Heroku Metrics: There and Back Again

Heroku Metrics: There and Back Again: "No system is perfect, and ours isn’t some magical exception. We’ve learned some things in this exercise that we think are worth pointing out.

Sharding and Partitioning Strategies Matter 

Our strategy of using the owner column for our shard/partitioning key was a bit unfortunate, and now hard to change. While we don’t currently see any ill effects from this, there are hypothetical situations in which this could pose a problem. For now, we have dashboards and metrics which we watch to ensure that this doesn’t happen and a lot of confidence that the systems we’ve built upon will actually handle it in stride.
Even still, a better strategy, likely, would have been to shard on owner + process_type (e.g. web), which would have spread the load more evenly across the system. In addition to the more even distribution of data, from a product perspective it would mean that in a partial outage, some of an application’s metrics would remain available.

Extensibility Comes Easily with Kafka 

The performance of our Postgres cluster doesn’t worry us. As mentioned, it’s acceptable for now, but our architecture makes it trivial to swap out, or simply add another data store to increase query throughput when it becomes necessary. We can do this by spinning up another Heroku app that uses shareable addons, starts consuming the summary topics and writes them to a new data store, with no impact to the Postgres store!
Our system is more powerful and more extensible because of Kafka."



'via Blog this'

A Two Month Debugging Story | Kevin Burke

A Two Month Debugging Story | Kevin Burke: "Instead we had to ship whatever logs we needed as artifacts from the test. This required turning on database logging for Postgres, then copying logs to the directory where artifacts get exported. Database logs across multiple containers shared the same filename, which meant only database logs from the first container became available as artifacts, so we also had to add a unique ID to each log file so they'd get exported properly.

This was a lot of steps, and took just enough debugging/setup time that I procrastinated for a while, hoping a better stack trace or error message might reveal itself, or that the problem would go away."



'via Blog this'

Agile is Dead – Again

Agile is Dead – Again: "Miko identifies Continuous Delivery as the successor to agile in the market.



 The paradigm of “Continuous Delivery (CD)” seems to be the logical successor to Agile. Continuous Delivery provides an umbrella term that does not specify methodology–and doesn’t require much of a manifesto. Everything you need to know is in the title–you just deliver shippable software in as continuous of a way as possible. This allows a team to pull whichever Agile principles and methods needed in order to achieve that goal. This addresses one of the complaints of Agile, which is that it became a religious movement with gurus–and that these highly paid Agile gurus would come with one-size-fits-all solutions for development teams which were hard to realistically fit to real-world development."



'via Blog this'

Scaling to 100M: MySQL is a Better NoSQL | Wix Engineering

Scaling to 100M: MySQL is a Better NoSQL | Wix Engineering: "The choice to use a NoSQL database is often based on hype, or a wrong assumption that relational databases cannot perform as well as a NoSQL database. Operational costs, as well as other stability and maturity concerns, are often overlooked by engineers when it comes to selecting a database. For more information about the limitations and shortcomings of different NoSQL (and SQL) engines, take a look at the Jepsen series of articles from Aphyr.
This post will explain why we’ve found that using MySQL for the key/value use case is better than most of the dedicated NoSQL engines, and provide guidelines to follow when using MySQL in this way."



'via Blog this'

Taking Docker to Production with Confidence | Voxxed

Taking Docker to Production with Confidence | Voxxed: "Many organizations developing software today use Docker in one way or another. If you go to any software development or DevOps conference and ask a big crowd of people “Who uses Docker?”, most people in the room will raise their hands. But if you now ask the crowd, “Who uses Docker in production?”, most hands will fall immediately. Why is it, that such a popular technology that has enjoyed meteoric growth is so widely used at early phases of the development pipeline, but rarely used in production?"



'via Blog this'

The InfoQ Podcast: Uber's Chief Systems Architect = No Node.js, no JSON over HTTP

The InfoQ Podcast: Uber's Chief Systems Architect on their Architecture and Rapid Growth: "In this week's podcast QCon chair Wesley Reisz talks to Matt Ranney who is the Chief Systems Architect at Uber, where he's helping build and scale everything he can. Previously, Matt was a founder and CTO of Voxer, probably the largest and busiest deployment of Node.js.

Key takeaways

Expanding a company and team at this rate is genuinely hard. Lots of mistakes have been made along the way.
Microservices allow companies to grow rapidly but have a cost in terms of aggregate velocity.
Uber is gradually moving its marketplace development from Node.js to Go and Java. Java is used for the map services.
Aggressive failure testing is used extensively in Uber.
Some early design choices - like using JSON over HTTP - make formal verification basically impossible."



'via Blog this'

Managing Technical Debt Using Total Cost of Ownership

Managing Technical Debt Using Total Cost of Ownership: "Total Cost of Ownership (TCO) can be used for investment decisions and financial benefit analysis. When applied to software it covers the initial development costs and subsequent maintenance costs until phase out of a product. TCO can support architectural decisions and management of technical debt.

Hans Sassenburg talked about total cost of ownership analysis at the Bits&Chips Software Engineering conference. In his talk Sassenburg showed several metaphors that can be used to "sell" the concept of TCO. The metaphor of technical dept is easy to understand, but people often find it difficult to see technical debt increasing during the product lifecycle and take action said Sassenburg.

InfoQ did an interview with Sassenburg about using the total cost of ownership concept for managing software development, the main causes of technical debt and how to reduce technical debt or prevent accumulation of debt, managing R&D investments and costs and improving the quality of software products."



'via Blog this'