Heroku Metrics: There and Back Again

Heroku Metrics: There and Back Again: "No system is perfect, and ours isn’t some magical exception. We’ve learned some things in this exercise that we think are worth pointing out.

Sharding and Partitioning Strategies Matter 

Our strategy of using the owner column for our shard/partitioning key was a bit unfortunate, and now hard to change. While we don’t currently see any ill effects from this, there are hypothetical situations in which this could pose a problem. For now, we have dashboards and metrics which we watch to ensure that this doesn’t happen and a lot of confidence that the systems we’ve built upon will actually handle it in stride.
Even still, a better strategy, likely, would have been to shard on owner + process_type (e.g. web), which would have spread the load more evenly across the system. In addition to the more even distribution of data, from a product perspective it would mean that in a partial outage, some of an application’s metrics would remain available.

Extensibility Comes Easily with Kafka 

The performance of our Postgres cluster doesn’t worry us. As mentioned, it’s acceptable for now, but our architecture makes it trivial to swap out, or simply add another data store to increase query throughput when it becomes necessary. We can do this by spinning up another Heroku app that uses shareable addons, starts consuming the summary topics and writes them to a new data store, with no impact to the Postgres store!
Our system is more powerful and more extensible because of Kafka."



'via Blog this'

A Two Month Debugging Story | Kevin Burke

A Two Month Debugging Story | Kevin Burke: "Instead we had to ship whatever logs we needed as artifacts from the test. This required turning on database logging for Postgres, then copying logs to the directory where artifacts get exported. Database logs across multiple containers shared the same filename, which meant only database logs from the first container became available as artifacts, so we also had to add a unique ID to each log file so they'd get exported properly.

This was a lot of steps, and took just enough debugging/setup time that I procrastinated for a while, hoping a better stack trace or error message might reveal itself, or that the problem would go away."



'via Blog this'

Understanding caching in Postgres - An in-depth guide - Madusudanan

Understanding caching in Postgres - An in-depth guide - Madusudanan: "Caching can be considered an important aspect in tuning database system performance.

While this post is mainly focused on postgres, it can be easily compared and understood with other database systems. "



'via Blog this'

Could PostgreSQL 9.5 be your next JSON database?

Could PostgreSQL 9.5 be your next JSON database?: "You can use PostgreSQL to create rich, complex JSON/JSONB documents within the database. But then if you are doing that, you may want to consider whether you are using PostgreSQL well. If the richness and complexity of those documents comes from relating the documents to each other then the relational model is often the better choice for data models that have intertwined data. The relational model also has the advantage that it handles that requirement without large scale duplication within the actual data. It also has literally decades of engineering expertise backing up design decisions and optimizations.

What JSON support in PostgreSQL is about is removing the barriers to processing JSON data within an SQL based relational environment. The new 9.5 features take down another barrier, adding just enough accessible, built-in and efficient functions and operators to manipulate JSONB documents.

PostgreSQL 9.5 isn't your next JSON database, but it is a great relational database with a fully fledged JSON story. The JSON enhancements arrive alongside numerous other improvements in the relational side of the database, "upsert", skip locking and better table sampling to name a few.

It may not be your next JSON database, but PostgreSQL could well be the next database you use to work with relational and JSON data side by side."



'via Blog this'

General database links

When is "ACID" ACID? Rarely.

tl;dr: ACID and NewSQL databases rarely provide true ACID guarantees by default, if they are supported at all. See the table.