Efficiency Over Speed: Getting More Performance Out of Kafka Consumer - SignalFx

Efficiency Over Speed: Getting More Performance Out of Kafka Consumer - SignalFx:



At the last Kafka meetup at LinkedIn in Mountain View, I presented some work we’ve done at SignalFx to get significant performance gains by writing our own consumer/client. This assume a significant baseline knowledge of how Kafka works. Note from the presentation are below along with the video embedded (start watching at 01:51:09). You can find the slides here (download for animations!).

The presentation was broken up into three parts:

• Why we wrote a consumer
• How modern hardware works
• Three optimizations on the consumer

Command Query Responsibility Segregation and Event Sourcing

CQRS: "Command Query Responsibility Segregation"



The change that CQRS introduces is to split that conceptual model into separate models for update and display, which it refers to as Command and Query respectively following the vocabulary ofCommandQuerySeparation. The rationale is that for many problems, particularly in more complicated domains, having the same conceptual model for commands and queries leads to a more complex model that does neither well.

By separate models we most commonly mean different object models, probably running in different logical processes, perhaps on separate hardware. A web example would see a user looking at a web page that's rendered using the query model. If they initiate a change that change is routed to the separate command model for processing, the resulting change is communicated to the query model to render the updated state.

There's room for considerable variation here. The in-memory models may share the same database, in which case the database acts as the communication between the two models. However they may also use separate databases, effectively making the query-side's database into a real-time ReportingDatabase. In this case there needs to be some communication mechanism between the two models or their databases.

The two models might not be separate object models, it could be that the same objects have different interfaces for their command side and their query side, rather like views in relational databases. But usually when I hear of CQRS, they are clearly separate models.

Lightbend's Lagom Will Run Java-Based Microservices at Scale - The New Stack

Lightbend's Lagom Will Run Java-Based Microservices at Scale - The New Stack:

Many microservice frameworks available today require developers to manually run scripts in order to start their services, or add an automated infrastructure. Lagom realized the need for developers to be able to utilize the tools they were used to, such as hot code reloads —Without having to install a new toolset locally in order to test an application. Lagom allows for developers to manage hundreds of services from a command line. From there, users are able to perform testing, update services via hot code fixes, and more.

Powering the Pieces

Lagom relies on a number of additional technologies, including the Lightbend’s own Play Framework, Akka, Akka Streams, Akka Clustering and ConductR for resilience and auto-scaling. By utilizing Netty, REST and WebSockets for communication with applications or devices that require access to its services, Lagom assures high performance. Finally, Lagom utilizes Cassandra as its persistence store.

Lightbend Reactive Platform Fast Data Architecture

For those looking to get a head start working with Lagom, be aware that the production release isn’t available until early March 2016. That being said, companies can check out Lightbend’s Reactive Platform to decompose any of their existing monolith applications into microservices, or to create new microservices.

“What we observed was that while many of these companies had unlimited engineering resources and talent, the far greater pool of Java developers needed a more opinionated framework — specifically for Java — that enabled the construction of microservices built to run and scale on the JVM,” said Hayes. It is this which stands as the reason Lightbend opted to start out by first creating a Java API for Lagom, followed by a Scala API.

With the full release of Lagom, the Java community may have a powerful new tool in its arsenal for creating, managing, and scaling microservices to meet the rigorous demands of today’s applications.

Reactive: Akka Streams, RxJava, Ratpack

How Akka Streams can be used to process the Wikidata dump in parallel and using constant memory with just your laptop.

Here at Intent HQ we use Wikipedia and Wikidata as sources of data. They are very important to us because they both encode an enormous amount of information in several languages that we use to build our Topic Graph.
Although the current process we have to process these dumps works well enough, we are always interested in finding new and better ways of doing our work. It’s because of that that we were very excited when we saw the Reactive Streams initiative ¹. We thought it could be used to process the largest encyclopædia in the world ².