Linux and Perfomance: Reducing latency spikes by tuning the CPU scheduler

Reducing latency spikes by tuning the CPU scheduler: "It turns out that this effect can be observed when scylla runs in a different cgroup from the java process, but not when they run in the same cgroup. The reason for this effect is yet to be discovered.



 To check to which group given process belongs to, one can run:

 cat /proc/$PID/cgroup

The cgroup subsystem named cpu is relevant for scheduling.



 A process can be forced under a certain cgroup at run time with cgclassify. For example, to force all scylla and java (scylla-jmx) processes under the root group (-w is important for the change to affect all threads):

 sudo cgclassify -g cpu:/ `pgrep -w scylla`

sudo cgclassify -g cpu:/ `pgrep -w java`



Linux has a feature called autogroup which will automatically associate each session with a different cgroup. Processes started in those sessions will inherit group affinity. Due to latency issues mentioned above, the scylla package will disable autogroup (See this commit).



 Forcing scylla and scylla-jmx under the same group and applying scheduler settings has the desired effect of reducing scheduling delays. Now the maximum delay is 1.8 ms:"



'via Blog this'

storage: A ZFS developer’s analysis of the good and bad in Apple’s new APFS file system | Ars Technica

A ZFS developer’s analysis of the good and bad in Apple’s new APFS file system | Ars Technica: "I'm not sure Apple absolutely had to replace HFS+, but likely they had passed an inflection point where continuing to maintain and evolve the 30+ year old software was more expensive than building something new. APFS is a product born of that assessment.

Based on what Apple has shown I'd surmise that its core design goals were:

• satisfying all consumers (laptop, phone, watch, etc.) 
• encryption as a first-class citizen 
• snapshots for modernized backup 

Those are great goals that will benefit all Apple users, and based on the WWDC demos APFS seems to be on track (though the macOS Sierra beta isn't quite as far along).



 In the process of implementing a new file system the APFS team has added some expected features. HFS was built when 400KB floppies ruled the Earth (recognized now only as the ubiquitous and anachronistic save icon). Any file system started in 2014 should of course consider huge devices, and SSDs—check and check. Copy-on-write (COW) snapshots are the norm; making the Duplicate command in the Finder faster wasn't much of a detour. The use case is unclear—it's a classic garbage can theory solution in search of a problem—but it doesn't hurt and it makes for a fun demo. The beach ball of doom earned its nickname; APFS was naturally built to avoid it.



 There are some seemingly absent or ancillary design goals: performance, openness, and data integrity. Squeezing the most IOPS or throughput out of a device probably isn't critical on watchOS, and it's relevant only to a small percentage of macOS users. It will be interesting to see how APFS performs once it ships (measuring any earlier would only misinform the public and insult the APFS team)."



'via Blog this'

The Fault in Our JARs: Why We Stopped Building Fat JARs

The Fault in Our JARs: Why We Stopped Building Fat JARs: "The Fault in Our JARs: Why We Stopped Building Fat JARs
JUN 16, 2016 / BY JONATHAN HABER

inShare
10
HubSpot’s backend services are almost all written in Java. We have over 1,000 microservices constantly being built and deployed. When it comes time to deploy and run one of our Java applications, its dependencies must be present on the classpath for it to work. Previously, we handled this by using the maven-shade-plugin to build a fat JAR. This takes the application and all of its dependencies and bundles them into one massive JAR. This JAR is immutable and has no external dependencies, which makes it easy to deploy and run. For years this is how we packaged all of our Java applications and it worked pretty well, but it had some serious drawbacks.

 Fat JAR Woes

 The first issue we hit is that JARs are not meant to be aggregated like this. There can be files with the same path present in multiple JARs and by default the shade plugin includes the first file in the fat JAR and discards the rest. This caused some really frustrating bugs until we figured out what was going on (for example, Jersey uses META-INF/services files to autodiscover providers and this was causing some providers to not get registered). Luckily, the shade plugin supports resource transformers that allow you to define a merge strategy when it encounters duplicate files so we were able to work around this issue. However, it’s still an extra "gotcha" that all of our developers need to be conscious of.

The other, bigger issue we ran into is that this process is slow and inefficient. Using one of our applications as an example, it contains 70 class files totalling 210KB when packaged as a JAR. But after running the shade plugin to bundle its dependencies, we end up with a fat JAR containing 101,481 files and weighing in at 158MB. Combining 100,000 tiny files into a single archive is slow. Uploading this JAR to S3 at the end of the build is slow. Downloading this JAR at deploy time is slow (and can saturate the network cards on our application servers if we have a lot of concurrent deploys)."



'via Blog this'

Python: Instagram Strikes a Sizable Blow in Silicon Valley’s Tabs Vs Spaces War | WIRED

Instagram Strikes a Sizable Blow in Silicon Valley’s Tabs Vs Spaces War | WIRED: "Rather than really moving away from Python or really trying to change the language, it found all sorts of small ways to tweak its Python code so that it could efficiently serve those 500 million people.

Most notably, using a tool called cprofile, Krieger and company worked to identify their slowest pieces of Python code. “We believe in measuring first before taking action,” says head of infrastructure Hui Ding. Then, using a second tool called Cython, they converted these pockets of slow code into C or C++. According to the company, this allows Instagram to run with 10 to 15 percent less processing power."



'via Blog this'

Scala thread overload meets Game of Thrones..

Remember, folks. Always recover your #scala Futures! (or have a human-friendly top-level handler) https://t.co/4UGC5pihxS

— Manuel Bernhardt (@elmanu) June 20, 2016

What the hell, @HBO @hbonow?!?#fail #GoT pic.twitter.com/jXFxW8qYng

— Josh Thomas (@jpthoma2) June 20, 2016

joejag/dotfiles: Every dev should have some dotfiles, these are mine

joejag/dotfiles: Every dev should have some dotfiles, these are mine: "Every dev should have some dotfiles, these are mine"



'via Blog this'

Everyday Git Aliases

Everyday Git Aliases: "Git gives you as much flexibility in how you construct your VCS workflow as it does for the commands you use on your local repo. In your gitconfig file you can add alises for your favourite commands, in this article I'll talk about mine. You can see my gitconfig on github."



'via Blog this'