Maintainable Python Applications: a Guide for Skeptical Java Developers · Code Without Rules

Maintainable Python Applications: a Guide for Skeptical Java Developers · Code Without Rules: "When you've been writing Java for a while switching to Python can make you a little anxious. Not only are you learning a new language with new idioms and tools, you're also dealing with a language with far less built-in safety. No more type checks, no more clear separation between public and private.

It's much easier to learn Python than Java, it's true, but it's also much easier to write unmaintainable code. Can you really build large scale, robust and maintainable applications in Python? I think you can, if you do it right.

The suggestions below will help get you started on a new Python project, or improve an existing project that you're joining. You'll need to keep up the best practices you've used in Java, and learn new tools that will help you write better Python"

'via Blog this'

infoQ: Architecture: Technologies for the Future of Software Engineering

Technologies for the Future of Software Engineering: "The Cloud, infrastructure as code, federated architectures with APIs and anti-fragile systems: these are technologies for developing software systems that are rapidly coming into focus, claimed Mary Poppendieck. She spoke about the future of software engineering at GOTO Berlin 2016.

If you have too much data to fit on one computer, you have two options: scale up or scale out. Scaling up by using a bigger computer is usually not the right direction to take, argued Poppendieck. You need to scale out by getting more computers and build systems of computers.

Poppendieck mentioned different ways to scale out:

Scale out files: an example is the approach that Google uses for searching. Files are split up into small pieces which are copied to multiple servers. Then the searching is done in parallel, and the results from all servers are combined into one search result.
Scale out architecture: Amazon does this by breaking transactions into services and using specific servers to do the services. If there’s a bottleneck, then you can replicate the service on multiple servers. Each service is owned by a semi-autonomous "two pizza" team.
Systems are moving more and more towards the cloud; there is a cloud in your future, said Poppendieck"

Having one central database creates a dependency problem as all applications depend on the database, and changing the database can impact many applications. Poppendieck stated that "the enterprise database is a massive dependency generator". Individual teams cannot deploy autonomously as their work has to be coordinated with all other teams sharing the database. Instead of having one database, you need a federated architecture where you split into local data stores appropriate to the needs of individual modules or services, and accessed only through APIs. APIs are replacing central shared databases and they enable the internet of things. You have to learn to do software engineering with APIs, argued Poppendieck. Look at APIs as products which are owned by a responsible team, and focus on the API customers to evolve and develop new capabilities.

We need to stop trying to make systems failure free and start thinking differently, said Poppendieck. Many of our current systems are fragile; they started as robust systems but over time they have become difficult to maintain. Systems nowadays need to be anti-fragile, and able to embrace failure, claimed Poppendieck. When things go wrong, systems should be capable of limiting damage and recovering from failures.

You get anti-fragile systems from the way you test systems, which is by failure injection to make something go wrong. Poppendieck mentioned that your systems need to isolate and automatically recover from failure, in order to achieve today’s expected levels of availability and robustness.

Poppendieck mentioned aspects that are essential nowadays for developing software. She said that you need a deployment pipeline to be able to do continuous delivery, and you need cross-functional teams including product management, testers, and operations to get the benefits promised by continuous delivery. The deployment pipeline depends on automated processes for testing, migration, and deployment. Continuous delivery requires all teams to communicate through the codebase by doing continuous integration to the trunk. Teams keep the software always production-ready; if that’s not the case you have to stop and make it so. While deployment is continuous, release is incremental by toggle or switch whenever a useful increment or capability is ready.

'via Blog this'

GitHub - heathermiller - Programming Models for Distributed Computation

GitHub - heathermiller/dist-prog-book: "Source repo for the book that I and my students in my course at Northeastern University, CS7680 Special Topics in Computing Systems: Programming Models for Distributed Computing, are writing on the topic of programming models for distributed systems.

This is a book about the programming constructs we use to build distributed systems. These range from the small, RPC, futures, actors, to the large; systems built up of these components like MapReduce and Spark. We explore issues and concerns central to distributed systems like consistency, availability, and fault tolerance, from the lens of the programming models and frameworks that the programmer uses to build these systems."

'via Blog this'

process - Why Agile Works

Why Agile Works: "In this podcast, recorded at Agile 2016, Shane Hastie, InfoQ Lead Editor for Culture & Methods, spoke to David Benz and Michael de la Maza about their minibook Why Agile Works, which is available at InfoQ.com."

'via Blog this'

non-O(n) musings: How to translate a large C project to Rust

non-O(n) musings: How to translate a large C project to Rust: "In October, I started working on translating CVS from C to Rust, and today I'd like to answer these questions about how that's going:
How does a Corrode-aided porting effort work, in practice?
How much progress have I made on CVS, and how hard would it be to do the same for other projects today?"

'via Blog this'

Death to JIRA | TechCrunch

Death to JIRA | TechCrunch: "I promised a better way. It is astonishingly simple. We already have an extremely powerful descriptive system which can be used to specify complex systems while including ambiguities, uncertainties, interwoven relationships, iterative levels of success, and an arbitrarily broad spectrum of scale and detail. It is called “prose.”

For some reason many companies today seem to be terrified of the prospect of writing more than a couple of paragraphs of clear and simple prose. But a well-written 8-page document can define the nuances of a complicated system far better than a whole cumbersome flotilla of interlinked JIRA tickets.

Indeed it’s easy to envision an automated system which takes a simple text document, parses it into sentences, bullet points, paragraphs, sections, and chapters, and tracks the estimation and progress of those elements — at whichever scale makes sense in that context — instead of tickets. (I suppose you could even automatically render them into JIRA tickets, if you really had to; but the point is that the master specification would be a single coherent prose document.)"

'via Blog this'

List of Consistency Models for Cloud Storage Services - CloudRail

List of Consistency Models for Cloud Storage Services - CloudRail

This article compares consistency models of all major cloud storage providers. The CloudRail Unified API makes integrating different cloud storage services into your code easy. It provides easy to use, abstracted interfaces that allow you to treat groups of those services similarly by using the exact same methods. Yet, the services are not 100% the same and one important distinctive attribute is the service’s consistency model. It can impact how you write your code and is thus something every developer should be aware of.:

'via Blog this'

build automation - SVN checkout ignore folder - Stack Overflow

build automation - SVN checkout ignore folder - Stack Overflow: "SVN checkout ignore folder"

'via Blog this'

C for Python Programmers

C for Python Programmers:



'via Blog this'

Picking an OpenStack Networking Solution | OpenStack Summit Videos

Picking an OpenStack Networking Solution | OpenStack Summit Videos: "OpenStack networking ecosystem is very rich with both open source and proprietary
solutions representing many different architectures to implement Neutron and networking services.

What are the high level differences between the various open source solutions? What should you look for
when picking a solution? Topics include L2/L3 gateways, L3 only networks, monitoring and debugging, containers, hybrid cloud, connecting legacy networks with new networks, NFV and service chaining, scaling, federation, and more."



'via Blog this'

Dockerizing the Hard Services: Neutron & Nova | OpenStack Summit Videos

Dockerizing the Hard Services: Neutron & Nova | OpenStack Summit Videos:

We'll talk about the benefits and pitfalls involved in successfully running complex services like Neutron and Nova inside of Docker containers.

Topics will include:

• What magic incantations are needed to run these services at all?
• How to prevent HA router failover on service restarts.
• How to prevent network namespaces from breaking everything.
• Lessons learned in how to manage configuration changes and deployments.

'via Blog this'

Docker: "I just want to run a container!" - Julia Evans

"I just want to run a container!" - Julia Evans: "here's the tl;dr:

•  you can run Docker containers without Docker 
• runC can run containers... but it doesn't have overlayfs 
• but overlay filesystems are important! 
• rkt has overlay filesystem support. 
• you need to start & supervise the containers! You can use any regular process supervisor to do that. 
• also you need to tell your computers which containers to run 
• software around the OCI standard is evolving but it's not there yet

As far as I can tell running containers without using Docker or Kubernetes or anything is totally possible today, but no matter what tools you use it’s definitely not as simple as "just run a container". Either way going through all these steps helps me understand what the actual components of running a container are and what all these different pieces of software are trying to do.

This landscape is pretty confusing but I think it’s not impossible to understand! There are only a finite number of different pieces of software to figure out the role of :)"



'via Blog this'

web: Understanding REST And RPC For HTTP APIs – Smashing Magazine

Understanding REST And RPC For HTTP APIs – Smashing Magazine: "Knowing the differences between REST and RPC can be incredibly useful when you are planning a new API, and it can really help when you are working on features for existing APIs. It’s best not to mix styles in a single API, because this could be confusing both to consumers of your API as well as to any tools that expect one set of conventions (REST, for example) and that fall over when it instead sees a different set of conventions (RPC). Use REST when it makes sense, or use RPC if it is more appropriate. Or use both and have the best of both worlds!"



'via Blog this'

web python: httpstat: Simple cURL Stats

httpstat: Simple cURL Stats: "I recently discovered httpstat, which is a single python file which (simply) graphically displays request event timing, from DNS lookup to content transfer:"



'via Blog this'

The SRE book in a comic pic.twitter.com/XJccRfFWnK

— Alice Goldfuss (@alicegoldfuss) September 23, 2016

BCC - Dynamic Tracing Tools for Linux Performance Monitoring, Networking and More

BCC - Dynamic Tracing Tools for Linux Performance Monitoring, Networking and More: "BCC (BPF Compiler Collection) is a powerful set of appropriate tools and example files for creating resourceful kernel tracing and manipulation programs. It utilizes extended BPF (Berkeley Packet Filters), initially known as eBPF which was one of the new features in Linux 3.15."



'via Blog this'

How Intel's open source Data Plane Development Kit enables high-performance Linux networking - TechRepublic

How Intel's open source Data Plane Development Kit enables high-performance Linux networking - TechRepublic: "To utilize processors more efficiently in Linux, developers bypass the Linux kernel and dedicate CPU cores to network processing. The CloudFlare example used Linux kernel tools to pin processes to a set of cores and actively managed memory queues. It took a bit of programming discipline to send unprocessed packets between two systems. Add any packet processing, such as IPS, and the performance should drop significantly.

 DPDK bypasses the Linux kernel and manages the NIC and CPU assignment directly. On a single Intel processor, it's possible to achieve throughput of up to 80Mpps or roughly 46Gbps using the smallest Ethernet Frames. I want to make sure I caveat the above comment and note that it was an Intel processor. DPDK is an open source project and currently has both Intel and ARM distributions."



'via Blog this'

Storage: How to roll back a btrfs snapshot and restore corrupt or missing files - TechRepublic

How to roll back a btrfs snapshot and restore corrupt or missing files - TechRepublic: "The btrfs (often called Butter FS) is a modern copy on write filesystem that offers some seriously handy features and focuses on fault tolerance and repair. The developers of btrfs have also gone to great lengths to make it easy to use. No other aspect of btrfs highlights all of this more than the ability to take snapshots of a subvolume and roll back to a previous working state.

I'll show how easy it is to do just that on an external flash drive (for the sake of simplicity). You might want to test this process on a similar device before you're ready to take this to a production machine."



'via Blog this'

Thomas Graf: Linux Networking Explained

Linux Networking Explained:



'via Blog this'

Linux Takes Over the World... SDN and NFV

Linux Takes Over the World, or at the Least, Becomes Quite Influe: "Open source projects are new to networking, but they’ve been cropping up all over the place in the last couple of years. And many of them are gravitating toward the Linux Foundation.

Some of them were originally independent groups. The Open Network Operating System (ONOS) for example, was founded by On.lab. But in October, it became part of the Linux Foundation. The Linux Foundation was already hosting the OpenDaylight Project, which some considered a rival to ONOS. But the two groups seem to be happily coexisting under the same host."



'via Blog this'

What Golang Is and Is Not

What Golang Is and Is Not: "“Innovative” is such an overused and abused word that it has lost a lot of power and meaning. All innovation is contextual and to use the word without context is foolhardy. In the context of language design Go was never an innovative language, nor was it presented as such, or anyone dishonest in representing it that way.

As a language Go was always explicitly a return to simplicity, and in many ways naivety, for sound reasons.

“There is nothing new under the sun” rings true in all languages since the 80’s.

Virtually everything we see in language design now that someone says is “innovative” has been explored in some form before. Go is certainly no exception, but remember it never claimed to be state of the art.

Regarding the language being youthful, of course it is, but the intention is not for the language itself to ‘mature’: no more complexity is going to be added, or at least it’s very unlikely.

It is not ‘missing’ comprehensions, or inheritance, or generics, they are omitted (and I pray, always will be). In some way, in the context of the current fashion of returning to more functional languages, or the evolution of good old languages to include more functional paradigms (I’m looking at you Javascript and Python for two examples) then in a tenuous convoluted way Go has ‘innovated’ by avoiding that trend."



'via Blog this'

Linux kernel tracing

Linux kernel tracing: "Linux kernel tracing"



'via Blog this'

Web: Top 10 Mistakes Node.js Developers Make

Top 10 Mistakes Node.js Developers Make: "Node.js has seen an important growth in the past years, with big companies such as Walmart or PayPal adopting it. More and more people are picking up Node and publishing modules to NPM at such a pace that exceeds other languages. However, the Node philosophy can take a bit to get used to, especially if you have switched from another language.
In this article we will talk about the most common mistakes Node developers make and how to avoid them. You can find the source code for the examples on github."



'via Blog this'

10 Myths of Enterprise Python | PayPal Engineering Blog

10 Myths of Enterprise Python | PayPal Engineering Blog: "Python has enjoyed many years of grassroots usage and support from developers across eBay. Even before official support from management, technologists of all walks went the extra mile to reap the rewards of developing in Python. I joined PayPal a few years ago, and chose Python to work on internal applications, but I’ve personally found production PayPal Python code from nearly 15 years ago.

Today, Python powers over 50 projects, .

For this introductory post, I’ll be focusing on the 10 myths I’ve had to debunk the most in eBay and PayPal’s enterprise environments."



'via Blog this'

Linux/C: gdb Debugging Full Example (Tutorial): Brendan Gregg

gdb Debugging Full Example (Tutorial): ncurses: "I'm a little frustrated with finding "gdb examples" online that show the commands but not their output. gdb is the GNU Debugger, the standard debugger on Linux. I was reminded of the lack of example output when watching the Give me 15 minutes and I'll change your view of GDB talk by Greg Law at CppCon 2015, which, thankfully, includes output!

It's well worth the 15 minutes.

It also inspired me to share a full gdb debugging example, with output and every step involved, including dead ends. This isn't a particularly interesting or exotic issue, it's just a routine gdb debugging session. But it covers the basics and could serve as a tutorial of sorts, bearing in mind there's a lot more to gdb than I used here.

 I'll be running the following commands as root, since I'm debugging a tool that needs root access (for now). Substitute non-root and sudo as desired. You also aren't expected to read through all this: I've enumerated each step so you can browse them and find ones of interest."



'via Blog this'

Java & C++ : More JVM Signal tricks - Thread control via mprotect · Jackson Davis

More JVM Signal tricks - Thread control via mprotect · Jackson Davis: "In my last post, I mentioned the JVM uses intentionally uses SIGSEGVS in other interesting ways. In this post I’ll give an overview of one of those other ways, synchronization for GC pauses.

If you ever look at generated assembly from Hotspot, you will find methods ending with very odd-looking tests like this:

 test %eax,0x16e71fa4(%rip) # 0x00007fce84071000

At first glance, this doesn’t seem to serve any real purpose right before the return. Rather than doing any useful work, its just a way of trying to read from 0x7fce84071000, the designated polling page. Modifying access to this page lets the JVM stop threads cleanly - in places where the state of the world is well-known."



'via Blog this'

Deciphering Glyph :: Python Packaging Is Good Now

Deciphering Glyph :: Python Packaging Is Good Now: "The real takeaway here though, is that although it’s still not perfect, other languages are no longer doing appreciably better.

 Go is still working through a number of different options regarding dependency management and vendoring, and, like Python extensions that require C dependencies, CGo is sometimes necessary and always a problem. Node has had its own well-publicized problems with their dependency management culture and package manager. Hackage is cool and all but everything takes a literal geological epoch to compile.

As always, I’m sure none of this applies to Rust and Cargo is basically perfect, but that doesn’t matter, because nobody reading this is actually using Rust.



My point is not that packaging in any of these languages is particularly bad. They’re all actually doing pretty well, especially compared to the state of the general programming ecosystem a few years ago; many of them are making regular progress towards user-facing improvements.

My point is that any commentary suggesting they’re meaningfully better than Python at this point is probably just out of date. Working with Python packaging is more or less fine right now. It could be better, but lots of people are working on improving it, and the structural problems that prevented those improvements from being adopted by the community in a timely manner have almost all been addressed.
Go! Make some virtualenvs! Hack some setup.pys! If it’s been a while and your last experience was really miserable, I promise, it’s better now.
"



'via Blog this'

NUMA Deep Dive Part 1: From UMA to NUMA - frankdenneman.nl

NUMA Deep Dive Part 1: From UMA to NUMA - frankdenneman.nl: "Non-uniform memory access (NUMA) is a shared memory architecture used in today's multiprocessing systems. Each CPU is assigned its own local memory and can access memory from other CPUs in the system. Local memory access provides a low latency - high bandwidth performance. While accessing memory owned by the other CPU has higher latency and lower bandwidth performance. Modern applications and operating systems such as ESXi support NUMA by default, yet to provide the best performance, virtual machine configuration should be done with the NUMA architecture in mind. If incorrect designed, inconsequent behavior or overall performance degradation occurs for that particular virtual machine or in worst case scenario for all VMs running on that ESXi host.

 This series aims to provide insights of the CPU architecture, the memory subsystem and the ESXi CPU and memory scheduler. Allowing you in creating a high performing platform that lays the foundation for the higher services and increased consolidating ratios. Before we arrive at modern compute architectures, it's helpful to review the history of shared-memory multiprocessor architectures to understand why we are using NUMA systems today."



'via Blog this'

Where the Database Market Goes From Here – tecosystems

Where the Database Market Goes From Here – tecosystems: "Given that the database market is subject to the same market forces as other enterprise categories, on premises software both specialized and general purpose is likely to be a tightening market over time. There is a great deal of revenue to be had in the category, without question, but it will be more difficult to obtain as there is more competition generally, more competition from open source specifically and on premises alternatives increasingly compete directly with service based alternatives.



These price pressures are one reason vendors are increasingly moving back towards general purpose datastores from specialized roots: the broader the functional capabilities, the wider the addressable market, at least in theory.

Even long time database incumbents, however, are scrambling to develop or acquire their way into service-based businesses because that is where much of the growth will occur. Whether adopted as more convenient stand-alone alternatives to on premises databases or deployed in conjunction with other cloud infrastructure, DBaaS offerings are attractive to both developers and their employers, if for very different reasons. Importantly, this is the case in spite of the fact that the DBaaS market is in its infancy; many popular databases are not yet available as services, and those that are don’t yet have the provider choice that they will ultimately. Which implies that the DBaaS market has been successful, to a degree, in spite of itself.

From a provider perspective, then, a choice is implied.



The existing spend on on premises relational solutions is measured in tens of billions of dollars, which is why many database providers today still regard their primary market competition as Oracle, even if they’re selling non-relational solutions. Vendors focused on trajectory, however, tend to see Amazon as the more important target, given that the most common report when talking to purveyors of on premises software is that a significant percentage of their existing customers are already in the cloud and most of those are on Amazon.

Implied choice or not, however, a legitimate market approach is also not to choose.



On premises providers in most cases will need to follow the lead of players like MongoDB, because competing with DBaaS players without an as-a-service option is a non-starter. Worse, competitors that operate as-a-service businesses have an enormous intelligence advantage over purely on premises competitors, thanks to the difference in available operational telemetry. But neither should on premises vendors deprecate their existing business; instead, they can differentiate from pure play as-a-service options by offering customers their choice of running in an existing datacenter or in the cloud.

 What is clear, however, is that a status quo approach in this market is one that will lead to diminishing returns over time. Choose your path carefully."



'via Blog this'

Don’t be Delta: Four pillars of high availability | InfoWorld

Don’t be Delta: Four pillars of high availability | InfoWorld: "Obviously no matter how great your software and servers, if you don't have redundant network infrastructure, you may not survive. This starts with your switches and routers, includes such equipment as load balancers, and goes a bit beyond. For instance, backup power -- from a reliable provider that routes your power appropriately.

High availability (HA) protects against such problems as a server or service failing.



Let's say the runtime your web service is built on does a core dump. No user should know -- they should simply be routed to another service. To accomplish this, you need to run multiple instances and load balancers, then replicate appropriately.

Disaster recovery (DR) protects against more unlikely events. Often, DR is implemented with relatively low expectations (sometimes hours or even days before recovery) and/or acceptable data loss. I’d argue that on the modern internet, recovery should be minutes at worst -- same goes for data accessibility. To accomplish DR, you need another datacenter that’s geographically distant and an ironclad failover scheme.



 You also need to get the data there.

If you use a cloud hosting provider like AWS, opt for servers in multiple regions in case disaster strikes. Some larger organizations go as far as using more than one cloud infrastructure provider."



'via Blog this'

Python: Please Fix Your Decorators · Homepage of Hynek Schlawack

Please Fix Your Decorators · Homepage of Hynek Schlawack: "If your Python decorator unintentionally changes the signatures of my callables or doesn’t work with class methods, it’s broken and should be fixed. Sadly most decorators are broken because the web is full of bad advice."



'via Blog this'

career: Five-essential-phone-screen-questions - steveyegge2

five-essential-phone-screen-questions - steveyegge2: "You have to probe all five areas; you can't skip any of them. Each area is a proxy for a huge body of knowledge, and failing it very likely means failing the interviews, even though the candidate did fine in the other areas.

Without further ado, here they are: The Five Essential Questions for the first phone-screen with an SDE candidate:



 1) Coding. The candidate has to write some simple code, with correct syntax, in C, C++, or Java.

 2) OO design. The candidate has to define basic OO concepts, and come up with classes to model a simple problem.

3) Scripting and regexes. The candidate has to describe how to find the phone numbers in 50,000 HTML pages.

 4) Data structures. The candidate has to demonstrate basic knowledge of the most common data structures.

5) Bits and bytes. The candidate has to answer simple questions about bits, bytes, and binary numbers. 




Please understand:   what I'm looking for here is a total vacuum in one of these areas. It's OK if they struggle a little and then figure it out. It's OK if they need some minor hints or prompting. I don't mind if they're rusty or slow. What you're looking for is candidates who are utterly clueless, or horribly confused, about the area in question.



 For example, you may find a candidate who decides that a Vehicle class should be a subclass of ParkingGarage, since garages contain cars. This is just busted, and it's un-fixable in any reasonable amount of training time.

Or a candidate might decide, when asked to search for phone numbers in a bunch of text files, to write a 2000-line C++ program, at which point you discover they've never heard of "grep", or at least never used it.



 When a candidate is totally incompetent in one of these Big Five areas, the chances are very high that they'll bomb horribly when presented with our typical interview questions. Last week I interviewed an SDE-2 candidate who made both of the mistakes above (a vehicle inheriting from garage, and the 2000-line C++ grep implementation.) He was by no means unusual, even for the past month. We've been bringing in many totally unqualified candidates.

The rest of this document describes each area in more detail, and gives example questions"



'via Blog this'

C/C++: Linkers and Loaders

Linkers and Loaders: "These are the manuscript chapters for my Linkers and Loaders, published by Morgan-Kaufman. See the book's web site for ordering information.
The text in these files is the unedited original manuscript. M-K has fine copy editors, who have fixed all the minor typos, spelling, and grammar errors in the printed book, but if you come across factual errors I'd still appreciate help getting all the details of linking and loading as complete and correct as possible. I will collect errata and fix them in subsequent printings.

The figures here are scans of hand-drawn sketches which have been redrawn for the book. You don't need to tell me I'm a lousy artist. I already know that."



'via Blog this'

Non-volatile Storage | the morning paper

Non-volatile Storage | the morning paper: "This invalidation is due to the arrival of high-speed, non-volatile storage devices, typically referred to as storage class memories (SCM). The performance of an SCM, at hundreds of thousands of iops, requires one or more many-core CPUs to saturate it. The most visible type of SCMs today are PCIe SSDs (SSDs attached via the PCI-express bus). Beyond PCI SSDs there are NVDIMMS, which have the performance characteristics of DRAM while simultaneously offering persistence – these tend to couple DRAM and flash on a DIMM and use a super-capacitor to provide enough power to flush the volatile contents of DRAM out to disk on a loss of power. And then of course there’s the even cheaper battery-backed distributed UPS solutions that we looked at yesterday…"



'via Blog this'

Simple Sketches for Diagramming Your Software Architecture | Voxxed

Simple Sketches for Diagramming Your Software Architecture | Voxxed: "If you’re working in an agile software development team at the moment, take a look around at your environment. Whether it’s physical or virtual, there’s likely to be a story wall or Kanban board visualising the work yet to be started, in progress and done. Visualising your software development process is a fantastic way to introduce transparency because anybody can see, at a glance, a high-level snapshot of the current progress.



 As an industry, we’ve become adept at visualising our software development process over the past few years – however, it seems we’ve forgotten how to visualise the actual software that we’re building. I’m not just referring to post-project documentation. This also includes communication during the software development process. Agile approaches talk about moving fast, and this requires good communication, but it’s surprising that many teams struggle to effectively communicate the design of their software."

'via Blog this'

Linux/Cloud: Puppet Enterprise Targets Application Orchestration

Puppet Enterprise Targets Application Orchestration: "Puppet Labs, known for configuration management of servers recently announced support for Application Management and Orchestration.

With the distributed nature of projects and advent of Microservices, applications need to be carefully orchestrated across multiple servers or Virtual Machines. Puppet Enterprise reduces the complexity of application delivery and deployment."



'via Blog this'

BGP and the System of Trust that Runs the Internet Pt. 1 - OpenDNS Umbrella Blog

BGP and the System of Trust that Runs the Internet Pt. 1 - OpenDNS Umbrella Blog: "What most Internet users don’t know, but many are beginning to discover, is the entire framework that holds the Internet together can be quite fragile. In fact the system in charge of correctly routing Internet traffic is largely built on a collective trust, or as The Washington Post recently described it, an ‘honor code’ of the Internet.



 The fact is we all rely on core protocols of the Internet like BGP (border gateway protocol) and DNS (domain name services) every day. They are a foundation of the Internet. Even companies that provide content on the Internet but don’t need BGP services directly, still rely on the service to get that content to their customers.



 This two-part series will cover BGP, what it is, how it works, and what global moving parts are involved. Part two of this series, publishing next week, will cover the technical and security issues surrounding BGP, and why they often have a drastic effect on a global scale."



'via Blog this'

Bruce Davie: Is SDN Necessary?

Is SDN Necessary?:



'via Blog this'

Security: Understanding and Hardening Linux Containers

Understanding and Hardening Linux Containers: "Operating System virtualisation is an attractive feature for efficiency, speed and modern application deployment, amid questionable security. Recent advancements of the Linux kernel have coalesced for simple yet powerful OS virtualisation via Linux Containers, as implemented by LXC, Docker, and CoreOS Rkt among others. Recent container focused start-ups such as Docker have helped push containers into the limelight. Linux containers offer native OS virtualisation, segmented by kernel namespaces, limited through process cgroups and restricted through reduced root capabilities, Mandatory Access Control and user namespaces. This paper discusses these container features, as well as exploring various security mechanisms. Also included is an examination of attack surfaces, threats, and related hardening features in order to properly evaluate container security. Finally, this paper contrasts different container defaults and enumerates strong security recommendations to counter deployment weaknesses-- helping support and explain methods for building high-security Linux containers. Are Linux containers the future or merely a fad or fantasy? This paper attempts to answer that question."



'via Blog this'

Linux/Unix: Understanding the bin, sbin, usr/bin , usr/sbin split

Understanding the bin, sbin, usr/bin , usr/sbin split: "On Tuesday 30 November 2010 15:58:00 David Collier wrote:
> I see that busybox spreads it's links over these 4 directories.
>
> Is there a simple rule which decides which directory each link lives
> in.....
>
> For instance I see kill is in /bin and killall in /usr/bin.... I don't
> have a grip on what might be the logic for that."



'via Blog this'

Extracting The Python Stack Using Ptrace

Extracting The Python Stack Using Ptrace: "I recently wrote a program that I think is pretty nifty, called Pystack. Pystack let's you ptrace(2) attach to an arbitrary running Python process and see the currently executing stack trace."



'via Blog this'

Google: Linux Shell Style Guide

Shell Style Guide:



"Which Shell to Use :   Bash is the only shell scripting language permitted for executables.
Executables must start with #!/bin/bash and a minimum number of flags.

Use set to set shell options so that calling your script as bash does not break its functionality.

Restricting all executable shell scripts to bash gives us a consistent shell language that's installed on all our machines.

The only exception to this is where you're forced to by whatever you're coding for. One example of this is Solaris SVR4 packages which require plain Bourne shell for any scripts.



 When to use Shell

link :  Shell should only be used for small utilities or simple wrapper scripts.
While shell scripting isn't a development language, it is used for writing various utility scripts throughout Google. This style guide is more a recognition of its use rather than a suggestion that it be used for widespread deployment.

Some guidelines:

•  If you're mostly calling other utilities and are doing relatively little data manipulation, shell is an acceptable choice for the task. 
• If performance matters, use something other than shell. 
• If you find you need to use arrays for anything more than assignment of ${PIPESTATUS}, you should use Python. 
• If you are writing a script that is more than 100 lines long, you should probably be writing it in Python instead. 

Bear in mind that scripts grow. Rewrite your script in another language early to avoid a time-consuming rewrite at a later date."



'via Blog this'

eklitzke.org: C++ and Thoughts On Java, Go, and Rust

C++ and Thoughts On Java, Go, and Rust: "Historically most programming situations that called for high performance, particularly in the field of systems programming, would have been written in C or C++. That has changed a lot over the last twenty five years or so. Java has increasingly been the programming language of choice for high performance server applications. Recently Go and Rust have made inroads as high performance alternatives to Java that claim to offer similar or better performance. Yet I still find that I really love writing C++, and at least for situations where speed or memory utilization are important I prefer writing C++ to any other language. In this post I will explore some of the aspects of C++ that I find appealing, and why Java, Go, and Rust don't make the cut for me."



'via Blog this'

Linux: Inline C/Assembly In Bash

Inline C/Assembly In Bash: "Recently I had the idea of trying to combine two of my favorite things: bash and assembly. My original idea was to try to make a way to inline assembly code into bash scripts. While I was researching this project I discovered Tavis Ormandy's ctype.sh. This is a really clever project that makes use of the nearly undocumented enable -f feature in bash to add new shell builtins that expose the functionality provided by dlopen(3). The project is extremely ingenious, and I would recommend browsing the code to anyone who interested in these kinds of low level things."



'via Blog this'

Good logging practice in Python « Victor Lin's coding notes

Good logging practice in Python « Victor Lin's coding notes: "In reality, logging is important. When you transfer money, there are transfer records. When an airplane is flying, black box (flight data recorder) is recording everything. If something goes wrong, people can read the log and has a chance to figure out what happened. Likewise, logging is important for system developing, debugging and running. When a program crashes, if there is no logging record, you have little chance to understand what happened. For example, when you are writing a server, logging is necessary."



'via Blog this'

I wish I knew these things when I learned Python | Bugra Akyildiz

I wish I knew these things when I learned Python | Bugra Akyildiz:



"I sometimes found myself asking myself how I cannot know simpler way of doing “this” thing in Python 3. When I seek solution, I of course find much more elegant, efficient and more bug-free code parts over time. In total(not just this post), the total sum of “those” things were far more than I expect/admit, but here is the first crop of features that was not obvious to me and learned later as I sought more efficient/simple/maintainable code."



'via Blog this'

Five Ways to Not Mess Up Microservices in Production

Five Ways to Not Mess Up Microservices in Production: "Ealier this month Takipi's Alex Zhitnitsky wrote about five ways not to mess up microservices in production (OK, "mess" wasn't exactly the word he used!). He builds on the output of the panel discussion we reported on a few months back as well as their own users, examining the top production problems with microservices and how to solve them. The emphasis, as we will see, is on distributed debugging and approaches which can help to make this a tractable effort."



'via Blog this'

Destroy All Ifs — A Perspective from Functional Programming – John A De Goes

Destroy All Ifs — A Perspective from Functional Programming – John A De Goes:



"I’ve long argued that, despite all its flaws, one of the great things about functional programming is its intrinsic inversion of control.
In an impure imperative language, if you call some function void doX(State *state), the function can do anything it wants. It can modify state (both passed in and global), it can delete files, it can read from the network, and it can even launch nukes!



In a pure functional language, however, if you call some function doX :: State -> IO (), then at most it’s going to return a value. It can’t modify what you pass it, and if you like, you can ignore the value returned by the function, in which case calling the function has no effect (aside from sucking up a little CPU and RAM).



Now granted, IO in Haskell is not a strong example of an abstraction that has superior reasoning properties, but the fundamental principle is sound: in pure functional programming languages, the control is inverted to the caller.
Thus, as you’re looking at the code, you have a clearer sense of what the functions you are calling can do, without having to unearth their foundations.



 I think this generalizes to the following principle:
Code that inverts control to the caller is generally easier to understand.
(I actually think this is a specialization of an even deeper principle — that code which destroys information is generally harder to reason about than code which does not.)



Viewed from this angle, you can see that if we embed conditionals deep into functions, and we call those functions, we have lost a certain amount of control: we feed in inputs, but they are combined in arbitrary and unknown ways to arrive at a decision (which code path to take) that has humongous ramifications on how the program behaves.
It’s no wonder that conditionals (and with them, booleans) are so widely despised!



 In an object-oriented programming language, it’s generally considered a good practice to replace conditionals with polymorphism.
In a similar fashion, in functional programming, it’s often considered good practice to replace boolean values with algebraic data types."



'via Blog this'

What kubernetes and cloud9 tell us about the new industry – anyone but Amazon – James Governor's Monkchips

What kubernetes and cloud9 tell us about the new industry – anyone but amazon – James Governor's Monkchips: "We used to talk about the Anyone but Microsoft club. Arguably the triumph of open source in the enterprise was a result of the same driver. Linux was a means to avoid Microsoft operating system domination. Vendors would push an alternative, and customers would support it partly as means to hedge their bets against too much domination by a single provider.

Today the dominant vendor scaring tech providers is clearly Amazon Web Services.

One facet of today’s Anyone but Amazon coalition is OpenStack. AWS dominance led pretty much every other major tech vendor, no matter how competitive to converge on OpenStack, as an open hedge.

When pondering the implications of Microsoft’s hiring of Brendan Burns the other day, it struck me another coalition is forming, changing the fault lines of the industry. Burns is one of the founders of the Kubernetes container cluster manager project. He was at Google but just took a job as product lead for the Azure Resource Manager. He has already publicly declared he will continue to work on Kubernetes. Kubernetes has also been enthusiastically adopted by Red Hat, through it’s OpenShift platform.

So now Microsoft, Red Hat, and Google Cloud Platform are all now aligned around Kubernetes. While at first glance this new alliance of strange bedfellows might seem to be a response to the rise of Docker and the Docker Pattern – and indeed there is no doubt the enthusiastic growth in Kubernetes is partly driven by concern that Docker will own too much real estate of the new infrastructure world I believe the overarching threat is Amazon.

As Stephen has explained – the biggest competitor to open source is Amazon. There is no doubt that Amazon EC2 Container Services is going to gain wide traction. Amazon can afford to be magnanimous about Docker’s rise in a way other vendors can’t. Docker is an implementation detail rather than a potential existential threat to AWS.

This week another shoe dropped, when Amazon announced it is acquiring Cloud9, the online IDE startup. Cloud9 created Ace, which also powers the GitHub editor."



'via Blog this'

Microsoft REST API Guidelines

api-guidelines/Guidelines.md at master · Microsoft/api-guidelines · GitHub:



"The Microsoft REST API Guidelines, as a design principle, encourages application developers to have resources accessible to them via a RESTful HTTP interface. To provide the smoothest possible experience for developers on platforms following the Microsoft REST API Guidelines, REST APIs SHOULD follow consistent design guidelines to make using them easy and intuitive.

This document establishes the guidelines Microsoft REST APIs SHOULD follow so RESTful interfaces are developed consistently."



'via Blog this'

Why Cisco's data center strategy depends on simplifying storage - TechRepublic

Why Cisco's data center strategy depends on simplifying storage - TechRepublic: "Cisco recently announced a MapR-based SAP HANA appliance. MapR is described as a converged data platform, and Cisco is leveraging the platform to create a dedicated x86-based SAN for the SAP HANA persistent layer.



A major difference between HyperFlex and the MapR-based SAP HANA appliance is the storage and compute nodes are not mixed. MapR creates a dedicated layer for storage separate from the application servers.

Cisco's storage strategy has a complicated history. The current set of technologies is focused on specific verticals. However, Cisco needs to place some consistency behind a single strategy before customers buy wholesale into a Cisco-only data center strategy."



'via Blog this'

Adam Leventhal's blog » I Love Go; I Hate Go

Adam Leventhal's blog » I Love Go; I Hate Go: "I’m sure time will, as it tends to, bring pragmatism. I appreciate that Go does have strong opinions (it’s just hard to remember that when I disagree with them). Weak opinions are what turn languages into unreadable mishmashes of overlapping mechanism. My favorite example of this is Perl. My first real programming job was in Perl. I was the most avid reader teenage of the Perl llama and camel books.



Ask me about my chat with Larry Wall, Perl’s creator, if you see a beer in my hand. In an interview, Larry said, “In Perl 6, we decided it would be better to fix the language than fix the user”.



Contrast this with Go’s statement on assertions:

Go doesn’t provide assertions. They are undeniably convenient, but our experience has been that programmers use them as a crutch to avoid thinking about proper error handling and reporting.

Perl wants to be whatever the user wants to be. The consumate pleaser. Go sees the user as flawed and the strictures of the language as the cure. It’s an authoritarian, steadfast in its ideals, yet too sensitive to find compromise (sound like anyone we all know?).

Despite my frustrations I really enjoy writing code in Go. It’s clean and supported by a great community and ecosystem."



'via Blog this'

Making OpenStack production ready with Kubernetes and OpenStack-Salt - Part 1 - Superuser

Making OpenStack production ready with Kubernetes and OpenStack-Salt - Part 1 - Superuser: "This tutorial introduces and explains how to build a workflow for life cycle management and operation of an enterprise OpenStack private cloud coupled with OpenContrail SDN running in Docker containers and Kubernetes.

The following blog post is divided into five parts, the first is an explanation of a deployment journey into a continuous DevOps workflow. The second offers steps on how to build and integrate containers with your build pipeline.



The third part details the orchestration of containers with a walkthrough of Kubernetes architecture, including plugins and prepping OpenStack for decomposition. In the fourth part, we introduce the tcp cloud theory of a “single source of truth” solution for central orchestration. In the fifth and final step we bring it all together, demonstrating how to deploy and upgrade of OpenStack with OpenContrail.

We decided to divide the process into two blog posts for better reading. This first post covers creating a continuous DevOps workflow and containers build."



'via Blog this'

Good to Great Python reads — jesse noller

Good to Great Python reads — jesse noller: "A COLLECTION OF PYTHON "MUST READS":"



'via Blog this'

Why Uber Engineering Switched from Postgres to MySQL - Uber Engineering Blog

Why Uber Engineering Switched from Postgres to MySQL - Uber Engineering Blog: "The early architecture of Uber consisted of a monolithic backend application written in Python that used Postgres for data persistence. Since that time, the architecture of Uber has changed significantly, to a model of microservices and new data platforms. Specifically, in many of the cases where we previously used Postgres, we now use Schemaless, a novel database sharding layer built on top of MySQL. In this article, we’ll explore some of the drawbacks we found with Postgres and explain the decision to build Schemaless and other backend services on top of MySQL."



'via Blog this'

Cloud: OpenStack will soon be able to run in containers on top of Kubernetes | TechCrunch

OpenStack will soon be able to run in containers on top of Kubernetes | TechCrunch: "OpenStack, the open source project that allows enterprises to run an AWS-like cloud computing service in their own data centers, added support for containers over the course of its last few releases. Running OpenStack itself on top of containers is a different problem, though.



Even though CoreOS has done some work on running OpenStack in containers thanks to its oddly named Stackanetes project, that project happened outside of the OpenStack community and the core OpenStack deployment and management tools.



 Soon, however, thanks to the work of Mirantis, Google and Intel, the OpenStack Fuel deployment tool will be able to use Kubernetes as its orchestration engine, too. Ideally, this will make it easier to manage OpenStack deployments at scale.



 “With the emergence of Docker as the standard container image format and Kubernetes as the standard for container orchestration, we are finally seeing continuity in how people approach operations of distributed applications,” said Mirantis CMO Boris Renski. “Combining Kubernetes and Fuel will open OpenStack up to a new delivery model that allows faster consumption of updates, helping customers get to outcomes faster.”



 This also means that OpenStack will soon be able to run in containers on Google’s Cloud — or really any cloud service that supports Kubernetes."



'via Blog this'

Linux: bcc tutorial by Brendan Gregg· iovisor/bcc

bcc/tutorial.md at master · iovisor/bcc · GitHub: "bcc Tutorial

This tutorial covers how to use bcc tools to quickly solve performance, troubleshooting, and networking issues. If you want to develop new bcc tools, see tutorial_bcc_python_developer.md for that tutorial.

It is assumed for this tutorial that bcc is already installed, and you can run tools like execsnoop successfully. See INSTALL.md. This uses enhancements added to the Linux 4.x series."



'via Blog this'

Building beautiful RESTful APIs using Flask, Swagger UI and Flask-RESTPlus - Michał Karzyński

Building beautiful RESTful APIs using Flask, Swagger UI and Flask-RESTPlus - Michał Karzyński: "This article outlines steps needed to create a RESTful API using Flask and Flask-RESTPlus. These tools combine into a framework, which automates common tasks:

• API input validation
• formatting output (as JSON)
• generating interactive documentation (with Swagger UI)
• turning Python exceptions into machine-readable HTTP responses

Flask 

 Flask is a web micro-framework written in Python. Since it’s a micro-framework, Flask does very little by itself. In contrast to a framework like Django, which takes the “batteries included” approach, Flask does not come with an ORM, serializers, user management or built-in internationalization. All these features and many others are available as Flask extensions, which make up a rich, but loosely coupled ecosystem.

The challenge, then, for an aspiring Flask developer lies in picking the right extensions and combining them together to get just the right set of functions. In this article we will describe how to use the Flask-RESTPlus extension to create a Flask-based RESTful JSON API.



 Flask-RESTPlus 

 Flask-RESTPlus aims to make building REST APIs quick and easy. It provides just enough syntactic sugar to make your code readable and easy to maintain. The killer feature of RESTPlus is its ability to automatically generate an interactive documentation for your API using Swagger UI.




Swagger UI 

 Swagger UI is part of a suite of technologies for documenting RESTful web services. Swagger has evolved into the OpenAPI specification, currently curated by the Linux Foundation. Once you have an OpenAPI description of your web service, you can use a variety of tool to generate documentation or even boilerplate code in a variety of languages. Take a look at swagger.io for more information."



'via Blog this'

REST etc: Don’t Version Your Web API - InfoQ

Don’t Version Your Web API: "For Lambla, none of these versioning techniques work on the open web. What we really want is something that can evolve, using contracts that smoothly can evolve with needed changes. For Lambla this is a well understood problem, and refers to the web; with many millions of microservices it has been running for 25 years without too many problems and no versioning with the exception of HTTP.

The first pillar for evolvable contracts is backward compatibility. Lambla uses HTML as an example where there are lot of elements that we are discouraged from using, but they are still supported by clients since we can’t update all web sites in the world. The same principles should apply to an API; as it evolves it must still support old formats.

The second pillar is forward compatibility. To achieve this, you have to ignore the unknown or what is not understood. Lambla refers to CSS as an example where new attributes can be handled without problems. To achieve this, fall-back rules are used to handle unknown attributes, an important way to get extensibility points.

XML is still commonly used, and often with XML schemas. To support evolvability here we must be flexible in content and Lambla therefore strongly recommends against validating schemas in servers as well as in clients. Instead we should just find the elements and attributes we need and ignore the rest.

To avoid versioning, we need to continue with support of all features, but we can’t keep all changes in an API forever. Old features that are rarely used should be removed. To know when they can be removed we need to use metrics and measure usage. We can then decide that when usage falls below for example, 1%, the feature can be removed."



'via Blog this'

ACID: How to Screw it Up! - DZone Database

ACID: How to Screw it Up! - DZone Database: "In my previous post, I described four applications (three implemented, one an example) that require, or at least strongly benefit from, strong ACID transactions. With that out of the way, we can now get to the fun bits. 

Today's claim: Most databases that claim to be 100% ACID compliant actually fudge definitions, goof up, or deliberately mislead. This applies to 30-year old giants as well as strapping upstarts; it's just a fact of life in databases.

I could make a giant chart with all of the products that claim to be ACID — and list where they fail. But that chart would be partial, soon-dated, and probably contain errors. It would certainly open up a discussion about specifics that might miss the forest for the trees.

So in lieu of that, here's general guidance about the ways many databases fail to meet that 100% ACID standard. This is how you can evaluate a new database's ACID promises. The content below will help you look  past the hype and/or marketing to get a sense of how mature the development is, what engineering tradeoffs have been made, and how serious the database developers are about backing up their promises. "



'via Blog this'

why i like netstat (h/t @icco) pic.twitter.com/PunGzBqjLg

— Julia Evans (@b0rk) July 19, 2016

some words about perf (this one's gonna need more than one page) pic.twitter.com/F4eqpCaaWe

— Julia Evans (@b0rk) July 18, 2016

a sketch about why I like dstat pic.twitter.com/lgQHm8Sq7J

— Julia Evans (@b0rk) July 18, 2016

networking: One way to make containers network: BGP - Julia Evans

One way to make containers network: BGP - Julia Evans: "Okay, so, again, I have 5 containers, all running things on port 4000. The key observation here is -- it's okay to run a bunch of things on the same port as long as they're on different IPs. Normally on one computer you only use one IP address. But that doesn't have to be true! You can have lots!

So. I have 5 containers, and I've assigned them IPs 10.0.1.101, 10.0.1.102, 10.0.1.103, 10.0.1.104, 10.0.1.105. Inside my computer, this is fine -- it's easy to imagine that I can just know which one is which.

But what do I do if I have another computer on the same network? How does that container know that 10.0.1.104 belongs to a container on my computer?

The Linux kernel knows about the BGP protocol. Calico knows about the Linux kernel. So Calico says "hey linux! Tell these other computers on the network to find these IP addresses here, okay?" And then all the traffic for those IPs comes to our computer, and everything is great.

To me, this seems pretty nice. It means that you can easily interpret the packets coming in and out of your machine (and, because we love tcpdump, we want to be able to understand our network traffic). I think there are other advantages but I'm not sure what they are.



 I find reading this networking stuff pretty difficult; more difficult than usual. For example, Docker also has a networking product they released recently. The webpage says they're doing "overlay networking". I don't know what that is, but it seems like you need etcd or consul or zookeeper. So the networking thing involves a distributed key-value store? Why do I need to have a distributed key-value store to do networking? There is probably a talk about this that I can watch but I don't understand it yet."



'via Blog this'

Architecture: The Log: What every software engineer should know about real-time data's unifying abstraction | LinkedIn Engineering

The Log: What every software engineer should know about real-time data's unifying abstraction | LinkedIn Engineering: "I joined LinkedIn about six years ago at a particularly interesting time. We were just beginning to run up against the limits of our monolithic, centralized database and needed to start the transition to a portfolio of specialized distributed systems. This has been an interesting experience: we built, deployed, and run to this day a distributed graph database, a distributed search backend, a Hadoop installation, and a first and second generation key-value store.

One of the most useful things I learned in all this was that many of the things we were building had a very simple concept at their heart: the log. Sometimes called write-ahead logs or commit logs or transaction logs, logs have been around almost as long as computers and are at the heart of many distributed data systems and real-time application architectures.

You can't fully understand databases, NoSQL stores, key value stores, replication, paxos, hadoop, version control, or almost any software system without understanding logs; and yet, most software engineers are not familiar with them. I'd like to change that. In this post, I'll walk you through everything you need to know about logs, including what is log and how to use logs for data integration, real time processing, and system building."



'via Blog this'

Linux debugging tools I love - Julia Evans

Linux debugging tools I love - Julia Evans: "

strace
A tool that traces system calls. My favorite thing. I have a bunch of posts with examples of how to use it on this blog. If you want to read what I think of it you should read the fanzine reference that you can read here.

strace is pretty broadly useful, but keep in mind it can really slow down your programs.

I would be remiss if I did not mention the even-more-awesome dtrace. Colin Jones has a nice introduction.



 dstat
A really simple tool that prints out how much data got sent over the network / written to disk every second. This is great when you suspect something is going on with network/disk usage and want to see what's happening in real time.

There's also iostat and netstat and atop and a bunch of other tools, but dstat is my favorite.



 tcpdump + wireshark
For spying on network traffic. I wrote an introduction explaining how to use them in tcpdump is amazing.

When using these, it really helps to have a basic understanding of how networking works. Luckily the basics ("what's the difference between IP and TCP and HTTP? what's a network packet?") are totally possible to pick up :D.



 perf 

Have a C program and want to know which functions it's spending the most time in? perf is a sampling profiler for Linux that can tell you that.

perf top gives you a live view of which functions are running right now, just like top. I like to use perf top no matter what language my programs are written in, just to see if I can understand anything from it. Sometimes it works!

node.js has built-in support for using perf to show you which Node function is running right now. You can also get this for JVM programs with perf-map-agent.

Brendan Gregg's website has the best introduction to perf I know.

You can use perf to generate amazing flame graphs.



 opensnoop 

Opensnoop is a new script that you can get as of Ubuntu 16.04. It's a delightfully simple tool -- it just shows you which files are being opened right now on your computer. And it's fast, unlike strace!

opensnoop also exists on OS X and does basically the same thing.

Go to the iovisor/bcc repo on github for installation instructions. It works using eBPF, which is a new thing that I will not explain yet here but Brendan Gregg has been writing about enthusiastically for some time. You don't need to know how it works to use it, though :)."



'via Blog this'

A checklist for Docker in the Enterprise | zwischenzugs

A checklist for Docker in the Enterprise | zwischenzugs: "Docker is extremely popular with developers, having gone as a product from zero to pretty much everywhere in a few years.

I started tinkering with Docker three years ago, got it going in a relatively small corp (700 employees) in a relatively unregulated environment. This was great fun: we set up our own registry, installed Docker on our development servers, installed Jenkins plugins to use Docker containers in our CI pipeline, even wrote our own build tool to get over the limitations of Dockerfiles.

I now work for an organisation working in arguably the most heavily regulated industry, with over 100K employees. The IT security department itself is bigger than the entire company I used to work for.



 I want to outline the areas that may be important to an enterprise when considering developing a Docker infrastructure."



'via Blog this'

Qcon: Much Faster Networking

Qcon: Much Faster Networking
David Riddoch talks about technologies that make very high performance networking possible on commodity servers and networks, with a special focus on kernel bypass technologies including sockets acceleration and NFV. These techniques give user-space applications direct access to the network adapter hardware, making possible sub-microsecond latencies and millions of messages per second per thread

Cloudfare: Why we use the Linux kernel's TCP stack - networking

Why we use the Linux kernel's TCP stack: "There are two general themes: first, there is no stable open-source partial kernel bypass technology yet. We hope Netmap will occupy this niche, and we are actively supporting it with our patches. Second, the Linux TCP stack has many critical features and very good debugging capabilities. It will take years to compete with this rich ecosystem.



For these reasons it's unlikely userspace networking will become mainstream. In practice I can think only of a few reasonable applications of kernel bypass techniques:

• Software switches or routers. Here you want to hand over network cards to the application, deal with raw packets and skip the kernel altogether. 
• Dedicated loadbalancers. Similarly, if the machine is only doing packet shuffling skipping the kernel makes sense. 

• Partial bypass for selected high throughput / low latency applications. This is the setup we use for our DDoS mitigations. Unfortunately I'm not aware of a stable open source TCP stack that fits this category.

 For the general user the Linux network stack is the right choice. Although it's less exciting than rewriting TCP stacks, we should focus on understanding the Linux stack performance and fixing its problems. There are some serious initiatives underway to improve the performance of the good old Linux TCP stack."



'via Blog this'

softwareyoga.: How to be invaluable employee?

How to be invaluable employee?

Or in other words how to bring more value to the company as a result of your work?

You got to show them how your work is making a difference to their world – A world where $, savings and time to market has precedence over exceptions, design patterns and interfaces.

Listed below are some aspects that will definitely get you recognized as a programmer who goes the extra mile beyond and above the official duties.

1. Understand what the end users want in the software. Talk to them. Figure out the things they don’t like in the existing system, things they would like to improve etc. You have to solve their problems using your technical skills.
2. Identify what modules or components have the most defects in your system. Come up with high level technical approach to solve the issues. Present the ideas to the management, explain what benefits it would bring to the company and how it can save time and cost. Use statistical figures to explain how your ideas will bring vast improvements.
3. Identify things that are being done manually or with complex workarounds. Eliminate them or handle them in a better way.
4. Speak to management on what they think are the biggest problem areas. A lot of the times, getting input from someone who is away from the technical stuff helps immensely.
5. Evaluate new technologies for your product and present to the team and management on what benefits they may bring.
6. Identify and improve workplace processes. You will surely have few issues regarding the way some things are done in the company. Figure a way to alleviate the issues.
7. Many a times, the project managers might over-promise or set extremely tight deadlines – not because they like sucking the last drop of blood out of you, but often because they don’t understand enough technically. Have a discussion with them on their expectations. With your technical knowledge, provide them inputs on whats possible realistically and how the scope can be altered so that everyone is a winner.
8. Draw comparisons of your work with other industries and bring in the best practices. Often, you can draw inspiration from the most unexpected places.
9. Mentor junior employees. Train other employees on various topics and practices. In most companies, you will have someone in the company who you aspire to be like some day. Meet them and ask them if they could be your mentor or if you could shadow them in their work.
10. Automate stuff outside of your product – Find people within your company who are not as tech savvy as you are. There will be some one out there who needs a helping hand. Solve people’s problems.

In conclusion, I believe that a programmer should get involved in other activities in a company, not just coding. It serves two purposes – provide you insights from a non-technical viewpoint and an understanding that programming is not the center of the universe.

Why is Python slow? « kmod's blog

Why is Python slow « kmod's blog: "Python spends almost all of its time in the C runtime



 This means that it doesn't really matter how quickly you execute the "Python" part of Python.  Another way of saying this is that Python opcodes are very complex, and the cost of executing them dwarfs the cost of dispatching them.  Another analogy I give is that executing Python is more similar to rendering HTML than it is to executing JS -- it's more of a description of what the runtime should do rather than an explicit step-by-step account of how to do it.



Update: why is the Python C runtime slow?

Here's the example I gave in my talk illustrating the slowness of the C runtime.  This is a for loop written in Python, but that doesn't execute any Python bytecodes:

import itertools
sum(itertools.repeat(1.0, 100000000))
The amazing thing about this is that if you write the equivalent loop in native JS, V8 can run it 6x faster than CPython.  In the talk I mistakenly attributed this to boxing overhead, but Raymond Hettinger kindly pointed out that CPython's sum() has an optimization to avoid boxing when the summands are all floats (or ints).  So it's not boxing overhead, and it's not dispatching on tp_as_number->tp_add to figure out how to add the arguments together.

My current best explanation is that it's not so much that the C runtime is slow at any given thing it does, but it just has to do a lot.  In this itertools example, about 50% of the time is dedicated to catching floating point exceptions.  The other 50% is spent figuring out how to iterate the itertools.repeat object, and checking whether the return value is a float or not.  All of these checks are fast and well optimized, but they are done every loop iteration so they add up.  A back-of-the-envelope calculation says that CPython takes about 30 CPU cycles per iteration of the loop, which is not very many, but is proportionally much more than V8's 5."



'via Blog this'

Why do we use the Linux kernel's TCP stack? - Julia Evans

Why do we use the Linux kernel's TCP stack? - Julia Evans: "I'm at PolyConf in Poland today, and I watched this super interesting talk by Leandro Pereira about Lwan, an ~8000 line of code web server. He talked about a bunch of the optimizations they'd done (improve CPU cache performance! be really careful about locking!). You can read more about the performance on the website & the links there.

It's a super cool project because it started out as a hobby project, and now he says it's getting to a state where it kinda actually really works and people are using it for real things. This web server is extremely fast -- it can do, in some benchmarks, 2 million requests per second.

Before I start talking about this -- of course practically nobody needs to do 2 million requests per second. I sure don't. But thinking about high performance computing is a really awesome way to understand the limits of computers better!



 I tracked him down to ask him questions later, and he mentioned that most of the time is spent talking to the Linux kernel and copying things back and forth.



 Then he said something really surprising: that in the Seastar HTTP framework, they wrote their own TCP stack, and it made everything several times times faster. What?!"



'via Blog this'

Linux and Perfomance: Reducing latency spikes by tuning the CPU scheduler

Reducing latency spikes by tuning the CPU scheduler: "It turns out that this effect can be observed when scylla runs in a different cgroup from the java process, but not when they run in the same cgroup. The reason for this effect is yet to be discovered.



 To check to which group given process belongs to, one can run:

 cat /proc/$PID/cgroup

The cgroup subsystem named cpu is relevant for scheduling.



 A process can be forced under a certain cgroup at run time with cgclassify. For example, to force all scylla and java (scylla-jmx) processes under the root group (-w is important for the change to affect all threads):

 sudo cgclassify -g cpu:/ `pgrep -w scylla`

sudo cgclassify -g cpu:/ `pgrep -w java`



Linux has a feature called autogroup which will automatically associate each session with a different cgroup. Processes started in those sessions will inherit group affinity. Due to latency issues mentioned above, the scylla package will disable autogroup (See this commit).



 Forcing scylla and scylla-jmx under the same group and applying scheduler settings has the desired effect of reducing scheduling delays. Now the maximum delay is 1.8 ms:"



'via Blog this'

storage: A ZFS developer’s analysis of the good and bad in Apple’s new APFS file system | Ars Technica

A ZFS developer’s analysis of the good and bad in Apple’s new APFS file system | Ars Technica: "I'm not sure Apple absolutely had to replace HFS+, but likely they had passed an inflection point where continuing to maintain and evolve the 30+ year old software was more expensive than building something new. APFS is a product born of that assessment.

Based on what Apple has shown I'd surmise that its core design goals were:

• satisfying all consumers (laptop, phone, watch, etc.) 
• encryption as a first-class citizen 
• snapshots for modernized backup 

Those are great goals that will benefit all Apple users, and based on the WWDC demos APFS seems to be on track (though the macOS Sierra beta isn't quite as far along).



 In the process of implementing a new file system the APFS team has added some expected features. HFS was built when 400KB floppies ruled the Earth (recognized now only as the ubiquitous and anachronistic save icon). Any file system started in 2014 should of course consider huge devices, and SSDs—check and check. Copy-on-write (COW) snapshots are the norm; making the Duplicate command in the Finder faster wasn't much of a detour. The use case is unclear—it's a classic garbage can theory solution in search of a problem—but it doesn't hurt and it makes for a fun demo. The beach ball of doom earned its nickname; APFS was naturally built to avoid it.



 There are some seemingly absent or ancillary design goals: performance, openness, and data integrity. Squeezing the most IOPS or throughput out of a device probably isn't critical on watchOS, and it's relevant only to a small percentage of macOS users. It will be interesting to see how APFS performs once it ships (measuring any earlier would only misinform the public and insult the APFS team)."



'via Blog this'

The Fault in Our JARs: Why We Stopped Building Fat JARs

The Fault in Our JARs: Why We Stopped Building Fat JARs: "The Fault in Our JARs: Why We Stopped Building Fat JARs
JUN 16, 2016 / BY JONATHAN HABER

inShare
10
HubSpot’s backend services are almost all written in Java. We have over 1,000 microservices constantly being built and deployed. When it comes time to deploy and run one of our Java applications, its dependencies must be present on the classpath for it to work. Previously, we handled this by using the maven-shade-plugin to build a fat JAR. This takes the application and all of its dependencies and bundles them into one massive JAR. This JAR is immutable and has no external dependencies, which makes it easy to deploy and run. For years this is how we packaged all of our Java applications and it worked pretty well, but it had some serious drawbacks.

 Fat JAR Woes

 The first issue we hit is that JARs are not meant to be aggregated like this. There can be files with the same path present in multiple JARs and by default the shade plugin includes the first file in the fat JAR and discards the rest. This caused some really frustrating bugs until we figured out what was going on (for example, Jersey uses META-INF/services files to autodiscover providers and this was causing some providers to not get registered). Luckily, the shade plugin supports resource transformers that allow you to define a merge strategy when it encounters duplicate files so we were able to work around this issue. However, it’s still an extra "gotcha" that all of our developers need to be conscious of.

The other, bigger issue we ran into is that this process is slow and inefficient. Using one of our applications as an example, it contains 70 class files totalling 210KB when packaged as a JAR. But after running the shade plugin to bundle its dependencies, we end up with a fat JAR containing 101,481 files and weighing in at 158MB. Combining 100,000 tiny files into a single archive is slow. Uploading this JAR to S3 at the end of the build is slow. Downloading this JAR at deploy time is slow (and can saturate the network cards on our application servers if we have a lot of concurrent deploys)."



'via Blog this'