Linux debugging tools I love - Julia Evans

Linux debugging tools I love - Julia Evans: "

strace
A tool that traces system calls. My favorite thing. I have a bunch of posts with examples of how to use it on this blog. If you want to read what I think of it you should read the fanzine reference that you can read here.

strace is pretty broadly useful, but keep in mind it can really slow down your programs.

I would be remiss if I did not mention the even-more-awesome dtrace. Colin Jones has a nice introduction.



 dstat
A really simple tool that prints out how much data got sent over the network / written to disk every second. This is great when you suspect something is going on with network/disk usage and want to see what's happening in real time.

There's also iostat and netstat and atop and a bunch of other tools, but dstat is my favorite.



 tcpdump + wireshark
For spying on network traffic. I wrote an introduction explaining how to use them in tcpdump is amazing.

When using these, it really helps to have a basic understanding of how networking works. Luckily the basics ("what's the difference between IP and TCP and HTTP? what's a network packet?") are totally possible to pick up :D.



 perf 

Have a C program and want to know which functions it's spending the most time in? perf is a sampling profiler for Linux that can tell you that.

perf top gives you a live view of which functions are running right now, just like top. I like to use perf top no matter what language my programs are written in, just to see if I can understand anything from it. Sometimes it works!

node.js has built-in support for using perf to show you which Node function is running right now. You can also get this for JVM programs with perf-map-agent.

Brendan Gregg's website has the best introduction to perf I know.

You can use perf to generate amazing flame graphs.



 opensnoop 

Opensnoop is a new script that you can get as of Ubuntu 16.04. It's a delightfully simple tool -- it just shows you which files are being opened right now on your computer. And it's fast, unlike strace!

opensnoop also exists on OS X and does basically the same thing.

Go to the iovisor/bcc repo on github for installation instructions. It works using eBPF, which is a new thing that I will not explain yet here but Brendan Gregg has been writing about enthusiastically for some time. You don't need to know how it works to use it, though :)."



'via Blog this'

A checklist for Docker in the Enterprise | zwischenzugs

A checklist for Docker in the Enterprise | zwischenzugs: "Docker is extremely popular with developers, having gone as a product from zero to pretty much everywhere in a few years.

I started tinkering with Docker three years ago, got it going in a relatively small corp (700 employees) in a relatively unregulated environment. This was great fun: we set up our own registry, installed Docker on our development servers, installed Jenkins plugins to use Docker containers in our CI pipeline, even wrote our own build tool to get over the limitations of Dockerfiles.

I now work for an organisation working in arguably the most heavily regulated industry, with over 100K employees. The IT security department itself is bigger than the entire company I used to work for.



 I want to outline the areas that may be important to an enterprise when considering developing a Docker infrastructure."



'via Blog this'

Qcon: Much Faster Networking

Qcon: Much Faster Networking
David Riddoch talks about technologies that make very high performance networking possible on commodity servers and networks, with a special focus on kernel bypass technologies including sockets acceleration and NFV. These techniques give user-space applications direct access to the network adapter hardware, making possible sub-microsecond latencies and millions of messages per second per thread

Cloudfare: Why we use the Linux kernel's TCP stack - networking

Why we use the Linux kernel's TCP stack: "There are two general themes: first, there is no stable open-source partial kernel bypass technology yet. We hope Netmap will occupy this niche, and we are actively supporting it with our patches. Second, the Linux TCP stack has many critical features and very good debugging capabilities. It will take years to compete with this rich ecosystem.



For these reasons it's unlikely userspace networking will become mainstream. In practice I can think only of a few reasonable applications of kernel bypass techniques:

• Software switches or routers. Here you want to hand over network cards to the application, deal with raw packets and skip the kernel altogether. 
• Dedicated loadbalancers. Similarly, if the machine is only doing packet shuffling skipping the kernel makes sense. 

• Partial bypass for selected high throughput / low latency applications. This is the setup we use for our DDoS mitigations. Unfortunately I'm not aware of a stable open source TCP stack that fits this category.

 For the general user the Linux network stack is the right choice. Although it's less exciting than rewriting TCP stacks, we should focus on understanding the Linux stack performance and fixing its problems. There are some serious initiatives underway to improve the performance of the good old Linux TCP stack."



'via Blog this'

softwareyoga.: How to be invaluable employee?

How to be invaluable employee?

Or in other words how to bring more value to the company as a result of your work?

You got to show them how your work is making a difference to their world – A world where $, savings and time to market has precedence over exceptions, design patterns and interfaces.

Listed below are some aspects that will definitely get you recognized as a programmer who goes the extra mile beyond and above the official duties.

1. Understand what the end users want in the software. Talk to them. Figure out the things they don’t like in the existing system, things they would like to improve etc. You have to solve their problems using your technical skills.
2. Identify what modules or components have the most defects in your system. Come up with high level technical approach to solve the issues. Present the ideas to the management, explain what benefits it would bring to the company and how it can save time and cost. Use statistical figures to explain how your ideas will bring vast improvements.
3. Identify things that are being done manually or with complex workarounds. Eliminate them or handle them in a better way.
4. Speak to management on what they think are the biggest problem areas. A lot of the times, getting input from someone who is away from the technical stuff helps immensely.
5. Evaluate new technologies for your product and present to the team and management on what benefits they may bring.
6. Identify and improve workplace processes. You will surely have few issues regarding the way some things are done in the company. Figure a way to alleviate the issues.
7. Many a times, the project managers might over-promise or set extremely tight deadlines – not because they like sucking the last drop of blood out of you, but often because they don’t understand enough technically. Have a discussion with them on their expectations. With your technical knowledge, provide them inputs on whats possible realistically and how the scope can be altered so that everyone is a winner.
8. Draw comparisons of your work with other industries and bring in the best practices. Often, you can draw inspiration from the most unexpected places.
9. Mentor junior employees. Train other employees on various topics and practices. In most companies, you will have someone in the company who you aspire to be like some day. Meet them and ask them if they could be your mentor or if you could shadow them in their work.
10. Automate stuff outside of your product – Find people within your company who are not as tech savvy as you are. There will be some one out there who needs a helping hand. Solve people’s problems.

In conclusion, I believe that a programmer should get involved in other activities in a company, not just coding. It serves two purposes – provide you insights from a non-technical viewpoint and an understanding that programming is not the center of the universe.

Why is Python slow? « kmod's blog

Why is Python slow « kmod's blog: "Python spends almost all of its time in the C runtime



 This means that it doesn't really matter how quickly you execute the "Python" part of Python.  Another way of saying this is that Python opcodes are very complex, and the cost of executing them dwarfs the cost of dispatching them.  Another analogy I give is that executing Python is more similar to rendering HTML than it is to executing JS -- it's more of a description of what the runtime should do rather than an explicit step-by-step account of how to do it.



Update: why is the Python C runtime slow?

Here's the example I gave in my talk illustrating the slowness of the C runtime.  This is a for loop written in Python, but that doesn't execute any Python bytecodes:

import itertools
sum(itertools.repeat(1.0, 100000000))
The amazing thing about this is that if you write the equivalent loop in native JS, V8 can run it 6x faster than CPython.  In the talk I mistakenly attributed this to boxing overhead, but Raymond Hettinger kindly pointed out that CPython's sum() has an optimization to avoid boxing when the summands are all floats (or ints).  So it's not boxing overhead, and it's not dispatching on tp_as_number->tp_add to figure out how to add the arguments together.

My current best explanation is that it's not so much that the C runtime is slow at any given thing it does, but it just has to do a lot.  In this itertools example, about 50% of the time is dedicated to catching floating point exceptions.  The other 50% is spent figuring out how to iterate the itertools.repeat object, and checking whether the return value is a float or not.  All of these checks are fast and well optimized, but they are done every loop iteration so they add up.  A back-of-the-envelope calculation says that CPython takes about 30 CPU cycles per iteration of the loop, which is not very many, but is proportionally much more than V8's 5."



'via Blog this'

Why do we use the Linux kernel's TCP stack? - Julia Evans

Why do we use the Linux kernel's TCP stack? - Julia Evans: "I'm at PolyConf in Poland today, and I watched this super interesting talk by Leandro Pereira about Lwan, an ~8000 line of code web server. He talked about a bunch of the optimizations they'd done (improve CPU cache performance! be really careful about locking!). You can read more about the performance on the website & the links there.

It's a super cool project because it started out as a hobby project, and now he says it's getting to a state where it kinda actually really works and people are using it for real things. This web server is extremely fast -- it can do, in some benchmarks, 2 million requests per second.

Before I start talking about this -- of course practically nobody needs to do 2 million requests per second. I sure don't. But thinking about high performance computing is a really awesome way to understand the limits of computers better!



 I tracked him down to ask him questions later, and he mentioned that most of the time is spent talking to the Linux kernel and copying things back and forth.



 Then he said something really surprising: that in the Seastar HTTP framework, they wrote their own TCP stack, and it made everything several times times faster. What?!"



'via Blog this'