The InfoQ Podcast: Uber's Chief Systems Architect = No Node.js, no JSON over HTTP

The InfoQ Podcast: Uber's Chief Systems Architect on their Architecture and Rapid Growth: "In this week's podcast QCon chair Wesley Reisz talks to Matt Ranney who is the Chief Systems Architect at Uber, where he's helping build and scale everything he can. Previously, Matt was a founder and CTO of Voxer, probably the largest and busiest deployment of Node.js.

Key takeaways

Expanding a company and team at this rate is genuinely hard. Lots of mistakes have been made along the way.
Microservices allow companies to grow rapidly but have a cost in terms of aggregate velocity.
Uber is gradually moving its marketplace development from Node.js to Go and Java. Java is used for the map services.
Aggressive failure testing is used extensively in Uber.
Some early design choices - like using JSON over HTTP - make formal verification basically impossible."



'via Blog this'

General Golang links

Quick Golang for Java Developers | Voxxed:



Beyond the Hype: 4 Years of Go in Production

NY Times: Gizmo Microservice Toolkit for GoLang

Introducing Gizmo

At The New York Times, our development teams have been adopting the Go programming language over the last three years to build better back-end services. In the past I’ve written about using Go for Elastic MapReduce streaming. I’ve also talked about using Go at GothamGo for news analysis and to improve our email and alert systems at the Golang NYC Meetup. We use Go for a wide variety of tasks, but the most common use throughout the company is for building JSON APIs.

When we first began building APIs with Go, we didn’t use any frameworks or shared interfaces. This meant that they varied from team to team and project to project with regard to structure, naming conventions and third-party tools. As we started building more and more APIs, the pains of microservices started to become apparent.

Around the time we reached this point, I came across Peter Bourgon’s talk from FOSDEM 2015, “Go and the Modern Enterprise,” and its accompanying blog post. A lot of what Peter said hit close to home for me and seemed very relevant to our situation at The Times. His description of the “Modern Enterprise” fit our technology teams quite well. We’re a consumer-focused company whose engineering group has more than doubled in size to a few hundred heads in recent years, and we have had a service-oriented architecture for a long time. We have also run into the same problems he brought up. As the need for more and more microservices arose, we needed a common set of tools to orchestrate and monitor them, as well as a common set of patterns and interfaces that enable developers to concentrate on business problems. An RFC for a new toolkit named “Go Kit” came out of the talk, and eventually open source development of it was under way.

Peter’s talk and the concept of Go Kit made me very excited but also somewhat dismayed. I knew a lot of the RPC and tracing technology involved would likely take a long time to be adopted throughout the company without some stepping stones to get us there. We also didn’t have a whole lot of time to wait around for the toolkit to be completed, so we decided to build our own set of tools that could bridge the gap and hopefully complement Go Kit by implementingsome of its “non-goals.”

It’s my pleasure to announce that as of today our toolkit, Gizmo, is open source. Gizmo offers four packages to help developers quickly configure and build microservice APIs and pubsub daemons.

GitHub: https://github.com/NYTimes/gizmo

Choose Boring Technology - Discourage "Cleverness"

Dan McKinley: Choose Boring Technology (Link to Slides)

Embrace Boredom.

Let's say every company gets about three innovation tokens. You can spend these however you want, but the supply is fixed for a long while. You might get a few more after you achieve a certain level of stability and maturity, but the general tendency is to overestimate the contents of your wallet. Clearly this model is approximate, but I think it helps.
If you choose to write your website in NodeJS, you just spent one of your innovation tokens. If you choose to use MongoDB, you just spent one of your innovation tokens. If you choose to use service discovery tech that's existed for a year or less, you just spent one of your innovation tokens. If you choose to write your own database, oh god, you're in trouble.
Any of those choices might be sensible if you're a javascript consultancy, or a database company. But you're probably not. You're probably working for a company that is at least ostensibly rethinking global commerce or reinventing payments on the web or pursuing some other suitably epic mission. In that context, devoting any of your limited attention to innovating ssh is an excellent way to fail. Or at best, delay success [1].
What counts as boring? That's a little tricky. "Boring" should not be conflated with "bad." There is technology out there that is both boring and bad [2]. You should not use any of that. But there are many choices of technology that are boring and good, or at least good enough. MySQL is boring. Postgres is boring. PHP is boring. Python is boring. Memcached is boring. Squid is boring. Cron is boring.
The nice thing about boringness (so constrained) is that the capabilities of these things are well understood. But more importantly, their failure modes are well understood. Anyone who knows me well will understand that it's only with a overwhelming sense of malaise that I now invoke the spectre of Don Rumsfeld, but I must.


Why discouraging cleverness is important in programming languages and codebases.

One of the other benefits of Go was widening the pool of backgrounds we can hire. We can take someone from any language background and have them ramped up on Go in weeks. With the Scala side there’s the JVM learning curve, the Java world of containers, black magic of JVM tuning etc…
New developers we’ve hired are ramped up in weeks vs months. We now have the majority of our services written in Go and one of the last holdouts to move to Go just wrote his first project and said to me afterward “Wow, I read through that library once and I knew exactly what it was doing , I’ve read the Scala version of that library four times and I still have no idea what it does, I can see why you guys like it so much”. That was one of our senior engineers who’s previously worked for one of the largest web properties in the world. This process was a complete bottoms up initiative from our development team who pushed for the move to Go.
We now process hundreds of thousands of messages per second and Terabytes of data per day with our GoLang services.
I’m not here to bash Scala, or ScalaZ (I love ValidationNel! ) but more to give some real world context of Scala in a production environment over 7 years with two companies. I still use Scala and love hacking in Scalding. Some of our more ambitious projects coming up will most likely be Scala based but I’m just not as sold on it anymore as the core language when trying to scale a growing engineering team. Go has since become my personal language of choice.