How Intel's open source Data Plane Development Kit enables high-performance Linux networking - TechRepublic: "To utilize processors more efficiently in Linux, developers bypass the Linux kernel and dedicate CPU cores to network processing. The CloudFlare example used Linux kernel tools to pin processes to a set of cores and actively managed memory queues. It took a bit of programming discipline to send unprocessed packets between two systems. Add any packet processing, such as IPS, and the performance should drop significantly.
DPDK bypasses the Linux kernel and manages the NIC and CPU assignment directly. On a single Intel processor, it's possible to achieve throughput of up to 80Mpps or roughly 46Gbps using the smallest Ethernet Frames. I want to make sure I caveat the above comment and note that it was an Intel processor. DPDK is an open source project and currently has both Intel and ARM distributions."
'via Blog this'
Wednesday, 24 August 2016
Storage: How to roll back a btrfs snapshot and restore corrupt or missing files - TechRepublic
How to roll back a btrfs snapshot and restore corrupt or missing files - TechRepublic: "The btrfs (often called Butter FS) is a modern copy on write filesystem that offers some seriously handy features and focuses on fault tolerance and repair. The developers of btrfs have also gone to great lengths to make it easy to use. No other aspect of btrfs highlights all of this more than the ability to take snapshots of a subvolume and roll back to a previous working state.
I'll show how easy it is to do just that on an external flash drive (for the sake of simplicity). You might want to test this process on a similar device before you're ready to take this to a production machine."
'via Blog this'
I'll show how easy it is to do just that on an external flash drive (for the sake of simplicity). You might want to test this process on a similar device before you're ready to take this to a production machine."
'via Blog this'
Monday, 22 August 2016
Linux Takes Over the World... SDN and NFV
Linux Takes Over the World, or at the Least, Becomes Quite Influe: "Open source projects are new to networking, but they’ve been cropping up all over the place in the last couple of years. And many of them are gravitating toward the Linux Foundation.
Some of them were originally independent groups. The Open Network Operating System (ONOS) for example, was founded by On.lab. But in October, it became part of the Linux Foundation. The Linux Foundation was already hosting the OpenDaylight Project, which some considered a rival to ONOS. But the two groups seem to be happily coexisting under the same host."
'via Blog this'
Some of them were originally independent groups. The Open Network Operating System (ONOS) for example, was founded by On.lab. But in October, it became part of the Linux Foundation. The Linux Foundation was already hosting the OpenDaylight Project, which some considered a rival to ONOS. But the two groups seem to be happily coexisting under the same host."
'via Blog this'
What Golang Is and Is Not
What Golang Is and Is Not: "“Innovative” is such an overused and abused word that it has lost a lot of power and meaning. All innovation is contextual and to use the word without context is foolhardy. In the context of language design Go was never an innovative language, nor was it presented as such, or anyone dishonest in representing it that way.
As a language Go was always explicitly a return to simplicity, and in many ways naivety, for sound reasons.
“There is nothing new under the sun” rings true in all languages since the 80’s.
Virtually everything we see in language design now that someone says is “innovative” has been explored in some form before. Go is certainly no exception, but remember it never claimed to be state of the art.
Regarding the language being youthful, of course it is, but the intention is not for the language itself to ‘mature’: no more complexity is going to be added, or at least it’s very unlikely.
It is not ‘missing’ comprehensions, or inheritance, or generics, they are omitted (and I pray, always will be). In some way, in the context of the current fashion of returning to more functional languages, or the evolution of good old languages to include more functional paradigms (I’m looking at you Javascript and Python for two examples) then in a tenuous convoluted way Go has ‘innovated’ by avoiding that trend."
'via Blog this'
As a language Go was always explicitly a return to simplicity, and in many ways naivety, for sound reasons.
“There is nothing new under the sun” rings true in all languages since the 80’s.
Virtually everything we see in language design now that someone says is “innovative” has been explored in some form before. Go is certainly no exception, but remember it never claimed to be state of the art.
Regarding the language being youthful, of course it is, but the intention is not for the language itself to ‘mature’: no more complexity is going to be added, or at least it’s very unlikely.
It is not ‘missing’ comprehensions, or inheritance, or generics, they are omitted (and I pray, always will be). In some way, in the context of the current fashion of returning to more functional languages, or the evolution of good old languages to include more functional paradigms (I’m looking at you Javascript and Python for two examples) then in a tenuous convoluted way Go has ‘innovated’ by avoiding that trend."
'via Blog this'
Thursday, 18 August 2016
Web: Top 10 Mistakes Node.js Developers Make
Top 10 Mistakes Node.js Developers Make: "Node.js has seen an important growth in the past years, with big companies such as Walmart or PayPal adopting it. More and more people are picking up Node and publishing modules to NPM at such a pace that exceeds other languages. However, the Node philosophy can take a bit to get used to, especially if you have switched from another language.
In this article we will talk about the most common mistakes Node developers make and how to avoid them. You can find the source code for the examples on github."
'via Blog this'
In this article we will talk about the most common mistakes Node developers make and how to avoid them. You can find the source code for the examples on github."
'via Blog this'
10 Myths of Enterprise Python | PayPal Engineering Blog
10 Myths of Enterprise Python | PayPal Engineering Blog: "Python has enjoyed many years of grassroots usage and support from developers across eBay. Even before official support from management, technologists of all walks went the extra mile to reap the rewards of developing in Python. I joined PayPal a few years ago, and chose Python to work on internal applications, but I’ve personally found production PayPal Python code from nearly 15 years ago.
Today, Python powers over 50 projects, .
For this introductory post, I’ll be focusing on the 10 myths I’ve had to debunk the most in eBay and PayPal’s enterprise environments."
'via Blog this'
Today, Python powers over 50 projects, .
For this introductory post, I’ll be focusing on the 10 myths I’ve had to debunk the most in eBay and PayPal’s enterprise environments."
'via Blog this'
Tuesday, 16 August 2016
Linux/C: gdb Debugging Full Example (Tutorial): Brendan Gregg
gdb Debugging Full Example (Tutorial): ncurses: "I'm a little frustrated with finding "gdb examples" online that show the commands but not their output. gdb is the GNU Debugger, the standard debugger on Linux. I was reminded of the lack of example output when watching the Give me 15 minutes and I'll change your view of GDB talk by Greg Law at CppCon 2015, which, thankfully, includes output!
It's well worth the 15 minutes.
It also inspired me to share a full gdb debugging example, with output and every step involved, including dead ends. This isn't a particularly interesting or exotic issue, it's just a routine gdb debugging session. But it covers the basics and could serve as a tutorial of sorts, bearing in mind there's a lot more to gdb than I used here.
I'll be running the following commands as root, since I'm debugging a tool that needs root access (for now). Substitute non-root and sudo as desired. You also aren't expected to read through all this: I've enumerated each step so you can browse them and find ones of interest."
'via Blog this'
It's well worth the 15 minutes.
It also inspired me to share a full gdb debugging example, with output and every step involved, including dead ends. This isn't a particularly interesting or exotic issue, it's just a routine gdb debugging session. But it covers the basics and could serve as a tutorial of sorts, bearing in mind there's a lot more to gdb than I used here.
I'll be running the following commands as root, since I'm debugging a tool that needs root access (for now). Substitute non-root and sudo as desired. You also aren't expected to read through all this: I've enumerated each step so you can browse them and find ones of interest."
'via Blog this'
Java & C++ : More JVM Signal tricks - Thread control via mprotect · Jackson Davis
More JVM Signal tricks - Thread control via mprotect · Jackson Davis: "In my last post, I mentioned the JVM uses intentionally uses SIGSEGVS in other interesting ways. In this post I’ll give an overview of one of those other ways, synchronization for GC pauses.
If you ever look at generated assembly from Hotspot, you will find methods ending with very odd-looking tests like this:
test %eax,0x16e71fa4(%rip) # 0x00007fce84071000
At first glance, this doesn’t seem to serve any real purpose right before the return. Rather than doing any useful work, its just a way of trying to read from 0x7fce84071000, the designated polling page. Modifying access to this page lets the JVM stop threads cleanly - in places where the state of the world is well-known."
'via Blog this'
If you ever look at generated assembly from Hotspot, you will find methods ending with very odd-looking tests like this:
test %eax,0x16e71fa4(%rip) # 0x00007fce84071000
At first glance, this doesn’t seem to serve any real purpose right before the return. Rather than doing any useful work, its just a way of trying to read from 0x7fce84071000, the designated polling page. Modifying access to this page lets the JVM stop threads cleanly - in places where the state of the world is well-known."
'via Blog this'
Monday, 15 August 2016
Deciphering Glyph :: Python Packaging Is Good Now
Deciphering Glyph :: Python Packaging Is Good Now: "The real takeaway here though, is that although it’s still not perfect, other languages are no longer doing appreciably better.
Go is still working through a number of different options regarding dependency management and vendoring, and, like Python extensions that require C dependencies, CGo is sometimes necessary and always a problem. Node has had its own well-publicized problems with their dependency management culture and package manager. Hackage is cool and all but everything takes a literal geological epoch to compile.
As always, I’m sure none of this applies to Rust and Cargo is basically perfect, but that doesn’t matter, because nobody reading this is actually using Rust.
My point is not that packaging in any of these languages is particularly bad. They’re all actually doing pretty well, especially compared to the state of the general programming ecosystem a few years ago; many of them are making regular progress towards user-facing improvements.
My point is that any commentary suggesting they’re meaningfully better than Python at this point is probably just out of date. Working with Python packaging is more or less fine right now. It could be better, but lots of people are working on improving it, and the structural problems that prevented those improvements from being adopted by the community in a timely manner have almost all been addressed.
Go! Make some virtualenvs! Hack some setup.pys! If it’s been a while and your last experience was really miserable, I promise, it’s better now.
"
'via Blog this'
Go is still working through a number of different options regarding dependency management and vendoring, and, like Python extensions that require C dependencies, CGo is sometimes necessary and always a problem. Node has had its own well-publicized problems with their dependency management culture and package manager. Hackage is cool and all but everything takes a literal geological epoch to compile.
As always, I’m sure none of this applies to Rust and Cargo is basically perfect, but that doesn’t matter, because nobody reading this is actually using Rust.
My point is not that packaging in any of these languages is particularly bad. They’re all actually doing pretty well, especially compared to the state of the general programming ecosystem a few years ago; many of them are making regular progress towards user-facing improvements.
My point is that any commentary suggesting they’re meaningfully better than Python at this point is probably just out of date. Working with Python packaging is more or less fine right now. It could be better, but lots of people are working on improving it, and the structural problems that prevented those improvements from being adopted by the community in a timely manner have almost all been addressed.
Go! Make some virtualenvs! Hack some setup.pys! If it’s been a while and your last experience was really miserable, I promise, it’s better now.
"
'via Blog this'
NUMA Deep Dive Part 1: From UMA to NUMA - frankdenneman.nl
NUMA Deep Dive Part 1: From UMA to NUMA - frankdenneman.nl: "Non-uniform memory access (NUMA) is a shared memory architecture used in today's multiprocessing systems. Each CPU is assigned its own local memory and can access memory from other CPUs in the system. Local memory access provides a low latency - high bandwidth performance. While accessing memory owned by the other CPU has higher latency and lower bandwidth performance. Modern applications and operating systems such as ESXi support NUMA by default, yet to provide the best performance, virtual machine configuration should be done with the NUMA architecture in mind. If incorrect designed, inconsequent behavior or overall performance degradation occurs for that particular virtual machine or in worst case scenario for all VMs running on that ESXi host.
This series aims to provide insights of the CPU architecture, the memory subsystem and the ESXi CPU and memory scheduler. Allowing you in creating a high performing platform that lays the foundation for the higher services and increased consolidating ratios. Before we arrive at modern compute architectures, it's helpful to review the history of shared-memory multiprocessor architectures to understand why we are using NUMA systems today."
'via Blog this'
This series aims to provide insights of the CPU architecture, the memory subsystem and the ESXi CPU and memory scheduler. Allowing you in creating a high performing platform that lays the foundation for the higher services and increased consolidating ratios. Before we arrive at modern compute architectures, it's helpful to review the history of shared-memory multiprocessor architectures to understand why we are using NUMA systems today."
'via Blog this'
Saturday, 13 August 2016
Where the Database Market Goes From Here – tecosystems
Where the Database Market Goes From Here – tecosystems: "Given that the database market is subject to the same market forces as other enterprise categories, on premises software both specialized and general purpose is likely to be a tightening market over time. There is a great deal of revenue to be had in the category, without question, but it will be more difficult to obtain as there is more competition generally, more competition from open source specifically and on premises alternatives increasingly compete directly with service based alternatives.
These price pressures are one reason vendors are increasingly moving back towards general purpose datastores from specialized roots: the broader the functional capabilities, the wider the addressable market, at least in theory.
Even long time database incumbents, however, are scrambling to develop or acquire their way into service-based businesses because that is where much of the growth will occur. Whether adopted as more convenient stand-alone alternatives to on premises databases or deployed in conjunction with other cloud infrastructure, DBaaS offerings are attractive to both developers and their employers, if for very different reasons. Importantly, this is the case in spite of the fact that the DBaaS market is in its infancy; many popular databases are not yet available as services, and those that are don’t yet have the provider choice that they will ultimately. Which implies that the DBaaS market has been successful, to a degree, in spite of itself.
From a provider perspective, then, a choice is implied.
The existing spend on on premises relational solutions is measured in tens of billions of dollars, which is why many database providers today still regard their primary market competition as Oracle, even if they’re selling non-relational solutions. Vendors focused on trajectory, however, tend to see Amazon as the more important target, given that the most common report when talking to purveyors of on premises software is that a significant percentage of their existing customers are already in the cloud and most of those are on Amazon.
Implied choice or not, however, a legitimate market approach is also not to choose.
On premises providers in most cases will need to follow the lead of players like MongoDB, because competing with DBaaS players without an as-a-service option is a non-starter. Worse, competitors that operate as-a-service businesses have an enormous intelligence advantage over purely on premises competitors, thanks to the difference in available operational telemetry. But neither should on premises vendors deprecate their existing business; instead, they can differentiate from pure play as-a-service options by offering customers their choice of running in an existing datacenter or in the cloud.
What is clear, however, is that a status quo approach in this market is one that will lead to diminishing returns over time. Choose your path carefully."
'via Blog this'
These price pressures are one reason vendors are increasingly moving back towards general purpose datastores from specialized roots: the broader the functional capabilities, the wider the addressable market, at least in theory.
Even long time database incumbents, however, are scrambling to develop or acquire their way into service-based businesses because that is where much of the growth will occur. Whether adopted as more convenient stand-alone alternatives to on premises databases or deployed in conjunction with other cloud infrastructure, DBaaS offerings are attractive to both developers and their employers, if for very different reasons. Importantly, this is the case in spite of the fact that the DBaaS market is in its infancy; many popular databases are not yet available as services, and those that are don’t yet have the provider choice that they will ultimately. Which implies that the DBaaS market has been successful, to a degree, in spite of itself.
From a provider perspective, then, a choice is implied.
The existing spend on on premises relational solutions is measured in tens of billions of dollars, which is why many database providers today still regard their primary market competition as Oracle, even if they’re selling non-relational solutions. Vendors focused on trajectory, however, tend to see Amazon as the more important target, given that the most common report when talking to purveyors of on premises software is that a significant percentage of their existing customers are already in the cloud and most of those are on Amazon.
Implied choice or not, however, a legitimate market approach is also not to choose.
On premises providers in most cases will need to follow the lead of players like MongoDB, because competing with DBaaS players without an as-a-service option is a non-starter. Worse, competitors that operate as-a-service businesses have an enormous intelligence advantage over purely on premises competitors, thanks to the difference in available operational telemetry. But neither should on premises vendors deprecate their existing business; instead, they can differentiate from pure play as-a-service options by offering customers their choice of running in an existing datacenter or in the cloud.
What is clear, however, is that a status quo approach in this market is one that will lead to diminishing returns over time. Choose your path carefully."
'via Blog this'
Friday, 12 August 2016
Don’t be Delta: Four pillars of high availability | InfoWorld
Don’t be Delta: Four pillars of high availability | InfoWorld: "Obviously no matter how great your software and servers, if you don't have redundant network infrastructure, you may not survive. This starts with your switches and routers, includes such equipment as load balancers, and goes a bit beyond. For instance, backup power -- from a reliable provider that routes your power appropriately.
High availability (HA) protects against such problems as a server or service failing.
Let's say the runtime your web service is built on does a core dump. No user should know -- they should simply be routed to another service. To accomplish this, you need to run multiple instances and load balancers, then replicate appropriately.
Disaster recovery (DR) protects against more unlikely events. Often, DR is implemented with relatively low expectations (sometimes hours or even days before recovery) and/or acceptable data loss. I’d argue that on the modern internet, recovery should be minutes at worst -- same goes for data accessibility. To accomplish DR, you need another datacenter that’s geographically distant and an ironclad failover scheme.
You also need to get the data there.
If you use a cloud hosting provider like AWS, opt for servers in multiple regions in case disaster strikes. Some larger organizations go as far as using more than one cloud infrastructure provider."
'via Blog this'
High availability (HA) protects against such problems as a server or service failing.
Let's say the runtime your web service is built on does a core dump. No user should know -- they should simply be routed to another service. To accomplish this, you need to run multiple instances and load balancers, then replicate appropriately.
Disaster recovery (DR) protects against more unlikely events. Often, DR is implemented with relatively low expectations (sometimes hours or even days before recovery) and/or acceptable data loss. I’d argue that on the modern internet, recovery should be minutes at worst -- same goes for data accessibility. To accomplish DR, you need another datacenter that’s geographically distant and an ironclad failover scheme.
You also need to get the data there.
If you use a cloud hosting provider like AWS, opt for servers in multiple regions in case disaster strikes. Some larger organizations go as far as using more than one cloud infrastructure provider."
'via Blog this'
Thursday, 11 August 2016
Python: Please Fix Your Decorators · Homepage of Hynek Schlawack
Please Fix Your Decorators · Homepage of Hynek Schlawack: "If your Python decorator unintentionally changes the signatures of my callables or doesn’t work with class methods, it’s broken and should be fixed. Sadly most decorators are broken because the web is full of bad advice."
'via Blog this'
'via Blog this'
career: Five-essential-phone-screen-questions - steveyegge2
five-essential-phone-screen-questions - steveyegge2: "You have to probe all five areas; you can't skip any of them. Each area is a proxy for a huge body of knowledge, and failing it very likely means failing the interviews, even though the candidate did fine in the other areas.
Without further ado, here they are: The Five Essential Questions for the first phone-screen with an SDE candidate:
1) Coding. The candidate has to write some simple code, with correct syntax, in C, C++, or Java.
2) OO design. The candidate has to define basic OO concepts, and come up with classes to model a simple problem.
3) Scripting and regexes. The candidate has to describe how to find the phone numbers in 50,000 HTML pages.
4) Data structures. The candidate has to demonstrate basic knowledge of the most common data structures.
5) Bits and bytes. The candidate has to answer simple questions about bits, bytes, and binary numbers.
Please understand: what I'm looking for here is a total vacuum in one of these areas. It's OK if they struggle a little and then figure it out. It's OK if they need some minor hints or prompting. I don't mind if they're rusty or slow. What you're looking for is candidates who are utterly clueless, or horribly confused, about the area in question.
For example, you may find a candidate who decides that a Vehicle class should be a subclass of ParkingGarage, since garages contain cars. This is just busted, and it's un-fixable in any reasonable amount of training time.
Or a candidate might decide, when asked to search for phone numbers in a bunch of text files, to write a 2000-line C++ program, at which point you discover they've never heard of "grep", or at least never used it.
When a candidate is totally incompetent in one of these Big Five areas, the chances are very high that they'll bomb horribly when presented with our typical interview questions. Last week I interviewed an SDE-2 candidate who made both of the mistakes above (a vehicle inheriting from garage, and the 2000-line C++ grep implementation.) He was by no means unusual, even for the past month. We've been bringing in many totally unqualified candidates.
The rest of this document describes each area in more detail, and gives example questions"
'via Blog this'
Without further ado, here they are: The Five Essential Questions for the first phone-screen with an SDE candidate:
1) Coding. The candidate has to write some simple code, with correct syntax, in C, C++, or Java.
2) OO design. The candidate has to define basic OO concepts, and come up with classes to model a simple problem.
3) Scripting and regexes. The candidate has to describe how to find the phone numbers in 50,000 HTML pages.
4) Data structures. The candidate has to demonstrate basic knowledge of the most common data structures.
5) Bits and bytes. The candidate has to answer simple questions about bits, bytes, and binary numbers.
Please understand: what I'm looking for here is a total vacuum in one of these areas. It's OK if they struggle a little and then figure it out. It's OK if they need some minor hints or prompting. I don't mind if they're rusty or slow. What you're looking for is candidates who are utterly clueless, or horribly confused, about the area in question.
For example, you may find a candidate who decides that a Vehicle class should be a subclass of ParkingGarage, since garages contain cars. This is just busted, and it's un-fixable in any reasonable amount of training time.
Or a candidate might decide, when asked to search for phone numbers in a bunch of text files, to write a 2000-line C++ program, at which point you discover they've never heard of "grep", or at least never used it.
When a candidate is totally incompetent in one of these Big Five areas, the chances are very high that they'll bomb horribly when presented with our typical interview questions. Last week I interviewed an SDE-2 candidate who made both of the mistakes above (a vehicle inheriting from garage, and the 2000-line C++ grep implementation.) He was by no means unusual, even for the past month. We've been bringing in many totally unqualified candidates.
The rest of this document describes each area in more detail, and gives example questions"
'via Blog this'
Wednesday, 10 August 2016
C/C++: Linkers and Loaders
Linkers and Loaders: "These are the manuscript chapters for my Linkers and Loaders, published by Morgan-Kaufman. See the book's web site for ordering information.
The text in these files is the unedited original manuscript. M-K has fine copy editors, who have fixed all the minor typos, spelling, and grammar errors in the printed book, but if you come across factual errors I'd still appreciate help getting all the details of linking and loading as complete and correct as possible. I will collect errata and fix them in subsequent printings.
The figures here are scans of hand-drawn sketches which have been redrawn for the book. You don't need to tell me I'm a lousy artist. I already know that."
'via Blog this'
The text in these files is the unedited original manuscript. M-K has fine copy editors, who have fixed all the minor typos, spelling, and grammar errors in the printed book, but if you come across factual errors I'd still appreciate help getting all the details of linking and loading as complete and correct as possible. I will collect errata and fix them in subsequent printings.
The figures here are scans of hand-drawn sketches which have been redrawn for the book. You don't need to tell me I'm a lousy artist. I already know that."
'via Blog this'
Wednesday, 3 August 2016
Non-volatile Storage | the morning paper
Non-volatile Storage | the morning paper: "This invalidation is due to the arrival of high-speed, non-volatile storage devices, typically referred to as storage class memories (SCM). The performance of an SCM, at hundreds of thousands of iops, requires one or more many-core CPUs to saturate it. The most visible type of SCMs today are PCIe SSDs (SSDs attached via the PCI-express bus). Beyond PCI SSDs there are NVDIMMS, which have the performance characteristics of DRAM while simultaneously offering persistence – these tend to couple DRAM and flash on a DIMM and use a super-capacitor to provide enough power to flush the volatile contents of DRAM out to disk on a loss of power. And then of course there’s the even cheaper battery-backed distributed UPS solutions that we looked at yesterday…"
'via Blog this'
'via Blog this'
Simple Sketches for Diagramming Your Software Architecture | Voxxed
Simple Sketches for Diagramming Your Software Architecture | Voxxed: "If you’re working in an agile software development team at the moment, take a look around at your environment. Whether it’s physical or virtual, there’s likely to be a story wall or Kanban board visualising the work yet to be started, in progress and done. Visualising your software development process is a fantastic way to introduce transparency because anybody can see, at a glance, a high-level snapshot of the current progress.
As an industry, we’ve become adept at visualising our software development process over the past few years – however, it seems we’ve forgotten how to visualise the actual software that we’re building. I’m not just referring to post-project documentation. This also includes communication during the software development process. Agile approaches talk about moving fast, and this requires good communication, but it’s surprising that many teams struggle to effectively communicate the design of their software."
'via Blog this'
As an industry, we’ve become adept at visualising our software development process over the past few years – however, it seems we’ve forgotten how to visualise the actual software that we’re building. I’m not just referring to post-project documentation. This also includes communication during the software development process. Agile approaches talk about moving fast, and this requires good communication, but it’s surprising that many teams struggle to effectively communicate the design of their software."
'via Blog this'
Linux/Cloud: Puppet Enterprise Targets Application Orchestration
Puppet Enterprise Targets Application Orchestration: "Puppet Labs, known for configuration management of servers recently announced support for Application Management and Orchestration.
With the distributed nature of projects and advent of Microservices, applications need to be carefully orchestrated across multiple servers or Virtual Machines. Puppet Enterprise reduces the complexity of application delivery and deployment."
'via Blog this'
With the distributed nature of projects and advent of Microservices, applications need to be carefully orchestrated across multiple servers or Virtual Machines. Puppet Enterprise reduces the complexity of application delivery and deployment."
'via Blog this'
BGP and the System of Trust that Runs the Internet Pt. 1 - OpenDNS Umbrella Blog
BGP and the System of Trust that Runs the Internet Pt. 1 - OpenDNS Umbrella Blog: "What most Internet users don’t know, but many are beginning to discover, is the entire framework that holds the Internet together can be quite fragile. In fact the system in charge of correctly routing Internet traffic is largely built on a collective trust, or as The Washington Post recently described it, an ‘honor code’ of the Internet.
The fact is we all rely on core protocols of the Internet like BGP (border gateway protocol) and DNS (domain name services) every day. They are a foundation of the Internet. Even companies that provide content on the Internet but don’t need BGP services directly, still rely on the service to get that content to their customers.
This two-part series will cover BGP, what it is, how it works, and what global moving parts are involved. Part two of this series, publishing next week, will cover the technical and security issues surrounding BGP, and why they often have a drastic effect on a global scale."
'via Blog this'
The fact is we all rely on core protocols of the Internet like BGP (border gateway protocol) and DNS (domain name services) every day. They are a foundation of the Internet. Even companies that provide content on the Internet but don’t need BGP services directly, still rely on the service to get that content to their customers.
This two-part series will cover BGP, what it is, how it works, and what global moving parts are involved. Part two of this series, publishing next week, will cover the technical and security issues surrounding BGP, and why they often have a drastic effect on a global scale."
'via Blog this'
Security: Understanding and Hardening Linux Containers
Understanding and Hardening Linux Containers: "Operating System virtualisation is an attractive feature for efficiency, speed and modern application deployment, amid questionable security. Recent advancements of the Linux kernel have coalesced for simple yet powerful OS virtualisation via Linux Containers, as implemented by LXC, Docker, and CoreOS Rkt among others. Recent container focused start-ups such as Docker have helped push containers into the limelight. Linux containers offer native OS virtualisation, segmented by kernel namespaces, limited through process cgroups and restricted through reduced root capabilities, Mandatory Access Control and user namespaces. This paper discusses these container features, as well as exploring various security mechanisms. Also included is an examination of attack surfaces, threats, and related hardening features in order to properly evaluate container security. Finally, this paper contrasts different container defaults and enumerates strong security recommendations to counter deployment weaknesses-- helping support and explain methods for building high-security Linux containers. Are Linux containers the future or merely a fad or fantasy? This paper attempts to answer that question."
'via Blog this'
'via Blog this'
Linux/Unix: Understanding the bin, sbin, usr/bin , usr/sbin split
Understanding the bin, sbin, usr/bin , usr/sbin split: "On Tuesday 30 November 2010 15:58:00 David Collier wrote:
> I see that busybox spreads it's links over these 4 directories.
>
> Is there a simple rule which decides which directory each link lives
> in.....
>
> For instance I see kill is in /bin and killall in /usr/bin.... I don't
> have a grip on what might be the logic for that."
'via Blog this'
> I see that busybox spreads it's links over these 4 directories.
>
> Is there a simple rule which decides which directory each link lives
> in.....
>
> For instance I see kill is in /bin and killall in /usr/bin.... I don't
> have a grip on what might be the logic for that."
'via Blog this'
Extracting The Python Stack Using Ptrace
Extracting The Python Stack Using Ptrace: "I recently wrote a program that I think is pretty nifty, called Pystack. Pystack let's you ptrace(2) attach to an arbitrary running Python process and see the currently executing stack trace."
'via Blog this'
'via Blog this'
Google: Linux Shell Style Guide
Shell Style Guide:
"Which Shell to Use : Bash is the only shell scripting language permitted for executables.
Executables must start with #!/bin/bash and a minimum number of flags.
Use set to set shell options so that calling your script as bash does not break its functionality.
Restricting all executable shell scripts to bash gives us a consistent shell language that's installed on all our machines.
The only exception to this is where you're forced to by whatever you're coding for. One example of this is Solaris SVR4 packages which require plain Bourne shell for any scripts.
When to use Shell
link : Shell should only be used for small utilities or simple wrapper scripts.
While shell scripting isn't a development language, it is used for writing various utility scripts throughout Google. This style guide is more a recognition of its use rather than a suggestion that it be used for widespread deployment.
Some guidelines:
Bear in mind that scripts grow. Rewrite your script in another language early to avoid a time-consuming rewrite at a later date."
'via Blog this'
"Which Shell to Use : Bash is the only shell scripting language permitted for executables.
Executables must start with #!/bin/bash and a minimum number of flags.
Use set to set shell options so that calling your script as bash does not break its functionality.
Restricting all executable shell scripts to bash gives us a consistent shell language that's installed on all our machines.
The only exception to this is where you're forced to by whatever you're coding for. One example of this is Solaris SVR4 packages which require plain Bourne shell for any scripts.
When to use Shell
link : Shell should only be used for small utilities or simple wrapper scripts.
While shell scripting isn't a development language, it is used for writing various utility scripts throughout Google. This style guide is more a recognition of its use rather than a suggestion that it be used for widespread deployment.
Some guidelines:
- If you're mostly calling other utilities and are doing relatively little data manipulation, shell is an acceptable choice for the task.
- If performance matters, use something other than shell.
- If you find you need to use arrays for anything more than assignment of ${PIPESTATUS}, you should use Python.
- If you are writing a script that is more than 100 lines long, you should probably be writing it in Python instead.
Bear in mind that scripts grow. Rewrite your script in another language early to avoid a time-consuming rewrite at a later date."
'via Blog this'
eklitzke.org: C++ and Thoughts On Java, Go, and Rust
C++ and Thoughts On Java, Go, and Rust: "Historically most programming situations that called for high performance, particularly in the field of systems programming, would have been written in C or C++. That has changed a lot over the last twenty five years or so. Java has increasingly been the programming language of choice for high performance server applications. Recently Go and Rust have made inroads as high performance alternatives to Java that claim to offer similar or better performance. Yet I still find that I really love writing C++, and at least for situations where speed or memory utilization are important I prefer writing C++ to any other language. In this post I will explore some of the aspects of C++ that I find appealing, and why Java, Go, and Rust don't make the cut for me."
'via Blog this'
'via Blog this'
Linux: Inline C/Assembly In Bash
Inline C/Assembly In Bash: "Recently I had the idea of trying to combine two of my favorite things: bash and assembly. My original idea was to try to make a way to inline assembly code into bash scripts. While I was researching this project I discovered Tavis Ormandy's ctype.sh. This is a really clever project that makes use of the nearly undocumented enable -f feature in bash to add new shell builtins that expose the functionality provided by dlopen(3). The project is extremely ingenious, and I would recommend browsing the code to anyone who interested in these kinds of low level things."
'via Blog this'
'via Blog this'
Good logging practice in Python « Victor Lin's coding notes
Good logging practice in Python « Victor Lin's coding notes: "In reality, logging is important. When you transfer money, there are transfer records. When an airplane is flying, black box (flight data recorder) is recording everything. If something goes wrong, people can read the log and has a chance to figure out what happened. Likewise, logging is important for system developing, debugging and running. When a program crashes, if there is no logging record, you have little chance to understand what happened. For example, when you are writing a server, logging is necessary."
'via Blog this'
'via Blog this'
I wish I knew these things when I learned Python | Bugra Akyildiz
I wish I knew these things when I learned Python | Bugra Akyildiz:
"I sometimes found myself asking myself how I cannot know simpler way of doing “this” thing in Python 3. When I seek solution, I of course find much more elegant, efficient and more bug-free code parts over time. In total(not just this post), the total sum of “those” things were far more than I expect/admit, but here is the first crop of features that was not obvious to me and learned later as I sought more efficient/simple/maintainable code."
'via Blog this'
"I sometimes found myself asking myself how I cannot know simpler way of doing “this” thing in Python 3. When I seek solution, I of course find much more elegant, efficient and more bug-free code parts over time. In total(not just this post), the total sum of “those” things were far more than I expect/admit, but here is the first crop of features that was not obvious to me and learned later as I sought more efficient/simple/maintainable code."
'via Blog this'
Five Ways to Not Mess Up Microservices in Production
Five Ways to Not Mess Up Microservices in Production: "Ealier this month Takipi's Alex Zhitnitsky wrote about five ways not to mess up microservices in production (OK, "mess" wasn't exactly the word he used!). He builds on the output of the panel discussion we reported on a few months back as well as their own users, examining the top production problems with microservices and how to solve them. The emphasis, as we will see, is on distributed debugging and approaches which can help to make this a tractable effort."
'via Blog this'
'via Blog this'
Destroy All Ifs — A Perspective from Functional Programming – John A De Goes
Destroy All Ifs — A Perspective from Functional Programming – John A De Goes:
"I’ve long argued that, despite all its flaws, one of the great things about functional programming is its intrinsic inversion of control.
In an impure imperative language, if you call some function void doX(State *state), the function can do anything it wants. It can modify state (both passed in and global), it can delete files, it can read from the network, and it can even launch nukes!
In a pure functional language, however, if you call some function doX :: State -> IO (), then at most it’s going to return a value. It can’t modify what you pass it, and if you like, you can ignore the value returned by the function, in which case calling the function has no effect (aside from sucking up a little CPU and RAM).
Now granted, IO in Haskell is not a strong example of an abstraction that has superior reasoning properties, but the fundamental principle is sound: in pure functional programming languages, the control is inverted to the caller.
Thus, as you’re looking at the code, you have a clearer sense of what the functions you are calling can do, without having to unearth their foundations.
I think this generalizes to the following principle:
Code that inverts control to the caller is generally easier to understand.
(I actually think this is a specialization of an even deeper principle — that code which destroys information is generally harder to reason about than code which does not.)
Viewed from this angle, you can see that if we embed conditionals deep into functions, and we call those functions, we have lost a certain amount of control: we feed in inputs, but they are combined in arbitrary and unknown ways to arrive at a decision (which code path to take) that has humongous ramifications on how the program behaves.
It’s no wonder that conditionals (and with them, booleans) are so widely despised!
In an object-oriented programming language, it’s generally considered a good practice to replace conditionals with polymorphism.
In a similar fashion, in functional programming, it’s often considered good practice to replace boolean values with algebraic data types."
'via Blog this'
"I’ve long argued that, despite all its flaws, one of the great things about functional programming is its intrinsic inversion of control.
In an impure imperative language, if you call some function void doX(State *state), the function can do anything it wants. It can modify state (both passed in and global), it can delete files, it can read from the network, and it can even launch nukes!
In a pure functional language, however, if you call some function doX :: State -> IO (), then at most it’s going to return a value. It can’t modify what you pass it, and if you like, you can ignore the value returned by the function, in which case calling the function has no effect (aside from sucking up a little CPU and RAM).
Now granted, IO in Haskell is not a strong example of an abstraction that has superior reasoning properties, but the fundamental principle is sound: in pure functional programming languages, the control is inverted to the caller.
Thus, as you’re looking at the code, you have a clearer sense of what the functions you are calling can do, without having to unearth their foundations.
I think this generalizes to the following principle:
Code that inverts control to the caller is generally easier to understand.
(I actually think this is a specialization of an even deeper principle — that code which destroys information is generally harder to reason about than code which does not.)
Viewed from this angle, you can see that if we embed conditionals deep into functions, and we call those functions, we have lost a certain amount of control: we feed in inputs, but they are combined in arbitrary and unknown ways to arrive at a decision (which code path to take) that has humongous ramifications on how the program behaves.
It’s no wonder that conditionals (and with them, booleans) are so widely despised!
In an object-oriented programming language, it’s generally considered a good practice to replace conditionals with polymorphism.
In a similar fashion, in functional programming, it’s often considered good practice to replace boolean values with algebraic data types."
'via Blog this'
What kubernetes and cloud9 tell us about the new industry – anyone but Amazon – James Governor's Monkchips
What kubernetes and cloud9 tell us about the new industry – anyone but amazon – James Governor's Monkchips: "We used to talk about the Anyone but Microsoft club. Arguably the triumph of open source in the enterprise was a result of the same driver. Linux was a means to avoid Microsoft operating system domination. Vendors would push an alternative, and customers would support it partly as means to hedge their bets against too much domination by a single provider.
Today the dominant vendor scaring tech providers is clearly Amazon Web Services.
One facet of today’s Anyone but Amazon coalition is OpenStack. AWS dominance led pretty much every other major tech vendor, no matter how competitive to converge on OpenStack, as an open hedge.
When pondering the implications of Microsoft’s hiring of Brendan Burns the other day, it struck me another coalition is forming, changing the fault lines of the industry. Burns is one of the founders of the Kubernetes container cluster manager project. He was at Google but just took a job as product lead for the Azure Resource Manager. He has already publicly declared he will continue to work on Kubernetes. Kubernetes has also been enthusiastically adopted by Red Hat, through it’s OpenShift platform.
So now Microsoft, Red Hat, and Google Cloud Platform are all now aligned around Kubernetes. While at first glance this new alliance of strange bedfellows might seem to be a response to the rise of Docker and the Docker Pattern – and indeed there is no doubt the enthusiastic growth in Kubernetes is partly driven by concern that Docker will own too much real estate of the new infrastructure world I believe the overarching threat is Amazon.
As Stephen has explained – the biggest competitor to open source is Amazon. There is no doubt that Amazon EC2 Container Services is going to gain wide traction. Amazon can afford to be magnanimous about Docker’s rise in a way other vendors can’t. Docker is an implementation detail rather than a potential existential threat to AWS.
This week another shoe dropped, when Amazon announced it is acquiring Cloud9, the online IDE startup. Cloud9 created Ace, which also powers the GitHub editor."
'via Blog this'
Today the dominant vendor scaring tech providers is clearly Amazon Web Services.
One facet of today’s Anyone but Amazon coalition is OpenStack. AWS dominance led pretty much every other major tech vendor, no matter how competitive to converge on OpenStack, as an open hedge.
When pondering the implications of Microsoft’s hiring of Brendan Burns the other day, it struck me another coalition is forming, changing the fault lines of the industry. Burns is one of the founders of the Kubernetes container cluster manager project. He was at Google but just took a job as product lead for the Azure Resource Manager. He has already publicly declared he will continue to work on Kubernetes. Kubernetes has also been enthusiastically adopted by Red Hat, through it’s OpenShift platform.
So now Microsoft, Red Hat, and Google Cloud Platform are all now aligned around Kubernetes. While at first glance this new alliance of strange bedfellows might seem to be a response to the rise of Docker and the Docker Pattern – and indeed there is no doubt the enthusiastic growth in Kubernetes is partly driven by concern that Docker will own too much real estate of the new infrastructure world I believe the overarching threat is Amazon.
As Stephen has explained – the biggest competitor to open source is Amazon. There is no doubt that Amazon EC2 Container Services is going to gain wide traction. Amazon can afford to be magnanimous about Docker’s rise in a way other vendors can’t. Docker is an implementation detail rather than a potential existential threat to AWS.
This week another shoe dropped, when Amazon announced it is acquiring Cloud9, the online IDE startup. Cloud9 created Ace, which also powers the GitHub editor."
'via Blog this'
Microsoft REST API Guidelines
api-guidelines/Guidelines.md at master · Microsoft/api-guidelines · GitHub:
"The Microsoft REST API Guidelines, as a design principle, encourages application developers to have resources accessible to them via a RESTful HTTP interface. To provide the smoothest possible experience for developers on platforms following the Microsoft REST API Guidelines, REST APIs SHOULD follow consistent design guidelines to make using them easy and intuitive.
This document establishes the guidelines Microsoft REST APIs SHOULD follow so RESTful interfaces are developed consistently."
'via Blog this'
"The Microsoft REST API Guidelines, as a design principle, encourages application developers to have resources accessible to them via a RESTful HTTP interface. To provide the smoothest possible experience for developers on platforms following the Microsoft REST API Guidelines, REST APIs SHOULD follow consistent design guidelines to make using them easy and intuitive.
This document establishes the guidelines Microsoft REST APIs SHOULD follow so RESTful interfaces are developed consistently."
'via Blog this'
Tuesday, 2 August 2016
Why Cisco's data center strategy depends on simplifying storage - TechRepublic
Why Cisco's data center strategy depends on simplifying storage - TechRepublic: "Cisco recently announced a MapR-based SAP HANA appliance. MapR is described as a converged data platform, and Cisco is leveraging the platform to create a dedicated x86-based SAN for the SAP HANA persistent layer.
A major difference between HyperFlex and the MapR-based SAP HANA appliance is the storage and compute nodes are not mixed. MapR creates a dedicated layer for storage separate from the application servers.
Cisco's storage strategy has a complicated history. The current set of technologies is focused on specific verticals. However, Cisco needs to place some consistency behind a single strategy before customers buy wholesale into a Cisco-only data center strategy."
'via Blog this'
A major difference between HyperFlex and the MapR-based SAP HANA appliance is the storage and compute nodes are not mixed. MapR creates a dedicated layer for storage separate from the application servers.
Cisco's storage strategy has a complicated history. The current set of technologies is focused on specific verticals. However, Cisco needs to place some consistency behind a single strategy before customers buy wholesale into a Cisco-only data center strategy."
'via Blog this'
Adam Leventhal's blog » I Love Go; I Hate Go
Adam Leventhal's blog » I Love Go; I Hate Go: "I’m sure time will, as it tends to, bring pragmatism. I appreciate that Go does have strong opinions (it’s just hard to remember that when I disagree with them). Weak opinions are what turn languages into unreadable mishmashes of overlapping mechanism. My favorite example of this is Perl. My first real programming job was in Perl. I was the most avid reader teenage of the Perl llama and camel books.
Ask me about my chat with Larry Wall, Perl’s creator, if you see a beer in my hand. In an interview, Larry said, “In Perl 6, we decided it would be better to fix the language than fix the user”.
Contrast this with Go’s statement on assertions:
Go doesn’t provide assertions. They are undeniably convenient, but our experience has been that programmers use them as a crutch to avoid thinking about proper error handling and reporting.
Perl wants to be whatever the user wants to be. The consumate pleaser. Go sees the user as flawed and the strictures of the language as the cure. It’s an authoritarian, steadfast in its ideals, yet too sensitive to find compromise (sound like anyone we all know?).
Despite my frustrations I really enjoy writing code in Go. It’s clean and supported by a great community and ecosystem."
'via Blog this'
Ask me about my chat with Larry Wall, Perl’s creator, if you see a beer in my hand. In an interview, Larry said, “In Perl 6, we decided it would be better to fix the language than fix the user”.
Contrast this with Go’s statement on assertions:
Go doesn’t provide assertions. They are undeniably convenient, but our experience has been that programmers use them as a crutch to avoid thinking about proper error handling and reporting.
Perl wants to be whatever the user wants to be. The consumate pleaser. Go sees the user as flawed and the strictures of the language as the cure. It’s an authoritarian, steadfast in its ideals, yet too sensitive to find compromise (sound like anyone we all know?).
Despite my frustrations I really enjoy writing code in Go. It’s clean and supported by a great community and ecosystem."
'via Blog this'
Subscribe to:
Posts (Atom)