The level of testing that is performed in CI can completely vary but the key fundamentals are that multiple integrations from different developers are done through out the day. The biggest advantage of following this approach is that if there are any errors then they are identified early in the cycle, typically soon after the commit. Finding the bugs closer to commit does make them much more easier to fix. This is explained well by Martin Fowler:Continuous Integrations doesn’t get rid of bugs, but it does make them dramatically easier to find and remove.
There are lots of tools that provide CI capabilities. Most common ones are Jenkins from CloudBees, Travis CI, Go from ThoughtWorks, and Bamboo from Atlassian.
Martin Fowler: Continuous Integration
The trouble with deferred integration is that it's very hard to predict how long it will take to do, and worse it's very hard to see how far you are through the process. The result is that you are putting yourself into a complete blind spot right at one of tensest parts of a project - even if you're one of the rare cases where you aren't already late.Continuous Integration completely finesses this problem. There's no long integration, you completely eliminate the blind spot. At all times you know where you are, what works, what doesn't, the outstanding bugs you have in your system.Bugs - these are the nasty things that destroy confidence and mess up schedules and reputations. Bugs in deployed software make users angry with you. Bugs in work in progress get in your way, making it harder to get the rest of the software working correctly.Continuous Integrations doesn't get rid of bugs, but it does make them dramatically easier to find and remove. In this respect it's rather like self-testing code. If you introduce a bug and detect it quickly it's far easier to get rid of. Since you've only changed a small bit of the system, you don't have far to look. Since that bit of the system is the bit you just worked with, it's fresh in your memory - again making it easier to find the bug. You can also usediff debugging - comparing the current version of the system to an earlier one that didn't have the bug.Bugs are also cumulative. The more bugs you have, the harder it is to remove each one. This is partly because you get bug interactions, where failures show as the result of multiple faults - making each fault harder to find. It's also psychological - people have less energy to find and get rid of bugs when there are many of them - a phenomenon that the Pragmatic Programmers call the Broken Windows syndrome.
As a result projects with Continuous Integration tend to have dramatically less bugs, both in production and in process. However I should stress that the degree of this benefit is directly tied to how good your test suite is. You should find that it's not too difficult to build a test suite that makes a noticeable difference. Usually, however, it takes a while before a team really gets to the low level of bugs that they have the potential to reach. Getting there means constantly working on and improving your tests.
On the Effectiveness of Unit Test Automation at Microsoft | http://t.co/cShVpwQ1n2
— Tony Ayres (@tayr_ie) November 15, 2014
Realizing quality improvement through test driven development | http://t.co/xla5BF2WYW
— Tony Ayres (@tayr_ie) November 15, 2014
Knightmare: A DevOps Cautionary Tale
This is the story of how a company with nearly $400 million in assets went bankrupt in 45-minutes because of a failed deployment.
The events of August 1, 2012 should be a lesson to all development and operations teams. It is not enough to build great software and test it; you also have to ensure it is delivered to market correctly so that your customers get the value you are delivering (and so you don’t bankrupt your company). The engineer(s) who deployed SMARS are not solely to blame here – the process Knight had set up was not appropriate for the risk they were exposed to. Additionally their process (or lack thereof) was inherently prone to error. Any time your deployment process relies on humans reading and following instructions you are exposing yourself to risk. Humans make mistakes. The mistakes could be in the instructions, in the interpretation of the instructions, or in the execution of the instructions.
Deployments need to be automated and repeatable and as free from potential human error as possible. Had Knight implemented an automated deployment system – complete with configuration, deployment and test automation – the error that cause the Knightmare would have been avoided.
A couple of the principles for Continuous Delivery apply here (even if you are not implementing a full Continuous Delivery process):
Releasing software should be a repeatable, reliable process.
Automate as much as is reasonable.
Busting 6 myths about DevOps
DevOps replaces AgileDevOps is absolutely compatible with Agile. In fact, DevOps is the logical continuation of the Agile journey that was started in 2001, because we now know that the real definition of “done” is not when development is done coding. Instead, code is only “done” when it has been fully tested and is operating in production as designed. (Note that Agile is not a prerequisite for adopting DevOps.)
DevOps replaces ITILAlthough some may view DevOps as backlash to ITIL (the IT Infrastructure Library) or ITSM (IT Service Management), DevOps and ITIL are compatible, too. ITIL and ITSM remain the best codifications of the processes that underpin IT Operations, and actually describe many of the capabilities needed in order for IT Operations to support a DevOps-style work stream.
In order to accommodate the faster lead times and higher deployment frequencies associated with DevOps, many areas of the ITIL processes require automation, specifically around the change, configuration, and release processes. Because we also require fast detection and recovery when service incidents occur, the ITIL disciplines of service design and incident and problem management remain as relevant as ever.
DevOps means NoOpsDevOps is sometimes incorrectly interpreted to be NoOps (i.e., IT operations is entirely eliminated). However, more precisely, DevOps will often put more responsibility on development to do code deployments and maintain service levels. This merely means that development is taking over many of the IT Operations and operations engineering functions.
In order to support fast lead times and enable developer productivity, DevOps does require many IT operations tasks to become self-service. In other words, instead of development opening up a work ticket and waiting for IT operations to complete the work, many of these activities will be automated so that developers can do it themselves (e.g. get a production-like dev environment or add a feature metric for production telemetry).
DevOps is only for open source softwareAlthough many of the DevOps success stories take place in organizations using software such as the LAMP stack (Linux, Apache web server, MySQL database, PHP or Python or Perl), organizations are implementing DevOps patterns using Microsoft .NET, SAP, and even COBOL applications running on mainframes and HP LaserJet firmware.
DevOps principles are universal, and they are largely independent of the underlying technology being used. Some of the DevOps patterns have specific technology requirements (e.g. able to support automated testing, able to expose configurations that can be checked into version control), which are often more prevalent in open source software.
DevOps is just “infrastructure as code” or automationWhile many DevOps patterns require automation, DevOps also requires shared goals and shared pain throughout the IT value stream. This goes far beyond just automation.
DevOps is only for startups and unicorns
DevOps is applicable and relevant to any organization that must increase flow of planned work through development while maintaining quality, reliability, and security for the customer.
In fact, DevOps is even more important for the horses than for the unicorns. After all, as Richard Foster states, “Of the Fortune 500 companies in 1955, 87 percent are gone. In 1958, the Fortune 500 tenure was 61 years; now it’s only 18 years.” (1)
We know that the downward spiral happens to every IT organization. However, most enterprise IT organizations will come up with countless reasons why they cannot adopt DevOps, or why it is not relevant for them.
One of the primary objections from horses is that all the unicorns (e.g. Google, Amazon, Twitter, Etsy) were born that way. In other words, unicorns were born doing DevOps.
In actuality, virtually every DevOps unicorn was once a horse and had all the problems associated with being a horse:
- Amazon, up until 2001, ran on the OBIDOS content delivery system, which became so problematic and dangerous to maintain that CTO Werner Vogels transformed their entire organization and code to a service-oriented architecture. (2)
- Twitter struggled to scale capacity on their front-end monolithic Ruby on Rails system in 2009, starting a multiyear project to progressively re-architect and replace it. (3)
- LinkedIn, six months after their successful IPO in 2011, struggled with problematic deployments so painful that they launched Operation InVersion, a two-month feature freeze, allowing them to overhaul their compute environments, deployments, and architecture. (4)
- Etsy, in 2009, according to Michael Rembetsy, “had to come to grips that they were living in a sea of their own engineering filth,” dealing with problematic software deployments and technical debt. They committed themselves to a cultural transformation. (5)
Simply put, all unicorns were once horses. DevOps is how any horse can become a unicorn, if they want to become one. And in fact, the list of enterprises adopting DevOps continues to grow. Christopher Little states, “If there’s anything that all horses [enterprise IT organizations] hate, it’s hearing stories about unicorns [DevOps shops]. Which is strange, because horses and unicorns are probably the same species. Unicorns are just horses with horns.”
- Facebook, in 2009, was at the breaking point for infrastructure operations. Barely able to keep up with user growth, code deployments were becoming increasingly dangerous and staff were continually firefighting. Jay Parikh and Pedro Canahuati started their transformation to make code safe to deploy again. (6)
What Is DevOps?
Historically, product managers, business analysts and software engineers would work together to organize a release plan, with user stories sequenced into iterations that could be re-prioritized at each iteration boundary. While every iteration is supposed to end with a “production ready” version of the system, it has not been common to actually release to production regularly.
More often, the output of an iteration makes it only as far as a “test” or “staging” environment, because actually pushing to production requires many more steps: bundling the code, packaging the product, provisioning the environment, and coordinating with the operations staff.
People began to realize that a “throw it over the wall” mentality from development to operations was just as much of a challenge as we used to have with throwing requirements over the wall and waiting for code to come back. People began to blur the lines between the development tasks, such as coding and operational deployment tasks such as server provisioning, hence the name “DevOps.”
I still don’t see a clear definition in there… As mentioned above, the term DevOps has been used for many areas: automated infrastructure provisioning tools (e.g. Chef and Puppet) and automation tools (e.g. continuous integration like Jenkins), as well as for establishing developer work environments to mirror production (e.g. Vagrant).
I use the term “DevOps” to define a set of practices, tools and policies that lead to improved quality and Automated Delivery (AD). In many ways, rapid and frequent deployment to production reduces risk, as any release contains fewer changes. And fixes for any issues that are found are easier to fix or, alternatively, the smaller changes are typically easier to roll back.
So, DevOps is the same as Automated or Continuous Deployment then?Almost. DevOps, in a way, is all the things that go into making AD/CD possible. In many ways, DevOps is about ensuring quality at all stages.
You can visualize DevOps as a conveyor belt, where many checks and balances are in place, at all stages, to ensure any bundle coming down the belt is removed if it’s not good enough, and delivered to the end of the belt (e.g. production) safely and reliably if it is.
Why should I care? What will DevOps do for me?DevOps practices and procedures lead to smoothing out the typically ‘bumpiest’ aspects of software development and deployment. By pulling infrastructure setup and awareness earlier in the development cycle, surprises from environmental differences are significantly reduced.
By establishing consistent, reliable and repeatable automated deployments, human error and the need for ‘rockstar firefighters’ are reduced. The Phoenix Project (Kim, Behr, Spafford) is a great read on the subject and chronicles an organization’s change to embracing these practices in an incremental and prioritized manner. One of the key takeaways from the book is that making the first step is difficult, but allows for the next step to be easier.
Defining DevOps and integrating it into your processes at all levelsMost of the talk, tools and processes around DevOps can be intimidating. The Herculean task of transitioning to fully automated, multiple production releases per day can be daunting. The truth is that level of automation is simply not needed for all products or companies.
How deeply clients incorporate DevOps into their practices varies wildly; some still want a formal ‘hand off’ process, which DevOps purists will rage against. For these clients, a fully automated endto-end system may convey far too much risk. Do what alleviates the greatest pain for your organization in the development to release cycle. Like agile programming taking incremental steps toward the end goal is the most effective and reliable way of reaching the goal.
Software product businesses commonly aim to be ahead of their competition by launching new products and features quickly to market. Agile product development teams are expected to adapt to business changes and challenges while maintaining high quality standards.
More clients are recognizing the need for, and asking for help with, developing effective strategies to continuously deploy new/updated software; keeping systems up around the clock; and quickly and seamlessly resolving operational issues quickly.
If you leverage DevOps practices to assist in smoothing the worst point in your cycle, you may discover that your pain level has been lowered to the point where your releases are in sync with the velocity of customer feedback, and you can hold and/or hone your existing processes.
Like traditional software development, introducing capabilities that are not needed or of high value is a waste of development resources. You know by now that many business demands can be managed with DevOps practices and processes. That’s why I’m excited by DevOps initiatives.
Microservices, DevOps, and the cost of change?
To be able to provide these kinds of services, and come up with new innovative ways to provide better customer service or better products, companies must rely on technology going forward. Or, their competition (be it large existing competitors, or competitors they don’t even know about yet) will smoke them. To accommodate this relatively newer embrace of technology as a differentiator, businesses have to be agile. But okay, Mr Posta, you say. I’ve been hearing that for the last 10 years about “agile”. WTF is different?
The difference is the technical and cultural solutions for this agility have been popping up organically and more specifically in the form of open collaboration, community and opensource. It’s not being driven (at least of yet…. although it’s fast becoming a “perceived goldmine” for vendors) by stuffy, know-it-all, committees coming up with egregious specs that they can then hand down to us from the castle of the all knowing.
But why should you care?
Well, your employer cares. To be able to achieve this “agility” you don’t need a special (and very expensive) SOA suite. You need to adopt a culture that embraces reducing thecost of change. Change should be constant. Your management structure, teams, developers, architects, DBAs, operations, build and release, security, and business people should be all on the same page with this effort. Otherwise, you end up loosening up “change” in one pocket (sorry) and tightening the noose in others.
Microservices and DevOps are really more about culture and reducing the cost of change than any particular technology. And if you stop and take a look around your current company/organizational structure, and really think about “what are the impediments to change”.. you’ll find them all over.
DevelopmentAre you not writing unit tests? Then you’re slowing yourself down. You cannot make changes to your code base without a hint of confidence that the changes you make don’t break existing functionality, or even the functionality you’re writing.
Are you not doing continuous integration? then you’re slowing yourself (and sinking deadlines) by not being able to communicate changes in your code base across your team. And when it comes time to integrate/merge/rebase with others on your team, you have a black hole (literally… how do you predict how long it will take to rectify these conflicts? you cannot..) in terms of time. So by slowing yourself, you incur a cost.
Are you creating all kinds of shared dependencies, custom libraries, shared domain logic, and home-grown frameworks? Then you might as well hang up the business right now. I’ve already written about the cost of code reuse abuse.
OperationsAre you hand crafting servers for deployment? Then have a look at some thoughts on that and how by definition this introduces inefficiencies and introduces human errors. Huge cost for change.
Are you deploying into application servers without treating them as atomic upgrades? See above about introducing human errors, but also magnify that by introducing technology errors.
InfrastructureDo you have shared infrastructure? Does it cost more to provision VMs that physical boxes? Does it take weeks to provision VMs? Do you not script your environments? Have repeatable automation for creating an environment from scratch? Have upgrade/rollback automation (for both applications, databases, etc)? Then you’re incuring a huge amount of cost to the business. You cannot be agile and support change with an environment like that.
At the end of the day, DevOps, Microservices, being “Agile”, etc, are about creating a culture that focuses on reducing the cost of change. Be wary of the vendors trying to capitalize on this, be wary of your organizations embracing this half-assed, and try to keep perspective and the goals in mind regardless of your role.
@abt_programming This is complete and utter truth.
— Bisse (@SveTob) May 18, 2015
Deploying Microservices to AWS at Gilt: Introducing ION-Roller
No comments:
Post a Comment