Written by: Jonathan Hood - @jonathan_hood
Published: 26 August 2016
Several years ago we were faced with a dilemma. The industry was changing and we could see it coming. Terms like SDN and NFV were starting to appear on the conference circuit. Carriers were becoming tired of vertically integrated monolithic devices and software stacks. Vendor lock in was becoming an issue at the forefront of everyone’s mind. A migration to more open source technologies was becoming evident both within our own company and those around us. These were the tidings of change. There was another real, very simple reality. Few, if any, knew exactly what was about to happen. What would our users expect from these new systems? How will we make them perform well? How are we going to deal with these new products that we haven’t even seen yet?
There are no simple answers here. In true engineer fashion, we decided the scientific method was best. We would hypothesize a solution and test that hypothesis with our users. In the software world this means more high-quality releases so that we can garner feedback on faster timelines. We couldn't wait months to get feedback on the functionality of our product - the industry was (and still is) simply moving too fast for timelines of that scale to be tenable. We needed feedback cycle timelines on the scale of days or weeks if we wanted to succeed.
So, we decided to adopt Continuous Delivery as a core principle of our software platform. We were even given a challenge by our leadership: to create releasable high-quality products in a matter of minutes. Doing this requires a fundamental rethink of how software is managed, tested and delivered. A small example: the versioning scheme of ADTRAN Mosaic Cloud Platform is a simple <year>.<release number> rather than more complicated schemes of the past. Even this simple step has removed previous manual error-prone steps from the release process making it more efficient, predictable and repeatable.
(Figure 1) Continuous Delivery Process
In our process, we decided on three key types of artifacts. Components include items like code libraries, micro services, user interfaces and other modular product pieces. Products (for example a G.fast DPU or Mosaic Cloud Platform) include combinations of these components along with some minor integration code if required. Systems include combinations of products required to achieve a full use case (for example Mosaic Cloud Platform and a set of Mosaic OS-based network elements). In addition many of ADTRAN's products require on-target validation of components and products. For example, a data plane component needs validation against the actual data plane hardware it is managing.
We expect high-quality from this process. Faster is not better if the result is sub-par. To this end we dictate and monitor high levels of test coverage at many levels of the system. Tests run fastest at the component level (where unit testing may be used) taking a matter of minutes to build and execute their full test pipeline. We use this capability to achieve fast feedback for our developers. Products by nature dictate longer build times and longer test times. These tests execute in production-like environments and focus on integration of components. System level tests then require longer test times again, but represent the most production-like environment for testing.
(Figure 2) Continuous Testing Stages
Enabling these forms of testing is a key reason behind the micro-services architecture chosen for Firefly. Using micro-services allows us to break the system along a wide variety of boundaries for the purposes of testing. We can create small and quick tests focused on a single micro service, slightly larger integration tests which combine some micro services together and bigger product or system level tests that compose the system in its full form. Each of these tests has its place, but by focusing most of our testing on smaller components of code we can achieve confidence in our quality on fast timelines.
When looking at overall cycle time, we rapidly noticed that our pipelines spent a significant amount of time waiting for limited test resources. Along the way we've begun the process of virtualizing our test infrastructure to make it more flexible. Our test pipelines can quickly request a virtual resource be allocated for any purpose. This has allowed us to both optimize our usage of expensive test infrastructure while also leveraging parallelism in our test process which speeds things up overall.
(Figure 3) Virtualized Test Infrastructure
Our developers, testers and the delivery pipeline itself can use this infrastructure to stage and execute integration tests in a single command. We consider this infrastructure so important that we have teams focused on making sure others can easily work with this infrastructure without caring about the details of deployment, container orchestration or system tear down. New components even come batteries-included with the ability to execute tests on our virtualized infrastructure. This allows our teams to go from nothing to a working and well-tested micro-service in a matter of hours or days.
How long does it really take to create a release? Not minutes yet. We can certainly do it in an hour or two. This is a constantly evolving process for us. If you’ve ever been told that DevOps and Continuous Delivery should force big changes in your process, believe it. We are constantly evolving to make things quicker and easier.
We have used this capability to great effect at several customer trials. In one instance we deployed a trial and discovered an issue, reproduced the issue, resolved the issue, created a new high-quality release, tested said release through our pipeline and gave it to the customer within a 24 hour period. This kind of speed and confidence would not be possible without our continuous delivery pipeline. In previous products the same confidence would take orders of magnitude longer - sometimes weeks or months depending on the scope of change required.
All of this so that we can get that rapid feedback on the product, we can deliver new features faster and we can get valuable user experience testing. It’s that important to us that the product does the right thing rather than just doing something. This mindset has changed how we approach software in a pretty fundamental way. We focus on fast feedback both internally and with our customers. This appeals to the engineers in us - we get to make a hypothesis and capture it in the product, but then test that with our users. We can do this quickly and adjust our assumptions as needed. It also appeals to the business side of our company. The "stickiness" of our product improves as many customers like the early look at the product and the opportunity to influence it at an early stage, and we improve productivity by focusing on features that bring value.
ADTRAN, Inc. is a leading global provider of networking and communications equipment. ADTRAN’s products enable voice, data, video and Internet communications across a variety of network infrastructures. ADTRAN solutions are currently in use by service providers, private enterprises, government organizations, and millions of individual users worldwide. For more information, please visit www.adtran.com.