Thoughts on code development, access technologies, and telecommunication networks from developers within ADTRAN’s R&D organization.




Thoughts on code development, access technologies, and telecommunication networks from developers within ADTRAN’s R&D organization.




Thoughts on code development, access technologies, and telecommunication networks from developers within ADTRAN’s R&D organization.


Written by: Nathan Alderson - @nathanalderson
Published: 30 September 2016

During World War II, with the advent of Blitzkrieg-style warfare, it suddenly became clear that entire divisions of armored vehicles needed to be able to advance quickly over varied and unpredictable terrain without losing speed. One common challenge was the need to cross streams and other obstacles without having to concentrate forces at vulnerable fords or bridges. This led to the introduction of armored vehicle-launched bridges (AVLBs). Rather than building a new bridge, these vehicles were capable of deploying improvised bridges quickly and on short notice.1 This AVLB from 1960 is capable of deploying a bridge spanning up to sixty feet in just three minutes. Once the bridge was laid, tanks, jeeps, troop carriers, and all manner of support vehicles could cross rapidly. Finally, the bridge layer itself could cross and retrieve the bridge, ready to deploy it again at the next obstacle.

An M60A1 Armored Vehicle Launched Bridge (AVLB), deploying its scissors-type bridge.2

Today, in the software industry, the advent of agile development practices has made it clear that software teams need to quickly develop and deploy features in an unpredictable market without losing speed. Like the streams and other obstacles presented by the terrain of a World War II battlefield, our teams often encounter barriers to their progress. They need tools like the AVLB which allow them to proceed, even when faced with these unpredictable challenges.

At ADTRAN, we have developed many such tools. Two of these have been particularly effective at enabling developers and non-developers alike to go faster and be more effective while working with our Firefly platform (introduced in a previous post).

The first is a tool we call kaylee. Named for the cheerful and highly competent ship's mechanic from the Firefly TV series, kaylee is a tool written in Python intended to make developers' lives easier. Kaylee does highly domain-specific orchestration of docker containers (via the excellent docker-py library). A single command, kaylee dev start, will start a number of docker containers on the local machine representing a functional Firefly instance. The user can then run kaylee dev install ... to install additional microservices (along with their dependencies) into that local instance. This would be tedious to do manually, but is simple in Python. The docker-py copy and get_archive functions only support copying files out of the container, so we first subprocess out to the docker CLI to copy the tarball in:

def _docker_copy(container, src_path, dest_path):["docker", "cp",
                     "{}:{}".format(container, dest_path)])

Then we install it:

def _install_app(self, container, tarball, src_path):
    app = os.path.join(src_path, os.path.basename(tarball))
    self._docker_exec(container, cmd="python {}/firefly-init/bin/firefly_init install {}"
                                     .format(self.basepath, app))

This was originally intended to allow a microservice to be rapidly deployed and redeployed from the local file system during development. However, we also included the ability for the tool to pull the microservice directly from our artifact repo, Nexus. As a result we have already seen it being used with some light scripting to build up more complex deployments for exploratory testing. Other simple commands exist for tasks like viewing the logs, obtaining an authentication token, and stopping the server. While some of these features could be accomplished with a more general tool like Docker Compose, our tool is able to be more useful by being custom-tailored for our ecosystem of containers. Also, by unifying under one tool, the learning curve is smaller for new teams and individuals as they come on board with our platform.

Besides starting and controlling local Firefly instances, kaylee can also scaffold out new projects. By running kaylee init and answering a few questions, a developer can quickly get a new Firefly microservice project bootstrapped in either of our two supported languages (Scala or Python). Things like the project name are automatically populated in all the right places.

~   » kaylee init
Which programming language would you like to use?
We can currently provide templates for Scala and Python.
This will be used to set up the project in your preferred programming language.
[scala] >
What would you like to name your project?
Use dashes to separate words.  For example: "performance-monitoring"
This will be use for lots of things, including your top-level project folder, your
package name, and will show up in documentation, config files, and probably other
places. We also use this value to derive several variants which will appear
in yet more contexts.
[example-project] > blog-posts-service

We maintain the scaffolds as tarballs on a centrally accessible server, so our team can easily keep them up to date with the latest best practices. The scaffolds are written as Jinja2 templates which are then rendered using the answers provided by the user during kaylee init. This prevents the less ideal approach which otherwise tends to occur, where new projects get started by copying the source code of an existing project and then modifying it. This leads to copy-paste errors as well as the propagation of outdated patterns.

The other tool I want to talk about is one we call Testbeds as a Service, or TBaaS (it rhymes with "sea bass"). TBaaS is a service that creates testbeds on demand in a private cloud. Testbeds are defined in JSON, and clients can use either a simple REST API or the provided web GUI to create a testbed. Here's a screenshot of the create-testbed dialog in the TBaaS web GUI.

Creating a TBaaS Testbed through the Web GUI

Testbeds are composed of one or more resources. You'll notice that the JSON object shown above contains a single resource named "my-virtualbox" which has type "VIRTUALBOX_OVA" and a single property indicating where TBaaS can download the required OVA file. Similarly, TBaaS can also launch generic docker containers by setting the type to "DOCKER_CONTAINER" and specifying the image name (which it will pull from either our internal docker repo or the main docker hub). This can be very useful. However, the real power comes when teams define their own resource types. For example:

Creating a Testbed with a Firefly Resource

Here we are creating a resource with type "FIREFLY." A Firefly resource already knows which Docker containers to pull and run, and how they should be configured and linked together. The properties it provides are specific to configuring a Firefly instance. The "Templates" dropdown provides JSON templates for a dozen or so different types of resources that have been built by various teams, including ADTRAN Mosaic Cloud Platform instances, virtualized TA5000 shelves, and mock NV8424's.

Creating a new resource is also done in Python, and is very straightforward: just extend the TestbedResource class and implement a handful of lifecycle methods (validate, init, wait, destroy). All resources are based on Docker containers. When the service constructs your resource, you will be passed a docker-py client object, which you can store and use in the other lifecycle methods to control your resource. Even the VirtualBox OVA resource type launches OVAs using a Docker container. Basically, if it can be dockerized (and what can't?), it can be a TBaaS resource. By the way, the Firefly resource actually shares an underlying library with kaylee to do all the heavy lifting.

Need a testbed with a cluster of three Mosaic CP management systems, an SDN controller, and some simulated DPU's (along with their Persistency Management Agents)? Fill out the JSON template, click a button, and a few minutes later you'll get back a bunch of IP addresses. We've even integrated both kaylee and TBaaS with a Gradle plugin, so Firefly microservices can write their integration tests right along with their code, and then run them either locally or on TBaaS with a single gradle command. The plugin takes care of starting a Firefly server, installing the microservice under test along with any declared dependencies, and running the tests. This also makes CI super easy!

Kaylee and TBaaS give our users a fast, convenient, and flexible way to deploy our software for a multitude of purposes. Developers use them to experiment during the development cycle. Both testers and the CI infrastructure use them to reproducibly run integration and system tests in a production-like environment. Teams use them to bring new engineers on board quickly. Sales engineers can use them to spin up the latest software for exploration or demos. Tech support engineers use them to create a sandboxed deployment with all the same component versions as a customer who is having issues. By building flexible and extensible tools and by supporting them with engineering resources, users are continually coming up with new uses for these tools. Just recently, a developer created a WAN emulator resource for TBaaS based on WANem to use when testing redundancy between geographically diverse datacenters.

We don't have time to stop and build a whole new bridge every time we hit an obstacle. With kaylee and TBaaS we don't have to–we just deploy one, instead.


ADTRAN, Inc. is a leading global provider of networking and communications equipment. ADTRAN’s products enable voice, data, video and Internet communications across a variety of network infrastructures. ADTRAN solutions are currently in use by service providers, private enterprises, government organizations, and millions of individual users worldwide. For more information, please visit

1 Armoured vehicle-launched bridge, Wikipedia

2 By Kevin Quihuis Jr., Public Domain, WikiMedia Commons

Archived Blog Posts