Thoughts on code development, access technologies, and telecommunication networks from developers within ADTRAN’s R&D organization.


Written by:
Nathan Alderson - @nathanalderson

Building complex systems is, by definition, hard. The more components, technologies, developers, product managers, and whiteboards that are involved in designing a system, the more likely it becomes that a project will struggle or fail to meet requirements, deadlines, and budgets. Dealing with this complexity is the practice of software architecture. A good architecture encapsulates complexity, facilitates collaboration, enables evolution, and supports a sustainable development cycle.

Describing the ideal architecture is, of course, impossible. Every system is different and calls for different choices. Even given a complete requirements description of a desired system (potentially a laughable thought in itself), there could be many viable architectures. Once a project reaches a certain level of complexity, however, one thing is certain: the architecture must be intentional. For a good discussion about why this is the case, take a look at the Scaled Agile Framework's discussion on Agile Architecture.

As we started the ADTRAN Mosaic Cloud Platform project, we knew we needed to be intentional about our architecture. However, we also knew that architecture–even good architecture–done poorly can lead to its own problems. So we began by capturing some principles about the architectural process that would guide us:

  • Architecture is vision: There are many definitions of architecture, but at its core, architecture is an idea–a shared understanding of the important parts of a system. Architects are responsible (in collaboration with others–see "Architecture is collaborative") for defining, communicating, and executing that vision.
  • Architecture is code: Architectural vision can–and often should–be captured by documents and diagrams, but we believe it is best conveyed by working software. Architects, therefore, don't just write documents and draw pictures, but build software. This also ensures that architects personally understand and experience the practical implications of their decisions.
  • Good architecture is supported by tools and infrastructure: Architectural decisions often affect the development process. Architects, therefore, should support their architectural decisions with tools, processes, and infrastructure so that the "right" path becomes the easy path.
  • Architecture is a craft: Like programming, software architecture is a skill which requires continual development and practice. The software industry changes rapidly, and architects have a responsibility to invest time and effort in the continual improvement of their craft. This may include practices such as investigating new technologies, contributing to open source projects, participating in user groups, and reading books or articles on related topics.
  • Architecture is collaborative: Architects should not work in a vacuum or "ivory tower," but should engage with as many stakeholders as possible, as early as possible. Collaborative solutions should be considered preferable to "ideal" solutions. Developer teams likely have real world experience that can influence architecture, and buy-in enables wider adoption.
  • Good architecture is cohesive, but flexible: Good architecture should allow as much autonomy as possible to developer teams and allow for adaptability and innovation as business needs change. However, it should also clearly define and enforce standards around those decisions where variability would hinder the effectiveness and cohesion of the system.

Capturing these ideals in writing was a good first step, but how do you turn these concepts into a reality? We decided to build a platform to encapsulate and support our architecture. In our definition, a platform represents several things:

  • A set of architectural principles upon which we can all agree.
  • A set of services and libraries which enable applications to be built on the architecture.
  • A process to deliver software utilizing the architectural principles.
  • A set of tooling which enforces the process while making development of products easy.

We also realized that we would need a dedicated team to support the platform. This team would be responsible for collaborating with all of the stakeholders in the ADTRAN Mosaic Cloud Platform and pouring the architectural runway needed to keep development moving forward. They would also build the tools, libraries, and processes necessary to make developing in line with the architectural vision easy, allowing developers to focus on adding value to the product rather than worrying about things like deployment models or database backups. Lastly, the team would also be responsible for helping developers and development teams come on board, providing training and assistance as necessary.

The platform that came out of this effort is Firefly, introduced in two previous posts (Part I and Part II), and my team is the primary team supporting it. We definitely aren't the only platform team at ADTRAN, but having worked as a dedicated platform team for some time, we've learned some tricks that we thought we'd share:

Write down your first principles. This includes your architectural philosophy as described in this post, but also the ideals underlying any platforms that you build. The principles behind Firefly were described in its second introduction post. Doing this and getting buy-in early will be invaluable as trade-offs must be made later and various decisions must be justified to stakeholders later on.

Get the right kind of people. Compared to a traditional team, there are some traits that are especially important for members of a platform team: the most important trait for a platform team member is fearlessness, followed closely by helpfulness. You need folks who aren't afraid of a blank page or new technology, and people who are willing to provide lots of support to others as they use the things your team builds.

Prototype often. Speaking of blank pages and new technology, you're not going to get everything right the first time. Our team tends to approach any new technologies and solutions with skepticism, so we always budget time to actually build a proof-of-concept or demo before we commit to something.

That said, don't get trapped in analysis paralysis. Some problems are so big that fully exploring all of the possibilities is impractical. Timebox your investigations and then commit to something with the knowledge that you can always change it later. Our team completely switched databases pretty early on because we realized our original choice wasn't right. If you are particularly unsure about a technical choice, consider introducing a layer of abstraction so you can pivot more easily down the road.

As a platform team supporting internal customers, it's also very important to us that other teams in our organization perceive us as highly responsive. We want to allow plenty of time to respond to design reviews, answer email questions, or go sit with someone for a debugging session. We're a scrum team, but we always allow a big buffer in our velocity to account for unanticipated support work that comes in during an iteration. We adjust the buffer size as needed based on the number of other teams–especially new teams–working with our platform. Of course, if a debugging session turns into a big investigation or a quick bug fix turns into a major feature request, we'll schedule that on the backlog with other priorities, but many things can be handled quickly and the goodwill earned by turning out a bug fix version of a library for a frustrated developer in a few days instead of weeks is worth the trade-off in predictability for us.

Finally, we've learned that whenever possible, physical, in-person contact can make a huge difference. For example, we make a habit of sending one or two team members to physically go visit other teams as they are coming on board with the platform or if they're struggling with some new feature. We call this practice "doing rounds" and we'll do it daily whenever there are teams that can benefit. We've gotten immense benefit from this both in terms of accelerating other teams and also by improving our own sense of how the platform is being used by others. In a more extreme form, we've even been known to embed a platform team member on another team to help them get up to speed quickly. Usually a single iteration is enough to bootstrap a team in this way. For other teams, we have instead chosen to schedule a one-or-two-day hacking session where our whole team and their whole team will find a place to sit together and build a prototype of their application (or even just a toy application) together.

Building complex systems will always be difficult. However, developing the right architecture will provide the road map you need to navigate through the maze of complexities. Furthermore, by embodying that architecture in a platform, and by supporting the platform with a team, you can turn that architectural map into a full-fledged GPS, constantly helping the user figure out where they are relative to the rest of the world and giving them customized directions on how to proceed toward their destination. We hope that some of these techniques might be helpful for you and your team. What other ways have you used to support your developers? I'd love to hear about your experiences over on Twitter. You'll find me at @nathanalderson.

Archived Blog Posts