Pascal Haakmat, software engineer at funda, shares his experience with the Backstage developer portal. These templates offer our developers pre-built components that are easy to use, accelerate integration with our CI/CD processes and promote consistency.
This is the second blog post in a series of four where funda's platform team shares their CI/CD process experience. In the previous post, we talked about our move from Bitbucket and Bamboo to Github and Azure DevOps. And we spoke about our usage of modular pipelines, to strike a balance between the need for team autonomy and the benefits of consistency across our organization.
Backstage
As a Platform team it is our responsibility to provide developers with tools and information that enable them to work autonomously within a shared architecture and towards a common goal.
To this end, we started exploring the use of Spotify’s Backstage last year, in parallel with the migration of our CI/CD systems. We mentioned this in this blog post on Golden Paths:
`Backstage is a fully-fledged developer portal that includes a software catalogue, a documentation repository, software scaffolding templates and other features.'
Backstage is one of the tools we're using to help developers navigate our software ecosystem. The portal helps developers document their services via TechDocs (Spotify’s homegrown docs-like-code solution built directly into Backstage), and read up on funda’s development methodology, collected in a series of Golden Paths. It also helps them to create new services and components, through a library of software templates.
Software templates
A Backstage software template is a YAML file with a list of parameters and a list of steps. The Backstage scaffolder reads this file and presents a web form user interface from the specification of each of the parameters. The specification is from react-jsonschema-form, a declarative web form builder around JSONSchema. This enables some control and customization over the presentation and layout of the form (see screenshot).
The way in which Backstage integrates JSONSchema/UISchema is interesting. It makes it possible to build forms with a rich user experience, including advanced elements like alternatives, conditionals, and dynamic lists. Unsurprisingly, that complexity also comes at a cost (as we found out).
The list of steps tells the Backstage scaffolder which actions to perform. The scaffolder ships with default actions for templating and interfacing with different source code repositories. Other actions are available as open source. Writing custom actions is done in Typescript and nodejs.
At funda we use a custom action, for example, to automatically create build and deploy pipelines in Azure Pipelines. This way, when developers create a new project using a software template, their build and deploy pipelines are already set up for them. We also use a generic processor action called k8s-processor, about which more below.
.NET Service API
Over the past years, funda has been moving from on-prem deployment of monolithic apps to containerized applications in Kubernetes. Additionally, we have moved our applications from .NET Framework to .NET Core, Flutter and Nuxt. To enable teams to manage their own infrastructure as code, we use Terraform. We use Helm charts to configure the deployment of a service to our Kubernetes clusters. Moving away from monolithic apps and centralized infrastructure increases teams' independence, allowing them to deliver more business value with less organizational drag.
Our initial .NET service template included the following components:
- Skeleton .NET service
o Storage (CosmosDb, SqlPaas, EntityFramework storage adapters for Cosmos DB and SQL)
o Endpoints (ASP.NET MVC or Minimal API)
o Unit Tests
o Component Tests
o Integration Tests - Terraform infrastructure
o Kubernetes config maps
o Storage (CosmosDb, Azure SQL databases)
o Azure Key Vault - Helm charts
o Routing
o Configuration - Azure DevOps build & deploy pipelines
When running the template, the Backstage scaffolder automatically creates build and deploy pipelines for the service in Azure DevOps (using the custom action mentioned above), so developers don’t have to do that.
Template tester
Testing Backstage software templates is a little unwieldy because every change needs to make a round-trip through Backstage. First, the changes need to be pushed to the template repo, then the template needs to be (re-)loaded in Backstage, before Backstage has to be told to render the template. Next, it needs to be verified that the rendered output actually builds and deploys correctly. And finally, this must be done for different permutations, depending on the selected features and options.
See also: How Golden Paths give our developers more time to actually develop
The rich feature set of our initial template added significantly to this testing load. As a mitigation, we automated some of it by creating the template-tester. This tool allowed us to render, build and test a template from the command line, locally, without having to make a round-trip through Backstage. It also ran as a checker for our template repository via a Github workflow, to ensure that new commits did not break the template.
Errors and quirks
Still, even with the template-tester in place, errors kept creeping into our templates. Sometimes the errors were caused by Backstage quirks. For example, we ran into a problem where we wanted the user to have the option to enter the name of a Helm chart, and expose that value in an environment variable, like so:
Then, to improve developer experience, we wanted to automatically remove whitespace off the ends of the string, which nunjucks can do with the trim function:
This seemed to work, until we noticed that the CHART_NAME environment variable did not always contain the expected value. In some cases, it was set to the literal value ${{ parameters.chartName | trim }}.
We discovered that this happened when the user left the input blank, which was possible because the chart name is an optional field. In that case, the trim function throws an error. Backstage catches that error, and just skips substitution of the expression.
The solution was to make sure that trim always receives some value, even if the user leaves the field blank:
Understanding and solving these sorts of issues requires fairly deep dives into the workings of nunjucks and Backstage. Arguably it would be better for Backstage to simply emit an error when the nunjucks engine emits an error, rather than to fail silently and continue.
Other problems we encountered involved the use of some of the more advanced JSONSchema features, like alternatives and conditionals. These rough edges will undoubtedly improve as Backstage matures, but they may soak up a lot of debugging time.
Modular templates
While Backstage teething problems caused us some headaches, the main source of challenges during template development was the maximalist design of our .NET service template. Our template included .NET source code for the service with a (conditional) storage component, component tests with Wiremock support, Terraform for the (conditional) storage infrastructure, as well as Dockerfiles and Helm charts for Kubernetes deployment. Despite the test tooling we developed, fully testing every aspect of the template still required a lot of manual work.
We realized that our template was trying to stretch across two dimensions at once:
A) Providing a lot of features to developers, and
B) Implementing each of those features across all the links in our CI/CD chain
The combination yielded a large complexity space, and a lot of that space offered only limited value to our developers. But all of it needed to be tested and maintained.
To address this we decided to simplify our approach and create modular templates. A modular template focuses on integrating a single feature into our CI/CD chain in a way that’s composable, allowing developers to mix and match modular templates according to their needs.
Being smaller and simpler, and containing no template logic, modular templates are easier to develop and understand. Perhaps most importantly, they are easier to understand for the developers who have to actually use the templates.
k8s processor
When a developer uses a modular template to add a feature to their existing application, we want the process to be as smooth as possible. Ideally, the template directly performs all the changes that need to be made to enable the new feature.
To support this, we developed the k8s-processor custom Backstage action. This action takes an existing repository (the target) and a modular template (the source), and sends them to a Kubernetes pod to be processed by an arbitrary script.
For example, when a developer uses our modular template to add a .NET service to an existing repository, the processor script for that modular template automatically calls dotnet sln add to add the new .csproj file to the existing .sln. We also automatically update the Helm values.yaml to create a Kubernetes Pod that can run the new service.
After the developer runs a modular template, they can view the changes via a pull request that is generated by Backstage.
Because our modular templates are simple, the changes they generate are also simple. This helps developers build a mental model of how the different parts of our applications fit together.
This way the modular templates not only help to standardize our code and increase developer productivity, but also contribute to the development of know-how within our organization.
Worth the CI/ CD journey
In moving from a (partially) on-prem development environment to a (fully) cloud-native development, we wanted to leverage the increased autonomy this affords teams, while providing and maintaining organization-wide structures and standards at the same time.
We adopted Backstage to support these goals, by providing a common catalogue for our services, documentation, and software templates. For teams to operate efficiently and autonomously, solid coordination up front is required. As they say, "good fences make good neighbours".
We encountered a fair number of issues during the development of our software templates. Some of these were typical “early adopter” challenges, while others are inherent to the fairly elaborate workflow for testing templates. But the biggest hurdle we faced was managing complexity. Our initial template combined lots of features and organizational best practices, while also leaving a lot of room for developer preference: a combination of conflicting goals that turned out to be difficult to understand and maintain.
To address this issue we moved to small, specialized modular templates, which also help developers gain a better understanding of how different aspects of our software landscape fit together. Although it took some time to figure out how we should work with templates to improve on our CI/CD processes, it was well worth it. We’re looking forward to growing our library of modular templates to help empower our developers in leveraging the full potential of our cloud-native CI/CD environment.
See also: How incident drills help us prepare for the inevitable
