Does this sound familiar? If so, you probably deal with a lot of frustration and time mucking about with all of the things that have to be done before you can actually get coding and move your project forward. In other words, everything you need to have installed and running so you can do your work.
Strangely this “time suck” is just accepted. It’s generally considered to be just part of the process of software development, buried within “setup time” and “environment configuration”. So your frustration rolls up into your development costs. But your frustration actually equates into hours, days and months of wasted time.
Breaking Down the Time Suck
It can take hours and even days for a developer to get their initial setup completed. And this wasted time isn’t just a one-time cost. It happens all the time, when you first start a project, as well as throughout a project’s lifecycle when:
- Languages get updated,
- Components get upgraded or switched around,
- New dependencies get added,
- Operating systems get updated.
All this wasted time adds up into big costs. Yet the costs are “invisible”. Your team is incurring them and you’re losing developer productivity and project velocity. And since you’re not tracking, accounting, or understanding the wasted time and associated costs, you’re incurring greater risk. These big hidden costs are a part of the reason why software projects go over budget. For example, in the likely occurrence that something breaks during a must-have upgrade, it cascades to effect deliverables, budgets and productivity. And, over time, your build requirements will grow, as your project increases in size and scope, so do your “invisible costs”.
Dependency Management & Environment Configuration
In our 2018 developer survey, 76% of respondents said that part, most, or all of their time was spent managing dependencies and development tools. In fact the issue of managing dependencies and developer environments has become so topical, even always insightful XKCD webcomic got in on the act:
Source: https://xkcd.com/1987/
Almost any software project of any real size and scope can quickly degrade into a mess on developer’s machines, as comically illustrated above, and all of your engineers will experience it. Organizations will need to begin to invest time in standardizing the process to get development environments configured easily. In other words, an investment is needed in build engineering.
The History of Build Engineering
Build engineering has no standard definition, but an early reference to the concept can be found in Stuart Feldman’s 1978 paper “Make – a program for maintaining computer programs.” Feldman states, “What is needed is a mechanism for keeping track of the status of the parts of a program, and for executing an efficient control sequence to ensure that the end product is correctly made from consistent pieces.“ Build engineering is just that: the process by which “…the end product is correctly made from consistent pieces.” The actual scope of build engineering is much larger than just the capabilities of make or other build tools/systems. Effectively build engineering is encompassing all of the tools, languages, processes, and components required before the real work can get done.
Very few standards and best practices exist today for build engineering. Often times it comes down to the knowledge of a few experienced engineers on a team, aka tribal knowledge. These build engineers document and codify the steps, and provide standards for build environments consisting of tools, languages, and dependent components. They may also build from source languages and dependencies to be provided as artifacts to their developers. In most cases, the origin of the “read me” is to get your setup done. Who wants to go through the dreaded readme steps just to set everything up before you can get productive? We should be working to distill all the steps to a single step whereby you specify your build and install it on-demand.
And with respect to languages, each one has its own way of doing things. Each language has its own set of tools or combination of tools to build it and its dependencies. And most of the tools that are built today are specific to a language, with the exception of some of the build systems that are polyglot. In addition, each language has its own component ecosystem and way to manage those components. Given all of this you can see the complexity that arises from modern polyglot environments. Oh and did I mention build flags, custom OS libraries, compiler versions, and other build engineering factors?
Reducing the Hidden Cost of Build Engineering
You can reduce your hidden build engineering costs, but first you have to know what they are. You need to track them and iteratively, budget and account for them. They need to be included in your onboarding process (and costs) as well as identified in software estimates. Ensure the time for build engineering is captured in project costs and project velocity estimates. We all need to practice rigour in tracking these processes so we can shine a light on how much time is wasted. We can’t improve the situation if we don’t elevate its importance and start thinking about solutions.
At ActiveState we’re working on a Platform that will enhance the quality of life and productivity for developers everywhere. Sound ambitious, yah, it is. We are building a Platform that will enable the automatic building, certifying and resolving of language distributions. And… some extra stuff to make life even easier on developers. So development teams can focus on coding and spend less time worrying about dependency hell, third party vulnerabilities, secret keys, and inconsistent developer environments.
We want to automate the build engineering process for open source languages. Our vision is to do this for every open source language and operating system platform on the planet (and beyond). We need your help to do it!
Interested in trying out Beta functionality? Create a free account. Want a demo and see if you’d like to help us define our roadmap? Schedule a Quick Call.
