The Integrator's Dilemma

The very first software developers built things from the ground up, by necessity. They wrote in what is called machine language-- simply a series of numbers representing computer operations. As time went on, various reusable components were developed. The "assembler" was surely one of the first-- simply a program that converted text like "mov, jump, add" into raw machine language.

We've come a long way since those days. The first step for most budding computer programmers is not learning machine language, or even assembly language. It is usually learning a high-level language, which will come with a set of reusable libraries. That high-level language will sit on top of an operating system.

The truth is, though we developers like to think of ourselves as innovators, we are all also, to a greater or a lesser degree, integrators, who stack together a set of pre-made components.

The Stack

The set of software components used in an application is usually known as the stack. The stack traditionally includes the operating system, the core libraries, the language, the database that is being used (if any), and the framework.

Different tiers of the stack need to be developed with different techniques. In the operating system, developers have to prioritize efficiency and stability. In contrast, at the top of the stack, the most important factor is usually development time and readability. It makes sense to use C for an operating system. For the top of the stack, in contrast, Ruby, Java, or another high-level language is more sensible. This is what people mean by using the "right tool for the job."

In my experience, developers tend to underestimate the amount of work that goes into components that they don't personally work on. At a conference, a low-level developer once told me that he could recreate the Android framework code in "a few months." Of course, this is utterly ridiculous. It took Google years to develop that framework code, even with a great team of engineers. Other companies that have developed mobile operating systems have taken similar amount of time.

There tends to be a cultural mismatch between developers who work on different levels of the stack. Low-level developers often consider the high-level ones to be idiots because of their inefficient code. Don't they know that it could be done faster and better? High-level developers tend to wonder why low-level code has the limitations it does. It's easy to support strings of indefinite length in Python; why does ext3 impose a maximum path component limit?

My Preferences

Personally, I prefer working on the middle or lower parts of the stack. It might seem like there is little scope for creativity here: don't we all pretty much know what an operating system is and does? However, what people often miss is that a big part of the fun is figuring out how to implement that functionality. The lower parts of the stack offer developers a chance to become an expert in something, which many people enjoy.

Moreover, a lot of the apparent opportunities for creativity at the top of the stack are illusory. It might seem fun to connect together low-level components in arbitrary ways to solve a challenging problem. But you probably won't be doing that when developing an application. You'll probably be working within a framework, and coloring inside the lines. Frameworks come in many shapes and sizes-- Ruby on Rails, Hibernate, Spring, the Android framework, node.js, and so forth.

The idea behind frameworks is to standardize the way applications are developed. This makes good capitalist sense. As much as possible, employers want to be able to treat developers like interchangeable parts, rather than rock stars who have to be carefully coddled. Using a framework helps ensure that other developers will understand the existing codebase.

The View from the Top

As I said earlier, most development positions are a mixture of development and integration. There are a few positions out there, though, that really are almost all integration. Here are two examples.

• Drupal developers can get a lot done without writing a line of code. A lot of sites can be built with no custom code at all, by stringing together pre-existing open source modules. Mike Crittenden writes about all of this in "Drupal's Golden Handcuffs."

• Many hardware-focused companies have what's called "product teams." These teams are responsible for integrating code from some upstream source with the internal code and hardware of the company. The upstream source might be Google, providing a base Android operating system. Or it might be Microsoft. Either way, the job of the product team is the same-- to get that code compiling on the company's hardware, and to debug issues that come up in testing.

I think that as time goes on, the top of the stack will be more dominated by integration, and less by innovation. If your company is making a phone, it doesn't make sense to re-invent the wheel. You're going to use a pre-existing operating system and adapt it. Similarly, if you're creating a web site, you're going to use a framework like Ruby on Rails, not hack around with CGI scripts and such. The day of the "in-house operating system" is drawing to a close, as Nokia and Blackberry are learning. Fewer and fewer companies are trying to be "full-stack."

It's the middle and the lower parts of the stack that will tend to foster more creativity. The middle in particular is probably the sweet spot for creativity. There is still some abstraction going on, but there is also a desire to create something re-usable and general.

Anyway, those are my opinions. Hopefully, I haven't fallen into the trap of believing my own level of the stack is the only important one. I honestly believe that every level of the stack is important, and that there are people who would enjoy working on each level. The key to being happy as a developer is to figuring out which one is right for you.