SCM and Software Integrity

A friend of mine emailed me about using control of /main versus continuous integration. He argued that a team of productive developers would swamp the select few who own /main. This would cause integration errors to pile up and slow the team down. He correctly identified this as a bottleneck. Using Goldratt's Theory of Constraints, bottlenecks limit the productivity of the system. Thus bottlenecks are to be eliminated (either by shifting resources or changing the process) if one hopes to improve productivity.

I used these controls as a way of achieving source code integrity, not that I recommended it. Achieving source code integrity can be achieve various ways. One philosophy is to place controls on /main and establish a process that is rigorously followed. This will work, but will be slower than ideal because of the bottleneck of the control. There might be situations I would favor such controls, but the situations I can think of involve low levels of trust towards the developer (off-shore development for instance).

I favor teamwork. In a functional team, all members share a common goal. A good team has a culture. I agree with McCarthy (see Software for your Head) that the software takes on the properties of the team. To achieve software integrity the team that develops the software should have integrity. In a team of individuals with integrity I expect the team to own their success. So a team like this will have ownership of checking in well written, well tested code. If a team member is unsure of their code I would expect them to seek out a better developer to validate their code before checking it in. In such an environment mistakes will happen, but they will self correct. If you do not trust your team then high levels of productivity are not possible as controls will slow your team down.

Keys to Software Development Success

After some thought and reflection I have deduced two key aspects to successful software development:

1) Focus on teamwork (lack of too many teamcide practices)
2) Use a process oriented around the integrity of the source code

The first key seems straight forward, but somehow many organizations take it for granted. The word teamcide might not be familiar to you (read Chapter 20 of Peopleware). DeMarco & Lister were attempting to write a chapter about how to make teams jell. They gave up and instead wrote a chapter on ways to prevent teamwork. It is very insightful. I have found that even when an organization goes out of its way to kill teamwork a strong leader can overcome it. First you must realize the problem before you can overcome it.

The second key aspect can be interpreted many ways. I will try to further explain myself. One of my first jobs was as a software developer for Motorola. Some VP had decided we needed to be CMM level 3 so every developer was forced to work 4 hours a week on process improvement. I was young and got involved as a capability area lead. After reaching level 2 I saw improvements and when I started a new team I carried over many of the practices that I had learned. When I look back on what worked two things come to mind:

1) Unit Testing
2) Strict control of /main in SCM

Both revolve around the concept of keeping the code in the best shape as possible. Unit testing prevented many bugs from ever being merged into main. In fact only two people could check out from main and not without running the entire suite of automated unit tests.

Jumping to present day; I work with a vendor that is struggling to produce quality code in a timely manner. When I look at their process I see no real SCM - all code is checked into main by anyone and there are no automated unit tests. All tests are GUI driven and run manually. The sad thing is they have a fairly strict process. They are not 100% compliant to it, but I bet with just a few tweaks they could achieve SEI CMM level 3 and still struggle to produce quality code in a timely manner.

Many people are caught up in the Agile craze these days. Agile is best described by the manifesto (http://agilemanifesto.org/):

- Individuals and interactions over processes and tools
- Working software over comprehensive documentation
- Customer collaboration over contract negotiation
- Responding to change over following a plan

I truly believe that you could follow this manifesto and still struggle, but I find it hard to believe that you could follow my keys and struggle. I might be naive, but almost all problems I have run into in my 16 years of professional software development experience come down to poor communication or teamwork (they are extremely related) and a lack of understanding of the importance of the source code. Many processes place the same emphasis on documents as the source code. In fact many people believe the same processes that work for documents work for source code. The only tools available to written documents that I can think of are: human reviews and version control. Source code has the unique ability to be compiled and tested all without the ever erring human mind. I am not saying human review is useless, it is just insufficient.

One thing I have said little about is using iterative development versus waterfall. Part of me thinks it isn't worth saying anything as the waterfall process has been completely debunked, but some people still cling to it. I will simply say that the you can never achieve high levels of source code integrity if you start with a big ball of untested code. The waterfall process will start with such poor source code integrity that you will never be able to reach a high enough level to acvhieve success. You must start small to achieve high integrity and grow a little at a time. The human brain isn't big enough to make large leaps in source code without loosing integrity.