Lean 101

Lean is a philosophy derived from manufacturing (see Lean.org for more details) that has made inroads into many things, including software development. When people try to adopt it they tend to focus on the techniques and often miss the big picture. When faced with a task, how you think about it will determine how you attempt to perform it.

Take laundry for example. The most common method is to first sort all the dirty laundry into piles of like colors. Then wash them one at a time and as they wash throw them in the drier. As the dry hang and fold and/or make piles for each person to take care of their own. This divide and conquer approach is usually tool #1 in our tool-belt.

A lean approach would start at a different place. Lean would start on value. Why do we do laundry? What's the end goal? For me it is clean clothes in my closet. If I could have elves do it for me while I slept I would be satisfied. Since I can't seem to find any nocturnal elves willing to work for free I am stuck doing it. I start with four laundry baskets (and some floors) filled with dirty clothes and I need a process for getting these clothes cleaned and put in the appropriate drawers and/or closets. The clothes need to be cleaned in the washing machine, dried (or laid out for those pesky clothes) and folded/hung in the correct location.

Lean tells us to pull value, instead of pushing it. What does that mean? Pushing can be thought of as focusing on the sub task. Let's take the first sub task when doing laundry - sorting. Pushing is doing all the sorting before starting a single load. You make sorting the goal before you move to the next task. This can lead to suboptimization. In lean we are trying to maximize value. One tool lean has is to minimize work in process (WIP). WIP isn't value and utilizes resources without providing value. In my domestic example WIP is piles of laundry on the floor. So how do I get laundry done when I don't have pre-made piles to throw in to the washing machine?

1. Take clothes that can be washed together from the dirty clothes hamper containing the most import clothes (mine) and toss them in the washer. 
2. When they are done toss them in the dryer.
3. Go to #1 until all clothes are done

A problem this approach can run into is when I focus on maximizing the amount of clothes I can stuff in the washer. Doing this is similar to maximizing the output of a machine on a factory floor. When I do this the clean, wet clothes have to wait for the dryer. As I am maximizing the amount of clothes I put in the washer my WIP is bigger than it should be. Kanban (a practice from lean manufacturing) uses WIP limits to prevent this from happening. I simply need to learn the amount I can place in the washer so that the washing (a fixed time on my machine) matches the drying time (a variable time on my fancy dryer). This is the Lean principle of Perfection.

In software development lean tells us to pull the value. Value is software that is being used by those that paid (at least indirectly) for the software. One common mistake when adopting a lean based software development approach is to force the business customer into breaking cohesive features into bite size ones that are of NO value to the business by themselves. Instead the development team needs to own the break down of features into bite size one and deliver the cohesive feature in a way that the business can use. Sometimes the feature can be delivered piecemeal, but often the feature is not useful until all parts (think CRUD) are done.

The other mistake is trying to to the entire feature at once. This is comparable to shoving all the clothes in the washer. You need to find the smallest unit that works for your group. In a small group (one-two developers) in can be tackled in extremely small increments (one test at a time in TDD). In larger groups more coordination will be needed. It will be up to each group to find the sweet spot (like how many pants can my drier dry in one wash cycle).

One thing to keep in mind when adopting a lean approach is there is NO ONE RIGHT WAY. Every situation is different and requires different practices and rules to be optimal. The key is the Perfection principle. Never be satisfied. Always look for ways to make the software better, faster and more valuable to the business. Sometimes you will try ides that cause a step backwards in productivity, but these are a step forward in knowledge. Remember perfection is not achievable, but with lean you should always be seeking it.

Post Agile

I cut my teeth as a software professional in the 90's as my division went from CMM level 1 to 3. I was a BDUF guy and would brow beat anyone who disagreed. Needless to say, I was unbearable in my youth! I eventually grew up (to some degree) and learned that I didn't know everything and began to question the BDUF mentality. I read Beck's Extreme Programming Explained in 2000 and stumbled on to Agile Manifesto in 2001. I wasn't convinced at first but the ideas bounced around for a while an by 2002 I was convinced.

I preached Agile to anyone that would listen. I wrote about it, taught it, practiced it and even lived it. I guess you could say I was an Agile disciple. My career has taken me places where agile was used in ways that I would call non-agile and in ways that seemed agile but didn't work well. I began to think that maybe agile wasn't the best way to work. Don't misunderstand me, it beat BDUF hands down, but there where things that I wasn't convinced of:

• Why do we need a definite sprint/cycle time?
• Why does the customer need to split his ideas to fit the developers way of working?
• Why spend time on micro-estimates?

I had read a lot of Lean books back in my agile craze (2002-2005) but never could find a practical approach. I had even had dinner with the Poppendieck's and questioned how they made the leap from the Lean principles (value, flow, pull, perfection) to their lean software principles in their book. They were convincing, but I was skeptical (a common theme).

In my current career I get to wear many hats. I support live systems as well as writing new software. It took a while for me to find a process that worked. I couldn't do my one week iteration as I couldn't commit to a single day due to the multi-hat issue. My customers were not very available and features were chunky. I did have a co-worker so I could pair and TDD so I went for it. We did one hour blocks, scheduled each morning based on our work load (he had his hats to juggle as well). We still used estimation, but I found that to be a waste of time as we never used the data. We used feature slices (take a feature and define a chunk of work that was visible and could be done in less than an hour, no internal only chunks. I have examples buried in this post http://mwpolen.blogspot.com/2007_04_01_archive.html) The results were good. We met dates even though some weeks had us meeting only five times. Our quality was good and the customers were happy.

Lately I have been reading a lot on kanban (9 stuffs to date) and like the way they focus on pull. When I find myself in an area where it is not clear what is right I tend to go to the root principles. Hmmm what are good root software principles? I'm not sure I have any, so I will use Lean's as documented in Lean Thinking (value, flow, pull perfection). Value and is easy to me as agile got the community focused on that years ago. Pull seems easy as well as I always pull the highest priority item and I think the agile community has done a good job here as well.

Flow...that's tricky. Software doesn't have pieces that come together in a manufacturing sense. I think software is design so I will use flow to mean the flow of the design as documented in code. I am thinking of a flow of a river and how I don't want trees caught on any rocks in my river (which is a metaphor for my design). I like that idea...probably been used before but my google skills can't find it. Then flow is a clear design (clear river path) which is represented by clean code.

All that leaves is perfection which I like to state as continuous improvement. This one is interesting to me. The thing I have always done through my career is try to improve. Even in my BDUF days I was always looking for ways to make a better design. Agile talked about this and tended to box this inside a practice commonly called a retrospective. I was a big fan, but I have found that retrospectives are hard to do correctly. I think the reason is that is you box the principle then it's easy to push the box around. I think continuous improvement has to be a focus every day, every hour. There is no time that is a bad time for improvement.

I think I have a better understanding of where I am philosophically. I'm not sure agile as a term works for me anymore. It did a great job pushing the software off of the waterfall tower. I will always think good thoughts about the movement and I think all software developers owe a great deal to those authors who banded together and pushed and pushed. Now it's a snowball picking up things they never intended, but that's the law of unintended consequences. I think lean and kanban can push us towards a being better and I hope they do as I still see lots of software that doesn't do the job it should. We have come a long way, but there is so much more to do. I guess that's what draws me to software development. No matter how good you are there is always room for improvement.

Handling Multiple Small Projects

Anyone who has worked in IT has had the situation in which you are assigned to more than one IT project. For this post I will use the term IT project to mean a body of work needed to accomplish a recognizeable business function. It may touch one system in one file and it may touch 7 systems in 11 different ways each. Typical IT management thinks like an accountant:

Let's see, I have 6 1-month projects and 3 developers  (Mark, Luke & John). I'll give 2 each and tell the customer that they will all be done in 2 months! Man I'm good.

Obviously this is a straw man, but it is the general thinking I have seen and read enough times to believe it is prevalent. The only way I know to analyze this is with some basic math. Let's assume that the estimates are accurate. Let's assume that each developer is equally productive (lets call it P). Let's call the work to be done on each project W1, W2...W6. Lets call the time to complete each project T1, T2...T6. So the basic formula is Tn = Wn/P.

All seems good, but IT work doesn't work like painting a fence. In painting a fence if you have painted 1/10 of the fence in one day you are pretty sure than in 10 days you wsill be done. IT work almost always follow an S curve.

You start off getting little done, followed by moments of massive productivity, followed by some painful moments of finishing/fixing the project. Agile/Lean techniques can help straighten out the curve, but it still isn't a line and I'm not sure it's a healthy goal.

So if Mark is working on projects 1 & 2 and 1 happens to get hard Mark is likely to spend more time on 1 than 2. Which means 2 suffers becuase 1 was got hard. So instead of finishing them both in two months they both slip a week or three. The other problem is that it is a known fact that focusing one one project at a time is fater than focusings on 2.

Another problem is the poor return on investment. By definition, IT projects are being funded by a business of some sort. This business usually spends money for one of two reasons, either it will save money or increase sales. Accounting 101: Time vlaue of Money (PV = FV/(1+i)^n) means the early you save/make the better off you are.

Scenario A (projects in parrallel)

PV1 = FV1/(1+i)^2

PV2 = FV2/(1+i)^2

Scenario B (projects serial)

Project 1 goes first

PV1 = FV1/(1+i)^1

PV2 = FV2/(1+i)^2

PV2 in both scenarios are equal and it we assume i>0 (which is true except in cases of deflation) then PV1 for scenario A is always less than PV1 for scenario B. BTW this even assumes the two scenarios take the same amout of time.

Therefore delivering 2 projects in 2 months is worse than delivering 1 project per month. The key in any time management course/book/workshop/blog is focus on the important things first. The key to multiple projects is getting the business to prioritize and have IT to focus. 

In my little example let's assume the number represents the priority. I would get the 3 developers together and have them decide the most productive way of tackling the list. They might decide they all can contribute to 1, but only one has the knowledge for 2 and while that is going on the other two can tackle 3 and so on and so forth. Then have them use an estimation technique (e.g. planning poker is fun and has on-line support).  Do some simple planning to come up with estimated completion dates for the customer and you are going to do better than most. Of course, if you have a true Agile team then simply work them in priority order.

Jim Shore on what seperates high performing agile teams from just good ones

I found a talk by Jim Shore at the Dallas chapter of the APLNtalk on what separates the great teams from the average agile teams (it is over an hour long):

http://jamesshore.com/Blog/Secrets-of-Agile-Success-Live-Recording.html

Since I had no good ideas this morning I just took notes.

1. High Bandwidth Communication

High performing teams sit together. People complain that it gets noisy. Pair programming can help with the noise because the adhoc communication is less chaotic (mostly between pairs and not cross the table all the time). They also are cross function and have an involved product owner. Get real-time feedback on features and priorities. Without one the questions take more time to get answered correctly.

Teasley study shows that sitting together doubles productivity and time to market is one third:
http://www.sciencedaily.com/releases/2000/12/001206144705.htm
If you separate a team because of convenience is it worth the cost.

2. Passion

Cult of quality: the believe they are special and are accomplishing something special.

Talks about teams forming: http://en.wikipedia.org/wiki/Forming-storming-norming-performing. In his opinion it takes months, up to and over 12, to get to performing. As teams really form they are inwardly focused and from the outside they look weird. He warns that you should use cross functional teams to prevent this teamness from harming the organization. Sometimes organization break up the team in the norming phase, before performing. The fifth stage is adjourning (breaking up) and the team members will go through some mourning. The team will be naturally self organizing (teams take ownership of work).

Someone asks if you lose one member does it take another year to incorporate a new team member. He says one of two things happen: assimilation or rejection. One person states that sitting together can speed up the time to get to the performing phase.

There is a focus on continuous improvement (much like Lean's perfection principle). They have pragmatic idealism: you don't stop doing something just because it is uncomfortable or hard. (e.g. TDD - it can be hard in certain situations, high performing teams don't give in to the desire to short cut the hard parts - reminds me of a recent Tim Ottinger post). The ideals or principles are pragmatic because they directly relate to a goal (e.g. TDD is there for the goal of high quality or in Lean terms low Muda.) Regular retrospectives are the one thing he would recommend to any agile team that isn't - not just going through the motions but selecting an area to improve each time. He conducts them for an hour.

Marick's missing motivators: Marick states that the agile manifesto (that he signed) is missing four values:

• skill - good at your job


• discipline - don't cut corners


• ease at work - don't make it hard to do the job (remove obstacles)


• joy - enjoy your work

He states Scrum is good because it focuses on ease, but not good enough to get work done (I agree of course :-). Talks about distributed teams. He recommends Pubs & Planes :-) He references some research by Monica Yap (might be this: http://ieeexplore.ieee.org/Xplore/login.jsp?url=/iel5/10705/33795/01609825.pdf?arnumber=1609825) that shows how hard distributed teams are.

3. Lash of Jeffries

Running tested features on a schedule every iteration. Ship like clockwork. Requires that done means done (or as he says "Done-Done"). He means that even though all the work is done it isn't done until it is a running tested feature. IMO: There are often gaps in a feature if it is split among people. He says the velocity is rock solid and the features are nearly defect free. They prevent defects from entering into the product. Flickr deploys software every day!

Asks the audience if anyone has gotten to this point and the room was quiet for a while. One guy says they are just now able to deliver every iteration on a Monday (with Tuesday as a back-up in case of any defects found with the push on Monday). If the user isn't ready for a feature have the ability to turn off the GUI (but deliver the feature anyway) You don't want long lived branches in your code. Sometimes the team will overwhelm the business team. Isn't that a cool place to be! He states that the business will adapt.

He tells a story from his book about a team working on an embedded application. They had 60k LOC, 51 defects (over 3 years of development) never had more than 2 open defects at a time. Delivered software after 6 months and supported engineering as well as new features. They delivered 21 defects. Capers Jones statistics said a best in class team working on this size project would have had 461 defects, found 95% and delivered 23. The team above found only 59%. They prevented defects!

4. Agile Engineering

Great teams use agile engineering. He busts on Scrum again because it lacks AE. AE Practices are TDD, continuous design, rigor & mindfulness. Software engineering practices taught in school come from large and long DoD projects doing some version of waterfall or spiral. These practices can't support weekly or even bi-weekly deliveries. Great teams avoid technical stories (taking time off to refactor). AE doesn't require an A-team. He works with ordinary people in great teams.

He does think every team needs someone who is passionate about quality and pays attention. This person doesn't need to be senior but needs courage to speak up when things are going wrong.

He didn't plug his book, but I was impressed so I will. He publishes added stuff to his book every Wednesday here: http://jamesshore.com/Agile-Book/

My take on YAGNI

Recently there has been a long going post on the Extreme Programming newsgroup (http://tech.groups.yahoo.com/group/extremeprogramming/?yguid=198635929) about YAGNI. For those of you who do not know what YAGNI means please read this: http://c2.com/xp/YouArentGonnaNeedIt.html. Basically it started when some guy said that he thought XP got some things wrong and one of them was YAGNI. This of course caused lots of people to inquire/question him and out of it came several good posts. I sent some of these posts to an old colleague and he thought some people were over zealous about YAGNI. I do not doubt that people can become over zealous, no matter what you are talking about, but in this case I think the opposite is the larger problem.

Let me start with a story. At work a project came up that was for the CEO, and thus had high priority,vague requirements and an unmovable deadline! I pitched tackling this problem in an iterative fashion (Agile is becoming such a buzzword that I rarely use it in public anymore). The IT lead agreed and off we were. The team was me and a colleague (new to agile techniques and principles but liked all he had read). We couldn't dedicate much time to it as we had 2-3 other projects that had to take larger chunks of our time during any given week. We didn't have a prioritized list yet (customer was working on it with the CEO), but we did have the notion that the important thing was the landing page (the CEO doesn't like to click unless compelled). In our first working session we set up the project and hooked it up to a test page and viola we had a landing page...what next. We knew the whole point of the project was to display data and this data was coming from multiple sources (some internal, some external and some typed in) so we thought database. Then we asked, do we need it? One day yes, but not now. So we created a class and hard coded all the starting values in the code behind. This was obviously throw away code as the data must come from a database, but we got a landing page with real data in from of the team lead for direction in 2 hours spread over 2 days. He loved it and seems willing to stand up to the political forces that want us to design it all up front!

The moral I choose to draw from this story is that not doing something now even though one day, maybe even tomorrow, you will do it is a powerful tool. One reason is that software developers love to get ahead of themselves. My colleague came into my cube yesterday with a technical solution for a problem that we don't have yet. It was good that he was excited, but his (and mine) tendency is to solve the problem we sees coming because the problem we imagine is somehow more romantic than the problem right in front of us.

Our manager, not in line with the IT lead, did not feel comfortable with our approach, even though we are the two best developers in the company, because he wanted us to have solved all problems he could envision. Forget the fact that the customer wasn't even sure what he wanted...lets go solve "the problem he should be asking for" and ignore what he needs right now. It is hard to fight these forces, but knowing the technique works gives us stength to resist and carry on (I hope :-)

I am not saying that you should be blind of the future. You should always use your best judgement with the best information possible. I am just saying, and I think YAGNI is saying, be careful of your tendency to solve tomorrow's problem when today's problem can be just as challenging. In fact our last working session my colleague (the agile neophyte) re-directed our task list because I was focusing too much on back-end data issues on not enough on giving the customer value. I loved it! Our next set of tasks is still database free. It's coming...but we don't need it yet.

Why I Recommend One-Week Iterations

Over the years I have worked with, talked to and/or read about many agile teams. Almost none of the teams did agile software development the same way which has reinforced my belief that there is no "one way" of doing agile. One aspect of agile development I have always been drawn to is iteration length. Most teams seem to be falling in the 2-3 week range. In fact, recently a friend of mine said they were doing scrum with 3 week iterations becuase "any thing shorter caused too much overhead." I tried to point out that that isn't always the case as it depends of what overhead you are doing. I am a firm believer in minimuzing overhead as much as possible. I have always found ways to decrease or shift overhead to take less time in an iteration than previously believed possible. My favorite technique is process impedance matching.

I believe that lean thinking should be applied to software development. Given the principles of flow and value along with agile development practices it is obvious to me that the shorter the iteration the better. One of the competing forces is as the team size increases so does the overhead. To combat this I recommend capping team size to 9 (I am a slave to the magic number :-) Another consideration is the team's schedule. All the agile teams I know of work Monday to Friday and have a daily standup in the AM (9AM seems to be a popular time). People also like to forget work on weekends...some would argue it is the point of the weekend. Given these factors releasing on a Friday make the most sense to me. Monday's typically are hard... catching up on emails from people you didn't get to on Friday, remembering what it was you were doing when you left Friday...so planning meetings are best suited for Monday's. As you plan you are refreshing your memory as to the features you plan on bulding. All of this seems to point to a week long schedule of Monday mornings are planning and Friday afternoons are releasing. It's a nice package for an iteration in my opinion. I am open to shorter, but longer seems wastefull.

Many people will immediately think some version of the following: "That can't be a good idea becuase I'm not doing. If it was a good idea I would have thought of it when we decided to do >1 week iterations" or "I am sure it is >1 week here for a good reason...I just don't know exactly what that reason is." Given that I would like to convince you that my belief should be your belief (one of my many weaknesses) let me try and lay out common arguments against the one week iteration.

A1) One week iterations will have too much overhead.
I have found that the most people say this because they they they have to cram all the overhead they currently do in 3-4 weeks into one thus it is obvious to them this makes no sense. This assumption does not hold because of the lean principle of waste reduction and thus the goal is to minimize overhead. I challenge everyone to examine what overhead is actually needed in EVERY week? We already have blocked of 2 half days (Monday morning planning and Friday afternoon release & retrospective) as well as the 15 min stand up every day. Everything else should be avoided or at least minimized. If your organization requires a weekly status report and meeting. Use process impedance matching (e.g. let the team lead do it for the whole team; rotate through the team). Most all obstacles can be overcome if you use a brainstorm session with you team. If you have trouble solving a problem use the many internet sources (websites, groups, blogs...) If you seem stuck it is probably due to a lack of true teamwork (The Core has some iteresting ideas).

A2) Weekly releases are wasteful
Usually this comes from a management type who is thinking that all releases must have the business customer present (or some overhead thing like A3). If your business customer isn't engaged enough to commit to 2 half days a week then improvise and have a team member play a surrogate customer in those meetings. Different people have different skills, but hopefully one member of your team can think from a customer's point of view long enough for you to move forward. Once the customer makes time then re-validate. I am not saying it isn't ideal to have a fully committed business customer, but don't throw the baby out with the bath water.

A3) We can't do X in one week iterations
X is usually some localized practice that has evolved from a problem that wasn't solve, but is now. Like passing all releases to some QA team. If your organization is one with a separate QA team that must "sign off" on all external releases then simply use process impedance matching and send them your final release (the one that has all features needed in the time allowed) and then they test and they few if any defects (if you used proper agile development techniques) . You fix them as they find them and send them one more release when they are done (if they can manage to test in one week...doubtful if your organization hasn't embraced continuous testing..then send them one per week without needing to change your process much). Pretty simple if your team thinks about how to fit what needs to be done for organization reasons with what has to be done for agile development reasons.

I am sure there are more arguments out there that don't neatly fit into one of the above. Please comment with them. I think Tom Gilb once challenged an audience that over lunch he could figure out a way to release early then they currently do (Gilb's guidance was 2-5% of budget/schedule for each release). He got at least one free lunch out of that challenge. I am willing to take the few who read this on :-)

Long Term Planning for Agile Projects

The other day my boss asked me about what do I think about how to do long term planning for an agile project. Of course I hve an opinion which I gladly share :-) I think agile software development makes planning much easier that traditional projects. Though the techniques are similar. The details here are for an IT environment although they can easily be varied for other software organizations. So here is my approach (beware it is looooong):

First start with a list of Features as defined by your customer that constitutes the project's scope. Next use the wide-band delphi estimation technique. The estimation team must consist of the customer(s) who requested the Features and at least 2 members of the agile team tasked with delivery (tech lead and team lead at a minimum) and at least one expert not on that team. The non-customer representation should be 5-7 in size not counting the moderator. I like to document the following for each Feature: description,assumptions, average estimate and standard deviation. Then depending on the risk deemed appropriate for the project determine the final number based on this:

A - Mean = 50% chance of actual work exceeding the estimate
B - Mean + 1 std dev = 16% chance of actual work exceeding the estimate
C - Mean + 2 std dev = 2% chance of actual work exceeding the estimate
D - Mean + 3 std dev = ~0% chance of actual work exceeding the estimate

I would never use A because I think this is gambling. I would never use D because I don't believe any estimate has that low a chance of being wrong. Of course all of this is based on statistics and we have all heard some variation on this quote:

There are three kinds of lies: lies, damn lies, and statistics. --Benjamin Disraeli

We are just putting confidence levels on our estimate. What might these numbers look like in reality... An estimate team of 7 estimates Feature X to take 4,5,5,6,6,7,9. Using B you get 8 and C you get 9. Some of you might be tempted to use decimal places...please do not. I am sure there is quote out there, but here is my general rule: let the level of precision of a number be based on its accuracy. Accuracy of an estimate is always low so please don't use decimals (precision) If you don't understand, please read this explanation of the difference between precision and accuracy.

So do you use 8 or 9 for Feature X? As I said above "depending on the risk deemed appropriate for the project ." This means that a high risk project should use the larger number. The riskier the project the safer the estimates need to be. A risky project usually has a high number of unknowns and a high degree of pressure. Unknowns makes estimation hard and pressure tends to make estimates low (and the project hard because of them). If you succumb to the pressure at the beginning of the project you are doomed.

Another pitfall can occur when the Features are too big. You should be keeps the estimates in single digits of person weeks. I doubt any estimate of week past this. What do you do when a Feature is too big? Help the customer break it into multiple Features. Repeat until all features come in as single digits. This tool is a mainstay of agile planning and will be needed for each sprint planning session so it is critical you understand how to do this. If you need help please read Mike Cohn's book "Agile Estimating and Planning" and use the abundance of experience in Yahoo Groups. I find these two to be the best in this area:
http://tech.groups.yahoo.com/group/extremeprogramming/
http://groups.yahoo.com/group/scrumdevelopment/

Next you get the customer's to rank them in priority. If there are multiple customers then you need a moderator who knows how to get multiple parties to agree and a simple technique or two to get them to put a number on each and every Feature. You will get at least one customer who says "I don't care, just get them all done" please be patience and ask them to participate in the ranking. Do not try and convince them of the importance, if they say something like this is an open forum they are pretty much not going (or probably even try) to understand in an open forum. So now you have a ranked list of Features with estimates.

At this point some of you may be wondering what unit of measure to use. I would go with person-weeks. Features should be big enough to take a week. Weeks are easy for people to grasp. Reasons not to use other units:

• Points can mean different things to different people and will usually confuse/frustrate your customer
• Hours are too fine grained and too precise for accurate estimates of large chunks of work
• Months are too long for most projects. I do not trust any estimate that is months away. If your project is huge then use two levels of estimates. Start with groups of Features...I like to call them a Capability. One by one split these Capabilities into Features and use the technique here. Then roll the features up by Capability (Excel excels at this ;-) to get a complete picture.

So what do you do with this list. This list feeds into the agile team's work. Each sprint they take enough Features off the top of the list to ensure complete sprint coverage (meaning the total person weeks exceeds the person weeks of your sprint). So if you have a team of 7 and you use a 1 week sprint length (my recommendation) then you need at least 7 person weeks of work to start your agile planning session. Out Feature X had 9 so that one Feature would be enough. The amount over you current capacity is determined by your team's performance. If the team has no history then estimate double the capacity. You might plan more than you do, but it is always OK to have extra stories on the backlog than to finish a Feature and have no work planned out.

Now you need to add a timeline to your list of Features. We will not put dates on each Feature. Instead we will draw a line with the Y axis as Feature person-weeks and the X axis as sprints. The slope is the bur rate we expect from our team. With a team of 7 and a sprint length of one week do not expect 7 person-weeks/sprint. This presumes 100% effeciency. Remember there are planning eets, demos, doctor appointments and other non-project work. If you have historical data then use that to inform you. If you do not the 50% on task is probably a good starting point. Our fictional project has 10 features with sizes 2,5,4,4,4,7,2,5,5,2 (total 40 person weeks). That menas we should be done in the middle of sprint 12 (40/3.5 = 11.4). Remember we are using estimates for this prediction. Instead of drawing one line I would draw two - pper and lower limit, and reality should fall in betweem (give your customer the upper limit). List use a burn rate of 3 and 4 for these two lines (if you are really into numbers you can use +- some percentage..say 15?). That gives us a lower limit of 10 (40/4) and an upper limit of 14 (40/3). So tell your management 10-14 sprints and use 14 as the end date. Now the sprints can start.

There are several ways to do Agile planning within sprints. The key for me is how you break up the work. One way to break this work up can be seen here: http://www.mountaingoatsoftware.com/sprint_backlog. I recommend slicing the Feature into Feature Slices. Feature Slices are stories that slice the Feature end to end. Using slices means each story is a mini-feature so each one would be recognizable from the customer's point of view and would have distinguishable value. This allows the customer to prioritize the work in a meaningful way. The link above is developer focused and helps the team's natural inclination to silo and sub-optimize.

An example will probably help. Lets say the Feature we are working on is a shopping cart (forgive any ignorant assumptions...I have never coded a shopping cart) Lets assume we already have a way to display products. One way to slice this is:

• Select one item from the catalogue and add it to the cart
• Display an item previously selected
• Add multiple items to the cart
• Sort items in the cart by product name
• Allow the customer to select the sort order (name, price, date selected)
• Sort items in the cart by Price
• I think you get the idea

Each one of these has a different value to the customer. He can order them as he wants. He can also draw a line and say don't show me the shopping cart until it can do these things. This puts the power in the right hands. Us technical folks like to have power over the order, but we don't own the work...we just do it! These should still be stories...meaning they should take no more than 2 days to do (ideally take 8 ideal hours or else) and completion is defined by executable tests. If the team can't promise that for the above list then add more differences that will split the work. For example, think of the story w/o the database...it would not have persistence. Then add persistence as a separate story. Think of why the story takes more than 2 days and split up the chunks in a way that the customer cares. For the above example the customer cares about persistence...think about the usefulness of a stateless shopping cart...not appealing :-)

That's about all the techniques you need. The rest is presentation. I am lazy so lets do an example. Our team has 7 members as is following a one week sprint. Here is their Sprint Burndown chart (Bar height shows work left, The line represents expected burndown; used to determine estimated completion time):

Notice that the backlog is in the person-weeks from the original estimate not in actuals (much like earned-value). Many people use actuals and thus the backlog grows and shrinks each sprint and this fluctuation will seem to the uninformed to be an ever changing target and some sort of shell game. In case it isn't obvious you start with the total estimated person-weeks for the all features. Each sprint in which you complete a feature (pass all feature tests) you subtract the estimate off the backlog.

The factors that will shape this chart are the quality of the original estimates, the quality of the story estimates each sprint and the productivity during each sprint. Usually, but not always, the factors even out. The key is to look at the numbers each sprint and understand the variance. You can show you management all the data, but I recommend only showing the customer the burndown. You can use the other data for the IT management so they can understand the factors (and it gives them metrics that tend to make upper management happy :-)

Data you should keep track of:

• Person-weeks per Feature (estimates)
• Points per feature (estimated in the sprint planning)
• Sprint each feature was shown complete (actual)
• Points finished each sprint (actual)
• Total Points required to complete each feature (actual)

With this data you can show:

• How good your original estimates were and even show limits and track this as the sprints complete and re-plan if the estimates are consistently tracking out of limit.
• How good your sprint planning estimates are
• How many points done each sprint (this should start low, track up after 2-4 weeks then track down near the end as the project winds down...yes the S curve did not die) I would not do much with this chart unless it never rose, trended down over several sprints or if there was a drastic change

How to do the above I leave as an exercise for the reader. What might some examples scenarios look like? Lets look at three scenarios:

1. Team is new and runs into problems they could not foresee (Dark Matter), will miss the 14 sprint end date given to customer.
2. Customer keeps adding scope each and every sprint, and the work will push past 14 sprints.
3. Feature estimates were much too low and the team can not be done in 14 sprints.

Scenario 1
I will use the following for Feature/story points (6,17,14,11,14,18,11,18,18,8). You can track along in Excel if you dare ;-)

The team has trouble with the technology, each other and agile software development techniques. The stories completed for the first 4 sprints are 5,6,6,6. They complete feature 1 in sprint 2 and feature 2 in sprint 4. Given the performance so far the team would need to average 13 points/sprint to finish by sprint 14. At the current rate it looks like it will take 23 sprints to finish. Things do not look good.

The team should talk about whether or not they think they can double their outcome. It should be obvious to the team lead that she needs to sit down with the customer and her management to determine a course of action. Upper management should be able to tell from the numbers that this talk is needed. I would not recommend they have the talk after the first 3 sprints. The first three sprints are typically lower than the rest. They may decide to let the team try to self correct for the next 2-3 sprints.

The team completes 7 points the next two sprints and complete feature 3. This is the best they have done so far, but it is not good enough to get the scope done by sprint 14 (more likely they will take 20). Management has the options of hiring a consultant and adding a senior person or two. The customer will need to be told that there is a risk to delivery and she needs to start thinking about potentially lengthening the time line or cutting scope.

Scenario 2
I will use the following for Feature/story points (9,22,18,22,22,33,11,27,29,13). Notice these numbers are higher than scenario 1. The team is experience with agile and the technology. The fist three sprints the team completes 9,13,12 points and features #1 & #2. Their backlog is at 33 and the estimated finish is sprint 18 with a current average of 2.3 person-weeks/sprint. To finish by sprint 14 the team will have to up their average to 3. The team talks about this at their weekly retrospective and think they can do it.

The team works most nights in print 4 and does 18 points to complete feature #3 worth 4 person-weeks. Things are looking up, but the next sprint their stories are down to 11 because they are tired. Sprint 6 is back to 15, but now the picture is starting to be clearer. They are expected to finish in sprint 16 and to finish by 14 they need to up their output to 3.1. The team wises up in the retrospective for sprint 6 and starts asking about why the work isn’t lining up with the estimates. They do a ratio of points per person-week by feature. It is averaging around 4.7 and tracking up. They ask the outside expert that helped them come up with the feature estimates and he notices that several assumptions made in the planning session are being violated by the stories. The customer is asking for more that was agreed. The team lead needs to sit down with the customer and the her management to work out a solution. Something has to give.

Scenario 3
Will follow the same path as scenario 2, but when the retrospective digs into the reason for the mismatch with the estimate it is becomes obvious that the original estimates were much too optimistic. The estimates did not take into account many of the realities that the team now knows. The team lead needs to sit down with the stakeholders, but the discussion should be about two things. What do we do to satisfy the customer and what do we do about improving the estimation process for next time.

For those brave enough to stay with me I hope this was coherent.

SCM and Software Integrity

A friend of mine emailed me about using control of /main versus continuous integration. He argued that a team of productive developers would swamp the select few who own /main. This would cause integration errors to pile up and slow the team down. He correctly identified this as a bottleneck. Using Goldratt's Theory of Constraints, bottlenecks limit the productivity of the system. Thus bottlenecks are to be eliminated (either by shifting resources or changing the process) if one hopes to improve productivity.

I used these controls as a way of achieving source code integrity, not that I recommended it. Achieving source code integrity can be achieve various ways. One philosophy is to place controls on /main and establish a process that is rigorously followed. This will work, but will be slower than ideal because of the bottleneck of the control. There might be situations I would favor such controls, but the situations I can think of involve low levels of trust towards the developer (off-shore development for instance).

I favor teamwork. In a functional team, all members share a common goal. A good team has a culture. I agree with McCarthy (see Software for your Head) that the software takes on the properties of the team. To achieve software integrity the team that develops the software should have integrity. In a team of individuals with integrity I expect the team to own their success. So a team like this will have ownership of checking in well written, well tested code. If a team member is unsure of their code I would expect them to seek out a better developer to validate their code before checking it in. In such an environment mistakes will happen, but they will self correct. If you do not trust your team then high levels of productivity are not possible as controls will slow your team down.

Keys to Software Development Success

After some thought and reflection I have deduced two key aspects to successful software development:

1) Focus on teamwork (lack of too many teamcide practices)
2) Use a process oriented around the integrity of the source code

The first key seems straight forward, but somehow many organizations take it for granted. The word teamcide might not be familiar to you (read Chapter 20 of Peopleware). DeMarco & Lister were attempting to write a chapter about how to make teams jell. They gave up and instead wrote a chapter on ways to prevent teamwork. It is very insightful. I have found that even when an organization goes out of its way to kill teamwork a strong leader can overcome it. First you must realize the problem before you can overcome it.

The second key aspect can be interpreted many ways. I will try to further explain myself. One of my first jobs was as a software developer for Motorola. Some VP had decided we needed to be CMM level 3 so every developer was forced to work 4 hours a week on process improvement. I was young and got involved as a capability area lead. After reaching level 2 I saw improvements and when I started a new team I carried over many of the practices that I had learned. When I look back on what worked two things come to mind:

1) Unit Testing
2) Strict control of /main in SCM

Both revolve around the concept of keeping the code in the best shape as possible. Unit testing prevented many bugs from ever being merged into main. In fact only two people could check out from main and not without running the entire suite of automated unit tests.

Jumping to present day; I work with a vendor that is struggling to produce quality code in a timely manner. When I look at their process I see no real SCM - all code is checked into main by anyone and there are no automated unit tests. All tests are GUI driven and run manually. The sad thing is they have a fairly strict process. They are not 100% compliant to it, but I bet with just a few tweaks they could achieve SEI CMM level 3 and still struggle to produce quality code in a timely manner.

Many people are caught up in the Agile craze these days. Agile is best described by the manifesto (http://agilemanifesto.org/):

- Individuals and interactions over processes and tools
- Working software over comprehensive documentation
- Customer collaboration over contract negotiation
- Responding to change over following a plan

I truly believe that you could follow this manifesto and still struggle, but I find it hard to believe that you could follow my keys and struggle. I might be naive, but almost all problems I have run into in my 16 years of professional software development experience come down to poor communication or teamwork (they are extremely related) and a lack of understanding of the importance of the source code. Many processes place the same emphasis on documents as the source code. In fact many people believe the same processes that work for documents work for source code. The only tools available to written documents that I can think of are: human reviews and version control. Source code has the unique ability to be compiled and tested all without the ever erring human mind. I am not saying human review is useless, it is just insufficient.

One thing I have said little about is using iterative development versus waterfall. Part of me thinks it isn't worth saying anything as the waterfall process has been completely debunked, but some people still cling to it. I will simply say that the you can never achieve high levels of source code integrity if you start with a big ball of untested code. The waterfall process will start with such poor source code integrity that you will never be able to reach a high enough level to acvhieve success. You must start small to achieve high integrity and grow a little at a time. The human brain isn't big enough to make large leaps in source code without loosing integrity.

Programming a Sudoku Solver

I programmed a Sudoku solver the other weekend. I was going to blog as I went, but there were too many interrupts and as the coding gets more interesting I forget to blog. I end up with a blog of the boring parts :-) The key thing is what did I learn about my little exercise?

The first thing I learned was how important discipline is. I started by slicing my problem into small slices with tests driving the code until I got to my first tough part...solving for a cell. I thought "Hey I can code this in no time and get this test to pass". An hour later I was running the debugger and asking myself "How did I get here?" when my wife called to help puts the kids in bed. In disgust, I killed the IDE and didn't return till the following morning. My own hubris caused me to abandon discipline! Once I got back into the habit of slicing the problem and driving the development with tests I was able to get the solver working and in no time I was able to solve all but the "wicked" level of Sudoku's.

The second lesson was the power of TDD. I had not programmed in over a year and the process drove me through to a solution in roughly 8 hours. I honestly believe that when we look back on this time for the history books, TDD will be the first true productivity catalyst since second order languages. If you did it right OO gets you 5-10%, Pair 10-15%, but I think TDD will get you 50-100%.

The third lesson happened when I got into a discussion with a co-worker. He is a fairly smart guy with out any experience in TDD. When I brought the subject up his face literally changes to one of disdain. He thought the idea of writing tests for each line of code was absolutely stupid. He assured me that code coverage tools would lead me to a better use of my testing time; good developers obviously shouldn't be spending too much time writing tests! I then remembered the lesson I learned from my last job...change is evil...even for smart people! People have strong beliefs that have been re-enforced as they live life and develop patterns to deal with this world. Therefore most experienced developers have a long list of things that must be done and things that should be avoided. For this one developer, and probably many like him, one of the things to avoid was writing too many tests. Maybe a programmers ego is threatened by tests. Maybe the thinking is "If I am good then I don't need some stupid test to double check my work and tell me I am doing a good job!" I have to admit, when I was a full-time developer I wrote the least amount of unit test code among my team. I wonder what I would have thought about TDD then? Hmm, whatever the thinking I find it interesting that I think TDD is the best thing to happen since iterative delivery and other equally intelligent people think it is a waste of time.

The fourth lesson revolves around feature splitting. The Sudoku solver feature set was pretty simple. It only allowed for a few methods (load, solve, check solution and print), but 90% of the code revolved around one feature (solve). It required thinking to break it down into its parts - something I call design. Lacking discipline, I chose the simpler method of just trying to code it and then refactor it into a decent design. I realize now that stopping to design and TDD were working in tandem to point me to doing the right thing! Of course you only learn these things by doing the wrong thing :-)

My last lesson has little to do with my coding exercise and more to do with several different posts and blogs I have read about TDD. Many people think they know how to write software. I have learned that we really don't "know", but we are uncovering better ways of doing it. Once we figure it out then we can automate the whole process and put us all out of work. So many people cling to simple minded beliefs that we have actually figured it out and all we need to do is to follow the age old wisdom of define it, design it, write it and test it! SEI seems to collect people with this mindset. In my opinion we must constantly question our preconceived notions about software development because just when you think it's truly a given, technology or some one smart shows you just how wrong you are.

Good Software Engineering Reading Material

I was going to send this as an email, but decided I wanted it to be more persentent...

Brad appleton on simplicity (again): http://groups.yahoo.com/group/scrumdevelopment/message/14045

Jim Grenning on embedded TDD:
http://www.objectmentor.com/resources/listArticles?key=topic&topic=Embedded%20Software


Kent Beck on Appreciative Inquiry (as applicable in stand-ups, but I like the idea in general):
http://groups.yahoo.com/group/extremeprogramming/message/120359

Ron Jefferies being himself (entertaining and educational at the same time):
http://groups.yahoo.com/group/scrumdevelopment/message/14033
http://groups.yahoo.com/group/scrumdevelopment/message/14036
http://groups.yahoo.com/group/extremeprogramming/message/120397
http://groups.yahoo.com/group/extremeprogramming/message/120398

A TDD example for non programmers (I didn't take the time to read it, but some people I respect liked it):
http://www.clarkeching.com/2006/04/test_driven_dev.html

And I leave you with this quote:
The intelligent man finds almost everything ridiculous, the sensible man hardly anything.
-- Johann Wolfgang von Goethe

Why Time Management is Important

Like most people my age, I have dabbled in many forms of self improvement. Once I tried Franklin Quest (now Franklin Covey) and eventually gave up on it. Once Outlook became the first thing I did everyday, I would say to myself, "How can they still be in business? Outlook does everything I need." Well, I think I now know why they still do business.

There are roughly three types of workers on software projects:
A - those that take on too much
B - those that take on the right amount
C - those that take on as little as possible

Type A workers alternate between type A and C if they are not well managed. Type B workers are great, but can't step up in a moment of crisis (gotta love those type As then!). Type C workers (those that aren't just recovering type As) are usually unmotivated for reasons beyond your control...not that you should give up on them. It should be obvious that your most valuable resource are your type As (If you don't know this then you shouldn't be in management :-)

Let's see if I can get to my point with a personal story. One of the unseen advantages of my current job is I get to do more work than I can possibly do. This has led me to go through the typical type A overwork cycle:

1. Work all the time until one of two things happen: A. Someone who cares and that you trust(my wife usually plays this role) tells you to stop or B. You check out
2. Work as little as possible to recover (or quit...never done this one but I know people that have)
3. Go to 1.

As you can see it is never ending. So in a short moment of clarity I stumbled on (remembered) a way to stop:

1. Write down all known tasks (if you are just starting 2 as I am it might take a while) - meetings are tasks!
Each Day:
1. Set a hard stop to your day (5pm is mine for now)
2. Block off the last 30 minutes (so now we are at 4:30; 30 minutes might not be enough for some of you)
3. Start each day by prioritizing all known tasks
4. Work on the most important tasks until the day is done (4:30 for me)
5. As new tasks are added; put them on the list and do a quick and dirty prioritization (today...not today) If it is today then add it at the top and see if all Top priority items can be done. If not then the last one on the list just got bumped!
7. During the last 30 minutes make sure you have have all known tasks (usually this means reading email)

In other words - Time Management. Other than being cathartic for me what was my point? Well if you remember my last post, I am trying to find the principles of software engineering. I have been pondering this for a while and haven't had anything worth saying until now (you judge its worth).

What is the most important thing in a software project? This answer is easy - the people on the project. I know this sounds like a cliche. When I taught agile software development people would always scoff and say something to the effect of "Well I don't know where you come from, but were I work I get the people I get!" A team of talented software engineers will succeed no matter how many stupid decisions are made by the uninformed, but a misaligned team of unmotivated and mismanaged software engineers will fail not matter how good decisions are made. It all comes down to the people. I have never been on a team, seen a team or even heard of a team that couldn't succeed. I have been, seen and heard of plenty of failures. I think you have two choices: be a victim or do something about it.

Well the people on most software projects are a mix of the three types above. If you don't manage them accordingly then you will wind up failing no matter what else you do. Given that the type As are the most valuable resource and managing them should be priority one. One thing type As do poorly is....Time Management!

Thus my circle of logic is complete...kind of.

What is Software Engineering?

Earlier in my career, I would say my career was software engineering. Then one day an interviewer asked "What do you mean by that?" This simple question was the beginning of my search to really understand what software engineering meant. My first answer was
awkward and used thoughts that can be found in the CMM. After thinking about it for a few years I began to believe that Software Engineering was an oxymoron. I used the term software development. Wasn't I clever ;-) Then I discovered Agile Software Development and I was hooked. I wrote about it, taught it, practiced it, loved it. This lasted for a few years, but the more I saw the more I realized there are lots of ways to be "Agile". Some were sound and others weren't, but they all could say (without lying) that they were Agile! I knew what worked for me, but did that always work? Off the shelf agile methods (like XP) work great in some places, but not so great in others. Many projects had constraints that most agile practitioners hadn't even considered (let alone little ole me).

My approach to new problems is to go back to first principles. In this case what were those principles. The agile alliance has these, but they seem more like good ideas from experience rather than principles. This got me thinking. Maybe there is something in that old term Software Engineering. Maybe it would help to dissect the term:

Software: Coded instructions (programs) that make a computer do useful work. (from www.krollontrack.com/legalresources/glossary.asp)

Engineering: Engineering is the application of science to the needs of humanity. This is accomplished through the application of knowledge, mathematics, and practical experience to the design of useful objects or processes. (From wikipedia)

Let's see if I can munge the above definitions to create something I like...

Software Engineering: The application of knowledge, techniques, and practical experience to make a computer do work deemed useful.

This still doesn't get me to the principles. Civil Engineers have a wealth of physical principles (Newtonian physics and all that). The best Software Engineering can counter with is Computer Science. It hasn't really progressed since second order programming languages and the best I can figure it's principles boil down to logic and some rule about a Turing machine. So I have just alienated my CS doctorate crowd. One less reader won't hurt me ;-o

I have thought on this and in the past the best I could come up with was to steal principles from other disciplines. I would rationalize it like the definition for software engineering found in the wikipedia. This defintion makes Software Engineering nothing more than a mutt grown from other more pure fields (science, management, physiology and my favorite the generic "engineering"). Well that just doesn't cut it for me. There are some rules behind this activity called Software Engineering and if you break them it causes you pain. Let see if I can find them....

Gimme a Date

I have been in the role of project manager for about 5 months now. One phrase I find myself repeating is "So, do you have a date when that will be done..." I used to hate to have to provide dates for things I barely understood. I felt that anyone who asked was obviously clueless. Now I find myself on the other side of this question...what has changed? One thing I have found is that if no date is asked for then the chance of it getting done is usually extremely low...close to ZERO. It seems most people aren't very organized and their job stretches them too thin. They get done what they think is most important. When someone reminds them of a date that is coming up or has just passed and and a task they promised to have done they typically up the priority of the given task. Why is this mindlessness needed?

For one thing we don't trust one another. My wife asks me to feed the kids. 15 minutes later she asks "Have you fed the kids?" This is with someone who trusts you. Take this to the office and know one trusts anyone. Everyone knows all of us are playing games of trying to look good to those that matter so we can get that promotion, that raise, that praise...you name it. The game isn't about how do WE win, but how do I win. This type of culture makes getting things done very difficult. You wind up having one meeting to get them to agree to a task and a date. Another meeting to get status (sometimes this even occurs weekly) then a final meeting to go over what they did. It isn't rare that in a given week I spend more time in meetings than getting any work done. The project I am currently on, we have more people in an oversight role that in a doer role. I wonder if we could calculate a performer to oversight ratio for projects. Then map that to success....hmm...it might be interesting.

One thing Agile projects try to do is limit no-win situations of date promising. The date is either the end of this sprint or it isn't. The focus then becomes what can the team deliver this sprint. We don't have a number of meetings with various stakeholders, each with their priorities and action items and dates. We have ONE backlog and ONE team. The team works the backlog and NOTHING ELSE. A stakeholder wants something...it goes on the backlog and gets prioritized along with ALL the other requests. None of this back room dealing junk where Bob gets his stuff this week, because we didn't promise Tim his stuff till next week and poor Jim didn't ask for a date so he can just wait. Besides Bob is always calling me and I want him off my back.

With Agile, we put all the cards on the table with all the stakeholders. Let the customer decide what the relevant priority is between Bob, Tim and Jim. Let the developers do the work and stop playing politics. Agile practices allow progress to be made because the rules are simple and well communicated. Anyone who wants work is told the same story. It isn't different because of who the asker is or the relationship to the answerer. It is always: Come to the planning meeting and we'll see where the work lands. The development team is only responsible for delivering what the owners have prioritized. When the simple rules are followed it works well. When the rules are tinkered with to soft peddle the approach. It works less and less well. In some cases it works worse that before. All I want to know is when will you be done?

Typical Oversimplification

A friend pointed out that my last post seems to paint the picture of PM as simply effectively communicating project status. This isn't the picture I was going after. I know that there isn't just one thing a PM needs to do in order to be successful. A few things I can think of off the top of my head:

- Establish a network of people in the company, for support and information
- Understand the goals and objectives of the customer(s)
- Establish trust with key stakeholders (customers,bosses, project workers,...)
- Do the mandatory tasks (Sit in meetings, schedules, plans, statuses, meeting minutes...)
- Mediate disputes and drive people to make a decision
- Manage risk and issues
- Monitor budget and other key metrics that the customer(s) care about
- Sheild the workers from unnecessary busy work and political gyrations
- Understand the true state of the project and communicate it effecticely to all key stakeholders

In my opinion, this last one is the most important. Of course you have to do all the other things, but if you fail in the last one your job will be frustrating and probably unsuccessful. If I was reading this my first thought would be so what do you mean by the "true state of the project" and "communicate it effecticely to all key stakeholders". These seem awfully vague. Let me try to elaborate.

Lets assume for a bit that we had a crystal ball that knew all and would answer any question about our project we wanted to know. Stop salivating...assuming we were intelligent to ask the right questions we could determine the a good understanding of the current state of the project. In fact if we had enough time I bet we could get a complete picture of all work on the project. This is what I mean by true state of the project. Of course we don't have any crystal balls...try telling that to your customer. So what do we do. Well I think the answer is different for everyone. I like to make a mental map around which I put various bits of information from various sources. As time goes on and I find out the reliability of various sources I adjust my mental map to get a better understanding of what is going on. I never rely on just one source. There are tools I have seen that let you map information with links...I forget their name...but then I doubt I would ever use it. Others seem to have success using them though.

Now on to communicating it effecticely to all key stakeholders. Communication is an interesting thing. Different mediums work better in different situations. When you want to tell folks you are buying the first round I find email works just fine. When you want to tell some one they aren't meeting expectations I would never use email. In between it all depends. Email is great for simple questions and simple answers. It is great for reporting status to a group of people and making sure everyone that needs the basics gets them on a regualr basis. It isn't the best communciation tool for ALL communication though. Alistair Cockburn has written about the modes of communication (http://www.agilemodeling.com/essays/communication.htm). It is important to use communication effectively and efficiently. Relying on just one mode (i.e. email or meetings) is bound to result in less information than is necessary to do the job. Then again I don't know everything...so I might just be wrong.

What is a project manager's true responsibility?

The PMBOK states this: "Project Management is the application of knowledge, skills, tools and techniques to project activities in order to meet or exceed stockholder needs and expectations from a project."So I guess coding is project management (PM) as well as design, analysis, architecting and a host of other non-management jobs. What isn't PM by this definition?

As you can tell I don't like it, but it is too easy to criticize and often difficult to be constructive. I like a good challenge. In my 14 years in the "software industry" (meaning jobs that involved writing software...I am not sure what other people mean by it) I have seen a wide variety of management. One type of PM, lets call him A, bossed around all the people on the project and acted in a way that everyone hated/feared him and the job got done...not well, but got done. Another type of PM, lets call him Z, provides status, updates the schedule, hold meetings and keeps the minutes, fills out forms, and other menial tasks. Of course these are two extremes and most PMs fall in between A and Z.

What makes PMing interesting to me is that the culture under which the PM operates has direct influence and control on how best to do the job. Take a type A PM and place him in an environment in which he has no political power and he will be frustrated and probably will quit. Take a type Z and place them in a position with ultimate authority and control and the same will probably happen to them. So the question is, does the environment/culture of an organization select in a Darwinian way the PMs that last? Now take into account the churn in today's job market, in which people leave jobs all the time and never stick around for long. One environment won't have any one group of PMs long enough to do natural selection...so what determines a PM success?

I don't have any answer to that question...not yet at least ;-) I do have an opinion on how a PM might be successful in all but the most oppressive environments. To me the key is information. A PM that knows little about what is really going on in their project is going to struggle no matter what tools/techniques they learn in PM camp. A PM that knows the most about the true state of their project, knows the tools (i.e. EVM, Schedule, TOC) and can use the tools in ways that show the state of the project with proper understanding their audience will be successful.

Some of you may be wondering what I mean by "show the state of the project with proper understanding their audience." I like using examples to illustrate my points. Lets take a situation in which the PM finds out that a mandated date will not be met and scope must be moved. Wrong ways to deal with this are: to hide it for fear of looking bad, to make the project immediately red and shoot an email up the chain. A proper way is to have a all forms of communication in synch that show the dates may be in jeopardy (like raising a risk, tightening the schedule so the date has only a few days of slack, bringing up the issue in meeting with people of authority...) Hopefully you get the point. What makes this difficult is finding out ASAP that dates will be slipping and putting that information in a sensible way in all project documentation. Enough for now....

Project management

I just quit my job at the Consortium and I start a new job Monday as a project manager. Coincidentally I had just bought "The Art of Project Management" by Scott Berkun based on a review by the SD newsletter, SD People and Projects. I'm almost half way and I disappointed so far. Mr. Berkun seems to have some good ideas, but I don't think he is truly aware of what I would call current best ideas in project management (namely TOC and Agile). He has mentioned Agile a few times, but he seems not to truly get it. Of course I may be judging him early, I still have 270 pages to go.

I think software has unique qualities that other mediums (i.e. concrete & steel) do not. The major difference is flexibility to change it's behavior in short order. Therefore software project management needs to take that uniqueness into account. Secondly, lean thinking has shown us that rapid time to market with best in class quality is achievable by rethinking the nature of production. I think lean thinking and agile can make this happen for software, we are just waiting for the Toyota of the software industry to show us how.

Software quality

Software quality is a tricky thing. There are many schools of thought. The formal methods camp says the only way to get quality software is to formally spec it and test against the spec using some formal testing method such as model-based testing. The agile camp says that the developers have to build quality in on a day by day basis based on feedback from the customer. There are problems with both approaches.

The formal methods approach assumes you can specify things formally for all systems. There are some systems where formal specs are impossible and there are a large number of systems in which formal specs are too expensive (most all IT applications come to mind).

The agile approach assumes the customer can tell when they get something they want. In some agile methods it is encouraged that the customer write the acceptance tests. This will work for some systems, but there are a large number of systems where this will not do (banking and pacemakers come to mind)

Is there a middle ground? I think there are ways to combine some of the formal methods approaches with the agile approaches for the subset that require it. Lets face it quality is in the mind of the buyer, not some objective thing (although there are people who think defects/LOC is meaningful. Just to dismiss this think of two fictional word processors. Both have the same LOC count and have 1 defect each. The first word processor's bug is that it crashes just before it saves. The second just after. Which has higher quality. Only an idiot or zealot, is there a difference, would say neither.

I say where it is warranted (two examples are when the cost of a bug is more than system development, the past has shown that the system needs better testing) you can make the acceptance tests in an agile approach use model-based testing. I haven't seen it work, so I am skeptical, but hopeful.

Integration sucks!

Well I got JavaChess working easily. There is just one problem...not only is it bad it is buggy! At first I couldn't play a complete game because an array kept going out of bounds. I fixed that and now it is trying to make illegal moves when mate is inevitable! I am thinking seriously about rewriting it using TDD. I will try to fix this bug, but the code is pretty hard to follow, not that it written poorly, it just wasn't written by me and it has NO tests!!!

Will I ever make any progress?

I feel like John Kerry! I can't make up my mind (sorry for those Kerryites out there, but I just couldn't help myself) I have scratched the idea of incorporating Crafty. It would be ugly, messy code and what would be the point? Jose already does this. I found a great article on chess programming. The last series has a link to the author's code, and it is written in Java! I have downloaded it and put the javachess package in my project. It compiles, but doesn't run. His code isn't written to be integrated, but that's a challenge I like! We shall see how long it takes to get working code.

Have a happy Thanksgiving! Remember to be thankful!