Life, Liberty, and the Pursuit of Agility

In this episode I’ll cover an extremely satisfying application of Lean and Agile on a large ‘Legacy’ effort I was part of a while back.

Please understand that I am not at liberty to discuss certain (that is, most) details. I must protect the guilty. I will give just the bare details – only what is necessary. I bet you don’t believe I will be able to give just the bare details, right?

So – this is about using the No Estimate Approach on an End-Of-Life Legacy Support project.

First, a few definitions:

What is “Legacy” code?

I am glad you asked.  There are many definitions of legacy code. Here is the one I use: Legacy code is ANY CODE THAT IS DOING ACTUAL WORK THAT YOU WANT TO KEEP WORKING. (Please note the clever use of “all caps” for emphasis.)  Does that make sense to you? Doesn’t matter – just pretend that is what it means.  Using my definition, most projects become legacy code as soon as you have some functionality in production. That’s when the bits hit the fan, usually, so to speak.  However, even if it isn’t yet in production, it could still be “legacy” if we have stuff working we don’t want to break.

What is Legacy Support?

Regardless of what you think legacy support is, for the purposes of this post I am using it to mean this: “Work done (bug fixes and minor enhancements – that sort of thing, or more serious stuff if needed) to extend and protect the life of code that is in use”.   I’m leaving out more extensive enhancements like “whole, new functionality”.  You can judge for yourself the difference between minor and “more extensive” enhancements.  Doesn’t really matter to me.

What is End-Of-Life?

For the purposes of this post, an End-Of-Life project is a project that is almost retired.  It’s still in use but going to be replaced sometime soon.   In software programming “Soon” means “perhaps sometime, but we have no clue when.”  The basic idea is that people are using it, and it is important enough to keep supporting, but we’ll stop doing big additions and improvements.  Some bugs will just be ignored, but any deemed to be important to fix will be fixed if possible.

Project History and Details

The project that we are discussing is is a large, line-of-business set of applications (‘The System’) that is critical to the users of the software for managing most aspects of their business.  The typical user of The System manages thousands to millions of their own customers – sales, billing, transactions of all sorts, bunches of “real-time” stuff, reporting, etc.  Typical line-of-business stuff.  I was told the code base was about 2 million lines of code [and that’s about enough, in my opinion – any more is just bragging.]

The System has been in production for 15 years or so and has moved through several different code bases.  The current code base is about 4 or 5 years old.  It uses several languages – 4 or 5 perhaps.  Typical stuff.

Enhancements and bug fixes are managed, worked on, and delivered using a waterfall approach, with a 3 month delivery schedule (that takes 6 months due to test and fix cycle, and other typical deadline issues).

Due to design, industry, and technical reasons it is deemed necessary to re-write The System in a new language with a new architecture.

The Path Going Forward

The current version of The System MUST continue to serve the user base, and even more – it has to be clear that we are still responsive to concerns and issues of the user base so they don’t start looking to competitors to solve problems while the “greenfield” version of The System is being developed.  It has to look and feel like “business as usual, only better”.

A “maintenance team” is put together to provide this service.

I’m not sure exactly how many people were working on The System before the End-Of-Life Maintenance, but it was something on the order of 20 to 25 people – maybe more, it was a while back and I didn’t keep track.

The new team was to be 6 people: 2 QA, 2 BA (Business analyst or “Product Owner”), 2 Dev.  I was one of the two Devs, and also the manager, or co-manager (more or less).  I “volunteered” for this assignment on one condition: I would be allowed to manage this effort my way – which means Lean/Agile/XP.

We’ll call this System EOL (for system end-of-life) .

The State of System EOL  on Day One

There are approximately 500 active bugs in the bug tracking system.  Some of these bugs have been pernicious. Some have never been addressed due to level, some have been introduced in the fixing of other bugs, some have been there a long time but just recently discovered – All the typical bug stuff.

500 Acitve Bugs???!

There are 500 active bugs! There are a number of reasons there are so many bugs.  Before we talk about that, I’d like to ask a question.

• Question: What is an appropriate number of known bugs for a project that is in production?
• Answer: Zero.

Got that? I’ll allow for any bugs that have easy work-arounds – you can leave those if you have to.  I’ll also allow for any bugs that were discovered since the last deployment.  But you better fix those NOW and then you’ll have zero bugs once again. Nice.

Remember that: Zero bugs.  Does that seem unreasonable to you? If it does, I’ll ask another question: Why bother reading my blog? Go do something else that seems reasonable to you.  Why waste your time on what I’ve got to say? I am NOT REASONABLE most of the time.  I want a better way to do stuff than whatever we used to think is “reasonable”.  Zero bugs is a “lofty goal” in some workplaces.  In others, it is business as usual. I prefer zero bugs.

So, 500 bugs is not just way too many, it is infinitely too many.  Or whatever 500 divided by zero is.  Anyone know?

One reason there are so many bugs: The old process.

There were several reasons all these bugs existed, but the one that is most pertinent to our discussion today is the process that was in place for assessing and fixing bugs. The way things were being done was about like this:

• Bug report is entered into the tracking system by a user, support person, developer
• The bug is reviewed at a “triage” meeting where its importance determined
• If it is important enough, it is assigned to an “investigator” who will assess it: that is, attempt to reproduce it, determine the likely cause and module that will probably contains the “bad code”,  and estimates the time to fix.
• After the assesment, the bug goes back to a “bug review” meeting to re-assess it’s importance and schedule the work to be done.
• If it is deemed important enought, it is scheduled to be fixed (either sooner or later based on the “cost” of fixing it and importance to the customers).  If it is not important enough, it is closed and it’s status and other details are communicated to the interested parties.
• It is assigned to a developer who works in the area of the “bad code”. Since the developer is a different person than the investigator, the developer starts from square one and “re-thinks” the stuff the investigator already thought about.
• In working on the bug, the developer discovers the problem is NOT in the area of code it was thought to be in, so he adds his comments and assigns it back to the bug review team.
• The bug review team looks at it again, and then re-assigns it to an investigator.
• And so on and so on…
• Once a bug is actually fixed, it goes thorugh QA where it is discovered that it didn’t fix what needed to be fixed.
• Go through whole process again.
• Once a bug is fixed and passes QA, it is deployed in the next deployment, and…
• The users discover that it didn’t fix what needed to be fixed, or the fix was only partial, or that something else is now broken.
• Even worse – some other things have been broken that are not yet discovered that will lead to other bug reports sometime soon.

Okay – you get the point.  I am not making this up.

There were other reasons System EOL was in this state, which included lack of unit tests, code neglect (huge methods, overwhelming complexity, fragility, coupling and cohesion problems), and so on… but we’ll solve one thing at a time.

Chutes and Ladders

Well.  Does that remind you of that game you used to play when you were a kid?  It should.  And it is JUST AS FRUSTRATING, at least conceptually.  Only difference is you get paid to play Bug Fixing Chutes and Ladders.  But overall – it is terrible and horrendous (and other bad words).

What to do???

Since I was allowed to manage System EOL in my own way – Here is what we did.

Value Stream Map

First day I brought in 6 copies of Mary and Tom Poppendiecks “Lean Software Development” book, and we all agreed to read it, and more or less follow their basic concept.  I’d been doing XP for years (more or less) but it’s a hard sell so I opted for the “easy sell”.  If Lean is good enough for Toyota, it’s good enough for me.

First thing we did was a value stream map on the  “Chutes and Ladders” methodolgy that was in use. If you haven’t seen a value stream map or know how to do one, take a look at Mary and Tom Poppendiecks “Lean Software Development” book.  The basic idea is that anything that is not directly adding value can (and should) be eliminated. For example: waiting in queues is a waste – it brings no value to the person interested in having the bug fixed.

I bet you can guess that we found a lot of NON-VALUE-ADDING behaviors.  So, we made a few rules.  For example…

First of all, a few rules:

• No Triage meetings or Bug Review meetings
• No Assesments for “where the code is broken”
• No prioritization or severity determination (a bug is a bug is a bug).  If it’s in the tracking system, fix it.
• No Estimation of effort. We gotta work on all the bugs.
• … and Other things

NOTE: No Estimates, No prioritization, No meetings.  These are wastes (at least in some situations, if not a lot) – bringing no value to the “customer”. Even worse,  waste usually removes value by using up time and “spirit”.  A team doing a lot of meetings, estimates, assessments, meetings, prioritization, meetings, waiting, etc. becomes NUMB. Avoid numbness.

An initial assessment:

We quickly read each bug in the tracking system, looking for duplicates, bugs no one had asked about in a year or more, and bugs where the description could not be understood (and there was no one who could clarify). This was done very quickly and eliminated about one half of the bugs.  NICE. 500 bugs cut down to 250 bugs.  No one can say that is not progress!

This took about a week (as I recall).  During this “first-week” the developers and QA took on bugs to fix, since there were plenty that were clear enough to work on – but spent some of their time helping the BA’s “weed out” the bugs that we removed.

Daily process we adopted:

• Daily morning 1/2 hour meeting.  Sitting. Quickly talk about some interesting things (movies we had seen, a great lunch place we found, etc) , and occasionally some work stuff as well.  Seriously – this was a time for the team to become comfortable with each other and build some common memories and start the day off on a pleasant note.  And everyone outside the team expected we’d do daily meetings and we didn’t want to disappoint them.
• Pick a bug, any bug.
• QA and Dev sit together and reproduce bug. Confer with BA as needed.
• QA and Dev examine code together, and create a characterization test to “clamp down” the code.
• Dev fixes code using TDD whenever possible, conferring with and pairing with QA and BA whenever needed
• QA and Dev test code to verify fix works as expected
• QA and Dev run all existing characterization tests
• Code is checked in and project is built
• QA runs all existing automated UA tests and any needed manual tests.
• Pick next bug – preferable one that is in some way related to last bug – but not critical. Just pick one, and repeat.

Release Schedule:

• About every two weeks a build of whatever was fixed at that point was made avaliable to customers.
• They could take the build if any bugs they were interested in were fixed, or they could ignore it.  It was up to them.

End Result:

Guess how long it took to clean this project of  bugs.  Give up?  3 months!

250 bugs in 3 months – that’s about 4 bugs a day!!! Well, we did have 17 “un-fixable” bugs.  Those were bugs that we weren’t allowed to fix for some business reason.  So still – an average of just under 4 bugs a day.

This is the power of eliminating waste.  All of our time was spent doing meaningful work, REAL WORK on bugs. No reviewing, no estimating, no “pre” investigation that wasn’t part of doing the fix, no passing code to QA that we were not confident would pass all tests, etc. In other words: NO WASTE.  Or at least: We were no longer doing any of the steps that we had identified as waste.

A Few Other Interesting Things:

Well, what we found is that in fixing some bugs, we’d incidently fix other bugs.  That is, some bugs were caused by a single underlying malfunction.  That happens, and it is a sweet bonus when it does.

We had no recidivism. Once delivered to the customers, we did not have a single fix that was “sent back”.  Once fixed it stayed fixed.  I think this can mostly be attributed to the QA and Dev working together, the unit tests put in place, and the “one-at-a-time” approach to “fix and build.” We would fix ONE BUG at a time. Fix,  Build, Unit Tests, Checked in, Built, Tested, confirmed.  Only then would next bug be tackled.

Conclusion

So… this is about no estimates.  Our rule: Eliminate waste.  In this case, estimates were a waste.  They were thought to provide a way to “manage” the work being done, but were in fact reducing the ability of the organization to “manage” getting work done.  I think they are like the dice in chutes and ladders? Probably not. Still, focus on the real work, don’t muddle things by playing chutes and ladders.

One little blog post cannot even begin to explore this complex subject.  Here is a start at beginning the first part. Or perhaps, a pre-beginning.

If you really want to figure this out quickly don’t wait for me to figure out how to write about this.  Just come and visit me sometime and I can quickly show you the whole thing.  It really isn’t hard.  Or, even easier, read the Agile Manifesto Values and Principles.  It’s all there.

A Few Paragraphs You Don’t Need To Read

It’s obviously not my job to tell you (teach you, show you,  guide you, coach you, whatever) how to be effective at software development.  My job is to write software, and coach and work on a team that is writing software – and to deploy that working software so people can use it.

I don’t write books, sell certifications, do trainings, or provide workshops for a living. I have nothing to sell you. [I’ve occasionally done introductory sessions about Agile stuff for a company or two.  I love doing that, but it is not my living or my job. ]

The company we work for makes irrigation products. Great company, great team, great product.  I truly love being able to come in to work every day to be with my team and do the work we do.  Point is: I make software for a living.

Estimates are NOT a deliverable

Regardless of how we define “estimate”*, it is not a deliverable in the world of software development.  I’ll write another post about that if you don’t agree .  (That’ll show you!)

The Basic “No Estimates” Approach

Here are the basics – in very simplified form told as a mini “case study”, based on a real situation from a long while back  (not my current job) – but I’ve changed the details so no one will get mad.

Players:

• Boss – you can think of Boss as the “Product Owner”, “Customer”, or “User” or some combination of these.  In any case, Boss is a “person(s)” who knows enough about the purpose of the software (or some part) to make reasonable decisions, and who is responsible and has authority for making those decisions.
• Woody – you can think of Woody as the person who guides the software development process – could be a devleoper on the team, or some other person who understands software development better than I do. We’ll call him Woody.
• Team – everyone else working on this “project”.

History:

• This was a while back, but Woody was already capable at “No Estimates Software Development”.
• This project was a new “internal” application.
• Project accessed existing databases, but also would likely need it’s own database.
• Boss had never worked with No-Estimate Software Development.
• There was a lengthy “requirements document”. It was believed that everything in the requirements doc was necessary. [Note: It was a mess, and if I had any influence I would have abandoned it].
• Team had all necessary skills and knowledge to do this work.

Approach for day one:

• Start with the huge list of way too much stuff that Boss says needs to get done (the original requirements doc, so to speak).
• Ask Boss to select one “small” thing that she felt was very important.
• That “small” thing becomes the “thing we are working on right now”: We’ll call it a “Story”, but on this job we didn’t use words like story.  We called them features.
• Team starts work on the Story.

Approach for day two and each day after that:

• Team works on the thing we are working on right now.  Nothing else.
• Team constantly assesses the work we are doing, and splits the current Story if it is not easy to understand or it is not clear how to start on it.  We check with Boss, but have no issue breaking things down.
• Team asks questions as soon as they come up, and get the answers right away.  Without this, projects drag on through endless muck.
• Team completes the Story we are working on right now (or the part of the Story that we split off)
• Team deploys the Story if it is deployable, and if it is not deployable then Team works with Boss to understand what needs to be added to it to make it deployable, does that, and then deploys.
• Team asks Boss to pick the next thing she thinks is very important.  It can be the other part of the previous story, or any other thing she thinks is important.

Result:

• First deployment happened in a little over a week.
• Project was deemed “done” in about a month and a week.
• As we started delivering usable software, the users and Boss started asking for different things than had been described in the requirements document.  The users and Boss started fine tuning what we had delievered – noting things that they wanted to be done differently.  This is a valuable and wonderful thing.  This is how requirements emerge.
• After about one quarter of the “features” described in the requirements list/document (including the additions the users and Boss had asked for), Boss decided the project was “done enough”.  The users were doing most of what they needed, and the rest wasn’t important enough to do, at least for the time being. There were other more important things to get done on another project.  The users were happy. Boss was happy. Team was happy. Woody was happy.
• Team was iterating (but not time-boxed), and incrementally delivering.

Observations:

• Boss wanted estimates, and also wanted the work done as quickly as possible.  I made the case that we could try to get some critical feature into production ASAP, and then revisit the idea of estimates later. After the first deployment, no requests were made for estimates.
• Estimates are part of the “comprehensive documentation” that we value less than “working software”.  If you don’t see that – please let me know.
• I am sure you saw the big win, right? We discovered the needed features. Most of what was thought to be needed was not actually needed.  Doing the work of writing the code in small pieces and getting it into real use made it possible to steer the work.  This is the “requirements emerge” concept.  I urge you and encourage you to think about it and experiment with it.  It is GOLDEN.
• Even if estimates were not liars, we just simply did not need them.

Well, that might be a little help.  It’s just one very basic example.  But it’s real.

This all just seems like “plain old Agile” to me.  Many people tell me I’m wrong.  What do I know?  I’d rather be wrong about what to call this and be getting a lot of work done than the other way around. Let me live in my delusions.

NOTES:

* For the purpose of this article, the sort of estimates I am discussing are the estimates typically asked for on many software development projects where a project, a feature, or a function, or a bug fix (or where a list of features or functions) are described and people are asked to come up with an approximate cost in time, money, or effort to do the work that will be required to provide the feature(s)/function(s)/capability(ies)/bug fix(es) being requested.

Disclaimer: There are many situations where estimates can be meaningful and useful.  This article is about situations where I don’t think they are typically meaningful or useful, and only in the realm of software development.

I’ve noticed that there seems to be a lot of people who feel they get a lot of value out of doing estimates*.  There also seems to be a growing number of people who suggest that estimates* are wasteful and even harmful to a project.  Whichever works for you, either way, use it. I have only one thing to suggest: Do your best to make sure you are not blinding yourself from finding new and better ways.

[Disclaimer: I am only talking about Estimates* as used in software development as part of planning or otherwise. See notes below]

I take the “not blinding myself” very seriously. Here is my Agile Maxim about that:

Agile Maxim #3 – Question Everything – put special focus on our deepest held beliefs. What we trust most hides our biggest problems.

[ You can see more thrilling and uplifting Agile Maxims here: Agile Maxims ]

You are perhaps thinking “But Woody – you seem to have a very deeply held belief that Estimates* are Useless and Wasteful – you should practice what you preach and Question That!”

Some will say “Woody: You should Practice what you Preach!!!”

You are absolutely right – I should practice what I preach. (By the way – I don’t believe that all estimates* and estimating* is useless and wasteful, just a great deal of how they are typically used in software development).  And of course, I have gotten to where I am in my thinking about estimates* by questioning the value and usefulness of estimates* over a long time – in both the realm of software development as well as other fields I have experience with.  Yet still, EVERY DAY, I re-scrutinize my thinking on the value of estimates* – and whether or not we should use them.  Sometimes more time, sometime less time – but it is rare that I don’t think about the usefulness of estimates* (or of not having estimates*) at some point in the day.  If estimates* bring value, and I am losing out on that value because I am too bone-headed to see it, I want to be set straight. And on the team I work with we constantly experiment and try  new things, old things, different things, same things – whatever we feel we need to do to learn and make things better.

So… here are the sort of things I ask myself or consider every day about estimates*:

• If I found that estimates* are indeed wasteful, what would I do?
• If I found that estimates* were misleading and giving us bad information, what would I do?
• What if estimates* never existed – if they were never invented… is there some other way we could work?
• What if there is a better way to get the right work done using some other process, would I still estimate*?

And it goes the other way:

• What if we were not able to make decisions unless we had estimates*, what would I change about our work?
• What if we needed estimates*, but the estimates* we were doing were not very good, how would I fix that?
• What if estimates* provide a way to see into the future and make GREAT decisions, what would I do?
• and so on…

And so on

This whole questioning process takes little time on a daily basis.  Occasionally, I scrutinize this sort of thing to a much deeper level – sometimes in a focused retrospective, or just thinking for a while, or in talking with other Agile-maniacs, or even by doing (or attending) a session at some user group or Agile Open event. I WANT TO KNOW THE TRUTH!  I want to steer and adjust my thinking.  I have no pride – if I learn a better way, I will evaluate how to incorporate it into what we’re doing – and experiment, reflect, tune, adjust, throw out, bring back, etc.  I am always open to THROW OUT THE OLD if it is no longer useful.

Bottom Line

I don’t care if you use estimates* or not.  If it brings value to you, do them. If you want my opinion: I suggest you find a way to make sure they help and don’t simply use them because “that is how we do things”.

To quote the poet:

Whatever gets you through the night, it’s alright, it’s alright.

NOTES:

* For the purpose of this article, the sort of estimates I am discussing are the estimates typically asked for on many software development projects where a project, a feature, or a function, or a bug fix (or where a list of features or functions) are described and people are asked to come up with an approximate cost in time, money, or effort to do the work that will be required to provide the feature(s)/function(s)/capability(ies)/bug fix(es) being requested.

Disclaimer: There are many situations where estimates can be meaningful and useful.  This article is about situations where I don’t think they are typically meaningful or useful, and only in the realm of software development.

Estimating Software Development

Anyone who knows me will know that the title of this post is meant as a bit of a joke.   For software development, it is my experience that estimating** is rarely useful and always easy.

Well… a little clarification:  estimating is NEVER useful but ALWAYS easy the way I have typically seen it done, which uses the WAG* method of providing some numbers that seem plausible and that will allow the decision makers (stakeholders and managers) to approve the start or continuance of a project with a clear conscience. That is, when things sour later on they can always blame the developers for giving them inaccurate estimates (and not living up to their “commitments”).

And the developers can always blame the “requirements people” for giving them unclear, incorrect, or incomplete requirements.

And the requirements people can always blame the stakeholders for providing bad direction about what was important and the developers for not “understanding the requirements”, and so on.

Regardless of who is part of this process, it is one big happy circle of blame that lets us all do the wrong thing and still keep our jobs (not always happy for everyone, actually, and sometimes not everyone will keep their job).

I have simplified it here, but the basic idea is sound.  As long as everyone is good at deflecting blame, and everyone is willing to continue on saying we’ll do it better “next-time” and after only a few people have been fired then everything is fine, right? Well, I don’t think it’s fine, of course – but many organizations seem to operate this way.

Let’s estimate a game of chess

Chess is a game that some of you may have seen.  It provides a very limited environment with only 6 different and charming pieces that have very limited ways in which they can be used, and there are only two players.  Simple. The whole game can be contained in an area of about a square foot or even less, and the complete rules can be written on the inside of the box lid.   Just like Candy Land, it is very easy to learn and play. Not so easy to win, perhaps – but that makes it interesting enough to be an engaging game for some.

So, here is a description of a simple “project” and I need your estimate:

Win a game of chess. Losing is not an option. Tying is not an option.  We will only make money if you win.  If you don’t win, the company is at risk, and you will lose your job (and mine).

Also, I have a diagram of a chess game with the exact position of the pieces as required at the end of the game.  To make money, it needs to end up just like the diagram.

Oh, one other thing: we don’t know yet who you will playing, but we’re pretty certain it will be someone who knows how to play the game, and might be pretty good at it.

That’s it!  There are only a few requirements:  That you win, and that the board matches the diagram I’ll give you.  Easy!

What I need to know is exactly how much this will cost me.  In this situation, cost is the number of moves you’ll need to meet the requirements.  Oh, also, we don’t have the diagram yet, but we’ll have it soon.  Still, you should be able to estimate it because it is basically the normal chess end-of-the-game arrangement with pieces here and there and so on, and you have the opponent in a checkmate.

Should be easy to estimate that, right?

Well, I hope you agree this is impossibly difficult, or actually: just plain impossible to do a meaningful estimate.  First of all, there is not enough information to do a good estimate, and even if we had all the information, there are many variables and a lot of stuff we have no control over. How close do you think your end results would come to the requirements?  Would you be willing to stake your job on making a decent estimate on a chess game in this situation?  Well, as long as you could blame someone else and promise to “all do better next time” you’ll probably be okay.  Or you could be a consultant and be paid and off to your next contract before someone actually has to play this game you helped them estimate.

Getting better at estimating is like pulling teeth

I hear it over and over:  Our estimates were not accurate and we had a lot of trouble because of that.  We need to get better at estimating.

It is similar in a way to saying “My tooth hurt, so we pulled it out.  Now I can’t chew as well as I used to.  We need to get better at pulling teeth”.  What is the real problem?  Shouldn’t we work on that instead?  Hint: Getting better at pulling teeth is not it.

So what would you do to get better at estimating the game of chess, based on the basic scenario we described above?   Could you get better at it?   How good would you have to be before it had real value? Is it worth spending time on getting better at it?  What is the real problem.  Hint: Getting better at estimates is not a solution to anything because having bad estimates it not the real problem.

My suggestion: Stop That!

I haven’t seen much value in estimates.  Actually, in the software development work I have done, I can’t recall any situation where estimates** of this sort were of value to the actual job at hand.  Someone wanted them, someone saw value in them, and sadly – important decisions were made based on them.  Do you think those were good decisions?  People often tell me that “even though we know the estimates are inaccurate, they are better than nothing”.  I prefer nothing, if there are better ways.  There are better ways.

Almost everyone tells me that estimates** are very important, but estimates are often a blind spot hiding something of actual importance and value.  This almost universal acceptance that estimates are important and useful is harmful.  A predictive approach such as waterfall itself depends on estimates – and estimates are how we make the predictions.  Don’t think “waterfall” and predictions are harmful?  You probably should be reading some other blog.

Estimates** of this sort are based on guesses about the time needed to do work we don’t yet understand.  Nothing about this gives me confidence that these estimates have value.

It seems I always make enemies when I suggest this, but that is not my intention.  I want to do effective work on meaningful stuff – that is my intention.  I don’t want to be spending time on work that has no value, causes misdirection and waste, and potentially destroys any chance of being effective.  That is painful.  There are MUCH BETTER WAYS to approach software development.

Do you need estimates?

It could be useful for you and your organization to question your use of estimates and explore new ways to think about your “need” for estimates.

One last thing:  Do not ask me for an estimate on how long it will take to eliminate estimates from your process.  It is too much like a game of chess, only very complicated.

Here is a start at clearing up our thinking.  Let’s do a little thought experiment.

Which would better help if we have decided to eliminate estimates and find a better way in our organization? :

• Get rid of all developers
• Get rid of all upper management.
• Get rid of all upper management and developers
• Something else more realistic

Who is most insistent that estimates are important?  Who do those people answer to? Are they going to fire you for questioning things like estimates? Are they open to scrutinizing the status quo or to examining why we do the things we do?

So what might be better?

What can we do instead of doing estimates?

Nothing.  Don’t do estimates.  Simple, clean, easy.  But that is not the question we should be asking, and that is not what I am suggesting.

The correct question is: since we think that estimates are important, and you say that they are not, how can we develop software without doing estimates?

The answer is… Well, I suspect no one reads this blog, so that doesn’t really matter.  This is just a clearing house for my thinking and has nothing to do with the reader.  However, I am sure to write some stuff someday about ways to get things done without estimates, and I guess some of my earlier posts cover some of this thinking. If someone is actually interested and asks for me to cover this, I might get to it sooner.  But I am sure you can figure it out without me – this is truly simple stuff.  Read the Agile Manifesto and Agile Principles, and use them to invent better ways.

Another hint: Get good at frequent incremental delivery of useful software.  That might be worth considering.  That takes a LOT of experience and open thinking.

NOTES:

* WAG = Wild Ass Guess.  A Wild Ass is a beast that is indigenous to parts of Africa and Asia.  A Guess is a guess.  Wild asses are about as good at making software estimates as anyone else.

** For the purpose of this article, the sort of estimates I am discussing are the estimates typically asked for on many projects where a list of features or functions are described and people are asked to come up with an approximate cost in time (working time or elapsed time) [sometimes it is phrased as “cost”, or “effort”] to do the work that will be required to provide the feature(s)/function(s)/capability(ies) being requested.

Disclaimer: There are many situations where estimates can be meaningful and useful.  This article is about situations where I don’t think they are meaningful or useful.

Requirements Gathering

In the traditional phased SDLC approach, a phase for  “Requirements Gathering” is done very early on in the process.  We can argue about all the issues with this some other time – I am just covering one issue here.

My opinion of this “phase” is that it is pretty much worthless: THERE IS NO WAY TO PROVE THAT THE REQUIREMENTS WE’VE GATHERED ARE COMPLETE AND CORRECT UNTIL WE HAVE WORKING CODE.  Working code, in this sense, means code that is actually being used to do real work for the real users. Does that make sense?  If not, you probably don’t want to read on, and probably shouldn’t bother reading anything I have ever written or will someday write.  Plain and simple.

Requirements Hunting and Gathering

One issue is that requirements will rot if not used promptly after collecting them.

Early humans are thought to have used a subsistence strategy where they hunted and gathered (and scavenged and stole) their food for immediate use.  If the food was not eaten shortly after it was obtained, it would rot or otherwise become useless.  The food needed to be put to work within minutes or hours, or at most a few days.  Otherwise, the effort of hunting and gathering would go to waste.

Just like the early hunter and gatherer folks, we have no way to “preserve” the requirements we’ve gathered.  We don’t even know if they are edible.  And yet we store them for a long time, and do a lot of work with them during that time, and base a lot of decisions and work on them (like estimates, budgeting, designs, architectures, ramping-up efforts, coding, testing, marketing, and so on) and all the time the requirements are rotting.

It isn’t until the code becomes true working code that we can get real information about the correctness and completeness.  Until then everything is just assumptions and guesses.  Unfortunately, by the time we learn that the requirements have indeed rotted it is too late.  They have become toxic, wasting our money, time, and effort, and killing the usefulness of the project they are supposed to define.

Early Humans understood this, and therefore rarely wrote software, and never ate mayonnaise that had not been “refrigerated after opening”.  Fortunately for Late Humans (us, that is), someone eventually figured out how to preserve foods so we could buy and eat them days, months, or years and years after they had been plucked, prepared, and packaged.   However, to this day, I have seen no way to preserve requirements.

Requirements MUST be used while fresh

So, my recommendation is: Eat your requirements as soon after you have picked them as possible.  Between the garden and the kitchen they have already started to lose their sweetness.  [Please imagine that there are many other very funny food based metaphors here.  I am not having trouble coming up with them as such, they just don’t seem all that meaningful, and they aren’t as funny as I thought they would be].

How do you do this with software?  You use a very short “discover, define, code/review/re-discover, deliver, discover more, repeat” approach during which we are continually discovering what it is we really need to do:

1. Keep your requirements at a very high and general level until just before use.  Details in the requirements are a smell of the rot.  Pay attention to that.  Understanding the business need is important, details about how that translates into software is not.
2. Pinch off a small but useful bit of one of the high level requirements – preferably something of immediate and high value to the end user, but anything of value will do.
   • Find something that you have a pretty good chance of turning into working code quickly – in a day or two if possible.
   • Prune as you go as needed.
   • Identifying small bits, pruning as you go, and getting something of value quickly are skills you must aquire.  It takes practice, dilligence, and close attention.
   • You don’t have to understand the requirement completely – you will discover the details and what is important as you code it up while partnering with the business experts/users.
   • Prioritizing is useful – you want to take on features in a prioritized manner, but don’t go overboard on this – close enough is close enough.  We’ll talk about this some day.
3. Code it up.  The code-it-up process is not just coding – we must work closely with the business people discovering the details as we go.
   • Code some tiny bit the business expert can see and use (click a button, see a list, choose an option, whatever)
   • Get her feedback, comments, and direction continuously – early on and frequently – every 10 minutes to 2 hours.  Every time you have something meaningful you can show get it in front of the business expert.
   • Sitting side-by-side works well when appropriate, but if that isn’t practical at least be within shouting distance.
   •  If you go longer than 4 hours without getting useful feedback you are wasting your time and need to learn how to work in smaller chunks.
   • You might discover you have completed some small but deliverable bit of funcitonality that is just a part of what you had chosen to work on.  If so, then deliver it, and go on to the next bit.
4. Deliver it.  That is – put the code into actual use with actual users.
   • You’ll discover more things about it as the users use it.
   • This will help expose the next thing to work on.  The real next priority.
   • If they don’t use it once delivered, you’ll have learned something else of value: It wasn’t of value. I’ll cover that some other day – but that is a magical and very valuable bit of knowledge to gain early.
5. Do it again tomorrow (or today, I suppose) with another bite-size chunk of functionality.
   • You are now steering the project toward the best possible result given the abilities and knowledge you and your team have.
   • Important things will become apparent and be added to the working code in a prioritized manner.
   • Useless things will show themselves or be ignored and you’ll waste as little time as possible on them.
   • You’ll get better and better at it.

Following this approach the requirements will not rot because you leave them “on the vine” until you are ready to eat and digest them.  Of course, what comes out after the eating and digesting is not covered in this article:  At some point all analogies break down.

[NOTE: Everything I say in this post and on this blog is based on my experiences, study, and observations.  It is MY OPINION. It is the way things seem to me.  It might be useful to you, might not – I don’t really care.  If you want to argue about it, that is fine.  I am suggesting that there are better ways to do almost everything we do in software development, and that it is worth thinking about it and doing something about it. You might already have better ways to do things, and I want to hear about it.]