Life, Liberty, and the Pursuit of Agility

In this episode I’ll cover an extremely satisfying application of Lean and Agile on a large ‘Legacy’ effort I was part of a while back.

Please understand that I am not at liberty to discuss certain (that is, most) details. I must protect the guilty. I will give just the bare details – only what is necessary. I bet you don’t believe I will be able to give just the bare details, right?

So – this is about using the No Estimate Approach on an End-Of-Life Legacy Support project.

First, a few definitions:

What is “Legacy” code?

I am glad you asked.  There are many definitions of legacy code. Here is the one I use: Legacy code is ANY CODE THAT IS DOING ACTUAL WORK THAT YOU WANT TO KEEP WORKING. (Please note the clever use of “all caps” for emphasis.)  Does that make sense to you? Doesn’t matter – just pretend that is what it means.  Using my definition, most projects become legacy code as soon as you have some functionality in production. That’s when the bits hit the fan, usually, so to speak.  However, even if it isn’t yet in production, it could still be “legacy” if we have stuff working we don’t want to break.

What is Legacy Support?

Regardless of what you think legacy support is, for the purposes of this post I am using it to mean this: “Work done (bug fixes and minor enhancements – that sort of thing, or more serious stuff if needed) to extend and protect the life of code that is in use”.   I’m leaving out more extensive enhancements like “whole, new functionality”.  You can judge for yourself the difference between minor and “more extensive” enhancements.  Doesn’t really matter to me.

What is End-Of-Life?

For the purposes of this post, an End-Of-Life project is a project that is almost retired.  It’s still in use but going to be replaced sometime soon.   In software programming “Soon” means “perhaps sometime, but we have no clue when.”  The basic idea is that people are using it, and it is important enough to keep supporting, but we’ll stop doing big additions and improvements.  Some bugs will just be ignored, but any deemed to be important to fix will be fixed if possible.

Project History and Details

The project that we are discussing is is a large, line-of-business set of applications (‘The System’) that is critical to the users of the software for managing most aspects of their business.  The typical user of The System manages thousands to millions of their own customers – sales, billing, transactions of all sorts, bunches of “real-time” stuff, reporting, etc.  Typical line-of-business stuff.  I was told the code base was about 2 million lines of code [and that’s about enough, in my opinion – any more is just bragging.]

The System has been in production for 15 years or so and has moved through several different code bases.  The current code base is about 4 or 5 years old.  It uses several languages – 4 or 5 perhaps.  Typical stuff.

Enhancements and bug fixes are managed, worked on, and delivered using a waterfall approach, with a 3 month delivery schedule (that takes 6 months due to test and fix cycle, and other typical deadline issues).

Due to design, industry, and technical reasons it is deemed necessary to re-write The System in a new language with a new architecture.

The Path Going Forward

The current version of The System MUST continue to serve the user base, and even more – it has to be clear that we are still responsive to concerns and issues of the user base so they don’t start looking to competitors to solve problems while the “greenfield” version of The System is being developed.  It has to look and feel like “business as usual, only better”.

A “maintenance team” is put together to provide this service.

I’m not sure exactly how many people were working on The System before the End-Of-Life Maintenance, but it was something on the order of 20 to 25 people – maybe more, it was a while back and I didn’t keep track.

The new team was to be 6 people: 2 QA, 2 BA (Business analyst or “Product Owner”), 2 Dev.  I was one of the two Devs, and also the manager, or co-manager (more or less).  I “volunteered” for this assignment on one condition: I would be allowed to manage this effort my way – which means Lean/Agile/XP.

We’ll call this System EOL (for system end-of-life) .

The State of System EOL  on Day One

There are approximately 500 active bugs in the bug tracking system.  Some of these bugs have been pernicious. Some have never been addressed due to level, some have been introduced in the fixing of other bugs, some have been there a long time but just recently discovered – All the typical bug stuff.

500 Acitve Bugs???!

There are 500 active bugs! There are a number of reasons there are so many bugs.  Before we talk about that, I’d like to ask a question.

• Question: What is an appropriate number of known bugs for a project that is in production?
• Answer: Zero.

Got that? I’ll allow for any bugs that have easy work-arounds – you can leave those if you have to.  I’ll also allow for any bugs that were discovered since the last deployment.  But you better fix those NOW and then you’ll have zero bugs once again. Nice.

Remember that: Zero bugs.  Does that seem unreasonable to you? If it does, I’ll ask another question: Why bother reading my blog? Go do something else that seems reasonable to you.  Why waste your time on what I’ve got to say? I am NOT REASONABLE most of the time.  I want a better way to do stuff than whatever we used to think is “reasonable”.  Zero bugs is a “lofty goal” in some workplaces.  In others, it is business as usual. I prefer zero bugs.

So, 500 bugs is not just way too many, it is infinitely too many.  Or whatever 500 divided by zero is.  Anyone know?

One reason there are so many bugs: The old process.

There were several reasons all these bugs existed, but the one that is most pertinent to our discussion today is the process that was in place for assessing and fixing bugs. The way things were being done was about like this:

• Bug report is entered into the tracking system by a user, support person, developer
• The bug is reviewed at a “triage” meeting where its importance determined
• If it is important enough, it is assigned to an “investigator” who will assess it: that is, attempt to reproduce it, determine the likely cause and module that will probably contains the “bad code”,  and estimates the time to fix.
• After the assesment, the bug goes back to a “bug review” meeting to re-assess it’s importance and schedule the work to be done.
• If it is deemed important enought, it is scheduled to be fixed (either sooner or later based on the “cost” of fixing it and importance to the customers).  If it is not important enough, it is closed and it’s status and other details are communicated to the interested parties.
• It is assigned to a developer who works in the area of the “bad code”. Since the developer is a different person than the investigator, the developer starts from square one and “re-thinks” the stuff the investigator already thought about.
• In working on the bug, the developer discovers the problem is NOT in the area of code it was thought to be in, so he adds his comments and assigns it back to the bug review team.
• The bug review team looks at it again, and then re-assigns it to an investigator.
• And so on and so on…
• Once a bug is actually fixed, it goes thorugh QA where it is discovered that it didn’t fix what needed to be fixed.
• Go through whole process again.
• Once a bug is fixed and passes QA, it is deployed in the next deployment, and…
• The users discover that it didn’t fix what needed to be fixed, or the fix was only partial, or that something else is now broken.
• Even worse – some other things have been broken that are not yet discovered that will lead to other bug reports sometime soon.

Okay – you get the point.  I am not making this up.

There were other reasons System EOL was in this state, which included lack of unit tests, code neglect (huge methods, overwhelming complexity, fragility, coupling and cohesion problems), and so on… but we’ll solve one thing at a time.

Chutes and Ladders

Well.  Does that remind you of that game you used to play when you were a kid?  It should.  And it is JUST AS FRUSTRATING, at least conceptually.  Only difference is you get paid to play Bug Fixing Chutes and Ladders.  But overall – it is terrible and horrendous (and other bad words).

What to do???

Since I was allowed to manage System EOL in my own way – Here is what we did.

Value Stream Map

First day I brought in 6 copies of Mary and Tom Poppendiecks “Lean Software Development” book, and we all agreed to read it, and more or less follow their basic concept.  I’d been doing XP for years (more or less) but it’s a hard sell so I opted for the “easy sell”.  If Lean is good enough for Toyota, it’s good enough for me.

First thing we did was a value stream map on the  “Chutes and Ladders” methodolgy that was in use. If you haven’t seen a value stream map or know how to do one, take a look at Mary and Tom Poppendiecks “Lean Software Development” book.  The basic idea is that anything that is not directly adding value can (and should) be eliminated. For example: waiting in queues is a waste – it brings no value to the person interested in having the bug fixed.

I bet you can guess that we found a lot of NON-VALUE-ADDING behaviors.  So, we made a few rules.  For example…

First of all, a few rules:

• No Triage meetings or Bug Review meetings
• No Assesments for “where the code is broken”
• No prioritization or severity determination (a bug is a bug is a bug).  If it’s in the tracking system, fix it.
• No Estimation of effort. We gotta work on all the bugs.
• … and Other things

NOTE: No Estimates, No prioritization, No meetings.  These are wastes (at least in some situations, if not a lot) – bringing no value to the “customer”. Even worse,  waste usually removes value by using up time and “spirit”.  A team doing a lot of meetings, estimates, assessments, meetings, prioritization, meetings, waiting, etc. becomes NUMB. Avoid numbness.

An initial assessment:

We quickly read each bug in the tracking system, looking for duplicates, bugs no one had asked about in a year or more, and bugs where the description could not be understood (and there was no one who could clarify). This was done very quickly and eliminated about one half of the bugs.  NICE. 500 bugs cut down to 250 bugs.  No one can say that is not progress!

This took about a week (as I recall).  During this “first-week” the developers and QA took on bugs to fix, since there were plenty that were clear enough to work on – but spent some of their time helping the BA’s “weed out” the bugs that we removed.

Daily process we adopted:

• Daily morning 1/2 hour meeting.  Sitting. Quickly talk about some interesting things (movies we had seen, a great lunch place we found, etc) , and occasionally some work stuff as well.  Seriously – this was a time for the team to become comfortable with each other and build some common memories and start the day off on a pleasant note.  And everyone outside the team expected we’d do daily meetings and we didn’t want to disappoint them.
• Pick a bug, any bug.
• QA and Dev sit together and reproduce bug. Confer with BA as needed.
• QA and Dev examine code together, and create a characterization test to “clamp down” the code.
• Dev fixes code using TDD whenever possible, conferring with and pairing with QA and BA whenever needed
• QA and Dev test code to verify fix works as expected
• QA and Dev run all existing characterization tests
• Code is checked in and project is built
• QA runs all existing automated UA tests and any needed manual tests.
• Pick next bug – preferable one that is in some way related to last bug – but not critical. Just pick one, and repeat.

Release Schedule:

• About every two weeks a build of whatever was fixed at that point was made avaliable to customers.
• They could take the build if any bugs they were interested in were fixed, or they could ignore it.  It was up to them.

End Result:

Guess how long it took to clean this project of  bugs.  Give up?  3 months!

250 bugs in 3 months – that’s about 4 bugs a day!!! Well, we did have 17 “un-fixable” bugs.  Those were bugs that we weren’t allowed to fix for some business reason.  So still – an average of just under 4 bugs a day.

This is the power of eliminating waste.  All of our time was spent doing meaningful work, REAL WORK on bugs. No reviewing, no estimating, no “pre” investigation that wasn’t part of doing the fix, no passing code to QA that we were not confident would pass all tests, etc. In other words: NO WASTE.  Or at least: We were no longer doing any of the steps that we had identified as waste.

A Few Other Interesting Things:

Well, what we found is that in fixing some bugs, we’d incidently fix other bugs.  That is, some bugs were caused by a single underlying malfunction.  That happens, and it is a sweet bonus when it does.

We had no recidivism. Once delivered to the customers, we did not have a single fix that was “sent back”.  Once fixed it stayed fixed.  I think this can mostly be attributed to the QA and Dev working together, the unit tests put in place, and the “one-at-a-time” approach to “fix and build.” We would fix ONE BUG at a time. Fix,  Build, Unit Tests, Checked in, Built, Tested, confirmed.  Only then would next bug be tackled.

Conclusion

So… this is about no estimates.  Our rule: Eliminate waste.  In this case, estimates were a waste.  They were thought to provide a way to “manage” the work being done, but were in fact reducing the ability of the organization to “manage” getting work done.  I think they are like the dice in chutes and ladders? Probably not. Still, focus on the real work, don’t muddle things by playing chutes and ladders.

I read a question posted on the Scrum Alliance Google Group, and jotted down a few thoughts about Planning Card Games.

The Basic Question Being Asked:

How do you describe a 1 to the team for Planning Poker®?

The value of Estimating using Card Games:

To me, the main value of story point estimating is as a tool for a team to learn how to break things down into “Sprint size” pieces.  As a team becomes skilled at breaking stories into smaller (yet still valuable and deliverable) pieces, the less need they have for using estimating games. Once the ability to “chip off a small, do-able, deliverable chunk” becomes easy for the team, the work flow changes a bit.  The team no longer needs estimating games – they simply choose important stuff, break off a few valuable, small pieces to work on, and then do the work and build/deploy.  Over and over again.

Many don’t see it that way, and use estimating games to make decisions about what work to bring into a Sprint. That is a legitimate use in some workplaces, and whatever works for you is the right thing.  For me, the scheduling of work for a Sprint is not nearly as important as breaking work into a “do-able” condition, and getting it done.  I’m interested in choosing what we work on based on the value to the “customer”,  not how big things are.  We can only do so much work at a time.  Finding valuable things to work on is where I put my focus.

Discovering What Those Numbers Mean To A Team

One approach that has worked for me in finding a way to size things is to do it during the first Sprint Planning Session:
Take all the Stories you are considering, and lay them out in “size order” on the table – No numbers, just “Is this bigger or smaller (or the same) as that”.  Once you have them laid out you can group them into the numbering system you use.  The group of the small items becomes 1, next group 2, and so on.  During the Sprint you will get a feel for if the 2’s were more or less double the 1’s, and so on.  “More or less” is fine – you don’t need precision.  You are just “getting a feel for it” anyway.

Over time, the team will be able to quickly gauge a basic size for each story – with the main goal being to find the ones that need to be broken down further so they can be taken on as work.  Once a story is taken on as work, you’ll discover the real nature of the story and it’s “size” – it is often found that it needs to be broken up into a few smaller chunks.

Still, What is the Answer to “How To Describe a 1 to the Team”

This doesn’t really answer your question: “How do you describe a 1 to the team for Planning Poker®?” – but it replaces it with a way to work with the numbers. In the past when I used story points and planning cards, I found this to be the definition that worked for me:  The “1” story point is the smallest thing that we can do discretely that brings value once built and delivered.  Smaller than 1 is the place to put things that are tiny, but not really stories.  We would mark something as zero if we felt it would “come for free” in the doing of another story.  Also, anything bigger than a 5 was put into a bucket of “Too big, must break down before we put any work on it”.  For me, usually 1 & 2 are considered for taking on as work, 3 & 5 need to be sliced up if at all possible (and if it can’t be easily sliced up it’s a signal that we need to learn how to better define and break up a story),  and 8+ is “Too Big – must be sliced up”.

Note: Planning Poker® is a registered trademark of Mountain Goat Software, LLC.

I wrote a short post called “Is The Daily Stand-Up Harming Your Team”.  After getting some feedback on it, I have refactored it and here is the new version.  (I have deprecated the old version):

The Purpose of Stand-Ups

Here is a short definition from “Planning Extreme Programming” (Beck/Fowler 2001):

Have a short meeting every day so everybody knows what’s going on, and what’s not.

The Daily Stand-Up meeting is meant to be a very short meeting with several related purposes:

• Team Alignment or “sync-up” – who is working on what, what’s been done, what do we need to discuss…
• Expose issues that need to be handled (impediments, etc.)
• Eliminate other meetings that can “hog time”
• Allow some other people not on the team to hear about things the team is doing
• etc – put your own things you value about Stand-Ups here.

Those all bundle up to one thing: Daily communication about important stuff.

Good Communication Is Essential for Software Development

Programming is done by individuals who must communicate with each other.  Finding ways to communicate well makes things a lot better.  The Agile Manifesto itself puts a lot of focus on the importance of good communication.  A quick glance at the Manifesto and Principles makes this clear:  Individuals and Interactions, Customer Collaboration, Responding to Change, business people and developers must work together daily, face-to-face conversation, self-organizing teams, the team reflects on how to become more effective.  Almost every Value and Principle somehow reiterates the need for good communication.

I “buy in” to all of this.  In my opinion, a BIG part of becoming and staying effective in software development is to continually improve our ability to communicate – team member to team member, team to customer, business people to team, and whatever combination you need.  As we find problems with traditional communication methods (such as meetings, emails, documents, charts, yelling,  etc.) we need to explore “better”.  Even more important, when we find something is working for us we should explore how to make it even better.  Whatever we do, no matter how good, there is always “better”.

A little Background:

A number of years ago (about 10 or 11 years I guess) I started visiting Agile/Scrum/XP development teams/departments and observing Stand-Ups and other meetings – just for the fun of it .  I’d been doing XP for several years, and loved visiting other teams – mostly for my own enjoyment, but also to learn how others were doing things. I have visited a number of teams and departments over the years – about 20 or so.  I am not a consultant as such – so these were casual visits.

What I noticed in visiting teams doing Stand-Ups

I noticed a few harmful things in a majority of the Daily Stand-Up meetings I observed. Things such as:

• Individuals taking a long time (5 or more minutes each rather than 2 minutes)
• Meeting taking too long (45-60 minute standups – more like “lean-ups” after a while)
• Too Much Detail, Not enough Detail
• People not showing up on time
• Too many people (10 or 20 people) – attendees are hearing about stuff that isn’t relevant to them.
• Mumbling and Grunting
• Discussions between 2 or 3 team members to resolve some aspect of something
• Yelling
• No one is interested in what the others are saying (and often without any reason to be interested)
• Issues brought up are never addressed
• Needed follow-up meetings never occur
• Same report every day
• Dogmatic enforcement of rules
• … I can go on and on. I used to take notes and had about 20 different issues that commonly showed up.

I have seen a lot of these same dysfunctions on teams I ‘ve worked with over the years.  Some of these teams have discovered ways to reduce or even eliminate most of these “Daily Stand-Up” dysfunctions.

There have been countless blog posts, book chapters, workshops, conference sessions, etc. on solving “Daily Stand-Up” problems – which is great.  It is a good thing to recognize problems and solve them.

But I think there is a “next step” we can take.  For me, when I see something causing this much trouble for so many different folks and teams, in so many dfferent ways, over and over again – I start to wonder if there is something we are all missing.

My first question was: Why do so many teams have so many problems with Stand-Ups???

The Daily Stand-up is a fixture in most Agile/XP/Scrum environments.  It has a very simple purpose, and is a very simple meeting – and yet, many teams continue to have these problems. Why???  Well,  that is too complex for me to cover completely here (I could cover some of that in a future post if anyone is interested) – what I want do do instead is share what I have discovered after investigating this for about 5 or 6 years. I’ve done a number of “5-whys” root-cause analysis sessions, and retrospectives on vairous Stand-Up problems, and I’ve read as much as I could find on the topic.

Here is what I think I have found:

At the bottom of many of my “5-why” Sessions

You’ve probably all done something like a 5-Whys analysis. From Wikipedia:

The 5 Whys is a question-asking technique used to explore the cause-and-effect relationships underlying a particular problem. The primary goal of the technique is to determine the root cause of a defect or problem.

With manny of the dysfunctions I was investigating, I would end up at about the same place.  Typical starting questions were: “Why do team members think the Stand-Up is boring”, or “Why do we have too many things we need to cover in the Stand-Up”, or “Why does our Stand-Up take so long”.  As often as not, the near the bottom of my “5-why” session was this question: “Why do we think we NEED Stand-Ups?”, and the answer I often end up with is “The Daily Stand-up exists because we are not in continuous alignment. We need the meeting because there are things we are not communicating during the day”.  So that leads to this question: “Why aren’t we in continuous alignment?”.

Why aren’t we in continuous alignment?

That is: If we could be in “continuous alignment”, and find we have nothing to communicate in the Daily Stand-Up, we wouldn’t have any of the dysfunctions of the Daily Stand-Up because we would no longer be holding them.

Would it be good to be in Continuous Alignment?  What if it was possible to effectively and efficiently stay in “continuous alignment” how would that change our thinking about the need for Daily Stand-ups?  Some teams have been very happy with the Daily Stand-Up, and find that it brings a lot of value.  Still, even in those environments, there is always room for improvement.  In Agile/Lean/XP/Scrum it is important to continuously scrutinize the practices we are using and innovate even better ways to do things.

It seems worthwhile to me to at least explore the possibility that there is something even better than well functioning Daily Stand-Ups.

Are You Saying We Should Just Stop Doing Our Daily Stand-Up?

Of course not.  But this hints at an important “next step”:  Let’s figure out how to have continuous alignment.  Some common practices that many teams already use are things like: Task Boards that everyone can see and that are dynamically updated as work gets done, sitting together (so you can hear anything imporant that is happening), pair-programming, switching up pairs frequently (several times a day), raising “blocking issues”(and solving them) immediately rather than waiting until a meeting,  TDD, CI, and whatever you can think of.  The more complete and meaningful you can be with dynamic communication the better chance you have of being in continuous alignment.

Of course – these practices will not necessarily elminate the need for the Daily Stand-Up – lots of teams use them and still find they have a need for the Stand-Up, but they will very likely resolve a lot of the traditional “Daily Stand-Up” issues I listed above.  A shorter Daily Stand-Up that is more meaningful for the team could be a worthwhileend result.

What are the Costs of “Continuous Alignment”

The “costs” for being in “continuous alignment” should be small – it’s a matter of discovering the things you need to communicate and a mechanism for doing having those communications.  Most Agile teams are already doing a lot of the practices that help provide good communication throughout the day – it’s just a matter of fine-tuning them, or “turning them up a notch”.  Becomming better at communication is a benefit in any team situation. We can choose the Daily Stand-Up over Continuous Alignment, vice versa, or both in some combination.  Each time we find something in a Daily Stand-up that should have been communicated earlier we should look into it and figure out how we could have communicated it dynamically.  The main cost is that you’ll need to pay a lot more attention for a while – but as you smooth things out it will become natural to gravitate to better forms of communication.

What My Experience Has Shown Me

If we have continuous alignment, there is little need for a Daily Stand-Up.  When it gets to the point where we gather in the morning for our Daily Stand-up and consistently find we have nothing useful to share it is likely that the practice can be retired.  In the “Lean” sense, a practice is waste if it brings no value.  Waste is not neutral, it diminishes the team – it is a cost that has no return.  With my current team we’ve worked without Stand-Ups for over a year and have found no need for them.

Here are some of the practices we follow that allow us to do this:

• We work together througout the day
• We meet at any time there is something important that needs to be addressed
• We pay attention to update our dynamic “information radiators” as we work
• Anytime we need help we ask for help
• Anytime someone asks for help, we help
• We meet face-to-face for all meetings whenever possible
• We all work on the same thing until it is done or there is nothing further that can be done with it (“one story at a time”)
• We solve blocking items immediately if possible, and communicate with each other when anything changes

My Advice – A Few Ideas

If you want my advice it is this:

• Examine your Daily Stand-Ups regularly
• Pay attention to anything that comes up in the Stand-Up that should have been communicated earlier
• Hold a retrospective with a focus on the Stand-Up – do this several times a year
• Find better “dynamic” communication techniques
• If you are not doing pair-programming, you should practice it and learn how to make it work for you
• Switch up pairs frequently – every few hours
• If you can’t stand the idea of pair-programming, do some Code Dojos and Katas to experiment with it
• Sit together – all team members working within easy talking distance of each other
• Do more work as a team – QA and Dev pairing, “Customer” and Dev pairing, QA/Dev/Customer teaming, and so on.
• Improve your “information radiators” and keep them updated throughout the day
• Try “Mob Programming” where everyone works at the same computer all together at the same time
• Talk with each other during the day

That’s just a start.  You will probably come up with better ideas than I have.  Please let me know what you find.

So: Are Daily Stand-Ups Harming Your Team?

Might be.  If there are better ways to do things, and you don’t at least explore them, then you might be stuck with practices that are not as useful as they could be.  In the old days, I heard a lof of “That’s the way we do things here” excuses.  More recently I see the “Cargo Cult” approach to things: “The books say this is the way to do it, so this is what we do”.  I’ve seen a lot of tricks suggested to fix problems with Daily Stand-Ups (like using a 10-lb talking-stick to keep things short) and feel that whenever you need to resort to a “trick” like that you probably should be looking for a root cause instead.

What do you think? Are Daily Stand-ups Harming Your Team?

NOTE: I have deprecated this version of this post.  I have put the new version at:

New Version of Are Daily Stand-Ups Harming Your Team?

This original post was not as clear as I had hoped, and I don’t mean for anyone to get the wrong idea. Thanks to Ron Jeffries and George Dinwiddie for their converstaions that helped me better organize my thinking and the purpose of this post. 

One little blog post cannot even begin to explore this complex subject.  Here is a start at beginning the first part. Or perhaps, a pre-beginning.

If you really want to figure this out quickly don’t wait for me to figure out how to write about this.  Just come and visit me sometime and I can quickly show you the whole thing.  It really isn’t hard.  Or, even easier, read the Agile Manifesto Values and Principles.  It’s all there.

A Few Paragraphs You Don’t Need To Read

It’s obviously not my job to tell you (teach you, show you,  guide you, coach you, whatever) how to be effective at software development.  My job is to write software, and coach and work on a team that is writing software – and to deploy that working software so people can use it.

I don’t write books, sell certifications, do trainings, or provide workshops for a living. I have nothing to sell you. [I’ve occasionally done introductory sessions about Agile stuff for a company or two.  I love doing that, but it is not my living or my job. ]

The company we work for makes irrigation products. Great company, great team, great product.  I truly love being able to come in to work every day to be with my team and do the work we do.  Point is: I make software for a living.

Estimates are NOT a deliverable

Regardless of how we define “estimate”*, it is not a deliverable in the world of software development.  I’ll write another post about that if you don’t agree .  (That’ll show you!)

The Basic “No Estimates” Approach

Here are the basics – in very simplified form told as a mini “case study”, based on a real situation from a long while back  (not my current job) – but I’ve changed the details so no one will get mad.

Players:

• Boss – you can think of Boss as the “Product Owner”, “Customer”, or “User” or some combination of these.  In any case, Boss is a “person(s)” who knows enough about the purpose of the software (or some part) to make reasonable decisions, and who is responsible and has authority for making those decisions.
• Woody – you can think of Woody as the person who guides the software development process – could be a devleoper on the team, or some other person who understands software development better than I do. We’ll call him Woody.
• Team – everyone else working on this “project”.

History:

• This was a while back, but Woody was already capable at “No Estimates Software Development”.
• This project was a new “internal” application.
• Project accessed existing databases, but also would likely need it’s own database.
• Boss had never worked with No-Estimate Software Development.
• There was a lengthy “requirements document”. It was believed that everything in the requirements doc was necessary. [Note: It was a mess, and if I had any influence I would have abandoned it].
• Team had all necessary skills and knowledge to do this work.

Approach for day one:

• Start with the huge list of way too much stuff that Boss says needs to get done (the original requirements doc, so to speak).
• Ask Boss to select one “small” thing that she felt was very important.
• That “small” thing becomes the “thing we are working on right now”: We’ll call it a “Story”, but on this job we didn’t use words like story.  We called them features.
• Team starts work on the Story.

Approach for day two and each day after that:

• Team works on the thing we are working on right now.  Nothing else.
• Team constantly assesses the work we are doing, and splits the current Story if it is not easy to understand or it is not clear how to start on it.  We check with Boss, but have no issue breaking things down.
• Team asks questions as soon as they come up, and get the answers right away.  Without this, projects drag on through endless muck.
• Team completes the Story we are working on right now (or the part of the Story that we split off)
• Team deploys the Story if it is deployable, and if it is not deployable then Team works with Boss to understand what needs to be added to it to make it deployable, does that, and then deploys.
• Team asks Boss to pick the next thing she thinks is very important.  It can be the other part of the previous story, or any other thing she thinks is important.

Result:

• First deployment happened in a little over a week.
• Project was deemed “done” in about a month and a week.
• As we started delivering usable software, the users and Boss started asking for different things than had been described in the requirements document.  The users and Boss started fine tuning what we had delievered – noting things that they wanted to be done differently.  This is a valuable and wonderful thing.  This is how requirements emerge.
• After about one quarter of the “features” described in the requirements list/document (including the additions the users and Boss had asked for), Boss decided the project was “done enough”.  The users were doing most of what they needed, and the rest wasn’t important enough to do, at least for the time being. There were other more important things to get done on another project.  The users were happy. Boss was happy. Team was happy. Woody was happy.
• Team was iterating (but not time-boxed), and incrementally delivering.

Observations:

• Boss wanted estimates, and also wanted the work done as quickly as possible.  I made the case that we could try to get some critical feature into production ASAP, and then revisit the idea of estimates later. After the first deployment, no requests were made for estimates.
• Estimates are part of the “comprehensive documentation” that we value less than “working software”.  If you don’t see that – please let me know.
• I am sure you saw the big win, right? We discovered the needed features. Most of what was thought to be needed was not actually needed.  Doing the work of writing the code in small pieces and getting it into real use made it possible to steer the work.  This is the “requirements emerge” concept.  I urge you and encourage you to think about it and experiment with it.  It is GOLDEN.
• Even if estimates were not liars, we just simply did not need them.

Well, that might be a little help.  It’s just one very basic example.  But it’s real.

This all just seems like “plain old Agile” to me.  Many people tell me I’m wrong.  What do I know?  I’d rather be wrong about what to call this and be getting a lot of work done than the other way around. Let me live in my delusions.

NOTES:

* For the purpose of this article, the sort of estimates I am discussing are the estimates typically asked for on many software development projects where a project, a feature, or a function, or a bug fix (or where a list of features or functions) are described and people are asked to come up with an approximate cost in time, money, or effort to do the work that will be required to provide the feature(s)/function(s)/capability(ies)/bug fix(es) being requested.

Disclaimer: There are many situations where estimates can be meaningful and useful.  This article is about situations where I don’t think they are typically meaningful or useful, and only in the realm of software development.

Dynamics of Software Development

Jim McCarthy wrote a book called “Dynamics of Software Development”.  If you don’t own it, you should.

I believe it was published in 1995, and I think I read it in 1996. There is little reason for you to do anything else until you get this book and read it.

He has an updated version published in 2006.  The 2006 version includes a CD of Jim’s 23 1/2 Rules of Thumb (For Shipping Great Software On Time.)

You can find this book used for about 1 cent for the 1995 version, and about 2 dollars for the 2006 version.  You have to pay shipping – but still, for 5 or 6 bucks you can get a lot good, old fashioned, unconventional wisdom.  I paid full price for both, and it was well worth it.

If you think the stuff I say is worthwhile, you just might like this book.  If you hate the stuff I say, then this book will give you even more great stuff to hate.

Even without taking into consideration how old this book is (almost 20 years! ) most of his insights are extemely relevent and useful (and downright powerful) if you are doing software development.  A few concepts might need to be updated for changes in technology, and some of it has been surpased by modern Agile thinkging (in my opinion), but overall it’s valuable reading.

Also, he has a set of clips from the  23 1/2 Rules of Thumb (For Shipping Great Software On Time) presentation at his site:  http://www.mccarthyshow.com/the-23-rules-of-thumb/

There you go.  Free advice you can use today.

The Agile Manifesto – Values and Principles are the foundation.

My Maxims are not meant to distract from or replace them.  My Maxims are just another way to for me to frame my thinking.

I spend a lot of time scrutinizing the things I value to make sure I can minimize my blind spots and unknowns, at least the ones that make the most difference. Seems I am usually my own worst enemy.

I share my Maxims so I can get feedback from you and learn where I can improve.  Please be kind.

Agile Open So Cal 2012 at UC Irvine.

Once again I have been able to participate in the Agile Open So Cal at UC Irvine.  Great fun, great people, great sessions, great etc.   Several people had seen me tweet these “Maxims” or heard me talk about them before, and asked me to present them again.  It takes VERY LITTLE URGING to get me to talk about Agile, Lean, Clean Code, or anything programming.  So, I proposed a session –  and here are the notes:

Title: The 8 Agile Maxims of Software Development

Byline: This time it’s personal.  With Woody, it’s always personal.

Here they be (in no particular order – you can mix and match):

1- It is in the doing of the work that we discover the work that we must do. Doing exposes reality.

I live this daily.  Thinking about stuff is obviously worthwhile – I don’t discount that. But doing is way more important.

2 – “Responding to Change” is impossible unless code is easy to change, easy to maintain, easy to fix, easy to enhance, easy to read, and easy to discard.

The “easy qualities” –  I learned them from the greatest programmer I have ever worked with: Fred Zuill, my little brother. Back in the 90’s he used to do a talk on the Qualities of Software that was pithy, meaningful, and wickedly sardonic.  If you ever get a chance to hear him speak, do it.  If you see him please remind him he owes me $18.

3 – Question Everything – put special focus on our deepest held beliefs. What we trust most hides our biggest problems.

I’m pretty good at getting comfortable in my ways.  Gotta work at keeping that from blocking improvement.  When I really believe something, I’m likely to be fooling myself.  Lets keep things uncomfortably wonky.

4 – “Working Software” is software that users are actually using. Until it’s in use it is truly useless.

This is my understanding of how “Working Software” should be thought of (as in the Agile Value of “Working Software over Comprehensive Documentation”).

Let’s not fool ourselves: “Potentially Deliverable” is a lot like “The check is in the mail”.

5 – Stress at work diminishes value. Crunch-time is a symptom of harmful and counter-productive attitudes.

Nuff said?  I hope so.

6 – We are the innovators of our process. Learn what works for others, prove it for our self, innovate beyond.

Just a suggestion: Don’t wait until you are an “expert” to innovate.  Just like Jello, there is always room for innovation.  (You remember those ads for Jello, don’t you?  Dang, you young people really missed out on the best days of television. You remember television, don’t you? Dang… I’m getting old, so it seems)

7 – The object isn’t to make great software, it’s to be in that wonderful state which makes great software inevitable – Robert Henri, paraphrased

This is a paraphrase of the well known quote from Robert Henri.  Just replace “great software” with the word “art” and you get the original (and much more meaningful) quote.  I replay this one over and over in my head all the time.  Wish I was the one who had said it! I was introduced to this quote many years ago by Donald Faast, an amazing show-card writer and sign man. I think he is in Colorado now.  If you see him, just say thanks for me if you would.

8 – The more we work at the work we do, the less capable we become -Repenning/Sterman – Make time for improving capability

Dang.  If you haven’t already read the paper “Nobody Ever Gets Credit for Fixing Problems that Never Happened: Creating and Sustaining Process Improvement” by Repenning and Sterman then please click on it and read that now: http://bit.ly/Qd3NmR – It’s a pdf file.

9 – I reserve the right to add, remove, change, improve. (This one has been added since the session).

I expanded on these greatly during the session, and invited feedback, observations, and discussion.  Unfortunately I was not able to take notes.  Overall, I feel it was well accepted and if it was not, my memory has already painted it much better than it actually was.  Thank you memory!

Remaining Puzzles, Recommendations, Next Steps:

I pointed out that these Maxims will change and grow, and invited the participants to add, remove, change, improve

Also… A BIG thanks to Drew LeSueur – @drewlesueur – from Integrum Tech

Drew has been very encouraging to me, and on his own he quickly put up a site and posted the Agile Maxims: http://agilemaxims.org/ –   Remember, this whole thing is just me thinking out-loud.

[NOTE: To make sure people are paying attention, I always purposefully put a hidden typo in my posts… see if you can find it.]

Below are my 8 Agile Maxims.

They are not in any particular order, although my favorite is probably the paraphrase from Rober Henri.

You’ll note that there are now Nine… the ninth one simple is a disclaimer that this is bound to grow, shrink, or otherwise change over time.

• It is in the doing of the work that we discover the work that we must do. Doing exposes reality.
• “Responding to Change” is impossible unless code is easy to change, easy to maintain, easy to fix, and easy to enhance.
• Question Everything – put special focus on our deepest held beliefs. What we trust most hides our biggest problems.
• “Working Software” is software that users are actually using. Until it’s in use it is truly useless.
• Stress at work diminishes value. Crunch-time is a symptom of harmful and counter-productive attitudes.
• We are the innovators of our process. Learn what works for others, prove it for our self, innovate beyond.
• The object isn’t to make great software, it’s to be in that wonderful state which makes great software inevitable – Henri
• The more we work at the work we do, the less capable we become -Repenning/Sterman – Make time for improving capability
• The unspoken Agile Maxim – I reserve the right to add, remove, change, improve.

I love Document Driven Development

One thing I really miss in Agile is having long, incomplete, boilerplate documents for me to “sign-off”.   I like to write code and deliver working software quickly, but even more I LOVE to read documents and like nothing better than to spend long days before a glowing flat-screen reading incomprehensible nonsense carefully crafted to appear complete and correct while still being essentially content-less.  Doesn’t matter what they are about, as long as they have small type and lots of words.  Of course, nothing is further from the truth.  If I want to read nonsense then I’ll read Edward Lear.  Or my own writings.

So documents are fine, but what about Sign-offs?

Even more, I love to take responsibility for documents that I did not write.  It gives me the feeling that I am a grown-up.  But apparently, I am very unsophisticated regarding sign-offs.  Or I was until recently…

I had one CEO show me the policy for documents at his organization (let’s call it “Anonymous Inc”) where there were 3 sign-off statements for a manager to sign on any document requiring her approval.  Oh joy!!!  It is in the spirit of Extreme Documents, I suppose.  If one signature is good, why not dial it up and try three?  More is always better.  In his defense, he was showing this to me as an example of something he wants to abolish – he understood the absurdity of it.

I approve this document

This typical sign-off is for the manager-type to “approve” the document.  In software development this often indicates that others can rely on this document as being “okay to use for doing whatever it is they do next based on this document”, and that it is Complete and Correct.  This is actually the third signature to apply after the next two required signatures have been taken care of.

I have read this document

At Anonymous Inc, besides approving the document you are also required to sign-off that you have read the document.  You’d think that approving something implies that you read it.  Guess what?  Apparently this isn’t as common as you might think, at least not at Anonymous Inc.  I suppose the “I have read this” signature solves the problem of someone later on saying “I just signed it, but I didn’t read it – if I knew what was in it I would not have signed it”.   I suspect that a lot of documents get “approved” without the approver having actually read it, and VP’s and upper management can always say “I was so busy I had my underling read it – he said is was okay so I signed it.  Let’s fire him, okay?”

I understand this document

Okay – “I approve this document” might make sense in some organizations.  “I have read it” at least keeps some people a bit more honest, I suppose – I’m not sure it would.  But “I understand it” is something I just do not understand.  Still, it is a very fun idea, and Anonymous Inc is all about fun.  Still, just because I say “I understand” doesn’t mean I do.  Any parent knows that.  “I understand” can also mean “shut up already”. But then, what do I know?

Have fun out there, and be careful when you sign-off.

Estimating Software Development

Anyone who knows me will know that the title of this post is meant as a bit of a joke.   For software development, it is my experience that estimating** is rarely useful and always easy.

Well… a little clarification:  estimating is NEVER useful but ALWAYS easy the way I have typically seen it done, which uses the WAG* method of providing some numbers that seem plausible and that will allow the decision makers (stakeholders and managers) to approve the start or continuance of a project with a clear conscience. That is, when things sour later on they can always blame the developers for giving them inaccurate estimates (and not living up to their “commitments”).

And the developers can always blame the “requirements people” for giving them unclear, incorrect, or incomplete requirements.

And the requirements people can always blame the stakeholders for providing bad direction about what was important and the developers for not “understanding the requirements”, and so on.

Regardless of who is part of this process, it is one big happy circle of blame that lets us all do the wrong thing and still keep our jobs (not always happy for everyone, actually, and sometimes not everyone will keep their job).

I have simplified it here, but the basic idea is sound.  As long as everyone is good at deflecting blame, and everyone is willing to continue on saying we’ll do it better “next-time” and after only a few people have been fired then everything is fine, right? Well, I don’t think it’s fine, of course – but many organizations seem to operate this way.

Let’s estimate a game of chess

Chess is a game that some of you may have seen.  It provides a very limited environment with only 6 different and charming pieces that have very limited ways in which they can be used, and there are only two players.  Simple. The whole game can be contained in an area of about a square foot or even less, and the complete rules can be written on the inside of the box lid.   Just like Candy Land, it is very easy to learn and play. Not so easy to win, perhaps – but that makes it interesting enough to be an engaging game for some.

So, here is a description of a simple “project” and I need your estimate:

Win a game of chess. Losing is not an option. Tying is not an option.  We will only make money if you win.  If you don’t win, the company is at risk, and you will lose your job (and mine).

Also, I have a diagram of a chess game with the exact position of the pieces as required at the end of the game.  To make money, it needs to end up just like the diagram.

Oh, one other thing: we don’t know yet who you will playing, but we’re pretty certain it will be someone who knows how to play the game, and might be pretty good at it.

That’s it!  There are only a few requirements:  That you win, and that the board matches the diagram I’ll give you.  Easy!

What I need to know is exactly how much this will cost me.  In this situation, cost is the number of moves you’ll need to meet the requirements.  Oh, also, we don’t have the diagram yet, but we’ll have it soon.  Still, you should be able to estimate it because it is basically the normal chess end-of-the-game arrangement with pieces here and there and so on, and you have the opponent in a checkmate.

Should be easy to estimate that, right?

Well, I hope you agree this is impossibly difficult, or actually: just plain impossible to do a meaningful estimate.  First of all, there is not enough information to do a good estimate, and even if we had all the information, there are many variables and a lot of stuff we have no control over. How close do you think your end results would come to the requirements?  Would you be willing to stake your job on making a decent estimate on a chess game in this situation?  Well, as long as you could blame someone else and promise to “all do better next time” you’ll probably be okay.  Or you could be a consultant and be paid and off to your next contract before someone actually has to play this game you helped them estimate.

Getting better at estimating is like pulling teeth

I hear it over and over:  Our estimates were not accurate and we had a lot of trouble because of that.  We need to get better at estimating.

It is similar in a way to saying “My tooth hurt, so we pulled it out.  Now I can’t chew as well as I used to.  We need to get better at pulling teeth”.  What is the real problem?  Shouldn’t we work on that instead?  Hint: Getting better at pulling teeth is not it.

So what would you do to get better at estimating the game of chess, based on the basic scenario we described above?   Could you get better at it?   How good would you have to be before it had real value? Is it worth spending time on getting better at it?  What is the real problem.  Hint: Getting better at estimates is not a solution to anything because having bad estimates it not the real problem.

My suggestion: Stop That!

I haven’t seen much value in estimates.  Actually, in the software development work I have done, I can’t recall any situation where estimates** of this sort were of value to the actual job at hand.  Someone wanted them, someone saw value in them, and sadly – important decisions were made based on them.  Do you think those were good decisions?  People often tell me that “even though we know the estimates are inaccurate, they are better than nothing”.  I prefer nothing, if there are better ways.  There are better ways.

Almost everyone tells me that estimates** are very important, but estimates are often a blind spot hiding something of actual importance and value.  This almost universal acceptance that estimates are important and useful is harmful.  A predictive approach such as waterfall itself depends on estimates – and estimates are how we make the predictions.  Don’t think “waterfall” and predictions are harmful?  You probably should be reading some other blog.

Estimates** of this sort are based on guesses about the time needed to do work we don’t yet understand.  Nothing about this gives me confidence that these estimates have value.

It seems I always make enemies when I suggest this, but that is not my intention.  I want to do effective work on meaningful stuff – that is my intention.  I don’t want to be spending time on work that has no value, causes misdirection and waste, and potentially destroys any chance of being effective.  That is painful.  There are MUCH BETTER WAYS to approach software development.

Do you need estimates?

It could be useful for you and your organization to question your use of estimates and explore new ways to think about your “need” for estimates.

One last thing:  Do not ask me for an estimate on how long it will take to eliminate estimates from your process.  It is too much like a game of chess, only very complicated.

Here is a start at clearing up our thinking.  Let’s do a little thought experiment.

Which would better help if we have decided to eliminate estimates and find a better way in our organization? :

• Get rid of all developers
• Get rid of all upper management.
• Get rid of all upper management and developers
• Something else more realistic

Who is most insistent that estimates are important?  Who do those people answer to? Are they going to fire you for questioning things like estimates? Are they open to scrutinizing the status quo or to examining why we do the things we do?

So what might be better?

What can we do instead of doing estimates?

Nothing.  Don’t do estimates.  Simple, clean, easy.  But that is not the question we should be asking, and that is not what I am suggesting.

The correct question is: since we think that estimates are important, and you say that they are not, how can we develop software without doing estimates?

The answer is… Well, I suspect no one reads this blog, so that doesn’t really matter.  This is just a clearing house for my thinking and has nothing to do with the reader.  However, I am sure to write some stuff someday about ways to get things done without estimates, and I guess some of my earlier posts cover some of this thinking. If someone is actually interested and asks for me to cover this, I might get to it sooner.  But I am sure you can figure it out without me – this is truly simple stuff.  Read the Agile Manifesto and Agile Principles, and use them to invent better ways.

Another hint: Get good at frequent incremental delivery of useful software.  That might be worth considering.  That takes a LOT of experience and open thinking.

NOTES:

* WAG = Wild Ass Guess.  A Wild Ass is a beast that is indigenous to parts of Africa and Asia.  A Guess is a guess.  Wild asses are about as good at making software estimates as anyone else.

** For the purpose of this article, the sort of estimates I am discussing are the estimates typically asked for on many projects where a list of features or functions are described and people are asked to come up with an approximate cost in time (working time or elapsed time) [sometimes it is phrased as “cost”, or “effort”] to do the work that will be required to provide the feature(s)/function(s)/capability(ies) being requested.

Disclaimer: There are many situations where estimates can be meaningful and useful.  This article is about situations where I don’t think they are meaningful or useful.