Life, Liberty, and the Pursuit of Agility

Lately I’ve seen/heard a lot of discussion about deadlines – and it seems that they are often misunderstood.  Fortunately, I am often misunderstood, and I misunderstand most things… so it seems perfect I should write a blog post about this.

I Worked As A Flyboy At A Newspaper In My Teens – [you can skip this part]

You might have heard of newspapers.  They used to have these things where you could read news that was printed on paper – thus “newspaper”.

In my teens (in the late 1960’s) I had a night job as a “Flyboy” and printer’s assistant for a small newspaper printer who had their own web press (a web press is a huge machine where you put in 1-ton rolls of paper at one end and output printed and folded newspapers at the other end).

What’s a flyboy? It is a boy who stands at the “output” end of the press and picks up an arm-full of newspapers as they come off the press, quickly “jogs” them into a neat stack, and bundles and ties them on a bundler/tier.  “Fly” because the papers “fly off the press”, “Boy” because no reasonable adult would take this job unless they had no other choice.

A Few Things About This Job  – [you can skip this part]

• It’s LOUD: you will soon loose your hearing.
• It’s DANGEROUS: the various parts of the machines (cutter, folder, bundler/tier, etc) are all capable of removing parts of your body, mostly fingers or hands.
• It’s DIRTY: the ink is still drying as you handle the papers. You get completely covered in toxic pigments that you have to scrub off with a scrub-brush at the end of a shift, and you’re breathing paper dust and toxic fumes the whole time.
• It’s a NIGHT JOB: Or, very often it’s a night job because the papers go out in the morning.
• It’s HARD: It takes a great deal of constant physical exertion so you are guaranteed to be worn out at the end of the shift.
• In other words, at the end of the day you are tired, dirty, injured, and nearly deaf – much like computer programming.
• You don’t want me to tell you the bad things about this job.
• This has nothing to do with the point of this story.

Besides being a flyboy and a printer’s assistant, I also would do typesetting for some of the less important content, and operate smaller presses to print advertising flyers that get “stuffed” into the papers at some point, and help run the cameras and to do various layout, stripping, and other tasks.  Most of this stuff is now considered a “dying trade”. And good riddance, I suppose, but I loved it – it seems I always loved books, printing, machines, noise, lettering, typesetting, doing stuff, craft, high skills, hand work, hard work, and late night radio.  Especially the shipping reports from England that come on at 2:00 am PST.

Point is:

The point is I don’t know a lot about the whole process except what I learned when I worked in this environment for about a year, and I think I remember just enough to make my point (hopefully).

Getting the newspaper ready for printing each day requires a lot of specific things get done ON TIME

All of the following list of work must get done on a very tight schedule every day:

• Before you can print you must make the printing plates.
• Before you make the plates you must do the layout, camera work (separations, halftones, etc), and stripping
• Before you do the layout, camera work, and stripping you must do the typesetting.
• Before you can typeset you must have edited copy
• Before you can edit copy you have to decide that a story merits getting into the newspaper
• Before you can decide which stories merit getting into the newspaper someone must write those stories (or at least the headline and some copy) and take photos
• Before you can write stories and take photos the reporters and photographers have to go to the scene of the story and gather news, do interviews, etc.
• Before you can write a story you have to decide what to write the story about – a “news events”.
• Often the “news events” happen throughout the day and we don’t know what they might be until they happen. Man Bites Dog. Nuclear Annihilation Avoided – Again, Major Caught In Drug Scandal – Again, and so on.
• And so on and so on.

There are a LOT of deadlines in there – and they ALL HAVE MEANING.  As a matter of fact, I’ve left out a lot of details just to keep this post small enough to fit on the Internet.  Also, a number of these things happen “in parallel” – a story can be “gathered”, written about, and edited while other stories are at different stages of “completion” and still all get into the same daily paper.

So I am going to talk about the classic “Printers Deadline”.

There is a HARD DEADLINE in Newspaper Printing

At least where I worked, we had a hard deadline.  To get the paper out to the “marketplace”, the trucks must leave sufficiently early before traffic starts so the papers can be where they need to be when people want to buy them.  That’s the business. So – the hard deadline is: The press must start early enough to start putting newspapers into trucks so they can leave on time.  It is a complex dance with a lot of variables.

To do this, each step along the way has a specific deadline: It’s the time all items for this step MUST BE DONE with enough time to do all the following steps and still be on the press when printing starts.

The bottom line: if a story isn’t ready and “on the press” when the press starts printing, it is NOT going to be in the newspaper (there are ways around this, but that’s the general idea).  That is a hard deadline.  A lot of things can be easily changed right up until that minute.  After that, it is costly in both time and money to make changes.

Sometimes someone will yell “Stop the Presses”, like you see in the movies – but that is rare.  There are occasional “Dewey Defeats Truman” mess-ups.

My point: Real Deadlines Exist

Real deadlines exist. When they do exist, we need a system or method in place to deal with them so we have a reasonable chance for success.  In the newspaper business they have the deadline thing figured out.  The deadline is obvious. Some people say this is where the term “deadline” was invented.  By the way, the newspaper industry has bigger problems than deadlines nowadays, or maybe it’s a different kind of “deadline”, and we can talk about that some other day.

How Newspapers Deal With So Many Deadlines

There are a number of things the newspaper publishers have found work nicely to allow for these serious deadlines.

• Create news features (stories) daily – work in daily iterations – each day, the current “most important” content is created
• Deploy daily – a whole year worth of news is delivered in small, useful, daily chunks
• Decisions are made at the last responsible moment – if a more important story comes in at the “last minute”, a less important story can be removed.
• If insufficient material is generated for a story for it to be interesting or useful, they can quit working on it at any time.
• Whatever is done by the daily deadline is what gets delivered.
• There is no “Yearly” deadline – it’s more like a continuous delivery system.
• They discover processes that allow them to work quickly – e.g.: you can buy some stories from a third party (such as… the AP)
• They can “spice things up” as needed to boost sales.  That is not really important to the point of this article.
• They ca add lines and cut lines from a story to fill the space as needed.
• If a really important story comes in, several people can work on it at the same time.
• Stories that can be worked on, are worked on.  Stories not ready to be worked on aren’t.
• Fillers and other non-news “content” can be done and ready to add as needed without effecting the output of more meaningful content.  For example, a whole weeks worth of comics can be ready each Monday.
• Reporters and photographers can be multi-purpose (they can wear more than one hat).  They can interview someone, take a picture, write a story.  Some can even do the typesetting in smaller houses.
• They only need enough good (or really, “bad”) news material to keep people happy.

News items have different schedules and different deadlines based on the nature of the content – a garden page item might take weeks to write, photograph, illustrate, layout, etc – and a hard news item often has at most about a day and sometimes as little as an hour to go from “this just in” to “press-ready”.  Almost any feature can be edited to reduce its size, increase its size, or rearranged to fit the space on a page.

Sometimes they “stop the presses” and re-set one of the plates to change or add content.  Also sometimes a “special edition” is needed to disseminate late breaking but important news.  Sometimes a flysheet can be printed on another press to add content at the last minute.  Lots of options.

In other words – there are many practices they’ve innovated over the years to be able meet those deadlines and do the job of delivering a sufficiently interesting and sell-able product.

Sometimes Deadlines Are Needed, and We Need Ways To Handle That

Well – this blog is about software development, not the newspaper business.  However, I think there are a few similarities.  What do you think?  You probably see other similarities, and a few differences between the newspaper business and computer programming.  Still, the newspaper model might have some use.

Do you have deadlines??? Learn how to deal with it.

This is a post about a trivial thing that I can estimate, and there is a non-trivial point I want to make at the end.  So you can just skip-to-the-end if that is the sort of person you are.

I can estimate how long it will take me to drive to work.

I drive to work 5 days a week.   I need to get to my workplace, but I also want to pick up my daily oatmeal and drink for breakfast at a fast-food drive-thru.  Simple stuff.

Why do I need an estimate?

I need an estimate so I can be sure to leave home early enough to get to work on time.

I like to get to work 15 minutes early, and I’d like to make sure that I am never late except in exceptional cases.  So if I can estimate a time that will get me there about 15 minutes early, but never later than 5 minutes early that would work for me.  I don’t mind if I am 20 minutes early, but I don’t really want to be 25 minutes early.  So, I’m looking for a time range that gets me to work between 7:40 and 7:55, with the ideal spot being 7:45.

I want to be on time to work, with sufficient time to not worry about being late, and still not waste too much time just sitting there tweeting about estimates.

I drive pretty much the same route every day with about 10 minor alternatives available. I can take advantage of these alternatives (turn right here, or turn right a block up, stop at this fast-food or at that fast-food, etc) to make allowances for congestion at traffic lights, schools (school days are busy, holidays are not), drive-thru lines, and stuff like that – so I make a lot of little decisions on the way. For example: I’ll turn right at a light if there are a lot of cars backed up and take a chance that the next street isn’t backed up.  That allows me adjust to get a better chance at a good drive time.

I can estimate how long this will take, and use that estimate most of the time.

Yesterday it took 40 minutes, the day before 43 minutes, the day before that 36 minutes (school holiday).  Over the last year, I’ve found that it takes about 40 minutes to drive to work on a typical day.

If I leave at 7:05, I will almost always get to work in my target time-frame.

All well and good.  And that works great for me for typical days.

I can take advantage of daily changes

I “re-estimate” daily to adjust for school holidays, weather, etc. and vary the time I leave for work to compensate.

To clarify: I re-estimate every day to adjust for daily differences.  I can “re-estimate” quickly.  The cost is about 10 seconds of my time.

On a school holiday, I can sleep in 5 minutes. If it is raining, I must leave 10 minutes early.  If it is snowing, something is really wrong because it never snows here.

Why Does This Work?

You already know:

• I have a lot of data – way more than is needed.
• I know all the variables (weather/holiday/etc)
• I know the work-arounds (alternative routes)
• It’s essentially the same thing every day
• There are almost no unknowns
• There is almost nothing new to learn
• I can trust the guy who is providing the numbers.  It’s me.

That is, it is trivial.

By the way –  I can’t control the variables to predict an exact time, so a range is sufficient.

Having the estimate is useful as it helps me optimize my sleep time and still reach work on time.

This is the Non Trivial Part: How Similar Is This To Computer Programming?

So, if it is useful to have an estimate, and we can make a meaningful estimate – then we have a good application of estimates.  That is the case with my “drive to work” example.

In my humble opinion, this is not similar to computer programming. We have the first part: We feel estimates would be useful, but we don’t have the second part – the estimates are not meaningful unless you are doing very trivial work.

With software development we don’t have much data, we don’t know the variables, we don’t know the work-arounds, each project is very different, there are lots of unknowns, there is much to learn, and I can’t trust anyone for “good numbers”, not even ME.  By the time we know enough about what we are doing to make useful estimates they are useless because we are DONE.

When we find our software development work is “easy to estimate” it is likely that it is trivial in nature.  That is, if it matches the nature of my “drive to work” example then I would question if that work is actually software development. In software development, when I find I am doing the same thing over and over, know all the variables, have few unknowns, have little to learn – then that’s the time I investigate ways to automate it and stop “programming” it.

Have fun.

And remember: This is ALL JUST MY OPINION.  Don’t do anything I say.  Don’t mention to anyone else that this makes sense to you, they will just say you are crazy.

Some other posts I’ve written on the topic of estimating:

• Estimation is Easy and Useful: Estimate a game of Chess
• Do Estimates Do What We Want Them To Do?
• No Estimate Programming Series – Intro Post
• No Estimate Approach For End-Of-Life Legacy Support
• Notes From A Conference Session On No Estimates in Software Development
• A Comment And Response from Estimate Chess Post
• Can We Code Without Estimates?

Can we write code, create software applications, without doing estimates*?  I think there are at least some cases where estimates are not needed.

In The Beginning

I bought a computer in 1982 or 83 for the purpose of managing aspects of a business I had.  My business was essentially a small custom manufacturing business of sorts, and I had very little money.  No capital, just cash flow. I would sell some work, get a deposit, buy the materials and supplies for the work, do the work, get the balance on delivery, pay some bills, pay the rent, and go on to the next job.  Every hour I had was needed to earn money,  or support the overhead of doing business.  Additionally, I would spend some time improving my capability to more effectively make money, reduce costs, or “take care of business”. Payback for that time needed to be quick, not in two weeks, but in two days, or two hours.  Every penny counted.

So I now had a computer, but none of the software I bought worked well enough to help me.  I wanted to do simple stuff – like mailing lists, simple bookkeeping, making estimates for manufacturing jobs, writing invoices –  stuff like that.  I also wanted to do some very specialized things, like controlling devices: material cutting plotters, measurement devices, and imaging equipment.  But software was expensive, and it was mostly useless in those days.

Computers can help us do bookkeeping, right?

One important reason I wanted a computer was to keep my books. I wanted to spend less time doing my bookkeeping.  I had bought an application that was supposed to do simple bookkeeping: Keep track of money spent, allocate it to varioius accounts, keep track of the checkbook, reconcile the checkbook, do income and expense reports,  profit and loss reports, year end stuff, and so on.  Trouble was – it didn’t work well at all.  It was difficult to use, made mistakes, and the data got corrupted after a couple months of using the application.  It was truly useless and worse: it wasted a lot of my time.  I tried several other applications that friends of mine were using – but none of them made things easier than doing the work by hand.  I wanted my computer and software to save time, and make it easier for me to focus on getting and doing work.  It wasn’t able to do that.

Earn as you Learn – Or At Least Try

So I decided to try writing my own “bookkeeping application”.  It’s just numbers, right?  Computers work with numbers, right?  Shouldn’t be too hard.  Before we go any further, I’ll “skip to the end” and let you know: It was not hard at all.  It turns out that computers ARE good at working with numbers, and you just have to find a reasonable way to let users interface with the computer, and some way to show stuff on a screen, and some way to send stuff to a printer, and some way to save data to a disk so you can use it the next time you want to work with it, and some way to do the calculations and other work actions.  These are all easy things.

I had no experience writing applications at that time beyond some games and little text editing apps and similar things that came from books I was learning from.  For my first project I needed something that would return back the time I spent right away.  Seemed the checkbook part of the bookkeeping app would be the place to start – you need to put expenses and income into the system before you can do much else, and it would save me some of the time it took to do that work by hand.

A Simple Checkbook Application – Find A Small Bit Of Potential Value

Balancing a checkbook is easy enough, it took me about 2 hours a week to do the hand entry of checks and deposits into the paper system my accountant had set up for me, and to update the balance.  It also took about 3 or 4 hours once a month to reconcile the checkbook.  I had always made mistakes in entry or calculation, so sometimes it would take longer.  I was spending about 11 to 14 hours (or more) each month doing the paperwork for my checkbook.  There was no Quicken in those days.

If I could write something that would allow me to enter a check into the system, enter a deposit into the system, reconcile the registry, catch mistakes, correct mistakes, and print out the details – that would save me 11 to 14 hours every month.  Well… not really: It would take some of my time to do the entry of checks and deposits, and to review the output, and correct mistakes.  So there is the value: A time savings of perhaps 8 hours a month.

I had NO IDEA how much time it would take to write that software.  I had nothing to base it on. It seemed worth a shot, but I had no way of knowing that.  The only sure things was that I’d learn a bit more about programming.  The key is: I had identified a small chunk of work with POTENTIAL value.

Learn By Doing, Decide Based on Real Time, Real Value

So, I had a vague idea about value.  I figured I could put in about an hour each night and I would soon learn if it was worth trying to write this code.  In doing the work I would learn the things I needed to learn to do the work, and I could make decisions about the value being delivered.  At any point I could decide to cancel the project or keep going.  The decision would be based on value delivered for time invested.  NOT value that MIGHT be delivered for time that MIGHT be needed.  That seems like a bad way to make decisions to me, at least in some cases. Real value, real time.  Take action, make decision, take action, make decision.  A small amount of real work delivers enough value upon which to make a decision.  Deliver early, then decide: Should we do more, or try again, or change directions, or expand or reduce… whatever.  Baby steps every step of the way.

The First Bit Of Functionality: Data Entry

The first value I could see: Ability to enter in checks (number, payee, date, amount, etc.), save as a file, and print it out on paper.  This would be exactly what I was currently doing with pen and paper, and I could do the totals by hand with a calculator just like I had been doing.  So that is where I started. Not much value, but provided a way to “amplify learning”.

Now, it seemed that this “functionality” could be “written” without writing any code.  The user (me) could just use the existing text editor to type in the data in a specific format.  How long did it take to code that?  No code, no time.  I just needed to define a format.  How robust was that?  Not very! It would be easy to make mistakes in data entry – but I needed to get real functionality quickly.  Same old same old.  That’s software development, so I’ve learned.  Once the code is in use the most painful stuff will be revealed, and we can fix that first.  Might never need to do anything more than the minimum.  Try to keep it that way.

Writing to a file was easy enough.  What to do next?

I wanted to calculate the balance of the checkbook. I would be very close to having something truly useful if I could do that. That would require that I take into consideration deposits as well. That was easy to do, of course. Deposits are positive, checks and withdrawals were negative.  Or something like that.

Again, I had only an hour a night or so to spend on this – so everything was very “bare bones”, but reading the file, reading the records, and accumulating a total was easy enough.  And essentially, the application was ready for production.  It was useful. Lame, but useful.  Value was delivered, and it showed the project was viable.  There were a number of features to add before it would be “easy to use”. I still often made data entry mistakes, but I could fix those by simply opening the file with the text editor and making the needed change to the data. Over time, I added much of that functionality, and eventually it became my bookkeeping application.  The development was paid for by the time I saved every month.

So, that was the start – No Estimates

I don’t think I did any estimates in this work.  If you think I did, please let me know and I’ll clarify the process.  Or maybe I was wrong!

The process for me: Choose something of potential value, find the simplest thing I can think of, and try to do it.  If it works, I go on to the next bit that seems to have value, if it doesn’t work I try something else. Repeat until the payback doesn’t seem worth it anymore – at which point I can move to another “project”.

This is a very minimal example. It’s real, and it is where I started.  I used this same basic pattern to write a lot of software for myself over the next 15 years.  I never had a need to estimate anything.  I didn’t even think of making estimates.   Someone once told me that necessity is the mother of invention.  I’m not sure about that, but I just haven’t had enough time to waste on inventing things that I don’t need.

I’ve used this same apporach on much larger projects, in various situations, and with a number of different projects types.  I think some form of this approach can work for just about any project.

Please don’t yell at me

Before you start yelling, and before the veins start popping out on your head, please understand: in this post I AM NOT SAYING THAT ESTIMATES ARE NOT NECESSARY FOR MANAGING SOFTWARE  DEVELOPMENT in your situation.  I am simply saying that estimates are not needed for designing, coding, testing, and delivering software into use.

The process you implement for managing the business of making software might require that you do estimates.   But that necessity has nothing to do with writing software – that necessity is imposed by people who haven’t yet invented a better way to manage their software development, their customer interactions, their collaborative abilities, etc.  Maybe you can find a better way.  Maybe there isn’t a better way.  Regardless, if there IS a better way, maybe you can take a tiny step every day that moves you to a better process that does not require that you do estimates.  That’s up to you.

Can We Code Without Estimates?

Still, I hope you can agree that we can write code without estimates.  If you don’t think we can, I’d like to hear about it – especially if I am wrong or am missing something.

Some other posts I’ve written on the topic of estimating:

• Estimation is Easy and Useful: Estimate a game of Chess
• Do Estimates Do What We Want Them To Do?
• No Estimate Programming Series – Intro Post
• No Estimate Approach For End-Of-Life Legacy Support
• Notes From A Conference Session On No Estimates in Software Development
• A Comment And Response from Estimate Chess Post

NOTES:

* For the purpose of this article, the sort of estimates I am discussing are the estimates typically asked for on many software development projects where a project, a feature, or a function, or a bug fix (or where a list of features or functions) are described and people are asked to come up with an approximate cost in time, money, or effort to do the work that will be required to provide the feature(s)/function(s)/capability(ies)/bug fix(es) being requested.

Disclaimer: There are many situations where estimates can be meaningful and useful.  This article is about situations where I don’t think they are typically meaningful or useful, and only in the realm of software development.

Cory House ( blog  , twitter) wrote a comment on my Estimate Chess post, and I’d like to double purpose it by turning it into a post.

First – thanks for reading my post, Cory, and for making comments.  I appreciate that very much!!!

Cory’s Comment

I can appreciate the feeling of fruitlessness in some estimates. It’s hard to get right, particularly when requirements are weak. However, I believe a professional developer must be willing to take the risk and do the hard work of estimating when requested. Drawing a hard line of “no estimating” without considering context doesn’t come across professional or accommodating.

Your chess example is entertaining, but it’s also a great example of a place where I feel it’s unfair to expect an estimate. Given, it’s gray exactly how your story would translate to the development world, but it seems to describe a request to estimate a project without even vaguely reasonable requirements. I think we all agree that’s a waste of time.

If given sufficient requirements, we, as professionals, should be able to estimate our work. The world expects no less from engineers in other fields. Multiple contractors recently estimated the cost to finish my basement despite the fact that I had no floor plan, no idea where lights would be, only a vague idea on what word work I’d like, a bathroom of unknown size and quality, etc. If one of them refused to provide an estimate, I’d have gone elsewhere.

I land plenty of dev work on contract because I’ve gotten good at asking the right questions and sniffing out complexity so that I can provide reasonable estimates. Being skilled in estimation sets you apart whether you’re independent or full-time. Steve McConnell wrote an excellent book on the subject. http://www.amazon.com/Software-Estimation-Demystifying-Practices-Microsoft/dp/0735605351   That said, while I disagree with your answer to “What can we do?” as “Nothing.”, I enjoyed reading your blog. It’s fun reading and certainly food for thought.

My Response:

Hello Cory! Thanks for commenting on my post!

Well, as you can guess, I don’t agree with your take on things – but that’s why I have a blog, and you have a blog. We all get to write about things as we see them.

Regarding professionalism: I believe it is professional to find a better way to do all the things we do, and eliminate things that are wasteful. I see estimates and the estimating process as mostly waste, and as a professional I see no need to “take on the hard work of estimating when requested”. More directly: estimating is not hard to do, in my opinion. I’m not clear on why you might think it is. However, I also believe that estimates as used in software development typically (and almost always) result in misinformation that is used to make bad decisions. Which is why I follow an Agile approach to development. The Agile approach, at least as I do it, provides a lot of value and eliminates a lot of waste. And that is very professional. We’ll just have to agree to disagree on our thinking about professionalism and estimates.

I don’t think I made it clear in the post: The Chess analogy is to show that something as simple as Chess becomes complicated quickly. It is a VERY SIMPLE GAME, with very few variables, and yet it becomes too difficult to predict when we add a few simple constraints. And even without those constraints, there is little we can predict. Otherwise we wouldn’t even play the game, right? Remember: Don’t stretch an analogy too far – it will always break. With software projects, anything that is non-trivial is much more complicated than the simple chess example.

Cory: “If given sufficient requirements, we, as professionals, should be able to estimate our work”.

My Response: This is where we really diverge. I see estimates as waste, in the Lean sense of the word, and most other ways. Estimates are not part of what the customer buys. They don’t add value to the product. In a value stream map estimates are on the “non-value” track. They are merely ONE WAY people have attempted to be able to arrange working relationships and get software created. It makes sense, and it partly works – still, there is nothing about them that makes them necessary, or even beneficial to the actual creation of software. We can write software without them. We can do whole projects without them. And if we can find a way to make BETTER SOFTWARE at less cost, and to more quickly get the right software into use without estimates, wouldn’t it be professional and worthwhile to at least consider doing it that way? Hopefully we can agree on that.

Cory: “The world expects no less from engineers in other fields. Multiple contractors recently estimated the cost to finish my basement despite the fact that I had no floor plan, no idea where lights would be, only a vague idea on what word work I’d like, a bathroom of unknown size and quality, etc. If one of them refused to provide an estimate, I’d have gone elsewhere.”

My Response: Here we continue to think differently. There are countless examples in life where estimates work to some degree. We’ll discuss that some other day – most, if not all of them are NOT useful as models for software development. Here is where I’d like to leave it for the moment: If you could have had the work done on your basement for less money and get higher quality, and perhaps get it done even quicker – would you be willing to consider doing that? I know that there would be no way to know it was cheaper if you had no estimates – but conceptually: If you could get all the benefits I describe would you consider learning to do business that way? I hope we can agree that would be better all other things being equal.

We have contracts, and estimates, and predictions, and all the trappings of doing business this way because most people haven’t found a way to do things better – we fail each other, we don’t deliver as expected, we don’t trust each other, so we find ways to do business that gives us a chance to do business. So… we can try to get good at estimates, and it if serves you and your customers, more power to you.

Lastly:

Cory: “That said, while I disagree with your answer to “What can we do?” as “Nothing.”, I enjoyed reading your blog. It’s fun reading and certainly food for thought.”

My Response: I didn’t ask “What can we do?”. I asked “What can we do instead of doing estimates?”, and the answer was simply put as “Nothing”. My point being that we do not need to do anything “instead” – estimates are often simply UNNEEDED. And then I posed the more meaningful question: “How can we develop software without doing estimates?”. The nuances of this are important, and perhaps I was being too cryptic. At the time I wrote this post it seemed few were willing to explore these things. I hope I was wrong, and now I am certainly finding more people willing to entertain and even experiment with finding better ways. And some doing purely “no estimate” projects.

Overall, I have found the Agile way works well. When you have customers who see the benefits and find it works better for them than the old phased, predictive approach – then it is truly sweet.

Thanks for your kind words. I try to be fun, and I’m “all about” food for thought!!!

Cheers!

I’ve done 12+ sessions at Agile and Developer “user groups”, Agile or Dev conferences, college classes, and code camps where the main topic (or a big component) of the session was Estimates in Software Development.

Here are the notes from one such session I facilitated last year at an Open Space event (Agile Open California, North)

Session Title: Estimates – Can’t Live With Them, Can We Live Without Them?

The title was my own.  I’ve been talking about, and holding sessions on this subject for several years, and the previous time I was at this event a couple years earlier I did a session on “5 Whys on Why we Estimate”.  It was a very contentious session, with almost everyone (except me) feeling estimates are important and needed for software development, and the problem is just that we’re no good at it… so we just need to get better at doing estimates.  I don’t agree (as you probably know).

The “we just need to get better at it” argument sounds a lot like what I used to hear in “post-mortem” and “lessons learned” sessions in waterfall/phased project environments. A typical “take-away” from those sessions were things like this: “We missed our deadlines and had a lot of rework because the requirents kept changing, even though we have a change control process in place.  We need to get better at controlling changes to requirements”.  I hope you see that is very likely a bad conclusion.  Essentially, that line of thinking is that “since control isn’t working, we need more control.”

I’ve been suggesting that if we keep failing at getting good at estimates even though we are working hard at becoming better at doing so (and have been doing so for countless years), then maybe we are expecting estimates to do something for us that just can’t be done.   Maybe there are better ways.  [Just to be clear, for me – I feel I have found a much better way and it is Agile Software Developent as presented in the Agile Manifesto (Values and Principles) – the Agile MVP.]

My position: We need to get to the bottom of what it is we want from estimates and predictions, and find a better way to get to what we really want. [HINT: What we reallly want is Sucessful Software Development Efforts (or something like that, perhaps?) .  Or, as long as we are wishing, let’s just wish to win the Mega Lottery and then spend our day’s doing something we really enjoy. Like taking hikes or arguing – whatever you like.]

Participants:

There were initially 17 participants.  Developers, ScrumMasters, Agile Coaches, Product Managers, etc.  Almost everyone had some experience with Agile/Scrum/XP/Lean.  A few people joined in as the session proceeded, and a few others left.

Purpose:

Explore and discuss what we want out of estimates and the problems we’re having with estimates.

[NOTE: Everything will be tainted because Woody Zuill is facilitating this session, and he leans slightly toward the “No Estimates” approach.]

Process:

My approach for this session: Quickly explain my basic thoughts about the failure of estimates and the purpose of the session, do an information gathering exercise.  Ask why we feel we need estimates, have the participants write their thoughts on sticky notes, group them, and then discuss.

Information Gathering: Why do we “need” estimates

• We gathered sticky notes to identify “why we ‘need’ estimates and put them on easel sheets
• We then gathered the notes into groupings.
• There were 2 main groupings: Estimates are for planning, and estimates are for manipulating
• NOTE: Tne notes below were written by the participants.  Some are a bit cryptic. Not to worry, okay?

• Here are the notes and the groupings:

Affinity Group: Estimates are for Planning

• How many resources do we allocate to do the work
• Length of engagement
• Costs
• To Predict Future Velocity
• You need estimates to determine budget
• To determine when we’ll finish
• Target implementation time-frame & duration for resource allocation & cost estimates
• When to commit to customer delivery
• to decide whether or not to do a task
• To Determine when a feature will be available
• To determine when we’ll finish
• To know when it is done
• It drives the business
• So we can prioritize
• Business needs it for planning
• Request funds for work
• To help balance the work
• Helps in managing resources
• Determine if end result is worth the cost
• To give the customer prediction – to meet customer expectations
• CEO needs it.

Affinity Group: “Gaming” or manipulating the system and related stuff / dysfunctions

• NOTE: I am not discussing these things in this post – sometime soon I’ll cover this stuff in detail.
• The keep your job
• To give a reason to yell at someone
• It’s “How we do things”
• Habit
• Need to get a programmer fired (setting her up for failure)
• To cover your own ass as a developer
• To pit people/teams against each other

Now for the discussion

After our gathering and grouping exercise, we began discussing what we felt this showed us.  During the discussion we just “followed the flow” as ideas were expressed.

We observed, as a group, that estimates are mostly for planning and making decisions about what to work on.

In the discussion, we were in alignment that we “need” estimates so we can plan our work, and so we can make decisions about which projects or features to work on.

After all the effort we put into estimating, we still rarely get things done “on schedule” and “within budget”.

Also, we still frequently ask (or our managers ask): “Are we there yet?”.

Why do we have these problems?  We’ve been doing estimates for years – shouldn’t we be great at it by now? Other similar or related things that we’ve seen:

• Progress isn’t visible
• We seem to be get more behind our schedule no matter what we do
• Money is running out and important things are not even close to being done.
• Stuff we thought was done turns out to not be.

Rarely are the estimates accurate enough to really count on for release dates, costs, etc.

How do we typically deal with “over budget” and “not on schedule” issues?

• We end up cutting features
• Pushing out deadlines and so on
• Hire more people
• Blaming each other about whose “fault” it is
• Agreeing to get “better” at gathering requirements and doing estimates “next time”

Still, others are dependent on estimates such as marketing, HR, Ops, our customers… and so on

• Do estimates really provide sufficient information for these folks to get real value?
• Or… are there a lot of decisions made based on faulty estimates???

So, With all these problems, why aren’t we looking for a better approach? What is wrong with our estimating process? Should we try to get better at it, or… ?”

One reason estimates provide little chance for usefulness:

• The Unknowns might invalidate any estimate we make – estimates by nature contain unknowns
• What is 4 x 8 + 7  / 2 * 83 + unknown?
  • The answer UNKNOWN
  • We can guess, or base our UNKNOWN on similar things.  Might work. Might not.
• Many important decisions are based on UNKNOWN, but we act as if we “KNOW” something
• The argument is often made that we just need to “have some relative estimate…”
  • It is likely that even this leads to bad decisions
  • For example, what good is relative decision based on “This UNKNOWN is bigger than that UNKNOWN?”

MY CONCLUSION AS PRESENTED:

So, if the Estimating process is flawed (and it might be), and getting “better at it” hasn’t worked… Do we really need estimates?  Can we innovate a better way?

• What I am suggesting is that we have to evaluate our dependence on estimates
• My own experiences have proven to me that estimates are not always needed.
  • I’ve been working without estimates for over 3 years with good results in corporate, “internal” projects
  • I’ve had several long term ongoing projects that did not use (or need) estimates – One of over 4 years on commercial software
  • I’ve done contract work with NO ESTIMATES. Many have told me “customers won’t do that”, but there are at least some that will.
    • When a customer sees that more “real” work gets done and more of the right thing gets done with less “busy work” they get on board.
    • Finding ways to pay a very low cost to prove something works, or doesn’t work is very attractive to most customers.  To me, this is a fundamental benefit of an Agile approach.
• My experiences are not proof that estimates are not needed.
  • I am not trying to change the world.  I do want MY jobs/contracts/customers to get the benefit of doing things in a better way.
  • I AM trying to invite conversation about this stuff, and encourage us to explore and replace these painful practices in our profession.

This discussion had more to do with posing the questions than answering them

• It is contextual (as usual)
• Lots of organizations feel they need estimates and you must work within the system.  Sorry for that – but I can’t fix that.
• Often, customers will not entertain using a company that does not provide an estimate.
  • They want to know what they’ll get, when they’ll get it, and how much it will cost.
  • It is a risky situation for everyone – and if you are in that world, there are still ways “sell” the benefits of finding a better way.
• I am suggesting is that we need to do some serious thinking about how to change this reality, and innovate ways to eliminate the dependence on estimates

I am proposing that we need to imagine a different world

• Do some “thought” experiments
  • IMAGINE:
    • What if we were to find that estimates have NO value (zero benefit), should we still do them?
      • What would that look like?
    • What if we were to find that estimates are harmful and are counter-productive?
      • What would you do?
    • What if the were to find out that estimates are fatally TOXIC
      • What would we do then?  How fast would we need to address this?

Take some action that will expose the reality about our estimates

• Can we find a way to wean ourselves from estimate dependency?
• A few things that have worked for me
  • Deliver value quickly and frequently (sound familiar?) – that is: put small chunks into real use early and often
    • Making decisions about “what to do” become less important when we can do “some of this” then “some of that” and see what works (or doesn’t work) quickly and cheaply.
    • People stop worrying about “are we there yet” when interesting (useful) things are continuously happening and being put into real use by real users.
      • It’s like bringing along some new toys and activities to hand to your kids on long trips when they start getting antsy.
  • Lead your customer/stakeholders/whatever to an understanding of what is important to deliver NEXT…
    • … we only have so much work that can be done right now – focus on that
    • … put all your effort into that
    • … and put little time on all else
  • You can introduce this a little at a time.
    • Thinking people who are open to this will slowly come around.  Or not.  Just depends.

Remember: I am just encouraging discussion.

What works for me might not make sense to you.  Maybe someday it will.  When that happens, you will have crossed over into the land of no estimates.  Beware.

In this episode I’ll cover an extremely satisfying application of Lean and Agile on a large ‘Legacy’ effort I was part of a while back.

Please understand that I am not at liberty to discuss certain (that is, most) details. I must protect the guilty. I will give just the bare details – only what is necessary. I bet you don’t believe I will be able to give just the bare details, right?

So – this is about using the No Estimate Approach on an End-Of-Life Legacy Support project.

First, a few definitions:

What is “Legacy” code?

I am glad you asked.  There are many definitions of legacy code. Here is the one I use: Legacy code is ANY CODE THAT IS DOING ACTUAL WORK THAT YOU WANT TO KEEP WORKING. (Please note the clever use of “all caps” for emphasis.)  Does that make sense to you? Doesn’t matter – just pretend that is what it means.  Using my definition, most projects become legacy code as soon as you have some functionality in production. That’s when the bits hit the fan, usually, so to speak.  However, even if it isn’t yet in production, it could still be “legacy” if we have stuff working we don’t want to break.

What is Legacy Support?

Regardless of what you think legacy support is, for the purposes of this post I am using it to mean this: “Work done (bug fixes and minor enhancements – that sort of thing, or more serious stuff if needed) to extend and protect the life of code that is in use”.   I’m leaving out more extensive enhancements like “whole, new functionality”.  You can judge for yourself the difference between minor and “more extensive” enhancements.  Doesn’t really matter to me.

What is End-Of-Life?

For the purposes of this post, an End-Of-Life project is a project that is almost retired.  It’s still in use but going to be replaced sometime soon.   In software programming “Soon” means “perhaps sometime, but we have no clue when.”  The basic idea is that people are using it, and it is important enough to keep supporting, but we’ll stop doing big additions and improvements.  Some bugs will just be ignored, but any deemed to be important to fix will be fixed if possible.

Project History and Details

The project that we are discussing is is a large, line-of-business set of applications (‘The System’) that is critical to the users of the software for managing most aspects of their business.  The typical user of The System manages thousands to millions of their own customers – sales, billing, transactions of all sorts, bunches of “real-time” stuff, reporting, etc.  Typical line-of-business stuff.  I was told the code base was about 2 million lines of code [and that’s about enough, in my opinion – any more is just bragging.]

The System has been in production for 15 years or so and has moved through several different code bases.  The current code base is about 4 or 5 years old.  It uses several languages – 4 or 5 perhaps.  Typical stuff.

Enhancements and bug fixes are managed, worked on, and delivered using a waterfall approach, with a 3 month delivery schedule (that takes 6 months due to test and fix cycle, and other typical deadline issues).

Due to design, industry, and technical reasons it is deemed necessary to re-write The System in a new language with a new architecture.

The Path Going Forward

The current version of The System MUST continue to serve the user base, and even more – it has to be clear that we are still responsive to concerns and issues of the user base so they don’t start looking to competitors to solve problems while the “greenfield” version of The System is being developed.  It has to look and feel like “business as usual, only better”.

A “maintenance team” is put together to provide this service.

I’m not sure exactly how many people were working on The System before the End-Of-Life Maintenance, but it was something on the order of 20 to 25 people – maybe more, it was a while back and I didn’t keep track.

The new team was to be 6 people: 2 QA, 2 BA (Business analyst or “Product Owner”), 2 Dev.  I was one of the two Devs, and also the manager, or co-manager (more or less).  I “volunteered” for this assignment on one condition: I would be allowed to manage this effort my way – which means Lean/Agile/XP.

We’ll call this System EOL (for system end-of-life) .

The State of System EOL  on Day One

There are approximately 500 active bugs in the bug tracking system.  Some of these bugs have been pernicious. Some have never been addressed due to level, some have been introduced in the fixing of other bugs, some have been there a long time but just recently discovered – All the typical bug stuff.

500 Acitve Bugs???!

There are 500 active bugs! There are a number of reasons there are so many bugs.  Before we talk about that, I’d like to ask a question.

• Question: What is an appropriate number of known bugs for a project that is in production?
• Answer: Zero.

Got that? I’ll allow for any bugs that have easy work-arounds – you can leave those if you have to.  I’ll also allow for any bugs that were discovered since the last deployment.  But you better fix those NOW and then you’ll have zero bugs once again. Nice.

Remember that: Zero bugs.  Does that seem unreasonable to you? If it does, I’ll ask another question: Why bother reading my blog? Go do something else that seems reasonable to you.  Why waste your time on what I’ve got to say? I am NOT REASONABLE most of the time.  I want a better way to do stuff than whatever we used to think is “reasonable”.  Zero bugs is a “lofty goal” in some workplaces.  In others, it is business as usual. I prefer zero bugs.

So, 500 bugs is not just way too many, it is infinitely too many.  Or whatever 500 divided by zero is.  Anyone know?

One reason there are so many bugs: The old process.

There were several reasons all these bugs existed, but the one that is most pertinent to our discussion today is the process that was in place for assessing and fixing bugs. The way things were being done was about like this:

• Bug report is entered into the tracking system by a user, support person, developer
• The bug is reviewed at a “triage” meeting where its importance determined
• If it is important enough, it is assigned to an “investigator” who will assess it: that is, attempt to reproduce it, determine the likely cause and module that will probably contains the “bad code”,  and estimates the time to fix.
• After the assesment, the bug goes back to a “bug review” meeting to re-assess it’s importance and schedule the work to be done.
• If it is deemed important enought, it is scheduled to be fixed (either sooner or later based on the “cost” of fixing it and importance to the customers).  If it is not important enough, it is closed and it’s status and other details are communicated to the interested parties.
• It is assigned to a developer who works in the area of the “bad code”. Since the developer is a different person than the investigator, the developer starts from square one and “re-thinks” the stuff the investigator already thought about.
• In working on the bug, the developer discovers the problem is NOT in the area of code it was thought to be in, so he adds his comments and assigns it back to the bug review team.
• The bug review team looks at it again, and then re-assigns it to an investigator.
• And so on and so on…
• Once a bug is actually fixed, it goes thorugh QA where it is discovered that it didn’t fix what needed to be fixed.
• Go through whole process again.
• Once a bug is fixed and passes QA, it is deployed in the next deployment, and…
• The users discover that it didn’t fix what needed to be fixed, or the fix was only partial, or that something else is now broken.
• Even worse – some other things have been broken that are not yet discovered that will lead to other bug reports sometime soon.

Okay – you get the point.  I am not making this up.

There were other reasons System EOL was in this state, which included lack of unit tests, code neglect (huge methods, overwhelming complexity, fragility, coupling and cohesion problems), and so on… but we’ll solve one thing at a time.

Chutes and Ladders

Well.  Does that remind you of that game you used to play when you were a kid?  It should.  And it is JUST AS FRUSTRATING, at least conceptually.  Only difference is you get paid to play Bug Fixing Chutes and Ladders.  But overall – it is terrible and horrendous (and other bad words).

What to do???

Since I was allowed to manage System EOL in my own way – Here is what we did.

Value Stream Map

First day I brought in 6 copies of Mary and Tom Poppendiecks “Lean Software Development” book, and we all agreed to read it, and more or less follow their basic concept.  I’d been doing XP for years (more or less) but it’s a hard sell so I opted for the “easy sell”.  If Lean is good enough for Toyota, it’s good enough for me.

First thing we did was a value stream map on the  “Chutes and Ladders” methodolgy that was in use. If you haven’t seen a value stream map or know how to do one, take a look at Mary and Tom Poppendiecks “Lean Software Development” book.  The basic idea is that anything that is not directly adding value can (and should) be eliminated. For example: waiting in queues is a waste – it brings no value to the person interested in having the bug fixed.

I bet you can guess that we found a lot of NON-VALUE-ADDING behaviors.  So, we made a few rules.  For example…

First of all, a few rules:

• No Triage meetings or Bug Review meetings
• No Assesments for “where the code is broken”
• No prioritization or severity determination (a bug is a bug is a bug).  If it’s in the tracking system, fix it.
• No Estimation of effort. We gotta work on all the bugs.
• … and Other things

NOTE: No Estimates, No prioritization, No meetings.  These are wastes (at least in some situations, if not a lot) – bringing no value to the “customer”. Even worse,  waste usually removes value by using up time and “spirit”.  A team doing a lot of meetings, estimates, assessments, meetings, prioritization, meetings, waiting, etc. becomes NUMB. Avoid numbness.

An initial assessment:

We quickly read each bug in the tracking system, looking for duplicates, bugs no one had asked about in a year or more, and bugs where the description could not be understood (and there was no one who could clarify). This was done very quickly and eliminated about one half of the bugs.  NICE. 500 bugs cut down to 250 bugs.  No one can say that is not progress!

This took about a week (as I recall).  During this “first-week” the developers and QA took on bugs to fix, since there were plenty that were clear enough to work on – but spent some of their time helping the BA’s “weed out” the bugs that we removed.

Daily process we adopted:

• Daily morning 1/2 hour meeting.  Sitting. Quickly talk about some interesting things (movies we had seen, a great lunch place we found, etc) , and occasionally some work stuff as well.  Seriously – this was a time for the team to become comfortable with each other and build some common memories and start the day off on a pleasant note.  And everyone outside the team expected we’d do daily meetings and we didn’t want to disappoint them.
• Pick a bug, any bug.
• QA and Dev sit together and reproduce bug. Confer with BA as needed.
• QA and Dev examine code together, and create a characterization test to “clamp down” the code.
• Dev fixes code using TDD whenever possible, conferring with and pairing with QA and BA whenever needed
• QA and Dev test code to verify fix works as expected
• QA and Dev run all existing characterization tests
• Code is checked in and project is built
• QA runs all existing automated UA tests and any needed manual tests.
• Pick next bug – preferable one that is in some way related to last bug – but not critical. Just pick one, and repeat.

Release Schedule:

• About every two weeks a build of whatever was fixed at that point was made avaliable to customers.
• They could take the build if any bugs they were interested in were fixed, or they could ignore it.  It was up to them.

End Result:

Guess how long it took to clean this project of  bugs.  Give up?  3 months!

250 bugs in 3 months – that’s about 4 bugs a day!!! Well, we did have 17 “un-fixable” bugs.  Those were bugs that we weren’t allowed to fix for some business reason.  So still – an average of just under 4 bugs a day.

This is the power of eliminating waste.  All of our time was spent doing meaningful work, REAL WORK on bugs. No reviewing, no estimating, no “pre” investigation that wasn’t part of doing the fix, no passing code to QA that we were not confident would pass all tests, etc. In other words: NO WASTE.  Or at least: We were no longer doing any of the steps that we had identified as waste.

A Few Other Interesting Things:

Well, what we found is that in fixing some bugs, we’d incidently fix other bugs.  That is, some bugs were caused by a single underlying malfunction.  That happens, and it is a sweet bonus when it does.

We had no recidivism. Once delivered to the customers, we did not have a single fix that was “sent back”.  Once fixed it stayed fixed.  I think this can mostly be attributed to the QA and Dev working together, the unit tests put in place, and the “one-at-a-time” approach to “fix and build.” We would fix ONE BUG at a time. Fix,  Build, Unit Tests, Checked in, Built, Tested, confirmed.  Only then would next bug be tackled.

Conclusion

So… this is about no estimates.  Our rule: Eliminate waste.  In this case, estimates were a waste.  They were thought to provide a way to “manage” the work being done, but were in fact reducing the ability of the organization to “manage” getting work done.  I think they are like the dice in chutes and ladders? Probably not. Still, focus on the real work, don’t muddle things by playing chutes and ladders.

I read a question posted on the Scrum Alliance Google Group, and jotted down a few thoughts about Planning Card Games.

The Basic Question Being Asked:

How do you describe a 1 to the team for Planning Poker®?

The value of Estimating using Card Games:

To me, the main value of story point estimating is as a tool for a team to learn how to break things down into “Sprint size” pieces.  As a team becomes skilled at breaking stories into smaller (yet still valuable and deliverable) pieces, the less need they have for using estimating games. Once the ability to “chip off a small, do-able, deliverable chunk” becomes easy for the team, the work flow changes a bit.  The team no longer needs estimating games – they simply choose important stuff, break off a few valuable, small pieces to work on, and then do the work and build/deploy.  Over and over again.

Many don’t see it that way, and use estimating games to make decisions about what work to bring into a Sprint. That is a legitimate use in some workplaces, and whatever works for you is the right thing.  For me, the scheduling of work for a Sprint is not nearly as important as breaking work into a “do-able” condition, and getting it done.  I’m interested in choosing what we work on based on the value to the “customer”,  not how big things are.  We can only do so much work at a time.  Finding valuable things to work on is where I put my focus.

Discovering What Those Numbers Mean To A Team

One approach that has worked for me in finding a way to size things is to do it during the first Sprint Planning Session:
Take all the Stories you are considering, and lay them out in “size order” on the table – No numbers, just “Is this bigger or smaller (or the same) as that”.  Once you have them laid out you can group them into the numbering system you use.  The group of the small items becomes 1, next group 2, and so on.  During the Sprint you will get a feel for if the 2’s were more or less double the 1’s, and so on.  “More or less” is fine – you don’t need precision.  You are just “getting a feel for it” anyway.

Over time, the team will be able to quickly gauge a basic size for each story – with the main goal being to find the ones that need to be broken down further so they can be taken on as work.  Once a story is taken on as work, you’ll discover the real nature of the story and it’s “size” – it is often found that it needs to be broken up into a few smaller chunks.

Still, What is the Answer to “How To Describe a 1 to the Team”

This doesn’t really answer your question: “How do you describe a 1 to the team for Planning Poker®?” – but it replaces it with a way to work with the numbers. In the past when I used story points and planning cards, I found this to be the definition that worked for me:  The “1” story point is the smallest thing that we can do discretely that brings value once built and delivered.  Smaller than 1 is the place to put things that are tiny, but not really stories.  We would mark something as zero if we felt it would “come for free” in the doing of another story.  Also, anything bigger than a 5 was put into a bucket of “Too big, must break down before we put any work on it”.  For me, usually 1 & 2 are considered for taking on as work, 3 & 5 need to be sliced up if at all possible (and if it can’t be easily sliced up it’s a signal that we need to learn how to better define and break up a story),  and 8+ is “Too Big – must be sliced up”.

Note: Planning Poker® is a registered trademark of Mountain Goat Software, LLC.

I wrote a short post called “Is The Daily Stand-Up Harming Your Team”.  After getting some feedback on it, I have refactored it and here is the new version.  (I have deprecated the old version):

The Purpose of Stand-Ups

Here is a short definition from “Planning Extreme Programming” (Beck/Fowler 2001):

Have a short meeting every day so everybody knows what’s going on, and what’s not.

The Daily Stand-Up meeting is meant to be a very short meeting with several related purposes:

• Team Alignment or “sync-up” – who is working on what, what’s been done, what do we need to discuss…
• Expose issues that need to be handled (impediments, etc.)
• Eliminate other meetings that can “hog time”
• Allow some other people not on the team to hear about things the team is doing
• etc – put your own things you value about Stand-Ups here.

Those all bundle up to one thing: Daily communication about important stuff.

Good Communication Is Essential for Software Development

Programming is done by individuals who must communicate with each other.  Finding ways to communicate well makes things a lot better.  The Agile Manifesto itself puts a lot of focus on the importance of good communication.  A quick glance at the Manifesto and Principles makes this clear:  Individuals and Interactions, Customer Collaboration, Responding to Change, business people and developers must work together daily, face-to-face conversation, self-organizing teams, the team reflects on how to become more effective.  Almost every Value and Principle somehow reiterates the need for good communication.

I “buy in” to all of this.  In my opinion, a BIG part of becoming and staying effective in software development is to continually improve our ability to communicate – team member to team member, team to customer, business people to team, and whatever combination you need.  As we find problems with traditional communication methods (such as meetings, emails, documents, charts, yelling,  etc.) we need to explore “better”.  Even more important, when we find something is working for us we should explore how to make it even better.  Whatever we do, no matter how good, there is always “better”.

A little Background:

A number of years ago (about 10 or 11 years I guess) I started visiting Agile/Scrum/XP development teams/departments and observing Stand-Ups and other meetings – just for the fun of it .  I’d been doing XP for several years, and loved visiting other teams – mostly for my own enjoyment, but also to learn how others were doing things. I have visited a number of teams and departments over the years – about 20 or so.  I am not a consultant as such – so these were casual visits.

What I noticed in visiting teams doing Stand-Ups

I noticed a few harmful things in a majority of the Daily Stand-Up meetings I observed. Things such as:

• Individuals taking a long time (5 or more minutes each rather than 2 minutes)
• Meeting taking too long (45-60 minute standups – more like “lean-ups” after a while)
• Too Much Detail, Not enough Detail
• People not showing up on time
• Too many people (10 or 20 people) – attendees are hearing about stuff that isn’t relevant to them.
• Mumbling and Grunting
• Discussions between 2 or 3 team members to resolve some aspect of something
• Yelling
• No one is interested in what the others are saying (and often without any reason to be interested)
• Issues brought up are never addressed
• Needed follow-up meetings never occur
• Same report every day
• Dogmatic enforcement of rules
• … I can go on and on. I used to take notes and had about 20 different issues that commonly showed up.

I have seen a lot of these same dysfunctions on teams I ‘ve worked with over the years.  Some of these teams have discovered ways to reduce or even eliminate most of these “Daily Stand-Up” dysfunctions.

There have been countless blog posts, book chapters, workshops, conference sessions, etc. on solving “Daily Stand-Up” problems – which is great.  It is a good thing to recognize problems and solve them.

But I think there is a “next step” we can take.  For me, when I see something causing this much trouble for so many different folks and teams, in so many dfferent ways, over and over again – I start to wonder if there is something we are all missing.

My first question was: Why do so many teams have so many problems with Stand-Ups???

The Daily Stand-up is a fixture in most Agile/XP/Scrum environments.  It has a very simple purpose, and is a very simple meeting – and yet, many teams continue to have these problems. Why???  Well,  that is too complex for me to cover completely here (I could cover some of that in a future post if anyone is interested) – what I want do do instead is share what I have discovered after investigating this for about 5 or 6 years. I’ve done a number of “5-whys” root-cause analysis sessions, and retrospectives on vairous Stand-Up problems, and I’ve read as much as I could find on the topic.

Here is what I think I have found:

At the bottom of many of my “5-why” Sessions

You’ve probably all done something like a 5-Whys analysis. From Wikipedia:

The 5 Whys is a question-asking technique used to explore the cause-and-effect relationships underlying a particular problem. The primary goal of the technique is to determine the root cause of a defect or problem.

With manny of the dysfunctions I was investigating, I would end up at about the same place.  Typical starting questions were: “Why do team members think the Stand-Up is boring”, or “Why do we have too many things we need to cover in the Stand-Up”, or “Why does our Stand-Up take so long”.  As often as not, the near the bottom of my “5-why” session was this question: “Why do we think we NEED Stand-Ups?”, and the answer I often end up with is “The Daily Stand-up exists because we are not in continuous alignment. We need the meeting because there are things we are not communicating during the day”.  So that leads to this question: “Why aren’t we in continuous alignment?”.

Why aren’t we in continuous alignment?

That is: If we could be in “continuous alignment”, and find we have nothing to communicate in the Daily Stand-Up, we wouldn’t have any of the dysfunctions of the Daily Stand-Up because we would no longer be holding them.

Would it be good to be in Continuous Alignment?  What if it was possible to effectively and efficiently stay in “continuous alignment” how would that change our thinking about the need for Daily Stand-ups?  Some teams have been very happy with the Daily Stand-Up, and find that it brings a lot of value.  Still, even in those environments, there is always room for improvement.  In Agile/Lean/XP/Scrum it is important to continuously scrutinize the practices we are using and innovate even better ways to do things.

It seems worthwhile to me to at least explore the possibility that there is something even better than well functioning Daily Stand-Ups.

Are You Saying We Should Just Stop Doing Our Daily Stand-Up?

Of course not.  But this hints at an important “next step”:  Let’s figure out how to have continuous alignment.  Some common practices that many teams already use are things like: Task Boards that everyone can see and that are dynamically updated as work gets done, sitting together (so you can hear anything imporant that is happening), pair-programming, switching up pairs frequently (several times a day), raising “blocking issues”(and solving them) immediately rather than waiting until a meeting,  TDD, CI, and whatever you can think of.  The more complete and meaningful you can be with dynamic communication the better chance you have of being in continuous alignment.

Of course – these practices will not necessarily elminate the need for the Daily Stand-Up – lots of teams use them and still find they have a need for the Stand-Up, but they will very likely resolve a lot of the traditional “Daily Stand-Up” issues I listed above.  A shorter Daily Stand-Up that is more meaningful for the team could be a worthwhileend result.

What are the Costs of “Continuous Alignment”

The “costs” for being in “continuous alignment” should be small – it’s a matter of discovering the things you need to communicate and a mechanism for doing having those communications.  Most Agile teams are already doing a lot of the practices that help provide good communication throughout the day – it’s just a matter of fine-tuning them, or “turning them up a notch”.  Becomming better at communication is a benefit in any team situation. We can choose the Daily Stand-Up over Continuous Alignment, vice versa, or both in some combination.  Each time we find something in a Daily Stand-up that should have been communicated earlier we should look into it and figure out how we could have communicated it dynamically.  The main cost is that you’ll need to pay a lot more attention for a while – but as you smooth things out it will become natural to gravitate to better forms of communication.

What My Experience Has Shown Me

If we have continuous alignment, there is little need for a Daily Stand-Up.  When it gets to the point where we gather in the morning for our Daily Stand-up and consistently find we have nothing useful to share it is likely that the practice can be retired.  In the “Lean” sense, a practice is waste if it brings no value.  Waste is not neutral, it diminishes the team – it is a cost that has no return.  With my current team we’ve worked without Stand-Ups for over a year and have found no need for them.

Here are some of the practices we follow that allow us to do this:

• We work together througout the day
• We meet at any time there is something important that needs to be addressed
• We pay attention to update our dynamic “information radiators” as we work
• Anytime we need help we ask for help
• Anytime someone asks for help, we help
• We meet face-to-face for all meetings whenever possible
• We all work on the same thing until it is done or there is nothing further that can be done with it (“one story at a time”)
• We solve blocking items immediately if possible, and communicate with each other when anything changes

My Advice – A Few Ideas

If you want my advice it is this:

• Examine your Daily Stand-Ups regularly
• Pay attention to anything that comes up in the Stand-Up that should have been communicated earlier
• Hold a retrospective with a focus on the Stand-Up – do this several times a year
• Find better “dynamic” communication techniques
• If you are not doing pair-programming, you should practice it and learn how to make it work for you
• Switch up pairs frequently – every few hours
• If you can’t stand the idea of pair-programming, do some Code Dojos and Katas to experiment with it
• Sit together – all team members working within easy talking distance of each other
• Do more work as a team – QA and Dev pairing, “Customer” and Dev pairing, QA/Dev/Customer teaming, and so on.
• Improve your “information radiators” and keep them updated throughout the day
• Try “Mob Programming” where everyone works at the same computer all together at the same time
• Talk with each other during the day

That’s just a start.  You will probably come up with better ideas than I have.  Please let me know what you find.

So: Are Daily Stand-Ups Harming Your Team?

Might be.  If there are better ways to do things, and you don’t at least explore them, then you might be stuck with practices that are not as useful as they could be.  In the old days, I heard a lof of “That’s the way we do things here” excuses.  More recently I see the “Cargo Cult” approach to things: “The books say this is the way to do it, so this is what we do”.  I’ve seen a lot of tricks suggested to fix problems with Daily Stand-Ups (like using a 10-lb talking-stick to keep things short) and feel that whenever you need to resort to a “trick” like that you probably should be looking for a root cause instead.

What do you think? Are Daily Stand-ups Harming Your Team?

NOTE: I have deprecated this version of this post.  I have put the new version at:

New Version of Are Daily Stand-Ups Harming Your Team?

This original post was not as clear as I had hoped, and I don’t mean for anyone to get the wrong idea. Thanks to Ron Jeffries and George Dinwiddie for their converstaions that helped me better organize my thinking and the purpose of this post. 

Here it is.  You weren’t looking for it, but you found it.

Mob Programming!