Life, Liberty, and the Pursuit of Agility

I love Document Driven Development

One thing I really miss in Agile is having long, incomplete, boilerplate documents for me to “sign-off”.   I like to write code and deliver working software quickly, but even more I LOVE to read documents and like nothing better than to spend long days before a glowing flat-screen reading incomprehensible nonsense carefully crafted to appear complete and correct while still being essentially content-less.  Doesn’t matter what they are about, as long as they have small type and lots of words.  Of course, nothing is further from the truth.  If I want to read nonsense then I’ll read Edward Lear.  Or my own writings.

So documents are fine, but what about Sign-offs?

Even more, I love to take responsibility for documents that I did not write.  It gives me the feeling that I am a grown-up.  But apparently, I am very unsophisticated regarding sign-offs.  Or I was until recently…

I had one CEO show me the policy for documents at his organization (let’s call it “Anonymous Inc”) where there were 3 sign-off statements for a manager to sign on any document requiring her approval.  Oh joy!!!  It is in the spirit of Extreme Documents, I suppose.  If one signature is good, why not dial it up and try three?  More is always better.  In his defense, he was showing this to me as an example of something he wants to abolish – he understood the absurdity of it.

I approve this document

This typical sign-off is for the manager-type to “approve” the document.  In software development this often indicates that others can rely on this document as being “okay to use for doing whatever it is they do next based on this document”, and that it is Complete and Correct.  This is actually the third signature to apply after the next two required signatures have been taken care of.

I have read this document

At Anonymous Inc, besides approving the document you are also required to sign-off that you have read the document.  You’d think that approving something implies that you read it.  Guess what?  Apparently this isn’t as common as you might think, at least not at Anonymous Inc.  I suppose the “I have read this” signature solves the problem of someone later on saying “I just signed it, but I didn’t read it – if I knew what was in it I would not have signed it”.   I suspect that a lot of documents get “approved” without the approver having actually read it, and VP’s and upper management can always say “I was so busy I had my underling read it – he said is was okay so I signed it.  Let’s fire him, okay?”

I understand this document

Okay – “I approve this document” might make sense in some organizations.  “I have read it” at least keeps some people a bit more honest, I suppose – I’m not sure it would.  But “I understand it” is something I just do not understand.  Still, it is a very fun idea, and Anonymous Inc is all about fun.  Still, just because I say “I understand” doesn’t mean I do.  Any parent knows that.  “I understand” can also mean “shut up already”. But then, what do I know?

Have fun out there, and be careful when you sign-off.

Estimating Software Development

Anyone who knows me will know that the title of this post is meant as a bit of a joke.   For software development, it is my experience that estimating** is rarely useful and always easy.

Well… a little clarification:  estimating is NEVER useful but ALWAYS easy the way I have typically seen it done, which uses the WAG* method of providing some numbers that seem plausible and that will allow the decision makers (stakeholders and managers) to approve the start or continuance of a project with a clear conscience. That is, when things sour later on they can always blame the developers for giving them inaccurate estimates (and not living up to their “commitments”).

And the developers can always blame the “requirements people” for giving them unclear, incorrect, or incomplete requirements.

And the requirements people can always blame the stakeholders for providing bad direction about what was important and the developers for not “understanding the requirements”, and so on.

Regardless of who is part of this process, it is one big happy circle of blame that lets us all do the wrong thing and still keep our jobs (not always happy for everyone, actually, and sometimes not everyone will keep their job).

I have simplified it here, but the basic idea is sound.  As long as everyone is good at deflecting blame, and everyone is willing to continue on saying we’ll do it better “next-time” and after only a few people have been fired then everything is fine, right? Well, I don’t think it’s fine, of course – but many organizations seem to operate this way.

Let’s estimate a game of chess

Chess is a game that some of you may have seen.  It provides a very limited environment with only 6 different and charming pieces that have very limited ways in which they can be used, and there are only two players.  Simple. The whole game can be contained in an area of about a square foot or even less, and the complete rules can be written on the inside of the box lid.   Just like Candy Land, it is very easy to learn and play. Not so easy to win, perhaps – but that makes it interesting enough to be an engaging game for some.

So, here is a description of a simple “project” and I need your estimate:

Win a game of chess. Losing is not an option. Tying is not an option.  We will only make money if you win.  If you don’t win, the company is at risk, and you will lose your job (and mine).

Also, I have a diagram of a chess game with the exact position of the pieces as required at the end of the game.  To make money, it needs to end up just like the diagram.

Oh, one other thing: we don’t know yet who you will playing, but we’re pretty certain it will be someone who knows how to play the game, and might be pretty good at it.

That’s it!  There are only a few requirements:  That you win, and that the board matches the diagram I’ll give you.  Easy!

What I need to know is exactly how much this will cost me.  In this situation, cost is the number of moves you’ll need to meet the requirements.  Oh, also, we don’t have the diagram yet, but we’ll have it soon.  Still, you should be able to estimate it because it is basically the normal chess end-of-the-game arrangement with pieces here and there and so on, and you have the opponent in a checkmate.

Should be easy to estimate that, right?

Well, I hope you agree this is impossibly difficult, or actually: just plain impossible to do a meaningful estimate.  First of all, there is not enough information to do a good estimate, and even if we had all the information, there are many variables and a lot of stuff we have no control over. How close do you think your end results would come to the requirements?  Would you be willing to stake your job on making a decent estimate on a chess game in this situation?  Well, as long as you could blame someone else and promise to “all do better next time” you’ll probably be okay.  Or you could be a consultant and be paid and off to your next contract before someone actually has to play this game you helped them estimate.

Getting better at estimating is like pulling teeth

I hear it over and over:  Our estimates were not accurate and we had a lot of trouble because of that.  We need to get better at estimating.

It is similar in a way to saying “My tooth hurt, so we pulled it out.  Now I can’t chew as well as I used to.  We need to get better at pulling teeth”.  What is the real problem?  Shouldn’t we work on that instead?  Hint: Getting better at pulling teeth is not it.

So what would you do to get better at estimating the game of chess, based on the basic scenario we described above?   Could you get better at it?   How good would you have to be before it had real value? Is it worth spending time on getting better at it?  What is the real problem.  Hint: Getting better at estimates is not a solution to anything because having bad estimates it not the real problem.

My suggestion: Stop That!

I haven’t seen much value in estimates.  Actually, in the software development work I have done, I can’t recall any situation where estimates** of this sort were of value to the actual job at hand.  Someone wanted them, someone saw value in them, and sadly – important decisions were made based on them.  Do you think those were good decisions?  People often tell me that “even though we know the estimates are inaccurate, they are better than nothing”.  I prefer nothing, if there are better ways.  There are better ways.

Almost everyone tells me that estimates** are very important, but estimates are often a blind spot hiding something of actual importance and value.  This almost universal acceptance that estimates are important and useful is harmful.  A predictive approach such as waterfall itself depends on estimates – and estimates are how we make the predictions.  Don’t think “waterfall” and predictions are harmful?  You probably should be reading some other blog.

Estimates** of this sort are based on guesses about the time needed to do work we don’t yet understand.  Nothing about this gives me confidence that these estimates have value.

It seems I always make enemies when I suggest this, but that is not my intention.  I want to do effective work on meaningful stuff – that is my intention.  I don’t want to be spending time on work that has no value, causes misdirection and waste, and potentially destroys any chance of being effective.  That is painful.  There are MUCH BETTER WAYS to approach software development.

Do you need estimates?

It could be useful for you and your organization to question your use of estimates and explore new ways to think about your “need” for estimates.

One last thing:  Do not ask me for an estimate on how long it will take to eliminate estimates from your process.  It is too much like a game of chess, only very complicated.

Here is a start at clearing up our thinking.  Let’s do a little thought experiment.

Which would better help if we have decided to eliminate estimates and find a better way in our organization? :

• Get rid of all developers
• Get rid of all upper management.
• Get rid of all upper management and developers
• Something else more realistic

Who is most insistent that estimates are important?  Who do those people answer to? Are they going to fire you for questioning things like estimates? Are they open to scrutinizing the status quo or to examining why we do the things we do?

So what might be better?

What can we do instead of doing estimates?

Nothing.  Don’t do estimates.  Simple, clean, easy.  But that is not the question we should be asking, and that is not what I am suggesting.

The correct question is: since we think that estimates are important, and you say that they are not, how can we develop software without doing estimates?

The answer is… Well, I suspect no one reads this blog, so that doesn’t really matter.  This is just a clearing house for my thinking and has nothing to do with the reader.  However, I am sure to write some stuff someday about ways to get things done without estimates, and I guess some of my earlier posts cover some of this thinking. If someone is actually interested and asks for me to cover this, I might get to it sooner.  But I am sure you can figure it out without me – this is truly simple stuff.  Read the Agile Manifesto and Agile Principles, and use them to invent better ways.

Another hint: Get good at frequent incremental delivery of useful software.  That might be worth considering.  That takes a LOT of experience and open thinking.

NOTES:

* WAG = Wild Ass Guess.  A Wild Ass is a beast that is indigenous to parts of Africa and Asia.  A Guess is a guess.  Wild asses are about as good at making software estimates as anyone else.

** For the purpose of this article, the sort of estimates I am discussing are the estimates typically asked for on many projects where a list of features or functions are described and people are asked to come up with an approximate cost in time (working time or elapsed time) [sometimes it is phrased as “cost”, or “effort”] to do the work that will be required to provide the feature(s)/function(s)/capability(ies) being requested.

Disclaimer: There are many situations where estimates can be meaningful and useful.  This article is about situations where I don’t think they are meaningful or useful.

I like to talk

I was given an opportunity to present a session at the Agile San Diego group last night.  I have some pretty clear ideas about how to be productive in developing software (and a fair number of solid successes in turning out useful software), and those who know me are sick of hearing this stuff.  I am always looking for an opportunity to find someone new to bore.  Thanks for stopping by. 

Agile Success seems elusive

Over the past 13 years I’ve been using (or sometimes fighting to be allowed to use) an Agile approach in software development.  For about a year I have been slowly gathering my thoughts into a talk on Agile Success.  I’ve given about 5 talks on this theme now – each very different, and the audience has mostly been folks new to Agile thinking.  

Agile San Diego is a group of highly sophisticated Agilists who are typically expert debaters and highly competent loud arguers –  I like it very much.  So I took this opportunity to solidify my materials and put it to the test.  The results were so-so, IMHO, but it is in the doing of a presentation that I discover the presentation that I must do.

Agile Success – Why Not Me Too?

In its current state, the talk is called “Agile Success – Why Not Me Too?”.  It’s mostly about the “Why Not”: we tend to block ourselves from achieving good result by hanging on to “old and busted” thinking and practices.  Perhaps we do this because the old ways seem so right (but they are oh so wrong) , or perhaps just because we are comfortable with them, or “our customers (or boss, or manager, or marketing) require it”,  or … well  – there are a lot of reasons we keep doing things the same old way – you get the idea, I am sure.

For this instance of the talk, my main goal was to show that we are typically our own worst enemy.  More specifically, it is the things we think are most solid, meaningful, useful, and productive in our process that are the most likely to be hampering our progress and wasting our time and money – and we don’t have the skills or openness or whatever to recognize this.  In other words, those are our blind spots. 

The Maxims

Here are my three main talking points.  I call them Maxims (to differentiate them from the Values and Principles):

Critical Maxim 1:
It is in the doing  of the work that we discover the work that we must do.

Critical Maxim 2:
Embracing change is IMPOSSIBLE if your code is not Easy to Read, Easy to Maintain, Easy to Grow, and Easy to Change.  Everything else is secondary.  Everything.  This applies to “lean start-ups” too!  Code becomes HARD really fast – often within a few hours if we are not paying attention.

Critical Maxim 3:
Question Everything

In my own work, I use the Agile Manifesto and Principles as a guide to evaluate everything.  I use them as a quick sanity test for every practice that I am using or thinking of using.  However, I also recognize that I am the inventor of the process I follow, and therefore use the 3 Maxims to keep from getting off track. 

Using Maxim 3, I also question the value of the Maxims and the Agile Manifesto/Principles themselves.  It is a vicious, never-ending continuous loop of vicious, never-ending continuous loop of… – so I don’t get too serious about it.  Whenever my brain starts bleeding I take a break, let things coagulate, and then get back to it.

Also, the 4 values, 12 principles, and 3 maxims are not written in stone.  It is likely that there are other important missing maxims/values/principles, and there might be errors and partial omissions.  But this is a living thing – adjustments are allowed – cells split, some things become vestigial – and so on.  Complete re-writes are allowed.  Nothing is sacred.  Continuous improvement must include improving the things we improved previously and think can’t be improved.  Throwing some things out and letting other things die off is okay.

The talk took less than an hour, plus about 1/2 hour for discussion.   Not bad for solving all the problems that ever were or ever could be.  Well, no one else saw it that way, but this is my blog and I can make up anything I want.

However, what I was trying to show is that my current thinking (which is based on real, every day cranking out of healthy, usable functionality) is that:

• most development efforts are chock-full of harmful practices that we hold sacred yet are massively wasteful or even counter-productive
• that we are the cause of most of the problems
• and there is a way to make things better.

I won’t cover the rest of it here unless enough people actually ask for more.  That has never happened yet, as I don’t think anyone reads this blog, and most people who attend my talks spend most of the time trying to find a way to respectfully withdraw. 

The most important thing:

For me, acceptance of Maxims #1 & #2 gives a clear way to evaluate 90% of the badness I have encountered in software development.  They are on an equal with the Agile Principles in the way I use them.  Maxim #3 keeps me honest about Maxims #1 & #2.  I have another 3 or 4 slides on Maxim #3 that guides the questioning process.  Perhaps I’ll write about that sometime soon.  It is not the 5 Whys, and I don’t think there is a Japanese word for it either.

There you go. 

Have a nice day.