Life, Liberty, and the Pursuit of Agility

Requirements Gathering

In the traditional phased SDLC approach, a phase for  “Requirements Gathering” is done very early on in the process.  We can argue about all the issues with this some other time – I am just covering one issue here.

My opinion of this “phase” is that it is pretty much worthless: THERE IS NO WAY TO PROVE THAT THE REQUIREMENTS WE’VE GATHERED ARE COMPLETE AND CORRECT UNTIL WE HAVE WORKING CODE.  Working code, in this sense, means code that is actually being used to do real work for the real users. Does that make sense?  If not, you probably don’t want to read on, and probably shouldn’t bother reading anything I have ever written or will someday write.  Plain and simple.

Requirements Hunting and Gathering

One issue is that requirements will rot if not used promptly after collecting them.

Early humans are thought to have used a subsistence strategy where they hunted and gathered (and scavenged and stole) their food for immediate use.  If the food was not eaten shortly after it was obtained, it would rot or otherwise become useless.  The food needed to be put to work within minutes or hours, or at most a few days.  Otherwise, the effort of hunting and gathering would go to waste.

Just like the early hunter and gatherer folks, we have no way to “preserve” the requirements we’ve gathered.  We don’t even know if they are edible.  And yet we store them for a long time, and do a lot of work with them during that time, and base a lot of decisions and work on them (like estimates, budgeting, designs, architectures, ramping-up efforts, coding, testing, marketing, and so on) and all the time the requirements are rotting.

It isn’t until the code becomes true working code that we can get real information about the correctness and completeness.  Until then everything is just assumptions and guesses.  Unfortunately, by the time we learn that the requirements have indeed rotted it is too late.  They have become toxic, wasting our money, time, and effort, and killing the usefulness of the project they are supposed to define.

Early Humans understood this, and therefore rarely wrote software, and never ate mayonnaise that had not been “refrigerated after opening”.  Fortunately for Late Humans (us, that is), someone eventually figured out how to preserve foods so we could buy and eat them days, months, or years and years after they had been plucked, prepared, and packaged.   However, to this day, I have seen no way to preserve requirements.

Requirements MUST be used while fresh

So, my recommendation is: Eat your requirements as soon after you have picked them as possible.  Between the garden and the kitchen they have already started to lose their sweetness.  [Please imagine that there are many other very funny food based metaphors here.  I am not having trouble coming up with them as such, they just don’t seem all that meaningful, and they aren’t as funny as I thought they would be].

How do you do this with software?  You use a very short “discover, define, code/review/re-discover, deliver, discover more, repeat” approach during which we are continually discovering what it is we really need to do:

1. Keep your requirements at a very high and general level until just before use.  Details in the requirements are a smell of the rot.  Pay attention to that.  Understanding the business need is important, details about how that translates into software is not.
2. Pinch off a small but useful bit of one of the high level requirements – preferably something of immediate and high value to the end user, but anything of value will do.
   • Find something that you have a pretty good chance of turning into working code quickly – in a day or two if possible.
   • Prune as you go as needed.
   • Identifying small bits, pruning as you go, and getting something of value quickly are skills you must aquire.  It takes practice, dilligence, and close attention.
   • You don’t have to understand the requirement completely – you will discover the details and what is important as you code it up while partnering with the business experts/users.
   • Prioritizing is useful – you want to take on features in a prioritized manner, but don’t go overboard on this – close enough is close enough.  We’ll talk about this some day.
3. Code it up.  The code-it-up process is not just coding – we must work closely with the business people discovering the details as we go.
   • Code some tiny bit the business expert can see and use (click a button, see a list, choose an option, whatever)
   • Get her feedback, comments, and direction continuously – early on and frequently – every 10 minutes to 2 hours.  Every time you have something meaningful you can show get it in front of the business expert.
   • Sitting side-by-side works well when appropriate, but if that isn’t practical at least be within shouting distance.
   •  If you go longer than 4 hours without getting useful feedback you are wasting your time and need to learn how to work in smaller chunks.
   • You might discover you have completed some small but deliverable bit of funcitonality that is just a part of what you had chosen to work on.  If so, then deliver it, and go on to the next bit.
4. Deliver it.  That is – put the code into actual use with actual users.
   • You’ll discover more things about it as the users use it.
   • This will help expose the next thing to work on.  The real next priority.
   • If they don’t use it once delivered, you’ll have learned something else of value: It wasn’t of value. I’ll cover that some other day – but that is a magical and very valuable bit of knowledge to gain early.
5. Do it again tomorrow (or today, I suppose) with another bite-size chunk of functionality.
   • You are now steering the project toward the best possible result given the abilities and knowledge you and your team have.
   • Important things will become apparent and be added to the working code in a prioritized manner.
   • Useless things will show themselves or be ignored and you’ll waste as little time as possible on them.
   • You’ll get better and better at it.

Following this approach the requirements will not rot because you leave them “on the vine” until you are ready to eat and digest them.  Of course, what comes out after the eating and digesting is not covered in this article:  At some point all analogies break down.

[NOTE: Everything I say in this post and on this blog is based on my experiences, study, and observations.  It is MY OPINION. It is the way things seem to me.  It might be useful to you, might not – I don’t really care.  If you want to argue about it, that is fine.  I am suggesting that there are better ways to do almost everything we do in software development, and that it is worth thinking about it and doing something about it. You might already have better ways to do things, and I want to hear about it.]

I’ve signed up for a few sessions at the SoCal Code Camp at USC in Los Angeles.  The dates of the camp are October 15th & 16th, and I will probably be there both days.  There are about 87 sessions listed so far, so you are sure to find something of interest.

Here are the sessions I am doing:

• Coding Dojo Part 1 – Hands on “mob” programming
• Coding Dojo Part 2 – Hands on “mob” programming
• Code Excellence Part 1 – Demonstration of cleaning, refactoring, and improving working code
• Code Excellence Part 2 – Demonstration of cleaning, refactoring, and improving working code
• Agile Success – An Introduction – An interactive discussion of how to make Agile a success for your project

I hope to see you there… whoever you are.  Let’s talk Agile and write some code.

Losing Interest?  Refill Your Cup Before it is Too Late

A boss of mine once said to me “I wish I had your passion and interest in coding – I lost that a long time ago”.  I think that is kind of sad, especially if you make your living making software.  I don’t think I could spend each work day doing something I was bored with – but of course, a lot of people do that, and I’ve worked with a lot of programmers who have lost interest in programming.  Seems they were once interested, and now they no longer are. 

Perhaps I’m just naturally interested in programming and can’t help myself, or maybe I have a mental disorder – but I seem to be able to stay engaged and still enjoy programming every day. I think part of the reason I keep my interest is that I continuously “refill the cup”.  When I sense I am starting to loose interest (or even long before that) I look for a way to refill the cup. 

One note: I don’t really think it is ever “too late” to regain interest.

I’d like to share a couple of ways that work for me: Intentional Practice/Effortful Study, and Programming as a Social Activity.

Intentional Practice and Effortful Study

My friend Jason Kerney did a talk on Intentional Practice at the San Diego .NET Development Group back in March.  I have a lot of respect for Jason.  He has an infectious passion for programming, clean code, and continuous improvement and I always learn a lot from him.

I had talked with Jason about this idea and like it a lot:  It’s important for us programmers to intentionally practice our craft.  I remembered reading an article on the topic somewhere, and tracked it down at Coding Horor: http://www.codinghorror.com/blog/2008/06/the-ultimate-code-kata.html. In this article from 2008 Jeff Atwood quotes an article by Steve Yegge (from 2005) called Practicing Programming.  You can read this article at http://sites.google.com/site/steveyegge2/practicing-programming. Steve refers to some other articles, and I think if you follow the references in those articles you eventually end up at Kevin Bacon – but you know all about that.  With all this great stuff already written about practicing our coding craft I won’t have to regurgitate that here.

Anyway – Jeff quotes a Scientific American article that uses the term: “Effortful Study”.  I like that, and suggest you read that article too, and then tell me what it is about.

I’ve been meeting up with a few friends at work, at code camps, and remotely during the last few years to intentionally practice programming.  We engage in effortful study to focus on very simple things that lead to code excellence. I have been learning a lot.  Hovever, even more imortant to me is that I feel rejuvinated after these study sessions.   Some of the Katas have been very simple, others have been very complicated.  Sometimes we move quickly to a good solution, sometimes we flounder about and don’t get anywhere.  Regardless, I have a great time interacting with my fellow programmers and find I’ve refilled my cup.  And that is the concept I call:

Programming as a Social Activity Just for the Fun of it

People get together to watch sports, go on bike rides, have dinner, go to movies, play games, get drunk, and lots of other fun stuff JUST FOR THE FUN OF IT.  Programming can be done the same way.  Two people or a whole group of people.  You can do simple Kata exercises, complicated code jam style contests, work on some Open Source project of mutual interest.  Whatever.  A friend of mine (the brilliant and dynamic Llewellyn Falco) used to have “Monday Night Programming” get togethers every week at his house.  What a great idea!  

I don’t know how or why this works for me, but it does – every time I do one of these code dojos or attend a coding gathering, or some similar activity it definately stimulates my interest in the work I love to do: programming. 

It might not work for you, but I love it.  Next time you run into me let’s sit down, open up the laptop, and code something – what do you say?  Come on, it could be fun.

I Like to Hike

I like to hike.  I hike whenever I can.  I prefer the trails and paths and wilderness, but I’ll hike on the roads and streets when I have no other option.  There is not much real wilderness around where I live anymore, but there are a lot of places that are remote enough that they seem wild.  Good enough for me.

I was out hiking today on the Coast to Crest Trail with my wife and our dog.  I’ll call him Rex, but that is not his real name (he’s asked me not to use his real name).  Rex is a great hiker, and at 9 years old he is still anxious to get out on the trail and can out-hike most.

We got about a mile up a pretty steep trail and Rex started favoring his left hind leg.  We’re not sure what happened, and he wouldn’t tell us.  We suspect he was bitten by a red ant since we couldn’t find a thorn or a cut or any other obvious issue – but we did notice a bunch of red ants on the trail.  Whatever it was, we decided to turn back and call off the hike while Rex was still in good enough shape to hike himself out of there.  Rex weighs about 45 pounds, and if he couldn’t make it under his own power I’d have to carry him.  I could probably handle it – but it would be very slow going.  Also, the whining and complaining is something neither my wife nor Rex would care to put up with.

As we headed back down the trail, I was thinking of other hiking I have done, and the risks and possible trouble you can get into out on a hike,  which eventually reminded me of a saying they have at the Grand Canyon:

Hiking the Grand Canyon: Getting to the Bottom – Optional. Getting Back to the Top – Mandatory

I’ve hiked in the Grand Canyon to the Colorado River and back out four times, including a “rim to rim” from the North Rim down to the Phantom Ranch and the River then up to the South Rim.  It is a lot of fun, and even though there are a lot of hikers out on those trails, it’s not hard to get a sense of wildenerss in that spectacular setting.  Some trails are lightly traveled and you can get very remote and lonely if you like.

These Grand Canyon hikes are “backwards” compared to most I’ve done – typically, we start at the bottom of some mountain and hike up – and once all the hard uphill work is done, we can come back in a relatively easier downhill mode.  At the Grand Canyon, we start at the top and hike down and take pride in our tremendous pace.  When it is time to return, it is all uphill and we start to remember how “pride cometh before a long trudge up”.  That is the way they built the place.

From the top of the Bright Angel Trail on the South Rim you can see many of the fabled milestones and exotic destinations such as the Mile-and-a-half Rest House, the Three Mile Rest House, Indian Gardens, Plateau Point, and the Colorado River itself – you can even see the Phantom Ranch from some viewing areas.

All these interesting and enticing places don’t look all that far away. They seem very close, and they are if you are a crow.  But this is deceptive – the distance for the non-crow hiker is greatly extended due to the long grades and switchbacks necessary to make the trail possible to hike.  Things that look close are many trail miles away.

I’ve seen a lot of casual hikers sprightly scamper down the first two or three miles (or more) thinking it’s an easy hike, only to find it takes a great deal more time and effort to hike back out again once they turn around and head back to the rim.  Especailly if you are wearing flip-flops or other inappropriate footwear.  And worse things can happen.  On one trip I met a man near the river (met is a bit of a lie.  He was lying on his back in the middle of the trail almost unconscious.  It was more like “stumbled upon” than “met”).  He was suffering from euvolemic hyponatremia (look it up).  At least that is what I think he was suffering from.  Anyway – he was suffering.  He should not have been down there.  We helped him with some electrolytes and salty snacks and then helped him slowly make the couple of remaining miles to the Phantom Ranch. He recovered, but had a hard several days.  As a side note he was hiking with his ex-wife.  Now THAT is interesting.

Another trick of the Canyon is that as you hike down, the temperature increases.  At the rim where it might be 65 degrees on the very same day it can be 100+ degrees at the river.  (On one day I was there it was 120f at the river and 70f at the rim.  Dang! ).   What starts out as a pleasant, cool, and easy down-hill hike turns into a stressful, hot, dry, and very lengthy up-hill trudge.  Once you have used up your water, water starts to seem more important to you.  And don’t forget the electrolytes (remember the euvolemic hyponatremia?)

There are about 400 rescues per year in the Canyon (not all are hikers, but a lot are), and they only do rescues for the really critical cases such as heat stroke, serious injuries, and other medical emergencies – a lot of hikers get rescued by other hikers who share their water, food, gear, time , and electrolytes.

The “backwards” nature of the hike (Down then Up, Cool then Hot, Seemingly Short, Actually Long, Easy then Hard) lulls hikers into going too far down the trail.  It all seems so easy and fun at first.  By the time you discover how HARD hiking back out is, you’ve already used up all your food, water, energy, happiness, and other resources.  You’ve spent everything and now comes the real work – hiking up and out of the Canyon in the heat.  And oddly, the distances are even longer hiking up than hiking down.  Funny how that works.

The reason to hike in the Grand Canyon (or anywhere) is slightly different for each hiker: Recreation, fitness, adventure, or whatever – [Note: in the world I live in we rarely hike out of necessity anymore.]  In the Canyon, a lot of people also have a goal to get to one or more of the milestones or well known points of interest, or all the way to the river.  But the real goal for every hiker has to be to get out alive and in reasonably good condition (aches and pains are acceptable, I suppose, but dehydration, heat exhaustion, heat stroke, and alarming leg swelling aren’t so nice) .  Getting out is Mandatory – All other goals are optional.  They have warning signs all over the place at the Canyon about the risk of misjuding your skill, ability, and the difficulty of the environment.  Still, many hikers end up in trouble.

What The Heck Does This Have To Do With Software Development???

You probably hoped I would forget to drag Software Development into this post, but…sorry.  Here goes.

I’ve seen this same basic scenario in Software Development.  The goal, the abilities of the organization, the apparent short distance to enticing and desirable “points of interest”, and the risks are all often misunderstood.  All the available money, time, and effort have often been expended by the time the real work of coding/testing/deploying begins and we realize how difficult everything is and how far we’ve gone down a winding, narrow trail we can no longer get back up.  And even worse, we have nothing useful to show for it except documents, diagrams, designs, and some code that can’t climb out of the test-and-fix cycle.  Whew.  Glad that is over.  Now – your assignment – go back over this post and equate all that stuff about hiking to some software development effort you have worked on.

I’ve seen and heard about a LOT of backwards, risky, hot, long, painful, death march trudges – Lucky for me I have worked on only a few of them.  Just remember – there is a way to work that isn’t backwards.  Let’s do that instead.

This turned into a long post. I think the corollaries are obvious.  Maybe they aren’t.  Maybe I am just full of it.

So, have a good night.  Tomorrow go do something fun – and stop wasting your time reading stuff on the Internet.  Go out hiking instead.  And by the way – hiking the Grand Canyon is FANTASTIC.  I recommend it to anyone still healthy enough to get out and hike – thousands of people hike the Canyon each year without problem.  Bring some electrolytes or at least some salty peanuts and cheese crackers or something.

Cheers!

What I did last Thursday and Friday

Last week I was able to spend two days at the AOSoCal 2011 event with 70 other Agile enthusiasts who are all practicing, thinking about, coaching, consulting, and working with Agile Software Development.  I had a great time, learned a lot, made a lot of friends, and can’t wait for the next one.

Like I said several times at the event, I felt surrounded by friends the whole time.  These are the sort of people I would hope can attend my funeral to balance out the other folks who will be there to gloat and trash-talk me.  Do people still say “trash-talk”? 

A little about Agile Open, and Open Space

Each Open Space event has a theme that defines the purpose of the gathering.  Our theme for AOSoCal 2011 was People, Principles and Performance.  That’s pretty wide open.  However, unlike many typical conferences, an Open Space event is invented by the people who show up. The sessions are not scheduled until the morning of the event. 

Everyone gathers in a circle, and the participants propose session topics by writing the topic on a piece of paper and announcing it to the gathering.   Typically, the proposer takes responsibility for the session, choosing a time and space to hold the session, and later facilities the session.  One by one the event schedule it built, and once that is done everyone selects the first session they will attend… and then off we go – the Space has been Opened.  Sessions can be added at any time, and some get combined or morph into something completely different.  Overall it is a very dynamic process.

You can learn more about the process at Open Space World, or at the Agile Open site, or the Agile Open California site, and about the AOSoCal event specifically at the AO Southern California site.

So what is it really like?

From my description it might be hard to get a picture of how this all works – but it does work, and rather nicely.  I encourage you to attend an Agile Open event – they are generally very inexpensive and then you could easily experience it for yourself.  All the people I’ve talked to who have attended these events have been very pleased they were able to join in. 

I have been to a variety of Open Space events and the details vary from one another – but overall I get a lot more of what I want from a conference than I ever get at the more traditional conferences.  One thing I have especially enjoyed is that the sessions are engaging, personal, and interactive – I find this keeps me energized throughout the entire two days.  Most sessions keep things lively using facilitated (or non-facilitated) discussions, hands-on activities, or interactive techniques of some sort.  I love that I get to spend a couple of days continuously interacting with fascinating people who are discovering, learning, experimenting, living, and inventing what Agile Software Development is all about.  How is that for a poorly formed sentence?

In addition, there are endless opportunities to just sit and talk, pair program, or take a walk (or whatever) with other attendees.  Overall, a lot of fun.

So, that is what I did last Thursday and Friday

Cheers!

Written: May 26, 2011

NOTE: This article relates to the domain of software development.  Nothing else.  Also, the idea of failing early, fast, often, and better does not apply in all situations.  Use with caution.  However, it has been very useful for me in software development, and I can recommend that you try it out.  Otherwise, leave me alone, okay?

Failure and Mistakes are a Valuable Part of Software Development

We are going to make mistakes, misteps, and “failures”.  It’s part of the process.  We need to recognize failure as a valuable learning and steering opportunity.  Here are a couple of quotes on mistakes/failures:

“Mistakes themselves are often the best teachers of all.” – James Anthony Froude –

“I have not failed. I’ve just found 10,000 ways that won’t work.”  – Thomas Edison –

That is pretty clear, I think.  If we buy into the idea that failures can bring value, we might want to focus on ways to increase that value – it’s just another application of the concept of “Extreme Programming” – if something brings value, let’s figure out a way get the maximum value for the time and effort spent.  Make sense?

Fail Early

If we believe that we can learn from our failures then let’s start the learning.  The sooner we have a chance to fail, the sooner we get the benefit of the learning that comes with it.

How do we fail early?  Take action and make something small (but useful) right away, and immediately deliver it to the the customer (or user, or whoever) and get real and useful feedback.  Dig in, do something of value, see what works and what doesn’t, learn from it, adjust our course, and march on.

I have seen this quote of Napoleon about his basic military strategy: “On s’engage et puis on voit” which I am told means “I engage and then I see”.  This might not work in all cases but has worked very well for me in software development.  I shorten it to “Engage And See”.   [N.B.: Unlike Napoleon, please attempt in all cases to avoid a disastrous winter campaign in Russia.]

Fail Fast

We want to fail fast so we can learn in a matter of minutes, hours, or days rather than weeks and months (or never).  There are lots of ways to approach this in software development.  One great example which can give us “instant” results is TDD (Test Driven Development/Design).  Before we write any code, we write a failing (and very small) test.  Of course, there is more to it than that – but for my purpose in this article, the key is “VERY SMALL WITH INSTANT FEEDBACK”.  With TDD, the test fails immediately, which is about the shortest feedback loop as we can have.  Once the test is working, any time we work on our code we will run the tests and learn immediately when we have broken something that was previously working.  Instant feedbcak.

This instant feedback is great, but we can’t get instant feedback for everything.  However, what we CAN do is find a way to take very small steps that lead to useful feedback as quickly as possible for everything we do.  For example, the “business people” should work directly with the programmers and round-trip each step/task/feature continuously – let’s work to get this feedback in a few minutes rather than having a meeting every two weeks. You get the idea.  For every aspect of the process we want to minimize the time it takes to discover we have a mistake.

Fail Often

Once we are good at the failing/feedback/learning loop, we can ramp up the learning by failing a lot.  The more things we try, the more failures we might have, the more chances to learn and steer.  We are NOT after failure for failure’s sake.  We ARE after the good things we can learn by trying things that stretch our abilities and understanding.  Another benefit is that we waste less time working on the wrong thing.

I’ve often seen it that a little step towards what the user needs will lead to a change in direction to a better result for the next “chunk” of work.  Sometimes it can lead to a complete about-face where the customer all of a sudden realizes they want to go in a completely different direction, which is also very valuable.  And of course, they most often say “that is great!  it’s exactly what we wanted”.  Well…

Anyways – if one failure is good, lots of failures can be better.

Fail Better

So: early, rapid, and frequent opportunities to fail is good.  All that is left is to find ways to maximize the results of each learning opportunity.  Inspect and adapt.  Here are a couple of ideas:

Finding better ways to communicate feedback is one area worth improving.  For example, sitting side-by-side with the customer to do a hands-on walk-through of a new feature might be (and generally is) more productive than a “shared screen” online session, and is way better than traditional demos.  Being with a real user while they really use the project is the best.   Try it.

Another area we can explore is how to expose hidden requirements.  These are the things that in traditionally managed projects the “requirements gatherers” and customers have trouble capturnig in a document and eventually lead to BIG failure at the end of a long death march style effort. Fun! One approach we can follow is to “take a stab at it”.  If there are things we don’t understand we can often get clarity quickly by coding up what we currently understand and getting it into the hands of a user.  This will often result in something that is close to (but not quite) what is needed – and will allow us to discover the right path.  I am talking about doing real code that does real things for the purpose of showing us where we have a “failure to communicate” – clarity ensues.

Conclusion

Anyways – these are just a few thoughts that will hopefully lead to further thinking and improvement in our ability to make progess through failure.  That could be my new motto: “Progress Through Failure For A Brighter Tomorrow.”  Well, probably not – but… you get the idea.

One Last Quote:

“Agile doesn’t prevent us from making mistakes, it gives us a way to reduce the cost of those mistakes.” – Llewellyn Falco, paraphrased