You Can’t Test Documents for Correctness or Completeness

[This is part of my series on the Communication Problems with Waterfall]

A common flaw of many develpment documents: You can’t test them.

It is a glaring flaw of the typical document generated in software development that there is no way to know if it is correct or complete.   We can do our work diligently and honestly and still end up with a lot of mistakes, missing pieces, extra (unneeded) pieces, and misunderstandings.

It would be nice to be able to “click a button” and prove that we have collected all the requirements, and that they are exactly what the customer needs and/or wants, or even that it is what the customer THINKS they need or want.  We can’t test an ER Diagram to prove that it will result in a database that stores everything we need and nothing more than we need, and allows us to create, retrieve, and use data with all the qualities we expect.  We can’t test object models and sequence diagrams (as far as I know, anyway) to prove our design is useful or good.  Well… you get the idea. 

All of these diagrams and documents are helpful in getting us to think about whatever it is we are working on, but like they say, the proof of the pudding is in the eating, and we don’t get to eat the pudding until late in a project if we are following a waterfall approach.  By the time most real problems are discovered it is very expensive and over-whelming to put things right.  This is part of the “Fail-Late” feature of the Waterfall: In Waterfall, risk and problems are by nature hidden and therefore get pushed forward to the following phase, and then the phase after that, until you have reached the end.  Eventually they are exposed, but not until it is too late to address them in an effective manner.  [I have about 3 pages of notes on “Fail-Late” that I will someday condense and post here if I get the chance.]

We try to gain confidence about our documents by being diligent.  We put processes in place to scrutinize them carefully.  We get as many eyes as possible to go over the documents and make sure everything is correct.  We have meetings where people discuss what we are trying to communicate in the document as we “hand them off” from one phase to another.  We have documents with audit trails built in so we can see what has changed over time.  We have change control boards and documented work-flow processes so we can follow and review and approve the changes, and so on ad nauseam.  No matter how careful we are, there is still a lot of room for mistakes and extra work, and even worse, the nature of the process provides opportunity to introduce even more mistakes.  And sadly, the mistakes and problems become compounded over time as complexity grows.

And still, there is no way to test these documents to determine their correctness or completeness. 

[Note: I am not speaking about documents that are actually used by the end user or in support of the end customer – such as training materials, help files, deployment instructions, and so on.  These are part of the deliverable to the customer and are an important deliverable of our development]

Finger Pointing and the “Solution”

Even worse, most of the mechanisms put in place to deal with problems when they come up are susceptible to becoming “finger-pointing” exercises.  These are the things like signing-off on a document, base-lining a deliverable for the next phase, review committees, and handing-off practices that projects using the waterfall usually follow.  I won’t cover those here and now, but perhaps in some future post.  The point is that at some point (usually very much later on) in the process we start discovering that something is wrong and we want to find out who is to blame.  Hopefully it is someone other than us.

The “solution” that is often suggested is to do more of the same sort of practices that are causing the problem in the first place.  After the cause is found we then look for a way to keep from making that mistake again – and that often means we add even more process and control into a system that is already over-burdened by process and control.

Restated: What is the real problem?

The Waterfall is a methodology that values these error-prone documents and work-flow processes.  However, since there is no way to put the documents into a test harness or automated user acceptance test suite to verify things are good (or to expose things that are bad) the results are often not acceptable.   Besides the inability to test these documents, the feedback loop is so long and ineffective that discovering problems takes a lot of time and only occurs after much of the work is done.  Once problems are exposed the cost of correcting them is high.

What is the solution?  Here are a few ideas:

  • Executable Documents: Strive to have documents that are executable tests – such as scenario, behavior, FIT, or story Tests,  or anything else you can click a button to run.  These tests can prove that what you are doing is correct and complete, or quickly expose mistakes and the lack of completeness. 
  • Buildable Code: We can test buildable code.  Therefore, lets get code as early as possible.  Preferably on the first day we know enough about ANYTHING the project is going to do.
  • Tests First: Don’t do work without first putting these tests into place, or at least create them concurrently with the work you are coding.
  • Continuous Integration: Once tests are passing, they must be kept passing.  This is one benefit of CI.
  • Involve the Users or Customer Early and Often: Get working code into the hands of real users or “user proxies” as soon as possible.  When a “user expert” and/or a Subject Matter Expert can actually see and use some small aspect of the software we can get feedback on what we are doing and then steer towards the desired result.  This is what guides us to real needs, the real requirements, quickly.  This is one form of the “feedback loop” that you hear so much about in the Agile world, and provides a discovery process that makes it difficult to leave something out while making it easy to verify correctness.  
  • Create Knowledge: Working, tested, and useable code is a knowledge creating tool that drives us towards a better understanding of what “correct” and “complete” means for the application we are writing.   Risk is often a lack of knowledge, so we can drive down risk by increasing knowledge.
  • Do Only Valuable Things: Try to do work the customer needs right now.  This means the code has a chance of getting tested in real world scenarios right away.  I have found that customers put off testing things that they don’t have an immediate need for.  This also keeps us from doing things the customer will never need.
  • Emergent Requirements Gathering, Design, and Architecture: Follow practices that allow your code and application to evolve, so the design can emerge as more is discovered.

One final note: We can create and test a prototype, and that can be very useful.  Although I rarely see prototypes being used, I like prototypes or spikes as a discovery and exploration tool, and they can easily be testable.  Prototypes introduce their own set of issues that we can discuss some other time. 


One Comment

  1. Adacelapkal:

    omg.. good work, bro

Leave a comment