{"id":127,"date":"2008-05-05T08:02:36","date_gmt":"2008-05-05T16:02:36","guid":{"rendered":"http:\/\/zuill.us\/WoodyZuill\/?p=127"},"modified":"2008-05-21T08:12:20","modified_gmt":"2008-05-21T16:12:20","slug":"you-cant-test-documents-for-correctness-or-completeness","status":"publish","type":"post","link":"https:\/\/zuill.us\/WoodyZuill\/2008\/05\/05\/you-cant-test-documents-for-correctness-or-completeness\/","title":{"rendered":"You Can’t Test Documents for Correctness or Completeness"},"content":{"rendered":"

[This is part of my series on the Communication Problems with Waterfall<\/a>]<\/strong><\/p>\n

A common flaw of many develpment documents: You can’t test them.<\/strong><\/p>\n

It is a glaring flaw of the\u00a0typical\u00a0document\u00a0generated in software development\u00a0that\u00a0there is no way to know if it is correct or complete.\u00a0\u00a0 We can do our work diligently and honestly and still end up with a lot of mistakes,\u00a0missing pieces, extra (unneeded) pieces, and misunderstandings.<\/p>\n

It would be nice to be able to “click a button” and prove that we have collected all the requirements, and that they are exactly what the customer needs and\/or wants, or even that it is what the customer THINKS they need or want.\u00a0 We can’t test an ER Diagram to prove that it will result in a database that stores everything we need and nothing more than we need, and allows us to create, retrieve,\u00a0and use\u00a0data with all the qualities we expect.\u00a0 We can’t test object models and sequence diagrams (as far as I know, anyway) to prove our design is useful or good.\u00a0 Well… you get the idea.\u00a0<\/p>\n

All of these diagrams and documents are helpful in getting us to think about whatever it is we are working on, but like they say, the proof of the pudding is in the eating, and we don’t get to eat the pudding\u00a0until\u00a0late in a project if we are following a waterfall approach.\u00a0 By the time most real problems are discovered\u00a0it is very expensive and over-whelming to put things right.\u00a0\u00a0This is part of the “Fail-Late” feature of the Waterfall: In Waterfall, risk and problems are\u00a0by nature hidden and therefore get pushed forward to the following phase, and then the phase after that, until you have reached the end.\u00a0 Eventually they are exposed, but not until it is too late to address them in an effective manner.\u00a0\u00a0[I have about 3 pages of notes on “Fail-Late”\u00a0that\u00a0I will someday condense and post here if I get the chance.]<\/p>\n

We try to\u00a0gain confidence about our documents by being diligent. \u00a0We put processes in place to\u00a0scrutinize them carefully.\u00a0 We\u00a0get as many eyes as possible to go over the documents and make sure everything is correct.\u00a0 We have meetings where people discuss\u00a0what we are trying to communicate in the document as we “hand them off” from one phase to another.\u00a0\u00a0We have documents with\u00a0audit trails built in so we can see what has changed over time.\u00a0 We have change control boards and documented work-flow processes\u00a0so we can follow and review and approve\u00a0the changes,\u00a0and so on ad nauseam.\u00a0 No matter how careful we are, there is still a lot of room for mistakes and extra work, and even worse, the nature of the process\u00a0provides opportunity to\u00a0introduce even more mistakes.\u00a0\u00a0And sadly, the mistakes\u00a0and problems become compounded over time as complexity grows.<\/p>\n

And still, there is no way to test these documents to determine their correctness or completeness.\u00a0<\/p>\n

[Note: I am not speaking about documents that are actually used by the end user or in\u00a0support of the end customer – such as training materials, help files, deployment instructions, and so on.\u00a0 These are part of the deliverable to the customer and are an important deliverable of our development]<\/p>\n

Finger Pointing and the “Solution”<\/strong><\/p>\n

Even worse, most of the mechanisms put in place to deal with problems when they come up are susceptible to becoming “finger-pointing” exercises.\u00a0 These are the things like signing-off on a document, base-lining a deliverable for the next phase, review committees, and handing-off practices that projects using the waterfall usually follow.\u00a0 I won’t cover those here and now, but perhaps in some future post.\u00a0 The point is that at some point (usually very much later on) in the process we start discovering that something is wrong and we want to find out who is to blame.\u00a0 Hopefully it is someone other than us.<\/p>\n

The “solution”\u00a0that is often suggested is to do more of the same sort of practices that are causing the problem in the first place.\u00a0\u00a0After the cause is found we then look for a way to keep from making that mistake again – and that often means we add even more process and control into a system that is already over-burdened by process and control.<\/p>\n

Restated: What is the real problem?<\/strong><\/p>\n

The Waterfall is a methodology that\u00a0values these\u00a0error-prone\u00a0documents and work-flow processes.\u00a0 However, since\u00a0there is no way to put the documents\u00a0into a test harness or automated user acceptance test suite to\u00a0verify things are good (or to\u00a0expose\u00a0things that are bad) the results are often not acceptable.\u00a0\u00a0 Besides the inability to test these documents,\u00a0the feedback loop is so long and ineffective that discovering\u00a0problems takes a lot of time and only occurs after much of the work is done.\u00a0\u00a0Once problems are exposed the cost of\u00a0correcting them\u00a0is high.<\/p>\n

What is the\u00a0solution?\u00a0 Here are a few ideas:<\/strong><\/p>\n