In this episode I’ll cover an extremely satisfying application of Lean and Agile on a large ‘Legacy’ effort I was part of a while back.<\/p>\n
Please understand that I am not at liberty to discuss certain (that is, most)\u00a0details. I must protect the guilty. I will give just the bare details – only what is necessary. I bet you don’t believe I will be able\u00a0to give just the bare details, right?<\/p>\n
So – this is about using the No Estimate Approach on an End-Of-Life Legacy Support project.<\/p>\n
First, a few definitions:<\/p>\n
I am glad you asked.\u00a0 There are many definitions of legacy code. Here is the one I use: Legacy code is ANY CODE THAT IS DOING ACTUAL WORK THAT YOU WANT TO KEEP WORKING. (Please note the clever use of “all caps” for emphasis.)\u00a0 Does that make sense to you? Doesn’t matter – just pretend that is what it means.\u00a0 Using my definition, most projects become legacy code as soon as you have some functionality in production. That’s when the bits hit the fan, usually, so to speak.\u00a0 However, even if it isn’t yet in production, it could still be “legacy” if we have stuff working we don’t want to break.<\/p>\n
Regardless of what you think legacy support is,\u00a0for the purposes of this post I am using it to mean this: “Work done\u00a0(bug fixes and minor enhancements – that sort of thing, or more serious stuff if needed)\u00a0to\u00a0extend and protect\u00a0the life of code that is in use”.\u00a0\u00a0 I’m leaving out more extensive enhancements like “whole, new functionality”.\u00a0 You can\u00a0judge for yourself the difference between minor and “more extensive” enhancements.\u00a0 Doesn’t really matter to me.<\/p>\n
For the purposes of this post, an End-Of-Life project is a project that is almost retired.\u00a0 It’s still in use but\u00a0going to be replaced sometime soon. \u00a0\u00a0In software programming “Soon” means “perhaps sometime, but we have no clue when.”\u00a0 The basic idea is that people are using it, and it is important enough to keep supporting, but we’ll stop doing big additions and improvements.\u00a0 Some bugs will just be ignored, but any deemed to be important to fix will be fixed if possible.<\/p>\n
The project that we are discussing is is a large, line-of-business set of applications (‘The System’<\/strong>)\u00a0that is critical to the users of the software for managing most aspects of their business.\u00a0 The typical user of The System<\/strong> manages thousands to millions\u00a0of their own customers – sales, billing, transactions of all sorts, bunches of “real-time” stuff, reporting, etc.\u00a0 Typical line-of-business stuff.\u00a0 I was told the code base was about 2 million lines of code [and\u00a0that’s about enough, in my opinion – any more is just bragging.]<\/p>\n The System<\/strong> has been in production for 15 years or so and has moved through several different code bases.\u00a0 The current code base is about 4 or 5\u00a0years old.\u00a0 It uses several languages – 4 or 5 perhaps.\u00a0 Typical stuff.<\/p>\n Enhancements and bug fixes are managed, worked on, and delivered using a waterfall approach, with a 3 month delivery schedule (that takes 6 months due to test and fix cycle, and other typical deadline issues).<\/p>\n Due to design, industry, and technical reasons it is deemed necessary to re-write The System<\/strong> in a new language with a new architecture.<\/p>\n The current version of The System<\/strong> MUST continue to serve the user base, and even more – it has to be clear that we are still\u00a0responsive to concerns and issues of the user base so they don’t start looking to competitors to solve problems while the “greenfield” version of The System<\/strong> is being developed.\u00a0 It has to look and feel like “business as usual, only better”.<\/p>\n A “maintenance team” is put together to provide this service.<\/p>\n I’m not sure exactly how many people were working on\u00a0The\u00a0System<\/strong>\u00a0before the End-Of-Life Maintenance, but it was something on the order of 20 to 25 people – maybe more, it was a while back and I didn’t keep track.<\/p>\n The new team was to be 6 people: 2 QA, 2 BA (Business analyst or “Product Owner”), 2 Dev.\u00a0 I was one of the two Devs, and also the manager, or co-manager\u00a0(more or less).\u00a0 I “volunteered” for this assignment on one condition: I would be allowed to manage this effort\u00a0my way – which means Lean\/Agile\/XP.<\/strong><\/p>\n We’ll call this System EOL<\/strong> (for system end-of-life)\u00a0.<\/p>\n There are approximately 500 active bugs in the bug tracking system<\/strong>.\u00a0 Some of these bugs have been pernicious. Some have never been addressed due to\u00a0level, some have been introduced in the fixing of other bugs, some have been there a long time but just recently discovered – All the typical bug stuff.<\/p>\n There are 500 active bugs!\u00a0There are a number of reasons there are so many bugs.\u00a0 Before we talk about that, I’d like to ask a question.<\/p>\n Got that? I’ll allow for any bugs that have easy work-arounds – you can leave those if you have to.\u00a0 I’ll also allow for any bugs that were discovered since the last deployment.\u00a0 But you better fix those NOW and then you’ll have zero bugs once again. Nice.<\/p>\n Remember that: Zero bugs.\u00a0 Does that seem unreasonable to you? If it does, I’ll ask another question: Why bother reading my blog? Go do something else that seems reasonable to you.\u00a0 Why waste your time on what I’ve got to say? I am NOT REASONABLE most of the time.\u00a0 I want a better way to do stuff than whatever we used to think is “reasonable”.\u00a0 Zero bugs is a “lofty goal” in some workplaces.\u00a0 In others, it is business as usual. I prefer zero bugs.<\/p>\n So, 500 bugs is not just way too many, it is infinitely too many.\u00a0 Or whatever 500 divided by zero is.\u00a0 Anyone know?<\/p>\n There were several reasons all these bugs existed, but the one that is most pertinent to our discussion today is the process that was in place for assessing and fixing bugs.\u00a0The way things were being done was\u00a0about like this:<\/p>\n Okay – you get the point.\u00a0 I am not making this up.<\/p>\n There were other reasons System EOL<\/strong><\/strong> was in this state, which included lack of unit tests, code neglect (huge methods, overwhelming complexity, fragility, coupling and cohesion problems),\u00a0and so on… but we’ll solve one thing at a time.<\/p>\n Well.\u00a0 Does that remind you of that game you used to play when you were a kid?\u00a0 It should.\u00a0 And it is JUST AS FRUSTRATING, at least conceptually.\u00a0 Only difference is you get paid to play Bug Fixing Chutes and Ladders.\u00a0 But overall – it is terrible and horrendous (and other bad words).<\/p>\n Since I was allowed to manage System\u00a0EOL<\/strong> in my own way – Here is what we did.<\/p>\n First day I brought in 6\u00a0copies of\u00a0Mary and Tom Poppendiecks “Lean Software Development” book, and we all agreed to read it, and more or less follow their basic concept.\u00a0 I’d been doing XP for years (more or less) but it’s a hard sell so I opted for the “easy sell”.\u00a0 If Lean is good enough for Toyota, it’s good enough for me.<\/p>\n First\u00a0thing we did was a value stream map on the \u00a0“Chutes and Ladders” methodolgy that was in use. If you haven’t seen a value stream map or know how to do one, take a look at Mary and Tom Poppendiecks “Lean Software Development” book.\u00a0 The basic idea is that anything that is not directly adding value can (and should) be eliminated. For example: waiting in queues is a waste – it brings no value to the person interested in having the bug fixed.<\/p>\n I bet you can guess that we found a lot of NON-VALUE-ADDING behaviors.\u00a0 So, we made a few rules.\u00a0 For example…<\/p>\n NOTE: No Estimates, No prioritization, No meetings.\u00a0 These are wastes (at least in some situations, if not a lot)\u00a0– bringing no value to the “customer”. Even worse,\u00a0 waste usually removes value by using up time and “spirit”.\u00a0 A team doing a lot of meetings, estimates, assessments, meetings, prioritization, meetings, waiting, etc.\u00a0becomes NUMB. Avoid numbness.<\/p>\n We quickly read each\u00a0bug in the tracking system,\u00a0looking for duplicates, bugs no one had asked about in a year or more, and bugs\u00a0where\u00a0the description\u00a0could not be understood (and there was no one who could clarify).\u00a0This was done very quickly and eliminated about one half of the bugs.\u00a0 NICE. 500 bugs cut down to 250 bugs.\u00a0 No one can say that is not progress!<\/p>\n This took about a week (as I recall).\u00a0 During this “first-week” the developers and QA took on bugs to fix, since there were plenty that were clear enough to work on – but spent some of their time helping the BA’s “weed out” the bugs that we removed.<\/p>\n Guess how long it took to clean this project of\u00a0 bugs.\u00a0 Give up?\u00a0 3 months!<\/p>\n 250 bugs in 3 months – that’s about 4 bugs a day!!! Well, we did have 17 “un-fixable” bugs.\u00a0 Those were bugs that we weren’t allowed to fix for some business\u00a0reason.\u00a0 So still – an average of\u00a0just under 4 bugs a day.<\/p>\n This is the power of eliminating waste.\u00a0 All of our time was spent doing meaningful work, REAL WORK on bugs. No reviewing, no estimating, no “pre” investigation that wasn’t part of doing the fix, no passing code to QA that we were not confident would pass all tests, etc. In other words:\u00a0NO WASTE.\u00a0 Or at least: We were no longer doing any of the steps that\u00a0we had identified as waste.<\/p>\n Well, what we found is that in fixing some bugs, we’d incidently fix other bugs.\u00a0 That is, some bugs were caused by a single underlying malfunction.\u00a0 That happens, and it is a sweet bonus when it does.<\/p>\n We had no recidivism. Once delivered to the customers, we\u00a0did not have a single fix that was “sent back”.\u00a0 Once fixed it stayed fixed.\u00a0 I think this can mostly be attributed to the QA and Dev working together, the unit tests put in place, and the “one-at-a-time” approach to “fix and build.” We would fix ONE BUG at a time.\u00a0Fix,\u00a0 Build, Unit Tests, Checked in, Built, Tested, confirmed.\u00a0 Only then would next bug be tackled.<\/p>\n So… this\u00a0is about no estimates.\u00a0\u00a0Our rule: Eliminate waste.\u00a0 In this case, estimates were a waste.\u00a0 They were thought to provide a way to “manage” the work being done, but were in fact reducing the ability of the organization to “manage” getting work done.\u00a0 I think they are like the dice in chutes and ladders? Probably not. Still, focus on the real work, don’t muddle things by playing chutes and ladders.<\/p>\n","protected":false},"excerpt":{"rendered":" In this episode I’ll cover an extremely satisfying application of Lean and Agile on a large ‘Legacy’ effort I was part of a while back. Please understand that I am not at liberty to discuss certain (that is, most)\u00a0details. I must protect the guilty. I will give just the bare details – only what is […]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[3,21,22,24,4],"tags":[],"class_list":["post-302","post","type-post","status-publish","format-standard","hentry","category-agile-stuff","category-estimating","category-no-estimating","category-noestimates","category-waterfall"],"_links":{"self":[{"href":"https:\/\/zuill.us\/WoodyZuill\/wp-json\/wp\/v2\/posts\/302","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/zuill.us\/WoodyZuill\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/zuill.us\/WoodyZuill\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/zuill.us\/WoodyZuill\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/zuill.us\/WoodyZuill\/wp-json\/wp\/v2\/comments?post=302"}],"version-history":[{"count":33,"href":"https:\/\/zuill.us\/WoodyZuill\/wp-json\/wp\/v2\/posts\/302\/revisions"}],"predecessor-version":[{"id":972,"href":"https:\/\/zuill.us\/WoodyZuill\/wp-json\/wp\/v2\/posts\/302\/revisions\/972"}],"wp:attachment":[{"href":"https:\/\/zuill.us\/WoodyZuill\/wp-json\/wp\/v2\/media?parent=302"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/zuill.us\/WoodyZuill\/wp-json\/wp\/v2\/categories?post=302"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/zuill.us\/WoodyZuill\/wp-json\/wp\/v2\/tags?post=302"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}The Path Going Forward<\/h4>\n
The State of System EOL \u00a0on Day One<\/h4>\n
500 Acitve Bugs???!<\/h4>\n
\n
One reason there are so many bugs: The old process.<\/h4>\n
\n
Chutes and Ladders<\/h4>\n
What to do???<\/h4>\n
Value Stream Map<\/h4>\n
First of all, a few rules:<\/h4>\n
\n
An initial assessment:<\/h4>\n
Daily process we adopted:<\/h4>\n
\n
Release Schedule:<\/h4>\n
\n
End Result:<\/h4>\n
A Few Other Interesting Things:<\/h4>\n
Conclusion<\/h4>\n