by Edward Campbell
I have written previously on partial adoption approaches to Agile/Iterative Software Development
Please see that article for a discussion of incremental approaches to adopting Agile/Iterative Software Development.
Let me apologize for skipping a restatement of the incremental adoption approach for Agile/Iterative Software Development techniques. Having been tasked with this very approach more than once in recent years and having grown in understanding, I am convinced that the best advice for most people is to pursue a full Agile/Iterative Software Development Capability project with a multi-season term and management commitment.
Without both of these conditions, I believe attempting to build a full Agile/Iterative Software Development Capability will fall prey to one or more of the failures I have witnessed.
This may seem extreme to some but is based on experience and understanding that was acquired at significant cost by myself and trusted professional associates.
Technical Debt is the progressive accumulation of rework required to bring a functioning, but less than reliable or unsustainable software system, to a sustainable level of documentation, reliability, testability, maintainability and consistency. This is often the result of years of, sometimes unintentional, neglect and mismanagement. The reasons for this are unavoidable outside organizational demands, downward pressure on budget and uncertainty or absence of longer term enterprise planning.
The role of technically inadequate management must be mentioned here. It is simply not sustainable to hide technical debt in a functioning system and attempt to address needed rework in a system by re-assigning new project resources to fix a failing older system under the guise of ‘other duties as assigned’. In the view of highly competent associates and from my own experience, this occurs not just frequently but as a matter of course in many organizations.
There are only a few ways to avoid undocumented technical debt in your next project: Work on an entirely new project or be severely underpaid as one of the maintainers of a newly deployed project with a long predicted lifespan. This could work if you are just starting your career and are enthusiastic about learning how large systems are built.
This is not to say that there are no resource openings on well-maintained up-to-date projects. Speaking from experience, well-maintained up-to-date projects have very low turnover. The developers, testers and operations people on such projects value the up-to-date tools, technical advancement and respect they have achieved. This also means that openings on such teams are infrequent.
To be sure this problem is widespread. Agencies of the US Government and US States have acknowledged publicly that they are dependent on antiquated systems, running undocumented, significantly patched, code bases that are decades old. All of the original developers are long gone. Knowledge of their critical code bases is spread unevenly across development, maintenance and operations staff.
The cost of replacing such systems is inflated by the need to first completely document what the current code base is actually doing and then implement and refactor that functionality in a newer, supported, language that will have abundant trained programmers available for the foreseeable future.
Commitment Abrogation is closely associated with Undocumented Or Unknown Technical Debt. Several professional associates have reported being recruited for, say C#/.NET, only arrive and be told they were needed for ‘a few weeks’ of help on Legacy Project X written in Obsolete Language Y.
If your first assignment is not what you were hired to do, it is just possible you will be ‘re-purposed’ repeatedly as management ‘discovers’ new undocumented and unknown technical debt.
One associate reported 13 partial or full team reassignments of more than a month to amend unforeseen requirements in maintaining the current system spanning multiple years. Needless to say, during this time the primary project was starved for resources and made insufficient progress. That did not prevent the actual obstructors of progress and concealers of technical debt from eventually killing the original project.
Beware this condition.
In most organizations, the computer service groups are like banks. They work well during times of normal demand for service. System wide exceptions, like all depositors of a bank showing up to withdraw their money at once, overwhelm them.
Service groups that provide consistently slow response to requests, do not have a defined network topology, do not have automated setup scripts for the kinds of systems they deploy, cannot safely firewall different classes of users, etc…, will likely not be able to provide support necessary to build an Agile/Iterative Software Development capability.
In short, if you did not have a working development box on your desk the week you started work that would build the application you were hired to work on, this should be a concern.
Slow, repetitive, burdensome approvals for software development tools, resources and access often requiring multiple presentations to achieve approval for free or low cost, low risk, open source tools.
This is also an extension a previous condition to be aware of: Inadequate Support. A separate development network with no access to the test or production networks eliminates many of the worries associated with open source, buggy or just plain worthless tools.
An example, Organization X expanded scope from the primary skills in their contract to include unsupported versions of previous languages, implementing new functionality in currently unsupported program code, porting unsupported code to a new operating system, new database systems and new security models.
Organization X management directed this scope creep over a multi-year period while repeatedly stating that in-scope work would resume ‘in a few weeks or months’. The direct result of this scope creep was insufficient time to complete the original contract deliverable.
In the best managed projects, the scope of the deliverable changes. Agile/Iterative Software Development is a capability constructed to handle such change. Each Iterative Development Cycle is conducted with an understanding of changed requirements, design changes needed to accommodate them, and any additional effort needed to implement. The final deliverable is adjusted by the Project Manager and Product Owner to balance delivered features and team productivity. Although this may require compromise from the user community, such balance results in significantly higher implementation and deployment rates.
At best, projected deliverable completion dates are threatened as the number or rate of changes increases. At worst, deliverables that meet all final requirements are put out of reach by date certain deadlines, if changes requirements are not
However, no approach will handle changes in scope that rise to the level of changes in skill sets. It is beyond the boundaries of good management and efficiency that you hire craftsmen of one skill and reassign them to build a product of a completely different skill set.
Sometimes, unscheduled rework is made necessary by failure to document existing systems, networks and a void of knowledge about the inventory of running production software. When this happens, inevitably significant resources must be diverted from other tasks to first documenting existing systems and then correcting any failures in meeting functional requirements before redeployment.
This puts management in the position of having to acknowledge and accept past misallocation of resources that has resulted in undocumented and unknown deficiencies in current production software. Hierarchical organizational structures do little to encourage responsibility for errors bubbling upward, particularly when information workers at the lower levels cannot report actual conditions and events that affected their ability to produce.
An example, Organization Y executives and those of other organizations simply mischaracterized lack of progress on primary project tasks as failures, despite actively diverting team time and resources to address technical debt rework made necessary by years of willful disregard and neglect.
This section outlines approaches for inventorying barriers to Agile/Iterative Software Development present in your situation and workarounds that might help in your efforts.
Here we will try to inventory the obstacles you face in implementing an Agile/Iterative Software Development capability. Our goal is to appraise risks and arrive at a set of conditions that realistically improves your chances of success with minimum impacts on the existing organization.
Appraise each risk according to the following scale:
1 – Handled by proper tools and process definition
2 – Requires special handling, but can be automated with effort
3 – Requires special handling and periodic resource time commitment
4 – Requires special handling and significant resource and some management time commitment
5 – High impact, requres special handling, resource and management commitment
There is no solution here except to embark on a documentation task. Fortunately, there is a robust and healthy marketplace of tools to assist in this effort.
Experience has shown that documenting embedded rules in program logic can require a large commitment in resources. If the business rules of the system you are documenting are well segregated in a central code location, your task will be significantly less complex. An excellent documentation tool for existing applications in many popular languages, database and web service implementation is Enterprise Architect. Please see section ‘Software Development Tools’ below for details.
The product of this documentation effort must provide a basis for communicating the size and complexity of the current solution domain. This should include a UML class diagram, Entity Relationship diagram and Component diagram that includes interaction with internal and external systems. Once you have a UML diagram, tools are available to use that model to emit class definitions in your chosen language. This will provide a rough file count, from which you may be able to gauge an estimated level of effort for re-coding and testing. From that you can derive a score. Start with a score of 5 and see if you can justify lowering by use of translation or documentation tools
Another rough way to score this risk is to take the number of lines of code that are undocumented divided by 5,000. If this number equals exceeds your developer count times the number of years in your contract your score is a 5. Many will quarrel with this calculation. Feel free to supply your own.
However you account for the effort to address Undocumented or Unknown Technical Debt, you will do best to not underestimate it.
Given that significant technical debt rarely occurs in isolation, you are likely to also encounter an over-tasked Support organization. Your success in documenting and creating the infrastructure needed for success likely depends to a significant degree on your support organization.
An important concept that will disentangle your development resources from your organization is a well-defined interface. Define an interface that you and your Support group can agree on. Your development group will supply either software in an agree format: collections of .dll files, .msi files, Docker containers, or VM images. Cloud-based development such as Visual Studio Team Services can provide much of the infrastructure a development team needs and relieve much of the initial burden of workstation and development server setup.
Here also, there are good and reliable tools for configuring the setup of developer workstations, Database, Web and build servers. Please see section ‘Software Development Tools’ below for details.
In addition, a set of automated test scripts, or manual test cases, to verify each tier of your installation can demonstrate successful installation and basic functionality, serves as an acceptance test for your Support/Ops/Deployment team to commence their work and make their agreed deliverable to the test team.
Score this by the degree you are dependent on your Support organization and the degree to which they are able to respond.
I wish I could say that a workaround for this existing condition had been found. If you are not permitted to work in the technology stack you were hired for, there follows serious doubt that any deliverable on your assigned project will ever be made.
My best counsel is to raise the re-assignments as placing your commitment to meeting your deliverable schedule in jeopardy. An experienced and articulate PMI certified Project Manager is your best resource in this case.
Scoring this risk is straight-forward but subjective: risk increases by 1 point for every 10% of your contract duration devoted to ‘other tasks as assigned’.
Also, I would suggest keeping your resume up to date, but omit work in outdated technologies.
Organizations with significant Technical Debt often accrue some of it by allowing development to occur in disparate development technologies. For example, a desktop program written in a 3GL might need some subsequent modification that is done in an unrelated scripting language. Some speed sensitive components, found to be too slow, may have been rewritten from the higher level language to a lower level language, say C++. The result is an increasingly complex code base that requires both antiquated coding skills and highly specific lower level coding skills.
This is a difficult situation to handle and will most likely require broad review to minimize the code base to fewer, more current, languages. Of course, a single language is preferable but that undertaking may be beyond available resources.
In many years of development, it has happened only once that a Project Manager secured a delay in the clock starting on a deliverable, for me. This was due to the failure of acquiring and deploying our initial development software configuration. My team presented what was needed in terms of software development tools, and for the best part of a year, we were delayed by meetings and indecision.
To my Manager’s great credit, the clock on our first deliverable did not start until the final software development tool was in place. That did not save the project but the locus of delay was clearly visible to any interested party.
After an initial review of your projects requirements, this risk is best scored by surveying software development tools that you think are needed. The score to assign is the percentage of tools that must be approved, adjusted for the difficulty in getting approval and acquisition. Let’s say 50% of the tools you identify must be justified, approved and acquired. If a written justification must be presented for each of these and your acquisition lead time is more than 1 month, I suggest adding 1 point for presentation delay and 1 for acquisition delay. Starting from a score of 2 or 3 (depending on the number of tools), this would yield a score of 4 or 5. It will be a significant risk, although it can be overcome by dedicated management support, see below.
Scope Creep is similar to Commitment Abrogation and scored the same way.
If you find yourself in a situation where you have gained knowledge about the root causes of failures in a software system, your best course is to document those root causes.
It will do no good to editorialize or rant. Simply state what you can support with facts.
Suffice it to say that if your score for all of the barriers above is 4 or 5, you can stop reading and reflect how best to describe your findings.
At this point, I suggest you should make a cogent and rational presentation that the risk factors are heavily against your project. I would also suggest that it is best to assume that the risk situation you are in has arisen despite good faith efforts to avoid it. But that the risks you have identified are real and must be dealt with.