Recent Articles

Adopting Agile/Iterative Software Development

Iterative Software Development

Choosing An Approach To Agile/Iterative Software Development

Adopting AGILE In Your SDLC Shop

iterDev Documentation Model

GitHub.com Projects

Convert Log4Net to NLog

Log4Net integration in ASP.NET MVC

Adopting Agile/Iterative Software Development|Agile|Iterative|iterDev.com

Adopting Agile/Iterative Software Development

by Edward Campbell
http://www.linkedin.com/in/edwardcampbellprojects/
edwardcampbellprojects@gmail.com
April 11, 2017

‘Adopting Agile/Iterative Software Development’ is a blueprint for adopting Agile Software Development in any organization. The approach and description are general; the tool selections and technologies are specifically based on my own project. This is a living document that chronicles my progress in implementing my own Agile/Iterative Software Development Capability. Some tools and approaches are based on available open-source tools that might not necessarily fit in your organization.

For more about the mainstream Agile methodology see: Top 10 Insights from the 11th Annual State of Agile Report

Building medium to large software systems that employ best practices in Continuous Integration, CI, and Continuous Deployment, CD, is the main focus of my professional efforts. If you or your company are interested in implementing an Agile/Iterative Software Development capability, I hope to hear from you. Email me at the link above. I am available to discuss your situation.

This article is the result of years of research, experience and collaboration with accomplished, diligent, like-minded developers. Their contributions to this document and our shared experience are gratefully acknowledged. This article is an incomplete work-in-progress. When I have a finished system, you will be able read about it first at http://iterdev.com . Thanks for reading.

It is my firm, but non-confrontational, assertion that ‘Agile Software Development’ should more aptly be called ‘Agile/Iterative Software Development’. This more descriptive name in this article is used to emphasize the iterative nature of successful systems development.

The basis for the name ‘Agile/Iterative Software Development’ is not new and comes originally from ‘Object-Oriented Analysis and Design With Applications’ by Grady Booch, p.13:

A complex system that works is invariably found to have evolved from a simple system that worked. … A complex system designed from scratch never works and cannot be patched up to make it work. You have to start over, beginning with a working system” *

* Booch cites this quotation as originally from: Gall, J 1986. Systematics: How Systems Really Work and How They Fail . Second Edition. Ann Arbor, MI: The General Systematics Press, p. 65

This observation still rings true today. Evolving working software through repeated iterations of the SDLC phases of Requirements Engineering, Design, Development, Integration and Test, is increasingly accepted as a best practice.

Agile Development is a nomenclature that expresses openness to accepting and adjusting to changing requirements during the development of a software system. Accepting and adjusting to changing requirements, of itself, demands Iterative Software Development techniques. This is the basis for much of my own practice for the past 20+ years. Please see http://iterdev.com/ for a fuller discussion of this subject.

And now, on to a blueprint for adopting Agile/Iterative Software Development.

Advantages of a Mature Agile Development Capability

The advantages of a mature Agile Development capability are straightforward:

  • Reduced software update, patch and hot fix deployment times
  • If implementing mandatory unit testing, increased automated build test coverage and enhanced ability to refactor existing code
  • If implementing Test Driven Development for bug fixes, increased automated regression testing
  • Lower bug reporting, rework and re-deployment rates
  • Reduced accumulation of Technical Debt.
  • Continuous improvement in Level of Effort accuracy for maintenance and new feature deployment
  • Increased productivity, morale and lower team turnover

A Big Picture Analogy

The Big Picture view of Agile/Iterative Software Development is that any business activity that is sufficiently costly and critical to business success is worthy of planning, resource allocation and investment to ensure success. So my big picture analogy is: the Planting-A-Garden analogy. But before we get there, a little background is in order.

Software Development, or software construction if you like, is somewhat like residential home construction, with the exception that no 2 software projects have more than a passing similarity (see Code Complete, Second Edition by Steve McConnell , chapter 1 – 3, for a thorough explanation). Except for some large multi-user/multi-tenant systems, the large majority of applications are different enough that developers on any other program take weeks or months to understand the code of the one you care about: Yours.

Perhaps a more useful observation about Software Development is that: From the beginning of the industrial age until the recent past, it was true that any foreman or manager could do any line worker’s job. The skills involved in a line worker’s job, however much encapsulated, were easily grasped by a replacement worker. This is no longer true. Software Development technical skills are simply too numerous and tightly coupled to be ‘picked up’ by a substitute. McConnell’s book meticulously notes, based on empirical data, that the difference in productivity on development teams between the least and most productive workers is often an order of magnitude or more.

Given these observations and the infamous history of software failures, it is not surprising that Software Development, as an activity, as undergone rigorous scrutiny and repeated attempts at evolution. Methodologies that promise to tame late deliveries, cost overruns and bad design have come and been overtaken by newer, sometimes better, versions. Each one promised to fix Software Development and address the shortcomings of its predecessors.

Enter Agile/Iterative Software Development in response. Agile/Iterative Software Development is based on a refreshing brevity of purpose and pioneered by successful developers rather than Business Process Management consultants. The Agile Manifesto needs no embellishment:


We are uncovering better ways of developing
software by doing it and helping others do it.
Through this work we have come to value:

Individuals and interactions over processes and tools
Working software over comprehensive documentation
Customer collaboration over contract negotiation
Responding to change over following a plan

That is, while there is value in the items on
the right, we value the items on the left more.

http://agilemanifesto.org/

For an in depth statement of principles see the ‘Principles behind that Agile Manifesto’ at http://agilemanifesto.org/principles.html .

Of course, this presents problems in contractual Software Development with government agencies and other fixed price, fixed deliverable arrangements. But these problems can be foreseen, prepared for and handled if management and clients place enough value on completion and delivering value. The situation for internal corporate Software Development teams makes adjusting deliverable requirements and schedules more possible and amenable. Trust, good faith and cooperation are the keys to success.

And now we arrive at the main assertion of this section: Building an Agile/Iterative Software Development capability that is mature, productive, predictable and resilient is akin to the Planting-A-Garden analogy. The Planting-A-Garden analogy and Agile/Iterative Software Development possess the following similar characteristics:

The first season’s product is mostly unknown, but can be maximized by the execution of thoughtful planning, preparation and execution. Accurate prediction of outcomes is hard to achieve. The critical product of this effort is a working method of producing iterations of a Deliverable Prototype. This is not the finished product but a partial implementation of a finished product that becomes more complete with each iteration.

The second season’s product will depend to a significant degree on preserving and reproducing the successful aspects of season 1. Accurate predictions are most likely those that mirror season 1 outcomes, and season 2 areas of focus.

It is in season 3, that tool understanding and usage mature to begin to support actionable business prediction.

Without doubt, these are general observations that can be exceeded by a greater intensity of effort and resources or thwarted by reluctance, dissembling or resistance. And for those who want to argue every point, a season in this discussion can be from: one or two calendar quarters up to a year, depending on your resource availability situation.

Still, the value of this analogy is to understand and set the expectation that building a capability is an undertaking in itself. Once that capability is built and functioning, its productivity and quality of output can be assessed, measured and optimized. Attempting a few uncoordinated or piecemeal steps at Agile/Iterative Software Development automation will not build a system, but yield something less.

The counsel of this document is that an Agile/Iterative Software Development capability is a multi-season, multi-disciplinary undertaking like the Planting-A-Garden analogy. If it is your project to implement an Agile/Iterative Software Development capability, your presentation to management should feature this simple truth: Agile/Iterative Software Development success comes from patient, multi-season effort, not from acquisition of a few tools.

Perhaps more to the point, if your shop is besieged regularly with emergency deployments, bug fixes, patches and late night testing, you may have lost, temporarily, the ability to plan and execute multi-season projects. In short if your house is on fire, you likely don’t have time to plant a garden that will yield results.

In the House-On-File situation, my experience is that adding an Agile/Iterative Software Development capability to your list of tasks without dedicated Management support and dedicated resources will end in disappointment. The disappointment may not be of your doing but that will not prevent outsiders from attributing it to you. Implementing an Agile/Iterative Software Development capability is not without peril.

Choosing an Approach To Agile/Iterative Software Development

Partial And Incremental Agile/Iterative Adoption

I have written previously on partial adoption approaches to Agile/Iterative Software Development

Please see that article for a discussion of incremental approaches to adopting Agile/Iterative Software Development.

In brief, my experience and counsel is that the most efficient and convincing way to achieve a working Agile/Iterative Software Development Capability is to pursue a small to medium project that implements all major features of Agile/Iterative Software Development.

The keys to this undertaking are aligning expectations with realities:

  • Substantial changes in software development take significant effort and time to produce.

  • An engaged and enthusiastic team is essential

  • Management commitment to provide support and stability is essential

  • Substantial change in software development tools occurs every year

These are the focus of the rest of this article.

A Plan To Implement An Agile/Iterative Software Development Capability

This section includes a straightforward listing of applicable resources and useful tools, discovered and used in current past projects. Each tool is listed by its general category and the specific tool that I have used. The purpose of this format is to stimulate thinking about tools whatever your technology stack and, if your development environment matches mine, these are tools that I have used and vetted.

Planning to Succeed

Software development tool technology changes every year. Maintaining your position in a competitive world necessarily includes keeping up with these changes. Like any handcrafted product the quality of your work product depends significantly on the quality of your tools.

In short, my counsel is to allocate sufficient time to your lead/best developers to evaluate new and updated tools to help your development capability.

The frequency of this research is up to you and management but, unless your projects are small, every project and team deserves a chance magnify their effort, product quality and current knowledge regularly. In the long run this will help minimize technical debt accruing to your product line, creeping brittleness in your code base and costly developer turnover.

Technology Stack

The technology stack described here is the Microsoft C#/.NET stack. Microsoft haters will find solace that some of the tools are actually written in competing stacks.

The only criteria for inclusion in this listing are utility, simplicity and effectiveness.

Management Commitment

It bears repeating that implementing an Agile/Iterative Software Development capability requires a commitment from your management. This is essential.

If you can reasonably expect your team to have a majority of their time dedicated to implementing Agile/Iterative development, you have a good chance, with the right tools and resources, of delivering both a working product and a working process. If not, then not.

Product Owner

An essential measure of commitment from you management is the appointment and empowerment of a Product Owner:

The (…) product owner is typically a project's key stakeholder. Part of the product owner responsibilities is to have a vision of what he or she wishes to build, and convey that vision to the scrum team. This is key to successfully starting any agile software development project. The agile product owner does this in part through the product backlog, which is a prioritized features list for the product.

The product owner is commonly a lead user of the system or someone from marketing, product management or anyone with a solid understanding of users, the market place, the competition and of future trends for the domain or type of system being developed.”

from https://www.mountaingoatsoftware.com/agile/scrum/roles/product-owner

More importantly to the daily functioning of an Agile/Iterative development team, the Product Owner is empowered by Management to make decisions. A disproportionate amount of rework and extra work can be avoided by having a Product Owner who can bless the approach, appearance, workflow and other questions about prototypes, posed by your developers in as they occur.

Conversely, if your Product Owner cannot make decisions but must rely on scheduling meetings, achieving consensus and routing that back to your team, much lost time will be risked.

Access to Tools

Developers need access to tools, for evaluation, for experimentation, for finding the next technique or visual effect that will make your product stand out. Network Support organizations fear, and with good cause, the installation of buggy or infected software.

Be prepared to research the safety of software development tools, download locations and development licenses to ensure that the tools you choose will not cause harm. Virus scanning is essential. Installing firewalls and port regulation between development sub-networks and test networks is likewise essential.

This is a critical task for the Project Manager/Lead Developer in trying to obtain the best tool set possible for your project.

See https://www.wibas.com/cmmi/decision-analysis-and-resolution-dar-cmmi-dev

Prototyping and Proofs of Concept

Agree with your Product Owner on a milestone to produce your project’s first deliverable prototype to test end to end development, verification and deployment.

See http://iterdev.com/Articles/iterDevDeliverablePrototype for a complete discussion of deliverable prototypes.

Development, Test, Production Environments (Minimum)

Most of my development has been in n-tier mission critical enterprise systems. This scale of development demands separate, fire-walled, development, test and production networks.

The Development network allows for experimentation and prototyping with few restrictions on local administrative users. This is the normal role group assignment for developers.

The Test network must be locked down to the same degree as the Production network and serves a critical role in compelling developers to adjust to Production level permissions settings. It is here that all Production deployment and testing problems are solved.

The Production network is accessed only by the Production Support group and the Test Group. The validated program code, database, test and deployment scripts from the Test network are the sole inputs to deploy a working Production system.

Without these separate environments successful Agile/Iterative Software Development is likely out of reach. If your organization cannot, or will not, support this level of resources, you should document each shortcoming for future use in a Lessons Learned activity.

Steps to Execute

1. Appraise your project deliverables into categories: Documents, Product, Database, Configuration, Build, Test and Deployment scripts.

2. Use your deliverable category list to research tools that might help and fit in with any current tools in use.

3. Define an acceptable interface to Production Support for product delivery: Assumptions, Location, Content.

4. Build and deploy your first deliverable prototype to the Development and Test environments

Measures of Success with Agile/Iterative Software Development

Employ Domain Driven Design in UML

Entreprise Architect, design with interfaces, dependency injection, Inversion of Control containers

Technology Stack

The primary technologies in my chosen stack are:

  • Domain Driven Design, using Enterprise Architect
  • Microsoft Windows 7, workstations.
  • Microsoft Server 2012, servers
  • C# 5
  • ASP.NET MVC
  • .NET CORE
  • LINQ
  • SQLServer 2016

These technologies inter-operate and cooperate to make development significantly more productive. If your choices differ you should be able to produce a reasoned Decision, Analysis, Resolution document to support their advantages over other available technology stacks.

Software Development Tools

These are my picks for Software Development Tool Categories, preferred Choice and Reasons

Tool Category

Choice

Reason

UML/DB Design

Enterprise Architect 13

Highly capable, floating licenses available. See:

https://en.wikipedia.org/wiki/Enterprise_Architect_(software)

Program Design Documentation

Enterprise Architect 13

Customizable output to virtually any documentation standard. See also http://iterdev.com/Articles/iterDevDocumentationModel

IDE

VS2017

Microsoft flagship development platform, the home of ASP.NET MVC development

IDE Productivity

CodeRush

A leading coding productivity add-on. See https://www.devexpress.com/products/coderush/

Code Documentation

Ghost Doc

Automatic documentation headers for methods, properties and files. Integrates with VS2017

ORM

Entity Framework 6, code first, LINQ

Microsoft integrated ORM, .NET integrated, SQL Server friendly

C#/LINQ scratchpad

LinqPad 5

Free, high productivity, works with any ODBC connectable database: http://linqpad.net

HTML

HTML5


CSS

Bootstrap 3

Ready to upgrade to CSS 4 when available

Source Control

Git

In house server running Bonobo Git Server

JavaScript

Jquery, JqueryUI

Wide adoption, wide library support, lowers browser incompatibilities, CSS like selector support,

Code Generator

CodeTrigger

Supports VS2017, Enterprise Architect. Generates useful prototype code: https://www.codetrigger.com

Test Harness

Xunit

Nunit follow on, supports VS2017, open software

Inversion of Control Container

MOQ

Well regarded, well integrated with VS2017, https://github.com/moq/moq4

Code Comparison Utility

Beyond Compare

Integrates with Git, excellent and efficient for Code Reviews, http://scootersoftware.com/

OS Scripting

Powershell 5

Microsoft’s replacement for batch files. De-facto interface for future MS Server products, wide support, well tested

Tool Category

Choice

Reason

Entity Mapping

AutoMapper

Useful to automate entity to entity mapping in MVC development, http://automapper.org/

Rules Engine

CodeEffects

Excellent rules engine with reflection powered rule creation UI, http://codeeffects.com/

Continuous Integration

TeamCity

Excellent support of Git projects, free for prototypes, https://www.jetbrains.com/teamcity/

Continuous Deployment

Octopus Deploy

Excellent integration with TeamCity, https://octopus.com/

Source Code Control

Git

Distributed source control built around making branching and merging easier. Very fast and integrates well with Beyond Compare, http://git-scm.com

Source Code Control - Special Consideration

Source code control is important enough to warrant special consideration. This section is devoted to special aspects of our chosen software version control tools, Git, GitKraken and PoshGit.

Git

Git is a distributed Source Code Control, SCC, system, originally designed and implemented by Linus Torvalds. Git represents a fundamental rethinking of SCC systems and emphasizes branching and merging as the central activities. Git is also much, much faster and more resilient than traditional central code respositories. In its short history, see https://en.wikipedia.org/wiki/Git Git has taken over as the repository for much of the world’s open source projects, see https://github , et. al..

In fact, Git is now the dominant backing store for Microsoft’s Team Foundation Server, in preference over Microsoft’s own Visual Source Safe product.

The best resource I have read on Git is: https://git-scm.com/book/en/v2

Bonobo Git Server

All of my source code is stored on a Git server that I adapted from a the Bonobo Git Server open source project: https://bonobogitserver.com/ . Using my own server or an inexpensive shared ASP.NET MVC web server, I can store large amounts of code and commit histories.

Furthermore, I can revert to the code state for any previous commit in a few seconds. Lastly, even if my server crashes, and it has, I am at most a few minutes to a few hours away from full restoration of my project. This, of course, depends on the number and size of the projects lost, and the number of locations where the most recent distributed repository resides. Since they are all on my backed up dev box, it would take 3 simultaneous drive failures for permanent code loss to occur and the loss would only represent the work from the second most recent repository.

I am a fan of Git.

PoshGit, Git Bash

Git is built on a command line interface. Every Git command and capability can be accessed from Git Bash, see https://en.wikipedia.org/wiki/Bash_(Unix_shell Unix shells can be daunting to learn and use.

As a Windows veteran, I much prefer PowerShell Git, PoshGit, from https://github.com/dahlbyk/posh-git.

Whichever shell you choose, you will be able to work through the details in the book.

GitKracken

A tool as powerful as Git comes with a learning curve, see https://git-scm.com/book/en/v2 . There are several visual Git clients and GitKracken is my favorite. This is not a free tool for commercial development but after you learn the command line interface it is a beautiful and refreshing upgrade. And you can use it for personal development free.

Get your free copy at: https://www.gitkraken.com/ .

Source Code Branching and Merging Model in Git

The branching model chosen is a simplified version of the excellent Git Branching Model originally published by Vincent Driessen at: http://nvie.com/posts/a-successful-git-branching-model/. This represents a simpler version of that branching model based on experience with small teams, by Drew Fradette:

At my work, we have been using a Git branching strategy based on Vincent Driessen’s  successful Git branching model . Over all, the strategy that Vincent proposes is very good and may work perfectly out of the box for many cases. However,  since starting to use it I have noticed a few problems as time goes on:

  • a LOT of branches  – Since Vincent’s model has a branch for each release version, over time you get a lot of branches lying around.
  •  - At any given time, we will have 3 identical entities:
Release branch, A master branch (after release-1.0 has been merged in), A version Tag (1.0)

Changes to every build  – We use Jenkins CI and we currently have to change each job to point to the new tag or branch. It get’s a little tedious and won’t scale in the long run.

Fortunately, these problems are easy to fix. In fact, the fixes simplify the model and allow simpler automation.

So, with these points in mind, here is my modified model: http://drewfradette.ca/a-simpler-successful-git-branching-model/

Simpler Successful Git Branching Model

Proofs of Concept

An important part of any plan to adopt Agile/Iterative Software Development is defining and overcoming technical barriers. In the case of replacing a brittle thick client written in an unmaintainable language with a thin client and services, we needed several proofs of concept to ensure success. Below are a few of our proofs of concept projects and some notable ones freely available on the internet:

MVC local peripheral control via Windows Service and Client: https://github.com/iterDevcom/WCFWindowsService https://github.com/iterDevcom/WCFWindowsService_MvcClient

MVC local peripheral control via JS: http://www.xarg.org/project/jquery-webcam-plugin/

MVC Web Service Access: https://github.com/iterdevcom

Editor Templates: https://github.com/edwardcampbellprojects/mvc_EditorTemplates_Test-01

Application Services: ToolTips: to be posted

Application Services: Logging: https://github.com/edwardcampbellprojects/mvc_Log4Net

Increasing responsiveness with Partial Views: to be posted

Angular 2 way data binding: to be posted

MVC Custom Scaffold Templates: to be posted

Testing (in-progress)

Unit Tests

Designing and Coding with Interfaces and Inversion of Control

Auto unit test coverage in Visual Studio

CodeUI tests in VS2017

xUnit

Business Rule Tests

Automated UI Test Scripts

User Acceptance Tests

CI, Continuous Integration (in-progress)

DevOps part 1: http://iterdev.com/Articles/iterDevProjectDeliverySystem

Smaller Teams/Projects Visual Studio 2015/2017 build scripts and Powershell

Medium to Large Teams/Projects: TeamCity, https://www.jetbrains.com/teamcity/

CD, Continuous Deployment (in-progress)

DevOps part 2

Smaller Teams/Projects Visual Studio 2015/2017 FTP/WebDeploy publishing

Medium to Large Teams/Projects: Octopus Deploy, https://octopus.com/

Team Selection (in-progress)

Project Manager: PMI trained, DAR qualified, Risk registering

Team Lead:

POC Authors

MVC/Bootstrap coder

Scripters (PSH, Test, T4, SQL, CodeTrigger)

Project Prototype (in-progress)

Integrate Proofs of Concept, Unit Tests, see http://iterdev.com/Articles/iterDevDeliverablePrototype

Automate Checkin Builds and unit testing

Automate test deployments on successful build and test

Planning an Agile/Iterative Project (in-progress)

Setting expectations: Planting a Garden

Management commitment

Resource commitment: Development assets, Tools,

Dedicated Team: Product Owner, Team

Funding

Building a Team: Requirements, Technology Stack, Preliminary Toolset

Training: PluralSight, YouTube

Deliverable Prototype Sprints: http://iterdev.com/Articles/iterDevDeliverablePrototype

Perilous Conditions (in-progress)

Excessive Agile Process: possibly overcome by auto documentation: EA

Hostile Departments in the critical path

No Tools or burdensome acquisition procedures

No Training

Management commitment

No Product Owner

No Dedicated Team

Indecision on Technology Stack

Underpowered Development boxes

No Experienced PM

No Experienced Project Lead

Under funding




© 2020 - iterDev.com, All Rights Reserved. 3.1.1.48