Death by architecture

One of the less mentioned dangers of a waterfall or engineer type approache to software development is the focus on the initial activity of defining and implementation the architecture of the system.

Usually the critic of the 'old' plan-drive methologies runs along the lines of inflexibility and inability to cope with the unknowns of real world software development. This issue is often adressed in 'mature' organizations by assembling a group of experts in the discipline of IT system construction (IT architects), and tasking them with defining the high level technical aspect of the system.

One of first task addressed by the architects is usually to define the non-functional requirements, like performance, availability, scalability, maintainability and so on. These input are used to define the architecture of the system. this includes selecting the technologies to use for building the system. The architecture will often include such things as layer models, security models, persistence models, domain model and a lot of other models for crosscutting concerns.

If the company defining the architecture also provides the platforms for deploying the system the architect will also be tempted to include a lot of enabling technologies in the architecture. This will include a potent cocktail of enterprise, high-availability, cluster, SOA...... middleware, platforms and frameworks which would (in theory) solve most of the non-functional challenges.

This all sounds very clever, but this approach can in the real world be devastation to a project progress. The problem is that the initial focus of the development project will be centered on creating the system architecture, instead of producing visible progress.

If you are lucky you will have access to people understanding the technologies making up the architecture. But more often you will a team with a more generic competence profile, and all the middleware and enterprise platform will remain badly understood black boxes, continually impeding the development of the system, while the promise result gained by using ready-made enterprise software never materializes.

During this phase the trust between the customer and the software development team will deteriorate, because the development team is unable to demonstrate any progress and and understand of the problem domain will not increase significantly. The moral of the development team will also go down, because the members are also unable to see, where the project is going.

Many project have never really gotten through this phase, and have died struggling to define and implement the 'right' architect. If you are able to fight yourself out of the architecture desert, the scars on the customer relationship and the team moral will quite deep and make the work ahead more difficult.

So, how should one avoid this?

As always, the solution is to use a more agile approach. This include following the agile best-practices of:

• Keep it simple: Do not introduce complications needed because of non-functional requirement, until you can't postpone them any more.
  
• Make it work, make it right, make it perform: Avoid focusing on concerns, which wouldn't have an effect until later in the project. You should off course not totally ignore later stage challeges, but the development should always be driven by the shortterm gains.
• Evolutionary development: Grow the system from a simple seed, nursing and growing the application on a continuous basis.
  

SCRUM vs. JIRA

In my role as JIRA evangelist, I have been struggling with making JIRA function efficiently in our departments SCRUM centric environment. The challenge here is that the central SCRUM concept of a sprint is very difficult to model in JIRA.

JIRA's way of partitioning the project into versions might at first glance appear as the same thing as sprints, but a deeper difference between SCRUM's and JIRA's view on software development lurks below. Where JIRA's concept of versions focus on the progress and roadmap of the application, the sprints in SCRUM acts as containers for sets of tasks, which the development team can work on. This indicates a fundamental focus in the two approaches to software projects:

• JIRA models an application focused approach to software development, with a task management aspect.
• SCRUM models a task based approach to software development, which hopefully reflects some kind of application development.
  

In an ideal world this would be two sides of the same thing, and is actually one of the premises SCRUM is build on. But in the real world cracks start to appear in the illusion, that SCRUM is a good way of managing application development.

The difficulty of JIRA to model SCRUM is just one of a number of symptoms that SCRUM is a less than perfect tool for application development. Even though pretty much all of our development teams have at this point succumbed to the euphoria of SCRUMs blessings, no one has really been able to consistently finish 'good' sprint. Usually the sprints ends with a lot of lose ends, and a very unclear picture of what was achieved compared to what was planned.

This is most commonly contributed to a number of more or less unjust factors, like external intervention, unclear goals, imprecise estimates etc. In my opinion all these annoying 'factors' are more an indicator of how fragile the SCRUM sprint is , because of all the implicit requirements the sprint has to its environment. These sprint prerequisites include:

• Stable team: The central task processing engine in SCRUM is the team. If the team isen't 100% dedicated towards completing the sprint, the sprint breaks. Causes could be the allocation of people for external crisis handling or changes to the partial allocation of team members.
• Stable sprint backlog: The task set to be processed in a sprint is defined by the sprint backlog. If this changes the sprint breaks. Causes could be the introduction of emergency tasks or just general reprioritizing of tasks by higher powers.
• No external dependencies: The SCRUM name implies that the team gangs up for completing the sprint at hand and without assistance from others. If the team isn't able to complete the task defined in the sprint alone they have to draw on external resources. If the external resources do not act fast the sprint breaks. This is a very likely result of having external dependecies, which will be outside the control of the SCRUM team and probably have other priorities.
• Clear goal: One of the cornerstones of the sprint is the team being able to share a common goal and everybody works hard at reaching this goal. If the sprint goal is unclear, much of the 'uniqueness' of the sprint concept disappears, and the sprint 'degenerates' into normal iterations or increments. In my experience the set of tasks very seldom sums up to a common goal, most of the time the sprint backlog is defined by the priorities in the product backlog, ad-hoc bugfixes, available resources and other unrelated tasks.
  
• Small, well estimated tasks: Even if the list of tasks in the sprint backlog remains stable, the ratio between the task estimates and the actual work used for completing the tasks has to be pretty stable to be able to predict the velocity of the team. If the velocity of the team is unpredictable the sprint breaks. A lot of the tasks needing to be solved in software development are very hard to estimate individualy. Some examples are bugfixes, analysis work, integration tasks, etc.
• Visual results: One of the valuable outcomes of good sprints should be clear and visible results of the sprint. If this isn't possible the value of the sprint diminishes significantly. But a lot of the work needing to be done in a software project are enablers for the development of the application, and as such doesn't produce any direct result. These types of tasks include: Framework creation, design and architectural work, documentation and specification work, education, etc

All of the points mentioned above are aspects of software development one should strive to improve. They all contribute to better and faster application development. But in the world I live in, very few of the points are stable enough to act as the foundation for a development method. SCRUMS needs every single one to work perfectly to succeed.

The result is that everybody is currently walking around crying SCRUM as the answer to every project management challenge, but nobody is really seeing it work correctly in their project. Any critical approach to improving this situation has been paralyzed by the blind followers of the SCRUM movement, who have hijacked agile development the last couple of years.