Efficient and fun software development the open source way

One of my constant sources of inspiration in how to work with software development, is the way successful Open Source (OS) projects structured. One obvious reason for using OS projects as inspiration, is of course the visibility of how the development works here. But other fundamental forces are at work here which makes OS projects very interesting, when considering how to build an efficient and robust platform for software development.

A interesting aspect of OS projects, it that they have to succeed under conditions which we would normally consider crippling in commercial software development. Some of these conditions are:

• Allocated resources: One of the cornerstone in every project management model is the ability to plan (and somewhat control) which resources are available when. In OS project people do work, when they have the time and interest.
• Control by management: In normal commercial projects you have the luxury of having roles,, which are dedicated to controlling the project. These are people like project managers, architects, QA, etc, and are backed by the company or organization. In OS projects there are no formal mechanisms for forcing people to do the 'right' thing.
  
• Localized team: Most commercial teams are placed in one location, and distributed teams are usually shunned, because these teams are notorious prone to inefficiency. OS projects are usually distributed, both geographically and in working hours.
  
• Sales organization: In commercial organizations, the usage of the products developed are helped on the way by a sales organization persuading customers to the virtues of the products. In OS software the sales organization is usually lacking.
  

Because of the absence of the listed factors in OS project a number of more fundamental qualities are more clearly visible and have to be addressed.

• Fun: Because of the voluntary nature of participating in OS projects, an OS project needs to be 'fun' to work in, or else it will died a silent dead because nobody contributes. Many of the following qualities are derived from this.
  
• Usage driven documentation: Where, the documentation in commercial projects are usually produced based on what non-users think is a good idea, OS documentation is much more driven by Just-In-Time need for documentation (here non-users are people like QA, project leaders, architects, process definers etc, which don't directly use the developed product or source). The result is, that OS documentation is usually much more relevant and updated than commercial documentation. The OS documentation production is also part of the daily work, so the infamous 'task' of documentation is much less pronounced in OS (eg. boring work isn't done i OS projects).
  
• Usage by product and web quality: Where a major part of commercial products success depends on the sales organization, OS projects have to depend more on the merits on the products themselves and the quality of the (typically web based) public interfaces to the consumers of the products. The result is that the usability and visibility of the project is essential, eg. documentation, bug/feature tracking, design/source need to be of high quality.
  
• Managerless: OS teams very seldom include non-software producing members, and are very efficient role vice, eg. everybody is producing actual software.
• High level of automation: Because boring tasks aren't performed in OS projects, these are either automated if they are necessary, or left out if the need of the task is difficult to see. Mature, high quality OS projects therefore have automated many of the repetitive, boring tasks done manually in many commercial projects.
• Self-organizing: Because projects teams in OS projects aren't defined by outside forces the OS teams usually have a much more organic approach to who-does-what. Everybody is in principle project manager, architect, developer, tester etc.
  
• Distributed team: OS projects are in nature spread all over the globe, and project members typically work at different times. To handle this OS projects needs to function efficiently without the people ever meetings of talking together.
  

For a OS project to be a success these factors have to handled efficiently.

Conclusion: Because of the more 'fundamental' nature of OS software development, a good OS development 'model' can be used as a solid foundation for a commercial development process, where higher order methods for improving software development like SCRUM, Unified Process, CMMI, internal processes, etc. can be added. If, on the other hand, the concerns exposed in OS projects aren't handled in projects focused on higher order development models, like SCRUM, UP, etc. it will be very difficult to make the project a success. And even worse, the forces ruining the project won't be understod, making improvement impossible (my blog SCRUM vs. JIRA and following discussion elaborates a bit on this).

Disclaimer: The differences described between commercial and OS projects are of course exaggerated, many commercial project are adopting more agile approached to software development, which have many similarities to OS development. OS projects are on the other hand seeing greater influence from commercial based team setups, where the advantages of more dedicated contributers becomes available.

Death by architecture

One of the less mentioned dangers of a waterfall or engineer type approache to software development is the focus on the initial activity of defining and implementation the architecture of the system.

Usually the critic of the 'old' plan-drive methologies runs along the lines of inflexibility and inability to cope with the unknowns of real world software development. This issue is often adressed in 'mature' organizations by assembling a group of experts in the discipline of IT system construction (IT architects), and tasking them with defining the high level technical aspect of the system.

One of first task addressed by the architects is usually to define the non-functional requirements, like performance, availability, scalability, maintainability and so on. These input are used to define the architecture of the system. this includes selecting the technologies to use for building the system. The architecture will often include such things as layer models, security models, persistence models, domain model and a lot of other models for crosscutting concerns.

If the company defining the architecture also provides the platforms for deploying the system the architect will also be tempted to include a lot of enabling technologies in the architecture. This will include a potent cocktail of enterprise, high-availability, cluster, SOA...... middleware, platforms and frameworks which would (in theory) solve most of the non-functional challenges.

This all sounds very clever, but this approach can in the real world be devastation to a project progress. The problem is that the initial focus of the development project will be centered on creating the system architecture, instead of producing visible progress.

If you are lucky you will have access to people understanding the technologies making up the architecture. But more often you will a team with a more generic competence profile, and all the middleware and enterprise platform will remain badly understood black boxes, continually impeding the development of the system, while the promise result gained by using ready-made enterprise software never materializes.

During this phase the trust between the customer and the software development team will deteriorate, because the development team is unable to demonstrate any progress and and understand of the problem domain will not increase significantly. The moral of the development team will also go down, because the members are also unable to see, where the project is going.

Many project have never really gotten through this phase, and have died struggling to define and implement the 'right' architect. If you are able to fight yourself out of the architecture desert, the scars on the customer relationship and the team moral will quite deep and make the work ahead more difficult.

So, how should one avoid this?

As always, the solution is to use a more agile approach. This include following the agile best-practices of:

• Keep it simple: Do not introduce complications needed because of non-functional requirement, until you can't postpone them any more.
  
• Make it work, make it right, make it perform: Avoid focusing on concerns, which wouldn't have an effect until later in the project. You should off course not totally ignore later stage challeges, but the development should always be driven by the shortterm gains.
• Evolutionary development: Grow the system from a simple seed, nursing and growing the application on a continuous basis.
  

SCRUM vs. JIRA

In my role as JIRA evangelist, I have been struggling with making JIRA function efficiently in our departments SCRUM centric environment. The challenge here is that the central SCRUM concept of a sprint is very difficult to model in JIRA.

JIRA's way of partitioning the project into versions might at first glance appear as the same thing as sprints, but a deeper difference between SCRUM's and JIRA's view on software development lurks below. Where JIRA's concept of versions focus on the progress and roadmap of the application, the sprints in SCRUM acts as containers for sets of tasks, which the development team can work on. This indicates a fundamental focus in the two approaches to software projects:

• JIRA models an application focused approach to software development, with a task management aspect.
• SCRUM models a task based approach to software development, which hopefully reflects some kind of application development.
  

In an ideal world this would be two sides of the same thing, and is actually one of the premises SCRUM is build on. But in the real world cracks start to appear in the illusion, that SCRUM is a good way of managing application development.

The difficulty of JIRA to model SCRUM is just one of a number of symptoms that SCRUM is a less than perfect tool for application development. Even though pretty much all of our development teams have at this point succumbed to the euphoria of SCRUMs blessings, no one has really been able to consistently finish 'good' sprint. Usually the sprints ends with a lot of lose ends, and a very unclear picture of what was achieved compared to what was planned.

This is most commonly contributed to a number of more or less unjust factors, like external intervention, unclear goals, imprecise estimates etc. In my opinion all these annoying 'factors' are more an indicator of how fragile the SCRUM sprint is , because of all the implicit requirements the sprint has to its environment. These sprint prerequisites include:

• Stable team: The central task processing engine in SCRUM is the team. If the team isen't 100% dedicated towards completing the sprint, the sprint breaks. Causes could be the allocation of people for external crisis handling or changes to the partial allocation of team members.
• Stable sprint backlog: The task set to be processed in a sprint is defined by the sprint backlog. If this changes the sprint breaks. Causes could be the introduction of emergency tasks or just general reprioritizing of tasks by higher powers.
• No external dependencies: The SCRUM name implies that the team gangs up for completing the sprint at hand and without assistance from others. If the team isn't able to complete the task defined in the sprint alone they have to draw on external resources. If the external resources do not act fast the sprint breaks. This is a very likely result of having external dependecies, which will be outside the control of the SCRUM team and probably have other priorities.
• Clear goal: One of the cornerstones of the sprint is the team being able to share a common goal and everybody works hard at reaching this goal. If the sprint goal is unclear, much of the 'uniqueness' of the sprint concept disappears, and the sprint 'degenerates' into normal iterations or increments. In my experience the set of tasks very seldom sums up to a common goal, most of the time the sprint backlog is defined by the priorities in the product backlog, ad-hoc bugfixes, available resources and other unrelated tasks.
  
• Small, well estimated tasks: Even if the list of tasks in the sprint backlog remains stable, the ratio between the task estimates and the actual work used for completing the tasks has to be pretty stable to be able to predict the velocity of the team. If the velocity of the team is unpredictable the sprint breaks. A lot of the tasks needing to be solved in software development are very hard to estimate individualy. Some examples are bugfixes, analysis work, integration tasks, etc.
• Visual results: One of the valuable outcomes of good sprints should be clear and visible results of the sprint. If this isn't possible the value of the sprint diminishes significantly. But a lot of the work needing to be done in a software project are enablers for the development of the application, and as such doesn't produce any direct result. These types of tasks include: Framework creation, design and architectural work, documentation and specification work, education, etc

All of the points mentioned above are aspects of software development one should strive to improve. They all contribute to better and faster application development. But in the world I live in, very few of the points are stable enough to act as the foundation for a development method. SCRUMS needs every single one to work perfectly to succeed.

The result is that everybody is currently walking around crying SCRUM as the answer to every project management challenge, but nobody is really seeing it work correctly in their project. Any critical approach to improving this situation has been paralyzed by the blind followers of the SCRUM movement, who have hijacked agile development the last couple of years.

Tales from the history of documentation, conclusion.

These tales are of course somewhat based on my own continuous battle for better documentation. The journey hasn't quite spanned that many years, but has instead crossed a number of different organizations with quality/configuration/project managers playing out the parts described in the previous tales.

These different kinds of managers often have a defining role in what kind of documentation should be produced in a project, even though they aren't actually involved in the production and especially the consumption of the documentation. The result is that the information exchange- and storage medias are not defined by the usability in the day-to-day work (by the common people), but instead is driven by theoretical quality and control concerns centered on a belief that every serious project must be build on a foundation of rock solid documentation.

It is easy for the people outside of the actual software production to fall into this trap. The documents produced by the development team is the only tangible product visible besides the actual software, which properly doesn't appear before the project has run for some time. And even when the software materializes, it is kind of magic, with strange things going on at the surface, produce by incomprehensible code woven by the developer.

The documentation becomes the managers sole media for defining and maintaining control of the development. After having started down this path most of the efforts to improve quality, efficiency and control will be channeled into 'improvements' to the documentation. The result usually lead to ever more documentation in increasingly complex structures. This again leads to decoupling of the software development and the formal documentation development.

If you don't recognize the described causes of 'documentation failure', try to look for the symptoms instead:

• To much time is spend on documentation.
• The relevant documentation does not exist.
• It is very difficult to find documentation you haven't got hardcode links to.
• The documentation isn't updated.
• Documentation production, review, rework tasks 'block' application progress.
  

So if you have identified any of the above bullets as problems in your current project, please don't try to address these by increasing documentation control, amount, rules, etc. Instead try to base the solution on the primary cause for documentation, support for the common peoples needs.

Tales from the history of documentation, chapter three

Years came and went and before long the concerns regarding the safety of the computer based documents where addressed. Safe archival systems where created, structures for storing the documents where defined, procedures for when and how to change the documents were made, elaborate software tools for controlling all the defined rules were implemented, template repositories where established to ensure standards for the documents where followed etc.

But the common people didn't use the formal computer based documentation frameworks that much. The complexity of it all was a bit overwhelming, and if you did manage to find something in the huge folder structures, it was properly out of date or wasn't really relevant for the common peoples daily life's.

To improve on this inability of the common people to embrace the electronic documents, documentation support roles were created, more documents were written on how to handle the other documents, people where sent on courses to learn the 'right way', rewards were given to the ones succumbing to the correct document usage, all to ensure that the common people where 'fixed' and finally brought to understand, what was in their best interest.

But even with this massive effort to perfect the documentation system, the common people didn't really use it much. They still preferred to use easier channels of information, which didn't have the complexity of the official documentation system.

Then one day a young enthusiastic fellow approached the community with a new media for communication and maintaining information. A new web based media, where everybody could easily find information and contribute to the documentation if they felt like it, he called it a wiki. He said that the wiki was a major improvement over the conventional document based ways of maintaining information. Information on the wiki was much easier to use, nothing was more than a mouse click or a text search away, and it was much easier kept up-to-date because the modification of the wiki was so easy and everything was changed directly in the central repository.

The experts and managers from the document organization listened to the young man and told him this wiki thing looked very nifty and he was very welcome to use it, but it could of course not be used for the more important information. After all, it wasn't possible to control who changed the documentation, who read the documentation, the documentation was difficult to print on real paper documents. In fact it didn't look at all like real documentation should, it was just a mess of information thrown together by people who didn't understand the finer art of document management.

And of course the primary goal of documents was the ability to store the documentation. When this had been perfected, you could look at the accessibility of the documents. And this issue had already been addressed though the previous efforts to guide the common people towards better understanding of the 'science of documents'.

The young man was pretty disillusioned by this lack of vision from the documentation experts and managers, but agreed to follow the decision to only use the wiki for things considered less important by the experts.

The years passed, and surprisingly enough wikis gradually started to replace the document based structures. Apparently the intangible concept of 'easy to use' and the possibility of everybody to contribute was found to be 'better' by the common people.

And this is where we stand today.

The End.

Tales from the history of documentation, chapter two

Generations came and went and before long many of the more long term issues regarding storing information had been addressed. Storage facilities (archives) had been established at central places, so the documentation could be kept for many, many years and review boards controlled any updates to the archives to ensure the quality of the documentation was maintained.

But the common people didn't use the huge archives that much. The complexity of it all was a bit overwhelming, and if you did manage to find something, it was properly out of date or wasn't really relevant for the common peoples daily life's.

So in an attempt to remedy this, procedures were carefully written on how to ensure updated and relevant documentation. Elaborate indexes were also created to document how to find the documentation. And all this was of course also stored in the archives, because this was the right place to keep important information.

To assist in facilitating the efficient proliferation of the documentation archival management system, an organization of archival and process experts was created with the responsibility of guiding people towards the information they needed. A host of managers was also need to ensure everything was handled correctly.

But even with this massive effort to perfect the documentation system, the common people didn't really use it much. They preferred to use more informal channels of information, which didn't have the complexity of the official documentation system.

Then one day a young enthusiastic fellow approached the community with a new media for maintaining documentation. A new electronic device called a computer was able to provide electronic versions of the document spreadsheets and other forms of paper documentation stored in the archives. He said that the computer was a major improvement over paper-based documentation. The electronic documents where much easier to edit and maintain. They could also be kept in computer based storage systems, avoiding the huge paper document archives. Copy and distribution was also much easier, just a click of a button.

The experts and managers from the archives listened to the young man and told him this computer thing looked very nifty and he was very welcome to use it, but it could of course not be used for the more important information. After all, it wasn't as safe as paper, it could loose power, the data could be corrupted or hacked, incompatibility issues would certainly arise. The control of the documentation would also be difficult to enforce, when copying and distribution was so easy.

The young man was pretty disillusioned by this lack of vision from the archive experts and managers, but agreed to follow the decision to only use the computer for things considered less important by the experts.

The years passed, and surprisingly enough computers gradually started to replace the paper documents. Apparently the intangible concept of 'easy to use' and the possibility of everybody to contribute was found to be 'better' by the common people. Soon most documentation was stored on computers and the archives were used more for historic documentation.

End of chapter two

Tales from the history of documentation, chapter one

One of the main battles I have been fighting during the years, is the fight against inefficient documentation. Inspired by these challenges I would like to tell a tale of the epic documentation struggle through the ages.

Once upon a time important information was stored by carving runes it into huge stones. This way of maintaining information was thought to work very well by the elders. It had worked for many generations, the documents would last for many, many years, the rune stones never moved, so the information should be easy to find and it was ensured that only the wise elders was allowed to contribute to the documentation.

But for some reason the writing on the stones weren't used that much by the common people, they mostly preferred to talk to other people with the relevant knowledge or just write notes in the dirt. When the common people were asked why they didn't use the stones, they answered, that it took too long to go to the stones and read the information and the information written there was out-of-date and wasn't really relevant for the common people.

The elders response to this was that the common people needed more teaching on how to use the rune stones and more rules regarding rune usage should be developed (and of course written on other rune stones). Another improvement would be the manufacture of even more stones to close the holes in the existing information and more effort should be put into updating the rune stones.

People were sent on rune stone courses, massive amounts of rune stones were produced, but the usage of the rune stones didn't improve much.

Then one day a young enthusiastic fellow approached the community with a new media for maintaining documentation. It just consisted of thin sheets of wood (he called it paper) and you could just write on it with small pieces of charcoal. He said that the paper was a major improvement to the rune stones, the paper was much 'easier' to used than the rune stones, everybody could contribute to the documentation, you could easily update the writing on the paper and you could bring the relevant documentation with you to use it where you needed it.

The elders listen to the young man and told him this paper thing looked very nifty and he was very welcome to use it, but it could of course not be used for the more important information. After all, it wasn't as safe as rune stone, it could burn, rot, bleach or blow away. Anybody could write on paper, which would be dangerous, who would control the changes. It would also be impossible to maintain the archives the rune stones provided, when everybody could just pick up a document and take it with him.

The young man was pretty disillusioned by this lack of vision from the elders, but agreed to follow the elders decision to only use the paper for things considered less important by the elders.

The years passed, and surprisingly enough paper gradually started to replace the rune stones. Apparently the intangible concept of 'easy to use' and the possibility of everybody to contribute was found to be 'better' by the common people. Soon all documentation was stored on paper and the rune stones retired to museums and parks.

End of chapter one