The cost and benefits of documentation

One of my first post on this blog was a rattle about what medias to use, when producing and consuming documentation. Recently I have been lucky enough to define what documentation tools should be used on my projects. The choice has of course fallen on a wiki based information system, more specifically Confluence, backed by JIRA for the more dynamic project information, like bugs, tasks, risks, changes ( I provide consulting services for both products ;-).

But the success in this area has made a new challenge visible, which until now has been hiding in the normal swamp of inefficient and unstructured documentation tooling.

One of the first impulses you might get after getting access to easier-to-use information system, could be to start to document everything, but this ignores one of the major shortfalls of documentation, namely producing and maintaining documentation has a cost. Even after the introduction of more efficient tools for accessing and maintaining documentation , the production and retrieval from the documentation system is far more cumbersome, than using simpler human-to-human information exchange.

On the other hand, documentation also has its benefits, but I find the costs and benefits of what documentation to produce when, are seldom handled in more than a intuitive manor. This results in very simple documentation models. Examples are:

1. More documentation is better: Heavy process methologies like Waterfall models and typical CMMI implementations are primarily build on the assumption that the mechanism for producing better software is driving the implementation by documentation (Documentation Driven Development, DDD). The idea is that more documentation -> more order and structure. You could say that the focuse here was the benefits of documentation.
   
2. All documentation is bad: This was seen in the initial XP and agile movements, which were a counter-reaction to the Waterfall model and other heavyweight methologies (DDD). The burden of documentation is the main focus here.

So in order to get a more balanced approach to when to retain a piece of information in form of documentation and when to stick to a direct person-to-person strategy, is here my list of forces to consider when choosing whether to create a piece of documentation, or not:

Benefits of documentation

• Doesn't change over time: This is one of the primary drives for creating documentation.
• Person independent: You do not need to have access to a particular person with the right knowledge to retrieve the documentation , you can just look it up.
• Scalable: In a pure person-to-person approach to information sharing, you'll quickly find a small subset of project members using a great deal of their time explaining project aspects to other project members. This can be alleviated somewhat by first attempting to looking information up in the project documentation, before turning to the project oracle in the particular area of interest.
• Geographical invariant: If a project team or stakeholders aren't all placed at the same location, the barrier to person-to-person information exchange rises significantly, thereby making documentation based information exchange more attractive. This is of course only the case if the documentation is accessible at all relevant sites, eg. properly Internet based, or at least intranet based.
  
• 24/7 accessibility: Documentation based information can be access all the time. So even when project members have different working hours, vacations etc. they will still (in principle) have access to information generated during their absence.
• Reference information: Where person-to-person information exchange usually varies according to the context it is used in, documentation based information never changes unless somebody actively updates the documentation. This makes documentation a more stable reference platform, than person based information, which has a tendency to vary more depending on who is delivering the information when. The variations in the information consumers interpretation of the information provided is of coarse another matter.

Costs of documentation

• Needs work: Compared to human memory based information, the production requires a significant amount of work. This isn't just a resource issue, but may also remove the focus from the real problem being solved, turning the effort into documentation production instead. An example is the development af an application design. The real problem being address here is the developing an efficient application design and applying this to the application implementation. But this can with sufficiently complex design guidelines and inefficient tools turn into a struggle to fill out document templates and placed the produced documents in the right configuration managed structure. The result often being outdated design documentation, never really driving, nor reflecting the application development.
  
• Outdated: One of the mentioned benefits of documentation based information, was the stability of the documentation over time. This may be a good ting in case of a static project, eg. project where the project information doesn't change over time, but projects are always changing, so the static aspect of documentation is degrades proportional to the dynamics of the contained information. The degrading is either caused by the documentation becoming outdated, or by the effort need to maintain the documentation.
• Information retrieval: Several of the benefits mentioned above addresses availability of documentation, eg. you can always find the documentation. The problem is that this apparent quality, doesn't translated into the ability of being able to find the information needed. The information retrieval capabilities of documentation are far inferior to a human ability to interpret and answer questions. This means that the apparent qualities of universal documentation retrieval functionality will be degraded severely by the inability to quickly find the relevant information.
• Reference interpretation: As mentioned under the documentation benefits, the static nature of document based references doesn't ensure consistent interpretation of the reference information. My claim is that it is actually often possible for an compentant human reference responsible to provide a much easier to understand explanation of how apply the reference information to the situation at hand. Eg. the design says the applcation should look like this, but how does this affect this bit of code I'm working on.

Need driven documentation

All in all there isn't any clear answers on what to document, and it is therefore very difficult to define a standard documentation structure (even though a lot of 'standard' process frameworks attempt to do this). Instead you should cultivate a more agile/intelligent approach in your organization as to what documentation should be generated.

One thing you have to keep in mind is that the (manual) production of a document doesn't contribute anything to adding knowledge to a project, the sole purpose is to retain information according to the the benefits mentioned earlier.

One way of generating just the right amount and type of documentation, is to try to avoid writing documentation before you need it. This doesn't necessary mean activity = document artifact, but that the documentation shouldn't be generated until the distribution and statical benefits of documentation based information artifact becomes apparent. A very efficient mechanism for achieving this, is to consider whether a piece of information should be found in the stored documentation, every time a person asks you a question regarding the project. If the information should be found in the documentation, refer the person to the documentation. This can have several outcomes:

• The relevant documentation is found, everything is great.
• The relevant documentation is found, but is obsolete. Here you would update the documentation most of the time. If you chose to not update the documentation, you should consider removed it, avoiding wasting another people time reading invalid documentation.
  
• The relevant documentation exists, but isn't found. Here you should consider whether the way of finding this bit of documentation, eg. add links to information, improve searchability, switch to more powerful tools.
• The relevant documentation doesn't exist. Consider whether the documentation should be produced at this time. Remember, on possible answer to this could be that it isn't worth the effort, I'll just answer the question.

If stick to an documentation generation approach like this, will properly only produce a minimum of documentation, but still over time generate the documentation needed to run the project. This is of course only possible in a very iterative project, very the needs of different kinds of documentation occurs on a repetitive bases, instead of deferring the usage of the documentation until right at the end.

Examples of need driven documentation
An example of this is the test specifications, these should be used continuously throughout the project, not just at an big bang acceptance test at the (apparent) final delivery at the project. Therefore test specification documentation should be produced right before or during the implementation of the functionality addressed in the tests. Using tests in this manor is the foundation for Test Driven Development, where you could says the TDD concept is actually a synergy between implementation and test, because the test specification should also be driven by implementation activities.

Another example is the production of architecture and design documentation. In the 'good old days' these where considered essential to a prober application implementation. But with the event of the more pragmatic agile approach to software development, the awareness of a more application development driven architecture and design emerged. The result is that the value of architecture and design documentation is greatly reduced, and the absence of such documentation isn't any longer considered a sure sign of an chaotic application.

Other documentation drivers
Of course you can't just ignore the more conventional drivers for generating documentation. We all start with a idea of what documentation could be useful in a project, the organization the development project is part of, usually also have some input to what should be produced. But my point here is that these inputs should all be seen in the context of the cost and benefits of documentation, and the decisions on what documentation to produce and maintain should should be based primarily on intelligent need driven ad-hoc decisions, as opposed to poorly understood upfront QA or organizational standards requirements/advices.