Release Notes
Filed under: Development, Management, Process, Product Management, Project Management, Quality Assurance, Templates
Software is never releases, it escapes. Usually when it escapes, it comes attached with a listing of caveats and other information for the user which are listed in release notes.
There is a big difference in the internal defect description and the external description. For instance, internally a bug may have the summary, “XML Parser occasionally dies horrible death when comment field improperly terminated”. The external release note could be, “The XML input functionality requires that the XML is well formed. Usage of poorly formed XML may result in termination of the application.”
The selection of content for release notes is defined by the culture of the organization and the relationship with the customer. I have been involved with the two extremes of release noting, where at one end we documented almost nothing, and the other we listed everything. When choosing your philosophy, keep in mind the following:
- Release Notes Leak
Each release note you write will end up in the hands of the sales force of the competition. Your competition will lie about your product, this just gives them more ammunition to confuse the customer and increase the duration of the sales cycle.
- Release Note Visibility
You need to know who has access to the release notes. Some release notes are published on the internet, while others are hidden inside of the product installer. If the release notes are visible to those who have not purchased the product, it may be time to get marketing involved in their creation as the notes could affect product demand. If the release notes are only visible to those who have purchased the product, you can be much more open.
- Does the customer care about this issue?
Yes, it is an important question to ask. Often dev will flag an issue for release noting because they think it is very important, but often they do not consider the program usage by the average user. I know that the program will crash if the XML being imported does not have the comment terminated, but I think it is safe to not release note this since the XML being imported is almost never edited by user. Development means well be asking for the release note, but after they have spent so much time working inside of the program, they often lose perspective.
- What happens if we don’t release note this?
There are consequences for customers finding bugs on their own. It is usually better for the customer to be provided advance notice of the issues so that they can plan their work flows around the defects.
- Will documenting this issue kill our chances of winning the product evaluation?
This is the one I hate the most. You should be proud of the product and acknowledge its short comings. The customer will find them out sooner or later. You will look more professional in the long run if you tell them the issues ahead of time rather than them finding them out themselves.
In one organization, I was specifically told not to mention any defects in the release notes because we were in the middle of product evaluations. This fostered a sense of mis-trust with the evaluators, who would constantly call and complain to me about the quality of the product. I finally convinced the organization to change their philosophy. Once we started producing real release notes containing useful information, their perception of our product quality by the evaluators actually went up. Before knowing the all the defects, they thought that almost all areas of the program were unstable. By telling them specifically what the supported scenarios were, it helped create trust and converted them from an evaluator to customer.
- Does this make the company and product look really bad?
If it makes the company look bad and the product bad, it is time to consider changing it from the “Non-Gating, Release Note” to “Gating”.
- Can this be explained concisely?
Why can the issue note be explained concisely? Perhaps it is to obscure of an issue to release note.
When in doubt, more is less. By trusting in your customers, they will trust in you.
A sample release note template is contained in the template section.
Formal Bug Triage
Filed under: Development, Management, Process, Product Management, Quality Assurance
The formal triage is when we get the usual suspects together to triage the bugs. These usually occur on regular interval during the main product release cycle and more frequently when we near a product release. It is important to have a set recurring meeting for the triage because it helps ensure that the meeting happens. There is nothing more demoralizing than entering a triage with a backlog 512 un-triaged issues because the triage has not happened for a while.
I have found for an effective bug triage you need Development, Quality Assurance, and Product Management roles filled in the room.
Development is there to help explain the impact of the bug. They are also provide push back for items which will take a long time, and often argue for including items which are very quick fixes. Development should discuss the risks and impact (including testing) of fixing the issue.
Quality Assurance represents the person who found the bug. They are there to provide information on the severity and context of the bug.
The Product Manager is the owner of the triage. They are the advocate for the customer and own the product. The final decision of severity, priority, and milestone is theirs.
In a good triage group, there will be health lively discussion between quality assurance and development over each bug. I love a good lively short debate on the important bugs. Development arguing that the bug is a trivial edge case, with Quality Assurance saying that it is the end of the world. Product management listens to the case for each bug, and makes the ultimate decision based on the customer’s needs, corporate delivery schedule, and product road map.
The triage room needs to have a projector to display the bug database and access to the latest build. It is usually good to do this in a room close to the Development and Quality Assurance team since it will allow you to easily grab a developer when they are needed to help explain a bug.
When triaging a bug, you need to ask yourself several questions (which are in no particular order) for each bug. The result of the answers to each of these questions will help you determine the priority, severity, and milestone for the issue.
- Will the customer ever find this issue?
- How often will a customer come across it?
- What is the impact of fixing (or not fixing) it?
- Are there political reasons for fixing (or not fixing) it?
- Does this bug block a customer demo scenario?
- Does this bug make the demos look bad?
- Is this bug a result of a real world user scenario?
- If a customer found this, would we need to immediately spin a hot fix?
- Is this a crazy bug which is only found under unrealistic QA environments?
- If we do not fix this now, what will be the impact on future releases?
- What is the impact to the current schedule of fixing it now?
- What is the schedule impact of fixing it in the future?
There are several people who should not be in the triage room. If a person has never run a development build, odds are they should not be included in the triage meeting request. Never let marketing, finance, or any CxO into the room. It will kill your ability to quickly go through the bugs.
