Keeping it clean
Filed under: Development, Management, Process, Product Management, Project Management
The bug databases which are used as they should be, are very susceptible to having a back log of old issues (bug rot). These old issues will grow and grow as time marches on. Any reasonable person who examines of the size and requests in the database results in the conclusion that they can’t all be resolved.
During a triage we ask, “When should we fix this issue?” The response typically is either, “Now” or “Next Release”. Many people are afraid to ask this question, “Will we ever fix it?” There are some issues that we will actually fix in the future, but that list will always be smaller than what we won’t fix in the future. If we are never going to fix them, why do we keep them around in an open state?
There are several reason people choose to push a bug to the “Next Release” instead of choosing not to fix it. By telling a person we aren’t planning on fixing your issue, it leads to longer triage meetings since you will have to have a more vigorous discussion on a bug closure. It is much easier to placate with a, “We’ll fix it in the future, we’ve got a release to get out the door right now”.
I’m a person who enjoys stats and the bug database. When the bug database is properly used by the developers, it is a good way for me to get a feel as to the quality and progress of a release. The result of choosing to move your issues to the “Next Release” instead of considering closure will cost you in the long run in the following ways:
-
Groundhog day
We have to constantly discuss these same issues again and again is insane. When we are releasing M6, we are moving many issues to M7 since we know that we won’t have time to resolve them in M6. Once we start development of M7, we go over all the M7, moving many of them to M8. Repeat this process again and again. This results in us having to discuss these same issues again and again. Those issues that we continually bump will never get fixed. Close them as “WONTFIX”, man up and just do it. -
Zarro boogs found
It clutters up the view for the developers. It is much easier for a bug to get forgotten when you have 200 assigned to you, instead of only 22. Seeing that my bug list empty, is just as satisfying as all of my unit tests as green. -
Component Health
Measuring the health of your components is much more difficult. With all of these old issues cluttering the database, the view of the software components health can become skewed. I could see in the database that there are 420 bugs against “Component A” which could be in a much better state than “Component B” with only 80 bugs against it. -
Projections
I use the database to do bug glide slope projections to help me determine if we are converging or diverging. Carrying a large amount of bugs make the projections more cluttered.
I will say that there has been a couple of times where I’ve broken the “WONTFIX” rule. There have been requests from the senior management team which are clearly impossible (usually defying the law of physics), and no amount of discussion can get them to drop it. These bugs get logged, and targetting approatiely. We typically visit these in the begining of the release cycle to ensure that they are given due consideration so that when we can provide an intelligent response.
When not to ship software
Filed under: Development, Product Management, Project Management
When planning a product release, you always choose a date. Choosing a date is difficult, and here are some dates which I suggest you try to avoid.
-
End of the Quarter
If you choose your shipping date as the last day of the quarter, your boss will tell the executive team in his next meeting. Your boss is now on the hook for delivering at the end of the quarter, whether the executives care or not.This isn’t the worst one because if you are late, you can usually get your way out of it by doing a demo to their boss. This demo, plus some nice words can usually buy you more time. The good thing about the demo is that it gives you a good wake up call of how far along you really are.
End of Fiscal Year
Never choose a date close to the end of the fiscal year. There is a chance that accounting may create forecasts of sales for what you are building. Once they make these forecasts, then it will be much more difficult to get the date modified.The end of fiscal year also causes problems due to vacations. Many companies limit vacation carry over, forcing people to burn up their vacation before year end.
Christmas
The last two weeks of December and first week of January is a bad time to try to get anything done. People have a lot going on in their lives, and are not as focused on work as they typically are. Even if your direct staff plans around the release, the accompanying staff will not. There is nothing more difficult than trying to get people around Christmas time. Also, executives will be off as well, so this will make it difficult to get sign off on a product delay.End of the Summer
People will take vacation in the summer, and even if you plan it so that each person is covered off during their vacation, it will result in pain. Shipping time is when you need all of your bench strength, and not people doing two jobs while covering someone.
In a future post I’ll try to address which date to choose.
Demo Driven Development
Filed under: Crazy, End User, Product Management, Project Management, Small Company
• Your company is living paycheck to paycheck.
• No one has actually paid for your product.
• Product direction changes each time sales visits a lead.
Given the above situation, it is time to plan a software release. The PMBOK doesn’t cover this kind of project. There is no chapter for near Armageddon events. Planning a software release for a company which is just starting out or in a death spiral is not covered in classical project management literature.
The development plan will be:
The name of the game is to make it looks like you have what the customers needs as quickly as possible. You do not know your market, you do not know your product, you know nothing. Through many failures, the company will discover the market. The hope is that there is enough time and cash to realize this goal.
Someone on the sales team manged to get on the calendar of someone with budget and a pain large enough that they will spend 1 hour to see if you can help them. The sales team has provided you with the release date, and a feature set.
As usual, break down the tasks, figure out what is possible, drop everything that is not (ensure that you record what was not done, and that it came in from sales, not the customer directly). Then get started. While developing it is important to remember that if you are successful, this demo will most likely become the foundations of your code base. Also, this software on demo day will not be touched by anyone outside of your company (in its current form), it just need to be good enough that they believe you can deploy it when they purchase it. I am not saying to fake the demo, I’m saying that you need to focus on the core pain points and that you can illustrate the solution to the potential customer. Many times, before the demo, the requirements will shift when the sales guys enlightens you more on the problem that the potential customer has. Those changes need to be incorporated. You are trying to demonstrate that your product can satisfy their needs. Do the changes as best as you can. Refusing to demonstrate a product which can help a potential customer is a waste of both your time and theirs. No one loves the churn, but you need to suck it up and do it.
During the demo, ensure that you are able to listen to the feedback while it is happening. This is the closest thing to users you have. They will provide you with the next set of work you need to do on your program. Record all their feedback (product ideas and bugs). Discuss internally all their ideas and make a plan for addressing the issues. Start working on them as soon as possible because you have up until the sales team schedules the next demo.
Once the next demo is scheduled, all of the focus shifts towards it. When demo goes well, but the potential customer is not interested (this is the most likely outcome), record all the feedback. Triage all of the features (including the previous demos) and begin work immediately based on what appears to be the most important in your potential customers eyes which is blocking the sale. Hopefully after several demos you will begin to see pattern and specific items and themes which re-occur. The future of the company depends on if you can solve the customers problem before the company closes shop.
Release Notes
Filed under: Development, Management, Process, Product Management, Project Management, Quality Assurance, Templates
Software is never releases, it escapes. Usually when it escapes, it comes attached with a listing of caveats and other information for the user which are listed in release notes.
There is a big difference in the internal defect description and the external description. For instance, internally a bug may have the summary, “XML Parser occasionally dies horrible death when comment field improperly terminated”. The external release note could be, “The XML input functionality requires that the XML is well formed. Usage of poorly formed XML may result in termination of the application.”
The selection of content for release notes is defined by the culture of the organization and the relationship with the customer. I have been involved with the two extremes of release noting, where at one end we documented almost nothing, and the other we listed everything. When choosing your philosophy, keep in mind the following:
- Release Notes Leak
Each release note you write will end up in the hands of the sales force of the competition. Your competition will lie about your product, this just gives them more ammunition to confuse the customer and increase the duration of the sales cycle.
- Release Note Visibility
You need to know who has access to the release notes. Some release notes are published on the internet, while others are hidden inside of the product installer. If the release notes are visible to those who have not purchased the product, it may be time to get marketing involved in their creation as the notes could affect product demand. If the release notes are only visible to those who have purchased the product, you can be much more open.
- Does the customer care about this issue?
Yes, it is an important question to ask. Often dev will flag an issue for release noting because they think it is very important, but often they do not consider the program usage by the average user. I know that the program will crash if the XML being imported does not have the comment terminated, but I think it is safe to not release note this since the XML being imported is almost never edited by user. Development means well be asking for the release note, but after they have spent so much time working inside of the program, they often lose perspective.
- What happens if we don’t release note this?
There are consequences for customers finding bugs on their own. It is usually better for the customer to be provided advance notice of the issues so that they can plan their work flows around the defects.
- Will documenting this issue kill our chances of winning the product evaluation?
This is the one I hate the most. You should be proud of the product and acknowledge its short comings. The customer will find them out sooner or later. You will look more professional in the long run if you tell them the issues ahead of time rather than them finding them out themselves.
In one organization, I was specifically told not to mention any defects in the release notes because we were in the middle of product evaluations. This fostered a sense of mis-trust with the evaluators, who would constantly call and complain to me about the quality of the product. I finally convinced the organization to change their philosophy. Once we started producing real release notes containing useful information, their perception of our product quality by the evaluators actually went up. Before knowing the all the defects, they thought that almost all areas of the program were unstable. By telling them specifically what the supported scenarios were, it helped create trust and converted them from an evaluator to customer.
- Does this make the company and product look really bad?
If it makes the company look bad and the product bad, it is time to consider changing it from the “Non-Gating, Release Note” to “Gating”.
- Can this be explained concisely?
Why can the issue note be explained concisely? Perhaps it is to obscure of an issue to release note.
When in doubt, more is less. By trusting in your customers, they will trust in you.
A sample release note template is contained in the template section.
Formal Bug Triage
Filed under: Development, Management, Process, Product Management, Quality Assurance
The formal triage is when we get the usual suspects together to triage the bugs. These usually occur on regular interval during the main product release cycle and more frequently when we near a product release. It is important to have a set recurring meeting for the triage because it helps ensure that the meeting happens. There is nothing more demoralizing than entering a triage with a backlog 512 un-triaged issues because the triage has not happened for a while.
I have found for an effective bug triage you need Development, Quality Assurance, and Product Management roles filled in the room.
Development is there to help explain the impact of the bug. They are also provide push back for items which will take a long time, and often argue for including items which are very quick fixes. Development should discuss the risks and impact (including testing) of fixing the issue.
Quality Assurance represents the person who found the bug. They are there to provide information on the severity and context of the bug.
The Product Manager is the owner of the triage. They are the advocate for the customer and own the product. The final decision of severity, priority, and milestone is theirs.
In a good triage group, there will be health lively discussion between quality assurance and development over each bug. I love a good lively short debate on the important bugs. Development arguing that the bug is a trivial edge case, with Quality Assurance saying that it is the end of the world. Product management listens to the case for each bug, and makes the ultimate decision based on the customer’s needs, corporate delivery schedule, and product road map.
The triage room needs to have a projector to display the bug database and access to the latest build. It is usually good to do this in a room close to the Development and Quality Assurance team since it will allow you to easily grab a developer when they are needed to help explain a bug.
When triaging a bug, you need to ask yourself several questions (which are in no particular order) for each bug. The result of the answers to each of these questions will help you determine the priority, severity, and milestone for the issue.
- Will the customer ever find this issue?
- How often will a customer come across it?
- What is the impact of fixing (or not fixing) it?
- Are there political reasons for fixing (or not fixing) it?
- Does this bug block a customer demo scenario?
- Does this bug make the demos look bad?
- Is this bug a result of a real world user scenario?
- If a customer found this, would we need to immediately spin a hot fix?
- Is this a crazy bug which is only found under unrealistic QA environments?
- If we do not fix this now, what will be the impact on future releases?
- What is the impact to the current schedule of fixing it now?
- What is the schedule impact of fixing it in the future?
There are several people who should not be in the triage room. If a person has never run a development build, odds are they should not be included in the triage meeting request. Never let marketing, finance, or any CxO into the room. It will kill your ability to quickly go through the bugs.
