Keeping it clean

The bug databases which are used as they should be, are very susceptible to having a back log of old issues (bug rot). These old issues will grow and grow as time marches on. Any reasonable person who examines of the size and requests in the database results in the conclusion that they can’t all be resolved.

During a triage we ask, “When should we fix this issue?” The response typically is either, “Now” or “Next Release”. Many people are afraid to ask this question, “Will we ever fix it?” There are some issues that we will actually fix in the future, but that list will always be smaller than what we won’t fix in the future. If we are never going to fix them, why do we keep them around in an open state?

There are several reason people choose to push a bug to the “Next Release” instead of choosing not to fix it. By telling a person we aren’t planning on fixing your issue, it leads to longer triage meetings since you will have to have a more vigorous discussion on a bug closure. It is much easier to placate with a, “We’ll fix it in the future, we’ve got a release to get out the door right now”.

I’m a person who enjoys stats and the bug database. When the bug database is properly used by the developers, it is a good way for me to get a feel as to the quality and progress of a release. The result of choosing to move your issues to the “Next Release” instead of considering closure will cost you in the long run in the following ways:

Groundhog day
We have to constantly discuss these same issues again and again is insane. When we are releasing M6, we are moving many issues to M7 since we know that we won’t have time to resolve them in M6. Once we start development of M7, we go over all the M7, moving many of them to M8. Repeat this process again and again. This results in us having to discuss these same issues again and again. Those issues that we continually bump will never get fixed. Close them as “WONTFIX”, man up and just do it.

Zarro boogs found
It clutters up the view for the developers. It is much easier for a bug to get forgotten when you have 200 assigned to you, instead of only 22. Seeing that my bug list empty, is just as satisfying as all of my unit tests as green.

Component Health
Measuring the health of your components is much more difficult. With all of these old issues cluttering the database, the view of the software components health can become skewed. I could see in the database that there are 420 bugs against “Component A” which could be in a much better state than “Component B” with only 80 bugs against it.

Projections
I use the database to do bug glide slope projections to help me determine if we are converging or diverging. Carrying a large amount of bugs make the projections more cluttered.

I will say that there has been a couple of times where I’ve broken the “WONTFIX” rule. There have been requests from the senior management team which are clearly impossible (usually defying the law of physics), and no amount of discussion can get them to drop it. These bugs get logged, and targetting approatiely. We typically visit these in the begining of the release cycle to ensure that they are given due consideration so that when we can provide an intelligent response.

Are we there yet

Asking developers on an hourly basis if they are done, will NOT make the fix come faster. Just because you are receiving pressure from above, doesn’t give you a license to kill productivity for all of those around you. Now is when you need to control the situation and get things done by helping your team. This is not the time to request for up to the minute status updates and TPS reports. Interrupting your developer’s concentration constantly doesn’t help them work. Try the following instead:

• Make sure the correct amount of people are dedicated to the problem. There are many times when less is more (I’m not just talking about the editor), where the right two people are infinitely more effective than the wrong 20 people.

• Distractions must be minimized to ensure their focus doesn’t drift from the current work to other priorities.

• Provide the stakeholders with accurate updates on progress. If there is any heat or anger related to the issue, take it for the team so that they can focus on fixing the issue.

• Ensure the developers understands the issue, context, and time lines.

• Remove roadblocks soon as quickly as you can (or ahead of time if possible).

• Get the developers what they need to get the job done right.

Clear the path in front of them and let them work, don’t hold them back.

When not to ship software

When planning a product release, you always choose a date. Choosing a date is difficult, and here are some dates which I suggest you try to avoid.

End of the Quarter
If you choose your shipping date as the last day of the quarter, your boss will tell the executive team in his next meeting. Your boss is now on the hook for delivering at the end of the quarter, whether the executives care or not.

This isn’t the worst one because if you are late, you can usually get your way out of it by doing a demo to their boss. This demo, plus some nice words can usually buy you more time. The good thing about the demo is that it gives you a good wake up call of how far along you really are.

End of Fiscal Year
Never choose a date close to the end of the fiscal year. There is a chance that accounting may create forecasts of sales for what you are building. Once they make these forecasts, then it will be much more difficult to get the date modified.

The end of fiscal year also causes problems due to vacations. Many companies limit vacation carry over, forcing people to burn up their vacation before year end.

Christmas
The last two weeks of December and first week of January is a bad time to try to get anything done. People have a lot going on in their lives, and are not as focused on work as they typically are. Even if your direct staff plans around the release, the accompanying staff will not. There is nothing more difficult than trying to get people around Christmas time. Also, executives will be off as well, so this will make it difficult to get sign off on a product delay.

End of the Summer
People will take vacation in the summer, and even if you plan it so that each person is covered off during their vacation, it will result in pain. Shipping time is when you need all of your bench strength, and not people doing two jobs while covering someone.

In a future post I’ll try to address which date to choose.

Sunshine Report

Sunshine Report: A status report which only reflects the positive feelings one is having.

For the most part, I have found that engineers tend to focus on problems in their status reports, but the first person who gave me sunshine reports took me by surprise. These next few paragraph detail my first enlightening experience with the sunshine report.

1st Status Report
When asked for his status, he responded that, “Everything is great, I’m looking into this one issue, but I’m confident that I’ll be able to get everything done on time.” My first thought was wow, a part of this project is going well. I recorded his report and moved on to my next task.

2nd Status Report
I asked him for his status report, he said again that everything is good. He was still looking into this one issue, but is confident that fixing this issue will resolve many other issues he has assigned to him. According to him, everything will still get completed on-time. At this point, warning bells started to sound. I was concerned that he is still on the same issue and that this issue is starting to spread to other sections of the code. But, he still seemed positive about it. I let it go to see where he would get to next week.

3rd Status Report
Now, this time I came to the meeting much more prepared. When I asked him how it was going, he was still one big happy ball of sunshine and confident that it would be done on time. I thought differently. He had just spent three weeks working on the same issue. I began to question him much more in-depth about his work. Driving down on the issues. After only about 5 minutes of drill down, he started to come to the realization that he was in over his head. We worked together to come up with a plan to get him back on track.

After that, I monitored his project more closely. Unfortunately on the next project, when prompted for a status report he still gave “Sunshine Reports”, but it just took a few minutes to cut through the sunshine to get to the truth.

Demo Driven Development

• Your company is living paycheck to paycheck.
• No one has actually paid for your product.
• Product direction changes each time sales visits a lead.

Given the above situation, it is time to plan a software release. The PMBOK doesn’t cover this kind of project. There is no chapter for near Armageddon events. Planning a software release for a company which is just starting out or in a death spiral is not covered in classical project management literature.

The development plan will be:

The name of the game is to make it looks like you have what the customers needs as quickly as possible. You do not know your market, you do not know your product, you know nothing. Through many failures, the company will discover the market. The hope is that there is enough time and cash to realize this goal.

Someone on the sales team manged to get on the calendar of someone with budget and a pain large enough that they will spend 1 hour to see if you can help them. The sales team has provided you with the release date, and a feature set.

As usual, break down the tasks, figure out what is possible, drop everything that is not (ensure that you record what was not done, and that it came in from sales, not the customer directly). Then get started. While developing it is important to remember that if you are successful, this demo will most likely become the foundations of your code base. Also, this software on demo day will not be touched by anyone outside of your company (in its current form), it just need to be good enough that they believe you can deploy it when they purchase it. I am not saying to fake the demo, I’m saying that you need to focus on the core pain points and that you can illustrate the solution to the potential customer. Many times, before the demo, the requirements will shift when the sales guys enlightens you more on the problem that the potential customer has. Those changes need to be incorporated. You are trying to demonstrate that your product can satisfy their needs. Do the changes as best as you can. Refusing to demonstrate a product which can help a potential customer is a waste of both your time and theirs. No one loves the churn, but you need to suck it up and do it.

During the demo, ensure that you are able to listen to the feedback while it is happening. This is the closest thing to users you have. They will provide you with the next set of work you need to do on your program. Record all their feedback (product ideas and bugs). Discuss internally all their ideas and make a plan for addressing the issues. Start working on them as soon as possible because you have up until the sales team schedules the next demo.

Once the next demo is scheduled, all of the focus shifts towards it. When demo goes well, but the potential customer is not interested (this is the most likely outcome), record all the feedback. Triage all of the features (including the previous demos) and begin work immediately based on what appears to be the most important in your potential customers eyes which is blocking the sale. Hopefully after several demos you will begin to see pattern and specific items and themes which re-occur. The future of the company depends on if you can solve the customers problem before the company closes shop.

Release Notes

Software is never releases, it escapes. Usually when it escapes, it comes attached with a listing of caveats and other information for the user which are listed in release notes.

There is a big difference in the internal defect description and the external description. For instance, internally a bug may have the summary, “XML Parser occasionally dies horrible death when comment field improperly terminated”. The external release note could be, “The XML input functionality requires that the XML is well formed. Usage of poorly formed XML may result in termination of the application.”

The selection of content for release notes is defined by the culture of the organization and the relationship with the customer.  I have been involved with the two extremes of release noting, where at one end we documented almost nothing, and the other we listed everything.  When choosing your philosophy, keep in mind the following:

Release Notes Leak

Each release note you write will end up in the hands of the sales force of the competition. Your competition will lie about your product, this just gives them more ammunition to confuse the customer and increase the duration of the sales cycle.

Release Note Visibility

You need to know who has access to the release notes. Some release notes are published on the internet, while others are hidden inside of the product installer. If the release notes are visible to those who have not purchased the product, it may be time to get marketing involved in their creation as the notes could affect product demand. If the release notes are only visible to those who have purchased the product, you can be much more open.

Does the customer care about this issue?

Yes, it is an important question to ask. Often dev will flag an issue for release noting because they think it is very important, but often they do not consider the program usage by the average user. I know that the program will crash if the XML being imported does not have the comment terminated, but I think it is safe to not release note this since the XML being imported is almost never edited by user. Development means well be asking for the release note, but after they have spent so much time working inside of the program, they often lose perspective.

What happens if we don’t release note this?

There are consequences for customers finding bugs on their own. It is usually better for the customer to be provided advance notice of the issues so that they can plan their work flows around the defects.

Will documenting this issue kill our chances of winning the product evaluation?

This is the one I hate the most. You should be proud of the product and acknowledge its short comings. The customer will find them out sooner or later. You will look more professional in the long run if you tell them the issues ahead of time rather than them finding them out themselves.

In one organization, I was specifically told not to mention any defects in the release notes because we were in the middle of product evaluations. This fostered a sense of mis-trust with the evaluators, who would constantly call and complain to me about the quality of the product. I finally convinced the organization to change their philosophy. Once we started producing real release notes containing useful information, their perception of our product quality by the evaluators actually went up. Before knowing the all the defects, they thought that almost all areas of the program were unstable. By telling them specifically what the supported scenarios were, it helped create trust and converted them from an evaluator to customer.

Does this make the company and product look really bad?

If it makes the company look bad and the product bad, it is time to consider changing it from the “Non-Gating, Release Note” to “Gating”.

Can this be explained concisely?

Why can the issue note be explained concisely? Perhaps it is to obscure of an issue to release note.

When in doubt, more is less. By trusting in your customers, they will trust in you.

A sample release note template is contained in the template section.

Past, People, and Process

The first software release at a new company is usually the hardest to plan.  The good thing is that it gets easier with time because you will gain knowledge of how things have been delivered in the past, an understanding of their process, and most importantly and knowledge of the people.

Past

When I enter a new organization, very rarely do they have historical delivery data, and when they do it is not of much use.  I usually get verbal dump saying that the releases are usually (insert time frame here) late.  When I ask why, sometimes there is a why.

The first order of business is to start keeping stats.  The best time to start is as soon as you walk in the building.  For each major feature which is in the release, track the original estimate, estimators, along with all of the refined and updated estimates for the break down of the work.  Ensure you gather the actual times it took along with the people who actually did the work.

It is important during crunch time to keep stats as well.  For each test cycle, stats such as the number incoming bugs, their severity, and the actual time it took to fix them are very valuable.  It is very difficult to estimate an incoming defect rate prior to a test cycle without this information.

People

From the data, over time you will start to see trends for your project and individuals.  You will see that your projects are typically X weeks late on delivery, and that developers on average Y percent off on their estimates.

Process

Another important aspect is process.  How well does the process help the product delivery?  Does senior management allow the team to finish a task, or are they constantly distracting them and trying to add additional projects during crunch time (I’ll save those horror stories for another post).

With this information, you will be able to asses the release situation more accurately and the risks will become more evident.  These stats are very important and will come into play in a future post.

• Categories

  • Big Company
  • Crazy
  • Development
  • End User
  • HR
  • Intro
  • Management
  • Process
  • Product Management
  • Project Management
  • Quality Assurance
  • Small Company
  • Templates

• Projzilla

  • Bringing Microsoft Project and Bugzilla Together

• Blogroll

  • A VC
  • Cranky Product Manager
  • Doug on the Eclipse CDT
  • Guy Kawasaki
  • How to be a good product manager
  • Lessons Learned
  • Lispian
  • Mark on Virtualization
  • OnStartups
  • Product Marketing
  • Rands In Repose
  • Startup Marketing
  • Steve Blank
  • Venture Hacks
  • Why does everything suck?

• Blog under the Creative Commons Attribution-NonCommercial-ShareAlike 3.0 License