Keeping it clean
Filed under: Development, Management, Process, Product Management, Project Management
The bug databases which are used as they should be, are very susceptible to having a back log of old issues (bug rot). These old issues will grow and grow as time marches on. Any reasonable person who examines of the size and requests in the database results in the conclusion that they can’t all be resolved.
During a triage we ask, “When should we fix this issue?” The response typically is either, “Now” or “Next Release”. Many people are afraid to ask this question, “Will we ever fix it?” There are some issues that we will actually fix in the future, but that list will always be smaller than what we won’t fix in the future. If we are never going to fix them, why do we keep them around in an open state?
There are several reason people choose to push a bug to the “Next Release” instead of choosing not to fix it. By telling a person we aren’t planning on fixing your issue, it leads to longer triage meetings since you will have to have a more vigorous discussion on a bug closure. It is much easier to placate with a, “We’ll fix it in the future, we’ve got a release to get out the door right now”.
I’m a person who enjoys stats and the bug database. When the bug database is properly used by the developers, it is a good way for me to get a feel as to the quality and progress of a release. The result of choosing to move your issues to the “Next Release” instead of considering closure will cost you in the long run in the following ways:
-
Groundhog day
We have to constantly discuss these same issues again and again is insane. When we are releasing M6, we are moving many issues to M7 since we know that we won’t have time to resolve them in M6. Once we start development of M7, we go over all the M7, moving many of them to M8. Repeat this process again and again. This results in us having to discuss these same issues again and again. Those issues that we continually bump will never get fixed. Close them as “WONTFIX”, man up and just do it. -
Zarro boogs found
It clutters up the view for the developers. It is much easier for a bug to get forgotten when you have 200 assigned to you, instead of only 22. Seeing that my bug list empty, is just as satisfying as all of my unit tests as green. -
Component Health
Measuring the health of your components is much more difficult. With all of these old issues cluttering the database, the view of the software components health can become skewed. I could see in the database that there are 420 bugs against “Component A” which could be in a much better state than “Component B” with only 80 bugs against it. -
Projections
I use the database to do bug glide slope projections to help me determine if we are converging or diverging. Carrying a large amount of bugs make the projections more cluttered.
I will say that there has been a couple of times where I’ve broken the “WONTFIX” rule. There have been requests from the senior management team which are clearly impossible (usually defying the law of physics), and no amount of discussion can get them to drop it. These bugs get logged, and targetting approatiely. We typically visit these in the begining of the release cycle to ensure that they are given due consideration so that when we can provide an intelligent response.
Release Notes
Filed under: Development, Management, Process, Product Management, Project Management, Quality Assurance, Templates
Software is never releases, it escapes. Usually when it escapes, it comes attached with a listing of caveats and other information for the user which are listed in release notes.
There is a big difference in the internal defect description and the external description. For instance, internally a bug may have the summary, “XML Parser occasionally dies horrible death when comment field improperly terminated”. The external release note could be, “The XML input functionality requires that the XML is well formed. Usage of poorly formed XML may result in termination of the application.”
The selection of content for release notes is defined by the culture of the organization and the relationship with the customer. I have been involved with the two extremes of release noting, where at one end we documented almost nothing, and the other we listed everything. When choosing your philosophy, keep in mind the following:
- Release Notes Leak
Each release note you write will end up in the hands of the sales force of the competition. Your competition will lie about your product, this just gives them more ammunition to confuse the customer and increase the duration of the sales cycle.
- Release Note Visibility
You need to know who has access to the release notes. Some release notes are published on the internet, while others are hidden inside of the product installer. If the release notes are visible to those who have not purchased the product, it may be time to get marketing involved in their creation as the notes could affect product demand. If the release notes are only visible to those who have purchased the product, you can be much more open.
- Does the customer care about this issue?
Yes, it is an important question to ask. Often dev will flag an issue for release noting because they think it is very important, but often they do not consider the program usage by the average user. I know that the program will crash if the XML being imported does not have the comment terminated, but I think it is safe to not release note this since the XML being imported is almost never edited by user. Development means well be asking for the release note, but after they have spent so much time working inside of the program, they often lose perspective.
- What happens if we don’t release note this?
There are consequences for customers finding bugs on their own. It is usually better for the customer to be provided advance notice of the issues so that they can plan their work flows around the defects.
- Will documenting this issue kill our chances of winning the product evaluation?
This is the one I hate the most. You should be proud of the product and acknowledge its short comings. The customer will find them out sooner or later. You will look more professional in the long run if you tell them the issues ahead of time rather than them finding them out themselves.
In one organization, I was specifically told not to mention any defects in the release notes because we were in the middle of product evaluations. This fostered a sense of mis-trust with the evaluators, who would constantly call and complain to me about the quality of the product. I finally convinced the organization to change their philosophy. Once we started producing real release notes containing useful information, their perception of our product quality by the evaluators actually went up. Before knowing the all the defects, they thought that almost all areas of the program were unstable. By telling them specifically what the supported scenarios were, it helped create trust and converted them from an evaluator to customer.
- Does this make the company and product look really bad?
If it makes the company look bad and the product bad, it is time to consider changing it from the “Non-Gating, Release Note” to “Gating”.
- Can this be explained concisely?
Why can the issue note be explained concisely? Perhaps it is to obscure of an issue to release note.
When in doubt, more is less. By trusting in your customers, they will trust in you.
A sample release note template is contained in the template section.
Past, People, and Process
Filed under: Development, Process, Project Management
The first software release at a new company is usually the hardest to plan. The good thing is that it gets easier with time because you will gain knowledge of how things have been delivered in the past, an understanding of their process, and most importantly and knowledge of the people.
- Past
When I enter a new organization, very rarely do they have historical delivery data, and when they do it is not of much use. I usually get verbal dump saying that the releases are usually (insert time frame here) late. When I ask why, sometimes there is a why.
The first order of business is to start keeping stats. The best time to start is as soon as you walk in the building. For each major feature which is in the release, track the original estimate, estimators, along with all of the refined and updated estimates for the break down of the work. Ensure you gather the actual times it took along with the people who actually did the work.
It is important during crunch time to keep stats as well. For each test cycle, stats such as the number incoming bugs, their severity, and the actual time it took to fix them are very valuable. It is very difficult to estimate an incoming defect rate prior to a test cycle without this information.
- People
From the data, over time you will start to see trends for your project and individuals. You will see that your projects are typically X weeks late on delivery, and that developers on average Y percent off on their estimates.
- Process
Another important aspect is process. How well does the process help the product delivery? Does senior management allow the team to finish a task, or are they constantly distracting them and trying to add additional projects during crunch time (I’ll save those horror stories for another post).
With this information, you will be able to asses the release situation more accurately and the risks will become more evident. These stats are very important and will come into play in a future post.
Formal Bug Triage
Filed under: Development, Management, Process, Product Management, Quality Assurance
The formal triage is when we get the usual suspects together to triage the bugs. These usually occur on regular interval during the main product release cycle and more frequently when we near a product release. It is important to have a set recurring meeting for the triage because it helps ensure that the meeting happens. There is nothing more demoralizing than entering a triage with a backlog 512 un-triaged issues because the triage has not happened for a while.
I have found for an effective bug triage you need Development, Quality Assurance, and Product Management roles filled in the room.
Development is there to help explain the impact of the bug. They are also provide push back for items which will take a long time, and often argue for including items which are very quick fixes. Development should discuss the risks and impact (including testing) of fixing the issue.
Quality Assurance represents the person who found the bug. They are there to provide information on the severity and context of the bug.
The Product Manager is the owner of the triage. They are the advocate for the customer and own the product. The final decision of severity, priority, and milestone is theirs.
In a good triage group, there will be health lively discussion between quality assurance and development over each bug. I love a good lively short debate on the important bugs. Development arguing that the bug is a trivial edge case, with Quality Assurance saying that it is the end of the world. Product management listens to the case for each bug, and makes the ultimate decision based on the customer’s needs, corporate delivery schedule, and product road map.
The triage room needs to have a projector to display the bug database and access to the latest build. It is usually good to do this in a room close to the Development and Quality Assurance team since it will allow you to easily grab a developer when they are needed to help explain a bug.
When triaging a bug, you need to ask yourself several questions (which are in no particular order) for each bug. The result of the answers to each of these questions will help you determine the priority, severity, and milestone for the issue.
- Will the customer ever find this issue?
- How often will a customer come across it?
- What is the impact of fixing (or not fixing) it?
- Are there political reasons for fixing (or not fixing) it?
- Does this bug block a customer demo scenario?
- Does this bug make the demos look bad?
- Is this bug a result of a real world user scenario?
- If a customer found this, would we need to immediately spin a hot fix?
- Is this a crazy bug which is only found under unrealistic QA environments?
- If we do not fix this now, what will be the impact on future releases?
- What is the impact to the current schedule of fixing it now?
- What is the schedule impact of fixing it in the future?
There are several people who should not be in the triage room. If a person has never run a development build, odds are they should not be included in the triage meeting request. Never let marketing, finance, or any CxO into the room. It will kill your ability to quickly go through the bugs.
The Hiring Process
Hiring is one of the biggest challenges faced. This process I have successfully used many times. This is a macro post of the process, detailed descriptions of the individual steps may follow in future posts. It is a combination of processes from Joel Spolsky, Michael Loop, Guy Kawasaki, and coworkers of mine. The following image shows the entire process:
Gathering Inputs
Once you have your req, it is time to get the word out. Get the word out to everyone that you are hiring. Ensure that people within the company know as well. Use sources such as linkedin as well to inform your network that you are hiring. Pretty much anyone who comes to me through a referral jumps straight to interview day. If someone in my network is willing to vouch for the candidate, I’ll interview them.
Give me all the possible candidates. HR offers to pre-screen the resumes, but I prefer to receive all of them. One of the best guys I worked with had a degree in geology, what are the odds he would have made it through the HR filter? So, give me all the resumes and I’ll make my own decision.
Lets start off first with the huge pile of resumes. I create a directory which all resumes go into. I go through each resume and sort them into three buckets, {No Interview, Maybe, Interview}. Once we have all the candidates, I will review all of the Maybe’s again against the Interview pile.
If you have made it past here, congratulations, your odds are hugely increased.
Prescreen
The purpose of the prescreen is to ensure that you are able to communicate with people in the real world and have represented yourself fairly accurately on your resume. I do not perform the prescreen. HR will contact you with a script, which is usually the same for each candidate. HR does not know the answers to the script, they are simply recording the results. When HR presents me with the results, I also ask for the “soft skills” rating. I’m looking to find out if the candidate can communicate their ideas to a non-technical person. The candidate does not need to be able to explain the complexities of their thesis to HR, but should be able to boil down their thesis a 30 second description of the problem or solution.
If the candidate appears to check out ok, HR will setup the interview.
Interview Day
The interview day is a long day. Many people advocate day long interviews, but I’ve never been able to block off that much time for my team. We typically do half day interviews. I have listed the interviews in the order which I prefer to run them, but due to scheduling constraints, the ordering usually changes a bit. I try to schedule all the interviews as closely together as possible. A rule I have during interviews is that people do not discuss the interview impressions until we get together for the review meeting.
The introduction is to give the candidate a context for the company along with help on how to calibrate your answers. Listen carefully in this, I’m giving the candidate subtle hints on what I am hoping to hear. Along with the brief company bio, I’ll usually do the elevator pitch for the company.
The management interview is where I am looking to see what level of developer you are. How do you estimate, how much testing do you do on your code, etc. The goal of this is to determine if I think you can work with the myself and the team. I’ll be looking to determine what areas you will be contributing to the team and the areas we will need to work together on.
Technical interview #1 is hard. Buckle up, it will be a ride. We want to know how much you actually know. This is where you hope you have not lied on your resume. We will pick an area of your resume and drill down on it. These questions will be in an area where the interviewer has an in-depth knowledge. We are looking to:
- Gauge your knowledge range
- How you have learned in the past
- Capability of getting things done
- Is the candidate afraid of saying “I don’t know”?
- How well do they respond to criticism?
Yes, everyone is on their best behavior during an interview, but it can be surprisingly easy to coax out truth when people are challenged.
The HR interview is where we are trying to sell you on the company. HR will present our company as a great place to work. They will discuss our benefits and determine what it will take for the candidate to choose us.
Technical interview #2 is my favorite interview. This is where we allow the candidate to choose a technical area, and explain it in-depth. I love this interview because it allows me to learn about a new topic and evaluate the candidate at the same time. The purpose of this is to figure out what the candidate has accomplished before, what challenges appeared, how they overcame the challenges, and that they can explain their accomplishments with pride to us.
Now that the interview is almost done, it is time to bring in the closer. No matter if the candidate is going to be made an offer, the closer always comes in to sell. Their purpose is to close the interview and sell the candidate on the position. The closer will typically give a product demonstration and answer any final questions the candidate has.
The review meeting should happen soon after the candidate leaves. Prior to the review meeting, all participants send their interview result notes to me. During the meeting, everyone speaks about the candidate and a consensus is usually found quickly.
Selection
The group who has performed all the interviews is gather again one last time to provide input on the candidates. We will rank the candidates and setup the final interviews.
Decision and Negotiation
Many places have a member of the senior management team who reviews each new hire. This persons concern is corporate culture. This is a rubber stamp interview. They are simply their to sell the company and get a feel for how you will contribute to the company culture. After this, we will meet to get down to negotiation. Here, we will discuss salary, benefits, options, vacation,and everything else. The letter of offer is not the place to do this. The letter of offer is to confirm in writing our agreement in the meeting.
Signing Off
It is important to get your legal and HR department to move quickly. Once you have a verbal agreement with the candidate, move as quickly as possible to get the offer out. What I find successful is to ask them to wait for a few minutes after our negotiation phase. I’ll go to HR and get them to print the offer letter at that time along with the employment contacts.
It is important for hiring manager to read the employment contract as much as the candidate. Very often these are copy / pasted and contain mistakes. If there are mistakes, you will unfortunately be the one responsible for fixing them.
When the candidate comes in to sign the contracts, go over the important areas of the contract with them. Ensure that they understand what they are signing.
Second Thoughts
After communicating with a candidate only three times, it is hard to ensure that you have selected the right candidate. Most employment contracts in my region have a standard 3 month at will employment clause, where either party can terminate employment for any reason with no notice. Termination during this phase is 100 times easier than afterwords, so ensure that you evaluate the candidate to ensure you have the right person.
At the four week mark, discuss with the employees supervisor and peers to determine how the candidate is doing. Most of the time, everyone will still be in the honeymoon period, but you may see some small signals.
At the ten week mark, discuss with the employees again to determine if we have chosen the right person.
