Release Notes
Filed under: Development, Management, Process, Product Management, Project Management, Quality Assurance, Templates
Software is never releases, it escapes. Usually when it escapes, it comes attached with a listing of caveats and other information for the user which are listed in release notes.
There is a big difference in the internal defect description and the external description. For instance, internally a bug may have the summary, “XML Parser occasionally dies horrible death when comment field improperly terminated”. The external release note could be, “The XML input functionality requires that the XML is well formed. Usage of poorly formed XML may result in termination of the application.”
The selection of content for release notes is defined by the culture of the organization and the relationship with the customer. I have been involved with the two extremes of release noting, where at one end we documented almost nothing, and the other we listed everything. When choosing your philosophy, keep in mind the following:
- Release Notes Leak
Each release note you write will end up in the hands of the sales force of the competition. Your competition will lie about your product, this just gives them more ammunition to confuse the customer and increase the duration of the sales cycle.
- Release Note Visibility
You need to know who has access to the release notes. Some release notes are published on the internet, while others are hidden inside of the product installer. If the release notes are visible to those who have not purchased the product, it may be time to get marketing involved in their creation as the notes could affect product demand. If the release notes are only visible to those who have purchased the product, you can be much more open.
- Does the customer care about this issue?
Yes, it is an important question to ask. Often dev will flag an issue for release noting because they think it is very important, but often they do not consider the program usage by the average user. I know that the program will crash if the XML being imported does not have the comment terminated, but I think it is safe to not release note this since the XML being imported is almost never edited by user. Development means well be asking for the release note, but after they have spent so much time working inside of the program, they often lose perspective.
- What happens if we don’t release note this?
There are consequences for customers finding bugs on their own. It is usually better for the customer to be provided advance notice of the issues so that they can plan their work flows around the defects.
- Will documenting this issue kill our chances of winning the product evaluation?
This is the one I hate the most. You should be proud of the product and acknowledge its short comings. The customer will find them out sooner or later. You will look more professional in the long run if you tell them the issues ahead of time rather than them finding them out themselves.
In one organization, I was specifically told not to mention any defects in the release notes because we were in the middle of product evaluations. This fostered a sense of mis-trust with the evaluators, who would constantly call and complain to me about the quality of the product. I finally convinced the organization to change their philosophy. Once we started producing real release notes containing useful information, their perception of our product quality by the evaluators actually went up. Before knowing the all the defects, they thought that almost all areas of the program were unstable. By telling them specifically what the supported scenarios were, it helped create trust and converted them from an evaluator to customer.
- Does this make the company and product look really bad?
If it makes the company look bad and the product bad, it is time to consider changing it from the “Non-Gating, Release Note” to “Gating”.
- Can this be explained concisely?
Why can the issue note be explained concisely? Perhaps it is to obscure of an issue to release note.
When in doubt, more is less. By trusting in your customers, they will trust in you.
A sample release note template is contained in the template section.
Past, People, and Process
Filed under: Development, Process, Project Management
The first software release at a new company is usually the hardest to plan. The good thing is that it gets easier with time because you will gain knowledge of how things have been delivered in the past, an understanding of their process, and most importantly and knowledge of the people.
- Past
When I enter a new organization, very rarely do they have historical delivery data, and when they do it is not of much use. I usually get verbal dump saying that the releases are usually (insert time frame here) late. When I ask why, sometimes there is a why.
The first order of business is to start keeping stats. The best time to start is as soon as you walk in the building. For each major feature which is in the release, track the original estimate, estimators, along with all of the refined and updated estimates for the break down of the work. Ensure you gather the actual times it took along with the people who actually did the work.
It is important during crunch time to keep stats as well. For each test cycle, stats such as the number incoming bugs, their severity, and the actual time it took to fix them are very valuable. It is very difficult to estimate an incoming defect rate prior to a test cycle without this information.
- People
From the data, over time you will start to see trends for your project and individuals. You will see that your projects are typically X weeks late on delivery, and that developers on average Y percent off on their estimates.
- Process
Another important aspect is process. How well does the process help the product delivery? Does senior management allow the team to finish a task, or are they constantly distracting them and trying to add additional projects during crunch time (I’ll save those horror stories for another post).
With this information, you will be able to asses the release situation more accurately and the risks will become more evident. These stats are very important and will come into play in a future post.
Best … Demo … Ever …
Filed under: Big Company, Crazy, Development, End User, Small Company
There are certain aspects of a good demo, but it all comes down to practice, practice, practice.
Lets set the stage. After starting coding four month ago, we were running out of cash. There was a potential VC deal on the horizon, but it wasn’t going to happen until we had a sale. This was `the demo` to our only real customer in the pipeline. We had all the right people in the room for the demo. The customer had made it clear in our last demo what the sales objections were going to be, and that they had to be resolved for this next demo. It was a hail mary pass with two seconds left on the clock.
Lets dissect the days before the demo.
Day: t – 14
We gave a demo to our early adopter evangelist on the inside. He provided valuable feedback and pointed out the areas which will be the objection points. We scheduled the demo two weeks from that day. Several show stopper bugs, but we have two weeks to fix them.
Day: t – 13 to t – 3
Code like you’ve never coded before. Keep cranking it out. Don’t stop. Fix all those bugs.
Day: t – 2
We created the demo script. Although the meeting was to be 1 hour in length, the demo part was scheduled only to be 10 minutes long. The script covered all the users important pain points along with the requested objection points. All of the show stopper bugs were resolved except for one. We still have one day till code freeze to fix that nasty bug.
Day: t -1
24h before the demo was to begin, we froze the code. From now on, it was nothing but practice. That nasty bug is still there.
Now, about that bug. If an item was dragged in our interface, it would crash the entire application. Think BSOD. The only way to prevent the crash was to drag it to an exact pixel co-ordinates. Think of having a 1280×1024 resolution screen and having to drag to an unmarked x,y co-ordinate. If you missed, the application could crash and the company could fail. No pressure.
Day: t – 0
After our demo’er practiced nearly for a day, we were ready. He typically was able to hit the pixel about 50% of the time. Hedging our bets, I made sure we had two applications running at the same time, with the second setup to be at the exact point right after the crash should occur in the script. If a crash should occur, hopefully we can <ALT-Tab> before anyone notices. The demo went perfectly. He hit the pixel right on, no crash!
Best demo ever!
Formal Bug Triage
Filed under: Development, Management, Process, Product Management, Quality Assurance
The formal triage is when we get the usual suspects together to triage the bugs. These usually occur on regular interval during the main product release cycle and more frequently when we near a product release. It is important to have a set recurring meeting for the triage because it helps ensure that the meeting happens. There is nothing more demoralizing than entering a triage with a backlog 512 un-triaged issues because the triage has not happened for a while.
I have found for an effective bug triage you need Development, Quality Assurance, and Product Management roles filled in the room.
Development is there to help explain the impact of the bug. They are also provide push back for items which will take a long time, and often argue for including items which are very quick fixes. Development should discuss the risks and impact (including testing) of fixing the issue.
Quality Assurance represents the person who found the bug. They are there to provide information on the severity and context of the bug.
The Product Manager is the owner of the triage. They are the advocate for the customer and own the product. The final decision of severity, priority, and milestone is theirs.
In a good triage group, there will be health lively discussion between quality assurance and development over each bug. I love a good lively short debate on the important bugs. Development arguing that the bug is a trivial edge case, with Quality Assurance saying that it is the end of the world. Product management listens to the case for each bug, and makes the ultimate decision based on the customer’s needs, corporate delivery schedule, and product road map.
The triage room needs to have a projector to display the bug database and access to the latest build. It is usually good to do this in a room close to the Development and Quality Assurance team since it will allow you to easily grab a developer when they are needed to help explain a bug.
When triaging a bug, you need to ask yourself several questions (which are in no particular order) for each bug. The result of the answers to each of these questions will help you determine the priority, severity, and milestone for the issue.
- Will the customer ever find this issue?
- How often will a customer come across it?
- What is the impact of fixing (or not fixing) it?
- Are there political reasons for fixing (or not fixing) it?
- Does this bug block a customer demo scenario?
- Does this bug make the demos look bad?
- Is this bug a result of a real world user scenario?
- If a customer found this, would we need to immediately spin a hot fix?
- Is this a crazy bug which is only found under unrealistic QA environments?
- If we do not fix this now, what will be the impact on future releases?
- What is the impact to the current schedule of fixing it now?
- What is the schedule impact of fixing it in the future?
There are several people who should not be in the triage room. If a person has never run a development build, odds are they should not be included in the triage meeting request. Never let marketing, finance, or any CxO into the room. It will kill your ability to quickly go through the bugs.
Worst Interview Ever
Filed under: Big Company, Crazy, HR, Management, Small Company
Some people are used to large companies, and others small. This candidate was from a large company and didn’t realize that a large company and small company operate differently. Many of the items individually are not that bad, but the combination of all of them all made for the worst interview ever.
We were interviewing for a developers. It was a typical software development role. We had already done one interview in the morning, and this candidate was scheduled for the afternoon time slot.
His resume was quite different than other resumes I had reviewed. A bullet point at the top of the resume was, “Actively recruited by BigCompany Inc.” It also appeared to be on company letter head. Weird, but he seemed to have some of the skills we need and came from the recruiter.
Johnny arrives, I welcome him to the office and offer him a beverage. The third sentence out of his mouth was, “Is my Mercedes ok parked out front?”. Sure it is, I park my 9 year old Toyota out there all the time.
Two sentences later, he says that he had won the university congressional medal of honor for high academics. He then proceeds to spend five minutes discussing how rare an award it is and that is presented to the student with the highest mark in the class. I asked my coworker in the interview about the medal (I obviously have never won such a distinction, but know that my coworker had pretty good grades). He said that he has one, two of the other developers have one, and another developer has two (Bachelor and Masters). Damn, now that I think of it, we really need to get a trophy case in the front lobby to show off all the medals of our employees. The rest of the interview went ok, he appeared to be an average candidate from the interview.
We had troubles finding candidates for the interview, so we were considering an offer to him. A bit of context before I get into his demands. Johnny, a software engineer currently has a job at a large company, had graduated two years ago from his undergraduate, and had two years of professional experience. His demands were the following:
Vacation
Four weeks vacation. This is a little on the high side. We were typically offering three weeks, but we can work with vacation and salary to come up with a package which works for everyone.
Parking
He wanted us to pay for his indoor parking for his Mercedes. The company did not pay for anyone’s indoor parking.
Salary
His salary expectations were 50% higher than what our range and the other candidates.
Guaranteed Salary
This was the kicker. He wanted a guarantee of 1 years salary no matter what happens. Yes, if for some reason the company should implode, or he gets fired, or any reason what so ever, he is guaranteed at least one year of salary.
Needless to say, this candidate was not hired.
The Hiring Process
Hiring is one of the biggest challenges faced. This process I have successfully used many times. This is a macro post of the process, detailed descriptions of the individual steps may follow in future posts. It is a combination of processes from Joel Spolsky, Michael Loop, Guy Kawasaki, and coworkers of mine. The following image shows the entire process:
Gathering Inputs
Once you have your req, it is time to get the word out. Get the word out to everyone that you are hiring. Ensure that people within the company know as well. Use sources such as linkedin as well to inform your network that you are hiring. Pretty much anyone who comes to me through a referral jumps straight to interview day. If someone in my network is willing to vouch for the candidate, I’ll interview them.
Give me all the possible candidates. HR offers to pre-screen the resumes, but I prefer to receive all of them. One of the best guys I worked with had a degree in geology, what are the odds he would have made it through the HR filter? So, give me all the resumes and I’ll make my own decision.
Lets start off first with the huge pile of resumes. I create a directory which all resumes go into. I go through each resume and sort them into three buckets, {No Interview, Maybe, Interview}. Once we have all the candidates, I will review all of the Maybe’s again against the Interview pile.
If you have made it past here, congratulations, your odds are hugely increased.
Prescreen
The purpose of the prescreen is to ensure that you are able to communicate with people in the real world and have represented yourself fairly accurately on your resume. I do not perform the prescreen. HR will contact you with a script, which is usually the same for each candidate. HR does not know the answers to the script, they are simply recording the results. When HR presents me with the results, I also ask for the “soft skills” rating. I’m looking to find out if the candidate can communicate their ideas to a non-technical person. The candidate does not need to be able to explain the complexities of their thesis to HR, but should be able to boil down their thesis a 30 second description of the problem or solution.
If the candidate appears to check out ok, HR will setup the interview.
Interview Day
The interview day is a long day. Many people advocate day long interviews, but I’ve never been able to block off that much time for my team. We typically do half day interviews. I have listed the interviews in the order which I prefer to run them, but due to scheduling constraints, the ordering usually changes a bit. I try to schedule all the interviews as closely together as possible. A rule I have during interviews is that people do not discuss the interview impressions until we get together for the review meeting.
The introduction is to give the candidate a context for the company along with help on how to calibrate your answers. Listen carefully in this, I’m giving the candidate subtle hints on what I am hoping to hear. Along with the brief company bio, I’ll usually do the elevator pitch for the company.
The management interview is where I am looking to see what level of developer you are. How do you estimate, how much testing do you do on your code, etc. The goal of this is to determine if I think you can work with the myself and the team. I’ll be looking to determine what areas you will be contributing to the team and the areas we will need to work together on.
Technical interview #1 is hard. Buckle up, it will be a ride. We want to know how much you actually know. This is where you hope you have not lied on your resume. We will pick an area of your resume and drill down on it. These questions will be in an area where the interviewer has an in-depth knowledge. We are looking to:
- Gauge your knowledge range
- How you have learned in the past
- Capability of getting things done
- Is the candidate afraid of saying “I don’t know”?
- How well do they respond to criticism?
Yes, everyone is on their best behavior during an interview, but it can be surprisingly easy to coax out truth when people are challenged.
The HR interview is where we are trying to sell you on the company. HR will present our company as a great place to work. They will discuss our benefits and determine what it will take for the candidate to choose us.
Technical interview #2 is my favorite interview. This is where we allow the candidate to choose a technical area, and explain it in-depth. I love this interview because it allows me to learn about a new topic and evaluate the candidate at the same time. The purpose of this is to figure out what the candidate has accomplished before, what challenges appeared, how they overcame the challenges, and that they can explain their accomplishments with pride to us.
Now that the interview is almost done, it is time to bring in the closer. No matter if the candidate is going to be made an offer, the closer always comes in to sell. Their purpose is to close the interview and sell the candidate on the position. The closer will typically give a product demonstration and answer any final questions the candidate has.
The review meeting should happen soon after the candidate leaves. Prior to the review meeting, all participants send their interview result notes to me. During the meeting, everyone speaks about the candidate and a consensus is usually found quickly.
Selection
The group who has performed all the interviews is gather again one last time to provide input on the candidates. We will rank the candidates and setup the final interviews.
Decision and Negotiation
Many places have a member of the senior management team who reviews each new hire. This persons concern is corporate culture. This is a rubber stamp interview. They are simply their to sell the company and get a feel for how you will contribute to the company culture. After this, we will meet to get down to negotiation. Here, we will discuss salary, benefits, options, vacation,and everything else. The letter of offer is not the place to do this. The letter of offer is to confirm in writing our agreement in the meeting.
Signing Off
It is important to get your legal and HR department to move quickly. Once you have a verbal agreement with the candidate, move as quickly as possible to get the offer out. What I find successful is to ask them to wait for a few minutes after our negotiation phase. I’ll go to HR and get them to print the offer letter at that time along with the employment contacts.
It is important for hiring manager to read the employment contract as much as the candidate. Very often these are copy / pasted and contain mistakes. If there are mistakes, you will unfortunately be the one responsible for fixing them.
When the candidate comes in to sign the contracts, go over the important areas of the contract with them. Ensure that they understand what they are signing.
Second Thoughts
After communicating with a candidate only three times, it is hard to ensure that you have selected the right candidate. Most employment contracts in my region have a standard 3 month at will employment clause, where either party can terminate employment for any reason with no notice. Termination during this phase is 100 times easier than afterwords, so ensure that you evaluate the candidate to ensure you have the right person.
At the four week mark, discuss with the employees supervisor and peers to determine how the candidate is doing. Most of the time, everyone will still be in the honeymoon period, but you may see some small signals.
At the ten week mark, discuss with the employees again to determine if we have chosen the right person.
