You are here

10 stress-free steps to fix even the scariest bugs

public://pictures/rouan.png
Rouan Wilsenach, Software Developer, TES Global

You've got a product bug, and the pressure is on. Your users are upset, there's a group of grumpy managers huffing nearby, and you're somehow expected to keep your cool and get to the bottom of what's gone wrong.

Fixing production bugs can be scary—and overwhelming. The trick is to create the room you need to get your job done and work methodically. Here are some tricks to help you stay calm and fix even the most terrifying production bug.

[ Get Report: Buyer’s Guide to Software Test Automation Tools ]

Give yourself the room you need to debug properly

Some people may thrive under pressure, but most developers work much better when they aren't stressed out. The first thing to do when faced with a scary production bug is to try create room for yourself to work without pressure. Here are some ways to do that.

1. Acknowledge the issue publicly

Unless the issue you've found is security-related, tell your users that something is wrong. As a user, nothing is worse than not knowing what's happening. Tell them, so they're in the know.

GitHub is a good example of a website with a status page that lets its users know when it's having issues. If you work in some ChatOps magic, you could send a bot a message to give your users status updates.

[ More on ChatOps: ChatOps essential guide: The basics, benefits, and challengesHow to put ChatOps to work in your organization ]

2. Find a quick fix

If you're lucky, you’ll get an idea for something you can do relatively quickly to patch up the issue. If this is the case, don't worry about finding the perfect solution—just ship that fix! Once it's live, you’ll have fewer affected users, and the grumbling managers can go grumble elsewhere while you dig a bit deeper to find out what's caused the problem—and fix it properly.

3. Find a workaround

Sometimes the functionality that's broken isn’t the only way to achieve a task. If that’s the case, use whatever communication channels you have available to let users know alternative ways of getting what they need so they can keep working. The workaround will probably be clumsy and slow, and it may involve doing things offline (e.g., over the phone or in person), but at least you've given them an option.

4. Switch off broken features

If the payment feature on your website is broken, remove the "Pay" button. If the search feature is broken, remove the search bar. Sure, users still won't be able to achieve what they want to, but at least the functionality that remains is something they can have confidence in, and you won't be inundated with new logs and alerts.

If the bug is continually causing damage, such as creating bad data, this is a great way to stop things from getting worse. In the worst-case scenario, you can put a big "we're down for maintenance" message up instead of your home page, and/or redirect to your company's Facebook or Twitter page.

Work methodically

Now that you've cut through a bit of the chaos, you should have more time to focus and get the job done. I find that having a list of things to work through helps me feel more in control. Here's a little list of steps to take when you’re starting to dig into the cause of a bug.

1. Get a second set of eyes on the problem

Two pairs of eyes are better than one, so find someone to come work with you. This will help keep you on track and make it less likely that you'll follow dead ends.

If you're lucky, your partner will know the system as well as you do. But if not, he or she will still help you stay on track—and will be learning about the system, which means there might be less pressure on you next time around. Even if all the second person does is shoo away nosy managers, you'll be grateful!

2. Reproduce the issue

The best way to set yourself up for success is to reproduce the issue, ideally locally. Bugs become much less mysterious once you can make them occur at will. It also means you'll know for sure when you've fixed them, because you'll be able to demonstrate the bug occurring—and then not occurring—before and after you apply the patch.

3. Eliminate the obvious

Taking five minutes to make a little list of the most obvious potential causes can be a huge timesaver, because it's often the things of which we subconsciously say, "Nah, that can't be it," that are to blame.

Developers tend to have very good intuition when it comes to what is causing issues. But they often dismiss their own suspicions without sharing them. Write them down to force yourself to consider them.

4. List all the layers and players

It can be easy to forget a part of a system that you don't often change or think about, such as a load balancer or a piece of middleware that validates a user's cookie. Making a list of the stack that a user request passes through can help you broaden your thinking and find potential culprits you might otherwise miss.

5. Find what's changed

Bugs that have existed for a long time without having a negative impact will likely have come to your attention because something changed. Look at the source control and deployment histories to see what has changed recently.

Are there any suspicious commit messages or deployments? I often find bugs faster this way.

If you don't spot anything obvious, remember to look at your list of layers and players, because the change might not have been made in the code you look after. Ask around to find out if other systems have recently had changes deployed.

6. Think about next time

Once you've fixed a bug, take a bit of time to make changes that will improve life for your future self. Write a test that will catch the bug, should it crop up again. Add the logging or documentation you wish you'd had. You'll thank yourself later!

[ Webinar: How to Fit Security Into Your Software Lifecycle With Automation and Integration ]

Bugs don't have to be scary

Fixing production bugs is often daunting because of the way we react to them. It doesn't have to be that way. Remember to start by focusing on doing things that remove pressure from the situation. Then, armed with a list of steps to help you stay calm and focused, you'll fix bugs faster and be less stressed while doing it.

[ Get Report: How to Get the Most From Your Application Security Testing Budget ]