Optimization During the Holidays: Risk Management

Running a testing program during the holiday season can be a little like trying to drink from a fire hose. You may have lots of traffic and many inbound activities giving you much greater exposure, but you also have the chaos of everything else. From the upsurge in gift buyers to the dreaded holiday code freeze, which can sometimes result in revenue opportunity loss, there is always something to contend with.

Having been through more than a dozen of these seasons in my ecommerce career, I felt it would be of value sharing a few things I have learned about the holiday season. The challenges of optimization during the holidays can be succinctly categorized into two areas: risk management and changes in customer behavior. In this post, I’ll cover risk management.

Risk Management Prevents Risky Business

For anyone who has worked for a retailer of any size, the yearly code freeze comes as no surprise. Conventional web development practices will advise locking down any changes to site code to minimize the risk of a show-stopping bug during the time of year that your company may be moving from red to black. While we all need to be conscious of risks, we also need to rethink what we call “development” and do a code classification exercise before the holidays to identify and determine what can be changed with minimal exposure to potential bugs.

Here’s one example. While driving the testing strategy at a top fashion retailer, I learned that the holidays accounted for the lion’s share of online revenue. The company had spent the last few years struggling to find their strategy footing during these times, while simultaneously trying to extinguish the many fires that sprang up. I knew that the coming season was the best opportunity to gather knowledge about their customers, as well as develop high impact campaigns. With that in mind, I approached our development team to review what I could and could not update after our November 1st code freeze. Their answer was, “Aside from changing banners and products, nothing.”

Despite this, I set out to take a deeper dive into how we could maintain the most development capability to make changes while keeping our risks low. To begin, we asked a few key questions for every testing activity that we wanted to perform:
<ul=class tri>

• Are we changing code, or content? – Editing text or images is very different from editing more in-depth functionality, and involves much less risk.
• Have the changes been made before? – Most testing platforms have reusable “actions” (in the case of Monetate) or “offers” (Adobe Target) that allow you to make content edits to a change that has already cycled through the QA process. Since only the content, such as an image or text, is changed, adjusting actions during the holidays is relatively low risk.
• Is the area we are changing on the primary shopping path? – Making changes to a content page is very different than changes to the product or category pages. Often, static-content pages won’t have any code that can affect other parts of the site. This is heavily dependent upon how the page was built, and what ecommerce platform you are working within, if any.

<ul=class tri> Following a series of discussions, we were ultimately able to get an agreement from our development team to continue testing in about 80% of the areas we had laid out on our roadmap. In fact, only our plans for more advanced UI testing, such as navigation elements and headers, were impacted.

Measuring risk internally for your own ecommerce operation now will help you to better develop your optimization roadmap for the rest of the year. Performing a code classification exercise is key. You will be able to determine which critical or higher-risk projects to tackle prior to your code freeze, and which projects can continue during the busy season with minimal impact to website sales and performance.