In this post I discuss the problem of sequencing from a customer first perspective and present a basic calculation upon which experimentation can occur. I explore a few interesting questions to advance sequence relevance, and I touch upon how customisation can offer a better solution.

What is Sequencing?

Suppose customer x is looking for a coffee table on Dunelm.com. There are typically 2 ways they could find what they are looking for: search or navigation. In each case, Dunelm shows customer x a list of products. How is the order of these generated? We define the sequence of a set of products to be the order that they are listed on an ecommerce webpage. Sequencing is the process of defining the sequence. In this post, I’m specifically interested in product sequencing on a lister page and not a search results page.

Why is sequencing important?

Numerous studies on customer behaviour onsite suggest that humans don’t spend much time scrolling and reading web pages. In the following UX research study conducted by the Nielsen Norman group, 74% of viewing time was spent in the top two screenfuls of page content. In an age of more distraction and increased customer demands, customers want easier shopping. As retailers, how can we show the customer what they want at the top of the page, without expecting them to scroll? This is the problem of product sequencing.

How could Dunelm solve the sequencing problem?

The easiest approach would be to randomly sequence our coffee tables, in the hope that customer x finds one they like near the top of the page. But why risk it. How we can do better?

To answer that, we need to dig down into what customer x wants and identify which coffee tables they are most likely to buy. As retailers, we’re aiming to improve their customer experience, making it easier to shop. Filtering is common on pages with a sufficient number of products, enabling customers to filter by colour, material, price or brand. Let’s assume filtering preferences have been selected and we’re still left with a number of products to sequence. What’s our approach?

Without knowing anything more about customer x, a good start is to consider products that have been most relevant for previous shoppers. Statistically, we’d expect the same products to have the highest probability of being bought by customer x.

We can track metrics such as Impressions, Click Through Rate, Conversion Rate, Product Returns or Cancellations, Bounce Rate, Time on Page, Customer Reviews, Product Availability, Product Exclusivity or Newness. The longer we track these for, the more information we have to base our decisions on.

A few notes to guide us and for further enquiry:

• A high conversion rate suggests that the product is popular amongst customers who view it. Lift these products to see whether they maintain a good conversion rate above the fold.
• A high % of returns or cancellations suggests that there are issues with these products, or that the product is not what the customer expects. Push further down the sequence until resolved.
• A high bounce rate could suggest that the imagery is misleading or that the product details are lacking.
• A low time on page may depend on the context. What is the content length? We may expect more customer consideration on high value items.
• Not all products have reviews – how should we show these to customers? How do customers weight both the number of reviews, with the average star rating? Is a 1 and a 5 better than two 3’s?
• If the product isn’t in stock, should we show it customers? Remove from the sequence altogether? We need to be clear with customers if a product isn’t currently available. How could we deal with products with limited stock?
• New products will have less customer behaviour data around them than products which have been on site longer. How should we balance showing customers what is new and exclusive vs the current best sellers?
• In a sale period, how can we best display our offers to customers in the product lister pages?
• What is the impact of pagination? Are customers more likely to look at the top of page 2 than the bottom of page 1? Our sequencing can be set up to reflect this.

Enter the Weighted Average

Suppose we’ve gathered the last 3 months of performance data for each of our coffee tables. Let’s look at an example to demonstrate what we can do with it. Below are 5 Dunelm coffee tables with mock data for some of the metrics we’ve discussed.

I wrote about the benefits of weighted averages in my post about Assessment Centre Scoring. First, there’s one issue to fix: scale. In our table, we see that some metrics have a much wider variance of scores. Average rating will always be between 0 and 5, Conversion Rate is a percentage between 0 and 100 and time on page could take any positive value, limited only by our reporting timeframe.

To fix this, we can use a normalisation method. Normalisation is the process of scaling values to between a standard range, typically between 0 and 1. Importantly, order is preserved, and we can normalise to a number of different underlying distributions. To keep things simple, we could use min-max feature scaling as follows.

We make an exception to this formula for any metrics where a low score is better, such as the bounce or return rates. In this case, I’ll use the following variant:

Normalisation complete, and with binary values in the Stock and New columns, we have the following table and can begin to sequence our products.

In a simple case, suppose we base the sequencing calculation on 3 metrics: Conversion Rate, Bounce Rate and whether the product is ‘New’ or not. Newness is a topic of discussion later on; for now, I’ve left it deliberately vague.

With our metrics decided, all that’s left to do is decide the weightings – which factor do we rate to be the most significant? For a vector x, of ranking factors of length n, and a vector w, of weightings summing to 1, we define the Rank Score R to be their dot product. That is:

Finally, sequence the products in descending order of R. As we saw in the Assessment Centre Scoring post, the weightings within vector w can hugely impact the final ranking. Taking the following as an example, we see that a greater emphasis on newness alters the sequencing significantly.

From here, experimenting with various weighting vectors and evaluating the relevance to customers is a sensible approach.

The basics covered; I’d like to explore how we can improve the customer experience with additional factors.

Reviews

How should we weight reviews in our sequencing calculation? Two metrics at hand are the number of reviews, and the average star rating. Which matters more to customers?

Take a look at the two review summaries below – which coffee table do you buy?

Easy, right? The Bromley dominates the Lancaster; it has more reviews and the average rating is higher. Factoring in the price differences confirms the decision.

What about these next two; which table do you prefer in this case?

Not so straightforward. The Fulton has more reviews than the Hastings, but a lower average rating. This example presents a dilemma, neither of the coffee tables dominates the other in both metrics.

Factoring in the prices of these two coffee tables may change your mind again?

Ask a friend which table they would buy and you may have different opinions. Is your view typical of a wider public sample?

Unfortunately, the experts are split on the issue. Academics found in this 2017 paper that the majority of people choosing between books or films were more rating inclined, that is, put greater weighting on the average star rating. However, this article by UX researchers at the Baymard Institute found that customers placed greater emphasis on the number of reviews.

And what about the situation with no reviews? This probably isn’t a coffee table for the risk averse, so how should we give it the opportunity to sell to interested customers?

With the lack of unanimous conclusion, the best advice for retailers is to conduct their own testing. The most helpful review ranking will consider both the average star rating and the number of reviews to highlight products recommended by the crowd.

Stock

How should stock levels impact the products sequence?

It is really annoying to find a product you like, only for it to be out of stock both online and in-store! Either customers should have the option to view only products in stock, or out of stock products should be removed from the lister by default. I think the first option is preferable, provided the toggle or filter is clear. This gives customers the option to wait for restocking of their favourite products.

Newness

Why upweight newness? Without it, the calculation is self-fulfilling for established products. Under a sensible ranking calculation, we’d expect historic bestsellers to feature near the top of the lister. They receive more customer views and sell in larger volume, so remain at the top! Newness also appeals to customers who want to keep on trend, we should give it a chance at the top until we have enough data to make a more informed decision on its popularity.

In the example above, Newness was binary, defined as a yes or a no, and encoded with a 0 or a 1. I stated that this was deliberately vague. When is a product no longer ‘new’? After a set number of days? After a set number of views? A step beyond this is to use one-hot encoding to bucket products into varying levels of newness and assign different w values to these. Here’s an example:

For increased precision and control over the uplift of new products, an exponential decay function could be applied to all products. For any time, t, the lift could be calculated by the following expression:

We would set k as required to vary how we push newness for different lister pages, depending on the importance of newness to the customer e.g. Cushion Covers vs Frying Pans.

Personalisation: Basket Analysis and Customer Segments

So far, our discussion has assumed that we don’t know anything about customer X. This all changes once they start adding products to their basket, thanks to the power of association rules.

Association rules can identify collections of products frequently bought together. Given a set of historic transaction data, we can find relations and could use this information to adapt our sequencing. In the example below are 6 different customer baskets each containing the product ‘Pillowcase 1’. We see that ‘Pillow 2’ is 4 times more frequently bought with ‘Pillowcase 1’ than either of the other pillows.

If customer X adds Pillowcase 1 to their basket and then navigates to the Pillow lister page, we could move Pillow 2 to the top of the sequence, with the expectation that this would give customer X an easier shopping experience. Another example where this method could be useful is for furniture collections or matching décor.

If customers had an account, then we could adjust and personalise the sequencing before any products are added to basket. Through the CRM lens of customer segments, we would be able to give greater weighting to products which appealed to similar customers. Whilst taking a sample of customers would mean fewer data points, we may find the sequencing relevancy is still improved.

A Final Note:

We’ve talked a lot about sequencing and the weighting we give to various metrics and factors. There seems no end to complexity we could apply. Sounds great, but it’s worth being wary; checking that each adjustment is better for the customer than before.

A final point to be aware of is the differentiation between local and global maxima. Changing the weightings is an optimisation problem, in each step we aim to improve page performance. Beware that we could be homing in on the wrong target, and that a better solution lies in the opposite direction. Are we stuck on a hill in the shadow of a mountain?

Until next time,

Scott