For a long time now, we’ve been communicating the power of Conversion Rate Optimisation (CRO) and it being a pivotal strategy for our clients to enhance their website’s User Experience (UX) with the end goal of driving measurable, improved performance KPIs.
Since conducting CRO tests, we’ve seen many clear-cut successes; however, in this blog article, I’ll share the work we’ve been doing on our most recent CRO test where the results haven’t been so clear cut. Although we were seeing positive conversion improvements online, the offline KPIs that followed on didn't match. This led us to take a wider view of analysis and determine if consumer behaviour and psychology were acting as a contributing factor.
Context for the CRO test
The CRO test in question is for a client’s website providing a service that, after submitting a 'sell my gold' request form, enables consumers to sell their unwanted gold by posting it to our client for a valuation. The consumer then has the choice of accepting the offer and receiving payment or rejecting the offer and having their gold posted back to them. The objective for our CRO test was clear; improve the UX on the 'sell my gold' request form to increase the conversion rate of form submissions.
Understanding the user journey
The website’s design and content were originally created with a streamlined user journey in mind, i.e. visitors who land already have a strong and clear intention to sell their unwanted gold. Users can quickly access all crucial information over one or two pages, evaluate our client’s credibility and once satisfied, there is a prominent sticky 'Sell My Gold Now' CTA, taking users to the 'sell my gold' form.
The simplicity of the website is advantageous, but it also poses a unique challenge. The effectiveness of the form is critical since it represents the key transitional element in the user journey, moving potential customers from just visiting the site, to actively engaging with our client’s services. Consideration of user behaviour played a significant role in our approach to this CRO test. We sought to understand potential barriers preventing users from completing the form and hypothesised that if we could reduce user friction, we should see significant increases in submission rates.
Testing new forms
To identify the optimal form design for maximising submissions, we set up a series of tests, using Varify to evenly distribute 25% of site traffic to each of the form variations:
- Variant 1. Control form: The original version of the sell my gold request form, as a baseline for comparison.
- Variant 2. Scaled form: A variant with a modified 'tightened up' layout to ensure all form fields and the submit button were visible without scrolling on page load – highlighting the ease of the form for users to fill in.
- Variant 3. Scaled form with no phone number form: A version that omitted the request for the customer’s telephone number entirely (research shows that users see telephone number as a barrier even if non-mandatory).
- Variant 4. Phone number explanation form: A variation that retained the phone number field but included supplementary text explaining why this information was required – providing reassurance to users if would not be used for any other purpose other than processing their request.
For completeness, all three variants that asked for the user’s telephone number were clearly labelled as a non-mandatory form field.
Monitoring performance and results
The test went live at the start of August and over the following two months of meticulous monitoring, we observed fluctuations in conversion rates across the forms. It’s worth noting that we did achieve statistical significance after just one month, however for due diligence, decided to extend the test to ensure that summer holidays over the testing period weren’t having a causal effect on results.
- Variant 2 - saw improvement increases over the control ranging from 1% to 6%.
- Variant 3 - saw improvement increases over the control ranging from 8% to 33%.
- Variant 4 - saw a decrease of performance against the control ranging from -1% to -7%.
It’s fair to say we were surprised Variant 4 always underperformed against the control. Typically, it’s best practice UX to inform the user for what purpose their telephone number is being asked for, especially if used as a reassurance that it won’t be used for other sales or marketing purposes. However, seeds of doubt around the credibility of the message could have arisen for users and may have played a factor. Equally, the phone message could've drawn more attention to the telephone field and distracted away from the visibility of the non-mandatory nature of the phone field - leading to the misunderstanding that users felt they had to submit their phone number to submit the form, not wanting to and so dropping off.
With two of the three variants always outperforming the control, we were proud of the results we were seeing. However, as I mentioned at the start of this article, the results weren’t as clear cut as they first appeared. Waiting for the CRO tests to achieve statistical significance post summer holidays also gave the 'sell my gold' pack requests time to mature, meaning we could see data on the percentage of customers (in each of the form variants) who were posting their unwanted gold to receive their cash offer. This ultimately is the key conversion for our client. Although form submissions on the website are of course important, this is just the lead generation part - the bit that really matters is how many packs are returned. 90%+ of these customers accept their cash offer.
Here’s where the plot thickens. Undeniably we were seeing great results of the CRO test, however as time went on and the pack requests matured, these results weren’t correlating to the rates customers were posting their gold selling packs. In fact, the control form resulted in a greater number of returned gold packs by customers which was never the winning CRO test on the website for form submissions. This outcome prompted deeper analysis into customer decision-making processes to 'complete the transaction'.
Unpacking the findings
The paradox of highest website form submissions but lower returns of gold selling packs from Variant 3 against the control can potentially be better understood through a psychological lens. By removing the telephone field, we made the form easier to complete. However, we may have inadvertently decreased perceived commitment from users. The act of providing a telephone number often fosters a sense of accountability and engagement, meaning users may subconsciously feel more motivated to follow through with the selling process if they share more personal information.
Additionally, users who submitted the control form could perceive our client as more credible and established, knowing they could easily and quickly be contacted on the phone with follow up communications during the selling process to then speed up the approving or rejecting of their cash offer.
To be continued…
The CRO test not only illustrates the complexities of user behaviour in seemingly simply digital interactions but also highlights the importance of thorough analysis when undertaking optimisation efforts and having all the facts to make a firm conclusion on whether a CRO test is a net success or not.
While streamlining the user journey across form submission is critical; understanding that customer behaviours can differ across digital and offline touch points is equally vital - the intent to complete the transaction can still fluctuate, new barriers can arise after the ones on the website have been removed.
We’ve since streamlined the CRO test by removing the two underperforming form variants and plan to further optimise the test by triggering an email to encourage consumers who don’t submit their telephone number to provide one.
