Subscribe | Free AI | CRO Audit | Cool Tools

Thank you for your feedback about the Optimize newsletter. We are going to ask the question below to get feedback from those who did not get a chance to reply.

Which Test Won?

On a product page for a product that costs $300, which button variation will lead to more checkout conversions? (see the answer below)

AI Search Visibility Analytics

AI Visibility tracks the mentions of your website, product, or brand within AI search tools like ChatGPT, Gemini, etc. This is a difficult problem to solve or get an exact answer for because there is no open door to view the data in AI search systems.

Of course, you can try testing by asking an AI tool like ChatGPT to see what answer it will produce. The caveat is that this only captures branded questions. It doesn’t account for non-branded conversations that eventually lead back to you. For example, a user might ask an AI about summer vacation planning, which later turns into sunscreen recommendations. That path is difficult to trace or attribute.

Another option is to check your Google Console account to see where the traffic originates. Filtering by brand name, product name can also fine-tune what people were searching for. You can also get some data from Google Analytics referral tracking data.

In general, it’s much easier to track visibility for branded terms than for broader, non-branded AI-driven queries. That’s why it’s important to clearly mention your brand in FAQs and on-site content, so it has a better chance of being included in AI-generated answers.

Here are some tools to help with AI visibility analytics

• Googe Console Free

• Google Analytics 4 Referral Tracking Free

• Otterly.ai $

• Rankscale.ai $

• Otterly.ai $

• Nightwatch.io $

• Peec.ai $$

• SE Ranking $$

• Rankability $$

• Soogle.io $$

• LLMrefs.com $$

• Writesonic.com $$

• Anthenahq.ai $$

• Surferseo.com $$

• Hall $$

• Scrunch.com $$$

• AirOps.com $$$

• SEMRush $$$$

• SeoClarity $$$$

Which Test Won Answer

This test is a bit of a trick question. The first clue is in the phrase “product that costs $300.” Higher-priced items are more likely to be one-time purchases, and in those cases, reducing friction and decision-making often increases conversion rates. Presenting shoppers with both “Add to Cart” and “Buy Now” can sometimes hurt that conversion rate and even AOV because it forces an extra decision. What is interesting about this test is the difference between how mobile and desktop users behave. Here are the results for the test below:

Mobile: Variation A won with a lift of 70.58% at 96% statistical confidence
Desktop: Variation B won with a lift of 123.76% at 98% statistical confidence

The combined results were not statistically accurate but showed a slightly negative result for Variation B because the site had significantly more mobile traffic.

The reasoning: On mobile, “Buy Now” is a one-click step, which makes life easier for mobile users. On a desktop, the shopper tends to browse more and use the “add to cart” button as a wishlist until they are ready to make a purchase. This behavior may delay checkout, but it often results in higher AOV. In contrast, the quick decision process led mobile users to check out faster, but resulted in lower AOV.

Note that, as with all tests, you need to test this on your site before implementing it. Due to many factors, a test result from one site does not indicate it will be the same on another site.