6 lessons from user testing AI Shopping assistants

An AI shopping assistant is a conversational tool built directly into an online store. It acts as a digital 'personal shopper,' helping customers navigate the site and find the right products or services based on their needs. For instance, it can compare products or find specific details about a single item.

Conversionista conducted usability tests on two leading ecommerce stores to see how real users interact with on-site AI shopping assistants. Participants in the test first used traditional browsing/search to fin products, then performed the same tasks using the AI shopping assistant, if not already used in the first round.

The findings show that trust is fragile, a single incorrect answer often drives users back to ChatGPT or traditional search. We could also observe that many shoppers miss these tools entirely because they are used to old habits in how to navigate a website.

Read our six key lessons to learn how to optimize an AI assistant that is accurate, easy to find, and truly helpful to the customer journey.

1. Old habits die hard (and kill discoverability)

You might think your new "Chat with AI" button is impossible to miss. Users disagree.

28% of consumers saying they prefer AI assistants for simple queries, although most participants in our test didn't even notice the on-site assistant.

• The filter rut: Users instinctively went for the search bar or category filters because that’s how they’ve learned to shop over the last 20 years.
• Banner blindness 2.0: Users are so focused on finding their specific product that they tunnel-vision past the AI features.

Focus on this: Don't just place the AI shopping assistant in the header or at the top of the menu; integrate it contextually where the user is already looking for guidance.

2. ChatGPT has set a very high bar

Here is the harsh reality: Your on-site assistant isn't competing with your competitor's search bar; it's competing with ChatGPT.

Since general AI usage has grown rapidly over all, users bring expectations of fluency and intelligence to your site. Already established AI assistants or chat-bots set the bar and create high expectations.

• The comparison: Users expect detailed answers and the ability to ask complex follow-up questions.
• The abandonment: When a retailer’s AI shopping assistant felt limited or "bot-like," users stated they would rather leave the site and ask ChatGPT for advice instead.

Focus on this: Build an assistant that excels at what only you can offer: deep domain knowledge, real constraints, and helping users complete real tasks faster than leaving your site ever could.

3. Trust is fragile and easily broken

In traditional e-commerce, a bad filter result is an annoyance. In AI e-commerce, a bad answer breaks trust entirely.

Data shows that the number one use case for AI shopping assistants is "answering questions about products" (44%). If your AI shopping assistant fails at this primary task, you are in trouble.

During our tests, technical issues and data gaps made the assistants feel "not production-ready":

• Wrong answers: In one test, a user asked for a foundation and was served a lip tint.
• Not trustworthy: For complex tasks, users felt the AI shopping assistant was too vague or missed essential items, forcing them to double-check everything manually.

Focus on this: Prioritize accuracy over cleverness: limit the assistant to well-covered data and make uncertainty explicit. It’s better to say “I don’t know” or ask a clarifying question than to give a confident but wrong answer that destroys trust. If users feel they need to double‑check the AI’s answers, you haven't saved them time – you've just added a layer of anxiety.

4. The assistant falls apart when the UX doesn’t support the journey

Even when the AI logic is strong, the experience can still break down. In our tests, predefined prompts and vague responses lowered expectations early and made the assistant feel more limited than intelligent.

Missing items, unclear quantities and being rushed into the cart before they were ready, disrupted users’ flow and confidence. This led to uncertainty whether the assistant fully understood the task and making it difficult to return to the assistant.

• Too fast too soon: Being pushed into the cart too early made users hesitant to trust the suggested products or the assistant’s understanding of their needs.

Focus on this: Design the AI assistant as part of the journey, not a shortcut to checkout. Users should feel guided, not pushed, with clear transitions between exploration and purchase. When the journey feels intentional and continuous, trust in both the assistant and its recommendations follows.

5. It moves us from keywords to conversation

Despite the challenges, the potential here is huge. We are seeing a shift from "orientation" based on keywords (e.g., searching for "white sneakers") to "conversational context" (e.g., "I am looking for white sneakers in leather that I can wear to the office").

When the AI worked as intended in our tests, it outperformed traditional browsing on speed and perceived relevance.

• The help to choose: Personal, "best-for-you" recommendations helped users reach decisions faster.
• The conversation: Users expressed a desire for scenarios like telling a site, "I'll visit a gala next week, please suggest the perfect top to match my colours and my dress".

Focus on this: Focus on "expert advice" use cases. The biggest value add isn't finding a product by name, but translating a user's problem into a product solution.

6. Users are forgiving... for now

Perhaps the most surprising finding was the users' willingness to try again. Despite bugs and navigational friction, almost all participants said they would try the service again in the future.

Users understand that we are in a transition period. They express a clear need for on-site AI shopping assistants and are willing to use them more as the technology matures. They want this to succeed because they want to avoid the hassle of traditional scrolling and filtering.

To optimize your AI-assistant strategy, consider these central findings

Discoverability: Users still rely on traditional search; you have to nudge them.

Trust: One wrong answer decreases the trust and can send them running to ChatGPT.

Complexity: AI-Assistants included in the test works best for simple "best for me" queries but struggles with complex, multi-step project planning.

Having an AI-Assistant that is trustworthy, transparent, and integrated into the user journey can boost your digital experience.

Would you like to know how your users actually perceive your digital experience? Reach out to us for a usability audit or conversion review today!