Shopify A/B Testing: Can Better Conversion Rates Hurt Accessibility?
An A/B test can improve conversion and still make a website worse. The numbers may say Variant B won, but if the new experience becomes harder to navigate with a keyboard, understand with a screen reader, or read with low vision, the experiment created a problem the conversion report never measured.
Conversion rate optimization is built around change. Teams test new buttons, popups, product cards, sticky bars, checkout messaging, navigation patterns, and landing-page layouts because small improvements can produce meaningful revenue.
Accessibility, meanwhile, depends heavily on consistency and predictable interactions. When those two disciplines are managed separately, a CRO experiment can quietly reintroduce accessibility problems into a storefront that previously worked well.
A winning test can still create a losing experience
Imagine a store testing a sticky “Add to Cart” bar on mobile. Variant B increases add-to-cart rate, so the team declares it the winner and rolls it out to everyone.
From a conversion perspective, the decision makes sense. But the new bar overlaps content when the page is zoomed, its button does not receive visible keyboard focus, and a screen reader announces the price update without enough context.
The test produced a measurable improvement for one group of users while introducing friction for another.
Traditional experimentation may never reveal that problem because the dashboard is designed to answer questions such as “Did more users click?” or “Did revenue increase?” It usually does not answer whether everyone could operate the new interface successfully.
That is why accessibility should be part of experiment QA, not something checked only during a major redesign.
Popups are a common place for problems to appear
Popups are popular CRO tools because their results are easy to measure. A discount modal can increase email signups, an exit-intent offer can recover visitors, and an upsell drawer can increase average order value.
The accessibility requirements are less visible.
When a modal opens, keyboard focus needs to behave logically. The user should be able to reach its controls, understand what appeared, close it, and return to the page without getting lost. W3C guidance specifically addresses focus order and modal behavior because unexpected focus movement can make dynamic interfaces confusing or unusable for keyboard and assistive-technology users.
A popup does not become a bad CRO idea simply because accessibility needs attention. It just means the experiment has more success criteria than conversion rate alone.
Small visual experiments can create real barriers
Not every accessibility regression comes from complex JavaScript. Some begin with a simple visual test.
A team might test a lighter button because it feels more premium, reduce text size to fit more information above the fold, move labels inside form fields to create a cleaner layout, or replace a text CTA with an icon.
Each decision may improve the visual design or even increase clicks, while also creating issues with contrast, form labels, or accessible button names.
These are not rare problems. WebAIM's 2026 analysis of one million home pages found detectable WCAG failures on 95.9% of them. Low contrast appeared on 83.9%, missing form labels on 51%, and empty buttons on 30.6%. Just six categories accounted for 96% of all automatically detected errors.
That is useful context for CRO teams because many experiments touch exactly those elements: colors, forms, links, buttons, and images.
Dynamic experiments are especially easy to miss
A/B testing tools often change a page after it has loaded. They may replace copy, inject a new component, reorder content, show a banner, or trigger a different experience based on user behavior.
Visually, everything may appear correct. Underneath, however, the order in which content is presented to assistive technology may no longer match what users see.
W3C notes that dynamically inserted content should preserve a logical reading and keyboard order. When new content changes the interface, where it appears in the DOM and how focus behaves can directly affect whether users can understand and operate the page.
This is why testing only with a mouse is not enough for interactive experiments.
Accessibility belongs inside the CRO workflow
The solution is not to make experiments slower or stop testing ambitious ideas. Accessibility checks can be surprisingly lightweight when they are part of the existing process.
Before launching a test, use the keyboard to move through the changed experience. Check whether focus is visible and follows a logical order. If the experiment introduces a popup or drawer, make sure users can enter, operate, and exit it properly. Check new colors for contrast, verify that buttons have meaningful names, and test changed forms with their labels and error messages.
Automated tools can help catch obvious problems, but they should not be the only test. WebAIM itself emphasizes that automated testing detects only a subset of accessibility failures and that a page with no detected errors is not automatically accessible.
For larger experiments, accessibility can simply become another line in the QA process alongside mobile testing, browser testing, analytics verification, and performance checks.
The best CRO result works for more people
Conversion optimization is ultimately about reducing friction. Accessibility is also about reducing friction, just from a perspective that many analytics dashboards do not automatically show.
That makes the two disciplines more closely related than they may first appear. A clearer form, more understandable navigation, predictable focus behavior, readable contrast, and controls that work without a mouse can improve the experience without working against conversion goals.
At Positron, we think CRO experiments should be judged by more than whether a metric moved in the right direction. The strongest test is one that improves performance without creating a new usability problem somewhere else.
A higher conversion rate is useful. A higher conversion rate on an experience more people can actually use is better.
