Overview
Shopify's Winter '26 Edition introduced Rollouts, a free, native A/B testing and progressive deployment tool built directly into the admin. If you have been wondering what Rollouts can and cannot do, how to run your first split test, or whether you still need a paid tool like Shoplift, Intelligems, or Convert, this FAQ has the answers. It is written for merchants who have never tested before and for those evaluating whether to keep their existing CRO stack.
Contents
Understanding Rollouts
What is Shopify Rollouts?
Rollouts is a centralized system for managing, scheduling, and testing changes to your online store before publishing them. It lets you prepare seasonal updates, sales, or campaigns in advance and schedule them to launch at a specific time, while also testing and comparing different sets of changes to understand what drives better results with your visitors. It is a native split testing and controlled deployment feature built directly into the Shopify admin, with no third-party app, no injected scripts, and no additional monthly fee.
Within Rollouts, you can control the percentage of traffic you want the rollout to apply to and can simultaneously launch more than one set of features. This in effect turns a rollout into an A/B test. Rollouts runs server-side on Shopify's own systems, meaning the split happens before the page renders, not after. That means zero page speed penalty and no script conflicts with your other apps.
What Shopify plans support Rollouts?
Rollouts is available on the Basic plan or higher. You must have the new version of Shopify Markets activated on the store. However, the A/B testing portion (adjusting the traffic percentage slider) has a plan requirement. If your store is on the Advanced plan or higher, you can adjust the Treatment percentage to split visitors between your changes and your live theme to run the rollout as an experiment.
Basic and standard Shopify plan merchants can still use Rollouts for scheduled deployments and drafting changes on their live theme. The full feature set favours Shopify Plus. Keep an eye on Shopify's official documentation as access continues to expand during early access.
What can Rollouts actually test?
Rollouts works inside the theme editor, so you can test any visual or structural change that the editor supports. This could be anything the theme editor supports: rearranging sections, changing images, updating text, modifying colours, adjusting layout settings. Common first tests include hero section variants, product grid layouts, navigation styles, and CTA placement.
Any changes you make in the theme editor apply only to that rollout; areas you don't customise automatically stay synchronised with your live theme. This means you can test specific sections (like a new hero banner or product page layout) without rebuilding your entire theme.
Setting Up Your First Test
How do I set up my first Rollout?
From your Shopify admin, go to Markets > Rollouts. Click Create rollout. In the Name section, add a name for the new rollout. Click Select launch date and time. Choose a launch date in the date picker, and then click Done. Optionally, adjust the percentage of traffic for the rollout; the default is 100%.
After you create and save the rollout, you can click Add changes on the rollout page to customize your published theme. You must save the rollout before you can create a set of changes. From there, the theme editor opens with your rollout context active. Any customizations you make in the theme editor apply to that rollout only. Areas of your theme that aren't customized in a rollout automatically stay synchronized with customizations made to your published theme.
Tip: Give it a clear, descriptive name. "Test 1" helps nobody. Something like "Homepage V2, Video Hero, March 2026" tells your team exactly what is being tested and when.
What traffic percentage should I start with?
Starting at 10% to 25% is the smart move for any test. You can increase the percentage later as you gain confidence. This graduated approach lets you catch problems before your full audience is affected.
Start at 10% for the first 48 hours. If your conversion rate and revenue hold steady, scale to 25%. Then 50%. This graduated approach catches problems before they reach your full audience. For stores with lower daily traffic (under 1,000 sessions per day), run tests for a minimum of 7 days to account for weekly traffic patterns, and for 14 days to achieve statistical significance.
How long should I run a test?
At minimum, let your test run a full seven days so you capture weekday and weekend shopping patterns. Don't call a winner after 100 visitors. Use Shopify's built-in confidence intervals and wait for clear results. If you have fewer than 1,000 daily visitors, plan for a two-week minimum to accumulate enough data. Rollouts tracks conversion rate, average order value, and revenue against your live store automatically, so check these metrics regularly to decide when you have a confident result.
Limitations and Third-Party Tools
What can't Rollouts do yet?
Shopify has been transparent about what Rollouts can't do yet: Theme Settings Changes (can't test different color schemes or typography scales), App Embeds (app blocks aren't rollout-aware; globally injected code affects all traffic), Liquid Template Edits (direct .liquid file changes require publishing a new theme version), Audience Segmentation (no targeting by device type or new vs. returning visitors), Custom Goals (limited to Shopify's standard metrics), and Product/Checkout Testing (can't test pricing display or checkout optimizations yet).
Rollouts does not currently show confidence intervals or statistical significance indicators in its analytics. That is one of its biggest limitations for stores that want to run disciplined experiments. Until Shopify adds this, you will need to export your data and run the math yourself, or use a free significance calculator. In the coming months, Rollouts will extend to more surfaces in Shopify so you can coordinate rollouts across products, discounts, and more.
How does Rollouts compare to Shoplift, Intelligems, and Convert?
Rollouts wins on cost (free vs. $50 to $500+/month), page speed (native vs. script injection), and setup time (minutes vs. hours). Third-party tools like Shoplift and Intelligems win on audience targeting, custom event tracking, full storefront testing, and Liquid code changes.
Here is a quick breakdown. Shoplift offers full-featured A/B testing with Bayesian analysis, device segmentation, and AI-powered variant generation. It is best for brands running a comprehensive testing programme. Intelligems specialises in price testing and profit optimisation alongside traditional CRO tests, and is best for brands that want to test pricing strategies, which Rollouts cannot currently do. Convert Experiences offers enterprise-grade testing with advanced segmentation, custom goals, and RPV tracking, and is best for sophisticated CRO teams that need granular targeting.
Should I cancel my testing app now?
Rollouts does not replace a mature A/B testing program. But it does replace the basic layer most stores were paying for. A store installed a testing app two years ago, uses it to split test homepage hero images, and never touches the advanced features. They are paying $200 to $400 per month for something Shopify now handles natively. If your current app is doing nothing beyond splitting theme traffic, you are likely overpaying.
If your testing app handles pricing experiments, checkout optimization, audience segmentation, or custom analytics, keep it. Rollouts is not a replacement for those capabilities. Audit your testing stack quarterly and keep only the tools where Rollouts cannot cover the use case.
SimGym and Advanced Strategy
What is SimGym and how does it work with Rollouts?
Alongside Rollouts, Shopify released SimGym, an AI tool that creates simulated shoppers to browse your store. These aren't bots running random clicks. Shopify built personas from aggregated data across billions of transactions on their platform. The idea is to catch problems before you run a live Rollout. SimGym flags friction in navigation, layout, and checkout flow so you can fix it before real customers see it.
The recommended workflow is a two-stage validation pipeline. Run a SimGym simulation first to get synthetic early signal, then set up a Rollout at a conservative traffic percentage (10 to 25%) to test with real customers. The first stage uses AI to catch the obvious problems before real visitors are exposed. The second stage uses real traffic to confirm the direction with statistical confidence. SimGym is still in research preview with a waitlist/approval flow.
What are good first tests to run with Rollouts?
If you have never A/B tested before, start with high-impact, above-the-fold changes on your most-visited pages. Good first tests include: Homepage hero (lifestyle image vs product image, to see which drives more clicks to collection pages), product page reviews placement (above the fold vs below, to see if social proof earlier increases add-to-cart rate), collection page grid layout (3-column vs 4-column), navigation style (mega menu vs simple dropdown), and CTA button copy ("Add to Cart" vs "Buy Now" vs "Add to Bag").
Tip: Don't go 50/50 on day one. Begin with 10% traffic, a 24 to 48 hour run time, and one clear hypothesis. Once you are comfortable reading results and acting on them, you can run bigger tests and layer in third-party tools if your programme outgrows what Rollouts offers.
View on app store