← Back to Documentation

🚀Getting Started with Abe CRO

Welcome to Abe CRO

This comprehensive guide will walk you through setting up your first A/B test from installation to analyzing results. Abe CRO is designed to be honestly simple—no complex setup, no confusing configurations, just straightforward testing that delivers real results.

Part 1: Installation and Setup

Step 1: Install from the Shopify App Store

Getting started is quick and easy:

  1. Navigate to the App Store: Visit the Shopify App Store listing for Abe CRO or search for "Abe CRO" in your Shopify admin's App Store
  2. Click "Add app": You'll see a button to add the app to your store
  3. Review Permissions: Shopify will show you what permissions Abe CRO needs. Here's what each permission does:
    • read_themes - Allows Abe to read your theme files to identify available templates
    • read_content - Enables access to your pages and content for preview generation
    • write_script_tags - Installs the embed block in your theme (required for test assignment)
    • write_pixels - Sets up the web pixel extension for tracking checkout events
    • read_orders - Tracks revenue and conversion data from completed orders
    • read_products - Accesses product information for preview thumbnails
  4. Confirm Installation: Click "Install app" to proceed
  5. Wait for Redirect: After installation completes, you'll be automatically redirected to the Abe CRO dashboard

Step 2: Understanding Automatic Setup

Once installed, Abe CRO automatically configures everything you need:

  • Embed Block Installation: The embed block is automatically added to your theme. This is a small JavaScript snippet that handles visitor assignment and template switching. You'll find it in your theme's code under "App embeds" or "Theme app extensions"
  • Web Pixel Setup: The web pixel extension is installed to track checkout events. This runs on your checkout pages and tracks when visitors start checkout and complete orders
  • Cookie Configuration: Tracking cookies are automatically configured. These cookies store visitor IDs and test assignments
  • Trial Activation: Your 14-day free trial begins immediately. You can use all features during the trial period

Step 3: Verify Installation

It's good practice to verify everything is working correctly:

  1. Check Theme Code:
    • Go to Online Store → Themes → Actions → Edit code
    • Look for "App embeds" or check your theme.liquid file
    • You should see "Abe CRO Tracker" or similar in your app embeds section
  2. Visit Your Storefront:
    • Open your store in a new browser tab (or use incognito mode)
    • Open browser developer tools (F12 or right-click → Inspect)
    • Go to the Console tab
    • You should see initialization messages from Abe CRO (these may be hidden in production mode)
  3. Check Cookies:
    • In developer tools, go to the Application tab (Chrome) or Storage tab (Firefox)
    • Navigate to Cookies → Your store domain
    • Look for a cookie named _cro_visitor_id
    • This cookie confirms tracking is working

Step 4: Explore the Dashboard

Take a moment to familiarize yourself with the Abe CRO interface:

  • Tests Page: This is your main dashboard where you'll see all your tests. If this is your first time, you'll see an empty state with a "Get Started" button
  • Roadmap: A visual timeline view of all your tests, both active and planned
  • Settings: Configure storefront password for preview generation, clear cache, and view documentation
  • Billing: View your current plan and manage your subscription

Part 2: Understanding Test Types

Template Tests (Recommended for Beginners)

Template tests allow you to test specific page templates with different template files. This is the most common type of test and is perfect for optimizing individual page types.

What You Can Test:

  • Product pages (e.g., product.default.json vs product.quick-view.json)
  • Collection pages (e.g., collection.default.json vs collection.grid.json)
  • Custom pages (e.g., page.about.json vs page.about-alternate.json)
  • Homepage (e.g., index.json vs index.alternate.json)
  • Cart page, search results, blog pages, and more

When to Use Template Tests:

  • You want to optimize specific page types without changing your entire theme
  • You're testing different layouts or designs for individual templates
  • You want to make targeted improvements to specific pages
  • You're on the Starter plan (template tests are available on all plans)

Theme Tests (Professional & Unlimited Plus Only)

Theme tests allow you to test entire themes against each other. This is perfect for major redesigns or comparing completely different theme options.

What You Can Test:

  • Your current live theme vs a new theme you're considering
  • Two different published themes to see which performs better
  • Major design overhauls across your entire storefront

When to Use Theme Tests:

  • You're considering a major theme redesign
  • You want to compare two different theme options
  • You're testing significant layout or design changes across the entire store
  • You want comprehensive, store-wide performance data

Part 3: Creating Your First Template Test (Step-by-Step)

Let's walk through creating a template test from start to finish. We'll use a product page test as an example, but the process is the same for any template type.

Step 1: Navigate to Create Test

  1. In your Abe CRO dashboard, click "Tests" in the left navigation
  2. If you have no tests yet, you'll see an empty state with a "Get Started" button—click it
  3. If you already have tests, click the "Create Test" button in the top right
  4. You'll be taken to the test creation form

Step 2: Choose Test Type

  1. At the top of the form, you'll see two options: "Template Test" and "Theme Test"
  2. For your first test, select "Template Test" (it's selected by default)
  3. Note: If you're on the Starter plan, "Theme Test" will be disabled with a message to upgrade

Step 3: Name Your Test

  1. In the "Test Name" field, enter a descriptive name
  2. Good examples:
    • "Product Page - New Layout Test"
    • "Collection Grid vs List View"
    • "Homepage Hero Section A/B - January 2025"
  3. Avoid vague names like: "Test 1", "New Test", "Product Test"
  4. Tip: Include what you're testing and when, so you can easily identify it later

Step 4: Select Template Type

  1. In the "Template Test Elements" section, you'll see a dropdown labeled "Template Type"
  2. Click the dropdown and select which type of template you want to test
  3. Available options:
    • Product: For testing product page templates
    • Collection: For testing collection listing pages
    • Page: For testing custom pages (About, Contact, etc.)
    • Index: For testing homepage templates
    • Cart: For testing cart page templates
    • Search: For testing search results pages
    • Blog: For testing blog listing pages
    • Article: For testing individual blog post templates
  4. For this example, select "Product"

Step 5: Choose Control Template

  1. After selecting a template type, the "Control Template" dropdown will populate
  2. The control is your current/default version—what visitors see now
  3. For most templates, "Default" will be the first option (representing your current template)
  4. Select "Default" to use your current product page template as the control
  5. Note: "Default" means no template suffix—your standard product page

Step 6: Choose Variant Template

  1. The "Variant Template" dropdown shows all available templates for the selected type
  2. This is the alternative version you want to test
  3. Select the template you want to test (e.g., "quick-view", "minimal", "detailed")
  4. Important: The variant cannot be the same as the control—the form will prevent this
  5. If you don't see the template you want, you may need to create it in your theme first

Step 7: Review Visual Previews

Once both control and variant are selected, you'll see:

  • Thumbnail Images: Small previews of how each version looks (if available)
  • Preview Links: Click "Preview Link →" to open each version in a new tab
  • Use these to verify you're testing the right templates before creating the test

Step 8: Configure Traffic Split

  1. Find the "Traffic Split" section
  2. You'll see two percentage fields: one for Control, one for Variant
  3. Recommended: 50/50 Split
    • Provides balanced sample sizes
    • Allows for faster statistical significance
    • Minimizes risk if variant performs poorly
  4. Alternative: 80/20 Split
    • Use when testing a significant change and want to minimize risk
    • 80% see control (current version), 20% see variant
    • Takes longer to get results but reduces potential negative impact
  5. The percentages must add up to 100%—the form validates this automatically

Step 9: Set Schedule (Optional)

You can schedule tests to start and end automatically, or activate them manually:

  • No Schedule (Manual): Leave both date fields empty. You'll activate the test manually when ready
  • Start Date Only: Set when the test should automatically start. You'll end it manually
  • Both Dates: Set start and end dates for fully automated testing
  • Timezone: Dates use your store's timezone (configured in Shopify settings)

Step 10: Configure Targeting (Optional)

Targeting allows you to filter which visitors are included in your test:

  • Device Type:
    • Leave unchecked to include all devices (recommended)
    • Check "Mobile", "Desktop", or "Tablet" to test only that device type
    • Useful when mobile and desktop experiences differ significantly
  • Login Status:
    • Leave unchecked to include all visitors (recommended)
    • Check "Logged In" to test only logged-in customers
    • Check "Logged Out" to test only visitors who aren't logged in
    • Useful when logged-in and logged-out experiences differ
  • Important: More targeting means smaller sample sizes. Ensure you have enough traffic for reliable results

Step 11: Add More Test Elements (Optional - Professional/Unlimited Plus)

If you're on Professional or Unlimited Plus, you can test multiple template types in one test:

  1. Click the "+ Add Another Test Element" button
  2. Select a different template type (you can't use the same type twice)
  3. Choose control and variant templates for this element
  4. Repeat as needed (up to your plan's limits)
  5. Note: Starter plan users can only add one template type per test

Step 12: Review and Create

  1. Review all your settings one more time
  2. Check that control and variant templates are correct
  3. Verify traffic split adds up to 100%
  4. Ensure test name is descriptive
  5. Click the "Create Test" button at the bottom of the form
  6. You'll be redirected to the test details page

Part 4: Activating Your Test

Step 1: Review Test Configuration

After creating your test, you'll be taken to the test details page. Here's what to check:

  1. Test Information: Verify test name, type, and status (should show "Draft")
  2. Test Elements: Confirm the correct templates are selected for control and variant
  3. Traffic Split: Verify the percentages are correct
  4. Schedule: Check start/end dates if you set them
  5. Targeting: Review any targeting filters you applied

Step 2: Preview Your Test

Before activating, preview both versions to ensure they work correctly:

  1. On the test details page, you'll see thumbnail previews of control and variant
  2. Click the "Preview Link →" under each thumbnail to open that version in a new tab
  3. Test the functionality:
    • Navigate through the page
    • Click links and buttons
    • Add items to cart (if applicable)
    • Check mobile view (resize browser or use device emulation)
  4. If anything looks wrong, you can still edit the test (if it's in Draft status)

Step 3: Activate the Test

Once you're confident everything is correct:

  1. On the test details page, look for the action buttons near the status badge
  2. Click "Activate" (or "Start Test" depending on your test status)
  3. You may see a confirmation dialog—confirm to proceed
  4. The test status will change from "Draft" to "Active"
  5. Your test is now live and collecting data!

Step 4: Verify Test is Running

After activation, verify the test is working:

  1. Visit Your Storefront: Open your store in an incognito/private browser window
  2. Check Assignment:
    • Open browser developer tools (F12)
    • Go to Application/Storage → Cookies
    • Look for a cookie named _cro_assignment_[your-test-id]
    • The value should be either "default" or "variant"
  3. Verify Template:
    • If assigned to variant, check the URL—it should have ?view=[templateSuffix] parameter
    • The page should render using the variant template
    • Refresh the page—you should stay in the same group
  4. Check Metrics: Return to the test details page and verify metrics are starting to appear (may take a few minutes)

Part 5: Understanding What Happens Behind the Scenes

How Visitor Assignment Works

When a visitor arrives at your store:

  1. First Visit:
    • Abe CRO checks if the visitor has an assignment cookie for this test
    • If no cookie exists, a random number is generated (0-100)
    • Based on your traffic split, the visitor is assigned to "control" or "variant"
    • Example: With 50/50 split, random number < 50 = control, ≄ 50 = variant
    • A cookie is set: _cro_assignment_[testPlanId] with the assignment value
    • This cookie lasts 365 days
  2. Returning Visitors:
    • The assignment cookie is read
    • Visitor is assigned to the same group as their first visit
    • This ensures consistent experience across sessions
  3. Multiple Tests:
    • If a visitor is in multiple active tests, they have separate assignment cookies for each
    • Each test assigns independently
    • A visitor could be in control for one test and variant for another

How Template Rendering Works

Once assigned, visitors see the appropriate version:

  • Template Tests:
    • The embed block detects which test applies to the current page
    • If visitor is assigned to "variant", it adds ?view=[templateSuffix] to the URL
    • Shopify's theme system uses this parameter to render the variant template
    • Example: /products/my-product?view=quick-view renders product.quick-view.json
    • This happens seamlessly—no page reload or flickering
  • Theme Tests:
    • For theme tests, visitors are redirected to ?preview_theme_id=[themeId]
    • Shopify's preview system renders the assigned theme
    • This applies to all pages across the entire storefront
    • Visitors see a consistent theme experience throughout their session

How Event Tracking Works

Abe CRO tracks events at different stages of the customer journey:

  • Page Views:
    • Tracked automatically when visitors view pages included in your test
    • Uses unique visitor IDs to avoid counting the same person multiple times
    • Only counts views of pages that match your test's template types
  • Add to Cart:
    • Tracked via the embed block when visitors click "Add to Cart" buttons
    • Only tracked if the visitor is on a page included in an active test
    • Event includes which test and variant the visitor is in
  • Checkout Started:
    • Tracked via the web pixel when visitors begin checkout
    • Works on all checkout pages (including one-page checkout)
    • Links back to the visitor's test assignments
  • Orders & Revenue:
    • Tracked via the web pixel when checkout completes
    • Includes order total as revenue
    • Attributed to all active tests the visitor is enrolled in

How Data is Collected and Stored

All tracking data flows through a secure process:

  1. Event Occurs: Visitor performs an action (page view, add to cart, etc.)
  2. Client-Side Tracking: Embed block or web pixel captures the event
  3. API Call: Event data is sent to Abe CRO's API endpoint
  4. Server Processing: Server validates the event and determines which tests it applies to
  5. Database Storage: Event is stored with test ID, variant, visitor ID, and timestamp
  6. Aggregation: Events are aggregated daily for performance
  7. Dashboard Display: Aggregated metrics appear in your dashboard

Part 6: Monitoring Your Test

Understanding the Test Details Page

The test details page is your command center for monitoring results:

  • Test Information Section:
    • Test name, type, and current status
    • Traffic split percentages
    • Schedule information (if set)
    • Targeting filters (if applied)
  • Test Elements Section:
    • Visual cards showing control and variant templates
    • Thumbnail previews of each version
    • Preview links to view each version
    • Collapsible section to save space
  • Metrics Section:
    • Default metrics: Visitors, Add to Carts, Orders, Revenue
    • Default ratio metrics: $ / Visitor, Product Views / Visitor, ATC %, CVR %
    • Toggle to show/hide additional metrics
    • Charts showing trends over time
  • Actions:
    • Activate, Pause, End Test links
    • Edit button (if test is editable)
    • Export to PDF or Excel

What to Monitor Daily

While your test is running, check these daily (but don't make decisions too early):

  1. Visitor Count:
    • Ensure visitors are being assigned to both control and variant
    • Check that the split is roughly matching your configured percentages
    • If one group has significantly fewer visitors, there may be an issue
  2. Revenue Per Visitor:
    • This is your most important metric
    • Compare control vs variant
    • Look for trends, not day-to-day fluctuations
  3. Conversion Rate:
    • Shows overall effectiveness
    • Higher conversion rate = more visitors completing purchases
    • Compare to Revenue Per Visitor for full picture
  4. Sample Size:
    • Ensure you're collecting enough data
    • Typically need 1,000+ visitors per group for reliable results
    • More visitors = more confidence in results

When to End a Test

Don't end tests too early. Here's when it's safe to make a decision:

  • Minimum Duration: 1-2 weeks (for high-traffic stores)
  • Recommended Duration: 2-4 weeks for most stores
  • Consider:
    • Full business cycles (weekly patterns, monthly patterns)
    • Seasonal variations
    • Marketing campaign periods
  • Signs You Have Enough Data:
    • 1,000+ visitors in each group (control and variant)
    • Clear trend in the metrics (not just random fluctuations)
    • Statistical significance indicators (if available)
    • Consistent results over multiple days

Part 7: Analyzing Results and Next Steps

Step 1: Review Overall Performance

When your test has run long enough, analyze the results:

  1. Compare Revenue Per Visitor:
    • This is the most important metric
    • If variant has higher $/Visitor, it's likely the winner
    • Calculate the difference and potential revenue impact
  2. Check Conversion Rate:
    • Higher conversion rate is good, but revenue matters more
    • A variant with lower conversion but higher AOV might still win on revenue
  3. Review Other Metrics:
    • Add to Cart Rate: Shows engagement
    • Average Order Value: Shows transaction value
    • Checkout Completion: Shows checkout effectiveness

Step 2: Make a Decision

Based on your analysis, decide:

  • Variant Wins:
    • If variant shows better revenue performance, implement it
    • Make the variant template your new default
    • Update your theme accordingly
    • Consider testing further improvements
  • Control Wins:
    • Keep your current template
    • Learn from what didn't work
    • Try a different variant in a new test
  • No Clear Winner:
    • Results are too close to call
    • Either version is acceptable
    • Consider testing a different change
    • Or run the test longer for more data

Step 3: End the Test

  1. On the test details page, click "End Test"
  2. Confirm the action
  3. The test status will change to "Ended"
  4. Data collection stops, but results remain available

Step 4: Export and Document

Save your results for future reference:

  1. Click "Export" on the test details page
  2. Choose PDF or Excel format
  3. Save the file with a descriptive name
  4. Share with your team or stakeholders
  5. Keep records for future test planning

Part 8: Best Practices for Your First Test

Start Simple

For your first test, keep it straightforward:

  • Test a single template type (e.g., just product pages)
  • Test one significant change at a time
  • Use a 50/50 traffic split for balanced results
  • Don't add targeting filters initially—get baseline data first
  • Choose a change that could actually impact conversions

Test Duration Guidelines

How long to run your test depends on traffic:

  • High-Traffic Stores (10,000+ visitors/day):
    • Minimum: 1 week
    • Recommended: 2 weeks
    • You'll get results faster due to high sample sizes
  • Medium-Traffic Stores (1,000-10,000 visitors/day):
    • Minimum: 2 weeks
    • Recommended: 3-4 weeks
    • Need more time to accumulate sufficient data
  • Low-Traffic Stores (<1,000 visitors/day):
    • Minimum: 3-4 weeks
    • Recommended: 4-6 weeks
    • May need to run longer to reach statistical significance

What to Monitor

Focus on metrics that matter to your business:

  • Revenue Per Visitor ($ / Visitor): The most important metric—shows actual business impact
  • Conversion Rate (CVR %): Overall effectiveness of your variant
  • Sample Size: Ensure you have enough visitors (typically 1,000+ per group for reliable results)
  • Statistical Significance: Look for metrics marked as significant (when available)
  • Trends: Look for consistent patterns, not day-to-day fluctuations

Avoid These Common Mistakes

  • Ending Too Early: Don't make decisions based on a few days of data
  • Making Changes Mid-Test: Don't modify templates while a test is active
  • Ignoring Revenue: Don't focus on vanity metrics—revenue is what matters
  • Overlapping Tests: Don't run multiple tests on the same templates simultaneously
  • Not Documenting: Export and save your results for future reference

Part 9: Next Steps After Your First Test

If Your Test Was Successful

  1. Implement the winning variant as your new default
  2. Update your theme to make the variant template the standard
  3. Consider testing further improvements
  4. Build on what worked—test related changes

If Your Test Didn't Show Improvement

  1. Don't be discouraged—every test teaches you something
  2. Analyze what didn't work and why
  3. Try a different approach or change
  4. Consider testing a different template type
  5. Use insights to inform your next test

Building a Testing Program

Successful CRO is an ongoing process:

  • Plan your next test while the current one runs
  • Use the roadmap view to coordinate multiple tests
  • Test systematically—one change at a time
  • Document everything—what you tested, why, and results
  • Share learnings with your team

Need Help?

If you run into any issues during setup or testing: