Creative A/B testing effectively increases the total conversions of an app. It works by publishing two versions of an app’s product detail page (PDP), and comparing the conversions (purchases) between them. The Growth > Creative A/B Testing tab in the Developer Dashboard enables developers of Meta Horizon Store apps to run A/B tests on various PDP assets.
Get the most out of your experiment by reading our Best Practices and FAQ before creating your test.
Note: If you’re looking for information on price A/B testing, see Price A/B Testing.
Prerequisites and setup
Developers with apps published in the Store, including Coming Soon and Pre-Order listings, can access the A/B tool on the left-hand side of the Developer Dashboard after selecting their app. You also need to have your test assets ready to upload and submit prior to starting the experiment.
You can test five PDP variables with the tool. These variables include: video trailer, description (short and long), cover art (square and landscape), screenshots, and 360° previews.
Cover art
Cover art (square): 1:1 static image that’s displayed across platforms to market your app to users in the Store.
Cover art (landscape): 16:9 static image that’s displayed across platforms to market your app to users in the Store.
Video trailer:
The video that displays at the top of your app’s PDP.
The thumbnail image that previews your video trailer.
Description
Short description: Description that is shown throughout the platform, primarily in VR.
Long description: Description that is shown throughout the platform, including the Store.
Screenshots
Each image should represent a unique scene highlighting the best part of the experience.
Note: Reordering existing screenshots does not require review and are automatically approved for testing.
360° previews
360° previews: A cubestrip or equirectangular graphic used as a skybox for your PDP.
All creative A/B tests must be created and submitted for review from the creative A/B testing tool.
In the left-side navigation, click Growth > Creative A/B Testing. The Creative A/B Testing tool is where you can create new and view current and past A/B tests, including the status, dates and outcomes for each test.
Click Create New.
Create your test by filling out the following information.
Name - The name of your creative A/B test. Please note that this is the name for the test itself for your reference, not the individual assets.
Audience split - The percentage of the audience that can see your test asset. You can choose from 10%, 25% or 50%.
Start date - Your desired start date for the test. You can also choose to start your test immediately following your submission’s approval.
Test type - Select the PDP variable(s) you are testing from the five provided options: Cover Art, Video Trailer, Descriptions, Screenshots, and 360° Previews.
Confidence Level - The statistical measure of the percentage of test results that can be expected to be within a specified range. For example, a Confidence Level of 80% means that the experiment result is guaranteed 80% of the time.
You can choose to automatically publish the test winner. This will trigger if your experiment reaches 80% confidence in statistical significance after 20 days. If you don’t select this option, you can manually publish your test results (see Monitor and review results).
Upload your test asset(s) in the provided boxes below your selection. To encourage statistically significant results, we recommend testing multiple asset types in the same experiment.
Note: When testing multiple asset types in the same experiment, the results show how the test assets as a whole performed against the control, so you will not be able to discern how individual assets performed.
Click Submit For Review.
Submit changes as necessary
After submission, your test is reviewed for approval. We send an email guiding you to your results once your experiment has reached statistical significance at the 80% confidence level. We send a second email after your test has ended. Results can be viewed by visiting the Results page for your test of the creative A/B testing tool. Even after statistical significance has been reached, results may fluctuate if your test is not completed or canceled. All tests are completed after 30 days.
To submit any requested changes, follow these steps:
Navigate to the Metadata Virtual Reality Checks (VRCs) section of the creative A/B testing tool to confirm that your app meets all metadata VRCs. If a red icon is listed next to a VRC, click on the item to learn more and resolve the issue.
After the requested changes have been made, select Save to resubmit your test.
Once approved, your creative A/B test begins on the chosen Start date, or immediately after approval, depending on your previous selection.
Monitor and review results
You can manually check your ongoing results throughout the test by visiting the creative A/B testing tool in the Developer Dashboard. If you need to cancel the test at any time, you can do so from the dashboard by clicking the overflow menu on the right side of each creative A/B test and selecting Cancel. Only one creative A/B test may run at a time per app and you can’t publish other app metadata changes while a test is active.
Once statistical significance has been reached, or your test has ended, you’ll get an email notification to view your results. All tests are completed after 30 days.
To review your creative A/B test results, follow these steps:
Click on the email link or visit the Results page for your test by clicking on your test in the A/B Testing dashboard. Even after statistical significance has been reached, results may fluctuate if your test is not completed or canceled.
If you’re satisfied with your test assets, you can publish those to your app’s PDP by selecting Publish Test Variant.
If your test was not statistically significant, but your test assets produced positive results, you may also publish it. If your A/B test is not completed and you would like to publish your test assets, you must first cancel your test.
