A project from
A/B Tests
how brands measure what content
converts best with their app users.
Outcome
Test natively, instead of guessing.
INTRO
the Big Picture
decision makers
founders
brand managers
digital marketers
task executioners
shopify developers
marketing designers
focus areas
increase revenue & build loyalty
maintain identity & web parity
manage integrations
PRODUCT
Product in Focus
How App & Product Relate
An app is built & maintained at Appbrewin 2 distinct stages.
What's Relevant for this Project
A/B Test sits in the Design section of the Product, where users manage content for their app.
Design Section
Pages Panel
Blocks Panel
Edit Panel
What was my Role
My job was to understand the core user problem and design an optimal solution on the product.
Co-Founder (CTO)
Product Roadmap
Tech Architecture
Product Manager
Problem Scoping
Customer Research
Product Designer
Solution Scoping
Final Design
3x Engineers
Development
PROBLEM
Outlining the Problem
What Users want to Achieve
Brands want to test which content converts or sells better on key pages of the app shopping journey.
product discovery
what content leads to more product exploration
high-value products
what content leads to higher order amounts
creative & messaging
what content triggers buying intent over browsing
sales & events
what content sells more – pushing sales or brand building
cross-sell / upsell
what content better pads orders with related products
notifications
what content prompts higher interest to purchase
prominent blocks
what CTA or search layouts lead to more actions
review prompts
when review prompts lead to more & better feedback
loyalty benefits
what & where loyalty benefits lead to higher revenue
How it Worked before Design
Users were able to test some app areas, by using hacks outside the product and applying the result on the product later.
What Problems are in Focus
Problem
No native way to reliably test content variations.
Problem
Workarounds & 3rd party tools are slow, inaccurate & costly.
SOLUTION
Mapping the Solution
How Problems were Addressed
Let users test natively any content they create / edit directly on the product, without alternatives or workarounds.
Solution
Allow users to create A/B tests against content block & pages.
Solution
One key metric to define test winner, with other relevant metrics for context.
Test Scope
Test Content Scope
Test Metrics
Constraint
Can't deviate from app framework, like testing app layout variations.
Constraint
Time limited for essentials only, like 2 test variants & limited active test editing.
Constraint
Low bandwidth for advanced metrics, like audience funnels or AI analysis.
Constraint
Project launched under legacy design system (updated design system used here).
DESIGN
Crafting the Interface
Access to A/B Tests
The scope of testing led to 2 distinct, yet similar, access points for blocks & pages each.
Block Editing Panel
Page Editing Panel
Creating a Test (Approach)
Creating has high scope of friction due to many inputs required. I tackled this mainly by disclosing controls in layers & steps throughout the process.
ITR-Drafted
ITR-Shipped
Creating a Test (Steps)
The test details & parameters are the same for both page & block tests, except content editing for each.
Test Details
Block / Page Variants
Test Parameters
Test Schedule
Creating a Test (Variants)
Controls for editing a page or block variant differ based on the editing scope of each.
Creating Variants
Block Variant
Page Variant
Test Statuses
After creating, a test can go through various stages, as outlined on the same panel where tests are accessed.
ITR-1
ITR-2
ITR-Shipped
Tracking Performance
The simpler test panel opens the full modal with all info & controls, including editing some test details.
ITR-1
ITR-2
ITR-Shipped (Performance)
ITR-Shipped (Settings)
Test History
This is a permanent record of all tests created on any theme, page or block, tracking all past test decisions.
OUTCOME
the Aftermath
What Changed
Outcome
Brands report testing as a useful way to optimise revenue & conversion on key areas.
Outcome
Brands report usefulness in optimizing content during promotions.
Outcome
Higher interest from prospective enterprise customers.
What to Improve Next
Up Next
Expand scope to themes, product content, notifications & audience targeting.
Up Next
Expand parameters to creating more than 2 variants & allow editing active tests.
Up Next
Better analysis with audience funnels & AI performance analysis.
MORE
