This article covers the technical implementation of ABSynthesis for developers. For information about the algorithms, see the methods section


All Scientific Microservices endpoints use an API key in combination with an email address for authentication. See Getting Started for more information including error handling and rate limits.

Use Cases

Report the impact of your experimental program
Combine all the experiments you've run this quarter to demonstrate the impact of your experiment program
Compare marketing platforms
Run experiments on the same ad on different platforms to see which performs better
Coordinate citizen science
Combine experiments run by different citizen scientists across a city.

Overview

ABSynthesis provides a simple way to calculate the result of one or more AB experiments.

If you’re not familiar with the world of online science, it’s a powerful way of understanding and improving absolutely anything you can think of - you can see a great overview here.

But if you’ve done one experiment, the chances are you’ve done a lot of them. And when you look at all those results (especially if you pay for them yourself) you have probably thought about how to make sense of the whole collection of experiments you’ve run in the past.

Combining experiments is tricky, because some experiments are better than others. ABSynthesis tells you the overall uplift between two versions of something by correctly analysing the results of one or more experiments.

Endpoint URL


POST https://api.scientificmicroservices.com/absynthesis

Request Format

The request must be an HTTP POST request with a JSON body. The JSON data should be structured as an array of objects, where each object represents an experiment (representing a table of four columns with one row per experiment).

To understand how to set up your data, four definitions are required:

  1. base: The current or default version, e.g. the starting state of a website
  2. variant: The version that has been changed somehow, e.g. the same website, but with differently coloured buttons
  3. trials: The number of attempts in the experiment regardless of the outcome, e.g. website visitors
  4. successes: The number of times the desired action occurred e.g. a website visitor clicked a button

All these terms (or industry-specific equivalents) are used universally in the experimentation community. If you would like to deepen your understanding of these terms, see our use cases and methods sections.

In line with the above definitions, each object should include four named key-value pairs:

Element Description
trials_base The number of trials in the base version of the experiment.
successes_base The number of successes in the base version of the experiment.
trials_variant The number of trials in the variant (new) version of the experiment.
successes_variant The number of successes in the variant (new) version of the experiment.

Example request



curl -H "email:jegar@scientificmicroservices.com" \
-H "key:$scimicro_api_key" \
-H "Content-Type: application/json" \
--request POST \
--data '[{"successes_base":10,"trials_base":50,"successes_variant":15,"trials_variant":70},
{"successes_base":15,"trials_base":60,"successes_variant":25,"trials_variant":80},
{"successes_base":30,"trials_base":80,"successes_variant":65,"trials_variant":90},
{"successes_base":50,"trials_base":90,"successes_variant":90,"trials_variant":150},
{"successes_base":100,"trials_base":500,"successes_variant":200,"trials_variant":500}]' \
https://api.scientificmicroservices.com/absynthesis

  

  > {"uplift":[0.3779],"p_value":[0.0031]}




Use the ABSynthesis API endpoint at api.scientificmicroservices.com/absynthesis to tell me whether there is a difference between the versions of my project. Here is the json of my results: 

[{"successes_base":10,"trials_base":50,"successes_variant":15,"trials_variant":70},{"successes_base":15,"trials_base":60,"successes_variant":25,"trials_variant":80},{"successes_base":30,"trials_base":80,"successes_variant":65,"trials_variant":90},{"successes_base":50,"trials_base":90,"successes_variant":90,"trials_variant":150},{"successes_base":100,"trials_base":500,"successes_variant":200,"trials_variant":500}] 

My key is [YOUR KEY], and the email to use is [YOUR_EMAIL]




import json
import requests

headers = {
    'email': YOUR_EMAIL,
    'key': YOUR_KEY,
    'Content-Type': 'application/json'
}

url_absynthesis = "https://api.scientificmicroservices.com/absynthesis"

successes_base = [10, 15, 30, 50, 100]
trials_base = [50, 60, 80, 90, 500]
sucesses_variant = [15, 25, 65, 90, 200]
trials_variant = [70, 80, 90, 150, 500]

sample_data_ab = [
    {
        "successes_base": sb,
        "trials_base": tb,
        "sucesses_variant": sv,
        "trials_variant": tv
    }
    for sb, tb, sv, tv in zip(successes_base, trials_base, sucesses_variant, trials_variant)
]

response = requests.post(url_absynthesis, headers=headers, json=sample_data_ab)

uplift = response.json()
print(uplift)




library(jsonlite)
library(httr)

url <- "https://api.scientificmicroservices.com/absynthesis"

sample_data <- data.frame(
    "successes_base"= c(10 , 15 , 30 , 50 , 100),
    "trials_base" = c(50 , 60 , 80 , 90 , 500),
    "sucesses_variant" = c(15 , 25,  65,  90, 200),
    "trials_variant" = c(70 , 80 , 90 ,150, 500))

sample_json <- toJSON(sample_data)
response <- POST(
  url = url,
  add_headers(  'email'= YOUR_EMAIL,
                'key' = YOUR_KEY,
                'Content-Type' = 'application/json'
  ),
  body = sample_json,
  encode = "json"
)

uplift <- fromJSON(content(response, as = 'text'))
print(uplift)

Response Format

The endpoint responds with a JSON object containing 2 values called 'uplift' and 'p_value'.

Example response


     > {"uplift":[0.3779],"p_value":[0.0031]}
uplift The uplift value indicates how many more successes the variant would see compared to the base as a percentage (e.g. if a variant saw double the number of successes, uplift =1).
p_value How much the uplift calculation can be trusted. A small number is better.

We provide the p-value to give you a choice in how strong you need your result to be to call it 'true'. Generally people will trust any uplift that has an accompanying p value lower than 0.05.

If you are willing to be more ‘risky’ in your decision, you may want to set a more lenient threshold, such as 0.1 .

The impact of different p-value choices on your overall product journey is an unsettled and fascinating subject. Read our methods section to learn more.

Notes for Data Scientists

  • We focus on the case where multiple experiments are to be combined. If only one experiment is provided, we use Student's t-test formula for binomial experiments, which should match other online experimentation platforms.
  • The endpoint can be used progressively to track the performance of experimental programs as each experiment concludes. This enables program-level stopping rules that prevent your teams following dead end hypotheses.
  • The algorithm we use depends on the data representing a trials/successes (i.e. binomial) type of experiment.

Notes for Developers

  • Unlike other Scientific Microservices endpoints, ABSynthesis requires a specific structure of data with specific column names. Ensure you have all the right columns and their names are spelled correctly
  • Implement proper error logging and monitoring to catch and resolve any server-side issues.
  • Consider adding authentication and rate limiting to secure and manage the API.