Using Optimized A/B Tests to Improve Online Software Services

Zoe Tseng, CISE Staff Writer

Have you ever wondered what exactly is happening when a website asks to track your cookies? CISE Faculty Affiliate Jinglong Zhao (Questrom) is working to help web-facing firms such as Google, Amazon, Netflix, and Facebook use cookies to improve their software services. Zhao works at the interface between optimization and econometrics. He designs field experiments for online platforms that aid in their product development process.

In a recent paper titled “Pigeonhole Design: Balancing Sequential Experiments from an Online Matching Perspective,” Zhao and his coauthor propose a new method called pigeonhole design for helping web-facing firms target their audience. The design addresses the challenges of heterogeneity in experiments. Heterogeneity is applying the same treatment to different covariates (user demographics in an online platform). The problem arises when the same treatment produces different effects across covariates.

The gold standard for addressing heterogeneity is a method called blocking. Blocking characterizes users with similar covariates into groups and then randomly assigns users from each block into the control and treatment groups.

The above diagram shows the gold standard in handling heterogeneity with the three covariates being hats, glasses, and pants. [cited from: Christopher Harshaw. “Balancing covariates in randomized experiments with the Gram–Schmidt Walk design”. Presented at the 2021 INFORMS Annual Meeting, Anaheim, CA, 2021.]
However, this method cannot always be used for online field experiments since the covariates are not always known until the user arrives at the website. Zhao likened the problem of heterogeneity to comparing apples to oranges. For example, an “apple” could be a 20-year-old male who comes to visit the Walmart website. He may view the “buy one get one free” coupon more favorably compared to the 50% off coupon, whereas the “orange” might be a 60-year-old female who views the 50% coupon more favorably.

“Imagine yourself running a large-scale randomized control trial. The heart of experimental design is to make an apples to apples comparison. You want to compare two units that are considered similar or the same,” Zhao said.

Given this, websites must learn how to target users by showing them the coupon they favor. This is where the pigeonholes come into play. The pigeonhole design divides the covariate space into smaller spaces called pigeonholes. It matches users who arrive at a pigeonhole into pairs so there is a roughly even amount of users in each group. One pigeonhole could be the 20~25-year-old men who visit the site at around 8pm, another pigeonhole could be the 50~60-year-old women who visit the website at around 1pm. Each pigeonhole might show different preferences for the coupon. Based on the data, the web-facing firm can better target their coupons to different customers.

Balancing three covariates sequentially: the challenge of this work.

During the experiment, Zhao and his team compared the results of the pigeonhole design to the matched-pair design and completely randomized design – both of which can be used as benchmarks. The matched-pair design would happen in a perfect world where the web-facing firm could see the next user before they arrive, enabling the user to be placed in a certain pigeonhole and given the treatment or control. The treatment group might be giving a user a 50% off coupon, and the control group might be giving the user a “buy 1 get one free” coupon.

CISE faculty affiliate Jinglong Zhao studies the interface between optimization and econometrics.

In contrast to the matched-pair design, a web-facing firm using a completely randomized design essentially ignores the covariance. 50% of users are randomly given the control and the other 50% are given the treatment.

Ideally, the web-facing firm would want performance as close to the matched-pair design. However, they cannot foresee the users that arrive at the site. Web-facing firms can only see the covariate data when the user arrives. Once the user arrives, they are placed into a pigeonhole that has covariates most similar to theirs. The treatment and control are given based on the previous user in that pigeonhole. So if a 20-year-old male arrives and is given the treatment, then the next person that fits into the same pigeonhole would be given the control instead.

The results of Zhao’s experiment show that the pigeonhole design closely matched the performance of matched-pair designs which is unachievable for online designs. It also outperforms the completely randomized design, enabling companies to target their audience with smaller sample sizes.

Zhao said the courses he took in his undergraduate years and while obtaining his Ph.D. influenced his interest in optimizing experimental design problems and led him to this type of work.

“I bring the approach of looking at experimental design questions through an optimization perspective,” Zhao said. “People don’t typically do this traditionally, which is why this area is so interesting.”

Zhao, Jinglong, and Zijie Zhou. “Stratifying Online Field Experiments Using the Pigeonhole Design.” arXiv preprint arXiv:2201.12936 (2022).

View all posts