On the one hand, PlanOut is straightforward and should be easy to reason about: the language does not permit user-defined functions and none of the operators use loops or recursion. Experiments are designed to provide apples-to-apples comparisons between two or more equivalent treatment groups, yet PlanOut programs are able to assign individuals to treatment groups using any number of users characteristics or complex logic. To analyze such experiments, one therefore needs to be able to determine what kinds of contrasts between the variables which define a treatment (the experimental contrasts), and the propensity to receive a treatment (the propensity scores).

What makes reasoning about PlanOut scripts tricky is that free variables may sit around unevaluated: we may have some expression such that there is no way to simplify it down to a base value without further information from the system. This post describes how the analyzer interprets expressions in the presence of unknown values.

When previously discussing online experiments, we talked about randomization errors. In that model, assignment to treatment or control is done by the system that selects user ids. We can think about these assignments as a series of tables. If we want to repeat an experiment, we have re-generate the tables, so we don’t end up experimenting on the same exact group every time. It is much more efficient and compact to have a procedure for generating assignments. A procedure also allows us to document the process we used for assignment, and even implies the kind of analyses we intend to do. It can be used for provenance, verification, and program synthesis.

In the CheckCell work, Emery often says “garbage in, garbage out.” However, when chatting with Dan about this idea, we realized that it was really a specific instance of a more general concept, which I will call “data-driven correctness.”

When I was chatting with Emery about the effect of the unit of randomization) on inference of causal estimators, he suggested I look into the information flow literature, since randomization parallels secure information. He initially suggested I look at Jif, but while perusing Stephen Chong’s publications, I came across a different paper that appears to have some important features and properties that we can build on.

This appears to be an older paper (published in 2006). What follows is part summary, part running commentary/reactions.

Wherein we discuss field experiments, controlled experiments, and online experiments. Includes a discussion of infrastructure for online experiments.