I was recently reviewing Church encodings, for no particular reason other than to refresh my memory. It’s been a while since I first learned about them and I was pleased that they seemed so much easier to understand this time around! The exercise I was reviewing got me thinking about an idiom I’d learned and used often in Lisp dialects.

What are the priniciples that make an experimentation system sound? We’ll discuss in a future blog post how to make the experimentation-analysis pipeline correct by construction, and what correctness means in this context. First, however, we need to to look at the set of basic assumptions that allow us to reason about the interactions between concurrently running experiments, variables or parameters not represented in a particular experimental procedure, and those parameters of interest. The underlying experimentation system must reflect these assumptions; if it does not, then all of our analyses may be invalid.

Like many (all?) static analysis tools, our PlanOut analyzer does not have to be complete, but it should always be sound. Therefore, we must ensure that we never return estimators that are incorrect.

Conditional random assignment presents a complication on this front – in order to reason about conditionally assigned variants, we need to determine what values of variables in a particular guard lead us down a particular path. To solve this problem in the general case, we need an SMT solver.

We would like to add assertions, annotations, and maybe even types to the PlanOut language. However, since it’s infeasible to rewrite our corpus of scripts in this updated language, we have a temporary solution: we are parsing in YAML config files to provide supplementary type information for inference.

Back in October, Brendan mentioned that some folks at Harvard were working on a new book for causal inference that unifies concepts from both the graphical models literature and the counterfactual reasoning literature.

So far, I’ve found the drafts they’ve posted online very accessible. In fact, I strted reading it after writing a few blog posts on PlanOut and what I’ve been learning from the social science causal inference literature, and this book has put things in terms similar to how I explain the concepts to myself.

One particularly cool insight is this very evocative image in their first chapter, used to describe the difference between causation and correlation: