Tests, Experiments, and Various Effects

Reading through some software engineering textbooks, I’ve felt vindicated in my hunch that SE has a distinctly “scientific” feel about it. By this I mean the empiricism, adherence to the scientific method, etc., all feels quite different from other subdisciplines of CS.

For about a year now, Kaleigh have been chatting more and more about how testing in software engineering feels so much like experimental design. As I’ve delved more into the SE literature, it’s become clear that that is no accident – the SE textbook that UMass uses uses “test” and “experiment” almost interchangably in the validation chapter, and I’ve seen similar language and approaches in test suite generation to experimental designs such as Latin squares.

To narrow things down a bit, I’ve been thinking more about how to take e.g. an individual acceptance test in Toybox and think about it in terms of experimentation. In order to do that, we need to reason about counterfactuals and even candidate underlying causal models. Below is a collection of perhaps overly-detailed musings meant to help focus my dissertation writing that are deeply intertwined with conversations we have been having around XAI…

[Read More]


What I've been doing all year(s)

Well, this blog post is only a few weeks late. Fortunately my excitement hasn’t waned: I’m psyched to write about a project I’ve effectively been working on since May of this year.

I’m working with John, Kaleigh, and David on a new testing framework for deep reinforcement learning. We presented a version of the work at NeurIPS this year, and are submitting it to a software engineering conference soon. It’s called ~~ Toybox ~~, it’s written in Rust, and there’s more cool information about it on my Projects page.

[Read More]


A Basis for a Mathematical Theory of Computation

It perhaps should not surprise me that there are not histories of PL the way there are histories of other sciences: computer science is such a new discipline and most overviews of how things came to be currently live in blog posts. I think it would be fun to have a history class that treats earlier academic papers as primary sources and strives to understand the cultural contexts of intellectual trends. Maybe this is too niche, but a history of the academic paper would also be very cool, and might help us as a community address better address its shortcomings in light of changing technologies, cultural norms, etc.

Anyway, I’m trying to read classic papers related to the topics covered by my thesis. Since my research is interdisciplinary, I’ve been thinking about how to address some of the common questions and misconceptions that come up when folks trained in, say, machine learning, interact with folks trained in programming lanaguages. Today I’m posting on John McCarthy’s 1963 paper A Basis for a Mathematical Theory of Computation (the link is to the correction).

TL/DR Version: The intro to this paper is interesting for the perspective it gives on the field at a point in time, but the particular formalisms have evolved since then. Students of PL would be better served working through Software Foundations to learn and appreciate McCarthy’s approach. A later paper, such as Wright and Felleisen’s “A Syntactic Approach to Type Soundness” would probably be a more useful primary source.

[Read More]


Replication vs. Reproducibility

Note: I am posting this from home today, where we cancelled our internet service. I am pushing over my phone’s mobile data. Since I’m not sure how much data I’ll be using, and since I don’t want to wait and then forget to post this, I’m not putting links into this particular post. Apologies!

Reproducibility is something I first started to care about while working with my first PhD supervisor, Lee Spector. One of the things that really drew me to PL was the growing community interest in reproducibility. However, it was only after looking at the ISSTA CfP and its discussion of reproducible studies that the distinction between replication and reproduction really started to gnaw at me.

[Read More]


Could Facebook Block Spam Phone Calls?

During one of my between-tasks pomodoro breaks, I read an article on Gizmodo about Facebook’s “People You May Know” feature. The obvious high-level perspective I have on this is the impending culture clash between science and policy: many in the tech industry have some kind of scientific training, which prizes testing hypotheses that may be wrong. Being wrong is not an inherently bad thing, because it leads to an improved understanding of the world. Of course, it’s always great when you’re right the first time, but learning from mistakes is part of scientific discourse. This ethos clashes with politicians and policy makers, who pay dearly for being wrong. The past elections and the rise of the 314Action PAC have had me thinking about the differences in perspectives between politicians who typically have legal training versus those with scientific training.

In any case, that’s not the discussion I want to be having about the Gizmodo article. I am more interested in whether so-called “shadow profiles” can be used to cut down on spam phone calls.

[Read More]