A Talk at TTIC on Experiment Design

Jessica Hullman of Northwestern University is visiting TTIC this coming Monday. She is giving a talk at 10am Central Time (11am Eastern) on “Hypothesizing About Effects in Experiment Design and Interpretation.”

Hullman is the first holder of one of two professorships in AI and Machine Learning created in a $5 million gift by IBM to Northwestern in honor of the retired IBM chairman Virginia Rometty, a Northwestern alumnus. Yes, the article says “chairman” but Rometty was the first female to lead of IBM as CEO and in other chairing capacities until her retirement at the end of 2020. Rometty is also vice chair (no “-man”) of Northwestern’s Board of Trustees.

Hullman’s webpage says that her research “addresses challenges and limitations that arise when people draw inductive inferences from data.” She has contributed multiple visualization and interaction techniques for improving reasoning under uncertainty from data-driven interfaces, as well as theoretical frameworks for understanding the role of visualization in statistical workflow. All this has been recognized with best paper awards at some top visualization and HCI venues, a Microsoft Faculty award, and NSF CAREER, Medium, and Small awards as PI, among others. (Are our quotes “transitive,” per question 1 here?)

The Talk

Here is the abstract of her talk:

Learning from data, such as the results of controlled experiments, requires one to reason about the likelihood of many competing explanations. However, people are boundedly rational agents who often engage in pattern-finding at the expense of recognizing uncertainty or considering potential sources of heterogeneity and variation in the effects they seek to discover. I will discuss graphical and theoretical tools we have developed to support analysts in hypothesizing and interpreting experimental effects.

First, motivated by the heavy reliance on average treatment effects in data-driven science, I will present Causal Quartets, sets of plots that depict possible patterns of variation compatible with an average treatment effect at the level of individual units. Second, I will discuss how a rational agent framework can be used to better design and interpret results of controlled human decision-making experiments, such as are increasingly used to understand the impacts of data displays and AI assistance.

Level K vs. K-1

The talk follows on from a recent paper of hers with Dongping Zhang and Jason Hartline, also of Northwestern University. It is titled, “Designing Shared Information Displays for Agents of Varying Strategic Sophistication.”

One technical aspect that comes through in the paper that attracts Ken’s attention is the express use of game theory and Level-k reasoning. This is connected with the idea of how deeply one considers another party’s possible reactions to one’s moves, alternating times. Ken’s student Tamal Biswas incorporated some aspects of this into his thesis work on Ken’s chess project, as reflected in the discussion of “deep traps” in this post.

An example of this played out in the fourth quarter of last Sunday’s NFL playoff game between the Buffalo Bills and Kansas City Chiefs. When KC forgot to put an 11th player on the field as the Bills were lining up for a punt, Buffalo’s coach quickly thought to take advantage by faking the punt and trying a running play for a first down. They had been mindful of wanting not to punt because the Bills had been unable to stop KC’s offense—KC had scored touchdowns on their previous three possessions (excluding an end-of-half kneel-down).

Alas, Buffalo was on “level ” to KC’s “level .” KC apparently anticipated the fake punt and imy moved to stop a running play. They stopped the runner well short of where he needed to reach for the first down and took over the ball deep in the Bills side of the field. That was not the decisive play of the game, but it showed how there is a game of chess behind the physical moves of the players. OK, the football case was only KC on level L2 versus the Bills on level L1, but the main cases in the paper are also L2 versus L1.

Open Problems

The talk is freely available at the above link. We look forward to seeing the analysis of visual information, which we interact with all the time online.