Project

Counterfactual image editing

Deep learning and causal inference to predict the effect of treatments on chest X-rays

• Causal AI and graphical models
• Healthcare
• Ongoing

A chest X-ray is mapped to a representation structured by a causal graph of clinical variables (such as fluid intake, diuretics and oedema). An intervention on the treatment variables produces a counterfactual image for the same patient.

Challenge

In intensive care, fluid therapy is adjusted on the basis of clinical and radiological signs, such as the oedema visible on chest X-rays. Clinicians would benefit from tools that show how the image of an individual patient would look under a different treatment. Standard deep learning models capture statistical associations and cannot answer this type of counterfactual question.

Approach

We combine deep generative models of medical images with a causal graph that encodes clinical knowledge about patient characteristics, treatments and outcomes. The encoder infers patient-specific latent factors from the observed image. An intervention on a treatment variable, such as the volume of administered fluids, is propagated through the causal graph, and the decoder generates the counterfactual image of the same patient.

Expected impact

A decision-support tool that allows clinicians to visualise the predicted effect of treatment choices on the individual patient, and a general framework for counterfactual reasoning on complex data such as images.