Qualify the reviewer

One line out of the PDA/FDA Joint Regulatory Conference has stuck with me, which is an FDA speaker's caution against using AI for anything you aren't qualified to check yourself (it's summarized in PharmTech, from conference reporting). It sounds like common sense, and it is, but I think it cuts against a lot of how the industry is talking about AI right now.

In my last note I argued that the number worth tracking is how many workflows you've redesigned so they can't run without AI, not how many agents you've deployed. I still believe that, but this is the honest counterweight to it. The more a workflow depends on AI, the more it depends on the person reviewing what the AI produced, and if that person couldn't have done the work themselves, the review is a signature and not a check.

I understand why this gets skipped. The pitch for a lot of AI tools is that they let a less experienced person do work that used to need a specialist, and for better or worse that's exactly the scenario the FDA is warning about. A junior person approving an AI-drafted protocol section or a data review they don't fully understand doesn't remove the expertise from the process, it just hides the fact that it's missing.

So when you're redesigning a workflow around AI, I'd design the review step first and ask who is actually qualified to catch it when the output is wrong, and whether that person has the time to do it properly. If the answer is nobody, you haven't redesigned the workflow, you've just moved the risk somewhere harder to see.

We need people making judgment calls, and the AI only makes that more important.