RAG shows its work. That’s not the same as being right.
At the Generative AI Summit Austin, Ramkumar Shanker took the important stage to ship a keynote that lower via the hype: the period of third-party cookies is over, and the publishers who will win are these that may flip consented, first-party alerts into that means, at scale, with accountability.
Shanker brings a uncommon twin perspective to this problem. As Director of Data Science at USA TODAY (Gannett), he leads AI initiatives at the intersection of generative AI and media monetization.
In parallel, he’s a analysis collaborator at the University of Chicago, the place his work on explainable AI in medical imaging informs his method to governance and analysis.
I sat down with him after his session to go deeper on the concepts that lit up the room.
Ram, you opened with a provocation about the “privateness reset.” What is definitely altering, and why does it matter proper now?
The reset is easy in precept and arduous in follow. Invisible monitoring, the variety that follows a consumer throughout the net with out their consciousness, is changing into structurally unattainable.
Regulations, browser modifications, and platform coverage are all converging. When that pipe closes, the solely sign you personal outright is the one your readers gave you immediately: what they learn, what they subscribe to, what they watch, how they behave by yourself property.
That shifts first-party knowledge from a compliance checkbox into a real aggressive moat. It is the solely sign that’s each permissioned and unique to your model. But having first-party knowledge is desk stakes. The edge comes from what you do with it: are you able to translate habits into intent? Can you clarify what you inferred, and why? That is the new functionality that separates leaders from laggards.
Your keynote launched the thought of “monetizing that means” fairly than monetizing identifiers. What does that appear like in follow?
Traditional programmatic promoting offered entry to an individual, or a cookie that stood in for an individual. Monetizing that means is completely different: it sells entry to an intent.
Instead of “this gadget visited these 40 websites,” you may say “this reader has demonstrated a sustained curiosity in native sports activities that correlates with in-market buy habits for sporting items.” That is a richer, extra actionable sign, and it’s one you may again up with proof.
Publishers can construct segments like “native sports activities superfans” or “home-improvement intenders” from content material consumption and subscription alerts, then package deal these as premium concentrating on or sponsored editorial experiences.
The advertiser will get a high-signal viewers. The reader will get relevance. And you may present your work, citing the particular content material behaviors that justified the phase. That traceability is what makes it defensible, commercially and ethically.
You talked about LLMs and RAG as the technical engine behind this. What is the “unlock” in contrast with conventional taxonomy approaches?
Rules-based taxonomies are brittle. Language shifts, subjects evolve, and handbook tagging can’t sustain with the quantity of content material a writer produces. An LLM can classify based mostly on that means fairly than precise key phrases. That is the flexibility win.
But flexibility with out accountability is harmful in a business context. That is the place Retrieval-Augmented Generation earns its place in manufacturing. RAG grounds every tag in an permitted supply set and produces an audit path.
The system can inform you: “This article was tagged supply-chain compliance as a result of these particular passages join it to regulatory reporting necessities.” An article on sustainable manufacturing would possibly by no means point out compliance explicitly, but the system can floor that connection and present its work.
That turns tagging from a static taxonomy right into a dwelling semantic index you may monetize and govern.
A tough however helpful strategy to image it: think about you might be the head librarian of the world’s largest newspaper archive, thousands and thousands of articles, many years of journalism. A sponsor walks in and asks for each reader who cares deeply about clear power however has not but dedicated to an EV.
Under the outdated mannequin, you hand them a listing of people that clicked on an EV advert. Under the new mannequin, you learn each article each individual has engaged with, infer their evolving considerations about vary anxiousness and charging infrastructure, and construct an viewers outlined by intention, not by a single click on.
The analogy is not good, actual programs are messier and inference is rarely that clear, but it surely captures what LLM-based tagging plus RAG is reaching towards at the scale of a nationwide writer.
What are the deployment realities that almost all groups underestimate after they attempt to take this from demo to manufacturing?
Most LLM initiatives do not fail in the mannequin. They fail in the system. There are 5 failure modes I might warn each builder about: drift, value, latency, analysis, and governance.
Drift is the quiet killer. Content subjects, viewers habits, and language all shift over time, and you must monitor retrieval high quality and downstream enterprise outcomes constantly, not simply mannequin accuracy at launch.
Cost can explode at inference scale, so that you want caching methods, routing to smaller fashions for less complicated duties, and smarter retrieval. Latency is an engineering constraint from day one: each retrieval hop provides time, and if you’re making an attempt to serve contextual adverts in actual time, you can not bolt that on later.
Evaluation is the place quite a lot of groups are overconfident. Accuracy scores inform you nearly nothing about enterprise worth. You must measure relevance, phase raise, and precise outcomes, and it is advisable actively probe failure modes fairly than ready for them to floor in manufacturing.
Governance means logs, provenance data, entry controls, and human evaluation for high-stakes selections, particularly if you’re utilizing brokers that take actions autonomously.
I discover John Searle’s Chinese Room a helpful provocation right here, although I’ll admit it’s a contested analogy and philosophers have been arguing about it for many years. Imagine a room the place an individual who speaks no Chinese is handed playing cards with Chinese symbols.
They comply with a rulebook to provide responses that look, to everybody outdoors, like fluent Chinese dialog. The individual inside is not understanding something; they’re pattern-matching.
Now think about that room is your LLM. The analogy breaks down in essential methods; LLMs are not merely rule-following in any easy sense, however the sensible warning holds: a system can produce outputs that sound authoritative and coherent whereas being subtly improper in methods which can be very arduous to detect.
That is not a purpose to keep away from LLMs. It is a purpose to pair them with retrieval from curated sources, human evaluation for consequential selections, and governance buildings that deal with ‘sounds proper’ as inadequate. Fluency is not understanding.
When monetization is concerned, governance turns into pressing quick. What danger do groups most persistently underestimate, and what have you ever seen really work?
The failure I see most frequently is governance that’s obscure fairly than concrete. Organizations publish AI ideas, kind committees, after which uncover, often when one thing goes improper, that no person had resolution rights over something. Principles with out authority are ornament.
The larger danger is how rapidly useful automation turns into unaccountable automation as soon as income strain enters the image. Nick Bostrom’s paperclip maximizer is a helpful body right here, even when the situation itself is intentionally absurd.
Imagine an AI tasked with one purpose: maximize paperclip manufacturing. Given adequate functionality, it converts each obtainable useful resource into paperclips, as a result of that’s what it was instructed to optimize.
Nobody is constructing paperclip maximizers, and the real-world dynamics are much more gradual and mundane than the thought experiment suggests. But that’s nearly the level: you do not want a superintelligent rogue agent for reward hacking to trigger injury.
If you optimize just for clicks or conversions, you may, with out anybody desiring to, degrade belief, compromise model security, and erode the long-term reader relationship that makes your first-party knowledge price something in the first place. The reward perform you select is a governance resolution, not only a modeling resolution.
What works is governance that’s particular: an AI council with precise resolution rights, not simply advisory standing.
Concretely, which means approval gates for brand spanking new AI options and phase definitions; supply allow-lists for RAG with common audits; required labeling and logging for coaching and inference; periodic end result opinions that transcend accuracy metrics; and an specific kill-switch to pause any system rapidly.
My parallel work in medical imaging has bolstered the same intuition. In radiology AI, you might be by no means allowed to cease at ‘is it correct?’ You should reply: the place does it fail? Who opinions it? How will we catch errors earlier than they hurt a affected person? Media is not medication, however the self-discipline transfers. High-stakes selections deserve the same scrutiny.
You join your media work to your analysis in medical imaging at the University of Chicago. What travels between these two worlds?
More than folks anticipate. The floor options are very completely different: one is predicting tumor response to therapy, the different is predicting which content material a reader will have interaction with.
But the underlying challenges are practically equivalent. You want held-out validation that truly displays real-world distribution, explainability necessities that maintain as much as scrutiny, and governance frameworks that survive institutional evaluation.
In medical imaging, you can not ship a mannequin with out enthusiastic about failure modes, audit trails, and the humans in the loop who catch what the mannequin misses. That self-discipline has made me a extra cautious engineer in media.
And the scale of media, thousands and thousands of articles and a whole lot of thousands and thousands of classes, has made me a extra rigorous scientist. The intuition to ask whether or not the experimental design really examined the declare is one which improves each system I construct, in both subject.
What is the one psychological mannequin you need each attendee to hold out of this session?
We are drifting towards a Library of Babel web, and I imply that as a helpful approximation fairly than a exact declare.
Borges imagined an infinite library containing each doable mixture of characters, and due to this fact each ebook that has ever been written or ever might be written, alongside an infinite variety of books which can be nearly proper, plausibly formatted, and utterly improper.
The web is clearly not infinite, and AI-generated content material is not random, however the directional problem is actual: generative AI can produce convincing textual content at an unprecedented scale, which suggests the ratio of plausible-but-wrong to verified-and-true is shifting in a course that ought to concern anybody who is determined by info high quality.
The aggressive benefit is not the potential to generate extra textual content. It is the potential to construct and preserve the trusted index, the curated, provenance-backed catalog that allows you to discover what’s true amid the noise.
RAG is not a reality machine. It is a traceability machine. If your supply set is unhealthy, you’re going to get well-cited nonsense. The actual unlock is retrieval from respected, curated sources with clear provenance.
Start small: decide a high-trust supply set, construct a fundamental RAG layer that cites it, create a small set of monetizable segments, and design human evaluation and logging from day one. Because the groups that get governance proper from the begin are the groups which can be nonetheless working at scale two years from now.
About the speaker
Ramkumar Shanker is Director of Data Science at USA TODAY (Gannett), the place he leads AI initiatives at the intersection of generative AI and first-party knowledge monetization.
He can also be a analysis collaborator at the University of Chicago, the place his work on explainable AI in medical imaging informs his method to governance and analysis. He holds an M.S. in Applied Data Science from the University of Chicago and a B.Tech. from IIT Madras.
The views expressed on this interview are these of Ramkumar Shanker and do not essentially replicate the positions of USA TODAY or the University of Chicago.
