Architecting Enterprise AI for Generative and Agentic Systems – with Ranjan Sinha of IBM
Maintaining legacy IT platforms has become a massive financial and operational drag. Research from Pegasystems Inc., in partnership with research firm Savanta, found that the average global enterprise wastes more than USD $370 million each year on technical debt.
The primary reason for this technical debt cited in the study is that legacy transformation projects are slow, resource‑intensive, and repeatedly fail, resulting in annual losses of around $56 million USD to maintain and integrate outdated systems.
In critical public‑sector infrastructure, a U.S. Department of Transportation review of AI for Intelligent Transportation Systems finds that legacy platforms — limited by restricted computational power, constrained data storage, and weak system documentation — create integration and compatibility challenges that raise infrastructure costs and prevent agencies from unlocking the full potential of AI-enabled safety and operational improvements.
Health care shows the same pattern. A 2011 editorial in Applied Clinical Informatics reports that health IT projects “fail at a rate up to 70% of the time,” when failure is defined to include cost overruns, delays, unmet objectives, or project abandonment.
The author links these outcomes to organizational complexity, resource constraints, and weak governance structures — arguing that trust, safety, and regulatory alignment must be addressed before digital and AI-driven systems can move from pilot programs into routine clinical use.
In a recent episode of Emerj’s ‘Vision to Value in Enterprise AI’ video podcast, Emerj Editorial Director Matthew DeMello sat down with Ranjan Sinha, IBM Fellow, Vice President and Chief Technology Officer for watsonx at IBM Research AI, to discuss the fundamental shift in how enterprises need to approach AI infrastructure.
Their conversation highlights two critical insights for enterprise adoption of agentic AI:
- Making AI-ready infrastructure a priority: As quantum computing becomes more commonplace, leaders will need to redesign AI infrastructure to treat AI as a mission-critical system, not to scale experiments on legacy foundations.
- Full-stack foundation for agentic AI: Investing in a governed, full-stack architecture to scale agentic AI beyond experimentation.
Listen to the full episode below:
Guest: Ranjan Sinha, IBM Fellow, Vice President and Chief Technology Officer for watsonx, IBM Research AI, IBM
Expertise: Enterprise AI and Data, Data and AI Platforms, AI Strategy, Responsible AI, Software Development
Brief Recognition: Ranjan works at the intersection of technology, research, product, and enterprise-scale AI use cases. He is a seasoned technical executive with experience building large-scale platforms from concept to production. He holds a Ph.D. in computer science from RMIT University and has received federal and university research grants, including during his time at the University of Melbourne.
Making AI-Ready Infrastructure a Priority
Ranjan opens the conversation by highlighting that the AI landscape is at a pivotal moment, pushing organizations to rethink the very foundation of their technology infrastructure. “It is no longer just the back end,” he explains. “Infrastructure is becoming the foundation for how we build and scale the next phase of AI.”
He notes that, until recently, most companies were running small-scale AI experiments, testing chatbots, recommendation engines, or automating simple enterprise tasks. But as organizations aim to expand these experiments across their core business operations, they are running into a critical roadblock: their existing technology stacks simply cannot support the demands of large-scale AI.
“So now, companies that want to scale these successes across their intact business are discovering that their current technology infrastructure simply cannot handle it.
It’s like trying to run a Formula One race on neighborhood streets: The foundation just wasn’t built for this level of performance, scale, and security requirements. So we are witnessing an inflection point where AI has moved beyond proof of concept demonstrations to mission-critical enterprise applications.”
-Ranjan Sinha, IBM Fellow, Vice President and Chief Technology Officer for watsonx, IBM Research AI at IBM
According to him, the shift from AI as a proof-of-concept to AI as a mission-critical business function marks a fundamental change. When AI models now directly influence business analytics, customer experiences, financial transactions, or operational decisions, infrastructure considerations go beyond computational power.
Reliability, security, governance, and trust across the entire stack have become central concerns. “The move from experimentation to AI as a business backbone fundamentally changes the infrastructure requirements,” Ranjan emphasizes.
He explains that enterprises are confronting unprecedented infrastructure challenges as they transition from AI experimentation to production-scale deployment. He says, AI workloads differ fundamentally from traditional computing: they are massively parallel, data-intensive, and increasingly demand real-time performance.
“This is why we’ve seen the shift from general-purpose CPUs to specialized accelerators like GPUs, NPUs, and dedicated inference engines,” he explains, noting that these accelerators are optimized for the matrix operations at the heart of AI, delivering dramatically higher throughput and energy efficiency.
Heterogeneous computing architectures, he said, are becoming essential for supporting diverse enterprise workloads, from training large language models to running AI inference at the edge.
In a recent research benchmark, Sinha pointed to IBM’s North Pole neuromorphic chip as an example of how alternative hardware architectures can improve energy efficiency and latency for AI workloads. He emphasized that these results were demonstrated in controlled research settings using a 3 billion-parameter model, highlighting the potential — rather than production-ready performance — of emerging computing paradigms.
On the quantum front, he explains that cloud-accessible quantum computers have enabled broader experimentation with quantum algorithms. Hardware has been scaled from five qubits to over 100, and roadmaps exist for building fault-tolerant quantum systems capable of handling larger computational problems.
Hybrid classical-quantum architectures, he notes, allow workloads to be split across CPUs, GPUs, and quantum processors, with classical systems handling data preprocessing, interfaces, and result interpretation.
For Ranjan, AI hardware advancements are not merely about speed — they are about enabling entirely new classes of problems to be solved efficiently, sustainably, and at scale.
Full-Stack Foundation for Agentic AI
Ranjan explains that building enterprise-ready agentic AI systems that can perceive, plan, reason, and act autonomously requires a purpose-built, full-stack infrastructure. Traditional AI setups aren’t sufficient for these advanced systems, so enterprises need an integrated approach that covers everything from data to models to agents.
He outlines a purpose-built, full-stack infrastructure in the following layers:
- AI Agents and Assistants – Interfaces through which users interact with AI systems, whether via text, voice, images, or dashboards. These agents not only automate work and deliver insights but also continuously improve through human-in-the-loop feedback, aligning their behavior with human preferences and values.
- Observability and Governance – Tools for monitoring, telemetry, and performance tracking, which enable enterprises to observe, govern, and tune AI systems as adoption scales.
- AI/ML Operations (MLOps) – Processes for building, managing, and maintaining AI systems while integrating them into workflows.
- Data Layer – The foundation of generative AI, requiring accuracy, governance, and scalability for both structured and unstructured data.
- Hybrid Cloud and AI Tools – The scalable base that supports all layers and enables safe enterprise deployment.
According to Ranjan, such a layered, full-stack approach enables enterprises to move safely and efficiently from experimentation to large-scale, agent-driven AI applications, supporting the next phase of generative and agentic AI.
In the end, he emphasizes that the future of AI will be driven by open strategies rather than closed, proprietary systems.
“So the internet succeeded because it was built on open standards that everyone could use and improve. And we are seeing the same thing happen with AI, through foundation models that anyone can customize for their specific needs, rather than building AI from scratch.
In fact, there are over a million models available through platforms like Hugging Face. So this implies that a small startup can access the same sophisticated AI capabilities as a Fortune 500 company.”
–Ranjan Sinha, IBM Fellow, Vice President and Chief Technology Officer for watsonx, IBM Research AI at IBM
He highlights that OpenAI does not have to compromise security or governance. Enterprises can safely adopt and scale AI through platforms that provide robust controls, compliance, and observability, enabling organizations to benefit from innovation while maintaining reliability and enterprise-grade standards.