|

Building a Multimodal RAG Pipeline with NVIDIA NeMo Retriever, Hosted NIMs, LanceDB, Reranking, and Grounded Generation

ByRicardo

In this tutorial, we construct a sophisticated multimodal retrieval-augmented era pipeline with NVIDIA NeMo Retriever. We start by configuring a Python 3.12 atmosphere, putting in the required packages, and performing offline PDF textual content extraction with out counting on a GPU or exterior API key. We then lengthen the workflow with hosted NVIDIA NIM endpoints to detect web page parts, extract tables, charts, and infographics, generate dense vector embeddings, and retailer the processed content material in LanceDB. Finally, we implement dense retrieval, vision-language reranking, metadata-filtered search, grounded response era with inline citations, and a light-weight recall-at-k analysis to validate retrieval high quality throughout multimodal doc content material.

Copy Code
import sys, os, subprocess, textwrap, json, time, warnings
warnings.filterwarnings("ignore")
assert sys.version_info[:2] == (3, 12), (
   f"nemo-retriever requires Python 3.12.x (discovered {sys.model.cut up()[0]}). "
   "Colab's default runtime is 3.12; in the event you modified it, change again."
)
def sh(cmd):
   print(f"$ {cmd}")
   subprocess.run(cmd, shell=True, examine=False)
strive:
   import nemo_retriever
   print("nemo-retriever already put in")
besides ImportError:
   sh("pip set up -q --ignore-installed PyJWT nemo-retriever openai")
import nemo_retriever
print("nemo-retriever model:", nemo_retriever.__version__)
from nemo_retriever import create_ingestor
strive:
   from nemo_retriever.io import to_markdown, to_markdown_by_page
besides ImportError:
   from nemo_retriever.frequent.io import to_markdown, to_markdown_by_page
strive:
   from nemo_retriever.retriever import Retriever
besides ImportError:
   from nemo_retriever.graph.retriever import Retriever
import pandas as pd
pd.set_option("show.max_colwidth", 160)
DOC = "multimodal_test.pdf"
if not os.path.exists(DOC):
   sh(f"curl -sL -o {DOC} "
      "https://uncooked.githubusercontent.com/NVIDIA/NeMo-Retriever/major/information/multimodal_test.pdf")
print("doc:", DOC, os.path.getsize(DOC), "bytes")
DOCS = [DOC]
print("n=== STAGE 1: offline textual content extraction (no API key) ===")
offline = (
   create_ingestor(run_mode="inprocess", allow_no_gpu=True)
   .recordsdata(DOCS)
   .extract(
       extract_text=True,
       extract_tables=False, extract_charts=False,
       extract_images=False, extract_infographics=False,
       use_page_elements=False,
       extract_page_as_image=False,
       technique="pdfium",
   )
)
df_offline = offline.ingest()
print("rows:", df_offline.form, "ncolumns:", checklist(df_offline.columns))
print("npage 1 textual content preview:n", df_offline.iloc[0]["text"][:400])

We configure the Python 3.12 atmosphere, set up NVIDIA NeMo Retriever, and import the required ingestion and retrieval parts. We obtain the pattern multimodal PDF and outline it because the enter doc for the pipeline. We then carry out CPU-based offline textual content extraction with PDFium and examine the extracted rows, columns, and web page content material.

Copy Code
from getpass import getpass
if not os.environ.get("NVIDIA_API_KEY"):
   strive:
       from google.colab import userdata
       os.environ["NVIDIA_API_KEY"] = userdata.get("NVIDIA_API_KEY")
   besides Exception:
       os.environ["NVIDIA_API_KEY"] = getpass("NVIDIA_API_KEY (nvapi-...): ").strip()
API_KEY = os.environ.get("NVIDIA_API_KEY", "").strip()
HAVE_KEY = API_KEY.startswith("nvapi-")
print("API key current:", HAVE_KEY)
PAGE_ELEMENTS_URL   = "https://ai.api.nvidia.com/v1/cv/nvidia/nemotron-page-elements-v3"
OCR_URL             = "https://ai.api.nvidia.com/v1/cv/nvidia/nemotron-ocr-v1"
TABLE_STRUCT_URL    = "https://ai.api.nvidia.com/v1/cv/nvidia/nemotron-table-structure-v1"
GRAPHIC_ELEM_URL    = "https://ai.api.nvidia.com/v1/cv/nvidia/nemotron-graphic-elements-v1"
EMBED_URL           = "https://combine.api.nvidia.com/v1/embeddings"
RERANK_URL          = "https://ai.api.nvidia.com/v1/retrieval/nvidia/llama-nemotron-rerank-vl-1b-v2/reranking"
CHAT_URL            = "https://combine.api.nvidia.com/v1"
EMBED_MODEL  = "nvidia/llama-nemotron-embed-1b-v2"
RERANK_MODEL = "nvidia/llama-nemotron-rerank-vl-1b-v2"
LLM_MODEL    = "nvidia/llama-3.3-nemotron-super-49b-v1.5"
LANCEDB_URI, TABLE = "./lancedb", "colab_demo"
df = df_offline
if HAVE_KEY:
   print("n=== STAGE 2: multimodal ingest by way of hosted NIMs ===")
   ing = (
       create_ingestor(
           run_mode="inprocess",
           allow_no_gpu=True,
           error_policy="gather",
       )
       .recordsdata(DOCS)
       .extract(
           extract_text=True,
           extract_tables=True,
           extract_charts=True,
           extract_infographics=True,
           extract_images=False,
           technique="pdfium",
           dpi=200,
           table_output_format="markdown",
           page_elements_invoke_url=PAGE_ELEMENTS_URL,
           ocr_invoke_url=OCR_URL,
           table_structure_invoke_url=TABLE_STRUCT_URL,
           graphic_elements_invoke_url=GRAPHIC_ELEM_URL,
           api_key=API_KEY,
           request_timeout_s=120.0,
           split_config={"textual content": {"max_tokens": 512, "overlap_tokens": 64}},
       )
       .dedup(content_hash=True, bbox_iou=True, iou_threshold=0.45)
       .embed(
           embedding_endpoint=EMBED_URL,
           model_name=EMBED_MODEL,
           embed_model_name=EMBED_MODEL,
           api_key=API_KEY,
           input_type="passage",
           inference_batch_size=16,
           nim_http_max_concurrent=8,
       )
       .vdb_upload(
           vdb_op="lancedb",
           vdb_kwargs={
               "uri": LANCEDB_URI,
               "table_name": TABLE,
               "overwrite": True,
               "create_index": True,
               "index_type": "IVF_HNSW_SQ",
               "metric": "l2",
           },
       )
   )
   t0 = time.time()
   df = ing.ingest(show_progress=True)
   print(f"ingested in {time.time()-t0:.1f}s -> {df.form}")

We securely load the NVIDIA API key and outline the hosted NIM endpoints for format detection, OCR, desk extraction, graphic evaluation, embedding, reranking, and era. We create a multimodal ingestion pipeline that extracts textual content, tables, charts, and infographics whereas making use of token-aware chunking and content material deduplication. We generate embeddings for the extracted content material and add the ensuing vectors and metadata to a LanceDB desk.

Copy Code
print("n=== Extraction inspection ===")
for col in ["tables", "charts", "infographics", "images"]:
   if col in df.columns:
       n = int(df[col].apply(lambda v: len(v) if isinstance(v, (checklist, tuple)) else 0).sum())
       print(f"  {col:<14} {n}")
pages = to_markdown_by_page(df)
print("npages rendered to markdown:", checklist(pages.keys()))
print("n--- web page 1 markdown (first 900 chars) ---n", pages[min(pages)][:900])
full_md = to_markdown(df)
if full_md:
   with open("extracted.md", "w") as f:
       f.write(full_md)
   print("nfull doc markdown -> extracted.md")
if HAVE_KEY:
   print("n=== STAGE 3: dense retrieval ===")
   retriever = Retriever(
       run_mode="service",
       top_k=5,
       rerank=False,
       vdb_kwargs={"uri": LANCEDB_URI, "table_name": TABLE},
       embed_kwargs={
           "embedding_endpoint": EMBED_URL,
           "model_name": EMBED_MODEL,
           "embed_model_name": EMBED_MODEL,
           "api_key": API_KEY,
           "input_type": "question",
       },
   )
   QUERIES = [
       "Given their activities, which animal is responsible for the typos in my documents?",
       "What is the most expensive gadget and how much does it cost?",
       "Which animal is at the beach?",
   ]
   def present(hits, label=""):
       print(f"n--- {label} ---")
       for i, h in enumerate(hits, 1):
           meta = h.get("metadata")
           if isinstance(meta, str):
               strive: meta = json.hundreds(meta)
               besides Exception: meta = {}
           web page = (meta or {}).get("page_number", "?")
           rating = h.get("_distance", h.get("rerank_score", ""))
           physique = " ".be part of(str(h.get("textual content", "")).cut up())[:180]
           print(f" {i}. p{web page} rating={rating}  {physique}")
   present(retriever.question(QUERIES[0]), "single question")
   for q, hits in zip(QUERIES, retriever.queries(QUERIES, top_k=3)):
       present(hits, q[:60])

We examine the extracted multimodal parts and convert the processed doc into page-level and full-document Markdown. We configure a dense retriever that embeds person queries and searches the LanceDB vector index for probably the most related doc chunks. We take a look at each particular person and batched queries whereas displaying web page numbers, similarity scores, and retrieved textual content previews.

Copy Code
if HAVE_KEY:
   print("n=== STAGE 4: retrieve + VL rerank ===")
   reranking = Retriever(
       run_mode="service",
       top_k=5,
       rerank=True,
       vdb_kwargs={"uri": LANCEDB_URI, "table_name": TABLE},
       embed_kwargs={
           "embedding_endpoint": EMBED_URL, "model_name": EMBED_MODEL,
           "embed_model_name": EMBED_MODEL, "api_key": API_KEY, "input_type": "question",
       },
       rerank_kwargs={
           "model_name": RERANK_MODEL,
           "invoke_url": RERANK_URL,
           "api_key": API_KEY,
           "refine_factor": 4,
           "batch_size": 16,
       },
   )
   strive:
       present(reranking.question(QUERIES[0]), "reranked")
   besides Exception as e:
       print("rerank unavailable, dense outcomes stand:", kind(e).__name__, str(e)[:160])
if HAVE_KEY:
   print("n=== STAGE 5: filtered retrieval ===")
   strive:
       hits = retriever.question(
           "gadget prices",
           top_k=5,
           vdb_kwargs={"the place": "textual content LIKE '%Cost%'"},
       )
       present(hits, "the place: textual content LIKE '%Cost%'")
   besides Exception as e:
       print("filter push-down failed:", kind(e).__name__, str(e)[:160])
   import lancedb
   tbl = lancedb.join(LANCEDB_URI).open_table(TABLE)
   print("nrows in LanceDB:", tbl.count_rows())
   print(tbl.to_pandas()[["text"]].head(3).to_string())

We create a vision-language reranking pipeline that retrieves a wider candidate set and reorders the outcomes in accordance with semantic relevance. We additionally apply a text-based filter to slender retrieval outcomes to chunks containing particular content material from the doc. We immediately examine the LanceDB desk to confirm the variety of saved information and study the listed textual content.

Copy Code
if HAVE_KEY:
   print("n=== STAGE 6: RAG reply ===")
   from openai import OpenAI
   shopper = OpenAI(base_url=CHAT_URL, api_key=API_KEY)
   def rag(query, ok=5):
       hits = retriever.question(query, top_k=ok)
       ctx = []
       for i, h in enumerate(hits, 1):
           meta = h.get("metadata")
           if isinstance(meta, str):
               strive: meta = json.hundreds(meta)
               besides Exception: meta = {}
           ctx.append(f"[{i}] (web page {(meta or {}).get('page_number','?')})n{h.get('textual content','')}")
       immediate = textwrap.dedent(f"""
           Answer the query utilizing ONLY the numbered context beneath.
           Cite the sources you used as [1], [2], and many others. If the context is
           inadequate, say so plainly.
           Context:
           {chr(10).be part of(ctx)}
           Question: {query}
           """)
       r = shopper.chat.completions.create(
           mannequin=LLM_MODEL,
           messages=[{"role": "user", "content": prompt}],
           temperature=0.0, max_tokens=512,
       )
       return r.decisions[0].message.content material, hits
   for q in QUERIES[:2]:
       strive:
           ans, _ = rag(q)
           print(f"nQ: {q}nA: {ans}n" + "-" * 70)
       besides Exception as e:
           print("era failed:", kind(e).__name__, str(e)[:200])
if HAVE_KEY:
   print("n=== Recall@ok examine ===")
   GOLD = [
       ("which animal is jumping onto a laptop", "Cat"),
       ("what does the chart show", "Gadgets"),
       ("which animal is at the beach", "Giraffe"),
   ]
   Okay = 5
   hit_lists = retriever.queries([q for q, _ in GOLD], top_k=Okay)
   bought = sum(
       any(exp.decrease() in str(h.get("textual content", "")).decrease() for h in hits)
       for (_, exp), hits in zip(GOLD, hit_lists)
   )
   print(f"recall@{Okay} = {bought}/{len(GOLD)} = {bought/len(GOLD):.2f}")
print("nDone. Artifacts: ./lancedb (vector desk), ./extracted.md (markdown).")

We mix the retrieved doc chunks with a hosted Nemotron language mannequin to generate solutions grounded solely within the equipped context. We embrace numbered supply references and web page metadata so the generated responses stay traceable to the unique doc. We conclude by calculating recall at ok for a small set of anticipated solutions and report the ultimate vector database and Markdown artifacts.

In conclusion, we created a full multimodal RAG system that transforms structured and unstructured PDF content material into searchable, citation-ready information. We used NeMo Retriever to coordinate extraction, deduplication, chunking, embedding, vector database indexing, retrieval, and reranking whereas retaining the Colab runtime light-weight by delegating mannequin inference to hosted NVIDIA NIM companies. We additionally generated grounded solutions with a Nemotron language mannequin and measured retrieval effectiveness with a easy recall-at-k take a look at. By finishing this workflow, we established a reusable basis for constructing doc intelligence purposes that course of textual content, tables, charts, and visible parts by means of a unified retrieval pipeline.

Check out the FULL CODES here. Also, be happy to observe us on Twitter and don’t overlook to hitch our 150k+ML SubReddit and Subscribe to our Newsletter. Wait! are you on telegram? now you can join us on telegram as well.

Need to associate with us for selling your GitHub Repo OR Hugging Face Page OR Product Release OR Webinar and many others.? Connect with us

The publish Building a Multimodal RAG Pipeline with NVIDIA NeMo Retriever, Hosted NIMs, LanceDB, Reranking, and Grounded Generation appeared first on MarkTechPost.

Similar Posts