Building an End-to-End Document Intelligence Pipeline with deepDoctection
In this tutorial, we implement a doc intelligence pipeline with deepDoctection 1.2.x that mixes structure detection, desk construction recognition, OCR, reading-order reconstruction, annotation linking, and structured export in a single workflow. We configure the analyzer explicitly with DocLayNet-based structure detection, Table Transformer construction recognition, and DocTR OCR, then examine the ensuing Page objects to grasp how deepDoctection represents textual content, figures, tables, relationships, provenance, and studying order. We additionally prolong the framework by registering customized object varieties and implementing our personal PipelineElement for extracting financial and date entities whereas classifying paperwork by their tabular traits. Finally, we assemble a customized pipeline manually with ServiceFactory, discover filtering and repair rollback, serialize processed pages, and rework doc annotations into ordered JSONL chunks appropriate for downstream RAG and retrieval techniques.
!pip set up -q "deepdoctection" "transformers>=5.2.0" "timm" "python-doctr" "pdfplumber" "networkx" "lxml"
import os
os.environ["DD_USE_TORCH"] = "True"
os.environ["DPI"] = "200"
os.environ["LOG_LEVEL"] = "INFO"
os.environ["ENABLE_DYNAMIC_OBJECT_TYPES"] = "False"
import json, re, textwrap
from pathlib import Path
from collections import Counter
import numpy as np
import matplotlib.pyplot as plt
from IPython.show import HTML, show
import deepdoctection as dd
print("deepdoctection:", dd.__version__)
import transformers.integrations.peft as _hf_peft
if _hf_peft.is_peft_available():
_hf_peft.is_peft_available = lambda: False
print("patched: PEFT adapter lookup disabled for from_pretrained")
!mkdir -p /content material/docs /content material/imgs
!wget -q -O /content material/docs/paper.pdf
!wget -q -O /content material/imgs/finance.png
https://uncooked.githubusercontent.com/deepdoctection/notebooks/major/pattern/finance/1bcac3899c9cb1c0b0f650b1431d3d52_7.png
PDF = Path("/content material/docs/paper.pdf")
PNG = Path("/content material/imgs/finance.png")
OUT = Path("/content material/out"); OUT.mkdir(exist_ok=True)
def present(img, w=16):
if img is None: return
plt.determine(figsize=(w, w * 1.3)); plt.axis("off"); plt.imshow(img); plt.present()
def analyze_any(pipe, path, **kw):
"""
Dispatch appropriately for a listing, a PDF, or a single picture file.
DoctectionPipe can stream a listing or a PDF from disk, however a *single*
picture has no reader — path= solely provides the file title / provenance, and
the pixels have to be handed in through bytes=. Without this you get:
ValueError: When passing a path to a single picture, bytes of the picture
have to be handed
"""
path = Path(path)
if path.is_dir():
kw.setdefault("file_type", [".jpg", ".png", ".jpeg", ".tif"])
return pipe.analyze(path=path, **kw)
if path.suffix.decrease() == ".pdf":
return pipe.analyze(path=path, **kw)
if path.suffix.decrease() in (".png", ".jpg", ".jpeg", ".tif"):
return pipe.analyze(path=path, bytes=path.read_bytes(), **kw)
elevate ValueError(f"unsupported enter: {path}")
We set up the required deepDoctection dependencies, configure its runtime atmosphere, and apply a compatibility patch for Transformers and PEFT. We obtain the pattern PDF and picture information that we use all through the tutorial and put together our output listing. We additionally outline helper capabilities to visualise pictures and constantly analyze directories, PDFs, and particular person picture information.
dd.print_model_infos(add_description=False, add_config=False, add_categories=False)
profile = dd.ModelCatalog.get_profile("Aryn/deformable-detr-DocLayNet/mannequin.safetensors")
print("nlayout mannequin classes:", profile.classes)
print("is registered:", dd.ModelCatalog.is_registered("Aryn/deformable-detr-DocLayNet/mannequin.safetensors"))
config_overwrite = [
"USE_ROTATOR=False",
"USE_LAYOUT=True",
"USE_LAYOUT_NMS=True",
"USE_TABLE_SEGMENTATION=True",
"USE_TABLE_REFINEMENT=False",
"USE_PDF_MINER=False",
"USE_OCR=True",
"USE_LAYOUT_LINK=True",
"LAYOUT.WEIGHTS=Aryn/deformable-detr-DocLayNet/model.safetensors",
"ITEM.WEIGHTS=deepdoctection/tatr_tab_struct_v2/model.safetensors",
"ITEM.FILTER=['table']",
"OCR.USE_DOCTR=True",
"OCR.USE_TESSERACT=False",
"OCR.USE_TEXTRACT=False",
"OCR.WEIGHTS.DOCTR_WORD=doctr/db_resnet50/db_resnet50-ac60cadc.pt",
"OCR.WEIGHTS.DOCTR_RECOGNITION=doctr/crnn_vgg16_bn/crnn_vgg16_bn-0417f351.pt",
"SEGMENTATION.THRESHOLD_ROWS=0.4",
"SEGMENTATION.THRESHOLD_COLS=0.4",
"SEGMENTATION.FULL_TABLE_TILING=True",
"WORD_MATCHING.RULE=ioa",
"WORD_MATCHING.THRESHOLD=0.3",
"WORD_MATCHING.MAX_PARENT_ONLY=True",
"TEXT_ORDERING.INCLUDE_RESIDUAL_TEXT_CONTAINER=True",
"TEXT_ORDERING.PARAGRAPH_BREAK=0.035",
"TEXT_ORDERING.BROKEN_LINE_TOLERANCE=0.003",
"LAYOUT_LINK.PARENTAL_CATEGORIES=['figure','table']",
"LAYOUT_LINK.CHILD_CATEGORIES=['caption']",
]
analyzer = dd.get_dd_analyzer(config_overwrite=config_overwrite)
print("n--- pipeline ---")
for sid, title in analyzer.get_pipeline_info().objects():
print(f"{sid} {title}")
print("n--- what this pipeline produces ---")
print(analyzer.get_meta_annotation())
We examine deepDoctection’s mannequin registry to confirm the structure mannequin and its supported doc classes. We explicitly configure the analyzer to mix structure detection, desk segmentation, DocTR OCR, phrase matching, reading-order reconstruction, and structure linking. We then initialize the analyzer and examine its pipeline elements and the annotation varieties that it produces.
df = analyze_any(analyzer, PDF, session_id="tutorial01", max_datapoints=3)
df.reset_state()
pages = checklist(df)
print(f"nparsed {len(pages)} pages")
web page = pages[0]
present(web page.viz(show_figures=True, show_residual_layouts=True, show_table_structure=True))
print("== narrative textual content ==")
print(textwrap.fill(web page.textual content[:900], 110))
print("n== structure blocks in studying order ==")
for doc_id, img_id, pno, ann_id, order, cat, txt in web page.chunks[:12]:
print(f"[{order:>3}] {str(cat):<15} {txt[:70]!r}")
print("n== class histogram ==")
print(Counter(a.category_name for a in web page.get_annotation()))
for fig in web page.figures:
linked = fig.get_relationship("layout_link")
print("determine", fig.annotation_id[:8], "-> caption ids:", [i[:8] for i in linked])
if web page.phrases:
w = web page.phrases[0]
print("nword:", w.characters, "| service:", w.service_id,
"| mannequin:", w.model_id, "| bbox:", [round(x) for x in w.bbox])
tbl_pages = [p for p in pages if p.tables]
if tbl_pages:
t = tbl_pages[0].tables[0]
print(f"desk {t.number_of_rows}x{t.number_of_columns}, "
f"max_row_span={t.max_row_span}, max_col_span={t.max_col_span}")
show(HTML(t.html))
for row in t.csv[:5]:
print([c[:22] for c in row])
for c in t.cells[:5]:
print(f" r{c.row_number} c{c.column_number} "
f"(span {c.row_span}x{c.column_span}) {c.textual content[:40]!r}")
else:
print("no desk on these pages — the finance.png pattern under has one")
We run the configured analyzer on the pattern PDF and materialize the ensuing pages from the lazy knowledge circulation. We examine narrative textual content, reading-order chunks, annotation classes, figure-caption relationships, phrase provenance, and bounding containers. We additionally entry detected tables by HTML, CSV, and particular person cell representations to look at their structured output.
@dd.object_types_registry.register("CustomKey")
class CustomKey(dd.ObjectTypes):
"""Custom abstract keys — have to be registered to be serialisable."""
MONEY_MENTIONS = "money_mentions"
DATE_MENTIONS = "date_mentions"
DOC_FLAVOUR = "doc_flavour"
@dd.object_types_registry.register("FlavourLabel")
class FlavourLabel(dd.ObjectTypes):
TABULAR = "tabular"
NARRATIVE = "narrative"
MIXED = "blended"
MONEY = re.compile(r"(?:[$€£]s?d[d,.]*|d[d,.]*s?(?:USD|EUR|GBP|million|bn))")
DATE = re.compile(r"b(?:d{1,2}[/-]d{1,2}[/-]d{2,4}|d{4}-d{2}-d{2}|"
r"(?:Jan|Feb|Mar|Apr|May|Jun|Jul|Aug|Sep|Oct|Nov|Dec)w*s+d{1,2},?s+d{4})b")
class EntityAndFlavourService(dd.PipelineElement):
def __init__(self, title="entity_flavour", tabular_ratio=0.25):
self.tabular_ratio = tabular_ratio
tremendous().__init__(title)
def serve(self, dp: dd.Image) -> None:
web page = dd.Page.from_image(dp, text_container=dd.LayoutLabel.WORD)
textual content = web page.text_no_line_break
cash = sorted(set(MONEY.findall(textual content)))
dates = sorted(set(DATE.findall(textual content)))
tables = web page.tables
table_area = sum((b[2] - b[0]) * (b[3] - b[1]) for b in (t.bbox for t in tables))
ratio = table_area / float(web page.width * web page.peak or 1)
taste = (FlavourLabel.TABULAR if ratio > self.tabular_ratio
else FlavourLabel.NARRATIVE if not tables
else FlavourLabel.MIXED)
self.dp_manager.set_summary_annotation(
summary_key=CustomKey.MONEY_MENTIONS, summary_name=CustomKey.MONEY_MENTIONS,
summary_value=cash)
self.dp_manager.set_summary_annotation(
summary_key=CustomKey.DATE_MENTIONS, summary_name=CustomKey.DATE_MENTIONS,
summary_value=dates)
self.dp_manager.set_summary_annotation(
summary_key=CustomKey.DOC_FLAVOUR, summary_name=flavour,
summary_score=spherical(ratio, 4))
def clone(self):
return self.__class__(self.title, self.tabular_ratio)
def get_meta_annotation(self) -> dd.MetaAnnotation:
return dd.MetaAnnotation(
image_annotations=(),
sub_categories={},
relationships={},
summaries=(CustomKey.MONEY_MENTIONS, CustomKey.DATE_MENTIONS, CustomKey.DOC_FLAVOUR),
)
for okay in (CustomKey.MONEY_MENTIONS, CustomKey.DATE_MENTIONS, CustomKey.DOC_FLAVOUR):
dd.Page.add_attribute_name(okay)
We register customized object varieties for extracted financial mentions, date mentions, and doc taste classifications. We implement a customized deepDoctection pipeline part that analyzes web page textual content and desk protection to generate these page-level summaries. We then expose the customized abstract fields as Page attributes in order that we will entry them instantly from processed paperwork.
from deepdoctection.analyzer import cfg, ServiceFactory
cfg.freeze(False)
cfg.USE_TABLE_SEGMENTATION = True
cfg.freeze(True)
elements = []
layout_detector = ServiceFactory.build_layout_detector(cfg, mode="LAYOUT")
elements.append(ServiceFactory.build_layout_service(cfg, detector=layout_detector, mode="LAYOUT"))
elements.append(ServiceFactory.build_layout_nms_service(cfg))
item_detector = ServiceFactory.build_layout_detector(cfg, mode="ITEM")
elements.append(ServiceFactory.build_sub_image_service(cfg, detector=item_detector, mode="ITEM"))
elements.append(ServiceFactory.build_table_segmentation_service(cfg, detector=item_detector))
word_detector = ServiceFactory.build_doctr_word_detector(cfg)
elements.append(ServiceFactory.build_doctr_word_detector_service(word_detector))
elements.append(ServiceFactory.build_text_extraction_service(cfg, ServiceFactory.build_ocr_detector(cfg)))
elements.append(ServiceFactory.build_word_matching_service(cfg))
elements.append(ServiceFactory.build_text_order_service(cfg))
elements.append(EntityAndFlavourService())
custom_pipe = dd.DoctectionPipe(pipeline_component_list=elements)
print("ncustom pipeline:", checklist(custom_pipe.get_pipeline_info().values()))
df2 = analyze_any(custom_pipe, PNG)
df2.reset_state()
fin_page = subsequent(iter(df2))
print("flavour :", fin_page.doc_flavour)
print("cash :", fin_page.money_mentions[:10])
print("dates :", fin_page.date_mentions[:10])
present(fin_page.viz(show_table_structure=True), w=13)
def skip_if_no_table(dp: dd.Image) -> bool:
return "desk" not in {a.category_name for a in dp.get_annotation()}
elements[-1].set_inbound_filter(skip_if_no_table)
det_sid = subsequent(sid for sid, n in analyzer.get_pipeline_info().objects()
if n.startswith("image_doctr"))
det_comp = analyzer.get_pipeline_component(service_id=det_sid)
df_undo = det_comp.undo(dd.DataFromList([p.base_image for p in pages]))
df_undo.reset_state()
undone = checklist(df_undo)
print("annotations earlier than/after undo:",
len(pages[0].get_annotation()),
len(dd.Page.from_image(undone[0]).get_annotation()))
We manually assemble a deepDoctection pipeline with ServiceFactory, combining structure evaluation, desk processing, OCR, textual content ordering, and our customized part. We execute this tradition pipeline on the monetary doc picture and examine the detected taste, financial values, dates, and desk construction. We additionally apply an inbound filter and reveal how we undo the annotations produced by a particular DocTR service.
for i, p in enumerate(pages):
p.save(image_to_json=False, path=OUT / f"page_{i}.json")
restored = dd.Page.from_file(str(OUT / "page_0.json"))
print("round-trip:", len(restored.get_annotation()), "of",
len(pages[0].get_annotation()), "annotations restored")
information = []
for p in pages:
for doc_id, img_id, pno, ann_id, order, cat, txt in p.chunks:
if txt and txt.strip():
information.append({"document_id": doc_id, "web page": pno, "order": order,
"class": str(cat), "annotation_id": ann_id, "textual content": txt})
for t in p.tables:
information.append({"document_id": p.document_id, "web page": p.page_number,
"order": -1, "class": "table_html",
"annotation_id": t.annotation_id, "textual content": t.html})
(OUT / "chunks.jsonl").write_text("n".be part of(json.dumps(r) for r in information))
print(f"n{len(information)} chunks -> {OUT/'chunks.jsonl'}")
print(json.dumps(information[0], indent=2)[:400])
We serialize every processed web page to JSON whereas preserving its structural annotations with out embedding the unique picture knowledge. We reload a saved web page and evaluate annotation counts to confirm that the structural data survives serialization. We lastly rework narrative chunks and desk HTML into JSONL information that we will use instantly in RAG, retrieval, and downstream document-processing pipelines.
In conclusion, we developed a sensible understanding of how deepDoctection orchestrates a number of document-analysis fashions and rule-based companies right into a configurable processing pipeline. We moved past merely operating a predefined analyzer by inspecting mannequin registrations, controlling particular person companies, accessing structured page-level annotations, extracting tables, creating customized abstract metadata, and composing our personal pipeline phases. We additionally examined how service filtering and undo operations have an effect on annotations, giving us finer management over advanced document-processing workflows. Finally, we serialized the processed doc construction. We generated RAG-ready chunks, giving us a reusable basis for constructing doc search, information extraction, retrieval-augmented era, and different production-oriented doc AI functions.
Check out the FULL CODES here. Also, be at liberty to observe us on Twitter and don’t neglect to affix our 150k+ML SubReddit and Subscribe to our Newsletter. Wait! are you on telegram? now you can join us on telegram as well.
Need to associate with us for selling your GitHub Repo OR Hugging Face Page OR Product Release OR Webinar and many others.? Connect with us
The publish Building an End-to-End Document Intelligence Pipeline with deepDoctection appeared first on MarkTechPost.
