Running_pre_trained_models_from_third_party_libraries.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Running pre-trained models from Layout-Parser\n",
    "\n",
    "[**Layout-Parser**](https://layout-parser.github.io/) provides several models trained on datasets like Publaynet but from many different areas as well.\n",
    "\n",
    "In this tutorial we will show you how to configure and run a model from this library within a **deep**doctection pipeline.\n",
    "\n",
    "Layout-Parser provides some pre-trained Detectron2 models for various document layout analysis tasks. Models from other libraries are available as well, but running Detectron2 models with **deep**doctection is particularly easy because model wrappers are already available.\n",
    "\n",
    "You can find the Layout-Parser catalog [here](https://github.com/Layout-Parser/layoutparser/blob/main/src/layoutparser/models/detectron2/catalog.py) .\n",
    "\n",
    "Let's download `faster_rcnn_R_50_FPN_3x` trained on Publaynet. Enter the URL into your browser and the download starts. We need the model weights and the config file. \n",
    "\n",
    "To make it easy we suggest to save config file in **deep**doctection's `.cache` directory.\n",
    "\n",
    "Assume you have saved the model at: \n",
    "\n",
    "`~/.cache/deepdoctection/weights/layoutparser/publaynet/model_final.pth`\n",
    "\n",
    "and the config file at:\n",
    "\n",
    "`~/.cache/deepdoctection/configs/layoutparser/publaynet/config.yml`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/janis/Documents/Repos/deepdoctection_pt/.venv/lib/python3.9/site-packages/tqdm/auto.py:22: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html\n",
      "  from .autonotebook import tqdm as notebook_tqdm\n",
      "\u001b[32m[0715 10:48.10 @file_utils.py:36]\u001b[0m  \u001b[32mINF\u001b[0m  \u001b[97mPyTorch version 2.1.2+cu121 available.\u001b[0m\n",
      "\u001b[32m[0715 10:48.10 @file_utils.py:74]\u001b[0m  \u001b[32mINF\u001b[0m  \u001b[97mDisabling Tensorflow because USE_TORCH is set\u001b[0m\n"
     ]
    }
   ],
   "source": [
    "import deepdoctection as dd\n",
    "from matplotlib import pyplot as plt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Adding model to the `ModelCatalog`\n",
    "\n",
    "Next we will be adding the model to the `ModelCatalog`. This is not required but it simplifies a lot of things. \n",
    "You need to pass a key, e.g. `layoutparser/publaynet/model_final.pth` and a `ModelProfile`. A `ModelProfile` stores meta data about the model. Make sure that `ModelProfile.name` and `ModelProfile.config` is equal to the relative path of the model weights and the config file. \n",
    "\n",
    "It is convenient to add the categories. You can find the categories of the model [here](https://github.com/Layout-Parser/layout-parser/blob/main/src/layoutparser/models/detectron2/catalog.py) as well.  Note however, that, unlike in Layout-Parser all categories in deepdoctection start with 1. You therefore have to increment all category ids by 1. Also please ensure that all category ids are strings.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "dd.ModelCatalog.register(\"layoutparser/publaynet/model_final.pth\",dd.ModelProfile(\n",
    "            name=\"layoutparser/publaynet/model_final.pth\",\n",
    "            description=\"Add some infos regarding the model\",\n",
    "            config=\"layoutparser/publaynet/config.yml\",\n",
    "            tp_model=False,\n",
    "            size = [],\n",
    "            categories={\n",
    "                1: dd.LayoutType.TEXT,\n",
    "                2: dd.LayoutType.TITLE,\n",
    "                3: dd.LayoutType.LIST,\n",
    "                4: dd.LayoutType.TABLE,\n",
    "                5: dd.LayoutType.FIGURE,\n",
    "            },\n",
    "            model_wrapper=\"D2FrcnnDetector\",\n",
    "        ))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "path_weights = dd.ModelCatalog.get_full_path_weights(\"layoutparser/publaynet/model_final.pth\")\n",
    "path_config = dd.ModelCatalog.get_full_path_configs(\"layoutparser/publaynet/model_final.pth\")\n",
    "categories = dd.ModelCatalog.get_profile(\"layoutparser/publaynet/model_final.pth\").categories\n",
    "\n",
    "d2_detector = dd.D2FrcnnDetector(path_config,path_weights,categories)\n",
    "image_layout = dd.ImageLayoutService(d2_detector)\n",
    "\n",
    "pipe = dd.DoctectionPipe([image_layout])\n",
    "\n",
    "path = \"/home/janis/Documents/Repos/notebooks/pics/samples\"\n",
    "df = pipe.analyze(path=\"/path/to/publaynet_dir\")\n",
    "df.reset_state()\n",
    "\n",
    "df_iter = iter(df)\n",
    "dp = next(df_iter)\n",
    "\n",
    "image = dp.viz()\n",
    "\n",
    "plt.figure(figsize = (25,17))\n",
    "plt.axis('off')\n",
    "plt.imshow(image)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![layoutparser_1.png](./pics/layoutparser_1.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can use some additional config parameters that are only available in **deep**doctection like `NMS_THRESH_CLASS_AGNOSTIC`. This can reduce the number of overlapping layout segments."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "path_weights = dd.ModelCatalog.get_full_path_weights(\"layoutparser/publaynet/model_final.pth\")\n",
    "path_config = dd.ModelCatalog.get_full_path_configs(\"layoutparser/publaynet/model_final.pth\")\n",
    "categories = dd.ModelCatalog.get_profile(\"layoutparser/publaynet/model_final.pth\").categories\n",
    "\n",
    "d2_detector = dd.D2FrcnnDetector(path_config,path_weights,categories,config_overwrite=[\"NMS_THRESH_CLASS_AGNOSTIC=0.001\"])\n",
    "image_layout = dd.ImageLayoutService(d2_detector)\n",
    "\n",
    "pipe = dd.DoctectionPipe([image_layout])\n",
    "\n",
    "df = pipe.analyze(path=\"/path/to/publaynet\")\n",
    "df.reset_state()\n",
    "\n",
    "df_iter = iter(df)\n",
    "dp = next(df_iter)\n",
    "\n",
    "image = dp.viz()\n",
    "\n",
    "plt.figure(figsize = (25,17))\n",
    "plt.axis('off')\n",
    "plt.imshow(image)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![layoutparser_2.png](./pics/layoutparser_2.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## A second example\n",
    "\n",
    "Let's have a look at a second example. We will be using the model trained on the NewspaperNavigator dataset. This dataset detects labels that have not been used in **deep**doctection before. As all categories are registered in a sub-class of `ObjectTypes` which in turn, is a sub-class of `Enum` we need to define a derived `ObjectTypes` class listing all new layout sections.   "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "@dd.object_types_registry.register(\"NewspaperType\")\n",
    "class NewspaperExtension(dd.ObjectTypes):\n",
    "    \"\"\"Additional Newspaper labels not registered yet\"\"\"\n",
    "\n",
    "    PHOTOGRAPH =\"Photograph\",\n",
    "    ILLUSTRATION = \"Illustration\",\n",
    "    MAP = \"Map\",\n",
    "    COMIC = \"Comics/Cartoon\",\n",
    "    EDITORIAL_CARTOON = \"Editorial Cartoon\",\n",
    "    HEADLINE = \"Headline\",\n",
    "    ADVERTISEMENT =  \"Advertisement\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There are two more things one needs to take care of.\n",
    "\n",
    "1.) If we want to display the layout result we need to characterize the type of detected sections. There are several classes. Most of them are barely bounded layout components that only contain text and we call them `Layout` structures. There are other components as well, like `Table` which inherit a more complex structure like having `Cell`s. There is a dictionary `IMAGE_ANNOTATION_TO_LAYOUTS` available that maps the `Enum` members to the specific classes.\n",
    "\n",
    "2.) When parsing the detected components into the `Page` format we need to add the components to the top level layout sections. While this not really important for the very small pipeline we have been creating, not adding them would also prevent the layout sections to be visualized."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "from deepdoctection.datapoint.view import IMAGE_ANNOTATION_TO_LAYOUTS, Layout\n",
    "\n",
    "IMAGE_ANNOTATION_TO_LAYOUTS.update({i: Layout for i in NewspaperExtension})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Everything else is pretty much straight forward."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "dd.ModelCatalog.register(\"layoutparser/newspaper/model_final.pth\",dd.ModelProfile(\n",
    "            name=\"layoutparser/newspaper/model_final.pth\",\n",
    "            description=\"layout detection \",\n",
    "            config=\"layoutparser/newspaper/config.yml\",\n",
    "            size=[],\n",
    "            tp_model=False,\n",
    "            categories={1: NewspaperExtension.PHOTOGRAPH,\n",
    "                        2: NewspaperExtension.ILLUSTRATION,\n",
    "                        3: NewspaperExtension.MAP,\n",
    "                        4: NewspaperExtension.COMIC,\n",
    "                        5: NewspaperExtension.EDITORIAL_CARTOON,\n",
    "                        6: NewspaperExtension.HEADLINE,\n",
    "                        7: NewspaperExtension.ADVERTISEMENT},\n",
    "            model_wrapper=\"D2FrcnnDetector\",\n",
    "        ))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\u001b[32m[0715 11:07.14 @detection_checkpoint.py:38]\u001b[0m  \u001b[32mINF\u001b[0m  \u001b[97m[DetectionCheckpointer] Loading from /media/janis/Elements/.cache/deepdoctection/weights/layoutparser/newspaper/model_final.pth ...\u001b[0m\n",
      "\u001b[32m[0715 11:07.14 @checkpoint.py:150]\u001b[0m  \u001b[32mINF\u001b[0m  \u001b[97m[Checkpointer] Loading from /media/janis/Elements/.cache/deepdoctection/weights/layoutparser/newspaper/model_final.pth ...\u001b[0m\n"
     ]
    }
   ],
   "source": [
    "path_weights = dd.ModelCatalog.get_full_path_weights(\"layoutparser/newspaper/model_final.pth\")\n",
    "path_config = dd.ModelCatalog.get_full_path_configs(\"layoutparser/newspaper/model_final.pth\")\n",
    "categories = dd.ModelCatalog.get_profile(\"layoutparser/newspaper/model_final.pth\").categories\n",
    "\n",
    "d2_detector = dd.D2FrcnnDetector(path_config,path_weights,categories,config_overwrite=[\"NMS_THRESH_CLASS_AGNOSTIC=0.8\",\"MODEL.ROI_HEADS.SCORE_THRESH_TEST=0.1\"])\n",
    "image_layout = dd.ImageLayoutService(d2_detector)\n",
    "\n",
    "page_parser = dd.PageParsingService(text_container = dd.LayoutType.WORD, # this argument is required but will not have any effect\n",
    "                                    floating_text_block_categories=[layout_item for layout_item in NewspaperExtension])\n",
    "pipe = dd.DoctectionPipe([image_layout],page_parsing_service = page_parser)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df = pipe.analyze(path=\"/path/to/dir/newspaper_layout\")\n",
    "df.reset_state()\n",
    "\n",
    "df_iter = iter(df)\n",
    "dp = next(df_iter)\n",
    "\n",
    "image = dp.viz()\n",
    "\n",
    "plt.figure(figsize = (25,17))\n",
    "plt.axis('off')\n",
    "plt.imshow(image)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![layoutparser_3.png](./pics/layoutparser_3.png)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "dp.layouts"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[Layout(active=True, _annotation_id='5b0bd0dd-300c-3303-ad45-ffcb50ba5af8', category_name=<NewspaperExtension.headline>, _category_name=<NewspaperExtension.headline>, category_id='6', score=0.9875668287277222, sub_categories={}, relationships={}, bounding_box=BoundingBox(absolute_coords=True, ulx=14.930139541625977, uly=194.06497192382812, lrx=518.3706665039062, lry=270.4627685546875, height=76.39779663085938, width=503.4405269622803)),\\n\",\n",
    "Layout(active=True, _annotation_id='b6bf8f1a-a62b-3958-8f9d-7c0dc7c79354', category_name=<NewspaperExtension.photograph>, _category_name=<NewspaperExtension.photograph>, category_id='1', score=0.9749446511268616, sub_categories={}, relationships={}, bounding_box=BoundingBox(absolute_coords=True, ulx=275.99072265625, uly=522.1495971679688, lrx=454.6565246582031, lry=775.3734741210938, height=253.223876953125, width=178.66580200195312)),\n",
    "       \" Layout(active=True, _annotation_id='7fd1431f-4e48-3ff5-9fe9-be7c11c11bab', category_name=<NewspaperExtension.photograph>, _category_name=<NewspaperExtension.photograph>, category_id='1', score=0.8084900379180908, sub_categories={}, relationships={}, bounding_box=BoundingBox(absolute_coords=True, ulx=12.471419334411621, uly=268.1596984863281, lrx=273.9352722167969, lry=506.079833984375, height=237.92013549804688, width=261.46385288238525)),\n",
    "       \" Layout(active=True, _annotation_id='94c10163-eb20-3694-afa0-1b6cac9efda4', category_name=<NewspaperExtension.advertisement>, _category_name=<NewspaperExtension.advertisement>, category_id='7', score=0.44110408425331116, sub_categories={}, relationships={}, bounding_box=BoundingBox(absolute_coords=True, ulx=8.226184844970703, uly=493.3304138183594, lrx=276.7277526855469, lry=779.7467041015625, height=286.4162902832031, width=268.5015678405762)),\n",
    "       \" Layout(active=True, _annotation_id='98a12ec2-4b2f-36a6-ab49-10accaac5912', category_name=<NewspaperExtension.illustration>, _category_name=<NewspaperExtension.illustration>, category_id='2', score=0.37421464920043945, sub_categories={}, relationships={}, bounding_box=BoundingBox(absolute_coords=True, ulx=0.0, uly=24.19247817993164, lrx=539.7670288085938, lry=261.5598449707031, height=237.36736679077148, width=539.7670288085938)),\n",
    "       \" Layout(active=True, _annotation_id='700c979a-4c6a-3218-ae14-fd8009768590', category_name=<NewspaperExtension.illustration>, _category_name=<NewspaperExtension.illustration>, category_id='2', score=0.23696725070476532, sub_categories={}, relationships={}, bounding_box=BoundingBox(absolute_coords=True, ulx=239.40988159179688, uly=81.54991149902344, lrx=298.70074462890625, lry=152.0034942626953, height=70.45358276367188, width=59.290863037109375)),\n",
    "       \" Layout(active=True, _annotation_id='aa7d72ce-46c5-3304-88a8-1403251ce0e3', category_name=<NewspaperExtension.advertisement>, _category_name=<NewspaperExtension.advertisement>, category_id='7', score=0.14747683703899384, sub_categories={}, relationships={}, bounding_box=BoundingBox(absolute_coords=True, ulx=260.8577575683594, uly=500.2983093261719, lrx=520.8097534179688, lry=781.0679321289062, height=280.7696228027344, width=259.9519958496094))]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "dd_pt_033",
   "language": "python",
   "name": "dd_pt_033"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.19"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}