Analyzer_Configuration.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Analyzer Configuration\n",
    "\n",
    "Document extraction from PDFs and scans is nothing that can be done in only one way. On the contrary, one needs flexibility in the extraction process and this flexibility will be provided by an extensive customization of **deep**doctection analyzer. \n",
    "\n",
    "We will now discuss the most important configuration options. We will assume familiarity with the [Get started notebook](./Get_Started.ipynb) as well as the [pipeline notebook](./Pipelines.ipynb). \n",
    "\n",
    "## How to change configuration\n",
    "\n",
    "Configuration can be done either directly in the `.yaml` file or by explicitly overwriting the default variables. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import deepdoctection as dd\n",
    "\n",
    "config_overwrite = [\"USE_TABLE_SEGMENTATION=False\",\"USE_OCR=False\"]\n",
    "\n",
    "analyzer = dd.get_dd_analyzer(config_overwrite=config_overwrite)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## High level Configuration\n",
    "\n",
    "The analyzer consists of four steps: Layout detection, table recognition, text extraction and recombination. You can switch each block on and off:\n",
    "\n",
    "```yaml\n",
    "USE_LAYOUT: True\n",
    "\n",
    "USE_TABLE_SEGMENTATION: True   # this will determine cells and rows/column of a detected table\n",
    "USE_TABLE_REFINEMENT: True     # this will guarantee that the table structure can be expressed in a valid html-format\n",
    "\n",
    "USE_PDF_MINER: False           # when processing a PDF file, it will first try to extract words using pdfplumber \n",
    "USE_OCR: True                  # using one of the three available OCR tools to be specified later  \n",
    "```\n",
    "\n",
    "## Layout models\n",
    "\n",
    "Layout detection uses either Tensorpack's Cascade-RCNN or Detectron2 Cascade-RCNN by default, depending on which DL framework PyTorch or Tensorflow has been installed. The model have been trained on Publaynet and therefore provide detection of one of the following segments: `text, title, table, figure, list`. You can filter any of the segments, e.g.\n",
    "\n",
    "```yaml\n",
    "PT:\n",
    "   LAYOUT:\n",
    "      WEIGHTS: layout/d2_model_0829999_layout_inf_only.pt\n",
    "      FILTER:\n",
    "         - figure\n",
    "```\n",
    "\n",
    "### Table transformer\n",
    "\n",
    "You can use the [table transformer model](https://github.com/microsoft/table-transformer) for table detection. \n",
    "\n",
    "\n",
    "```yaml\n",
    "PT:\n",
    "   LAYOUT:\n",
    "      WEIGHTS: microsoft/table-transformer-detection/pytorch_model.bin\n",
    "   PAD:\n",
    "      TOP: 60\n",
    "      RIGHT: 60\n",
    "      BOTTOM: 60\n",
    "      LEFT\": 60\n",
    "\n",
    "```\n",
    "\n",
    "Table transformer requires image padding for more accurate results. The default padding provided might not be optimal. You can tweak and change it according to your needs.\n",
    "\n",
    "\n",
    "### Custom model\n",
    "\n",
    "A custom model can be added as well, but it needs to be registered. The same holds true for some special categories. We refer to [this notebook](./Running_pre_trained_models_from_third_party_libraries.ipynb) for adding your own or third party models.\n",
    "\n",
    "## Table segmentation\n",
    "\n",
    "Table segmentation, e.g. the determination of cells, rows and columns as well as multi-span cells can be done in two different ways. \n",
    "\n",
    "*The original **deep**doctection process*: With one cell detector and one row-column detector each, the basis for the table structure is created. Then, by superimposing cells to rows and columns, the row and column number of the cell is determined. This admittedly rather short description should not hide the fact that this derivation consists of many intermediate steps. Row and column numbers must be stretched to completely overlay the table. If necessary, some overlapping rows and columns must be removed. An overlay rule of cells to rows and columns must be set, etc. \n",
    "\n",
    "*The Table Transformer process*: Here, rows and columns as well as multi-spanning cells are detected in only one detector. In contrast to the process proposed in the original repo, cells are simply detected by overlapping rows and columns. Then it is looked which of the simple cells can be replaced by multi-spanning cells. In a last step, the row and column number of a cell is determined. \n",
    "\n",
    "Table transformer is only available with PyTorch.\n",
    "\n",
    "The default configuration uses the **deep**doctection process.\n",
    "\n",
    "```yaml\n",
    "SEGMENTATION:\n",
    "  ASSIGNMENT_RULE: ioa  # iou is another cell/row/column overlapping rule\n",
    "  THRESHOLD_ROWS: 0.4\n",
    "  THRESHOLD_COLS: 0.4\n",
    "  FULL_TABLE_TILING: True # in order to guarantee that the table is completely covered with rows and columns, resp.\n",
    "  REMOVE_IOU_THRESHOLD_ROWS: 0.001\n",
    "  REMOVE_IOU_THRESHOLD_COLS: 0.001\n",
    "  STRETCH_RULE: equal  # how to stretch row/columns: left is another choice\n",
    "```\n",
    "\n",
    "As a rule of thumb for the configuration of the segmentation the following can be stated: The better the detectors work for the use case, the higher the thresholds should be chosen to leverage the results.\n",
    "\n",
    "To perform table segmentation with Table transformer, the configuration must be set as follows:\n",
    "\n",
    "```yaml\n",
    "PT:\n",
    "   ITEM:\n",
    "     WEIGHTS: microsoft/table-transformer-structure-recognition/pytorch_model.bin\n",
    "     FILTER:\n",
    "        - table # model detects tables which are redundant and must be filtered\n",
    "```\n",
    "\n",
    "In our own experience, it has been shown that the recognition of multi spanning cells does not work reliably for tables that do not originate from medical articles. If you can do without determining multi-spanning cells/headers, it is recommended to filter them.\n",
    "\n",
    "```yaml\n",
    "PT:\n",
    "   ITEM:\n",
    "     WEIGHTS: microsoft/table-transformer-structure-recognition/pytorch_model.bin\n",
    "     FILTER:\n",
    "        - table # model detects tables which are redundant and must be filtered\n",
    "        - column_header\n",
    "        - projected_row_header\n",
    "        - spanning\n",
    "```\n",
    "\n",
    "## Text extraction\n",
    "\n",
    "There are four different options for text extraction.\n",
    "\n",
    "### PDFPlumber\n",
    "\n",
    "Extraction with pdfplumber. This requires native PDF documents where the text can be extracted from the byte encoding. Scans and text from images are not included here. \n",
    "\n",
    "```yaml\n",
    "USE_PDF_MINER: True\n",
    "```\n",
    "\n",
    "The remaining three are all OCR methods.\n",
    "\n",
    "It is possible to select PdfPlumber in combination with an OCR (exception: DocTr). If no text was extracted with PdfPlumber, the OCR service will be called, otherwise it will be omitted. There is currently not option to grap everything with OCR that cannot be extracted with PdfPlumber. It is all or nothing.  \n",
    "\n",
    "```yaml\n",
    "USE_PDF_MINER: True\n",
    "USE_OCR: True\n",
    "OCR:\n",
    "  USE_TESSERACT: True\n",
    "  USE_DOCTR: False\n",
    "  USE_TEXTRACT: False\n",
    "```\n",
    "\n",
    "### Tesseract\n",
    "\n",
    "Tesseract has its own configuration file `conf_tesseract.yaml`, which is also located in the `.cache`. Here you can enter all parameters that are also valid via the Tesseract CLI. We refer to Tesseract's documentation. \n",
    "\n",
    "### DocTr\n",
    "\n",
    "DocTr is a powerful OCR library with different models. Only one model is currently registered for PyTorch/Tensorflow, but there are more pre-trained models that can also be used in this framework after registration. \n",
    "\n",
    "DocTr uses a textline detector and a text recognizer whose models can both be loaded. These are both included in the default configuration. \n",
    "\n",
    "```yaml\n",
    "OCR:\n",
    "  WEIGHTS:\n",
    "    DOCTR_WORD:\n",
    "      TF: doctr/db_resnet50/tf/db_resnet50-adcafc63.zip\n",
    "      PT: doctr/db_resnet50/pt/db_resnet50-ac60cadc.pt\n",
    "    DOCTR_RECOGNITION:\n",
    "      TF: doctr/crnn_vgg16_bn/tf/crnn_vgg16_bn-76b7f2c6.zip\n",
    "      PT: doctr/crnn_vgg16_bn/pt/crnn_vgg16_bn-9762b0b0.pt\n",
    "```\n",
    "\n",
    "### AWS Textract\n",
    "\n",
    "Textract is the AWS OCR solution that can be accessed via an API. This is a paid service and requires an AWS account, installation of the AWS CLI, and a token. We refer to the official documentation to access the service via API.\n",
    "\n",
    "## Word matching\n",
    "\n",
    "We have already discussed word matching in the pipeline notebook, we can cover the main topics quickly here.\n",
    "\n",
    "We determine which layout segments should be considered as candidates for matching words. A relation is then created using an overlap rule. Layout segments that are not listed are not available as candidates for a relation.\n",
    "\n",
    "```yaml\n",
    "WORD_MATCHING:\n",
    "  PARENTAL_CATEGORIES:\n",
    "    - text\n",
    "    - title\n",
    "    - list\n",
    "    - cell  # Note, that there is no relationship between tables and words. The reason is, that we want to relate cells and words. \n",
    "  RULE: ioa  # choose iou otherwise\n",
    "  THRESHOLD: 0.6\n",
    "```\n",
    "\n",
    "## Text ordering\n",
    "\n",
    "We have also dealt with reading order in more detail in the pipeline notebook.\n",
    "\n",
    "The layout sections that contain text and that need to be sorted need to be configured. If the section is listed in `TEXT_BLOCK_CATEGORIES`, the assigned words are run through the sorting algorithm.\n",
    "\n",
    "The layout sections that are to be merged into continuous text across the entire page must also be configured. All sections listed in `FLOATING_TEXT_BLOCK_CATEGORIES` are taken into account.\n",
    "\n",
    "The reading order is determined heuristically. The following parameters can be changed, but we refer to the API documentation for their meaning.\n",
    "\n",
    "```yaml\n",
    "TEXT_ORDERING:\n",
    "  INCLUDE_RESIDUAL_TEXT_CONTAINER: False\n",
    "  STARTING_POINT_TOLERANCE: 0.005\n",
    "  BROKEN_LINE_TOLERANCE: 0.003\n",
    "  HEIGHT_TOLERANCE: 2.0\n",
    "  PARAGRAPH_BREAK: 0.035\n",
    "```\n",
    "\n",
    "Words that have not been assigned to a layout segment are initially no longer displayed. To avoid losing words or even phrases, set\n",
    "\n",
    "```yaml\n",
    "TEXT_ORDERING:\n",
    "  INCLUDE_RESIDUAL_TEXT_CONTAINER: True\n",
    "```\n",
    "\n",
    "When doing this, words will be grouped into lines of synthetic text and the lines are treated and arranged as layout segments."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "dd_pt_033",
   "language": "python",
   "name": "dd_pt_033"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.20"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}