Layoutlm_v1_on_custom_token_classification.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# LayoutLMv1 for financial report NER\n",
    "\n",
    "This notebook is the beginning of a series of training and evaluation scripts for the LayoutLM family of models.\n",
    "\n",
    "The goal is to train the models LayoutLMv1, LayoutLMv2, LayoutXLM and LayoutLMv3 for token classification on a custom dataset. The goal is to give a realistic setting and to document the findings.\n",
    "\n",
    "We use the self-labeled dataset Funds Report Front Page Entities (FRFPE), which can be downloaded from [Huggingface](https://huggingface.co/datasets/deepdoctection/fund_ar_front_page). \n",
    "\n",
    "For experimentation, we use the W&B framework, which is integrated into the training and evaluation. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/janis/Documents/Repos/deepdoctection_pt/.venv/lib/python3.9/site-packages/tqdm/auto.py:22: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html\n",
      "  from .autonotebook import tqdm as notebook_tqdm\n",
      "\u001b[32m[0712 18:07.41 @file_utils.py:36]\u001b[0m  \u001b[32mINF\u001b[0m  \u001b[97mPyTorch version 2.1.2+cu121 available.\u001b[0m\n",
      "\u001b[32m[0712 18:07.41 @file_utils.py:74]\u001b[0m  \u001b[32mINF\u001b[0m  \u001b[97mDisabling Tensorflow because USE_TORCH is set\u001b[0m\n"
     ]
    }
   ],
   "source": [
    "import deepdoctection as dd\n",
    "from collections import defaultdict\n",
    "import wandb\n",
    "from transformers import LayoutLMTokenizerFast"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Defining object types\n",
    "\n",
    "**FRFPE** contains categories that have not been defined in **deep**doctection. These must first be made known to the framework. `TokenClasses.other` has already been defined."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "@dd.object_types_registry.register(\"ner_first_page\")\n",
    "class FundsFirstPage(dd.ObjectTypes):\n",
    "\n",
    "    REPORT_DATE = \"report_date\"\n",
    "    UMBRELLA = \"umbrella\"\n",
    "    REPORT_TYPE = \"report_type\"\n",
    "    FUND_NAME = \"fund_name\"\n",
    "\n",
    "dd.update_all_types_dict()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Download the dataset and save it to "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "PosixPath('/media/janis/Elements/.cache/deepdoctection/datasets/FRFPE')"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dd.get_dataset_dir_path() / \"FRFPE\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Visualization and display of ground truth"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "path = dd.get_dataset_dir_path() / \"FRFPE\" / \"40952248ba13ae8bfdd39f56af22f7d9_0.json\"\n",
    "\n",
    "page = dd.Page.from_file(path)\n",
    "page.image =  dd.load_image_from_file(path.parents[0]  / \"image\" / page.file_name.replace(\"pdf\",\"png\"))\n",
    "#page.viz(interactive=True,show_words=True)  # close interactive window with q"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "word: GFG, category: umbrella, bio: B\n",
      "word: Funds, category: umbrella, bio: I\n",
      "word: Société, category: other, bio: O\n",
      "word: d, category: other, bio: O\n",
      "word: ', category: other, bio: O\n",
      "word: Investissement, category: other, bio: O\n",
      "word: à, category: other, bio: O\n",
      "word: Capital, category: other, bio: O\n",
      "word: Variable, category: other, bio: O\n",
      "word: incorporated, category: other, bio: O\n",
      "word: in, category: other, bio: O\n",
      "word: Luxembourg, category: other, bio: O\n",
      "word: Luxembourg, category: other, bio: O\n",
      "word: R, category: other, bio: O\n",
      "word: ., category: other, bio: O\n",
      "word: C, category: other, bio: O\n",
      "word: ., category: other, bio: O\n",
      "word: S, category: other, bio: O\n",
      "word: ., category: other, bio: O\n",
      "word: B60668, category: other, bio: O\n",
      "word: Unaudited, category: other, bio: O\n",
      "word: Semi-Annual, category: report_type, bio: B\n",
      "word: Report, category: report_type, bio: I\n",
      "word: as, category: other, bio: O\n",
      "word: at, category: other, bio: O\n",
      "word: 30.06.2021, category: report_date, bio: B\n"
     ]
    }
   ],
   "source": [
    "for word in page.words:\n",
    "    print(f\"word: {word.characters}, category: {word.token_class}, bio: {word.tag}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will not use the `word.tag`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Defining Dataflow and Dataset\n",
    "\n",
    "We define dataflow and use the interface for the `CustomDataset`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "@dd.curry\n",
    "def overwrite_location_and_load(dp, image_dir, load_image):\n",
    "    image_file = image_dir / dp.file_name.replace(\"pdf\",\"png\")\n",
    "    dp.location = image_file.as_posix()\n",
    "    if load_image:\n",
    "        dp.image = dd.load_image_from_file(image_file)\n",
    "    return dp\n",
    "\n",
    "class NerBuilder(dd.DataFlowBaseBuilder):\n",
    "\n",
    "    def build(self, **kwargs) -> dd.DataFlow:\n",
    "        load_image = kwargs.get(\"load_image\", False)\n",
    "\n",
    "        ann_files_dir = self.get_workdir()\n",
    "        image_dir = self.get_workdir() / \"image\"\n",
    "\n",
    "        df = dd.SerializerFiles.load(ann_files_dir,\".json\")   # get a stream of .json files\n",
    "        df = dd.MapData(df, dd.Image.from_file)   # load .json file\n",
    "\n",
    "        df = dd.MapData(df, overwrite_location_and_load(image_dir, load_image))\n",
    "\n",
    "        if self.categories.is_filtered():\n",
    "            df = dd.MapData(\n",
    "                df,\n",
    "                dd.filter_cat(\n",
    "                    self.categories.get_categories(as_dict=False, filtered=True),\n",
    "                    self.categories.get_categories(as_dict=False, filtered=False),\n",
    "                ),\n",
    "            )\n",
    "        df = dd.MapData(df,dd.re_assign_cat_ids(cat_to_sub_cat_mapping=self.categories.get_sub_categories(\n",
    "                                                 categories=dd.LayoutType.WORD,\n",
    "                                                 sub_categories={dd.LayoutType.WORD: dd.WordType.TOKEN_CLASS},\n",
    "                                                 keys = False,\n",
    "                                                 values_as_dict=True,\n",
    "                                                 name_as_key=True)))\n",
    "\n",
    "        return df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "ner = dd.CustomDataset(name = \"FRFPE\",\n",
    "                 dataset_type=dd.DatasetType.TOKEN_CLASSIFICATION,\n",
    "                 location=\"FRFPE\",\n",
    "                 init_categories=[dd.LayoutType.TEXT, dd.LayoutType.TITLE, dd.LayoutType.LIST, dd.LayoutType.TABLE,\n",
    "                                  dd.LayoutType.FIGURE, dd.LayoutType.LINE, dd.LayoutType.WORD],\n",
    "                 init_sub_categories={dd.LayoutType.WORD: {dd.WordType.TOKEN_CLASS: [FundsFirstPage.REPORT_DATE,\n",
    "                                                                                     FundsFirstPage.REPORT_TYPE,\n",
    "                                                                                     FundsFirstPage.UMBRELLA,\n",
    "                                                                                     FundsFirstPage.FUND_NAME,\n",
    "                                                                                     dd.TokenClasses.OTHER],\n",
    "                                                           dd.WordType.TAG: []}},\n",
    "                 dataflow_builder=NerBuilder)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Defining a split and saving the split distribution as W&B artifact \n",
    "\n",
    "The ground truth contains some layout sections `ImageAnnotation` that we need to explicitly filter out.\n",
    "\n",
    "We define a split with ~90% train, ~5% validation and ~5% test samples.\n",
    "\n",
    "To reproduce the split later we save the split as a W&B artifact."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\u001b[32m[0712 18:08.06 @base.py:259]\u001b[0m  \u001b[32mINF\u001b[0m  \u001b[97mWill use the same build setting for all dataflows\u001b[0m\n",
      "|                                                                                                                                                                                                                                                                                                   |357/?[00:00<00:00,26702.93it/s]\n",
      "|                                                                                                                                                                                                                                                                                                      |357/?[00:05<00:00,70.27it/s]\n",
      "\u001b[32m[0712 18:08.12 @base.py:314]\u001b[0m  \u001b[32mINF\u001b[0m  \u001b[97m___________________ Number of datapoints per split ___________________\u001b[0m\n",
      "\u001b[32m[0712 18:08.12 @base.py:315]\u001b[0m  \u001b[32mINF\u001b[0m  \u001b[97m{'test': 15, 'train': 327, 'val': 15}\u001b[0m\n"
     ]
    }
   ],
   "source": [
    "ner.dataflow.categories.filter_categories(categories=dd.LayoutType.WORD)\n",
    "\n",
    "merge = dd.MergeDataset(ner)\n",
    "merge.buffer_datasets()\n",
    "merge.split_datasets(ratio=0.1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "ArtifactManifestEntry(path='split.table.json', digest='Y4OPSqZ/Z3PlQYSWBBDbxw==', size=18543, local_path='/home/janis/.local/share/wandb/artifacts/staging/tmpkptwl7lb', skip_cache=False)"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "out = merge.get_ids_by_split()\n",
    "\n",
    "table_rows=[]\n",
    "for split, split_list in out.items():\n",
    "    for ann_id in split_list:\n",
    "        table_rows.append([split,ann_id])\n",
    "table = wandb.Table(columns=[\"split\",\"annotation_id\"], data=table_rows)\n",
    "\n",
    "wandb.init(project=\"FRFPE_layoutlmv1\")\n",
    "\n",
    "artifact = wandb.Artifact(merge.dataset_info.name, type='dataset')\n",
    "artifact.add(table, \"split\")\n",
    "\n",
    "wandb.log_artifact(artifact)\n",
    "wandb.finish()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## LayoutLMv1 training\n",
    "\n",
    "Next we setup training with LayoutLMv1."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "path_config_json = dd.ModelCatalog.get_full_path_configs(\"microsoft/layoutlm-base-uncased/pytorch_model.bin\")\n",
    "path_weights = dd.ModelCatalog.get_full_path_weights(\"microsoft/layoutlm-base-uncased/pytorch_model.bin\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "metric = dd.get_metric(\"f1\")\n",
    "metric.set_categories(sub_category_names={\"word\": [\"token_class\"]})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Remember id to label mapping:\n",
    "\n",
    "``` \n",
    "0: <FundsFirstPage.REPORT_DATE>,\n",
    "1: <FundsFirstPage.REPORT_TYPE>,\n",
    "2: <FundsFirstPage.UMBRELLA>,\n",
    "3: <FundsFirstPage.FUND_NAME>,\n",
    "4: <TokenClasses.OTHER>\n",
    "```\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "dd.train_hf_layoutlm(path_config_json,\n",
    "                     merge,\n",
    "                     path_weights,\n",
    "                     config_overwrite=[\"max_steps=2000\",\n",
    "                                       \"per_device_train_batch_size=8\",\n",
    "                                       \"eval_steps=100\",\n",
    "                                       \"save_steps=400\",\n",
    "                                       \"use_wandb=True\",\n",
    "                                       \"wandb_project=FRFPE_layoutlmv1\",\n",
    "                                      ],\n",
    "                     log_dir=\"/path/to/dir/Experiments/FRFPE/layoutlmv1\",\n",
    "                     dataset_val=merge,\n",
    "                     metric=metric,\n",
    "                     use_token_tag=False,\n",
    "                     pipeline_component_name=\"LMTokenClassifierService\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Further exploration of evaluation\n",
    "\n",
    "### Evaluation with confusion matrix\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\u001b[32m[0712 18:20.04 @eval.py:112]\u001b[0m  \u001b[32mINF\u001b[0m  \u001b[97mBuilding multi threading pipeline component to increase prediction throughput. Using 2 threads\u001b[0m\n",
      "\u001b[32m[0712 18:20.12 @eval.py:226]\u001b[0m  \u001b[32mINF\u001b[0m  \u001b[97mPredicting objects...\u001b[0m\n",
      "100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 15/15 [00:05<00:00,  2.95it/s]\n",
      "\u001b[32m[0712 18:20.17 @eval.py:208]\u001b[0m  \u001b[32mINF\u001b[0m  \u001b[97mStarting evaluation...\u001b[0m\n",
      "\u001b[32m[0712 18:20.17 @accmetric.py:429]\u001b[0m  \u001b[32mINF\u001b[0m  \u001b[97mConfusion matrix: \n",
      " \u001b[36m|    predictions ->  |   1 |   2 |   3 |   4 |   5 |\n",
      "|     ground truth | |     |     |     |     |     |\n",
      "|                  v |     |     |     |     |     |\n",
      "|-------------------:|----:|----:|----:|----:|----:|\n",
      "|                  1 |  41 |   0 |   0 |   0 |   4 |\n",
      "|                  2 |   0 |  19 |   0 |   0 |   8 |\n",
      "|                  3 |   0 |   0 |  20 |   4 |  14 |\n",
      "|                  4 |   0 |   0 |   0 |  25 |   5 |\n",
      "|                  5 |   0 |   0 |   1 |   1 | 657 |\u001b[0m\u001b[0m\n"
     ]
    }
   ],
   "source": [
    "categories = ner.dataflow.categories.get_sub_categories(categories=\"word\",\n",
    "                                                        sub_categories={\"word\": [\"token_class\"]},\n",
    "                                                        keys=False)[\"word\"][\"token_class\"]\n",
    "\n",
    "path_config_json = \"/path/to/dir/Experiments/FRFPE/layoutlmv1/checkpoint-1600/config.json\"\n",
    "path_weights = \"/path/to/dir/Experiments/FRFPE/layoutlmv1/checkpoint-1600/model.safetensors\"\n",
    "\n",
    "layoutlm_classifier = dd.HFLayoutLmTokenClassifier(path_config_json,\n",
    "                                                   path_weights,\n",
    "                                                   categories=categories)\n",
    "\n",
    "tokenizer_fast = LayoutLMTokenizerFast.from_pretrained(\"microsoft/layoutlm-base-uncased\")\n",
    "pipe_component = dd.LMTokenClassifierService(tokenizer_fast,\n",
    "                                             layoutlm_classifier,\n",
    "                                             use_other_as_default_category=True)\n",
    "metric = dd.get_metric(\"confusion\")\n",
    "metric.set_categories(sub_category_names={\"word\": [\"token_class\"]})\n",
    "evaluator = dd.Evaluator(merge, pipe_component, metric)\n",
    "_ = evaluator.run(split=\"val\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "###  Visualizing predictions and ground truth"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "result = evaluator.compare(interactive=True, split=\"val\", show_words=True)\n",
    "sample = next(iter(result))\n",
    "sample.viz()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Evaluation on test set\n",
    "\n",
    "Comparing the evaluation results of eval and test split we see a deterioration of `fund_name`  F1-score (too many erroneous as `umbrella`). All remaining labels are slightly worse."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\u001b[32m[0712 18:24.09 @eval.py:112]\u001b[0m  \u001b[32mINF\u001b[0m  \u001b[97mBuilding multi threading pipeline component to increase prediction throughput. Using 2 threads\u001b[0m\n",
      "\u001b[32m[0712 18:24.17 @eval.py:226]\u001b[0m  \u001b[32mINF\u001b[0m  \u001b[97mPredicting objects...\u001b[0m\n",
      "100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 15/15 [00:05<00:00,  2.92it/s]\n",
      "\u001b[32m[0712 18:24.22 @eval.py:208]\u001b[0m  \u001b[32mINF\u001b[0m  \u001b[97mStarting evaluation...\u001b[0m\n",
      "\u001b[32m[0712 18:24.22 @accmetric.py:429]\u001b[0m  \u001b[32mINF\u001b[0m  \u001b[97mConfusion matrix: \n",
      " \u001b[36m|    predictions ->  |   1 |   2 |   3 |   4 |   5 |\n",
      "|     ground truth | |     |     |     |     |     |\n",
      "|                  v |     |     |     |     |     |\n",
      "|-------------------:|----:|----:|----:|----:|----:|\n",
      "|                  1 |  53 |   0 |   0 |   0 |   2 |\n",
      "|                  2 |   0 |  20 |   0 |   0 |  12 |\n",
      "|                  3 |   0 |   0 |  25 |   6 |  10 |\n",
      "|                  4 |   0 |   0 |   4 | 292 |  33 |\n",
      "|                  5 |   0 |   0 |   4 |   8 | 482 |\u001b[0m\u001b[0m\n"
     ]
    }
   ],
   "source": [
    "categories = ner.dataflow.categories.get_sub_categories(categories=\"word\",\n",
    "                                                        sub_categories={\"word\": [\"token_class\"]},\n",
    "                                                        keys=False)[\"word\"][\"token_class\"]\n",
    "\n",
    "path_config_json = \"/path/to/dir/FRFPE/layoutlmv1/checkpoint-1600/config.json\"\n",
    "path_weights = \"/path/to/dir/FRFPE/layoutlmv1/checkpoint-1600/model.safetensors\"\n",
    "\n",
    "layoutlm_classifier = dd.HFLayoutLmTokenClassifier(path_config_json,\n",
    "                                                   path_weights,\n",
    "                                                   categories=categories)\n",
    "\n",
    "tokenizer_fast = LayoutLMTokenizerFast.from_pretrained(\"microsoft/layoutlm-base-uncased\")\n",
    "pipe_component = dd.LMTokenClassifierService(tokenizer_fast,\n",
    "                                             layoutlm_classifier,\n",
    "                                             use_other_as_default_category=True)\n",
    "\n",
    "\n",
    "evaluator = dd.Evaluator(merge, pipe_component, metric)\n",
    "_ = evaluator.run(split=\"test\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\u001b[32m[0712 18:24.26 @eval.py:112]\u001b[0m  \u001b[32mINF\u001b[0m  \u001b[97mBuilding multi threading pipeline component to increase prediction throughput. Using 2 threads\u001b[0m\n",
      "\u001b[32m[0712 18:24.34 @eval.py:226]\u001b[0m  \u001b[32mINF\u001b[0m  \u001b[97mPredicting objects...\u001b[0m\n",
      "100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 15/15 [00:05<00:00,  2.91it/s]\n",
      "\u001b[32m[0712 18:24.39 @eval.py:208]\u001b[0m  \u001b[32mINF\u001b[0m  \u001b[97mStarting evaluation...\u001b[0m\n",
      "\u001b[32m[0712 18:24.39 @accmetric.py:429]\u001b[0m  \u001b[32mINF\u001b[0m  \u001b[97mConfusion matrix: \n",
      " \u001b[36m|    predictions ->  |   1 |   2 |   3 |   4 |   5 |\n",
      "|     ground truth | |     |     |     |     |     |\n",
      "|                  v |     |     |     |     |     |\n",
      "|-------------------:|----:|----:|----:|----:|----:|\n",
      "|                  1 |  53 |   0 |   0 |   0 |   2 |\n",
      "|                  2 |   0 |  20 |   0 |   0 |  12 |\n",
      "|                  3 |   0 |   0 |  25 |   6 |  10 |\n",
      "|                  4 |   0 |   0 |   4 | 292 |  33 |\n",
      "|                  5 |   0 |   0 |   4 |   8 | 482 |\u001b[0m\u001b[0m\n"
     ]
    }
   ],
   "source": [
    "metric = dd.get_metric(\"confusion\")\n",
    "metric.set_categories(sub_category_names={\"word\": [\"token_class\"]})\n",
    "\n",
    "evaluator = dd.Evaluator(merge, pipe_component, metric)\n",
    "_ = evaluator.run(split=\"test\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "result = evaluator.compare(interactive=True, split=\"test\", show_words=True)\n",
    "sample = next(iter(result))\n",
    "sample.viz()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "wandb.finish()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Changing training parameters and settings\n",
    "\n",
    "Already the first training run delivers satisfactory results. The following parameters could still be changed:\n",
    "\n",
    "**Training specific** (Check the Transformers doc to know more about these parameters): \n",
    "`per_device_train_batch_size`\n",
    "`max_steps` \n",
    "\n",
    "**General**: \n",
    "`sliding_window_stride` - LayoutLMv1 accepts a maximum of 512 tokens. For samples containing more tokens a sliding window can be used: Assume a training sample contains 600 tokens. Without sliding window the last 88 tokens are not considered in the training. If a sliding window of 16 is set, 88 % 16 +1= 9 samples are generated. \n",
    "\n",
    "Caution: \n",
    "- The number `per_device_train_batch_size` can thereby turn out to increase very fast and lead to Cuda OOM.\n",
    "- If a sample occurs that generates multiple training points due to the sliding windows setting, all other samples in the batch will be ignored and only the one data point with all its windows will be considered in this step. If you train with a dataset where the number of tokens is high for many samples, you should choose `per_device_train_batch_size` to be rather small to ensure that you train with the whole dataset. \n",
    "\n",
    "To avoid the situation that due to the `sliding_window_stride` the batch size becomes arbitrarily large one can select the additional `max_batch_size`:\n",
    "Setting this parameter causes a selection of max_batch_size samples to be randomly sent to the GPU from the generated sliding window samples.\n",
    "\n",
    "E.g. another training configuration might look like this:\n",
    "\n",
    "\n",
    "```\n",
    "dd.train_hf_layoutlm(path_config_json,\n",
    "                     merge,\n",
    "                     path_weights,\n",
    "                     config_overwrite=[\"max_steps=4000\",\n",
    "                                       \"per_device_train_batch_size=8\",\n",
    "                                       \"eval_steps=100\",\n",
    "                                       \"save_steps=400\",\n",
    "                                       \"sliding_window_stride=16\",\n",
    "                                       \"max_batch_size=8\",\n",
    "                                       \"use_wandb=True\",\n",
    "                                       \"wandb_project=funds_layoutlmv1\"],\n",
    "                         log_dir=\"/path/to/dir/Experiments/ner_first_page_v1_2\",\n",
    "                         dataset_val=merge,\n",
    "                         metric=metric,\n",
    "                         use_token_tag=False,\n",
    "                         pipeline_component_name=\"LMTokenClassifierService\")\n",
    "``` \n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.19"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}