docs(ingest): custom transformer remote executor (#12864)

Author: anshbansal

metadata-ingestion/docs/transformer/dataset_transformer.md

@@ -1632,4 +1632,24 @@ After running `datahub ingest -c <path_to_recipe>`, our MCEs will now have the f
 ],
 ```
 
+### Using this in the remote executor (DataHub Cloud only)
+
+Build the image with your transformer
+```
+docker build -t acryldata:customtransform1 -f metadata-ingestion/examples/transforms/example.Dockerfile metadata-ingestion/examples/transforms
+```
+
+Test it works
+```
+docker run -it --rm acryldata:customtransform1 bash
+```
+
+Inside the docker container
+```
+source venv/bin/activate
+datahub ingest -c ./custom_transformer/recipe.dhub.yaml
+```
+
+If you use this image for remote executor then you can set `file:///datahub-executor/custom_transformer` as an extra pip dependency to install the transformer in your ingestion.
+
 All the files for this tutorial may be found [here](../../examples/transforms/).

metadata-ingestion/examples/transforms/example.Dockerfile

@@ -0,0 +1,18 @@
+FROM 795586375822.dkr.ecr.us-west-2.amazonaws.com/datahub-executor:v0.3.8.2-acryl
+
+COPY setup.py custom_transformer/setup.py
+COPY custom_transform_example.py custom_transformer/custom_transform_example.py
+COPY owners.json custom_transformer/owners.json
+COPY recipe.dhub.yaml custom_transformer/recipe.dhub.yaml
+
+USER root
+
+RUN chown -R root:root custom_transformer/* && \
+    chmod -R u+rw custom_transformer && \
+    python3 -m venv venv && \
+    . venv/bin/activate && \
+    pip install uv && \
+    uv pip install 'acryl-datahub' && \
+    uv pip install -e 'file:///datahub-executor/custom_transformer'
+
+USER datahub

metadata-ingestion/examples/transforms/owners.json

@@ -0,0 +1,6 @@
+[
+    "urn:li:corpuser:athos",
+    "urn:li:corpuser:porthos",
+    "urn:li:corpuser:aramis",
+    "urn:li:corpGroup:the_three_musketeers"
+  ]  

metadata-ingestion/examples/transforms/recipe.dhub.yaml

@@ -0,0 +1,11 @@
+source:
+  type: demo-data
+  config: {}
+
+transformers:
+  - type: "custom_transform_example_alias"
+    config:
+       owners_json: /datahub-executor/custom_transformer/owners.json
+
+sink:
+  type: "console"

metadata-ingestion/examples/transforms/setup.py

@@ -1,7 +1,12 @@
-from setuptools import find_packages, setup
+from setuptools import setup
 
 setup(
-    name="custom_transform_example",
+    name="custom_transformer",
     version="1.0",
-    packages=find_packages(),
+    py_modules=["custom_transform_example"],
+    entry_points={
+        "datahub.ingestion.transformer.plugins": [
+            "custom_transform_example_alias = custom_transform_example:AddCustomOwnership",
+        ],
+    },
 )