Skip to content

Commit 235a041

Browse files
ryota-cloudryota-cloud
committed
Adding sources and documents
1 parent 89989b0 commit 235a041

File tree

4 files changed

+127
-0
lines changed

4 files changed

+127
-0
lines changed

‎datahub-web-react/src/app/ingest/source/builder/sources.json

+7
Original file line numberDiff line numberDiff line change
@@ -333,5 +333,12 @@
333333
"description": "Import Nodes and Relationships from Neo4j.",
334334
"docsUrl": "https://datahubproject.io/docs/generated/ingestion/sources/neo4j/",
335335
"recipe": "source:\n type: 'neo4j'\n config:\n uri: 'neo4j+ssc://host:7687'\n username: 'neo4j'\n password: 'password'\n env: 'PROD'\n\nsink:\n type: \"datahub-rest\"\n config:\n server: 'http://localhost:8080'"
336+
},
337+
{
338+
"urn": "urn:li:dataPlatform:vertexai",
339+
"name": "vertexai",
340+
"displayName": "VertexAI",
341+
"docsUrl": "https://datahubproject.io/docs/generated/ingestion/sources/vertexai/",
342+
"recipe": "source:\n type: vertexai\n config:\n tracking_uri: tracking_uri"
336343
}
337344
]

‎metadata-ingestion/docs/sources/vertexai/README.md

+1
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
Ingesting metadata from VertexAI requires using the **vertexai** module.

‎metadata-ingestion/docs/sources/vertexai/vertexai_pre.md

+109
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,109 @@
1+
### Prerequisites
2+
To understand how BigQuery ingestion needs to be set up, first familiarize yourself with the concepts in the diagram below:
3+
<p align="center">
4+
<img width="70%" src="https://github.com/datahub-project/static-assets/raw/main/imgs/integrations/bigquery/source-bigquery-setup.png"/>
5+
</p>
6+
7+
There are two important concepts to understand and identify:
8+
- *Extractor Project*: This is the project associated with a service-account, whose credentials you will be configuring in the connector. The connector uses this service-account to run jobs (including queries) within the project.
9+
- *Bigquery Projects* are the projects from which table metadata, lineage, usage, and profiling data need to be collected. By default, the extractor project is included in the list of projects that DataHub collects metadata from, but you can control that by passing in a specific list of project ids that you want to collect metadata from. Read the configuration section below to understand how to limit the list of projects that DataHub extracts metadata from.
10+
11+
#### Create a datahub profile in GCP
12+
1. Create a custom role for datahub as per [BigQuery docs](https://cloud.google.com/iam/docs/creating-custom-roles#creating_a_custom_role).
13+
2. Follow the sections below to grant permissions to this role on this project and other projects.
14+
15+
##### Basic Requirements (needed for metadata ingestion)
16+
1. Identify your Extractor Project where the service account will run queries to extract metadata.
17+
18+
| permission                       | Description                                                                                                                         | Capability                                                               |
19+
|----------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|---------------------------------------------------------------------------------------------------------------|
20+
| `bigquery.jobs.create`           | Run jobs (e.g. queries) within the project. *This only needs for the extractor project where the service account belongs*           |                                                                                                               |
21+
| `bigquery.jobs.list`             | Manage the queries that the service account has sent. *This only needs for the extractor project where the service account belongs* |                                                                                                               |
22+
| `bigquery.readsessions.create`   | Create a session for streaming large results. *This only needs for the extractor project where the service account belongs*         |                                                                                                               |
23+
| `bigquery.readsessions.getData` | Get data from the read session. *This only needs for the extractor project where the service account belongs*                       |
24+
2. Grant the following permissions to the Service Account on every project where you would like to extract metadata from
25+
26+
:::info
27+
28+
If you have multiple projects in your BigQuery setup, the role should be granted these permissions in each of the projects.
29+
30+
:::
31+
| Permission | Description | Capability | Default GCP Role Which Contains This Permission |
32+
|----------------------------------|-----------------------------------------------------------------------------------------------------------------|-------------------------------------|---------------------------------------------------------------------------|
33+
| `bigquery.datasets.get` | Retrieve metadata about a dataset. | Table Metadata Extraction | [roles/bigquery.metadataViewer](https://cloud.google.com/bigquery/docs/access-control#bigquery.metadataViewer) |
34+
| `bigquery.datasets.getIamPolicy` | Read a dataset's IAM permissions. | Table Metadata Extraction | [roles/bigquery.metadataViewer](https://cloud.google.com/bigquery/docs/access-control#bigquery.metadataViewer) |
35+
| `bigquery.tables.list` | List BigQuery tables. | Table Metadata Extraction | [roles/bigquery.metadataViewer](https://cloud.google.com/bigquery/docs/access-control#bigquery.metadataViewer) |
36+
| `bigquery.tables.get` | Retrieve metadata for a table. | Table Metadata Extraction | [roles/bigquery.metadataViewer](https://cloud.google.com/bigquery/docs/access-control#bigquery.metadataViewer) |
37+
| `bigquery.routines.get` | Get Routines. Needs to retrieve metadata for a table from system table. | Table Metadata Extraction | [roles/bigquery.metadataViewer](https://cloud.google.com/bigquery/docs/access-control#bigquery.metadataViewer) |
38+
| `bigquery.routines.list` | List Routines. Needs to retrieve metadata for a table from system table. | Table Metadata Extraction | [roles/bigquery.metadataViewer](https://cloud.google.com/bigquery/docs/access-control#bigquery.metadataViewer) |
39+
| `resourcemanager.projects.get` | Retrieve project names and metadata. | Table Metadata Extraction | [roles/bigquery.metadataViewer](https://cloud.google.com/bigquery/docs/access-control#bigquery.metadataViewer) |
40+
| `bigquery.jobs.listAll` | List all jobs (queries) submitted by any user. Needs for Lineage extraction. | Lineage Extraction/Usage Extraction | [roles/bigquery.resourceViewer](https://cloud.google.com/bigquery/docs/access-control#bigquery.resourceViewer) |
41+
| `logging.logEntries.list` | Fetch log entries for lineage/usage data. Not required if `use_exported_bigquery_audit_metadata` is enabled. | Lineage Extraction/Usage Extraction | [roles/logging.privateLogViewer](https://cloud.google.com/logging/docs/access-control#logging.privateLogViewer) |
42+
| `logging.privateLogEntries.list` | Fetch log entries for lineage/usage data. Not required if `use_exported_bigquery_audit_metadata` is enabled. | Lineage Extraction/Usage Extraction | [roles/logging.privateLogViewer](https://cloud.google.com/logging/docs/access-control#logging.privateLogViewer) |
43+
| `bigquery.tables.getData` | Access table data to extract storage size, last updated at, data profiles etc. | Profiling | |
44+
| `datacatalog.policyTags.get` | *Optional* Get policy tags for columns with associated policy tags. This permission is required only if `extract_policy_tags_from_catalog` is enabled. | Policy Tag Extraction | [roles/datacatalog.viewer](https://cloud.google.com/data-catalog/docs/access-control#permissions-and-roles) |
45+
46+
47+
#### Create a service account in the Extractor Project
48+
49+
1. Setup a ServiceAccount as per [BigQuery docs](https://cloud.google.com/iam/docs/creating-managing-service-accounts#iam-service-accounts-create-console)
50+
and assign the previously created role to this service account.
51+
2. Download a service account JSON keyfile.
52+
Example credential file:
53+
54+
```json
55+
{
56+
"type": "service_account",
57+
"project_id": "project-id-1234567",
58+
"private_key_id": "d0121d0000882411234e11166c6aaa23ed5d74e0",
59+
"private_key": "-----BEGIN PRIVATE KEY-----\nMIIyourkey\n-----END PRIVATE KEY-----",
60+
"client_email": "[email protected]",
61+
"client_id": "113545814931671546333",
62+
"auth_uri": "https://accounts.google.com/o/oauth2/auth",
63+
"token_uri": "https://oauth2.googleapis.com/token",
64+
"auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",
65+
"client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/x509/test%suppproject-id-1234567.iam.gserviceaccount.com"
66+
}
67+
```
68+
69+
3. To provide credentials to the source, you can either:
70+
71+
Set an environment variable:
72+
73+
```sh
74+
$ export GOOGLE_APPLICATION_CREDENTIALS="/path/to/keyfile.json"
75+
```
76+
77+
_or_
78+
79+
Set credential config in your source based on the credential json file. For example:
80+
81+
```yml
82+
credential:
83+
project_id: project-id-1234567
84+
private_key_id: "d0121d0000882411234e11166c6aaa23ed5d74e0"
85+
private_key: "-----BEGIN PRIVATE KEY-----\nMIIyourkey\n-----END PRIVATE KEY-----\n"
86+
client_email: "[email protected]"
87+
client_id: "123456678890"
88+
```
89+
90+
##### Profiling Requirements
91+
92+
To profile BigQuery external tables backed by Google Drive document, you need to grant document's "Viewer" access to service account's email address (`client_email` in credentials json file). To find the Google Drive document linked to BigQuery table, open the BigQuery console, locate the needed table, select "Details" from the drop-down menu in the top-right corner and refer "Source" field . To share access of Google Drive document, open the document, click "Share" in the top-right corner, add the service account's email address that needs "Viewer" access. ![Google Drive Sharing Dialog](https://github.com/datahub-project/static-assets/raw/main/imgs/integrations/bigquery/google_drive_share.png)
93+
94+
### Lineage Computation Details
95+
96+
When `use_exported_bigquery_audit_metadata` is set to `true`, lineage information will be computed using exported bigquery logs. On how to setup exported bigquery audit logs, refer to the following [docs](https://cloud.google.com/bigquery/docs/reference/auditlogs#defining_a_bigquery_log_sink_using_gcloud) on BigQuery audit logs. Note that only protoPayloads with "type.googleapis.com/google.cloud.audit.BigQueryAuditMetadata" are supported by the current ingestion version. The `bigquery_audit_metadata_datasets` parameter will be used only if `use_exported_bigquery_audit_metadat` is set to `true`.
97+
98+
Note: the `bigquery_audit_metadata_datasets` parameter receives a list of datasets, in the format $PROJECT.$DATASET. This way queries from a multiple number of projects can be used to compute lineage information.
99+
100+
Note: Since bigquery source also supports dataset level lineage, the auth client will require additional permissions to be able to access the google audit logs. Refer the permissions section in bigquery-usage section below which also accesses the audit logs.
101+
102+
### Profiling Details
103+
104+
For performance reasons, we only profile the latest partition for partitioned tables and the latest shard for sharded tables.
105+
You can set partition explicitly with `partition.partition_datetime` property if you want, though note that partition config will be applied to all partitioned tables.
106+
107+
### Caveats
108+
109+
- For materialized views, lineage is dependent on logs being retained. If your GCP logging is retained for 30 days (default) and 30 days have passed since the creation of the materialized view we won't be able to get lineage for them.

‎metadata-ingestion/docs/sources/vertexai/vertexai_recipe.yml

+10
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,10 @@
1+
source:
2+
type: vertexai
3+
config:
4+
project_id: "acryl-poc"
5+
region: "us-west2"
6+
7+
sink:
8+
type: "datahub-rest"
9+
config:
10+
server: "http://localhost:8080"

0 commit comments

Comments
 (0)