feat(ingest/looker): Added delta-lake to platform naming convention involving 2 parts

[k7ragav]

Checklist

Added delta-lake to platforms that follow the naming convention of 'db.table_name`.

• The PR conforms to DataHub's Contributing Guideline (particularly Commit Message Format)
• Links to related issues (if applicable)
• Tests for the changes have been added/updated (if applicable)
• Docs related to the changes have been added/updated (if applicable). If a new feature has been added a Usage Guide has been added for the same.
• For any breaking change/potential downtime/deprecation/big changes an entry has been made in Updating DataHub

[hsheth2]

Based on the logic here

datahub/metadata-ingestion/src/datahub/ingestion/source/delta_lake/source.py

Lines 204 to 212 in b74ba11

	logger.debug(f"Ingesting table {table_name} from location {path}")
	if self.source_config.relative_path is None:
	browse_path: str = (
	strip_s3_prefix(path) if self.source_config.is_s3 else path.strip("/")
	)
	else:
	browse_path = path.split(self.source_config.base_path)[1].strip("/")
	data_platform_urn = make_data_platform_urn(self.source_config.platform)

- it does not seem like delta-lake is always going to have a two-part identifier

[k7ragav]

I thought they would also have database.tablename identifier in Looker while using the sql_table_name. And will not require the DEFAULT_DATABASE that is mentioned in the Looker connection
I saw some docs here as well https://docs.delta.io/latest/delta-batch.html

But let me know what would be the best way forward. We have database.tablename in sql_table_name and would like that to be the identifier as well for upstream lineage

[hsheth2]

@k7ragav looking at our golden files, delta-lake seems to have / in the dataset urns and can have an arbitrary number of parts

datahub/metadata-ingestion/tests/integration/delta_lake/delta_lake_minio_mces_golden.json

Line 6 in 9fb2df1

"aspects": [

What do your delta-lake urns look like? Are you using our delta-lake ingestion source, or something else?

Overall - I don't think this change does what we want. That said, my understanding of how delta-lake works is somewhat weak.

[k7ragav]

@hsheth2 We do not use the delta-lake ingestion from DataHub. We ingest it using the emitters. And we do not have the file paths as it is shown in the golden dataset. We have it like this urn:li:dataset:(urn:li:dataPlatform:delta-lake,databasename.tablename,DEV
I think with the proposed change, it will only matter based on what is provided in the sql_table_name right? Or did I misunderstand it. Because that is why I added it to _platform_names_have_2_parts cause in sql _table_name , it works when it is database.tablename for Delta Lake queries in Looker
Let me know what you think.

[k7ragav]

@hsheth2 Do you have some suggestions on how to move forward based on the explanation below?

@hsheth2 We do not use the delta-lake ingestion from DataHub. We ingest it using the emitters. And we do not have the file paths as it is shown in the golden dataset. We have it like this urn:li:dataset:(urn:li:dataPlatform:delta-lake,databasename.tablename,DEV I think with the proposed change, it will only matter based on what is provided in the sql_table_name right? Or did I misunderstand it. Because that is why I added it to _platform_names_have_2_parts cause in sql _table_name , it works when it is database.tablename for Delta Lake queries in Looker Let me know what you think.

[treff7es]

I briefly checked how our different sources work right now.

1. delta-lake: This source ingests delta lake files from cloud storage right now, and here the platform is delta-lake. In the above case, you have delta-lake tables registered in some metastore, and you use some query execution engine like Databricks/Trino/Presto/Hive to query the delta-lake tables from Looker. Because of this, I think the preferable platform should be hive/presto/trino/databricks and not the underlying file format. These tables could also be iceberg tables or pure parquet files, if I get it right. Am I right?
2. Then, the urn creation should be aligned with what the source-specific ingestions generate. This is mostly catalog_name.schema.table.

@k7ragav, what do you think about this?

[k7ragav]

@treff7es

1. Yes. You're right, we do not store in the cloud and store in a metastore. Does this platform 'delta-lake' is only when the location is in a cloud? (like s3)? If I am understanding correclty, if that is the case, then we cannot use it the way I described it above?
2. So if we use a catalog (for example, let's say hive), so that the URN will contain hive.database.tablename instead of database.tablename?

[treff7es]

@treff7es

1. Yes. You're right, we do not store in the cloud and store in a metastore. Does this platform 'delta-lake' is only when the location is in a cloud? (like s3)? If I am understanding correclty, if that is the case, then we cannot use it the way I described it above?
2. So if we use a catalog (for example, let's say hive), so that the URN will contain hive.database.tablename instead of database.tablename?

Here is an example of how this looks like in the Hive tests ->

datahub/metadata-ingestion/tests/integration/hive/hive_mces_all_db_golden.json

Line 109 in 96758e2

"urn": "urn:li:dataset:(urn:li:dataPlatform:hive,db1._test_table_underscore,PROD)",

urn:li:dataset:(urn:li:dataPlatform:hive,db1._test_table_underscore,PROD)
In the above example, platform is: hive db name:: db1 and table name _test_table_underscore
In your case, this would look like:
urn:li:dataset:(urn:li:dataPlatform:hive,database.tablename,PROD)