feat(ingestion) Adding vertexAI ingestion source (v1 - model group and model)

[ryota-cloud]

Checklist

• The PR conforms to DataHub's Contributing Guideline (particularly Commit Message Format)
• Links to related issues (if applicable)
• Tests for the changes have been added/updated (if applicable)
• Docs related to the changes have been added/updated (if applicable). If a new feature has been added a Usage Guide has been added for the same.
• For any breaking change/potential downtime/deprecation/big changes an entry has been made in Updating DataHub

[codecov]

Codecov Report

Attention: Patch coverage is 84.81481% with 41 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
...ingestion/src/datahub/ingestion/source/vertexai.py	82.70%	41 Missing ⚠️

📢 Thoughts on this report? Let us know!

🚀 New features to boost your workflow:

• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[hsheth2]

Every entity needs container aspects

[hsheth2]

do we actually need to create a credentials file?

[ryota-cloud]

good point, not needed while GCPcredential need it, deleted

[hsheth2]

a bit confused - it looks like we're still writing the credentials to a file

[hsheth2]

this is not a blocker - but we should not be writing credentials to disk if we can avoid it

[ryota-cloud]

this file is actually used to create credentials object using service_account.Credentials util, which feed into aitplatform.init()

        credentials = (
            service_account.Credentials.from_service_account_file(
                self.config._credentials_path
            )
            if self.config.credential
            else None
        )

[hsheth2]

right - but we're getting the config as json or as a true file path in the original GCPCredential

then we take that credential and write it to a new file, storing the file path in self.config._credentials_path. and then we load self.config._credentials_path again.

that flow is pretty strange

[ryota-cloud]

not so unusual to see this pattern (cred part only is written to temp file and loaded), but I understand your point of avoiding yet another file write, how about changing it to something like below,

     credentials = (
            service_account.Credentials.from_service_account_info(
                self.config.get_credentials(). --> passing dict
            )

[hsheth2]

a bit confused - it looks like we're still writing the credentials to a file

[hsheth2]

this is not a blocker - but we should not be writing credentials to disk if we can avoid it

[hsheth2]

why are we using a platformResource here? or is this just a dummy value?

[ryota-cloud]

this is placeholder, since actor info is missing in Vertex Training job itself.
MLflow connector is using "urn:li:corpuser:datahub. I can change to it for now.

[hsheth2]

Approving to unblock you, but noting that we agreed a number of things will need to be done in follow ups

• Fixing the way the unit tests are set up
• Moving to use MCPW.construct_many
• Changing the audit stamp urns to not be platformResources
• Changing the auth flow to avoid writing credentials to a file
• (couple others that I might be forgetting - basically all pending "conversation" in the github PR review)

[ryota-cloud]

thanks for review! @hsheth2
couple of follows-up

• Moving to use MCPW.construct_many
  already changed in this PR (last commit)

• Changing the audit stamp urns to not be platformResources
  changed to "urn:li:corpuser:datahub" using MLFlow example

• Changing the auth flow to avoid writing credentials to a file
  feat(ingestion) Adding vertexAI ingestion source (v1 - model group and model)  #12632 (comment)