🐛 Destination Databricks: Fix the S3 table path to use schema from the configured schema instead of the default.

[tanawatpan]

What

The Airbyte Databricks Connector for S3 currently constructs table paths using a default schema value when multiple schemas contain tables with the same name, leading to potential data overlap or collisions in S3.

How

Instead of using a default schema value, the connector will use the schema in the configured Destination Namespace. This change allows for distinct and schema-specific paths for each table.

🚨 User Impact 🚨

the schema of the S3 Destination location will be derived from the Destination Namespace (if specified).

Test

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
airbyte-docs	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Feb 22, 2024 4:32am

[github-actions]

Before Merging a Connector Pull Request

Wow! What a great pull request you have here! 🎉

To merge this PR, ensure the following has been done/considered for each connector added or updated:

• PR name follows PR naming conventions
• Breaking changes are considered. If a Breaking Change is being introduced, ensure an Airbyte engineer has created a Breaking Change Plan.
• Connector version has been incremented in the Dockerfile and metadata.yaml according to our Semantic Versioning for Connectors guidelines
• You've updated the connector's metadata.yaml file any other relevant changes, including a breakingChanges entry for major version bumps. See metadata.yaml docs
• Secrets in the connector's spec are annotated with airbyte_secret
• All documentation files are up to date. (README.md, bootstrap.md, docs.md, etc...)
• Changelog updated in docs/integrations/<source or destination>/<name>.md with an entry for the new version. See changelog example
• Migration guide updated in docs/integrations/<source or destination>/<name>-migrations.md with an entry for the new version, if the version is a breaking change. See migration guide example
• If set, you've ensured the icon is present in the platform-internal repo. (Docs)

If the checklist is complete, but the CI check is failing,

1. Check for hidden checklists in your PR description

2. Toggle the github label checklist-action-run on/off to re-run the checklist CI.

[tanawatpan]

@gisripa What do you think?

[gisripa]

@tanawatpan We had to the pin the version

airbyte/airbyte-integrations/connectors/destination-databricks/metadata.yaml

Line 14 in 0adf266

dockerImageTag: 1.1.0 # pinning due to CDK incompatibility, see https://github.com/airbytehq/alpha-beta-issues/issues/2596

due to a library incompatibility and didn't have bandwidth to upgrade this connector. How do you feel about running a forked build for now (if it builds), we will be revamping this connector very soon.

[tanawatpan]

@gisripa Sure, I'll run a forked build for now and watch for the updated connector.

[anchitagarwal]

Hi @gisripa how soon is the revamp for databricks connector? Is there a roadmap or issue I can track?