Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

WIP: Load CDK BulkLoad interface + MSSQL V2 Usage #55671

Conversation

johnny-schmidt
Copy link
Contributor

What

Migrates MSSQL to the new LoadPipeline CDK interface.

Practical implication: you can now do the upload-to-azure part in parallel while still limiting the load-into-mssql step to 1 thread per stream.

  • creates a BulkLoad LoadStrategy to support it
  • some other modifications to make it actually work
  • BulkLoad ITs are all passing
  • injection is hacked in -- some Bulk Load stuff will need to be behind a custom Condition that checks the config (so StandardInsert is probably broken until that)
  • I haven't done any performance testing/tuning yet

Copy link

vercel bot commented Mar 10, 2025

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name Status Preview Comments Updated (UTC)
airbyte-docs ✅ Ready (Inspect) Visit Preview 💬 Add feedback Mar 10, 2025 11:32pm

@octavia-squidington-iii octavia-squidington-iii added area/connectors Connector related issues CDK Connector Development Kit connectors/destination/mssql-v2 labels Mar 10, 2025
@johnny-schmidt johnny-schmidt force-pushed the jschmidt/s3v2/s3-uses-new-interface branch 3 times, most recently from d622f38 to 8a2188f Compare March 10, 2025 16:03
@johnny-schmidt johnny-schmidt force-pushed the jschmidt/mssql-v2/bulk-load-uses-new-interface branch from 7da16c8 to 1414b6a Compare March 10, 2025 16:07
@johnny-schmidt johnny-schmidt force-pushed the jschmidt/s3v2/s3-uses-new-interface branch from 8a2188f to 9b08cdb Compare March 10, 2025 16:20
@johnny-schmidt johnny-schmidt force-pushed the jschmidt/mssql-v2/bulk-load-uses-new-interface branch from 1414b6a to c36ec24 Compare March 10, 2025 16:27
@johnny-schmidt johnny-schmidt force-pushed the jschmidt/mssql-v2/bulk-load-uses-new-interface branch from c36ec24 to 4e1205b Compare March 10, 2025 20:49
@johnny-schmidt johnny-schmidt force-pushed the jschmidt/s3v2/s3-uses-new-interface branch from 9b08cdb to ea12643 Compare March 10, 2025 21:07
@johnny-schmidt johnny-schmidt force-pushed the jschmidt/mssql-v2/bulk-load-uses-new-interface branch from 4e1205b to 3033c27 Compare March 10, 2025 21:49
@johnny-schmidt johnny-schmidt force-pushed the jschmidt/mssql-v2/bulk-load-uses-new-interface branch from 6215ea1 to 452e0fd Compare March 10, 2025 22:25
@johnny-schmidt johnny-schmidt force-pushed the jschmidt/s3v2/s3-uses-new-interface branch from ea12643 to 1d98570 Compare March 10, 2025 22:51
@johnny-schmidt johnny-schmidt force-pushed the jschmidt/mssql-v2/bulk-load-uses-new-interface branch from 452e0fd to b750061 Compare March 10, 2025 22:52
@johnny-schmidt johnny-schmidt force-pushed the jschmidt/mssql-v2/bulk-load-uses-new-interface branch from 35a7e69 to 6e292e9 Compare March 10, 2025 23:21
@johnny-schmidt johnny-schmidt force-pushed the jschmidt/s3v2/s3-uses-new-interface branch 2 times, most recently from 1f77bde to 1b72de1 Compare March 12, 2025 18:13
@johnny-schmidt johnny-schmidt force-pushed the jschmidt/s3v2/s3-uses-new-interface branch 6 times, most recently from f1a6b7b to 53cbe5a Compare March 17, 2025 22:34
@johnny-schmidt
Copy link
Contributor Author

Closing b/c the rebase after beta was so gnarly I rewrote/cherry-picked

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/connectors Connector related issues CDK Connector Development Kit connectors/destination/mssql-v2
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants