feat(source-stripe): migrate to low-code cdk #53687

lazebnyi · 2025-02-14T03:03:19Z

What

To migrate source-stripe connector relying on custom python implementation to manifest based using low code cdk and latest CDK features.

Closes airbytehq/airbyte-internal-issues#11544
Closes airbytehq/airbyte-internal-issues#11683
Closes airbytehq/airbyte-internal-issues#11752

How

The connector supports 7 groups of streams, all migrated to a low-code architecture:

Base Incremental Streams
- Standard incremental streams that track state and support pagination.
State Condition Streams
- These streams adapt their behavior based on whether a state is set:
- No State: Runs the base stream as a full sync.
- With State: Uses an event stream to fetch only relevant events (event types provided via parameters).
Strict State Condition Streams
- Similar to state condition streams but with stricter handling:
- No State: Runs the base stream. The endpoints strictly enforce allowed query parameters and return an error for unexpected ones.
- With State: Uses an event stream to fetch only relevant events (event types provided via parameters).
Base Incremental Partition Streams
- Incremental streams that support partitioning, designed to handle large datasets more efficiently.
- These streams rely on state condition streams as parent streams.
Partitioned State Condition Streams
- Streams with state-aware partitioning behavior:
- No State: Runs a partition stream associated with the parent state condition stream.
- With State: Uses the event stream to fetch partitioned event data (event types provided via parameters).
Lazy Read State Condition Streams
- Streams that support lazy loading to optimize performance:
- No State: Fetches the first N records from the parent, then switches to normal pagination if additional data is needed.
- With State: Uses the event stream to fetch only relevant events (event types provided via parameters).
Base Incremental Partition Streams with Lazy Read Logic
- Combines partitioning and lazy reading for improved performance in large datasets.
- These streams inherit state condition streams as parent streams and leverage lazy read logic.

Implementation details:

Streams relying on state-driven behavior are implemented using StateDelegatingStream, ensuring state management and event-based retrieval.
Streams utilizing lazy read logic are powered by LazySimpleRetriever, enabling efficient data fetching when dealing with large payloads.
Parent cursor value injection is applied to streams that lack their own cursors in responses — the parent’s cursor value is injected to maintain incremental sync integrity.

User Impact

No

Can this PR be safely reverted and rolled back?

YES 💚
NO ❌

vercel · 2025-02-14T03:03:26Z

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
airbyte-docs	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Mar 14, 2025 6:24pm

lazebnyi · 2025-02-14T03:08:10Z

/format-fix

Format-fix job started... Check job output.

✅ Changes applied successfully. (1e0c4a8)

…b.com:airbytehq/airbyte into lazebnyi/source-stripe-move-basic-to-low-code

lazebnyi · 2025-03-13T22:45:08Z

/format-fix

Format-fix job started... Check job output.

✅ Changes applied successfully. (d1d69b8)

…om:airbytehq/airbyte into lazebnyi/source-stripe-migrate-to-low-code

lazebnyi · 2025-03-14T12:05:57Z

/format-fix

Format-fix job started... Check job output.

✅ Changes applied successfully. (3f56559)

…om:airbytehq/airbyte into lazebnyi/source-stripe-migrate-to-low-code

lazebnyi · 2025-03-14T14:27:52Z

/format-fix

Format-fix job started... Check job output.

✅ Changes applied successfully. (2e0e5da)

lazebnyi · 2025-03-14T15:49:15Z

Regression tests are looking good. Only auto-generated fields are changed - https://github.com/airbytehq/airbyte/actions/runs/13858784655/job/38781757496

maxi297

LGTM! Should we do another round of regression test for incremental syncs? What is the rollout plan?

airbyte-integrations/connectors/source-stripe/pyproject.toml

maxi297 · 2025-03-14T17:03:19Z

...-integrations/connectors/source-stripe/unit_tests/integration/test_external_account_cards.py

@@ -211,7 +212,7 @@ def test_given_no_state_when_read_then_use_external_accounts_endpoint(self, http
        output = self._read(_config().with_start_date(_A_START_DATE), _NO_STATE)
        most_recent_state = output.most_recent_state
        assert most_recent_state.stream_descriptor == StreamDescriptor(name=_STREAM_NAME)
-        assert most_recent_state.stream_state == AirbyteStateBlob(updated=int(_NOW.timestamp()))
+        assert int(most_recent_state.stream_state.updated) == int(_NOW.timestamp())


Does that means that the format of the state has changed from int to str? Does that means that we can't revert this change? We should at least make sure to be explicit on Can this PR be safely reverted and rolled back? in the PR description

Does that means that the format of the state has changed from int to str?

Yes, the low-code CDK stores states as a string.

Does that means that we can't revert this change?

I don't think so, but I need to double-check to be 100% sure.

natikgadzhi

Since you're 99% there, why not switch to manifest-only completely, leaving manifest.yaml at top level, and keeping unit tests?

octavia-squidington-iii added area/connectors Connector related issues connectors/source/stripe labels Feb 14, 2025

vercel bot deployed to Preview February 14, 2025 03:08 View deployment

vercel bot deployed to Preview February 14, 2025 03:16 View deployment

lazebnyi and others added 24 commits February 19, 2025 16:06

Moved created streams to low code

818c7e3

Add YamlDeclarativeSource

c11edc8

Add YamlDeclarativeSource

04800e1

Add migrated streams to source

4fcb77a

Add basic streams

e3850d4

Add changelog

c3b0303

Merge branch 'master' into lazebnyi/source-stripe-move-basic-to-low-code

24a8949

chore: auto-fix lint and format issues

ca7bded

Add TODO

9c4c4a6

Merge branch 'lazebnyi/source-stripe-move-basic-to-low-code' of githu…

0aeada0

…b.com:airbytehq/airbyte into lazebnyi/source-stripe-move-basic-to-low-code

Bump CDK version to dev

3d50f65

Metadata tag updated

e618182

Fix unittests

e07384c

chore: auto-fix lint and format issues

16f9a9d

Update delivery_estimate field type in shipping_rates

fd80299

Merge branch 'lazebnyi/source-stripe-move-basic-to-low-code' of githu…

5f654cd

…b.com:airbytehq/airbyte into lazebnyi/source-stripe-move-basic-to-low-code

chore: auto-fix lint and format issues

a22bcb8

Update to last version CDK

8fff856

Merge branch 'lazebnyi/source-stripe-move-basic-to-low-code' of githu…

5c88cda

…b.com:airbytehq/airbyte into lazebnyi/source-stripe-move-basic-to-low-code

Rollback configured catalog

27bd26f

Fix typo

95aeff7

Fix expected record for shipping_rates

27f0bcc

Update abnormal state

c4b0bbd

Apply default schema normalization

8afdde1

lazebnyi added 2 commits March 13, 2025 23:31

Update CDK dependency

39b47f9

Merge master to branch

8528327

vercel bot deployed to Preview March 13, 2025 22:42 View deployment

Merge branch 'master' into lazebnyi/source-stripe-migrate-to-low-code

ce5687c

chore: auto-fix lint and format issues

d1d69b8

vercel bot deployed to Preview March 13, 2025 22:55 View deployment

lazebnyi added 2 commits March 14, 2025 00:56

Add coverage ignore file

6ce7fe1

Merge branch 'lazebnyi/source-stripe-migrate-to-low-code' of github.c…

fdab28a

…om:airbytehq/airbyte into lazebnyi/source-stripe-migrate-to-low-code

vercel bot deployed to Preview March 14, 2025 00:06 View deployment

Update path in .coveragerc

0ff37c3

vercel bot deployed to Preview March 14, 2025 11:08 View deployment

Fix 400 error handling

3835e2a

chore: auto-fix lint and format issues

3f56559

vercel bot deployed to Preview March 14, 2025 12:18 View deployment

lazebnyi added 2 commits March 14, 2025 15:27

Fix catalog changes

484418e

Merge branch 'lazebnyi/source-stripe-migrate-to-low-code' of github.c…

04b2da0

…om:airbytehq/airbyte into lazebnyi/source-stripe-migrate-to-low-code

chore: auto-fix lint and format issues

2e0e5da

vercel bot deployed to Preview March 14, 2025 14:40 View deployment

lazebnyi requested a review from maxi297 March 14, 2025 14:46

maxi297 approved these changes Mar 14, 2025

View reviewed changes

lazebnyi added 2 commits March 14, 2025 19:11

Update CDK dependency

9860e4f

Update version to 5.9.0-rc.1

9271e59

vercel bot deployed to Preview March 14, 2025 18:24 View deployment

natikgadzhi reviewed Mar 16, 2025

View reviewed changes

This was referenced Mar 16, 2025

feat(source-stripe): migrate events, shipping_rates, balance_transactions, files and file_links to low-code #54162

Closed

Source Stripe: Add tax_rates stream #44184

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(source-stripe): migrate to low-code cdk #53687

feat(source-stripe): migrate to low-code cdk #53687

lazebnyi commented Feb 14, 2025 •

edited

Loading

vercel bot commented Feb 14, 2025 •

edited

Loading

lazebnyi commented Feb 14, 2025 •

edited by github-actions bot

Loading

lazebnyi commented Mar 13, 2025 •

edited by github-actions bot

Loading

lazebnyi commented Mar 14, 2025 •

edited by github-actions bot

Loading

lazebnyi commented Mar 14, 2025 •

edited by github-actions bot

Loading

lazebnyi commented Mar 14, 2025

maxi297 left a comment

maxi297 Mar 14, 2025

lazebnyi Mar 14, 2025 •

edited

Loading

natikgadzhi left a comment

feat(source-stripe): migrate to low-code cdk #53687

Are you sure you want to change the base?

feat(source-stripe): migrate to low-code cdk #53687

Conversation

lazebnyi commented Feb 14, 2025 • edited Loading

What

How

Implementation details:

User Impact

Can this PR be safely reverted and rolled back?

vercel bot commented Feb 14, 2025 • edited Loading

lazebnyi commented Feb 14, 2025 • edited by github-actions bot Loading

lazebnyi commented Mar 13, 2025 • edited by github-actions bot Loading

lazebnyi commented Mar 14, 2025 • edited by github-actions bot Loading

lazebnyi commented Mar 14, 2025 • edited by github-actions bot Loading

lazebnyi commented Mar 14, 2025

maxi297 left a comment

Choose a reason for hiding this comment

maxi297 Mar 14, 2025

Choose a reason for hiding this comment

lazebnyi Mar 14, 2025 • edited Loading

Choose a reason for hiding this comment

natikgadzhi left a comment

Choose a reason for hiding this comment

lazebnyi commented Feb 14, 2025 •

edited

Loading

vercel bot commented Feb 14, 2025 •

edited

Loading

lazebnyi commented Feb 14, 2025 •

edited by github-actions bot

Loading

lazebnyi commented Mar 13, 2025 •

edited by github-actions bot

Loading

lazebnyi commented Mar 14, 2025 •

edited by github-actions bot

Loading

lazebnyi commented Mar 14, 2025 •

edited by github-actions bot

Loading

lazebnyi Mar 14, 2025 •

edited

Loading