feat(source-Monday): Update CDK to v6 #55225

tolik0 · 2025-03-06T12:30:49Z

What

Update CDK to v6.

How

Migrated boards and items stream to StateDelegatingStream.
Update custom components to match abstract classes from the CDK

User Impact

The sync should become quicker as the source is now concurrent.

Can this PR be safely reverted and rolled back?

YES 💚
NO ❌

vercel · 2025-03-06T12:30:54Z

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
airbyte-docs	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Mar 21, 2025 1:35pm

natikgadzhi · 2025-03-09T19:31:09Z

@tolik0 is this ready to go? If so, @dbgold17 can you review this and see if we should merge this first and apply your changes on top, so you get to work with the new CDK?

dbgold17

Hi @tolik0 , I left a bunch of comments, a lot of small things and then some which have to do with me not understanding the changes you're making. Would you be able to write a couple sentences on the PR description about the structural changes you needed to make in order to update the CDK to version 6? It would be super helpful for me and for others looking at this code in the future :)

Also, how have you tested these changes? Do you feel confident that they won't break anything for uses?

airbyte-integrations/connectors/source-monday/source_monday/source.py

airbyte-integrations/connectors/source-monday/source_monday/run.py

dbgold17 · 2025-03-10T05:38:16Z

airbyte-integrations/connectors/source-monday/pyproject.toml

-python = "^3.9,<3.12"
-airbyte-cdk = "0.78.6" # Breaks with newer versions of the CDK
+python = ">=3.10,<3.12"
+airbyte_cdk = { path = "/home/anatolii/airbytehq/airbyte/airbyte-cdk/python/airbyte-python-cdk/", develop = true }


this should be replaced with a public package version right?

Added dev version

dbgold17 · 2025-03-10T05:39:28Z

airbyte-integrations/connectors/source-monday/metadata.yaml

@@ -10,7 +10,7 @@ data:
  connectorSubtype: api
  connectorType: source
  definitionId: 80a54ea2-9959-4040-aac1-eee42423ec9b
-  dockerImageTag: 2.1.13
+  dockerImageTag: 2.2.0


is there consensus about whether to bump minor or patch number in this version?

This should be the minor update

dbgold17 · 2025-03-10T05:43:29Z

airbyte-integrations/connectors/source-monday/source_monday/extractor.py

-            return extracted if isinstance(extracted, list) else [extracted]
-        return []
+        decoded_response = self.decoder.decode(response)
+        for response_body in decoded_response:


why are you now looping over this response where we weren't before?

The underlying CDK class changed

airbyte-integrations/connectors/source-monday/source_monday/graphql_requester.py

dbgold17 · 2025-03-10T05:58:34Z

airbyte-integrations/connectors/source-monday/source_monday/item_pagination_strategy.py

+        response: requests.Response,
+        last_page_size: int,
+        last_record: Optional[Record],
+        last_page_token_value: Optional[Any]) -> Optional[Tuple[Optional[int], Optional[int]]]:
        """


update docstring to conform with new interface.

dbgold17 · 2025-03-10T05:58:35Z

airbyte-integrations/connectors/source-monday/source_monday/item_pagination_strategy.py

@@ -72,7 +78,11 @@ def __post_init__(self, parameters: Mapping[str, Any]):
        self._page: Optional[int] = self.start_from_page
        self._sub_page: Optional[int] = self.start_from_page

-    def next_page_token(self, response, last_records: List[Mapping[str, Any]]) -> Optional[Tuple[Optional[int], Optional[int]]]:
+    def next_page_token(self,


why is this change necessary? It doesn't seem like you're using any of these arguments.

If it's only in order to conform to some interface spec, can you note that in a comment?

Yes, the underlying class in the CDK changed, so we need to align with it

dbgold17 · 2025-03-10T06:15:32Z

airbyte-integrations/connectors/source-monday/source_monday/extractor.py

+        response_decoded = self.decoder.decode(response)
+        for response_body in response_decoded:
+            if not response_body["data"]["boards"]:
+                yield from []


do you actually want to return a generator object with nothing in it here or just continue?

dbgold17 · 2025-03-10T06:17:12Z

airbyte-integrations/connectors/source-monday/source_monday/extractor.py

@@ -37,32 +37,30 @@ class MondayActivityExtractor(RecordExtractor):
    decoder: Decoder = JsonDecoder(parameters={})

    def extract_records(self, response: requests.Response) -> List[Record]:


should the return type annotation be updated?

tolik0 · 2025-03-10T09:58:14Z

@dbgold17 Sorry, this PR is not in review yet. I still need to add StateDelegatingRetriever, which hasn’t been merged yet.
I'm currently busy handling P1 issues, but I’ll ping you once it’s ready.
Thank you!

dbgold17 · 2025-03-10T17:11:54Z

@dbgold17 Sorry, this PR is not in review yet. I still need to add StateDelegatingRetriever, which hasn’t been merged yet. I'm currently busy handling P1 issues, but I’ll ping you once it’s ready. Thank you!

Sounds good, sorry for the premature review!

…n-router

tolik0 · 2025-03-18T14:51:04Z

/format-fix

Format-fix job started... Check job output.

✅ Changes applied successfully. (1264ab2)

tolik0 · 2025-03-19T14:21:35Z

/format-fix

Format-fix job started... Check job output.

✅ Changes applied successfully. (39c39bf)

tolik0 · 2025-03-19T17:05:45Z

/format-fix

Format-fix job started... Check job output.

✅ Changes applied successfully. (727c1a1)

tolik0 · 2025-03-20T10:25:07Z

/format-fix

Format-fix job started... Check job output.

✅ Changes applied successfully. (9d4234d)

tolik0 · 2025-03-20T18:31:22Z

Regression test results:

… into tolik0/source-monday/add-grouping-partition-router

brianjlai

one suggestions on refactoring try_extract_records() to process records iteratively since the underlying call became a generator, but otherwise no other comments and looks good to me!

brianjlai · 2025-03-20T19:03:41Z

airbyte-integrations/connectors/source-monday/source_monday/extractor.py

-                return [x for x in extracted if x]
-            return extracted if isinstance(extracted, list) else [extracted]
-        return []
+        decoded_response = self.decoder.decode(response)


because self.decoder.decode(response) has been changed to become a generator, can we refactor this method to also iterate over the generator and yield from extracted below instead of storing them in the result list?

brianjlai · 2025-03-20T19:08:36Z

airbyte-integrations/connectors/source-monday/source_monday/extractor.py

@@ -117,4 +117,4 @@ def extract_records(self, response: requests.Response) -> List[Record]:
                if display_value and not text:
                    values["text"] = display_value

-        return result
+        yield from result


In addition to the above comment, if we refactor self.try_extract_records() to be an iterable, we should also yield record at the end of each iteration of the for loop so we don't have to keep everything in memory with result

brianjlai · 2025-03-20T19:13:40Z

airbyte-integrations/connectors/source-monday/source_monday/manifest.yaml

+concurrency_level:
+  type: ConcurrencyLevel
+  default_concurrency: "{{ config.get('num_workers', 4) }}"
+  max_concurrency: 40


Monday actually has a pretty detailed section on rate limits: https://developer.monday.com/api-reference/docs/rate-limits

40 actually seems like an acceptable threshold since they call out only allowing for at most 40 concurrent requests at any given moment. But let's directly reference https://developer.monday.com/api-reference/docs/rate-limits#concurrency-limit this in the comment

tolik0 · 2025-03-21T13:24:41Z

/format-fix

Format-fix job started... Check job output.

✅ Changes applied successfully. (6adeeb1)

brianjlai

i had a couple small questions, but i'm fine with just answering or addressing and no need to block.

brianjlai · 2025-03-21T17:38:07Z

airbyte-integrations/connectors/source-monday/source_monday/extractor.py

+            else:
+                path = [p.eval(self.config) for p in field_path]
+                if "*" in path:
+                    extracted = dpath.values(body, path)


small nit/question, why the switch to use dpath.values() instead of dpath.util.values()? same for the below get() as well

Updated according to the latest implementation of DpathExtractor. Here is the PR for the util change: #38847

brianjlai · 2025-03-21T17:48:56Z

airbyte-integrations/connectors/source-monday/source_monday/extractor.py

-            result = self.try_extract_records(response, self.field_path_incremental)
-
-        for record in result:
-            if "updated_at" in record:


why don't we have to edit some of the fields on each record object anymore?

I moved this to transformations

tolik0 added 7 commits November 21, 2024 15:29

Add WaitTimeFromHeader backoff strategy

9b58f82

Update version

c11239c

Update CDK version to 0.90.0

2382b81

Revert CDK update

163d3dc

Update page size for items stream

1a3c390

Update page size to 10

7de9fa8

Update CDK to v6

c9cc1b5

octavia-squidington-iii added area/connectors area/documentation connectors/source/monday labels Mar 6, 2025

vercel bot deployed to Preview March 6, 2025 12:34 View deployment

dbgold17 reviewed Mar 10, 2025

View reviewed changes

tolik0 marked this pull request as draft March 10, 2025 09:55

dbgold17 mentioned this pull request Mar 12, 2025

[WIP] upgrade source monday to cdk v6 #55712

Draft

2 tasks

tolik0 added 2 commits March 18, 2025 16:32

Add StateDelegatingStream

4434e5d

Merge branch 'master' into tolik0/source-monday/add-grouping-partitio…

633bcaa

…n-router

vercel bot deployed to Preview March 18, 2025 14:39 View deployment

tolik0 added 2 commits March 18, 2025 16:45

Update Monday version

7fde793

Update spec

c2036a2

octavia-squidington-iii and others added 2 commits March 18, 2025 14:52

chore: auto-fix lint and format issues

1264ab2

Update poetry.lock

ef34439

vercel bot deployed to Preview March 18, 2025 15:11 View deployment

tolik0 added 2 commits March 18, 2025 17:19

Small refactoring

f9f53cc

Fix types in custom record extractor

fd46351

tolik0 marked this pull request as ready for review March 18, 2025 15:25

Fix all unit tests

663b6f0

chore: auto-fix lint and format issues

39c39bf

vercel bot deployed to Preview March 19, 2025 14:32 View deployment

Fix comments from review

e20cc84

chore: auto-fix lint and format issues

727c1a1

vercel bot deployed to Preview March 19, 2025 17:16 View deployment

Delete testing print

675ffd9

vercel bot deployed to Preview March 19, 2025 17:43 View deployment

Fix state migration for boards stream

fbf1503

vercel bot deployed to Preview March 19, 2025 18:11 View deployment

Fix unit test coverage

fbe8c79

chore: auto-fix lint and format issues

9d4234d

vercel bot deployed to Preview March 20, 2025 10:37 View deployment

Fix state migration test

5c7cc55

vercel bot deployed to Preview March 20, 2025 11:27 View deployment

Merge branch 'tolik0/source-monday/add-wait-time-from-header-backoff'…

23c957a

… into tolik0/source-monday/add-grouping-partition-router

tolik0 requested a review from brianjlai March 20, 2025 18:36

Update CDK version

62fce0b

vercel bot deployed to Preview March 20, 2025 18:50 View deployment

brianjlai reviewed Mar 20, 2025

View reviewed changes

Refactor MondayIncrementalItemsExtractor

edccab3

chore: auto-fix lint and format issues

6adeeb1

vercel bot deployed to Preview March 21, 2025 13:35 View deployment

tolik0 requested a review from brianjlai March 21, 2025 15:23

brianjlai approved these changes Mar 21, 2025

View reviewed changes

		@@ -37,32 +37,30 @@ class MondayActivityExtractor(RecordExtractor):
		decoder: Decoder = JsonDecoder(parameters={})

		def extract_records(self, response: requests.Response) -> List[Record]:

feat(source-Monday): Update CDK to v6 #55225

Are you sure you want to change the base?

feat(source-Monday): Update CDK to v6 #55225

Conversation

tolik0 commented Mar 6, 2025 • edited Loading

What

How

User Impact

Can this PR be safely reverted and rolled back?

vercel bot commented Mar 6, 2025 • edited Loading

natikgadzhi commented Mar 9, 2025

dbgold17 left a comment

Choose a reason for hiding this comment

dbgold17 Mar 10, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tolik0 commented Mar 10, 2025

dbgold17 commented Mar 10, 2025

tolik0 commented Mar 18, 2025 • edited by github-actions bot Loading

tolik0 commented Mar 19, 2025 • edited by github-actions bot Loading

tolik0 commented Mar 19, 2025 • edited by github-actions bot Loading

tolik0 commented Mar 20, 2025 • edited by github-actions bot Loading

tolik0 commented Mar 20, 2025

brianjlai left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

brianjlai Mar 20, 2025 • edited Loading

Choose a reason for hiding this comment

tolik0 commented Mar 21, 2025 • edited by github-actions bot Loading

brianjlai left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tolik0 commented Mar 6, 2025 •

edited

Loading

vercel bot commented Mar 6, 2025 •

edited

Loading

dbgold17 Mar 10, 2025 •

edited

Loading

tolik0 commented Mar 18, 2025 •

edited by github-actions bot

Loading

tolik0 commented Mar 19, 2025 •

edited by github-actions bot

Loading

tolik0 commented Mar 19, 2025 •

edited by github-actions bot

Loading

tolik0 commented Mar 20, 2025 •

edited by github-actions bot

Loading

brianjlai Mar 20, 2025 •

edited

Loading

tolik0 commented Mar 21, 2025 •

edited by github-actions bot

Loading