fix(low-code cdk): add type and parameters resolving for dynamic streams

[lazebnyi]

Fixed: https://github.com/airbytehq/airbyte-internal-issues/issues/11699

Summary by CodeRabbit

• New Features

  • Dynamic stream configurations now automatically include enriched type and parameter information, enabling more flexible and consistent parameter merging.
  • Introduced a new optional parameter for prioritizing parent parameters during parameter merging.

• Tests

  • Updated test coverage validates the revised stream configuration structure and ensures accurate handling of dynamic streams.

[lazebnyi]

/autofix

Auto-Fix Job Info

This job attempts to auto-fix any linting or formating issues. If any fixes are made,
those changes will be automatically committed and pushed back to the PR.

Note: This job can only be run by maintainers. On PRs from forks, this command requires
that the PR author has enabled the Allow edits from maintainers option.

PR auto-fix job started... Check job output.

✅ Changes applied successfully.

[coderabbitai]

📝 Walkthrough

Walkthrough

The pull request updates the dynamic stream configuration process in the manifest declarative source. It adds a transformation step where dynamic streams are modified by calling the ManifestComponentTransformer.propagate_types_and_parameters method. This method now accepts an optional use_parent_parameters flag to control how parent and component-specific parameters are merged. Additionally, the stream template configuration in the HTTP components resolver tests has been updated to include a $parameters key, and assertions have been adjusted to verify the newly structured dynamic stream data.

Changes

File(s)	Change Summary
airbyte_cdk/sources/declarative/manifest_declarative_source.py airbyte_cdk/sources/declarative/parsers/manifest_component_transformer.py	Updated dynamic stream processing by adding a transformation step in _dynamic_stream_configs and introducing an optional use_parent_parameters parameter in propagate_types_and_parameters to manage parameter precedence.
unit_tests/sources/declarative/resolvers/test_http_components_resolver.py	Revised test configurations by adding a $parameters key in stream templates and updating assertions to verify dynamic stream name resolution.

Sequence Diagram(s)

sequenceDiagram
    participant DS as ManifestDeclarativeSource
    participant MCT as ManifestComponentTransformer
    DS->>DS: Resolve components for dynamic streams
    DS->>MCT: Call propagate_types_and_parameters("", dynamic_stream, {}, False)
    MCT-->>DS: Return enriched dynamic_stream
    DS->>DS: Continue processing transformed dynamic_stream

Loading

Possibly related PRs

• feat(HttpComponentsResolver): added stream slices to HttpComponentsResolver #175: The changes in the main PR, which involve modifying the _dynamic_stream_configs method to apply transformations using the propagate_types_and_parameters method, are directly related to the updates in the retrieved PR that introduce a new parameter to this method, affecting how it processes components.
• feat(low-code cdk): add transformation to dynamic schema loader #176: The changes in the main PR, which involve modifying the _dynamic_stream_configs method to apply transformations using the ManifestComponentTransformer, are related to the changes in the retrieved PR that introduce a new parameter to the propagate_types_and_parameters method, as both involve enhancing the transformation capabilities of components within the Airbyte framework.
• fix(low-code): add wrong dynamic stream name type validation #305: The changes in the main PR and the retrieved PR are related as both modify the _dynamic_stream_configs method of the ManifestDeclarativeSource class, with the main PR adding a transformation step and the retrieved PR introducing a validation check for the stream name.

Suggested labels

enhancement

Suggested reviewers

• maxi297 (wdyt?)
• darynaishchenko (wdyt?)
• aldogonzalez8 (wdyt?)

---

📜 Recent review details

Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 70e6665 and 8fef455.

📒 Files selected for processing (1)

• airbyte_cdk/sources/declarative/manifest_declarative_source.py (1 hunks)

🚧 Files skipped from review as they are similar to previous changes (1)

• airbyte_cdk/sources/declarative/manifest_declarative_source.py

⏰ Context from checks skipped due to timeout of 90000ms (9)

• GitHub Check: Check: 'source-pokeapi' (skip=false)
• GitHub Check: Check: 'source-amplitude' (skip=false)
• GitHub Check: Check: 'source-shopify' (skip=false)
• GitHub Check: Check: 'source-hardcoded-records' (skip=false)
• GitHub Check: Pytest (Fast)
• GitHub Check: Pytest (All, Python 3.11, Ubuntu)
• GitHub Check: SDM Docker Image Build
• GitHub Check: Pytest (All, Python 3.10, Ubuntu)
• GitHub Check: Analyze (python)

---

🪧 Tips

Chat

There are 3 ways to chat with CodeRabbit:

‼️ IMPORTANT
Auto-reply has been disabled for this repository in the CodeRabbit settings. The CodeRabbit bot will not respond to your replies unless it is explicitly tagged.

• Files and specific lines of code (under the "Files changed" tab): Tag @coderabbitai in a new review comment at the desired location with your query. Examples:
  • @coderabbitai generate unit testing code for this file.
  • @coderabbitai modularize this function.
• PR comments: Tag @coderabbitai in a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples:
  • @coderabbitai gather interesting stats about this repository and render them as a table. Additionally, render a pie chart showing the language distribution in the codebase.
  • @coderabbitai read src/utils.ts and generate unit testing code.
  • @coderabbitai read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.
  • @coderabbitai help me debug CodeRabbit configuration file.

Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments.

CodeRabbit Commands (Invoked using PR comments)

• @coderabbitai pause to pause the reviews on a PR.
• @coderabbitai resume to resume the paused reviews.
• @coderabbitai review to trigger an incremental review. This is useful when automatic reviews are disabled for the repository.
• @coderabbitai full review to do a full review from scratch and review all the files again.
• @coderabbitai summary to regenerate the summary of the PR.
• @coderabbitai generate docstrings to generate docstrings for this PR.
• @coderabbitai resolve resolve all the CodeRabbit review comments.
• @coderabbitai configuration to show the current CodeRabbit configuration for the repository.
• @coderabbitai help to get help.

Other keywords and placeholders

• Add @coderabbitai ignore anywhere in the PR description to prevent this PR from being reviewed.
• Add @coderabbitai summary to generate the high-level summary at a specific location in the PR description.
• Add @coderabbitai anywhere in the PR title to generate the title automatically.

CodeRabbit Configuration File (.coderabbit.yaml)

• You can programmatically configure CodeRabbit by adding a .coderabbit.yaml file to the root of your repository.
• Please see the configuration documentation for more information.
• If your editor has YAML language server enabled, you can add the path at the top of this file to enable auto-completion and validation: # yaml-language-server: $schema=https://coderabbit.ai/integrations/schema.v2.json

Documentation and Community

• Visit our Documentation for detailed information on how to use CodeRabbit.
• Join our Discord Community to get help, request features, and share feedback.
• Follow us on X/Twitter for updates and announcements.