Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

🚨Source Fauna: Migrate to poetry #41051

Merged
merged 33 commits into from
Mar 14, 2025
Merged
Show file tree
Hide file tree
Changes from 22 commits
Commits
Show all changes
33 commits
Select commit Hold shift + click to select a range
9697bff
migrate to poetry
btkcodedev Jul 8, 2024
dafc636
format
btkcodedev Jul 8, 2024
d79ef11
update docs
btkcodedev Jul 8, 2024
6d64d2e
update docs
btkcodedev Jul 8, 2024
78af479
update dependency
btkcodedev Jul 8, 2024
d180db0
Merge branch 'master' into btkcodedev/faunaMigrtepoetry
ChristoGrab Jul 8, 2024
480416a
pin airbyte cdk to 0.80.0
btkcodedev Jul 9, 2024
f7e9217
Merge branch 'btkcodedev/faunaMigrtepoetry' of https://github.com/air…
btkcodedev Jul 9, 2024
7f6a861
update cdk
btkcodedev Jul 10, 2024
e823571
Merge branch 'master' into btkcodedev/faunaMigrtepoetry
btkcodedev Jul 11, 2024
5372d79
update cdk
btkcodedev Jul 11, 2024
67f4078
chore: set airbyte cdk same as previous
btkcodedev Jul 15, 2024
0c007ba
poetry lock
btkcodedev Jul 15, 2024
4accc24
add docker to dev dependency
btkcodedev Jul 15, 2024
0f48499
update dependency
btkcodedev Jul 15, 2024
e162449
fix: pin python to 3.9 for pyairbyte to work
btkcodedev Jul 19, 2024
f1c8383
pin to 3.10.12 python
btkcodedev Jul 19, 2024
2f3036e
revert to default
btkcodedev Jul 19, 2024
1bc0937
Merge branch 'master' into btkcodedev/faunaMigrtepoetry
natikgadzhi Oct 14, 2024
272c8b6
Update airbyte-integrations/connectors/source-fauna/pyproject.toml
natikgadzhi Oct 14, 2024
1519d87
Update airbyte-integrations/connectors/source-fauna/pyproject.toml
natikgadzhi Oct 14, 2024
cd6f658
Update airbyte-integrations/connectors/source-fauna/pyproject.toml
natikgadzhi Oct 14, 2024
34c9251
Update airbyte-integrations/connectors/source-fauna/pyproject.toml
natikgadzhi Oct 14, 2024
24a8be4
Update airbyte-integrations/connectors/source-fauna/metadata.yaml
natikgadzhi Oct 14, 2024
87ccb68
Resolve conflicts by deleting setup.py for Poetry migration
devin-ai-integration[bot] Mar 1, 2025
c415941
Update Python version requirement, add Dockerfile with 4.0.0 base ima…
devin-ai-integration[bot] Mar 1, 2025
c781845
Add updated poetry.lock file
devin-ai-integration[bot] Mar 1, 2025
c2686c1
Revert to python-connector-base:3.0.2 and downgrade airbyte-cdk to <1.0
devin-ai-integration[bot] Mar 1, 2025
3f27745
Remove Dockerfile and update base image in metadata.yaml
devin-ai-integration[bot] Mar 1, 2025
e2de185
Update airbyte-integrations/connectors/source-fauna/metadata.yaml
natikgadzhi Mar 3, 2025
e7e6dda
Merge branch 'master' into btkcodedev/faunaMigrtepoetry
natikgadzhi Mar 3, 2025
f7b7735
Merge branch 'master' into btkcodedev/faunaMigrtepoetry
btkcodedev Mar 13, 2025
9f949c4
update tests
btkcodedev Mar 13, 2025
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 0 additions & 6 deletions airbyte-integrations/connectors/source-fauna/.dockerignore

This file was deleted.

38 changes: 0 additions & 38 deletions airbyte-integrations/connectors/source-fauna/Dockerfile

This file was deleted.

62 changes: 55 additions & 7 deletions airbyte-integrations/connectors/source-fauna/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -120,22 +120,70 @@ python main.py read --config secrets/config.json --catalog integration_tests/con

### Locally running the connector docker image

#### Build

**Via [`airbyte-ci`](https://github.com/airbytehq/airbyte/blob/master/airbyte-ci/connectors/pipelines/README.md) (recommended):**

#### Use `airbyte-ci` to build your connector
The Airbyte way of building this connector is to use our `airbyte-ci` tool.
You can follow install instructions [here](https://github.com/airbytehq/airbyte/blob/master/airbyte-ci/connectors/pipelines/README.md#L1).
Then running the following command will build your connector:

```bash
airbyte-ci connectors --name=source-fauna build
airbyte-ci connectors --name source-fauna build
```
Once the command is done, you will find your connector image in your local docker registry: `airbyte/source-fauna:dev`.

##### Customizing our build process
When contributing on our connector you might need to customize the build process to add a system dependency or set an env var.
You can customize our build process by adding a `build_customization.py` module to your connector.
This module should contain a `pre_connector_install` and `post_connector_install` async function that will mutate the base image and the connector container respectively.
It will be imported at runtime by our build process and the functions will be called if they exist.

Here is an example of a `build_customization.py` module:
```python
from __future__ import annotations

from typing import TYPE_CHECKING

if TYPE_CHECKING:
# Feel free to check the dagger documentation for more information on the Container object and its methods.
# https://dagger-io.readthedocs.io/en/sdk-python-v0.6.4/
from dagger import Container


async def pre_connector_install(base_image_container: Container) -> Container:
return await base_image_container.with_env_variable("MY_PRE_BUILD_ENV_VAR", "my_pre_build_env_var_value")

async def post_connector_install(connector_container: Container) -> Container:
return await connector_container.with_env_variable("MY_POST_BUILD_ENV_VAR", "my_post_build_env_var_value")
```

An image will be built with the tag `airbyte/source-fauna:dev`.
#### Build your own connector image
This connector is built using our dynamic built process in `airbyte-ci`.
The base image used to build it is defined within the metadata.yaml file under the `connectorBuildOptions`.
The build logic is defined using [Dagger](https://dagger.io/) [here](https://github.com/airbytehq/airbyte/blob/master/airbyte-ci/connectors/pipelines/pipelines/builds/python_connectors.py).
It does not rely on a Dockerfile.

If you would like to patch our connector and build your own a simple approach would be to:

1. Create your own Dockerfile based on the latest version of the connector image.
```Dockerfile
FROM airbyte/source-fauna:latest

**Via `docker build`:**
COPY . ./airbyte/integration_code
RUN pip install ./airbyte/integration_code

# The entrypoint and default env vars are already set in the base image
# ENV AIRBYTE_ENTRYPOINT "python /airbyte/integration_code/main.py"
# ENTRYPOINT ["python", "/airbyte/integration_code/main.py"]
```
Please use this as an example. This is not optimized.

2. Build your image:
```bash
docker build -t airbyte/source-fauna:dev .
# Running the spec command against your patched connector
docker run airbyte/source-fauna:dev spec
```

#### Run

Then run any of the connector commands as follows:
Expand Down Expand Up @@ -178,4 +226,4 @@ You've checked out the repo, implemented a million dollar feature, and you're re
4. Make the connector documentation and its changelog is up to date (`docs/integrations/sources/fauna.md`).
5. Create a Pull Request: use [our PR naming conventions](https://docs.airbyte.com/contributing-to-airbyte/resources/pull-requests-handbook/#pull-request-title-convention).
6. Pat yourself on the back for being an awesome contributor.
7. Someone from Airbyte will take a look at your PR and iterate with you to merge it into master.
7. Someone from Airbyte will take a look at your PR and iterate with you to merge it into master.
46 changes: 26 additions & 20 deletions airbyte-integrations/connectors/source-fauna/metadata.yaml
Original file line number Diff line number Diff line change
@@ -1,9 +1,29 @@
data:
ab_internal:
ql: 100
sl: 100
connectorBuildOptions:
baseImage: docker.io/airbyte/python-connector-base:1.2.2@sha256:57703de3b4c4204bd68a7b13c9300f8e03c0189bffddaffc796f1da25d2dbea0
connectorSubtype: database
connectorTestSuitesOptions:
- suite: unitTests
- suite: acceptanceTests
testSecrets:
- fileName: config.json
name: SECRET_SOURCE-FAUNA_CREDS
secretStore:
alias: airbyte-connector-testing-secret-store
type: GSM
- fileName: config-deletions.json
name: SECRET_SOURCE-FAUNA_DELETION_CREDS
secretStore:
alias: airbyte-connector-testing-secret-store
type: GSM
connectorType: source
definitionId: 3825db3e-c94b-42ac-bd53-b5a9507ace2b
dockerImageTag: 0.1.1
dockerImageTag: 0.1.2
dockerRepository: airbyte/source-fauna
documentationUrl: https://docs.airbyte.com/integrations/sources/fauna
githubIssueLabel: source-fauna
icon: fauna.svg
license: MIT
Expand All @@ -18,26 +38,12 @@ data:
oss:
enabled: true
releaseStage: alpha
documentationUrl: https://docs.airbyte.com/integrations/sources/fauna
remoteRegistries:
pypi:
enabled: true
packageName: airbyte-source-fauna
supportLevel: community
tags:
- language:python
- cdk:python
ab_internal:
sl: 100
ql: 100
supportLevel: community
connectorTestSuitesOptions:
- suite: unitTests
- suite: acceptanceTests
testSecrets:
- name: SECRET_SOURCE-FAUNA_CREDS
fileName: config.json
secretStore:
type: GSM
alias: airbyte-connector-testing-secret-store
- name: SECRET_SOURCE-FAUNA_DELETION_CREDS
fileName: config-deletions.json
secretStore:
type: GSM
alias: airbyte-connector-testing-secret-store
metadataSpecVersion: "1.0"
Loading
Loading