🎉 Destination Redshift: make purgeStagingData optional

[edgao]

What

Stacked onto #8607 while I wait for the acceptance test to finish.

Addresses #8552

How

S3StreamCopier now accepts an option to not delete staging data. This means that adding this capability to e.g. Snowflake should be pretty straightforward (or at least, will be once #8820 is done).

Also, I think this gets us a tiny step closer to a happy future where we have some kind of consistent CopyDestinationConfig that gets used across redshift/snowflake/databricks/etc. Something along these lines:

{
  ... insert redshift/snowflake/etc properties...
  "load_strategy": {
    "type": "COPY", // or INSERT
    "purgeStagingData": true,
    "backing_storage": {
      "type": "S3", // GCS, azure blob store, etc
      ... insert S3DestinationConfig properties ...
    }
  }
}

Recommended reading order

1. S3StreamCopier + factory + test
2. spec.json + RedshiftCopyS3Destination
3. Everything else

🚨 User Impact 🚨

nope

Pre-merge Checklist

Updating a connector

Community member or Airbyter

• Grant edit access to maintainers (instructions)
• Secrets in the connector's spec are annotated with airbyte_secret
• Unit & integration tests added and passing. Community members, please provide proof of success locally e.g: screenshot or copy-paste unit, integration, and acceptance test output. To run acceptance tests for a Python connector, follow instructions in the README. For java connectors run ./gradlew :airbyte-integrations:connectors:<name>:integrationTest.
• Code reviews completed
• Documentation updated
  • Connector's README.md
  • Connector's bootstrap.md. See description and examples
  • Changelog updated in docs/integrations/<source or destination>/<name>.md including changelog. See changelog example
• PR name follows PR naming conventions

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

• Create a non-forked branch based on this PR and test the below items on it
• Build is successful
• Credentials added to Github CI. Instructions.
• /test connector=connectors/<name> command is passing.
• New Connector version released on Dockerhub by running the /publish command described here
• After the new connector version is published, connector version bumped in the seed directory as described here
• Seed specs have been re-generated by building the platform and committing the changes to the seed spec files, as described here

[edgao]

/test connector=connectors/destination-redshift

🕑 connectors/destination-redshift https://github.com/airbytehq/airbyte/actions/runs/1589561219
✅ connectors/destination-redshift https://github.com/airbytehq/airbyte/actions/runs/1589561219
Python tests coverage:

	 ---------- coverage: platform linux, python 3.8.10-final-0 -----------
	 Name                                                              Stmts   Miss  Cover
	 -------------------------------------------------------------------------------------
	 main_dev_transform_catalog.py                                         3      3     0%
	 main_dev_transform_config.py                                          3      3     0%
	 normalization/__init__.py                                             4      0   100%
	 normalization/destination_type.py                                    13      0   100%
	 normalization/transform_catalog/__init__.py                           2      0   100%
	 normalization/transform_catalog/catalog_processor.py                143     77    46%
	 normalization/transform_catalog/destination_name_transformer.py     124      6    95%
	 normalization/transform_catalog/reserved_keywords.py                 13      0   100%
	 normalization/transform_catalog/stream_processor.py                 494    313    37%
	 normalization/transform_catalog/table_name_registry.py              174     34    80%
	 normalization/transform_catalog/transform.py                         45     26    42%
	 normalization/transform_catalog/utils.py                             33      7    79%
	 normalization/transform_config/__init__.py                            2      0   100%
	 normalization/transform_config/transform.py                         146     32    78%
	 -------------------------------------------------------------------------------------
	 TOTAL                                                              1199    501    58%

[sherifnada]

what's the use of nested in this context?

[edgao]

just to indicate that these tests are specific to the Config record, rather than the factory itself. might actually make more sense to just move the record into its own file though?

[edgao]

/publish connector=connectors/destination-redshift

🕑 connectors/destination-redshift https://github.com/airbytehq/airbyte/actions/runs/1593330144
✅ connectors/destination-redshift https://github.com/airbytehq/airbyte/actions/runs/1593330144