Skip to content

Commit 049eb06

Browse files
jrhizorlmossmansherifnadaigrankovagrubberr
authored andcommitted
upgrade to Gradle 7.3.1 / Java 17 (airbytehq#7964)
* upgrade gradle * upgrade to Java 17 (and fix a few of the node versioning misses) * oops * try to run a different format version * fix spotless by upgrading / reformatting some files * fix ci settings * upgrade mockito to avoid other errors * undo bad format * fix "incorrect" sql comments * fmt * add debug flag * remove * bump * bump jooq to a version that has a java 17 dist * fix * remove logs * oops * revert jooq upgrade * fix * set up java for connector test * fix yaml * generate std source tests * fail zombie job attempts and add failure reason (airbytehq#8709) * fail zombie job attempts and add failure reason * remove failure reason * bump gcp dependencies to pick up grpc update (airbytehq#8713) * Bump Airbyte version from 0.33.9-alpha to 0.33.10-alpha (airbytehq#8714) Co-authored-by: jrhizor <[email protected]> * Change CDK "Caching" header to "nested streams & caching" * Update fields in source-connectors specifications: file, freshdesk, github, google-directory, google-workspace-admin-reports, iterable (airbytehq#8524) Signed-off-by: Sergey Chvalyuk <[email protected]> Co-authored-by: Serhii Chvaliuk <[email protected]> Co-authored-by: Sherif A. Nada <[email protected]> * move S3Config into destination-s3; update dependencies accordingly (airbytehq#8562) Co-authored-by: Lake Mossman <[email protected]> Co-authored-by: jrhizor <[email protected]> Co-authored-by: Sherif A. Nada <[email protected]> Co-authored-by: Iryna Grankova <[email protected]> Co-authored-by: Serhii Chvaliuk <[email protected]> Co-authored-by: Edward Gao <[email protected]>
1 parent dabb6c8 commit 049eb06

File tree

28 files changed

+227
-185
lines changed

28 files changed

+227
-185
lines changed

.github/workflows/gke-kube-test-command.yml

+2-2
Original file line numberDiff line numberDiff line change
@@ -46,11 +46,11 @@ jobs:
4646

4747
- uses: actions/setup-java@v1
4848
with:
49-
java-version: '14'
49+
java-version: '17'
5050

5151
- uses: actions/setup-node@v1
5252
with:
53-
node-version: '14.7'
53+
node-version: '16.13.0'
5454

5555
- name: Fix EC-2 Runner
5656
run: |

.github/workflows/gradle.yml

+24-8
Original file line numberDiff line numberDiff line change
@@ -79,7 +79,7 @@ jobs:
7979

8080
- uses: actions/setup-java@v1
8181
with:
82-
java-version: "14"
82+
java-version: "17"
8383

8484
- uses: actions/setup-node@v1
8585
with:
@@ -96,7 +96,11 @@ jobs:
9696
run: |
9797
mkdir -p ~/.gradle/
9898
cat > ~/.gradle/gradle.properties <<EOF
99-
org.gradle.jvmargs=-Xmx8g -Xss4m
99+
org.gradle.jvmargs=-Xmx8g -Xss4m --add-exports jdk.compiler/com.sun.tools.javac.api=ALL-UNNAMED \
100+
--add-exports jdk.compiler/com.sun.tools.javac.file=ALL-UNNAMED \
101+
--add-exports jdk.compiler/com.sun.tools.javac.parser=ALL-UNNAMED \
102+
--add-exports jdk.compiler/com.sun.tools.javac.tree=ALL-UNNAMED \
103+
--add-exports jdk.compiler/com.sun.tools.javac.util=ALL-UNNAMED
100104
org.gradle.workers.max=8
101105
org.gradle.vfs.watch=false
102106
EOF
@@ -206,7 +210,7 @@ jobs:
206210

207211
- uses: actions/setup-java@v1
208212
with:
209-
java-version: "14"
213+
java-version: "17"
210214

211215
- uses: actions/setup-node@v1
212216
with:
@@ -216,7 +220,11 @@ jobs:
216220
run: |
217221
mkdir -p ~/.gradle/
218222
cat > ~/.gradle/gradle.properties <<EOF
219-
org.gradle.jvmargs=-Xmx8g -Xss4m
223+
org.gradle.jvmargs=-Xmx8g -Xss4m --add-exports jdk.compiler/com.sun.tools.javac.api=ALL-UNNAMED \
224+
--add-exports jdk.compiler/com.sun.tools.javac.file=ALL-UNNAMED \
225+
--add-exports jdk.compiler/com.sun.tools.javac.parser=ALL-UNNAMED \
226+
--add-exports jdk.compiler/com.sun.tools.javac.tree=ALL-UNNAMED \
227+
--add-exports jdk.compiler/com.sun.tools.javac.util=ALL-UNNAMED
220228
org.gradle.workers.max=8
221229
org.gradle.vfs.watch=false
222230
EOF
@@ -330,7 +338,7 @@ jobs:
330338

331339
- uses: actions/setup-java@v1
332340
with:
333-
java-version: "14"
341+
java-version: "17"
334342

335343
- uses: actions/setup-node@v1
336344
with:
@@ -343,7 +351,11 @@ jobs:
343351
run: |
344352
mkdir -p ~/.gradle/
345353
cat > ~/.gradle/gradle.properties <<EOF
346-
org.gradle.jvmargs=-Xmx8g -Xss4m
354+
org.gradle.jvmargs=-Xmx8g -Xss4m --add-exports jdk.compiler/com.sun.tools.javac.api=ALL-UNNAMED \
355+
--add-exports jdk.compiler/com.sun.tools.javac.file=ALL-UNNAMED \
356+
--add-exports jdk.compiler/com.sun.tools.javac.parser=ALL-UNNAMED \
357+
--add-exports jdk.compiler/com.sun.tools.javac.tree=ALL-UNNAMED \
358+
--add-exports jdk.compiler/com.sun.tools.javac.util=ALL-UNNAMED
347359
org.gradle.workers.max=8
348360
org.gradle.vfs.watch=false
349361
EOF
@@ -411,7 +423,7 @@ jobs:
411423

412424
- uses: actions/setup-java@v1
413425
with:
414-
java-version: "14"
426+
java-version: "17"
415427

416428
- uses: actions/setup-node@v1
417429
with:
@@ -425,7 +437,11 @@ jobs:
425437
run: |
426438
mkdir -p ~/.gradle/
427439
cat > ~/.gradle/gradle.properties <<EOF
428-
org.gradle.jvmargs=-Xmx8g -Xss4m
440+
org.gradle.jvmargs=-Xmx8g -Xss4m --add-exports jdk.compiler/com.sun.tools.javac.api=ALL-UNNAMED \
441+
--add-exports jdk.compiler/com.sun.tools.javac.file=ALL-UNNAMED \
442+
--add-exports jdk.compiler/com.sun.tools.javac.parser=ALL-UNNAMED \
443+
--add-exports jdk.compiler/com.sun.tools.javac.tree=ALL-UNNAMED \
444+
--add-exports jdk.compiler/com.sun.tools.javac.util=ALL-UNNAMED
429445
org.gradle.workers.max=8
430446
org.gradle.vfs.watch=false
431447
EOF

.github/workflows/publish-cdk-command.yml

+1-1
Original file line numberDiff line numberDiff line change
@@ -23,7 +23,7 @@ jobs:
2323
python-version: ${{ matrix.python-version }}
2424
- uses: actions/setup-java@v1
2525
with:
26-
java-version: '14'
26+
java-version: '17'
2727
- name: Checkout Airbyte
2828
uses: actions/checkout@v2
2929
- name: Build CDK Package

.github/workflows/release-airbyte-os.yml

+2-2
Original file line numberDiff line numberDiff line change
@@ -17,11 +17,11 @@ jobs:
1717
fetch-depth: 0
1818
- uses: actions/setup-java@v1
1919
with:
20-
java-version: '14'
20+
java-version: '17'
2121

2222
- uses: actions/setup-node@v1
2323
with:
24-
node-version: '14.7'
24+
node-version: '16.13.0'
2525
- name: Save Old Version
2626
id: old_version
2727
run: |

.github/workflows/test-command.yml

+3
Original file line numberDiff line numberDiff line change
@@ -64,6 +64,9 @@ jobs:
6464
# Beside PyEnv, this does not set any runtimes up because it uses an AMI image that has everything pre-installed. See https://github.com/airbytehq/airbyte/issues/4559/
6565
- name: Install Pyenv
6666
run: python3 -m pip install virtualenv==16.7.9 --user
67+
- uses: actions/setup-java@v1
68+
with:
69+
java-version: '17'
6770
- name: Write Integration Test Credentials
6871
run: ./tools/bin/ci_credentials.sh ${{ github.event.inputs.connector }}
6972
env:

airbyte-db/lib/src/main/resources/configs_database/schema.sql

+2-2
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
-- tables
2-
CREATE
2+
CREATE
33
TABLE
44
IF NOT EXISTS AIRBYTE_CONFIGS(
55
id BIGINT GENERATED BY DEFAULT AS IDENTITY PRIMARY KEY,
@@ -11,7 +11,7 @@
1111
);
1212

1313
-- indices
14-
CREATE
14+
CREATE
1515
UNIQUE INDEX IF NOT EXISTS airbyte_configs_type_id_idx ON
1616
AIRBYTE_CONFIGS(
1717
config_type,

airbyte-db/lib/src/main/resources/jobs_database/schema.sql

+2-2
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
-- types
2-
CREATE
2+
CREATE
33
TYPE JOB_STATUS AS ENUM(
44
'pending',
55
'running',
@@ -27,7 +27,7 @@ CREATE
2727
);
2828

2929
-- tables
30-
CREATE
30+
CREATE
3131
TABLE
3232
IF NOT EXISTS AIRBYTE_METADATA(
3333
KEY VARCHAR(255) PRIMARY KEY,

airbyte-db/lib/src/test/java/io/airbyte/db/instance/configs/migrations/V0_30_22_001__Store_last_sync_state_test.java

+7-1
Original file line numberDiff line numberDiff line change
@@ -158,7 +158,13 @@ public void testCopyData() throws SQLException {
158158
.withConnectionId(CONNECTION_2_ID)
159159
.withState(new State().withState(Jsons.deserialize("{ \"cursor\": 3 }"))));
160160

161-
final OffsetDateTime timestamp = OffsetDateTime.now();
161+
final OffsetDateTime timestampWithFullPrecision = OffsetDateTime.now();
162+
/*
163+
* The AWS CI machines get a higher precision value here (2021-12-07T19:56:28.967213187Z) vs what is
164+
* retrievable on Postgres or on my local machine (2021-12-07T19:56:28.967213Z). Truncating the
165+
* value to match.
166+
*/
167+
final OffsetDateTime timestamp = timestampWithFullPrecision.withNano(1000 * (timestampWithFullPrecision.getNano() / 1000));
162168

163169
database.query(ctx -> {
164170
V0_30_22_001__Store_last_sync_state.copyData(ctx, STD_CONNECTION_STATES, timestamp);

airbyte-integrations/bases/base-java/Dockerfile

+1-1
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
ARG JDK_VERSION=14.0.2
1+
ARG JDK_VERSION=17.0.1
22
FROM openjdk:${JDK_VERSION}-slim
33
COPY --from=airbyte/integration-base:dev /airbyte /airbyte
44

airbyte-integrations/bases/base-standard-source-test-file/Dockerfile

+1-1
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
ARG JDK_VERSION=14.0.2
1+
ARG JDK_VERSION=17.0.1
22
FROM openjdk:${JDK_VERSION}-slim
33

44
# Install Docker to launch worker images. Eventually should be replaced with Docker-java.

airbyte-integrations/bases/standard-source-test/Dockerfile

+1-1
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
ARG JDK_VERSION=14.0.2
1+
ARG JDK_VERSION=17.0.1
22
FROM openjdk:${JDK_VERSION}-slim
33

44
# Install Docker to launch worker images. Eventually should be replaced with Docker-java.

airbyte-integrations/connectors/source-jdbc/build.gradle

+1-1
Original file line numberDiff line numberDiff line change
@@ -47,7 +47,7 @@ dependencies {
4747
testFixturesRuntimeOnly 'org.junit.jupiter:junit-jupiter-engine:5.4.2'
4848
testFixturesImplementation 'org.junit.jupiter:junit-jupiter-api:5.4.2'
4949
testFixturesImplementation 'org.junit.jupiter:junit-jupiter-params:5.4.2'
50-
testFixturesImplementation group: 'org.mockito', name: 'mockito-junit-jupiter', version: '3.4.6'
50+
testFixturesImplementation group: 'org.mockito', name: 'mockito-junit-jupiter', version: '4.0.0'
5151

5252
implementation files(project(':airbyte-integrations:bases:base-java').airbyteDocker.outputs)
5353
}

airbyte-integrations/infrastructure/ssh_tunnel/module/sql/postgresql-01-dbcreate.sql

+1-1
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
-- generic setup for a brand new empty postgresql RDS
2-
CREATE
2+
CREATE
33
ROLE integrationtest_rw;
44

55
GRANT integrationtest_rw TO airbyte;
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
-- create the specific user we want to use from airbyte
2-
CREATE
2+
CREATE
33
USER testcaseuser WITH password 'ThisIsNotTheRealPassword.PleaseSetThisByHand';
44

55
GRANT integrationtest_rw TO testcaseuser;

airbyte-scheduler/app/Dockerfile

+1-1
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
ARG JDK_VERSION=14.0.2
1+
ARG JDK_VERSION=17.0.1
22
FROM openjdk:${JDK_VERSION}-slim AS scheduler
33

44
ENV APPLICATION airbyte-scheduler

airbyte-scheduler/persistence/src/main/resources/job_history_purge.sql

+6-6
Original file line numberDiff line numberDiff line change
@@ -18,8 +18,8 @@ WHERE
1818
) counts ON
1919
jobs.scope = counts.scope
2020
WHERE
21-
-- job must be at least MINIMUM_AGE_IN_DAYS old or connection has more than EXCESSIVE_NUMBER_OF_JOBS
22-
(
21+
/* job must be at least MINIMUM_AGE_IN_DAYS old or connection has more than EXCESSIVE_NUMBER_OF_JOBS */
22+
(
2323
jobs.created_at <(
2424
TO_TIMESTAMP(
2525
?,
@@ -29,8 +29,8 @@ WHERE
2929
OR counts.jobCount >?
3030
)
3131
AND jobs.id NOT IN(
32-
-- cannot be the most recent job with saved state
33-
SELECT
32+
/* cannot be the most recent job with saved state */
33+
SELECT
3434
job_id AS latest_job_id_with_state
3535
FROM
3636
(
@@ -69,8 +69,8 @@ WHERE
6969
stateRecency = 1
7070
)
7171
AND jobs.id NOT IN(
72-
-- cannot be one of the last MINIMUM_RECENCY jobs for that connection/scope
73-
SELECT
72+
/* cannot be one of the last MINIMUM_RECENCY jobs for that connection/scope */
73+
SELECT
7474
id
7575
FROM
7676
(

airbyte-server/Dockerfile

+1-1
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
ARG JDK_VERSION=14.0.2
1+
ARG JDK_VERSION=17.0.1
22
FROM openjdk:${JDK_VERSION}-slim AS server
33

44
EXPOSE 8000

airbyte-server/src/test/resources/migration/schema.sql

+4-4
Original file line numberDiff line numberDiff line change
@@ -1,9 +1,9 @@
11
-- extensions
2-
CREATE
2+
CREATE
33
EXTENSION IF NOT EXISTS "uuid-ossp";
44

55
-- types
6-
CREATE
6+
CREATE
77
TYPE JOB_STATUS AS ENUM(
88
'pending',
99
'running',
@@ -31,7 +31,7 @@ CREATE
3131
);
3232

3333
-- tables
34-
CREATE
34+
CREATE
3535
TABLE
3636
AIRBYTE_METADATA(
3737
KEY VARCHAR(255) PRIMARY KEY,
@@ -73,7 +73,7 @@ CREATE
7373
);
7474

7575
-- entries
76-
INSERT
76+
INSERT
7777
INTO
7878
AIRBYTE_METADATA
7979
VALUES(

airbyte-workers/Dockerfile

+1-1
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
ARG JDK_VERSION=14.0.2
1+
ARG JDK_VERSION=17.0.1
22
FROM openjdk:${JDK_VERSION}-slim AS worker
33

44
ARG DOCKER_BUILD_ARCH=amd64

airbyte-workers/build.gradle

+1-1
Original file line numberDiff line numberDiff line change
@@ -31,7 +31,7 @@ dependencies {
3131
implementation project(':airbyte-protocol:models')
3232
implementation project(':airbyte-scheduler:persistence')
3333

34-
testImplementation 'org.mockito:mockito-inline:2.13.0'
34+
testImplementation 'org.mockito:mockito-inline:4.0.0'
3535
testImplementation 'org.postgresql:postgresql:42.2.18'
3636
testImplementation "org.flywaydb:flyway-core:7.14.0"
3737
testImplementation 'io.temporal:temporal-testing:1.6.0'

build.gradle

+6-6
Original file line numberDiff line numberDiff line change
@@ -14,7 +14,7 @@ buildscript {
1414
plugins {
1515
id 'base'
1616
id 'pmd'
17-
id 'com.diffplug.spotless' version '5.7.0'
17+
id 'com.diffplug.spotless' version '6.0.0'
1818
id 'com.github.hierynomus.license' version '0.16.1'
1919
}
2020

@@ -103,7 +103,7 @@ spotless {
103103

104104
importOrder()
105105

106-
eclipse('4.16.0').configFile(rootProject.file('tools/gradle/codestyle/java-google-style.xml'))
106+
eclipse('4.21.0').configFile(rootProject.file('tools/gradle/codestyle/java-google-style.xml'))
107107

108108
licenseHeaderFile createJavaLicenseWith(rootProject.file('LICENSE_SHORT'))
109109
removeUnusedImports()
@@ -132,7 +132,7 @@ def Task getDockerBuildTask(String artifactName, String projectDir) {
132132
def buildPlatform = System.getenv('DOCKER_BUILD_PLATFORM') ?: 'linux/amd64'
133133
def alpineImage = System.getenv('ALPINE_IMAGE') ?: 'alpine:3.4'
134134
def postgresImage = System.getenv('POSTGRES_IMAGE') ?: 'postgres:13-alpine'
135-
def jdkVersion = System.getenv('JDK_VERSION') ?: '14.0.2'
135+
def jdkVersion = System.getenv('JDK_VERSION') ?: '17.0.1'
136136
def buildArch = System.getenv('DOCKER_BUILD_ARCH') ?: 'amd64'
137137

138138
inputDir = file("$projectDir/build/docker")
@@ -178,8 +178,8 @@ subprojects {
178178
apply plugin: 'java'
179179
apply plugin: 'jacoco'
180180

181-
sourceCompatibility = JavaVersion.VERSION_14
182-
targetCompatibility = JavaVersion.VERSION_14
181+
sourceCompatibility = JavaVersion.VERSION_17
182+
targetCompatibility = JavaVersion.VERSION_17
183183

184184
repositories {
185185
mavenCentral()
@@ -300,7 +300,7 @@ subprojects {
300300
testRuntimeOnly 'org.junit.jupiter:junit-jupiter-engine:5.7.2'
301301
testImplementation 'org.junit.jupiter:junit-jupiter-api:5.7.2'
302302
testImplementation 'org.junit.jupiter:junit-jupiter-params:5.7.2'
303-
testImplementation 'org.mockito:mockito-junit-jupiter:3.12.4'
303+
testImplementation 'org.mockito:mockito-junit-jupiter:4.0.0'
304304
testImplementation 'org.assertj:assertj-core:3.21.0'
305305

306306
}

docs/connector-development/testing-connectors/standard-source-tests.md

-32
Original file line numberDiff line numberDiff line change
@@ -2,35 +2,3 @@
22

33
Test methods start with `test`. Other methods are internal helpers in the java class implementing the test suite.
44

5-
## testGetSpec
6-
7-
Verify that a spec operation issued to the connector returns a valid spec.
8-
9-
## testCheckConnection
10-
11-
Verify that a check operation issued to the connector with the input config file returns a success response.
12-
13-
## testDiscover
14-
15-
Verifies when a discover operation is run on the connector using the given config file, a valid catalog is output by the connector.
16-
17-
## testFullRefreshRead
18-
19-
Configuring all streams in the input catalog to full refresh mode, verifies that a read operation produces some RECORD messages.
20-
21-
## testIdenticalFullRefreshes
22-
23-
Configuring all streams in the input catalog to full refresh mode, performs two read operations on all streams which support full refresh syncs. It then verifies that the RECORD messages output from both were identical.
24-
25-
## testIncrementalSyncWithState
26-
27-
This test verifies that all streams in the input catalog which support incremental sync can do so correctly. It does this by running two read operations on the connector's Docker image: the first takes the configured catalog and config provided to this test as input. It then verifies that the sync produced a non-zero number of RECORD and STATE messages. The second read takes the same catalog and config used in the first test, plus the last STATE message output by the first read operation as the input state file. It verifies that no records are produced \(since we read all records in the first sync\). This test is performed only for streams which support incremental. Streams which do not support incremental sync are ignored. If no streams in the input catalog support incremental sync, this test is skipped.
28-
29-
## testEmptyStateIncrementalIdenticalToFullRefresh
30-
31-
If the source does not support incremental sync, this test is skipped. Otherwise, this test runs two syncs: one where all streams provided in the input catalog sync in full refresh mode, and another where all the streams which in the input catalog which support incremental, sync in incremental mode \(streams which don't support incremental sync in full refresh mode\). Then, the test asserts that the two syncs produced the same RECORD messages. Any other type of message is disregarded.
32-
33-
## testEntrypointEnvVar
34-
35-
In order to launch a source on Kubernetes in a pod, we need to be able to wrap the entrypoint. The source connector must specify its entrypoint in the AIRBYTE_ENTRYPOINT variable. This test ensures that the entrypoint environment variable is set.
36-

0 commit comments

Comments
 (0)