Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

ERROR[m i.a.w.g.DefaultCheckConnectionWorker(run):125 - Unexpected error while checking connection: io.airbyte.workers.exception.WorkerException: null #32459

Closed
goldengrisha opened this issue Nov 13, 2023 · 2 comments
Closed

ERROR[m i.a.w.g.DefaultCheckConnectionWorker(run):125 - Unexpected error while checking connection: io.airbyte.workers.exception.WorkerException: null #32459

goldengrisha opened this issue Nov 13, 2023 · 2 comments
Labels
area/platform issues related to the platform community Stale team/platform-move type/bug Something isn't working

Comments

@goldengrisha
Copy link

What method are you using to run Airbyte?

Kubernetes

Platform Version or Helm Chart Version

0.44.2

What step the error happened?

During the Sync

Revelant information

Hello, please help, it worked for us for 1 month but now it doesn't.

Relevant log output

2023-11-13 13:46:53 �[32mINFO�[m i.a.w.t.TemporalAttemptExecution(get):138 - Cloud storage job log path: /workspace/376/0/logs.log
2023-11-13 13:46:53 �[32mINFO�[m i.a.w.t.TemporalAttemptExecution(get):141 - Executing worker wrapper. Airbyte version: 0.44.2
2023-11-13 13:46:53 �[32mINFO�[m i.a.a.c.AirbyteApiClient(retryWithJitter):179 - Attempt 0 to save workflow id for cancellation
2023-11-13 13:46:53 �[32mINFO�[m i.a.c.EnvConfigs(getEnvOrDefault):1222 - Using default value for environment variable SIDECAR_KUBE_CPU_LIMIT: '2.0'
2023-11-13 13:46:53 �[32mINFO�[m i.a.c.i.LineGobbler(voidCall):149 - 
2023-11-13 13:46:53 �[32mINFO�[m i.a.c.EnvConfigs(getEnvOrDefault):1222 - Using default value for environment variable SOCAT_KUBE_CPU_LIMIT: '2.0'
2023-11-13 13:46:53 �[32mINFO�[m i.a.c.i.LineGobbler(voidCall):149 - ----- START CHECK -----
2023-11-13 13:46:53 �[32mINFO�[m i.a.c.EnvConfigs(getEnvOrDefault):1222 - Using default value for environment variable SIDECAR_KUBE_CPU_REQUEST: '0.1'
2023-11-13 13:46:53 �[32mINFO�[m i.a.c.i.LineGobbler(voidCall):149 - 
2023-11-13 13:46:53 �[32mINFO�[m i.a.c.EnvConfigs(getEnvOrDefault):1222 - Using default value for environment variable SOCAT_KUBE_CPU_REQUEST: '0.1'
2023-11-13 13:46:53 �[32mINFO�[m i.a.c.EnvConfigs(getEnvOrDefault):1222 - Using default value for environment variable LAUNCHDARKLY_KEY: ''
2023-11-13 13:46:53 �[32mINFO�[m i.a.c.EnvConfigs(getEnvOrDefault):1222 - Using default value for environment variable FEATURE_FLAG_CLIENT: ''
2023-11-13 13:46:53 �[32mINFO�[m i.a.c.EnvConfigs(getEnvOrDefault):1222 - Using default value for environment variable OTEL_COLLECTOR_ENDPOINT: ''
2023-11-13 13:46:53 �[32mINFO�[m i.a.w.p.KubeProcessFactory(create):103 - Attempting to start pod = source-mssql-check-376-0-hgeco for airbyte/source-mssql:2.0.1 with resources io.airbyte.config.ResourceRequirements@6d6ce9a7[cpuRequest=,cpuLimit=,memoryRequest=,memoryLimit=,additionalProperties={}] and allowedHosts io.airbyte.config.AllowedHosts@1185a439[hosts=[10.1.4.100, *.datadoghq.com, *.datadoghq.eu, *.sentry.io],additionalProperties={}]
2023-11-13 13:49:22 �[32mINFO�[m i.a.w.t.TemporalAttemptExecution(get):138 - Cloud storage job log path: /workspace/376/0/logs.log
2023-11-13 13:49:03 �[32mINFO�[m i.a.w.t.TemporalAttemptExecution(lambda$getCancellationChecker$6):231 - Running sync worker cancellation...
2023-11-13 13:49:03 �[32mINFO�[m i.a.w.t.TemporalAttemptExecution(lambda$getCancellationChecker$6):235 - Interrupting worker thread...
2023-11-13 13:49:03 �[32mINFO�[m i.a.w.t.TemporalAttemptExecution(lambda$getCancellationChecker$6):238 - Cancelling completable future...
2023-11-13 13:49:03 �[33mWARN�[m i.a.c.t.CancellationHandler$TemporalCancellationHandler(checkAndHandleCancellation):60 - Job either timed out or was cancelled.
2023-11-13 13:49:03 �[1;31mERROR�[m i.a.w.g.DefaultCheckConnectionWorker(run):125 - Unexpected error while checking connection: 
io.airbyte.workers.exception.WorkerException: null
	at io.airbyte.workers.process.KubeProcessFactory.create(KubeProcessFactory.java:148) ~[io.airbyte-airbyte-commons-worker-0.44.2.jar:?]
	at io.airbyte.workers.process.AirbyteIntegrationLauncher.check(AirbyteIntegrationLauncher.java:106) ~[io.airbyte-airbyte-commons-worker-0.44.2.jar:?]
	at io.airbyte.workers.general.DefaultCheckConnectionWorker.run(DefaultCheckConnectionWorker.java:69) ~[io.airbyte-airbyte-commons-worker-0.44.2.jar:?]
	at io.airbyte.workers.general.DefaultCheckConnectionWorker.run(DefaultCheckConnectionWorker.java:43) ~[io.airbyte-airbyte-commons-worker-0.44.2.jar:?]
	at io.airbyte.workers.temporal.TemporalAttemptExecution.lambda$getWorkerThread$5(TemporalAttemptExecution.java:195) ~[io.airbyte-airbyte-workers-0.44.2.jar:?]
	at java.lang.Thread.run(Thread.java:1589) ~[?:?]
Caused by: java.lang.InterruptedException
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1681) ~[?:?]
	at java.util.concurrent.LinkedBlockingDeque.pollFirst(LinkedBlockingDeque.java:515) ~[?:?]
	at java.util.concurrent.LinkedBlockingDeque.poll(LinkedBlockingDeque.java:677) ~[?:?]
	at io.airbyte.workers.process.KubePortManagerSingleton.take(KubePortManagerSingleton.java:67) ~[io.airbyte-airbyte-commons-worker-0.44.2.jar:?]
	at io.airbyte.workers.process.KubeProcessFactory.create(KubeProcessFactory.java:106) ~[io.airbyte-airbyte-commons-worker-0.44.2.jar:?]
	... 5 more
2023-11-13 13:49:03 �[32mINFO�[m i.a.w.t.TemporalAttemptExecution(get):163 - Stopping cancellation check scheduling...
2023-11-13 13:49:03 �[33mWARN�[m i.t.i.a.ActivityTaskExecutors$BaseActivityTaskExecutor(execute):114 - Activity failure. ActivityId=6225e49e-dc45-3e69-93d5-3997cf330fd6, activityType=RunWithJobOutput, attempt=1
java.util.concurrent.CancellationException: null
	at java.util.concurrent.CompletableFuture.cancel(CompletableFuture.java:2510) ~[?:?]
	at io.airbyte.workers.temporal.TemporalAttemptExecution.lambda$getCancellationChecker$6(TemporalAttemptExecution.java:241) ~[io.airbyte-airbyte-workers-0.44.2.jar:?]
	at io.airbyte.commons.temporal.CancellationHandler$TemporalCancellationHandler.checkAndHandleCancellation(CancellationHandler.java:59) ~[io.airbyte-airbyte-commons-temporal-0.44.2.jar:?]
	at io.airbyte.workers.temporal.TemporalAttemptExecution.lambda$getCancellationChecker$7(TemporalAttemptExecution.java:244) ~[io.airbyte-airbyte-workers-0.44.2.jar:?]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:577) ~[?:?]
	at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:358) ~[?:?]
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?]
	at java.lang.Thread.run(Thread.java:1589) ~[?:?]
2023-11-13 13:49:03 �[32mINFO�[m i.a.c.i.LineGobbler(voidCall):149 - 
2023-11-13 13:49:03 �[32mINFO�[m i.a.c.i.LineGobbler(voidCall):149 - ----- END CHECK -----
2023-11-13 13:49:03 �[32mINFO�[m i.a.w.t.TemporalAttemptExecution(lambda$getWorkerThread$5):198 - Completing future exceptionally...
io.airbyte.workers.exception.WorkerException: Unexpected error while getting checking connection.
	at io.airbyte.workers.general.DefaultCheckConnectionWorker.run(DefaultCheckConnectionWorker.java:127) ~[io.airbyte-airbyte-commons-worker-0.44.2.jar:?]
	at io.airbyte.workers.general.DefaultCheckConnectionWorker.run(DefaultCheckConnectionWorker.java:43) ~[io.airbyte-airbyte-commons-worker-0.44.2.jar:?]
	at io.airbyte.workers.temporal.TemporalAttemptExecution.lambda$getWorkerThread$5(TemporalAttemptExecution.java:195) ~[io.airbyte-airbyte-workers-0.44.2.jar:?]
	at java.lang.Thread.run(Thread.java:1589) ~[?:?]
Caused by: io.airbyte.workers.exception.WorkerException
	at io.airbyte.workers.process.KubeProcessFactory.create(KubeProcessFactory.java:148) ~[io.airbyte-airbyte-commons-worker-0.44.2.jar:?]
	at io.airbyte.workers.process.AirbyteIntegrationLauncher.check(AirbyteIntegrationLauncher.java:106) ~[io.airbyte-airbyte-commons-worker-0.44.2.jar:?]
	at io.airbyte.workers.general.DefaultCheckConnectionWorker.run(DefaultCheckConnectionWorker.java:69) ~[io.airbyte-airbyte-commons-worker-0.44.2.jar:?]
	... 3 more
Caused by: java.lang.InterruptedException
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1681) ~[?:?]
	at java.util.concurrent.LinkedBlockingDeque.pollFirst(LinkedBlockingDeque.java:515) ~[?:?]
	at java.util.concurrent.LinkedBlockingDeque.poll(LinkedBlockingDeque.java:677) ~[?:?]
	at io.airbyte.workers.process.KubePortManagerSingleton.take(KubePortManagerSingleton.java:67) ~[io.airbyte-airbyte-commons-worker-0.44.2.jar:?]
	at io.airbyte.workers.process.KubeProcessFactory.create(KubeProcessFactory.java:106) ~[io.airbyte-airbyte-commons-worker-0.44.2.jar:?]
	at io.airbyte.workers.process.AirbyteIntegrationLauncher.check(AirbyteIntegrationLauncher.java:106) ~[io.airbyte-airbyte-commons-worker-0.44.2.jar:?]
	at io.airbyte.workers.general.DefaultCheckConnectionWorker.run(DefaultCheckConnectionWorker.java:69) ~[io.airbyte-airbyte-commons-worker-0.44.2.jar:?]
	... 3 more
2023-11-13 13:49:03 �[32mINFO�[m i.a.c.i.LineGobbler(voidCall):149 - 
2023-11-13 13:49:03 �[33mWARN�[m i.t.i.w.ActivityWorker$TaskHandlerImpl(logExceptionDuringResultReporting):365 - Failure during reporting of activity result to the server. ActivityId = 6225e49e-dc45-3e69-93d5-3997cf330fd6, ActivityType = RunWithJobOutput, WorkflowId=connection_manager_49e7d51e-6d9d-454d-92ec-058631f2b2e0, WorkflowType=ConnectionManagerWorkflow, RunId=c0374877-3388-44f2-b179-3bce806f8dc6
io.grpc.StatusRuntimeException: NOT_FOUND: invalid activityID or activity already timed out or invoking workflow is completed
	at io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:271) ~[grpc-stub-1.52.1.jar:1.52.1]
	at io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:252) ~[grpc-stub-1.52.1.jar:1.52.1]
	at io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:165) ~[grpc-stub-1.52.1.jar:1.52.1]
	at io.temporal.api.workflowservice.v1.WorkflowServiceGrpc$WorkflowServiceBlockingStub.respondActivityTaskFailed(WorkflowServiceGrpc.java:3866) ~[temporal-serviceclient-1.17.0.jar:?]
	at io.temporal.internal.worker.ActivityWorker$TaskHandlerImpl.lambda$sendReply$1(ActivityWorker.java:320) ~[temporal-sdk-1.17.0.jar:?]
	at io.temporal.internal.retryer.GrpcRetryer.lambda$retry$0(GrpcRetryer.java:52) ~[temporal-serviceclient-1.17.0.jar:?]
	at io.temporal.internal.retryer.GrpcSyncRetryer.retry(GrpcSyncRetryer.java:67) ~[temporal-serviceclient-1.17.0.jar:?]
	at io.temporal.internal.retryer.GrpcRetryer.retryWithResult(GrpcRetryer.java:60) ~[temporal-serviceclient-1.17.0.jar:?]
	at io.temporal.internal.retryer.GrpcRetryer.retry(GrpcRetryer.java:50) ~[temporal-serviceclient-1.17.0.jar:?]
	at io.temporal.internal.worker.ActivityWorker$TaskHandlerImpl.sendReply(ActivityWorker.java:315) ~[temporal-sdk-1.17.0.jar:?]
	at io.temporal.internal.worker.ActivityWorker$TaskHandlerImpl.handleActivity(ActivityWorker.java:252) ~[temporal-sdk-1.17.0.jar:?]
	at io.temporal.internal.worker.ActivityWorker$TaskHandlerImpl.handle(ActivityWorker.java:206) ~[temporal-sdk-1.17.0.jar:?]
	at io.temporal.internal.worker.ActivityWorker$TaskHandlerImpl.handle(ActivityWorker.java:179) ~[temporal-sdk-1.17.0.jar:?]
	at io.temporal.internal.worker.PollTaskExecutor.lambda$process$0(PollTaskExecutor.java:93) ~[temporal-sdk-1.17.0.jar:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?]
	at java.lang.Thread.run(Thread.java:1589) ~[?:?]
2023-11-13 13:49:22 �[32mINFO�[m i.a.w.t.TemporalAttemptExecution(get):141 - Executing worker wrapper. Airbyte version: 0.44.2
2023-11-13 13:49:22 �[32mINFO�[m i.a.a.c.AirbyteApiClient(retryWithJitter):179 - Attempt 0 to save workflow id for cancellation
2023-11-13 13:49:22 �[32mINFO�[m i.a.c.i.LineGobbler(voidCall):149 - 
2023-11-13 13:49:22 �[32mINFO�[m i.a.c.EnvConfigs(getEnvOrDefault):1222 - Using default value for environment variable SIDECAR_KUBE_CPU_LIMIT: '2.0'
2023-11-13 13:49:22 �[32mINFO�[m i.a.c.EnvConfigs(getEnvOrDefault):1222 - Using default value for environment variable SOCAT_KUBE_CPU_LIMIT: '2.0'
2023-11-13 13:49:22 �[32mINFO�[m i.a.c.i.LineGobbler(voidCall):149 - ----- START CHECK -----
2023-11-13 13:49:22 �[32mINFO�[m i.a.c.i.LineGobbler(voidCall):149 - 
2023-11-13 13:49:22 �[32mINFO�[m i.a.c.EnvConfigs(getEnvOrDefault):1222 - Using default value for environment variable SIDECAR_KUBE_CPU_REQUEST: '0.1'
2023-11-13 13:49:22 �[32mINFO�[m i.a.c.EnvConfigs(getEnvOrDefault):1222 - Using default value for environment variable SOCAT_KUBE_CPU_REQUEST: '0.1'
2023-11-13 13:49:22 �[32mINFO�[m i.a.c.EnvConfigs(getEnvOrDefault):1222 - Using default value for environment variable LAUNCHDARKLY_KEY: ''
2023-11-13 13:49:22 �[32mINFO�[m i.a.c.EnvConfigs(getEnvOrDefault):1222 - Using default value for environment variable FEATURE_FLAG_CLIENT: ''
2023-11-13 13:49:22 �[32mINFO�[m i.a.c.EnvConfigs(getEnvOrDefault):1222 - Using default value for environment variable OTEL_COLLECTOR_ENDPOINT: ''
2023-11-13 13:49:22 �[32mINFO�[m i.a.w.p.KubeProcessFactory(create):103 - Attempting to start pod = source-mssql-check-376-0-uladd for airbyte/source-mssql:2.0.1 with resources io.airbyte.config.ResourceRequirements@55fb4b09[cpuRequest=,cpuLimit=,memoryRequest=,memoryLimit=,additionalProperties={}] and allowedHosts io.airbyte.config.AllowedHosts@46f22378[hosts=[10.1.4.100, *.datadoghq.com, *.datadoghq.eu, *.sentry.io],additionalProperties={}]
2023-11-13 13:52:22 �[32mINFO�[m i.a.w.t.TemporalAttemptExecution(get):138 - Cloud storage job log path: /workspace/376/0/logs.log
2023-11-13 13:51:32 �[32mINFO�[m i.a.w.t.TemporalAttemptExecution(lambda$getCancellationChecker$6):231 - Running sync worker cancellation...
2023-11-13 13:51:32 �[32mINFO�[m i.a.w.t.TemporalAttemptExecution(lambda$getCancellationChecker$6):235 - Interrupting worker thread...
2023-11-13 13:51:32 �[32mINFO�[m i.a.w.t.TemporalAttemptExecution(lambda$getCancellationChecker$6):238 - Cancelling completable future...
2023-11-13 13:51:32 �[33mWARN�[m i.a.c.t.CancellationHandler$TemporalCancellationHandler(checkAndHandleCancellation):60 - Job either timed out or was cancelled.
2023-11-13 13:51:32 �[32mINFO�[m i.a.w.t.TemporalAttemptExecution(get):163 - Stopping cancellation check scheduling...
2023-11-13 13:51:32 �[1;31mERROR�[m i.a.w.g.DefaultCheckConnectionWorker(run):125 - Unexpected error while checking connection: 
io.airbyte.workers.exception.WorkerException: null
	at io.airbyte.workers.process.KubeProcessFactory.create(KubeProcessFactory.java:148) ~[io.airbyte-airbyte-commons-worker-0.44.2.jar:?]
	at io.airbyte.workers.process.AirbyteIntegrationLauncher.check(AirbyteIntegrationLauncher.java:106) ~[io.airbyte-airbyte-commons-worker-0.44.2.jar:?]
	at io.airbyte.workers.general.DefaultCheckConnectionWorker.run(DefaultCheckConnectionWorker.java:69) ~[io.airbyte-airbyte-commons-worker-0.44.2.jar:?]
	at io.airbyte.workers.general.DefaultCheckConnectionWorker.run(DefaultCheckConnectionWorker.java:43) ~[io.airbyte-airbyte-commons-worker-0.44.2.jar:?]
	at io.airbyte.workers.temporal.TemporalAttemptExecution.lambda$getWorkerThread$5(TemporalAttemptExecution.java:195) ~[io.airbyte-airbyte-workers-0.44.2.jar:?]
	at java.lang.Thread.run(Thread.java:1589) ~[?:?]
Caused by: java.lang.InterruptedException
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1681) ~[?:?]
	at java.util.concurrent.LinkedBlockingDeque.pollFirst(LinkedBlockingDeque.java:515) ~[?:?]
	at java.util.concurrent.LinkedBlockingDeque.poll(LinkedBlockingDeque.java:677) ~[?:?]
	at io.airbyte.workers.process.KubePortManagerSingleton.take(KubePortManagerSingleton.java:67) ~[io.airbyte-airbyte-commons-worker-0.44.2.jar:?]
	at io.airbyte.workers.process.KubeProcessFactory.create(KubeProcessFactory.java:106) ~[io.airbyte-airbyte-commons-worker-0.44.2.jar:?]
	... 5 more
2023-11-13 13:51:32 �[33mWARN�[m i.t.i.a.ActivityTaskExecutors$BaseActivityTaskExecutor(execute):114 - Activity failure. ActivityId=6225e49e-dc45-3e69-93d5-3997cf330fd6, activityType=RunWithJobOutput, attempt=2
java.util.concurrent.CancellationException: null
	at java.util.concurrent.CompletableFuture.cancel(CompletableFuture.java:2510) ~[?:?]
	at io.airbyte.workers.temporal.TemporalAttemptExecution.lambda$getCancellationChecker$6(TemporalAttemptExecution.java:241) ~[io.airbyte-airbyte-workers-0.44.2.jar:?]
	at io.airbyte.commons.temporal.CancellationHandler$TemporalCancellationHandler.checkAndHandleCancellation(CancellationHandler.java:59) ~[io.airbyte-airbyte-commons-temporal-0.44.2.jar:?]
	at io.airbyte.workers.temporal.TemporalAttemptExecution.lambda$getCancellationChecker$7(TemporalAttemptExecution.java:244) ~[io.airbyte-airbyte-workers-0.44.2.jar:?]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:577) ~[?:?]
	at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:358) ~[?:?]
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?]
	at java.lang.Thread.run(Thread.java:1589) ~[?:?]
2023-11-13 13:51:32 �[32mINFO�[m i.a.w.t.TemporalAttemptExecution(lambda$getWorkerThread$5):198 - Completing future exceptionally...
io.airbyte.workers.exception.WorkerException: Unexpected error while getting checking connection.
	at io.airbyte.workers.general.DefaultCheckConnectionWorker.run(DefaultCheckConnectionWorker.java:127) ~[io.airbyte-airbyte-commons-worker-0.44.2.jar:?]
	at io.airbyte.workers.general.DefaultCheckConnectionWorker.run(DefaultCheckConnectionWorker.java:43) ~[io.airbyte-airbyte-commons-worker-0.44.2.jar:?]
	at io.airbyte.workers.temporal.TemporalAttemptExecution.lambda$getWorkerThread$5(TemporalAttemptExecution.java:195) ~[io.airbyte-airbyte-workers-0.44.2.jar:?]
	at java.lang.Thread.run(Thread.java:1589) ~[?:?]
Caused by: io.airbyte.workers.exception.WorkerException
	at io.airbyte.workers.process.KubeProcessFactory.create(KubeProcessFactory.java:148) ~[io.airbyte-airbyte-commons-worker-0.44.2.jar:?]
	at io.airbyte.workers.process.AirbyteIntegrationLauncher.check(AirbyteIntegrationLauncher.java:106) ~[io.airbyte-airbyte-commons-worker-0.44.2.jar:?]
	at io.airbyte.workers.general.DefaultCheckConnectionWorker.run(DefaultCheckConnectionWorker.java:69) ~[io.airbyte-airbyte-commons-worker-0.44.2.jar:?]
	... 3 more
Caused by: java.lang.InterruptedException
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1681) ~[?:?]
	at java.util.concurrent.LinkedBlockingDeque.pollFirst(LinkedBlockingDeque.java:515) ~[?:?]
	at java.util.concurrent.LinkedBlockingDeque.poll(LinkedBlockingDeque.java:677) ~[?:?]
	at io.airbyte.workers.process.KubePortManagerSingleton.take(KubePortManagerSingleton.java:67) ~[io.airbyte-airbyte-commons-worker-0.44.2.jar:?]
	at io.airbyte.workers.process.KubeProcessFactory.create(KubeProcessFactory.java:106) ~[io.airbyte-airbyte-commons-worker-0.44.2.jar:?]
	at io.airbyte.workers.process.AirbyteIntegrationLauncher.check(AirbyteIntegrationLauncher.java:106) ~[io.airbyte-airbyte-commons-worker-0.44.2.jar:?]
	at io.airbyte.workers.general.DefaultCheckConnectionWorker.run(DefaultCheckConnectionWorker.java:69) ~[io.airbyte-airbyte-commons-worker-0.44.2.jar:?]
	... 3 more
2023-11-13 13:51:32 �[32mINFO�[m i.a.c.i.LineGobbler(voidCall):149 - 
2023-11-13 13:51:32 �[32mINFO�[m i.a.c.i.LineGobbler(voidCall):149 - ----- END CHECK -----
2023-11-13 13:51:32 �[32mINFO�[m i.a.c.i.LineGobbler(voidCall):149 - 
2023-11-13 13:51:32 �[33mWARN�[m i.t.i.w.ActivityWorker$TaskHandlerImpl(logExceptionDuringResultReporting):365 - Failure during reporting of activity result to the server. ActivityId = 6225e49e-dc45-3e69-93d5-3997cf330fd6, ActivityType = RunWithJobOutput, WorkflowId=connection_manager_49e7d51e-6d9d-454d-92ec-058631f2b2e0, WorkflowType=ConnectionManagerWorkflow, RunId=c0374877-3388-44f2-b179-3bce806f8dc6
io.grpc.StatusRuntimeException: NOT_FOUND: invalid activityID or activity already timed out or invoking workflow is completed
	at io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:271) ~[grpc-stub-1.52.1.jar:1.52.1]
	at io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:252) ~[grpc-stub-1.52.1.jar:1.52.1]
	at io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:165) ~[grpc-stub-1.52.1.jar:1.52.1]
	at io.temporal.api.workflowservice.v1.WorkflowServiceGrpc$WorkflowServiceBlockingStub.respondActivityTaskFailed(WorkflowServiceGrpc.java:3866) ~[temporal-serviceclient-1.17.0.jar:?]
	at io.temporal.internal.worker.ActivityWorker$TaskHandlerImpl.lambda$sendReply$1(ActivityWorker.java:320) ~[temporal-sdk-1.17.0.jar:?]
	at io.temporal.internal.retryer.GrpcRetryer.lambda$retry$0(GrpcRetryer.java:52) ~[temporal-serviceclient-1.17.0.jar:?]
	at io.temporal.internal.retryer.GrpcSyncRetryer.retry(GrpcSyncRetryer.java:67) ~[temporal-serviceclient-1.17.0.jar:?]
	at io.temporal.internal.retryer.GrpcRetryer.retryWithResult(GrpcRetryer.java:60) ~[temporal-serviceclient-1.17.0.jar:?]
	at io.temporal.internal.retryer.GrpcRetryer.retry(GrpcRetryer.java:50) ~[temporal-serviceclient-1.17.0.jar:?]
	at io.temporal.internal.worker.ActivityWorker$TaskHandlerImpl.sendReply(ActivityWorker.java:315) ~[temporal-sdk-1.17.0.jar:?]
	at io.temporal.internal.worker.ActivityWorker$TaskHandlerImpl.handleActivity(ActivityWorker.java:252) ~[temporal-sdk-1.17.0.jar:?]
	at io.temporal.internal.worker.ActivityWorker$TaskHandlerImpl.handle(ActivityWorker.java:206) ~[temporal-sdk-1.17.0.jar:?]
	at io.temporal.internal.worker.ActivityWorker$TaskHandlerImpl.handle(ActivityWorker.java:179) ~[temporal-sdk-1.17.0.jar:?]
	at io.temporal.internal.worker.PollTaskExecutor.lambda$process$0(PollTaskExecutor.java:93) ~[temporal-sdk-1.17.0.jar:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?]
	at java.lang.Thread.run(Thread.java:1589) ~[?:?]
2023-11-13 13:52:22 �[32mINFO�[m i.a.w.t.TemporalAttemptExecution(get):141 - Executing worker wrapper. Airbyte version: 0.44.2
2023-11-13 13:52:22 �[32mINFO�[m i.a.a.c.AirbyteApiClient(retryWithJitter):179 - Attempt 0 to save workflow id for cancellation
2023-11-13 13:52:22 �[32mINFO�[m i.a.c.i.LineGobbler(voidCall):149 - 
2023-11-13 13:52:22 �[32mINFO�[m i.a.c.EnvConfigs(getEnvOrDefault):1222 - Using default value for environment variable SIDECAR_KUBE_CPU_LIMIT: '2.0'
2023-11-13 13:52:22 �[32mINFO�[m i.a.c.EnvConfigs(getEnvOrDefault):1222 - Using default value for environment variable SOCAT_KUBE_CPU_LIMIT: '2.0'
2023-11-13 13:52:22 �[32mINFO�[m i.a.c.i.LineGobbler(voidCall):149 - ----- START CHECK -----
2023-11-13 13:52:22 �[32mINFO�[m i.a.c.i.LineGobbler(voidCall):149 - 
2023-11-13 13:52:22 �[32mINFO�[m i.a.c.EnvConfigs(getEnvOrDefault):1222 - Using default value for environment variable SIDECAR_KUBE_CPU_REQUEST: '0.1'
2023-11-13 13:52:22 �[32mINFO�[m i.a.c.EnvConfigs(getEnvOrDefault):1222 - Using default value for environment variable SOCAT_KUBE_CPU_REQUEST: '0.1'
2023-11-13 13:52:22 �[32mINFO�[m i.a.c.EnvConfigs(getEnvOrDefault):1222 - Using default value for environment variable LAUNCHDARKLY_KEY: ''
2023-11-13 13:52:22 �[32mINFO�[m i.a.c.EnvConfigs(getEnvOrDefault):1222 - Using default value for environment variable FEATURE_FLAG_CLIENT: ''
2023-11-13 13:52:22 �[32mINFO�[m i.a.c.EnvConfigs(getEnvOrDefault):1222 - Using default value for environment variable OTEL_COLLECTOR_ENDPOINT: ''
2023-11-13 13:52:22 �[32mINFO�[m i.a.w.p.KubeProcessFactory(create):103 - Attempting to start pod = source-mssql-check-376-0-vcsxu for airbyte/source-mssql:2.0.1 with resources io.airbyte.config.ResourceRequirements@72c0c749[cpuRequest=,cpuLimit=,memoryRequest=,memoryLimit=,additionalProperties={}] and allowedHosts io.airbyte.config.AllowedHosts@1652e0df[hosts=[10.1.4.100, *.datadoghq.com, *.datadoghq.eu, *.sentry.io],additionalProperties={}]
2023-11-13 13:54:32 �[32mINFO�[m i.a.w.t.TemporalAttemptExecution(lambda$getCancellationChecker$6):231 - Running sync worker cancellation...
2023-11-13 13:54:32 �[32mINFO�[m i.a.w.t.TemporalAttemptExecution(lambda$getCancellationChecker$6):235 - Interrupting worker thread...
2023-11-13 13:54:32 �[32mINFO�[m i.a.w.t.TemporalAttemptExecution(lambda$getCancellationChecker$6):238 - Cancelling completable future...
2023-11-13 13:54:32 �[33mWARN�[m i.a.c.t.CancellationHandler$TemporalCancellationHandler(checkAndHandleCancellation):60 - Job either timed out or was cancelled.
2023-11-13 13:54:32 �[32mINFO�[m i.a.w.t.TemporalAttemptExecution(get):163 - Stopping cancellation check scheduling...
2023-11-13 13:54:32 �[1;31mERROR�[m i.a.w.g.DefaultCheckConnectionWorker(run):125 - Unexpected error while checking connection: 
io.airbyte.workers.exception.WorkerException: null
	at io.airbyte.workers.process.KubeProcessFactory.create(KubeProcessFactory.java:148) ~[io.airbyte-airbyte-commons-worker-0.44.2.jar:?]
	at io.airbyte.workers.process.AirbyteIntegrationLauncher.check(AirbyteIntegrationLauncher.java:106) ~[io.airbyte-airbyte-commons-worker-0.44.2.jar:?]
	at io.airbyte.workers.general.DefaultCheckConnectionWorker.run(DefaultCheckConnectionWorker.java:69) ~[io.airbyte-airbyte-commons-worker-0.44.2.jar:?]
	at io.airbyte.workers.general.DefaultCheckConnectionWorker.run(DefaultCheckConnectionWorker.java:43) ~[io.airbyte-airbyte-commons-worker-0.44.2.jar:?]
	at io.airbyte.workers.temporal.TemporalAttemptExecution.lambda$getWorkerThread$5(TemporalAttemptExecution.java:195) ~[io.airbyte-airbyte-workers-0.44.2.jar:?]
	at java.lang.Thread.run(Thread.java:1589) ~[?:?]
Caused by: java.lang.InterruptedException
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1681) ~[?:?]
	at java.util.concurrent.LinkedBlockingDeque.pollFirst(LinkedBlockingDeque.java:515) ~[?:?]
	at java.util.concurrent.LinkedBlockingDeque.poll(LinkedBlockingDeque.java:677) ~[?:?]
	at io.airbyte.workers.process.KubePortManagerSingleton.take(KubePortManagerSingleton.java:67) ~[io.airbyte-airbyte-commons-worker-0.44.2.jar:?]
	at io.airbyte.workers.process.KubeProcessFactory.create(KubeProcessFactory.java:106) ~[io.airbyte-airbyte-commons-worker-0.44.2.jar:?]
	... 5 more
2023-11-13 13:54:32 �[33mWARN�[m i.t.i.a.ActivityTaskExecutors$BaseActivityTaskExecutor(execute):114 - Activity failure. ActivityId=6225e49e-dc45-3e69-93d5-3997cf330fd6, activityType=RunWithJobOutput, attempt=3
java.util.concurrent.CancellationException: null
	at java.util.concurrent.CompletableFuture.cancel(CompletableFuture.java:2510) ~[?:?]
	at io.airbyte.workers.temporal.TemporalAttemptExecution.lambda$getCancellationChecker$6(TemporalAttemptExecution.java:241) ~[io.airbyte-airbyte-workers-0.44.2.jar:?]
	at io.airbyte.commons.temporal.CancellationHandler$TemporalCancellationHandler.checkAndHandleCancellation(CancellationHandler.java:59) ~[io.airbyte-airbyte-commons-temporal-0.44.2.jar:?]
	at io.airbyte.workers.temporal.TemporalAttemptExecution.lambda$getCancellationChecker$7(TemporalAttemptExecution.java:244) ~[io.airbyte-airbyte-workers-0.44.2.jar:?]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:577) ~[?:?]
	at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:358) ~[?:?]
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?]
	at java.lang.Thread.run(Thread.java:1589) ~[?:?]
2023-11-13 13:54:32 �[32mINFO�[m i.a.w.t.TemporalAttemptExecution(lambda$getWorkerThread$5):198 - Completing future exceptionally...
io.airbyte.workers.exception.WorkerException: Unexpected error while getting checking connection.
	at io.airbyte.workers.general.DefaultCheckConnectionWorker.run(DefaultCheckConnectionWorker.java:127) ~[io.airbyte-airbyte-commons-worker-0.44.2.jar:?]
	at io.airbyte.workers.general.DefaultCheckConnectionWorker.run(DefaultCheckConnectionWorker.java:43) ~[io.airbyte-airbyte-commons-worker-0.44.2.jar:?]
	at io.airbyte.workers.temporal.TemporalAttemptExecution.lambda$getWorkerThread$5(TemporalAttemptExecution.java:195) ~[io.airbyte-airbyte-workers-0.44.2.jar:?]
	at java.lang.Thread.run(Thread.java:1589) ~[?:?]
Caused by: io.airbyte.workers.exception.WorkerException
	at io.airbyte.workers.process.KubeProcessFactory.create(KubeProcessFactory.java:148) ~[io.airbyte-airbyte-commons-worker-0.44.2.jar:?]
	at io.airbyte.workers.process.AirbyteIntegrationLauncher.check(AirbyteIntegrationLauncher.java:106) ~[io.airbyte-airbyte-commons-worker-0.44.2.jar:?]
	at io.airbyte.workers.general.DefaultCheckConnectionWorker.run(DefaultCheckConnectionWorker.java:69) ~[io.airbyte-airbyte-commons-worker-0.44.2.jar:?]
	... 3 more
Caused by: java.lang.InterruptedException
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1681) ~[?:?]
	at java.util.concurrent.LinkedBlockingDeque.pollFirst(LinkedBlockingDeque.java:515) ~[?:?]
	at java.util.concurrent.LinkedBlockingDeque.poll(LinkedBlockingDeque.java:677) ~[?:?]
	at io.airbyte.workers.process.KubePortManagerSingleton.take(KubePortManagerSingleton.java:67) ~[io.airbyte-airbyte-commons-worker-0.44.2.jar:?]
	at io.airbyte.workers.process.KubeProcessFactory.create(KubeProcessFactory.java:106) ~[io.airbyte-airbyte-commons-worker-0.44.2.jar:?]
	at io.airbyte.workers.process.AirbyteIntegrationLauncher.check(AirbyteIntegrationLauncher.java:106) ~[io.airbyte-airbyte-commons-worker-0.44.2.jar:?]
	at io.airbyte.workers.general.DefaultCheckConnectionWorker.run(DefaultCheckConnectionWorker.java:69) ~[io.airbyte-airbyte-commons-worker-0.44.2.jar:?]
	... 3 more
2023-11-13 13:54:32 �[32mINFO�[m i.a.c.i.LineGobbler(voidCall):149 - 
2023-11-13 13:54:32 �[32mINFO�[m i.a.c.i.LineGobbler(voidCall):149 - ----- END CHECK -----
2023-11-13 13:54:32 �[32mINFO�[m i.a.c.i.LineGobbler(voidCall):149 - 
2023-11-13 13:54:32 �[33mWARN�[m i.t.i.w.ActivityWorker$TaskHandlerImpl(logExceptionDuringResultReporting):365 - Failure during reporting of activity result to the server. ActivityId = 6225e49e-dc45-3e69-93d5-3997cf330fd6, ActivityType = RunWithJobOutput, WorkflowId=connection_manager_49e7d51e-6d9d-454d-92ec-058631f2b2e0, WorkflowType=ConnectionManagerWorkflow, RunId=c0374877-3388-44f2-b179-3bce806f8dc6
io.grpc.StatusRuntimeException: NOT_FOUND: invalid activityID or activity already timed out or invoking workflow is completed
	at io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:271) ~[grpc-stub-1.52.1.jar:1.52.1]
	at io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:252) ~[grpc-stub-1.52.1.jar:1.52.1]
	at io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:165) ~[grpc-stub-1.52.1.jar:1.52.1]
	at io.temporal.api.workflowservice.v1.WorkflowServiceGrpc$WorkflowServiceBlockingStub.respondActivityTaskFailed(WorkflowServiceGrpc.java:3866) ~[temporal-serviceclient-1.17.0.jar:?]
	at io.temporal.internal.worker.ActivityWorker$TaskHandlerImpl.lambda$sendReply$1(ActivityWorker.java:320) ~[temporal-sdk-1.17.0.jar:?]
	at io.temporal.internal.retryer.GrpcRetryer.lambda$retry$0(GrpcRetryer.java:52) ~[temporal-serviceclient-1.17.0.jar:?]
	at io.temporal.internal.retryer.GrpcSyncRetryer.retry(GrpcSyncRetryer.java:67) ~[temporal-serviceclient-1.17.0.jar:?]
	at io.temporal.internal.retryer.GrpcRetryer.retryWithResult(GrpcRetryer.java:60) ~[temporal-serviceclient-1.17.0.jar:?]
	at io.temporal.internal.retryer.GrpcRetryer.retry(GrpcRetryer.java:50) ~[temporal-serviceclient-1.17.0.jar:?]
	at io.temporal.internal.worker.ActivityWorker$TaskHandlerImpl.sendReply(ActivityWorker.java:315) ~[temporal-sdk-1.17.0.jar:?]
	at io.temporal.internal.worker.ActivityWorker$TaskHandlerImpl.handleActivity(ActivityWorker.java:252) ~[temporal-sdk-1.17.0.jar:?]
	at io.temporal.internal.worker.ActivityWorker$TaskHandlerImpl.handle(ActivityWorker.java:206) ~[temporal-sdk-1.17.0.jar:?]
	at io.temporal.internal.worker.ActivityWorker$TaskHandlerImpl.handle(ActivityWorker.java:179) ~[temporal-sdk-1.17.0.jar:?]
	at io.temporal.internal.worker.PollTaskExecutor.lambda$process$0(PollTaskExecutor.java:93) ~[temporal-sdk-1.17.0.jar:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?]
	at java.lang.Thread.run(Thread.java:1589) ~[?:?]
2023-11-13 13:56:22 �[32mINFO�[m i.a.w.t.TemporalAttemptExecution(get):138 - Cloud storage job log path: /workspace/376/0/logs.log
2023-11-13 13:56:22 �[32mINFO�[m i.a.w.t.TemporalAttemptExecution(get):141 - Executing worker wrapper. Airbyte version: 0.44.2
2023-11-13 13:56:22 �[32mINFO�[m i.a.a.c.AirbyteApiClient(retryWithJitter):179 - Attempt 0 to save workflow id for cancellation
2023-11-13 13:56:22 �[32mINFO�[m i.a.c.i.LineGobbler(voidCall):149 - 
2023-11-13 13:56:22 �[32mINFO�[m i.a.c.EnvConfigs(getEnvOrDefault):1222 - Using default value for environment variable SIDECAR_KUBE_CPU_LIMIT: '2.0'
2023-11-13 13:56:22 �[32mINFO�[m i.a.c.i.LineGobbler(voidCall):149 - ----- START CHECK -----
2023-11-13 13:56:22 �[32mINFO�[m i.a.c.EnvConfigs(getEnvOrDefault):1222 - Using default value for environment variable SOCAT_KUBE_CPU_LIMIT: '2.0'
2023-11-13 13:56:22 �[32mINFO�[m i.a.c.i.LineGobbler(voidCall):149 - 
2023-11-13 13:56:22 �[32mINFO�[m i.a.c.EnvConfigs(getEnvOrDefault):1222 - Using default value for environment variable SIDECAR_KUBE_CPU_REQUEST: '0.1'
2023-11-13 13:56:22 �[32mINFO�[m i.a.c.EnvConfigs(getEnvOrDefault):1222 - Using default value for environment variable SOCAT_KUBE_CPU_REQUEST: '0.1'
2023-11-13 13:56:22 �[32mINFO�[m i.a.c.EnvConfigs(getEnvOrDefault):1222 - Using default value for environment variable LAUNCHDARKLY_KEY: ''
2023-11-13 13:56:22 �[32mINFO�[m i.a.c.EnvConfigs(getEnvOrDefault):1222 - Using default value for environment variable FEATURE_FLAG_CLIENT: ''
2023-11-13 13:56:22 �[32mINFO�[m i.a.c.EnvConfigs(getEnvOrDefault):1222 - Using default value for environment variable OTEL_COLLECTOR_ENDPOINT: ''
2023-11-13 13:56:22 �[32mINFO�[m i.a.w.p.KubeProcessFactory(create):103 - Attempting to start pod = source-mssql-check-376-0-nvqai for airbyte/source-mssql:2.0.1 with resources io.airbyte.config.ResourceRequirements@7d05c67b[cpuRequest=,cpuLimit=,memoryRequest=,memoryLimit=,additionalProperties={}] and allowedHosts io.airbyte.config.AllowedHosts@2e0f9c1f[hosts=[10.1.4.100, *.datadoghq.com, *.datadoghq.eu, *.sentry.io],additionalProperties={}]
2023-11-13 13:58:32 �[32mINFO�[m i.a.w.t.TemporalAttemptExecution(lambda$getCancellationChecker$6):231 - Running sync worker cancellation...
2023-11-13 13:58:32 �[32mINFO�[m i.a.w.t.TemporalAttemptExecution(lambda$getCancellationChecker$6):235 - Interrupting worker thread...
2023-11-13 13:58:32 �[32mINFO�[m i.a.w.t.TemporalAttemptExecution(lambda$getCancellationChecker$6):238 - Cancelling completable future...
2023-11-13 13:58:32 �[33mWARN�[m i.a.c.t.CancellationHandler$TemporalCancellationHandler(checkAndHandleCancellation):60 - Job either timed out or was cancelled.
2023-11-13 13:58:32 �[1;31mERROR�[m i.a.w.g.DefaultCheckConnectionWorker(run):125 - Unexpected error while checking connection: 
io.airbyte.workers.exception.WorkerException: null
	at io.airbyte.workers.process.KubeProcessFactory.create(KubeProcessFactory.java:148) ~[io.airbyte-airbyte-commons-worker-0.44.2.jar:?]
	at io.airbyte.workers.process.AirbyteIntegrationLauncher.check(AirbyteIntegrationLauncher.java:106) ~[io.airbyte-airbyte-commons-worker-0.44.2.jar:?]
	at io.airbyte.workers.general.DefaultCheckConnectionWorker.run(DefaultCheckConnectionWorker.java:69) ~[io.airbyte-airbyte-commons-worker-0.44.2.jar:?]
	at io.airbyte.workers.general.DefaultCheckConnectionWorker.run(DefaultCheckConnectionWorker.java:43) ~[io.airbyte-airbyte-commons-worker-0.44.2.jar:?]
	at io.airbyte.workers.temporal.TemporalAttemptExecution.lambda$getWorkerThread$5(TemporalAttemptExecution.java:195) ~[io.airbyte-airbyte-workers-0.44.2.jar:?]
	at java.lang.Thread.run(Thread.java:1589) ~[?:?]
Caused by: java.lang.InterruptedException
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1681) ~[?:?]
	at java.util.concurrent.LinkedBlockingDeque.pollFirst(LinkedBlockingDeque.java:515) ~[?:?]
	at java.util.concurrent.LinkedBlockingDeque.poll(LinkedBlockingDeque.java:677) ~[?:?]
	at io.airbyte.workers.process.KubePortManagerSingleton.take(KubePortManagerSingleton.java:67) ~[io.airbyte-airbyte-commons-worker-0.44.2.jar:?]
	at io.airbyte.workers.process.KubeProcessFactory.create(KubeProcessFactory.java:106) ~[io.airbyte-airbyte-commons-worker-0.44.2.jar:?]
	... 5 more
2023-11-13 13:58:32 �[32mINFO�[m i.a.w.t.TemporalAttemptExecution(get):163 - Stopping cancellation check scheduling...
2023-11-13 13:58:32 �[33mWARN�[m i.t.i.a.ActivityTaskExecutors$BaseActivityTaskExecutor(execute):114 - Activity failure. ActivityId=6225e49e-dc45-3e69-93d5-3997cf330fd6, activityType=RunWithJobOutput, attempt=4
java.util.concurrent.CancellationException: null
	at java.util.concurrent.CompletableFuture.cancel(CompletableFuture.java:2510) ~[?:?]
	at io.airbyte.workers.temporal.TemporalAttemptExecution.lambda$getCancellationChecker$6(TemporalAttemptExecution.java:241) ~[io.airbyte-airbyte-workers-0.44.2.jar:?]
	at io.airbyte.commons.temporal.CancellationHandler$TemporalCancellationHandler.checkAndHandleCancellation(CancellationHandler.java:59) ~[io.airbyte-airbyte-commons-temporal-0.44.2.jar:?]
	at io.airbyte.workers.temporal.TemporalAttemptExecution.lambda$getCancellationChecker$7(TemporalAttemptExecution.java:244) ~[io.airbyte-airbyte-workers-0.44.2.jar:?]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:577) ~[?:?]
	at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:358) ~[?:?]
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?]
	at java.lang.Thread.run(Thread.java:1589) ~[?:?]
2023-11-13 13:58:32 �[32mINFO�[m i.a.c.i.LineGobbler(voidCall):149 - 
2023-11-13 13:58:32 �[32mINFO�[m i.a.w.t.TemporalAttemptExecution(lambda$getWorkerThread$5):198 - Completing future exceptionally...
io.airbyte.workers.exception.WorkerException: Unexpected error while getting checking connection.
	at io.airbyte.workers.general.DefaultCheckConnectionWorker.run(DefaultCheckConnectionWorker.java:127) ~[io.airbyte-airbyte-commons-worker-0.44.2.jar:?]
	at io.airbyte.workers.general.DefaultCheckConnectionWorker.run(DefaultCheckConnectionWorker.java:43) ~[io.airbyte-airbyte-commons-worker-0.44.2.jar:?]
	at io.airbyte.workers.temporal.TemporalAttemptExecution.lambda$getWorkerThread$5(TemporalAttemptExecution.java:195) ~[io.airbyte-airbyte-workers-0.44.2.jar:?]
	at java.lang.Thread.run(Thread.java:1589) ~[?:?]
Caused by: io.airbyte.workers.exception.WorkerException
	at io.airbyte.workers.process.KubeProcessFactory.create(KubeProcessFactory.java:148) ~[io.airbyte-airbyte-commons-worker-0.44.2.jar:?]
	at io.airbyte.workers.process.AirbyteIntegrationLauncher.check(AirbyteIntegrationLauncher.java:106) ~[io.airbyte-airbyte-commons-worker-0.44.2.jar:?]
	at io.airbyte.workers.general.DefaultCheckConnectionWorker.run(DefaultCheckConnectionWorker.java:69) ~[io.airbyte-airbyte-commons-worker-0.44.2.jar:?]
	... 3 more
Caused by: java.lang.InterruptedException
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1681) ~[?:?]
	at java.util.concurrent.LinkedBlockingDeque.pollFirst(LinkedBlockingDeque.java:515) ~[?:?]
	at java.util.concurrent.LinkedBlockingDeque.poll(LinkedBlockingDeque.java:677) ~[?:?]
	at io.airbyte.workers.process.KubePortManagerSingleton.take(KubePortManagerSingleton.java:67) ~[io.airbyte-airbyte-commons-worker-0.44.2.jar:?]
	at io.airbyte.workers.process.KubeProcessFactory.create(KubeProcessFactory.java:106) ~[io.airbyte-airbyte-commons-worker-0.44.2.jar:?]
	at io.airbyte.workers.process.AirbyteIntegrationLauncher.check(AirbyteIntegrationLauncher.java:106) ~[io.airbyte-airbyte-commons-worker-0.44.2.jar:?]
	at io.airbyte.workers.general.DefaultCheckConnectionWorker.run(DefaultCheckConnectionWorker.java:69) ~[io.airbyte-airbyte-commons-worker-0.44.2.jar:?]
	... 3 more
2023-11-13 13:58:32 �[32mINFO�[m i.a.c.i.LineGobbler(voidCall):149 - ----- END CHECK -----
2023-11-13 13:58:32 �[32mINFO�[m i.a.c.i.LineGobbler(voidCall):149 - 
2023-11-13 13:58:33 �[33mWARN�[m i.t.i.w.ActivityWorker$TaskHandlerImpl(logExceptionDuringResultReporting):365 - Failure during reporting of activity result to the server. ActivityId = 6225e49e-dc45-3e69-93d5-3997cf330fd6, ActivityType = RunWithJobOutput, WorkflowId=connection_manager_49e7d51e-6d9d-454d-92ec-058631f2b2e0, WorkflowType=ConnectionManagerWorkflow, RunId=c0374877-3388-44f2-b179-3bce806f8dc6
io.grpc.StatusRuntimeException: NOT_FOUND: invalid activityID or activity already timed out or invoking workflow is completed
	at io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:271) ~[grpc-stub-1.52.1.jar:1.52.1]
	at io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:252) ~[grpc-stub-1.52.1.jar:1.52.1]
	at io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:165) ~[grpc-stub-1.52.1.jar:1.52.1]
	at io.temporal.api.workflowservice.v1.WorkflowServiceGrpc$WorkflowServiceBlockingStub.respondActivityTaskFailed(WorkflowServiceGrpc.java:3866) ~[temporal-serviceclient-1.17.0.jar:?]
	at io.temporal.internal.worker.ActivityWorker$TaskHandlerImpl.lambda$sendReply$1(ActivityWorker.java:320) ~[temporal-sdk-1.17.0.jar:?]
	at io.temporal.internal.retryer.GrpcRetryer.lambda$retry$0(GrpcRetryer.java:52) ~[temporal-serviceclient-1.17.0.jar:?]
	at io.temporal.internal.retryer.GrpcSyncRetryer.retry(GrpcSyncRetryer.java:67) ~[temporal-serviceclient-1.17.0.jar:?]
	at io.temporal.internal.retryer.GrpcRetryer.retryWithResult(GrpcRetryer.java:60) ~[temporal-serviceclient-1.17.0.jar:?]
	at io.temporal.internal.retryer.GrpcRetryer.retry(GrpcRetryer.java:50) ~[temporal-serviceclient-1.17.0.jar:?]
	at io.temporal.internal.worker.ActivityWorker$TaskHandlerImpl.sendReply(ActivityWorker.java:315) ~[temporal-sdk-1.17.0.jar:?]
	at io.temporal.internal.worker.ActivityWorker$TaskHandlerImpl.handleActivity(ActivityWorker.java:252) ~[temporal-sdk-1.17.0.jar:?]
	at io.temporal.internal.worker.ActivityWorker$TaskHandlerImpl.handle(ActivityWorker.java:206) ~[temporal-sdk-1.17.0.jar:?]
	at io.temporal.internal.worker.ActivityWorker$TaskHandlerImpl.handle(ActivityWorker.java:179) ~[temporal-sdk-1.17.0.jar:?]
	at io.temporal.internal.worker.PollTaskExecutor.lambda$process$0(PollTaskExecutor.java:93) ~[temporal-sdk-1.17.0.jar:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?]
	at java.lang.Thread.run(Thread.java:1589) ~[?:?]
2023-11-13 14:02:22 �[32mINFO�[m i.a.w.t.TemporalAttemptExecution(get):138 - Cloud storage job log path: /workspace/376/0/logs.log
2023-11-13 14:02:22 �[32mINFO�[m i.a.w.t.TemporalAttemptExecution(get):141 - Executing worker wrapper. Airbyte version: 0.44.2
2023-11-13 14:02:22 �[32mINFO�[m i.a.a.c.AirbyteApiClient(retryWithJitter):179 - Attempt 0 to save workflow id for cancellation
2023-11-13 14:02:22 �[32mINFO�[m i.a.c.i.LineGobbler(voidCall):149 - 
2023-11-13 14:02:22 �[32mINFO�[m i.a.c.EnvConfigs(getEnvOrDefault):1222 - Using default value for environment variable SIDECAR_KUBE_CPU_LIMIT: '2.0'
2023-11-13 14:02:22 �[32mINFO�[m i.a.c.EnvConfigs(getEnvOrDefault):1222 - Using default value for environment variable SOCAT_KUBE_CPU_LIMIT: '2.0'
2023-11-13 14:02:22 �[32mINFO�[m i.a.c.i.LineGobbler(voidCall):149 - ----- START CHECK -----
2023-11-13 14:02:22 �[32mINFO�[m i.a.c.i.LineGobbler(voidCall):149 - 
2023-11-13 14:02:22 �[32mINFO�[m i.a.c.EnvConfigs(getEnvOrDefault):1222 - Using default value for environment variable SIDECAR_KUBE_CPU_REQUEST: '0.1'
2023-11-13 14:02:22 �[32mINFO�[m i.a.c.EnvConfigs(getEnvOrDefault):1222 - Using default value for environment variable SOCAT_KUBE_CPU_REQUEST: '0.1'
2023-11-13 14:02:22 �[32mINFO�[m i.a.c.EnvConfigs(getEnvOrDefault):1222 - Using default value for environment variable LAUNCHDARKLY_KEY: ''
2023-11-13 14:02:22 �[32mINFO�[m i.a.c.EnvConfigs(getEnvOrDefault):1222 - Using default value for environment variable FEATURE_FLAG_CLIENT: ''
2023-11-13 14:02:22 �[32mINFO�[m i.a.c.EnvConfigs(getEnvOrDefault):1222 - Using default value for environment variable OTEL_COLLECTOR_ENDPOINT: ''
2023-11-13 14:02:22 �[32mINFO�[m i.a.w.p.KubeProcessFactory(create):103 - Attempting to start pod = source-mssql-check-376-0-ophof for airbyte/source-mssql:2.0.1 with resources io.airbyte.config.ResourceRequirements@72c0c749[cpuRequest=,cpuLimit=,memoryRequest=,memoryLimit=,additionalProperties={}] and allowedHosts io.airbyte.config.AllowedHosts@2728b1eb[hosts=[10.1.4.100, *.datadoghq.com, *.datadoghq.eu, *.sentry.io],additionalProperties={}]
2023-11-13 14:04:32 �[32mINFO�[m i.a.w.t.TemporalAttemptExecution(lambda$getCancellationChecker$6):231 - Running sync worker cancellation...
2023-11-13 14:04:32 �[32mINFO�[m i.a.w.t.TemporalAttemptExecution(lambda$getCancellationChecker$6):235 - Interrupting worker thread...
2023-11-13 14:04:32 �[32mINFO�[m i.a.w.t.TemporalAttemptExecution(lambda$getCancellationChecker$6):238 - Cancelling completable future...
2023-11-13 14:04:32 �[33mWARN�[m i.a.c.t.CancellationHandler$TemporalCancellationHandler(checkAndHandleCancellation):60 - Job either timed out or was cancelled.
2023-11-13 14:04:32 �[1;31mERROR�[m i.a.w.g.DefaultCheckConnectionWorker(run):125 - Unexpected error while checking connection: 
io.airbyte.workers.exception.WorkerException: null
	at io.airbyte.workers.process.KubeProcessFactory.create(KubeProcessFactory.java:148) ~[io.airbyte-airbyte-commons-worker-0.44.2.jar:?]
	at io.airbyte.workers.process.AirbyteIntegrationLauncher.check(AirbyteIntegrationLauncher.java:106) ~[io.airbyte-airbyte-commons-worker-0.44.2.jar:?]
	at io.airbyte.workers.general.DefaultCheckConnectionWorker.run(DefaultCheckConnectionWorker.java:69) ~[io.airbyte-airbyte-commons-worker-0.44.2.jar:?]
	at io.airbyte.workers.general.DefaultCheckConnectionWorker.run(DefaultCheckConnectionWorker.java:43) ~[io.airbyte-airbyte-commons-worker-0.44.2.jar:?]
	at io.airbyte.workers.temporal.TemporalAttemptExecution.lambda$getWorkerThread$5(TemporalAttemptExecution.java:195) ~[io.airbyte-airbyte-workers-0.44.2.jar:?]
	at java.lang.Thread.run(Thread.java:1589) ~[?:?]
Caused by: java.lang.InterruptedException
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1681) ~[?:?]
	at java.util.concurrent.LinkedBlockingDeque.pollFirst(LinkedBlockingDeque.java:515) ~[?:?]
	at java.util.concurrent.LinkedBlockingDeque.poll(LinkedBlockingDeque.java:677) ~[?:?]
	at io.airbyte.workers.process.KubePortManagerSingleton.take(KubePortManagerSingleton.java:67) ~[io.airbyte-airbyte-commons-worker-0.44.2.jar:?]
	at io.airbyte.workers.process.KubeProcessFactory.create(KubeProcessFactory.java:106) ~[io.airbyte-airbyte-commons-worker-0.44.2.jar:?]
	... 5 more
2023-11-13 14:04:32 �[32mINFO�[m i.a.w.t.TemporalAttemptExecution(get):163 - Stopping cancellation check scheduling...
2023-11-13 14:04:32 �[33mWARN�[m i.t.i.a.ActivityTaskExecutors$BaseActivityTaskExecutor(execute):114 - Activity failure. ActivityId=6225e49e-dc45-3e69-93d5-3997cf330fd6, activityType=RunWithJobOutput, attempt=5
java.util.concurrent.CancellationException: null
	at java.util.concurrent.CompletableFuture.cancel(CompletableFuture.java:2510) ~[?:?]
	at io.airbyte.workers.temporal.TemporalAttemptExecution.lambda$getCancellationChecker$6(TemporalAttemptExecution.java:241) ~[io.airbyte-airbyte-workers-0.44.2.jar:?]
	at io.airbyte.commons.temporal.CancellationHandler$TemporalCancellationHandler.checkAndHandleCancellation(CancellationHandler.java:59) ~[io.airbyte-airbyte-commons-temporal-0.44.2.jar:?]
	at io.airbyte.workers.temporal.TemporalAttemptExecution.lambda$getCancellationChecker$7(TemporalAttemptExecution.java:244) ~[io.airbyte-airbyte-workers-0.44.2.jar:?]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:577) ~[?:?]
	at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:358) ~[?:?]
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?]
	at java.lang.Thread.run(Thread.java:1589) ~[?:?]
2023-11-13 14:04:32 �[32mINFO�[m i.a.w.t.TemporalAttemptExecution(lambda$getWorkerThread$5):198 - Completing future exceptionally...
io.airbyte.workers.exception.WorkerException: Unexpected error while getting checking connection.
	at io.airbyte.workers.general.DefaultCheckConnectionWorker.run(DefaultCheckConnectionWorker.java:127) ~[io.airbyte-airbyte-commons-worker-0.44.2.jar:?]
	at io.airbyte.workers.general.DefaultCheckConnectionWorker.run(DefaultCheckConnectionWorker.java:43) ~[io.airbyte-airbyte-commons-worker-0.44.2.jar:?]
	at io.airbyte.workers.temporal.TemporalAttemptExecution.lambda$getWorkerThread$5(TemporalAttemptExecution.java:195) ~[io.airbyte-airbyte-workers-0.44.2.jar:?]
	at java.lang.Thread.run(Thread.java:1589) ~[?:?]
Caused by: io.airbyte.workers.exception.WorkerException
	at io.airbyte.workers.process.KubeProcessFactory.create(KubeProcessFactory.java:148) ~[io.airbyte-airbyte-commons-worker-0.44.2.jar:?]
	at io.airbyte.workers.process.AirbyteIntegrationLauncher.check(AirbyteIntegrationLauncher.java:106) ~[io.airbyte-airbyte-commons-worker-0.44.2.jar:?]
	at io.airbyte.workers.general.DefaultCheckConnectionWorker.run(DefaultCheckConnectionWorker.java:69) ~[io.airbyte-airbyte-commons-worker-0.44.2.jar:?]
	... 3 more
Caused by: java.lang.InterruptedException
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1681) ~[?:?]
	at java.util.concurrent.LinkedBlockingDeque.pollFirst(LinkedBlockingDeque.java:515) ~[?:?]
	at java.util.concurrent.LinkedBlockingDeque.poll(LinkedBlockingDeque.java:677) ~[?:?]
	at io.airbyte.workers.process.KubePortManagerSingleton.take(KubePortManagerSingleton.java:67) ~[io.airbyte-airbyte-commons-worker-0.44.2.jar:?]
	at io.airbyte.workers.process.KubeProcessFactory.create(KubeProcessFactory.java:106) ~[io.airbyte-airbyte-commons-worker-0.44.2.jar:?]
	at io.airbyte.workers.process.AirbyteIntegrationLauncher.check(AirbyteIntegrationLauncher.java:106) ~[io.airbyte-airbyte-commons-worker-0.44.2.jar:?]
	at io.airbyte.workers.general.DefaultCheckConnectionWorker.run(DefaultCheckConnectionWorker.java:69) ~[io.airbyte-airbyte-commons-worker-0.44.2.jar:?]
	... 3 more
2023-11-13 14:04:32 �[32mINFO�[m i.a.c.i.LineGobbler(voidCall):149 - 
2023-11-13 14:04:32 �[32mINFO�[m i.a.c.i.LineGobbler(voidCall):149 - ----- END CHECK -----
2023-11-13 14:04:32 �[32mINFO�[m i.a.c.i.LineGobbler(voidCall):149 - 
2023-11-13 14:04:32 �[33mWARN�[m i.t.i.w.ActivityWorker$TaskHandlerImpl(logExceptionDuringResultReporting):365 - Failure during reporting of activity result to the server. ActivityId = 6225e49e-dc45-3e69-93d5-3997cf330fd6, ActivityType = RunWithJobOutput, WorkflowId=connection_manager_49e7d51e-6d9d-454d-92ec-058631f2b2e0, WorkflowType=ConnectionManagerWorkflow, RunId=c0374877-3388-44f2-b179-3bce806f8dc6
io.grpc.StatusRuntimeException: NOT_FOUND: invalid activityID or activity already timed out or invoking workflow is completed
	at io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:271) ~[grpc-stub-1.52.1.jar:1.52.1]
	at io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:252) ~[grpc-stub-1.52.1.jar:1.52.1]
	at io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:165) ~[grpc-stub-1.52.1.jar:1.52.1]
	at io.temporal.api.workflowservice.v1.WorkflowServiceGrpc$WorkflowServiceBlockingStub.respondActivityTaskFailed(WorkflowServiceGrpc.java:3866) ~[temporal-serviceclient-1.17.0.jar:?]
	at io.temporal.internal.worker.ActivityWorker$TaskHandlerImpl.lambda$sendReply$1(ActivityWorker.java:320) ~[temporal-sdk-1.17.0.jar:?]
	at io.temporal.internal.retryer.GrpcRetryer.lambda$retry$0(GrpcRetryer.java:52) ~[temporal-serviceclient-1.17.0.jar:?]
	at io.temporal.internal.retryer.GrpcSyncRetryer.retry(GrpcSyncRetryer.java:67) ~[temporal-serviceclient-1.17.0.jar:?]
	at io.temporal.internal.retryer.GrpcRetryer.retryWithResult(GrpcRetryer.java:60) ~[temporal-serviceclient-1.17.0.jar:?]
	at io.temporal.internal.retryer.GrpcRetryer.retry(GrpcRetryer.java:50) ~[temporal-serviceclient-1.17.0.jar:?]
	at io.temporal.internal.worker.ActivityWorker$TaskHandlerImpl.sendReply(ActivityWorker.java:315) ~[temporal-sdk-1.17.0.jar:?]
	at io.temporal.internal.worker.ActivityWorker$TaskHandlerImpl.handleActivity(ActivityWorker.java:252) ~[temporal-sdk-1.17.0.jar:?]
	at io.temporal.internal.worker.ActivityWorker$TaskHandlerImpl.handle(ActivityWorker.java:206) ~[temporal-sdk-1.17.0.jar:?]
	at io.temporal.internal.worker.ActivityWorker$TaskHandlerImpl.handle(ActivityWorker.java:179) ~[temporal-sdk-1.17.0.jar:?]
	at io.temporal.internal.worker.PollTaskExecutor.lambda$process$0(PollTaskExecutor.java:93) ~[temporal-sdk-1.17.0.jar:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?]
	at java.lang.Thread.run(Thread.java:1589) ~[?:?]
@goldengrisha goldengrisha added area/platform issues related to the platform needs-triage type/bug Something isn't working labels Nov 13, 2023
@octavia-squidington-iii
Copy link
Collaborator

At Airbyte, we seek to be clear about the project priorities and roadmap. This issue has not had any activity for 180 days, suggesting that it's not as critical as others. It's possible it has already been fixed. It is being marked as stale and will be closed in 20 days if there is no activity. To keep it open, please comment to let us know why it is important to you and if it is still reproducible on recent versions of Airbyte.

@octavia-squidington-iii
Copy link
Collaborator

This issue was closed because it has been inactive for 20 days since being marked as stale.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
area/platform issues related to the platform community Stale team/platform-move type/bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@timroes @goldengrisha @marcosmarxm @davinchia @octavia-squidington-iii