Unable to upgrade a connector from a private registry

[philippeboyd]

What method are you using to run Airbyte?

Docker

Platform Version or Helm Chart Version

0.50.34

What step the error happened?

Other

Revelant information

We used to be able to upgrade connectors from private registry as long as they are pulled on the machine prior.
Now we're unable to upgrade a connector from private registry we get the error

Error upgrading <Custom Connector Nam> to version 1.2.3: Internal Server Error: Get Spec job failed.

When executing docker images my image is present on the machine.

Something either changed in io.airbyte.config.persistence.ActorDefinitionVersionResolver and/or io.airbyte.config.specs.RemoteDefinitionsProvider

Relevant log output

2023-11-29 18:49:03 ERROR i.a.c.p.ActorDefinitionVersionResolver(fetchRemoteActorDefinitionVersion):91 - Failed to fetch registry entry for us-docker.pkg.dev/<custom_connector_path>:1.2.3
2023-11-29 18:49:04 ERROR i.a.s.a.ApiHelper(execute):44 - Unexpected Exception
java.lang.IllegalStateException: Get Spec job failed.
	at com.google.common.base.Preconditions.checkState(Preconditions.java:502) ~[guava-31.1-jre.jar:?]
	at io.airbyte.commons.server.converters.SpecFetcher.getSpecFromJob(SpecFetcher.java:17) ~[io.airbyte-airbyte-commons-server-0.50.34.jar:?]
	at io.airbyte.commons.server.handlers.helpers.ActorDefinitionHandlerHelper.getSpecForImage(ActorDefinitionHandlerHelper.java:149) ~[io.airbyte-airbyte-commons-server-0.50.34.jar:?]
	at io.airbyte.commons.server.handlers.helpers.ActorDefinitionHandlerHelper.defaultDefinitionVersionFromUpdate(ActorDefinitionHandlerHelper.java:122) ~[io.airbyte-airbyte-commons-server-0.50.34.jar:?]
	at io.airbyte.commons.server.handlers.SourceDefinitionsHandler.updateSourceDefinition(SourceDefinitionsHandler.java:284) ~[io.airbyte-airbyte-commons-server-0.50.34.jar:?]
	at io.airbyte.server.apis.SourceDefinitionApiController.lambda$updateSourceDefinition$9(SourceDefinitionApiController.java:157) ~[io.airbyte-airbyte-server-0.50.34.jar:?]
	at io.airbyte.server.apis.ApiHelper.execute(ApiHelper.java:27) ~[io.airbyte-airbyte-server-0.50.34.jar:?]
	at io.airbyte.server.apis.SourceDefinitionApiController.updateSourceDefinition(SourceDefinitionApiController.java:157) ~[io.airbyte-airbyte-server-0.50.34.jar:?]
	at io.airbyte.server.apis.$SourceDefinitionApiController$Definition$Exec.dispatch(Unknown Source) ~[io.airbyte-airbyte-server-0.50.34.jar:?]
	at io.micronaut.context.AbstractExecutableMethodsDefinition$DispatchedExecutableMethod.invoke(AbstractExecutableMethodsDefinition.java:371) ~[micronaut-inject-3.10.1.jar:3.10.1]
	at io.micronaut.context.DefaultBeanContext$4.invoke(DefaultBeanContext.java:594) ~[micronaut-inject-3.10.1.jar:3.10.1]
	at io.micronaut.web.router.AbstractRouteMatch.execute(AbstractRouteMatch.java:303) ~[micronaut-router-3.10.1.jar:3.10.1]
	at io.micronaut.web.router.RouteMatch.execute(RouteMatch.java:111) ~[micronaut-router-3.10.1.jar:3.10.1]
	at io.micronaut.http.context.ServerRequestContext.with(ServerRequestContext.java:103) ~[micronaut-http-3.10.1.jar:3.10.1]
	at io.micronaut.http.server.RouteExecutor.lambda$executeRoute$14(RouteExecutor.java:659) ~[micronaut-http-server-3.10.1.jar:3.10.1]
	at reactor.core.publisher.FluxDeferContextual.subscribe(FluxDeferContextual.java:49) ~[reactor-core-3.5.5.jar:3.5.5]
	at reactor.core.publisher.InternalFluxOperator.subscribe(InternalFluxOperator.java:62) ~[reactor-core-3.5.5.jar:3.5.5]
	at reactor.core.publisher.FluxSubscribeOn$SubscribeOnSubscriber.run(FluxSubscribeOn.java:194) ~[reactor-core-3.5.5.jar:3.5.5]
	at io.micronaut.reactive.reactor.instrument.ReactorInstrumentation.lambda$init$0(ReactorInstrumentation.java:62) ~[micronaut-runtime-3.10.1.jar:3.10.1]
	at reactor.core.scheduler.WorkerTask.call(WorkerTask.java:84) ~[reactor-core-3.5.5.jar:3.5.5]
	at reactor.core.scheduler.WorkerTask.call(WorkerTask.java:37) ~[reactor-core-3.5.5.jar:3.5.5]
	at io.micronaut.scheduling.instrument.InvocationInstrumenterWrappedCallable.call(InvocationInstrumenterWrappedCallable.java:53) ~[micronaut-context-3.10.1.jar:3.10.1]
	at java.util.concurrent.FutureTask.run(FutureTask.java:317) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?]
	at java.lang.Thread.run(Thread.java:1589) ~[?:?]

[mcornejo]

I had the same issue when I was trying to build the docker image using an apple silicon (and airbyte was running in an amd64). I fixed it by building the docker image using buildx: docker buildx build --platform linux/amd64 -t my_tag .

[philippeboyd]

Hi @mcornejo I'm building on macOS Intel (x86_64) and Airbyte is also deployed on a x86_64

The error is due to the fact that Airbyte still http requests the image from the private registry Failed to fetch registry entry ...

[oskarryn]

Seems that fetching registry entry fails even when pulling not from a private registry: #34198
I confirm to hit the same bug on 0.50.43 with private registry

[octavia-squidington-iii]

At Airbyte, we seek to be clear about the project priorities and roadmap. This issue has not had any activity for 180 days, suggesting that it's not as critical as others. It's possible it has already been fixed. It is being marked as stale and will be closed in 20 days if there is no activity. To keep it open, please comment to let us know why it is important to you and if it is still reproducible on recent versions of Airbyte.

[octavia-squidington-iii]

This issue was closed because it has been inactive for 20 days since being marked as stale.