Error Loop from Removing Relation

[mertalpt]

Steps to reproduce

1. Deploy kafka-k8s (3/stable, revno. 56) and zookeeper-k8s
2. Deploy a Kafka interface consumer
3. Relate everything together
4. Break the relation between kafka-k8s and the consumer charm

Note that the issue might stem from my charm in development improperly consuming the Kafka interface.

Expected behavior

Relation is broken and kafka-k8s settles to active & idle.

Actual behavior

Error loop in kafka-k8s.

Versions

Operating system: 22.04 (inside multipass with the charm-dev image)

Juju CLI: 3.5.1-genericlinux-amd64

Juju agent: 3.5.1

Charm revision: 56

microk8s: 1.29.5 revno 6884

Log output

Juju debug log:

unit-kafka-k8s-0: 15:36:21 ERROR unit.kafka-k8s/0.juju-log kafka-client:18: Uncaught exception while in charm code:
Traceback (most recent call last):
  File "/var/lib/juju/agents/unit-kafka-k8s-0/charm/./src/charm.py", line 333, in <module>
    main(KafkaCharm)
  File "/var/lib/juju/agents/unit-kafka-k8s-0/charm/venv/ops/main.py", line 456, in main
    _emit_charm_event(charm, dispatcher.event_name)
  File "/var/lib/juju/agents/unit-kafka-k8s-0/charm/venv/ops/main.py", line 144, in _emit_charm_event
    event_to_emit.emit(*args, **kwargs)
  File "/var/lib/juju/agents/unit-kafka-k8s-0/charm/venv/ops/framework.py", line 351, in emit
    framework._emit(event)
  File "/var/lib/juju/agents/unit-kafka-k8s-0/charm/venv/ops/framework.py", line 853, in _emit
    self._reemit(event_path)
  File "/var/lib/juju/agents/unit-kafka-k8s-0/charm/venv/ops/framework.py", line 943, in _reemit
    custom_handler(event)
  File "/var/lib/juju/agents/unit-kafka-k8s-0/charm/src/events/provider.py", line 124, in _on_relation_broken
    self.charm.auth_manager.delete_user(username=username)
  File "/var/lib/juju/agents/unit-kafka-k8s-0/charm/src/managers/auth.py", line 194, in delete_user
    self.workload.run_bin_command(bin_keyword="configs", bin_args=command)
  File "/var/lib/juju/agents/unit-kafka-k8s-0/charm/src/workload.py", line 104, in run_bin_command
    return self.exec(command=command, env=parsed_opts or None)
  File "/var/lib/juju/agents/unit-kafka-k8s-0/charm/src/workload.py", line 71, in exec
    raise e
  File "/var/lib/juju/agents/unit-kafka-k8s-0/charm/src/workload.py", line 67, in exec
    output, _ = process.wait_output()
  File "/var/lib/juju/agents/unit-kafka-k8s-0/charm/venv/ops/pebble.py", line 1540, in wait_output
    raise ExecError[AnyStr](self._command, exit_code, out_value, err_value)
ops.pebble.ExecError: non-zero exit code 1 executing ['/opt/kafka/bin/kafka-configs.sh', '--bootstrap-server=kafka-k8s-0.kafka-k8s-endpoints:9092', '--command-config=/etc/kafka/client.properties', '--alter', '--entity-type=users', '--entity-name=relation-18', '--delete-config=SCRAM-SHA-512'], stdout="Error while executing config command with args '--bootstrap-server=kafka-k8s-0.kafka-k8s-endpoints:9092 --command-config=/etc/kafka/client.properties --alter --entity-type=users --entity-name=relation-18 --delete-config=SCRAM-SHA-512'\njava.util.concurrent.ExecutionException: org.apache.kafka.common.errors.ResourceNotFoundException: Attempt to delete a user credential that does not exist\n\tat java.base/java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:396)\n\tat java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2096)\n\tat org.apache.kafka.common.internals.KafkaFutureImpl.get(KafkaFutureImpl.java:180)\n\tat kafka.admin.ConfigCommand$.alterUserScramCredentialConfigs(ConfigCommand.scala:466)\n\tat kafka.admin.ConfigCommand$.alterConfig(ConfigCommand.scala:429)\n\tat kafka.admin.ConfigCommand$.processCommand(ConfigCommand.scala:328)\n\tat kafka.admin.ConfigCommand$.main(ConfigCommand.scala:97)\n\tat kafka.admin.ConfigCommand.main(ConfigCommand.scala)\nCaused by: org.apache.kafka.co" [truncated]
unit-kafka-k8s-0: 15:36:21 ERROR juju.worker.uniter.operation hook "kafka-client-relation-broken" (via hook dispatching script: dispatch) failed: exit status 1

Additional context

Original thread

[github-actions]

https://warthogs.atlassian.net/browse/DPE-4818

[zmraul]

Problem is located in this check. Apparently it's not working.

[mertalpt]

@zmraul That check looks into stderr while the debug logs indicate that the message is printed to stdout. Should be a simple fix if that is indeed the problem.