CC @ejona86, he may have some comments about why this was added.

Oh, this is a different one than I thought. I thought you had discovered ManagedChannelOrphanWrapper, which is on the channel creation flow.

I'm surprised you noticed this one and not the ManagedChannelOrphanWrapper exception. Why would this one be more noticable?

Regarding ClientTransportLifecycleManager, I think it'd probably be better to just get rid of the getShutdownThrowable() API and have each callsite convert the status to an exception when necessary. I am surprised by your earlier PR though, because I feel like NettyClientHandler.channelInactive() would have always been called, and it calls getShutdownThrowable() unconditionally. So it seems making the throwable lazily initialized wouldn't have changed the frequency it was created.

We should make cancelPing(Throwable) become cancelPing(Status) to avoid that throwable creation. That leaves just createStream() as being a concern. But because the RPC is failing, it we'll probably be creating exceptions anyway. Not that great, but probably not that bad.

I'm surprised you noticed this one and not the ManagedChannelOrphanWrapper exception.

We did, but we disabled ENABLE_ALLOCATION_TRACKING. Looks like a debugging feature, not something that should be turned on in production.

I feel like NettyClientHandler.channelInactive() would have always been called, and it calls getShutdownThrowable() unconditionally

I'll only be able to check on Thursday, sorry.

I've been busy tracking other Exceptions allocations when closing a channel. It turns out there's also a bug in Netty, netty/netty#13569, whose consequences are even worse in grpc-netty when using shutdownNow because it causes a double channel closing (one from the graceful shutdown and one from the forceful one), hence 2 SslClosedEngineExceptions every time. See #10527 that's one of the causes.