HTTPCLIENT-2358 Implement a mutual authentication capable SPNEGO scheme #615

stoty · 2025-01-23T09:58:27Z

No description provided.

ok2c · 2025-01-23T12:51:47Z

@stoty I cannot contribute here much. My only wish, and I understand it is a big ask, to also create a compatibility test similar to those we have for BASIC / DIGEST with Squid and Apache HTTPD

https://github.com/apache/httpcomponents-client/tree/master/httpclient5-testing/src/test/java/org/apache/hc/client5/testing/compatibility

stoty · 2025-01-23T13:10:08Z

I was afraid you're going to say that...

There are two ways to go about that:

Try to find an docker image with a KDC and web server, and model the test on the Squid test
Initialize a local (non-dockerized) KDC with Kerby, start Jetty, and use that.

I don't know which of the two is more work, but I know where to copy the test setup from for the second one.

Do you have a preference ?

ok2c · 2025-01-23T13:22:09Z

I was afraid you're going to say that...

@stoty Kek.

Anyways, We already test for compatibility with Jetty in core by running it in a Docker container

https://github.com/apache/httpcomponents-core/blob/master/httpcore5-testing/src/test/java/org/apache/hc/core5/testing/compatibility/JettyCompatIT.java

I t would be my preferred option but I do not know how difficult it is to pack extra Jetty dependencies into a Docker container. I was also hoping Apache HTTPD or Ngnix might have Kerberos support.

If it is too much effort, disregard my request.

stoty · 2025-01-23T13:32:21Z

It is a reasonable request, and it IS helpful to catch any regressions, etc.
I will try to find and existing docker image that is close enough.

garydgregory · 2025-01-23T17:55:18Z

Apache Kerby implements Kerberos, you can get inspiration (copy) how they do it: https://directory.apache.org/kerby/

stoty · 2025-01-24T08:43:07Z

Kerby is great for in-JVM tests, but for dockerized tests it's probably easier to use MIT kerberos.

stoty · 2025-01-24T11:51:19Z

After poking around a bit, it seems that neither Apache Httpd, not Nginx supports SPNEGO out of the box.
While both have third party SPNEGO modules, creating a test image with a full working setup looks like a lot of work.

For now, I plan to make a local test with Kerby + Jetty, without Docker.

michael-o · 2025-01-28T12:21:29Z

A full coverage test contains JGSS (via Tomcat my SpnegoAuthenticator), MIT Kerberos (via mod_auth_gssapi), Microsoft Kerberos (via IIS (SSPI)). Virtually impossible to automate. I will do manual testing anyway.

I have everything in place at work to test.

michael-o

This is much less code review. Thanks. I already see some points.

httpclient5/src/main/java/org/apache/hc/client5/http/auth/MutualKerberosConfig.java

stoty · 2025-01-28T15:13:55Z

Thanks.
I've made the changes to MutualKerberosConfig.

While I don't use Tomcat, testing Jgss with Jetty is doable, in fact one of the ways I tested this was via the SPNEGO test in HttpCLient->Calcite Avatica->Phoenix PQS.
It does bring in a ton of extra test dependencies, and I couldn't really find where to fit the that integration test in the project.

Fix GSSCredential handling

stoty · 2025-02-20T23:10:14Z

I have added tests for direct connection to httpd + mod_auth_spnego, @michael-o .
The tests have immediately found a problem, which I also fixed.

The async client cannot use the Subject set with Subject.callAs to figure out the Kerberos credentials (probably due to the auth being run from a different thread)
Is that acceptable, or should we extract and store the Subject ?

Async still works if we create and set KerbersCredentials. (as in the new tests I added)

stoty · 2025-02-22T19:42:29Z

I have added tests for SPNEGO authentication for Squid.

I feel that this is ready now, @michael-o .

michael-o · 2025-02-26T10:51:50Z

I have added tests for direct connection to httpd + mod_auth_spnego, @michael-o . The tests have immediately found a problem, which I also fixed.

The async client cannot use the Subject set with Subject.callAs to figure out the Kerberos credentials (probably due to the auth being run from a different thread) Is that acceptable, or should we extract and store the Subject ?

Async still works if we create and set KerbersCredentials. (as in the new tests I added)

Which bug exactly?

httpclient5/src/main/java/org/apache/hc/client5/http/impl/auth/MutualGssSchemeBase.java

michael-o

I am picking this up again and want to complete next month.
Structural issues:

Since in both auth and impl.path packages contains multiple new classes there should be in respective new subpackages gss for tidiness.
Don't use the term "Kerberos" in classnames becausee we perform SPNEGO with GSS-API, correctly it should be called "Gss"
SPNEGO in class names should be written consistently, In PascalCase and not mixed case.
Drop the prefix "Mutual" in class names altogether because one can disable on request making it a contradiction and the other party could pontentially deny it.

stoty · 2025-02-27T13:13:57Z

I can do that, but that would mean the the three formerly mutual classes only differ in case and package from the old deprecated classes. I can see that causing a lot of gried for users down the line.

Since in both auth and impl.path packages contains multiple new classes there should be in respective new subpackages gss for tidiness.

Sure.

Don't use the term "Kerberos" in classnames becausee we perform SPNEGO with GSS-API, correctly it should be called "Gss"

Sure.

SPNEGO in class names should be written consistently, In PascalCase and not mixed case.

Only the old deprecated classes use SPNego, and fixing those would make it even easier to mix them up with the new ones. (apart from breaking backwards compatibility, if we care about that)

Drop the prefix "Mutual" in class names altogether because one can disable on request making it a contradiction and the other party could pontentially deny it.

I can do that, but then then the only difference between the old and new classes will be the packages and the case.
Machines don't care, but I see that causing grief for users.

stoty · 2025-02-27T13:50:21Z

Thanks for the review.
I have made the suggested changes, but I am not happy with how close the old new class names are.

I have also left KerberosCredentials alone, because it is used by both the new and old code.
Should I make an identical child class with GssCredentials name ?

michael-o · 2025-02-27T13:58:28Z

I can do that, but that would mean the the three formerly mutual classes only differ in case and package from the old deprecated classes. I can see that causing a lot of gried for users down the line.

Since in both auth and impl.path packages contains multiple new classes there should be in respective new subpackages gss for tidiness.

Sure.

Don't use the term "Kerberos" in classnames becausee we perform SPNEGO with GSS-API, correctly it should be called "Gss"

Sure.

SPNEGO in class names should be written consistently, In PascalCase and not mixed case.

Only the old deprecated classes use SPNego, and fixing those would make it even easier to mix them up with the new ones. (apart from breaking backwards compatibility, if we care about that)

Drop the prefix "Mutual" in class names altogether because one can disable on request making it a contradiction and the other party could pontentially deny it.

I can do that, but then then the only difference between the old and new classes will be the packages and the case. Machines don't care, but I see that causing grief for users.

Don't rename old ones, only new ones. If old ones are reused maye it would be better to copy them?

michael-o · 2025-02-27T13:59:12Z

Thanks for the review. I have made the suggested changes, but I am not happy with how close the old new class names are.

I have also left KerberosCredentials alone, because it is used by both the new and old code. Should I make an identical child class with GssCredentials name ?

Yes, please. Don't break old deprecated, but add new classes for the new impl.

stoty · 2025-02-27T14:22:25Z

I can do that, but then then the only difference between the old and new classes will be the packages and the case. Machines don't care, but I see that causing grief for users.

Don't rename old ones, only new ones. If old ones are reused maye it would be better to copy them?

I wasn't clear. The classes are different, but their names are almost identical.

As the Scheme is not enabled by default, users have to add some boilerplate code to enable SPNEGO at all.
It is now too easy to use the old SPNegoScheme instead of the new SpnegoScheme in that boilerplate IMO.

i have renamed KerberCredentials, and added a backwards compatibility subclass.

httpclient5/src/main/java/org/apache/hc/client5/http/auth/KerberosCredentials.java

httpclient5/src/main/java/org/apache/hc/client5/http/auth/gss/GssConfig.java

httpclient5/src/main/java/org/apache/hc/client5/http/auth/KerberosConfig.java

httpclient5/src/main/java/org/apache/hc/client5/http/auth/gss/GssConfig.java

httpclient5/src/main/java/org/apache/hc/client5/http/impl/auth/gss/GssSchemeBase.java

michael-o · 2025-02-27T15:29:00Z

httpclient5/src/main/java/org/apache/hc/client5/http/impl/auth/gss/GssSchemeBase.java

+            case TOKEN_SENT:
+                if (challenged) {
+                    state = State.TOKEN_READY;
+                } else if (mutualAuth) {


This is wrong. The only loop condition is gssContext.isEstablished(). The acceptor can send still a token, even if not mutual auth is peformed and even in an error condition an acceptor context can produce a token (e.g. MIT Kerberos), if necessary I can show code. HAve seen this years ago.
Note that requesting mutual auth is only a hint to GSS-API, wether the security context will provide is upto the mech.

How should we handle the case when we request mutual auth, but don't get it ?
Should we just fail, or should we split the option , one to request it, and one to require that it succeeds ?

Read https://datatracker.ietf.org/doc/html/rfc7546, there is even code depicting how you handle the loop. It should answer almost all questions.

The spec kind of leaves this to the implementation:

If a flag
was requested but is not available and that feature is necessary for
the application protocol, the initiator must destroy the security
context and not use the security context for application traffic.

If I interpret that losely, this means that we could split the mutualAuth, because the caller may ask for mutual Auth, but may not consider it necessary (though admittedly this not make a lot of sense).

If the initiator is expecting a context token (that is, the previous
call to GSS_Init_sec_context() returned GSS_S_CONTINUE_NEEDED) but
does not receive such a token within a reasonable amount of time
after transmitting the previous output_token to the acceptor, the
initiator should assume that the acceptor's state is invalid and fail
the GSS negotiation. Again, it may be appropriate for the initiator
to report this error condition to the acceptor via the application's
communication channel.

This one is even more interesting, as if we implemented this strictly, we could never work with Squid (and I expect that that there are many more similarly broken servers.)

I lean towards not making mutual authentication optional, as the user has no way to check the status, and if it is optional, then why even bother in the first place ? It would catch misconfigured/malicious servers that send an incorrect response, but not ones that just plain ignore the request.

On the other hand, I lean towards adding a strict flag to the config, so that we require that the GSS negotiation completes, so that we have a standards-compliant mode. This of course is implied when mutual Auth is requested.

I will read this one as well, check other impls and share my thoughts on this.

Yes, the current code should comply with that.
"ignoreMissingToken" is the non-strict flag (defaults to off), but I am open to a better name.

ignoreIncompleteSecurityContext?

I am not sure about the warning, that can be a really high volume, and the user has already disabled it explicitly.
I am currently logging a DEBUG message for this.

Lets go for WARNING and see whether we will receive complaints, then we still can reduce. I'd eager to see who is faulty, this will give everyone a chance to report to the producer of the crap.

The old scheme had also other issues beside the mutual auth.

So you are saying, if we request mutual auth we shall check for it after the context has been established and if not then we should fail? Including a switch to make it soft?

Yes. currently these are requestMutualAuth and requireMutualAuth.

I am not sure about the warning, that can be a really high volume, and the user has already disabled it explicitly.
I am currently logging a DEBUG message for this.

Lets go for WARNING and see whether we will receive complaints, then we still can reduce. I'd eager to see who is faulty, this will give everyone a chance to report to the producer of the crap.

FTR: I checked the source of Squid, man...
Look at https://github.com/squid-cache/squid/blob/0c90595b9d63930f07313a60eb3c7ec9859251bf/src/auth/negotiate/kerberos/negotiate_kerberos_auth.cc#L750-L754, completion is never really checked...
I also failed to find where the output token goes, it is simply not processed after it has been converted to Base 64, moreover they use a lot of raw Kerberos 5 APIs which aren't required at all, one can do everything through GSS-API these days, even obtaining the PAC data. For me: Squid is broken.

httpclient5/src/main/java/org/apache/hc/client5/http/impl/auth/gss/GssSchemeBase.java

httpclient5/src/main/java/org/apache/hc/client5/http/impl/auth/gss/SpnegoScheme.java

michael-o · 2025-02-27T15:37:15Z

httpclient5/src/main/java/org/apache/hc/client5/http/impl/auth/gss/SpnegoScheme.java

+
+    @Override
+    public boolean isConnectionBased() {
+        return true;


This is wrong and false. If there is only one roundtrip then you don't need to bind the GSS context to the connection otherwise you have to, see also https://github.com/gssapi/mod_auth_gssapi?tab=readme-ov-file#gssapiconnectionbound

Right, fixing.
Note that this was copied from the old one.
Should we fix it there as well ?

No, leave as is.

Sorry, I misread. We should fix the copy, not the original one. My bad!

Thinking about this again, This is not a problem when the completion done after one roundtrip. So I guess when the security context says continue needed than this needs to be bound to the connection. WDYT?

TBH I'm a bit hazy on what connectionBased does.

Looking at the main auth loop in ProtocolExec.execute(), It looks like AuthScheme (and with it GssContext) is bound to the whole authentication conversation until it either fails or succeeds, regardless of this setting.

Best I can tell, isConnectionBased is exclusively used for NTLM, where the server seems to bind the user authentication to the HTTP connection.

AFAIU, NTLM is the only auth that does this.
For Kerberos, we need to do full authentication for every single request (that's why it's so insanely slow and expensive, and that's why an AUTH cookie is usually created for the session after successfully authenticating).

Based on this, I think that the right setting is false, just like for every other method, excpet NTLM.

Hmm. Setting it to false causes some dockerized Proxy tests to fail.
I'm going to need to more time to investige.

stoty · 2025-02-27T17:24:36Z

I have fixed the small issues.
It's getting late, I'll try to do the loop condition tomorrow.

…or non-compliant servers

stoty · 2025-02-28T09:26:49Z

I have added a new config option for non-standard servers, and rewritten the established checks.

Please take another look @michael-o .

httpclient5/src/main/java/org/apache/hc/client5/http/impl/auth/gss/GssSchemeBase.java

Update some debug/exception messages

stoty · 2025-02-28T14:07:58Z

I have split the mutual auth config option to request and require.
Having both takes no effort on our part, but allows for set and pray mutual authentication (though I doubt that value of that).

I can drop the request one, and reqest unconditionally if you prefer.

stoty · 2025-02-28T14:23:46Z

I opened https://bugs.squid-cache.org/show_bug.cgi?id=5485 for the Squid issues.

michael-o · 2025-02-28T14:36:12Z

I opened https://bugs.squid-cache.org/show_bug.cgi?id=5485 for the Squid issues.

Thanks, very good. mod_auth_gssapi is the gold standard for me written by very knowledgeable people.

michael-o · 2025-03-05T10:39:32Z

httpclient5/src/main/java/org/apache/hc/client5/http/auth/gss/GssConfig.java

+import org.apache.hc.core5.annotation.ThreadingBehavior;
+
+/**
+ * Immutable class encapsulating GSS configuration options for the new mutual auth capable


...for the new {@link SpnegoScheme}.

michael-o · 2025-03-05T10:41:16Z

httpclient5/src/main/java/org/apache/hc/client5/http/auth/gss/GssCredentials.java

+
+    @Override
+    public Principal getUserPrincipal() {
+        return null;


Can you add a TODO than the principal can be obtained from the GSSCredential?

michael-o · 2025-03-05T10:41:36Z

httpclient5/src/main/java/org/apache/hc/client5/http/impl/auth/gss/GssSchemeBase.java

+
+    private static final Logger LOG = LoggerFactory.getLogger(GssSchemeBase.class);
+    private static final String NO_TOKEN = "";
+    private static final String KERBEROS_SCHEME = "HTTP";


michael-o · 2025-03-05T10:41:43Z

httpclient5/src/main/java/org/apache/hc/client5/http/impl/auth/gss/GssSchemeBase.java

+
+    // The GSS spec does not specify how long the conversation can be. This should be plenty.
+    // Realistically, we get one initial token, then one maybe one more for mutual authentication.
+    private static final int MAX_GSS_CHALLENGES = 3;


michael-o · 2025-03-05T10:42:08Z

httpclient5/src/main/java/org/apache/hc/client5/http/impl/auth/gss/GssSchemeBase.java

+            try {
+                 hostname = dnsResolver.resolveCanonicalHostname(host.getHostName());
+            } catch (final UnknownHostException ignore) {
+            }


Still open...

michael-o · 2025-03-05T10:43:04Z

httpclient5/src/main/java/org/apache/hc/client5/http/impl/auth/gss/SpnegoScheme.java

+ * SPNEGO (Simple and Protected GSSAPI Negotiation Mechanism) authentication
+ * scheme.
+ * <p>
+ * This is the new mutual authentication capable Scheme which replaces the old deprecated non mutual


I think is more than enough to mention that this superseded the other class, not necessary to mention its capabilities here.

michael-o · 2025-03-05T10:45:03Z

httpclient5/src/main/java/org/apache/hc/client5/http/impl/auth/gss/SpnegoScheme.java

+
+    @Override
+    public boolean isConnectionBased() {
+        return true;


Thinking about this again, This is not a problem when the completion done after one roundtrip. So I guess when the security context says continue needed than this needs to be bound to the connection. WDYT?

michael-o · 2025-03-05T11:02:57Z

httpclient5/src/main/java/org/apache/hc/client5/http/impl/auth/gss/GssSchemeBase.java

+            if (LOG.isDebugEnabled()) {
+                final HttpClientContext clientContext = HttpClientContext.cast(context);
+                final String exchangeId = clientContext.getExchangeId();
+                LOG.debug("{} GSS error: too many challenges received. Infinite loop ?", exchangeId);


This is actually a good question. I have the feeling that this should be at warning lever since an upper layer exception cannot reflect this.

michael-o · 2025-03-05T11:04:36Z

httpclient5/src/main/java/org/apache/hc/client5/http/impl/auth/gss/GssSchemeBase.java

+            return;
+        }
+
+        final byte[] challengeToken = Base64.decodeBase64(authChallenge == null ? null : authChallenge.getValue());


Why bother Base64 if we can do the ternary check outside?

I'm not sure what you mean.
The ternary here prevents an NPE from authChallenge.getValue() is authChallange is null.
Base64.decodeBase64 returns null for null, so we don't have to check for a null value.

authChallenge == null ? null : Base64.decodeBase64(authChallenge.getValue());

michael-o · 2025-03-05T11:05:36Z

httpclient5/src/main/java/org/apache/hc/client5/http/impl/auth/gss/GssSchemeBase.java

+            case UNINITIATED:
+                setGssCredential(HttpClientContext.cast(context).getCredentialsProvider(), host, context);
+                queuedToken = generateToken(challengeToken, KERBEROS_SCHEME, gssHostname);
+                if (challengeToken != null) {


This check should be done before the generateToken() method, no?

Doesn't really matter, generateToken() does not change challengeToken, and this only gates some logging.
I will switch them around.

It makes sense in a way because you avoid the object creation in the method.

michael-o · 2025-03-05T11:07:16Z

httpclient5/src/main/java/org/apache/hc/client5/http/impl/auth/gss/GssSchemeBase.java

+                        LOG.debug("{} Internal GSS error: token received when none was sent yet: {}", exchangeId, challengeToken);
+                    }
+                    // TODO Should we fail ? That would break existing tests that send a token
+                    // in the first response, which is against the RFC.


OK.
Will need to change test for this.

michael-o · 2025-03-05T11:07:52Z

httpclient5/src/main/java/org/apache/hc/client5/http/impl/auth/gss/GssSchemeBase.java

+                        if (LOG.isDebugEnabled()) {
+                            final HttpClientContext clientContext = HttpClientContext.cast(context);
+                            final String exchangeId = clientContext.getExchangeId();
+                            LOG.debug("{} GSS Context is not established, but continuing because GssConfig.ignoreMissingToken is true.", exchangeId);


Warning, not debug as agreed.

michael-o · 2025-03-05T11:08:59Z

httpclient5/src/main/java/org/apache/hc/client5/http/impl/auth/gss/GssSchemeBase.java

+                        if (LOG.isDebugEnabled()) {
+                            final HttpClientContext clientContext = HttpClientContext.cast(context);
+                            final String exchangeId = clientContext.getExchangeId();
+                            LOG.debug("{} Did not receive required challenge and GssConfig.ignoreMissingToken is false.",


I'd drop the part from "...and". If not set explicitly, no need to mention.

michael-o · 2025-03-05T11:09:10Z

httpclient5/src/main/java/org/apache/hc/client5/http/impl/auth/gss/GssSchemeBase.java

+                        }
+                        state = State.FAILED;
+                        throw new AuthenticationException(
+                                "Did not receive required challenge and GssConfig.ignoreMissingToken is false.");


michael-o · 2025-03-05T11:10:00Z

httpclient5/src/main/java/org/apache/hc/client5/http/impl/auth/gss/GssSchemeBase.java

+                    state = State.FAILED;
+                    // TODO should we have specific exception(s) for these ?
+                    throw new AuthenticationException(
+                            "GSSContext is not established.");


You haven't disposed the security context.

Adding a finally block for that.

michael-o · 2025-03-05T11:11:43Z

httpclient5/src/main/java/org/apache/hc/client5/http/impl/auth/gss/GssSchemeBase.java

+                            LOG.debug("{} GSSContext MutualAuthState is false, but continuing because GssConfig.requireMutualAuth is false.",
+                                exchangeId);
+                        }
+                        state = State.FAILED;


I do not fully understand this. If mutual auth isn't required, why fail if it is not available?

This is a bug, I'm fixing it. (We don't have a test case for this path)

stoty · 2025-03-05T12:37:01Z

I realize that I somehow didn't upload the changes for GSSSchemeBase in my last pach, that's why a lot of your earlier comments were not addressed.

set isConnectionBased() to false fix logic in needsAuthentication()

stoty · 2025-03-08T08:57:30Z

I hope I got all the fixes we discussed, @michael-o .

Based on my digging, I have set isConnectionBased() to false.

I have also fixed the logic in the needAuthentication() to handle proxy + target authentication correctly.
(Should that be split to a separate JIRA ?)

stoty · 2025-03-19T12:15:03Z

Were you able to check my latest patch @michael-o ?

stoty marked this pull request as draft January 23, 2025 09:58

stoty mentioned this pull request Jan 25, 2025

HTTPCLIENT-1625 Completely overhaul GSS-API-based authentication backend #577

Closed

michael-o requested changes Jan 28, 2025

View reviewed changes

ok2c force-pushed the master branch from 6eac9d2 to 05497d5 Compare February 10, 2025 17:57

stoty force-pushed the HTTPCLIENT-2358 branch 2 times, most recently from be6688d to ce46c45 Compare February 20, 2025 22:17

stoty added 2 commits February 20, 2025 23:19

HTTPCLIENT-2358 Implement a mutual authentication capable SPNEGO scheme

51de855

MutualKerberosConfig changes

1b28a5a

stoty force-pushed the HTTPCLIENT-2358 branch 3 times, most recently from 2d87e73 to 2d7dc5e Compare February 20, 2025 22:50

Add containerized apache httpd + mod_auth_gssapi tests

5a11bce

Fix GSSCredential handling

stoty force-pushed the HTTPCLIENT-2358 branch from 2d7dc5e to 5a11bce Compare February 20, 2025 23:00

Add SPNEGO tests for Squid

2d8f404

stoty marked this pull request as ready for review February 22, 2025 19:39

stoty requested a review from michael-o February 22, 2025 19:42

stoty commented Feb 26, 2025

View reviewed changes

httpclient5/src/main/java/org/apache/hc/client5/http/impl/auth/MutualGssSchemeBase.java Outdated Show resolved Hide resolved

michael-o requested changes Feb 27, 2025

View reviewed changes

Rename and move around classes

82c9603

renames KerberosCredentials

bbdb403

michael-o self-requested a review February 27, 2025 14:56

michael-o requested changes Feb 27, 2025

View reviewed changes

fixing the samller issues based on the review

dd87c1a

rewrite GSS established logic, and add new option for compatibility f…

41c5203

…or non-compliant servers

stoty requested a review from michael-o February 28, 2025 09:26

michael-o reviewed Feb 28, 2025

View reviewed changes

httpclient5/src/main/java/org/apache/hc/client5/http/impl/auth/gss/GssSchemeBase.java Outdated Show resolved Hide resolved

Add separate flags for requesting and checking mutual auth status.

7318bf3

Update some debug/exception messages

michael-o requested changes Mar 5, 2025

View reviewed changes

apply discussed fixes

0bf4eca

set isConnectionBased() to false fix logic in needsAuthentication()

stoty requested a review from michael-o March 8, 2025 08:57

HTTPCLIENT-2358 Implement a mutual authentication capable SPNEGO scheme #615

Are you sure you want to change the base?

HTTPCLIENT-2358 Implement a mutual authentication capable SPNEGO scheme #615

Conversation

stoty commented Jan 23, 2025

ok2c commented Jan 23, 2025

stoty commented Jan 23, 2025

ok2c commented Jan 23, 2025

stoty commented Jan 23, 2025

garydgregory commented Jan 23, 2025

stoty commented Jan 24, 2025

stoty commented Jan 24, 2025

michael-o commented Jan 28, 2025 • edited Loading

michael-o left a comment

Choose a reason for hiding this comment

stoty commented Jan 28, 2025 • edited Loading

stoty commented Feb 20, 2025 • edited Loading

stoty commented Feb 22, 2025

michael-o commented Feb 26, 2025

michael-o left a comment

Choose a reason for hiding this comment

stoty commented Feb 27, 2025

stoty commented Feb 27, 2025

michael-o commented Feb 27, 2025

michael-o commented Feb 27, 2025

stoty commented Feb 27, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

stoty Feb 27, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

michael-o Feb 28, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

michael-o Feb 28, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

stoty commented Feb 27, 2025

stoty commented Feb 28, 2025

stoty commented Feb 28, 2025

stoty commented Feb 28, 2025

michael-o commented Feb 28, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

stoty Mar 5, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

michael-o commented Jan 28, 2025 •

edited

Loading

stoty commented Jan 28, 2025 •

edited

Loading

stoty commented Feb 20, 2025 •

edited

Loading

stoty Feb 27, 2025 •

edited

Loading

michael-o Feb 28, 2025 •

edited

Loading

michael-o Feb 28, 2025 •

edited

Loading

stoty Mar 5, 2025 •

edited

Loading