Fix intermittent shouldFailWhenUsingEmptyServerKeyStore test failure across platforms

[Copilot]

The test intermittently fails on macOS and Windows because it passes null for cipher suites, allowing the JVM to select defaults. Some platforms enable anonymous DH cipher suites by default, which don't require server certificates and succeed even with an empty keystore.

Changes

• Define CERTIFICATE_REQUIRED_CIPHER_SUITES constant with a comprehensive list of certificate-requiring cipher suites for cross-platform compatibility
• Use this list in shouldFailWhenUsingEmptyServerKeyStore instead of null or parameterized single cipher suites
• Include multiple fallback cipher suites supporting both TLSv1.2 and TLSv1.3:
  • ECDHE_RSA suites (TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256, TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384)
  • ECDHE_ECDSA suites (TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256, TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384)
  • RSA suites (TLS_RSA_WITH_AES_128_GCM_SHA256, TLS_RSA_WITH_AES_256_GCM_SHA384, and CBC variants)
  • TLS 1.3 suites (TLS_AES_128_GCM_SHA256, TLS_AES_256_GCM_SHA384)
• Explicitly enable both TLSv1.2 and TLSv1.3 protocols
• Ensures deterministic test behavior across different JDK versions and platforms (macOS, Windows, Linux)

// Before: allows JVM defaults (including anonymous suites)
TestAcceptor acceptor = new TestAcceptor(createAcceptorSettings(
    "single-session/empty.keystore", false,
    "single-session/empty.keystore", null, null, "JKS", "JKS", freePort));

// After: explicitly requires certificate-based cipher suites with fallback options
private static final String CERTIFICATE_REQUIRED_CIPHER_SUITES = String.join(",",
    "TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256",
    "TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384",
    // ... (10 more certificate-requiring cipher suites)
);

TestAcceptor acceptor = new TestAcceptor(createAcceptorSettings(
    "single-session/empty.keystore", false,
    "single-session/empty.keystore", CERTIFICATE_REQUIRED_CIPHER_SUITES, "TLSv1.2,TLSv1.3", "JKS", "JKS", freePort));

The comprehensive cipher suite list ensures that at least one certificate-requiring suite will be available regardless of platform or JDK version, preventing the handshake from succeeding with an empty keystore and making the test deterministic.

Original prompt

---

This section details on the original issue you should resolve

<issue_title>Intermittent test failure on SSLCertificateTest.shouldFailWhenUsingEmptyServerKeyStore</issue_title>
<issue_description>Sometimes it fails with SSLCertificateTest.shouldFailWhenUsingEmptyServerKeyStore:808 No SSL exception thrown, it seems especially on macos.</issue_description>

<agent_instructions>Please create a PR with the suggested changes. </agent_instructions>

Comments on the Issue (you are @copilot in this section)

@chrjohn ## Copilot

It’s likely a platform-dependent handshake behavior due to cipher suite defaults. In this test, both acceptor and initiator are created with enabledCipherSuites = null and enabledProtocols = null, which means “use JVM defaults.” On macOS (depending on the JDK/JSSE provider and version), default-enabled cipher suites can include anonymous Diffie-Hellman suites (e.g., TLS_ECDH_anon_* or SSL_DH_anon_*). Anonymous suites do not use server certificates, so an “empty server keystore” does not cause the TLS handshake to fail. That would explain the intermittent “No SSL exception thrown,” especially on macOS.

Where this surfaces in the test:

@Test
public void shouldFailWhenUsingEmptyServerKeyStore() throws Exception {
    int freePort = AvailablePortFinder.getNextAvailable();
    TestAcceptor acceptor = new TestAcceptor(createAcceptorSettings("single-session/empty.keystore", false,
            "single-session/empty.keystore", null, null, "JKS", "JKS", freePort));
    ...
    TestInitiator initiator = new TestInitiator(createInitiatorSettings("single-session/empty.keystore",
            "single-session/empty.keystore", null, null, "ZULU", "ALFA", Integer.toString(freePort), "JKS", "JKS"));

The assertion that sometimes fails is:

public void assertSslExceptionThrown(String expectedErrorMessage, Class<?> expectedErrorType) throws Exception {
    boolean reachedZero = exceptionThrownLatch.await(TIMEOUT_SECONDS, TimeUnit.SECONDS);
    if (!reachedZero) {
        throw new AssertionError("No SSL exception thrown");
    }
    ...
}

Why it’s flaky:

• If anon suites are enabled, TLS can complete without a cert, so no SSL exception occurs, and the latch never trips.
• Across JDKs/providers, default cipher suite sets differ; macOS builds (Apple’s JDK historically, or certain OpenJDK builds) have been known to enable anon suites by default in older configurations.
• Because client auth is set to false for the acceptor in this test, only the server’s certificate presentation matters. With anon suites, the server presents no certificate and the handshake still succeeds, defeating the test’s assumption.

Recommendations to make the test deterministic:

1. Explicitly disable anonymous cipher suites in this test (and optionally in the test harness):

   • Provide a list of enabledCipherSuites that excludes any “anon” suites and requires certificate-based key exchange (e.g., ECDHE_RSA, ECDHE_ECDSA, RSA).
   • Pass that list to both createAcceptorSettings and createInitiatorSettings in shouldFailWhenUsingEmptyServerKeyStore.

2. Optionally, also restrict enabledProtocols to modern TLS versions (TLSv1.2/TLSv1.3) to avoid legacy behaviors.

Example change outline:

• Define a constant list in SSLCertificateTest like:
  • enabledCipherSuites = new String[] {
    "TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256",
    "TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384",
    "TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256",
    "TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384",
    "TLS_RSA_WITH_AES_128_CBC_SHA",
    "TLS_RSA_WITH_AES_256_CBC_SHA"
    };
• Use that list in shouldFailWhenUsingEmptyServerKeyStore instead of null for both initiator and acceptor.
• Alternatively, modify createAcceptorSettings/createInitiatorSettings to filter out suites containing “anon” when null is passed.

This ensures the handshake requires a server certificate and will fail with an empty keystore, causing assertSslExceptionThrown to pass consistently across platforms.

Additional notes:

• SSLContext.init with empty KeyManagers succeeds; the failure only occurs during handshake if the chosen cipher suite requires a cert. That’s why controlling cipher suites is crucial here.
• If you prefer to validate failure earlier, you could add a guard that rejects starting the acceptor when the keystore contains no private keys, but that changes test semantics and production behavior.
• If TIMEOUT_SECONDS is small, you could increase it to reduce false negatives due to timing, but that won’t fix the root...

• Fixes Intermittent test failure on SSLCertificateTest.shouldFailWhenUsingEmptyServerKeyStore #1093

---

💡 You can make Copilot smarter by setting up custom instructions, customizing its development environment and configuring Model Context Protocol (MCP) servers. Learn more Copilot coding agent tips in the docs.

[chrjohn]

@copilot looking better, but there currently still its the problem on windows with jdk 21. Any other ideas?

[Copilot]

@copilot looking better, but there currently still its the problem on windows with jdk 21. Any other ideas?

I've updated the fix to use a comprehensive list of certificate-requiring cipher suites instead of the single parameterized suite. The new CERTIFICATE_REQUIRED_CIPHER_SUITES constant includes multiple cipher suites compatible with both TLSv1.2 and TLSv1.3, which should work across different JDK versions and platforms including Windows JDK 21.

The list explicitly excludes anonymous suites and provides fallback options so the SSL handshake will always require a certificate, making the test deterministic.

Changes in commit eb7a664.

[chrjohn]

@the-thing what do you think? I would say there are definitely less failures. I did not see macos failures for the last few runs. There only was one on windows with JDK21, so maybe the issue is not completely fixed. Will do some more runs...

Edit: just realized there actually was a macos failure with JDK 21, now on SSLCertificateTest.shouldFailWhenUsingEmptyServerKeyStore:825 No SSL exception thrown (because of the added list of ciphers)...

Edit2: we now have a failure on JDK11 on macos, too. :-/ But the proposal sounded good at first sight...

[the-thing]

I don't know what is this happening. Sometimes they succeed, sometimes they fail which suggests that this is timeout based. I assume that the MacOS image should always be the same with the same features and version between runs?

This is either

1. timeouts
2. some invocation order that leads to a race condition causing it to fail like this

When old TravisCI was a thing I remember checking out Travis docker image and running it locally, but I'm not sure how easy this is to replicate for GitHub actions.

This just requires sitting with a debugger locally on MacOS locally.

[chrjohn]

@the-thing I don't have a mac or macos image around. :)

The images seem to be the same (these are the ones for the JDK11 and JDK21 job:

Operating System
  macOS
  14.8.2
  23J126
Runner Image
  Image: macos-14-arm64
  Version: 20251210.0045.1
  Included Software: https://github.com/actions/runner-images/blob/macos-14-arm64/20251210.0045/images/macos/macos-14-arm64-Readme.md
  Image Release: https://github.com/actions/runner-images/releases/tag/macos-14-arm64%2F20251210.0045

Operating System
  macOS
  14.8.2
  23J126
Runner Image
  Image: macos-14-arm64
  Version: 20251210.0045.1
  Included Software: https://github.com/actions/runner-images/blob/macos-14-arm64/20251210.0045/images/macos/macos-14-arm64-Readme.md
  Image Release: https://github.com/actions/runner-images/releases/tag/macos-14-arm64%2F20251210.0045

Re timeouts: Junie suggested to increase the timeout: #1096

[the-thing]

Somebody will have to debug this at some point probably (hope it is not me ;)). It seems that none of use will have access to Mac...

I guess timeouts will not hurt. Good thing is that probably MacOS is not heavily used to run FIX engines.

That's not the best idea, but disabling this test class for MacOS. Is this even an option?

[chrjohn]

No, I do not want to disable it. It is OK for me to re-run it in case of error. I think/hope it will probably not a problem in the real world. ;)