Problem connecting with client cert chain

[WebDust21]

Firstly, I want to say thank-you for (so far) the only working SSL library that I've yet found! It might be the biggest in compiled size, but all the other ones don't work!
My apologies in advance for a lengthy issue, but I'm trying to provide as much detail as I can. I know effectively nothing about SSL, which is why I'm putting this up to those who know about it!

Describe the error
It appears (from server logs) that when a PEM cert chain is provided to SSLClientESP32 for the client certificate, it simply does not send the certificates--causing the server to reject and close the connection. (Server is a self-hosted TBMQ FWIW.)
If a singular PEM is provided to SSLClientESP32, it successfully negotiates the SSL handshake--although the X.509 automatic device provisioning on TBMQ fails without at least 2 certs in the chain (device + group).

Expected behavior
SSLClientESP32 successfully negotiates the SSL handshake when provided with a PEM cert chain for the client cert.

Screenshots
Here's the verbose debug output from SSLClientESP32. Unfortunately very devoid of any specific error info...

[436789][D][SSLClientESP32.cpp:114] connect(): Connecting to [server]
[436789][V][ssl_lib_client.cpp:156] start_ssl_client(): Free internal heap before TLS 127740
[436793][I][ssl_lib_client.cpp:158] start_ssl_client(): Connecting to [server]
[437057][V][ssl_lib_client.cpp:172] start_ssl_client(): Seeding the random number generator
[437058][V][ssl_lib_client.cpp:181] start_ssl_client(): Setting up the SSL/TLS structure...
[437062][V][ssl_lib_client.cpp:204] start_ssl_client(): Loading CA cert
[437073][V][ssl_lib_client.cpp:260] start_ssl_client(): Loading CRT cert
[437084][V][ssl_lib_client.cpp:269] start_ssl_client(): Loading private key
[437093][V][ssl_lib_client.cpp:280] start_ssl_client(): Setting hostname for TLS session...
[437093][V][ssl_lib_client.cpp:293] start_ssl_client(): Setting up IO callbacks...
[437097][V][ssl_lib_client.cpp:296] start_ssl_client(): Performing the SSL/TLS handshake...
[437686] ### Got Data: 0  <- from TinyGSM
[438066] ### Got Data: 0
[438398] ### Got Data: 0
[448676] ### Closed:  0
[557136][E][SSLClientESP32.cpp:121] connect(): start_ssl_client: -1
[557136][V][ssl_lib_client.cpp:349] stop_ssl_socket(): Cleaning SSL connection.
SSLClient Error #-1 = ERROR - Generic error

The server logs meanwhile appear to indicate that SSLClientESP32 simply does not send the client certificates if they are chained--despite the mbedTLS "mbedtls_x509_crt_parse" indicating that it supports chained PEM-format certificates.

SSLClientESP32/src/ssl_lib_client.cpp

Line 262 in 7906f38

ret = mbedtls_x509_crt_parse(&ssl_client->client_cert, (const unsigned char *)cli_cert, strlen(cli_cert) + 1);

-> https://github.com/espressif/mbedtls/blob/09bba150d0d822aad2e58d71723f5407da5c21e0/include/mbedtls/x509_crt.h#L495-L529

/**
 * \brief          Parse one DER-encoded or one or more concatenated PEM-encoded
 *                 certificates and add them to the chained list.
 *
 *                 For CRTs in PEM encoding, the function parses permissively:
 *                 if at least one certificate can be parsed, the function
 *                 returns the number of certificates for which parsing failed
 *                 (hence \c 0 if all certificates were parsed successfully).
 *                 If no certificate could be parsed, the function returns
 *                 the first (negative) error encountered during parsing.
 *
 *                 PEM encoded certificates may be interleaved by other data
 *                 such as human readable descriptions of their content, as
 *                 long as the certificates are enclosed in the PEM specific
 *                 '-----{BEGIN/END} CERTIFICATE-----' delimiters.
 *
 * \note           If #MBEDTLS_USE_PSA_CRYPTO is enabled, the PSA crypto
 *                 subsystem must have been initialized by calling
 *                 psa_crypto_init() before calling this function.
 *
 * \param chain    The chain to which to add the parsed certificates.
 * \param buf      The buffer holding the certificate data in PEM or DER format.
 *                 For certificates in PEM encoding, this may be a concatenation
 *                 of multiple certificates; for DER encoding, the buffer must
 *                 comprise exactly one certificate.
 * \param buflen   The size of \p buf, including the terminating \c NULL byte
 *                 in case of PEM encoded data.
 *
 * \return         \c 0 if all certificates were parsed successfully.
 * \return         The (positive) number of certificates that couldn't
 *                 be parsed if parsing was partly successful (see above).
 * \return         A negative X509 or PEM error code otherwise.
 *
 */
int mbedtls_x509_crt_parse(mbedtls_x509_crt *chain, const unsigned char *buf, size_t buflen);

Here's a verbose server log dump compare between an unsuccessful SSLClientESP32 connection (with chained client cert), and a successful Paho MQTT client connection (with chained client cert), both on TLSv1.2:

The left side is the SSLClientESP32 server log--and this where the SSLClientESP32 log ends (not visible in the diff: last two lines are the server's SSL context being closed, i.e. "Closing outbound of SSLEngine", and "Closing inbound of SSLEngine"). Notice that no certificate is sent. Right side is Paho MQTT with the exact same cert files--and you can see from the scrollbar that the SSL handshake is just getting started--and this is exactly where the client cert chain is sent.

Server logs above this point are pretty much the same between the two.

Please complete the following information:

• Setup: SSLClientESP32@^2.0.3 -> TinyGSM -> SIM7080G

Additional context
Hmm...is this an mbedTLS issue?

SSLClientESP32/src/ssl_lib_client.cpp

Line 277 in 7906f38

mbedtls_ssl_conf_own_cert(&ssl_client->ssl_conf, &ssl_client->client_cert, &ssl_client->client_key);

-> https://github.com/espressif/mbedtls/blob/09bba150d0d822aad2e58d71723f5407da5c21e0/include/mbedtls/ssl.h#L3387-L3425

/**
 * \brief          Set own certificate chain and private key
 *
 * \note           own_cert should contain in order from the bottom up your
 *                 certificate chain. The top certificate (self-signed)
 *                 can be omitted.
 *
 * \note           On server, this function can be called multiple times to
 *                 provision more than one cert/key pair (eg one ECDSA, one
 *                 RSA with SHA-256, one RSA with SHA-1). An adequate
 *                 certificate will be selected according to the client's
 *                 advertised capabilities. In case multiple certificates are
 *                 adequate, preference is given to the one set by the first
 *                 call to this function, then second, etc.
 *
 * \note           On client, only the first call has any effect. That is,
 *                 only one client certificate can be provisioned. The
 *                 server's preferences in its CertificateRequest message will
 *                 be ignored and our only cert will be sent regardless of
 *                 whether it matches those preferences - the server can then
 *                 decide what it wants to do with it.
 *
 * \note           The provided \p pk_key needs to match the public key in the
 *                 first certificate in \p own_cert, or all handshakes using
 *                 that certificate will fail. It is your responsibility
 *                 to ensure that; this function will not perform any check.
 *                 You may use mbedtls_pk_check_pair() in order to perform
 *                 this check yourself, but be aware that this function can
 *                 be computationally expensive on some key types.
 *
 * \param conf     SSL configuration
 * \param own_cert own public certificate chain
 * \param pk_key   own private key
 *
 * \return         0 on success or MBEDTLS_ERR_SSL_ALLOC_FAILED

"on client...only one client certificate can be provisioned"????

I guess the real question is: is it possible for mbedTLS to support a client cert chain on an SSL connection? And if so, this is probably the wrong place to ask that ;-)

[alkonosst]

Hello @WebDust21 , I'm really sorry for the extremely late response! I hope you made some progress with your issue 🤞.

I had the same limitation when trying to use a certificate chain to connect to Thingsboard via X.509 Certificate Based Authentication Thingsboard via X.509 Certificate Based Authentication. ESP32 just couldn't make the connection to the server.

My go-to alternative (what I've been using until now for a couple of years), is to use a certificate bundle and connect to a MQTT server using a token generated on the broker. That way I don't have to change the certificate for each device, and just change a token (which is a string). Besides that, I also can use the same certificate bundle to connect to other HTTPS services.

I hope it helps!

[WebDust21]

I haven't made any progress, apart from using a Python script (Paho MQTT) to do the initial provisioning connection, after which I can connect fine with just the client cert (no intermediate chained cert).

Different cert per device is no problem at all; I use a Python script to run automatic cert generation tools for each device. Means I know exactly which device is going to use what cert, etc.

Was just hoping to see if there was some solution to the cert chain stuff on the ESP32 side, but it appears it goes further downstream than this library.

[alkonosst]

@WebDust21 I'm sorry to read that. At this moment I can't investigate further because I'm working full-time on a company an I don't have too much free time... I will look at at in some (I hope) near future.

Also, I realized that ESP32 Release v3.0.0 broke this library. I haven't seen specifically where, but maybe now there is an update for the Mbed TLS library.

I will left this issue open to remind me to investigate it later.

[WebDust21]

I shall have to test ESP32 Release v3.0.0 it when it trickles down to PlatformIO; note that this isn't specifically a problem with your library. The underlying ESP32 mbedTLS implementation (WiFiClientSecure) also fails with chained client certs--and your library is just leveraging the underlying mbedTLS functionality in the ESP framework.

So the end result is not surprising; what is surprising is how many other TLS libraries can't connect to the Thingsboard server regardless (chained client cert or not).