Fails to build on OpenSSL-4.x-dev

[mouse07410]

Problem

OpenSSL-4.x officially removed support for Engine API from the code. And the current libp11 code does not seem to allow to build a Provider without building the Engine first.

.  .  .
libp11 has been configured with the following options:

Version:                 0.4.17_git
libp11 directory:        /Users/ur20980/openssl-4/lib
Engine directory:        /Users/ur20980/openssl-4/lib/engines-4/
Provider directory:      /Users/ur20980/openssl-3/lib/ossl-modules
Default PKCS11 module:   /opt/local/lib/p11-kit-proxy.dylib
API doc support:         no
Static PKCS#11 engine    no

Host:                    aarch64-apple-darwin25.2.0
Compiler:                clang
Preprocessor flags:      -g 
Compiler flags:          -O3 -std=gnu18 -isysroot /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk -I/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include -pthread
Linker flags:            -g 
Libraries:               -lpthread 

OPENSSL_CFLAGS:          -O3 -std=gnu18 -isysroot /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk -I/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include -g -I/Users/ur20980/openssl-4/include
OPENSSL_LIBS:            -L/Users/ur20980/openssl-4/lib -lssl -lcrypto
.  .  .

libtool: compile:  clang -DHAVE_CONFIG_H -I. -g -O3 -std=gnu18 -isysroot /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk -I/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include -g -I/Users/ur20980/openssl-4/include -O3 -std=gnu18 -isysroot /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk -I/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include -pthread -MT libp11_la-p11_ec.lo -MD -MP -MF .deps/libp11_la-p11_ec.Tpo -c p11_ec.c -o libp11_la-p11_ec.o >/dev/null 2>&1
p11_pkey.c:530:33: error: call to undeclared function 'EVP_PKEY_meth_find'; ISO C99 and later do not support implicit function declarations [-Wimplicit-function-declaration]
  530 |         orig_meth = (EVP_PKEY_METHOD *)EVP_PKEY_meth_find(EVP_PKEY_RSA);
      |                                        ^
p11_pkey.c:530:14: warning: cast to 'EVP_PKEY_METHOD *' (aka 'struct evp_pkey_method_st *') from smaller integer type 'int' [-Wint-to-pointer-cast]
  530 |         orig_meth = (EVP_PKEY_METHOD *)EVP_PKEY_meth_find(EVP_PKEY_RSA);
      |                     ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
p11_pkey.c:531:2: error: call to undeclared function 'EVP_PKEY_meth_get_sign'; ISO C99 and later do not support implicit function declarations [-Wimplicit-function-declaration]
  531 |         EVP_PKEY_meth_get_sign(orig_meth,
      |         ^
p11_pkey.c:533:2: error: call to undeclared function 'EVP_PKEY_meth_get_decrypt'; ISO C99 and later do not support implicit function declarations [-Wimplicit-function-declaration]
  533 |         EVP_PKEY_meth_get_decrypt(orig_meth,
      |         ^
p11_pkey.c:537:13: error: call to undeclared function 'EVP_PKEY_meth_new'; ISO C99 and later do not support implicit function declarations [-Wimplicit-function-declaration]
  537 |         new_meth = EVP_PKEY_meth_new(EVP_PKEY_RSA,
      |                    ^
p11_pkey.c:537:11: error: incompatible integer to pointer conversion assigning to 'EVP_PKEY_METHOD *' (aka 'struct evp_pkey_method_st *') from 'int' [-Wint-conversion]
  537 |         new_meth = EVP_PKEY_meth_new(EVP_PKEY_RSA,
      |                  ^ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  538 |                 EVP_PKEY_FLAG_AUTOARGLEN);
      |                 ~~~~~~~~~~~~~~~~~~~~~~~~~
p11_pkey.c:540:2: error: call to undeclared function 'EVP_PKEY_meth_copy'; ISO C99 and later do not support implicit function declarations [-Wimplicit-function-declaration]
  540 |         EVP_PKEY_meth_copy(new_meth, orig_meth);
      |         ^
p11_pkey.c:542:2: error: call to undeclared function 'EVP_PKEY_meth_set_sign'; ISO C99 and later do not support implicit function declarations [-Wimplicit-function-declaration]
  542 |         EVP_PKEY_meth_set_sign(new_meth,
      |         ^
p11_pkey.c:544:2: error: call to undeclared function 'EVP_PKEY_meth_set_decrypt'; ISO C99 and later do not support implicit function declarations [-Wimplicit-function-declaration]
  544 |         EVP_PKEY_meth_set_decrypt(new_meth,
      |         ^
p11_pkey.c:674:33: error: call to undeclared function 'EVP_PKEY_meth_find'; ISO C99 and later do not support implicit function declarations [-Wimplicit-function-declaration]
  674 |         orig_meth = (EVP_PKEY_METHOD *)EVP_PKEY_meth_find(EVP_PKEY_EC);
      |                                        ^
p11_pkey.c:674:14: warning: cast to 'EVP_PKEY_METHOD *' (aka 'struct evp_pkey_method_st *') from smaller integer type 'int' [-Wint-to-pointer-cast]
  674 |         orig_meth = (EVP_PKEY_METHOD *)EVP_PKEY_meth_find(EVP_PKEY_EC);
      |                     ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
p11_pkey.c:675:2: error: call to undeclared function 'EVP_PKEY_meth_get_sign'; ISO C99 and later do not support implicit function declarations [-Wimplicit-function-declaration]
  675 |         EVP_PKEY_meth_get_sign(orig_meth,
      |         ^
p11_pkey.c:678:13: error: call to undeclared function 'EVP_PKEY_meth_new'; ISO C99 and later do not support implicit function declarations [-Wimplicit-function-declaration]
  678 |         new_meth = EVP_PKEY_meth_new(EVP_PKEY_EC, 0);
      |                    ^
p11_pkey.c:678:11: error: incompatible integer to pointer conversion assigning to 'EVP_PKEY_METHOD *' (aka 'struct evp_pkey_method_st *') from 'int' [-Wint-conversion]
  678 |         new_meth = EVP_PKEY_meth_new(EVP_PKEY_EC, 0);
      |                  ^ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
p11_pkey.c:680:2: error: call to undeclared function 'EVP_PKEY_meth_copy'; ISO C99 and later do not support implicit function declarations [-Wimplicit-function-declaration]
  680 |         EVP_PKEY_meth_copy(new_meth, orig_meth);
      |         ^
p11_pkey.c:682:2: error: call to undeclared function 'EVP_PKEY_meth_set_sign'; ISO C99 and later do not support implicit function declarations [-Wimplicit-function-declaration]
  682 |         EVP_PKEY_meth_set_sign(new_meth,
      |         ^
2 warnings and 14 errors generated.
make[2]: *** [libp11_la-p11_pkey.lo] Error 1
$
$ openssl4 version
OpenSSL 4.0.0-dev  (Library: OpenSSL 4.0.0-dev )
$

Suggested solution

Split the Provider code out cleanly, and make it possible to build Provider only via make target.

[mtrojnar]

Isn't it a little early to address compatibility issues against a branch that has not even reached its alpha stage?

[mouse07410]

Isn't it a little early to address compatibility issues against a branch that has not even reached its alpha stage?

Your repo - your decision. And it is true that OpenSSL v4 hasn't been released yet.

However, since Engine API has been removed for good (as was stated earlier) - it means that neither libp11 engine, nor libp11 provider are usable with the current OpenSSL master, and won't be usable with the upcoming releases until this issue is addressed.

Unless you expect OpenSSL to re-introduce the API they declared deprecated quite a while ago, and now finally removed altogether? Or plan to maintain libp11 only up to OpenSSL v3.x?

Anyway, my job is to point at the problem. What to do about it - if anything at all - is somebody else's headache.