the conflict of protobuf version between apache orc and tensorflow

[j2cms]

when use tensorflow 1.15.5, it depend protobuf, the default version is 3.19.6
but the apache orc use protobuf 3.5.1 ( it seems too old)

they meet conflict as below:

Python(3937,0x7ff85c27bfc0) malloc: *** error for object 0x600002e394d8: pointer being freed was not allocated
Python(3937,0x7ff85c27bfc0) malloc: *** set a breakpoint in malloc_error_break to debug

I found the similarity issues [https://github.com//issues/1425]

can we upgrade PROTOBUF version to 3.19.6?

[dongjoon-hyun]

Hi, @j2cms .

Which Apache ORC versions are you using now? IIUC, Apache ORC v2.x uses more higher versions of protobuf.

• Apache ORC 1.9.5
  

  orc/java/pom.xml

  Line 135 in c6904de

  <version>3.22.3</version>

• Apache ORC 2.0.0
  

  orc/java/pom.xml

  Line 81 in 46eb6ff

  <protobuf.version>3.25.3</protobuf.version>

• Apache ORC 2.1.0
  

  orc/java/pom.xml

  Line 82 in 2cb1394

  <protobuf.version>3.25.5</protobuf.version>

[dongjoon-hyun]

BTW, FYI, Apache ORC community doesn't maintain PyPi packages.

[j2cms]

hi, @dongjoon-hyun , thank you for the reply.

I use the latest version of Apache ORC, I think the reason is PROTOBUF_VERSION in cmake_modules is 3.5.1

orc/cmake_modules/ThirdpartyToolchain.cmake

Line 27 in b9641c6

set(PROTOBUF_VERSION "3.5.1")

can you upgrade it to 3.19.6 or later ?

[dongjoon-hyun]

Thank you, @j2cms .

I searched the community activity. It seems that it was tried here.

• ORC-1662: [C++] Upgrade protobuf to 3.21.12 #1857

[dongjoon-hyun]

It seems that we deprecated HDFS support to upgrade C++ protobuf at ORC 2.0.1 but we couldn't delete it in Apache Orc 2.x because we follow the semantic versioning.

• ORC-1669: [C++] Deprecate HDFS support #1885

In this case, we need to wait until 3.0.0 because 2.0 and 2.1 is already out with BUILD_LIBHDFSPP.

cc @wgtmac

[dongjoon-hyun]

I filed the following.

• [C++] Remove HDFS support #2136

[wgtmac]

@j2cms If you have an installed protobuf somewhere, perhaps you may want to set one of these environment variables before building to avoid the default one.

[j2cms]

hi, @dongjoon-hyun , thank you for the reply.

I use the latest version of Apache ORC, I think the reason is PROTOBUF_VERSION in cmake_modules is 3.5.1

orc/cmake_modules/ThirdpartyToolchain.cmake

Line 27 in b9641c6

set(PROTOBUF_VERSION "3.5.1")
can you upgrade it to 3.19.6 or later ?

I changed it to 3.19.6 as above and compile orc locally (on my MacOS), success.
then, I compile pyorc use the orc I compiled,

finally It works. no protobuf version conflict with tensorflow now

maybe we should upgrade it to 3.19.6, not 3.21.12 in #1857

[dongjoon-hyun]

We started to discuss Apache ORC 2.2 release.

After releasing 2.2, we can start to re-discuss

• ORC-1662: [C++] Upgrade protobuf to 3.21.12 #1857
• [C++] Remove HDFS support #2136

in the scope of Apache ORC 3.0.