Discussions on rocBLAS and Tensile

[littlewu2508]

hip-4.1 is already in gentoo and hip-4.2 is almost done.

I've tried rocBLAS-4.2 (skipping 4.1) out and find some issues that need discussion.

1. I would like to go back bundle tensile in virtualenv rather than use a standalone tensile package. dev-util/Tensile has two much configuration and C++ source files installed, and their default location is in python site-packages directory.This brings QA concerns, and placing them in the right place requires hard working. Thus it is left messy and put into science overlay rather than gentoo portage, and rocBLAS is in science because of it. We want rocBLAS to be in portage (https://bugs.gentoo.org/732952).
2. The build process becomes longer because TensileCreateLibrary is excuted twice for some unknown reasons. Need fixes.
3. rocBLAS-4.2 uses a cmake command which generates an incorrect build.ninja.
4. As 6aa5e41 described there are existing bugs I haven't noticed about rocBLAS which needs to be fixed.

I will put more information and try to fix the above problems after investigation and reviewing patches from @fuag15.

[littlewu2508]

hip-4.1 is already in gentoo and hip-4.2 is almost done.

I've tried rocBLAS-4.2 (skipping 4.1) out and find some issues that need discussion.

1. I would like to go back bundle tensile in virtualenv rather than use a standalone tensile package. dev-util/Tensile has two much configuration and C++ source files installed, and their default location is in python site-packages directory.This brings QA concerns, and placing them in the right place requires hard working. Thus it is left messy and put into science overlay rather than gentoo portage, and rocBLAS is in science because of it. We want rocBLAS to be in portage (https://bugs.gentoo.org/732952).

rocBLAS-4.1.0::science now bundles Tensile, and rocBLAS-4.2 is planned to land on ::gentoo.

2. The build process becomes longer because TensileCreateLibrary is excuted twice for some unknown reasons. Need fixes.

In the first TensileCreateLibrary run yaml library logic files are read and generates Tensile library manifests, without compilation. The second time during src_compile, it builds the code objects. But parsing and processing the library logic is still executed twice.

3. rocBLAS-4.2 uses a cmake command which generates an incorrect  build.ninja.

Fixed with a patch from upstream.

4. As [6aa5e41](https://github.com/justxi/rocm/commit/6aa5e4147f2239953350efdcdbc4851b6ddd05cb) described there are existing bugs I haven't noticed about rocBLAS which needs to be fixed.

Fixed

[littlewu2508]

@heroxbd By debugging TensileCreateLibrary I understand how the code objects are created:

1. Find the configuration files named as _.yaml, which are the library logic. The architecture name can be r9nano (gfx803), vega10 (gfx900), vega20 (gfx906), arcturus (gfx908), aldebaran (gfx90a), navi10 (gfx1010), navi11 (gfx1011), navi12 (gfx1012), navi21 (gfx1030), and hip (gfx000).
2. The "hip" solution will generate HIP/C++ source code compatible for all archs (gfx000), while others generated architecture specific assemblies.
3. Clang++ from llvm-roc compiles assemblies. Compiled objects with same arch are bundled together to Tensile_gfx***.co, the ASM kernel code objects. If multiple non-source (hip, gfx000) arch exitst, there will be more than one code objects.
4. Source kernels are (by default) merged together as a large Kernel.cpp. According to llvm AMDGPU target feature, there can be xnack enabled (gfx***:xnack+) or disabled (gfx***:xnack-), so during compilation features can be appended to arch names. Then Kernels.cpp is compiled to Kernels.o by hipcc, and gets separated to different offload architectures by clang-offload-bundler, resulting in Kernels.so-000-<arch_with_features>.hasco.

[littlewu2508]

The question is, TensileCreateLibrary by default process all supported archs, and compile Kernels.cpp with all possible --offload-arch, which makes the compilation very slow.

However, specifying the arch can be a little complicated: there are two kind of "archs", the first one without features is for asm code objects, and the second one with features is for source code objects. And they have different range of support. Take rocm-4.2 for example, one can only parse gfx803, gfx900, gfx906:xnack-, gfx908:xnack- to --architecture, which control the selection of yaml library logic, and by that unwanted arch are filtered. But for source code objects, the supported ISA also includes gfx1010 and gfx1011, thus will output Kernels.so-000-gfx1010.hsaco.

Also, for gfx906 and gfx908, it will force disable :xnack while force enabled it for gfx90a when compiling Kernels.cpp.

So, there are two sets of arch: the first one can be parsed from command line via "--architecture", which transform to arch names ( vega, arcturus, navi, etc), and select the wanted arch, which only affects the asm code objects. Meanwhile all "hip" arch yamls are selected, and will generated Kernels.so-000-gfx****.hsaco for all supported ISA.