Skip to content

Performance comparisons: nc mesh vs conf. mesh #18

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
artv3 wants to merge 1 commit into
base: master
Choose a base branch
from
+25 −4
Draft

Performance comparisons: nc mesh vs conf. mesh #18

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 4 additions & 2 deletions machine-configs/lassen.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@ function setup_xlc()
MPIF77=mpixlf
mpi_info_flag="-qversion=verbose"

CFLAGS="-O2 -qmaxmem=-1 -qarch=auto -qcache=auto -qtune=auto"
CFLAGS="-O2"
# CFLAGS="-O3 -qarch=auto -qcache=auto -qhot -qtune=auto"
# CFLAGS+=" -qipa=threads"

Expand All @@ -44,10 +44,12 @@ function setup_gcc()

MPICC=mpigcc
MPICXX=mpig++
which $MPICXX
$MPICXX --version
MPIF77=mpigfortran
mpi_info_flag="--version"

CFLAGS="-O3 -mcpu=native -mtune=native"
CFLAGS="-O3"
FFLAGS="$CFLAGS"
TEST_EXTRA_CFLAGS=""
# TEST_EXTRA_CFLAGS+=" -std=c++11 -fdump-tree-optimized-blocks"
Expand Down
16 changes: 16 additions & 0 deletions package-builders/hypre.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -73,19 +73,35 @@ function hypre_build()
my_cflags="-g"
fi
echo "Building $pkg, sending output to ${pkg_bld_dir}_build.log ..." && {
local CUDA_MAKE_OPTS=()
if [[ -n "$CUDA_ENABLED" ]]; then
CUDA_MAKE_OPTS=(
"--with-cuda"
"--with-gpu-arch=70"
"--enable-device-memory-pool"
Copy link
Collaborator

@tomstitt tomstitt May 26, 2022 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we know if this is as performant as using Umpire? Also, does this disable the use of unified memory?

Copy link

@liruipeng liruipeng May 26, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No. We suggest using Umpire if you can. We probably will remove our device memory pool implementation in near future since even CUDA now has the "official" memory pool.

Copy link
Collaborator Author

@artv3 artv3 May 26, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The default for hypre is no unified memory (ref. docs).
--enable-unified-memory Use unified memory for allocating the memory
(default is NO).

If I recall correctly I think Umpire does improve the performance a bit, not sure by how what factor compared to the internal memory pool framework.

Copy link
Collaborator Author

@artv3 artv3 May 26, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@liruipeng do you know how much faster it is to use umpire?

Copy link

@liruipeng liruipeng May 26, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@liruipeng do you know how much faster it is to use umpire?

Hard to give a number. It highly depends on the application.

# "CUCC=$cuda_home/bin/nvcc"
)
xcompiler="-Xcompiler="
optim_flags="$cxx11_flag $xcompiler\"$CFLAGS\""
else
echo "${magenta}INFO: Building $pkg without CUDA ...${none}"
fi
cd "$pkg_bld_dir/src" && \
if [[ -e config/Makefile.config ]]; then
make distclean
fi && \
CUCC="$cuda_home/bin/nvcc -ccbin=$MPICXX" \
./configure \
CC="$MPICC" \
CXX="$MPICXX" \
CFLAGS="$my_cflags" \
CXXFLAGS="$my_cflags" \
$HYPRE_EXTRA_CONFIG \
"${CUDA_MAKE_OPTS[@]}" \
${hypre_big_int:+--enable-bigint} \
${hypre_mixed_int:+--enable-mixedint} \
${hypre_debug:+--enable-debug} \
--with-MPI \
--disable-fortran \
--without-fei && \
make -j $num_proc_build
Expand Down
3 changes: 3 additions & 0 deletions tests/mfem_ex1/ex1.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -81,6 +81,9 @@ int main(int argc, char *argv[])
Array<int> nxyz;
Mesh *mesh = make_mesh(myid, num_procs, dim, level, par_ref_levels, nxyz);

//Go through NC mesh machinery
mesh->EnsureNCMesh();

// 5. Refine the serial mesh on all processors to increase the resolution. In
// this example we do 'ref_levels' of uniform refinement. We choose
// 'ref_levels' to be the largest number that gives a final mesh with no
Expand Down
4 changes: 2 additions & 2 deletions tests/mfem_ex1/ex1.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -39,7 +39,7 @@ function run_tests()
# 'max_dofs_proc' can be set on the command line
local l_max_dofs_proc=${max_dofs_proc:-4200000}
# 'mfem_devs' can be set on the command line
local l_mfem_devs=(${mfem_devs:-cpu})
local l_mfem_devs=(${mfem_devs:-cuda})
local dim=3
local args=
for dev in ${l_mfem_devs[@]}; do
Expand Down Expand Up @@ -92,6 +92,6 @@ libceed_branch=${libceed_branch:-master}
# Only with HIP
#packages="hip metis hypre mfem-hip"

packages=${packages:-metis hypre mfem}
packages=${packages:-metis cuda hypre mfem}

test_required_packages=${packages}