Minimal reproducible example for Libfuzzer

[akhikolla]

hello @tdhock,

Here is the Testharness, Makefile used to compile the code using libfuzzer.

rcpp_read_out_of_bound_DeepState_TestHarness.cpp :

#include <fstream>
#include <RInside.h>
#include <iostream>
#include <RcppDeepState.h>
#include <qs.h>
#include <DeepState.hpp>
#include <ctime>

int rcpp_read_out_of_bound(int rbound);

TEST(testSAN_deepstate_test,rcpp_read_out_of_bound_test){
  RInside R;
  std::cout << "input starts" << std::endl;
  std::ofstream rbound_stream;
  int rbound  = RcppDeepState_int();
  rbound_stream.open("/home/akhila/R/x86_64-pc-linux-gnu-library/3.6/RcppDeepState/testpkgs/testSAN/inst/testfiles/rcpp_read_out_of_bound/inp>
  rbound_stream << rbound;
  std::cout << "rbound values: "<< rbound << std::endl;
  rbound_stream.close();
  std::cout << "input ends" << std::endl;
  try{
     rcpp_read_out_of_bound(rbound);
  }
  catch(Rcpp::exception& e){
    std::cout<<"Exception Handled"<<std::endl;
  }
}

When we run the makefile :

akhila@ml-experiments:~/R/x86_64-pc-linux-gnu-library/3.6/RcppDeepState/testpkgs/testSAN/inst/testfiles/rcpp_read_out_of_bound$ make -f libfuzz.Makefile 
clang++ -g -fsanitize=address,fuzzer -I/usr/share/R/include -I/home/akhila/R/x86_64-pc-linux-gnu-library/3.6/Rcpp/include -I/home/akhila/R/x86_64-pc-linux-gnu-library/3.6/RcppArmadillo/include -I/home/akhila/R/x86_64-pc-linux-gnu-library/3.6/qs/include -I/home/akhila/R/x86_64-pc-linux-gnu-library/3.6/RInside/include -I/home/akhila/R/x86_64-pc-linux-gnu-library/3.6/RcppDeepState/include /home/akhila/R/x86_64-pc-linux-gnu-library/3.6/RcppDeepState/testpkgs/testSAN/inst/testfiles/rcpp_read_out_of_bound/rcpp_read_out_of_bound_DeepState_TestHarness.cpp -o /home/akhila/R/x86_64-pc-linux-gnu-library/3.6/RcppDeepState/testpkgs/testSAN/inst/testfiles/rcpp_read_out_of_bound/rcpp_read_out_of_bound_DeepState_TestHarness_LF.o -c
clang++ -g -fsanitize=address,fuzzer -o /home/akhila/R/x86_64-pc-linux-gnu-library/3.6/RcppDeepState/testpkgs/testSAN/inst/testfiles/rcpp_read_out_of_bound/rcpp_read_out_of_bound_DeepState_TestHarness_LF /home/akhila/R/x86_64-pc-linux-gnu-library/3.6/RcppDeepState/testpkgs/testSAN/inst/testfiles/rcpp_read_out_of_bound/rcpp_read_out_of_bound_DeepState_TestHarness_LF.o -I/home/akhila/R/x86_64-pc-linux-gnu-library/3.6/RcppDeepState/include -I/home/akhila/.RcppDeepState/deepstate-master/build_libfuzzer -I/home/akhila/.RcppDeepState/deepstate-master/src/include -L/home/akhila/R/x86_64-pc-linux-gnu-library/3.6/RInside/lib -Wl,-rpath=/home/akhila/R/x86_64-pc-linux-gnu-library/3.6/RInside/lib -L/usr/share/R/lib -Wl,-rpath=/usr/share/R/lib -L/home/akhila/.RcppDeepState/deepstate-master/build_libfuzzer -Wl,-rpath=/home/akhila/.RcppDeepState/deepstate-master/build_libfuzzer -lR -lRInside -ldeepstate -ldeepstate_LF -I/usr/share/R/include -I/home/akhila/R/x86_64-pc-linux-gnu-library/3.6/Rcpp/include -I/home/akhila/R/x86_64-pc-linux-gnu-library/3.6/RcppArmadillo/include -I/home/akhila/.RcppDeepState/deepstate-master/src/include /home/akhila/R/x86_64-pc-linux-gnu-library/3.6/RcppDeepState/testpkgs/testSAN/src/*.cpp
cd /home/akhila/R/x86_64-pc-linux-gnu-library/3.6/RcppDeepState/testpkgs/testSAN/inst/testfiles/rcpp_read_out_of_bound && ./rcpp_read_out_of_bound_DeepState_TestHarness_LF --fuzz --fuzz_save_passing 
DEBUG: INFO: libFuzzer ignores flags that start with '--'

DEBUG: INFO: Seed: 3529313378

DEBUG: INFO: Loaded 1 modules   (1693 inline 8-bit counters): 
DEBUG: 1693 [604720, 604dbd), 
DEBUG: 

DEBUG: INFO: Loaded 1 PC tables (1693 PCs): 
DEBUG: 1693 [5b41d8,5baba8), 
DEBUG: 

DEBUG: INFO: -max_len is not provided; libFuzzer will not generate inputs larger than 4096 bytes

input starts
rbound values: 0
input ends
R is already initialized
    #0 0x52d7f1 in __sanitizer_print_stack_trace (/home/akhila/R/x86_64-pc-linux-gnu-library/3.6/RcppDeepState/testpkgs/testSAN/inst/testfiles/rcpp_read_out_of_bound/rcpp_read_out_of_bound_DeepState_TestHarness_LF+0x52d7f1)
    #1 0x478948 in fuzzer::PrintStackTrace() (/home/akhila/R/x86_64-pc-linux-gnu-library/3.6/RcppDeepState/testpkgs/testSAN/inst/testfiles/rcpp_read_out_of_bound/rcpp_read_out_of_bound_DeepState_TestHarness_LF+0x478948)
    #2 0x45db4c in fuzzer::Fuzzer::ExitCallback() (/home/akhila/R/x86_64-pc-linux-gnu-library/3.6/RcppDeepState/testpkgs/testSAN/inst/testfiles/rcpp_read_out_of_bound/rcpp_read_out_of_bound_DeepState_TestHarness_LF+0x45db4c)
    #3 0x7f0f9a38ca26 in __run_exit_handlers /build/glibc-ZN95T4/glibc-2.31/stdlib/exit.c:108:8
    #4 0x7f0f9a38cbdf in exit /build/glibc-ZN95T4/glibc-2.31/stdlib/exit.c:139:3
    #5 0x7f0f9a964c2a in Rf_initialize_R (/lib/libR.so+0x269c2a)
    #6 0x7f0f9a95f4ec in Rf_initEmbeddedR (/lib/libR.so+0x2644ec)
    #7 0x7f0f9a6e804f in RInside::initialize(int, char const* const*, bool, bool, bool) /tmp/RtmpppBlNo/R.INSTALL1906e07ad5449/RInside/src/RInside.cpp:155:21
    #8 0x7f0f9a6e8483 in RInside::RInside() /tmp/RtmpppBlNo/R.INSTALL1906e07ad5449/RInside/src/RInside.cpp:61:15
    #9 0x55e306 in DeepState_Test_testSAN_deepstate_test_rcpp_read_out_of_bound_test() /home/akhila/R/x86_64-pc-linux-gnu-library/3.6/RcppDeepState/testpkgs/testSAN/inst/testfiles/rcpp_read_out_of_bound/rcpp_read_out_of_bound_DeepState_TestHarness.cpp:12:11
    #10 0x556cf8 in DeepState_Run_testSAN_deepstate_test_rcpp_read_out_of_bound_test() /home/akhila/R/x86_64-pc-linux-gnu-library/3.6/RcppDeepState/testpkgs/testSAN/inst/testfiles/rcpp_read_out_of_bound/rcpp_read_out_of_bound_DeepState_TestHarness.cpp:11:1
    #11 0x56f7f7 in DeepState_RunTestNoFork (/home/akhila/R/x86_64-pc-linux-gnu-library/3.6/RcppDeepState/testpkgs/testSAN/inst/testfiles/rcpp_read_out_of_bound/rcpp_read_out_of_bound_DeepState_TestHarness_LF+0x56f7f7)
    #12 0x56f60a in LLVMFuzzerTestOneInput (/home/akhila/R/x86_64-pc-linux-gnu-library/3.6/RcppDeepState/testpkgs/testSAN/inst/testfiles/rcpp_read_out_of_bound/rcpp_read_out_of_bound_DeepState_TestHarness_LF+0x56f60a)
    #13 0x45f151 in fuzzer::Fuzzer::ExecuteCallback(unsigned char const*, unsigned long) (/home/akhila/R/x86_64-pc-linux-gnu-library/3.6/RcppDeepState/testpkgs/testSAN/inst/testfiles/rcpp_read_out_of_bound/rcpp_read_out_of_bound_DeepState_TestHarness_LF+0x45f151)
    #14 0x45e895 in fuzzer::Fuzzer::RunOne(unsigned char const*, unsigned long, bool, fuzzer::InputInfo*, bool*) (/home/akhila/R/x86_64-pc-linux-gnu-library/3.6/RcppDeepState/testpkgs/testSAN/inst/testfiles/rcpp_read_out_of_bound/rcpp_read_out_of_bound_DeepState_TestHarness_LF+0x45e895)
    #15 0x461071 in fuzzer::Fuzzer::ReadAndExecuteSeedCorpora(std::__Fuzzer::vector<fuzzer::SizedFile, fuzzer::fuzzer_allocator<fuzzer::SizedFile> >&) (/home/akhila/R/x86_64-pc-linux-gnu-library/3.6/RcppDeepState/testpkgs/testSAN/inst/testfiles/rcpp_read_out_of_bound/rcpp_read_out_of_bound_DeepState_TestHarness_LF+0x461071)
    #16 0x461519 in fuzzer::Fuzzer::Loop(std::__Fuzzer::vector<fuzzer::SizedFile, fuzzer::fuzzer_allocator<fuzzer::SizedFile> >&) (/home/akhila/R/x86_64-pc-linux-gnu-library/3.6/RcppDeepState/testpkgs/testSAN/inst/testfiles/rcpp_read_out_of_bound/rcpp_read_out_of_bound_DeepState_TestHarness_LF+0x461519)
    #17 0x4501ee in fuzzer::FuzzerDriver(int*, char***, int (*)(unsigned char const*, unsigned long)) (/home/akhila/R/x86_64-pc-linux-gnu-library/3.6/RcppDeepState/testpkgs/testSAN/inst/testfiles/rcpp_read_out_of_bound/rcpp_read_out_of_bound_DeepState_TestHarness_LF+0x4501ee)
    #18 0x479032 in main (/home/akhila/R/x86_64-pc-linux-gnu-library/3.6/RcppDeepState/testpkgs/testSAN/inst/testfiles/rcpp_read_out_of_bound/rcpp_read_out_of_bound_DeepState_TestHarness_LF+0x479032)
    #19 0x7f0f9a36a0b2 in __libc_start_main /build/glibc-ZN95T4/glibc-2.31/csu/../csu/libc-start.c:308:16
    #20 0x424f8d in _start (/home/akhila/R/x86_64-pc-linux-gnu-library/3.6/RcppDeepState/testpkgs/testSAN/inst/testfiles/rcpp_read_out_of_bound/rcpp_read_out_of_bound_DeepState_TestHarness_LF+0x424f8d)

make: *** [libfuzz.Makefile:5: /home/akhila/R/x86_64-pc-linux-gnu-library/3.6/RcppDeepState/testpkgs/testSAN/inst/testfiles/rcpp_read_out_of_bound/libfuzzer_rcpp_read_out_of_bound_log] Error 77

The code aborts because of an internal call to exit() from the RInside object initialization.

[tdhock]

hi @eddelbuettel I know this is not easily reproducible (without installing libfuzzer,deepstate) but do you have any idea why exit is being called here? libfuzzer does not like that
From the stack trace it looks like it is being called when RInside is instantiated, RInside R; which calls RInside::initialize, Rf_initEmbeddedR, Rf_initialize_R, and then exit. Why would R call exit and how can we avoid that? I'm pretty confused here.

[eddelbuettel]

The call sequence is from #1 to #20, isn't it? I think the exit() you see in #3 and #4 may be spurious.

But in any event, and you could check that via grep -r, we don't call exit() in Rcpp or RInside and I doubt R does. So I don't know, and I don't have any leads.

[tdhock]

Hi Dirk thanks for the quick response, that helped actually.
There are three places I see exit in
https://github.com/wch/r-source/blob/9c9e8ab03b499f0da3c490238d40fb5b9f904fb4/src/unix/system.c
I bet the culprit is https://github.com/wch/r-source/blob/9c9e8ab03b499f0da3c490238d40fb5b9f904fb4/src/unix/system.c#L197 which calls exit if num_initialized is non-zero, which I think means that R has already been initialized. Is libfuzzer possibly running the test harness code / RInside instantiation multiple times in the same process @akhikolla ? If you could try fixing by moving the Rinside instantiation outside/before the TEST() block.
The num_initialized is defined here

/* Protection against embedded misuse, PR#15420 */
static int num_initialized = 0;

This R-devel bug https://bugs.r-project.org/bugzilla/show_bug.cgi?id=15420 did not really help me much to understand what is going on.

[akhikolla]

I don't think libfuzzer instantiate R more than once. Libfuzzer is an in-process fuzzer which means fuzzing happens in only one process, for every test case, the process isn't restarted but the input values are changed in the memory.
I can work on changing the R instantiation inside the testharness and maybe I'll create the main function just for R initialization?

[eddelbuettel]

That could help. Try to do setup only once, and then just call workloads. You cannot run the <construct,destruct> pair in a loop as R was never built for that; I think some internal may not reset.

BTW I think your repo is not complete and the example is not self-contained: rbound_stream.open("/home/akhila/R/x86_64-pc-linux-gnu-library/3.6/RcppDeepState/testpkgs/testSAN/inst/testfiles/rcpp_read_out_of_bound/inp>

[akhikolla]

Thanks for your suggestion. Test Fixtures in deepstate provide SetUp() and TearDown() where we can initialize an object, destruct an object respectively.
I apologize. I didn't update it properly.

Here is the complete code:

#include <fstream>
#include <RInside.h>
#include <iostream>
#include <RcppDeepState.h>
#include <qs.h>
#include <DeepState.hpp>
#include <ctime>

int rcpp_read_out_of_bound(int rbound);

TEST(testSAN_deepstate_test,rcpp_read_out_of_bound_test){
  RInside R;
  std::cout << "input starts" << std::endl;
  std::ofstream rbound_stream;
  int rbound  = RcppDeepState_int();
  rbound_stream.open("/home/akhila/R/x86_64-pc-linux-gnu-library/3.6/RcppDeepState/testpkgs/testSAN/inst/testfiles/rcpp_read_out_of_bound/inputs/rbound");
  rbound_stream << rbound;
  std::cout << "rbound values: "<< rbound << std::endl;
  rbound_stream.close();
  std::cout << "input ends" << std::endl;
  try{
     rcpp_read_out_of_bound(rbound);
  }
  catch(Rcpp::exception& e){
    std::cout<<"Exception Handled"<<std::endl;
  }
}

[eddelbuettel]

What I meant is whether this file is public or not:

/home/akhila/R/x86_64-pc-linux-gnu-library/3.6/RcppDeepState/testpkgs/testSAN/inst/testfiles/rcpp_read_out_of_bound/inputs/rbound"

The path you use is not. You could access a file in a package via system.file(..., package=....) .

[akhikolla]

The file is not public. The testharness which I presented is automatically generated by the R code and Yes I make use of system.file to get the path and I write the obtained path to the Testharness.cpp file. For every Rcpp function in the package, the R code generates a Testharness and makes a call to the rcpp function inside the testharness.

[eddelbuettel]

The file is not public.

Is there a deeper philosphical reason I am not seeing? Everything else here is open source.

[akhikolla]

The R code uses system.file() to get the path to the "rbound" file in the RcppDeepState package and writes it to the Testharness.
The Testharness files generated are system-specific and paths provided in the file are system-specific too.
These Testharness files, makefile are written by R.
There is no specific reason why we are not using system.file in the Testharness.cpp directly.

All the inputs passed to the TestHarness by deepstate are usually stored in binary files(.crash/.fail/.pass files). Reading data from these files is not possible so we created a file(rbound) that stores the inputs provided by RcppDeepState_int().
We just save the inputs so that we know which input is causing the error on the code.

[FabrizioSandri]

FabrizioSandri/RcppDeepState#7