Benchmark, FIO, GSoC, RTEMS, Software

[GSoC Phase-1] Porting FIO to RTEMS

Hi all, This post covers most of my learnings during phase one of GSoC with RTEMS. My main goal of phase 1 was to port an IO benchmark to RTEMS. Towards this, various benchmarks were compared and fio was finalized for the import. FIO is a pretty much widely accepted and highly configurable IO benchmarking tool, which supports a number of large number of different OS and now RTEMS is among one of them.

Following were the major considerations i learned while porting a 3rd party user-space tool to RTEMS(I’ll try to generalize as much as possible, but there will be few fio specific portions too):

Cross-Compiling fio for RTEMS

Most of the applications nowadays, have support for cross-compilation and fio was no exception. So, first step here is the generation of appropriate  RTEMS toolchain for the desired architecture(In my case, I’ll be working with BeagleBone Black so, the architecture will be arm).

Generation of toolchain

In case of RTEMS, required toolchain can be generated via RTEMS Source Builder(RSB). Complete documentation and detailed steps can be found here: https://docs.rtems.org/branches/master/rsb/source-builder.html . Following are the outline steps:

Now, under the directory sandbox/5/bin/ there will be the generated compiler(arm-rtems5-gcc), linker etc. The path [workspace path]/sandbox/5 has to be passed to fio through a variable $TOOL_PATH. So, set the variable  like:

Building BSP

FIO works mainly on pthreads. so we need to provide posix support, which will be done via building the Board support package with –enable-posix option:

Now, the toolchain is ready with posix support and can be passed to fio.

Configuring compiler and linker calls

This step is actually very application specific and requires you to provide with all flags/options/directories required to cross compile an application. In my case, following were how my compiler and linker call look likes:

and thus i need to add following to fio’s makefile:

Just notice here the TOOL_PATH variable that we earlier defined, is here used to pass the path for the include directory which contains all the BSP specific library files(here, mainly posix ones are required) to the loader.

Configuring FIO

FIO has to be configured via:

the cc path here points to our rtems-specific compiler.

Note:-  –disable-optimization flag here removes the -O3, U_FORTIFY_SOURCE, D_FORTIFY_SOURCE flags from the compiler/loader calls. We had an issue by using the U_FORTIFY_SOURCE, D_FORTIFY_SOURCE compiler optimizations. Details of which can be found here https://lists.rtems.org/pipermail/devel/2018-May/021747.html

and then make call looks like:

Here, V=1 turns on verbose output.

Invoking FIO as a shell command from RTEMS

After cross compiling fio and resolving all it’s dependencies few linker errors like the following will appear due to missing RTEMS configuration.

Here, In our case, fio is configured as a shell command so first rtems-shell has to be initialized then sd card has to be mounted before starting the shell. Then, when user invokes fio from shell command, it will just call fio’s constructors and will then call fio’s main function. Following was the configuration used by us:

Notice here, that upon call to fio rtems_bsd_command_fio is called. Here’s the defination of this function:

Note:- RTEMS doesn’t support application constructors, so they have to be called like normal functions right before calling the main. Same is true for destructors, they are called as main ends. mainwrapper function is used for this purpose.

Notice rtems_bsd_program_lock() function here, this function along with few others are used for resource tracking and cleanup. It will be covered in detail in next section.

Implementing memory cleanup and Resource Tracking

FIO, just like most other user space tools expects a memory management system that cleanup the resources when fio exits. However, RTEMS just to be consize and Real time doesn’t have any such prebuilt memory management system, so we need to externally implement this. Most of the work here can be done simply via userspace-header-gen.py script(credits to Christian mauderer and Sebestian huber). Detailed instructions regarding porting of user space utilities are documented here: https://git.rtems.org/rtems-libbsd/tree/CONTRIBUTING.md#n151 . Here’s an overview of the steps:

  • Replace getopt()/getopt_long()/getopt_long_only() with their reentrant versions like getopt_r(),getopt_long_r() and getopt_long_only_r() respectively. Normal getopt versions uses global state variables to pass information so, when the application is called second time getopt state isn’t reset(another consequence of absence of memory cleanup). Moreover, if a function isn’t reentrant it has to be called with an interrtupt lock.

 

  • Complie the program as usual. That will generate object files corresponding to each source files. Put all the object files(.c) in one folder. and call userspace-header-gen.py script as:

    In my case it was: uka_in@Madaari:~/development/benchmark/sandbox/rtems-libbsd$ python ./userspace-header-gen.py ../../fio/fio/*.o -p fio

    Note:-  In case of error like the one below, one may need to analyse all the object file maybe via dwarfdump command.

    These are probably due to because of empty object file and thus missing the debug info. Also, one might have to install pyelf tools to full all the package dependencies required by the script.

  • It will generate one rtems-bsd-PROGNAME-MODULE-data.h per object file, one
    rtems-bsd-PROGNAME-namespace.h and one rtems-bsd-PROGNAME-data.h. These files have to be included in all source files as(at the beginning of source file)

    and then

    at the end of the sorce file.

  • After this, fio’s main function is to be called like:

    rtems_bsd_program_lock() is used to begin resource tracking. It will keep a log of all memory locations allocated via malloc, calloc,mmap etc. and then rtems_bsd_program_unlock(); releases all the memory allocated to the programe.

  • Apart from memory, this script also keeps a log of all global and static variables which are reinitialized each time the application is invoked.

Conclusion

Phase 1 concludes by adding the support for the following fio ioengines to RTEMS:

  • psync – Uses basic pread, pwrite I/O
  • vsync – Uses readv , writev I/O
  • sync – Uses read , write I/O
  • null – Doesn’t transfer any data. Just pretends to. Mainly used for debugging,testing purpose
  • filecreate – Simply creates the files and don’t IO them. Used to benchmark the time taken to create a file
  • ftruncate – Uses ftruncate() operations in response to write operations.
Tagged , ,

1 thought on “[GSoC Phase-1] Porting FIO to RTEMS

Leave a Reply

Your email address will not be published.

This site uses Akismet to reduce spam. Learn how your comment data is processed.