Hi all, This post covers most of my learnings during phase of GSoC with RTEMS. My main goal of phase 1 was to port an IO benchmark to RTEMS. Towards this, various benchmarks were compared and fio was finalized for the import. FIO is a pretty much widely accepted and highly configurable IO benchmarking tool, which supports a number of large number of different OS and now RTEMS is among one of them.
Following were the major considerations i learned while porting a 3rd party user-space tool to RTEMS(I’ll try to generalize as much as possible, but there will be few fio specific portions too):
Cross-Compiling fio for RTEMS
Most of the applications nowadays, have support for cross-compilation and fio was no exception. So, first step here is the generation of appropriate RTEMS toolchain for the desired architecture(In my case, I’ll be working with BeagleBone Black so, the architecture will be arm).
Generation of toolchain
In case of RTEMS, required toolchain can be generated via RTEMS Source Builder(RSB). Complete documentation and detailed steps can be found here: https://docs.rtems.org/branches/master/rsb/source-builder.html . Following are the outline steps:
– Jump into your workspace and make a directory named sandbox This folder will contain all our project files – cd sandbox/ – export sandbox=$PWD – git clone git://git.rtems.org/rtems-source-builder.git – git clone git://git.rtems.org/rtems.git – git clone git://git.rtems.org/rtems-libbsd.git === Building desired toolset for ARM === – cd rtems-source-builder/rtems/ – ../source-builder/sb-check – ../source-builder/sb-set-builder --prefix=$sandbox/5 5/rtems-arm – cd $sandbox – export PATH=$PWD/5/bin:$PATH
Now, under the directory sandbox/5/bin/ there will be the generated compiler(arm-rtems5-gcc), linker etc. The path [workspace path]/sandbox/5 has to be passed to fio through a variable $TOOL_PATH. So, set the variable like:
export TOOL_PATH=/home/uka_in/development/benchmark/sandbox/5
Building BSP
FIO works mainly on pthreads. so we need to provide posix support, which will be done via building the Board support package with –enable-posix option:
=== Building Beagleboneblack BSP === – cd $sandbox – mkdir beagleboneblack – cd rtems – ./bootstrap -c && ./bootstrap – ../rtems-source-builder/source-builder/sb-bootstrap – cd .. – cd beagleboneblack/ – ../rtems/configure --prefix=$sandbox/5 --target=arm-rtems5 --enable-posix --disable-networking --enable-rtemsbsp=beagleboneblack – make – make install – cd $sandbox
Now, the toolchain is ready with posix support and can be passed to fio.
Configuring compiler and linker calls
This step is actually very application specific and requires you to provide with all flags/options/directories required to cross compile an application. In my case, following were how my compiler and linker call look likes:
../../sandbox/5/bin/arm-rtems5-gcc -o fio.o -std=gnu99 -Wwrite-strings -Wall -Wdeclaration-after-statement -g -ffast-math -D_GNU_SOURCE -include config-host.h -O3 -I. -I. -DBITS_PER_LONG=32 -I /home/uka_in/development/benchmark/sandbox/5/arm-rtems5/beagleboneblack/lib/include -ffunction-sections -fdata-sections -g -mcpu=cortex-a8 -fno-strict-aliasing -ffreestanding -fno-common -w -DHAVE_RTEMS_SCORE_CPUOPTS_H=1 -DHAVE_RTEMS_H=1 -DHAVE_DLFCN_H=1 -DHAVE_RTEMS_PCI_H=1 -DHAVE_RTEMS_RTEMS_DEBUGGER_H=1 -g -DFIO_VERSION='"fio-3.6-60-g62d6a-dirty"' -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -DFIO_INTERNAL -DFIO_INC_DEBUG -c fio.c
../../sandbox/5/bin/arm-rtems5-gcc -B /home/uka_in/development/benchmark/sandbox/5/arm-rtems5/beagleboneblack/lib -specs bsp_specs -qrtems -Wl,--gc-sections -std=gnu99 -Wwrite-strings -Wall -Wdeclaration-after-statement -g -ffast-math -D_GNU_SOURCE -include config-host.h -O3 -I. -I. -DBITS_PER_LONG=32 -I /home/uka_in/development/benchmark/sandbox/5/arm-rtems5/beagleboneblack/lib/include -ffunction-sections -fdata-sections -g -mcpu=cortex-a8 -fno-strict-aliasing -ffreestanding -fno-common -w -DHAVE_RTEMS_SCORE_CPUOPTS_H=1 -DHAVE_RTEMS_H=1 -DHAVE_DLFCN_H=1 -DHAVE_RTEMS_PCI_H=1 -DHAVE_RTEMS_RTEMS_DEBUGGER_H=1 -g -DFIO_VERSION='"fio-3.6-60-g62d6a-dirty"' -o fio crc/crc16.o crc/crc32.o crc/crc32c-arm64.o crc/crc32c-intel.o crc/crc32c.o crc/crc64.o crc/crc7.o crc/fnv.o crc/md5.o crc/murmur3.o crc/sha1.o crc/sha256.o crc/sha3.o crc/sha512.o crc/test.o crc/xxhash.o lib/axmap.o lib/bloom.o lib/flist_sort.o lib/gauss.o lib/getrusage.o lib/hweight.o lib/ieee754.o lib/lfsr.o lib/memalign.o lib/memcpy.o lib/mountcheck.o lib/num2str.o lib/output_buffer.o lib/pattern.o lib/prio_tree.o lib/rand.o lib/rbtree.o lib/strntol.o lib/zipf.o gettime.o ioengines.o init.o stat.o options.o log.o time.o filesetup.o eta.o verify.o memory.o io_u.o parse.o fio_sem.o rwlock.o pshared.o client.o server.o smalloc.o filehash.o profile.o debug.o engines/cpu.o engines/mmap.o engines/sync.o engines/null.o engines/ftruncate.o engines/filecreate.o iolog.o backend.o libfio.o flow.o cconv.o gettime-thread.o helpers.o json.o idletime.o td_error.o profiles/tiobench.o profiles/act.o io_u_queue.o filelock.o workqueue.o rate-submit.o optgroup.o helper_thread.o steadystate.o engines/net.o fio.o os/rtems/rtems-init.o -lm -Wl,-Bstatic -L. -lbsd -Wl,-Bdynamic -lbsd -lm -lz
and thus i need to add following to fio’s makefile:
LDFLAGS += -B $(TOOL_PATH)/arm-rtems5/beagleboneblack/lib -specs bsp_specs -qrtems -Wl,--gc-sections LIBS += -Wl,-Bstatic -L. -lbsd -Wl,-Bdynamic -lm -lz CFLAGS += -I $(TOOL_PATH)/arm-rtems5/beagleboneblack/lib/include -ffunction-sections -fdata-sections -g -mcpu=cortex-a8 -fno-strict-aliasing -ffreestanding -fno-common -w -DHAVE_RTEMS_SCORE_CPUOPTS_H=1 -DHAVE_RTEMS_H=1 -DHAVE_DLFCN_H=1 -DHAVE_RTEMS_PCI_H=1 -DHAVE_RTEMS_RTEMS_DEBUGGER_H=1 -g
Just notice here the TOOL_PATH variable that we earlier defined, is here used to pass the path for the include directory which contains all the BSP specific library files(here, mainly posix ones are required) to the loader.
Configuring FIO
FIO has to be configured via:
/fio$ ./configure --cc=../../sandbox/5/bin/arm-rtems5-gcc --disable-optimizations --extra-cflags=-O3 --disable-pmem
the cc path here points to our rtems-specific compiler.
Note:- –disable-optimization flag here removes the -O3, U_FORTIFY_SOURCE, D_FORTIFY_SOURCE flags from the compiler/loader calls. We had an issue by using the U_FORTIFY_SOURCE, D_FORTIFY_SOURCE compiler optimizations. Details of which can be found here https://lists.rtems.org/pipermail/devel/2018-May/021747.html
and then make call looks like:
$ make CROSS_COMPILE=../../sandbox/5/bin/arm-rtems5- V=1
Here, V=1 turns on verbose output.
Invoking FIO as a shell command from RTEMS
After cross compiling fio and resolving all it’s dependencies few linker errors like the following will appear due to missing RTEMS configuration.
../sandbox/BBB/arm-rtems5/beagleboneblack/lib/librtemscpu.a(libcsupport_a-rtems_putc.o): In function `rtems_putc': /home/uka_in/development/benchmark/sandbox/BBB/arm-rtems5/c/beagleboneblack/cpukit/libcsupport/../../../../../../rtems/c/src/../../cpukit/libcsupport/src/rtems_putc.c:30: undefined reference to `BSP_output_char' /home/uka_in/development/benchmark/sandbox/BBB/arm-rtems5/c/beagleboneblack/cpukit/libcsupport/../../../../../../rtems/c/src/../../cpukit/libcsupport/src/rtems_putc.c:30: undefined reference to `BSP_output_char' /home/uka_in/development/benchmark/sandbox/5/lib/gcc/arm-rtems5/7.3.0/../../../../arm-rtems5/lib/libg.a(lib_a-init.o): In function `__libc_init_array': /home/uka_in/development/benchmark/sandbox/rtems-source-builder/rtems/build/arm-rtems5-gcc-7.3.0-newlib-3.0.0-x86_64-linux-gnu-1/build/arm-rtems5/newlib/libc/misc/../../../../../gcc-7.3.0/newlib/libc/misc/init.c:37: undefined reference to `_init'
Here, In our case, fio is configured as a shell command so first rtems-shell has to be initialized then sd card has to be mounted before starting the shell. Then, when user invokes fio from shell command, it will just call fio’s constructors and will then call fio’s main function. Following was the configuration used by us:
#include <sys/param.h>
#include <sys/stat.h>
#include <sys/socket.h>
#include <assert.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <fcntl.h>
#include <ifaddrs.h>
#include <sysexits.h>
#include <machine/rtems-bsd-commands.h>
#include <net/if.h>
#include <rtems/bdbuf.h>
#include <rtems/console.h>
#include <rtems/ftpd.h>
#include <rtems/media.h>
#include <rtems/shell.h>
#include <rtems/telnetd.h>
#include <rtems.h>
#include <rtems/printer.h>
#include <rtems/stackchk.h>
#include <rtems/bsd/bsd.h>
#include <rtems/bsd/modules.h>
#include <rtems/dhcpcd.h>
#include <rtems/console.h>
#include <rtems/shell.h>
rtems_bsd_command_fio(int argc, char *argv[]);
static rtems_status_code
media_listener(rtems_media_event event, rtems_media_state state,
const char *src, const char *dest, void *arg)
{
if (dest != NULL) {
printf(", dest = %s", dest);
}
if (arg != NULL) {
printf(", arg = %p\n", arg);
}
return RTEMS_SUCCESSFUL;
}
rtems_shell_cmd_t rtems_shell_fio_Command = {
.name = "fio",
.usage = "fio --help",
.topic = "user",
.command = rtems_bsd_command_fio
};
static void
early_initialization(void)
{
rtems_status_code sc;
sc = rtems_bdbuf_init();
assert(sc == RTEMS_SUCCESSFUL);
sc = rtems_media_initialize();
assert(sc == RTEMS_SUCCESSFUL);
sc = rtems_media_listener_add(media_listener, NULL);
assert(sc == RTEMS_SUCCESSFUL);
sc = rtems_media_server_initialize(
200,
32 * 1024,
RTEMS_DEFAULT_MODES,
RTEMS_DEFAULT_ATTRIBUTES
);
assert(sc == RTEMS_SUCCESSFUL);
}
void
Init(rtems_task_argument arg)
{
rtems_status_code sc;
puts("\n*** FIO - Flexible I/O tester ***\n\n");
early_initialization();
rtems_bsd_initialize();
/* Let the callout timer allocate its resources */
sc = rtems_task_wake_after(2);
assert(sc == RTEMS_SUCCESSFUL);
sc = rtems_shell_init("SHLL", 16 * 1024, 1, CONSOLE_DEVICE_NAME,
false, true, NULL);
assert(sc == RTEMS_SUCCESSFUL);
assert(0);
}
#define DEFAULT_NETWORK_DHCPCD_ENABLE
#define CONFIGURE_MICROSECONDS_PER_TICK 1000
#define CONFIGURE_MAXIMUM_DRIVERS 32
#define CONFIGURE_FILESYSTEM_DOSFS
#define CONFIGURE_MAXIMUM_PROCESSORS 32
/*
* Configure LibBSD.
*/
#define RTEMS_BSD_CONFIG_NET_PF_UNIX
#define RTEMS_BSD_CONFIG_NET_IP_MROUTE
#define RTEMS_BSD_CONFIG_NET_IP6_MROUTE
#define RTEMS_BSD_CONFIG_NET_IF_BRIDGE
#define RTEMS_BSD_CONFIG_NET_IF_LAGG
#define RTEMS_BSD_CONFIG_NET_IF_VLAN
#define RTEMS_BSD_CONFIG_BSP_CONFIG
#define RTEMS_BSD_CONFIG_INIT
#include <machine/rtems-bsd-config.h>
#define CONFIGURE_APPLICATION_NEEDS_CLOCK_DRIVER
#define CONFIGURE_APPLICATION_NEEDS_CONSOLE_DRIVER
#define CONFIGURE_APPLICATION_NEEDS_STUB_DRIVER
#define CONFIGURE_APPLICATION_NEEDS_ZERO_DRIVER
#define CONFIGURE_APPLICATION_NEEDS_LIBBLOCK
#define CONFIGURE_LIBIO_MAXIMUM_FILE_DESCRIPTORS 32
#define CONFIGURE_MAXIMUM_USER_EXTENSIONS 1
#define CONFIGURE_UNLIMITED_ALLOCATION_SIZE 32
#define CONFIGURE_UNLIMITED_OBJECTS
#define CONFIGURE_UNIFIED_WORK_AREAS
#define CONFIGURE_STACK_CHECKER_ENABLED
/* Turn cache off */
#define CONFIGURE_BDBUF_BUFFER_MAX_SIZE (4 * 1024)
#define CONFIGURE_BDBUF_MAX_READ_AHEAD_BLOCKS 0
#define CONFIGURE_BDBUF_CACHE_MEMORY_SIZE (1 * 1024 * 1024)
#define CONFIGURE_RTEMS_INIT_TASKS_TABLE
#define CONFIGURE_INIT_TASK_STACK_SIZE (128 * 1024)
#define CONFIGURE_INIT_TASK_INITIAL_MODES RTEMS_DEFAULT_MODES
#define CONFIGURE_INIT_TASK_ATTRIBUTES RTEMS_FLOATING_POINT
#define CONFIGURE_INIT
#define CPU_STACK_MINIMUM_SIZE (128*1024)
#include <rtems/confdefs.h>
#define CONFIGURE_SHELL_COMMANDS_INIT
#include <bsp/irq-info.h>
#include <rtems/netcmds-config.h>
#define CONFIGURE_SHELL_USER_COMMANDS \
&bsp_interrupt_shell_command, \
&rtems_shell_ARP_Command, \
&rtems_shell_HOSTNAME_Command, \
&rtems_shell_PING_Command, \
&rtems_shell_ROUTE_Command, \
&rtems_shell_NETSTAT_Command, \
&rtems_shell_IFCONFIG_Command, \
&rtems_shell_TCPDUMP_Command, \
&rtems_shell_SYSCTL_Command, \
&rtems_shell_VMSTAT_Command, \
&rtems_shell_fio_Command
#define CONFIGURE_SHELL_COMMAND_CPUINFO
#define CONFIGURE_SHELL_COMMAND_CPUUSE
#define CONFIGURE_SHELL_COMMAND_PERIODUSE
#define CONFIGURE_SHELL_COMMAND_STACKUSE
#define CONFIGURE_SHELL_COMMAND_PROFREPORT
#define CONFIGURE_SHELL_COMMAND_CP
#define CONFIGURE_SHELL_COMMAND_PWD
#define CONFIGURE_SHELL_COMMAND_LS
#define CONFIGURE_SHELL_COMMAND_LN
#define CONFIGURE_SHELL_COMMAND_LSOF
#define CONFIGURE_SHELL_COMMAND_CHDIR
#define CONFIGURE_SHELL_COMMAND_CD
#define CONFIGURE_SHELL_COMMAND_MKDIR
#define CONFIGURE_SHELL_COMMAND_RMDIR
#define CONFIGURE_SHELL_COMMAND_CAT
#define CONFIGURE_SHELL_COMMAND_MV
#define CONFIGURE_SHELL_COMMAND_RM
#define CONFIGURE_SHELL_COMMAND_MALLOC_INFO
#define CONFIGURE_SHELL_COMMAND_SHUTDOWN
#define CONFIGURE_SHELL_COMMAND_FDISK
#define CONFIGURE_SHELL_COMMAND_BLKSTATS
#define CONFIGURE_SHELL_COMMAND_BLKSYNC
#define CONFIGURE_SHELL_COMMAND_MSDOSFMT
#define CONFIGURE_SHELL_COMMAND_DF
#define CONFIGURE_SHELL_COMMAND_MOUNT
#define CONFIGURE_SHELL_COMMAND_UNMOUNT
#define CONFIGURE_SHELL_COMMAND_MSDOSFMT
#define CONFIGURE_SHELL_COMMAND_EDIT
#define CONFIGURE_SHELL_COMMAND_GETENV
#define CONFIGURE_SHELL_COMMAND_SETENV
#define CONFIGURE_SHELL_COMMAND_UNSETENV
#include <rtems/shellconfig.h>
Notice here, that upon call to fio rtems_bsd_command_fio is called. Here’s the defination of this function:
#ifdef __rtems__
void act_register(void);
void tiobench_register(void);
void fio_syncio_register(void);
void fio_filecreate_register(void);
void fio_null_register(void);
void prio_tree_init(void);
void fio_syncio_register_ft(void);
void fio_client_hash_init(void);
void fio_syncio_unregister(void);
void tiobench_unregister(void);
void fio_filecreate_unregister(void);
void fio_null_unregister(void);
void act_unregister(void);
void fio_syncio_unregister_ft(void);
static int
mainwrapper(int argc, char *argv[])
{
int err=0;
/* Constructors */
act_register();
tiobench_register();
fio_syncio_register();
fio_filecreate_register();
fio_null_register();
prio_tree_init();
fio_syncio_register_ft();
fio_client_hash_init();
err = main(argc, argv, (char *)NULL);
/* Destructors */
fio_syncio_unregister();
tiobench_unregister();
fio_cpuio_unregister();
fio_filecreate_unregister();
fio_mmapio_unregister();
fio_null_unregister();
act_unregister();
fio_netio_unregister();
fio_syncio_unregister_ft();
}
RTEMS_LINKER_RWSET(bsd_prog_fio, char);
rtems_bsd_command_fio(int argc, char *argv[])
{
int exit_code;
void *data_begin;
size_t data_size;
data_begin = RTEMS_LINKER_SET_BEGIN(bsd_prog_fio);
data_size = RTEMS_LINKER_SET_SIZE(bsd_prog_fio);
rtems_bsd_program_lock();
exit_code = rtems_bsd_program_call_main_with_data_restore("fio",
mainwrapper, argc, argv, data_begin, data_size);
rtems_bsd_program_unlock();
return exit_code;
}
#endif /* __rtems__ */
Note:- RTEMS doesn’t support application constructors, so they have to be called like normal functions right before calling the main. Same is true for destructors, they are called as main ends. mainwrapper function is used for this purpose.
Notice rtems_bsd_program_lock() function here, this function along with few others are used for resource tracking and cleanup. It will be covered in detail in next section.
Implementing memory cleanup and Resource Tracking
FIO, just like most other user space tools expects a memory management system that cleanup the resources when fio exits. However, RTEMS just to be consize and Real time doesn’t have any such prebuilt memory management system, so we need to externally implement this. Most of the work here can be done simply via userspace-header-gen.py script(credits to Christian mauderer and Sebestian huber). Detailed instructions regarding porting of user space utilities are documented here: https://git.rtems.org/rtems-libbsd/tree/CONTRIBUTING.md#n151 . Here’s an overview of the steps:
- Replace getopt()/getopt_long()/getopt_long_only() with their reentrant versions like getopt_r(),getopt_long_r() and getopt_long_only_r() respectively. Normal getopt versions uses global state variables to pass information so, when the application is called second time getopt state isn’t reset(another consequence of absence of memory cleanup). Moreover, if a function isn’t reentrant it has to be called with an interrtupt lock.
- Complie the program as usual. That will generate object files corresponding to each source files. Put all the object files(.c) in one folder. and call userspace-header-gen.py script as:
python ./userspace-headerspace-gen.py [path to folder]/*.o -p [program name]
In my case it was: uka_in@Madaari:~/development/benchmark/sandbox/rtems-libbsd$ python ./userspace-header-gen.py ../../fio/fio/*.o -p fio
Note:- In case of error like the one below, one may need to analyse all the object file maybe via dwarfdump command.
Traceback (most recent call last): | File "./userspace-header-gen.py", line 515, in <module> | uhg.generate_header(dataoutfilename, globdataoutfile, namespaceoutfile) | File "./userspace-header-gen.py", line 421, in generate_header | raise NoDwarfInfoError() | __main__.NoDwarfInfoError: Input file has no DWARF info. |
These are probably due to because of empty object file and thus missing the debug info. Also, one might have to install pyelf tools to full all the package dependencies required by the script.
- It will generate one `rtems-bsd-PROGNAME-MODULE-data.h` per object file, one
`rtems-bsd-PROGNAME-namespace.h` and one `rtems-bsd-PROGNAME-data.h`. These files have to be included in all source files as(at the beginning of source file)#ifdef __rtems__ #include <machine/rtems-bsd-user-space.h> #include <machine/rtems-bsd-program.h> #include "os/rtems/headers/rtems-bsd-PROGNAME-namespace.h" #endif /* __rtems__ */
and then
#ifdef __rtems__ #include "os/rtems/headers/rtems-bsd-PROGNAME-MODULE-data.h" #endif /* __rtems__ */
at the end of the sorce file.
- After this, fio’s main function is to be called like:
int exit_code; void *data_begin; size_t data_size; data_begin = RTEMS_LINKER_SET_BEGIN(bsd_prog_fio); data_size = RTEMS_LINKER_SET_SIZE(bsd_prog_fio); rtems_bsd_program_lock(); exit_code = rtems_bsd_program_call_main_with_data_restore("fio", mainwrapper, argc, argv, data_begin, data_size); rtems_bsd_program_unlock();rtems_bsd_program_lock() is used to begin resource tracking. It will keep a log of all memory locations allocated via malloc, calloc,mmap etc. and then rtems_bsd_program_unlock(); releases all the memory allocated to the programe.
- Apart from memory, this script also keeps a log of all global and static variables which are reinitialized each time the application is invoked.
Conclusion
Phase 1 concludes by adding the support for the following fio ioengines to RTEMS:
- psync – Uses basic pread, pwrite I/O
- vsync – Uses readv , writev I/O
- sync – Uses read , write I/O
- null – Doesn’t transfer any data. Just pretends to. Mainly used for debugging,testing purpose
- filecreate – Simply creates the files and don’t IO them. Used to benchmark the time taken to create a file
- ftruncate – Uses ftruncate operations in response to write operations.