Hi, In this post we will explore FIO’s RTEMS port and how it can be used to benchmark RTEMS filesystems and drivers.
First, let’s have a quick look at all the RTEMS filesystems:
RTEMS FILESYSTEMS
RTEMS supports mainly two types of filesystems: Network and Physical fs. Benchmarking support for networking filesystems isn’t yet available. However, nearly every other physical filesystem can be benchmarked and contrasted. Heap-based file systems are those which uses malloc() for file allocation. In other words, they reside completely on heap memory. These are mainly used to provide basic directory/file management even if there is no dedicated physical storage for files. Further, they facilitate mounting of other file systems too. For benchmarking IMFS/M-IMFS no external device is needed, unlike block-based fs which need either a RAM-disk or a flash device to work with. Further detailed information of these filesystems is available here: https://devel.rtems.org/wiki/Developer/FileSystems
Now, let’s move on to the benchmarking section (for which this post is dedicated to):
Step 1# Preparation of the benchmarking tool
As of today(02/08/2018) FIO’s RTEMS port isn’t yet merged into the fio’s official repository(here’s the corresponding thread: https://www.spinics.net/lists/fio/msg07157.html). I’ll update this section, once it gets merged. Till then we can use my repository https://github.com/madaari/fio/tree/paper . Please stick to the ‘paper‘ branch as it will be much more stable then the master one. For benchmarking purposes, it’s always recommended to use only one version of the tool throughout the results.
Preparation of RTEMS toolchain
Toolchain, required for cross-compiling fio for desired architecture (like ARM in this example) can be generated by using RTEMS Source Builder(RSB).
Please note that fio’s build for RTEMS has been tested with RSB version 5(commit id :25f4db09c85a52fb1640a29f9bdc2de8c2768988), and it may not work with older versions.
Moreover, For enabling POSIX support(required by fio), build the BSP using RTEMS v5 with –enable-posix option. After that, if needed(like for using SD card driver for BeagleBone Black) one may also need to build rtems-libbsd for the desired BSP.
Cross-Compiling FIO
Fetch the fio repository and then configure it as shown below:
Variable to be passed for toolchain path is TOOL_PATH_PREFIX, which in this case would be,
$ export TOOL_PATH_PREFIX=/home/uka_in/development/sandbox/5
After setting up the variable, the next step would be to configure and build fio as:
$ make clean $ ./configure --cc=$TOOL_PATH_PREFIX/bin/arm-rtems5-gcc --disable-optimizations --extra-cflags=-O3 $ make fio CROSS_COMPILE=$TOOL_PATH_PREFIX/bin/arm-rtems5- V=1
By now, you will have the fio binary ready which can be loaded to the target using the bootloader like u-boot.
Note:- While using u-boot to load up the fio binary use the following configuration:
boot=fatload mmc 0 0x81000000 $app ; fatload mmc 0 0x88000000 ${DTB_INSTALL_NAME} ; bootm 0x81000000 - 0x88000000"
where $app is the name of the image file to be loaded. Just make sure, there is enough space between the loading and executing addresses. Otherwise, I would recommend using this script after making above mentioned changes(mainly change the load address from 0x80800000 to 0x81000000). The main reason for this is to avoid the following error during uncompressing:
## Booting kernel from Legacy Image at 80800000 ... Image Name: RTEMS Created: 2018-07-15 12:05:41 UTC Image Type: ARM Linux Kernel Image (gzip compressed) Data Size: 1930907 Bytes = 1.8 MiB Load Address: 80000000 Entry Point: 80000000 Verifying Checksum ... OK ## Flattened Device Tree blob at 88000000 Booting using the fdt blob at 0x88000000 Uncompressing Kernel Image ... Error: inflate() returned -5 Image too large: increase CONFIG_SYS_BOOTM_LEN Must RESET board to recover
It was because during gunzip(at loadaddr i.e 0x80000000) uboot overwrite the gzipped kernel image(at 0x80800000) thus an error during uncompressing. It’s resolved via a small change in uenv.txt: `boot=fatload mmc 0 0x81000000 rtems-app.img ; fatload mmc 0 0x88000000 am335x-boneblack.dtb ; bootm 0x81000000 – 0x88000000`
Step 2# Setting up the Benchmarking Environment
The user might want to customize the rtems configuration for simulating different environments (to notice their effect on benchmarking stats) or might want to set up a filesystem on any mounted device like flash, RAM-disk. This sub-section will begin with a small description of rtems-init.c file and will then describe the changes required for benchmarking different file systems and device drivers.
Just like any for any other rtems application, rtems-init.c file is the main configuration file with Init() function, device drivers, and libblock/cache settings. Making a change in this file will not in any way affect the benchmarking tool aka fio but would alter the benchmarking environment.
Evaluating the effect of cache size
For example, if someone wanted to evaluate the effect of cache size on the stats, he can change the libblock and cache settings from the following lines:
#define CONFIGURE_BDBUF_BUFFER_MAX_SIZE (1024) #define CONFIGURE_BDBUF_MAX_READ_AHEAD_BLOCKS 0 #define CONFIGURE_BDBUF_CACHE_MEMORY_SIZE (10 * 1024)
Note:- In case of using DOSFS, don’t set the cache size below 8k that might have an adverse effect on the rtems-shell application(In my case, it didn’t even load up ). Also, Heap-based filesystems like IMFS/M-IMFS are virtually unaffected by any change in cache settings. That has a pretty straight forward reason: Heap-based fs just don’t use libblock API. Further, information/description of these options can be found in RTEMS documentation.
Setting up RAM-disk
For benchmarking on a RAM-disk, the following configuration is required:
//ramdisk path can be /dev/rda and mount path can be /mnt
//This configuration is for setting up a ramdisk with RFS as filesystem
Init(){
...
rv = rtems_rfs_format(RAMDISK_PATH, &rfs_config);
assert(rv == 0);
rv = mount_and_make_target_path(
RAMDISK_PATH,
MOUNT_PATH,
RTEMS_FILESYSTEM_TYPE_RFS,
RTEMS_FILESYSTEM_READ_WRITE,
NULL
);
assert(rv == 0);
...
}
rtems_ramdisk_config rtems_ramdisk_configuration[] ={
{ .block_size = 512, .block_num = 131072*2 }
};
size_t rtems_ramdisk_configuration_size = RTEMS_ARRAY_SIZE(rtems_ramdisk_configuration);
#define CONFIGURE_APPLICATION_EXTRA_DRIVERS RAMDISK_DRIVER_TABLE_ENTRY
Setting the RAM-disk block size other then 512B might not work, in that case, rtems_rfs_format won’t successfully exit. Also, note that for IMFS you don’t need to set up the RAM-disk since Heap-based fs allocate on RAM itself! Also, please carefully choose the RAM-disk size. for example, In my case, BeagleBone Black has 512MB of RAM, while RTEMS BBB BSP uses only 216MB of it. Use it’s wise to uses only 128MB(50% of RAM available) for RAM-disk. For benchmarking, it’s always better to use larger-sized files for consistent bandwidth stats. So, a very small RAM-disk size is also not viable.
As a general observation, the cache has a negative effect on RAM-disk bandwidth!!
Setting up the SD card
For benchmarking the SD card driver, one can use the media_server(config given below) to mount the card. Please note that the SD card driver acts as a bottle-neck and thus evaluating RTEMS filesystems on an SD-card isn’t really a good idea.
#include <rtems/media.h>
static rtems_status_code
media_listener(rtems_media_event event, rtems_media_state state,
const char *src, const char *dest, void *arg)
{
if (dest != NULL) {
printf(", dest = %s", dest);
}
if (arg != NULL) {
printf(", arg = %p\n", arg);
}
return RTEMS_SUCCESSFUL;
}
static void
early_initialization(void)
{
rtems_status_code sc;
sc = rtems_bdbuf_init();
assert(sc == RTEMS_SUCCESSFUL);
sc = rtems_media_initialize();
assert(sc == RTEMS_SUCCESSFUL);
sc = rtems_media_listener_add(media_listener, NULL);
assert(sc == RTEMS_SUCCESSFUL);
sc = rtems_media_server_initialize(
200,
32 * 1024,
RTEMS_DEFAULT_MODES,
RTEMS_DEFAULT_ATTRIBUTES
);
assert(sc == RTEMS_SUCCESSFUL);
}
Benchmarking IMFS
IMFS being the default filesystem, is pretty easy to benchmark. However, there are some settings of IMFS which might need to be taken care of:
Init(){
...
rv = mount(NULL, "/mnt", "imfs", RTEMS_FILESYSTEM_READ_WRITE, NULL);
assert(rv == 0);
...
}
#define CONFIGURE_USE_IMFS_AS_BASE_FILESYSTEM
#define CONFIGURE_FILESYSTEM_IMFS
#define CONFIGURE_IMFS_MEMFILE_BYTES_PER_BLOCK 512
Also, please don’t try to mount RAM-disk using IMFS, it would fail: https://lists.rtems.org/pipermail/users/2018-July/032466.html
IMFS block size determines the size of the file that can be created. 512B block size allows maximum possible file size. However, for some unknown(at least to me) reasons benchmarking with a file size greater then 80MB-82MB raises an IO error. Maybe on some other boards or with other RAM sizes this limit gets changed.
Benchmarking RFS
For benchmarking RFS, the first step would be to set up a RAM disk with rtems_rfs_format and RTEMS_FILESYSTEM_TYPE_RFS settings. Next step would be to add RFS configuration like:
#include <rtems/rtems-rfs-format.h>
static const rtems_rfs_format_config rfs_config[] = {
{ .block_size = 1024 }
};
//use rfs_config as a parameter to rfs_format
Init(){
...
rv = rtems_rfs_format(RAMDISK_PATH, &rfs_config);
...
}
#define CONFIGURE_FILESYSTEM_RFS
Setting RFS block size other than 1024 might raise an error as well !!
Benchmarking DOSFS
For DOSFS, msdos_format() and RTEMS_FILESYSTEM_TYPE_DOSFS(while mounting RAM-disk) can be used. like:
#include <rtems/dosfs.h>
Init(){
...
rv = msdos_format(RAMDISK_PATH);
assert(rv == 0);
...
}
#define CONFIGURE_FILESYSTEM_DOSFS
Default block size in case of DOSFS is 512B and I couldn’t find a way to change it: https://lists.rtems.org/pipermail/users/2018-July/032475.html
Possible pitfalls while benchmarking on dosfs:
- Please make sure while benchmarking, files to be created should not lie in the mount root directory i.e ‘/mnt’ instead use a directory like ‘/mnt/1’. This is because there’s an upper limit on the number of files that can be created in the root directory of the FAT file system.
- Make sure cache size when using DOSFS is more than 8Kb
General observation: DOSFS being very old, is more optimized then RFS
General settings
Working with multiple files
When benchmarking on many files concurrently, don’t forget to set the following parameter:
#define CONFIGURE_LIBIO_MAXIMUM_FILE_DESCRIPTORS 320
where 320 is the maximum number of files that can be opened concurrently.
Verification
Many times, you may want to verify your settings either of the filesystem or of the block size. In that case, It’s very handy to have the following shell commands:
#define CONFIGURE_SHELL_COMMAND_BLKSTATS #define CONFIGURE_SHELL_COMMAND_CPUINFO #define CONFIGURE_SHELL_COMMAND_MKRFS #define CONFIGURE_SHELL_MOUNT_RFS #define CONFIGURE_SHELL_MOUNT_MSDOS #define CONFIGURE_SHELL_MOUNT_DOSFS #define CONFIGURE_SHELL_COMMAND_MSDOSFMT #define CONFIGURE_SHELL_COMMAND_MOUNT #define CONFIGURE_SHELL_COMMAND_UNMOUNT // for editor #define CONFIGURE_SHELL_COMMAND_EDIT
Step 3# Selecting the Job configuration file
After setting up the environment and the tool now it’s time to select the job configuration file for FIO.
Job configuration files are used to set IO type, directory, IO size, block size, and numerous other settings. Please have a look at fio documentation once for detailed explanation of job file configuration parameters. Here I will cover only a few of them, which I have actually used.
Selecting the IOengine
Following are the different ioengines that can be used with fio’s RTEMS port:
- sync – Uses basic read() and write() system calls for IO and lseek() is used to position IO. fsync and fdatasync is used to sync the file in case of Buffered IO
- psync – Uses pread() and pwrite() system calls. for IO
- vsync – Uses vectored read and write operations(readv and writev)
- ftruncate – Uses ftruncate() to set file size and then use write() for IO
- filecreate – Just create empty files – used to evaluate the latency while creating a file
IOengine determines how the IO will take place. like in case of `sync` read() and write() calls are used to do IO.
Selecting IO direction
IO direction refers to whether you want to read or write from a file, that too in a random or sequential way. Following are different IO types available to choose from:
- read – only read a file in a sequential way
- write – only write a file in a sequential way
- rw – do both sequential read write operations
- randread – Read a file in a random manner
- randwrite – Write a file in a random manner
- randrw – random read and write file
The percentage of read and write operations in mixed io types is by default 50-50%. However, it can be changed.
Other parameters
- size :- It is the size of the total IO operation. If there is only one file, then size is equal to filesize.
- nrfiles :- Number of files over which IO is uniformly distributed
- thread :- Number of threads to use. Use 1 if not sure.
- direct :- Use to bypass cache while IO. Note that this option won’t work on RTEMS.
- directory :- directory to use while creating files
- ss_ramp :- used to warm-up the benchmarking tool, before actually start io. It should be used if benchmarking on smaller sized files.
- bs :- IO block size
Here is the sample configuration file:
[global] ioengine=sync size=80M rw=write directory=/mnt thread=1 ss_ramp=5 [imfs-write-cfg1-4k] bs=4k
The configuration file can be entered in the RTEMS shell directly by using the file editor. and then can be supplied to fio as a parameter like $ fio config_file_name
Step 4# Interpreting the results
Following is sample output in case of RTEMS:
fat-randrw-sync-16k: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=sync, iodepth=1
fio-3.6-149-g82eb4-dirty
Starting 1 thread
fat-randrw-sync-16k: Laying out IO file (1 file / 100MiB)
Jobs: 1 (f=1): [m(1)][90.9%][r=5204KiB/s,w=5220KiB/s][r=325,w=326 IOPS][eta 00m:01s]
fat-randrw-sync-16k: (groupid=0, jobs=1): err= 0: pid=1: Fri Jan 1 00:00:30 1988
read: IOPS=314, BW=5025KiB/s (5146kB/s)(48.9MiB/9966msec)
clat (usec): min=128, max=4330, avg=1521.40, stdev=1010.44
lat (usec): min=129, max=4331, avg=1522.66, stdev=1010.44
clat percentiles (usec):
| 1.00th=[ 139], 5.00th=[ 200], 10.00th=[ 314], 20.00th=[ 562],
| 30.00th=[ 783], 40.00th=[ 1057], 50.00th=[ 1336], 60.00th=[ 1647],
| 70.00th=[ 2008], 80.00th=[ 2474], 90.00th=[ 3032], 95.00th=[ 3425],
| 99.00th=[ 3916], 99.50th=[ 4113], 99.90th=[ 4293], 99.95th=[ 4293],
| 99.99th=[ 4359]
bw ( KiB/s): min= 4332, max= 5587, per=98.07%, avg=4928.16, stdev=363.67, samples=19
iops : min= 270, max= 349, avg=307.58, stdev=22.89, samples=19
write: IOPS=328, BW=5250KiB/s (5376kB/s)(51.1MiB/9966msec)
clat (usec): min=88, max=10844, avg=1545.92, stdev=1109.49
lat (usec): min=90, max=10846, avg=1548.11, stdev=1109.48
clat percentiles (usec):
| 1.00th=[ 109], 5.00th=[ 182], 10.00th=[ 293], 20.00th=[ 537],
| 30.00th=[ 783], 40.00th=[ 1045], 50.00th=[ 1352], 60.00th=[ 1696],
| 70.00th=[ 2040], 80.00th=[ 2507], 90.00th=[ 3097], 95.00th=[ 3458],
| 99.00th=[ 4113], 99.50th=[ 4228], 99.90th=[10028], 99.95th=[10290],
| 99.99th=[10814]
bw ( KiB/s): min= 4752, max= 5630, per=97.82%, avg=5134.53, stdev=245.69, samples=19
iops : min= 297, max= 351, avg=320.53, stdev=15.18, samples=19
lat (usec) : 100=0.20%, 250=7.53%, 500=10.31%, 750=10.48%, 1000=9.70%
lat (msec) : 2=31.00%, 4=29.53%, 10=1.17%, 20=0.06%
cpu : usr=100.00%, sys=100.00%, ctx=18446744073709551615, majf=18446744073709551615, minf=18446744073709551615
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=3130,3270,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
READ: bw=5025KiB/s (5146kB/s), 5025KiB/s-5025KiB/s (5146kB/s-5146kB/s), io=48.9MiB (51.3MB), run=9966-9966msec
WRITE: bw=5250KiB/s (5376kB/s), 5250KiB/s-5250KiB/s (5376kB/s-5376kB/s), io=51.1MiB (53.6MB), run=9966-9966msec
There are numerous parameters like Bandwidth, latency distribution, IOPS, clat, issued rwts through which one can get an overview view of how the device performs under different conditions. Complete description of these parameters can be found on the fio documentation. Please note that ctx,majf and minf fields are misconfigured for RTEMS.
In case of any Query or for reporting any bugs in fio’s RTEMS port:
Please send an e-mail to me(<dev.madaari@gmail.com>) along with <users@rtems.org> and <fio@vger.kernel.org> on CC. Also, attach complete fio output, job configuration file, and a small description of your RTEMS environment settings(If possible, attach complete rtems-init.c file).
ToDo:- I’ve got some really cool benchmarking statistics for RTEMS filesystems. But, as of now, I couldn’t publicize them as we have an EwiLi paper in progress. So, once publicized, I can further extend this post with the results and their interpretations. However, RAW data can always be found in my GitHub notes here:https://gist.github.com/madaari
Thanks!