ARM, Benchmark, File system, FIO, GSoC, RTEMS, Software

Benchmarking RTEMS Filesystems using FIO

Hi, In this post we will explore FIO’s RTEMS port and how it can be used to benchmark RTEMS filesystems and drivers.

First, let’s have a quick look at all the RTEMS filesystems:

RTEMS FILESYSTEMS

RTEMS supports mainly two types of filesystems : Network and Physical fs. Benchmarking support for networking filesystems isn’t yet available. However, nearly every other physical filesystem can be benchmarked and contrasted. Heap based file systems are those which uses malloc() for file allocation. In other words, they reside completely on the heap memory. These are mainly used to provide basic directory/file management even if there is no dedicated physical storage for files.Further, they facilitate mounting of other file systems too. For benchmarking IMFS/M-IMFS no external device is needed, unlike block based fs which need either a RAM-disk or a flash device to work with. Further detailed information of these filesystems is available here: https://devel.rtems.org/wiki/Developer/FileSystems

Now, let’s move on to the benchmarking section (for which this post is dedicated to):

Step 1# Preparation of the benchmarking tool

As of today(02/08/2018) FIO’s RTEMS port isn’t yet merged into the fio’s official repository(here’s the corresponding thread: https://www.spinics.net/lists/fio/msg07157.html). I’ll update this section, once it gets merged. Till then we can use my repository https://github.com/madaari/fio/tree/paper . Please stick to the ‘paper‘ branch as it will be much more stable then the master one. For benchmarking purposes, it’s always recommended to use only one version of the tool throughout the results.

Preparation of RTEMS toolchain

Toolchain, required for cross-compiling fio for desired architecture (like arm in this example) can be generated by using RTEMS Source Builder(RSB).

Please note that fio’s build for RTEMS has been tested with RSB version 5(commit id :25f4db09c85a52fb1640a29f9bdc2de8c2768988), and it may not work with older versions.

Moreover, For enabling POSIX support(required by fio), build the BSP using RTEMS v5 with –enable-posix option. After that, if needed(like for using SD card driver for BeagleBone Black) one may also need to build rtems-libbsd for the desired BSP.

Cross-Compiling FIO

Fetch the fio repository and then configure it as shown below:

Variable to be passed for toolchain path is TOOL_PATH_PREFIX, which in this case would be,

After setting up the variable, next step would be to configure and build fio as:

By now, you will have the fio binary ready which can be loaded to the target using the bootloader like u-boot.


Note:- While using u-boot to load up the fio binary use the following configuration: 

where $app is the name of image file to be loaded. Just make sure, there is enough space between the loading and executing address. Otherwise, i would recommend to use this script after making above mentioned changes(mainly change the load address from 0x80800000 to 0x81000000). Main reason for this is to avoid the following error during uncompressing:

It was because during gunzip(at loadaddr i.e 0x80000000) uboot overwrite the gzipped kernel image(at 0x80800000) thus an error during uncompressing. It’s resolved via a small change in uenv.txt: boot=fatload mmc 0 0x81000000 rtems-app.img ; fatload mmc 0 0x88000000 am335x-boneblack.dtb ; bootm 0x81000000 - 0x88000000


Step 2# Setting up the Benchmarking Environment

User might want to customize the rtems configuration for simulating different environments (to notice their effect on benchmarking stats) or might want to setup a filesystem on any mounted device like flash, RAM-disk. This sub-section will begin with a small description of rtems-init.c file and will then describe the changes required for benchmarking different file systems and device drivers.

Just like any for any other rtems application, rtems-init.c file is the main configuration file with Init() function, device drivers and libblock/cache settings. Making change in this file will not in any way affect the benchmarking tool aka fio but would alter the benchmarking environment.

Evaluating the effect of cache size

For example, if someone wanted to evaluate the effect of cache size on the stats, he can change the libblock and cache settings from the following lines:

Note:- In case of using DOSFS, don’t set the cache size below 8k that might have an adverse effect on the rtems-shell application(In my case, it didn’t even load up ). Also, Heap based filesystems like IMFS/M-IMFS are virtually unaffected by any change in cache settings. That has a pretty straight forward reason: Heap based fs just don’t use libblock API. Further, information/description of these options can be found in RTEMS documentation.

Setting up RAM-disk

For benchmarking on a RAM-disk, following configuration is required:

Setting the RAM-disk block size other then 512B might not work, in that case rtems_rfs_format won’t successfully exit. Also, Note that for IMFS you don’t need to set up the RAM-disk, since Heap-based fs allocate on RAM itself! Also, please carefully choose the RAM-disk size. for example, In my case BeagleBone Black has 512MB of RAM, while RTEMS BBB BSP uses only 216MB of it. Use it’s wise to uses only 128MB(50% of RAM available) for RAM-disk. For benchmarking, it’s always better to use larger sized files for consistent bandwidth stats. So, very small RAM-disk size is also not viable.

As a general observation, cache has negative effect on RAM-disk bandwidth!!

Setting up SD card

For benchmarking the SD card driver, one can use the media_server(config given below) to mount the card. Please note that, SD card driver act as a bottle-neck and thus evaluating RTEMS filesystems on a SD-card isn’t really a good idea.

Benchmarking IMFS

IMFS being the default filesystem, is pretty easy to benchmark. However, there are some settings of IMFS which might need to be taken care of:

Also, please don’t try to mount RAM-disk using IMFS, it would fail: https://lists.rtems.org/pipermail/users/2018-July/032466.html 

IMFS block size determines the size of the file that can be created. 512B block size allows maximum possible file size. However, for some unknown(atleast to me) reasons benchmarking with a file size greater then 80MB-82MB raises an IO error. Maybe on some other boards or with other RAM size this limit gets changed.

Benchmarking RFS

For benchmarking RFS, first step would be to set up a RAM disk with rtems_rfs_format and RTEMS_FILESYSTEM_TYPE_RFS settings. Next step would be to add RFS configuration like:

Setting RFS block size other then 1024 might raise an error as well !!

Benchmarking DOSFS

For DOSFS, msdos_format() and RTEMS_FILESYSTEM_TYPE_DOSFS(while mounting RAM-disk) can be used. like:

Default block size in case of DOSFS is 512B and i coundn’t find a way to change it: https://lists.rtems.org/pipermail/users/2018-July/032475.html

Possible pitfalls while benchmarking on dosfs:

  • Please make sure while benchmarking, files to be created should not lie in the mount root directory i.e ‘/mnt’ instead use a directory like ‘/mnt/1’. This is because there’s an upper limit on the number of files that can be created in the root directory of the FAT file system.
  • Make sure cache size when using DOSFS is more then 8Kb

General observation: DOSFS being very old, is more optimized then RFS

General settings

Working with multiple files

When benchmarking on many files concurrently, don’t forget to set the following parameter:

where 320 is the maximum number of files that can be opened concurrently.

Verification

Many times, you may want to verify your settings either of filesystem or of block size. In that case, It’s very handy to have the following shell commands:

Step 3# Selecting the Job configuration file

After setting up the environment and the tool now it’s time to select the job configuration file for FIO.

Job configuration files are used to set IO type, directory, IO size, block size and numerous other settings. Please have a look at fio documentation once for detailed explanation of job file configuration parameters. Here i will cover only few of them, which i have actually used.

Selecting the IOengine

Following are the different ioengines that can be used with fio’s RTEMS port:

  • sync – Uses basic read() and write() system calls for IO and lseek() is used to position IO. fsync and fdatasync is used to sync the file in case of Buffered IO
  • psync – Uses pread() and pwrite() system calls. for IO
  • vsync – Uses vectored read and write operations(readv and writev)
  • ftruncate – Uses ftruncate() to set file size and then use write() for IO
  • filecreate – Just create empty files – used to evaluate the latency while creating a file

IOengine determines how the IO will take place. like in case of sync read() and write() calls are used to do IO.

Selecting IO direction

IO direction refers to whether you want to read or write from a file, that too in a random or sequential way. Following are different IO types available to choose from:

  • read – only read a file in a sequential way
  • write  – only write a file in a sequential way
  • rw – do both sequential read write operations
  • randread – Read a file in a random manner
  • randwrite – Write a file in a random manner
  • randrw – random read and write file

Percentage of read and write operations in mixed io types is by default 50-50%. However, it can be changed.

Other parameters

  • size :-  It is the size of the total IO operation. If there is only one file, then size is equal to filesize.
  • nrfiles :- Number of files over which IO is uniformly distributed
  • thread :- Number of threads to use. Use 1 if not sure.
  • direct :- Use to bypass cache while IO. Note that this option won’t work on RTEMS.
  • directory :- directory to use while creating files
  • ss_ramp :- used to warm-up the benchmarking tool, before actually start io. It should be used if benchmarking on smaller sized files.
  • bs :- IO block size

Here is the sample configuration file:

Configuration file can be entered in the RTEMS shell directly by using the file editor. and then can be supplied to fio as a parameter like $ fio config_file_name

Step 4# Interpreting the results

Following is sample output in case of RTEMS:

There are numerous parameters like Bandwidth, latency distribution, IOPS, clat, issued rwts through which one can get a overview view of how the device performs under different conditions. Complete discription of these parameters can be found on the fio documentation. Please note that ctx,majf and minf fields are misconfigured for RTEMS.


In case of any Query or for reporting any bugs in fio’s RTEMS port:

Please send an e-mail to me(<dev.madaari@gmail.com>) along with <users@rtems.org> and <fio@vger.kernel.org> on CC. Also, attach complete fio output , job configuration file and a small description of your RTEMS environment settings(If possible, attach complete rtems-init.c file).

ToDo:- I’ve got some really cool benchmarking statistics for RTEMS filesystems. But, as of now I couldn’t publicize them as we have an EwiLi paper in progress. So, once publicized, i can further extend this post with the results and their interpretations. However, RAW data can always  be found in my github notes here:https://gist.github.com/madaari

Thanks!

Tagged , ,

1 thought on “Benchmarking RTEMS Filesystems using FIO

Leave a Reply

Your email address will not be published.

This site uses Akismet to reduce spam. Learn how your comment data is processed.