Fixing Linux I/O write performance problems

In a previous tip, I have described how to analyze write performance problems on a Linux server, using top and iotop. Here, I'll explain what you can do to optimize your server for writing.

If you have used top and iotop to determine that you have a write performance problem, there are a few things that you can do. First of all, there is the design of your server. Too many servers are installed with one huge partition with the operating system and data intensive processes together. This is not ideal because there will be only one file system active, one process that flushes data from RAM to disk and only one journaling process. So if your server suffers from bad write performance, backup all your data and get the installation disk. While installing, make sure that the data that is needed by the write-intensive process is on a different partition than the rest of your files.

By default, most Linux servers are tuned for average use – for a use-case in which there are as many reads as there are writes. But, your server may need different settings. ,

While installing your server with custom partitioning, the file system you are using also plays a role. There are two file systems that offer considerable better performance with regard to writes, these are XFS and Ext4. Fortunately, Ext4 is the default file system on all recent Linux distributions, just make sure that your partition is formatted with this file system.

But, the act of installing your server with different partitions may not be enough. By default, most Linux servers are tuned for average use – for a use-case in which there are as many reads as there are writes. But, your server may need different settings.

There are two parameters that relate to the current performance behavior. First, there is the I/O scheduler, a setting that exists for each device you have in your server. Next, there is the journaling mode your file system is using. Let's have a look at the I/O-scheduler first.

I/O scheduler settings
Every device has its own I/O-scheduler. You can find the current settings in the /sys/block/<device>/queue directory, in the file scheduler. To show the current contents, use the cat scheduler command. This gives you a result that looks as in the following listing:

root@texas:/sys/block/sda/queue# cat scheduler

noop anticipatory deadline [cfq]

As you can see, there are four different settings available, and currently the cfq setting is selected. That means that the I/O scheduler allocates equal time slots for reads and for writes, which means bad performance if your server is writing most of the time. So let's start by changing this setting to deadline. Using deadline, means that the scheduler is delaying writes as long as it can, with the purpose of writing as efficiently as possible. To change the current setting, you can use the following command:

echo deadline > /sys/block/sda/queue/scheduler

Journaling settings
You are likely to see some progress after applying this change. But there is more. All modern file systems are using journaling. Journaling ensures the integrity of a file system. There are three different modes in journaling, in which data=journal is the most secure, but definitely the slowest method, and data=writeback is the least secure, but absolutely the fastest mode. The default value is normally set to data=ordered, which is the ideal compromise between speed and data integrity.

For performance reasons it is a good idea, especially if iotop gives you a high amount of data throughput for the kjournald2 process (the process that takes care of journaling). To apply this option, you must use it while mounting the file system, so you have to include it in your fstab. The following line shows how a file system can be mounted with this option applied to it:

/dev/sda3 / ext4 acl,user_xattr,data=writeback  1 1 

Many servers that suffer from bad performance, actually suffer from bad write performance, but fixing the problem is not difficult. Here we've covered how to optimize your server for better writes, applying some simple design guidelines and optimization parameters.

ABOUT THE AUTHOR: Sander van Vugt is an author and independent technical trainer, specializing in Linux since 1994. Vugt is also a technical consultant for high-availability (HA) clustering and performance optimization, as well as an expert on SLED 10 administration.