Last weekend I had problems with heavily active server. I check and see that iostat is showing high disk activity service time was about 100. Also that caused cpu load. I try to find which process is causing disk activity. For this I turn on kernel messages about I/O. Normally for old kernels you could not directly see which process is causing heavy disk activity.
echo 1 > /proc/sys/vm/block_dump
dmesg | egrep "READ|WRITE|dirtied" | egrep -o '([a-zA-Z]*)' | sort | uniq -c | sort -rn | head
I found kjournald causing disk activity. And I know many files are created in short term for that server. So I decide to tune filesystem. Filesystem is ext3. I decide to move writeback journaling. The writeback mode does not preserve data ordering when writing to the disk, so commits to the journal may happen before the file system is written to. This method is faster because only the meta data is journaled, but is not quite as neurotic about protecting your data as the default.
I edit fstab and add writeback option
LABEL=/ / ext3 defaults,data=writeback 1 1
Before reboot run this command.If not your server won't boot.
tune2fs -o journal_data_writeback /dev/sda1