Disk I/O is another potential bottleneck that can kill the performance of a parallel program. Disk access is about 1,000,000 times slower than memory access in the worst case (completely random), and about 10 or 20 times slower in the best case (completely sequential).

I/O is generally avoided even in serial programs, but the problem can be far worse for parallel programs. Imagine 300 processes in a parallel job competing for access to the same files.

Using local disks on the compute nodes might improve performance, especially if the processes are spread out across as many nodes as possible. However, this will require distributing the data beforehand. Another issue is that modern nodes have multiple cores, and there may be other processes competing for disk even though you chose to spread out your own job. You could request exclusive access to nodes if I/O is really an issue, but leaving 7 cores idle on 300 8-core machines is not good utilization of resources.

Modern clusters usually have a high-speed shared file system utilizing Redundant Arrays of Parallel Disks (RAIDs). In theory, a group of 20 disks grouped together in a RAID can be read 20 times faster than a single disk of the same type. Performance gains for write access are not as simple, but can also be significant.

Depending on how much I/O your jobs must do, you may be better off using a high-speed shared RAID, or you may be better off distributing data to local disks before computations begin. Only experimentation with each job on each cluster will reveal the optimal strategy.