[Home] [Services] [Philosophy] [Experience] [Software] [Publications] [Tips] [Contact]


PAPP - Portable Assembly Pre-Processor
xml-format - XML Beautifier
Diskimage Tools
Matlab Installer

Why CentOS and FreeBSD?

Redhat Enterprise Linux and it's free twin, CentOS, are the de facto standard operating systems for HPC (High Performance Computing) clusters. They are very stable, have strong support for HPC system software like Infiniband drivers, parallel file systems, etc., and are the only GNU/Linux platforms officially by many commercial software vendors.

FreeBSD's unparalleled stability, near-optimal efficiency, and easy software management via ports and packages make it an ideal platform for high performance computing (HPC) clusters. There is no better platform for running huge computational jobs that may require weeks or months of uninterrupted up time. FreeBSD is the only operating system we've found that offers enterprise stability combined with a top-tier software management system (FreeBSD ports).

Many FreeBSD HPC clusters are in use today, serving science, engineering, and other disciplines. FreeBSD is a supported platform on Amazon's EC2 virtual machine service. It is also a little-known fact that the special effects for the movie "Matrix" were rendered on a FreeBSD cluster.

FreeBSD can run most Linux binaries natively (with better performance than Linux in some cases), using its CentOS-compatible Linux compatibility module. This module is *NOT* an emulation layer. It simple adds Linux system calls to the FreeBSD kernel so that it can run Linux binaries directly. Hence, there is no performance penalty. The only cost is a small amount of memory and disk used to house the module and Linux software.

The main disadvantage of enterprise Linux platforms (compared to FreeBSD or community Linux distributions such as Debian and Gentoo) is the outdated base installations and the limited and outdated collection of packages available in the Yum repository. (Stability and long-term binary compatibility in enterprise Linux systems is maintained by running older, time-tested versions of system software.)

We've had great success using pkgsrc to manage more up-to-date open source software on RHEL/CentOS. The pkgsrc system is well-supported on Linux, offers far more packages than Yum, and can install a complete set of packages that are almost completely independent from the base Linux installation. This allows the base system (including RPMs from Yum) to be updated without breaking software installed by pkgsrc.

Cluster Admin

Cluster-admin is a suite of tools for building and managing RHEL/CentOS and FreeBSD HPC clusters.

It is the only portable cluster management suite we are aware of and can be easily adapted to other platforms.

It automates the process of configuring a head node and compute nodes for a high performance computing cluster, and managing configuration and software after installation.

Screen shot of a small cluster built with cluster-admin:

[Ganglia Screen Shot]

Cluster-admin automates the setup of a cluster using either HTCondor, SLURM or Torque schedulers, the Open MPI parallel programming suite, and the Ganglia web-based network monitoring suite. It also helps synchronize system files on the compute nodes, manage user accounts, and manage software on compute nodes.

The design philosophy centers on simplicity and minimizing interdependence of the cluster nodes. Each compute node contains a fully independent operating system installation and critical software installations on its own local storage. Compared with clusters that utilize shared storage more extensively, this strategy increases the initial cluster setup time slightly in exchange for simpler management, less "noise" on the local network, fewer single points of failure, and fewer bottlenecks.

Core design principals:

  • Speed and Simplicity: Our efforts are focused on basic functionality, robustness, fast and easy setup and management. No fluff to make it look fancy.
  • Portability: We aim to make the system portable to any POSIX operating system. Ultimately, we plan to support heterogeneous clusters, where different nodes can opaquely run different operating systems, while being managed with the exact same commands. The pkgsrc portable package manager plays a major role in making this possible, by managing both system and scientific software on any POSIX platform. Many system-dependent differences are encapsulated in the auto-admin tools, on which cluster-admin depends, so the cluster-admin scripts can remain cleaner and more generic. Much work remains to be done in this area, but it will remain a core principal and steady improvement should be expected.
  • Non-interference with core operating system: Unlike some other cluster management systems, cluster-admin does not depend on hacks to the base operating system, and all OS updates can be applied using the standard tools that come with the OS (through the cluster-admin interface to avoid issues). Critical security update just released for your OS? Install it immediately without worrying about breaking your cluster. You don't have to wait for us to port it to a modified OS image.
  • Never require a cluster shutdown. All system updates can be applied to a live cluster. Compute nodes are all set to draining state at once and updated when they become idle. This ensures that all new jobs will land on updated nodes.

Implementation of this design is facilitated by leveraging the extensive systems management tools provided by the FreeBSD base system, including the ports system which automates the installation of nearly every mainstream open source application in existence. The pkgsrc package manager is used on other platforms.

The cluster-admin tools are written primarily as Bourne shell scripts using standard Unix tools to configure the system, and utilizing ports/packages for all software management.

[cluster diagram] In many clusters, the head node can be multi-homed (have two network interfaces) and serve as the gateway for the cluster. Cluster-admin supports this configuration, but be aware that it complicates the setup of the head node as well as configuration of services running on the head node, including the scheduler and the Ganglia resource monitor.

The recommended hardware configuration uses a single network interface on all nodes, including the head node, and a separate router. Many network switches have built-in routing capability. If you're using a simple switch without routine capability for your cluster, you can use an inexpensive hardware router, or quickly and cheaply build a sophisticated firewall router using another PC and pfSense or OPNsense.

To get started, do a basic FreeBSD installation on your head node including all source code and ports. Then install cluster-admin via ports or packages and run "man cluster-setup".