Acādix Consulting, LLC
Bioinformatics Consulting
Unix Software Development

Acadix Software Auto-admin Desktop-installer SPCM HPC Cluster Management Another Programmer's Editor Diskimage Tools Personal Github Site Auer Lab Github Site

SPCM - Simple, Portable Cluster Manager The SPCM cluster manager is a free, open source integrated tool set for managing a simple HPC (High Performance Computing) cluster. It is the only portable cluster management suite we are aware of and is designed to be easily adapted to most POSIX platforms. SPCM automates the process of configuring a head node, compute nodes, file servers, and visualization nodes. Most common management tasks can be performed using a simple menu interface, while additional tasks are supported by command-line tools. SPCM automatically installs and integrates the SLURM scheduler and the Ganglia web-based network monitoring suite with the Apache web server. Status SPCM is currently alpha-quality with a moratorium on major new features until the existing code reaches a high level of cleanliness and robustness. Only a few new features, such as integrating parallel file systems, are planned for the more distant future. SPCM was developed over many years as a tool to manage production HPC clusters at the University of Wisconsin -- Milwaukee. While the code still needs some cleanup and redesign, most of it is fairly evolved and stable. Efforts for the foreseeable future will focus on refactoring and improving the user interface. Screen Shots SPCM main menu: Landing page of a small cluster built with SPCM: Ganglia on a small cluster built with SPCM: Design Philosophy The design philosophy centers on simplicity and performance. These ideals are achieved in-part by minimizing interdependence of cluster nodes. Each compute node contains a fully independent operating system installation and critical software installations on its own local storage. Compared with clusters that utilize shared storage more extensively, this strategy increases the initial cluster setup time slightly in exchange for simpler management, less "noise" on the local network, fewer single points of failure, and fewer bottlenecks. Core design principals: Simplicity: Efforts are focused on objective measures such as reliability, speed, easy management. No peacock feathers that would drain man-hours from improving functionality and add needless complexity and more bugs. We intend to keep the "Simple" in SPCM and will not let it fall victim to creeping feature syndrome. Portability: SPCM should be easy to port to any POSIX operating system. This is a challenge given the differences in sysadmin tools across various POSIX systems, but the basic design minimizes barriers to supporting different tools. The long-term plan is to support heterogeneous clusters, where different nodes can opaquely run different operating systems, while being managed with the same tools. There is currently limited support for this and using FreeBSD file servers and visualization nodes in predominantly CentOS clusters has been tested. The pkgsrc portable package manager plays a major role in making this possible, by managing both system and scientific software on any POSIX platform. Many system-dependent differences are encapsulated in the auto-admin tools, on which SPCM depends, so the SPCM scripts can remain cleaner and more generic. Much work remains to be done in this area, but it will remain a core principal and steady improvement should be expected. Non-interference with core operating system: Unlike many cluster management systems, SPCM does not depend on hacks to the base operating system, but sits on top of the standard base system and package manager for each OS. Critical security updates just released for your OS? Install them immediately without worrying about breaking your cluster. You don't have to wait for us to update a custom OS image. SPCM clusters never never need to be shut down. All system updates can be applied to a live cluster. Compute nodes are set to draining state and updated when they become idle. This ensures that all new jobs will land on updated nodes. The head node, file servers, and visualization nodes can all be updated and rebooted without adversely affecting running jobs. Implementation of this design is facilitated by leveraging the extensive systems management tools provided by the FreeBSD base system, including the ports system which automates the installation of nearly every mainstream open source application in existence. The pkgsrc package manager is used on other platforms. Yum is used for the most basic system services and commercial software support on RHEL/CentOS. The SPCM tools are written almost entirely in POSIX Bourne shell using standard Unix tools to configure the system, and utilizing ports/packages for all software management. In many clusters, the head node is multi-homed (has two network interfaces) and serves as the gateway for the cluster. SPCM supports this configuration, but be aware that it complicates the setup of the head node as well as configuration of many services running on the head node, including the scheduler and the Ganglia resource monitor. The recommended hardware configuration uses a single network interface on every node, including the head node, and a separate router/gateway. Many network switches have built-in routing capability that can serve this purpose. If you're using a simple switch without routing capability for your cluster, you can use an inexpensive hardware router or quickly and cheaply build a sophisticated firewall router using any PC with two network adapters and pfSense or OPNsense. You'll probably want to disable the DHCP server on the router. Running the DHCP server on the head node instead will facilitate PXE installations on the other nodes using tools integrated into SPCM. It's best to keep the load on the head node as low as possible to ensure good response times for shell sessions and for the scheduler. Hence, the head node should not double as a compute node or as a file server for large amounts of data. We generally house /home on the head node, but with a very small quota (e.g. 250 GiB) and store scientific data on a separate file server. A compute node can also be a file server and this has the advantage that jobs running on that node have direct access to the disks rather than using the network via NFS, Gluster, etc. If you do this, be aware that ZFS by default will consume most or all available RAM, thus competing with computational processes. To prevent performance problems, you can limit ZFS adaptive read cache (ARC) to a few gigabytes and subtract the same amount from RealMemory for that node in your slurm.conf. Currently Supported Platforms: FreeBSD and RHEL/CentOS Redhat Enterprise Linux and it's free twin, CentOS, are the de facto standard operating systems for HPC clusters. They are more stable than bleeding-edge distributions, have strong support for HPC system software like Infiniband drivers, parallel file systems, etc., and are the only GNU/Linux platforms officially supported by most commercial software products. The main disadvantages of enterprise Linux platforms (compared to FreeBSD or community Linux distributions such as Debian and Gentoo) are use of outdated kernels and packages available in the Yum repository. (Stability and long-term binary compatibility in enterprise Linux systems is maintained by running older, time-tested, and heavily patched versions of system software.) We've had great success using pkgsrc to manage more up-to-date open source software on RHEL/CentOS. The pkgsrc system is well-supported on Linux, offers far more packages than Yum, and can install a complete set of packages that are almost completely independent from the base Linux installation. FreeBSD's unparalleled reliability, near-optimal efficiency, and easy software management via the FreeBSD ports collection make it an ideal platform for HPC clusters. There is no better platform for running scientific software that requires modern development tools and lengthy uninterrupted up time. FreeBSD is the only operating system we've found that offers enterprise stability and system management features (binary updates, fully-integrated ZFS, etc) combined with top-tier development tools and software management (Clang/LLVM base compiler, FreeBSD ports, etc.). FreeBSD is the basis of many products used in HPC including FreeNAS, Isilon, NetApp, OPNSense, Panasas, and pfSense. Many FreeBSD HPC clusters are in use today, serving science, engineering, and other disciplines. FreeBSD is a supported platform on Amazon's EC2 virtual machine service. It is also a little-known fact that the special effects for the movie "Matrix" were rendered on a FreeBSD cluster. FreeBSD can run most Linux binaries natively (with better performance than Linux in some cases), using its CentOS-based Linux compatibility module. This module is *NOT* an emulation layer. It simple adds Linux system calls to the FreeBSD kernel so that it can run Linux binaries directly. Hence, there is no performance penalty. The only added cost is a small kernel module and modest amount of disk used to house the module and Linux software. Getting started: Do a basic RHEL/CentOS minimal or FreeBSD installation on your head node. Configure the network as desired. Either place all nodes behind a separate gateway (recommended) or configure the head node as a gateway (somewhat more involved, see above). A LAN address of 192.168.x.2 is recommended for the head node. Be sure to choose a LAN IP range that does not overlap with the WAN. Download and run the cluster-bootstrap script to begin head node setup. Once the head node is configured, run cluster-admin and follow the menu options for installing and configuring other nodes. Github