Getting Started with the Intel Xeon Phi on Ubuntu 14.04/Linux Kernel 3.13.0

Peter Hardware, OpenCL 25 Comments

You may already know that the Intel MPSS (Manycore Platform Software Stack) officially only supports the RedHat and SUSE Linux distros. Using an enterprise distro might be very interesting if your company is running a large server environment or short on specialized people and you rely on the distro official support for more complicated tasks.

Not all companies use enterprise Linux distributions. Ubuntu has a large share of the Linux distro market (if not the largest). A while back, I needed to setup a machine running Ubuntu 14.04 and MPSS 3.4.x and could not find any documentation running the newest versions of Ubuntu/Linux Kernel/MPSS. In this blog, I will try to document how to get the Intel Xeon Phi running on Kernel 3.13.0 (which is not officially supported by the MPSS to the date of this post). The kernel module was only tested on Ubuntu 14.04, but it should compile and run without any problems in any system running Kernel 3.13.0 (not tested with newer versions).

Building and Installing Kernel Modules

To start, you will need to download the MPSS 3.4.x kernel module sources, it can be download from Intel's website but only as part of the complete set of sources for MPSS platform. As we do not need all sources and want to make it easier, you can just clone the MPSS 3.4.x module sources from my GitHub repository, it is ready to be built for Linux Kernel 3.13.0.

After downloading the module source, one can build it by simply running make, but do not forget to first install build-essential (includes tools such as GCC and Make, required to compile the module) and also the kernel headers that is contained within package kernel-headers-generic on Ubuntu, these packages can be installed with the command sudo apt-get install build-essential linux-headers-generic. To install (as root), just execute make install. The correct kernel module for MPSS is now installed on your system, but it might be necessary to update the list of modules available manually by running sudo depmod.

There is module called mic_host that comes with the Linux Kernel tree, but we will not use it, we will use the one provided by Intel instead. Before loading the now installed module mic, let's ensure that the mic_host module is not loaded by trying to unload it with the command sudo rmmod mic_host. To avoid its loading during boot, make sure to blacklist it by creating the file /etc/modprobe.d/blacklist-mic-host.conf and inserting blacklist mic_host in the first line. Now the MIC module can be safely loaded by running modprobe mic, or you can simply reboot the system that it will be automatically loaded by the kernel.

Be sure also to verify if your Phi will require an update of its flash and SMC bootloader. Because this is a sensitive task, refer to original Intel readme.txt, also available on Intel's website.

Converting and Installing RPM Packages

With the kernel module already installed and loaded, it is time to install the remaining MPSS tools. As with the MPSS kernel module, there is no native support for Ubuntu which uses the DEB packaging system, but only for RedHat and SUSE (both use RPM packaging system). The simplest way to install MPSS tools is probably to convert the RPM packages to DEB packages.

Debian-based distributions (which is the case of Ubuntu) have a tool called alien to convert RPM packages into DEB ones. This tool does not only recompresses the files within the RPM package into a DEB package, but has also the ability to convert installation scripts, that are processed during installation to fix, for example, permissions of newly installed files. The alien tool can be installed by running sudo apt-get install alien.

In the case of Linux MPSS 3.4.2 binaries, it is necessary to download a tar package containing all RPM packages from Intel's website and uncompress it by executing the following command:

After uncompressing the file, there will be a new directory called mpss-3.4.2, with about 50 RPM packages. Inside the mpss-3.4.2 directory, you need to start the conversion of packages, executing the command alien --scripts *.rpm. This process may take a several minutes to complete.

After the conversion is completed, we simply install all DEB packages but this time not with apt-get, since it only installs packages that are available in the repositories defined for Ubuntu. To install the DEB packages we use the command dpkg -i *.deb - this may take a few minutes.

The MPSS tools install some libraries in the system. Because Ubuntu is switching to a different method of organizing libraries within the filesystem, it will not find MPSS libraries directly, as a workaround to this behavior we will create a new file /etc/ld.so.conf.d/zz_x86_64-compat.conf and write /usr/lib64 in there so that the system will recognize the directory /usr/lib64 as a standard search directory for libraries.

All applications must be able to look for the libraries it is linked to in the filesystem. To make search for libraries faster on Linux, it contains a library cache that will be searched before running the application. After a new library is installed in the system, this cache must be updated in order for applications to find the libraries it depends on. This cache is updated by running ldconfig.

Configuring the Coprocessor

The coprocessor configuration described in this section is almost the same as the one in the MPSS Users Guide from Intel's website with a few modifications, such as the nfs service which is called by nfs-kernel-server in Ubuntu and the change from /srv to /var (changing /srv to /var is not entirely necessary but I used it to comply with long-term server files storage location in most modern distros).

First of all, note that everything in this section will be run on the host-side (not on the Phi) and as root user or using sudo. It is up to you if you prefer running as root or using sudo, but you need root permissions for all the following steps. Since the communication is done via root user, first an SSH key is created for root user with ssh-keygen command, and you can leave it without a password if desired.

The MIC should have a configuration tree, this configuration can be initialized to defaults with the command micctrl --initdefaults.

Even though Intel does not officially support Ubuntu, for some reason they have a script to initialize the MPSS service for Ubuntu, and this file is originally installed in /etc/mpss/mpss.ubuntu, but we will copy it to /etc/init.d/mpss so that it will be together with the other service initialization scripts in Ubuntu. It is also necessary to enable the service start during boot and stop during reboot/shutdown, which can be done by executing update-rc.d mpss defaults 99 10.

Instead of rebooting the system many times during the configuration process, we will simply start/stop services whenever necessary. At this point it is necessary to start the MPSS service with command service mpss start.

We will now start the NFS configuration, which is the easiest way to share files between the host system and the coprocessor. On Ubuntu, the NFS service can be installed by running apt-get install nfs-kernel-server.

Next, the directory /micNfs must be created by running mkdir /micNfs. It should be then configured for NFS export by adding the line /micNfs 172.31.1.1(rw,no_root_squash) to the file /etc/exports. Later the coprocessor machine should be allowed to mount this filesystem, and this can be done by adding the line ALL:172.31.1.1 to file /etc/hosts.allow. Note that we are assuming that the configuration is being done for the first Intel Xeon Phi on the system (IP: 172.31.1.1), if you have more than one Phi installed, you may have to change this IP address.

Now the filesystem can be exported by running exportfs -a. After that, the NFS filesystem should be added to MPSS configuration, which is simply done with the micctrl tool, more specifically micctrl --addnfs=172.31.1.254:/micNfs --dir=/micNfs.

Now the MPSS service must be stopped to finish the remaining configuration: service mpss stop. A user for MIC called micuser with user ID 400 is then created along with a group of same name by running useradd -U -m -u 400 micuser. The group ID of micuser is then changed to 400 as well with groupmod -g 400 micuser.

A directory to hold files for micuser must be created (mkdir -p /var/michome/micuser) and given the correct permissions (chown micuser /var/michome/micuser).

The parent directory for micuser NFS home should also be exported by adding the line /var/michome 172.31.0.0/16(rw) to the file /etc/exports. The directory shall now be exported (exportfs -a) and the NFS service restarted service nfs-kernel-server restart.

The MPSS should be configure to mount the /home directory on the Phi's OS from host's /var/michome by running micctrl --addnfs=/var/michome --dir=/home. Finally, the MPSS service is restarted (service mpss restart) and NFS filesystems mounted (mount -a).

OpenCL

To run OpenCL applications on the Phi, there are two additional toolsets required. The first set contains the OpenCL Drivers and Runtime and can be obtained here, this is necessary to actually execute OpenCL applications. To compile OpenCL applications for Intel, you will require also the Intel Parallel Studio XE. It is possible to compile OpenCL applications simply with G++, but OpenCL headers are only available as part of the Intel Parallel Studio XE. If you have OpenCL header files from another vendor they might work as well, but if you want to profile your code, it is strongly advised to get the Intel Parallel Studio XE (note that it requires the purchase of a license, with a 30-day trial also available).

Before install the OpenCL Drivers and Runtime on Ubuntu, you will need the lsb-core package, obtainable via the package manager (apt-get install lsb-core).

Since the installation process for both toolsets is straightforward and no specific steps are required on Ubuntu, a walkthrough the installation will not be covered here, instead refer to Intel's official documentation.

The only thing you will have to change or add after the installation of both toolset is the ICD (Installable Client Driver) file for OpenCL. Simply create a new file /etc/OpenCL/vendors/intel64.icd or modify the existing one to contain only one line /opt/intel/opencl-1.2-4.5.0.8/lib64/libintelocl.so (note that you may have to change opencl-1.2-4.5.0.8 in case you obtained a different version of OpenCL Drivers and Runtime).

Facebooktwittergoogle_plusredditlinkedinmail