You may already know that the Intel MPSS (Manycore Platform Software Stack) officially only supports the RedHat and SUSE Linux distros. Using an enterprise distro might be very interesting if your company is running a large server environment or short on specialized people and you rely on the distro official support for more complicated tasks.
Not all companies use enterprise Linux distributions. Ubuntu has a large share of the Linux distro market (if not the largest). A while back, I needed to setup a machine running Ubuntu 14.04 and MPSS 3.4.x and could not find any documentation running the newest versions of Ubuntu/Linux Kernel/MPSS. In this blog, I will try to document how to get the Intel Xeon Phi running on Kernel 3.13.0 (which is not officially supported by the MPSS to the date of this post). The kernel module was only tested on Ubuntu 14.04, but it should compile and run without any problems in any system running Kernel 3.13.0 (not tested with newer versions).
Building and Installing Kernel Modules
To start, you will need to download the MPSS 3.4.x kernel module sources, it can be download from Intel’s website but only as part of the complete set of sources for MPSS platform. As we do not need all sources and want to make it easier, you can just clone the MPSS 3.4.x module sources from my GitHub repository, it is ready to be built for Linux Kernel 3.13.0.
After downloading the module source, one can build it by simply running make, but do not forget to first install build-essential (includes tools such as GCC and Make, required to compile the module) and also the kernel headers that is contained within package kernel-headers-generic on Ubuntu, these packages can be installed with the command sudo apt-get install build-essential linux-headers-generic. To install (as root), just execute make install. The correct kernel module for MPSS is now installed on your system, but it might be necessary to update the list of modules available manually by running sudo depmod.
There is module called mic_host that comes with the Linux Kernel tree, but we will not use it, we will use the one provided by Intel instead. Before loading the now installed module mic, let’s ensure that the mic_host module is not loaded by trying to unload it with the command sudo rmmod mic_host. To avoid its loading during boot, make sure to blacklist it by creating the file /etc/modprobe.d/blacklist-mic-host.conf and inserting blacklist mic_host in the first line. Now the MIC module can be safely loaded by running modprobe mic, or you can simply reboot the system that it will be automatically loaded by the kernel.
Be sure also to verify if your Phi will require an update of its flash and SMC bootloader. Because this is a sensitive task, refer to original Intel readme.txt, also available on Intel’s website.
Converting and Installing RPM Packages
With the kernel module already installed and loaded, it is time to install the remaining MPSS tools. As with the MPSS kernel module, there is no native support for Ubuntu which uses the DEB packaging system, but only for RedHat and SUSE (both use RPM packaging system). The simplest way to install MPSS tools is probably to convert the RPM packages to DEB packages.
Debian-based distributions (which is the case of Ubuntu) have a tool called alien to convert RPM packages into DEB ones. This tool does not only recompresses the files within the RPM package into a DEB package, but has also the ability to convert installation scripts, that are processed during installation to fix, for example, permissions of newly installed files. The alien tool can be installed by running sudo apt-get install alien.
In the case of Linux MPSS 3.4.2 binaries, it is necessary to download a tar package containing all RPM packages from Intel’s website and uncompress it by executing the following command:
tar xf mpss-3.4.2-linux.tar
After uncompressing the file, there will be a new directory called mpss-3.4.2, with about 50 RPM packages. Inside the mpss-3.4.2 directory, you need to start the conversion of packages, executing the command alien --scripts *.rpm. This process may take a several minutes to complete.
After the conversion is completed, we simply install all DEB packages but this time not with apt-get, since it only installs packages that are available in the repositories defined for Ubuntu. To install the DEB packages we use the command dpkg -i *.deb – this may take a few minutes.
The MPSS tools install some libraries in the system. Because Ubuntu is switching to a different method of organizing libraries within the filesystem, it will not find MPSS libraries directly, as a workaround to this behavior we will create a new file /etc/ld.so.conf.d/zz_x86_64-compat.conf and write /usr/lib64 in there so that the system will recognize the directory /usr/lib64 as a standard search directory for libraries.
All applications must be able to look for the libraries it is linked to in the filesystem. To make search for libraries faster on Linux, it contains a library cache that will be searched before running the application. After a new library is installed in the system, this cache must be updated in order for applications to find the libraries it depends on. This cache is updated by running ldconfig.
Configuring the Coprocessor
The coprocessor configuration described in this section is almost the same as the one in the MPSS Users Guide from Intel’s website with a few modifications, such as the nfs service which is called by nfs-kernel-server in Ubuntu and the change from /srv to /var (changing /srv to /var is not entirely necessary but I used it to comply with long-term server files storage location in most modern distros).
First of all, note that everything in this section will be run on the host-side (not on the Phi) and as root user or using sudo. It is up to you if you prefer running as root or using sudo, but you need root permissions for all the following steps. Since the communication is done via root user, first an SSH key is created for root user with ssh-keygen command, and you can leave it without a password if desired.
The MIC should have a configuration tree, this configuration can be initialized to defaults with the command micctrl --initdefaults.
Even though Intel does not officially support Ubuntu, for some reason they have a script to initialize the MPSS service for Ubuntu, and this file is originally installed in /etc/mpss/mpss.ubuntu, but we will copy it to /etc/init.d/mpss so that it will be together with the other service initialization scripts in Ubuntu. It is also necessary to enable the service start during boot and stop during reboot/shutdown, which can be done by executing update-rc.d mpss defaults 99 10.
Instead of rebooting the system many times during the configuration process, we will simply start/stop services whenever necessary. At this point it is necessary to start the MPSS service with command service mpss start.
We will now start the NFS configuration, which is the easiest way to share files between the host system and the coprocessor. On Ubuntu, the NFS service can be installed by running apt-get install nfs-kernel-server.
Next, the directory /micNfs must be created by running mkdir /micNfs. It should be then configured for NFS export by adding the line /micNfs 172.31.1.1(rw,no_root_squash) to the file /etc/exports. Later the coprocessor machine should be allowed to mount this filesystem, and this can be done by adding the line ALL:172.31.1.1 to file /etc/hosts.allow. Note that we are assuming that the configuration is being done for the first Intel Xeon Phi on the system (IP: 172.31.1.1), if you have more than one Phi installed, you may have to change this IP address.
Now the filesystem can be exported by running exportfs -a. After that, the NFS filesystem should be added to MPSS configuration, which is simply done with the micctrl tool, more specifically micctrl --addnfs=172.31.1.254:/micNfs --dir=/micNfs.
Now the MPSS service must be stopped to finish the remaining configuration: service mpss stop. A user for MIC called micuser with user ID 400 is then created along with a group of same name by running useradd -U -m -u 400 micuser. The group ID of micuser is then changed to 400 as well with groupmod -g 400 micuser.
A directory to hold files for micuser must be created (mkdir -p /var/michome/micuser) and given the correct permissions (chown micuser /var/michome/micuser).
The parent directory for micuser NFS home should also be exported by adding the line /var/michome 172.31.0.0/16(rw) to the file /etc/exports. The directory shall now be exported (exportfs -a) and the NFS service restarted service nfs-kernel-server restart.
The MPSS should be configure to mount the /home directory on the Phi’s OS from host’s /var/michome by running micctrl --addnfs=/var/michome --dir=/home. Finally, the MPSS service is restarted (service mpss restart) and NFS filesystems mounted (mount -a).
OpenCL
To run OpenCL applications on the Phi, there are two additional toolsets required. The first set contains the OpenCL Drivers and Runtime and can be obtained here, this is necessary to actually execute OpenCL applications. To compile OpenCL applications for Intel, you will require also the Intel Parallel Studio XE. It is possible to compile OpenCL applications simply with G++, but OpenCL headers are only available as part of the Intel Parallel Studio XE. If you have OpenCL header files from another vendor they might work as well, but if you want to profile your code, it is strongly advised to get the Intel Parallel Studio XE (note that it requires the purchase of a license, with a 30-day trial also available).
Before install the OpenCL Drivers and Runtime on Ubuntu, you will need the lsb-core package, obtainable via the package manager (apt-get install lsb-core).
Since the installation process for both toolsets is straightforward and no specific steps are required on Ubuntu, a walkthrough the installation will not be covered here, instead refer to Intel’s official documentation.
The only thing you will have to change or add after the installation of both toolset is the ICD (Installable Client Driver) file for OpenCL. Simply create a new file /etc/OpenCL/vendors/intel64.icd or modify the existing one to contain only one line /opt/intel/opencl-1.2-4.5.0.8/lib64/libintelocl.so (note that you may have to change opencl-1.2-4.5.0.8 in case you obtained a different version of OpenCL Drivers and Runtime).
Comments 25
Pingback: Getting Started with the Intel Xeon Phi on Ubuntu 14.04/Linux Kernel 3.13.0 • Atlanta Tech Blogs
Hey ArrayFire guys … this your best blog post ever!!! Thanks a lot.
There is some experimental OpenCL 2.0 for Xeon Phi?
I don’t remember Intel reading anything about OpenCL 2.0 on Xeon Phi so far, not even about Linux support yet, perhaps they will be able to tell more precisely what are the plans for OpenCL 2.0 for Phi/Linux.
Upon using any of the “micctrl” commands I get “Warning: No Mic card detected or specified in command line” is it not detecting my xeon phi?
I think so, but you can also try “micinfo” to check if some Phi is detected, also take a look at “dmesg”, it may give you some hints if your Phi was detected or not.
mmm, why things are never easy…
The instruccions above do not work for me. I did follow the instruccions: I went to the git download the MPSS 3.4.2 module, compile… everything went well until “Converting and Installing RPM Packages”. I go to the intel website and: MPSS-3.4.3 and not 3.4.2!
So I download now the src from Intel (from 3.4.3) but the $make command do not compile:
make -C /lib/modules/3.13.0-46-generic/build M=/home/jofre/Downloads/mpss-3.4.3/src/mpss-modules-3.4.3 modules
INSTALL_MOD_PATH=
make[1]: Entering directory `/usr/src/linux-headers-3.13.0-46-generic’
/home/jofre/Downloads/mpss-3.4.3/src/mpss-modules-3.4.3/Kbuild:10: *** building for host, but $(MIC_CARD_ARCH) is unset. Stop.
make[1]: *** [_module_/home/jofre/Downloads/mpss-3.4.3/src/mpss-modules-3.4.3] Error 2
make[1]: Leaving directory `/usr/src/linux-headers-3.13.0-46-generic’
make: *** [modules] Error 2
Any ideas?
For info, I installed Lubuntu 14.04.2 LTS, and then install the kernel and headers of 3.13 branch, and changed grub to boot from it.
All the best.
Part of the problem is the MIC_CARD_ARCH variable, which must be set in the Makefile. To set this, edit the make file and fin the line
export MIC_CARD_ARCH :=
and set it to
export MIC_CARD_ARCH := k1om
I have no idea what that architecture code means or why it has to be set that way. But that’s what the author did in his patched version on github.
Problem solved. I found the solution here:
https://software.intel.com/en-us/forums/topic/532329#comment-1815945
Jofre
Stupid questions =): how do I connect to the mic? I try ssh root@ip but it is asking me password..
Pingback: Xeon Phi reference | meditation by maverick
Has anyone tried this card for 3D rendering on Blender?
I have tried to follow the instructions but got stuck quite early on at the make install, I guess because of version incompatibility. I am on Ubuntu 14.4 but I get this error
install: cannot stat ‘Module.symvers’: No such file or directory
make: *** [kdev_install] Error 1
I am a newbie on Linux but I am here to learn, the hard way. P.O.
I looked a bit closer and I get stuck already ad the make. I also see that I have kernel version 3.19.0.32 (or 25 or 31 depending on where you read) so I guess there are differences. Typically the errors I get during the make look like this
In file included from include/linux/kobject.h:21:0,
from include/linux/module.h:16,
from /home/po/Downloads/mpss-modules-master/include/micint.h:39,
from /home/po/Downloads/mpss-modules-master/host/linux.c:39:
include/linux/sysfs.h:473:20: note: expected ‘struct kernfs_node *’ but argument is of type ‘struct sysfs_dirent *’
static inline void sysfs_notify_dirent(struct kernfs_node *kn)
^
/home/po/Downloads/mpss-modules-master/host/linux.c: In function ‘mic_read’:
/home/po/Downloads/mpss-modules-master/host/linux.c:150:18: error: ‘struct file’ has no member named ‘f_dentry’
dev_t dev = filp->f_dentry->d_inode->i_rdev;
^
/home/po/Downloads/mpss-modules-master/host/linux.c: In function ‘mic_ioctl’:
/home/po/Downloads/mpss-modules-master/host/linux.c:163:12: error: ‘struct file’ has no member named ‘f_dentry’
dev = filp->f_dentry->d_inode->i_rdev;
^
/home/po/Downloads/mpss-modules-master/host/linux.c: In function ‘mic_fasync’:
/home/po/Downloads/mpss-modules-master/host/linux.c:184:6: error: void value not ignored as it ought to be
rc = __f_setown(filp, task_pid(current), PIDTYPE_PID, 0);
I would be most grateful if someone could tell me what to do next. Can I downgrade to Kernel version 3.13.0? I am running Ubuntu 14.04.3 LTS
P.O.
I updated the kernel module sources to work with Linux kernel 3.19.0
used in Ubuntu 14.04.3. You can find the updated sources on GitHub:
https://github.com/gjvdbraak/mpss-modules
Great! I will give it a try sometime next week. For the record I could find an older version of Ubuntu 14.4 with a 3.13 kernel and could follow the instruction almost (but only almost :-() all the way. The make and make install seems to work but there are error messages when I try to contact the card. I will try again with your updated modules. I am installing this card on a MAC PRO (mid-2010) running Linux so it may or may not be working. I am here to learn though.
Please allow me to ask a very basic Newbie question: Do I need to have the card in the machine while building the framework?, I have installed it only after I had done all the work.
I have patched the latest MPSS module 3.6.1 for Ubuntu 14.10 (kernel 4.2.0). You can find the sources here:
https://github.com/huanzhang12/mpss-modules
I rolled back to kernel 4.2.0, cloned the files and when attempting to make (with make MIC_CARD_ARCH=k1o, as in the readme), I get the following errors:
make -C /lib/modules/4.2.0-040200-generic/build M=/home/kevin/mpss-modules modules
INSTALL_MOD_PATH=
make[1]: Entering directory `/usr/src/linux-headers-4.2.0-040200-generic’
Makefile:659: Cannot use CONFIG_CC_STACKPROTECTOR_STRONG: -fstack-protector-strong not supported by compiler
/home/kevin/mpss-modules/Kbuild:13: *** $(MIC_CARD_ARCH) must be l1om or k1om. Stop.
make[1]: *** [_module_/home/kevin/mpss-modules] Error 2
make[1]: Leaving directory `/usr/src/linux-headers-4.2.0-040200-generic’
make: *** [modules] Error 2
Is simply loading the older kernel (4.2 from 4.4) not enough to properly compile?
Good analysis . Apropos , if anyone is wanting a a form , my friend came across a sample form here
http://goo.gl/mCcUZlHello,
I am trying to install 4 Xeon Phi 7120p on a Ubuntu 16.04 LTS with kernel 4.4.0-62-generic. I found the latest MPSS 3.8 supports
SUSE* Linux* Enterprise Server 12 SP2 64-bit | 4.4.21-69-default
So, does this guide works for Ubuntu 16.04?
Thank you!
I try to install it on my ubuntu 16.04, i didn’t achieve to build the git repo
I finally switched to CentOS 7.1 because I need to install MPSS 3.8 and OFED 3.18.2
Hello, I am trying to make a Co-processor: Intel Corporation Xeon Phi coprocessor SE10/7120 series (rev 20) (from lspci) work with Ubuntu 16.04 LTS with kernel 4.8.0-54-generic.
I have tried with:
https://github.com/pentschev/mpss-modules
and got the error
[…]/include/mic_common.h:701:23: error: passing argument 1 of ‘sysfs_notify_dirent’ from incompatible pointer type [-Werror=incompatible-pointer-types]
sysfs_notify_dirent(mic_ctx->sysfs_state);
[…]
and
https://github.com/huanzhang12/mpss-modules and https://software.intel.com/en-us/articles/intel-manycore-platform-software-stack-mpss#lx38rel
both with the error
[…]host/tools_support.c:67:5: error: implicit declaration of function ‘page_cache_release’ [-Werror=implicit-function-declaration]
page_cache_release(pages[j]);
[…]
Do anyone have any idea to solve this issue and be able to use the Xeon Phi through OpenCL?
Thank you in advance
Get CUDA device and enjoy the same programming-model-complextity, higher efficiency and most of typical problems solved.
Post Scriptum: I am professional, please do not hesitate to trust me.