
Setting up Intel MPI on an HPC cluster can be a bit of a process, but it's definitely doable. First, you'll need to load the Intel MPI module, which can be done by running the command "module load intel-mpi".
The Intel MPI module includes the Intel MPI library, which is a high-performance implementation of the Message Passing Interface (MPI) standard. This library is designed to work with the Intel compilers and can be used to develop parallel applications.
To execute Intel MPI applications on an HPC cluster, you'll need to use the mpirun command. This command is used to launch MPI applications and can be configured to use various features such as process binding and thread pinning. For example, you can use the "-bind-to-core" option to bind each MPI process to a specific core on the node.
You might enjoy: Intel Killer Wifi 6e
Installation and Setup
To install Intel MPI, you'll need to use an environment module, which makes available all the tools needed to compile and execute MPI programs.
Intel MPI is available on all HPC systems at LRZ that support parallel processing in their batch queuing setup. This means you can rely on it for your parallel processing needs.
To configure your environment for the default version of Intel MPI, use module spider intel-oneapi-mpi to check what module(s) to load first. This will give you the necessary information to proceed with the installation.
You'll then need to load the required modules, followed by the intel-oneapi-mpi module. Note that this module conflicts with the default loaded MVAPICH installations, but Lmod will automatically replace it with the correct one when you use module load intel-oneapi-mpi.
Versions
To install Intel MPI, you'll want to know about the available versions. Intel MPI may be used as an alternative to, but not in conjunction with, the MVAPICH MPI libraries.
You can view the available modules using the command `module spider intel-oneapi-mpi`. This will give you a list of the versions that are currently available.
Here are the versions currently available at OSC:
Feel free to contact OSC Help if you need other versions for your work.
Setup

To configure your environment for the default version of Intel MPI, use module spider intel-oneapi-mpi to check what module(s) to load first.
You'll need to load the necessary modules, which can be found by using the module spider command. This will give you the information you need to proceed.
To load the required modules, use module load [module name and version] to load what modules you need, then use module load intel-oneapi-mpi/[version] to load intelmpi.
Note that Intel MPI may conflict with the default loaded MVAPICH installations, so Lmod will automatically replace with the correct one when you use module load intel-oneapi-mpi.
Execution and Running
You can use either the SLURM srun command or the mpiexec command to start up your program inside a SLURM script or interactive salloc environment. The mpiexec command is always used for starting up Intel MPI programs, even if you use the mpiexec command.
To start up your program, you can use the mpiexec -n 32 ./myprog.exe command, which will start up 32 MPI tasks, using as many cores of the system as possible. The same is done if you issue the srun -n 32 ./myprog.exe command.
Sometimes MPI tasks need more memory per task than is available per core, in which case you need to reserve more resources in your job and leave cores idling. You can do this by using the srun --cpus-per-task=2 -n 32 ./myprog.exe command.
To execute hybrid-parallel programs that use both MPI and OpenMP for parallelization, the number of cores used by each MPI task is usually equal to the number of OpenMP threads to be used by that task. You can set the number of OpenMP threads using the export OMP_NUM_THREADS=4 command.
To perform pinning of the OpenMP threads, you can use the compiler-specific pinning mechanism, such as the KMP_AFFINITY environment variable for Intel compilers. However, this will usually only work well on systems with Intel processors.
You can run your MPI jobs using the mpirun command, which allows you to specify the number of processes to run in parallel. For example, you can use the -np 4 option to specify that 4 processes will be run in parallel. Adjust this number based on your job's requirements and the number of available processors on your compute nodes.
You can get help on the Intel(R) MPI Benchmarks from the command line using the component name and the -help parameter. For example, you can run the IMB-MPI1 -help command to see the available command-line parameters.
Environment and Settings
You can control how environment variables are transmitted to the started MPI tasks using the mpiexec command, which takes options like -genv to set specific variables. For example, you can set MY_VAR_1 to value1 and MY_VAR_2 to value2.
To reduce the memory footprint of Intel MPI, you can use environment variables like I_MPI_SHM_CELL_FWD_SIZE, I_MPI_SHM_CELL_FWD_NUM, and I_MPI_SHM_CELL_EXT_SIZE to control the size and number of cells used for shared memory. This can significantly reduce memory consumption, as seen in tests on single SuperMUC-NG compute nodes.
Custom memory allocators can also be controlled using environment variables like I_MPI_MALLOC and I_MPI_SHM_HEAP, which can help reduce memory consumption. Disabling these allocators can be done using specific commands and variables, as described in Intel's documentation.
Availability and Restrictions
Intel MPI is designed to be highly customizable, but it's essential to select the right version for your application's needs to achieve maximum performance.
Always verify your environment and dependencies to avoid compatibility issues, as incompatible versions of MPI or incorrect flags during compilation can lead to errors or performance degradation.
You should check your application's requirements to ensure you're using the correct version of Intel MPI.
To avoid compatibility issues, verify your environment and dependencies before proceeding with the installation or compilation process.
Make sure to check the documentation for your specific application to determine the recommended version of Intel MPI.
Access
To access the Intel MPI, you're in luck - it's available to all OSC users. If you have any questions, don't hesitate to reach out to OSC Help.
Intel MPI is a powerful tool that can help you understand the performance of your cluster system. The generated benchmark data can fully characterize performance, including node performance, network latency, and throughput.
Here's a breakdown of what you can expect from the Intel MPI benchmarks:
- Performance of a cluster system, including node performance, network latency, and throughput
- Efficiency of the MPI implementation
Handling Environment Variables
Handling environment variables is a crucial aspect of optimizing your Intel MPI performance. You can control how environment variables are transmitted to the started MPI tasks using the mpiexec command.
To set environment variables, you can use the -genv option followed by the variable name and value. For example, mpiexec -genv MY_VAR_1 value1 -genv MY_VAR_2 value2 -n 12 ./myprog.exe.
There are also a number of I_MPI_* variables that allow you to extensively configure and optimize at compile as well as run time. These variables can be used to control the behavior of Intel MPI in various ways.
For instance, to reduce the MPI footprint, you can use the I_MPI_SHM_CELL_FWD_SIZE, I_MPI_SHM_CELL_FWD_NUM, I_MPI_SHM_CELL_BWD_SIZE, and I_MPI_SHM_CELL_BWD_NUM variables. These variables can be set at the beginning of the SLURM processing job scripts.
Here are some of the most relevant environment variables for reducing shared memory consumption:
- I_MPI_SHM_CELL_FWD_SIZE – size of forward cells
- I_MPI_SHM_CELL_FWD_NUM – number of forward cells per rank
- I_MPI_SHM_CELL_BWD_SIZE – size of backward cells
- I_MPI_SHM_CELL_BWD_NUM – number of backward cells per rank
- I_MPI_SHM_CELL_EXT_SIZE – size of extended cells
- I_MPI_SHM_CELL_EXT_NUM_TOTAL – total number of extended cells per computational node
By adjusting these variables, you can significantly reduce the shared memory consumption of your MPI application.
Support
The Intel MPI Library is a robust and versatile tool that offers a range of support features to help you develop and deploy high-performance applications.
It supports the MPI-3 interface, which is a significant upgrade over previous versions. You can use the new mpi_f08 Fortran interface with the Intel Fortran compiler to take advantage of this support.
With the Intel MPI Library, you can develop applications that can run on multiple cluster interconnects chosen by the user at run time. This allows you to deliver maximum end-user performance without having to change the software or operating environment.
The library is designed to automatically tune for the latest Intel platforms, achieving the best latency, bandwidth, and scalability. This means you can reduce the time to market by linking to one library and deploying on the latest optimized fabrics.
Here are some key benefits of using the Intel MPI Library:
- Develop applications that can run on multiple cluster interconnects chosen by the user at run time.
- Quickly deliver maximum end-user performance without having to change the software or operating environment.
- Achieve the best latency, bandwidth, and scalability through automatic tuning for the latest Intel platforms.
- Reduce the time to market by linking to one library and deploying on the latest optimized fabrics.
The Intel MPI Library is part of the Intel oneAPI HPC Toolkit, making it an essential tool for anyone looking to develop high-performance applications on Intel platforms.
Building
Building with Intel MPI can be a bit complex, but don't worry, I've got you covered. To make it easier, Intel has defined several environment variables that you can use during compilation.
You can use $MPI_CFLAGS for C programs, $MPI_CXXFLAGS for C++ programs, $MPI_FFLAGS for Fortran programs, and $MPI_F90FLAGS for Fortran 90 programs. Additionally, you can use $MPI_LIBS when linking your program to Intel MPI.
For C programs, use $MPI_CFLAGS during compilation. This is a straightforward process, especially if your application is already set up to use mpicc.
The following table lists options that can be used with compiler wrappers in addition to usual switches for optimization:
To build the Intel MPI Benchmarks, follow these steps: Set the CC variable to point to the appropriate compiler wrapper, either mpiicc or mpicc. Then, run one or more Makefile commands, such as make clean to remove legacy binary object files and executable files, or make all to build all executable files available.
To build "x64" executable files, check that the Include, Lib, and Path environment variables are set to the Intel(R) MPI Library installation directory. Then, open the ".vcproj" or ".vcxproj" file for the component you would like to build, and change the Solution Platforms dialog box to "x64".
Modules and Loading
To use Intel MPI, you'll need to load the appropriate module. This can be done using the module load command, making sure to load the version that fits your needs for optimal performance.
The available Intel MPI modules include those for Haswell nodes, which can be loaded using the module load command. You can load the module to use the Intel MPI module.
You can also load the Intel oneAPI toolchain that includes Intel MPI by using the module load command. This will load the oneAPI toolkit along with Intel MPI, allowing you to develop and execute parallel applications with MPI support.
Non-Blocking Calls
Non-Blocking Calls are not as asynchronous as you might think. They don't spawn a separate thread for communication, so the memory transfer will still happen in the main program thread.
Using shared memory, the CPU will need cycles to transfer the data, which typically occurs during a Waitall call. This can be a bottleneck in your program.
If you're using RDMA, the transfer can happen asynchronously, resulting in a slight improvement. However, for more asynchronous behavior, you'll want to use threading.
One thread can perform the Waitall call while other threads perform calculations, allowing for more efficient use of CPU resources.
Available Modules
Available modules can be a crucial part of your workflow.
Intel MPI modules are available for specific nodes, such as Haswell nodes.
Loading a Module
Loading a module is a straightforward process. To load the Intel MPI module, use the module load command.
You can load the module based on your application requirements. Make sure to load the version that fits your needs for optimal performance in your parallel applications.
The module load command is a simple way to access the tools and libraries you need. This command will load the module and make its contents available for use.
oneAPI and Tools
Intel MPI is also available as part of the Intel oneAPI toolkit, which integrates a set of libraries, compilers, and tools designed for high-performance computing (HPC) and AI applications.
To access Intel MPI with oneAPI, you can load the Intel oneAPI toolchain that includes Intel MPI by loading the module. This will load the oneAPI toolkit along with Intel MPI, allowing you to develop and execute parallel applications with MPI support.
Loading OneAPI
You can load the Intel oneAPI toolchain that includes Intel MPI by loading the module.
Intel MPI is also available as part of the Intel oneAPI toolkit, which integrates a set of libraries, compilers, and tools designed for high-performance computing (HPC) and AI applications.
To load the Intel oneAPI toolchain that includes Intel MPI, you can load the module like so: This will load the oneAPI toolkit along with Intel MPI, allowing you to develop and execute parallel applications with MPI support.
Loading the module loads the oneAPI toolkit along with Intel MPI, making it ready for use.
One Library with Multi-Fabric Support
The Intel MPI Library is a multifabric message-passing library that implements the open-source MPICH specification. It's designed to create, maintain, and test advanced, complex applications that perform better on HPC clusters based on Intel processors.
Developing applications that can run on multiple cluster interconnects chosen by the user at run time is a key benefit. This allows for maximum end-user performance without having to change the software or operating environment.
Quickly delivering maximum end-user performance is crucial for many applications. By linking to one library and deploying on the latest optimized fabrics, developers can reduce the time to market.
The Intel MPI Library achieves the best latency, bandwidth, and scalability through automatic tuning for the latest Intel platforms. This is done without requiring manual intervention or changes to the application code.
Here are some key benefits of using the Intel MPI Library with multiple fabric support:
- Develop applications that can run on multiple cluster interconnects chosen by the user at run time.
- Quickly deliver maximum end-user performance without having to change the software or operating environment.
- Achieve the best latency, bandwidth, and scalability through automatic tuning for the latest Intel platforms.
- Reduce the time to market by linking to one library and deploying on the latest optimized fabrics.
Scalability and Performance
The Intel MPI library is designed to deliver maximum application performance, even when you change or upgrade to new interconnects, without requiring major modifications to the software or operating systems.
It has been verified to scale up to 262,000 processes, making it suitable for large-scale applications.
Thread safety allows for tracing hybrid multithreaded MPI applications for optimal performance on multicore and many-core Intel architecture.
With support for multi-endpoint communications, an application can efficiently split data communication among threads, maximizing interconnect utilization.
The mpiexec.hydra process manager improves start scalability, making it easier to start parallel jobs.
Here's a quick rundown of the library's scalability features:
- Scaling verified up to 262,000 processes
- Thread safety for optimal performance on multicore and many-core Intel architecture
- Multi-endpoint communications for efficient data splitting and interconnect utilization
- Improved start scalability with mpiexec.hydra process manager
Scalability
Scalability is a crucial aspect of any high-performance computing solution. It allows you to quickly deliver maximum application performance without requiring major modifications to the software or operating systems.
The library has been verified to scale up to 262,000 processes, which is a testament to its robustness and ability to handle large-scale applications.
Thread safety is also a key feature, allowing you to trace hybrid multithreaded MPI applications for optimal performance on multicore and many-core Intel architecture.
Support for multi-endpoint communications enables an application to efficiently split data communication among threads, maximizing interconnect utilization.
The mpiexec.hydra process manager improves start scalability, making it easier to start parallel jobs.
Here are some key scalability features:
- Scaling verified up to 262,000 processes
- Thread safety for optimal performance on multicore and many-core Intel architecture
- Support for multi-endpoint communications for maximum interconnect utilization
- Improved start scalability through the mpiexec.hydra process manager
Interconnect Independence
Interconnect Independence is a crucial aspect of scalability and performance. By providing an accelerated, universal, multifabric layer for fast interconnects via OFI, Intel MPI Library allows developers to write MPI code independent of the fabric, knowing it will run efficiently on whatever network they choose at runtime.
This means you can focus on writing code without worrying about the underlying network infrastructure. The library dynamically establishes connections only when needed, reducing memory footprint and automatically choosing the fastest transport available.
Here are some key benefits of Interconnect Independence:
- Develop MPI code independent of the fabric, knowing it will run efficiently on whatever network you choose at run time.
- Use a two-phase communication buffer enlargement capability to allocate only the memory space required.
Tune
Tuning your cluster or application parameters can significantly impact performance. Use mpitune to adjust these parameters.
Sometimes the library's default parameters just aren't delivering the best results. Use mpitune to make adjustments and then fine-tune them iteratively until you achieve the highest performance.
Adjusting parameters is an iterative process, and it's essential to be patient and persistent. You may need to try several combinations before finding the optimal settings.
Iterative adjustments can be time-consuming, but the payoff is worth it. With mpitune, you can make informed decisions and optimize your parameters for maximum performance.
Frequently Asked Questions
What is Intel MPI?
The Intel MPI Library is a powerful tool for creating and optimizing high-performance applications on HPC clusters. It's based on the open-source MPICH specification and supports Intel and compatible processors.
Is MPI still used?
Yes, MPI is still widely used today, from laptops to the world's largest computers, making it a staple in scientific parallel computing. Its continued adoption is a testament to its enduring impact on the field.
Featured Images: pexels.com

