How to run commands at the same time with moreutils parallel command-line utility?
Most of the CPUs nowadays are multi-cores. parallel runs the specified command, passing it a single one of the specified arguments. This is repeated for each argument. Jobs may be run in parallel. (The default is to run one job per CPU.)
Install moreutils parallel command
There are two parallel utilities available in Debian Linux. If you simply try to run parallel command, you get this error:
To install moreutils parallel which is used in this tutorial use:
apt install moreutils
More more information check this discussion: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=597050
Syntax for moreutils parallel command:
parallel [options] [command]-- [argument ...]
parallel [options]-- [command ...]
Options available for moreutils parallel command:
| Options |
Description |
-j maxjobs |
Use to limit the number of jobs that are run at the same time. |
-l maxload |
Wait as needed to avoid starting new jobs when the system's load average is not below the specified limit. |
-i |
Normally the command is passed the argument at the end of its command line. With this option, any instances of "{}" in the command are replaced with the argument. |
-n |
Number of arguments to pass to a command at a time. Default is 1. Incompatible with -i. |
Run multiple command in parallel with moreutils parallel command in Linux
To run multiple independent command in parallel or using multiple CPU cores, we have the following syntax:
parallel -j [number of jobs at a time] -- "command 1" "command 2" "command 3"
Let's try running three commands using 1, 2, 3 jobs at a time using parallel command and take a look at the results of the time command for the time taken to finish the processes.
Time taken when
- Single job :
4.20s
- Two jobs :
1.32s
- Three jobs :
1.20s
You can see that time taken way shorter when using multiple processors.
Misc: To run a processes at the same time until all files have been processed.
parallel -j 3 cat -- *
Run subshells on each processor with moreutils parallel command in Linux
To run multiple subshells on a (specified) processor, we have the following syntax:
parallel sh -c "[command]" -- [processor number]
Let's try running a command by specifying the processor to use (on a system with 3 processor):
You can see that if you try to use more processors than available, it will run after a process is finished.
Tip: You can also find out the number of processors available in your system by this tricky method.
Conclusion
In this tutorial, we explored parallel command to run multiple processs on multiple CPU cores. This saves so much time while dealing with heavy time consuming processes.
