Updated July 1, 2023
Introduction to Python Multiprocessing
Multiprocessing is somewhat of a computerized version of multitasking. Multitasking is the process of handling several tasks at the same time efficiently. Similarly, Multiprocessing in Python is the ability to handle more than one process simultaneously. In real life, a multitasker is very successful at his work; similarly, a Python program using multiprocessing is far better than one not using it.
Python Multiprocessing uses parallel processing, in which code from a single program can be executed on the different cores of a computer using parallel code.
Multiprocessing in Python
Python has a module named multiprocessing which helps us write parallel code, thus resulting in parallel computing. The following classes in Python multiprocessing help us create a parallel program:
- Process
- Queue
- Pool
- Lock
In parallel programming, a code is run on different cores. We can know the number of cores in our system in the following way:
Code:
import multiprocessing
print("The number of cores in the system is",multiprocessing.cpu_count())Output:
How can we create a parallel program using the different classes?
Below are various points to build parallel programming:
1. Process
Code:
import numpy as np
from multiprocessing import Process
numbers = [2.1,7.5,5.9,4.5,3.5]
def print_func(element=5):
print('Square of the number : ', np.square(element))
if __name__ == "__main__": # confirmation that the code is under main function
procs = []
proc = Process(target=print_func) # instantiating without any argument
procs.append(proc)
proc.start()
for number in numbers:
proc = Process(target=print_func, args=(number,))
procs.append(proc)
proc.start()
for proc in procs:
proc.join()Output:
The following are the two important functions of the Process class:-
- start(): The start() function initiates the process object’s processing as it has been created.
- join(): The join() function instructs the process to complete its execution. Without the join() function, the process would remain idle and not terminate.
It is essential to call the join() function after the termination of the process to free the resources for future processes; otherwise, it has to be done manually. The args keyword is used to send an argument through the process.
2. Queue
Code:
from multiprocessing import Queue
objects = ["John",34,6578.9,True]
counter = 1
# We would instantiate a queue object
queue = Queue()
print('Pushing items to queue:')
for o in objects:
print('Object No: ', counter, ' ', o)
queue.put(o)
counter = counter + 1
print('\nPopping items from queue:')
counter = 1
while not queue.empty():
print('Object No: ', counter, ' ', queue.get())
counter = counter + 1Output:
Python Multiprocessing has a Queue class that helps to retrieve and fetch data for processing following FIFO(First In First Out) data structure. They are handy for storing Python pickle objects and ease sharing objects among different processes, thus helping parallel programming.
They are passed as a parameter to the Process target function to enable the process to consume that data during execution. There are mainly two functions that help us store and fetch data to and from the Queue:-
- put: put() function allows to insert data into the Queue.
- get: get() function allows to retrieve data from the Queue.
In the FIFO data structure, the stored element must be retrieved first.
FIFO data structure is analogous to Customer Service calls, whereby the customers being the elements, the ones who call the earliest, have to wait for the least to get the call connected to the customer service personnel.
3. Pool
Code:
from multiprocessing import Pool
import time
work = (["1", 5], ["2", 2], ["3", 1], ["4", 3])
def work_log(work_data):
print(" Process %s waiting %s seconds" % (work_data[0], work_data[1]))
time.sleep(int(work_data[1]))
print(" Process %s Finished." % work_data[0])
def pool_handler():
p = Pool(2)
p.map(work_log, work)
if __name__ == '__main__':
pool_handler()Output:
Python Multiprocessing Pool class helps in the parallel execution of a function across multiple input values. The variable work when declared; it is mentioned that Process 1, Process 2, Process 3, and Process 4 shall wait for 5,2,1,3 seconds, respectively. During execution, the processes discussed above wait for the interval mentioned above of time, as evident from the print statements’ orders.
4. Lock
Python Multiprocessing Lock class allows code to claim lock so that no other process can work on a similar code. There are two important functions of lock as follows: –
- acquire: acquire() function claims the lock
- release: release() function releases the lock
Let us consolidate all the things that we have learned into a single example:-
Code:
from multiprocessing import Lock, Process, Queue, current_process
import time
import queue
def do_job(tasks_to_do, tasks_finished):
while True:
try:
'''
get_nowait() function raises queue empty exception if the queue is empty.
queue(False) function also does the same task.
'''
task = tasks_to_do.get_nowait()
except queue.Empty:
break
else:
'''
if no exception is raised, add the task completion
message to tasks_finished queue
'''
print(task)
tasks_finished.put(task + ' is finished by ' + current_process().name)
time.sleep(0.25)
return True
def main():
number_of_task = 6
number_of_processes = 2
tasks_to_do = Queue()
tasks_finished = Queue()
processes = []
for i in range(number_of_task):
tasks_to_do.put("Task no " + str(i))
# creating processes
for w in range(number_of_processes):
p = Process(target=do_job, args=(tasks_to_do, tasks_finished))
processes.append(p)
p.start()
# completing process
for p in processes:
p.join()
# print the output
while not tasks_finished.empty():
print(tasks_finished.get())
return True
if __name__ == '__main__':
main()Output:
Conclusion
It is time to draw closure to this article as we have discussed the basic concepts of Multiprocessing in Python. Next time you write a big complex code, remember to apply its multiprocessing concepts to appreciate it powerfully.
Recommended Articles
This is a guide to Python Multiprocessing. Here we discuss the introduction, multiprocessing, and how we can create a parallel program using different classes. You can also go through our other related articles to learn more –
