Updated April 3, 2023
Introduction to Python Memory Error
Memory Error is a kind of error in python that occurs when where the memory of the RAM we are using could not support the execution of our code since the memory of the RAM is smaller and the code we are executing requires more than the memory of our existing RAM, this often occurs when a large volume of data is fed to the memory and when the program has run out of memory in processing the data.
Syntax of Python Memory Error
When performing an operation that generates or using a big volume of data, it will lead to a Memory Error.
Code:
## Numpy operation which return random unique values
import numpy as np
np.random.uniform(low=1,high=10,size=(10000,100000))Output:
For the same function, let us see the Name Error.
Code:
## Functions which return values
def calc_sum(x,y):
op = x + y
return(op)The numpy operation of generating random numbers with a range from a low value of 1 and highest of 10 and a size of 10000 to 100000 will throw us a Memory Error since the RAM could not support the generation of that large volume of data.
How does Memory Error Works?
Most often, Memory Error occurs when the program creates any number of objects, and the memory of the RAM runs out. When working on Machine Learning algorithms most of the large datasets seems to create Memory Error. Different types of Memory Error occur during python programming. Sometimes even if your RAM size is large enough to handle the datasets, you will get a Memory Error. This is due to the Python version you might be using some times; 32-bit will not work if your system is adopted to a 64-bit version. In such cases, you can go uninstall 32-bit python from your system and install the 64-bit from the Anaconda website. When you are installing different python packages using the pip command or other commands may lead to improper installation and throws a Memory Error.
In such cases, we can use the conda install command in python prompt and install those packages to fix the Memory Error.
Example:
Another type of Memory Error occurs when the memory manager has used the Hard disk space of our system to store the data that exceeds the RAM capacity. Upon working, the computer stores all the data and uses up the memory throws a Memory Error.
Avoiding Memory Errors in Python
The most important case for Memory Error in python is one that occurs during the use of large datasets. Upon working on Machine Learning problems, we often come across large datasets which, upon executing an ML algorithm for classification or clustering, the computer memory will instantly run out of memory. We can overcome such problems by executing Generator functions. It can be used as a user-defined function that can be used when working with big datasets.
Generators allow us to efficiently use the large datasets into many segments without loading the complete dataset. Generators are very useful in working on big projects where we have to work with a large volume of data. Generators are functions that are used to return an iterator. Iterators can be used to loop the data over. Writing a normal iterator function in python loops the entire dataset and iters over it. This is where the generator comes in handy it does not allow the complete dataset to loop over since it causes a Memory Error and terminates the program.
The generator function has a special characteristic from other functions where a statement called yield is used in place of the traditional return statement that returns the output of the function.
A sample Generator function is given as an example:
Code:
def sample_generator():
for i in range(10000000):
yield i
gen_integ= sample_generator()
for i in gen_integ:
print(i)Output:
In this sample generator function, we have generated integers using the function sample generator, which is assigned to the variable gen_integ, and then the variable is iterated. This allows us to iter over one single value at a time instead of passing the entire set of integers.
In the sample code given below, we have tried to read a large dataset into small bits using the generator function. This kind of reading would allow us to process large data in a limited size without using up the system memory completely.
Code:
def readbits(filename, mode="r", chunk_size=20):
with open(filename, mode) as f:
while True:
data = f.read(chunk_size)
if not data:
break
yield data
def main():
filename = "C://Users//Balaji//Desktop//Test"
for bits in readbits(filename):
print(bits) Output:
There is another useful technique that can be used to free memory while we are working on a large number of objects. A simple way to erase the objects that are not referenced is by using a garbage collector or gc statement.
Code:
import gc
gc.collect()The import garbage collector and gc.collect() statement allows us to free the memory by removing the objects which the user does not reference.
There are additional ways in which we can manage the memory of our system CPU where we can write code to limit the CPU usage of memory.
Code:
import resource
def limit_memory(Datasize):
min_, max_ = resource.getrlimit(resource.RLIMIT_AS)
resource.setrlimit(resource.RLIMIT_AS, (Datasize, max_)) This allows us to manage CPU usage to prevent Memory Error.
Some of the other techniques that can be used to overcome the Memory Error are to limit our sample size of we are working on, especially while performing complex machine learning algorithms. Or we could update our system with more memory, or we can use the cloud services like Azure, AWS, etc. that provides the user with strong computing capabilities.
Another way is to use the Relational Database Management technique where open-source databases like MySQL are available free of cost. It can be used to store large volumes of data; also, we can adapt to big data storage services to effectively work with large volumes.
Conclusion
In detail, we have seen the Memory Error that occurs in the Python programming language and the techniques to overcome the Name Error. The main take away to remember in python Memory Error is the memory usage of our RAM where the operations are taking place, and efficiently using the above-mentioned techniques will allow us to overcome the Memory Error.
Recommended Articles
This is a guide to Python Memory Error. Here we discuss the introduction, working and avoiding memory errors in python, respectively. You may also have a look at the following articles to learn more –
