Kodeclik Blog
Python Memory Errors: Why they happen and how to fix them
A Python memory error means your program has run out of working memory. Learn about the many ways to fix memory errors and ensure your program’s execution is not stalled.
Python memory errors can be a rare occurrence for many programmers but they do happen in several situations. A memory error means that your program has run out of working memory and cannot continue or proceed. As a result, the program execution stops and your operating system or programming environment will indicate that the program has run out of memory.
There are four main reasons memory errors happen.
Reason 1: Program tries to load excessively large datasets
One common reason Python programs run out of memory is because the program is attempting to load a massive dataset into main memory and the size of the dataset is larger than the available memory size. One way to fix this problem is to do distributed processing on your dataset. For instance, load only a segment of the dataset at a time, process it, and then load a different segment. Not all problems are amenable to such distributed processing, however clever algorithms can overcome the limitations of memory.
Reason 2: Memory is limited by 32-bit Python
A second reason memory errors happen is because, even though your computer has ample memory, the version of Python you are running could be old and can address only 32-bit memory (as opposed to 64-bit memory). With 32 bits, your programs can address only 2^32 addresses which is about 4GB. Now, of the 4GB, your operating system and other essential services on your computer will use up memory as well leaving much less memory available to your Python environment. Reinstalling the latest Python packages will usually alleviate this issue.
Reason 3: Invalid memory management by installed packages
If the above two reasons are not the root cause of your program, it could be the case that a package or combination of packages could be the source of poor memory management. Reinstall packages using a state-of-the-art package management tool such as Conda or PyPi.
Reason 4: Unfettered object creation
Finally, your program might be creating new objects in a rampant manner without disposing of them even when they are unused. As a result the program rapidly runs of memory. For instance, you might be using a package like numpy and creating a series of large matrices that may not be used. This could be another cause of Python memory problems
How to fix memory problems in Python
To fix memory problems we can use a package such as “gc”. Here is a Python program to import this library and get a count of the number of objects using the Python get_count() method:
import gc
print(gc.get_count())The output when we run it (your output would be different) is:
(599, 1, 1)The sequence of numbers indicate the number of objects allocated in each generation (with 599 being the youngest generation).
We can use the gc.collect() method to do garbage collection. Here’s how that would work:
import gc
print(gc.get_count())
gc.collect()
print(gc.get_count())The output is:
(599, 1, 1)
(0, 0, 0)Note that the garbage collector has cleaned up a huge number of objects.
This blogpost has highlighted four key situations where Python memory errors happen and how to fix them. Keep this information handy as you embark on ambitious projects!
To learn more about the inner workings of Python, learn about the Python vars() function.
Interested in more things Python? See our blogpost on Python's enumerate() capability. Also if you like Python+math content, see our blogpost on Magic Squares. Finally, master the Python print function!
Want to learn Python with us? Sign up for 1:1 or small group classes.
