Metpy Static Stability - MemoryError: Unable to allocate 13.4 GiB for an array with shape (19, 1825, 180, 288) and data type float64 - out-of-memory

Question

I have downloaded GFDL-ESM model global data for temperature for 2010-2014 period of around 3.5 GB. I want to calcuate static stability using the METPY library.
My dataset looks like this Temperature Data
The code I am using is as follows:
import xarray as xr
import numpy as np
import pandas as pd
import geopandas as gp
import matplotlib.pyplot as plt
import cartopy
import cartopy.crs as ccrs
from cartopy import feature as cf
import netCDF4 as nc
import seaborn as sns
import glob
from datetime import datetime, timedelta
import cartopy.feature as cfeature
import matplotlib.gridspec as gridspec
import metpy.calc as mpcalc
from metpy.units import units
from metpy.plots.declarative import *
from netCDF4 import num2date
import scipy.ndimage as ndimage
from scipy.ndimage import gaussian_filter
from siphon.ncss import NCSS
from metpy.cbook import get_test_data
from metpy.interpolate import cross_section
import math
all=glob.glob(" {path} /ta_Eday_GFDL-ESM4_historical_r1i1p1f1_gr1_20100101-20141231.nc")
all
ds= xr.open_mfdataset(all).metpy.parse_cf()
ds
t=ds['ta']
pt = mpcalc.static_stability( t.plev*units.millibar , t*units.K )
After running this program the error I am getting is as follows:
--------
MemoryError Traceback (most recent call last)
~\AppData\Local\Temp\ipykernel_14188\3773388047.py in <cell line: 1>()
----> 1 pt = mpcalc.static_stability( t.plev*units.millibar , t*units.K )
~\miniconda3\lib\site-packages\metpy\xarray.py in wrapper(*args, **kwargs)
1542 )
1543
-> 1544 return func(*bound_args.args, **bound_args.kwargs)
1545 return wrapper
~\miniconda3\lib\site-packages\metpy\xarray.py in wrapper(*args, **kwargs)
1233
1234 # Evaluate inner calculation
-> 1235 result = func(*bound_args.args, **bound_args.kwargs)
1236
1237 # Wrap output based on match and match_unit
~\miniconda3\lib\site-packages\metpy\units.py in wrapper(*args, **kwargs)
294 def wrapper(*args, **kwargs):
295 _check_units_inner_helper(func, sig, defaults, dims, *args, **kwargs)
--> 296 return func(*args, **kwargs)
297
298 return wrapper
~\miniconda3\lib\site-packages\metpy\calc\thermo.py in static_stability(pressure, temperature, vertical_dim)
3125
3126 """
-> 3127 theta = potential_temperature(pressure, temperature)
3128
3129 return - mpconsts.Rd * temperature / pressure * first_derivative(
~\miniconda3\lib\site-packages\metpy\xarray.py in wrapper(*args, **kwargs)
1233
1234 # Evaluate inner calculation
-> 1235 result = func(*bound_args.args, **bound_args.kwargs)
1236
1237 # Wrap output based on match and match_unit
~\miniconda3\lib\site-packages\metpy\units.py in wrapper(*args, **kwargs)
294 def wrapper(*args, **kwargs):
295 _check_units_inner_helper(func, sig, defaults, dims, *args, **kwargs)
--> 296 return func(*args, **kwargs)
297
298 return wrapper
~\miniconda3\lib\site-packages\metpy\calc\thermo.py in potential_temperature(pressure, temperature)
137
138 """
--> 139 return temperature / exner_function(pressure)
140
141
~\miniconda3\lib\site-packages\metpy\xarray.py in wrapper(*args, **kwargs)
1233
1234 # Evaluate inner calculation
-> 1235 result = func(*bound_args.args, **bound_args.kwargs)
1236
1237 # Wrap output based on match and match_unit
~\miniconda3\lib\site-packages\metpy\units.py in wrapper(*args, **kwargs)
294 def wrapper(*args, **kwargs):
295 _check_units_inner_helper(func, sig, defaults, dims, *args, **kwargs)
--> 296 return func(*args, **kwargs)
297
298 return wrapper
~\miniconda3\lib\site-packages\metpy\calc\thermo.py in exner_function(pressure, reference_pressure)
95
96 """
---> 97 return (pressure / reference_pressure).to('dimensionless')**mpconsts.kappa
98
99
~\miniconda3\lib\site-packages\pint\quantity.py in __truediv__(self, other)
1339
1340 def __truediv__(self, other):
-> 1341 return self._mul_div(other, operator.truediv)
1342
1343 def __rtruediv__(self, other):
~\miniconda3\lib\site-packages\pint\quantity.py in wrapped(self, *args, **kwargs)
137 elif isinstance(other, list) and other and isinstance(other[0], type(self)):
138 return NotImplemented
--> 139 return f(self, *args, **kwargs)
140
141 return wrapped
~\miniconda3\lib\site-packages\pint\quantity.py in wrapped(self, *args, **kwargs)
117 def ireduce_dimensions(f):
118 def wrapped(self, *args, **kwargs):
--> 119 result = f(self, *args, **kwargs)
120 try:
121 if result._REGISTRY.auto_reduce_dimensions:
~\miniconda3\lib\site-packages\pint\quantity.py in _mul_div(self, other, magnitude_op, units_op)
1311 other = other.to_root_units()
1312
-> 1313 magnitude = magnitude_op(new_self._magnitude, other._magnitude)
1314 units = units_op(new_self._units, other._units)
1315
MemoryError: Unable to allocate 13.4 GiB for an array with shape (19, 1825, 180, 288) and data type float64
Please help.
​

Answer 1

As indicated by the MemoryError, the problem is that the calculation you're requesting needs to allocate a large 13GB array, which isn't fitting in the memory on your system.
When you use open_mfdataset, xarray lazily loads the data, meaning it doesn't actually read all of the values into memory until they are requested.
Unfortunately, MetPy's calculations operate eagerly, meaning when you call static_stability on this full dataset, it's going to iterate over all of these values and needs to store the full final result into memory.
The Dask library tries to address this by allowing for "chunking" arrays that are bigger than can fit in memory and smartly handling computations. Unfortunately, while xarray can use Dask internally, MetPy's support for Dask right now is not particularly robust; I'm not sure how well 1static_stability` would work when given a large Dask array.
If Dask doesn't work here with MetPy, one work-around is to only calculate on a subset of the data (e.g. whatever size you'd be doing analysis on, like a single time or a single vertical level) and iterate over these subsets.