Btukfyl

Problem

seaborn heatmap dies in jupyter notebook and in python script form.

Literature review

seaborn failing, is the closest I can find to a similar problem, but this isn't exactly the same issue. Mine is just a standalone seaborn won't print the heatmap problem.

Symptoms

import numpy as np; np.random.seed(0)

import seaborn as sns; sns.set()

import matplotlib.pyplot as plt



M=np.ones((32000,32000))



ax = sns.heatmap(M)

plt.show()

With the above code, I get a kernel stopped running message in the jupyter notebook, and a [1]+ Killed: 9 python bible_ai_construction.py in my terminal after waiting 20 minutes for the heatmap.

Overarching Goal

My hope is to be able to investigate each states relationship to each other state in a dataset. I am supplied with each state, as well as a list of all other states it is related to, small example below:

d={'a':['b','c','e'],'b':['a','c'],'c':['a'],'d':['a','e','b'],'e':['a','c']}

#set the new dictionary to one_hot_encoding representation

d={'a':[0,1,1,0,1],'b':[1,0,1,0,0],'c':[1,0,0,0,0],'d':[1,1,0,0,1],'e':[1,0,1,0,0]}

#construct matrix

M=np.matrix(list(map(lambda x:x, d.values())))

#apply M to seaborn heatmap, i.e., M.shape->(32000,32000)

Matrix with dimensions of 32000x32000 will have over billion elements.
On my computer this numpy array grows to over 8Gb in memory.

Seaborn/Matplotlib will not be able to plot such huge arrays.

You should perform some aggregation methods on your matrix, binning etc.

Simply plotting such big array will have no advantage, no one will be able to deduce any information from such detailed chart.

I think you should preview the large data in some sort of a subsampled/summarized form
and then have a detailed view connected via mouse interactions with the larger
image. You can implement this using https://github.com/AaronWatters/jp_doodle and in fact I implemented something like this (using much smaller data) as the final example in this notebook: https://github.com/AaronWatters/jp_doodle/blob/master/notebooks/Simple%20Python%20Examples.ipynb

Here is a screen shot

the left image is the image at full resolution,
the middle image is a 3x3 detail from the full resolution image and the
left swatch is a color sample in the middle of the detail. It's a little
hard to see, but there is a gray square in the full image that shows
the location of the detail which can be dragged using the mouse.

In your case you would show the subsampled image at left and stream the
full size detail region slice from the Python kernel. See
https://github.com/AaronWatters/jp_doodle/blob/master/notebooks/misc/image%20streaming%20demo.ipynb for an example of streaming data
from numpy arrays to a jp_doodle widget.