[Nix-dev] GCC 5, Matplotlib, and tcmalloc don't work together

[Andreas Herrmann]

Thanks for your response, Vladimir.

On Wednesday 01 June 2016 16:01:58 Vladimír Čunát wrote:
> On 06/01/2016 11:30 AM, Andreas Herrmann wrote:
> > Maybe it's just a bad idea to mix allocators and I was lucky before.
> 
> Deallocating memory that was allocated by a different allocator
> certainly means trouble, and immediate crash is the better option IMO.
My understanding was the following, and please correct me if I'm wrong. I thought that Python was using the allocator it was compiled with (glibc) and my C++ module would be using the allocator it was compiled with (tcmalloc). pybind11 (similar to boost-python) would tell Python to call back into the C++ module to release any object coming out of that module such that the C++ runtime could take care to call the destructor and properly free the memory through, in that case, tcmalloc. That way those two allocators would coexist without getting into each other's way. So far, I never had any allocator related issues and no memory leaks so I assumed that the above picture was correct.

Now, the issue only appears with matplotlib.pylab which, through the qt backend, introduces its own C++ module. Previously, they were compiled with two different compilers with different C++ ABI versions (pre C++11, and post C++11). Now, after the update they're both compiled with the same GCC and C++ ABI version. I assume that this causes some conflict. It's probably more a C++ runtime issue than a Python issue - I don't know.

For me it's not a big deal. The Python module only does some post-processing where performance is not critical. Thanks to Nix I can have two separate versions of my C++ library without any effort. 


> BTW, tcmalloc compares itself to glibc-2.3; IIRC since then the default
> allocator has improved significantly, in particular in multi-threaded usage.

I found notable speedups between tcmalloc and glibc malloc (whichever version was in nix at that point) about a year ago for my use-case. Since then I haven't checked again.

Best, Andreas