[Nix-dev] Debugging Symbols

[Kevin Cox]

Hello all, I'm creating a bit of noise this morning.

I noticed that almost all Nix packages are stripped of debugging
symbols and was hoping to discuss a possible solution that would both
keep the closure size minimal and allow post-hoc debugging of crashes.
The small closure size speaks for itself and since systemd helpfully
stores coredumps of all crashes it would be nice to be able to look
into them without having to recompile and observe the crash again.

The solution is obviously seperate debug files which allows the
debugging symbols to be kept separate from the binary itself. I see
that there was an earlier attempt but it appears to have failed due to
profile linking issues (https://github.com/NixOS/nixpkgs/blob/master/ni
xos/modules/config/debug-info.nix). I also think that this approach
isn't ideal because it requires the program to be "installed" which is
generally not the case for my system services.

Here is my proposal and please feel free to critique it.
- Change the strip helper to copy the debugging info first (and start
building with max debug info (-ggdb)).
- Store the debugging info in it's output ("symbols" or "debug") under
the same file path of the original binary.
- Embed the path of the debug symbols in the .gnu_debuglink section
using a hacky path.

The assumptions I am under are that there can only be one debug output
(because nix isn't dynamic enough for determining outputs at build
time) and that directories with paths continue to work in gdb.

A bit of background is that the .gnu_debuglink section is intended to
contain the basename of the debugging symbols which is looked up in the
global debug directories. However experimentation shows that while the
tools make it difficult to create a .gnu_debuglink with slashes in it
gdb will faithfully concatenate the paths and try to read the symbols
there. Using this we can set the global debug directory to /nix/store,
effectively allowing us to specify any path as the debug symbols.

That means for a file /nix/store/xxx-foo/bin/exe the build scripts
should copy the debugging symbols over to /nix/store/yyy-foo-
debug/bin/exe.debug and set the debug link to yyy-foo-
debug/bin/exe.debug. Then whenever gdb is started with exe or a
crashdump of exe it will automatically pick up the debugging symbols
(assuming they are installed).

The one downside to this approach is that installing the debugging
symbols is non-obvious. However it would be trivial to create a simple
script that takes an executable as an argument and reads the
.gnu_debuglink section out of it then realizes the path. Assuming that
you have that path in a binary cache or a store derivation it should
create those files without issue.

Another approach that I tested was to create a small python extension
(a couple of lines, below for reference), either embedded in the binary
or alongside the executable on the filesystem, that tried to load the
debug symbols by path, then if it failed printed a message with the
realization command. That way the use could just copy the command to
get the debugging symbols. However this wasn't ideal as it took more
space in the executable (roughly 6x, which really isn't much) and
required including the whole store in the "auto-load safe-path" in
order to have a seamless experience. Of course the advantage to this is
if you don't have auto-load enabled for a particular file you get a
nice message from gdb telling you how to enable it.

f=" /nix/store/yyy-foo-debug/bin/exe"
try:gdb.execute("symbol-file"+f);print("Debugging symbols loaded")
except:print('Error loading debugging symbols. They can be downloaded
with "nix-store -r"'+f);raise

I'm also going to look into other gdb extension options to do this from
a trusted extension with less size required in the actual packages.
I'll post to the list if I find something promising.

TL;DR debugging symbols for all binaries, can be downloaded after the
fact, works automatically in gdb.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.science.uu.nl/pipermail/nix-dev/attachments/20160215/f85068d4/attachment-0001.html 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 213 bytes
Desc: This is a digitally signed message part
Url : http://lists.science.uu.nl/pipermail/nix-dev/attachments/20160215/f85068d4/attachment-0002.bin 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/x-pkcs7-signature
Size: 5658 bytes
Desc: not available
Url : http://lists.science.uu.nl/pipermail/nix-dev/attachments/20160215/f85068d4/attachment-0003.bin