[Nix-dev] Unicode locale for build environments

[Benno Fünfstück]

Hello list,

right now, the stdenv appears to not set any locale. I think this means
that the locale defaults to C, which specifies ASCII as the character
encoding. For example, python then defaults to `ASCII` so it will fail if
any script tries to open a file with non-ascii characters:

$ nix-shell --pure -p python36 --command 'python -c "import locale;
print(locale.getpreferredencoding())"'
ANSI_X3.4-1968

Just recently, I've hit a build that failed due to that:

Traceback (most recent call last):
  File "nix_run_setup.py", line 8, in <module>
    exec(compile(getattr(tokenize, 'open',
open)(__file__).read().replace('\\r\\n', '\\n'), __file__, 'exec'))
  File "setup.py", line 20, in <module>
    long_description=open('README.rst').read(),
  File
"/nix/store/i5ixvcy4i6jqzlzy9aajdhf3wliixvh1-python3-3.6.1/lib/python3.6/encodings/ascii.py",
line 26, in decode
    return codecs.ascii_decode(input, self.errors)[0]
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 542:
ordinal not in range(128)

As UTF-8 is the nowadays almost always used (I have yet to see a source
archive that does not use UTF-8), I propose that we make the stdenv support
UTF-8 by default. Would this be a feasible approach? (whether to use
C.UTF-8  or some other UTF-8 locale like en_US.UTF-8 still needs to be
decided)

Regards,
Benno
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mailman.science.uu.nl/pipermail/nix-dev/attachments/20170625/ecb2890b/attachment.html>