[Nix-dev] CPU optimized packages

Fri Feb 8 00:14:26 CET 2013

But how to do this on a per package basis?
I don't want to rebuild my entire system, I already had my "fun" with 
Gentoo in the past :-)

> Eelco Dolstra <mailto:eelco.dolstra at logicblox.com>
> 7 februari 2013 21:11
> Hi,
>
>
> This is actually easy to do in the stdenv branch, since you can say 
> something like:
>
> stdenv.userHook = ''
> NIX_CFLAGS_COMPILE+=" -march=corei7"
> '';
>
> in ~/.nixpkgs/config.nix.
>
> Shea Levy <mailto:shea at shealevy.com>
> 7 februari 2013 19:47
> On 02/07/2013 10:09 AM, Danny Wilson wrote:
>> Great!
>> Maybe instead of depending on time, it could use `md5sum 
>> /proc/cpuinfo`  or something that identifies the machine?
>>
>> I get the concepts but I'm very new to the nix language. How would I 
>> modify an existing package to use this in my ~/.nixpkgs/config.nix ?
>
> Unfortunately, this isn't currently easy to do, we'd need to add some 
> hooks in stdenv to allow adding some custom CFLAGS. Can you open an 
> issue on github.com/nixos/nixpkgs for this so I can have a reminder 
> about adding this?
>
>>
>>
>>
>>> Shea Levy <mailto:shea at shealevy.com>
>>> 7 februari 2013 16:01
>>> OK, well technically that is possible, with something like this:
>>>
>>> extra_cflags = import (pkgs.runCommand "cflags" {} ''
>>> mkdir $out
>>> echo "" | gcc -O3 -march=native -mtune=native -v -E - 2>&1 |grep cc1 
>>> |sed -r 's/.*? - -(.*)$/-\1/' > $out/flags
>>> echo "builtins.readFile ./flags" > $out/default.nix
>>> '');
>>>
>>> Basically, what that does is build a derivation which creates a nix 
>>> expression, then imports that nix expression. One problem with this 
>>> is that the result is saved in the nix store, meaning if you share 
>>> nix stores between machines it won't work (because it will run once 
>>> on one machine and the same result will be used on different 
>>> machines). To get around this, you can make the derivation depend on 
>>> the current time, which will force it to calculate the cflags every 
>>> time you build something. Also, it's possible that in the future 
>>> builds run in chroot will be inside namespaces that block how gcc 
>>> detects the host system, so we should make this not run in chroot:
>>>
>>> extra_cflags = import (pkgs.runCommand "cflags" {time = 
>>> builtins.currentTime; __noChroot = true;} ''
>>> mkdir $out
>>> echo "" | gcc -O3 -march=native -mtune=native -v -E - 2>&1 |grep cc1 
>>> |sed -r 's/.*? - -(.*)$/-\1/' > $out/flags
>>> echo "builtins.readFile ./flags" > $out/default.nix
>>> '');
>>>
>>> I'm not sure of your level of understanding of nix, so please let me 
>>> know if this needs more explanation.
>>>
>>> Thanks,
>>> Shea
>>>
>>> On 02/07/2013 09:53 AM, Danny Wilson wrote:
>>>
>>> Danny Wilson <mailto:danny at prime.vc>
>>> 7 februari 2013 15:53
>>> Well in this case I am the user. And I don't want to fill this in 
>>> for every different server I might have.
>>>
>>> My suggestion was to make -march=native pure, by expanding it on the 
>>> system configuring itself at nix expression evaluation time:
>>>
>>> $ echo "" | gcc -O3 -march=native -mtune=native -v -E - 2>&1 |grep 
>>> cc1 |sed -r 's/.*? - -(.*)$/-\1/'
>>> -march=corei7 -mcx16 -msahf -mno-movbe -maes -mno-pclmul -mpopcnt 
>>> -mno-abm -mno-lwp -mno-fma -mno-fma4 -mno-xop -mno-bmi -mno-tbm 
>>> -mno-avx -msse4.2 -msse4.1 --param l1-cache-size=32 --param 
>>> l1-cache-line-size=64 --param l2-cache-size=4096 -mtune=generic -O3
>>>
>>>
>>>
>>>
>>> Shea Levy <mailto:shea at shealevy.com>
>>> 7 februari 2013 15:33
>>> Hi Danny,
>>>
>>> Unfortunately, there's not a good answer to this because 
>>> -march=native is inherently impure. Is there any reason you can't 
>>> tell users "Find out what -march=native means on your system and 
>>> then fill in nixpkgs.config.extra_cflags with the result?" or some 
>>> such (Note that config.extra_cflags doesn't currently exist AFAIK, 
>>> but could)?
>>>
>>> ~Shea
>>>
>>> On 02/07/2013 07:36 AM, Danny Wilson wrote:
>>>
>>> Danny Wilson <mailto:danny at prime.vc>
>>> 7 februari 2013 13:36
>>> With help of the gentoo wiki ( 
>>> http://en.gentoo-wiki.com/wiki/Hardware_CFLAGS )
>>> I found this produces a nice GCC option list:
>>>
>>> echo "" | gcc -O3 -march=native -mtune=native -v -E - 2>&1 |grep cc1 
>>> |sed -r 's/.*? - -(.*)$/-\1/'
>>>
>>>
>>>
>>>
>>> _______________________________________________
>>> nix-dev mailing list
>>> nix-dev at lists.science.uu.nl
>>> http://lists.science.uu.nl/mailman/listinfo/nix-dev
>>> Danny Wilson <mailto:danny at prime.vc>
>>> 7 februari 2013 12:37
>>> I'd like some packages to be optimized for my server CPU. GCC has
>>> -march=native for this, but that would not generate a cpu specific 
>>> nixos
>>> hash.
>>>
>>> Is there a nix expression available which adds something like this:
>>> gcc -march=native -Q --help=target |grep enabled
>>>
>>> to CFLAGS and correctly hashing it?
>>>
>>>
>>>
>>> _______________________________________________
>>> nix-dev mailing list
>>> nix-dev at lists.science.uu.nl
>>> http://lists.science.uu.nl/mailman/listinfo/nix-dev
>>> ------------------------------------------------------------------------
>>
>>
>>
>>
>> _______________________________________________
>> nix-dev mailing list
>> nix-dev at lists.science.uu.nl
>> http://lists.science.uu.nl/mailman/listinfo/nix-dev
>
> Danny Wilson <mailto:danny at prime.vc>
> 7 februari 2013 16:09
> Great!
> Maybe instead of depending on time, it could use `md5sum 
> /proc/cpuinfo`  or something that identifies the machine?
>
> I get the concepts but I'm very new to the nix language. How would I 
> modify an existing package to use this in my ~/.nixpkgs/config.nix ?
>
>
>
>
>
> _______________________________________________
> nix-dev mailing list
> nix-dev at lists.science.uu.nl
> http://lists.science.uu.nl/mailman/listinfo/nix-dev
> Shea Levy <mailto:shea at shealevy.com>
> 7 februari 2013 16:01
> OK, well technically that is possible, with something like this:
>
> extra_cflags = import (pkgs.runCommand "cflags" {} ''
> mkdir $out
> echo "" | gcc -O3 -march=native -mtune=native -v -E - 2>&1 |grep cc1 
> |sed -r 's/.*? - -(.*)$/-\1/' > $out/flags
> echo "builtins.readFile ./flags" > $out/default.nix
> '');
>
> Basically, what that does is build a derivation which creates a nix 
> expression, then imports that nix expression. One problem with this is 
> that the result is saved in the nix store, meaning if you share nix 
> stores between machines it won't work (because it will run once on one 
> machine and the same result will be used on different machines). To 
> get around this, you can make the derivation depend on the current 
> time, which will force it to calculate the cflags every time you build 
> something. Also, it's possible that in the future builds run in chroot 
> will be inside namespaces that block how gcc detects the host system, 
> so we should make this not run in chroot:
>
> extra_cflags = import (pkgs.runCommand "cflags" {time = 
> builtins.currentTime; __noChroot = true;} ''
> mkdir $out
> echo "" | gcc -O3 -march=native -mtune=native -v -E - 2>&1 |grep cc1 
> |sed -r 's/.*? - -(.*)$/-\1/' > $out/flags
> echo "builtins.readFile ./flags" > $out/default.nix
> '');
>
> I'm not sure of your level of understanding of nix, so please let me 
> know if this needs more explanation.
>
> Thanks,
> Shea
>
> On 02/07/2013 09:53 AM, Danny Wilson wrote:
>
> Danny Wilson <mailto:danny at prime.vc>
> 7 februari 2013 15:53
> Well in this case I am the user. And I don't want to fill this in for 
> every different server I might have.
>
> My suggestion was to make -march=native pure, by expanding it on the 
> system configuring itself at nix expression evaluation time:
>
> $ echo "" | gcc -O3 -march=native -mtune=native -v -E - 2>&1 |grep cc1 
> |sed -r 's/.*? - -(.*)$/-\1/'
> -march=corei7 -mcx16 -msahf -mno-movbe -maes -mno-pclmul -mpopcnt 
> -mno-abm -mno-lwp -mno-fma -mno-fma4 -mno-xop -mno-bmi -mno-tbm 
> -mno-avx -msse4.2 -msse4.1 --param l1-cache-size=32 --param 
> l1-cache-line-size=64 --param l2-cache-size=4096 -mtune=generic -O3
>
>
>
>
> ------------------------------------------------------------------------

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.science.uu.nl/pipermail/nix-dev/attachments/20130208/b049db36/attachment-0001.html 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: postbox-contact.jpg
Type: image/jpeg
Size: 1412 bytes
Desc: not available
Url : http://lists.science.uu.nl/pipermail/nix-dev/attachments/20130208/b049db36/attachment-0003.jpg 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: postbox-contact.jpg
Type: image/jpeg
Size: 1178 bytes
Desc: not available
Url : http://lists.science.uu.nl/pipermail/nix-dev/attachments/20130208/b049db36/attachment-0004.jpg 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: compose-unknown-contact.jpg
Type: image/jpeg
Size: 770 bytes
Desc: not available
Url : http://lists.science.uu.nl/pipermail/nix-dev/attachments/20130208/b049db36/attachment-0005.jpg 