[Nix-dev] Binary patches, again

[Lluís Batlle]

Hello,

I wanted to give a try to binary patches. For the first time I used
nix-push, so I may have done something wrong.

I created an archive directory 'archive1' having the nar files for a
kdelibs-4.4.1 derivation. And an 'archive2' directory for a
kdelibs-4.4.2 derivation. I run the generate-patches.pl between its
manifests, and I got that all differences amounted to a total a bit
over 900KB. That would have been around 20MB without patches.

  /nix/store/hp1b00jzpsx3l6d15856pgb0cv8d1nqp-phonon-4.3.80
  USE 1 1 1 /nix/store/skjbqyn1hn5qwxa23jpgzn6cg770272l-phonon-4.3.80
  /nix/store/hp1b00jzpsx3l6d15856pgb0cv8d1nqp-phonon-4.3.80 <-
/nix/store/skjbqyn1hn5qwxa23jpgzn6cg770272l-phonon-4.3.80
    size 2770; full size 507823
  /nix/store/lmrmfmvqzpgbz4bscmplnfbn1wvq0zw8-kdelibs-4.4.2
  USE 1 1 1 /nix/store/qy3jz1v8n9gqck717ck8wg5n5qphv46c-kdelibs-4.4.1
  /nix/store/lmrmfmvqzpgbz4bscmplnfbn1wvq0zw8-kdelibs-4.4.2 <-
/nix/store/qy3jz1v8n9gqck717ck8wg5n5qphv46c-kdelibs-4.4.1
    size 942590; full size 18876140
  /nix/store/qkvabv7nb7zirv0sx1pwgwavqzpblrfi-attica-0.1.2
  USE 1 1 1 /nix/store/vvy78gsp9sqi61k2rag90vpmp74j2sjz-attica-0.1.2
  /nix/store/qkvabv7nb7zirv0sx1pwgwavqzpblrfi-attica-0.1.2 <-
/nix/store/vvy78gsp9sqi61k2rag90vpmp74j2sjz-attica-0.1.2
    size 342; full size 138297
  /nix/store/j568chaicfamf3b6zl14m1yraxmb96xq-shared-desktop-ontologies-0.2
  USE 1 1 1 /nix/store/h26glcsbg29l50g3x6h1drvvmwmdg4w2-shared-desktop-ontologies-0.2
  /nix/store/j568chaicfamf3b6zl14m1yraxmb96xq-shared-desktop-ontologies-0.2
<- /nix/store/h26glcsbg29l50g3x6h1drvvmwmdg4w2-shared-desktop-ontologies-0.2
    size 306; full size 76328
  /nix/store/n9zpjpy61fckq1ql1z0sxap9d1dp8vvm-soprano-2.4.0.1
  USE 1 1 1 /nix/store/765xqcvghdg85wb4w6pmd19vjygh8i6k-soprano-2.4.0.1
  /nix/store/n9zpjpy61fckq1ql1z0sxap9d1dp8vvm-soprano-2.4.0.1 <-
/nix/store/765xqcvghdg85wb4w6pmd19vjygh8i6k-soprano-2.4.0.1
    size 623; full size 902319
  /nix/store/mxdxbggpjdm1r9jxndmwq3m7wk46mxs6-polkit-qt-0.9.3
  USE 1 1 1 /nix/store/fcq0q4xg1bzh6clbi3sc1a94jl8dy4sq-polkit-qt-0.9.3
  /nix/store/mxdxbggpjdm1r9jxndmwq3m7wk46mxs6-polkit-qt-0.9.3 <-
/nix/store/fcq0q4xg1bzh6clbi3sc1a94jl8dy4sq-polkit-qt-0.9.3
    size 375; full size 58394
  /nix/store/1v443595i336m0kp3wasycsw4dgglpjw-strigi-0.7.1
  USE 1 1 1 /nix/store/djb9qh6ibnvcl0jsr0cps69awm38pg9p-strigi-0.7.1
  /nix/store/1v443595i336m0kp3wasycsw4dgglpjw-strigi-0.7.1 <-
/nix/store/djb9qh6ibnvcl0jsr0cps69awm38pg9p-strigi-0.7.1
    size 547; full size 862677

I don't know if that would increase the hydra store a lot, but it
would be nice having that enabled.
As 'urkud' in irc pointed that there are still side effects in the
store paths (nixbld1, nixbld2, ... strings in the resulting output,
impurities), we could use the database in nix-store for hashes of the
output store paths. If they match, use the patch. If not, then don't.
I guess that database exists and it is used by nix-store to check if
output paths have been modified.

Looking at a manifest, I see:
patch {
  StorePath: /nix/store/1v443595i336m0kp3wasycsw4dgglpjw-strigi-0.7.1
  NarURL: http://patata//1q1g8bnjm2d89pqqlnqr2w6y8jl673fd7dsl0gw25anp5zhpgnkq.nar-bsdiff
  Hash: sha256:1q1g8bnjm2d89pqqlnqr2w6y8jl673fd7dsl0gw25anp5zhpgnkq
  NarHash: sha256:18dxbzn7fikglaz582q0slfcb26awmdz87z34kf6kgj0w3s9k2yb
  Size: 547
  BasePath: /nix/store/djb9qh6ibnvcl0jsr0cps69awm38pg9p-strigi-0.7.1
  BaseHash: sha256:06g24ni1v3zapzf5rby1sgbi753yrihzc1gzxwdlg36x36avwnns
  Type: nar-bsdiff
}

Is the BaseHash already what I am talking about? If it is, why isn't
hydra creating patches?

I see the patches are used mostly distribution-related
(nix-push/nix-pull). Would there be any chance of being used in
nix-copy-closure? That would help in people having advantage of
patches, without having a nar-distribution system started up, in a
more peer-to-peer style.

Regards,
Lluís.