[Nix-dev] Re: Auto-updating sources: handling the general case

[Michael Raskin]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 08/20/2010 02:29 AM, Yury G. Kudryashov wrote:
>> What can be done:
>> For general case there is update-upstream data. There are some fixes
>> that can be made:
> You should not assume that people know what is "update-upstream data".

Well, all relevant details are described explicitly in this mail.
Something that requires a ton of attributes that show underlying
implementation and three files per expression, but does update packages.

>> 3) The resulting source data (url etc.) can be put back right into Nix
>> expression if we require marking the generated attribute set by putting
>> something like ``autoGeneratedClause = "auto-update";''. The hard part
>> (writing regexp for finding such attrSet in the expression text) is
>> already done (in CL-PPCRE tree syntax - debugging this as a
>> string-expressed regexp is too hard). Or maybe we want to fix
>> "--parse-only --xml"? In that case we will need a lot of position
>> information, though.
> Hm... Didn't understood. Can we start by disscussing what should be in nix 
> expressions, not how should we parse them?

Isn't this commutative?

>> 4) The attributes should be designed from the beginning for different
>> methods in different cases. I.e. there is a "method" where we read
>> sourceforge page and find the latest release (optionally: filter by
>> filename, process nested groups); there is a method where project
>> news/download page contains a direct link to the latest version; etc.
> We can have some attributes for the "general case" and functions that 
> generate these attributes for specific cases.

Unfortunately, it may complicate things more than needed.. There are
multiple "simple cases", and some "quite simple". Uniting them can lead
to complex ones.

Example: SF.net = link + redirect (or regular substitution); GNOME-like
project = two levels of directory listing. Does unifying have any merit
here? There can also be FTP listings where a separate approach may be
needed. And so on.

>> 5) Version extraction should also be split into different handling
>> methods.
>>
>> I guess the thing requiring most synchronization is naming.
> Yes. I think that we should start by disscussing what metadata we want to 
> store in the expression, then how should we store it.
> 
> I think that the following metadata should be extractable from nix 
> expression:
> 
> 1. How can one obtain the list of all available versions (an FTP directory, 
> a page with download links etc.)?
>   a) stable versions;
>   b) snapshots;
>   c) betas;
>   d) development repository (+some well-known branch names?) <-- this could 
> be integrated with repo-manager.
> 
> This "stability" information could be used to test pre-releases in branches.

I'd say that it would be nice if we had a branch for undertested updates
not called "trunk"...

> Michael, could you please describe the attributes update-upstream-data uses? 
> We can use these names as a starting point instead of reinventing the wheel.

Currently, it uses the following model:
1) The page is fetched
Attr: downloadPage
2) All links (link targets) from the page are extracted
3) Links are filtered by a regular expression to leave only source code
packages
Attr: sourceRegexp; blacklistRegexp
4) Links are fed to a command choosing one of them. Unfortunately, in
practice this command sometimes changes the link, not just chooses one
(for example, to go around SF.net redirects)
Attr: choiceCommand
5) If link is relative, it is treated as relative to download page.
6) Version is extracted by regex replacements; in the URL it can be
replaced with a reference
Attr: versionExtractorSedScript, versionReferenceCreator
7) mirror:// version of URL is made, if necessary
Attr: mirrorSedScript
8) new hash is calculated
9) new name for expression is calculated using version
Attr: baseName

There are also special code paths for SF.net and for updating fetchgit
expressions.

> 2 (optional). Does upstream signs the releases? If yes, key id+fingerprint 
> and URL of the pgp-signature.
> 
> P.S.: I don't think that package upgrades should be performed by computers. 
> I think that each new version must be reviewed before commiting to svn.

Well, I guess we should have capability for fully-automated package
upgrades and test runs (I.e. update everything and see whether Quake
screenshot in a VM can be taken and not pure black). How we should _use_
it is another question (I am actually all for a daemon upgrading
packages in micro-branches, Hydra building this and then maintainer
testing the package and merging or disapproving this change after
testing the package - whether this is practical is another question).
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.15 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iQEcBAEBAgAGBQJMbb0UAAoJEE6tnN0aWvw3CrEIAKiY/g77K0Heh9MesCqYldTk
HkQpD6ddFvD29PfjSuLNb6lfzVPAWeQqj6ELpVq6umJJgWZq09DJ08ZQU8F2LPwE
pdaHEWcd8PszmDtm4KoLMblr4nAXXxWFsoBvb99mhJ58IR8HqBCfBcxX2gG0dfPU
vQERrAiZPyukCJk8VBw7OHgs07VRKYjRKJsxG2PcmtGyuvoIOIsfYygffckroysA
4yIWd9ZZa1BVxlagahLJgrpsPGKKOeSYApTNddur6X/6pK8ek1xv2++KE/htgZFD
MLB+uEfAijq+JIgSibj2Vkx/1PkQ0Jk82ZKXw1O1vUKXsDio3TUO4VMJb9tnnQQ=
=3S2e
-----END PGP SIGNATURE-----