[Nix-dev] Going through hell with systemd timers
zimbatm
zimbatm at zimbatm.com
Fri Jan 29 00:50:08 CET 2016
We really need to get rid of that SSL_CERT_FILE environment variable...
related issue: https://github.com/NixOS/nixpkgs/issues/8486
On Thu, 28 Jan 2016 at 15:06 4levels <4levels at gmail.com> wrote:
> Dear Guillaume,
>
> you saved my day (and sleepless last night)!
> This line worked all the magic for me:
>
> environment = {
> inherit (config.environment.variables) SSL_CERT_FILE;
> };
>
> Make sure to pass through our office whenever you're near Gent, Belgium,
> we're keeping a bottle of champagne chilled for you ;-)
>
> And again nix-dev proves to be the most valuable resource in our NixOs /
> NixOps experience!
>
> Kind regards to you all, and as before, keep up the amazing work and great
> attitude..
>
>
> Erik
>
> On Thu, Jan 28, 2016 at 11:19 AM Guillaume Maudoux (Layus) <
> layus.on at gmail.com> wrote:
>
>> Hi,
>>
>> My experience with urlwatch was that the SSL_CERT_FILE env var was
>> missing.
>> This may also be your issue if you are using the network.
>>
>> It is however always possible to run the service manually, and see the
>> logs.
>> A service declaration with a startAt attribute creates two services, .service
>> and .timer.
>> You can start your service with # systemctl start <service>.service and
>> see the logs in journald.
>> (No need to wait for the timer.)
>>
>> I made urlwatch work with the following snippet :
>>
>> systemd.services.urlwatch = rec {
>> description = "Run urlwatch (${startAt})";
>> startAt = "hourly";
>> environment = {
>> inherit (config.environment.variables) SSL_CERT_FILE;
>> };
>>
>> serviceConfig = {
>> User = "layus"; # should use a user unit...
>> ExecStart = "${urlwatch}/bin/urlwatch -v";
>> };
>> };
>>
>> For debugging, I used:
>> # systemctl start urlwatch.service : Start the service once.
>> $ systemctl status urlwatch.service -l -n 1000 : See the systemd logs
>> for the last run, up to 1000 full lines.
>> # journalctl -xef --unit=urlwatch : Print all the logs, -f follows the
>> output in real time.
>>
>> A very simple trick is indeed to dump the environment at the start of the
>> script.
>>
>> Layus.
>>
>> Le 28/01/16 10:59, 4levels a écrit :
>>
>> Hi Zimbatm (is that your name :-)
>>
>> I'm currently trying to debug by using printenv to view the differences
>> in both environments. I'm not yet familiar with using nix-shell :-s
>> But since I cannot even deploy or rebuild switch anymore (see my other
>> email to this list) I'm pretty stuck.
>>
>> I'm currently repartitioning and reinstalling the Vultr machines (*sigh*)
>> to see if that brings any changes..
>>
>> Thank you for your pointer to the environment vars!
>>
>> Erik
>>
>> On Thu, Jan 28, 2016 at 10:56 AM zimbatm < <zimbatm at zimbatm.com>
>> zimbatm at zimbatm.com> wrote:
>>
>>> One common error with system services are missing environment variable.
>>> When testing with your shell you will have $HOME set for example.
>>>
>>> On Thu, 28 Jan 2016 09:43 4levels < <4levels at gmail.com>4levels at gmail.com>
>>> wrote:
>>>
>>>> Hi Exi,
>>>>
>>>> thank you for your reply.
>>>>
>>>> This is the timers config I'm using (note that I'm starting this every
>>>> 5 minutes to troubleshoot, is supposed to run every 2 hours or so)
>>>>
>>>> backup = {
>>>> description = "Backup service";
>>>> after = [ "network.target" "mysql.target" ];
>>>> path = [ pkgs.procps pkgs.gawk pkgs.nettools pkgs.mysql pkgs.php pkgs.duplicity pkgs.postfix ];
>>>> script =
>>>> ''
>>>> ./s3Backup.sh
>>>> '';
>>>> startAt = "*-*-* *:0/5:00";
>>>>
>>>>
>>>>
>>>> And the contents of the s3Backup.sh script:
>>>>
>>>> s3Backup = name:
>>>> ''
>>>> #!${pkgs.bash}/bin/bash
>>>>
>>>> ${builtins.readFile ./src/envrc}
>>>>
>>>> # Your GPG key
>>>> GPG_KEY=
>>>>
>>>> # export PATH="$PATH:/var/setuid-wrappers:/run/current-system/sw/bin:/run/current-system/sw/sbin"
>>>>
>>>> # Set up some variables for logging
>>>> LOGFILE="/var/lib/projects/${name}/log/duplicity-backup.log"
>>>> DAILYLOGFILE="/var/lib/projects/${name}/log/duplicity-backup.daily.log"
>>>> FULLBACKLOGFILE="/var/lib/projects/${name}/log/duplicity-backup.full.log"
>>>> HOST=`hostname`
>>>> DATE=`date +%Y-%m-%d`
>>>> MAILADDR="dev at domain.com"
>>>> TODAY=$(date +%d%m%Y)
>>>>
>>>> # The S3 destination followed by bucket name
>>>> DEST="s3://s3.amazonaws.com/projects-backup-eu-west/${name} <http://s3.amazonaws.com/projects-backup-eu-west/$%7Bname%7D>"
>>>>
>>>> is_running=$(ps -ef | grep duplicity | grep python | wc -l)
>>>>
>>>> if [ ! -f $FULLBACKLOGFILE ]; then
>>>> touch $FULLBACKLOGFILE
>>>> fi
>>>>
>>>> if [ $is_running -eq 0 ]; then
>>>> # Clear the old daily log file
>>>> cat /dev/null > ''${DAILYLOGFILE}
>>>>
>>>> # Trace function for logging, don't change this
>>>> trace () {
>>>> stamp=`date +%Y-%m-%d_%H:%M:%S`
>>>> echo "$stamp: $*" >> ''${DAILYLOGFILE}
>>>> }
>>>>
>>>> # Dump $PATH
>>>> trace "Current PATH: $PATH"
>>>>
>>>> # How long to keep backups for
>>>> OLDER_THAN="1M"
>>>>
>>>> # The source of your backup
>>>> SOURCE=/var/lib/projects/${name}
>>>>
>>>> FULL=
>>>> tail -1 ''${FULLBACKLOGFILE} | grep ''${TODAY} > /dev/null
>>>> if [ $? -ne 0 -a $(date +%d) -eq 1 ]; then
>>>> FULL=full
>>>> fi;
>>>>
>>>> trace "Backup for local filesystem started"
>>>>
>>>> trace "... removing old backups"
>>>>
>>>> duplicity remove-older-than ''${OLDER_THAN} ''${DEST} --s3-use-new-style >> ''${DAILYLOGFILE} 2>&1
>>>>
>>>> trace "... backing up filesystem"
>>>>
>>>> duplicity \
>>>> ''${FULL} \
>>>> -v9 \
>>>> --s3-use-new-style --s3-european-buckets --no-encryption \
>>>> --include=/var/lib/projects/${name}/data/backup \
>>>> --exclude=/** \
>>>> --allow-source-mismatch \
>>>> ''${SOURCE} ''${DEST} >> ''${DAILYLOGFILE} 2>&1
>>>>
>>>> trace "Backup for local filesystem complete"
>>>> trace "------------------------------------"
>>>>
>>>> # Send the daily log file by email
>>>> BACKUPSTATUS=`cat "$DAILYLOGFILE" | grep Errors | awk '{ print $2 }'`
>>>> if [ "$BACKUPSTATUS" != "0" ]; then
>>>> echo -e "Subject: Duplicity Backup Log for $HOST - $DATE - ${name}\n\n$(cat $DAILYLOGFILE)" | sendmail $MAILADDR
>>>> elif [ "$FULL" = "full" ]; then
>>>> echo "$(date +%d%m%Y_%T) Full Back Done" >> $FULLBACKLOGFILE
>>>> fi
>>>>
>>>> # Append the daily log file to the main log file
>>>> cat "$DAILYLOGFILE" >> $LOGFILE
>>>>
>>>> fi
>>>>
>>>> unset AWS_ACCESS_KEY_ID
>>>> unset AWS_SECRET_ACCESS_KEY
>>>> unset PASSPHRASE
>>>> '';
>>>>
>>>>
>>>> On Thu, Jan 28, 2016 at 10:34 AM exi <e-nixos at wthack.de> wrote:
>>>>
>>>>> Hi Erik,
>>>>>
>>>>> does duplicity use an ssh connection? Does it depend on your ssh
>>>>> passphrase to be present? Do you use a ssh agent?
>>>>>
>>>>> "BackendException" from the traceback looks more like a connection
>>>>> issue than a nix issue.
>>>>> Which user is running the timer command?
>>>>> Could you post your timer config?
>>>>>
>>>>> Regards,
>>>>>
>>>>> exi
>>>>>
>>>>>
>>>>> On 28.01.2016 09:07, 4levels wrote:
>>>>>
>>>>> Hi Nix-Devs,
>>>>>
>>>>> yesterday I came to a point of really wanting to break something out
>>>>> of sheer frustration over failing systemd timer calls.
>>>>>
>>>>> I've setup a duplicity backup script over s3 that works flawlessly
>>>>> when invoked from terminal, but fails misrably when being called from a
>>>>> timer.
>>>>>
>>>>> I've tried everything I know, including but not limited to adding my
>>>>> full user $PATH to the script, adding all possible related packages to the
>>>>> path directive, .. nothing seems to work.
>>>>>
>>>>> The duplicity error is very vague (BackendException) and when adding
>>>>> maximum verbosity to the duplicity call ( -v9 ) I do get some error which
>>>>> seems to be related to a very old duplicity bug. Since duplicity uses
>>>>> python (the version I could trace seems to be 2.7) with python-boto for the
>>>>> s3 backend - the issue seems to be related to this, but I can't figure out
>>>>> what could be the reason since all required packages are installed and
>>>>> operational from the commandline.
>>>>>
>>>>> Has anyone experience with running python-based code in systemd timer
>>>>> calls (without being bitten)?
>>>>>
>>>>> On top of that, Github went down for a couple of hours last night and
>>>>> to make things even worse, NixOps cannot finish a deploy on any of the 5
>>>>> machines I'm managing with it anymore, with a vague error message:
>>>>>
>>>>> v-ams02...> updating GRUB 2 menu...
>>>>> v-ams02...> Died at
>>>>> /nix/var/nix/profiles/system/bin/switch-to-configuration line 264.
>>>>> v-ams02...> error: unable to activate new configuration
>>>>>
>>>>> Kind regards.
>>>>>
>>>>> Erik
>>>>>
>>>>> Duplicity error with maximum verbosity:
>>>>> Backend error detail: Traceback (most recent call last):
>>>>> File
>>>>> "/nix/store/ap2bv0p5m8napigg7f6yciap4nm61ap8-duplicity-0.7.02/bin/.duplicity-wrapped",
>>>>> line 1519, in <module>
>>>>> with_tempdir(main)
>>>>> File
>>>>> "/nix/store/ap2bv0p5m8napigg7f6yciap4nm61ap8-duplicity-0.7.02/bin/.duplicity-wrapped",
>>>>> line 1513, in with_tempdir
>>>>> fn()
>>>>> File
>>>>> "/nix/store/ap2bv0p5m8napigg7f6yciap4nm61ap8-duplicity-0.7.02/bin/.duplicity-wrapped",
>>>>> line 1354, in main
>>>>> action = commandline.ProcessCommandLine(sys.argv[1:])
>>>>> File
>>>>> "/nix/store/ap2bv0p5m8napigg7f6yciap4nm61ap8-duplicity-0.7.02/lib/python2.7/site-packages/duplicity/commandline.py",
>>>>> line 1070, in ProcessCommandLine
>>>>> backup, local_pathname = set_backend(args[0], args[1])
>>>>> File
>>>>> "/nix/store/ap2bv0p5m8napigg7f6yciap4nm61ap8-duplicity-0.7.02/lib/python2.7/site-packages/duplicity/commandline.py",
>>>>> line 961, in set_backend
>>>>> globals.backend = backend.get_backend(bend)
>>>>> File
>>>>> "/nix/store/ap2bv0p5m8napigg7f6yciap4nm61ap8-duplicity-0.7.02/lib/python2.7/site-packages/duplicity/backend.py",
>>>>> line 223, in get_backend
>>>>> obj = get_backend_object(url_string)
>>>>> File
>>>>> "/nix/store/ap2bv0p5m8napigg7f6yciap4nm61ap8-duplicity-0.7.02/lib/python2.7/site-packages/duplicity/backend.py",
>>>>> line 209, in get_backend_object
>>>>> return factory(pu)
>>>>> File
>>>>> "/nix/store/ap2bv0p5m8napigg7f6yciap4nm61ap8-duplicity-0.7.02/lib/python2.7/site-packages/duplicity/backends/_boto_single.py",
>>>>> line 161, in __init__
>>>>> self.resetConnection()
>>>>> File
>>>>> "/nix/store/ap2bv0p5m8napigg7f6yciap4nm61ap8-duplicity-0.7.02/lib/python2.7/site-packages/duplicity/backends/_boto_single.py",
>>>>> line 187, in resetConnection
>>>>> raise BackendException(err.message)
>>>>>
>>>>>
>>>>>
>>>>> !DSPAM:56a9cc5f200881139745903!
>>>>>
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> nix-dev mailing listnix-dev at lists.science.uu.nlhttp://lists.science.uu.nl/mailman/listinfo/nix-dev
>>>>>
>>>>>
>>>>> !DSPAM:56a9cc5f200881139745903!
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> _______________________________________________
>>>> nix-dev mailing list
>>>> nix-dev at lists.science.uu.nl
>>>> http://lists.science.uu.nl/mailman/listinfo/nix-dev
>>>>
>>>
>>
>> _______________________________________________
>> nix-dev mailing listnix-dev at lists.science.uu.nlhttp://lists.science.uu.nl/mailman/listinfo/nix-dev
>>
>> _______________________________________________
>> nix-dev mailing list
>> nix-dev at lists.science.uu.nl
>> http://lists.science.uu.nl/mailman/listinfo/nix-dev
>>
> _______________________________________________
> nix-dev mailing list
> nix-dev at lists.science.uu.nl
> http://lists.science.uu.nl/mailman/listinfo/nix-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.science.uu.nl/pipermail/nix-dev/attachments/20160128/3caddd42/attachment-0001.html
More information about the nix-dev
mailing list