[Nix-dev] Going through hell with systemd timers
Guillaume Maudoux (Layus)
layus.on at gmail.com
Thu Jan 28 11:18:01 CET 2016
Hi,
My experience with urlwatch was that the SSL_CERT_FILE env var was missing.
This may also be your issue if you are using the network.
It is however always possible to run the service manually, and see the logs.
A service declaration with a startAt attribute creates two services,
.service and .timer.
You can start your service with |# systemctl start <service>.service|
and see the logs in journald.
(No need to wait for the timer.)
I made urlwatch work with the following snippet :
|systemd.services.urlwatch = rec { description = "Run urlwatch
(${startAt})"; startAt = "hourly"; environment = { inherit
(config.environment.variables) SSL_CERT_FILE; }; serviceConfig = { User
= "layus"; # should use a user unit... ExecStart =
"${urlwatch}/bin/urlwatch -v"; }; }; |
For debugging, I used:
|# systemctl start urlwatch.service| : Start the service once.
|$ systemctl status urlwatch.service -l -n 1000| : See the systemd logs
for the last run, up to 1000 full lines.
|# journalctl -xef --unit=urlwatch| : Print all the logs, -f follows the
output in real time.
A very simple trick is indeed to dump the environment at the start of
the script.
Layus.
Le 28/01/16 10:59, 4levels a écrit :
> Hi Zimbatm (is that your name :-)
>
> I'm currently trying to debug by using printenv to view the
> differences in both environments. I'm not yet familiar with using
> nix-shell :-s
> But since I cannot even deploy or rebuild switch anymore (see my other
> email to this list) I'm pretty stuck.
>
> I'm currently repartitioning and reinstalling the Vultr machines
> (*sigh*) to see if that brings any changes..
>
> Thank you for your pointer to the environment vars!
>
> Erik
>
> On Thu, Jan 28, 2016 at 10:56 AM zimbatm <zimbatm at zimbatm.com
> <mailto:zimbatm at zimbatm.com>> wrote:
>
> One common error with system services are missing environment
> variable. When testing with your shell you will have $HOME set for
> example.
>
>
> On Thu, 28 Jan 2016 09:43 4levels <4levels at gmail.com
> <mailto:4levels at gmail.com>> wrote:
>
> Hi Exi,
>
> thank you for your reply.
>
> This is the timers config I'm using (note that I'm starting
> this every 5 minutes to troubleshoot, is supposed to run every
> 2 hours or so)
>
> backup = {
> description = "Backup service";
> after = [ "network.target" "mysql.target" ];
> path = [ pkgs.procps pkgs.gawk pkgs.nettools pkgs.mysql pkgs.php pkgs.duplicity pkgs.postfix ];
> script =
> ''
> ./s3Backup.sh
> '';
> startAt = "*-*-* *:0/5:00";
>
>
>
> And the contents of the s3Backup.sh script:
>
> s3Backup = name:
> ''
> #!${pkgs.bash}/bin/bash
>
> ${builtins.readFile ./src/envrc}
>
> # Your GPG key
> GPG_KEY=
>
> # export PATH="$PATH:/var/setuid-wrappers:/run/current-system/sw/bin:/run/current-system/sw/sbin"
>
> # Set up some variables for logging
> LOGFILE="/var/lib/projects/${name}/log/duplicity-backup.log"
> DAILYLOGFILE="/var/lib/projects/${name}/log/duplicity-backup.daily.log"
> FULLBACKLOGFILE="/var/lib/projects/${name}/log/duplicity-backup.full.log"
> HOST=`hostname`
> DATE=`date +%Y-%m-%d`
> MAILADDR="dev at domain.com <mailto:dev at domain.com>"
> TODAY=$(date +%d%m%Y)
>
> # The S3 destination followed by bucket name
> DEST="s3://s3.amazonaws.com/projects-backup-eu-west/${name}
> <http://s3.amazonaws.com/projects-backup-eu-west/$%7Bname%7D>"
>
> is_running=$(ps -ef | grep duplicity | grep python | wc -l)
>
> if [ ! -f $FULLBACKLOGFILE ]; then
> touch $FULLBACKLOGFILE
> fi
>
> if [ $is_running -eq 0 ]; then
> # Clear the old daily log file
> cat /dev/null > ''${DAILYLOGFILE}
>
> # Trace function for logging, don't change this
> trace () {
> stamp=`date +%Y-%m-%d_%H:%M:%S`
> echo "$stamp: $*" >> ''${DAILYLOGFILE}
> }
>
> # Dump $PATH
> trace "Current PATH: $PATH"
>
> # How long to keep backups for
> OLDER_THAN="1M"
>
> # The source of your backup
> SOURCE=/var/lib/projects/${name}
>
> FULL=
> tail -1 ''${FULLBACKLOGFILE} | grep ''${TODAY} > /dev/null
> if [ $? -ne 0 -a $(date +%d) -eq 1 ]; then
> FULL=full
> fi;
>
> trace "Backup for local filesystem started"
>
> trace "... removing old backups"
>
> duplicity remove-older-than ''${OLDER_THAN} ''${DEST} --s3-use-new-style >> ''${DAILYLOGFILE} 2>&1
>
> trace "... backing up filesystem"
>
> duplicity \
> ''${FULL} \
> -v9 \
> --s3-use-new-style --s3-european-buckets --no-encryption \
> --include=/var/lib/projects/${name}/data/backup \
> --exclude=/** \
> --allow-source-mismatch \
> ''${SOURCE} ''${DEST} >> ''${DAILYLOGFILE} 2>&1
>
> trace "Backup for local filesystem complete"
> trace "------------------------------------"
>
> # Send the daily log file by email
> BACKUPSTATUS=`cat "$DAILYLOGFILE" | grep Errors | awk '{ print $2 }'`
> if [ "$BACKUPSTATUS" != "0" ]; then
> echo -e "Subject: Duplicity Backup Log for $HOST - $DATE - ${name}\n\n$(cat $DAILYLOGFILE)" | sendmail $MAILADDR
> elif [ "$FULL" = "full" ]; then
> echo "$(date +%d%m%Y_%T) Full Back Done" >> $FULLBACKLOGFILE
> fi
>
> # Append the daily log file to the main log file
> cat "$DAILYLOGFILE" >> $LOGFILE
>
> fi
>
> unset AWS_ACCESS_KEY_ID
> unset AWS_SECRET_ACCESS_KEY
> unset PASSPHRASE
> '';
>
>
> On Thu, Jan 28, 2016 at 10:34 AM exi <e-nixos at wthack.de
> <mailto:e-nixos at wthack.de>> wrote:
>
> Hi Erik,
>
> does duplicity use an ssh connection? Does it depend on
> your ssh passphrase to be present? Do you use a ssh agent?
>
> "BackendException" from the traceback looks more like a
> connection issue than a nix issue.
> Which user is running the timer command?
> Could you post your timer config?
>
> Regards,
>
> exi
>
>
> On 28.01.2016 09:07, 4levels wrote:
>> Hi Nix-Devs,
>>
>> yesterday I came to a point of really wanting to break
>> something out of sheer frustration over failing systemd
>> timer calls.
>>
>> I've setup a duplicity backup script over s3 that works
>> flawlessly when invoked from terminal, but fails misrably
>> when being called from a timer.
>>
>> I've tried everything I know, including but not limited
>> to adding my full user $PATH to the script, adding all
>> possible related packages to the path directive, ..
>> nothing seems to work.
>>
>> The duplicity error is very vague (BackendException) and
>> when adding maximum verbosity to the duplicity call ( -v9
>> ) I do get some error which seems to be related to a very
>> old duplicity bug. Since duplicity uses python (the
>> version I could trace seems to be 2.7) with python-boto
>> for the s3 backend - the issue seems to be related to
>> this, but I can't figure out what could be the reason
>> since all required packages are installed and operational
>> from the commandline.
>>
>> Has anyone experience with running python-based code in
>> systemd timer calls (without being bitten)?
>>
>> On top of that, Github went down for a couple of hours
>> last night and to make things even worse, NixOps cannot
>> finish a deploy on any of the 5 machines I'm managing
>> with it anymore, with a vague error message:
>>
>> v-ams02...> updating GRUB 2 menu...
>> v-ams02...> Died at
>> /nix/var/nix/profiles/system/bin/switch-to-configuration
>> line 264.
>> v-ams02...> error: unable to activate new configuration
>>
>> Kind regards.
>>
>> Erik
>>
>> Duplicity error with maximum verbosity:
>> Backend error detail: Traceback (most recent call last):
>> File
>> "/nix/store/ap2bv0p5m8napigg7f6yciap4nm61ap8-duplicity-0.7.02/bin/.duplicity-wrapped",
>> line 1519, in <module>
>> with_tempdir(main)
>> File
>> "/nix/store/ap2bv0p5m8napigg7f6yciap4nm61ap8-duplicity-0.7.02/bin/.duplicity-wrapped",
>> line 1513, in with_tempdir
>> fn()
>> File
>> "/nix/store/ap2bv0p5m8napigg7f6yciap4nm61ap8-duplicity-0.7.02/bin/.duplicity-wrapped",
>> line 1354, in main
>> action = commandline.ProcessCommandLine(sys.argv[1:])
>> File
>> "/nix/store/ap2bv0p5m8napigg7f6yciap4nm61ap8-duplicity-0.7.02/lib/python2.7/site-packages/duplicity/commandline.py",
>> line 1070, in ProcessCommandLine
>> backup, local_pathname = set_backend(args[0], args[1])
>> File
>> "/nix/store/ap2bv0p5m8napigg7f6yciap4nm61ap8-duplicity-0.7.02/lib/python2.7/site-packages/duplicity/commandline.py",
>> line 961, in set_backend
>> globals.backend = backend.get_backend(bend)
>> File
>> "/nix/store/ap2bv0p5m8napigg7f6yciap4nm61ap8-duplicity-0.7.02/lib/python2.7/site-packages/duplicity/backend.py",
>> line 223, in get_backend
>> obj = get_backend_object(url_string)
>> File
>> "/nix/store/ap2bv0p5m8napigg7f6yciap4nm61ap8-duplicity-0.7.02/lib/python2.7/site-packages/duplicity/backend.py",
>> line 209, in get_backend_object
>> return factory(pu)
>> File
>> "/nix/store/ap2bv0p5m8napigg7f6yciap4nm61ap8-duplicity-0.7.02/lib/python2.7/site-packages/duplicity/backends/_boto_single.py",
>> line 161, in __init__
>> self.resetConnection()
>> File
>> "/nix/store/ap2bv0p5m8napigg7f6yciap4nm61ap8-duplicity-0.7.02/lib/python2.7/site-packages/duplicity/backends/_boto_single.py",
>> line 187, in resetConnection
>> raise BackendException(err.message)
>>
>>
>>
>> !DSPAM:56a9cc5f200881139745903!
>>
>>
>>
>> _______________________________________________
>> nix-dev mailing list
>> nix-dev at lists.science.uu.nl
>> <mailto:nix-dev at lists.science.uu.nl>
>> http://lists.science.uu.nl/mailman/listinfo/nix-dev
>>
>>
>> !DSPAM:56a9cc5f200881139745903!
>>
>>
>
>
> _______________________________________________
> nix-dev mailing list
> nix-dev at lists.science.uu.nl <mailto:nix-dev at lists.science.uu.nl>
> http://lists.science.uu.nl/mailman/listinfo/nix-dev
>
>
>
> _______________________________________________
> nix-dev mailing list
> nix-dev at lists.science.uu.nl
> http://lists.science.uu.nl/mailman/listinfo/nix-dev
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.science.uu.nl/pipermail/nix-dev/attachments/20160128/e23e9a2b/attachment-0001.html
More information about the nix-dev
mailing list