[Nix-dev] Going through hell with systemd timers

[4levels]

Hi Exi,

thank you for your reply.

This is the timers config I'm using (note that I'm starting this every 5
minutes to troubleshoot, is supposed to run every 2 hours or so)

backup = {
  description = "Backup service";
  after = [ "network.target" "mysql.target" ];
  path = [ pkgs.procps pkgs.gawk pkgs.nettools pkgs.mysql pkgs.php
pkgs.duplicity pkgs.postfix ];
  script =
  ''
      ./s3Backup.sh
  '';
  startAt = "*-*-* *:0/5:00";



And the contents of the s3Backup.sh script:

s3Backup = name:
  ''
    #!${pkgs.bash}/bin/bash

    ${builtins.readFile ./src/envrc}

    # Your GPG key
    GPG_KEY=

    # export PATH="$PATH:/var/setuid-wrappers:/run/current-system/sw/bin:/run/current-system/sw/sbin"

    # Set up some variables for logging
    LOGFILE="/var/lib/projects/${name}/log/duplicity-backup.log"
    DAILYLOGFILE="/var/lib/projects/${name}/log/duplicity-backup.daily.log"
    FULLBACKLOGFILE="/var/lib/projects/${name}/log/duplicity-backup.full.log"
    HOST=`hostname`
    DATE=`date +%Y-%m-%d`
    MAILADDR="dev at domain.com"
    TODAY=$(date +%d%m%Y)

    # The S3 destination followed by bucket name
    DEST="s3://s3.amazonaws.com/projects-backup-eu-west/${name}"

    is_running=$(ps -ef | grep duplicity  | grep python | wc -l)

    if [ ! -f $FULLBACKLOGFILE ]; then
      touch $FULLBACKLOGFILE
    fi

    if [ $is_running -eq 0 ]; then
      # Clear the old daily log file
      cat /dev/null > ''${DAILYLOGFILE}

      # Trace function for logging, don't change this
      trace () {
        stamp=`date +%Y-%m-%d_%H:%M:%S`
        echo "$stamp: $*" >> ''${DAILYLOGFILE}
      }

      # Dump $PATH
      trace "Current PATH: $PATH"

      # How long to keep backups for
      OLDER_THAN="1M"

      # The source of your backup
      SOURCE=/var/lib/projects/${name}

      FULL=
      tail -1 ''${FULLBACKLOGFILE} | grep ''${TODAY} > /dev/null
      if [ $? -ne 0 -a $(date +%d) -eq 1 ]; then
        FULL=full
      fi;

      trace "Backup for local filesystem started"

      trace "... removing old backups"

      duplicity remove-older-than ''${OLDER_THAN} ''${DEST}
--s3-use-new-style >> ''${DAILYLOGFILE} 2>&1

      trace "... backing up filesystem"

      duplicity \
        ''${FULL} \
        -v9 \
        --s3-use-new-style --s3-european-buckets --no-encryption \
        --include=/var/lib/projects/${name}/data/backup \
        --exclude=/** \
        --allow-source-mismatch \
        ''${SOURCE} ''${DEST} >> ''${DAILYLOGFILE} 2>&1

      trace "Backup for local filesystem complete"
      trace "------------------------------------"

      # Send the daily log file by email
      BACKUPSTATUS=`cat "$DAILYLOGFILE" | grep Errors | awk '{ print $2 }'`
      if [ "$BACKUPSTATUS" != "0" ]; then
        echo -e "Subject: Duplicity Backup Log for $HOST - $DATE -
${name}\n\n$(cat $DAILYLOGFILE)" | sendmail $MAILADDR
      elif [ "$FULL" = "full" ]; then
        echo "$(date +%d%m%Y_%T) Full Back Done" >> $FULLBACKLOGFILE
      fi

      # Append the daily log file to the main log file
      cat "$DAILYLOGFILE" >> $LOGFILE

    fi

    unset AWS_ACCESS_KEY_ID
    unset AWS_SECRET_ACCESS_KEY
    unset PASSPHRASE
  '';



On Thu, Jan 28, 2016 at 10:34 AM exi <e-nixos at wthack.de> wrote:

> Hi Erik,
>
> does duplicity use an ssh connection? Does it depend on your ssh
> passphrase to be present? Do you use a ssh agent?
>
> "BackendException" from the traceback looks more like a connection issue
> than a nix issue.
> Which user is running the timer command?
> Could you post your timer config?
>
> Regards,
>
> exi
>
>
> On 28.01.2016 09:07, 4levels wrote:
>
> Hi Nix-Devs,
>
> yesterday I came to a point of really wanting to break something out of
> sheer frustration over failing systemd timer calls.
>
> I've setup a duplicity backup script over s3 that works flawlessly when
> invoked from terminal, but fails misrably when being called from a timer.
>
> I've tried everything I know, including but not limited to adding my full
> user $PATH to the script, adding all possible related packages to the path
> directive, .. nothing seems to work.
>
> The duplicity error is very vague (BackendException) and when adding
> maximum verbosity to the duplicity call ( -v9 ) I do get some error which
> seems to be related to a very old duplicity bug.  Since duplicity uses
> python (the version I could trace seems to be 2.7) with python-boto for the
> s3 backend - the issue seems to be related to this, but I can't figure out
> what could be the reason since all required packages are installed and
> operational from the commandline.
>
> Has anyone experience with running python-based code in systemd timer
> calls (without being bitten)?
>
> On top of that, Github went down for a couple of hours last night and to
> make things even worse, NixOps cannot finish a deploy on any of the 5
> machines I'm managing with it anymore, with a vague error message:
>
> v-ams02...> updating GRUB 2 menu...
> v-ams02...> Died at
> /nix/var/nix/profiles/system/bin/switch-to-configuration line 264.
> v-ams02...> error: unable to activate new configuration
>
> Kind regards.
>
> Erik
>
> Duplicity error with maximum verbosity:
> Backend error detail: Traceback (most recent call last):
>   File
> "/nix/store/ap2bv0p5m8napigg7f6yciap4nm61ap8-duplicity-0.7.02/bin/.duplicity-wrapped",
> line 1519, in <module>
>     with_tempdir(main)
>   File
> "/nix/store/ap2bv0p5m8napigg7f6yciap4nm61ap8-duplicity-0.7.02/bin/.duplicity-wrapped",
> line 1513, in with_tempdir
>     fn()
>   File
> "/nix/store/ap2bv0p5m8napigg7f6yciap4nm61ap8-duplicity-0.7.02/bin/.duplicity-wrapped",
> line 1354, in main
>     action = commandline.ProcessCommandLine(sys.argv[1:])
>   File
> "/nix/store/ap2bv0p5m8napigg7f6yciap4nm61ap8-duplicity-0.7.02/lib/python2.7/site-packages/duplicity/commandline.py",
> line 1070, in ProcessCommandLine
>     backup, local_pathname = set_backend(args[0], args[1])
>   File
> "/nix/store/ap2bv0p5m8napigg7f6yciap4nm61ap8-duplicity-0.7.02/lib/python2.7/site-packages/duplicity/commandline.py",
> line 961, in set_backend
>     globals.backend = backend.get_backend(bend)
>   File
> "/nix/store/ap2bv0p5m8napigg7f6yciap4nm61ap8-duplicity-0.7.02/lib/python2.7/site-packages/duplicity/backend.py",
> line 223, in get_backend
>     obj = get_backend_object(url_string)
>   File
> "/nix/store/ap2bv0p5m8napigg7f6yciap4nm61ap8-duplicity-0.7.02/lib/python2.7/site-packages/duplicity/backend.py",
> line 209, in get_backend_object
>     return factory(pu)
>   File
> "/nix/store/ap2bv0p5m8napigg7f6yciap4nm61ap8-duplicity-0.7.02/lib/python2.7/site-packages/duplicity/backends/_boto_single.py",
> line 161, in __init__
>     self.resetConnection()
>   File
> "/nix/store/ap2bv0p5m8napigg7f6yciap4nm61ap8-duplicity-0.7.02/lib/python2.7/site-packages/duplicity/backends/_boto_single.py",
> line 187, in resetConnection
>     raise BackendException(err.message)
>
>
>
> !DSPAM:56a9cc5f200881139745903!
>
>
>
> _______________________________________________
> nix-dev mailing listnix-dev at lists.science.uu.nlhttp://lists.science.uu.nl/mailman/listinfo/nix-dev
>
>
> !DSPAM:56a9cc5f200881139745903!
>
>
>
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.science.uu.nl/pipermail/nix-dev/attachments/20160128/0b6b6613/attachment.html