[Nix-dev] What monitoring tools do you currently prefer?

[Mathijs Kwik]

Hi all,

I know most of you run servers on nixos, so I would like your opinion on
monitoring/guarding system performance and health.

I've used monit in the past to monitor certain processes, which is quite
nice, but I think nowadays systemd is capable of setting the same kinds
of limits, and since it uses cgroups this will be even more thorough as
it catches forked children too.

But as I don't fully trust the performance of the cloud hosting I'm
using, I would like to gain some insights into more generic stats as
well. Things like long io response times that cause dbs to hang might be
nice to know about. Also anomalies like tcp queue overflows (indicating
synfloods) and stuff like that might be nice to find.

Tools like sysstat/sar come to mind, but as these have been around for
eons, there might be more modern alternatives that I don't know about
yet.

What do you use and why?

Regards,
Mathijs