I recently logged into the Admin mail account of a Zimbra mail server I manage to see if there was anything noticeable to look at. It’s not an account I log in much, since it usually only holds daily reports and email notifications when services stop and start.
Well, what a surprise when I suddenly see over a million emails in the inbox! 1.000.000! And to my surprise, the subject of 99% of them was a variations of the following:
Disk /snap/core/13425 at 100% on mail.domain.com: Disk /snap/certbot/2192 at 100% on mail.domain.com: Disk /snap/canonical-livepatch/146 at 100% on mail.domain.com: Disk /snap/core20/1518 at 100% on mail.domain.com:
And the content of these emails was basically one line like the ones you usually find in the log files, eg:
mail zimbramon[22102]: 22102:crit: Disk warning: mail.domain.com: /snap/canonical-livepatch/146 on device /dev/loop6 at 100% mail zimbramon[22102]: 22102:crit: Disk warning: mail.domain.com: /snap/core/13425 on device /dev/loop3 at 100%
Cause
This is caused because Zimbra monitors the existing disks and alerts when these are full, but these /dev/loopX disks have nothing to do with Zimbra and don´t need to be monitored. They are automatically created if you have snap installed and they all are read-only disks. You can see these disks if you run the command df -h:
root@mail:~# df -h Filesystem Size Used Avail Use% Mounted on udev 5.9G 0 5.9G 0% /dev tmpfs 1.2G 115M 1.1G 10% /run /dev/mapper/mail--vg-root 245G 122G 111G 53% / tmpfs 5.9G 0 5.9G 0% /dev/shm tmpfs 5.0M 0 5.0M 0% /run/lock tmpfs 5.9G 0 5.9G 0% /sys/fs/cgroup /dev/loop0 64M 64M 0 100% /snap/core20/1738 /dev/loop1 9.0M 9.0M 0 100% /snap/canonical-livepatch/164 /dev/loop3 9.0M 9.0M 0 100% /snap/canonical-livepatch/146 /dev/loop2 64M 64M 0 100% /snap/core20/1778 /dev/loop4 45M 45M 0 100% /snap/certbot/2582 /dev/loop5 117M 117M 0 100% /snap/core/14399 /dev/loop6 45M 45M 0 100% /snap/certbot/2618 /dev/sda1 720M 164M 520M 24% /boot tmpfs 1.2G 0 1.2G 0% /run/user/999 tmpfs 1.2G 0 1.2G 0% /run/user/1000
Solution
Since this issue is not critical, but the emails can very quickly become very annoying, the best thing to do is to exclude all these loop devices from Zimbras monitoring. This can be achieved by running the following command as the Zimbra user (run su - zimbra as root):
zmlocalconfig -e zmstat_df_excludes='/dev/loop0:/dev/loop1:/dev/loop2:/dev/loop3:/dev/loop4:/dev/loop5:/dev/loop6'
Update the command with the loop devices you see on your server. After this, restart the stat data collectors by running:
zmstatctl restart
This should solve your issue.
Additional Tip
Upon restart of the server or over time you might find new/different loop devices and the emails for these coming in again. If this happens, just run the above commands again updating the list of devices to exclude.
Zimbra Wiki: https://wiki.zimbra.com/wiki/Disable_Disk_Space_Monitoring_for_Loop_Devices