昨天晚上zabbix服务器报一台主机的磁盘空间超过利用率，恰好我电脑开着，所以登录上去看了下，排查过程如下：
因为知道这台服务器的具体业务功能，正常情况下是不会立即达到监控线的，通过top查看了下进程，发现sytemd-journal这个进程，怀疑日志文件被大量的写入。
systemd-journald是一个改进型日志管理服务，可以收集来自内核、系统早期启动阶段的日志、系统守护进程在启动和运行中的标准输出和错误信息，还有syslog的日志。该日志服务仅仅把日志集中保存在单一结构的日志文件/run/log中，由于日志是经历过压缩和格式化的二进制数据，所以在查看和定位的时候很迅速。默认情况下并不会持久化保存日志，只会保留一个月的日志。另外，一些rsyslog无法收集的日志也会被journal记录到.
[root@ops-monitor-01 /var/log/journal] # du -sh /var/log/journal/ 1.5G /var/log/journal/ 说明:果然，这个目录比之前多了很多东西，本来通过删除清理下里面的日志，突然想到journactl有个高级的功能，可以做，于是有了下面的操作 [root@ops-monitor-01 /var/log/journal] # journalctl --vacuum-size=500M Deleted archived journal /var/log/journal/3c6ff97c6fbe4598b53fd04e08937468/system@ef03f01eefec47f5ada60bd1dba2ab24-0000000000000001-000550f7deac7403.journal (64.0M). Deleted archived journal /var/log/journal/3c6ff97c6fbe4598b53fd04e08937468/user-1000@0b70f7779047455090ed3bdeb41372e9-00000000000047fb-000551307d518b3e.journal (8.0M). Deleted archived journal /var/log/journal/3c6ff97c6fbe4598b53fd04e08937468/system@ef03f01eefec47f5ada60bd1dba2ab24-0000000000013fcd-0005535c3692ab03.journal (48.0M). Deleted archived journal /var/log/journal/3c6ff97c6fbe4598b53fd04e08937468/user-1000@0b70f7779047455090ed3bdeb41372e9-00000000000140be-0005535fdff3ea1d.journal (8.0M). Deleted archived journal /var/log/journal/3c6ff97c6fbe4598b53fd04e08937468/system@ef03f01eefec47f5ada60bd1dba2ab24-000000000001f484-000555c082ebcdf7.journal (64.0M). Deleted archived journal /var/log/journal/3c6ff97c6fbe4598b53fd04e08937468/user-1000@0b70f7779047455090ed3bdeb41372e9-000000000001fb4d-000555d51cf51bc6.journal (8.0M). Deleted archived journal /var/log/journal/3c6ff97c6fbe4598b53fd04e08937468/system@ef03f01eefec47f5ada60bd1dba2ab24-00000000000315f3-00055824d9a1b26c.journal (96.0M). Deleted archived journal /var/log/journal/3c6ff97c6fbe4598b53fd04e08937468/user-1000@0b70f7779047455090ed3bdeb41372e9-0000000000032b7f-0005585387ff50f0.journal (8.0M). Deleted archived journal /var/log/journal/3c6ff97c6fbe4598b53fd04e08937468/system@ef03f01eefec47f5ada60bd1dba2ab24-000000000004ac87-00055a8926aba5d2.journal (104.1M). Deleted archived journal /var/log/journal/3c6ff97c6fbe4598b53fd04e08937468/user-1000@0b70f7779047455090ed3bdeb41372e9-0000000000051652-00055b1a3b4b15ab.journal (8.0M). Deleted archived journal /var/log/journal/3c6ff97c6fbe4598b53fd04e08937468/system@ef03f01eefec47f5ada60bd1dba2ab24-0000000000069762-00055ced73056d1d.journal (112.1M). Deleted archived journal /var/log/journal/3c6ff97c6fbe4598b53fd04e08937468/user-1000@0b70f7779047455090ed3bdeb41372e9-000000000006a14c-00055cf8d0f034cb.journal (8.0M). Deleted archived journal /var/log/journal/3c6ff97c6fbe4598b53fd04e08937468/system@ef03f01eefec47f5ada60bd1dba2ab24-0000000000089211-00055e2ef50709d8.journal (128.1M). Deleted archived journal /var/log/journal/3c6ff97c6fbe4598b53fd04e08937468/user-1000@0b70f7779047455090ed3bdeb41372e9-000000000008b23a-00055e319abb77e8.journal (8.0M). Deleted archived journal /var/log/journal/3c6ff97c6fbe4598b53fd04e08937468/system@ef03f01eefec47f5ada60bd1dba2ab24-00000000000bc349-00055e8eb47900a7.journal (128.1M). Deleted archived journal /var/log/journal/3c6ff97c6fbe4598b53fd04e08937468/user-1000@0b70f7779047455090ed3bdeb41372e9-00000000000d57a3-00055e8ef8591843.journal (8.0M). Deleted archived journal /var/log/journal/3c6ff97c6fbe4598b53fd04e08937468/system@ef03f01eefec47f5ada60bd1dba2ab24-00000000000f99cd-00055e8f57466708.journal (128.1M). Vacuuming done, freed 937.0M of archived journals on disk. [root@ops-monitor-01 /var/log/journal] # du -sh /var/log/journal/ 513M /var/log/journal/ [root@ops-monitor-01 /var/log/journal] # df -h Filesystem Size Used Avail Use% Mounted on /dev/vda1 99G 76G 18G 82% / devtmpfs 1.9G 0 1.9G 0% /dev tmpfs 1.9G 100K 1.9G 1% /dev/shm tmpfs 1.9G 488K 1.9G 1% /run tmpfs 1.9G 0 1.9G 0% /sys/fs/cgroup /dev/vdb 197G 5.9G 181G 4% /war /dev/vdc1 100G 60G 41G 60% /mnt/yum tmpfs 380M 0 380M 0% /run/user/1000
到此，问题已经接近，需要做进一步了解的朋友可以查看下journalctl的帮助命令，相信会有收获。
