Monitoring Linux system resource consumption
Introduction
Data about system resources can be obtained from virtual files in the directory
/proc
ls /proc
1 18 21 31279 43 629 acpi kallsyms self 10 18349 22 32 44 659 asound kcore slabinfo 1051 18421 23 327 45 660 buddyinfo keys softirqs 1055 19 2375 33 46 662 bus key-users stat 11 19194 24 368 482 675 cgroups kmsg swaps 11527 19359 2400 369 4925 676 cmdline kpagecount sys 11529 19402 2402 379 499 680 consoles kpageflags sysrq-trigger 11532 19570 279 380 517 684 cpuinfo loadavg sysvipc 11762 19711 282 393 5677 686 crypto locks timer_list 11764 2 283 394 571 7 devices mdstat timer_stats 11765 20 284 395 575 8 diskstats meminfo tty 11818 20027 285 396 577 8413 dma misc uptime 11819 20182 2858 397 579 8419 driver modules version 13 20311 287 398 581 8790 execdomains mounts vmallocinfo 14 20374 289 399 583 8815 fb mtrr vmstat 14600 20415 291 4 5834 8817 filesystems net zoneinfo 15 20475 292 400 5836 9 fs pagetypeinfo 16 20518 29343 401 585 9214 interrupts partitions 1672 20712 2996 402 588 9250 iomem sched_debug 1694 20839 30 403 6 9251 ioports schedstat 17 20840 31 41 60 97 irq scsi
meminfo contains information about memory
cat meminfo
MemTotal: 8009160 kB MemFree: 5474752 kB MemAvailable: 7043488 kB Buffers: 2108 kB Cached: 1830824 kB SwapCached: 0 kB Active: 1191552 kB Inactive: 907404 kB Active(anon): 310848 kB Inactive(anon): 45940 kB Active(file): 880704 kB Inactive(file): 861464 kB Unevictable: 0 kB Mlocked: 0 kB SwapTotal: 2097148 kB SwapFree: 2097148 kB …
cpu contains information about the processor
cat cpuinfo
processor : 0 vendor_id : GenuineIntel cpu family : 6 model : 142 model name : Intel(R) Core(TM) i7-10510U CPU @ 1.80GHz stepping : 12 cpu MHz : 2304.012 cache size : 8192 KB …
Linux has a convenient program top which can show data from these virtual files in real time, as well as the utility free which shows free system resources.
This guide uses knowledge from the following articles:
Monitoring total memory consumption
Monitoring total memory and processor consumption with top
#!/bin/bash
top -b | grep '%Cpu(s)\|KiB Mem'
Pay attention to the output of your top. On CentOS you may have KiB by default, and on Ubuntu MiB
If you intend to do some calculations, you need to understand what top outputs
top - 14:41:49 up 20:18, 2 users, load average: 0.51, 0.32, 0.22 Tasks: 386 total, 1 running, 385 sleeping, 0 stopped, 0 zombie %Cpu(s): 0.8 us, 1.0 sy, 0.0 ni, 98.0 id, 0.0 wa, 0.0 hi, 0.1 si, 0.0 st KiB Mem : 8008504 total, 4713352 free, 1021040 used, 2274112 buff/cache KiB Swap: 8257532 total, 8257532 free, 0 used. 6646096 avail Mem
From the manual for top the following conclusions can be drawn:
- Used CPU is 100 minus the id (idle) value
- Used memory is the used value in the line that starts with KiB Mem
| us, user | time running un-niced user processes |
| sy, system | time running kernel processes |
| ni, nice | time running niced user processes |
| id, idle | time spent in the kernel idle handler |
| wa, IO-wait | time waiting for I/O completion |
| hi | time spent servicing hardware interrupts |
| si | time spent servicing software interrupts |
| st | time stolen from this vm by the hypervisor |
Based on these considerations, you can use sed , awk and bc to pull the required values out of the top output and calculate resource consumption.
#!/bin/bash
IDLE=$(top -b -n 1 | grep '%Cpu(s)' | sed 's/id.*//;s/^.*\,//')
MEM=$(top -b -n 1 | grep 'KiB Mem' | awk '{print $8}')
CPU=$(echo "100-$IDLE" | bc -l)
echo "$(date '+%Y-%m-%d %H:%M:%S.%3N'),$CPU, $MEM" >> log.txt
Comments on the script
- Line 1: declaration of the bash script
- Line 2:
- The value before id has to be assigned to the variable IDLE
- top is started in batch mode (-b) once (-n 1)
- grep selects the line with %Cpu(s) from the top output
- The first sed condition discards everything to the right of id, including id itself
- The second sed condition discards everything to the left of the last comma
- Line 3:
- top is started in batch mode (-b) once (-n 1)
- awk finds the required column
- Line 4:
- bc helps to calculate the difference of numbers
- Line 5:
- The date, CPU consumption and memory consumption are written to log.txt
Memory consumption by individual processes
An example of a script that, for a certain number of seconds, will extract data about the memory and processor usage by several specific processes from the top readings.
#!/bin/bash
START=$(date +%s)
DURATION=2
let COUNTER=$(date +%s)-$START
while [ $COUNTER -lt $DURATION ]; do
echo $(date '+Y-%m-%d %H:%M:%S,%3N') >> log3.txt
top -b -n 1 | grep 'node\|postgres\|nginx' | awk '{print $9,$10}' >> log.txt
let COUNTER=$(date +%s)-$START
done
By default top outputs the columns in the following order
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
free
You can find the amount of free memory with the command free
free
total used free shared buff/cache available Mem: 32538152 7937972 19124512 423056 5475668 23841896 Swap: 2097148 0 2097148
If you combine free with watch you get real monitoring
watch free -m
Every 2.0s: free -m Sun Jan 17 16:20:10 2021 total used free shared buff/cache available Mem: 7820 1121 4339 56 2359 6361 Swap: 8063 0 8063
Every two seconds the value will be updated
df
Disk state
df
df -t ext4
-h human readable format
df -h
Filesystem Size Used Avail Use% Mounted on devtmpfs 3.9G 0 3.9G 0% /dev tmpfs 3.9G 0 3.9G 0% /dev/shm tmpfs 3.9G 89M 3.8G 3% /run tmpfs 3.9G 0 3.9G 0% /sys/fs/cgroup /dev/mapper/centos-root 17G 9.1G 8.0G 54% / /dev/sda1 1014M 238M 777M 24% /boot tmpfs 783M 0 783M 0% /run/user/1000
Article author: Andrei Olegovich