“ENOSPC: no space left on device”: disk, inodes and file watchers
The same error covers three different limits. Two minutes with df tells you which one, and the fix is different for each.
Which ENOSPC do you have?
| Check | Shows | Problem |
|---|---|---|
df -h | a filesystem at 100% Use | The disk (or that partition) is full. |
df -i | 100% IUse while df -h has free space | Out of inodes: too many files, usually tiny ones. |
| The error text | System limit for number of file watchers reached | Not a disk problem: the inotify watch limit (dev servers, VS Code, Jest). |
df -h /tmp /dev/shm | a small tmpfs at 100% | A RAM-backed folder filled, not your main disk. |
Note which mount point is full. / full and /home fine are different problems, and Docker usually lives in /var/lib/docker on /.
A full disk: find the space
df -h # which filesystem
sudo du -xh / --max-depth=2 2>/dev/null | sort -h | tail -15
sudo du -xh /var --max-depth=2 2>/dev/null | sort -h | tail -15
ncdu -x / # interactive, if installed-x keeps du on one filesystem so it doesn’t wander into other mounts. The usual suspects, and the safe way to clear each:
| Where | Safe cleanup |
|---|---|
systemd journal (/var/log/journal) | sudo journalctl --vacuum-size=200M, and set SystemMaxUse= in journald.conf |
Old log files (/var/log/*.gz, app logs) | Delete rotated files; fix logrotate. Truncate a live log with : > file.log rather than deleting it. |
Docker (/var/lib/docker) | docker system df to see, then docker system prune (add -a for unused images, --volumes only if you’re sure) |
| Package caches | sudo apt clean, npm cache clean --force, pip cache purge, old kernels with sudo apt autoremove |
| node_modules, build folders | Delete and reinstall when needed |
| Core dumps, old backups, database dumps | Move off the server or delete |
Deleted files that still take space
If df says full but du can’t find the space, a program is still holding a deleted file open (a classic after deleting a huge log). The space comes back only when the program closes it:
sudo lsof -nP +L1 | sort -k7 -n | tail # open files with zero links
sudo systemctl restart the-service # releases the spaceAlso check whether root’s reserved blocks matter: ext4 keeps about 5% for root, so ordinary users can hit ENOSPC a little before df reads 100%.
Out of inodes
Every file uses one inode, and a filesystem has a fixed number. Millions of tiny files (session files, mail queues, cache shards, a runaway node_modules) can use them all while gigabytes are free. Find the folder with the most files:
sudo find / -xdev -type f 2>/dev/null | cut -d/ -f2-3 | sort | uniq -c | sort -n | tailDelete what’s safe (old PHP sessions, cache folders), then stop whatever creates them. The inode count can’t be raised on an existing ext4 filesystem.
“System limit for number of file watchers reached”
Error: ENOSPC: System limit for number of file watchers reached, watch '/app/src'Dev servers and editors ask Linux to watch every file for changes, and the default limit can be small. Check and raise it:
cat /proc/sys/fs/inotify/max_user_watches
echo fs.inotify.max_user_watches=524288 | sudo tee /etc/sysctl.d/90-inotify.conf
sudo sysctl --systemAlso tell your tools to ignore node_modules, .git and build folders, which cuts the watches needed. Inside Docker the limit comes from the host, so raise it there.
Keep it from happening again
- Cap logs: journald
SystemMaxUse, logrotate withmaxsize, Docker’smax-sizelog option. - Prune Docker on a schedule; a cron line such as weekly
docker image prune -fhelps. - Alert at 80%, not 100%. Many apps (databases especially) misbehave before the disk is completely full.
Process simply printed Killed or exited with code 137? That’s memory, not disk: see exit code 137.
Docker images, build cache and logs filling the disk? See what docker system prune deletes.
Questions
What does ENOSPC mean?
ENOSPC is the system error 'no space left on device'. It appears when a disk or partition is full, when the filesystem has run out of inodes, or (with the words 'file watchers') when a program hits the Linux inotify watch limit.
Why does it say no space left on device when df shows free space?
Check df -i. If inode use is at 100%, there are too many files even though there are free bytes. Also check that the full filesystem is the one you are looking at, such as /tmp or /var, and whether a deleted file is still held open (lsof +L1).
How do I fix System limit for number of file watchers reached?
Raise fs.inotify.max_user_watches, for example to 524288 in /etc/sysctl.d/90-inotify.conf followed by sudo sysctl --system, and exclude node_modules and build folders from watching.
Is it safe to delete files in /var/log?
Old rotated logs (.gz, .1) are safe to delete. For a log a program is still writing, truncate it with : > file.log instead of deleting it, or the space is not freed until the program restarts. Use journalctl --vacuum-size for the systemd journal.
Published .