Sunday, May 24, 2009

fork: Not enough space.. swap space limit exceeded.... tmpfs

During normal business hour, IHAC got a message on his terminal window as:
bash: fork: Not enough space
Experienced that the system performance degraded, and later on became un-responsive. Executing simple command still responded, but take a long time to finish. From /var/adm/messages file, "df -k" command output, and /etc/vfstab file there is the clue:
root@sun1: # cat /var/adm/messages
(snip)
Feb 24 10:06:41 sun1 tmpfs: [ID 518458 kern.warning] WARNING: /tmp: File system full, swap space limit exceeded
Feb 24 10:06:46 sun1 last message repeated 264 times

(snip)
root@sun1: df -k
Filesystem kbytes used avail capacity Mounted on
/dev/vx/dsk/bootdg/rootvol 68392666 24266489 43442251 36% /
/proc 0 0 0 0% /proc
mnttab 0 0 0 0% /etc/mnttab
fd 0 0 0 0% /dev/fd
swap 69801872 184 69801688 1% /var/run
dmpfs 69801688 0 69801688 0% /dev/vx/dmp
dmpfs 69801688 0 69801688 0% /dev/vx/rdmp
swap 69813720 69801688 12032 100% /tmp

(snip)
root@sun1: # cat /etc/vfstab
(snip)
swap - /tmp tmpfs - yes -
(snip)

For some reason, the /tmp filesystem filled up. Since the system un-responsived the last resort was rebooting the server (with hard reset anyway)./tmp filesystem by default, is a memory based file system (tmpfs), and its mounted without any size limitation. Furthermore it is the possibility that every user can use the whole /tmp space since its set publicly writeable.This case happened could be because an errant application or someone dumping file into /tmp. To avoid this happens again, its suggested to modify the /tmp entry in /etc/vfstab to limit to adequate size (e.g 1024Mb).
root@sun1: cat /etc/vfstab
(snip)
swap - /tmp tmpfs - yes size=1024m
(snip)
Unfortunately, system reboot is required to activate the changes.


No comments:

Post a Comment