Re: unable to find what is filling up /tmp (870 Views)
Reply
Super Advisor
NDO
Posts: 536
Registered: ‎12-10-2003
Message 1 of 17 (962 Views)
Accepted Solution

unable to find what is filling up /tmp

Hi all!

 

It seems rather easy problem to solve, but I am struglling to find out what is wrong, please can you help:

I have a server running 11.23 in which:

 #bdf /tmp
Filesystem          kbytes    used   avail %used Mounted on
/dev/vg00/lvol8    1048576 1012592   35984   97% /tmp

 But when I search for big files in it:

find /tmp -xdev -size +10000c -exec ls -lrt {} \;
-rwx------   1 root       sys          11914 Feb 17 11:09 /tmp/fr/vconf11
-r-xr-xr-x   1 root       sys          98152 Jun 29  2011 /tmp/fr/evainfo_hpux11_23_IA
-rw-------   1 root       sys          35959 Jun 20  2011 /tmp/mana06178
mceldev[721]/ #

#du -kx /tmp | sort -rn | head -30
496     /tmp
272     /tmp/fr
8       /tmp/backup
8       /tmp/PrinterSoftwt
8       /tmp/.oracle
0       /tmp/lost+found
0       /tmp/ign_configure
0       /tmp/.AgentSockets
mceldev[722]/ #

 and :

 

/tmp #du -sk *
0       67MQLuQXL5
0       ACL.log
8       AUTO.err
8       Core-sd.log
8       PrinterSoftwt
8       backup
8       cc-sd.log
8       da-sd.log
0       dhcpfifo.any
0       dhcpfifo.root
8       docs-sd.log
272     fr
0       ign_configure
0       ipv6agt.crashlog
8       javagui-sd.log
0       llbdbase.dat
0       lost+found
8       ma-sd.log
40      mana06178
8       media_info
8       media_info.tmp
8       mover_files
8       portmap.file
8       rpcbind.file
8       sensor_info
16      set_8265-151211_145156_basenv.ksh
8       system
8       system.prev
8       tmp_cur_shel_8265-151211_145156.env
0       tomala_swap.txt
0       wd_action.dbg

 Pls can you help

 

FR

Acclaimed Contributor
Dennis Handly
Posts: 25,094
Registered: ‎03-06-2006
Message 2 of 17 (956 Views)

Re: unable to find what is filling up /tmp

>It seems rather easy problem to solve

 

It appears someone created a large file then it was removed while still opened.

You need to use lsof to find it and the process that has it opened.

Frequent Advisor
Chirag Parikh
Posts: 51
Registered: ‎11-06-2007
Message 3 of 17 (950 Views)

Re: unable to find what is filling up /tmp

Run "fuser -cu /tmp" to find the process id which is using /tmp file system and kill that processor ID. then check bdf for /tmp, it should be normal and actual usage.

 

Regards,

Chirag Parikh

Honored Contributor
Patrick Wallek
Posts: 13,774
Registered: ‎06-21-2000
Message 4 of 17 (934 Views)

Re: unable to find what is filling up /tmp

To find unlinked files that are still occupying space use:

 

# lsof +L1

 

 

Super Advisor
NDO
Posts: 536
Registered: ‎12-10-2003
Message 5 of 17 (932 Views)

Re: unable to find what is filling up /tmp

I dont have at moment lsof, I am downloading it, but when I do fuser -cu /tmp, I got the following:

 

/tmp #fuser -cu /tmp
/tmp:     1387o(root)    1366o(root)    1377o(root)    1389o(root)    1416o(root)   22320o(root)   16842o(root)   10474o(root)   11626o(root)   12057o(root)   10340o(root)   14448o(root)   12535o(root)   13737o(root)   10206o(root)   10660o(root)   10956o(root)   11264o(root)   13130o(root)   15175o(root)   15954o(root)   18712o(root)   19735o(root)   17746o(root)   20809o(root)   24511o(root)   21992o(root)   23192o(root)   27204o(root)   25832o(root)    7028o(root)   28684o(root)     284o(root)    3570o(root)    1907o(root)    5252o(root)   14631o(root)   16717o(root)    8829o(root)    2935o(root)   12657o(root)   10728o(root)   18815o(root)   20964o(root)   27811o(root)   23172o(root)   25436o(root)   13414o(root)     306o(root)   16176o(root)    5469o(root)    8033o(root)   10710o(root)   27819o(root)   18991o(root)   21854o(root)    7392o(root)    4215o(root)   24827o(root)   13967o(root)    1022o(root)    5384o(root)   17315o(root)   10668o(root)   24187o(root)   20725o(root)   27753o(root)   12872o(root)   20631o(root)    1582o(root)    9094o(root)   16745o(root)   12522o(root)   24578o(root)    2844o(root)   28584o(root)   11235o(root)    3339o(root)   24167o(root)    7037o(root)   19806o(root)   15491o(root)    7908o(root)   28637o(root)   17194o(root)   21960o(root)   11786o(root)   26747o(root)    6813o(root)   27094o(root)   21955o(root)    1844o(root)   16867o(root)    2514o(root)   28261o(root)    7797o(root)   29537o(root)    5334o(root)   13183o(root)   18575o(root)   24026o(root)   10314o(root)   16703o(root)   11008o(root)   22434o(root)   22458o(root)   20895o(root)    4336o(root)   22290o(root)   22303o(root)   22427o(root)   22350o(root)   22413o(root)   22581o(root)   22193o(root)   22406o(root)   22686o(root)   22450o(root)   22614o(root)   22849o(root)   22499o(root)   22590o(root)   22719o(root)   22599o(root)   22940o(root)   22669o(root)   22696o(root)   22881o(root)   22703o(root)   22984o(root)   22858o(root)   22771o(root)   22866o(root)   23408o(root)   23157o(root)   22947o(root)   22959o(root)   22966o(root)   23386o(root)   23120o(root)   23038o(root)   23220o(root)   23244o(root)   23140o(root)   23213o(root)   23129o(root)   23233o(root)   23258o(root)   23682o(root)   23498o(root)   23308o(root)   23395o(root)   23512o(root)   23421o(root)   23669o(root)   23477o(root)   23523o(root)   24025o(root)   23539o(root)   23589o(root)   23695o(root)   23775o(root)   23760o(root)   24107o(root)   23708o(root)   23785o(root)   24035o(root)   23792o(root)   23807o(root)   23927o(root)   24051o(root)   24016o(root)   24121o(root)   24326o(root)   24138o(root)   24289o(root)   24131o(root)   24152o(root)   24437o(root)   24607o(root)   24208o(root)   24593o(root)   24300o(root)   24309o(root)   24412o(root)   24380o(root)   24397o(root)   24419o(root)   24693o(root)   24568o(root)   24580o(root)   24485o(root)   24859o(root)   24670o(root)   24660o(root)   24682o(root)   24707o(root)   24844o(root)   24757o(root)   24869o(root)   24937o(root)   24885o(root)   24961o(root)   24947o(root)   24976o(root)    6332c(root)    8286c(root)

 its a lot of stuff

 

 

FR

Honored Contributor
Patrick Wallek
Posts: 13,774
Registered: ‎06-21-2000
Message 6 of 17 (928 Views)

Re: unable to find what is filling up /tmp

Unless you want to check all of those processes, I recommend using LSOF.  I am not surprised that you  have lots of processes accessing /tmp. 

Super Advisor
NDO
Posts: 536
Registered: ‎12-10-2003
Message 7 of 17 (918 Views)

Re: unable to find what is filling up /tmp

Hi

 

 

I have downloaded and install lsof, but its a lot of stuff : 194 entries:

 

/usr/local/bin #./lsof +aL1 /tmp | more
COMMAND   PID USER   FD   TYPE DEVICE  SIZE/OFF NLINK NODE NAME
sh        306 root    1u   REG 64,0x8 345698744     0  301 /tmp (/dev/vg00/lvol8)
sh       2844 root    1u   REG 64,0x8 345698744     0  301 /tmp (/dev/vg00/lvol8)
sh       2935 root    1u   REG 64,0x8 345698744     0  301 /tmp (/dev/vg00/lvol8)
sh       3339 root    1u   REG 64,0x8 345698744     0  301 /tmp (/dev/vg00/lvol8)
sh       3570 root    1u   REG 64,0x8 345698658     0  301 /tmp (/dev/vg00/lvol8)
sh       4215 root    1u   REG 64,0x8 345698744     0  301 /tmp (/dev/vg00/lvol8)
sh       4336 root    1u   REG 64,0x8 345698830     0  301 /tmp (/dev/vg00/lvol8)
sh       5252 root    1u   REG 64,0x8 345698658     0  301 /tmp (/dev/vg00/lvol8)
sh       5334 root    1u   REG 64,0x8 345698830     0  301 /tmp (/dev/vg00/lvol8)
sh       5384 root    1u   REG 64,0x8 345698744     0  301 /tmp (/dev/vg00/lvol8)
sh       5469 root    1u   REG 64,0x8 345698744     0  301 /tmp (/dev/vg00/lvol8)
sh       6813 root    1u   REG 64,0x8 345698830     0  301 /tmp (/dev/vg00/lvol8)
sh       7028 root    1u   REG 64,0x8 345698658     0  301 /tmp (/dev/vg00/lvol8)
sh       7037 root    1u   REG 64,0x8 345698744     0  301 /tmp (/dev/vg00/lvol8)
sh       7392 root    1u   REG 64,0x8 345698744     0  301 /tmp (/dev/vg00/lvol8)
sh       7797 root    1u   REG 64,0x8 345698830     0  301 /tmp (/dev/vg00/lvol8)
sh       7908 root    1u   REG 64,0x8 345698830     0  301 /tmp (/dev/vg00/lvol8)
sh       8033 root    1u   REG 64,0x8 345698744     0  301 /tmp (/dev/vg00/lvol8)
sh       8829 root    1u   REG 64,0x8 345698744     0  301 /tmp (/dev/vg00/lvol8)
sh       9094 root    1u   REG 64,0x8 345698744     0  301 /tmp (/dev/vg00/lvol8)
sh      10206 root    1u   REG 64,0x8 345698658     0  301 /tmp (/dev/vg00/lvol8)
sh      10314 root    1u   REG 64,0x8 345698830     0  301 /tmp (/dev/vg00/lvol8)
sh      10340 root    1u   REG 64,0x8 345698658     0  301 /tmp (/dev/vg00/lvol8)
sh      10474 root    1u   REG 64,0x8 345698572     0  301 /tmp (/dev/vg00/lvol8)
sh      10660 root    1u   REG 64,0x8 345698658     0  301 /tmp (/dev/vg00/lvol8)
sh      10668 root    1u   REG 64,0x8 345698744     0  301 /tmp (/dev/vg00/lvol8)
sh      10710 root    1u   REG 64,0x8 345698744     0  301 /tmp (/dev/vg00/lvol8)
sh      10728 root    1u   REG 64,0x8 345698744     0  301 /tmp (/dev/vg00/lvol8)
sh      10956 root    1u   REG 64,0x8 345698658     0  301 /tmp (/dev/vg00/lvol8)
sh      11008 root    1u   REG 64,0x8 345698830     0  301 /tmp (/dev/vg00/lvol8)
sh      11235 root    1u   REG 64,0x8 345698744     0  301 /tmp (/dev/vg00/lvol8)
sh      11264 root    1u   REG 64,0x8 345698658     0  301 /tmp (/dev/vg00/lvol8)
sh      11626 root    1u   REG 64,0x8 345698658     0  301 /tmp (/dev/vg00/lvol8)
sh      11786 root    1u   REG 64,0x8 345698830     0  301 /tmp (/dev/vg00/lvol8)
sh      12057 root    1u   REG 64,0x8 345698658     0  301 /tmp (/dev/vg00/lvol8)

 I did this:

 

#./lsof +aL1 /tmp | wc -l
194

 and if I do ps -ef | grep pid, it returns an entry that was in the crontab, but its no longer there

 

FR

Acclaimed Contributor
Dennis Handly
Posts: 25,094
Registered: ‎03-06-2006
Message 8 of 17 (916 Views)

Re: unable to find what is filling up /tmp

>but it's a lot of stuff:

>sh 306 root 1u REG 64,0x8 345698744 0 301 /tmp (/dev/vg00/lvol8)

 

This appears to be the same large file each time, used by the shell.

 

>and if I do ps -ef | grep pid, it returns an entry that was in the crontab, but its no longer there

 

It doesn't matter if it's no longer in crontab, if it is still running.

 

What does this show:

ps -f -p 306 -p 2844 -p 2935 -p 11786 -p 12057

Super Advisor
NDO
Posts: 536
Registered: ‎12-10-2003
Message 9 of 17 (910 Views)

Re: unable to find what is filling up /tmp

its the same stuff:

 

ps -f -p 306 -p 2844 -p 2935 -p 11786 -p 12057
     UID   PID  PPID  C    STIME TTY       TIME COMMAND
    root 12057     1  0  Apr 13  ?        78:22 sh -c /usr/sbin/ping 192.168.0.85 | while read pong; do echo "$(date): $pong"; done > /tmp/fr/ping_date.txt
    root  2935     1  0  Apr 13  ?        77:46 sh -c /usr/sbin/ping 192.168.0.85 | while read pong; do echo "$(date): $pong"; done > /tmp/fr/ping_date.txt
    root   306     1  0  Apr 13  ?        78:46 sh -c /usr/sbin/ping 192.168.0.85 | while read pong; do echo "$(date): $pong"; done > /tmp/fr/ping_date.txt
    root 11786     1  0  Apr 13  ?        76:41 sh -c /usr/sbin/ping 192.168.0.85 | while read pong; do echo "$(date): $pong"; done > /tmp/fr/ping_date.txt
mceldev[800]/usr/local/bin #

 

Honored Contributor
Patrick Wallek
Posts: 13,774
Registered: ‎06-21-2000
Message 10 of 17 (908 Views)

Re: unable to find what is filling up /tmp

Does this file exist: /tmp/fr/ping_date.txt

 

If not, that is probably your culprit.  Do you really need all of those processes running that are pinging the same IP address?  They have been running since April 13.  I would look and see if you can kill those processes.

Acclaimed Contributor
Dennis Handly
Posts: 25,094
Registered: ‎03-06-2006
Message 11 of 17 (896 Views)

Re: unable to find what is filling up /tmp

>it's the same stuff:

 

So it's not very useful, kill them all!  You don't really need 194 processes pinging the same IP.  ;-)

Also, you really should be sending this info to a pipe so you can do data reduction on the fly.  Or only send errors when ping fails.

 

>Does this file exist: /tmp/fr/ping_date.txt?

 

That's the missing file, since it's so large.

 

>I would look and see if you can kill those processes.

 

I would just wave bye-bye to the unmutual script and kill them all.

You could check to see if all of them are the same IP and output file.

Super Advisor
NDO
Posts: 536
Registered: ‎12-10-2003
Message 12 of 17 (891 Views)

Re: unable to find what is filling up /tmp

Hi

 

The /tmp/fr/ping_date.txt file does not exist, and I am not quite sure how can kill 194 processes at the same time, please can you give a hint?

 

FR

Honored Contributor
Patrick Wallek
Posts: 13,774
Registered: ‎06-21-2000
Message 13 of 17 (887 Views)

Re: unable to find what is filling up /tmp

You can try to script the process. Otherwise you will just have to go through the list and kill the appropriate processes.

Super Advisor
NDO
Posts: 536
Registered: ‎12-10-2003
Message 14 of 17 (884 Views)

Re: unable to find what is filling up /tmp

 

 

does this:

 

 

ps -ef |awk '/pong/{print $2}' | xargs kill -15

 

 

will do the job?

Valued Contributor
RAJD1
Posts: 60
Registered: ‎10-09-2011
Message 15 of 17 (877 Views)

Re: unable to find what is filling up /tmp

Nandinho,

 

Try using lsof on /tmp with sorting SIZE column.

 

# lsof /tmp > lsof_tmp.out

# cat lsof_tmp.ou | sort -k 8 | more 

 

You will see the large size , the program using, try to check that program with the PID given. May be you need to bounce that program/application or to kill it .

 

Hth,

Raj D.

 

Acclaimed Contributor
Dennis Handly
Posts: 25,094
Registered: ‎03-06-2006
Message 16 of 17 (870 Views)

Re: unable to find what is filling up /tmp

[ Edited ]

>ps -ef | awk '/pong/{print $2}' | xargs kill -15;   will do the job?

 

Yes, unless you want to limit it to be safe:

ps -fu root | awk '/ping.*pong/ {print $2}' | xargs kill

 

>Try using lsof on /tmp with sorting SIZE column.

> cat lsof_tmp.out | sort -k 8

 

I count column 7.  Also no need to use an evil cat:

sort -k 7 lsof_tmp.out

Super Advisor
NDO
Posts: 536
Registered: ‎12-10-2003
Message 17 of 17 (852 Views)

Re: unable to find what is filling up /tmp

 

Hi

 

Thank you Dennis, I used your command, but with a pipe, (I think you missed it)

 

ps -fu root | awk '/ping.*pong/ {print $2}'| xargs kill

 

It worked fine, problem solved.

 

Thank you

 

FR

The opinions expressed above are the personal opinions of the authors, not of HP. By using this site, you accept the Terms of Use and Rules of Participation.