Finding files to trim (318 Views)
Reply
Super Advisor
TheJuiceman
Posts: 462
Registered: ‎02-20-2002
Message 1 of 16 (318 Views)
Accepted Solution

Finding files to trim

Hey gang,

/ on one of our systems is slowly growing. I have a two part question to ask the group....

1) Does anyone have a script that will look in / and find and list the largest files in / excluding all other mounted volumes? A script that would be portable or would require no editing as volumes are added/deleted would be preferred.

2) I was wondering if anyone had a script that would look for files to trim or delete on / or various other volumes (like /var or /opt). Is there a list of files or filetypes to look for that would qualify for occasional purging? This is for 11.11 systems.

Thanks for your help.
Acclaimed Contributor
James R. Ferguson
Posts: 21,184
Registered: ‎07-06-2000
Message 2 of 16 (318 Views)

Re: Finding files to trim

Hi:

The following script will list (and sum) the size of all directories except mounted ones:

# ls -il / | perl -nae 'if ($.>1 && $F[0]!=2 && $F[1]=~/^d/) {$v=`du -ks $F[9]`;print $v;@s=split(/\s+/,$v);$t+=$s[0]};END{print "$t\n"}'

This code relies on the fact that a directory that is a mountpoint will *always* have an inode=2. Remember, inode values are only unique within a filesystem.

Another way to list directory sizes is :

# du -xk /var|sort -k1nr|more

...This will restrict 'du' to reporting directory size in the order largest-to-smallest beneath a mountpoint.

The idea of both the above is to identify directories which have significant space.

As for looking for "large" files you could do something like:

# find /pathname -xdev -type f -size +10000c -exec ls -l {} \; | sort -k5nr

This would find all files in /pathname (without crossing mountpoints) whose size exceeds 10,000 characters. The output would be a stnadard 'ls' file listing sorted from largest to smallest by size.

As for trimming and/or deleting files in '/var/ and /'opt', the place to concentrate is '/var'. '/opt' should be static. See the manpages for 'hier(5)' for details.

That being said, files in:

/var/preserve --- are old 'vi' recovery files and if old can be deleted.

/var/adm/sw --- do *NOT* manage this manually. The *only* safe way to reclaim space is to use 'cleanup'. To do otherwise you run the risk of corrupting your Installed Product Database and you will not be able to patch your server again!

/var/adm/wtmp, /var/adm/btmp --- these are logs of successful and unsuccessful logins respectively. Trim them to zero by redirecting '/dev/null' to them or trim them with 'fwtmp' by converting them to ASCII to edit them and reconverting them to binary after they have been trimmed.

/var/adm/crash --- should only have unanalyzed crash dumps from high-priority machine checks. If you find something here that's old you can delete it unless you want HP Suppport to analyze it.

/var/adm/mail --- have your users cleanup their mail folders and this won't be a problem.

Regards!

...JRF...
Super Advisor
TheJuiceman
Posts: 462
Registered: ‎02-20-2002
Message 3 of 16 (318 Views)

Re: Finding files to trim

Hey James,

Wow...what a great response!!! That deserves 20 points!!!

I have another question along these lines...

I have a file that contains a list of directories. I want to get a sort these files by size so that the largest file is displayed first. Something like this...

find /{this is where I want to check the list of dirs in my file) -depth -type f -exec ll {} \; >> /tmp/file1

sort -r -n -k 5,5 /tmp/file1 > /tmp/file2

How can I write this so that it will go through my file and do the find on each of the entries in the file so I can then do the sort?

Thanks for the help.
Honored Contributor
RAC_1
Posts: 5,920
Registered: ‎03-21-2002
Message 4 of 16 (318 Views)

Re: Finding files to trim

There are some simple commands that will do the work. You need to be careful about trimming the files. you should understnad what are those files and what processes might be accessing them.

du -kx / | sort -n --> Will list dir sorted in ascending order

Once you note the dir that are using most of the space, run du on them again to see what is holding the space.
There is no substitute to HARDWORK
Super Advisor
TheJuiceman
Posts: 462
Registered: ‎02-20-2002
Message 5 of 16 (318 Views)

Re: Finding files to trim

Is there way to get a sort of just the files? Here is what I'm trying to do...

I have a file...FILE1 that contains the following...

entry1
entry2
entry3

I want to run a find on the entries in the file so I can collect and sort the output.

Also getting all the information that a ls -l gives would help tremendously.
Honored Contributor
Arunvijai_4
Posts: 4,194
Registered: ‎04-07-2005
Message 6 of 16 (318 Views)

Re: Finding files to trim

# du -skx /usr/* | sort -nr | head -10

(List the top 10 files)

# ls -lR | sort +4 -5nr | more

(List all files starting with biggest)

-Arun
"A ship in the harbor is safe, but that is not what ships are built for"
Super Advisor
TheJuiceman
Posts: 462
Registered: ‎02-20-2002
Message 7 of 16 (318 Views)

Re: Finding files to trim

I REALLY REALLY need this to do the find on the entries in the file. How can I do that? Thanks.
Honored Contributor
RAC_1
Posts: 5,920
Registered: ‎03-21-2002
Message 8 of 16 (318 Views)

Re: Finding files to trim

for i in $(do
find ${i} -exec ll -d {} \; | sort -nrk5
done
There is no substitute to HARDWORK
Honored Contributor
Devender Khatana
Posts: 2,278
Registered: ‎03-01-2004
Message 9 of 16 (318 Views)

Re: Finding files to trim

Hi,

#du -xk `cat /home/hp/file1` |sort -rnk1 |pg

Where /home/hp/file1 is the file and you are executing the command at any location. The contents of the file are with absolute path like

/var
/opt
/home

HTH,
Devender
Impossible itself mentions "I m possible"
Acclaimed Contributor
James R. Ferguson
Posts: 21,184
Registered: ‎07-06-2000
Message 10 of 16 (318 Views)

Re: Finding files to trim

Hi (again):

Perhaps this is what you are seeking (based upon your last post, above):

cat filelist.sh
#!/usr/bin/sh
while read LINE
do
find ${LINE} -xdev -type f | xargs ls -l
done < filelist | \
sort -k5nr
exit 0

...The 'filelist' is a simple file with directories of interest listed one per line. For example:

# cat filelist
/etc
/sbin
/stand

Note that you don't have to use any temporary files (only pipes) and that I have avoided the more expensive '-exec' with 'find' which would spawn a separate process for every 'ls' request. The 'xargs' bundles multiple requests and greatly reduces subprocess initiation. The 'find' traversing directories is brutal enough on any system!

Regards!

...JRF...
Regards!

...JRF...
Honored Contributor
Devender Khatana
Posts: 2,278
Registered: ‎03-01-2004
Message 11 of 16 (318 Views)

Re: Finding files to trim

Hi,

Apart from this I always use the SAM to trim the log files. This I think is the best way to this, as it also shows you the existing size of all files in one windows itself.

HTH,
Devender
Impossible itself mentions "I m possible"
Honored Contributor
Yogeeraj_1
Posts: 4,613
Registered: ‎11-25-2001
Message 12 of 16 (318 Views)

Re: Finding files to trim

hi,

if you go in SAM, you will get a few system logfiles that you can trim using the same interface. However, the list exclude maybe larger log files that could be generated by application programs.

beware that there are some files that you cannot just delete! You must document yourself fully before doing so. E.g. Oracle Redolog files cannot be just deleted!!

hope this helps too!

kind regards
yogeeraj
No person was ever honoured for what he received. Honour has been the reward for what he gave (clavin coolidge)
Honored Contributor
Ted Buis
Posts: 1,199
Registered: ‎12-05-2001
Message 13 of 16 (318 Views)

Re: Finding files to trim

It is very easy to add files to the list in SAM that can be monitored and trimmed. HP can know which application files you have that may need to be monitored.
Mom 6
Super Advisor
TheJuiceman
Posts: 462
Registered: ‎02-20-2002
Message 14 of 16 (318 Views)

Re: Finding files to trim

Thank you all for the help!!! I was able to get what I needed done!!!
Esteemed Contributor
Arturo Galbiati
Posts: 830
Registered: ‎02-10-2003
Message 15 of 16 (318 Views)

Re: Finding files to trim

Hi,
cat your_file|xargs -i find . -name {} -exec ll {} \;

this will find the files contained into your_file.

HTH,
ART
Super Advisor
TheJuiceman
Posts: 462
Registered: ‎02-20-2002
Message 16 of 16 (318 Views)

Re: Finding files to trim

thanks
The opinions expressed above are the personal opinions of the authors, not of HP. By using this site, you accept the Terms of Use and Rules of Participation.