Re: replacing ls from part of script (225 Views)
Reply
Frequent Advisor
Matthew Pegge_1
Posts: 90
Registered: ‎11-21-2002
Message 1 of 9 (225 Views)
Accepted Solution

replacing ls from part of script

We have a backup script that tars up some files to a filesystem which then gets backed up to tape daily.
The piece of script is :

cd $PROPATHFILES
rc=0
if [ "`ls MB* X* OC* JO* CV* 2>/dev/null | grep -E -v '^XX|^XM|^XOX|^[0-9]'`" ]
then

#
# Create a file list.
# Done this way to ignore XX and XM files but include X* files
#

ls MB* X* OC* JO* CV* 2>/dev/null | grep -E -v '^XX|^XM|^XOX|^[0-9]' > $TMP/
CISAM_check_file_list
echo "Backing up CISAM files"
tar cvf - `cat $TMP/CISAM_check_file_list` | gzip -3 -c > $BACKUP_DIR/backup
/CISAM_backup.tar.gz
rc=$?
[ $rc -ne 0 ] && echo "Error $rc from 'gzip' - $TMP/CISAM_check_file_list"

else
echo "No CISAM files to backup in $PROPATHFILES"
fi

The problem being that the number of files the ls command finds is now too big and we get the following error:

sh: /usr/bin/ls: The parameter list is too long.

Any suggestions on a solution?
Honored Contributor
Rodney Hills
Posts: 1,679
Registered: ‎09-09-1996
Message 2 of 9 (225 Views)

Re: replacing ls from part of script

Use another grep on the ls output to select the files first.

example-
cd $PROPPATHFILES
rc=0
ls | grep -e "^MB" -e "^X" -e "^OC" -e "^JO" -e "^CV" >/tmp/holdout2 | grep -E -v '^XX|^XM|^XOX|^[0-9]' >$TMP/CISAM_check_file_list
if [ -s $TMP/CISAM_check_file_list ] ; then
echo "Backing up CISAM files"
tar ...

HTH

-- Rod Hills
There be dragons...
Advisor
T Dockery
Posts: 27
Registered: ‎11-16-1997
Message 3 of 9 (225 Views)

Re: replacing ls from part of script

try "echo" instead of ls. As in:
echo MB* X* OC* JO* CV* | grep ...
Honored Contributor
Rodney Hills
Posts: 1,679
Registered: ‎09-09-1996
Message 4 of 9 (225 Views)

Re: replacing ls from part of script

Side note-
The "tar" command you are using may also get the error "parameter list is too long" if their are too many files.

You may want to look at "pax" or even "cpio", since both of them can take a list of files names for back up.

example-
cat $TMP/CISAM_check_file | pax -w -f - | gzip -3 -c >$BACKUP_DIR/backup/CISAM_backup.pax.gz

My 2 cents...

-- Rod Hills
There be dragons...
Honored Contributor
Rodney Hills
Posts: 1,679
Registered: ‎09-09-1996
Message 5 of 9 (225 Views)

Re: replacing ls from part of script

On my first reply, I accidentely left a ">/tmp/holdout" on the line, which should not be there.

Correct line-
ls | grep -e "^MB" -e "^X" -e "^OC" -e "^JO" -e "^CV" | grep -E -v '^XX|^XM|^XOX|^[0-9]' >$TMP/CISAM_check_file_list

-- Rod Hills
There be dragons...
Honored Contributor
Kent Ostby
Posts: 1,836
Registered: ‎09-26-2000
Message 6 of 9 (225 Views)

Re: replacing ls from part of script

The other way to go would be to split out your ls into multiple lines.

ls MB* > /tmp/filelist
ls OC* >> /tmp/filelist
ls JO* >> /tmp/filelist
ls CV* >> /tmp/filelist
ls X* | grep -E -v '^XX|^XM|^XOX|^[0-9] >> /tmp/filelist

Then do your checks on that file.

Its not as "clean" but it is straightforward.

Best regards,

Kent M. Ostby
"Well, actually, she is a rocket scientist" -- Steve Martin in "Roxanne"
Frequent Advisor
TOMAS BERNABEU
Posts: 72
Registered: ‎01-19-1998
Message 7 of 9 (225 Views)

Re: replacing ls from part of script


Hi !
use find command

example :
# clear backup file :
>$BACKUP_DIR/backup/CISAM_backup.tar.gz
# find my pattern and tar :
for in MB OC JO CV pattern pattern ....
do
find $PROPATHFILES -name "$i*" -exec tar -xvf $BACKUP_DIR/backup/CISAM_backup.tar {} \ ;
-print 1>/tmp/my_list_of_files 2>/tmp/my_errors

done
Acclaimed Contributor
James R. Ferguson
Posts: 21,184
Registered: ‎07-06-2000
Message 8 of 9 (225 Views)

Re: replacing ls from part of script

Hi Matthew:

In lieu of using 'ls' to collect the files meeting your criteria, you could use 'find':

# cd yourdir
# find . -path "./*" -prune -name "MB* X* OC* JO* CV*" -type f > $TMP/CISAM_check_file_list

This 'find' will not descend subdirectories so it will mimic a simple 'ls'.

Regards!

...JRF...
Frequent Advisor
TOMAS BERNABEU
Posts: 72
Registered: ‎01-19-1998
Message 9 of 9 (225 Views)

Re: replacing ls from part of script

Hi !

The tar with -r

example :
# clear backup file :
>$BACKUP_DIR/backup/CISAM_backup.tar.gz
# find my pattern and tar :
for in MB OC JO CV pattern pattern ....
do
find $PROPATHFILES -name "$i*" -exec tar -rvf $BACKUP_DIR/backup/CISAM_backup.tar {} \ ;
-print 1>/tmp/my_list_of_files 2>/tmp/my_errors

done
The opinions expressed above are the personal opinions of the authors, not of HP. By using this site, you accept the Terms of Use and Rules of Participation.