FTP Folder cleanup script (253 Views)
Reply
Advisor
gnana prakasam
Posts: 25
Registered: ‎01-16-2010
Message 1 of 2 (253 Views)

FTP Folder cleanup script

Hi all,

 

Pls go through and suggest me in the" #Get the individual recipient from the parsed file..." section 

I need the file name like this txt1.txt txt2.txt txt3.txt.

 

#!/bin/ksh -v
##################################################################################
# Log File manipulation functions
# To be sourced in main job script files
##################################################################################

##################################################################################
# FUNCTION keep_n_most_recent_days
##################################################################################
# Parameters
##################################################################################
# 1.Log File Partial Name (unique application descriptor)
# 2.How many days worth of log file should remain in place
##################################################################################
# Check the application related log files and delete all logs more than n days old
##################################################################################
function keep_n_most_recent_days {

   # Identify log files to be dropped in specified margin and execute deletion...
   find $JOBLOG -name $1\* -atime $2 | xargs rm -f 2>/dev/null

   # Delete any outstanding files out of specificed margin (after the weekend)...
   find $JOBLOG -name $1\* -mtime +$2 | xargs rm -f 2>/dev/null

   return

}
##################################################################################
# FUNCTION keep_n_most_recent_only
##################################################################################
# Parameters
##################################################################################
# 1.Log File Partial Name (unique application descriptor)
# 2.How many most recent log files should remain in place
##################################################################################
# Check the application related log files and delete all but n most recent
##################################################################################
function keep_n_most_recent_only {

   # How many application related log files we have?
   let count=`ls $JOBLOG/$1* | wc -l`

   # If we are to keep more that the existing ones...
   if ((count <= $2)); then

      # ...all existing log files should be kept in that case.
      echo 'not needed'> /dev/null 2>&1

   else

      # How many do we need to delete?
      let files_to_delete=count-$2

      let i=1


      while ((i <= files_to_delete)); do

         # loop through and pick the older log files
         current_file=`ls -tr $JOBLOG/$1* | head -n${files_to_delete} | head -n$i | tail -n1`

         # save the log file names into an array
         command_array[(i-1)]=${current_file}

         let i=i+1

      done

      # Array should have ${#command_array[*]} members

      # Now that we have the old log file name
      # we can go ahead and delete them
      for buffer in ${command_array[*]}
      do
         rm $buffer> /dev/null 2>&1
      done

   fi

   return

}
##################################################################################
# FUNCTION keep_n_most_recent_only_flex
##################################################################################
# Parameters
##################################################################################
# 1.Log File Partial Name (unique application descriptor)
# 2.How many most recent log files should remain in place
# 3.What directory we should look at (improvement over keep_n_most_recent_only)
##################################################################################
# Check the application related log files and delete all but n most recent
##################################################################################
function keep_n_most_recent_only_flex {

   # How many application related log files we have?
   let count=`ls $3/$1* | wc -l`

   # If we are to keep more that the existing ones...
   if ((count <= $2)); then

      # ...all existing log files should be kept in that case.
      echo 'not needed'> /dev/null 2>&1

   else

      # How many do we need to delete?
      let files_to_delete=count-$2

      let i=1


      while ((i <= files_to_delete)); do

         # loop through and pick the older log files
         current_file=`ls -tr $3/$1* | head -n${files_to_delete} | head -n$i | tail -n1`

         # save the log file names into an array
         command_array[(i-1)]=${current_file}

         let i=i+1

      done

      # Array should have ${#command_array[*]} members

      # Now that we have the old log file name
      # we can go ahead and delete them
      for buffer in ${command_array[*]}
      do
         rm $buffer> /dev/null 2>&1
      done

   fi

   return

}
##################################################################################
##################################################################################
# FUNCTION TO DELETE FILES IN FTPOUT FOLDER
##################################################################################
# Parameters
##################################################################################
# 1.FTP File Partial Name (unique application descriptor)
# 2.How many days worth of FTP file should remain in place
# 3.FTP Server Name
# 4.FTP User Name
# 5.FTP Password
# 6.FTP LOG File
##################################################################################
# Check the application related FTP files and delete all FTP more than n days old
##################################################################################
function keep_FTP_n_most_recent_days {

   # Identify ftp files to be dropped in specified margin and execute deletion...
   find $FTPOUT -name $1\* -atime $2 |awk -F/ '{print $NF}' > $FTPOUT/FTP_Delete.txt
   find $FTPOUT -name $1\* -atime $2 | xargs rm -f 2>/dev/null

   # Delete any outstanding files out of specificed margin (after the weekend)...
   find $FTPOUT -name $1\* -mtime +$2 |awk -F/ '{print $NF}' >> $FTPOUT/FTP_Delete.txt
   find $FTPOUT -name $1\* -mtime +$2 | xargs rm -f 2>/dev/null

   #Get the line number, parse out empty spaces and empty lines...

   let count=`cat $FTPOUT/FTP_Delete.txt | tr -s "\012" | tr -d " " | wc -l`

   #Reset counter
   let i=1

   #Loop in...
   while ((i <= count)); do

        #Get the individual recipient from the parsed file...
        filename=`cat $FTPOUT/FTP_Delete.txt | tr -s "\012" | tr -d " " | head -n$i | tail -n1`
        #Time to call the delete ftp file function...
        delete_FTP_Files $3 $4 $5 $filename $6

        let i=i+1

   done

   return

}
#################################################################################
##################################################################################
##################################################################################
# FUNCTION TO DELETE FILES IN FTP FOLDER
##################################################################################
# Parameters
##################################################################################
# 1.FTP Server Name
# 2.FTP User Name
# 3.FTP Password
# 4.FTP File Name
# 5.FTP LOG File
##################################################################################
# Delete the FTP files from the destination
##################################################################################
function delete_FTP_Files {

FTPServer=$1
FTPUserID=$2
FTPUserPswd=$3

echo $4
echo $5
#FTP Command Section
ftp -v -n<<EOF>$5
   open $FTPServer
   user $FTPUserID $FTPUserPswd
   delete $4
bye
EOF


}
#################################################################################


Please use plain text.
Acclaimed Contributor
Dennis Handly
Posts: 24,779
Registered: ‎03-06-2006
Message 2 of 2 (249 Views)

Re: FTP Folder cleanup script

>I need the file name like this txt1.txt txt2.txt txt3.txt.

 

What do you mean?  All on one line?  Or only with the string "txt"?

 

Some optimizations you can make:

>find $JOBLOG -name $1\* -atime $2 | xargs rm -f 2>/dev/null
>find $JOBLOG -name $1\* -mtime +$2 | xargs rm -f 2>/dev/null

 

Combine the two:

find $JOBLOG -name $1\* \( -atime $2 -o  -mtime +$2 \) -exec rm -f {} + 2>/dev/null

 

Instead of using let, just use (( )):

(( files_to_delete=count-$2 ))

 

Instead of using this pattern, use while read:

>      while ((i <= files_to_delete)); do
>          current_file=`ls -tr $JOBLOG/$1* | head -n${files_to_delete} | head -n$i | tail -n1`

 

ls -tr $JOBLOG/$1* | head -n${files_to_delete} | while read file; do

 

I don't think you need a for-loop here:

>for buffer in ${command_array[*]}

 

rm -f  "${command_array[@]}" 2> /dev/null

 

You probably want to combine these so you don't search 4 times:

   find $FTPOUT -name $1\* -atime $2 |awk -F/ '{print $NF}' > $FTPOUT/FTP_Delete.txt
   find $FTPOUT -name $1\* -atime $2 | xargs rm -f 2>/dev/null
   find $FTPOUT -name $1\* -mtime +$2 |awk -F/ '{print $NF}' >> $FTPOUT/FTP_Delete.txt
   find $FTPOUT -name $1\* -mtime +$2 | xargs rm -f 2>/dev/null

 

I.e. Do one find and put it into a file.  Use that with xargs to remove.  Then pass through awk.

Please use plain text.
The opinions expressed above are the personal opinions of the authors, not of HP. By using this site, you accept the Terms of Use and Rules of Participation