Automate FTP (98 Views)
Reply
Esteemed Contributor
Coolmar
Posts: 1,074
Registered: ‎12-21-2004
Message 1 of 28 (98 Views)
Accepted Solution

Automate FTP

Hi,

I need help with figuring out how to script the following. There is an FTP server that contains a file that I need to download. A new one is put there almost every day...I don't know when, it is random. If there is not a new file put there, then the old one remains until the new arrives (different name each time). I know how to automatically ftp, but want I want to do is:

1. only download the file if it is new (in other words, I don't have it downloaded already)

2. email myself as a notification that a new file has been obtained.

Thanks!
Please use plain text.
Honored Contributor
Rick Garland
Posts: 4,470
Registered: ‎06-25-1997
Message 2 of 28 (98 Views)

Re: Automate FTP

An option to automate ftp file transfers without using the $HOME/.netrc file...

ftp -n -v hostname > $LOG << ENDFTP
user username passwd
prompt off
binary
cd wherever
lcd wherever
mget *
quit


If you are wanting to do some comparisons for newer file, will need a script that contains logic to compare the date/time stamp of the file to be downloaded.

Please use plain text.
Esteemed Contributor
Coolmar
Posts: 1,074
Registered: ‎12-21-2004
Message 3 of 28 (98 Views)

Re: Automate FTP

Thanks Rick for the response. I already have the script and it works fine...it is the "logic" that I am looking for.

S.
Please use plain text.
Exalted Contributor
Steven E. Protter
Posts: 33,806
Registered: ‎08-15-2002
Message 4 of 28 (98 Views)

Re: Automate FTP

Hello,

The ftp client is not very good at knowing whether the file is newer or not.

You could however insert some code into the excellent response you have already received.

ls -la | awk '{print $5 $6 $7 $8} > /tmp/localfile

# now you can compare the time and date stamp and size to what you already have on your local system.

I'm not sure it will be 100% accurate.

A better methodology perhaps would be for the ftp server to run a command like this

find /ftpdir -mtime -exec 'ls -1 > /ftp/downloadlist'

or maybe

find /ftpdir -ctime -exec 'ls -1 > /ftp/downloadlist'

Then you download that list every day, feed it into your ftp scripot and get the new files.

A little help at ftp side is needed.

Or you can download and compare later, but it sounds like you are trying to conserver bandwitch.

Hope I've given you an idea on how to approach the problem.

As for the email part:

echo -e "To: $EMAIL_TO" > $fmesg
echo -e "From: $EMAIL_FROM" >> $fmesg
echo -e "Cc: prottertemp@investmenttool.com" >> $fmesg
echo -e "Subject: $EMAIL_SUBJECT" >> $fmesg
echo -e "" >> $fmesg
/bin/cat $fbody >> $fmesg
cat $fmesg | /usr/sbin/sendmail -t

> $fbody
rm -f $fbody
> $fmesg
rm -f $fmesg
> $ftext
rm -f $ftext

just set the variables and decide what the text is. fmesg is a temporary holding file for the message. Should be pretty obvious.

This is production code from a Linux box, but it is also running on HP boxes I used to admin in the US. Linux boxes are mine, I'm allowed to peak.

Good Luck,

SEP
Steven E Protter
Owner of ISN Corporation
http://isnamerica.com
http://hpuxconsulting.com
Sponsor: http://hpux.ws
Twitter: http://twitter.com/hpuxlinux
Founder http://newdatacloud.com
Please use plain text.
Honored Contributor
Dave La Mar
Posts: 829
Registered: ‎03-27-2001
Message 5 of 28 (98 Views)

Re: Automate FTP

Sally -
There are probably more eloquent ways to do this, but ...

From and ftp session:
1. Set the prompt off by issuing the ftp prompt command.
2. ls yourfile* dummy
The above will list yourfile with date/timestamp in the file dummy on your local machine.
3. Quit the ftp.
4. Compare your old file to that in the dummy file created in #2.
5. Based on the comparison, go on with your business. i.e. set off the email and second ftp job to pick up the file.

Best of luck.

Regards,

dl
"I'm not dumb. I just have a command of thoroughly useless information."
Please use plain text.
Acclaimed Contributor
James R. Ferguson
Posts: 21,184
Registered: ‎07-06-2000
Message 6 of 28 (98 Views)

Re: Automate FTP

Hi Sally:

See the man pages for 'ftp(1)'. You have a 'newer' variation of 'get':

# newer filename

...gets the file only if the modification time of the remote file is more recent that the file on the current system.

Regards!

...JRF...
Please use plain text.
Honored Contributor
Rick Garland
Posts: 4,470
Registered: ‎06-25-1997
Message 7 of 28 (98 Views)

Re: Automate FTP

Can use the 'ftp newer '

Get the file only if the modification time of the remote file is
more recent that the file on the current system. If the file
does not exist on the current system, the remote file is
considered newer. Otherwise, this command is identical to get.

As to sending mail, can have a job that periodically checks the date/time of the file. If newer, then send mail.


DATE=`ll | awk '{if (($7 == "9")) print $9}'`
if [ $DATE != date +%e` ]
then
mailx ....
fi

Hope that helps.
Please use plain text.
Honored Contributor
Rick Garland
Posts: 4,470
Registered: ‎06-25-1997
Message 8 of 28 (98 Views)

Re: Automate FTP

BTW, I use the $7 == "9" as an example. Today's date being 9
Please use plain text.
Acclaimed Contributor
A. Clay Stephenson
Posts: 17,825
Registered: ‎07-16-1998
Message 9 of 28 (98 Views)

Re: Automate FTP

Actually the FTP is extremely good at knowing whether a file is newer or not iff you use the Net::FTP Perl client but that should not be necessary if I can depend upon you being truthful about the filename changing everytime. All we have to do is list the remote directory and compare those names to the one found in the destination directory on the local machine. Any files not found should be retrieved and mail should be sent. I already have a Perl script which will do all of the hard work, we just need to add some shell logic.

ftp.sh
--------------------------------
#!/usr/bin/sh

PATH=${PATH}:/usr/local/bin
export PATH

typeset PROG=${0##*/}


typeset TDIR=${TMPDIR:-/var/tmp}
typeset PID=${?}
typeset T1=${TDIR}/X${PID}_1.txt

trap 'eval rm -f ${T1}' 0 1 2 15

typeset SRCDIR="/home/cstephen"
typeset REMHOST="bugs"
typeset DESTDIR="/tmp/cstephen"
typeset U="cstephen"
typeset P="-p secret"
typeset ADDRESS="mmouse@disney.com"


typeset -i STAT=0

# make sure destination directory exists"

if [[ ! -d "${DESTDIR}" ]]
then
echo "Destination ${DESTDIR} does not exist." >&2
exit 1
fi

cd "${DESTDIR}"
STAT=${?}
if [[ ${STAT} -ne 0 ]]
then
echo "Cannot cd to ${DESTDIR}; status ${STAT}." >&2
exit 2
fi

#Get a list of files in directory ${DIR} on remotehost {REMHOST}
#Login as user ${U} using password ${P}.
#If a .netrc file is located on the client then no passwd is needed.

ftpget.pl -h ${REMHOST} -l ${U} ${P} -L '' -d "${SRCDIR}" > ${T1}
STAT=${?}
if [[ ${STAT} -eq 0 ]] # dirlist okay
then
if [[ -s ${T1} ]]
then
typeset FNAME=""
cat ${T1} | while read FNAME
do
if [[ ! -f ${FNAME} ]]
then
ftpget.pl -h ${REMHOST} -l ${U} ${P} -d "${SRCDIR}" "${FNAME}"
STAT=${?}
if [[ ${STAT} -eq 0 ]]
then
echo "New File: ${FNAME}" | mailx -s "New File" ${ADDRESS}
else
echo "Failed to get ${FNAME}; status ${STAT}" >&2
exit ${STAT}
fi
fi
done
fi
else
echo "${PROG} dirlist failed; status ${STAT}" >&2
fi
exit ${STAT}

-------------------------------
That should do it if I haven't made any typo's. You can add a patterm to the ftpget.pl -L argument to match only certain patterns. e.g -L 'Data*.txt'. Using Perl is a much better choice because you get error checking for free -- something that is very tedious in the shell. Note that you can also setup a .netrc file on your end and you don't need to pass the password. Man .netrc for details. If you want to do this more secure there is also a Net::SFTP module. You can then make a cron wrapper script for this and you are done.

Here's the perl piece, ftpget.pl; the above script assumes that you install it in /usr/local/bin. I suggest that you invoke it as ftpget.pl -u for full usage because you may want to set binary or ASCII mode for the gets.






If it ain't broke, I can fix that.
Please use plain text.
Honored Contributor
Dave La Mar
Posts: 829
Registered: ‎03-27-2001
Message 10 of 28 (98 Views)

Re: Automate FTP

Sally -
James nailed it with newer.
James -
This is a great tip! Wasn't aware of it, but will definitely use it.

Thanks.

And Sally, please no points on this reply, just wanted to acknowledge anothers great contribution.

Regards,

dl
"I'm not dumb. I just have a command of thoroughly useless information."
Please use plain text.
Esteemed Contributor
Coolmar
Posts: 1,074
Registered: ‎12-21-2004
Message 11 of 28 (98 Views)

Re: Automate FTP

Thanks for all the responses....I think James hit it with "newer". I am going to try that.

S.
Please use plain text.
Esteemed Contributor
Coolmar
Posts: 1,074
Registered: ‎12-21-2004
Message 12 of 28 (98 Views)

Re: Automate FTP

hmmmmmm....I sent my previous reponse too soon. I am wondering, would the files have to have the same name with "newer"? The problem is that the name always changes...the one consistent thing is that it is always a tar file. However, I can't say "newer *.tar". What I want to accomplish is to check the ftp server throughout the day for the new tar file. If the tar file that exists is what I already have in my directory, then done. If it is a new file, that I don't have...download it.

Thanks,
Please use plain text.
Honored Contributor
Rick Garland
Posts: 4,470
Registered: ‎06-25-1997
Message 13 of 28 (98 Views)

Re: Automate FTP

Use mget (instead of get) and specify 'newer *.tar'

The mget accepts wildcards.

Please use plain text.
Esteemed Contributor
Coolmar
Posts: 1,074
Registered: ‎12-21-2004
Message 14 of 28 (98 Views)

Re: Automate FTP

I tried that, and this was what came back:

ftp> mget newer *.tar
newer: The system cannot find the file specified.
mget dat-4624.tar? y
200 PORT command successful.
150 Opening ASCII mode data connection for dat-4624.tar(7710720 bytes).


Then it starts "getting" the file. I must not be using the correct syntax or something. Shouldn't it determine that there is only one tar file there, so that would be the one....and it is not newer than what I have and therefore not download it?
Please use plain text.
Honored Contributor
Rick Garland
Posts: 4,470
Registered: ‎06-25-1997
Message 15 of 28 (98 Views)

Re: Automate FTP

What happens when you turn off the prompt?

prompt off
mget newer *.tar
...
Please use plain text.
Esteemed Contributor
Coolmar
Posts: 1,074
Registered: ‎12-21-2004
Message 16 of 28 (98 Views)

Re: Automate FTP

It starts getting the file. However, I already have that one, so I don't want it to get it.

prompt off
Interactive mode off.
ftp> mget newer *.tar
newer: The system cannot find the file specified.
local: dat-4624.tar remote: dat-4624.tar
200 PORT command successful.
150 Opening ASCII mode data connection for dat-4624.tar(7710720 bytes).
Please use plain text.
Esteemed Contributor
Coolmar
Posts: 1,074
Registered: ‎12-21-2004
Message 17 of 28 (98 Views)

Re: Automate FTP

Is there an equivalent to using grep within an ftp session? Like, if I could list the files and grep on the tar files, that would leave that one...then awk everything but the name of the file to a variable...and mget newer $var ? or is that asking too much of poor ftp?
Please use plain text.
Acclaimed Contributor
A. Clay Stephenson
Posts: 17,825
Registered: ‎07-16-1998
Message 18 of 28 (98 Views)

Re: Automate FTP

The most important glaring problem with these other approaches is that there is zero error checking. If you use the method I suggested and check the exit status of the Perl script itself then if it's zero then all is well -- meaning the complete file got there intact. The good news (maybe) is that you really don't have to know any Perl at all. All of the logic is still in the shell script that calls the Perl script. You simply have to feed it arguments and check the exit status.
If it ain't broke, I can fix that.
Please use plain text.
Honored Contributor
Rick Garland
Posts: 4,470
Registered: ‎06-25-1997
Message 19 of 28 (98 Views)

Re: Automate FTP

If you use the -g option (noglobbing) instead of the newer, will that still download the file?



Please use plain text.
Esteemed Contributor
Coolmar
Posts: 1,074
Registered: ‎12-21-2004
Message 20 of 28 (98 Views)

Re: Automate FTP

You mean start the seesion with -g then mget *tar?

ftp -g
mget *.tar

BTW, what does "globbing mean"


Please use plain text.
Honored Contributor
Rick Garland
Posts: 4,470
Registered: ‎06-25-1997
Message 21 of 28 (98 Views)

Re: Automate FTP

My mistake, you do want globbing for the mget command.

You will want filename expansion for the mget sequence.

This is a tricky one. I would have thought the 'newer' would work. This is how the MAN quotes it.
Please use plain text.
Esteemed Contributor
Coolmar
Posts: 1,074
Registered: ‎12-21-2004
Message 22 of 28 (98 Views)

Re: Automate FTP

I think it would work if it would take wildcards, but it seems to want the exact filename. Newer, is where those error messages are coming from.
Please use plain text.
Acclaimed Contributor
James R. Ferguson
Posts: 21,184
Registered: ‎07-06-2000
Message 23 of 28 (98 Views)

Re: Automate FTP

Hi (again) Sally:

OK. It appears that 'mget' and 'newer' do not mix. 'newer' is a variation of the simple 'get'. Re-reading your original post, I see that you said, "...then the old one remains until the new arrives (different name each time)."

So, I assume that you don't know what the new file's name will be. I missed that the first time.

One workaround is to query the remote server for the file(s) you might find. You CAN use globbing to do this (e.g. 'ls /tmp/my*'). The file list so obtained must be written to a file on the local server. Then, have your script start a second 'ftp' session with the remote server to perform either 'get filename' or 'newer filename' to transfer the *exact* file of interest.

I have used this technique of runnning a query of a directory on a remote server; building an indexed list for a user; and then allowing the user to see and select files by that index in a second menu driving a second 'ftp' session.

BTW, "globbing" is the term used to describe the shell's expansion of filenames based on metacharacter notation (e.g. 'ls my*' would return all files beginning with "my").

Regards!

...JRF...
Please use plain text.
Esteemed Contributor
Coolmar
Posts: 1,074
Registered: ‎12-21-2004
Message 24 of 28 (98 Views)

Re: Automate FTP

It won't let me ftp -g as "anonymous". So I can't use the globbing or no globbing..whichever it was. I tried to do an ls *.tar >file and it won't do that either. How do you print a listing of a directory to a file on the local system?
Please use plain text.
Esteemed Contributor
Coolmar
Posts: 1,074
Registered: ‎12-21-2004
Message 25 of 28 (98 Views)

Re: Automate FTP

Got it working! I didn't realize that you don't need the ">" when listing the output to a file.

Anyway, the final part of my problem is how to send out an email if it actually downloads a file. No email is needed, of course, if it finds nothing, but if it does find something and downloads, I need an email.

Thanks!
Please use plain text.
The opinions expressed above are the personal opinions of the authors, not of HP. By using this site, you accept the Terms of Use and Rules of Participation