Re: ftp.proxy usage for downloading dependencies of vim depot (2269 Views)
Reply
Valued Contributor
chindi
Posts: 391
Registered: ‎07-24-2008
Message 1 of 11 (2,388 Views)
Accepted Solution

ftp.proxy usage for downloading dependencies of vim depot

Hi  guys ,

We dont have direct internet connection from our servers.

We need to install depothelper and then vim .

I have installed depothelper and now when we are trying to download vim we are not able to access it through proxy , export http_proxy=http://33.xx.xx.xx:8080 

 

Have read in forum ;

 it's not really a transparent proxy, it's an ftp gateway and you can fool applications into using it simply by changing the destination address and embedding the real FTP server login.

 

I need to know how do we use it in our case ???

Honored Contributor
Matti_Kurkela
Posts: 6,271
Registered: ‎12-02-2001
Message 2 of 11 (2,347 Views)

Re: ftp.proxy usage for downloading dependencies of vim depot

Looks like the depothelper script can use FTP only, so setting the http_proxy variable will have no effect to it.

 

Can you connect to the Porting Archive FTP site like this?

$ ftp -i -n 33.xx.xx.xx 8080

[... some messages from your proxy and/or the FTP server here...]

ftp> user anonymous@hpux.connect.org.uk depothelper@localhost
331 Anonymous login ok, send your complete email address as your password
230 Anonymous access granted, restrictions apply
ftp>

 If you can do that with the standard HP-UX ftp command, then the depothelper should work with your FTP proxy.

You should then add the option "-p 33.xx.xx.xx:8080" to your depothelper command, to specify the FTP proxy.

 

But if your proxy is specifically a HTTP->FTP gateway only, then it might not be able to work as a plain FTP->FTP proxy. In that case, you might have to modify the ftp_file() function in the depothelper script to make it work with your strange proxy.

 

Or maybe your proxy system also includes a true FTP proxy functionality, in some different port?

MK
Valued Contributor
chindi
Posts: 391
Registered: ‎07-24-2008
Message 3 of 11 (2,340 Views)

Re: ftp.proxy usage for downloading dependencies of vim depot

[ Edited ]

Hi Matti ,

 

root #/ >ftp -i -n 33.xx.xx.xx 8080
Connected to 33.xx.xx.xx 

 

421 Service not available, remote server has closed connection
ftp> bye

Valued Contributor
chindi
Posts: 391
Registered: ‎07-24-2008
Message 4 of 11 (2,339 Views)

Re: ftp.proxy usage for downloading dependencies of vim depot

[ Edited ]

Hi Matti ,

 

Can you help me with ftp_file function modification ?

I have downoaded ftp.proxy software and installed the same.

How do we start or rather use this functionality ? if my direct internet connection is not working 

 

Honored Contributor
Matti_Kurkela
Posts: 6,271
Registered: ‎12-02-2001
Message 5 of 11 (2,322 Views)

Re: ftp.proxy usage for downloading dependencies of vim depot

The FTP proxy software would have to be installed to a system that has unrestricted access to the FTP servers you want to use (for example hpux.connect.org.uk), or else it won't solve your problem.

 

Do you mean this FTP proxy software:

http://www.ftpproxy.org

 

It must be added /etc/inetd.conf and optionally to /etc/services. An example can be found here:

http://www.ftpproxy.org/faq/index.php?qid=6&frommostpopular=yes

 

 Note that the example includes /usr/sbin/tcpd, i.e. it assumes TCP wrappers are installed and in use. If your proxy system is HP-UX and you prefer to use inetd.sec instead of TCP wrappers, you could replace /usr/sbin/tcpd in the inetd configuration line with the full pathname of the ftp.proxy binary.

 

If your system uses xinetd instead of traditional inetd, an example configuration file can be found here:

http://www.ftpproxy.org/faq/index.php?qid=3&catid=10

 

(Note that the xinetd example configuration includes the -m option to ftp.proxy, which is not required for the basic functionality of the ftp.proxy and requires extra set-up. You may want to omit the -m option if you don't know for sure you'll need it.)

 

-------------

 

With your original proxy set-up, can you download a file from some FTP server on the Internet, using the HP-UX command line? If that is possible, please post an example of such a command.

 

If you can show how to work with your special proxy, then maybe someone can suggest the necessary modifications to the depothelper script. But without knowing all the details about how your special proxy must be accessed, it will be impossible to help you in modifying the script.

MK
Valued Contributor
chindi
Posts: 391
Registered: ‎07-24-2008
Message 6 of 11 (2,319 Views)

Re: ftp.proxy usage for downloading dependencies of vim depot

Hi Matti ,

 

root #/ >wget ftp://es368618:gu_IM4zu@ftp.usa.hp.com
--13:50:35-- ftp://es368618:*password*@ftp.usa.hp.com/
=> `index.html.7'
Connecting to 33.xx.xx.xx:8080... connected.
Proxy request sent, awaiting response... 200 OK
Length: unspecified [text/html]

[ <=> ] 1,327 --.--K/s

13:50:43 (684.21 KB/s) - `index.html.7' saved [1327]

root #/ >cat index.html.7
<HTML>
<HEAD>
<TITLE>FTP / at ftp.usa.hp.com.</TITLE>
</HEAD>
<BODY>
<H2>FTP / at ftp.usa.hp.com</H2>
<HR>
<H4><PRE>
230------------------------------------------------------------------
230- HPRC F T P S e r v e r
230------------------------------------------------------------------
230- Welcome to HP's ftp service. Use of this service requires
230- agreement with HP representatives and implies consent to the
230- Terms of Use (see http://ftp.usa.hp.com/TermsOfService.html).
230- You are user #329 at Fri Jul 26 08:23:49 2013.
230------------------------------------------------------------------
230- NOTE: If your FTP client implementation supports PASSIVE
230- mode, it is recommended that you enable it prior to
230- continuing.
230------------------------------------------------------------------
230 User es368618 logged in

</PRE></H4>
<HR>
<PRE>
[DIRECTORY] 07/25/2003 07:12 J <A HREF="./"><B>.</B></A>
[DIRECTORY] 07/25/2003 07:12 J <A HREF="../"><B>..</B></A>
[FILE] 07/25/2003 07:01 J <A HREF=".welcome">.welcome</A>
[FILE] 07/25/2003 07:14 J <A HREF="evamanager-capture-2013Jul25111921.zip">evamanager-capture-2013Jul25111921.zip</A>
</PRE>
<HR>
</BODY>
</HTML>

Honored Contributor
Matti_Kurkela
Posts: 6,271
Registered: ‎12-02-2001
Message 7 of 11 (2,290 Views)

Re: ftp.proxy usage for downloading dependencies of vim depot

I see, your proxy in port 8080 is indeed a HTTP proxy. Even if it allows connections to FTP servers, any programs wishing to use it must understand HTTP, since HTTP is the protocol used to talk with the proxy itself.

 

Wget is a very suitable tool for your situation: since it can take all its input from the command line, it is very easy to use in scripts.

 

The depothelper script uses its ftp_file() function to download software. This is the part that does the actual download:

         if [ "$proxy_site" = "" ]
         then
            ( 
            echo "user anonymous $progname@localhost"
            echo "binary"
            echo "get $1 $bfile"
            echo "quit"
            ) | ftp -i -n $site >/dev/null 2>&1
         else
            ( 
            echo "user anonymous@$site $progname@localhost"
            echo "binary"
            echo "get $1 $bfile"
            echo "quit"
            ) | ftp -i -n $proxy_site $proxy_port 2>&1
         fi

 

The variables are all either descriptively named or easily identifiable elsewhere in the script:

  • $proxy_site = the hostname or IP address of the proxy, or empty if the -p option is not used
  • $proxy_port = the port number of the proxy
  • $progname = the name of the depothelper script
  • $1 = the full pathname of the file to download (documented at the beginning o the ftp_file() function)
  • $bfile = the name of the file to download, without the directory path (a.k.a basename of the file)
  • $site = the hostname to connect to (= hpux.connect.org.uk)

The original script uses two different ftp command lines, depending on whether a FTP proxy must be used or not. But since wget can take its proxy information through environment variables, it should be possible to just export the proxy variables if required, and then use the same wget command line to support both cases.

 

If you replace the part of the depothelper script listed above with this part, the script should work with your proxy:

        if [ "$proxy_site" = "" ]
        then
           unset ftp_proxy
           unset http_proxy
        else
           export ftp_proxy="http://$proxy_site:$proxy_port"
           export http_proxy="http://$proxy_site:$proxy_port"
        fi 
        wget -O $bfile --user=anonymous --password="$progname@localhost" \
           ftp://$site$1 >/dev/null 2>&1

 

NOTE: after the modification, the modified script will not work at all if wget is not installed and usable.

 

After the modification, the syntax of the depothelper commands is unchanged. The -p option must still be included to the depothelper command line to use a proxy, just as before.

MK
Valued Contributor
chindi
Posts: 391
Registered: ‎07-24-2008
Message 8 of 11 (2,282 Views)

Re: ftp.proxy usage for downloading dependencies of vim depot

Hi  Matti ,

 

Our proxy does not need username & password .

 

root #/ >depothelper -p 33.xx.xx.xx:8080 -n -a ia64 -c /app_dumps/bluefish -o 11.31 bluefish-2.2.1
======================================================================================================================================================
Package-version Comment Download Install
======================================================================================================================================================
ia64-11.31 Package list Downloading... wget: missing URL
Usage: wget [OPTION]... [URL]...

Try `wget --help' for more options.
ia64-11.31 Package list FAILED Not installed
======================================================================================================================================================
ERROR: Can't get list of ia64-11.31 packages from hpux.connect.org.uk - aborted

 


root #/ >wget hpux.connect.org.uk
--11:39:39-- http://hpux.connect.org.uk/
=> `index.html.9'
Connecting to 33.xx.xx.xx:8080... connected.
Proxy request sent, awaiting response... 200 OK
Length: 10,031 (9.8K) [text/html]

100%[============================================================================================================================>] 10,031 --.--K/s

11:39:39 (693.71 KB/s) - `index.html.9' saved [10031/10031]

 

 

Valued Contributor
chindi
Posts: 391
Registered: ‎07-24-2008
Message 9 of 11 (2,275 Views)

Re: ftp.proxy usage for downloading dependencies of vim depot

Thanks again Matti .

Honored Contributor
Matti_Kurkela
Posts: 6,271
Registered: ‎12-02-2001
Message 10 of 11 (2,269 Views)

Re: ftp.proxy usage for downloading dependencies of vim depot

[ Edited ]

> Our proxy does not need username & password .

 

The username & password is not for the proxy, but for the hpux.connect.org.uk site.

 

I don't know for sure if the username has special meaning to the site or not, but I tried to replicate the functionality of the original script as exactly as I could.

 

The FTP pathnames in the package list files used by the depothelper script may not match exactly the HTTP URLs used by the site, so the wget version of the depothelper must use ftp:// URLs.

 

> ia64-11.31 Package list Downloading... wget: missing URL
> Usage: wget [OPTION]... [URL]...

> Try `wget --help' for more options.

 

This error looks like you may have an extra space at the end of the wget line in the modified script. The "\" character must be the last character on that line.

 

Alternatively, you may write the wget command as a single long line, so that the last two lines of the modified part will look like this:

fi 
wget -O $bfile --user=anonymous --password="$progname@localhost" ftp://$site$1 >/dev/null 2>&1

 

(I hate extremely long lines in scripts, so I try to break them into multiple lines using "\", but that is just my style preference.)

 

Edit: Looks like the site will auto-split the long wget command line, even if it is inside a "insert code" block. Anyway, "wget ... >/dev/null 2>&1" should all be a single long line.

MK
Valued Contributor
chindi
Posts: 391
Registered: ‎07-24-2008
Message 11 of 11 (2,249 Views)

Re: ftp.proxy usage for downloading dependencies of vim depot

Thanks Matti
The opinions expressed above are the personal opinions of the authors, not of HP. By using this site, you accept the Terms of Use and Rules of Participation.