SCP is slow in 1 out of 3 directions!!! different than most things (1703 Views)
Reply
Frequent Advisor
marvin51796
Posts: 95
Registered: ‎03-12-2012
Message 1 of 35 (1,703 Views)

SCP is slow in 1 out of 3 directions!!! different than most things

[ Edited ]

ok if i do a scp test1:/tmp/junk /tmp/junk it takes over 5 minutes from machine test2, if from test1 i run the same thing copying from test2 to test1 (scp test2:/tmp/junk /tmp/junk)  it takes 5 seconds. We have verified /etc/hosts tables, DNS entries are correct.

 

We have been working with HP, Cisco, Epic  and no one knows what is causing this? So i am sending out this help call from anyone with any ideas? ...

these are HP-UX blades, running 11vi3.

thanks for any ideas.

 

 

P.S. this thread has been moved from HP-UX>System Administration to HP-UX>networking - HP Forums moderator

 

Honored Contributor
Bill Hassell
Posts: 14,205
Registered: ‎05-29-2000
Message 2 of 35 (1,688 Views)

Re: SCP is slow in 1 out of 3 directions!!! different than most things

I would start with a verification of the ssh versions on each machine:

 

ssh -V

 

If they all match, check the KNRG (random number generator) module on each system:

 

swlist -a readme -a mod_time -a revision -a software_spec -l fileset OS-Core.RNG-DKRN

 

(they should all match)

Also check both systems with top during the slow transfer period.

Does one of systems have exceedingly large amounts of CPU time?

 

Finally, make sure that name resolution is identical on both systems. Use nsquery rather than nslookup:

 

nsquery hosts systema
nsquery hosts systema.mydomain.com
nsquery hosts 10.11.12.13

nsquery hosts systemb
nsquery hosts systemb.mydomain.com
nsquery hosts 10.11.12.14

 

Be sure to query the remote AND the local host by name, fully qualified name and by IP address.

Respected Contributor
Ken Grabowski
Posts: 261
Registered: ‎12-08-2003
Message 3 of 35 (1,685 Views)

Re: SCP is slow in 1 out of 3 directions!!! Different than most things

I have seen this happen when the routing tables are wrong. Often multiple routes in the table causing one host to try multiple paths with each packet sent. What does netstat -rn show on test1?  Try a traceroute from test1 to test2 and see what it reports back.

Frequent Advisor
marvin51796
Posts: 95
Registered: ‎03-12-2012
Message 4 of 35 (1,683 Views)

Re: SCP is slow in 1 out of 3 directions!!! different than most things

all of the ssh -v, swlist -a readme -a mod_time -a revision -a software_spec -l fileset OS-Core.RNG-DKRN, and the nsquery's have worked fine so far, I am having my Networking team look into the routes...

 

Frequent Advisor
marvin51796
Posts: 95
Registered: ‎03-12-2012
Message 5 of 35 (1,679 Views)

Re: SCP is slow in 1 out of 3 directions!!! different than most things

we checked routing tables and stuff, i was told it is on the same layer 2, no hopping, tables all look good according to our head NE person.

 

Respected Contributor
Ken Grabowski
Posts: 261
Registered: ‎12-08-2003
Message 6 of 35 (1,669 Views)

Re: SCP is slow in 1 out of 3 directions!!! different than most things

So what was the output from netstat -in and the results of traceroute from test1 to test2?

Frequent Advisor
marvin51796
Posts: 95
Registered: ‎03-12-2012
Message 7 of 35 (1,663 Views)

Re: SCP is slow in 1 out of 3 directions!!! different than most things

traceroutes only showed 1 jump, to the other machine, both directions...and is it a netstat -rn or netstat -in, the -rn showed routing tables, but the network engineers said they were fine, and if we go to one of the other new machines, that has this same problem with NO additional routes, it still fails.

 

Respected Contributor
Ken Grabowski
Posts: 261
Registered: ‎12-08-2003
Message 8 of 35 (1,659 Views)

Re: SCP is slow in 1 out of 3 directions!!! different than most things

Sorry, netstat -rn.

Frequent Advisor
marvin51796
Posts: 95
Registered: ‎03-12-2012
Message 9 of 35 (1,656 Views)

Re: SCP is slow in 1 out of 3 directions!!! different than most things

ok, the netstat -rn just shows the routes, correctly accoding to the Head Network Engineer, they match up, cluster packages are good, etc...

Respected Contributor
Ken Grabowski
Posts: 261
Registered: ‎12-08-2003
Message 10 of 35 (1,654 Views)

Re: SCP is slow in 1 out of 3 directions!!! different than most things

Marvin, if you dont post results then there is no point to asking for help here.

Frequent Advisor
marvin51796
Posts: 95
Registered: ‎03-12-2012
Message 11 of 35 (1,652 Views)

Re: SCP is slow in 1 out of 3 directions!!! different than most things

ok not sure what you are wanting, but here is what it looks like...

khupepc001:/# netstat -rn
Routing tables
Destination           Gateway            Flags Refs Interface  Pmtu
127.0.0.1             127.0.0.1          UH    0    lo0       32808
172.27.104.54         172.27.104.54      UH    0    lan900    32808
172.27.111.246        172.27.111.246     UH    0    lan902    32808
172.27.104.215        172.27.104.215     UH    0    lan900:1  32808
172.27.103.214        172.27.103.214     UH    0    lan901    32808
172.27.103.215        172.27.103.215     UH    0    lan901:1  32808
172.27.111.240        172.27.111.246     U     2    lan902     1500
172.27.103.192        172.27.103.214     U     3    lan901     1500
172.27.103.192        172.27.103.215     U     3    lan901:1   1500
172.27.104.0          172.27.104.54      U     3    lan900     1500
172.27.104.0          172.27.104.215     U     3    lan900:1   1500
127.0.0.0             127.0.0.1          U     0    lo0       32808
default               172.27.104.1       UG    0    lan900     1500

 

and on the other machine is

khepx019:/# netstat -rn
Routing tables
Destination           Gateway            Flags Refs Interface  Pmtu
127.0.0.1             127.0.0.1          UH    0    lo0       32808
172.27.104.37         172.27.104.37      UH    0    lan900    32808
172.27.103.207        172.27.103.207     UH    0    lan901    32808
172.27.104.0          172.27.104.37      U     2    lan900     1500
172.27.103.0          172.27.103.207     U     2    lan901     1500
127.0.0.0             127.0.0.1          U     0    lo0       32808
default               172.27.104.1       UG    0    lan900     1500

 

 

the problem is pulling a file from khupepc001 when you are logged into khepx019..

if this is not what you are asking please let me know, i am a little confused on this part.

 

Frequent Advisor
marvin51796
Posts: 95
Registered: ‎03-12-2012
Message 12 of 35 (1,650 Views)

Re: SCP is slow in 1 out of 3 directions!!! different than most things

here is the netstat -rn from another system it still doesnt work from.


Routing tables
Destination           Gateway            Flags Refs Interface  Pmtu
127.0.0.1             127.0.0.1          UH    0    lo0       32808
172.27.104.55         172.27.104.55      UH    0    lan900    32808
172.27.111.247        172.27.111.247     UH    0    lan902    32808
172.27.103.212        172.27.103.212     UH    0    lan901    32808
172.27.111.240        172.27.111.247     U     2    lan902     1500
172.27.103.192        172.27.103.212     U     2    lan901     1500
172.27.104.0          172.27.104.55      U     2    lan900     1500
127.0.0.0             127.0.0.1          U     0    lo0       32808
default               172.27.104.1       UG    0    lan900     1500

Respected Contributor
Ken Grabowski
Posts: 261
Registered: ‎12-08-2003
Message 13 of 35 (1,646 Views)

Re: SCP is slow in 1 out of 3 directions!!! different than most things

which of these is test1? and which address do you aim at when doing the test that is slow? 

Frequent Advisor
marvin51796
Posts: 95
Registered: ‎03-12-2012
Message 14 of 35 (1,644 Views)

Re: SCP is slow in 1 out of 3 directions!!! different than most things

we are logged into khepx019, and are doing a scp khupepc001:/tmp/aectst /tmp/aectst and it takes like 5 minutes, if i am logged into either of them and do a scp /tmp/aectst khupepc001:/tmp/aectst  it takes 5 seconds.

and according to our EMR people this is a major problem.

 

Frequent Advisor
marvin51796
Posts: 95
Registered: ‎03-12-2012
Message 15 of 35 (1,641 Views)

Re: SCP is slow in 1 out of 3 directions!!! different than most things

i guess the commands are really setup like this

 

if logged into khepx019 and isues a scp khupepc001:/tmp/aectst /tmp/aectst it takes 5 minutes

if logged into khepx019 and issues a scp /tmp/aectst khupepc001:/tmp/aectst it take 5 seconds

if logged into khupepc001 and issues a scp /tmp/aectst khepx019:/tmp/aectst it takes 5 seconds

hopefully this is a little clearer.

 

Respected Contributor
Ken Grabowski
Posts: 261
Registered: ‎12-08-2003
Message 16 of 35 (1,640 Views)

Re: SCP is slow in 1 out of 3 directions!!! different than most things

From  khepx019 try using individual IP addresses not host names and use each of the IP's that they have. See if there is any difference in the time it takes.

Frequent Advisor
marvin51796
Posts: 95
Registered: ‎03-12-2012
Message 17 of 35 (1,639 Views)

Re: SCP is slow in 1 out of 3 directions!!! different than most things

logged into khepx019 doing a scp 172.27.104.54:/tmp/aectst /tmp/aectst takes the standard 5 minutes.

doing a push from khupepc001 scp /tmp/aectst 172.27.104.37:/tmp/aectst takes 5 seconds

doing a push from khepx019 scp /tmp/aectst 172.27.104.54:/tmp/aectst takes 5 seconds.

Respected Contributor
Ken Grabowski
Posts: 261
Registered: ‎12-08-2003
Message 18 of 35 (1,634 Views)

Re: SCP is slow in 1 out of 3 directions!!! different than most things

whats the time for:

 khepx019 doing a scp 172.27.103.214:/tmp/aectst /tmp/aectst

 

Frequent Advisor
marvin51796
Posts: 95
Registered: ‎03-12-2012
Message 19 of 35 (1,632 Views)

Re: SCP is slow in 1 out of 3 directions!!! different than most things

5 min

Respected Contributor
Ken Grabowski
Posts: 261
Registered: ‎12-08-2003
Message 20 of 35 (1,631 Views)

Re: SCP is slow in 1 out of 3 directions!!! different than most things

Also, while the 5 minute scp is occurring, what is the connect on the remote system?  netstat -an and look for the "ESTABLISHED" line from the originating server. 

Frequent Advisor
marvin51796
Posts: 95
Registered: ‎03-12-2012
Message 21 of 35 (1,628 Views)

Re: SCP is slow in 1 out of 3 directions!!! different than most things


Active Internet connections (including servers)
Proto Recv-Q Send-Q  Local Address          Foreign Address        (state)
tcp        0      0  127.0.0.1.49163        *.*                     LISTEN
tcp        0      0  *.135                  *.*                     LISTEN
tcp        0      0  *.50006                *.*                     LISTEN
tcp        0      0  *.49157                *.*                     LISTEN
tcp        0      0  127.0.0.1.49154        *.*                     LISTEN
tcp        0      0  *.515                  *.*                     LISTEN
tcp        0      0  *.4045                 *.*                     LISTEN
tcp        0      0  *.49159                *.*                     LISTEN
tcp        0      0  127.0.0.1.49163        127.0.0.1.706           ESTABLISHED
tcp        0     48  172.27.104.37.22       172.27.104.54.61319     ESTABLISHED
tcp        0      0  127.0.0.1.706          127.0.0.1.49163         ESTABLISHED
tcp        0      0  127.0.0.1.7161         *.*                     LISTEN
tcp        0      0  *.49166                *.*                     LISTEN
tcp        0      0  172.27.104.37.10443    172.27.104.52.50896     FIN_WAIT_2
tcp        0      0  *.49160                *.*                     LISTEN
tcp        0      0  172.27.104.37.10443    172.27.104.247.55537    ESTABLISHED
tcp        0      0  *.22                   *.*                     LISTEN
tcp        0      0  172.27.103.207.52972   172.27.103.214.22       ESTABLISHED
tcp        0      0  172.27.104.37.22       172.25.114.131.54409    ESTABLISHED
tcp        0      0  *.901                  *.*                     LISTEN
tcp        0      0  *.7815                 *.*                     LISTEN
tcp        0      0  *.587                  *.*                     LISTEN
tcp        0      0  *.6112                 *.*                     LISTEN
tcp        0      0  *.5303                 *.*                     LISTEN
tcp        0      0  *.25                   *.*                     LISTEN
tcp        0      0  *.49158                *.*                     LISTEN
tcp        0      0  172.27.104.37.10443    172.26.104.52.60958     ESTABLISHED
tcp        0      0  *.111                  *.*                     LISTEN
tcp        0      0  172.27.104.37.49156    172.27.16.122.389       ESTABLISHED
tcp        0      0  *.2121                 *.*                     LISTEN
tcp        0      0  *.6389                 *.*                     LISTEN
tcp        0      0  *.2049                 *.*                     LISTEN
tcp        0      0  172.27.104.37.52979    172.27.104.37.1712      TIME_WAIT
tcp        0      0  *.5989                 *.*                     LISTEN
t\

Respected Contributor
Ken Grabowski
Posts: 261
Registered: ‎12-08-2003
Message 22 of 35 (1,621 Views)

Re: SCP is slow in 1 out of 3 directions!!! different than most things

[ Edited ]

What catches my attention is the scp from khepx019 appears to create the following connection:

tcp        0      0  172.27.103.207.52972   172.27.103.214.22       ESTABLISHED

 

While khepx019 has:

172.27.103.207        172.27.103.207     UH    0    lan901    32808
172.27.103.0          172.27.103.207     U     2    lan901     1500

 

khupepc001 has:

172.27.103.214        172.27.103.214     UH    0    lan901    32808
172.27.103.192        172.27.103.214     U     3    lan901     1500

 

That would seem to indicate that your APA pseudo interface 901 on khupepc001 is trying to route traffic to a specific host, unless 172.27.103.192 is a different router.

 

I notice that on both non-khepx019 hosts your 900 based interfaces use a standard self and subnet route but the pseudo 901 and 902 interfaces appear to route to specific hosts.  These may be proper configurations, but since they are different from default configurations, and different from khepx019, this is where I would focus for now. I see nothing else out of the ordinary.

 

APA configurations can be rather complex, and I would hesitate to tell you to change something here without a lot of testing.  I would suggest going over the  /etc/rc.config.d/netconf,  /etc/rc.config.d/hp_apaconf, /etc/rc.config.d/hp_apaportconf, and if used /etc/lanmon/lanconfig.ascii file with APA experts at HP support.

 

 

Acclaimed Contributor
Dennis Handly
Posts: 25,091
Registered: ‎03-06-2006
Message 23 of 35 (1,606 Views)

Re: SCP is slow in 1 out of 4 directions! different than most things

[ Edited ]

>if logged into khupepc001 and issues a scp /tmp/aectst khepx019:/tmp/aectst it takes 5 seconds

 

To be complete, can you reverse this and get the 4th direction and time?

Honored Contributor
Matti_Kurkela
Posts: 6,271
Registered: ‎12-02-2001
Message 24 of 35 (1,603 Views)

Re: SCP is slow in 1 out of 3 directions!!! Different than most things

Modern blade servers should all have gigabit or better NICs. But are all the network links between khupepc001 and khepx019 gigabit, or are there any 100 Mbps hops? If there are, check for duplex mismatches on those hops. On 100 Mbps ethernet, disabling autonegotiation typically meant the NIC would completely stop sending autonegotiation messages and reacting to them. This led to duplex mismatches if one end of a link was forced to Full-Duplex and the other was on autonegotiation, since in the absence of autonegotiation messages from the other end, the autonegotiation logic must assume Half-Duplex. But on gigabit ethernet, "disabling autonegotiation" works a little differently: the NIC will still send autonegotiation messages, but will only offer a single option so there is nothing to choose. Some gigabit NIC chips will use the new disabling method on slower speeds too, others won't. The autonegotiation is a mandatory part of the gigabit Ethernet specification on copper cables, so it cannot be completely disabled in gigabit mode. To check the speed and duplex settings on a physical lanN interface on HP-UX, run "lanadmin -x N" where N is the number of the lanN interface (the number only: leave out the "lan" prefix).
MK
Frequent Advisor
marvin51796
Posts: 95
Registered: ‎03-12-2012
Message 25 of 35 (1,600 Views)

Re: SCP is slow in 1 out of 3 directions!!! Different than most things

Well just an update for everyone, to keep you in the loop. We have talked to Cisco and apparently they had us open up some buffers to max, and this cut the time in half down to 2 1/2 to 3 minutes....but apparently going from HP to Cisco equipment there is a problem with it handling burst copy. No one is sure what we should do now, but Cisco is looking into it......

 

Side not we did check all of the Duplex issues the first night this happened, and MTU settings , and setups on the OA's....WE, meaning HP, Epic didnt see any missmatches.

 

The opinions expressed above are the personal opinions of the authors, not of HP. By using this site, you accept the Terms of Use and Rules of Participation.