Re: login hanging (2245 Views)
Reply
Trusted Contributor
jerrym
Posts: 134
Registered: ‎02-12-2008
Message 1 of 25 (2,401 Views)
Accepted Solution

login hanging

I've got the infamous login hang requiring ctrl-c to finally get in.

Moved user .profile to .profile.no. Moved /etc/profile to /etc/profile.no

Just the one newly built box 11.11 with patch bundles.

 

As root I can su user without problem. But when su - user , sourcing env, it hangs and I have

to do cntrl-c two or three times and wait a while to get login prompt.

 

I've delt with problems like this before with resolve and path and nfs. But can't find anything

that would be causing this.

 

Acclaimed Contributor
Dennis Handly
Posts: 25,067
Registered: ‎03-06-2006
Message 2 of 25 (2,399 Views)

Re: login hanging

>I've got the infamous login hang requiring ctrl-c to finally get in.

 

Any infamous login hang that I've seen usually can't be ctrl-C.

You need to add "set -x" to your .profile to see what the last lines are.



Valued Contributor
Naj
Posts: 169
Registered: ‎06-16-2011
Message 3 of 25 (2,389 Views)

Re: login hanging

Hi,

Did you try ssh under debug mode?

try this

#ssh abc.om -v

we can get somehint from there

Thanks


____________________________________________
:: Really appreciate if you could assign some points.
:: Don't know how to assign point? Click the KUDOS! star!
Trusted Contributor
jerrym
Posts: 134
Registered: ‎02-12-2008
Message 4 of 25 (2,375 Views)

Re: login hanging

I did the set -x and it goes through all he way to the end of /etc/profile. I even moved my .profile and the /etc/profile so they would not get sourced at all. Tried to run ssh host -v but that works find. It's just that I do not get a login prompt unless I do a ctrl-c several times. The last thing I did on the server was doing a patch commit to clear out the save dir.  Does doing a patch commit require a reboot??

Acclaimed Contributor
Dennis Handly
Posts: 25,067
Registered: ‎03-06-2006
Message 5 of 25 (2,373 Views)

Re: login hanging

>I even moved my .profile and the /etc/profile so they would not get sourced at all.

 

And it still hung?  Where is your $HISTFILE pointing to?

 

>I do not get a login prompt unless I do a ctrl-c several times.

 

Do the following (on another window) to see which process is hanging:

UNIX95=EXTENDED_PS ps -H -fu your-hung-username

 

>Does doing a patch commit require a reboot?

 

No, it just fiddles with the IPD.

Trusted Contributor
jerrym
Posts: 134
Registered: ‎02-12-2008
Message 6 of 25 (2,371 Views)

Re: login hanging

I do know that as root if I  su user  it does not hang. If as root I  su - user  it hangs. - its sourcing something.

Could you explain UNIX95. I vaguely remember something about that. How are you saying to do it on another window??

 

 

Acclaimed Contributor
Dennis Handly
Posts: 25,067
Registered: ‎03-06-2006
Message 7 of 25 (2,369 Views)

Re: login hanging

[ Edited ]

>If as root I  su - user  it hangs. - its sourcing something.

 

Right, you said that.  What shell are you using?  (I've been assuming sh/ksh.)

 

>Could you explain UNIX95. I vaguely remember something about that.

 

Nothing to explain, just do it.  ;-)

It enables the -H option, for a hierarchical listing of processes.

http://h30499.www3.hp.com/t5/tag/UNIX95/tg-p

 

>How are you saying to do it on another window?

 

One window is hung, do it in another, it doesn't matter if you are root or any other user.

Trusted Contributor
jerrym
Posts: 134
Registered: ‎02-12-2008
Message 8 of 25 (2,366 Views)

Re: login hanging

[ Edited ]

okay. ksh

 

I am currently hung on one window with user moorej. Im running in another window

as root.

 

 # UNIX95=EXTENDED_PS ps -H -fu moorej
UID        PID  PPID  C    STIME TTY          TIME CMD
moorej   19309 19307  0 16:54:24 ?           00:00 sshd: moorej@pts/0
moorej   19312 19309  0 16:54:24 pts/0       00:00   -ksh

 

 

What's interesting is that I su - phamn who is running csh and that session comes right up. What gives with ksh??

 

 

Acclaimed Contributor
Dennis Handly
Posts: 25,067
Registered: ‎03-06-2006
Message 9 of 25 (2,362 Views)

Re: login hanging

[ Edited ]

>What's interesting is that I su - phamn who is running csh and that session comes right up.

 

(When you use the scummy C shell, it's not tricky, so it can't fail like this. ;-)

 

I asked above, what is $HISTFILE?  For old OS versions, if it was over NFS, you could get hangs.

Do you have all your NFS, automounter, etc patches?

But if you moved aside ~/.profile , then HISTFILE wouldn't be set.

 

Trusted Contributor
jerrym
Posts: 134
Registered: ‎02-12-2008
Message 10 of 25 (2,359 Views)

Re: login hanging

Correct. no histfile

This server is a new build with all patches. Login was working before. Seems after the server was rebooted that this problem started.

 

I am out of tricks and possible things to check. There is no truss on this server or I would try to use that. I will have to look into loading it.

Acclaimed Contributor
Dennis Handly
Posts: 25,067
Registered: ‎03-06-2006
Message 11 of 25 (2,354 Views)

Re: login hanging

>no histfile

 

You could try setting it to a local filesystem.

 

>I will have to look into loading it.
You may have to put a "sleep 120" at the end of your ~/.profile so you have time to attach to your process before it hangs.



Trusted Contributor
jerrym
Posts: 134
Registered: ‎02-12-2008
Message 12 of 25 (2,349 Views)

Re: login hanging

[ Edited ]

I'm running truss on the sshd process following forks but I dont really see anything. I can see my ctrl-c and returns but it then just sleeps.  I am using Putty, but the problem does not occur on other 11.11 or 11.23 11.31 boxes. I just don't get a prompt.

 

 

Trusted Contributor
jerrym
Posts: 134
Registered: ‎02-12-2008
Message 13 of 25 (2,346 Views)

Re: login hanging

[ Edited ]

Holy crap! It is the .sh_history over nfs mount.  A default histfile does get created even though I am not specifying one. I saw it in the truss ouput.  I changed it to be on a local filesystem, as you suggested, and now it works.  WHY!!!!

 

I still don't know why it only does this on this one box??

 

Thanks Dennis.

Honored Contributor
V. Nyga
Posts: 4,874
Registered: ‎09-20-2002
Message 14 of 25 (2,337 Views)

Re: login hanging

Hi,

 

check /etc/profile.

And read 'man profile' - "If the file /etc/profile exists, it is executed by the shell for every user who logs in."

In a cold installation there will of course exists a default /etc/profile.

 

HTH

V.

*** Say 'Thanks' with Kudos ***
Honored Contributor
Hakki Aydin Ucar
Posts: 1,313
Registered: ‎08-16-2006
Message 15 of 25 (2,335 Views)

Re: login hanging

Can you issue the command ;

3 bdf  (OR jsut in case of got stuck "bdf -l")

and post here.

also you can issue the command (when logon with root)

# echo $0

Valued Contributor
Arunabha Banerjee
Posts: 88
Registered: ‎04-22-2009
Message 16 of 25 (2,316 Views)

Re: login hanging

Please check your DNS server entry in /etc/resolv.conf file and confirm it is working properly or not. Otherwise disable DNS in sshd_config file. Also check /var/adm/wtmp file size, may it is very big.

 

 

Trusted Contributor
jerrym
Posts: 134
Registered: ‎02-12-2008
Message 17 of 25 (2,310 Views)

Re: login hanging

[ Edited ]

Nyga, read farther back. Already isolated /etc/profile. They are the same as other servers we have.

 

Hakki, it is not a stale nfs.  Other users could login using same home nfs mount. Automounter Configs

are same as other servers.

 

Arunabha, I have seen problems with DNS in the past. But resolv.conf is the same as other servers here.

Also, I had two servers with slow login not perm hang that had 1-2 gig size wtmps files. But this is not the case

here.

 

This is something with writing the history file to the home dir nfs mount with this server. Other 11.11 server

is fine.

Honored Contributor
Laurent Menase
Posts: 1,079
Registered: ‎11-06-2003
Message 18 of 25 (2,299 Views)

Re: login hanging

the 2 common problems are:

1) dns not responding, and nswitch.conf configuring it.

2) nfs when the account is mounted over NFS, or mail directory is nfs mounted,

 

check dmesg

 

To tusc/truss rlogind you need to either configure it in inetd.conf or just tusc

inetd with options -p  -f -E -o resfile.out  pidofinetd,

 

then try a rlogin and give the output.

 

 

Acclaimed Contributor
Dennis Handly
Posts: 25,067
Registered: ‎03-06-2006
Message 19 of 25 (2,294 Views)

Re: login hanging

>A default histfile does get created even though I am not specifying one.

 

Are you sure?  I thought the default was some temp file or just memory?  What was the name?

 

> I changed it to be on a local filesystem, as you suggested, and now it works.  WHY!!!!

>I still don't know why it only does this on this one box?

 

You don't have the right patches on it?  Or there was a networking glitch?

Do you have the RPC lock demons working correctly on the client and the server?

 

(Perhaps your control-C works because you have your NFS mounted with INTR?)

I've had this problem over and over for more than a decade because I need a shared history file.)

 

>Hakki, it is not a stale nfs.  Other users could login using same home nfs mount. 

>This is something with writing the history file to the home dir nfs mount with this server.

 

Right, it is the RPC lock demons.  Are these other users using ksh?  Are they on the same machine?

 

You may need the NFS guru Dave to help you.

Trusted Contributor
jerrym
Posts: 134
Registered: ‎02-12-2008
Message 20 of 25 (2,286 Views)

Re: login hanging

The .sh_history file gets created by default.

 

It's the patch bundles in their ignite build that they have been using. The other box has the same.

 

The export is /nfshomes with no options.  Homes is  auto.direct  file with    /nfshomes unity:/nfshomes

pretty straight forward.

 

RPC lock daemons should be doing whatever the default is.

 

Any users have the same problem unless the histfile is redirected to local filesystem.

 

Who is Dave?

 


 

Acclaimed Contributor
Dennis Handly
Posts: 25,067
Registered: ‎03-06-2006
Message 21 of 25 (2,281 Views)

Re: login hanging

[ Edited ]

>The .sh_history file gets created by default.

 

Hmm, I wasn't aware there was a file based default.

 

>RPC lock daemons should be doing whatever the default is.

 

The problem is they get tired and stop working correctly and hang.  ;-)

 

>users have the same problem unless the histfile is redirected to local filesystem.

 

As expected if the RPC lock demons aren't working.

 

>Who is Dave?

 

Dave Olker

http://h30499.www3.hp.com/t5/user/viewprofilepage/user-id/3154

Trusted Contributor
jerrym
Posts: 134
Registered: ‎02-12-2008
Message 22 of 25 (2,277 Views)

Re: login hanging

I've had to restart them in the past before. But I restarted them on this box already.

 

nfs.server, client stop/start

 

But I did not restart them on the nfs server. It would cause an outag I'm sure.

 

Acclaimed Contributor
Dennis Handly
Posts: 25,067
Registered: ‎03-06-2006
Message 23 of 25 (2,253 Views)

Re: login hanging

>I've had to restart them in the past before.

 

That's using a hammer, you need a scalpel:

1) Kill rpc.statd and rpc.lockd ON BOTH SYSTEMS.
        $ ps -ef | grep rpc.lockd
        $ kill <rpc.lockd pid>
        $ ps -ef | grep rpc.statd
        $ kill <rpc.statd pid>

2) Remove all entries in /etc/sm and /etc/sm.bak ON BOTH SYSTEMS.
        $ rm -r /var/statmon/sm /var/statmon/sm.bak

3) Restart rpc.statd (first), then rpc.lockd, on both systems:
        $ /usr/sbin/rpc.statd
        $ /usr/sbin/rpc.lockd

 

http://h30499.www3.hp.com/t5/LVM-and-VxVM/korn-shell-hangs-while-using-NFS-home/m-p/2617417/highligh...

Trusted Contributor
jerrym
Posts: 134
Registered: ‎02-12-2008
Message 24 of 25 (2,245 Views)

Re: login hanging

Dennis. YOU ARE THE MAN!!!   It worked.  No more hangs. It took two logins to finally clear but

then it stopped hanging.

 

Thank you soo much for not giving up on me. It was really starting to bother me that users may

start logging in and getting hung sessions. 

 

I will remember this behavior with rpc.

 

 

 

Acclaimed Contributor
Dennis Handly
Posts: 25,067
Registered: ‎03-06-2006
Message 25 of 25 (2,237 Views)

Re: login hanging

>Thank you so much for not giving up on me.

 

Unfortunately I've had decades of experience with this RPC hang and my shared history file.  :-(

The opinions expressed above are the personal opinions of the authors, not of HP. By using this site, you accept the Terms of Use and Rules of Participation.