04-11-2012 11:13 AM
I have looked in the forums and found a lot of people that have had the problem where they cannot unmount a filesystem.
They all say to install lsof and run "lsof /dev/vgbck/lv01 " to find the process that is hanging it. NOPE. No process.
One says to run "UNIX=95 fuser /dev/vgbck/lv01" nope.
UNIX=95 lsof? Nope
cd / before unmounting? Nope
NFS in use? Nope
my only GUESS is a process called postmaster used by sfmdb. But I can find nothing on it.
It appears that if I remove the filesystem from /etc/fstab and reboot, the filesystem is no longer mounted. However, if I add it back into fstab and remount, it hangs.
And .....it hangs on whatever mount point I gave it. I mounted it to a different spot. That seemed to work ok. Then I tried unmounted it. Still ok.... Then I mount the unmounted filesystem to /entbck. BOMB. It saysit is already mounted....but it is NOT.
"UX:vxfs mount:ERROR: V0-3-21264: /dev/vgbck/lv01 is already mounted, /entbck is busy, allowable number of mount points exceeded."
itanium rx2200-i2 hpux11.3. onlineJfs is running. I also have glance plus.
I have tried this on TWO identical itanium boxes. I have tried this on an EMC filesystem, and an unused vg00 filesystem.
My only dumb solution is to remove the mount point from /etc/fstab, rebooot, wait 10 minutes, then add it back into /etc/fstab.
I find this error a bit crazy. Any ideas where the problem lies?
Solved! Go to Solution.
04-11-2012 12:26 PM
Well, sfmdb is part of System Fault Manager, but I don't think that should be touching anything other than maybe /opt, /usr, /var, and /tmp.
Try running lsof and fuser on the mountpoint, not the LV's device file. Examples:
# fuser -cu /tmp
/tmp: 2797o(root) 2787o(root) 2779o(root) 2771o(root)
# /usr/local/bin/lsof /tmp COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME postmaste 2745 sfmdb 6u unix 64,0x4 0t0 1326 /tmp/.s.PGSQL.10864 (0x58122380) snmpdm 2757 root 5u unix 64,0x4 0t0 1353 /tmp/.AgentSockets/A (0x58122c80) hp_unixag 2771 root 1r DIR 64,0x4 8192 2 /tmp hp_unixag 2771 root 2r DIR 64,0x4 8192 2 /tmp ipv6agt 2779 root 1r DIR 64,0x4 8192 2 /tmp ipv6agt 2779 root 2r DIR 64,0x4 8192 2 /tmp ipv6agt 2779 root 3u REG 64,0x4 0 9 /tmp/ipv6agt.crashlog mib2agt 2787 root 1r DIR 64,0x4 8192 2 /tmp mib2agt 2787 root 2r DIR 64,0x4 8192 2 /tmp trapdesta 2797 root 1r DIR 64,0x4 8192 2 /tmp trapdesta 2797 root 2r DIR 64,0x4 8192 2 /tmp
Post the output here so we can take a gander.
04-11-2012 12:32 PM - edited 04-11-2012 12:49 PM
UNIX=95 lsof /entbck
UNIX=95 fuser /entbck
UNIX=95 lsof /dev/vgbck/lv01
UNIX=95 fuser /dev/vgbck/lv01
umount: cannot unmount /dev/vgbck/lv01 : Device busy
umount: return error 1.
When I say "NOTHING", I really am getting a null response. I get the device name or mount point, followed by a colon, then the unix prompt.
And I tried fuser -c and fuser -cu.
04-11-2012 01:00 PM
Yeah I tried that too. I had it in there, but I removed it. It had no affect. And it was not nfs mounted anywhere.
Strange enough, a REBOOT will not fix it. But a reboot with it flat out removed from /etc/fstab works for a FEW minutes until I add it back into fstab. It's crazy. I really wish I could find some clue as to why it says it is busy, yet is not.
I ran "what /sbin/umount" it is from PHCO_38752
I should look up that patch.
04-12-2012 05:15 AM
Thanks for the NFS advice. I know it can't be NFS though. I have a number of other vg00 filesystems that are not part of the standard hpux. One of them has this problem too. It is /dev/vg00/lvol10 mounted to /dumplog. This one never used NFS.
But it seems like the /etc/fstab file is involved a bit too much. Listen to THIS craziness. I removed /entbck from the fstab file. I rebooted. I made a new mount point /entbckL. I could unmount and mount to it no problem. I added /entbckL to /etc/fstab. Seems ok. Now I unmounted it, commented out the /entbckL from the fstab file and tried to mount the unmounted filesystem to /entbck......WAIT A MINUTE......
I found that if I removed the /entbck mount point and recreated it, things begin to work again. I still do not know the cause. But I have a really DUMB workaround.
1. remove the filesystem from fstab
3. delete the mount point, and recreate it (aka rmdir/mkdir).
4. wait 120 seconds for the stealth process of evil to get done taking over stuff.
(oh this step is a little wishy-washy).
5. mount the disk
6. edit /etc/fstab.
This is NOT really the solution because I have to reboot.
04-13-2012 04:53 AM - edited 04-16-2012 12:19 PM
yep. My choices are: possibly corrupt a filesystem, crash it, or reboot. I was hoping to find the process that is hanging on, and staying hidden. The forceful unmount should kill it. But I would rather identify it.
edit: GOSH... This is terribly inconvenient. forcing an unmount fails too. So my ONE AND ONLY OPTION is to comment out the filesystem in /etc/fstab, then reboot.
Very, very wrong....correction....WEIRD.
I have had a support call into HP for the past 2 days. I have not gotten an answer. I guess they think it is weird too.
edit...past 5 days and counting.
I wouldn't be whining about it, except that these commands are BROKEN? fuser, umount, lsof?
04-26-2012 05:55 AM
I'm am certain that eventually I will find my error. I just have not found it yet.
There was an update to VXFS as of 9 March 2012. I thought that would fix my problem. It did not.
Getting the filesystem to properly unmount is a bit like pulling a bone away from a really BIG junkyard dog.
For your information, here is my hand to hand combat with the filesystem.
....oh... cut and paste does not work? Ok... at least I can TYPE.
I will call the filesystem /dev/vgbck/lv01 /back
I put in patches. And reboot. I still cannot unmount /back
I remove /back from the /fstab and reboot. Now I cannot MOUNT /back because something says that /dev/vgbck/lv01 IS IN USE ALREADY! <chomp goes that dog>
What's using it?
lsof -V /dev/vgbck/lv01 <nothing>
fuser /dev/vgbck/lv01 <nothing>
lsof -V /back <nothing>
....yadda yadda yadda ....<nothing>
OK ....chomp goes that GHOST dog
I flat out remove the mount point. (THERE. Try to use THAT) And I reboot.
I can't mount it because the mount point is gone. I deleted it.
So I mkdir /back. And I run mount /dev/vgbck/lv01 /back
I run umount /back
It UN mounts fine.
mount/unmount/mount/unmount......yep. No problems anymore.
So I update /etc/fstab to have it mount.
I unmount /back
I run mount -a
It is fine.
mount/unmount/mount/unmount.....STILL...NO problems anymore.
I reboot? NOT YET.... I need to work on other stuff. But I bet it will start to fail again after a reboot.
the force options do not work.
the vxumount -oforce /back says it is UNlicensed. This is silly since I have onlineJFS installed, and the vxfs type (via ftyp -V /dev/vgbck/lv01) says it is type SEVEN.
Whatever.... I guess this will look more like a blog, than a solution.
So for at least the next day or two, the umount command will work for me.
PS: Yyou just KNOW it has to be something stupid. I always hope the cause of the error is something I did. At least that gives me something I can solve.
05-03-2012 05:35 AM - edited 05-03-2012 05:36 AM
I tried to set up an rc file to mount the UNMOUNTABLE filesystem after everything else is up.
I kept the filesystem out of fstab.
After everything is up I ran (via rc script):
/usr/sbin/vgchange -a y /dev/vgbck
/usr/sbin/fsck -F vxfs -y /dev/vgbck/lv01
/usr/sbin/mount -F vxfs -olargefiles,delaylog,datainlog /dev/vgbck/lv01
IT WORKED! IT DID NOT WORK! It died after I finished writing up my solution.
Oh. Back to square one.
Well the good news is that I can unmount file filesystem with this in a mere 2 reboots.
It seems a bit harsh.
I also looked into fuser and lsof. the file system is "in use." Yet lsof and fuser say it is NOT in use. I don't even care if it is in use or not. I just want to be able to unmount it. (And FORCING the mount fails too by the way).
I have a big database running. I know logical volume /dev/vgtst1/lv01 is is use. I run lsof /dev/vgtst1/lv01.Lsof says that nothing is running on it. I run lsof -V /dev/vgtst1/lv01. It says the filesystem is not in use. I repeat this with fuser. Nope. Both commands say this is not in use.
ok...FINE. I KNOW this is in use. I wonder if the mount point for the ROOT filesystem is in use? YES. Yes it is. I get a lot of stuff. I run lsof -V / | grep vgtst'\/lv01 and Ah HA! There is is. So I can find a process for a logical volume that I know is in use. All I have to do is search for the filesystem I want to unmount.
Ok.... So now I know that lvof -V / will show all the hooks. I cannot find anything that is use /back and or /dev/vgbck/lv01. Back to square one....again.
So for this entry:
rc script and removal from /etc/fstab did not solve my problem. It fixed it only for a few minutes.
And the lsof -V command can't find any processes unless I run it on ROOT, and grep for stuff.
For earlier entries:
forcing the umount fails.
Putting in patches for Vxfs did not fix it.
05-07-2012 04:52 AM - edited 05-07-2012 06:59 AM
Ok. thanks to HP, I found the problem.
In the first second of the error (so many months ago), I figured it was NFS. I immediately made sure I didn't have anything NFS mounted. That didn't even take a second. I saw I had nothing NFS mounted.
Why I could not unmount? IT WAS NFS! But in a weird way.
You see, I have /etc/dfs/dfstab setup so I could mount this one, certain filesystem. But it was NOT umounted up. Well? I guess that's close enough. Just the small desire to someday NFS mount this one file system was good enough for the system to decide to never allow it to unmount, and never tell me why.
How HP found it? I told them that I thought is was something in the startup. They had me shutdown to single user mode and run each startup command one at a time. They are a LOT of those. But I did it anyway. And WHILE I was running every rc/S#### link, I would try to unmount that filesytem /back. Well I got to the NFS server and it decided to not allow me to unmount filesystem /back! A HA!
Solution: Maybe there is a patch on this. But here is MY solution. It will take about 5 seconds to implement.
1. unmount filesytem. Error?
2. make sure the filesystem is not in use. via lsof -V /back and lsof -V /dev/vgbck/lv01 and fsuser.
3. Still error? Stop the NFS server via /etc/init.d/nfs.server stop
4. There is no step 4. THAT is it.
No wait a second. Forum person Mkdlxk told me I should grep for the filesystem in /etc/dfs. but he didn't tell me anything else.
Even if I: "removed the /back from that /etcdfs/dfstab, and ran exportfs -a" it would have no affect. I still would have a unmountable filesystem.
I would have to also stop and start the nfs server.
THE REAL SOLUTION is below. This was MY solution. I put in way too much effort on this.
05-07-2012 06:41 AM
You said that you removed it from /etc/dfs/sharetab, so I didn't go any farther down that route. I take it you put it back in?
An 'unshareall' should do the trick. You shouldn't have to completely stop NFS.
05-07-2012 06:57 AM - edited 05-07-2012 07:00 AM
Yep. This is how the forums work for me. I ask a question. I get answers that don't help or no response for weeks.
I dig and dig and finally get a solution.
I post the solution.
15 minutes later I get the answer I was looking for when I first entered the question.
Thanks. It is too bad the timing of that is always off. Perhaps the next time I have a problem, I should pretend I solved it. Then I will get the solution immediately.
unshareall is the solution.
I knew it would be a simple solution. I knew it was a bonehead problem. I just could not read minds.
05-07-2012 07:03 AM
05-07-2012 07:13 AM - edited 05-07-2012 07:18 AM
OH......It is just big communications problem here. You did fine. You gave me the true solution. I actually HOPED that since I had my GOOFY solution, some one would immediate give the REAL solution. And you did. THANK YOU.
What I know as absolute trues are now not so absolute.
Some things that are not true anymore.
fuser will tell you if something is in use.
Lsof will tell you if something is in use.
NFS will not get in the way of a filesystem because the filesystem is not nfs-mounted anywhere.
And there is a new command called unshareall that has probably been in this universe for years.
05-07-2012 10:59 AM
>I immediately made sure I didn't have anything NFS mounted.
I also mentioned that you must not have it NFS exported. That won't show up by lsof.
>And there is a new command called unshareall that has probably been in this universe for years.
Only for 11.31.
05-07-2012 11:14 AM - edited 05-07-2012 11:14 AM
If the NFS daemons truely do not hold open filesystems that contain NFS exports (suggested by the lack of help from fuser or lsof), then I'm guessing a list of NFS exports is consulted when an unmount is attempted. If this is the case, maybe a request for enhancement should be created to give a more useful error message? I'm sure this bites most people new to 11.31 at some point.
05-07-2012 11:20 AM
I ran bdf on the client box. It was not there.
I removed had it NFS exported. BUT....
I emptied /etc/dfs/dfstab
I ran "exportfs -a"
And the 11.3 apparently decided that wasn't good enough release the NFS export. Too bad 11.3 did NOT tell me so.
05-07-2012 11:26 AM
05-07-2012 11:26 AM
"If the NFS daemons truely do not hold open filesystems that contain NFS exports (suggested by the lack of help from fuser or lsof), then I'm guessing a list of NFS exports is consulted when an unmount is attempted. If this is the case, maybe a request for enhancement should be created to give a more useful error message? I'm sure this bites most people new to 11.31 at some point."
Kris, you are SO RIGHT. I got bit!
I know where to look now. I'll view the contents of /etc/dfs/sharetab to make sure the filesystem to unmount is not listed there.
I see I can also run: "unshare /back" in the event I need to unhook an NFS exported filesystem yet keep others still NFS mounted to other client computers.
05-07-2012 01:02 PM
Like /etc/mnttab is used by the mount/umount commands to track currently mounted filesystems, /etc/dfs/sharetab is used by the share/unshare commands to track currently shared filesystems.
Neither file should be edited by hand in any normal situation.
If you edit them, it will cause confusion, unless you really know what you're doing.
Tools like fuser and lsof will only tell you if something is being used by any userspace process: the kernel NFS server is not an userspace process, although some of its components may appear as processes in the ps -ef listing.
05-07-2012 04:43 PM
>I'm sure this bites most people new to 11.31 at some point.
I don't think this has changed for 11.31, except you need to look at /etc/dfs/sharetab instead of /etc/xtab.
>I don't think "exportfs -a" unexports/unshares things that have gone missing from dfstab
Right, exportfs -a only adds filesystems, unless -u is used. I assume that -au uses the info in /etc/dfs/sharetab.
>Neither file should be edited by hand in any normal situation.
It probably doesn't matter since mnttab is a device file (on 11.31) and can't be edited.