Root file system is mounted read-only??? (2219 Views)
Reply
Super Advisor
Qcheck
Posts: 282
Registered: ‎11-09-2005
Message 1 of 18 (2,219 Views)
Accepted Solution

Root file system is mounted read-only???

Recently one of our servers rebooted and the root file system is mounted read only. We can't touch any files in that file system. How can I mount rw?

When I tried # mount -n -o remount /
mount: block device /dev/VolGroup00/LogVol00 is write-protected, mounting read-only.

O/S is 5.1. Any help?
Please use plain text.
Honored Contributor
Tim Nelson
Posts: 3,727
Registered: ‎06-01-2000
Message 2 of 18 (2,219 Views)

Re: Root file system is mounted read-only???

how about

mount -o remount,rw /

Please use plain text.
Super Advisor
Qcheck
Posts: 282
Registered: ‎11-09-2005
Message 3 of 18 (2,219 Views)

Re: Root file system is mounted read-only???

No lick Tim:

# mount -o remount,rw /
mount: block device /dev/VolGroup00/LogVol00 is write-protected, mounting read-only.

Do u think hard disk is bad?
Please use plain text.
Super Advisor
Qcheck
Posts: 282
Registered: ‎11-09-2005
Message 4 of 18 (2,219 Views)

Re: Root file system is mounted read-only???

Sorry I meant no Luck.
Please use plain text.
Trusted Contributor
Gerardo Arceri
Posts: 130
Registered: ‎07-15-2009
Message 5 of 18 (2,219 Views)

Re: Root file system is mounted read-only???

try before remounting:
lvchange -pw /dev/VolGroup00/LogVol00

Are you sure that /dev/VolGroup00/LogVol00 is your root filesystem ?
Please use plain text.
Esteemed Contributor
Modris Bremze
Posts: 337
Registered: ‎09-11-2009
Message 6 of 18 (2,219 Views)

Re: Root file system is mounted read-only???

LVM probably has a good reason for disabling writes to /. If you say the server rebooted - was it unexpectedly? If so, did you look at dmesg or /var/log/messages output for any LVM related errors?

Running fsck on a mounted filesystem is not very safe, it being a read-only filesystem could also make it not very productive. You could try booting OEL installation CD1 in recovery mode and running fsck on your LV then.
Please use plain text.
Super Advisor
Qcheck
Posts: 282
Registered: ‎11-09-2005
Message 7 of 18 (2,219 Views)

Re: Root file system is mounted read-only???

Thank you Gerardo and Medris. Yes /dev/VolGroup00/LogVol00 is my / filesystem and when I tried the command, "lvchange -pw /dev/VolGroup00/LogVol00", it errors out as w as an invalid argument even though I saw in the man page.

And I don't see any errors in the log files abt lvm, however I will check thoroughly. I couldn't able to right to / file system, that is how I found the issue. Yes, I know I should boot the server in rescue mode but I am working remotely. Notified this to the management in the afternoon. Probably tomorrow they will decide how to follow up.

Thanks again for all your help.
Please use plain text.
Super Advisor
Qcheck
Posts: 282
Registered: ‎11-09-2005
Message 8 of 18 (2,219 Views)

Re: Root file system is mounted read-only???

This is what I noticed in the log files:

Jun 14 18:11:10 rac3 kernel: SCSI device sda: 859525120 512-byte hdwr sectors (440077 MB)
Jun 14 18:11:10 rac3 kernel: sda: Write Protect is off
Jun 14 18:11:10 rac3 kernel: SCSI device sda: drive cache: write back w/ FUA
Jun 14 19:15:46 rac3 kernel: sd 0:0:0:0: SCSI error: return code = 0x08000002
Jun 14 19:15:46 rac3 kernel: sda: Current: sense key: Hardware Error
Jun 14 19:15:46 rac3 kernel: Add. Sense: Internal target failure
Jun 14 19:15:46 rac3 kernel:
Jun 14 19:15:46 rac3 kernel: Info fld=0x0
Jun 14 19:15:46 rac3 kernel: end_request: I/O error, dev sda, sector 356269
Jun 14 19:15:46 rac3 kernel: Aborting journal on device dm-0.
Jun 14 19:15:47 rac3 kernel: ext3_abort called.
Jun 14 19:15:47 rac3 kernel: EXT3-fs error (device dm-0): ext3_journal_start_sb: Detected aborted journal
Jun 14 19:15:47 rac3 kernel: Remounting filesystem read-only
Please use plain text.
Honored Contributor
P Muralidhar Kini
Posts: 897
Registered: ‎03-14-2010
Message 9 of 18 (2,219 Views)

Re: Root file system is mounted read-only???

Hi Pratibha,

>> This is what I noticed in the log files:
This looks like a hardware error on that device.

Regards,
Murali
Let There Be Rock - AC/DC
Please use plain text.
Respected Contributor
AnthonySN
Posts: 511
Registered: ‎09-30-2006
Message 10 of 18 (2,219 Views)

Re: Root file system is mounted read-only???

was there any san lun presented to the server.
we faced a similar issue when the san lun was unpresented and we rebooted the server but there was an entry in the fstab of the san mounted partition hence it would boot and mount the filesystem in readonly.
Please use plain text.
Respected Contributor
AnthonySN
Posts: 511
Registered: ‎09-30-2006
Message 11 of 18 (2,219 Views)

Re: Root file system is mounted read-only???

Jun 14 18:11:10 rac3 kernel: SCSI device sda: 859525120 512-byte hdwr sectors (440077 MB)
>>440077 MB approx 430 GB means its not local disk.

Jun 14 19:15:47 rac3 kernel: EXT3-fs error (device dm-0): ext3_journal_start_sb: Detected aborted journal
Jun 14 19:15:47 rac3 kernel: Remounting filesystem read-only
>>device dm-0 means a san disk

so check your san disk which may be unpresented

Please use plain text.
Valued Contributor
Thomas Callahan
Posts: 91
Registered: ‎03-04-2005
Message 12 of 18 (2,219 Views)

Re: Root file system is mounted read-only???

DM-0 doesn't strictly mean it's a SAN disk. Device-Mapper can manage anything - including scsi, local attached, in the nose, or SAN. In fact, all LVM devices are managed under DM, and those are just logical pointers, per se.

If it IS a SAN disk, and has been unpresented, there is no reason the server should still be able to see the disk. There should be no path to the disk if it has been "properly" unpresented.

This looks more like a volume failure, more specifically an EXT3 failure. I would reboot in rescue mode, and run an fsck on the disk to attempt to clear the errors. If that fails, you may be looking at a restore from backup.
Please use plain text.
Super Advisor
Qcheck
Posts: 282
Registered: ‎11-09-2005
Message 13 of 18 (2,219 Views)

Re: Root file system is mounted read-only???

Thomas, Thank you and everyone who is responding to this.

In order to reboot in rescue mode, I need the boot disk right? I am afraid the client has that. Physically I can't access the server as it is 3 hrs away and we r supporting remotely.

So now, over the weekend, the server went down, tried to reboot I guess and this is where it stopped:

"Reboot and Select proper Boot device
or Insert Boot Media in selected Boot device and press a key"

So If I ask them to power off and on the button, do you think it will come up? Can I try that first as they don't have the disk1 to mount? Please let me know asap?

Thanks again.
Please use plain text.
Respected Contributor
AnthonySN
Posts: 511
Registered: ‎09-30-2006
Message 14 of 18 (2,219 Views)

Re: Root file system is mounted read-only???

check the bios for boot order change, any usb disk or floppy attached to the server.
Please use plain text.
Super Advisor
Qcheck
Posts: 282
Registered: ‎11-09-2005
Message 15 of 18 (2,219 Views)

Re: Root file system is mounted read-only???

SASJ, Thank you for the response.

We asked the client to power cycle the server and everything came up clean. Now, curious of what happened? Now, no read-only file system. I can write to the / file system. Happies....but wht happened? Nothing suspicious in the log file.
Please use plain text.
Respected Contributor
AnthonySN
Posts: 511
Registered: ‎09-30-2006
Message 16 of 18 (2,219 Views)

Re: Root file system is mounted read-only???

that is indeed gr8 news...
All is Well that ends Well.
cheers...
Please use plain text.
Valued Contributor
Thomas Callahan
Posts: 91
Registered: ‎03-04-2005
Message 17 of 18 (2,219 Views)

Re: Root file system is mounted read-only???

Sounds like whatever is presenting your storage hiccuped or the server hiccuped.

The screen you said it was stuck at was a BIOS screen where it was looking for a Boot Device but not finding one.

I would ask the client where the storage is coming from, and if there were any problems with that storage at the time this occurred.
Please use plain text.
Super Advisor
Qcheck
Posts: 282
Registered: ‎11-09-2005
Message 18 of 18 (2,219 Views)

Re: Root file system is mounted read-only???

After power cycling the server, problem has gone away......
Please use plain text.
The opinions expressed above are the personal opinions of the authors, not of HP. By using this site, you accept the Terms of Use and Rules of Participation