System went read-only suddenly

My web server (Linux, Debian Wheezy) has suddenly gone read-only, MySQL have crashed (but Apache didn't) - giving errors when pressing tab to complete file name as:

bash: cannot create temp file for here-document: Read-only file system

Also, when I restart Apache it won't turn on anymore.

What I tried to do : I've replaced the hard disk , moved web files again ; error repeated after 1 week.

Logs shows nothing, one notice that I've seen many failed attempts to access the server through ssh.

df output:

Filesystem 1K-blocks Used Available Use% Mounted on rootfs 249773956 2328024 234758164 1% / udev 10240 0 10240 0% /dev tmpfs 406336 200 406136 1% /run /dev/disk/by-uuid/e45e30eb-efa4-4cd9-aaf9-c6cbe46aa41c 249773956 2328024 234758164 1% / tmpfs 5120 0 5120 0% /run/lock tmpfs 2489760 0 2489760 0% /run/shm /dev/sdb1 249773956 2303784 234782404 1% /mnt/sdb1 

mount -n output:

sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime) proc on /proc type proc (rw,nosuid,nodev,noexec,relatime) udev on /dev type devtmpfs (rw,relatime,size=10240k,nr_inodes=506431,mode=755) devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000) tmpfs on /run type tmpfs (rw,nosuid,noexec,relatime,size=406336k,mode=755) /dev/disk/by-uuid/e45e30eb-efa4-4cd9-aaf9-c6cbe46aa41c on / type ext4 (ro,relatime,errors=remount-ro,user_xattr,barrier=1,data=ordered) tmpfs on /run/lock type tmpfs (rw,nosuid,nodev,noexec,relatime,size=5120k) tmpfs on /run/shm type tmpfs (rw,nosuid,nodev,noexec,relatime,size=2489760k) rpc_pipefs on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw,relatime) /dev/sdb1 on /mnt/sdb1 type ext4 (rw,relatime,user_xattr,barrier=1,data=ordered) 

1 Answer

Your root file system is mounted read-only. This likely happened on a reboot. There are a few options:

• Configure the system to fix errors during reboot. On Ubuntu this is controlled by the FSCKFIX option in the file /etc/default/rcS.
• Reboot in recovery mode and run fsck -f /dev/disk/by-uuid/e45e30eb-efa4-4cd9-aaf9-c6cbe46aa41c and reboot again.
• Boot from a recovery disk and run fsck as shown above.

All options will require your system to be down for a bit. However, it appears it is effectively down now.

It may be possible to run fsck without rebooting and remount the system rw. I would run a test fsck without enabling fixing problems to see how broken the file system