I did not know why my NAS said one of the Basic volumes as 'Crashed' and I could not write files to it. At the beginning I though it should be easy task so I logged the system through SSH directly and umounted the filesystem and run a 'e2fsck' on it. It got some errors and fixed them then I could mount it without any issue, and of course I could write it without any problem.
It was fixed, right? Yes, until I rebooted the NAS.
After the reboot, I found this volume still was marked as 'Crashed' and the filesystem became readonly again. And this time I wanted to fix it permanently.
When I tried to assembly the raid array, I got below error message:
dsm> mdadm -Av /dev/md3 /dev/sdd3 mdadm: looking for devices for /dev/md3 mdadm: /dev/sdd3 is identified as a member of /dev/md3, slot 0. mdadm: device 0 in /dev/md3 has wrong state in superblock, but /dev/sdd3 seems ok mdadm: added /dev/sdd3 to /dev/md3 as 0 mdadm: /dev/md3 has been started with 1 drive. dsm> e2fsck /dev/md3 e2fsck 1.42.6 (21-Sep-2012) 1.42.6-5644: is cleanly umounted, 809/91193344 files, 359249805/364756736 blocks
I checked the raid array and the disk partition, while could not find any issue:
dsm> mdadm -D /dev/md3 /dev/md3: Version : 1.2 Creation Time : Thu Feb 4 15:03:34 2016 Raid Level : raid1 Array Size : 1459026944 (1391.44 GiB 1494.04 GB) Used Dev Size : 1459026944 (1391.44 GiB 1494.04 GB) Raid Devices : 1 Total Devices : 1 Persistence : Superblock is persistent Update Time : Sat Dec 30 16:13:47 2017 State : clean Active Devices : 1 Working Devices : 1 Failed Devices : 0 Spare Devices : 0 Name : Gen8:3 UUID : d0796c7b:c9b0ab70:211e65c0:843891e2 Events : 40 Number Major Minor RaidDevice State 0 8 51 0 active sync /dev/sdd3 dsm> mdadm -E /dev/sdd3 /dev/sdd3: Magic : a92b4efc Version : 1.2 Feature Map : 0x0 Array UUID : d0796c7b:c9b0ab70:211e65c0:843891e2 Name : Gen8:3 Creation Time : Thu Feb 4 15:03:34 2016 Raid Level : raid1 Raid Devices : 1 Avail Dev Size : 2918053888 (1391.44 GiB 1494.04 GB) Array Size : 2918053888 (1391.44 GiB 1494.04 GB) Data Offset : 2048 sectors Super Offset : 8 sectors State : clean Device UUID : a9eafb18:84f3974f:42751688:7136418d Update Time : Sat Dec 30 16:59:19 2017 Checksum : faa9b8c3 - correct Events : 40 Device Role : Active device 0 Array State : A ('A' == active, '.' == missing)
So for some unknown reason, I always got the wrong state in the /proc/mdstat:
dsm> cat /proc/mdstat Personalities : [linear] [raid0] [raid1] [raid10] [raid6] [raid5] [raid4] md3 : active raid1 sdd3(E) 1459026944 blocks super 1.2 [1/1] [E] md2 : active raid1 sdc3 17175149551 blocks super 1.2 [1/1] [U] md1 : active raid1 sdc2 sdd2 2097088 blocks [12/2] [UU__________] md0 : active raid1 sdc1 sdd1 2490176 blocks [12/2] [UU__________] unused devices: <none>
There was an '[E]' letter and I though maybe it meant 'Error', so how to clear it?
I searched and read lots of posts, and below one save me:
I found similar solutions before reaching this post, while I did not dare to try it as I had not copied all the data, but the above information gave me more confidence and I run below commands:
dsm> mdadm -Cf /dev/md3 -e1.2 -n1 -l1 /dev/sdd3 -ud0796c7b:c9b0ab70:211e65c0:843891e2 mdadm: /dev/sdd3 appears to be part of a raid array: level=raid1 devices=1 ctime=Thu Feb 4 15:03:34 2016 Continue creating array? y mdadm: array /dev/md3 started. dsm> cat /proc/mdstat Personalities : [linear] [raid0] [raid1] [raid10] [raid6] [raid5] [raid4] md3 : active raid1 sdd3 1459026944 blocks super 1.2 [1/1] [U] md2 : active raid1 sdc3 17175149551 blocks super 1.2 [1/1] [U] md1 : active raid1 sdc2 sdd2 2097088 blocks [12/2] [UU__________] md0 : active raid1 sdc1 sdd1 2490176 blocks [12/2] [UU__________] unused devices: <none> dsm> e2fsck -pvf -C0 /dev/md3 809 inodes used (0.00%, out of 91193344) 5 non-contiguous files (0.6%) 1 non-contiguous directory (0.1%) # of inodes with ind/dind/tind blocks: 0/0/0 Extent depth histogram: 613/84/104 359249805 blocks used (98.49%, out of 364756736) 0 bad blocks 180 large files 673 regular files 127 directories 0 character device files 0 block device files 0 fifos 0 links 0 symbolic links (0 fast symbolic links) 0 sockets ------------ 800 files dsm> cat /etc/fstab none /proc proc defaults 0 0 /dev/root / ext4 defaults 1 1 /dev/md3 /volume2 ext4 usrjquota=aquota.user,grpjquota=aquota.group,jqfmt=vfsv0,synoacl 0 0 /dev/md2 /volume1 ext4 usrjquota=aquota.user,grpjquota=aquota.group,jqfmt=vfsv0,synoacl 0 0 dsm> mount /dev/md3 dsm> df -h Filesystem Size Used Avail Use% Mounted on /dev/root 2.3G 676M 1.6G 31% / /tmp 2.0G 120K 2.0G 1% /tmp /run 2.0G 2.5M 2.0G 1% /run /dev/shm 2.0G 0 2.0G 0% /dev/shm none 4.0K 0 4.0K 0% /sys/fs/cgroup /dev/bus/usb 2.0G 4.0K 2.0G 1% /proc/bus/usb /dev/md2 16T 14T 2.5T 85% /volume1 /dev/md3 1.4T 1.4T 21G 99% /volume2 dsm> ls /volume2 @eaDir @tmp aquota.group aquota.user downloads lost+found synoquota.db
After that I checked the volume state again and the annoying 'Crashed' became 'Normal'.