记录一下raid5坏盘更换硬盘恢复的全过程

前言

有个实验需求，需要存放一些数据，所以找了4块10.9T的盘用软raid组了raid5。
刚开始同步数据就死了一块盘。

一、环境

操作系统：ubuntu22.04.5
raid控制卡型号： Intel Corporation C600/X79 series chipset SATA RAID Controller
刚组好时的状态

sde8:64010.9T0disk ├─md1249:124031.1T0raid5 └─md1259:12500B0md sdf8:80010.9T0disk ├─md1249:124031.1T0raid5 └─md1259:12500B0md sdg8:96010.9T0disk ├─md1249:124031.1T0raid5 └─md1259:12500B0md sdh8:112010.9T0disk ├─md1249:124031.1T0raid5 └─md1259:12500B0md

二、处理记录

1.检查raid当前状态

cat /proc/mdstat

root@ubuntu:/var/log# cat /proc/mdstatPersonalities:[raid1][raid6][raid5][raid4][raid0][raid10]md124:active raid5 sde[3]sdf[2](F)sdg[1]sdh[0]33398808576blocks super external:/md125/0 level5, 32k chunk, algorithm0[4/3][U_UU]md125:inactive sdf[3](S)sdg[2](S)sdh[1](S)sde[0](S)20804blocks super external:imsm md126:active raid1 sda[1]sdb[0]927911936blocks super external:/md127/0[2/2][UU]md127:inactive sdb[1](S)sda[0](S)10402blocks super external:imsm unused devices:<none>

raid降级
sdf[2] (F) sdf盘failed
[U_UU] 丢失了第二块盘

2.检查硬盘状态

lsblk检查发现硬盘还在，去机房检查硬盘灯也正常。
试着重启一下，看看啥情况
启动过程中发现了sdf盘异常的日志

启动后检查硬盘列表

sde8:64010.9T0disk ├─md1249:124031.1T0raid5 /hdd-data └─md1259:12500B0md sdf8:80010.9T0disk ├─md1249:124031.1T0raid5 /hdd-data └─md1259:12500B0md sdg8:96010.9T0disk ├─md1249:124031.1T0raid5 /hdd-data └─md1259:12500B0md sdh8:112012.7T0disk ├─sdh18:113016M0part └─sdh28:114012.7T0part sdi8:128012.7T0disk ├─sdi18:129016M0part └─sdi28:130012.7T0part

坏掉的盘直接读取不到了，盘符推位了。。

3.关机换盘

替换对应坏盘，更换为全新同型号硬盘
启动服务器，启动后查看硬盘信息lsblk

lslk sde8:64010.9T0disk ├─md1249:124031.1T0raid5 /hdd-data └─md1259:12500B0md sdf8:80010.9T0disk sdg8:96010.9T0disk ├─md1249:124031.1T0raid5 /hdd-data └─md1259:12500B0md sdh8:112010.9T0disk ├─md1249:124031.1T0raid5 /hdd-data └─md1259:12500B0md

新的sdf盘已经出现，符合预期

4.进行恢复

先检查raid状态

root@ubuntu:~# cat /proc/mdstatPersonalities:[raid1][raid6][raid5][raid4][raid0][raid10]md124:active raid5 sde[2]sdg[1]sdh[0]33398808576blocks super external:/md125/0 level5, 32k chunk, algorithm0[4/3][U_UU]md125:inactive sdh[2](S)sdg[1](S)sde[0](S)15603blocks super external:imsm md126:active raid1 sda[1]sdb[0]927911936blocks super external:/md127/0[2/2][UU]md127:inactive sdb[1](S)sda[0](S)10402blocks super external:imsm unused devices:<none>

执行命令把盘加回去
sudo mdadm --manage /dev/md125 --add /dev/sdf

root@ubuntu:~# sudo mdadm --manage /dev/md125 --add /dev/sdfmdadm: added /dev/sdf

5.查看加入结果及恢复状态

先检查raid状态
cat /proc/mdstat

root@ubuntu:~# cat /proc/mdstatPersonalities:[raid1][raid6][raid5][raid4][raid0][raid10]md124:active raid5 sdf[4]sde[2]sdg[1]sdh[0]33398808576blocks super external:/md125/0 level5, 32k chunk, algorithm0[4/3][U_UU][>....................]recovery=0.0%(1341244/11132936192)finish=829.9minspeed=223540K/sec md125:inactive sdf[3](S)sdh[2](S)sdg[1](S)sde[0](S)16708blocks super external:imsm md126:active raid1 sda[1]sdb[0]927911936blocks super external:/md127/0[2/2][UU]md127:inactive sdb[1](S)sda[0](S)10402blocks super external:imsm unused devices:<none>

查看硬盘状态

avg-cpu: %user %nice %system %iowait %steal %idle0.960.002.120.920.0095.99Device r/s rkB/s rrqm/s %rrqm r_await rareq-sz w/s wkB/s wrqm/s %wrqm w_await wareq-sz d/s dkB/s drqm/s %drqm d_await dareq-sz f/s f_await aqu-sz %util dm-00.000.000.000.000.000.0025.0064.000.000.000.722.560.000.000.000.000.000.000.000.000.021.40loop00.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.00loop10.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.00loop20.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.00loop30.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.00loop40.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.00loop50.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.00loop60.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.00md1240.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.00100.10md1260.000.000.000.000.000.0016.0064.000.000.000.564.000.000.000.000.000.000.000.000.000.010.50sda0.000.000.000.000.000.0025.0064.000.000.000.522.560.000.000.000.000.000.009.000.670.021.30sdb0.000.000.000.000.000.0025.0064.000.000.000.482.560.000.000.000.000.000.009.000.670.021.40sdc0.000.000.000.000.000.001.000.000.000.000.000.000.000.000.000.000.000.001.000.000.000.00sdd0.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.00sde375.00235200.0058242.0099.3661.09627.200.000.000.000.000.000.000.000.000.000.000.000.000.000.0022.91100.00sdf0.000.000.000.000.000.00398.00230600.0058444.0099.3212.16579.400.000.000.000.000.000.000.000.004.8493.10sdg377.00235620.0058242.0099.3649.62624.990.000.000.000.000.000.000.000.000.000.000.000.000.000.0018.71100.00sdh386.00236652.0058235.0099.3425.95613.090.000.000.000.000.000.000.000.000.000.000.000.000.000.0010.0287.90

符合预期，从e\g\h向f盘灌数据

6.收尾工作

恢复完成，重建后收尾工作# 更新 mdadm 配置文件sudomdadm--detail--scan|sudotee-a/etc/mdadm/mdadm.conf# 更新 initramfssudoupdate-initramfs-u

总结

记录一下raid5换盘的步骤记录。