Wikimedia Cloud Services team/EnhancementProposals/SpareDisks

Problem Statement

WMCS servers currently do not have hot spares disks added to their RAID configuration.

We depend on Dell's 24-hour support contract to deliver new replacements to us.

Since we standardize on RAID-10, a single disk failure means that the whole volume has no redundancy anymore and it's at the mercy of the remaining disk in the pair that failed to keep working until a replacement disk arrives, is actually installed and completes the rebuilding process. This means we could be running without redundancy for a few days.

Due to the way that RAID-10 and SSDs work, wear levels could be the same for both SSDs in a pair. So both are at high risk of failure, meaning the remaining disk could fail at any moment too.

Additionally, miscommunication with Dell support or shipping delays could mean additional delays. We also don't have 24x7 DC-Ops staff to start working on a disk replacement immediately and there are organizational challenges in monitoring/alerting that could add even more delays.

Proposal

Adopt a new RAID standard where 2 disks are defined as hot spares. They will become active immediately after a failure is detected by the RAID controller, giving all teams more time to react and reducing the time window where we are without redundancy in our RAID volumes.

In essence, trade storage capacity for faster MTTR.

Technical Impact

Here is the current situation for our various servers:

Server	Vendor	Disks	Type	Raw Capacity	Current RAID-10	RAID-10 w/ spares	Variation	Current Disk Usage
labvirt1001	HP	16	15k 300GB	4.8TB	2.4TB	2.1TB	-12.5%	1.2TB
labvirt1002	HP	16	15k 300GB	4.8TB	2.4TB	2.1TB	-12.5%	1.2TB
labvirt1003	HP	16	15k 300GB	4.8TB	2.4TB	2.1TB	-12.5%	0.9TB
labvirt1004	HP	16	15k 300GB	4.8TB	2.4TB	2.1TB	-12.5%	1.2TB
labvirt1005	HP	16	15k 300GB	4.8TB	2.4TB	2.1TB	-12.5%	0.9TB
labvirt1006	HP	16	15k 300GB	4.8TB	2.4TB	2.1TB	-12.5%	0.7TB
labvirt1007	HP	16	15k 300GB	4.8TB	2.4TB	2.1TB	-12.5%	1.3TB
labvirt1008	HP	16	15k 300GB	4.8TB	2.4TB	2.1TB	-12.5%	0.5TB
cloudvirt1009	HP	16	15k 300GB	4.8TB	2.4TB	2.1TB	-12.5%	-
cloudvirt1012	HP	6	SSD 1.6TB	9.6TB	4.8TB	3.2TB	-33%	-
cloudvirt1013	HP	6	SSD 1.6TB	9.6TB	4.8TB	3.2TB	-33%	0.3TB
cloudvirt1014	HP	6	SSD 1.6TB	9.6TB	4.8TB	3.2TB	-33%	0.5TB
cloudvirt1015	Dell	10	SSD 1.6TB	16TB	8TB	6.4TB	-20%	-
cloudvirt1016	Dell	10	SSD 1.6TB	16TB	8TB	6.4TB	-20%	2.1TB
cloudvirt1017	Dell	10	SSD 1.6TB	16TB	8TB	6.4TB	-20%	1.5TB
cloudvirt1018	Dell	10	SSD 1.6TB	16TB	8TB	6.4TB	-20%	1.1TB
cloudvirt1019	HP	10	SSD 1.6TB	16TB	8TB	6.4TB	-20%	1.5TB
cloudvirt1020	HP	10	SSD 1.6TB	16TB	8TB	6.4TB	-20%	-
cloudvirt1021	Dell	10	SSD 1.6TB	16TB	8TB	6.4TB	-20%	1.3TB
cloudvirt1022	Dell	10	SSD 1.6TB	16TB	8TB	6.4TB	-20%	2.8TB
cloudvirt1023	Dell	10	SSD 1.6TB	16TB	8TB	6.4TB	-20%	2.4TB
cloudvirt1024	Dell	10	SSD 1.6TB	16TB	8TB	6.4TB	-20%	0.2TB
cloudvirt1025	Dell	6	SSD 1.8TB	10.8TB	5.4TB	3.6TB	-33%	0.5TB
cloudvirt1026	Dell	6	SSD 1.8TB	10.8TB	5.4TB	3.6TB	-33%	1.1TB
cloudvirt1027	Dell	6	SSD 1.8TB	10.8TB	5.4TB	3.6TB	-33%	1TB
cloudvirt1028	Dell	6	SSD 1.8TB	10.8TB	5.4TB	3.6TB	-33%	1TB
cloudvirt1029	Dell	6	SSD 1.8TB	10.8TB	5.4TB	3.6TB	-33%	0.5TB
cloudvirt1030	Dell	6	SSD 1.8TB	10.8TB	5.4TB	3.6TB	-33%	1.5TB
cloudvirtan1001	Dell	12	7.2k 4TB	48TB	24TB	20TB	-17%	-
cloudvirtan1002	Dell	12	7.2k 4TB	48TB	24TB	20TB	-17%	-
cloudvirtan1003	Dell	12	7.2k 4TB	48TB	24TB	20TB	-17%	-
cloudvirtan1004	Dell	12	7.2k 4TB	48TB	24TB	20TB	-17%	-
cloudvirtan1005	Dell	12	7.2k 4TB	48TB	24TB	20TB	-17%	-
labstore1004	Dell	26	7.2k 2TB	52TB	26TB	24TB	-8%	-
labstore1005	Dell	26	7.2k 2TB	52TB	26TB	24TB	-8%	-
labstore1006	HP	12	7.2k 6TB	72TB	36TB	30TB	-17%	-
labstore1007	HP	12	7.2k 6TB	72TB	36TB	30TB	-17%	-
cloudstore1008	Dell	12	7.2k 6TB	72TB	36TB	30TB	-17%	-
cloudstore1009	Dell	12	7.2k 6TB	72TB	36TB	30TB	-17%	-

Important notes:

Nominal disk capacity used (base 10 sizes published by vendor)
Disks dedicated to operating system are ignored
cloudvirtan* are "owned" by the Analytics team and may or may not be sized to allow for spares
labstore100{4,5,6,7} are scheduled to be decommissioned with instances on cloudvirt10{19,20} as the replacement

Timeline

If this proposal is accepted, we could adopt two strategies for deployment:

Piggyback on our migration to Stretch/Mitaka on the hypervisors and take each reimage opportunity to modify the RAID configuration
Schedule downtime for the remaining hypervisors that are already in Stretch/Mitaka so they're drained, reimaged and put back into production

Since draining hypervisors is a very disruptive process, the complete implementation of this proposal would have to take into account how much downtime we are comfortable with.

It's expected to be a year goal if not more.

Voting

Please add more stakeholders as needed. Vote Yes/No and a justification.

Name	Vote	Comment
Andrew Bogott	-	-
Arturo Borrero	Yes	We should increase robustness and resilience of CloudVPS. I know this involves capacity/budget/refresh planning.
Brooke Storm	-	-
Bryan Davis	Yes	Support for the piggyback strategy + investigating "converged infrastructure" idea of re-purposing cloudvirt local storage as Ceph storage that is exposed back to the cloudvirts for instance storage.
Giovanni Tirloni	Yes	Reasons: Engineer time is more expensive than cost of spare disks. Lack of redundancy is unacceptable in face of data loss (which has already occurred). We cannot maintain any meaningful SLA with humans in the critical path.

Decision

While some team members didn't cast their vote formally in this document, the majority seemed to agree that adopting hot spare disks was a good strategy in a meeting held on Feb 12.

We will:

Reconfigure RAID arrays to add hot spares at every opportunity we have (reimages, moving hypervisors to the new eqiad1 region, etc)
We will not drain existing hypervisors simply to reconfigure RAID because it's a too time-consuming process
New servers being bought for codfw and only serving dev/test purposes will have only 1 hot spare disk instead of 2 (like in production hypervisors)
Investigate using the hypervisors as Ceph nodes themselves in a "converged infrastructure" approach

RAID configuration

Initial RAID configuration is done using the last two disks as hot spares. As disks fail and get replaced the hot spares will end up being in different slots though.

Dell servers

Reboot and reconfigure it through the UI interface.

Remember to go into 'Advanced' while creating the volume and select 'Add hot spares' and 'Initialize'.

When selecting the disks for the volume, leave the last 2 disks unselected. Unintuitively, a new window will pop up after the volume is creating asking you to select the spare disks.

HP servers (Gen8)

There is an issue pressing F3 to delete existing RAID volume. If possible, run commands from Linux.

Show current status:

=> ctrl slot=0 pd all show

Smart Array P420i in Slot 0 (Embedded)

   array A

      physicaldrive 1I:1:1 (port 1I:box 1:bay 1, SAS, 146 GB, OK)
      physicaldrive 1I:1:2 (port 1I:box 1:bay 2, SAS, 146 GB, OK)

   array B

      physicaldrive 1I:1:3 (port 1I:box 1:bay 3, SAS, 300 GB, OK)
      physicaldrive 1I:1:4 (port 1I:box 1:bay 4, SAS, 300 GB, OK)
      physicaldrive 1I:1:5 (port 1I:box 1:bay 5, SAS, 300 GB, OK)
      physicaldrive 1I:1:6 (port 1I:box 1:bay 6, SAS, 300 GB, OK)
      physicaldrive 1I:1:7 (port 1I:box 1:bay 7, SAS, 300 GB, OK)
      physicaldrive 1I:1:8 (port 1I:box 1:bay 8, SAS, 300 GB, OK)
      physicaldrive 1I:1:9 (port 1I:box 1:bay 9, SAS, 300 GB, OK)
      physicaldrive 1I:1:10 (port 1I:box 1:bay 10, SAS, 300 GB, OK)
      physicaldrive 1I:1:11 (port 1I:box 1:bay 11, SAS, 300 GB, OK)
      physicaldrive 1I:1:12 (port 1I:box 1:bay 12, SAS, 300 GB, OK)
      physicaldrive 1I:1:13 (port 1I:box 1:bay 13, SAS, 300 GB, OK)
      physicaldrive 2I:1:14 (port 2I:box 1:bay 14, SAS, 300 GB, OK)
      physicaldrive 2I:1:15 (port 2I:box 1:bay 15, SAS, 300 GB, OK)
      physicaldrive 2I:1:16 (port 2I:box 1:bay 16, SAS, 300 GB, OK)
      physicaldrive 2I:1:17 (port 2I:box 1:bay 17, SAS, 300 GB, OK)
      physicaldrive 2I:1:18 (port 2I:box 1:bay 18, SAS, 300 GB, OK)

Delete existing RAID volume:

=> ctrl slot=0 ld 2 delete forced

Warning: Deleting an array can cause other array letters to become renamed.
         E.g. Deleting array A from arrays A,B,C will result in two remaining
         arrays A,B ... not B,C

Create new RAID volume (leaving 2 disks for spares):

=> ctrl slot=0 create type=ld raid=1+0 drives=1I:1:3,1I:1:4,1I:1:5,1I:1:6,1I:1:7,1I:1:8,1I:1:9,1I:1:10,1I:1:11,1I:1:12,1I:1:13,2I:1:14,2I:1:15,2I:1:16

Add hot spares to the array:

=> ctrl slot=0 array B add spares=2I:1:17,2I:1:18

Show RAID volume:

=> ctrl slot=0 ld 2 show          

Smart Array P420i in Slot 0 (Embedded)

   array B

      Logical Drive: 2
         Size: 1.9 TB
         Fault Tolerance: 1+0
         Heads: 255
         Sectors Per Track: 32
         Cylinders: 65535
         Strip Size: 256 KB
         Full Stripe Size: 1792 KB
         Status: OK
         Caching:  Enabled
         Unique Identifier: 600508B1001C2FD858B7C664FB32BECD
         Disk Name: /dev/sdb 
         Mount Points: None
         Logical Drive Label: 045EEA760014380311954D026CC
         Mirror Group 1:
            physicaldrive 1I:1:3 (port 1I:box 1:bay 3, SAS, 300 GB, OK)
            physicaldrive 1I:1:4 (port 1I:box 1:bay 4, SAS, 300 GB, OK)
            physicaldrive 1I:1:5 (port 1I:box 1:bay 5, SAS, 300 GB, OK)
            physicaldrive 1I:1:6 (port 1I:box 1:bay 6, SAS, 300 GB, OK)
            physicaldrive 1I:1:7 (port 1I:box 1:bay 7, SAS, 300 GB, OK)
            physicaldrive 1I:1:8 (port 1I:box 1:bay 8, SAS, 300 GB, OK)
            physicaldrive 1I:1:9 (port 1I:box 1:bay 9, SAS, 300 GB, OK)
         Mirror Group 2:
            physicaldrive 1I:1:10 (port 1I:box 1:bay 10, SAS, 300 GB, OK)
            physicaldrive 1I:1:11 (port 1I:box 1:bay 11, SAS, 300 GB, OK)
            physicaldrive 1I:1:12 (port 1I:box 1:bay 12, SAS, 300 GB, OK)
            physicaldrive 1I:1:13 (port 1I:box 1:bay 13, SAS, 300 GB, OK)
            physicaldrive 2I:1:14 (port 2I:box 1:bay 14, SAS, 300 GB, OK)
            physicaldrive 2I:1:15 (port 2I:box 1:bay 15, SAS, 300 GB, OK)
            physicaldrive 2I:1:16 (port 2I:box 1:bay 16, SAS, 300 GB, OK)
         Drive Type: Data
         LD Acceleration Method: Controller Cache

Show all disks (confirm there are 2 spares):

=> ctrl slot=0 pd all show

Smart Array P420i in Slot 0 (Embedded)

   array A

      physicaldrive 1I:1:1 (port 1I:box 1:bay 1, SAS, 146 GB, OK)
      physicaldrive 1I:1:2 (port 1I:box 1:bay 2, SAS, 146 GB, OK)

   array B

      physicaldrive 1I:1:3 (port 1I:box 1:bay 3, SAS, 300 GB, OK)
      physicaldrive 1I:1:4 (port 1I:box 1:bay 4, SAS, 300 GB, OK)
      physicaldrive 1I:1:5 (port 1I:box 1:bay 5, SAS, 300 GB, OK)
      physicaldrive 1I:1:6 (port 1I:box 1:bay 6, SAS, 300 GB, OK)
      physicaldrive 1I:1:7 (port 1I:box 1:bay 7, SAS, 300 GB, OK)
      physicaldrive 1I:1:8 (port 1I:box 1:bay 8, SAS, 300 GB, OK)
      physicaldrive 1I:1:9 (port 1I:box 1:bay 9, SAS, 300 GB, OK)
      physicaldrive 1I:1:10 (port 1I:box 1:bay 10, SAS, 300 GB, OK)
      physicaldrive 1I:1:11 (port 1I:box 1:bay 11, SAS, 300 GB, OK)
      physicaldrive 1I:1:12 (port 1I:box 1:bay 12, SAS, 300 GB, OK)
      physicaldrive 1I:1:13 (port 1I:box 1:bay 13, SAS, 300 GB, OK)
      physicaldrive 2I:1:14 (port 2I:box 1:bay 14, SAS, 300 GB, OK)
      physicaldrive 2I:1:15 (port 2I:box 1:bay 15, SAS, 300 GB, OK)
      physicaldrive 2I:1:16 (port 2I:box 1:bay 16, SAS, 300 GB, OK)
      physicaldrive 2I:1:17 (port 2I:box 1:bay 17, SAS, 300 GB, OK, spare)
      physicaldrive 2I:1:18 (port 2I:box 1:bay 18, SAS, 300 GB, OK, spare)

HP servers (Gen9)

Enter the HP RAID configuration:

Reboot server
Press ESC+9 to enter menu
Select RAID controler on slot 1
Select option to open configuration utility
Wait for "error: no such device: HPEZCD260" message to disappear

Verify current situation:

=> controller slot=1 ld 1 show  

Smart Array P840 in Slot 1

   Array A

      Logical Drive: 1
         Size: 7.3 TB
         Fault Tolerance: 1+0
         Heads: 255
         Sectors Per Track: 32
         Cylinders: 65535
         Strip Size: 256 KB
         Full Stripe Size: 1280 KB
         Status: OK
         MultiDomain Status: OK
         Caching:  Disabled
         Unique Identifier: 600508B1001CB0D3B3EFD3A1715AB007
         Disk Name: /dev/sdd 
         Mount Points: None
         Logical Drive Label: 0110F6E7PDNNF0ARH8015T82C5
         Mirror Group 1:
            physicaldrive 1I:1:5 (port 1I:box 1:bay 5, SATA SSD, 1.6 TB, OK)
            physicaldrive 1I:1:6 (port 1I:box 1:bay 6, SATA SSD, 1.6 TB, OK)
            physicaldrive 1I:1:7 (port 1I:box 1:bay 7, SATA SSD, 1.6 TB, OK)
            physicaldrive 1I:1:8 (port 1I:box 1:bay 8, SATA SSD, 1.6 TB, OK)
            physicaldrive 2I:1:1 (port 2I:box 1:bay 1, SATA SSD, 1.6 TB, OK)
         Mirror Group 2:
            physicaldrive 2I:1:2 (port 2I:box 1:bay 2, SATA SSD, 1.6 TB, OK)
            physicaldrive 2I:1:3 (port 2I:box 1:bay 3, SATA SSD, 1.6 TB, OK)
            physicaldrive 2I:1:4 (port 2I:box 1:bay 4, SATA SSD, 1.6 TB, OK)
            physicaldrive 2I:2:1 (port 2I:box 2:bay 1, SATA SSD, 1.6 TB, OK)
            physicaldrive 2I:2:2 (port 2I:box 2:bay 2, SATA SSD, 1.6 TB, OK)
         Drive Type: Data
         LD Acceleration Method: SSD Smart Path

Delete existing array:

=> ctrl slot=1 array A delete forced

Warning: Deleting the specified device(s) will result in data being lost.
         Continue? (y/n) y

Confirm all disks are now unassigned:

=> ctrl slot=1 pd all show

Smart Array P840 in Slot 1

   Unassigned

      physicaldrive 1I:1:5 (port 1I:box 1:bay 5, SATA SSD, 1.6 TB, OK)
      physicaldrive 1I:1:6 (port 1I:box 1:bay 6, SATA SSD, 1.6 TB, OK)
      physicaldrive 1I:1:7 (port 1I:box 1:bay 7, SATA SSD, 1.6 TB, OK)
      physicaldrive 1I:1:8 (port 1I:box 1:bay 8, SATA SSD, 1.6 TB, OK)
      physicaldrive 2I:1:1 (port 2I:box 1:bay 1, SATA SSD, 1.6 TB, OK)
      physicaldrive 2I:1:2 (port 2I:box 1:bay 2, SATA SSD, 1.6 TB, OK)
      physicaldrive 2I:1:3 (port 2I:box 1:bay 3, SATA SSD, 1.6 TB, OK)
      physicaldrive 2I:1:4 (port 2I:box 1:bay 4, SATA SSD, 1.6 TB, OK)
      physicaldrive 2I:2:1 (port 2I:box 2:bay 1, SATA SSD, 1.6 TB, OK)
      physicaldrive 2I:2:2 (port 2I:box 2:bay 2, SATA SSD, 1.6 TB, OK)

Create a new array / logic device (leaving two disks to be spares later):

=> ctrl slot=1 create type=ld drives=1I:1:5,1I:1:6,1I:1:7,1I:1:8,2I:1:1,2I:1:2,2I:1:3,2I:1:4 raid=1+0 forced

Warning: SSD Over Provisioning Optimization will be performed on the physical
         drives in this array. This process may take a long time and cause this
         application to appear unresponsive. Continue? (y/n)y

Add the last 2 disks as spares:

=> ctrl slot=1 array all add spares=2I:2:1,2I:2:2

Resources

Add papers, external links, etc, that support proposal

This article is issued from Wikimedia. The text is licensed under Creative Commons - Attribution - Sharealike. Additional terms may apply for the media files.