The figure below shows that from March to July of 2021 the removal of the weak drives provided a considerable reduction in the resulting HDD failure rate.Assessing the performance of a hard drive and diagnosing possible issues that may occur in its functioning is a highly specialized task that requires dedicated tools in order to get the job done properly. The current DHM thresholds was used on a 1,500 drive system with HDDs over 5 years old which had shown a 5% AFR in January 0f 2020 and was reduced to a 2% AFR in July of 2021 after March 2021. The DHM score was used to remove drives from the arrays before they failed. A weighting system was used to determine a total DHM score.
This method combined drive log sense page data and errors observed in HDDs by the controller (drive error monitoring system, DEMS). Mahmoud Jibbe and Charles Binford from NetApp discussed drive health monitoring (DHM) that they developed to reduce the failure rates of HDD systems for HDDs older than 5 years. Seagate FARM tools for their drives are available through GitHub. Seagate says that they have used this HDD failure prediction method with Tencent, Google and other data center customers. There is about a 0.16% false positive rate and a 36% predicted failure rate within a 7-day predictive window. Using machine learning algorithms FARM log data from customers seems to show good predictive capabilities. These metrics include information from the device and head level. The FARM log size is 96KB and the log structure is similar to ATA device statistics.Ībout 170 different FARM metrics are available in the log for SATA and about 140 for SAS HDDs. They say that FARM provides daily monitoring of a lot more parameters than SMART and that these internal metrics provide better analysis and device management at scale. Paul Brunett and Matt Shumway from Seagate gave a talk about Field Accessible Reliability Metrics (FARM) that Seagate has used with some customers to improve HDD failure prediction. The company keeps track of the annualized failure rate for particular types of drives.īackblaze Average HDD temperature versus Storage Capacity Image from 2021 SNIA SDC There are various types of drive failure and drives that have SMART attributes above certain thresholds are also removed. 255 pairs of data are collected for each drive each day. Backblaze collected SMART self-monitoring data from each drive once per day since 2013. 20 HDDs in a data center share parts of any file. The company has its own storage server design that has evolved over the years. They have 4 data centers in California, Arizona and Holland with 1.8EB of total storage and 178,166 active HDDs and a total of 260,461 HDDs that they have used. There is also a Command Duration Limits (CDL) feature that may be added to SAS and SATA HDDs that should allow higher performance while controlling tail latency.Īndrew Klein from cloud storage provider Backblaze, spoke about their insights from 250,000 hard disk drives used in their data centers. Different drive vendors have different command scheduling policies resulting in behavior differences between drive vendors and drive models. This method allows the user to indicate to the HDD the commands that must be executed quickly and the drive makes a best effort to do so. SATA HDDs have a feature that can enable better HDD performance while controlling tail latency.
This sacrifices potential performance, but reduces the possible tail latency. For optimal performance, this command queue must be managed to control the individual actuator load.ĭamien Le Moal from Western Digital suggested that for HDDs tail latency (the time needed to complete the slowest operation) can be controlled using the HDD at a low queue depth. 32 commands can be queued in the shared HDD queue. Seagate’s split actuator SATA HDD has each sector reachable by only one actuator. They pointed out that without additional actuators latency driven workloads can’t economically utilize HDD capacity gains once IOPS/TB drops below minimal levels. Tim Walker from Seagate and Paolo Valente from the University of Modena gave further insights on multi-actuator optimizations using Linux for SATA HDDs.
0 kommentar(er)
