Fix random reboot of NETAPP DSA E2800 (dual controller) by updating its Firmware

[#1]

Summary:

FW 11.50.2 must be used for Single Controllers and 11.60.2 for Dual Controllers

Overview:

In Q1 of the year 2020 the BU BT-VS announced that NetApp forces to update all systems E2800 to newer firmware than 11.40.x.

At that time, it was strongly recommended to go to version 11.50.2 and in February 2020 recommendation was given for 11.50.R2 

The version 11.50.R2 was forced by NetApp to

• get proper NetApp support services while valid warranty contract is active
• and to fix vulnerability aspects.

 

General Note:

Newer NetApp controller firmware might be approved and qualified by BOSCH. 

• Details can always be found at the BOSCH product catalogue and Release Note

All customers owning a NetApp storage system with valid warranty agreement and registered unit, do have access to the public announcement of NetApp.

_________________________________________________

• BVMS ≤10.1 – check the table in 7. History chapter from DSA E2800 Release note
• BVMS ≥ 11.0 see the BVMS Release Notes → In DVRs and other devices section in the table from 2.4 Versions chapter

For example:

BVMS 11.0 Release Note:

BVMS 11.1.1 Release Note:

BVMS 12.0 Release Note (valid also for v12.0.1)

Depending on the version you need, you can access this E-Series SANtricity OS Controller Software Link for Downloads. Be aware that you can access this link only if you have a NetApp "Customer" account (not a guest account) set up in NetApp NOW connected with one NetApp system S/N assigned to the NetApp account (during registration or afterwards by technical case). Otherwise, BOSCH Central Technical Support team can offer the service to hand-out

Go to Where to get the Controller Firmware section below within this article to find more detailed explanations.

_________________________________________________

Please follow our BOSCH Knowledge Base and monitor for updates at this article for important news. 
The BOSCH sub model ID 356 ensures that the NetApp system is optimized for 24/7 video recording.

Potential for Data Inconsistency Issues on E-Series Storage Systems

Unfortunately, BOSCH customers contacted the BOSCH technical support where in rare cases customers detected random reboots of DSA E2800 systems running on 11.50.2 or 11.50.R2.

With the controller firmware 11.50.1 and newer version, issues were fixed by NetApp, which enforce an immediate update of all E2800 iSCSI Storage Systems (their controllers). 

After this, the v11.60.2 was released to solve this issue. 

Products that were affected:

Random reboots are possible on NETAPP DSA E2800 controllers running 8.30, 8.40, 11.30, 11.40, and 11.50 versions of E-Series SANtricity OS controller software. 

• Dual controller E2800 iSCSI storage systems using above mentioned older NetApp Firmware versions
• We request customers facing a reboot / unexpected iSCSI connection reset to upgrade unit to the firmware v11.60.2 or to get in contact with BOSCH Technical Support. 

Please take in consideration the following:

• Simplex/ Single controller E2800 iSCSI storage system : According to the BOSCH Product Management and alignment with NetApp: Simplex controller systems E2800 can be operated on NetApp Firmware 11.50.2 only

NOTE: See also the Release Note of SANtricity OS Controller Software for latest information 

Example product variant:

• DSA-N2C8X4-12AT Dual controller unit 12x4TB High-performance and high-capacity storage system base unit with iSCSI disk arrays, dual controller. DSA E2800, 12 x 4 TB HDD, Order number DSA-N2C8X4-12AT

There are other models with various HDD capacity (e.g. 8TB hard drives and larger) available.
For more details visit the BOSCH product website

Important note:
• In general it is strongly recommended to upgrade to latest approved software (e.g. BVMS, VRM), whenever it is possible (keeping mentioned restrictions/limitations in mind), with the listed newest NetApp controller software, NVSRAM file, disk firmware and where needed Expansion Shelf Module (ESM) listed by NetApp. Fixes and comprehensive support services can only offered for the latest approved and secure versions.

Summary of the issue

NetApp® has become aware of issues that could occur when an E-Series controller reboots at certain points during the evacuation of data from a drive that is performed as part of the sequence when a drive is being failed by the controller.

As a result, there is a small possibility of data inconsistency on controllers running certain versions of E-Series SANtricity® OS controller software.

Issue Description

The overall probability of these issues occurring is very low, but RAID 1 volumes have a higher probability of encountering the issues than other RAID levels or NetApp Dynamic Disk Pools.

These issues do not affect traditional RAID volume groups that have no global hot spare drives because drive evacuation does not occur in this configuration.

For information about fixes in these releases, view the readme notes for each revision release.

Workaround

No workaround available. Controller Firmware update is needed.
HDD Firmware update is not in relation to the described issue here, but it is in general recommended to install the latest offered HDD firmware that is recommended by BOSCH and NetApp.
The same applies for HDD firmware.

Solution: Update the NetApp Controller Firmware

Upgrade E-Series SANtricity OS controller software to the latest applicable revision release for each platform as soon as possible.

• Dual controller E2800 - firmware v11.60.2
• Single/ Simplex controller systems E2800 can be operated on NetApp firmware 11.50.2 only

Or reach out to your local BOSCH technical team and BOSCH BT-SC/ETP-MKP team to ensure that the relevant NetApp Firmware has been approved to be usable for 24/7 video recording and replay use case.

Where to get the Controller Firmware

• All customers with a full NetApp NOW support account and valid warranty on their system can download the controller firmware by using the own NetApp user account. The version offered at the NetApp NOW platform should be certified and allowed by BOSCH for video use cases. 
  
  
• In case the download option is not offered to one of our customers, we kindly ask to make sure that the unit is registered on the company and that the NetApp NOW account belongs to the company that has registered the unit. In all other cases, please ask for help to the BOSCH Support organization or local Bosch Technical Support. Tech Support able to provide the firmware in some emergency situations is the Technical Support Solution Group.
• Where and why should 3rd party devices be registered?
  
  
• A NetApp SANtricity firmware / controller firmware package for E2800 model can be downloaded from the NetApp NOW website. 
  Revisions can be found at https://mysupport.netapp.com/NOW/cgi-bin/software/
  See also: https://mysupport.netapp.com/products/web/ECMLP2854621.html 

Additional information is available at NetApp and accessible after user registration.

• First go to https://mysupport.netapp.com/ to register. 
• Product Release Notes from NetApp: https://mysupport.netapp.com/ecm/ecm_download_file/ECMLP2842060
• NetApp Support Bulletin, please view the following URL: https://kb.netapp.com/Support_Bulletins/Customer_Bulletins/SU358
  (As the URL can change any time by NetApp, search for the KB id 1086731)

-----------------------------------------------------------------------------------------------------------------

General Instructions for Downloading E-Series Disk Firmware:

NetApp uses multiple suppliers for physical disk drives, and each has its own disk drive firmware. Disk firmware updates are published on an as-needed basis; therefore, it is possible that although newer disk firmware files are available, they are not applicable to any of the physical drives in your system.

Step	Action
1	Verify the Product ID and Firmware level of the disk drives using the Storage Array Profile from within SANtricity ES Storage Manager.
2	Determine if the Firmware level on the download site is higher than the current firmware on your drive. Proceed to step 3 if there is a later version. If your firmware is the current version there is no need to download the file.
3	Extract the .zip or .gz file for the Drive Identifier matching the Product ID to a temporary location on the system where SANtricity ES Storage Manager is installed.

 

Instructions for Installing E-Series Disk Firmware

Before you begin
Downloading firmware incorrectly could result in damage to the drives or loss of data. If you have any questions contact Technical Support before proceeding.

• Verify that you have downloaded the correct dlp package for the Product ID of the drive.
• Verify that the storage array is in an Optimal State
• Verify that no configuration changes are running on the storage array
• Stop all I/O to the storage array before the download.
• Do not make any configuration changes to the storage array while downloading the firmware.

Installing the Firmware

• From the Array Management Window of SANtricity ES Storage Manager, select Upgrade > Drive/ATA Translator Firmware
• Follow the instructions for selecting and downloading the firmware

-----------------------------------------------------------------------------------------------------------------

 

Steps to update

The E-Series SANtricity^® OS package includes data for simplex controller and duplex controller systems and the firmware file itself.

• NVSRAM File for for duplex.
• NVSRAM file for simplex
• and the controller firmware

1. Download the latest SANtricity OS software files from the NetApp Support Site to your management client.
2. From SANtricity System Manager, select Support> Upgrade Center.
3. In the area labeled “SANtricity OS Software upgrade,” click NetApp Support.
4. On the NetApp Support Site, click the Downloadstab, and then select Software.
5. Locate E-Series/EF-Series SANtricity OS (Controller Firmware).
6. For the platform, select E2800, and click Go!
7. Select the version of SANtricity OS (Controller Firmware) you want to install, and click
   View & Download.
8. Follow the online instructions to complete the file download.

Attention: Risk of data loss or risk of damage to the storage array — Do not make changes to the storage array while the upgrade is occurring. Maintain power to the storage array.