Potential for Data Inconsistency Issues on E-Series Storage Systems
With the controller firmware 11.50.1 and newer version an issues are fixed by NetApp, which enforce an immediate update of all E2800 iSCSI Storage Systems (their controllers). In July 2019 BOSCH announced the need to update from all former NetApp controller firmware 08.30.40.00 to newer versions. These newer versions should be certified and approved by BOSCH. In 2019 there was also an announcement by NetApp. All customers owning a NetApp storage system with valid warranty agreement and registered unit do have access to the public announcemnt of NetApp. This was announced by NetApp on their support websites (see https://mysupport.netapp.com/).
Bosch approval 31st of January 2018
Bosch approval 7th of May 2019
Bosch approval 20th of September 2019
outlook May 2020
NetApp Firmware 08.30.40.00 no longer allowed for usage
NetApp Firmware: 11.50.R1 no longer allowed for usage
NetApp Firmware: 11.50.2 no longer allowed for usage
NetApp works on a global release of a version newer than 11.60.1 - Please ensure that Bosch has certified any newer version before installing!
Please follow our BOSCH Knowledge Base and monitor for updates at this article for important news. The BOSCH submodel ID 356 ensures that the NetApp system is optimized for 24/7 video recording. See article here: ℹ️ https://community.boschsecurity.com/t5/Security-Video/TB-VS-date-2020-04-22-New-Firmware-11-50-3R1P3-is-available-for/ta-p/12778
All DSA E-Series (E2800 12-bay) and DSA E-Series (E2800 60-bay)
Example product variants:
DSA-N2E8X4-12AT Base unit 12x4TB High-performance and high-capacity storage system base unit with iSCSI disk arrays, single controller. DSA E2800, 12 x 4 TB HDD, Order number DSA-N2E8X4-12A
DSA-N2C8X4-12AT Dual controller unit 12x4TB High-performance and high-capacity storage system base unit with iSCSI disk arrays, dual controller. DSA E2800, 12 x 4 TB HDD, Order number DSA-N2C8X4-12AT
Note: There are other models with various HDD capacity (e.g. 8TB harddrives and larger) available For more details visit the BOSCH product website
Summary of issue
NetApp® has become aware of issues that could occur when an E-Series controller reboots at certain points during the evacuation of data from a drive that is performed as part of the sequence when a drive is being failed by the controller. As a result, there is a small possibility of data inconsistency on controllers running certain versions of E-Series SANtricity® OS controller software.
First announced in 06/2019: The issues are possible on E-Series controllers running 8.30, 8.40, 11.30, 11.40, and 11.50 versions of E-Series SANtricity OS controller software. The overall probability of these issues occurring is very low, but RAID 1 volumes have a higher probability of encountering the issues than other RAID levels or NetApp Dynamic Disk Pools. These issues do not affect traditional RAID volume groups that have no global hot spare drives because drive evacuation does not occur in this configuration. For information about fixes in these releases, view the readme notes for each revision release.
No workaround available. Controller Firmware update is needed. HDD Firmware update is not in relation to the descdribed issue here, but it is in general recommended to install the latest offered HDD firmware that is recommended by BOSCH and NetApp. The same applies for HDD firmware.
Solution: Update the NetApp Controller Firmware
Upgrade E-Series SANtricity OS controller software to the latest applicable revision release for each platform as soon as possible. Please reach out to your local BOSCH technical team and BOSCH BT-SC/ETP-MKP team to ensure that the relevant NetApp Firmware has been approved to be usable for 24/7 video recording and replay use case. Even the global NetApp Firmware can be installed on BOSCH submodel ID 356
Where to get the Controller Firmware
All customers with a full NetApp NOW support account and valid warranty on their system can download the controller firmware by using the own NetApp user account. The verison offered at the NetApp NOW platform should be certified and allowed by BOSCH for video use cases. See also: https://community.boschsecurity.com/t5/Security-Video/TB-VS-date-2020-04-22-New-Firmware-11-50-3R1P3-is-available-for/ta-p/12778
In caase the download option is not offered to one of our customers, we kindly ask to make sure that the unit is registered on the company and that the NetApp NOW account belongs to the company that has registered the unit. In all other cases, please recout out to the BOSCH Support organisation or local Bosch Technical Support desk.
A NetApp SANtricity firmware / controller firmware package for E2800 model can be downloaded from the NetApp NOW website. Revisions can be found at https://mysupport.netapp.com/NOW/cgi-bin/software/ See also: https://mysupport.netapp.com/products/web/ECMLP2854621.html
Additional information is available at NetApp and accessible after user registration. First go to https://mysupport.netapp.com/ to register. Product Release Notes from NetApp: https://mysupport.netapp.com/ecm/ecm_download_file/ECMLP2842060
NetApp Support Bulletin, please view the following URL: https://kb.netapp.com/app/answers/answer_view/a_id/1086731 (As the URL can change any time by NetApp, search for the KB id 1086731)
Steps to update
The E-Series SANtricity ® OS package includes data for simplex controller and duplex controller systems and the firmware file itself.
NVSRAM File for for duplex.
NVSRAM file for simplex
and the controller firmware
Download the latest SANtricity OS software files from the NetApp Support Site to your management client.
From SANtricity System Manager, select Support > Upgrade Center .
In the area labeled “SANtricity OS Software upgrade,” click NetApp Support .
On the NetApp Support Site, click the Downloads tab, and then select Software .
Locate E-Series/EF-Series SANtricity OS (Controller Firmware) .
For the platform, select E2800 , and click Go!
Select the version of SANtricity OS (Controller Firmware) you want to install, and click View & Download .
Follow the online instructions to complete the file download.
Attention: Risk of data loss or risk of damage to the storage array — Do not make changes to the storage array while the upgrade is occurring. Maintain power to the storage array.
In rare cases userc might have seen random reboots of the NetApp E2800 contorllers in a
simplex controller E2800 iSCSI storage system or
dual controller E2800 iSCSI storage system.
For example Order number:
DSA-N6C8X4-60AT, DSA-N6C8X8-60AT, DSA-N6C8XA-60AT, DSA-N6C8XC-60AT, DSA-N6C8XC-60AT as well as others like:
Commercial Type No. DSA-N2E8X4-12AT / Product No. F.01U.346.434, Commercial Type No. DSA-N2C8X4-12AT / Product No. F.01U.346.474, Commercial Type No. DSA-N2E8X8-12AT / Product No. F.01U.346.477, Commercial Type No. DSA-N2C8X8-12AT / Product No. F.01U.346.479, Commercial Type No. DSA-N2E8XC-12AT / Product No. F.01U.361.550, Commercial Type No. DSA-N2C8XC-12AT / Product No. F.01U.361.549,
Only for those useres recognizing such a E2800 iSCSI sysstem controller did show an unplanned reboot, can consult the NetApp Major Event Log (MEL). Customers reported this to BOSCH using a NetApp controller firmware 11.40, 11.50.2 or 11.50.R2. Based on that Bosch immediately contacted NetApp and requested support solutions from NetApp.
A basic step when dealing with technical questions for NetApp E2800 iSCSI storage systems is, to register the system with Serial No. by using the following registration form: https://resources-boschsecurity-cdn.azureedge.net/public/documents/Registration_DIP_600_Special_enUS_9007222920871435.docx NetApp and BOSCH want to inform, that we expect a new SANtricity firmware, provided by NetApp, which will contains a fix for the controller random reboot issue which were reported back to Bosch by some customers. The firmware version / SANtricitiy firmware for E2800 iSCSI models is delayed and planned for May 2020.
As this NetApp SANtricity firmware is desigend by NetApp for BOSCH use cases with 24/7 video recording + replay and even worst case situations during a running reconstruct, this Firmware is not yet available via NetApp NOW. NetApp agreed with BOSCH to cover all improvements and fixes in a global and public release around June/July 2020. BOSCH will inform here in this article about the comming release when available.
The password for the local admin and diagnostic port is set to the same starting with E2800 controller firmware 11.40.2 or later.
In case a user does not know his password any longer, it is not possible to reset and re-configure this password by using the diagnostic port (serial port on the rear side near the uplink ports of the controller). With controller firmware older than 11.40.2 it was possible to reset the Administrator Password needed for configuration. But older NetApp controller firmware cannot and must not be used any longer due to other technical reasons. When accessing the Management Port IP to enter the WEB GUI of the NetApp E2800 system for configuraiton or support data collection, the Administrator Password is required. For the time being this is reported to NetApp to work out a solution for all NetApp customers and BOSCH.
Solution and Procedure:
Right now, BOSCH and our customers have no documented instruciton or procedure to remove / reset the password. We are working ont hat with high preassure and awareness. We kindly ask all customers requiring access to the configuraiton but having not the Administrator Password on hand for configuraiton, to reach out to BOSCH support and ask to sent the requewst immediately up to a Levle 3 Ticket to the Gatekeeper team at BOSCH BT-SC/ETP-MKP2 or BT-VS Support in the BU.