In rare cases userc might have seen random reboots of the NetApp E2800 contorllers in a
simplex controller E2800 iSCSI storage syste (updated: According BOSCH Product Managemetn and alignment with NetApp Simplex controller systems E2800 can be operated on NetApp Firmware 11.50.2 Updates will be published till 8th of February 2021
dual controller E2800 iSCSI storage systems usng below mentioned older NetApp Firmware versions
This article is valid for Dualcontroller NetApp E2800 systems. Only for those useres recognizing such a E2800 iSCSI dual controller sysstem show an unplanned reboot, can consult the NetApp Major Event Log (MEL). Customers reported this to BOSCH using a NetApp
dual controller firmware 11.40, dual controller firmware11.50.2
dual controller firmware11.50.R2.
We request customers facing a reboot / unexpected iSCSI connection reset to get in contact with BOSCH Technical Support. For Dual Controller systems a new version is approved and offered by BOSCH.
Simplex / Singel Controller systems E2800 can be and should be used with Frimware 11.50.2 which can and must be requested via BOSCH Technical support. See also the BOSCH product catalogue for latest information status update 2021-02-01: https://resources-boschsecurity-cdn.azureedge.net/public/documents/DSA_E2800_Release_Note_enUS_79775057035.pdf These data are currently updated and therefore this KB-article will be enhanced and updated between 2021-01-10 and 2021-02-08.
A basic step when dealing with technical questions for NetApp E2800 iSCSI storage systems is, to register the system with Serial No. by using the following registration form: https://resources-boschsecurity-cdn.azureedge.net/public/documents/Registration_DIP_600_Special_enUS_9007222920871435.docx NetApp and BOSCH want to inform, that we expect a new SANtricity firmware, provided by NetApp, which will contains a fix for the controller random reboot issue which were reported back to Bosch by some customers. The firmware version / SANtricitiy firmware for E2800 iSCSI models is delayed and planned for May 2020.
As this NetApp SANtricity firmware is desigend by NetApp for BOSCH use cases with 24/7 video recording + replay and even worst case situations during a running reconstruct, this Firmware is not yet available via NetApp NOW. NetApp agreed with BOSCH to cover all improvements and fixes in a global and public release around June/July 2020. BOSCH will inform here in this article about the comming release when available.
The attached documents show the different design options that are available when designing a BVMS system.
For BVMS 10.1, we have extended the blueprints with Person Identification, the Access Management System, and updated the Enterprise overview.
Newer NetApp controller firmware might be approved and qualified by BOSCH. This article only refers to a introduction in time of a 11.50 firmware generation. (e.g. status 08/2020 there is an 11.60.2 qualified by BOSCH) Details can always be found at the BOSCH product catalogue: https://commerce.boschsecurity.com/us/en/DSA-E-Series-E2800-60-bay/p/34919710859/ and
Potential for Data Inconsistency Issues on E-Series Storage Systems
With the controller firmware 11.50.1 and newer version an issues are fixed by NetApp, which enforce an immediate update of all E2800 iSCSI Storage Systems (their controllers). In July 2019 BOSCH announced the need to update from all former NetApp controller firmware 08.30.40.00 to newer versions. These newer versions should be certified and approved by BOSCH. In 2019 there was also an announcement by NetApp. All customers owning a NetApp storage system with valid warranty agreement and registered unit do have access to the public announcemnt of NetApp. This was announced by NetApp on their support websites (see https://mysupport.netapp.com/).
Bosch approval 31st of January 2018
Bosch approval 7th of May 2019
Bosch approval 20th of September 2019
newer NetApp firmware versions:
NetApp Firmware 08.30.40.00 no longer allowed for usage
NetApp Firmware: 11.50.R1 no longer allowed for usage
NetApp Firmware: 11.50.2 no longer allowed for usage
BOSCH qualifies and approves new global NetAp frmware vversions for E2800 controllers. Please always use the latest approved NetApp global firmware where possible
Please follow our BOSCH Knowledge Base and monitor for updates at this article for important news. The BOSCH submodel ID 356 ensures that the NetApp system is optimized for 24/7 video recording.
Products that were affected:
All DSA E-Series (E2800 12-bay) and DSA E-Series (E2800 60-bay)
Example product variants:
DSA-N2E8X4-12AT Base unit 12x4TB High-performance and high-capacity storage system base unit with iSCSI disk arrays, single controller. DSA E2800, 12 x 4 TB HDD, Order number DSA-N2E8X4-12A
DSA-N2C8X4-12AT Dual controller unit 12x4TB High-performance and high-capacity storage system base unit with iSCSI disk arrays, dual controller. DSA E2800, 12 x 4 TB HDD, Order number DSA-N2C8X4-12AT
Note: There are other models with various HDD capacity (e.g. 8TB harddrives and larger) available For more details visit the BOSCH product website
Summary of issue
NetApp® has become aware of issues that could occur when an E-Series controller reboots at certain points during the evacuation of data from a drive that is performed as part of the sequence when a drive is being failed by the controller. As a result, there is a small possibility of data inconsistency on controllers running certain versions of E-Series SANtricity® OS controller software.
First announced in 06/2019: The issues are possible on E-Series controllers running 8.30, 8.40, 11.30, 11.40, and 11.50 versions of E-Series SANtricity OS controller software. The overall probability of these issues occurring is very low, but RAID 1 volumes have a higher probability of encountering the issues than other RAID levels or NetApp Dynamic Disk Pools. These issues do not affect traditional RAID volume groups that have no global hot spare drives because drive evacuation does not occur in this configuration. For information about fixes in these releases, view the readme notes for each revision release.
No workaround available. Controller Firmware update is needed. HDD Firmware update is not in relation to the descdribed issue here, but it is in general recommended to install the latest offered HDD firmware that is recommended by BOSCH and NetApp. The same applies for HDD firmware.
Solution: Update the NetApp Controller Firmware
Upgrade E-Series SANtricity OS controller software to the latest applicable revision release for each platform as soon as possible. Please reach out to your local BOSCH technical team and BOSCH BT-SC/ETP-MKP team to ensure that the relevant NetApp Firmware has been approved to be usable for 24/7 video recording and replay use case. Even the global NetApp Firmware can be installed on BOSCH submodel ID 356
Where to get the Controller Firmware
All customers with a full NetApp NOW support account and valid warranty on their system can download the controller firmware by using the own NetApp user account. The verison offered at the NetApp NOW platform should be certified and allowed by BOSCH for video use cases. See also: https://community.boschsecurity.com/t5/Security-Video/TB-VS-date-2020-04-22-New-Firmware-11-50-3R1P3-is-available-for/ta-p/12778
In caase the download option is not offered to one of our customers, we kindly ask to make sure that the unit is registered on the company and that the NetApp NOW account belongs to the company that has registered the unit. In all other cases, please recout out to the BOSCH Support organisation or local Bosch Technical Support desk.
A NetApp SANtricity firmware / controller firmware package for E2800 model can be downloaded from the NetApp NOW website. Revisions can be found at https://mysupport.netapp.com/NOW/cgi-bin/software/ See also: https://mysupport.netapp.com/products/web/ECMLP2854621.html
Additional information is available at NetApp and accessible after user registration. First go to https://mysupport.netapp.com/ to register. Product Release Notes from NetApp: https://mysupport.netapp.com/ecm/ecm_download_file/ECMLP2842060
NetApp Support Bulletin, please view the following URL: https://kb.netapp.com/app/answers/answer_view/a_id/1086731 (As the URL can change any time by NetApp, search for the KB id 1086731)
Steps to update
The E-Series SANtricity ® OS package includes data for simplex controller and duplex controller systems and the firmware file itself.
NVSRAM File for for duplex.
NVSRAM file for simplex
and the controller firmware
Download the latest SANtricity OS software files from the NetApp Support Site to your management client.
From SANtricity System Manager, select Support > Upgrade Center .
In the area labeled “SANtricity OS Software upgrade,” click NetApp Support .
On the NetApp Support Site, click the Downloads tab, and then select Software .
Locate E-Series/EF-Series SANtricity OS (Controller Firmware) .
For the platform, select E2800 , and click Go!
Select the version of SANtricity OS (Controller Firmware) you want to install, and click View & Download .
Follow the online instructions to complete the file download.
Attention: Risk of data loss or risk of damage to the storage array — Do not make changes to the storage array while the upgrade is occurring. Maintain power to the storage array.
BVMS Lite and BVMS Viewer are BVMS editions that you can download and activate free-of-charge.
How can I set-up a basic (live and recorded video) BVMS system?
First, you need to download the software package, active the BVMS Lite license and install the software. This is described in this article: BVMS - Activating a license.
Second, you need to prepare an iSCSI environment which is suitable for recording video. Any Windows Server based operating system will do. This is described in this article: BVMS - Configuring a Microsoft iSCSI target.
Last, you need to add cameras to the system and start the recording. This is described in this youtube video: How to add a new camera using Configuration Client (BVMS).
Now, have a look at the Operator Client quick guide and you're ready to go!
Where can I get more information on advanced functionality?
Once the software (configuration client or operator client) is running you can press F1 at any time to open the embedded software help! All of the advanced functionality BVMS offers is explained in the help files.
Status February 25, 2017
BOSCH offers and confirms that since February 2017 the firmware version 3.180.05-1562 can be used with the LSI MegaRAID Controller. In case there are newer versions supported and recommended with BOSCH DIVAR IP this will be announced in cooperation with the Product Management.
For DIVAR IP units (e.g. DIVAR IP 6000) BOSCH BT Security & Safety Systems offers an firmware update for the internal MegaRADI Controller.
It could happen that in the Megaraid Software the user can see the following: potential non-optimal configuration due, PD commissioned as Emergency Spare error.
Here we provide the steps to update a DIVAR IP system:
Using MegaRAID Storage Manager utility under OS
1) Open MSM, Right click on Supermicro MegaRAID controller to be updated and click Update Controller Firmware
2) Click Browse to search for new firmware
3) Select the new MegaRAID controller firmware
4) Click OK to continue
5) Check the “Confirm” box and click OK to continue
*** Wait for around 1~2 minutes to complete
6) Click OK once firmware update completed
7) Reboot the system and check firmware version in controller OPROM banner during boot. At the section Revision you can see version 3.180.05-1562
😎 Check firmware version using MSM in OS When selecting the "Supermicro SMC 2208 (Bus1, Dev 0) device you can see on the right the "Firmware Version" = 3.180.05-1562
The firmware *.rom filw is available via the following link - inside the ZIP file:
This Firmware Version is subject of change. Any new version must be announce by BOSCH PRM responsible for DIVAR IP via the BT-ST/ETP-MKP2 organisation and BOSCH support at BT-ASA here in the BOSCH Knowledge Base.
The RAID combines two or more physical drives into a logical unit presented as a single hard drive to the operating system. There are currently six basic RAID levels: RAID 0, RAID 1, RAID 0+1, RAID 1+0, RAID 3, RAID 4, RAID 5 and RAID 6.
The scope of this article is to provide basic information for the levels RAID 5 and RAID 6 and to compare them from point of view of performance and security.
Hot spare is a drive that acts as a stand by drive in RAID 1, RAID 5 or RAID 6 volume. It is fully functional drive that contains no data and is not used during normal operation. If a drive from the volume fails, the controller reconstructs the data from the failed drive to the hot spare drive.
A RAID 5 array is designed to protect against the failure of a single disk within the array. Because of the way that RAID 5 works, the total capacity of one disk is lost to overhead. If, for example, a RAID 5 array contained five 10TB disks, then the array’s usable capacity would be 40TB.
A RAID 5 (with Hot Spare disk) array can be configured to treat one of the disks as a hot spare. Then one of the disks is reserved as a replacement in the event that a disk fails. For the above example with five 10TB disks, this would decrease the example array’s usable capacity to 30TB.
A RAID 6 array is designed to protect against two simultaneous disk failures. However, the price for this extra protection is that two disks' worth of capacity is lost to overhead. As such, a RAID 6 array made up of five 10TB disks would have a usable capacity of 30TB because 20 TB is lost to overhead.
The performance during Normal Operation is measured in IOPS (Input/output operations per second) and as a sum for all the disks (excluding the Hot Spares and decreased for writing parity data) in the array. As a rule of the thumb, the higher the overhead associated with writing parity data (in the above example RAID 5 with Hot Spare causes the same overhead like RAID 6) the lower the IOPS.
The reason for implementing RAID arrays is to secure the data. The level of protection does not directly correlate with the overhead. From the above example both RAID 5 with Hot Spare and RAID 6 have same capacity, but offer different level of protection. In case of failure of RAID 5 array with Hot Spare, the Hot Spare is activated and the rebuild process start immediately. The system can recover from a single disk failure and during the recovery, process is vulnerable to second disk failure. Therefore, RAID 5 and RAID 5 with Hot Spare disk offer the same level of protection – single disk failure. In contrast, if a disk fails at RAID 6 array, the recovery will start only after the faulty disk is replaced manually. However, if during the recovery process second disk fails, the RAID 6 array will stay functional.
The password for the local admin and diagnostic port is set to the same starting with E2800 controller firmware 11.40.2 or later.
In case a user does not know his password any longer, it is not possible to reset and re-configure this password by using the diagnostic port (serial port on the rear side near the uplink ports of the controller). With controller firmware older than 11.40.2 it was possible to reset the Administrator Password needed for configuration. But older NetApp controller firmware cannot and must not be used any longer due to other technical reasons. When accessing the Management Port IP to enter the WEB GUI of the NetApp E2800 system for configuraiton or support data collection, the Administrator Password is required. For the time being this is reported to NetApp to work out a solution for all NetApp customers and BOSCH.
Solution and Procedure:
Right now, BOSCH and our customers have no documented instruciton or procedure to remove / reset the password. We are working ont hat with high preassure and awareness. We kindly ask all customers requiring access to the configuraiton but having not the Administrator Password on hand for configuraiton, to reach out to BOSCH support and ask to sent the requewst immediately up to a Levle 3 Ticket to the Gatekeeper team at BOSCH BT-SC/ETP-MKP2 or BT-VS Support in the BU.
In many cases a minimum and maximum retention time needs to be defined in a video surveillance systems due to legal requirements. While the minimum retention time defines the time period for how long video recordings need to be stored, the maximum retention time defines after which period of time the recordings have to be deleted. Thus, the minimum retention time is going to influence the amount of storage needed. The higher the minimum retention time the more storage space is required.
Hence, the storage space needs to be large enough to store the recordings for the minimum retention. For the maximum retention time this doesn’t have to be the case. Still users might be confused why recordings gaps might appear sort-of randomly, if the system does not have enough storage space to keep all recordings until the maximum retention time is reached. To understand what is going on we have to remember the principle of the VRM block assignment first.
For each camera in the system the BOSCH Video Recording Manager (VRM) generates a list of recording blocks (LUNs) on which the camera can next record. Therefore, the VRM makes an estimation based on the data rate and the amount of data of each camera in the system (global optimization). Basically, the VRM predicts when which camera needs a new block and always lists the block which will be the oldest block at the time the camera needs to record on the next block. One could think of it as a “next oldest block” estimation done by the VRM. But the prediction of the VRM might differ from the reality (mainly because of variance in recording bitrate) and this can cause recording gaps if the storage space is not large enough to support the maximum retention time.
Let’s have a closer look on the following two cases:
Sufficient storage space for maximum retention time
Insufficient storage space for maximum retention time
Sufficient storage space for maximum retention time
In case of sufficient storage space to fulfill the maximum retention time for every camera in the system no random recording gaps will appear, because the VRM will always assign a block containing recordings, which are older than the maximum retention time. Thus, for each camera the recording blocks will be kept until the maximum retention time is reached as illustrated in Figure 1.
Figure 1: Enough storage space to cover the maximum retention time for each camera of the system
Insufficient storage space for maximum retention time
In case the system is designed such that the storage space is not large enough to store all recordings from all cameras until the maximum retention time is reached, the VRM will of course still do its estimation and predict the oldest recording block when a camera will ask for a new block. Assuming an ideal setup (with ideal network connection where each camera has the same data rate and all cameras record the same amount of video data simultaneously), the oldest block would always be assigned by the VRM. Hence, no recordings gaps should appear for recordings older than the minimum retention time, compare Figure 2. This is was most customers falsely assume or expect.
Figure 2: Customer expectation of the system behaviour in case of insufficient storage space to cover the maximum retention time for each camera of the system
However, in reality the stated assumptions do not apply. Network connection, data rate, amount of recorded video data, etc. varies. Thus, the “next oldest block” estimation of the VRM can differ from reality. Since each camera already got its block list from the VRM and records according to this block list, it can happen that not the truly oldest block is used and recording gaps appear as shown in Figure 3.
Figure 3: System behaviour in case of insufficient storage space to cover the maximum retention time for each camera of the system
How to avoid or minimize this effect
To avoid this effect of random recording gaps simply add enough storage to your system. To get the best out of your system in terms of storage usage, the optimum would be to set the maximum retention time to storage limit, see Figure 4, but that is almost impossible to realize in practice.
Figure 4: In principle a maximum retention time set to the storage limit would avoid random recording gaps
Option 1 to minimize the effect in practice is to estimate the maximum retention time so that it will not exceed the storage limit of the system as illustrated in Figure 5.
Figure 5: Maximum retention very close to the storage limit will minimize the random recording gaps
Another less recommended option is to set a smaller time difference between the minimum and maximum retention time. But especially when the minimum retention time is shifted closer to the maximum retention time that introduces the risk that the VRM cannot free up storage space in case the minimum retention time is reached, which might result in a recording stop. Thus, we recommend to go for the first option.
One last hint: Changing the retention time on a running system is not going to influence the retention time of already recorded blocks. but will of cousre only be applied to new recorded video footage. Hence, changing the retention time is no option for an immediate change of required storage.
This article describes how to set LUNs on a target to Read Only via Configuration Manager.
If you want to move a Storage from one VRM to another, this procedure is needed in Order to retain your video footage until the desired Retention Time is over. All you need is the Configuration Manager on a PC in the same Network as the VRM. If you don’t have the Configuration Manager, you can download it here.
Please be aware that setting the LUNs to Read only will decrease the Capacity of the VRM, which might lead to a lower Minimum Retention Time then desired. Calculate upfront, if the available Storage is enough!
Start the Configuration Manager and add the local VRM to the System.
Change to the Tab My devices and select the Target on the Storage Device you want to move.
Set all LUNs to the Type Read Only. It should look like this then.
Now press the button Set and acknowledge it in the next window.
After, the window looks like this and the LUNs are now in Read Only Mode.
Now you need to wait until your desired Retention time passed (eg 30 days), afterwarts you can remove the Storage from the VRM and add it to another VRM (In this Process the Storage needs to be Factory defaulted to work).
In this article we cover the following basic questions:
How to Factory Default a NetApp E-Seires unit by using BOSCH tools and NetApp WEB GUI?
How to download the NetApp Support Bundle of a reachable online NetApp Storage Array
How to Factory Default a NetApp E-Series that you bought via BOSCH sales channel? Using the BOSCH Configuraiton Manager 6.00 or 6.01 the NetApp models E2600, E2700 and E2800 can be managed. Especially we recommend to use the BOSCH Configuration Manager version 06.01 when using a NetApp E2800 to have all models (also former models E2700 and E2600) supported. Also in the BOSCH Configuraiton Manager 6.01 the Basic configuration for the initial setup is helpful as well as the Factoury Default option and Clear option is available. Of cours the NetApp E2800 offers also a WebGUI by using the IP of the management port of a controller. The following screenshot made from the BOSCH Conmfiguration Manager shows the options available in the tab "Basic Configuration" when a NetApp E-Series (DSA E-Series) is already added to a VRM system in the Configuraiton Manager. The Button "Defaults" is used to trigger the "Reset Storage Array" mechanism of NetApp. At a E2800 all configuration is eliminated but the Management Port IP still remains to ensure that the WEB GUI of NetApp and the communiction of the BOSCH Configuraiton Manager can still work.
How to download the NetApp Support Bundle of a reachable online NetApp Storage Array In the Tab "My Devices" of the Configuration Manager 6.01 and newer versions you can also now download the Support Bundle (collection of logfiles) from a NetApp E2800 stroage system. Select the NetApp E2800 in your device tree that is added to a VRM system and right-click on it. There you find "File Download" and "Maintenance Log..." - Choose that to download the NetApp Support Bundle.
In August 2018 (10-08-2018) the VRM version 03.71.0029 was released.
The Video Recording Manager 03.71.0029 is fully supported with BVMS 8.0 and Product Management of BVMS and VRM recommend to use this VRM version instead of the former Release version of VRM 03.70.0056.
Changes / Bug Fixes:
One of the main bugfix reasons to use VRM 03.71.0029 is a fix in regards to correct display and replay of recorded clips in continous and alarm recording mode. This fix is listed on page 2 of hte attached Relase Letter.
For Troubleshooting and support reasons it is essential to double-check a reported gap in recording and to analyze on Level 2 and Level 3 support side what circoumstanced could lead to a video gap. See in the following chapter what kind of data a trained BOSCH partner, Installer or Video expert should provide to the BOSCH Technical Support to advise on next steps. The data described here below can and should be collected before the software VRM is changed and updated to a latger version. Note: Video Recording Manager version 03.71.0029 is not the latest available version of VRM, but in combination with other 3rd party implemantation or usage of special BOSCH VMS software version (e.g. BVMS 8.0) this VRM version 03.71.0029 may be required.
VRM logging to collect for in depth expert troubleshooting
In certain situations and troubleshooting scenarios extended logfiles might be required. The BOSCH Level 1 and BOSCH Level 2 team will assist all users on how to collect these data and cooperate with the BOSCH Level 3 Support where needed.
Backup the configuration of BVMS and VRM (BVMS elements and VRM config.xml file)
Enable debug logging of the VRM Depending on the used Configuraiton Software the debug logging need to be enabled to get an extended logging informaiton. In case a gap in recording happened in the past it is anyhow helpful to enable the debug logging for a defined time for future incidents. To analyze the already occured video recording gap the available VRM logging must be collected.
Depending on 32-Bit or 64-Bit version of the VRM software, the loggings are found in the "primary" or "secondary" sub-directory structure. Surch for ...\Bosch\Video Recording Manager\VRM Server folder at your VRM server to find a similar directory view like shown in the screenshot here below:
Inside the directory "log" the standard logging data of the VRM are found and need to bre collected. In addition to the standard logfiles of the relevant day, the debug logging from a defined time period need to be provided in case debug logging was enabled prior to an incident.
Here the "debug" logfiles are found: The debug loggins are saved in a special directory "debug".
The Spanhistory logging does provide details of the storage usage. It describes which internal "storage block" was used. It describes the IP of the target, the LUN and the block used for a recording. The Spanhistory of the day where the recording was done and shows issues (e.g. the video reocriding gap), the related spanhistory logfiles is required.
As seen in an earlier screenshot above the VRM configuration file is found in the directory: ...\Bosch\Video Recording Manager\VRM Server\primaryAll versions of the config.xml need to be provided to the Technical Support.