Potential for Data Inconsistency Issues on E-Series Storage Systems
With the controller firmware 11.50.1 and newer version like NetApp controller firmware 11.50.2 an issues are fixed by NetApp, which enforce an immediate update of all E2800 iSCSI Storage Systems (their controllers). In July 2019 BOSCH announced the need to update from all former NetApp controller firmware 08.30.40.00 to newer versions. At that time the Firmware for NetApp E2800 controller recommended was version NetApp Firmware 11.50R1. This is also announced by NetApp on their support websites (see https://mysupport.netapp.com/).
Bosch approval 31st of January 2018
Bosch approval 7th of May 2019
Bosch approval 20th of September 2019
NetApp Firmware 08.30.40.00 no longer allowed for usage
NetApp Firmware: 11.50.R1 no longer allowed for usage
NetApp Firmware: 11.50.2 currently shipped by Netapp via BOSCH
NetApp Firmware: 11.50.2P1 11.50.3 11.60
Please follow our BOSCH Knowledge Base and monitor for updates at this article for important news. Eventhough NetApp inbuild functions for BOSCH video optimized recording 24/7 and generic firmware is used at E2800 iSCSI storage systems, BOSCH is evaluating new firmware versions provided by NetApp on a regular bases. The BOSCH submodel ID 356 ensures that the NetApp system is optimized for 24/7 video recording.
All DSA E-Series (E2800 12-bay) and DSA E-Series (E2800 60-bay)
Example product variants:
DSA-N2E8X4-12AT Base unit 12x4TB High-performance and high-capacity storage system base unit with iSCSI disk arrays, single controller. DSA E2800, 12 x 4 TB HDD, Order number DSA-N2E8X4-12A
DSA-N2C8X4-12AT Dual controller unit 12x4TB High-performance and high-capacity storage system base unit with iSCSI disk arrays, dual controller. DSA E2800, 12 x 4 TB HDD, Order number DSA-N2C8X4-12AT
Note: There are other models with various HDD capacity (e.g. 8TB harddrives and larger) available For more details visit the BOSCH product website
Summary of issue
NetApp® has become aware of issues that could occur when an E-Series controller reboots at certain points during the evacuation of data from a drive that is performed as part of the sequence when a drive is being failed by the controller. As a result, there is a small possibility of data inconsistency on controllers running certain versions of E-Series SANtricity® OS controller software.
First announced in 06/2019: The issues are possible on E-Series controllers running 8.30, 8.40, 11.30, 11.40, and 11.50 versions of E-Series SANtricity OS controller software. The overall probability of these issues occurring is very low, but RAID 1 volumes have a higher probability of encountering the issues than other RAID levels or NetApp Dynamic Disk Pools. These issues do not affect traditional RAID volume groups that have no global hot spare drives because drive evacuation does not occur in this configuration. For information about fixes in these releases, view the readme notes for each revision release.
No workaround available. Controller Firmware update is needed. HDD Firmware update is not in relation to the descdribed issue here, but it is in general recommended to install the latest offered HDD firmware that is recommended by BOSCH and NetApp. The same applies for HDD firmware.
Solution: Update the NetApp Controller Firmware
Upgrade E-Series SANtricity OS controller software to the latest applicable revision release for each platform as soon as possible.
status 06/2019: controller firmware version minimum 11.50R1
status 10/2019: controller firmware version to be used 11.50.2
Where to get the Controller Firmware
All customers with a full NetApp NOW support account can download the controller firmware by using the own NetApp user account.
All other customers are requested to contact the BOSCH Support organisation. The controller firmware package can be downloaded fro mthe NetApp website directly. Revisions can be found at https://mysupport.netapp.com/NOW/cgi-bin/software/ See also: https://mysupport.netapp.com/products/web/ECMLP2854621.html
Additional information is available at NetApp and accessible after user registration. First go to https://mysupport.netapp.com/ to register. Product Release Notes from NetApp: https://mysupport.netapp.com/ecm/ecm_download_file/ECMLP2842060
NetApp Support Bulletin, please view the following URL: https://kb.netapp.com/app/answers/answer_view/a_id/1086731 (As the URL can change any time by NetApp, search for the KB id 1086731)
Steps to update
The E-Series SANtricity ® OS package includes data for simplex controller and duplex controller systems and the firmware file itself.
NVSRAM File for for duplex.
NVSRAM file for simplex
and the controller firmware
Download the latest SANtricity OS software files from the NetApp Support Site to your management client.
From SANtricity System Manager, select Support > Upgrade Center .
In the area labeled “SANtricity OS Software upgrade,” click NetApp Support .
On the NetApp Support Site, click the Downloads tab, and then select Software .
Locate E-Series/EF-Series SANtricity OS (Controller Firmware) .
For the platform, select E2800 , and click Go!
Select the version of SANtricity OS (Controller Firmware) you want to install, and click View & Download .
Follow the online instructions to complete the file download.
Attention: Risk of data loss or risk of damage to the storage array — Do not make changes to the storage array while the upgrade is occurring. Maintain power to the storage array.
In many cases a minimum and maximum retention time needs to be defined in a video surveillance systems due to legal requirements. While the minimum retention time defines the time period for how long video recordings need to be stored, the maximum retention time defines after which period of time the recordings have to be deleted. Thus, the minimum retention time is going to influence the amount of storage needed. The higher the minimum retention time the more storage space is required.
Hence, the storage space needs to be large enough to store the recordings for the minimum retention. For the maximum retention time this doesn’t have to be the case. Still users might be confused why recordings gaps might appear sort-of randomly, if the system does not have enough storage space to keep all recordings until the maximum retention time is reached. To understand what is going on we have to remember the principle of the VRM block assignment first.
For each camera in the system the BOSCH Video Recording Manager (VRM) generates a list of recording blocks (LUNs) on which the camera can next record. Therefore, the VRM makes an estimation based on the data rate and the amount of data of each camera in the system (global optimization). Basically, the VRM predicts when which camera needs a new block and always lists the block which will be the oldest block at the time the camera needs to record on the next block. One could think of it as a “next oldest block” estimation done by the VRM. But the prediction of the VRM might differ from the reality (mainly because of variance in recording bitrate) and this can cause recording gaps if the storage space is not large enough to support the maximum retention time.
Let’s have a closer look on the following two cases:
Sufficient storage space for maximum retention time
Insufficient storage space for maximum retention time
Sufficient storage space for maximum retention time
In case of sufficient storage space to fulfill the maximum retention time for every camera in the system no random recording gaps will appear, because the VRM will always assign a block containing recordings, which are older than the maximum retention time. Thus, for each camera the recording blocks will be kept until the maximum retention time is reached as illustrated in Figure 1.
Figure 1: Enough storage space to cover the maximum retention time for each camera of the system
Insufficient storage space for maximum retention time
In case the system is designed such that the storage space is not large enough to store all recordings from all cameras until the maximum retention time is reached, the VRM will of course still do its estimation and predict the oldest recording block when a camera will ask for a new block. Assuming an ideal setup (with ideal network connection where each camera has the same data rate and all cameras record the same amount of video data simultaneously), the oldest block would always be assigned by the VRM. Hence, no recordings gaps should appear for recordings older than the minimum retention time, compare Figure 2. This is was most customers falsely assume or expect.
Figure 2: Customer expectation of the system behaviour in case of insufficient storage space to cover the maximum retention time for each camera of the system
However, in reality the stated assumptions do not apply. Network connection, data rate, amount of recorded video data, etc. varies. Thus, the “next oldest block” estimation of the VRM can differ from reality. Since each camera already got its block list from the VRM and records according to this block list, it can happen that not the truly oldest block is used and recording gaps appear as shown in Figure 3.
Figure 3: System behaviour in case of insufficient storage space to cover the maximum retention time for each camera of the system
How to avoid or minimize this effect
To avoid this effect of random recording gaps simply add enough storage to your system. To get the best out of your system in terms of storage usage, the optimum would be to set the maximum retention time to storage limit, see Figure 4, but that is almost impossible to realize in practice.
Figure 4: In principle a maximum retention time set to the storage limit would avoid random recording gaps
Option 1 to minimize the effect in practice is to estimate the maximum retention time so that it will not exceed the storage limit of the system as illustrated in Figure 5.
Figure 5: Maximum retention very close to the storage limit will minimize the random recording gaps
Another less recommended option is to set a smaller time difference between the minimum and maximum retention time. But especially when the minimum retention time is shifted closer to the maximum retention time that introduces the risk that the VRM cannot free up storage space in case the minimum retention time is reached, which might result in a recording stop. Thus, we recommend to go for the first option.
One last hint: Changing the retention time on a running system is not going to influence the retention time of already recorded blocks. but will of cousre only be applied to new recorded video footage. Hence, changing the retention time is no option for an immediate change of required storage.
Status February 25, 2017
BOSCH offers and confirms that since February 2017 the firmware version 3.180.05-1562 can be used with the LSI MegaRAID Controller. In case there are newer versions supported and recommended with BOSCH DIVAR IP this will be announced in cooperation with the Product Management.
For DIVAR IP units (e.g. DIVAR IP 6000) BOSCH BT Security & Safety Systems offers an firmware update for the internal MegaRADI Controller.
It could happen that in the Megaraid Software the user can see potential non-optimal configuration due, PD commissioned as Emergency Spare error.
There is an article as well from hardware partner: http://www.supermicro.com.tw/support/faqs/faq.cfm?faq=19518
Here we provide the steps to update a DIVAR IP system:
Using MegaRAID Storage Manager utility under OS
1) Open MSM, Right click on Supermicro MegaRAID controller to be updated and click Update Controller Firmware
2) Click Browse to search for new firmware
3) Select the new MegaRAID controller firmware
4) Click OK to continue
5) Check the “Confirm” box and click OK to continue
*** Wait for around 1~2 minutes to complete
6) Click OK once firmware update completed
7) Reboot the system and check firmware version in controller OPROM banner during boot. At the section Revision you can see version 3.180.05-1562
😎 Check firmware version using MSM in OS When selecting the "Supermicro SMC 2208 (Bus1, Dev 0) device you can see on the right the "Firmware Version" = 3.180.05-1562
The firmware *.rom filw is available via the following link - inside the ZIP file:
This Firmware Version is subject of change. Any new version must be announce by BOSCH PRM responsible for DIVAR IP via the BT-ST/ETP-MKP2 organisation and BOSCH support at BT-ASA here in the BOSCH Knowledge Base.
BVMS Lite is a BVMS edition which can be downloaded and activated free-of-charge. How can I set-up a basic (live and recorded video) BVMS Lite system?
First, you need to download the software package, active the BVMS Lite license and install the software. This is described in this article: BVMS - Activating a license.
Second, you need to prepare an iSCSI environment which is suitable for recording video. Any Windows Server based operating system will do. This is described in this article: BVMS - Configuring a Microsoft iSCSI target.
Last, you need to add cameras to the system and start the recording. This is described in this youtube video: How to add a new camera using Configuration Client (BVMS).
Now, have a look at the Operator Client quick guide and you're ready to go!
Where can I get more information on advanced functionality?
Once the software (configuration client or operator client) is running you can press F1 at any time to open the embedded software help! All of the advanced functionality BVMS offers is explained in the help files.
This article describes how to set LUNs on a target to Read Only via Configuration Manager.
If you want to move a Storage from one VRM to another, this procedure is needed in Order to retain your video footage until the desired Retention Time is over. All you need is the Configuration Manager on a PC in the same Network as the VRM. If you don’t have the Configuration Manager, you can download it here.
Please be aware that setting the LUNs to Read only will decrease the Capacity of the VRM, which might lead to a lower Minimum Retention Time then desired. Calculate upfront, if the available Storage is enough!
Start the Configuration Manager and add the local VRM to the System.
Change to the Tab My devices and select the Target on the Storage Device you want to move.
Set all LUNs to the Type Read Only. It should look like this then.
Now press the button Set and acknowledge it in the next window.
After, the window looks like this and the LUNs are now in Read Only Mode.
Now you need to wait until your desired Retention time passed (eg 30 days), afterwarts you can remove the Storage from the VRM and add it to another VRM (In this Process the Storage needs to be Factory defaulted to work).
In this article we cover the following basic questions:
How to Factory Default a NetApp E-Seires unit by using BOSCH tools and NetApp WEB GUI?
How to download the NetApp Support Bundle of a reachable online NetApp Storage Array
How to Factory Default a NetApp E-Series that you bought via BOSCH sales channel? Using the BOSCH Configuraiton Manager 6.00 or 6.01 the NetApp models E2600, E2700 and E2800 can be managed. Especially we recommend to use the BOSCH Configuration Manager version 06.01 when using a NetApp E2800 to have all models (also former models E2700 and E2600) supported. Also in the BOSCH Configuraiton Manager 6.01 the Basic configuration for the initial setup is helpful as well as the Factoury Default option and Clear option is available. Of cours the NetApp E2800 offers also a WebGUI by using the IP of the management port of a controller. The following screenshot made from the BOSCH Conmfiguration Manager shows the options available in the tab "Basic Configuration" when a NetApp E-Series (DSA E-Series) is already added to a VRM system in the Configuraiton Manager. The Button "Defaults" is used to trigger the "Reset Storage Array" mechanism of NetApp. At a E2800 all configuration is eliminated but the Management Port IP still remains to ensure that the WEB GUI of NetApp and the communiction of the BOSCH Configuraiton Manager can still work.
How to download the NetApp Support Bundle of a reachable online NetApp Storage Array In the Tab "My Devices" of the Configuration Manager 6.01 and newer versions you can also now download the Support Bundle (collection of logfiles) from a NetApp E2800 stroage system. Select the NetApp E2800 in your device tree that is added to a VRM system and right-click on it. There you find "File Download" and "Maintenance Log..." - Choose that to download the NetApp Support Bundle.
In August 2018 (10-08-2018) the VRM version 03.71.0029 was released.
The Video Recording Manager 03.71.0029 is fully supported with BVMS 8.0 and Product Management of BVMS and VRM recommend to use this VRM version instead of the former Release version of VRM 03.70.0056.
Changes / Bug Fixes:
One of the main bugfix reasons to use VRM 03.71.0029 is a fix in regards to correct display and replay of recorded clips in continous and alarm recording mode. This fix is listed on page 2 of hte attached Relase Letter.
For Troubleshooting and support reasons it is essential to double-check a reported gap in recording and to analyze on Level 2 and Level 3 support side what circoumstanced could lead to a video gap. See in the following chapter what kind of data a trained BOSCH partner, Installer or Video expert should provide to the BOSCH Technical Support to advise on next steps. The data described here below can and should be collected before the software VRM is changed and updated to a latger version. Note: Video Recording Manager version 03.71.0029 is not the latest available version of VRM, but in combination with other 3rd party implemantation or usage of special BOSCH VMS software version (e.g. BVMS 8.0) this VRM version 03.71.0029 may be required.
VRM logging to collect for in depth expert troubleshooting
In certain situations and troubleshooting scenarios extended logfiles might be required. The BOSCH Level 1 and BOSCH Level 2 team will assist all users on how to collect these data and cooperate with the BOSCH Level 3 Support where needed.
Backup the configuration of BVMS and VRM (BVMS elements and VRM config.xml file)
Enable debug logging of the VRM Depending on the used Configuraiton Software the debug logging need to be enabled to get an extended logging informaiton. In case a gap in recording happened in the past it is anyhow helpful to enable the debug logging for a defined time for future incidents. To analyze the already occured video recording gap the available VRM logging must be collected.
Depending on 32-Bit or 64-Bit version of the VRM software, the loggings are found in the "primary" or "secondary" sub-directory structure. Surch for ...\Bosch\Video Recording Manager\VRM Server folder at your VRM server to find a similar directory view like shown in the screenshot here below:
Inside the directory "log" the standard logging data of the VRM are found and need to bre collected. In addition to the standard logfiles of the relevant day, the debug logging from a defined time period need to be provided in case debug logging was enabled prior to an incident.
Here the "debug" logfiles are found: The debug loggins are saved in a special directory "debug".
The Spanhistory logging does provide details of the storage usage. It describes which internal "storage block" was used. It describes the IP of the target, the LUN and the block used for a recording. The Spanhistory of the day where the recording was done and shows issues (e.g. the video reocriding gap), the related spanhistory logfiles is required.
As seen in an earlier screenshot above the VRM configuration file is found in the directory: ...\Bosch\Video Recording Manager\VRM Server\primaryAll versions of the config.xml need to be provided to the Technical Support.