NEC HYDRAstor HS8 Technical Overview and more

When in our country people hear the name of the company NEC, the first thing people of the older generation recall is this:

Those who are slightly younger are:

(under the cut a lot of pictures)

NEC Corporation

It sounds like "N-si." And in English it stands for N ippon E lectric C orporation. Although in our country they usually pronounce as NEC :).

Headquarters Apartment - Tokyo, Japan (in the picture)
Turnover for 2018 - $ 29 billion
Total employees 109,000+
The main plants are located in Japan, several plants in Malaysia
65% of turnover - sales within Japan
18 offices in 18 countries
4 research centers in 4 countries

The company was registered in Tokyo in 1899 and the first years it was engaged in the production of telephone equipment.

Today, NEC produces a fairly wide range of solutions, which can be found on the company's official website.

But I would like to tell the respected community about the NEC HYDRAstor deduplication storage line for backups, and most of all about the older HS8 family.

NEC HYDRAstor HS8

What surprised me from the very beginning was that the 5th generation of these systems was already released, and in Russia almost nothing is known about them. At least when communicating with people from vendors, integrators or customers, I often saw surprise as the first reaction when mentioning these solutions.

In the pictures below, the current NEC HYDRAstor HS8 lineup and the main characteristics of the models are presented.

HYDRAstor storage systems are systems built on the basis of scale-out architecture. Accordingly, NEC HS8 storage is a horizontally scalable system and can be expanded up to 165 nodes to increase capacity and performance. This allows you to start with a small, less expensive configuration (Nr: single-mode) and then expand it sequentially as your growing backup needs. As can be seen from the pictures, HS8 clusters can contain two types of nodes. Hybrid nodes (Hybrid Node), combining the functions of controllers (Accelerator Node), interacting with backup servers, as well as the data storage function (Storage Node). And separate storage nodes (Storage Node), which are used to increase the available disk space in the storage system. Depending on the system model and customer requirements, the ratio of Hybrid and Storage nodes in the cluster may vary.

^{2 - One (1) Terabyte (TB) = 1,000,000,000,000 bytes.} ^{3 - Assumption: full backup weekly and incremental backup daily, and 3 months retention period to achieve 20: 1 data reduction ratio;} ^{9 + 3 resiliency level.}

It is possible to configure nodes with 8TB disks upon customer request. The maximum performance per node according to the vendor is 61TB / h or 72TB / h at 1HN + 1SN, but this is subject to the use of deduplication at the source (Deduped Transfer) and data that lends itself well to deduplication. Like many vendors, this is more of a marketing figure - the so-called Best Case. And the worst case is 450MB / s per node on 12 SATA disks - provided that the data cannot be duplicated and compressed at all. Those. all non-compressed and non-deduplicated flow goes to SATA drives, which is called the Worst Case. Thus, the above described is a kind of extreme values, and the reality in each particular case will be somewhere in between.

The appearance of the HS8-5000 nodes in front is presented below. In fact, these are NEC's own servers with disks and network cards. CPU in all nodes of the E5-2600v3 generation.

Rear view of a hybrid node with internal and external ports on 1Gbe. Internal ports are used for intracluster connections between HydraStor nodes. External ports are used for external client connections to the cluster (for example, for backup media servers). On the right are two SAS disks that host the node's operating system.

Rear view of a hybrid node with internal and external ports on 1Gbe and two 10Gbe external ports.

Rear view of a hybrid node with internal and external ports on 1Gbe and four 10Gbe external ports.

Rear view of a stack of nodes with internal ports on 1Gbe.

Internal connections in a two-node cluster consisting of two hybrid nodes without the use of switches. When upgrading from a single-node configuration to two node clusters, you need to plan for 4 hours down time to reconfigure internal LSNs (Logical storage nodes). Data already written to the cluster is not lost.

Internal connections in a two-node cluster consisting of a hybrid node and a hundred-node node without using switches.

Schematic connection of a cluster consisting of more than 2 nodes using intra-cluster Ethernet switches.

For work inside the cluster, L2 NEC switches of own production are used. The following is an example of stacking NEC intra-cluster switches, which are used to connect the internal ports of HydraStor nodes. Depending on the size of the cluster, two to three 48 port Ethernet switches per telecommunication cabinet can be used. A total of 39 single-unit switches per cluster (including “Top-of-Rack” L3 switches). Switches are bundled with the cluster as part of a single solution.

An example of a port switching table on internal Ethernet switches.

HNxxxx-M: Connect to Maintenance Port of Hybrid Node

HNxxxx-1/2/3/4: Connect to iLAN1 / 2/3/4 Port of Hybrid Node

SNxxxx-M: Connect to Maintenance Port of Storage Node

SNxxxx-1/2/3/4: Connect to iLAN1 / 2/3/4 Port of Storage Node

L3xxxxx-xx: Connect to L3 switches

Through L3 switches connectivity between telecommunication cabinets in large configurations is provided.

Example of delivery of a ready-made NEC HydraStor cluster for installation in a customer's telecommunications cabinet. 15 nodes per cabinet, up to 11 cabinets in maximum configuration.

HYDRAstor uses inline global deduplication within the entire cluster - DataRedux .

During deduplication, data is divided into blocks of variable length, which allows you to more accurately respond to file changes during subsequent backups. For unique blocks, the LZO compression algorithm is additionally used to reduce the size of the stored data.

Changing some data in the middle of the file does not lead to the formation of new unique blocks, as with deduplication using blocks of constant length. The algorithm quite accurately determines the changed part of the file, and all subsequent blocks remain unchanged and deduplicated.

NEC HS8 storage offers a high level of redundancy of stored data that is superior to that used by conventional storage. Recorded data is divided into many blocks, adding special checksums to these blocks, and then distributed across disks at various nodes. The NEC HS8 storage system provides data recovery even in the event of failures at several points simultaneously. It can cope not only with simultaneous failures of 3 disks, but also of nodes, preventing loss of information in case of failure. To protect data, Erasure Coding algorithms are used.

Due to the fact that RAID and dedicated HS disks are not used on HYDRAStor, recovery after failures (disks or nodes) is much faster than when using RAID. Since the recovery goes from many to many, using a collective space for data recovery.

Stored data is periodically checked for integrity. If it turns out that the data is damaged, then it is automatically restored from the checksums.

Distributed Resilient Data (DRD) is:

1. User-defined protection against disk or node failure

Protection against 3 concurrent failures by default
The ability to use several levels (1-6) of protection for various applications in one system (different file systems with the desired level of protection for different applications)

2. Better protection at lower cost.

By default, 75% of the capacity for data storage and 25% for their protection is used.
1.5 times more reliable than RAID 6 with lower costs and faster recovery
No need for spare disks (Hot Spare), recovery goes to free space in the cluster

3. Faster recovery with less degradation in performance

Only data is recovered, not the entire disk
Data is restored in many-to-many mode, unlike RAID

The table below shows the dependence of the number of allowed disk or node failures for one and two-node configurations.

A different level of protection than the default (3) can be assigned to each newly created file system. The higher the level specified by Resilience, the more reliable the data is protected from loss, but also the greater the cost of disk space required for this protection. Compared with competitive solutions, approximately the same 25% of disk space costs are incurred by competitors when using Raid6, which allows you to lose no more than 2 disks from one Raid group. HydraStor can lose up to 3 disks without data loss with the same 25% of the disk space spent on protecting stored data.

A similar table, only for monogonode configurations.

Net capacity on HS8 nodes without deduplication and compression.

Adding new nodes to the cluster:

The addition of new HN (Hybrid Node) linearly increases cluster performance and capacity.
Adding new SN (Storage Node) linearly increases cluster capacity
When adding new nodes, the process of redistributing stored data to new nodes automatically starts

NEC HS8 storage capacity can be easily increased as the amount of data increases. NEC HS8 storage can safely store data for extended periods of time using distributed fault tolerance technology. Data automatically migrates from the old node to the new one during replacement. This allows you to change nodes in real time without creating inconvenience to users. Substitution of nodes also significantly reduces costs, prolonging the life of the existing system, compared with the complete replacement of the system with a new one. Nodes of three generations can simultaneously be in one cluster.

Universal Express I / O - lightweight I / O protocol NEC (installed by a separate plug-in on the backup media server)

1. Has less overhead than NFS / CIFS, which means it’s more productive

2. Works with many backup applications, for example: RMAN, NetWorker, NetVault etc.

3. Supports the following functions:

Data compression during transmission
Transmission Data Encryption
Network Load Balancing

Universal Express I / O Deduped Transfer is a separately licensed Universal Express I / O functionality that allows you to enable deduplication on the source.

Deduped Transfer has several limitations and will not work if the number of competitive backup jobs exceeds:

50 parallel backups to one Backup server (the 51st backup will be performed without deduplication on the source)
100 parallel backups per Hybryd node (the 101st backup will be performed without deduplication on the source)

Deduped Transfer requires an additional license for Universal Express I / O or OST Suite (OpenStorage - Suite).

Requirements for the Backup server using Deduped Transfer:

Synchronized Access Grid - allows you to enable access to one file system through several hybrid nodes at the same time. The feature requires a separate license. In fact, a single namespace for a particular file system is enabled through several physical hybrid nodes. At the same time, the client may not know through which specific node it is currently connected to. To increase the performance of such a file system, you can simply add another hybrid node to the Synchronized Node Group. Simultaneous recording to the same file from different nodes is not possible.

Synchronized Access Grid simplifies system design and provides good performance without the need to create separate file systems on each hybrid node.

Dynamic I / O - adaptive balancing of external flows. Dynamic distribution of backup streams across all external (front-end) Ethernet ports of all Hybrid nodes.

Dynamic I / O works if the cluster has Synchronized Access Grid and Universal Express I / O licenses. Data streams are evenly balanced across multiple hybrid nodes and / or subnets within a cluster. Within a single-mode configuration, you can also configure thread balancing on the physical ports of the node (without using the Synchronized Access Grid license), but you will need to use different subnets on each physical external port of the node. However, an alternative to both licenses within a small single-mode configuration may be the collection of several external interfaces in the Bonding Port . The following aggregation types are supported:

balance-rr - sequentially throws packets, from the first to the last interface
balance-alb
active-backup - one of the interfaces is active. If the active interface fails (link down, etc.), another interface replaces the active one. No additional switch configuration required
IEEE 802.3ad or LACP (requires configuration on the customer switch side)

Naturally, in addition to Bonding and Universal Express I / O, you can configure Jumbo Frames along the entire route of traffic (on the network interfaces of HydraStor inclusive).

For integration with Veritas NetBackup , OST plugins for various operating systems are supported. If Universal Express I / O is not used, then OST provides good performance over CIFS or NFS protocols. NetBackup is also able to balance its Jobs across multiple NEC HS nodes. To enable deduplication on the source, a Deduped Transfer license is also required. In addition, Optimized Copy functionality is supported, which allows only unique blocks to be replicated to a remote site.

OST Auto Image Replication is a functionality that automates the import of a backup catalog to a Backup server at a remote site. This allows you to accelerate data recovery at a remote site in the event of a complete failure of the original site.

OST plugin - FC configuration - a configuration is available for HYDRAStor, in which you can write (read) data to hybrid nodes including the FiberChannel protocol. Initially, only NetBackup with an OST plugin was supported. But recently a patch was released (available now), the functionality of which will be added to the next software release, and it already implements work with FC using the Universal Express IO protocol . This greatly expands the range of backup software that can work with data on a cluster via FC via Universal Express IO.

The figure below shows the configuration of hybrid nodes with FC HBA cards installed.

Failover Functionality for Business Continuity is the Failover mechanism of the HYDRAstor cluster. Hybrid nodes (HNs) monitor each other's state and in case of failure of one of the nodes, a healthy node intercepts the input / output from the broken node to itself.

When using the NFS protocol, I / O does not stop and sessions on the Backup server are not reset. In other cases, reconnection is required.

A slightly more illustrative example than in the previous picture. Not only file systems move from node to node, but also the IP addresses through which these file systems are exported.

WAN-optimized replication

Sends only unique data
Significantly reduces network load between sites
Eliminates tape maintenance costs
Protects data through on-the-fly encryption

1-to-1 bidirectional topology supported for replication

N-to-M bidirectional topology (or many to many) is also supported

You can copy data to 16 other systems with a single-node HYDRAstor configuration. As well as copying data to 50 other systems in a multi-node HYDRAstor configuration.

Space Reclamation - a place on a cluster, like on many deduplication cores, is not freed up immediately after deleting files or file systems. The Space Reclamation process is required, which proceeds in two stages: analysis and search for more unused data blocks and the actual process of removing these blocks. Space Reclamation can be started either manually or according to a custom schedule (for example, during hours when active data backups are not running).

WORM (HydraLock) - Write Once Read Many. A technology that allows over time to block files from making changes to them and further store them for a specified number of years. By default, data is stored in unchanged state for 10 years.

Encryption - HYDRAStor supports data encryption for transfers and disks. But this functionality is not available in Russia in accordance with our legislation.

List of separately purchased optional Hydrastor HS8 licenses

Almost all the functionality from the list is parsed above in the article. In addition to DirectDataShadow , which allows direct backup of volumes from NEC's M-Series storage systems (without additional servers and backup software).

When purchasing NEC HYDRAstor, separate licensing of deduplication + compression (DataRedux), as well as file access protocols Cifs, NFS and object access, are not required.

Monitoring and Notification - the system administrator can receive notifications from the system to the mail via SMTP (a), as well as via SNMP (b) or view them in the control interface ©.

HS Data Reduction Estimation Tool - a vendor command-line utility, available on request. Used to test deduplicability and compressibility of customer data. Available under Windows and Linux. The picture below shows an example of listing a BAT file that can be run with some frequency by hand or put into the task scheduler in Windows, for example to simulate a daily data backup. In fact, using the utility you can make an approximate estimate of the capacity of HydraStor, which will be required for storing customer backups.

Example of running the utility from CMD on Windows:

hsSimulator.exe -d <tempDBfile> -o <CSVfile 1> <Files/directories to be measured (before update)> hsSimulator.exe -d <tempDBfile> -o <CSVfile 2> <Files/directories to be measured (after update)>

An example of a BAT file for automating the launch of a scheduled utility:

 @echo off set h=%TIME:~0,2% if "%h:~0,1%" equ " " set h=0%h:~1,1% set m=%TIME:~3,2% set s=%TIME:~6,2% set dd=%DATE:~0,2% set mm=%DATE:~3,2% set yyyy=%DATE:~6,4% c:\hsSimulator\hsSimulator.exe -dc:\hsSimulator\temp.db ^ -oc:\hsSimulator\test-%dd%-%mm%-%yyyy%_%h%.%m%.%s%.csv c:\temp

As a result of the work of the BAT file, we get a CSV file with the date and time in the name: test-10-06-2019_17.30.11.csv

Single Node HYDRAStor HS3-50

NEC also has a small HYDRAStor HS3-50, which differs from older HS8 models in lower performance and lack of clustering support. Those. can be used only as one separate node. This model also uses a CPU generation E5-2600v3. The functionality is fully comparable with the single-model HS8-5001S. It can be licensed in multiples of 8TB, and the maximum raw capacity is 24TB. It is positioned for small installations and remote company offices with the ability to replicate data on older systems to the central office.

Net capacity on HS3 nodes without deduplication and compression:

HYDRAStor Virtual Appliance

HYDRAStor Virtual Appliance is a virtual app with the same software as on the iron versions of HS3 / HS8 and with the possibility of replication to these iron versions. Scalable to 15TB usable capacity. Installation requirements are presented in the picture below. It is positioned for small companies and remote offices as well as the HS3-50.

Upon request, the vendor can provide trial licenses for the HYDRAStor Virtual Appliance for 90 days.

^{Footnotes in tables} ^{1. Typical backup compression, depending on the type of data.} ^{2. Capacity calculated based on: 1 GB = 1,000,000,000 bytes, 1 TB = 1,000 GB.} ^{3. Calculation for HS VA on the VMware ESXi host server with 2 Intel Xeon E5-2470 @ 2.30GHz (8 cores each), 96GB of memory, 6x SATA HDD (7,200RPM 4TB) in RAID6} ^{4. The data compression ratio is assumed to be 20: 1.} ^{5. The type of CPU must be Nehalem-C (Westmere) or higher (eg, Intel Xeon Processor E56xx / L56xx / X56xx) with a frequency of 1.5GHz or higher.} ^{6. The capacitance values are calculated based on: 1GiB = 1,073,741,824 bytes, 1 GB = 1,000,000,000 bytes, 1 TB = 1,000 GB.}

NEC HYDRAStor HS6

NEC has another cheaper solution than the HS8 family of HYDRAstor systems. It is positioned for archival storage (primarily media content). It uses deduplication not as a variable, but as a fixed block. Otherwise, the same technologies are available as for the HS8. In the last column in the Capacity line is a typo, referring to Petabytes, not Terabytes.