A groundbreaking tool has been introduced to aid developers in verifying image and server profile schema checks for NetBackup Catalog Images. This innovative Data Availability Validation Tool ensures data availability across Host Operating Systems remains application-agnostic, with implementation depending on the underlying hardware architecture and storage topology. The tool supports two distinct architectural models to achieve this: Shared-Disk using Cluster File System (CFS) and Shared-Nothing using Flexible Storage Sharing (FSS).

How It Works

Shared-Disk using Cluster File System (CFS)

In the Shared-Disk model, multiple nodes connect to the same set of storage shelves. For configurations limited to two nodes, direct connections to the storage shelves are feasible. For setups exceeding two nodes, a switch is required to interconnect nodes and shelves. The hardware model suitable for this architecture is the 53xx series.

Shared-Nothing using Flexible Storage Sharing (FSS)

Conversely, in the Shared-Nothing model, each node is independently connected to its storage shelves. Data sharing and mirroring occur at the file-system level using Info Scale FSS technology. This model maintains CVM and CFS for volumes and filesystems, respectively. While it’s possible to qualify Info Scale FSS on regular VxVM and VxFS, the preferred approach for simplified management and minimal maintenance is to utilize CVM and CFS across all hardware models and data sharing architectures.

Image Types and Relationships

An image can be of different types, captured using a label like vxos.image.type. Here are some key image types:

1. Base: Used to build working images, suitable for dev-test environments but not for production or container runtime. Example: base NetBackup binary image.
2. Application: Used to instantiate and run containers, containing necessary application logic. Example: NetBackup image for container execution.
3. Add-On: Provides additional libraries and binaries, non-runnable independently. Example: third-party OST plugin for NetBackup.
4. Hotfix: Contains new binaries and libraries for patching application images, non-runnable independently. Example: EEB for NetBackup.
5. Internal: Non-user-facing images, used only for internal purposes.

Image Relationships

Complex applications may require multiple dependent containers, introducing parent-child relationships among images. For instance, a Velocity container might depend on Log Stash, Elastic Search, and MongoDB containers.

This introduces three properties: dependency type, location, and requirement.

1. Dependency Type: Images can specify parent, peer, and child dependencies. Example: a Velocity image might list WebServer and LogAggregator as peers, and MongoDB as a child.
2. Dependency Requirement: Dependencies can be optional or mandatory. Internal image dependencies are mandatory, while user-facing dependencies can vary. For example, an add-on image might be optional, while certain hotfixes could be mandatory.

High Availability

HA Enabler Platform

From an appliance perspective, the platform must deliver High Availability (HA) and Docker Container Orchestration without requiring container images to be aware of the HA technology. This flexibility ensures that any future changes to the HA provider will be seamless and non-disruptive to image providers. The Host OS will manage the container stack, including IP addresses and filesystem mounts. Application health monitoring inside the container should also be managed by the Host OS.

HA Enabled Containers

The platform supports all server instances defined in the Functional Requirement section, along with infrastructural components such as Auth Service, MongoDB, Hardware Monitoring, and AutoSupport.

Storage and Network Configuration

Image developers are not required to embed VCS components for application, network, and storage monitoring within containers. All HA management and monitoring will occur from the Host OS.

Storage Requirements:

1. Applications’ required filesystems must be created, formatted, and mounted on the Host OS.
2. These mount points should be exported and mounted from the Host OS.
3. The Host OS must fulfill the container’s volume requirements, avoiding volume plugins unless they offer specific benefits.

The platform ensures fast failover by making data volumes accessible to all Host OS instances (parallel service group). For exclusive Host OS access (failover instance), developers must specify this in the server profile.

Network Requirements:

1. Application IP addresses should be configured on the Host OS.
2. Required container application ports should be forwarded from the Host OS to the container.
3. The suitability of using macvlan remains under consideration, balancing support for bonds, VLANs, etc.

Orchestration Policy

Users can configure priority, CPU, and memory settings to manage load balancing and resource constraints for containers. These parameters limit resource usage upon container start and assist VCS in balancing compute nodes. The parameters are integrated into the service group’s properties, created to manage the container, with CPU and memory ranges dictated by the Host OS’s capabilities.

Maintenance Mode for Containers

Containers needing maintenance, whether for administrative purposes or internal operations like upgrades, will have their service group attribute set to “Enabled = 0,” suspending VCS monitoring to prevent false alarms.

Profile Schema Utilization

1. Understand whether it can cater to an image or not. If an image complies to a new profile scheme that does not maintain compatibility with the schema that the platform has knowledge of, then the platform cannot use the image. In this case, the platform can be upgraded to acquire ability to understand the image.
2. Understand whether a compute node can host containers of that image type-version pair. This situation may arise during rolling upgrade of the platform where the compute nodes are upgraded one at a time to minimize the application downtime OR an image is installed and loaded on some nodes but not everywhere. That is, either a compute node does not have knowledge of the image, or the backward compatibility is broken.
3. Understand whether same QoS can be provided to the application containers across compute nodes. This may happen when the backward compatibility is maintained but not all compute nodes are upgraded to have knowledge of new attributes or property of the profile.
4. All the filesystem required by the application should be created, formatted, and mounted on the Host OS and
5. The Host OS mount points for the application should be exported and mounted from the Host OS.
6. The Host OS will need to satisfy container’s requirements for volume, will not use volume plugin unless it can provide flexibility to choose LUNs etc.

Hidden Values in etcd

A notable aspect of this tool is the presence of hidden directory structures within etcd, such as directories or keys starting with an underscore. These hidden elements are not listed by conventional etcdctl tools and require specific knowledge to locate. For example, the /_coreos.com directory contains internal information about the fleet.

Transactions in etcd

Transactions are crucial for atomic operations within etcd, allowing a series of commands to be executed as a single, atomic transaction. This process involves defining the key and its value pre-modification, specifying commands upon successful comparison, and alternative commands if the comparison fails.

For more detailed information on this innovative tool and its capabilities, visit NetBackup Catalog Images Tool.

Published by: Martin De Juan