Disjoint L2 domains on Cisco UCS

This post will cover disjoint L2 domains on Cisco UCS-B and why/when you would implement it.

Scenario 1: multi-tenant infrastructure

Suppose your current UCS-B infrastructure is configured to support a single, physically L2 separated, tenant:

Single tenant UCS-B infrastructure

And suppose you need to share the UCS-B infrastructure with multiple, physically L2 separated, tenant:

Multi-tenant UCS-B infrastructure

Scenario 2: different physically separated, security zones

Suppose your current UCS-B infrastructure is configured to support a single L2 uplink:

Single L2 domain UCS-B infrastructure

And suppose you need some Virtual Machine on a different, physically L2 isolated, security zone:

Disjoint L2 domain UCS-B infrastructure

Requiremens for a working disjoint L2 domains

The configuration is pretty simple, but you need to be 100% sure about:

  1. All port related to the new port-channel must be in shutdown state, otherwise, after adding new port-channels on Fabric Interconnects, UCS-B will use them to carry old VLANs too and it will be a mess because new port-channels don’t carry old VLANs.
  2. VLANs must be unique: you cannot configure the same VLAN on more than one uplink/domain, otherwise, UCS-B won’t know which uplink/domain owns wich VLAN.
  3. Each server must have a vnic dedicated to a single uplink/domain: you cannot use a single vnic to carry VLAN from multiple uplink/domain, otherwise UCS-B won’t know to what uplink pins the vnic.

Configure disjoint L2 domains with a zero-downtime approach

  1. On Nexus switches, configure all ports dedicated to the new port-channels in shutdown state.
  2. On UCS Manageri (Equipment -> Chassis -> Fabric Interconnect A|B -> Physical Ports -> Ethernet Ports), configure all ports dedicated to the new port-channels as “Uplink Port” and disable them: Configure uplink ports on UCS-B
  3. On UCD Manager (LAN -> LAN Cloud -> Fabirc A|B -> Port Channels), add a new port-channel assigning the ports dedicated to the new port-channel, and disable the port-channel: Add port-channels on UCS-B
  4. On UCS Manager (LAN -> Lan Cloud -> VLANs), add all VLANs assigned to the new port-channel: Add VLANs on UCS-B
  5. On UCS Manager (LAN -> right click -> LAN Uplinks Manager -> VLANs -> VLAN Manager -> Fabric A|B), assign all VLANs (both old one and new one) to a port-channel. This is the most critical phase, so be sure you assign each VLAN to a single uplink/port-channel on both Fabric Interconnects: Bind VLANs to port-channels on UCS-B
  6. On UCS Manager, enable port-channels.
  7. On Cisco Nexus, configure and enable virtual port-channels