Connecting an external entity to Cisco ACI (external L3Out)

Abstract

In this post we'll show some scenario for L3Out or, in other words, how to manage static routes from a Cisco ACI fabric to external networks.

L3Out scenario on Cisco ACI

Topology:

  • VM1 connected to VLAN181, bridge domain BD_181 (VRFA).
  • VM2 connected to VLAN181, bridge domain BD_181 (VRFA), and to an external network.
  • VM3 connected to VLAN180, bridge domain BD_180 (VRFA).
  • VM4 connected to VLAN182, bridge domain BD_182 (VRFB).
  • VM5 connected to VLAN183, bridge domain BD_182 (BRFB).

Each IP address is defined as 10.0.<bridge domain ID>.<100 + VM ID>/24.

Always remember the Cisco ACI Workflow.

Steps to implement the topology:

  • Create an empty tenant
    • Create VRFA and VRFB
    • Create the bridge domains BD_180, BD_181, BD_182:
      • Associate the correct VRF to each bridge domain.
      • Define a subnet IP address to each bridge domain.
    • Create a new application profile.
      • Create an EPG for each bridge domain EPG_180, EPG_181, EPG_182:
        • Associate the right bridge domain to each EPG.
          • Associate the right domain to each EPG.
          • Associate all required static ports and VLAN to each EPG.

Intra EPG communication (VM1 to VM2)

Always remember the Cisco ACI Workflow.

Because VM1 and VM2 are in the same EPG, the can reach each other without any other configuration. VM2 will allows routing to the external world, but this step will be discussed later.

Intra VRF communication (VM3 to VM1 and VM2)

Always remember the Cisco ACI Workflow.

Because each bridge domain is associated to a dedicated EPG, to allow communication between VM3 and VM1/VM2, we have two paths:

  • define a contract between EPG_181 and EPG_181;
  • configure VRFA as unenforced.

Intra bridge domain communication from different EPGs (VM4 to VM5)

Always remember the Cisco ACI Workflow.

VM4 and VM5 are in different VLANs, attached to different EPGs, but both EPGs are attached to the same bridge domain. Because of that, both VM are in the same L2 broadcast domain, but communication is now controlled by contracts. Again, communication between VM4 and VM5 can allowed:

  • defining a contract between EPG_182 and EPG_183;
  • configuring VRFB as unenforced.

Inter VRF communication (VM1 and VM2 to VM4 and VM5)

Always remember the Cisco ACI Workflow.

In this case we have EPGs from different VRF that should communicate, in other words we must configure route leaking between VRFA and VRFB. To do that we must:

  • configure subnets (under each bridge domain) to be shared (“Shared between VRFs”);
  • configure contracts between EPGs which are under different VRFs (it’s not a full mesh, in our case). Contracts are needed even if VRFs are set as unenforced because contracts defines the mutual redistribution.

Contracts should be mono-directional, because both directions must be explicit (I suspect a bug, because it does not make sense). In other words, configure the following mono-directional contracts:

  • from EPG_180 to EPG_182;
  • from EPG_180 to EPG_183;
  • from EPG_181 to EPG_182;
  • from EPG_181 to EPG_183;
  • from EPG_182 to EPG_180;
  • from EPG_182 to EPG_181;
  • from EPG_183 to EPG_180;
  • from EPG_183 to EPG_181.

A full mesh requires n(n-1)/2=6 for each direction. In this case we have “only” 4*2 contracts.

On each leaf we can see inter VRF routes and leaked (pervasive) routes:

Leaf_101# show ip route vrf AD:VRFA
IP Route Table for VRF "AD:VRFA"
'*' denotes best ucast next-hop
'**' denotes best mcast next-hop
'[x/y]' denotes [preference/metric]
'%<string>' in via output denotes VRF <string>

10.0.180.0/24, ubest/mbest: 1/0, attached, direct, pervasive
    *via 10.0.232.66%overlay-1, [1/0], 00:21:56, static, tag 4294967295
10.0.180.1/32, ubest/mbest: 1/0, attached, pervasive
    *via 10.0.180.1, vlan27, [1/0], 01:47:12, local, local
10.0.181.0/24, ubest/mbest: 1/0, attached, direct, pervasive
    *via 10.0.232.66%overlay-1, [1/0], 00:22:02, static, tag 4294967295
10.0.181.1/32, ubest/mbest: 1/0, attached, pervasive
    *via 10.0.181.1, vlan1, [1/0], 01:46:25, local, local
10.0.182.0/24, ubest/mbest: 1/0, attached, direct, pervasive
    *via 10.0.232.66%overlay-1, [1/0], 00:08:31, static, tag 4294967295
Leaf_101# show ip route vrf AD:VRFB
IP Route Table for VRF "AD:VRFB"
'*' denotes best ucast next-hop
'**' denotes best mcast next-hop
'[x/y]' denotes [preference/metric]
'%<string>' in via output denotes VRF <string>

10.0.180.0/24, ubest/mbest: 1/0, attached, direct, pervasive
    *via 10.0.232.66%overlay-1, [1/0], 00:08:34, static, tag 4294967295
10.0.181.0/24, ubest/mbest: 1/0, attached, direct, pervasive
    *via 10.0.232.66%overlay-1, [1/0], 00:08:34, static, tag 4294967295
10.0.182.0/24, ubest/mbest: 1/0, attached, direct, pervasive
    *via 10.0.232.66%overlay-1, [1/0], 00:22:11, static, tag 4294967295
10.0.182.1/32, ubest/mbest: 1/0, attached, pervasive
    *via 10.0.182.1, vlan12, [1/0], 01:45:56, local, local

Routing to an external devices

Always remember the Cisco ACI Workflow.

Currently an “anycast address” cannot be used to forward packets from and to external routers. Because in this scenario VLAN 181 is now leading to external networks, delete BD_181, EPG_181 and all related contracts.

Then we must:

  • create an external routed domain, bound to a valid AAEP;
  • create an external routed network (L3 Out) bound to VRFA and to the external routed domain created above;
    • create a logical node profile;
      • add one node for each involved leaf;
        • add a default gateway on each node pointing to the VM2;
      • add one logical interface profile for each involved, selecting a SVI interface and giving to them an unique IP address and the same virtual IP address (secondary address);
    • create an external network;
  • associate the BD_180 to the L3 Out created above;
  • assign a contract between the BD_180 and the external network (contract is not needed if VRF is unenforced).

VM3 can now reach VM2 and networks behind VM2, but VM1 can also reach VM3 and networks behind VM2.

Let’s explain better this scenario:

  • VM1 uses BD_181 as default gateway;
  • when VM1 needs to reach networks behind VM2, still uses BD_181 as gateway, even if VM1 is directly attached. Using the traditional equipments (Catalyst, NX-OS switches), the “no ip redirect” is mandatory for most operating system.
  • Witch Cisco ACI, this scenario works configuring a L3Out using a SVI, and “no ip redirect” is not needed anymore.

From VRFB external networks are not reachable yet, because static are not redistributed into VRFB.

Posted on 24 Mar 2018 by Andrea.
  • Gmail icon
  • Twitter icon
  • Facebook icon
  • LinkedIN icon
  • Google+ icon