Today I would like to share my experience with configuring external L2 connectivity in Cisco ACI. As the reader probably knows, there are 2 approaches so far: either consider external L2 segment as an EPG itself or configure an External Bridged Network object (aka L2Out).
The lab has the following setup:
SW1 and SW2 are Nexus 3000 switches that would be used as hosts thus requiring nothing more than routed interface setup. APIC has 4.2(7l) firmware in this lab; specific switch models are irrelevant for the purpose of the discussion. Access policies and related entities are preconfigured since they are out of scope of this article.
Let’s create the network objects required for EPG connectivity: VRF and bridge domain (BD).
Figure 2. VRF setting – everything left as default
Figure 3. First step of BD config – all is default as well
Since we don’t require L3 connectivity, we could disable unicast routing within BD as well as uRPF. As for the other options, they could be left as default:
Now we can create an application profile (AP), TestAP, and get down to enabling L2 connectivity using the first method: assigning ports to EPGs SW1, SW2 and creating a contract.
It’s also important not to forget to associate the EPGs with domains, otherwise the policy would not be deployed. Let’s assign the EPG to the corresponding physical ports:
Having done everything (almost) on the ACI side, we can switch over to configuring N3k:
SW1(config)# interface ethernet1/46
SW1(config-if)# shutdown
SW1(config-if)# no switchport
SW1(config-if)# ip add 192.168.0.1/24
SW1(config-if)# mac-address 0000.0000.0001SW2(config)# interface ethernet1/46
SW2(config-if)# shutdown
SW2(config-if)# no switchport
SW2(config-if)# ip add 192.168.0.2/24
SW2(config-if)# mac-address 0000.0000.0002MAC addresses are assigned specific values so that it would be easier to distinguish the endpoint entries. IP addresses cannot be used for this purpose in our setup because we have disabled unicast routing on BD plus traffic is switched rather than routed.
This is still not enough to enable connectivity between SW1 and SW2 though because ACI employs whitelisting model via contracts. Let’s configure bidirectional ICMP connectivity:
Notice the scope of the contract: it would allow to confine connectivity within the AP instead of VRF. Although not relevant for SW1-SW2 reachability in this specific case, restricting the scope in general helps to avoid unexpected connectivity between endpoints in the same VRF. The only thing left is to assign the contract to the corresponding EPGs:
Time to test the connectivity between switches SW1 and SW2:
SW1# ping 192.168.0.2
PING 192.168.0.2 (192.168.0.2): 56 data bytes
36 bytes from 192.168.0.1: Destination Host Unreachable
Request 0 timed out
36 bytes from 192.168.0.1: Destination Host Unreachable
Request 1 timed out
64 bytes from 192.168.0.2: icmp_seq=2 ttl=254 time=2.259 ms
64 bytes from 192.168.0.2: icmp_seq=3 ttl=254 time=2.061 ms
64 bytes from 192.168.0.2: icmp_seq=4 ttl=254 time=2.138 msThe process is rather simple, isn’t it? Configuring L2Out instead of EPG SW2 would require much fewer steps now since most of the config is already in place. First, we need to remove static assignment from EPG SW2. Second, we’ll use this free port for L2Out:
I didn’t manage to configure L2Out to accept frames in VLAN 1, either tagged or untagged. When VLAN 1 was configured on L2Out, no endpoints were learned whatsoever. Since no faults were raised, I’m not sure whether there is any limitation regarding using VLAN 1 for L2Out. Anyway, I switched to SVI plus trunk configuration on SW2:
SW2(config)# interface ethernet1/46
SW2(config-if)# switchport
SW2(config-if)# switchport mode trunk
SW2(config-vlan)# interface vlan 150
SW2(config-if)# mac-address 0000.0000.0002
SW2(config-if)# ip address 192.168.0.2/24
SW2(config-if)# no shutdown
At the end, EPG should be assigned to L2 External Domain, otherwise the corresponding policies would not be deployed.
Since connectivity is governed by contracts, L2Out endpoints should be classified to external EPG:
EPG SW1 has already been configured, so only external EPG SW2 has to be assigned the contract to allow connectivity:
So far so good, everything seems to be in place, so let’s test connectivity once more:
SW1# ping 192.168.0.2
PING 192.168.0.2 (192.168.0.2): 56 data bytes
Request 0 timed out
Request 1 timed out
Request 2 timed out
Request 3 timed out
Request 4 timed outWell, something is obviously broken. One way to check if there’s something wrong with the contract is to disable policy enforcement in VRF. Once this is done, connectivity should be checked again. If a contract is the issue, then the ping should go through because contract enforcement is disabled. If SW1 cannot reach SW2 even with policy enforcement disabled, then something is wrong with L2Out configuration.
SW1# ping 192.168.0.2
PING 192.168.0.2 (192.168.0.2): 56 data bytes
64 bytes from 192.168.0.2: icmp_seq=0 ttl=254 time=1.713 ms
64 bytes from 192.168.0.2: icmp_seq=1 ttl=254 time=1.457 ms
64 bytes from 192.168.0.2: icmp_seq=2 ttl=254 time=1.397 ms
64 bytes from 192.168.0.2: icmp_seq=3 ttl=254 time=1.415 ms
64 bytes from 192.168.0.2: icmp_seq=4 ttl=254 time=1.548 msIt is definitely the policy who muddies the waters. However, everything was fine when we configured EPG in lieu of L2Out, so there must be no mistake in filtering, right? As always, the devil is in details. The contract is the issue, sure, but the filters at the same time are fine as well as the contract application. Looking back, I think there is only one place left where something can go wrong with such a config; however, it took me several hours to start desperately changing all the options available in tenant trying to find out what exactly blocks the connectivity. Funny enough, no faults were raised within the tenant meaning there were no contradictions within object model.
Remember the scope of the contract? We set it as “Application Profile” initially, but what AP does L2Out belong to? I don’t have an answer for that, unfortunately; however, you must be getting a feeling that it’s not natural for L2Out to have a contract with such a scope, so let’s change it to something bigger, “VRF”, for instance, and verify the connectivity again:
SW1# ping 192.168.0.2
PING 192.168.0.2 (192.168.0.2): 56 data bytes
64 bytes from 192.168.0.2: icmp_seq=0 ttl=254 time=1.807 ms
64 bytes from 192.168.0.2: icmp_seq=1 ttl=254 time=1.528 ms
64 bytes from 192.168.0.2: icmp_seq=2 ttl=254 time=1.412 ms
64 bytes from 192.168.0.2: icmp_seq=3 ttl=254 time=1.446 ms
64 bytes from 192.168.0.2: icmp_seq=4 ttl=254 time=1.391 msAt last, we managed to make L2Out work as intended. Was it easier than configuring L2 connectivity via EPG? Not quite: additional steps were needed to provision such an object (separate domain, EPG domain assignment) and the result turned out to be more rigid (single VLAN per L2Out as a whole). I also had hard time finding any documentation regarding L2Out, mostly bumping into various blogposts or forums. There was a positive point though: I’ve come across an interesting book that might prove itself useful for ACI troubleshooting.
You might be asking yourself: what about L3Out? As you could guess, it is subject to the same behavior as L2Out: contract with the scope of ”Application profile” is not going to be rendered for L3Out as well.
Kudos for review: Anastasiia Kuraleva
Networking and IT 