FCoE

  1. FCoE
    1. Fabric-provided MAC (FPMA)
    2. QoS
    3. FCoE NPV
  2. FCoE initialization protocol (FIP)
    1. FIP header
    2. FIP descriptors
    3. FIP snooping
  3. Design
    1. FCoE topologies
  4. Enhanced vPC (EvPC)
  5. Nexus 7000

FCoE

  • Ethertype = 0x8906
  • transports FC frame by providing data plane
    • MTU = 2160 ≡ jumbo frames
    • no fragmentation
  • FC forwarder (FCF): logical FC switch with own MAC
  • VLAN per VSAN
    • 1-1 mapping
    • VLAN 1002 by default
    • STP lite
      • sends agreement/proposal BPDUs
      • MAC flushing is suppressed
      • dedicated MST instance per VSAN (best practice)
    • no MAC learning, broadcast, flooding
    • no PVLAN
  • only between DCBX switches: parameters are negotiated via LLDP TLV
    • switches do not have to be FCF
  • hop-by-hop forwarding in Ethernet: src/dst MAC between FCF change, FCID remains
    • last dst MAC – FPMA
  • does not participate in vPC
  • requires Cat6e or Cat7 for copper to fullfil BER on 10Gbps
 0                   1                   2                   3
 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|           Ethertype           |Version|        Reserved       |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                            Reserved                           |
|                                                               |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                  Reserved                     |      SOF      |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
/                                                               /
\                      FC frame with FC CRC                     \
/                                                               /
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|      EOF      |                  Reserved                     |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
(config)# feature lldp

; allows connecting to itself
(config)# fcoe veloopback
(config-vlan)# fcoe vsan <N>
; enable DCBX
(config-if)# lldp tlv-select dcbxp

; only FCoE on CNA
(config-if)# shutdown lan
(config)# vsan database
(config-vsan-db)# vsan <N> interface vfc <M>
(config)# interface vfc <M>
(config-if)# switchport trunk allowed vsan <N>

; 1-1 mapping
(config-if)# bind interface <INTF>
# show vlan fcoe
# show interface <INTF> fcoe
# show fcoe database

Fabric-provided MAC (FPMA)

  • removes need for FCID-MAC mapping on devices
  • locally unique within L2 segment
  • assigned by FCF to ENode via FIP
  • FCF does not use FPMA
  • format: FC-MAP (0efc.xx) + FCID
    • FC-MAP: pool of 256 unique addresses (in case one VLAN services several fabrics)
(config)# fcoe fcmap <FCMAP>

QoS

  • classes
    • class-fcoe
      • no-drop
      • MTU 2158 (N6k), 2112 (MDS, N7k)
      • cannot be deleted
      • CoS = 3, qos-group = 1
    • class-default
      • MTU = 1500 by default
      • qos-group = 0
  • CoS = 3 by default
  • DCBX standards: pass QoS parameters (system qos) to CNA
    • IEEE 802.1Qbb: PFC for lossless Ethernet (enabled on all interface by default)
    • IEEE 802.1Qau: ECN
    • IEEE 802.1Qaz: scheduler + negotiation
; PFC disabled by default
# show interface priority-flow-control

FCoE NPV

  • connection to FCF – P2P only, vFC – VNP (trunk by default)
(config)# feature fcoe-npv

(config)# interface vfc <N>
(config-if)# switchport mode np

FCoE initialization protocol (FIP)

  • control plane
  • Ethertype = 0x8914
  • uses globally assigned MAC from CNA (ENode MAC)
  • ports: VN, VF, VE
  • exchange is initiated by ENode
  • not supported by 1st gen CNA: cannot be connected to FEX
  • phases
    1. VLAN discovery
      • optional
      • over native VLAN
      • mcast frame to All_FCF_MAC (0110.1801.0002) from ENode
      • response contains list of available VLANs
    2. FCF discovery
      • over FCoE VLAN
      • mcast frame to All_FCF_MAC (solicitation) from ENode
      • response: advertisement from FCF with their MAC, FIP timeout, FCF priority
        • VLAN per vFC ≡ VSAN
        • FCFs periodically send FCP discovery advertisements to All_ENode_MAC (0110.1801.0001)
        • response is padded with zeroes up to FCoE max MTU to verify its consistency
    3. Fabric login
      • establish virtual link with ENode
      • FLOGI/FDISC, unicast to FCF
      • single transaction: FCF election, fabric login, FCID allocation
; 128 default
(config)# fcoe fcf-priority <N>

; 8s default
(config)# fcoe fka-adv-period <sec>

FIP header

 0                   1                   2                   3
 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|           Ethertype           |Version|        Reserved       |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|      FIP operation code       |    Reserved   |  FIP Subcode  |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|    Descriptor list length     |F|S|    Reserved   |C|D|R|A|S|F|
+                               |P|P|               | | |P| | | +
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
/                                                               /
\                      Descriptor list (TLV)                    \
/                                                               /
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                       Padding (optional)                      |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

Version: = 1
Descriptor list length: 4-byte words
FP: fabric provided, 1 ≡ FPMA
SP: server provided, always 0
C: controlling FCF (VA/VE capable)
D: FDF (VA capable)
RP: 1 ≡ REC (recorded), 0 ≡ P2P (can be intercepted); ≈ IPv6 DAD
A: available for FLOGI
S: solicited
F: FCF
Codes:

  • 0x0001
    • 0x01: discovery solicitation
    • 0x02: discovery advertisement
  • 0x0002
    • 0x01: virtual link installation request
    • 0x02: virtual link installation reply
  • 0x0003
    • 0x01: FIP keepalive
    • 0x02: FIP clear virtual link
  • 0x0004
    • 0x01: FIP VLAN request
    • 0x02: FIP VLAN notification
  • 0x0005
    • 0x01: N_port_ID probe request
    • 0x02: N_port_ID probe reply
    • 0x03: N_port_ID claim notification
    • 0x04: N_port_ID claim response
    • 0x05: N_port_ID beacon
  • 0xfff8-0xfffe: vendor specific

FIP descriptors

  • 0-127: critical
    • 1: priority (1 byte)
    • 2: MAC address
    • 3: FC-MAP
    • 4: name ID (8 bytes)
    • 5: fabric (FC-MAP + fabric name)
    • 6: max FCoE size
    • 7: FLOGI
    • 8: NPIV FDISC
    • 9: LOGO
    • 10: ELP
    • 11: Vx_port identification
    • 12: FKA_ADV_period
    • 13: vendor ID
    • 14: VLAN
    • 15: FC-4 attributes
  • 128-255: non-critical
    • 128: clear virtual link reason code
    • 241-254: vendor specific

FIP snooping

  • Nexus 4000: IBM chassis blade
  • installs ACL that fixes path from FCF to CNA
  • protection from FCF MAC spoofing
  • FC-BB-5 D.4.1/D.4.2

Design

  • unified fabric ports: trunk, STP edge
  • native VLAN cannot be FCoE VLAN
  • manual pruning only for FCoE VLAN
  • equal QoS for no-drop

FCoE topologies

  • forwarding is based on FSPF, may not match STP
  • multi-hop: several FCF
  • single-hop
    • directly attached
    • FEX
    • remotely attached: FCoE pass-through

Enhanced vPC (EvPC)

  • FCoE on FEX
  • only one parent switch maps FEX to VSAN and binds vFC

Nexus 7000

  • FCoE only (F3 card)
  • no-drop policy must be applied in admin VDC first; otherwise enabling storage VDC may lead to crash
; F3 card
(config)# license fcoe module <N>

(config)# install feature-set fcoe

; enable FEX FCoE
(config)# allow feature-set <FEX_NUMBER>

(config)# system qos
(config-sys-qos)# service-policy type network-qos default-nq-7e-policy

(config)# vdc FCoE_VDC type storage
(config-vdc)# allocate shared interface <INTF>
(config-vdc)# allocate fcoe-vlan-range <N> from vdc DEFAULT