FCoE
- Ethertype = 0x8906
- transports FC frame by providing data plane
- MTU = 2160 ≡ jumbo frames
- no fragmentation
- FC forwarder (FCF): logical FC switch with own MAC
- VLAN per VSAN
- 1-1 mapping
- VLAN 1002 by default
- STP lite
- sends agreement/proposal BPDUs
- MAC flushing is suppressed
- dedicated MST instance per VSAN (best practice)
- no MAC learning, broadcast, flooding
- no PVLAN
- only between DCBX switches: parameters are negotiated via LLDP TLV
- switches do not have to be FCF
- hop-by-hop forwarding in Ethernet: src/dst MAC between FCF change, FCID remains
- last dst MAC – FPMA
- does not participate in vPC
- requires Cat6e or Cat7 for copper to fullfil BER on 10Gbps
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Ethertype |Version| Reserved |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Reserved |
| |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Reserved | SOF |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
/ /
\ FC frame with FC CRC \
/ /
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| EOF | Reserved |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
(config)# feature lldp
; allows connecting to itself
(config)# fcoe veloopback
(config-vlan)# fcoe vsan <N>
; enable DCBX
(config-if)# lldp tlv-select dcbxp
; only FCoE on CNA
(config-if)# shutdown lan
(config)# vsan database
(config-vsan-db)# vsan <N> interface vfc <M>
(config)# interface vfc <M>
(config-if)# switchport trunk allowed vsan <N>
; 1-1 mapping
(config-if)# bind interface <INTF>
# show vlan fcoe
# show interface <INTF> fcoe
# show fcoe database
Fabric-provided MAC (FPMA)
- removes need for FCID-MAC mapping on devices
- locally unique within L2 segment
- assigned by FCF to ENode via FIP
- FCF does not use FPMA
- format: FC-MAP (0efc.xx) + FCID
- FC-MAP: pool of 256 unique addresses (in case one VLAN services several fabrics)
(config)# fcoe fcmap <FCMAP>
QoS
- classes
- class-fcoe
- no-drop
- MTU 2158 (N6k), 2112 (MDS, N7k)
- cannot be deleted
- CoS = 3, qos-group = 1
- class-default
- MTU = 1500 by default
- qos-group = 0
- class-fcoe
- CoS = 3 by default
- DCBX standards: pass QoS parameters (system qos) to CNA
- IEEE 802.1Qbb: PFC for lossless Ethernet (enabled on all interface by default)
- IEEE 802.1Qau: ECN
- IEEE 802.1Qaz: scheduler + negotiation
; PFC disabled by default
# show interface priority-flow-control
FCoE NPV
- connection to FCF – P2P only, vFC – VNP (trunk by default)
(config)# feature fcoe-npv
(config)# interface vfc <N>
(config-if)# switchport mode np
FCoE initialization protocol (FIP)
- control plane
- Ethertype = 0x8914
- uses globally assigned MAC from CNA (ENode MAC)
- ports: VN, VF, VE
- exchange is initiated by ENode
- not supported by 1st gen CNA: cannot be connected to FEX
- phases
- VLAN discovery
- optional
- over native VLAN
- mcast frame to All_FCF_MAC (0110.1801.0002) from ENode
- response contains list of available VLANs
- FCF discovery
- over FCoE VLAN
- mcast frame to All_FCF_MAC (solicitation) from ENode
- response: advertisement from FCF with their MAC, FIP timeout, FCF priority
- VLAN per vFC ≡ VSAN
- FCFs periodically send FCP discovery advertisements to All_ENode_MAC (0110.1801.0001)
- response is padded with zeroes up to FCoE max MTU to verify its consistency
- Fabric login
- establish virtual link with ENode
- FLOGI/FDISC, unicast to FCF
- single transaction: FCF election, fabric login, FCID allocation
- VLAN discovery
; 128 default
(config)# fcoe fcf-priority <N>
; 8s default
(config)# fcoe fka-adv-period <sec>
FIP header
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Ethertype |Version| Reserved |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| FIP operation code | Reserved | FIP Subcode |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Descriptor list length |F|S| Reserved |C|D|R|A|S|F|
+ |P|P| | | |P| | | +
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
/ /
\ Descriptor list (TLV) \
/ /
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Padding (optional) |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
Version: = 1
Descriptor list length: 4-byte words
FP: fabric provided, 1 ≡ FPMA
SP: server provided, always 0
C: controlling FCF (VA/VE capable)
D: FDF (VA capable)
RP: 1 ≡ REC (recorded), 0 ≡ P2P (can be intercepted); ≈ IPv6 DAD
A: available for FLOGI
S: solicited
F: FCF
Codes:
- 0x0001
- 0x01: discovery solicitation
- 0x02: discovery advertisement
- 0x0002
- 0x01: virtual link installation request
- 0x02: virtual link installation reply
- 0x0003
- 0x01: FIP keepalive
- 0x02: FIP clear virtual link
- 0x0004
- 0x01: FIP VLAN request
- 0x02: FIP VLAN notification
- 0x0005
- 0x01: N_port_ID probe request
- 0x02: N_port_ID probe reply
- 0x03: N_port_ID claim notification
- 0x04: N_port_ID claim response
- 0x05: N_port_ID beacon
- 0xfff8-0xfffe: vendor specific
FIP descriptors
- 0-127: critical
- 1: priority (1 byte)
- 2: MAC address
- 3: FC-MAP
- 4: name ID (8 bytes)
- 5: fabric (FC-MAP + fabric name)
- 6: max FCoE size
- 7: FLOGI
- 8: NPIV FDISC
- 9: LOGO
- 10: ELP
- 11: Vx_port identification
- 12: FKA_ADV_period
- 13: vendor ID
- 14: VLAN
- 15: FC-4 attributes
- 128-255: non-critical
- 128: clear virtual link reason code
- 241-254: vendor specific
FIP snooping
- Nexus 4000: IBM chassis blade
- installs ACL that fixes path from FCF to CNA
- protection from FCF MAC spoofing
- FC-BB-5 D.4.1/D.4.2
Design
- unified fabric ports: trunk, STP edge
- native VLAN cannot be FCoE VLAN
- manual pruning only for FCoE VLAN
- equal QoS for no-drop
FCoE topologies
- forwarding is based on FSPF, may not match STP
- multi-hop: several FCF
- single-hop
- directly attached
- FEX
- remotely attached: FCoE pass-through
Enhanced vPC (EvPC)
- FCoE on FEX
- only one parent switch maps FEX to VSAN and binds vFC
Nexus 7000
- FCoE only (F3 card)
- no-drop policy must be applied in admin VDC first; otherwise enabling storage VDC may lead to crash
; F3 card
(config)# license fcoe module <N>
(config)# install feature-set fcoe
; enable FEX FCoE
(config)# allow feature-set <FEX_NUMBER>
(config)# system qos
(config-sys-qos)# service-policy type network-qos default-nq-7e-policy
(config)# vdc FCoE_VDC type storage
(config-vdc)# allocate shared interface <INTF>
(config-vdc)# allocate fcoe-vlan-range <N> from vdc DEFAULT