Fibre Channel

1. Fibre Channel
   1. FC-0
   2. FC-1
   3. FC-2
      1. FC frame
         1. Frame control (F_CTL)
      2. FC fragmentation
         1. SEQ_CNT
         2. Relative offset
      3. FC classes
      4. FC flow control
      5. Fibre Channel ID (FCID)
      6. World wide name (WWN)
      7. Alias
   4. FC-3
   5. FC-4
2. Port types
   1. Rate mode
3. FC link services
   1. BLS
      1. Abort sequence (ABTS)
      2. BA_RJT
   2. ELS
      1. Discover address (ADISC)
      2. FC ARP (FARP)
      3. Fabric login (FLOGI)
      4. PDISC
      5. Port login (PLOGI)
      6. State change registration (SCR)
      7. GA_NXT
      8. Process login (PRLI)
      9. Logout (LOGO)
      10. Process logout (PRLO)
      11. State change notification (SCN)
      12. Registered state change notification (RSCN)
      13. Loop initialize (LINIT)
      14. Fabric discovery (FDISC)
      15. Switch RSCN (SW_RSCN)
      16. REC
      17. Reinstate recovery qualifier (RRQ)
      18. Link services accept (LS_ACC)
      19. Link services reject (LS_RJT)
      20. Format
          1. FLOGI, LS_ACC, FDISC, PLOGI, PDISC
             1. PLOGI, PLOGI LS_ACC Common Service parameters
             2. FLOGI Common Service parameters
             3. FLOGI LS_ACC Common Service parameters
             4. PLOGI & PLOGI LS_ACC Class Service parameters
             5. FLOGI Class Service parameters
             6. FLOGI LS_ACC Class Service parameters
             7. FLOGI LS_ACC Services availability
             8. Clock synchronization QoS
          2. PRLI, PRLI LS_ACC
             1. PRLI Service parameter page
             2. PRLI LS_ACC Service parameter page
          3. REC
          4. REC LS_ACC
          5. RRQ
          6. LS_RJT
4. VSAN
   1. Trunking protocols
   2. Dynamic port VSAN membership (DPVM)
   3. Inter-VSAN routing (IVR)
5. LUN isolation
   1. Masking
   2. Mapping
   3. Zoning
      1. Smart zoning
      2. Enhanced zoning
      3. Autozone
6. FCP
   1. FCP write
   2. FCP_CMND
   3. FCP_DATA
   4. FCP_XFER_RDY
   5. FCP_RSP
   6. FCP_CONF
   7. Sequence retransmission request (SRR)
      1. Exchange error recovery
      2. Sequence error recovery
      3. SRR link service
      4. SRR Accept
   8. FCP IUs
      1. FCP_CMND IU
      2. FCP_XFER_RDY IU
      3. FCP_RSP IU
         1. FCP_RSP_INFO
      4. FCP_RJT
7. Virtualization
   1. N_port ID virtualization (NPIV)
   2. N_port virtualizer (NPV)
   3. VM-aware FC
8. Slow drain
   1. Fabric performance impact notification (FPIN)
   2. Dynamic ingress rate limiting (DIRL)
   3. TxWait
   4. Slowport-monitor
   5. Link reset/link reset response primitives (LR/LRR)
   6. Congestion-isolation
9. FC Port-channel
10. FC domain
    1. Principal switch
    2. Persistent FCID
    3. Fabric initialization
    4. Link initializtion
    5. Switch fabric internal link service (SW_ILS)
       1. Exchange link parameters (ELP)
          1. Class F service parameters
          2. Class 2,3 interconnect port parameters
       2. Exchange switch capabilities (ESC)
       3. Exchange peer parameters (EPP)
       4. Exchange fabric parameters (EFP)
       5. Domain ID assigned (DIA)
       6. Request domain ID (RDI)
       7. SW_ACC
       8. SW_RJT
11. Cisco fabric services (CFS)
12. Design
13. Security
    1. FC-SP-2
    2. FC authentication
    3. Port-security
    4. Fabric binding
14. Telemetry

• transport for SCSI
• host bus adapter (HBA)
  • flow control
  • sequencing
  • segmentation
  • error correction
  • own BIOS ⇒ boot from SAN
• fabric shortest path first (FSPF)
• I/O operations per second (IOPS)
• topology
  • point-to-point
    • bidirectional link
  • arbitrated loop
    • unidirectional ring, ≈ hub
    • max 127 devices
      • 127, if there is switch
      • 126, if not fabric-attached
      • practical limit = 18
    • reinit when a new device is added
      • device reset or link-level error – disruptive
    • used within JBOD, NAS, blade
  • fabric
    • bidirectional link, full-duplex, switched
    • AL can be attached to fabric
    • 16M devices
    • 239 switches: 0x00 ≡ FC_AL, 0xf0-0xfe ≡ reserved, 0xff ≡ well-known
• channels: input, output, Tx → Rx, Rx → Tx
• cut-through, store & forward
• up to 10 km between devices
• can use fiber or twinax, not twisted pair
• hop ≡ ISL, not switch
• lossless ≡ no drops because of congestion

FC-0

• physics
• serial: DB9, SMF, MMF
  • non-return to zero (NRZ): light ≡ 1, dark ≡ 0, up to 25 Gbaud
  • pulse amplitude modulation (PAM4): QSFP

FC-1

• encoding
  • 8b/10b
    • D/Kxx.y: D – data, K – special char, xx = 5 LSB, y = 3 MSB
  • 64b/66b
    • 10/16 Gbps
    • synchronization, guarantee for bits to change
      • preamble = 01 – data
      • preamble = 10 – 8-bit type + control info
• synchronization
• bit error rate
  • 15 error burst in 5 mins

; default: if threshold is exceeded, shutdown port
(config-if)# switchport ignore bit-errors

FC-2

• flow control, QoS
• MTU
  • 2112 bytes payload
  • 2148 bytes FC frame
  • must be multiple of 4 bytes
  • no fragmentation or reassembly in fabric ⇒ must match within whole fabric
• fragmentation and reassembly performed by hosts
  • method must match per sequence
• words
  • data words
    • between SOF and EOF
    • SYN ≈ NTP
    • BB_SC: credit recovery (buffer-to-buffer state change)
  • ordered set
    • delimiters
    • primitive signals: transmitted when senders wants to transmit
      • idle: sync, interframe spacing (24 bytes)
      • R_RDY: acknowledgement for buffer
      • VC_RDY, SYN, BB_SC
    • primitive sequence: transmitted unless response is received
      • LR: link reset
      • LRR: link reset response
      • OLS: ofline state (init, NOS received, enter offline)
      • NOS: not operational (failure, offline)
    • begin with K28.5
  • 4 character long (40 bits)
• sequence
  • bidirectional
  • consists of frames
  • guarantees in-order delivery
  • data is passed to ULP only after all frames are received ≈ transaction
• exchange
  • between host and disk, can be several
  • consists of sequences
  • per ULP, but ULPs use same FC-connection
• keepalive not required due to RSCN

FC frame

 0                   1                   2                   3
 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                             Idles                             |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                      Start of frame (SOF)                     |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+--+
|     R_CTL     |                     D_ID                      |  |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+  |
|     CS_CTL    |                     S_ID                      |  |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+  |
|     TYPE      |                    F_CTL                      |   \
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+    >   FC
|     SEQ_ID    |     DF_CTL    |            SEQ_CNT            |   /  header
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+  |
|             OX_ID             |             RX_ID             |  |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+  |
|                           Parameter                           |  |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+--+
\                                                               \
/                             Data                              /
\                                                               \
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                              CRC                              |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                       End of frame (EOF)                      |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|             Idles             |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

R_CTL: routing control

• ROUTING (4 bit) || INFORMATION (4 bit)
• ROUTING
  • 0x0: device data frame
  • 0x2: extended link services (ELS)
  • 0x3: FC-4 link data
  • 0x4: video data
  • 0x5: extended headers
  • 0x8: basic link services (BLS)
  • 0xc: link control frame
  • 0xf: extended routing
• INFORMATION
  • 0x0: uncategorized
  • 0x1: solicited data
  • 0x2: unsolicited control
  • 0x3: solicited control
  • 0x4: unsolicited data
  • 0x5: data descriptor
  • 0x6: unsolicited command
  • 0x7: command status

D_ID: destination ID
S_ID: source ID
SEQ_CNT: sequence count
OX_ID: originator exchange ID
RX_ID: responder exchange ID

CS_CTL: class specification/priority

• 0x80: preference
  • 0 ≡ frame delivered with no preference
  • 1 ≡ may be delivered with preference
• 0x40: reserved
• 0x3F: DSCP

TYPE: data structure

• ROUTING = 0x2 or 0x8
  • 0x00: basic link service (BLS)
  • 0x01: extended link service (ELS)
• ROUTING = 0x4
  • 0x60: FC-AV
  • 0x61: ARINC 818
• ROUTING = 0x0 or 0x3
  • 0x05: IPv4, IPv6, ARP over FC
  • 0x08: FCP
  • 0x0a: additional FCP features
  • 0x14: FC SATA tunnelling protocol
  • 0x18: SBCCS
  • 0x1b: SBCCS channel to control unit
  • 0x1c: SBCCS control unit to channel
  • 0x20: FC Common transport
  • 0x22: switch fabric internal link services (SW_ILS)
  • 0x25: interfabric router internal link services
  • 0x28: NVMe-oF
  • 0x40: HIPPI-FP
  • 0x48: MIL-SID-1553
  • 0x49: ASM
  • 0x58: virtual interface
  • 0xde: generic FC features
  • 0xdf: RNID general topology discovery page ID
  • 0xe0-0xff: vendor specific

DF_CTL: data field

• 0x80: reserved
• 0x40: 1 ≡ ESP header and trailer are present, 0 ≡ no ESP header or trailer
• 0x20: 0 ≡ no network header, 1 ≡ network header present; for video_data and device_data
• 0x1c: reserved
• 0x03:
  • 00: no device_header
  • 01: 16-bytes device_header
  • 10: 32-bytes device_header
  • 11: 64-bytes device_header
  • for video_data and device_data

Frame control (F_CTL)

   0     1     2     3     4     5     6     7
+-----+-----+-----+-----+-----+-----+-----+-----+
|ECTX |SCTX | 1S  | LS  | ES  | Rsv |CSCTL| SI  |
+-----+-----+-----+-----+-----+-----+-----+-----+
| Reserved  | ACK_Form  |       Reserved        |
+-----+-----+-----+-----+-----+-----+-----+-----+
| Reserved  |    ASC    | ROP | ER  |Fill bytes |
+-----+-----+-----+-----+-----+-----+-----+-----+

ECTX: exchange context, 0 ≡ originator, 1 ≡ responder
SCTX: sequence context, 0 ≡ originator, 1 ≡ recipient
1S: first sequence, 1 ≡ TRUE, 0 = FALSE
LS: last sequence, 1 ≡ last sequence of exchange
ES: end sequence, 1 ≡ last frame of sequence
CSCTL: 0 ≡ field in FC header is CS_CTL, 1 ≡ field is Priority
SI: sequence initiative, 0 ≡ hold, 1 ≡ transfer
ACK_Form:

• 00: no assistance provided
• 01: ACK_1 required
• 10: reserved
• 11: ACK_0 required

ASC: abort sequence condition

• ACK frame: from recipient
  • 00: continue sequence
  • 01: abort sequence, perform ABTS
  • 10: stop sequence
  • 11: obsolete
• data frame: from initiator
  • 00: abort, discard multiple sequences
  • 01: abort, discard single sequence
  • 10: process policy with infinite buffers
  • 11: obsolete

ROP: relative offset present, 1 ≡ parameter field in FC header – relative offset
ER: exchange reassembly, 0 ≡ exchange between 2 N_ports, 1 ≡ between multiple N_ports
Fill bytes: bytes of fill after payload

FC fragmentation

SEQ_CNT

• segmentation and reassembly method, uses SEQ_CNT field in FC header to concat frames within sequence
• mandatory for N_port
• mode
  • normal:
    • per-sequence counter
    • SEQ_CNT bit in Common Service parameters = 0 in PLOGI
  • continuously increasing
    • per-exchange counter
• F_CTL relative offset present = 0

Relative offset

• segmentation and reassembly method, uses Parameter in FC header to concat frames within sequence
• uses CIRO and RRO in Common Service parameters + relative offset by information category in PLOGI
• mode
  • continuously increasing: uses SEQ_CNT field in FC header, F_CTL
  • random: does not use SEQ_CNT or F_CTL
• F_CTL relative offset present = 1

FC classes

• class information is carried in SOF
• class 1
  • acknowledged
  • connection-oriented link
  • full BW
  • between nodes, ≈ telephone crossbar, deprecated
• class 2
  • acknowledged
  • packet-oriented (connectionless), routed
  • flow control: end-to-end, link
• class 3
  • unacknowledged: ULP must have retransmit functionality
  • packet-oriented, routed, no in-order delivery
  • flow control: link; no BW/latency guarantee
  • default
• class 4
  • acknowledged
  • connection-oriented
  • partial BW, VC setup
• class 6
  • acknowledged
  • connection-oriented
  • mcast, unidirectional
• class F
  • acknowledged
  • packet-oriented
  • between switches

FC flow control

• credit model, 1 credit = 1 frame
  • Tx cannot send more than credits are available
  • credits are replenished by acknowledge from Rx (R_RDY acknowledge)
  • credit size ≈ RTT/(transmit time per frame)
    • delay: 5 µs / 100 km
    • 1 BB/km at 2 Gbps
• link
  • physical connection
  • BB_credit (buffer-to-buffer)
  • BB_credit_CNT: number of sent and yet unacknowledged frames
  • classes: 2, 3, F
  • initial credits: via FLOGI
    • F_port: 32 default
    • E/TE_port: 250 (8 Gbps) or 500 (16/32 Gbps)
• end-to-end
  • EE_credit
  • HBA
  • classes: 1, 2
  • initial credits: via PLOGI
• prevents drops because of buffer overflow
• retransmit – per sequence
• if frame is not received within E_D_TOV (error detection timeout value) – drop all frames in sequence + notify destination node (source is not nofitfied by fabric, that is ULP task)

; number of BB_credit
# show interface bbcredit

; how many times BB_credit reached 0
# show interface counters

Fibre Channel ID (FCID)

• dynamically acquired, routable, 24 bits
• topologies
  • P2P
    • one port assigns 0x000000, another – 0x000001
  • AL
    • because of 8b/10b only 127 addresses are available
    • 1 address must be assigned to FL_port
    • addresses are assigned during initialization
    • LIP storm: if a device is added/removed – initialize again using loop initialization primitive
    • AL_PA: arbitrated loop physical address
  • fabric
    • domain ID: unique for switch (≈ switch address, L1 IS-IS)
    • area ID:
      • port group
      • every AL receives its own unique ID
    • domains 0x00, 0xf0-0xff are reserved for switch services ⇒ max 239 switches

	8 bit	8 bit	8 bit
fabric	domain ID	area ID	device
private loop	0x00	0x00	AL_PA
public loop	domain ID	area ID	AL_PA

World wide name (WWN)

• hardcoded, 64-bit
• types
  • node WWN (nWWN)
  • port WWN (pWWN)
  • fabric WWN (fWWN): FC switch port WWN
• name servers translate WWN → FCID
• network address authority (NAA)
  • = 1, 2, 5: 8-byte WWN
  • = 6: 16-byte WWN

 0                   1                   2                   3
 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|FMT_ID |   N_port/F_port ID    |              OUI              |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|      OUI      |                  HBA specific                 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

Format ID:

• 1 ≡ MAC-48: NAA || reserved(12) || MAC
• 2 ≡ extended: NAA || vendor assigned || MAC-48
• 3 ≡ locally assigned
• 4 ≡ IPv4: NAA || reserved(28) || IPv4
• 5 ≡ registered: NAA || OUI || vendor assigned
• 6 ≡ registered extended: NAA || OUI || vendor assigned(36) || extension(64)
• 12 ≡ EUI-64: NAA(2) || OUI without U/L and I/G || vendor assigned(40)

Alias

• replace pWWN with a text string
• modes
  • basic: alias is translated immediately (default)
  • enhanced: alias can be used and stored as part of config
• alias-to-pWWN mapping: 1-1 only
• spreaded over CFS

; legacy, no CFS support, local to VSAN config, zoning only
(config)# fcalias name <NAME> vsan <N>
(switch-config-fcalias)# member pwwn <PWWN>

(config)# device-alias import fcalias vsan <N>

(config)# device-alias mode enhanced
(config)# device-alias database
(config-device-alias-db)# device-alias name <TEXT> pwwn <PWWN>

(config)# device-alias commit

# show device-alias merge status
# show device-alias merge conflicts

FC-3

• conceptual: common services to ULP
• compressing, encryption in HBA
• multipathing
• extended link service (ELS): fcping, FLOGI, PLOGI, PRLI, PRLO

FC-4

• application protocols, upper-layer (ULP)
  • Fibre Channel Protocol (FCP) – SCSI-3
  • IPFC
  • Fibre connection ESCON (enterprise system connection) – FICON, mainframes
• layer between upward API and FC, ≈ driver installation

Port types

1. E_port
   • fabric expansion
   • ISL
   • class 2, 3, F
2. F_port
   • fabric port
   • connected to N_port only
   • class 2, 3
3. FL_port
   • fabric loop port
   • public loop
   • connected to NL_port, FL_port (only one active FL_port in segment)
   • class 2, 3
4. TE_port
   • trunk expansion
   • extended ISL
   • VSAN, QoS, fctrace
5. NP_port
   • node proxy port
6. TF_port
   • ≈ TE
   • connected to TN/TNP_port
   • uses EVFP
7. TNP_port
   • on NPV towards FC switch with NPIV
8. SD_port
   • SPAN destination port
9. ST_port
   • RSPAN port
10. Fx_port
    • after initialization becomes F_port or FL_port
11. B_port
    • bridge port
    • ≈ E_port
    • FCIP: long distances
12. G_port
    • generic port
    • autoconfiguration
13. V*_port
    • virtual port
    • FCoE

(config)# system default switchport mode F
(config)# system default switchport shutdown

; allows the port to consume license, by default ports are eligible
; acquire ≡ statically allocate license (e.g., server powered off)
(config-if)# port-license [acquire]

; on SD_port, preserve EISL headers, disabled by default
(config-if)# switchport encap eisl

(config-if)# switchport mode auto|E|F|FL|Fx|TL|NP|SD

; auto default, M depends on switch/LC
(config-if)# switchport speed <N>|auto [max <M>]

; LED beacon, simplifies finding the port
(config-if)# switchport beacon

Rate mode

• fixed BW is allocated in backplane for port group
• modes
  • dedicated
    • port has guaranteed BW
    • ISL, storage/tape array, high-BW servers, port-channels
  • shared
    • (BW pool) – (BW allocated to dedicated)

; shared by default
(config-if)# switchport rate-mode shared|dedicated

# show port-resources

FC link services

• basic link service (BLS): low-level, single frame
• extended link services (ELS): 2-sequence, prohibited before PLOGI
• FC services:
  • 0xffffff: bcast
  • 0xfffffe: login server
  • 0xfffffd: fabric controller; principal switch election, RSCN, FSPF RIB
  • 0xfffffc: name server
  • 0xfffffb: time server; time sync within fabric
  • 0xfffffa: mgmt server; fabric mgmt from any switch in fabric (e.g., zone config)
  • 0xfffff9: QoS facilitator; class 4
  • 0xfffff8: alias server
  • 0xfffff7: security key distribution server
  • 0xfffff6: clock synchronization server
  • 0xfffff5: mcast
  • 0xfffff1-0xfffff4: reserved
  • 0xfffff0: N_port controller

BLS

Abort sequence (ABTS)

• abort only last sequence or exchange
  • ABTS is sent within the exchange to be aborted: SEQ_ID matches
• mandatory
• can be sent even if count of active sequences = max concurrent sequences
• Sequence Initiative not required for ABTS to be transmitted
• always transfers Sequence Initiative to allow receiver to reply (BA_ACC, BA_RJT)
• ABTS effect depends on Exchange Error Policy
• no payload
• FC header parameter
  • 0x0000FF00: abort reason code
  • 0x00000001: abort type, 0 ≡ abort exchange, 1 ≡ abort sequence

   0     1     2     3     4     5     6     7
+-----+-----+-----+-----+-----+-----+-----+-----+
|                SEQ_ID validity                |
+-----+-----+-----+-----+-----+-----+-----+-----+
|      SEQ_ID of last deliverable sequence      |
+-----+-----+-----+-----+-----+-----+-----+-----+
|                    Reserved                   |
|                                               |
+-----+-----+-----+-----+-----+-----+-----+-----+
|                     OX_ID                     |
|                                               |
+-----+-----+-----+-----+-----+-----+-----+-----+
|                     RX_ID                     |
|                                               |
+-----+-----+-----+-----+-----+-----+-----+-----+
|                  Low SEQ_CNT                  |
|                                               |
+-----+-----+-----+-----+-----+-----+-----+-----+
|                 High SEQ_CNT                  |
|                                               |
+-----+-----+-----+-----+-----+-----+-----+-----+

SEQ_ID validity:

• 0x80: SEQ_ID of last deliverable sequence is valid
• 0x00: ignored, whole exchange is aborted

Low SEQ_CNT

• 0x0000 if abort exchange
• Recovery Qualifier if abort sequence
• SEQ_CNT of the last frame to be successfully received

High SEQ_CNT

• 0xffff if abort exchange
• ABTS SEQ_CNT if abort sequence

BA_RJT

   0     1     2     3     4     5     6     7
+-----+-----+-----+-----+-----+-----+-----+-----+
|                    Reserved                   |
+-----+-----+-----+-----+-----+-----+-----+-----+
|                  Reason code                  |
+-----+-----+-----+-----+-----+-----+-----+-----+
|              Reason explanation               |
+-----+-----+-----+-----+-----+-----+-----+-----+
|                Vendor specific                |
+-----+-----+-----+-----+-----+-----+-----+-----+

Reason code:

• 0x01: invalid command code
• 0x03: logical error
• 0x05: logical busy
• 0x07: protocol error
• 0x09: unable to perform command request
• 0xff: vendor specific error

Reason explanation

• 0x00: no additional info
• 0x03: invalid OX_ID-RX_ID combination
• 0x05: sequence aborted, no sequence info provided

ELS

; local to switch
# show flogi database

; global for fabric
# show fcns database

Discover address (ADISC)

• uses FCID of destination node ⇒ cannot be used for initial name resolution
• updates peer via PLOGI session about own changed FCID
• informational

FC ARP (FARP)

• can be used before PLOGI
• WWN → FCID, IP → FCID
• not used by FCP

Fabric login (FLOGI)

• sent by N_port to obtain FCID, BB_credit, MTU, service class
• source FCID = 0x00
• when switch receives FLOGI, corresponding port becomes F_port
  • time to wait for FLOGI after link init – 700 ms (Cisco)
• LS_ACC carries assigned FCID in frame D_ID

PDISC

• exchange parameters between N_ports
• does not affect open sequences and exchanges
• connection reset is required to apply new parameters ⇒ informational role only

Port login (PLOGI)

• register FCID-WWN mapping
• request list of accessible devices (SCSI scan)
• PLOGI to fabric: list of available WWNs for FCP
• PLOGI to device: capability list

State change registration (SCR)

• subscribe to fabric changes, subscription on controller (e.g., addition of new target)

GA_NXT

• NS query

Process login (PRLI)

• establish FC-4 session
• session parameters are negotiated via ELS PRLI and LS_ACC, via SCSI MODE SENSE and MODE SELECT
• using SCSI for parameter negotiation requires FCP session to be modified – usually LUN 0 sends parameters for the specific session within the device (proprietary) without network involvement

Logout (LOGO)

Process logout (PRLO)

State change notification (SCN)

Registered state change notification (RSCN)

• replaces SCN
• notify host about changes in fabric
  • disk join/leave
  • NS registration change
  • new zone
  • IP change
• zone scope

; multiplexes RSCN instead of sending them one by one
(config)# rscn multi-pid vsan <N>

; timeoutfor mux, 2000 ms default, must match in whole fabric
(config)# rscn event-tov <ms> vsan <N>

Loop initialize (LINIT)

Fabric discovery (FDISC)

• request additional FCID (≈ FLOGI)

Switch RSCN (SW_RSCN)

• distribute changes of FCNS DB

REC

• request from initiator to target about exchange state
• optional
• FC header Parameter = task retry ID or 0x00000000

Reinstate recovery qualifier (RRQ)

• Recovery Qualifier:
  • FQXID, low SEQ_CNT, high SEQ_CNT
  • it is prohibited to use SEQ_CNT from range after receving BA_ACC during R_A_TOV
• RRQ resets Recovery Qualifier

Link services accept (LS_ACC)

Link services reject (LS_RJT)

Format

FLOGI, LS_ACC, FDISC, PLOGI, PDISC

 0                   1                   2                   3
 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                       LS command code                         |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                                                               |
|                   Common service parameters                   |
|                                                               |
|                                                               |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                         N_port Name                           |
|                                                               |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                     Node name / Fabric name                   |
|                                                               |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                                                               |
|                  Class 1/6 Service parameters                 |
|                                                               |
|                                                               |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                                                               |
|                   Class 2 Service parameters                  |
|                                                               |
|                                                               |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                                                               |
|                   Class 3 Service parameters                  |
|                                                               |
|                                                               |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                                                               |
|                   Class 4 Service parameters                  |
|                                                               |
|                                                               |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                                                               |
|                      Vendor version level                     |
|                                                               |
|                                                               |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                      Services availability                    |
|                                                               |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                   Login extension data length                 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                      Login extension data                     | - words #32-61
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                    Clock synchronization QoS                  |
|                                                               |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

LS command code: code || 0x000000

• 0x02: LS_ACC
• 0x03: PLOGI
• 0x04: FLOGI
• 0x50: PDISC
• 0x51: FDISC

N_port name: pWWN, not used by F_port
Node name: nWWN or sWWN
Login extension data length: words

PLOGI, PLOGI LS_ACC Common Service parameters

   0     1     2     3     4     5     6     7
+-----+-----+-----+-----+-----+-----+-----+-----+
|                     0x2020                    |
|           (FC-PH version – obsolete)          |
+-----+-----+-----+-----+-----+-----+-----+-----+
|                   BB_Credits                  |
|                                               |
+-----+-----+-----+-----+-----+-----+-----+-----+
|CIRO | RRO |VVVL | NFP |BBCM |EDTOV|EELPI| Rsv |
+-----+-----+-----+-----+-----+-----+-----+-----+
|PTAG | QDB |SecBt|CSPC |RTTOV| DHD |SEQCT| PB  |
+-----+-----+-----+-----+-----+-----+-----+-----+
|     BB_SC_N     | B2B receive data field size |
+-----+-----+-----+-----+-----+-----+-----+-----+
|          Reserved           | AHS | Reserved  |
+-----+-----+-----+-----+-----+-----+-----+-----+
|      Nx_port total concurrent sequences       |
+-----+-----+-----+-----+-----+-----+-----+-----+
|    Relative offset by information category    |
|                                               |
+-----+-----+-----+-----+-----+-----+-----+-----+
|                                               |
|                 E_D_TOV value                 |
|                                               |
|                                               |
+-----+-----+-----+-----+-----+-----+-----+-----+

BB_credit: default = 1
CIRO: continuously increasing relative offset, default = 0 (not supported)
RRO: random relative offset, default = 0 (not supported)
VVVL: valid vendor version level, 0 ≡ no info (default), 1 ≡ field valid
NFP: 0 ≡ N_port, 1 ≡ F_port
BBCM: BB_credit management, 0 ≡ N/F_port, 1 ≡ L_port
EDTOV: E_D_TOV resolution, 0 ≡ 1 ms (default), 1 ≡ 1 ns
EELPI: energy efficient LPI mode supported, 0 ≡ not supported (default)
PTAG: priority tagging supported, 0 ≡ not supported (default)
QDB: query data buffer conditions, 0 ≡ no buffer condition to report (default)
SecBt: security bit, 0 (default)
CSPC: clock synchronizationprimitive capable, 0 ≡ not capable (default)
RTTOV: R_T_TOV value, 0 ≡ 100 ms (default), 1 ≡ 100 µs
DHD: dynamic half-duplex supported, 0 ≡ not supported (default)
SEQCT: SEQ_CNT/vendor specific, 0 ≡ normal rules (default), 1 ≡ increasing SEQ_CNT
PB: payload bit, 0 ≡ length=116 bytes (default), 1 ≡ length=256 bytes + login extension length
AHS: application header support, 0 ≡ not supported (default)
Buffer-to-beffer receive data field size: MTU, units – words, 256 ≤ x ≤ 2112

FLOGI Common Service parameters

   0     1     2     3     4     5     6     7
+-----+-----+-----+-----+-----+-----+-----+-----+
|                     0x2020                    |
|           (FC-PH version – obsolete)          |
+-----+-----+-----+-----+-----+-----+-----+-----+
|                   BB_Credits                  |
|                                               |
+-----+-----+-----+-----+-----+-----+-----+-----+
|NPIV | VF  |VVVL | NFP |BBCM |NSSB |EELPI| Rsv |
+-----+-----+-----+-----+-----+-----+-----+-----+
|PTAG | QDB |SecBt|CSPC |RTTOV| DHD |VenSp| PB  |
+-----+-----+-----+-----+-----+-----+-----+-----+
|     BB_SC_N     | B2B receive data field size |
+-----+-----+-----+-----+-----+-----+-----+-----+
|                                               |
|                                               |
|                                               |
|                   Reserved                    |
|                                               |
|                                               |
|                                               |
|                                               |
+-----+-----+-----+-----+-----+-----+-----+-----+

NPIV: multiple N_port ID support, 0 ≡ not supported (default), 1 ≡ NPIV-capable
VF: virtual fabrics, 0 ≡ not supported (default)
NSSB: name server session begin, 0 ≡ not requested (default), 1 ≡ requested
VenSp: vendor specific

FLOGI LS_ACC Common Service parameters

   0     1     2     3     4     5     6     7
+-----+-----+-----+-----+-----+-----+-----+-----+
|                     0x2020                    |
|           (FC-PH version – obsolete)          |
+-----+-----+-----+-----+-----+-----+-----+-----+
|                   BB_Credits                  |
|                                               |
+-----+-----+-----+-----+-----+-----+-----+-----+
| CA  | VF  |NPIVA| NFP |NSSS |EDTOV|EELPI| BF  |
+-----+-----+-----+-----+-----+-----+-----+-----+
|PTAG | QDB |SecBt|CSPC |RTTOV| DHD |VenSp| PB  |
+-----+-----+-----+-----+-----+-----+-----+-----+
|     BB_SC_N     | B2B receive data field size |
+-----+-----+-----+-----+-----+-----+-----+-----+
|                                               |
|                 R_A_TOV value                 |
|                                               |
|                                               |
+-----+-----+-----+-----+-----+-----+-----+-----+
|                                               |
|                 E_D_TOV value                 |
|                                               |
|                                               |
+-----+-----+-----+-----+-----+-----+-----+-----+

CA: clean address, 0 ≡ no info (default), 1 ≡ wasn’t used during R_A_TOV
NPIVA: multiple N_port ID assignment, 0 ≡ not supported (default)
NSSS: name server session started, 0 ≡ not established (default), 1 ≡ established
BF: broadcast supported by fabric, 0 ≡ not supported (default)
R_A_TOV, E_D_TOV: ms

PLOGI & PLOGI LS_ACC Class Service parameters

   0     1     2     3     4     5     6     7
+-----+-----+-----+-----+-----+-----+-----+-----+
| CV  |                 Reserved                |
+-----+-----+-----+-----+-----+-----+-----+-----+
|Prio |Pref |DSQoS|           Reserved          |
+-----+-----+-----+-----+-----+-----+-----+-----+
|        Reserved       |iACK0| Rsv |iACKG| Rsv |
+-----+-----+-----+-----+-----+-----+-----+-----+
|     Reserved    |iCSEC|        Reserved       |
+-----+-----+-----+-----+-----+-----+-----+-----+
|rACK0| Rsv |rXIDI|   rEPS    | Rsv |    CPS    |
+-----+-----+-----+-----+-----+-----+-----+-----+
|        Reserved       |rCSEC|rSUE |  Reserved |
+-----+-----+-----+-----+-----+-----+-----+-----+
|     Reserved    |   Receive data field size   |
+-----+-----+-----+-----+-----+-----+-----+-----+
|                    Reserved                   |
+-----+-----+-----+-----+-----+-----+-----+-----+
|           Total concurrent sequences          |
+-----+-----+-----+-----+-----+-----+-----+-----+
|               Nx_port EE_credit               |
+-----+-----+-----+-----+-----+-----+-----+-----+
|                    Reserved                   |
+-----+-----+-----+-----+-----+-----+-----+-----+
|          Open sequences per exchange          |
+-----+-----+-----+-----+-----+-----+-----+-----+
|                   Reserved                    |
|                                               |
+-----+-----+-----+-----+-----+-----+-----+-----+

i ≡ initiator control, r ≡ recipient control
CV: class validity, 0 ≡ not supported (default), 1 ≡ valid
Priority: 0 ≡ not supported (Fx) | non-zero may be tolerated (Nx) – default
Preference: 0 ≡ no preferred delivery (Fx) | non-zero CS_CTL may be tolerated (Nx) – default
DSQoS: DiffServ QoS, 0 ≡ not supported (default)
ACK0: ACK_0 capable, class 2 only, 0 ≡ incapable (default)
ACKG: ACK generation assistance, class 2 only, 0 ≡ no assistance (default)
CSEC: clock sync ELS capable, 0 ≡ not capable (default)
XIDI: X_ID interlock, class 2 only, 1 ≡ required (default)
EPS: error policy support, 00 ≡ discard only (default), 10 ≡ discard/process
SUE: SEQ_ID unique per exchange, 0 ≡ unique per SID/DID (default), 1 ≡ exchange only
Concurrent sequences: 1 default
EE_credit: 1 default
Open sequences: 1 default
CPS: categories per sequence

• 00 ≡ 1 cat/seq
• 01 ≡ 2 cat/seq (default)
• 11 ≡ > 2 cat/seq

FLOGI Class Service parameters

   0     1     2     3     4     5     6     7
+-----+-----+-----+-----+-----+-----+-----+-----+
| CV  |     Reserved    |SeqD |     Reserved    |
+-----+-----+-----+-----+-----+-----+-----+-----+
|Prio |Pref |DSQoS|                             |
+-----+-----+-----+                             +
|                    Reserved                   |
+                 +-----+-----+-----+-----+-----+
|                 |iCSEC|                       |
+-----+-----+-----+-----+                       +
|                    Reserved                   |
+                       +-----+-----+-----+-----+
|                       |rCSEC|     Reserved    |
+-----+-----+-----+-----+-----+-----+-----+-----+
|                                               |
|                   Reserved                    |
|                                               |
+-----+-----+-----+-----+-----+-----+-----+-----+
|           Total concurrent sequences          |
+-----+-----+-----+-----+-----+-----+-----+-----+
|               Nx_port EE_credit               |
|                                               |
+-----+-----+-----+-----+-----+-----+-----+-----+
|                    Reserved                   |
+-----+-----+-----+-----+-----+-----+-----+-----+
|          Open sequences per exchange          |
+-----+-----+-----+-----+-----+-----+-----+-----+
|                   Reserved                    |
|                                               |
+-----+-----+-----+-----+-----+-----+-----+-----+

SeqD: sequential delivery, 0 ≡ out-of-order allowed (default), 1 ≡ in-order requested

FLOGI LS_ACC Class Service parameters

   0     1     2     3     4     5     6     7
+-----+-----+-----+-----+-----+-----+-----+-----+
| CV  |     Reserved    |SeqD |     Reserved    |
+-----+-----+-----+-----+-----+-----+-----+-----+
|Prio |Pref |DSQoS|                             |
+-----+-----+-----+                             +
|                    Reserved                   |
+                 +-----+-----+-----+-----+-----+
|                 |iCSEC|                       |
+-----+-----+-----+-----+                       +
|                    Reserved                   |
+                       +-----+-----+-----+-----+
|                       |rCSEC|                 |
+-----+-----+-----+-----+-----+                 +
|                                               |
|                                               |
|                                               |
|                                               |
|                                               |
|                   Reserved                    |
|                                               |
|                                               |
|                                               |
|                                               |
+-----+-----+-----+-----+-----+-----+-----+-----+

SeqD: 1 ≡ fabric shall deliver class 2 & 3 in-order

FLOGI LS_ACC Services availability

   0     1     2     3     4     5     6     7
+-----+-----+-----+-----+-----+-----+-----+-----+
|                                               |
|                                               |
|                   Reserved                    |
|                                               |
|                                               |
|                                               |
+                                   +-----+-----+
|                                   | CS  |SKDS |
+-----+-----+-----+-----+-----+-----+-----+-----+
| Reserved  | MS  | TS  | DS  |     Reserved    |
+-----+-----+-----+-----+-----+-----+-----+-----+

CS: clock synchronization server
SKDS: security key distribution server
MS: management server
TS: time server
DS: directory server (name server)

Clock synchronization QoS

   0     1     2     3     4     5     6     7
+-----+-----+-----+-----+-----+-----+-----+-----+
|                 CS_QoS Request                |
+-----+-----+-----+-----+-----+-----+-----+-----+
|                  CS Accuracy                  |
+-----+-----+-----+-----+-----+-----+-----+-----+
| Reserved  |               CS_MSB              |
+-----+-----+-----+-----+-----+-----+-----+-----+
| Reserved  |               CS_LSB              |
+-----+-----+-----+-----+-----+-----+-----+-----+
|                                               |
|               CS Update period                |
|                                               |
|                                               |
+-----+-----+-----+-----+-----+-----+-----+-----+

CS QoS Request:

• 0x00 ≡ fields have no meaning
• 0x01 ≡ fields contain QoS parameters
• 0x02-0xff ≡ reserved

CS Accuracy: mantissa (3 bits) + exponent (5 bits)
CS_MSB: meaningful MSB bit position in clock (63 → … → 0), 110111b → MSB of byte 1 (byte 0, byte 1, …)
CS_LSB: meaningful LSB bit position in clock, 001000b → LSB of byte 6

PRLI, PRLI LS_ACC

   0     1     2     3     4     5     6     7
+-----+-----+-----+-----+-----+-----+-----+-----+
|                LS Command code                |
+-----+-----+-----+-----+-----+-----+-----+-----+
|                  Page length                  |
+-----+-----+-----+-----+-----+-----+-----+-----+
|                Payload length                 |
|                                               |
+-----+-----+-----+-----+-----+-----+-----+-----+
\                                               \
/            Service parameter pages            /
\                                               \
+-----+-----+-----+-----+-----+-----+-----+-----+

LS Command code: 0x20 ≡ PRLI, 0x02 ≡ PRLI LS_ACC
Page length: bytes, = 0x10
Payload length: bytes, size of PLRI
Service parameter page: per FCP process between devices

PRLI Service parameter page

   0     1     2     3     4     5     6     7
+-----+-----+-----+-----+-----+-----+-----+-----+
|                   Type code                   |
+-----+-----+-----+-----+-----+-----+-----+-----+
|                    Reserved                   |
+-----+-----+-----+-----+-----+-----+-----+-----+
|OPAV |RPAV | EIP |                             |
+-----+-----+-----+                             +
|                    Reserved                   |
+-----+-----+-----+-----+-----+-----+-----+-----+
|                                               |
|         Originator process associator         |
|                   (not used)                  |
|                                               |
+-----+-----+-----+-----+-----+-----+-----+-----+
|                                               |
|         Responder process associator          |
|                   (not used)                  |
|                                               |
+-----+-----+-----+-----+-----+-----+-----+-----+
|                    Reserved                   |
|                                               |
+                                   +-----+-----+
|                                   |           |
+-----+-----+-----+-----+-----+-----+           +
|               Service parameters              |
+-----+-----+-----+-----+-----+-----+-----+-----+

Type code: 0x08 ≡ FCP
OPAV: originator process associator valid, = 0
RPAV: responder process associator valid, = 0
EIP: establish image pair, 0 ≡ request parameters only, 1 ≡ request to establish session
Parameters

• 0x001: WRITE FCP_XFER_RDY DISABLED, 1 ≡ support unsolicited data
• 0x002: READ FCP_XFER_RDY DISABLED, = 1
• 0x00c: obsolete
• 0x010: TARGET FUNCTION
• 0x020: INITIATOR FUNCTION
• 0x040: DATA OVERLAY ALLOWED
  • 1 ≡ capable to transmit data with same buffer offset per command
  • 0 ≡ only sequentially increasing buffer offset
• 0x080: CONFIRMED COMPLETION ALLOWED, 1 ≡ support FCP_CONF
• 0x100: RETRY, 1 ≡ SRR support
• 0x200: TASK RETRY ID REQUESTED
  • 1 ≡ FC Parameter in FCP_CMND is task retry ID
  • 0 ≡ FC Parameter is zero

PRLI LS_ACC Service parameter page

   0     1     2     3     4     5     6     7
+-----+-----+-----+-----+-----+-----+-----+-----+
|                   Type code                   |
+-----+-----+-----+-----+-----+-----+-----+-----+
|                    Reserved                   |
+-----+-----+-----+-----+-----+-----+-----+-----+
|OPAV |RPAV | IPE | Rsv |          ARC          |
+-----+-----+-----+-----+-----+-----+-----+-----+
|                    Reserved                   |
+-----+-----+-----+-----+-----+-----+-----+-----+
|                                               |
|         Originator process associator         |
|                   (not used)                  |
|                                               |
+-----+-----+-----+-----+-----+-----+-----+-----+
|                                               |
|         Responder process associator          |
|                   (not used)                  |
|                                               |
+-----+-----+-----+-----+-----+-----+-----+-----+
|                    Reserved                   |
|                                               |
+                                   +-----+-----+
|                                   |           |
+-----+-----+-----+-----+-----+-----+           +
|          Service parameters response          |
+-----+-----+-----+-----+-----+-----+-----+-----+

Type code: 0x08 ≡ FCP
IPE: image pair established

• 1 ≡ confirm
• 0 ≡ parameters only
• valid only if PRLI EIP = 1

ARC: accept response code

• 0x1: no error
• 0x2: no resources, may retry
• 0x3: target initializing, may retry
• 0x5: cannot establish image pair due to configuration, cannot be retried
• 0x6: executed, some parameters not set as requested
• 0x7: cannot process multipage PLRI, single-page only
• 0x8: service parameters invalid

REC

   0     1     2     3     4     5     6     7
+-----+-----+-----+-----+-----+-----+-----+-----+
|                LS Command code                |
+-----+-----+-----+-----+-----+-----+-----+-----+
|                                               |
|                    Reserved                   |
|                                               |
|                                               |
+-----+-----+-----+-----+-----+-----+-----+-----+
|                                               |
|           Exchange originator S_ID            |
|                                               |
+-----+-----+-----+-----+-----+-----+-----+-----+
|                     OX_ID                     |
|                                               |
+-----+-----+-----+-----+-----+-----+-----+-----+
|                     RX_ID                     |
|                                               |
+-----+-----+-----+-----+-----+-----+-----+-----+

LS Command code: 0x13
Exchange originator S_ID: 3rd party query, S_ID = source FCID for exchange
RX_ID: 0xffff ≡ unassigned

REC LS_ACC

   0     1     2     3     4     5     6     7
+-----+-----+-----+-----+-----+-----+-----+-----+
|                LS Command code                |
+-----+-----+-----+-----+-----+-----+-----+-----+
|                                               |
|                    Reserved                   |
|                                               |
+-----+-----+-----+-----+-----+-----+-----+-----+
|                     OX_ID                     |
|                                               |
+-----+-----+-----+-----+-----+-----+-----+-----+
|                     RX_ID                     |
|                                               |
+-----+-----+-----+-----+-----+-----+-----+-----+
|                    Reserved                   |
+-----+-----+-----+-----+-----+-----+-----+-----+
|                                               |
|         Originator address identifier         |
|                                               |
+-----+-----+-----+-----+-----+-----+-----+-----+
|                    Reserved                   |
+-----+-----+-----+-----+-----+-----+-----+-----+
|                                               |
|          Responder address identifier         |
|                                               |
+-----+-----+-----+-----+-----+-----+-----+-----+
|                                               |
|                    FC4VALUE                   |
|                                               |
|                                               |
+-----+-----+-----+-----+-----+-----+-----+-----+
|                                               |
|                    E_STAT                     |
|                                               |
|                                               |
+-----+-----+-----+-----+-----+-----+-----+-----+

LS Command code: 0x02
FC4VALUE: bytes received/sent in exchange (FCP)
E_STAT

• 0x00000001: ESB owner, 0 ≡ originator, 1 ≡ responder
• 0x00000002: sequence initiative, 1 ≡ holds, 0 ≡ other holds
• 0x00000004: completion, 0 ≡ open, 1 ≡ complete
• 0x00000008: ending condition, 0 ≡ normal, 1 ≡ abnormal
• 0x00000010: recovery qualifier, 0 ≡ none, 1 ≡ active
• 0x00000060: exchange error policy
  • 00: abort, discard multiple sequences
  • 01: abort, discard single sequence
  • 10: process with infinite buffers
• 0x00000080: originator X_ID invalid, 0 ≡ valid
• 0000000100: responder X_ID invalid

RRQ

   0     1     2     3     4     5     6     7
+-----+-----+-----+-----+-----+-----+-----+-----+
|                LS Command code                |
+-----+-----+-----+-----+-----+-----+-----+-----+
|                                               |
|                    Reserved                   |
|                                               |
|                                               |
+-----+-----+-----+-----+-----+-----+-----+-----+
|                                               |
|           Exchange originator S_ID            |
|                                               |
+-----+-----+-----+-----+-----+-----+-----+-----+
|                     OX_ID                     |
|                                               |
+-----+-----+-----+-----+-----+-----+-----+-----+
|                     RX_ID                     |
|                                               |
+-----+-----+-----+-----+-----+-----+-----+-----+
|                                               |
≈               Obsolete (32 bytes)             ≈
|                                               |
+-----+-----+-----+-----+-----+-----+-----+-----+

LS Command code: 0x12

LS_RJT

   0     1     2     3     4     5     6     7
+-----+-----+-----+-----+-----+-----+-----+-----+
|                LS Command code                |
+-----+-----+-----+-----+-----+-----+-----+-----+
|                                               |
|                    Reserved                   |
|                                               |
|                                               |
+-----+-----+-----+-----+-----+-----+-----+-----+
|                  Reason code                  |
+-----+-----+-----+-----+-----+-----+-----+-----+
|              Reason explanation               |
+-----+-----+-----+-----+-----+-----+-----+-----+
|                Vendor specific                |
+-----+-----+-----+-----+-----+-----+-----+-----+

LS Command code: 0x01
Reason code:

• 0x01: invalid command code
• 0x03: logical error
• 0x05: logical busy
• 0x07: protocol error
• 0x09: unable to perform command request
• 0x0b: command not supported
• 0x0e: command already in progress
• 0xff: vendor specific error

Reason explanation

• 0x00: no additional info
• 0x15: invalid originator S_ID
• 0x17: invalid OX_ID-RX_ID combination
• 0x1e: PLOGI required
• 0x2c: request not supported

VSAN

• separates topologies, services and protocols
• ANSI T.11
• FCID can be the same in different VSANs
• 256 VSAN per switch
  • VSAN 1: default VSAN, in-band management
  • VSAN 4094: isolated VSAN (if VSAN is deleted, its port are assigned to this VSAN)
• EISL header
  • 8 bytes
    • QoS: 3 bits
    • VSAN ID: 12 bits
    • MPLS flag: 1 bit
    • TTL: 8 bits
  • between SOF and FC header

(config)# vsan database
(config-vsan-db)# vsan <N> [name <NAME>]

; 1 ≡ standard-based; 2,3 ≡ Brocade native (core PID 0,1); 4 ≡ McData native
(config-vsan-db)# vsan <N> interop <M>

; routing LB, src-dst-ox-id default
(config-vsan-db)# vsan <N> loadbalancing src-dst-id|src-dst-ox-id

; assign interface to VSAN
(config-vsan-db)# vsan <N> interface <INTF>

Trunking protocols

• only E_ports by default
• trunking auto ≡ passive mode, Cisco-only
• protocols
  1. E-TE: Cisco EPP (PTP)
  2. TF-TN: exchange virtual fabrics protocol (EVFP)
  3. TF-TNP: Cisco EPP (PTP)
  4. E, F port-channel: Cisco EPP (PCP)
  5. TF port-channel: Cisco EPP (PTP + PCP)
  6. TF-TNP 3rd party: EVFP

(config)# system default switchport trunk mode on|off|auto

# show vsan membership
# show vsan usage

Dynamic port VSAN membership (DPVM)

• assigns to VSAN, based on pWWN/nWWN during FLOGI
• pWWN > nWWN
• uses CFS: coordinated mode, physical scope

(config)# feature dpvm

(config)# dpvm database
(config-dpvm-db)# pwwn <WWN> vsan <N>

(config)# dpvm commit
(config)# dpvm activate [force]
(config)# dpvm autolearn
(config)# dpvm distribute

# show dpvm pending
# show dpvm database
# show dpvm ports vsan <N>

Inter-VSAN routing (IVR)

• Cisco proprietary
• edge switch rewrites VSAN header only
  • FCID can be rewritten as well (IVR NAT) to unused domain ID (not switch domain ID)
• RIB:
  • 4096 IVR entries max
  • current VSAN, source domain ID, destination domain ID, next-hop VSAN
• traffic control between VSANs
  • name server for configured edge nodes
  • RSCN for configured edge nodes
  • FSPF for domain IDs, used by configured end nodes
• default zone – always deny
• IVR zone copy is created for every member VSAN for hard zoning enforcement after activation
• CFS distribution: zones, zonesets, VSAN topology, service groups, AFID
• VSAN topology
  • group of VSANs that use IVR
  • can be split into service groups: no traffic between service groups
  • autonomous fabric ID (AFID): distinguish between different VSANs with same number

(config)# feature ivr

; disabled by default
(config)# ivr distribute

; disabled by default
(config)# ivr nat

; use CFS
(config)# ivr vsan-topology auto

; manual config
(config)# ivr vsan-topology database
(config-ivr-topology-db)# autonomous-fabric-id <AFID> switch <SWWN> vsan-ranges <LIST>

(config)# autonomous-fabric-id database
(config-afid-db)# switch <SWWN> autonomous-fabric-id <AFID> vsan <LIST>

(config)# ivr vsan-topology activate

(config)# ivr service-group name <SG>
(config-ivr-sg)# autonomous-fabric-id <AFID> switcvsan-ranges <LIST>

(config)# ivr zone name <ZONE>
(config-ivr-zone)# member pwwn <PWWN> vsan <N>

(config)# ivr zoneset name <ZONESET>
(config-ivr-zoneset-zone)# member <ZONE>

; if VSAN has no active zoneset, force ≡ disruptive
(config)# ivr zoneset activate name <ZONESET> [force]

# show ivr vsan-topology
# show ivr zone
# show ivr zoneset
# show ivr internal fcid-rewrite-list

LUN isolation

Masking

• array level
• access permission based on pWWN
• not suitable for JBOD: no management controller
• security measure, additional to zoning

Mapping

• host HBA level

Zoning

• unicast only
• VSAN level
• fabric security, usually enforced on fabric edge
• types
  • soft: restrictions on name server level (vulnerable to spoofing, manual config)
  • hard: frame filtering + soft zoning (soft – fallback for hard)
• zone set
  • group of zones
  • only 1 set is active, active set cannot be modified
  • 500 max
• TCAM utilization: n(n-1) ⇒ 1-1 zoning as best practice

; prohibit traffic forwarding in default zone
(config)# no zone default-zone permit vsan <N>

(config)# zone name <ZONE> vsan <N>
(config-zone)# member device-alias <ALIAS>
(config-zone)# member domain-id <ID> port-number <M>
(config-zone)# member fcalias <FCALIAS>
(config-zone)# member fcid <FCID>
(config-zone)# member fwwn|pwwn <WWN> [lun <LUN>]
(config-zone)# member interface <INTF>

(config)# zoneset name <ZONESET> vsan <N>
(config-zoneset)# member <ZONE>

; CFS distributes active zoneset
(config)# zoneset activate name <ZONESET> vsan <N>

; CFS distributes all zonesets
(config)# zoneset distribute full vsan <N>

; solve merge conflicts, replaces zones
(config)# zoneset import|export interface <INTF> vsan <N>

# show zoneset brief
# show zoneset active
# show zoneset vsan <N>

# zoneset distribute vsan <RANGE>
# zoneset import interface <INTF> vsan <RANGE>
# zoneset export vsan <RANGE>

; after merge, copy active zoneset (in RAM) to config (NVRAM)
# zone copy active-zoneset full-zoneset vsan <N>

Smart zoning

• distinguishes initiators and targets
• initiator → target flows are permitted, other flows are prohibited
• both can be used for replication
• 1-1, 1-n (multiple targets); n-n not recommended

(config)# system default zone smart-zone enable
(config)# zone smart-zoning enable vsan <N>

; receives device type from name server
(config)# zone convert smart-zoning zone <ZONE> vsan <N>
(config)# zone convert smart-zoning fcalias <FCALIAS> vsan <N>
(config)# zone convert smart-zoning zoneset <ZONESET> vsan <N>
(config)# zone convert smart-zoning vsan <N>

; disable smart zoning
(config)# clear zone smart-zoning zone <ZONE> vsan <N>
(config)# clear zone smart-zoning fcalias <FCALIAS> vsan <N>
(config)# clear zone smart-zoning zoneset <ZONESET> vsan <N>
(config)# clear zone smart-zoning vsan <N>

(config-zone)# member pwwn <WWN> both|init|target

Enhanced zoning

• VSAN-wide lock
• all modifications require commit
• enough to enable on one switch
• merge:
  • restrict: DBs must match
  • allow: fail only if contents of zone, fcalias with the same name do not match

(config)# zone mode enhanced vsan <N>
(config)# zone commit vsan <N>

; allow by default; fail if there are differences
(config)# zone merge-control restrict vsan <N>

# clear zone lock vsan <N>

Autozone

• fabric, consisting of single switch
• up to 100 attached devices
• VSAN 1
• scanning is performed once in 5 mins
• zones:
  • initiator → all targets
  • target → all initiators

# autozone --update

FCP

• FC-4, ULP = SCSI-3
• no support
  • read: phase-collapse
  • write: immediate data
• supports
  • write: unsolicited data (size – via SCSI Disconnect-Reconnect mode page), 1 PDU per SCSI write
• information unit (IU)
  • ≡ PDU, in Data field
  • R_CTL ROUTING = device data frame
  • single sequence
• only initiator or target can transmit at any moment of time ≡ hold F_CTL Sequence Initiative
• load-balancing: per SCSI command over different FCP sessions

FCP write

1. FCP_CMND
2. FCP_XFER_RDY
3. FCP_DATA
4. FCP_RSP

FCP_CMND

• FC header Parameter = task ID (command retry supported) or zero
• SCSI command, TMF request
• R_CTL ROUTING = device data frame, INFORMATION = unsolicited command
• uses command reference number (CRN) for in-order delivery
  • CRN ≈ iSCSI CmdSN (ITT ≈ FQXID)
  • CRN per LUN
  • increment by 1 per command, wrapping counter
  • CRN = 0
    • TMF request
    • SCSI command that does not support precise (in-order) delivery (INQUIRY, TEST UNIT READY)

FCP_DATA

• IU for Data-out and Data-in
• SI bit is set in FC header along with Data-out, so that target may respond
• R_CTL INFORMATION = solicited data (even for unsolicited burst)
• Data-in does not transfer SI bit in FC header, only FCP_RSP does that
• FC header Parameter = relative offset value
• no format defined
  • detected using FC header
  • ULP – in Data field (size > 0)

FCP_XFER_RDY

• acknowledge readiness for FCP_DATA (1:1 mapping)
• FC header Parameter = 0
• R_CTL INFORMATION = data descriptor

FCP_RSP

• SCSI status indicator after SCSI command and TMF
• FC header Parameter = 0
• R_CTL INFORMATION = command status

FCP_CONF

• initiator acknowledges receiving FCP_RSP, if requested to do so
• RSP and CONF are mapped to each other using FQXID
• R_CTL INFORMATION = solicited control
• FC header Parameter = 0
• no data, FC header only
• does not support TMF and SCSI-linked commands (only last command in chain)

Sequence retransmission request (SRR)

• FC-4 link service
• roles
  • initiator requests read data retransmit
  • initiator requests target to request write data retransmit
  • initiator requests FCP_RSP retransmit
• optional
• requires support for REC and retry task ID (inside FC header Parameter)
• does not support bidir commands
• always transfers Sequence Initiative to target to allow retransmit (SRR and referenced exchange)
• retransmit is not per IU (as in iSCSI), but per data accoding to relative offset (starting byte)
• FCP_RJT received ≡ initiator aborts referenced exchange

Exchange error recovery

• Recovery Abort
• mandatory

Sequence error recovery

• optional

SRR link service

 0                   1                   2                   3
 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|         FC-4 Link service Command code (0x14000000)           |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|             OX_ID             |             RX_ID             |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                        Relative offset                        |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|  R_CTL for IU |                   Reserved                    |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

OX_ID, RX_ID: exchange to be retransmitted
Relative offset:

• ignored for FCP_RSP
• first byte offset to retransmit
• 2 LSB = 00 ≡ 4-byte boundary
• last byte is always retransmitted

SRR Accept

 0                   1                   2                   3
 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|         FC-4 Link service Command code (0x02000000)           |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

FCP IUs

FCP_CMND IU

   0     1     2     3     4     5     6     7
+-----+-----+-----+-----+-----+-----+-----+-----+
|                                               |
≈               FCP_LUN (8 bytes)               ≈
|                                               |
+-----+-----+-----+-----+-----+-----+-----+-----+
|                      CRN                      |
+-----+-----+-----+-----+-----+-----+-----+-----+
| Rsv |           Priority          |    ATTR   |
+-----+-----+-----+-----+-----+-----+-----+-----+
|                   TMF flags                   |
+-----+-----+-----+-----+-----+-----+-----+-----+
|     Additional FCP_DB length      |RDDAT|WRDAT|
+-----+-----+-----+-----+-----+-----+-----+-----+
|                                               |
≈              FCP_CDB (16 bytes)               ≈
|                                               |
+-----+-----+-----+-----+-----+-----+-----+-----+
\                                               \
/              Additional FCP_CDB               /
\                                               \
+-----+-----+-----+-----+-----+-----+-----+-----+
|                                               |
≈                FCP_DL (4 bytes)               ≈
|                                               |
+-----+-----+-----+-----+-----+-----+-----+-----+
|                                               |
≈      FCP_BIDIRECTIONAL_READ_DL (4 bytes)      ≈
|                                               |
+-----+-----+-----+-----+-----+-----+-----+-----+

CRN: command reference number
Priority: priority to execute task from queue, SIMPLE only
ATTR: task attribute

• 00: SIMPLE
• 01: HEAD OF QUEUE
• 10: ORDERED
• 11: ACA

TMF flags

• flags
  • 0x02: abort task set
  • 0x04: clear task set
  • 0x10: logical unit reset
  • 0x40: clear ACA
• only 1 bit can be set
• if ≠ 0, then it’s not a SCSI cmd
• abort task is done via ABTS

Additional FCP_DB length: units – 4-byte words; 0, if length(CDB) ≤ 16
RDDAT: 1 ≡ read
WRDAT: 1 ≡ write
Additional FCP_CDB: omitted, if length (CDB) ≤ 16 or TMF
FCP_DL: bytes, estimate how much data is expected; if bidir – then write data
FCP_BIDIRECTIONAL_READ_DL: bytes, bidir only (otherwise omitted)

FCP_XFER_RDY IU

   0     1     2     3     4     5     6     7
+-----+-----+-----+-----+-----+-----+-----+-----+
|                                               |
≈              FCP_DATA_RO (4 bytes)            ≈
|                                               |
+-----+-----+-----+-----+-----+-----+-----+-----+
|                                               |
≈            FCP_BURST_LEN (4 bytes             ≈
|                                               |
+-----+-----+-----+-----+-----+-----+-----+-----+
|                                               |
≈               Reserved (4 bytes)              ≈
|                                               |
+-----+-----+-----+-----+-----+-----+-----+-----+

DATA_RO: relative offset; position of the first byte from frame, relative to all data
BURST_LEN: bytes, how much data can be sent in FCP_DATA as a response

FCP_RSP IU

   0     1     2     3     4     5     6     7
+-----+-----+-----+-----+-----+-----+-----+-----+
|                                               |
≈              Reserved (8 bytes)               ≈
|                                               |
+-----+-----+-----+-----+-----+-----+-----+-----+
|               Retry delay timer               |
|                                               |
+-----+-----+-----+-----+-----+-----+-----+-----+
|                     Flags                     |
+-----+-----+-----+-----+-----+-----+-----+-----+
|               SCSI status code                |
+-----+-----+-----+-----+-----+-----+-----+-----+
|                                               |
|                   FCP_RESID                   |
|                                               |
|                                               |
+-----+-----+-----+-----+-----+-----+-----+-----+
|                                               |
|                  FCP_SNS_LEN                  |
|                                               |
|                                               |
+-----+-----+-----+-----+-----+-----+-----+-----+
|                                               |
|                  FCP_RSP_LEN                  |
|                                               |
|                                               |
+-----+-----+-----+-----+-----+-----+-----+-----+   
|                                               |
|                  FCP_RSP_INFO                 |
|                                               |
|                                               |
+-----+-----+-----+-----+-----+-----+-----+-----+
|                                               |
|                  FCP_SNS_INFO                 |
|                                               |
|                                               |
+-----+-----+-----+-----+-----+-----+-----+-----+
|                                               |
|         FCP_BIDIRECTIONAL_READ_RESID          |
|                                               |
|                                               |
+-----+-----+-----+-----+-----+-----+-----+-----+ 

Retry delay timer:

• codes from SAM-4 specification
• why failed and how long to wait before retry
• busy:
  • 0x0000: no additional info
  • 0x0001-0xfffd: µs, how long to wait before another command
  • 0xfffe: client has to stop sending
  • 0xffff: too many initiator ports serviced
• task set full
  • 0x0000: no additional info
  • 0x0001-0xfffe: wait until command completes or µs timeout expires
  • 0xffff: too many initiator ports serviced

Flags:

• 0x01: FCP_RSP_LEN_VALID
  • TMF: 1
  • command: 0 ≡ FCP success, 1 ≡ FCP fail
• 0x02: FCP_SNS_LEN_VALID
• 0x04: FCP_RESID_OVER: FCP_RESID valid, overflow
• 0x08: FCP_RESID_UNDER: FCP_RESID valid, underflow
• 0x10: FCP_CONF_REQ: 1 ≡ target requests CONF IU from initiator
• 0x20: FCP_BIDI_READ_RESID_OVER
• 0x40: FCP_BIDI_READ_RESID_UNDER
• 0x80: FCP_BIDI_RSP: 1 ≡ bidir fields are valid

SCSI Status code

• valid if FCP_RSP_LEN_VALID = 0
• if not GOOD, SCSI sense data in FCP_SNS_INFO
• code
  • 0x00: GOOD
  • 0x02: CHECK CONDITION
  • 0x04: CONDITION MET
  • 0x08: BUSY
  • 0x10: INTERMEDIATE
  • 0x14: INTERMEDIATE CONDITION MET
  • 0x18: RESERVATION CONFLICT
  • 0x28: TASK SET FULL
  • 0x30: ACA ACTIVE
  • 0x40: TASK ABORTED

FCP_RESID:

• underflow: bytes that were not received
• overlow: bytes that were not transmitted over expected limit, because there are no buffers

FCP_SNS_LEN: bytes
FCP_RSP_LEN: bytes (4 or 8)
FCP_RSP_INFO: omitted if FCP_RSP_LEN_VALID = 0; TMF status
FCP_SNS_INFO: omitted if FCP_SNS_LEN_VALID = 0

FCP_RSP_INFO

 0                   1                   2                   3
 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|            Reserved           | FCP_RSP_CODE  |   Reserved    |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

Code:

• 0x00: TMF complete
• 0x01: FCP_DATA length is different from FCP_BURST_LEN
• 0x02: FCP_CMND fields invalid
• 0x03: FCP_DATA parameter mismatch with FCP_DATA_RO
• 0x04: TMF rejected
• 0x05: TMF failed
• 0x09: TMF incorrect LUN

FCP_RJT

   0     1     2     3     4     5     6     7
+-----+-----+-----+-----+-----+-----+-----+-----+
|                                               |
|  FC-4 Link service Command code (0x01000000)  |
|                                               |
|                                               |
+-----+-----+-----+-----+-----+-----+-----+-----+
|                    Reserved                   |
+-----+-----+-----+-----+-----+-----+-----+-----+
|                  Reason code                  |
+-----+-----+-----+-----+-----+-----+-----+-----+
|              Reason explanation               |
+-----+-----+-----+-----+-----+-----+-----+-----+
|                Vendor specific                |
+-----+-----+-----+-----+-----+-----+-----+-----+

Reason code:

• 0x01: invalid FCP FC-4 link service command code
• 0x03: logical error
• 0x05: logical busy
• 0x07: protocol error
• 0x09: unable to perform command request
• 0x0b: command not supported
• 0xff: vendor specific error

Reason explanation

• 0x00: no additional info
• 0x03: invalid OX_ID-RX_ID combination
• 0x2a: unable to supply requested data

Virtualization

N_port ID virtualization (NPIV)

• allows N_port to use several FCID on a single link
• zoning and LUN masking per VM
• hosts sends FLOGI for itself, then FDISC per VM

(config)# feature npiv

; permit to bundle TF ports in port-channel
(config)# feature fport-channel-trunk

(config-if)# switchport mode F

N_port virtualizer (NPV)

• Cisco proprietary
• NPIV idea, applied to switch: aggregates N_port via multiple logins
  • initial FLOGI NPV-capable switch does on itw own behalf
  • host FLOGI are translated into FDISC
  • upstream switch – NPIV-capable
  • uplink – NP_port, native FC
• ≈ FEX: no local switching, does not require FCID, does not enforce zoning
  • all devices are on the same F_port ⇒ no zoning between them on upstream switch
• does not support arbitrated loop
• load-balancing:
  • modes
    • auto
      • selects uplink with min load
      • no reselection if new uplink is added
    • traffic map
      • upstream link is set manually per F_port
  • if uplink fails, connectivity is disrupted → relogin required
  • disruptive LB: reinit server interface on upstream link event

; disruptive: erase config + reload
(config)# feature npv

(config)# npv auto-load-balance disruptive
(config)# npv traffic-map server-interface <INTF> external-interface <INTF>

(config-if)# switchport mode NP

# show npv flogi-table

VM-aware FC

• add VMID to frame
• distinguish VMs, connected to datastore, on fabric level
• requires support from OS and HBA

Slow drain

• device issue, accept speed < source send speed ⇒ congestion in fabric, affecting other devices
• root causes on edge
  • software problem
  • HBA problem
  • HBA-held buffer: VM shutdown – not graceful
  • speed mismatch
  • storage overload
• root causes on ISL
  • B2B credits are not sufficient for the ISL length
  • slow drain at edge
• solution:
  • reduce timeout for edge and fabric
  • VOQ
• action:
  • congestion-drop: drop frame, if it is not forwarded through switch within 500 ms (default)
  • no-credit-drop: Tx B2B = 0 for 500 ms ⇒ drop all egress frames + drop ingress frames that should have been forwarded via this port + drop new frames, forwarded through this port
  • link reset, link flap
  • error-disable link

Fabric performance impact notification (FPIN)

• fabric notifies end devices about intermediate overload → hosts can change MPIO logic
• old devices are not compatible (e.g., no driver available)
• standard

Dynamic ingress rate limiting (DIRL)

• Cisco proprietary
• does not require support from end devices
• within single switch
• limits BB_credit for host with slow drain ≡ rate limit – decreases read command intensity
• port-channel on F_port towards NPV is worse than several uplinks: rate limit is applied on all links in contrast to 1 link, mapped to slow-draining host

TxWait

• tracks Tx BB_credit on hardware level (1 ns interval)
• if BB_credit stay 0 for 2.5 µs ⇒ increment TxWait

# show interface counter
# show process creditmon txwait-history

Slowport-monitor

• tracks Tx BB_credit on hardware level (1 ms interval)

(config)# system timeout slowport-monitor

# show process creditmon slowport-monitor-events

Link reset/link reset response primitives (LR/LRR)

• if LR is received but Tx has a frame → 90ms LR Rcvd timer
  • if during timer all frames are sent → LRR
  • otherwise NOS
• LR is sent if port does not have Tx BB_credits for long
  • E_port: 1.5s
  • F_port: 1s

# attach module <N>
module-N# show port-config internal link-events

# slot <N> show port-config internal link-events

# show process creditmon credit_loss_events

Congestion-isolation

• uses extended receiver ready (ER_RDY)
• ISL is split into 4 VL: each VL has its own BB_credit
  1. QoS 7: control traffic
  2. QoS 5,6: unused
  3. QoS 2,3,4: slow traffic
  4. QoS 0,1: normal traffic
• slow device definition
  • PMON
  • manually

(config)# system fc flow-control er_rdy
(config)# feature congestion-isolation
(config)# congestion isolation include pwwn <PWWN> vsan <VSAN>

FC Port-channel

• mode
  • on
    • no dynamic protocol
    • interface with settings different from port-channel → suspended
    • if member is added – manual port-channel flap to apply new config
    • does not detect misconfiguration
    • default
    • no supported on F_port
  • active
    • negotiation with PCP
    • interface with settings different from port-channel → isolated
    • if member is added/removed – autorecover
    • synchronized link startup
    • misconfig detection
    • if on mode is detected – switch to on
• deskew
  • time difference between traffic, passing different links
  • relevant for long distance
  • the lower, the better
• link failure does not affect RIB
• up to 16 links in bundle
• if miswired – port-channel down, no standalone links
• load-balancing:
  • src-dst FCID
  • src-dst FCID + OX_ID (flow does not include *X_ID)
• first operational port (FOP): forwards control plane traffic

(config-if)# channel-group <N>

(config)# interface port-channel <N> | san-port-channel <N>
(config-if)# channel mode on|active

# show san-port-channel

FC domain

• principal switch election
• fabric reconfiguration: all switches simultaneously elect principal switch after fabric restart (RCF received)
• domain ID distribution: guarantee unique domain ID per switch
• FCID allocation: unique FCID
  • if FCID is assigned manually, it is included into domain ID request to inform principal switch
  • modes
    • preferred: if requested and assigned FCID conflict, use assigned FCID
    • static: recommended; if requested and assigned FCID conflict, switch → isolated
  • applying FCID:
    • reconfigure fabric (RCF): disruptive
    • build fabric (BF): not disruptive
• VSAN scope
• auto-reconfiguration:
  • if enabled for both fabrics: FCID change, disruptive
  • if one fabric does not have it enabled: ISL → isolated

(config)# fcdomain rfc-reject vsan <N>

(config-if)# fcdomain rfc-reject vsan <N>

# show fcdomain# show fcroute unicast

Principal switch

• master switch for name database, zone, zoneset
• assigns subordinate FCID (domain ID)
  • manually configured
  • random that is not already assigned
• election
  1. lowest priority: 128 default
  2. lower sWWN
     • if principal is elected using sWWN, new principal assign itself priority = 2
• preemptive
• authoritative time source
• election process
  1. empty domain ID list
  2. send BF
  3. send EFP SW_ILS for F_S_TOV (fabric stability timeout value)
  4. if better principal switch is received – F_S_TOV reset + transmit new information further (≈ STP); reception is acknowledged by SW_ACC
  5. F_S_TOV expired ≡ principal switch elected

Persistent FCID

• preserve assigned FCID value throughout reloads
• device can connect to any port without changing FCID
• enabled by default

Fabric initialization

Link initializtion

1. power-on
2. send OLS during R_T_TOV (receiver transmitter timeout value) at max supported speed
3. if no response during R_T_TOV, sent NOS and try lower speed
4. is response received → bit level sync and word alignment
5. once 3 consecutive OLS are received without error → send LR
6. once 3 consecutive LE are received without error → send LRR
7. once 3 consecutive LRR are received without error → send idle ≡ port is ready

Switch fabric internal link service (SW_ILS)

• ELS
• only between FC switches
• BF, RCF

Exchange link parameters (ELP)

• exchange B2B credits, timers, QoS
• destination = fabric controller address

   0     1     2     3     4     5     6     7
+-----+-----+-----+-----+-----+-----+-----+-----+
|                                               |
≈         SW_ILS Command code (4 bytes)         ≈
|                                               |
+-----+-----+-----+-----+-----+-----+-----+-----+
|                    Revision                   |
+-----+-----+-----+-----+-----+-----+-----+-----+
|  B  | VF  | FCF | FDF |    EEO    |           |
+-----+-----+-----+-----+-----+-----+           +
|                    Reserved                   |
+-----+-----+-----+-----+-----+-----+-----+-----+
|                    BB_SC_N                    |
+-----+-----+-----+-----+-----+-----+-----+-----+
|                                               |
≈                R_A_TOV (4 bytes)              ≈
|                                               |
+-----+-----+-----+-----+-----+-----+-----+-----+
|                                               |
≈                E_D_TOV (4 bytes)              ≈
|                                               |
+-----+-----+-----+-----+-----+-----+-----+-----+
|                                               |
≈    Requestor/Responder Port Name (8 bytes)    ≈
|                     pWWN                      |
+-----+-----+-----+-----+-----+-----+-----+-----+
|                                               |
≈   Requestor/Responder Switch Name (8 bytes)   ≈
|                     sWWN                      |
+-----+-----+-----+-----+-----+-----+-----+-----+
|                                               |
≈     Class F Service Parameters (16 bytes)     ≈
|                                               |
+-----+-----+-----+-----+-----+-----+-----+-----+
|                                               |
≈ Class 1 Interconnect Port Parameter (4 bytes) ≈
|                                               |
+-----+-----+-----+-----+-----+-----+-----+-----+
|                                               |
≈ Class 2 Interconnect Port Parameter (4 bytes) ≈
|                                               |
+-----+-----+-----+-----+-----+-----+-----+-----+
|                                               |
≈ Class 3 Interconnect Port Parameter (4 bytes) ≈
|                                               |
+-----+-----+-----+-----+-----+-----+-----+-----+
|                                               |
≈              Reserved (20 bytes)              ≈
|                                               |
+-----+-----+-----+-----+-----+-----+-----+-----+
|             ISL Flow control mode             |
|                                               |
+-----+-----+-----+-----+-----+-----+-----+-----+
|       Flow control mode parameter length      |
|                                               |
+-----+-----+-----+-----+-----+-----+-----+-----+
\                                               \
/            Flow control parameters            /
\                                               \
+-----+-----+-----+-----+-----+-----+-----+-----+

Command code: 0x10000000 (ELP), 0x20000000 (ELP SW_ACC)
B: bridge port, 0 ≡ E_port, 1 ≡ B_port
VF: virtual fabrics, 0 ≡ no tagged frames, 1 ≡ tagged frames supported
FCF: 1 ≡ controlling switch
FDF: 1 ≡ FCDF

EEO: energy efficient operation

• 00: no support
• 01: fast wake
• 11: fast wake and energy efficient

R_A_TOV: resource allocation timeout, common for whole fabric
Class 1 – obsolete

ISL Flow control mode

• 0x0001: vendor specific
• 0x0002: R_RDY flow control
• 0x0003-0x1fff: vendor specific
• 0x2000: VC_RDY flow control
• 0xae02: AE use

Class F service parameters

   0     1     2     3     4     5     6     7
+-----+-----+-----+-----+-----+-----+-----+-----+
| CV  |                                         |
+-----+                                         +
|                    Reserved                   |
|                                               |
|                                               |
+-----+-----+-----+-----+-----+-----+-----+-----+
|  Reserved | XII |                             |
+-----+-----+-----+                             +
|                    Reserved                   |
+-----+-----+-----+-----+-----+-----+-----+-----+
|            Receive data field size            |
|                                               |
+-----+-----+-----+-----+-----+-----+-----+-----+
|             Concurrent sequences              |
|                                               |
+-----+-----+-----+-----+-----+-----+-----+-----+
|                   EE_credit                   |
|                                               |
+-----+-----+-----+-----+-----+-----+-----+-----+
|          Open sequences per exchange          |
|                                               |
+-----+-----+-----+-----+-----+-----+-----+-----+
|                    Reserved                   |
|                                               |
+-----+-----+-----+-----+-----+-----+-----+-----+

CV: class valid, = 1
XII: X_ID interlock, 1 ≡ init sends bcast frame at the start of sequence, target responds with its address in ACK, then data flows

Class 2,3 interconnect port parameters

   0     1     2     3     4     5     6     7
+-----+-----+-----+-----+-----+-----+-----+-----+
| CV  |    Reserved     | SEQ |                 |
+-----+-----+-----+-----+-----+                 +
|                    Reserved                   |
+-----+-----+-----+-----+-----+-----+-----+-----+
|            Receive data field size            |
|                                               |
+-----+-----+-----+-----+-----+-----+-----+-----+

SEQ: sequential delivery, 1 ≡ in-order guarantee, 0 ≡ out-of-order

Exchange switch capabilities (ESC)

• negotiate IGP
• discover peer vendor ID

 0                   1                   2                   3
 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Command code  |   Reserved    |        Payload length         |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                        Vendor ID string                       |
|                                                               |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+--+
|                   Vendor ID string protocol                   |   \
|                                                               |   >  protocol
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+   / descriptors
|            Reserved           |          Protocol ID          |  |    (list)
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+--+

Command code: 0x30 (ESC), 0x02 (ESC SW_ACC)
Vendor ID string protocol = 0, if not vendor specific

Exchange peer parameters (EPP)

• Cisco proprietary
• trunk negotiation

Exchange fabric parameters (EFP)

• principal switch election
• PS_priority, PS_WWN, Domain_ID_list
• domain ID list
  • NULL during PS election
  • appended after every domain ID assigned
  • initiated by PS
  • SS forward everywhere except upstream ISL

 0                   1                   2                   3
 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Command code  | Record length |        Payload length         |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                    Reserved                   |  PS priority  |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                            PS name                            |
|                              sWWN                             |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+--+
|  Record type  |   Domain ID   |                               |  |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+                               +   \
|                            Reserved                           |    > domain ID
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+   /    list
|                      sWWN for domain ID                       |  |
|                                                               |  |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+--+

Command code: 0x11 (EFP), 0x02 (EFP SW_ACC)
PS priority

• 0x00: reserved
• 0x01: highest
• 0x02: switch was BS prior to BF
• 0xff: not capable to be PS

Record type

• 0x01: domain ID list
• 0x02: obsolete

Domain ID assigned (DIA)

• principal assigns itself domain ID and transmits DIA frames
• not forwarded beyond recipient switch
• when SS is assigned domain ID, it forwards DIA everywhere except ISL towards PS

 0                   1                   2                   3
 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                      SW_ILS Command code                      |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                          Switch name                          |
|                                                               |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                            Reserved                           |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 

Command code: 0x12000000 (DIA), 0x02000000 (DIA SW_ACC)

Request domain ID (RDI)

• requests domain ID through ISL towards PS after receiving DIA
• if SS does not accept assigned domain ID from PS, SS → isolated
• domain ID = 0x00 ≡ any ID, no preference for ID
• intermediate switches forward RDI over ISL towards PS

 0                   1                   2                   3
 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Command code  |   Reserved    |        Payload length         |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                          Switch name                          |
|                                                               |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+--+
|                   Reserved                    |   Domain ID   |   > list
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+--+

Command code: 0x13000000 (RDI), 0x02000000 (RDI SW_ACC)

SW_ACC

• acknowledges reception
• contains domain ID as a reply for RDI
• intermediate switches forward to destination

SW_RJT

   0     1     2     3     4     5     6     7
+-----+-----+-----+-----+-----+-----+-----+-----+
|                                               |
|        SW_ILS Command code (0x01000000)       |
|                                               |
|                                               |
+-----+-----+-----+-----+-----+-----+-----+-----+
|                    Reserved                   |
+-----+-----+-----+-----+-----+-----+-----+-----+
|                  Reason code                  |
+-----+-----+-----+-----+-----+-----+-----+-----+
|              Reason explanation               |
+-----+-----+-----+-----+-----+-----+-----+-----+
|                Vendor specific                |
+-----+-----+-----+-----+-----+-----+-----+-----+

Reason code:

• 0x01: invalid SW_ILS command code
• 0x02: invalid revision level
• 0x03: logical error
• 0x04: invalid payload size
• 0x05: logical busy
• 0x07: protocol error
• 0x09: unable to perform command request
• 0x0b: command not supported
• 0x0c: invalid attachment
• 0xff: vendor specific error

Reason explanation

• 0x00: no additional info
• 0x01: class F service parameter error
• 0x03: class n service parameter error
• 0x04: unknown flow control code
• 0x05: invalid flow control parameters
• 0x0d: invalid pWWN
• 0x0e: invalid sWWN
• 0x0f: R_A_TOV or E_D_TOV mismatch
• 0x10: invalid domain ID list
• 0x19: command already in progress
• 0x1e: PLOGI required
• 0x29: insufficient resources available
• 0x2a: domain ID not available
• 0x2b: invalid domain ID
• 0x2c: request not supported
• 0x2d: link parameters not yet established
• 0x2e: requested domain IDs not available
• 0x2f: E_port is isolated
• 0x31: authZ failed
• 0x32: authC failed
• 0x33: imcompatible security attribute
• 0x34: checks in progress
• 0x35: policy summary not equal
• 0x36: FC-SP zoning summary not equal
• 0x41: invalid data length
• 0x42: unsupported command
• 0x44: not authorized
• 0x45: invalid request
• 0x46: fabric changing
• 0x47: update not staged
• 0x48: invalid zoneset format
• 0x49: invalid data
• 0x4a: unable to merge
• 0x4b: zoneset size not supported
• 0x50: unable to verify connection
• 0x58: requested application not supported

Cisco fabric services (CFS)

• P2P
• scopes
  • logical: VSAN
  • physical
  • set of VSAN: for IVR
• modes
  • coordinated distribution: only 1 distribution per fabric at any moment of time (fabric lock)
  • uncoordinated distribution: several distributions while there is no coordinated distribution
  • unrestricted uncoordinated distribution: several distributions even if coordinated is active
• CFS region: physical switches, not per VSAN

# show cfs merge status name device-alias

Design

• fanout ratio
  • number of servers per storage device port
  • factors:
    • HBA queue depth
    • storage IOPS
    • port throughput
  • 7:1 – 15:1
• enable disruptive load-balancing only if links have to be recalculated: reinit, FLOGI to upstream
• 1-1 zoning to reduce TCAM utilization
• SAN A/B to protect from drops during FSPF convergence and configuration errors
• IVR for multisite fabric: errors are confined within VSAN
  • VSAN per site
  • transit VSAN

Security

FC-SP-2

• authC FC entity: switch-switch, switch-device, device-device
• session encryption: IKEv2, ESP, CT (common transport protocol, ≠ AH)
  • ESP: data
  • CT: control frames
• policy is distributed within fabric

FC authentication

• methods
  1. CHAP: password
  2. FC-AP: FC certificate authC protocol
  3. FC-PAP: secure remote password (SRP) ≡ hash(salt, secret) as password
  4. FC-EAP
  5. IKEv2-AUTH

Port-security

• deny at FLOGI
• uses pWWN, nWWN, sWWN
• auto-learn:
  • ≈ sticky
  • changes are not applied until auto-learn is disabled
  • disabled by default
  • does not overwrite configuration
  • denied only if request is received on port that is configured for other device
• per VSAN
• CFS distributes active DB
• split DB into configuration and active

(config)# feature port-security

(config)# port-security database vsan <N>
(config-port-security)# swwn <SWWN> interface <INTF>
(config-port-security)# any-wwn interface <INTF>

; enable auto-learning, no-auto-learn ≡ disable auto-learning + enable port-security
(config)# port-security activate vsan <N> [no-auto-learn]

; disabled auto-learning
(config)# no port-security auto-learn vsan <N>

(config)# port-security distribute

; active → configuration DB
# port-security database copy vsan <N>

# show port-security database
# show port-security violation

Fabric binding

• ISL is established only between authorized switches
• exchange fabric membership data protocol (EFMD)
• no support for CFS
• uses sWWN
• per VSAN
• port security subset
• if sWWN is absent from DB – ISL is isolated in VSAN

(config)# feature fabric-binding

(config)# fabric-binding database vsan <N>
(config-fabric-binding)# swwn <SWWN> [domain <M>]

(config)# fabric-binding activate vsan <N>

Telemetry

• ≈ Netflow for SCSI/NVMe
• SAN analytics requires license
  • extra license for DCNM is used as collector
• incompatible with IVR
• requires NTP

(config)# feature analytics

(config-if)# analytics type fc-scsi|fc-nvme|fc-all