Switching hardware

1. Switching device manager
2. Power over Ethernet (PoE)
3. Interfaces
   1. Switch virtual interface (SVI)
   2. Wedged interface
   3. Subinterfaces
   4. Null0
   5. GRE
   6. Tunnel
4. Errors
   1. Err-disable
   2. Interface error counters
   3. Excessive CPU utilization
5. SFP
6. Hardware queue
7. Thresholds
8. Redundancy
   1. Non-stop forwarding (NSF)
   2. Non-stop routing (NSR)
   3. Stackwise
   4. Stackwise+
   5. Stackwise-480
   6. FlexStack
   7. StackWise Virtual (VSS)
9. Config archive
10. Embedded event manager (EEM)
11. Dynamic VLAN
12. Boot
13. License
14. Alias
15. Generic online diagnostics (GOLD)
16. In-service software upgrade (ISSU)
    1. Software maintenance upgrade (SMU)
    2. Electrical programmable logical devices (EPLD)
17. Callhome
18. Cisco fabric services (CFS)
19. Cisco Nexus
    1. NX-API
    2. NX-OS RBAC
    3. Adjacency manager (AM)
    4. Host mobility manager (HMM)
    5. PBR
    6. L3 MTU
    7. L2 MTU
    8. Upgrade
    9. Checkpoint
    10. Power-on auto provisioning (POAP)
    11. Port Profile
    12. Graceful insertion and removal
    13. Unified ports
    14. N5k
    15. N7k
        1. Virtual device context (VDC)
        2. Storage VDC
        3. IDS

Switching device manager

• templates:
  1. access: max ACL
  2. default
  3. dual-ipv4-and-ipv6: IPv6 support
  4. routing: IPv4 routing
  5. vlan: max CAM, disable hardware routing

; reload is required
(config)# sdm prefer <TEMPLATE>

; TCAM resource distribution across the functions
# show sdm prefer

Power over Ethernet (PoE)

• implementations:
  • Cisco inline
    • FE only
    • max wattage: 7.7W
  • IEEE 802.3af
    • initial wattage: 12.95W
    • max wattage: 15.40W (2.4W of losses)
  • IEEE 802.3at
    • initial wattage: 25.50W (4 wires), 51W (8 wires, non-standard)
    • max wattage: 34.20W
  • IEEE 802.3bt (UPoE)
    • max wattage: 60W (8 wires), 90W
    • PoE pass-through (e.g., 3560CX): power sensors, cameras
• consumption
  • phone: 5W standby, 7W on call
  • AP: 12W, 802.11ac – 22W
• if client does not request specific power level with CDP/LLDP, it is allocated initial wattage level
• PoE client:
  • classes:
    • 0 ≡ no info
    • 1 ≡ max 4W
    • 2 ≡ max 7W
    • 3 ≡ max 15.4W
    • 4 ≡ max 30W
  • discovery:
    • Cisco inline: send tone that is echoed back through client via closed circuit
    • IEEE: measure resistance (kΩ), value → class
• power delivery type:
  • A:
    • over 2 unused wires
    • up to 100 Mbps
    • Cat3, Cat4, Cat5e
  • B:
    • phantom power
    • 2-10V initially; if device is PoE-capable – 48V, otherwise – 0V
• perpetual PoE: provides power during software-initiated reboot
• fast PoE: provides last negotiated power level during boot prior to negotiation

(config-if)# power inline never

; real power supply, provided by auto mode for PoE-capable
; useful if CDP is disabled, does not account for power loss in cable
(config-if)# power inline consumption <mW>

; max valid request for power from client, no client – no power
(config-if)# power inline auto max <mW>

; reserve from power bank regardless of whether client is actually present
; ignores consumption
(config-if)# power inline static max <mW>

; perpetual PoE + fast PoE
(config-if)# power inline port poe-ha

Interfaces

; turn Ethernet into unnumbered interface
(config-if)# medium p2p

; 300s default, interval to calculate statistics over
(config-if)# load-interval <sec>

Switch virtual interface (SVI)

• create SVI ≠ create VLAN
• up/up:
  1. VLAN exists
  2. not admin down
  3. at least one port (access/trunk) in STP forwarding (either access or voice VLAN)
• down/down:
  1. VLAN does not exist
  2. VLAN shutdown/suspended
• up/down
  1. VLAN exists
  2. no port with VLAN in STP forwarding
• SVI autostate: SVI does not switch to up/up, unless at least one port in VLAN is up/up and not blocked by STP

; on SVI, state – permanently active
(config-if)# no autostate

; physical interface, disable affecting SVI autostate
; useful on ports to sniffers, sniffer exists ≠ VLAN active
(config-if)# switchport autostate exclude

; NX-OS, enable SVIs
(config)# feature interface-vlan

Wedged interface

• interface with full input queue
• does not forward ingress traffic

Subinterfaces

• router physical interface does not accept tagged frames, only subints do (required for QoS)
• FastEthernet0/0.0 ≡ FastEthernet0/0

Null0

• static routes via Null0 are connected ⇒ affected by network or redistribute connected

GRE

• IP 47
• line status is based upon destination reachability by default
• may be not supported in HW ⇒ processed by CPU
• can transport several protocols at once
• IPsec – on-demand

# enable IPv4 forwarding
$ echo 1 > /proc/sys/net/ipv4/ip_forward

# load kernel module
$ modprobe ip_gre

# add tunnel interface
$ ip tunnel add <GRE> mode gre remote <remoteIP> local <localIP>
$ ip addr add <IP> dev <GRE>
$ ip link set <GRE> up

# add route via tunnel
$ ip route add <route> via <nexthop>

# disable RPF check (strong send and receive)
$ echo 0 > /proc/sys/net/ipv4/conf/all/rp_filter
$ echo 0 > /proc/sys/net/ipv4/conf/<GRE>/rp_filter

Tunnel

• DF-bit from passenger packet by default
  • Cisco: not copied to tunnel header
  • Juniper: copied to tunnel header
• PMTUD value is unavailable through show commands
  • value is used even after disabling the feature
  • low value can cause 100% CPU util due to fragmentation – attack opportunity
  • when ICMP Fragmentation Needed is received, accounts for only external IP header, does not include GRE ⇒ PMTUD affects all tunnels with the same endpoints
• ToS byte is copied from inner DSCP to outer DSCP

; copies DF-bit from passenger to transport
(config-if)# tunnel path-mtu-discovery

; shows fragmentation according with PMTUD value
# debug tunnel

Errors

Err-disable

• modes:
  • ARP inspection
  • BPDU guard
  • DHCP rate limit: via DHCP snooping
  • DTP flag: change of encapsulation
  • GBIC invalid
  • inline power
  • link flap
  • loopback: short circuit to itself
  • PAgP flap
  • Port Security violation
  • Security violation: 802.1x
  • small frame
  • storm control
  • UDLD

(config)# errdisable detect cause all|<MODE>

; autorecover interface from err-disable
(config)# errdisable recovery cause all|<MODE>

; 300s default
(config)# errdisable recovery interval <sec>

# show interface status err-disabled

; services for err-disable
# show errdisable detect

; services with auto-recovery, ports in err-disabled and cause for error
# show errdisable recovery

Interface error counters

1. Align-Err:
   • L1 issue
   • alignment misses byte boundary
2. FCS-Err:
   • L1 issue, duplex mismatch
   • only CRC is incorrect
3. Xmit-Err:
   • speed mismatch
   • egress buffer overflow
4. Rev-Err:
   • congestion, duplex mismatch
   • ingress buffer overflow
5. UnderSize:
   • less than 64 bytes
   • CRC is correct
6. Single-Col:
   • single collision before successful transmission
   • duplex mismatch, overflow for half-duplex
7. Multi-Col:
   • more than one collision before successful transmission
8. Late-Col:
   • collision after 64 bytes
   • duplex mismatch, cable is too long
9. Excess-Col:
   • after 16 successive collisions
   • too long cable, duplex mismatch, too many users
10. Carri-Sen:
    • increased on every attempt to send
11. Runts:
    • less than 64 bytes
    • CRC is incorrect
    • L1 issue, duplex mismatch
12. Giants:
    • > 1518 bytes, incorrect CRC
    • host NIC failure

; counter for 5s: load/interrupt load
# show processes cpu
# show memory
# show interfaces
# show controllers
# show platform
# show module

; routers, burned-in info and EEPROM
# show diagnostics

; shortened diag
# show inventory

Excessive CPU utilization

• ARP input process
• Net background process: buffer distribution for interfaces
• IP background process: interface status change
• TCP timer process:
  • every TCP session to device
  • resets embryonic state (no ACK after SYN+ACK)
• 5% is normal, up to 10%
• reasons
  • TCAM overload
  • frequent IGP/FHRP/SNMP
  • debug

SFP

; hidden command, allows using non-Cisco SFP
(config)# service unsupported-transceiver

; disable err-disable for non-Cisco SFP
(config)# no errdisable detect cause gbic-invalid

Hardware queue

• transmit ring
• does not involve CPU ⇒ no QoS
• FIFO
• the faster interface is, the deeper queue is:
  • shallow queue ≡ more CPU interrupts ≡ more wait state
  • deep queue ≡ penalty for voice, real-time traffic

; 60 default, output hardware queue
(config-if)# tx-ring-limit <pkts>

; software queue, defaults: 75 in, 40 out
(config-if)# hold-queue <pkts> in|out

Thresholds

(config)# process cpu threshold type process rising <%> interval <sec>

; low threshold, high = low + 5%
(config)# memory free low-watermark IO|processor <KB>

Redundancy

• modular switch with 2 supervisors
• master – the first switch to boot, no preemption after failover
  • C4k: L2 failover < 1s
  • C6k: L2 failover < 0-3s
• modes
  • Route Processor Redundancy (RPR)
    • standby is partially initialized
    • module reboot on failover
    • 2 mins
  • RPR+
    • route engine is fully initialized
    • modules are not reloaded
    • IOS must match
    • 30s
  • Stateful switchover (SSO)
    • full initialization
    • 1s
• SRM – single router mode
• DRM – dual router mode

(config)# redundancy
(config-red)# mode rpr|rpr-plus|sso

# show redundancy status

Non-stop forwarding (NSF)

• along with SSO, uses CEF that was built by failed master
• requests RIB information from peer
• indicates that SSO happened, not peer disappeared

Non-stop routing (NSR)

• protocol data is synced between RPs
• seamless switchover for control plane

Stackwise

• source stripping
• switching through master ≡ through backplane
• master election
  • process
    1. priority: manual config
    2. switch with configured interfaces > blank switch
    3. better feature set: crypto IP services > IP services > crypto IP base > IP base
    4. longest uptime
    5. min MAC
  • 20s delay after boot before election – allow heavier IOS to load
• common IP, configuration, CAM
• IOS must match (feature set does not have to, but recommended)
• up to 9 switches
• no master preemption
• IOS is different:
  1. master sends IOS from Flash along with config
  2. master downloads IOS via TFTP
  3. if subordinate switch does not support image, master puts it into suspended state
• supports RPR+
• physical stack cables – Cisco only

; 15 default
(config)# switch <N> priority <M>
(config)# switch <N> renumber <NEW_NUM>

; list of switches in stack, stack ports and their status
# show switch [stack-ports]

; closed stack ring – Full config, half of ring – Half (½ Full)
# show switch stack-ring speed
# show platform stack manager all

Stackwise+

• local switching
• destination stripping
• C3750X

Stackwise-480

• 480 Gbps backplane
• C3850

FlexStack

• up to 4 switches
• C2960S

StackWise Virtual (VSS)

• distribution-level campus switches
• virtual switch link (VSL) ≡ etherchannel (8×10GE max)
  • control and mgmt plane are prioritized
  • dual-active detection: VSL is down, but control plane is up
    • Standby → Active
    • Active: non-VSL ports → err-disable
    • detection:
      • fast-hello link
      • enhanced PAgP (includes VSS ID in Update) via another switch (upstream/downstream)
• 2 devices only
• SSO + NSF
• no preemption

(config)# redundancy
(config-red)# mode sso|rpr

; Active and Standby sync options
(config-red)# main-cpu
(config-r-mc)# auto-sync running-config 

; DOMAIN must match
(config)# switch virtual domain <DOMAIN>

; switch number in VSS
(config-vs-domain) switch <N>

; off default, recovery IP for mgmt on switch incase of dual-active
(config-vs-domain) dual-active recovery ip address <IP>

; on default
(config-vs-domain) dual-active detection pagp

; enable ePAgP on etherchannel <N>
(config-vs-domain) dual-active detection pagp trust channel <N>

(config-vs-domain) dual-active detection fast-hello

(config)# interface port-channel 1
(config-if)# switchport

; switch N – owner of the po1, different port-channels on switches
(config-if)# switch virtual link <N>

; on physical interface
(config-if)# dual-active fast-hello

; enable VSS
# switch convert mode virtual
# show switch virtual [link]
# show switch virtual role
# show switch virtual port-channel
# show switch virtual dual-active

Config archive

(config)# archive
(config-archive)# path <PATH>

; disabled by default
(config-archive)# time-period <mins>

(config-archive)# write-memory

; 10 default, number of local archives
(config-archive)# maximum <N>

; merges config, if tie – use value from startup-config
# copy startup-config running-config

; replaces running-config with startup-config, list – dry run ≡ only show the difference
# configure replace <PATH> [list]

; replacement must be confirmed within time window, otherwise – revert back
# configure replace <PATH> time <mins>

; confirm replacement after using time keyword
# configure confirm

; change time window for confirm
# config revert now|timer <mins>

; does not show the output, rewrites destination file contents
# show run | redirect <URL>

; shows the output, rewrites destination file contents
# show run | tee <URL>

; does not show the output, appends to destination file contents
# show run | append <URL>

; pipes after first pipe – logical OR in regex
# show run | include <S1>|<S2>|<S3>

Embedded event manager (EEM)

• custom events and reactions to events
• triggers: syslog, SNMP trap, IOS cmd

; NAME appears in Syslog
(config)# event manager applet <NAME> [override <SYSTEM_POLICY>]

; sync no ≡ run along with CLI command being executed, do not waitfor cmd
; skip no ≡ run command instead of skipping it
; occurs 1 ≡ 1 event is enough for event to be triggered
(config-applet)# event cli pattern <REGEX> sync no skip no occurs 1

(config-applet)# action <ACTION> syslog priority <LEVEL> msg <MESSAGE>

; permit manual run
(config-applet)# event none

(config)# event environment <VARNAME> <VALUE>

; if manual run is not permitted, syslog about "not registered to event none"
# event manager run <NAME>

Dynamic VLAN

• access ports only
• based on client MAC
• deprecated, replaced by 802.1x
• PortFast is enabled automatically
• not compatible with port-security, 802.1x, SPAN, Etherchannel
• all VLANs must be defined on all switches (e.g., distributed via VTP)
• VLAN membership policy server: VMPS

; up to 3 secondary servers
(config)# vmps server <IP> [primary]

(config-if)# switchport mode access
(config-if)# switchport access vlan dynamic

# show vmps
# vmps reconfirm

Boot

• sequence
  1. POST
  2. bootstrap: ROM → RAM
  3. bootstrap selects and loads IOS
  4. IOS searches for startup-config
• ROMMON: ROM monitor
  • invoked on Break during boot (old) or external Flash missing (new)
• BootROM: RxBoot, boot helper (legacy devices) – replaced by ROMMON
• confreg:
  • 0x2100: load ROMMON in lieu of IOS
  • 0x2102: load IOS, default
  • 0x2142: ignore startup-config

; selects first file on Flash, if path not specified 
(config)# boot system flash [<FILE>]
(config)# boot system tftp <FILE> <IP>

# config-register <VALUE>
# show version

; run first-run setup script
# setup

License

; unique device ID: product ID + serial number
# show license udi
# show license feature
# license install <PATH>

Alias

(config)# alias <MODE> <CMD>

Generic online diagnostics (GOLD)

• proactive
  • boot diagnostics
  • health monitor: non-disruptive tests
  • periodic background
• reactive
  • on-demand, scheduled
  • disruptive: used during manufacturing

; complete default, bypass ≡ skip bootup test
(config)# diagnostic bootup level complete|bypass

# show diagnostic boot level

# diagnostic start module <N> test <ID>|<NAME>|all|non-disruptive

In-service software upgrade (ISSU)

• sequence
  • Nexus
    • N7K
      1. BIOS
      2. boot standby, → active
      3. boot old active
      4. data module
      5. CMP on supervisor
    • N9K
      1. supervisor with switchover
      2. line card
      3. fabric module
      4. system controller
      5. FEX
    • kickstart: Linux
    • system: NX-OS
• N9k:
  • requires 2 system controllers and 2 fabric modules
  • divides LC and FM into 2 group, upgrade once group at a time

Software maintenance upgrade (SMU)

• patch to fix defect
• issued by TAC
• not supported on classic IOS (monolith), MDS

; runtime
# install activate <PATH>

; applied after reboot
# install commit <PATH>

Electrical programmable logical devices (EPLD)

• FPGA, CPLD

# show install all impact epld <PATH>
# show version module <N> epld

; Xbar utilization
# show hardware fabric-utilization detail

# install module <N> epld <PATH>
# install all epld <PATH> parallel xbar-module <SLOT>

Callhome

• sent on trigger: page, e-mail (up to 50 addresses), TAC case
• format
  • short: paging
  • full
  • XML: TAC

(config)# callhome
(config-callhome)# destination-profile <PROFILE> format xml|full|short
(config-callhome)# destination-profile <PROFILE> email-addr <EMAIL>

; webhook
(config-callhome)# destination-profile <PROFILE> http <URL>

(config-callhome)# destination-profile <PROFILE> transport-method email|http
(config-callhome)# destination-profile <PROFILE> alert-group <ALERT>

(config-callhome)# alert-group <ALERT> user-def-cmd <CLI>

(config-callhome)# transport email from <EMAIL>
(config-callhome)# transport mail-server <IP> use-vrf <VRF>

(config-callhome)# enable

Cisco fabric services (CFS)

• synchonize within region
  • AAA, RBAC
  • Syslog
  • NTP
  • SCSI
  • VSAN, IVR
  • iSNS
• can discover CFS-capable devices
• designated manager: applies config locally and distributes it across CFS
• CFSoFC
  • FC-2
  • fabric controller address
• FC > IP > Ethernet

(config)# role distribute

; CFSoFC, on default
(config)# cfs distribute

; CFSoEthernet
(config)# cfs eth distribute

; CFSoIP
(config)# cfs ipv4 distribute
(config)# cfs ipv4 mcast-address <IP>

; disables discovery
(config)# cfs static-peers
(config-cfs-static)# ip address <IP>

; create region
(config)# cfs region <N>

; add service to region
(config-cfs-region)# ntp

# show cfs status
# show cfs application
# show cfs lock
# show cfs static peers

Cisco Nexus

NX-API

• sandbox, CLI commands via HTTP
• nginx
• XML, JSON
• authC cookie timeout – 10 mins

(config)# feature nxapi
(config)# nxapi sandbox
(config)# nxapi https port <N>

# show ... | xml|json

; run Python script
# source <PATH>

NX-OS RBAC

• descending order: rule 2 evaluated earlier than rule 1
• the most privileged union across all roles is applied
  • permit > deny across roles
• privilege compatibility:
  • 0 ≡ all show, some exec
  • 14 ≡ vdc-admin
  • 15 ≡ network-admin
  • translates privilege level from AAA into role priv-X
• rule parameters
  • command
  • feature
  • feature group
  • SNMP OID

(config)# feature privilege

Adjacency manager (AM)

# show forwarding <PREFIX>
# show forwarding adjacency

Host mobility manager (HMM)

• installs host route into RIB
• based on ARP table

; hmm makes sure that route is known via HMM in lieu of recursive routing, EVPN
(config)# track <N> ip route <HOST_ROUTE> reachability [hmm]

# show fabric forwarding ip local-host-db [vrf <NAME>]

PBR

; off default, enables hit counts
(config)# route-map <RMAP> pbr-statistics

L3 MTU

; L3 port
(config-if)# mtu <N> 

L2 MTU

• N5k and N7k only
• N9k – global jumbo MTU

(config)# policy-map type network-qos <PMAP>
(config-pmap-nq)# class type network-qos class-default
(config-pmap-c-nq)# match qos-group 0
(config-pmap-c-nq)# mtu 9216

(config)# system qos
(config-sys-qos)# service-policy type network-qos jumbo

# show queueing interface <INTF>

Upgrade

• mode
  • non-disruptive
    • only control plane is down, data plane remains active
    • upgrade time does not exceed 80s (N5k) or 120s (N9k)
    • no topology change
  • disruptive
    • graceful stop is impossible (e.g., failure)
    • EPLD upgrade
• upgrade targets
  • FEX
  • kickstart image
  • system image
  • supervisor BIOS
  • module BIOS
  • module image
• vPC
  • not disrupted during upgrade
  • reset after upgrade – different SW version
• disabled guestshell
• enhanced ISSU: supervisor and LC software run in container

; enhanced ISSU
(config)# boot mode lxc

# install all kickstart <PATH> system <PATH> [impact]
# show incompatibility system

Checkpoint

• create
  • manual
  • automatic: prefix system-…
    • on feature disabled
    • on L3 protocol instance disabled and removed
    • on license expiration
  • not created on VDC manipulation
• restore only from own checkpoint: switch, VDC
• unique names, no overwrite
• 10 per VDC
• network-admin only

; create
# checkpoint <NAME>
# show checkpoint
# rollback run checkpoint <NAME>
# clear checkpoint database
# no checkpoint <NAME>

Power-on auto provisioning (POAP)

• uses DHCP to obtain IP and TFTP, downloads config script from TFTP, then gets image + actual config via TFTP or HTTP
  • DHCP client ID = SN
  • path – DHCP option
  • script: Python, TCL
  • DHCP lease ≥ 1h
• invoked if no startup-config is found
• phases
  1. power up
  2. USB discovery: except N9k
  3. DHCP discovery
     • ports are defaulted
     • N9K -EX and newer: L3 ports
  4. script execution
  5. post-install reload

```shell
; N9k, enable POAP after reboot without erasing startup-config, e.g., for upgrade
# boot poap enable
```

Port Profile

• if profile is changed, changes are applied to interfaces
• types:
  • Ethernet
  • SVI
  • MPLS TE
  • MPLS PW
  • port-channel
  • BDI

(config)# port-profile type <TYPE> <PROFILE>
(config-port-prof)# state enable

(config-if)# inherit port-profile <PROFILE>

# show run interface <INTF> expand-port-profile

Graceful insertion and removal

• BGP, EIGRP, OSPF, IS-IS, vPC, FabricPath
• ≈ IS-IS overload bit
• creates snapshot on switchover
• profile names
  • maintenance-mode
  • normal-mode

; use default profile
(config)# system mode maintenance

; use custom profile
(config)# system mode maintenance dont-generate-profile

# show snapshot

; custom profile
# configure maintenance profile <NAME>
(config-mm-profile)# router ospf <OSPF>
(config-mm-profile-router)# max-metric router lsa

Unified ports

• can be either Ethernet or native FC
• per slot (N5k) or per port (N9k)

; counted from the end
(config)# slot <N>
(config-slot)# port <M> type ethernet|fc

N5k

• VoQ: per port per CoS
• ingress queueing
• class-default: always CoS = 0 and qos-group = 0
• CoS-to-queue only
  • CoS setting – via network-qos matching on qos-group
  • no DSCP-to-CoS

N7k

• set dscp also sets CoS (3 MSB): this CoS is used to select egress queue even for access

Virtual device context (VDC)

• each VDC is allocated MAC

; disable prepending admin VDC to name
(config)# no vdc combined-hostname

(config)# vdc <NAME>
(config-vdc)# allocate interface <INTF>

# show feature-set
# show vdc [detail]
# show vdc membership

; system MAC pool
# show sprom backplane

# switchto <NAME>
# switchback

Storage VDC

• single instance
• SAN configuration

(config)# install feature-set fcoe
(config)# vdc fcoe_vdc type storage
(config-vdc)# allocate fcoe-vlan-range <RANGE> from vdc <VDC>

; FC – in storage VDC, Ethernet – usual VDC
(config-vdc)# allocate shared interface <INTF>

IDS

• ASIC verifies that packets do not have incorrect fields
  • src = dst
  • version ≠ 4 or 6
  • size in header ≠ actual size
• needs tuning for BFD (src IP = dst IP)

; has to be disabled for BFD
(config)# hardware ip verify address identical