Design

  1. Goals
  2. Layers
    1. Access layer
    2. Distribution layer
    3. Core/backbone layer
    4. Layer architecture
    5. Service types
  3. Scoping
  4. Resiliency
  5. Microloops
    1. Link-state IGP microloop
    2. BGP microloop
  6. DC tiers
  7. Cabling
  8. Topologies
    1. Cube
    2. Toroid
  9. Management models
    1. FCAPS
    2. ITIL
    3. PPDIOO
    4. TMN

Goals

  • reliability
  • scalability
  • manageability

Layers

Access layer

  • L2 connectivity
  • uplink oversubscription – 20:1
  • low cost per port
  • large number of ports
  • scalable uplinks
  • stub (EIGRP), totally stub (OSPF)
  • high-availability, failover ≈ 200 ms
  • service multiplexing: data, voice, video
  • security enforcement
  • QoS
  • PoE

Distribution layer

  • connects access layer devices with each other
  • inter-distribution link
  • large L3 throughput
  • uplink oversubscription – 4:1
  • security enforcement, filtering
  • aggregation towards core layer: protect from flaps in access
  • PBR
  • QoS
  • scalability, high-availability, failover ≈ 200 ms
  • fast links to access and core
  • L2/L3 border

Core/backbone layer

  • traffic is forwarded at max efficiency
  • very large L3 throughput
  • no CPU-heavy traffic manipulation
  • redundant, high-availability, failover ≈ 200ms
  • advanced QoS

Layer architecture

  • built with switch pairs
  • every switch with 2 uplinks
  • distribution layer pair has inter-switch link
  • VLANs are terminated on distribution, do not span over core
  • switch block ≡ access layer + 2 distribution switches
    • every block is L3 connected to 2 core switches
  • full mesh in core
  • collapsed core: distribution ≡ core

Service types

  • local:
    • via access layer
    • same segment/VLAN
  • remote:
    • via access and distribution layers
    • different segments/VLANs
  • enterprise:
    • via access, distribution and core layers
    • central to all sites

Scoping

  • region
  • PoP, building
  • priority
  • consumption
  • VLAN

Resiliency

  • using SSO (forward through failure) with BFD (forward around failure) makes no sense: BFD would be triggered in any case
  • shallow timers might cause instability because of false positives
  • times have to be balanced on overlay and underlay, otherwise – blackhole

Microloops

  • during reconvergence
  • common for ring/square topology, less common for triangle
  • solution: FRR, TI-LFA (LFA would work only for triangle topologies)
  • results in lower TTL, but does not reach 0
  • negative effect on VoIP

BGP microloop

  • during reconvergence
  • between spine (≡ cluster) and superspine (did not process update + sends default)
  • solution: discard route for cluster prefixes on spine ⇒ no bounce to superspine

DC tiers

  • certified by UptimeInstitute
  • levels:
    • Tier 1: no requirements
    • Tier 2:
      • UPS, several generators, component-level redundancy
      • downtime for maintenance (e.g., power)
    • Tier 3:
      • N+1 power, cooling in active/passive mode
      • on-site generators can supply energy for long time (≈ days)
      • vulnerable to unplanned outage
      • no downtime for maintenance
    • Tier 4:
      • several N+1 systems
      • physical isolation between redundant components
  • HVAC can consume more energy, then IT equipment

Cabling

  • TIA-942
  • cables should have capacity for higher bandwidth in future (OM4)
  • zones:
    • entrance room (ER)
      • demarcation
      • termination point for external channels
      • allows external access
    • main distribution area (MDA)
      • network core
    • horizontal distribution area (HDA)
      • network distribution
      • SAN core
    • equipment distribution area (EDA)
      • network access, servers, array
    • zone distribution area (ZDA)
      • between HDA and EDA
      • provides extra scalability
    • vertical/backbone
      • ER-MDA, MDA, HDA
    • horizontal
      • MDA-EDA
  • computer room
    • MDA + HDA + EDA + ZDA
    • no external access (e.g., for ISP personnel)
  • DAC is more power-efficient than UTP/STP

Topologies

Cube

  • includes stacked cube and hypercube
  • advantage: low number of hops between any 2 nodes
  • suitable for redundant core

Toroid

  • used in fabric, HPC
  • every node has 4 links ⇒ useful for large port count
  • oversubscribed ⇒ blocking fabric ⇒ load has to be localized

Management models

FCAPS

  • ISO
  • components
    • fault management
    • configuration management
    • accounting management
    • performance management
    • security management

ITIL

  • IT infrastructure library

PPDIOO

  • prepare, plan, design, implement, operate, optimize

TMN

  • ITU
  • telecommunications management network