Hierarchical Network Design

• predictable behavior
• low maintenance
• high availability
• recover from failure and topology changes in a pre-determined manner
• scale to support future expansion and upgrades
• design around traffic flows rather than a particular type of traffic
• keep end users at consistent distances from resources
• cisco approach enables designers to organize the network into distinct layers of devices
• access
• distribution
• core
• the resulting network is:
• efficient
• intelligent
• scalable
• easily managed
• access layer switches are aggregated at the distribution switch layer
• distribution layer switches are aggregated as the core switch layer

Access Layer:

• end users connected
• low cost per switch port
• high port density
• scalable uplinks to higher layers
• user access functions:
• VLAN membership
• traffic and protocol filtering
• QoS
• resiliency through multiple uplinks

Distribution Layer:

• interconnection between the campus network’s access and core layers
• aggregation of multiple access-layer devices
• high layer 3 throughput for packet handling
• security and policy-based connectivity functions through access lists or packet filters
• QoS
• scalable and resilient high-speed links to the core and access layer
• switches capable of handling the total volume of throughput from all connected devices
• high port density of high speed links to to support the collection of access layer switches
• VLAN’s and broadcast domains converge at this layer and require routing, filtering and security
• usually a layer 3 boundary where routing meets the VLAN’s of the access layer

Core Layer:

• connectivity of all distribution layer devices
• AKA backbone
• needs to switch traffic as efficiently as possible
• very high throughput at layer 3
• no costly or unnecessary packet manipulations (ACL’s, filtering)
• Redundancy and resilience
• advanced QoS
• designed with simplicity and efficiency in mind

Switch Block:

• a group of access layer switches together with their distribution layer switches
• all switch blocks connect into the core block
• balanced mix of layer 2 and layer 3
• distribution layer shields the switch block from certain failures or conditions in other parts of the network.  eg.  broadcasts are not propagated from the switch block into the core and other switch blocks
• STP is confined to each switch block where a VLAN is bounded
• VLAN’s should not be extended beyond distribution switches
• the distribution layer should always be the boundary of VLAN’s, subnets and broadcasts
• VLAN traffic should not traverse the network core
• when sizing a switch block, consider:
• port density for access layer switches
• traffic types and patterns
• amount of layer 3 switching capacity at the distribution layer
• number of users connected to access layer switches
• geographic boundaries of of subnets or VLAN’s
• size of spanning tree domains
• usually no more than 2000 users should be placed within a single switch block, although sizing should be based primarily on:
• traffic types and behaviour
• size and number of common workgroups
• a switch block is too large if:
• the routers (MLS’s) at the distribution layer become traffic bottlenecks, possibly due to high volumes of inter VLAN traffic, intensive CPU processing or switching times required by policy or security functions
• broadcast or multicast traffic slows the switches in the block
• best practice is for all layer 2 connectivity to be contained within the access layer

Core Block:

• the campus network’s backbone
• required to connect 2 or more switch blocks in a campus network
• must be as efficient and resilient as possible as all traffic passing to and from all switch blocks must cross it
• carries more traffic than any other block
• links to/from the distribution layer can be L3 or L2 (using a small vlan bounded by the switches, and an SVI to provide routing)
• for sizing core switches, each one must be able to handle each of it incoming distribution links at 100% capacity

Collapsed Core:

• core is collapsed into distribution layer
• dist and core functions provide by the same switch devices
• used in smaller campus networks where a separate core is not warranted
• not an independent building block but integrated into the distribution layer of individual switch blocks
• each access layer switch has a redundant link to each dist/core layer switch
• all L3 subnets in the access layer terminate at the dist switches L3 ports
• dist/core switches connect to each other by one or more L3 links for redundancy/failover

Dual Core:

• connects 2 or more switch blocks in a redundant fashion
• independent from any other switch block
• 2 identical, redundant switches
• redundant links connect the distribution layer of a switch block to each of the dual core switches
• routing protocols plus provide equal cost load balancing between dist and core switches

InterVLAN Routing

• requires that routing be enabled for the layer 3 protocol – eg. IP Routing
• requires static routes or a dynamic routing protocol
• by default every port on most catalyst switch platforms is layer 2, and on 6500’s every port is layer 3 by default
• to determine the mode a switch port is in: Switch# show interface type mod/num switchport
• if the status is Switchport: Disabled then it’s layer 3
• to swap ports between layer 2 and 3: Switch# (config-if) [no] switchport
• an etherchannel can also become a layer 3 port
• to configure a Switched Virtual Interface (SVI):
• Switch(config)# vlan 100
• Switch(config-if)# ip address ip-address mask [secondary]
• Switch(config-if)# no shutdown

Multilayer Switching with CEF

• route once and switch many.  The Route Processor (RP) receives the first packet and a routing decision is made.  The Switching Engine (SE) then listens to see if it can switch the packet in both directions, and if so then subsequent packets can be switched directly to the destination port, bypassing the RP.
• AKA Netflow Switching or Route Cache Switching
• CEF has taken over from Netflow switching as it is more efficient, and runs by default in hardware
• The layer 3 engine maintains routing information, and the routing table is reformatted into a list with the most specific route first for each destination subnet – this is called the Forwarding Information Base
• the switch examines the FIB for the longest match (ie. most specific) destination route for a packet
• the FIB also contains the next hop address for each entry
• the FIB is dynamically updated upon changes to the routing table, or next hop addresses change or age-out of the ARP table

• display the FIB table: Switch# show ip cef [type mod/num | vlan vlan-id] [detail]
• packets that cannot be switched in hardware are marked as “CEF punt” and sent to the layer 3 engine.  some conditions can lead to this:
• no entry in FIB
• FIB table is full
• TTL expired
• MTU exceeded
• ICMP redirect is involved
• encapsulation type is not supported
• packets are tunneled and require encryption or compression
• an ACL with a log option is triggered
• NAT’ing is taking place
• aCEF (accelerated) is where CEF is multiple layer 3 forwarding engines, but only subsection of the FIB table is known each engine
• dCEF (disrtibuted) is as aCEF but with the whole FIB table on each engine
• the Adjacency Table is the list of IP to MAC mappings for every next hop entry in the FIB table.  view details with:
• Switch# show adjacency [type mod/num | vlan vlan-id ] [summary | detail]
• Packet Re-write – After finding valid FIB and adjacency table entries the switch must re-write the packet header before it can be forward the packet.   This is done in real time by the re-write packet engine.  The following changes are made:
• L2 dest address – changed to next-hop MAC
• L2 src address – changed to out bound L3 switch interface’s MAC
• L3 IP TTL – decremented by 1
• L3 IP checksum – recalculated due to changes to the IP header
• L2 frame checksum – recalculated due to changes to the L2 and L3 headers
• CEF is enabled by default on all CEF capable switches
• disable CEF on 3750 series interface: no ip route-cache cef
• disable CEF on 4500 series interface: no ip cef
• verify CEF: show ip cef

Using DHCP with a multilayer switch

• configuring an IOS DHCP server
• switch(config)# ip dhcp excluded- address start-ip and ip
• switch(config)# ip dhcp pool pool-name
• switch(config-dhcp)# network ip-address subnet-mask
• switch(config-dhcp)#  default-router ip address [ip-address2] [ip-address3]
• switch(config-dhcp)# lease {infinite | {days [hours [minutes]]}}
• switch(config-dhcp)# exit
• check leases with: show ip dhcp binding
• configuring a DHCP relay
• switch(config)# interface vlan 5
• switch(config-if)# ip address 192.168.1.1 255.255.255.0
• switch(config-if)# ip helper-address 192.168.199.4
• switch(config-if)# exit

Protecting The Spanning Tree Protocol Topology

There are 2 conditions that can occur to disrupt the loop free topology even when STP is running:

1. BPDU’s suddenly being received on a port that shouldn’t be receiving them – can result in unexpected re-convergence results
2. BPDU’s suddenly stop being received on a port that should be receiving them – a switch can then make incorrect assumptions about the topology and unintentionally create loops

Protecting against unexpected BPDU’s

Root Guard

• protects against a rogue switch connecting to the network and becoming the root bridge
• controls where candidate root bridges can be connected and found on a netwok
• if superior BPDU’s are received on a switch port with Root Guard enabled, the local switch will not allow the new switch to become the root
• puts the port in the root-inconsistent STP state while superior BPDU’s are being received, meaning no data can be sent or received on that port, but the port can listen to BPDU’s received to detect a new root advertising itself
• a root guard protected port can only forward or relay BPDU’s, it cannot be used to receive them
• prevents a port from ever becoming a root port
• is disabled by default, and is enabled only on a per port basis with the command: Switch(config-if)# spanning-tree guard root
• port goes back to it’s normal state after it stops receiving superior BPDU’s
• show ports in the state of root-inconsistent  by issuing: Switch# show spanning-tree inconsistentports

BPDU Guard

• protects ports that PortFast enabled, to prevent loops forming if a switch was accidentally connected to a port that should only be used by a workstation
• should be enabled on all port that have PortFast enabled
• if any BPDU (superior to the current root or not) is received on a port running BPDU guard, the port is put into errdisable state
• disabled on all ports by default
• can be configured as a global default, meaning that all ports that have PortFast enabled will also have BPDU Guard automatically enabled.  Command: Switch(config)# spanning-tree portfast bpdugaurd default
• can be enabled or disabled on a per port basis: Switch(config-if)# [no] spanning-tree bpduguard enable
• port remains in the errdisble state even after BPDU’s have stopped being received
• never enable on any switch uplink where the root bridge is located, as a switch with multiple uplinks may receive BPDU’s on any of those ports

Protecting against sudden loss of BPDU’s

Loop Guard

• blocking port + BPDU’s stop being received + last received BPDU ages out = port starts forward + bridging loop occurs
• loop guard keeps track of BPDU activity on non-designated ports
• moves port to loop-inconsistent state when BDPU’d stop being received, which effectively puts the port in the blocking state
• port is allowed to move through the normal STP states when BPDU’s start being received again (automatically)
• disabled by default on all switch ports
• enable as a global default: Switch(config)# spanning-tree loopguard default
• enable or disable per port:  Switch(config-if)# [no] spanning-tree guard loop
• although configured on a port the loop guard corrective action is limited only to the VLAN affected, it doesn’t block the entire port
• you can enable on all switch ports and the switch will figure out which ports are non-designated (usually alternative root ports and blocking ports)

UDLD (Unidirectional Link Detection)

• if a switch link becomes uni-directional perhaps due to faulty hardware in a GBIC of SFP, then this can cause a bridging loop as BPSU’d may not be received at on end of the link 
• Cisco only
• monitors ports to see if a link is truly bidirectional
• switch sends layer 2 UDLD frames identifying its switch port at regular intervals, and UDLD expects the frame to be echoed back with the far end switch port’s identification addded.  if received, all is OK, but if the echoed frames are not seen then the link must be unidirectional
• message interval is configurable, and the default is 15 seconds
• aim is to detect before STP has time to move a blocked port into a forwarding state
• 2 modes of operation:
• normal – action is to mark the port as having an undetermined state and generate a syslog message
• aggressive – switch takes action to re-establish the link by sending out UDLD frames once a second for 8 seconds, if no echo fame received back then the port is moved to errdisable state
• configured on a per port basis, although can be enabled globally for all fibre optic switch ports
• disabled by default
• to enable globally: Switch(config)# udld {enable | aggressive | message time seconds}
• enable or disable per port:  Switch(config-if)# udld {enable | aggressive | disable}
• can be safely enabled on all switch ports as UDLD is only enabled globally on fibre optic ports because copper/twisted pair connections do not have the physical layer conditions that allow unidirectional links to occur
• UDLD operates on each link of an etherchannel indpendently

Using BPDU Filtering to diable STP on a port

• used in special cases where you want to prevent BPDU’s from being sent or processed on one or more ports
• disabled on all switch ports by default
• configure as a global default for all ports: Switch(config)# spanning-tree portfast bpdufilter default
• enable or disable per port:  Switch(config-if)# spanning-tree bpdufilter {enable | disable}
• only enable on ports where a single host is connected making loops impossible

STP Root Bridge

Root bridge location should be determined as part of the design process.  Adjustments to the STP configuration can be made for things like using redundant links to load balance in parrallel, or configuring STP to converge quickly and predictably if a major topology change occurs.

note – if STP has been disabled for any reason, enable it by issuing the following commands:

Switch(config)# spanning-tree vlan vlan-id

Switch(config-if)# spanning-tree vlan vlan-id  (use if disabled for a specific VLAN on a specific port)

Root Bridge Placement considerations:

• switch speed – default settings may result in the slowest switch becoming root and having to handle a lot of traffic
• redundancy – default settings may result in a switch that isn’t ideal for the job becoming the root
• location – default settings may result in a switch in a distant location from a large section of the network becoming root, meaning a lot of traffic having too go the long  way round to pass through the root

Root Bridge Configuration – to prevent surpirises

• always configure one switch as the root bridge in a determined fashion
• always configure one switch as secondary root bridge in case the primary fails

Generally the root bridge should always be placed near the centre of a layer 2 network.  eg.  a distribution layer switch is a better choice than an access layer switch, and a switch near to a server farm is a good choice as in both cases they would be expected to handle most traffic.

There are two ways to manually configure a switch as a root bridge:

• manually set the bridge priority to a low value, so it wins the election.  Make sure that the bridge priority value of the other is higher.
• Switch(config)# spanning-tree vlan vlan-list prority bridge piority
• causing the would be root bridge to choose it’s own priority based on assumptions about other switched in the network
• Switch(config)# spanning-tree vlan vlan-id root {primary | secondary | diameter diameter}

Spanning Tree Customization

Tuning the root path cost – this can be modified from the default value, using the following command:

Switch(config-if)# spanning-tree [vlan vlan-id] cost cost   (cost can be 1-65535)

To view the port cost of an interface:

Switch# show spanning-tree interface type mod/num  [cost]

Tuning the port ID – a switch port number is fixed, but it port ID can be changed using the port priority:

Switch(config-if)# spanning-tree [vlan vlan-list] port-priority port-prority

Tuning Spanning Tree Convergence

Manually Modifying STP Timers

Switch(config)# spanning-tree [vlan vlan-id ] hello-time seconds

Switch(config)# spanning-tree [vlan vlan-id ] forward-time seconds

Switch(config)# spanning-tree [vlan vlan-id ] max-age seconds

Automatically Configuring STP timers

Switch(config)# spanning-tree vlan vlan-list root {primary | secondary} [diameter diameter [hello-time hello-time]]

The above command adjusts STP timers according to the forumlas specified in 802.1D standard, by only giving the network’s diameter (max number of switches that layer 2 traffic will traverse)., an dan optional hello time.

Redundant Link Convergence

PortFast

• enables fast connectivity to be established on access-layer switch ports to workstations that are booting
• if not configured then delay will be 30 seconds from a port coming up before it is in forwarding mode (15 secs listening to learning + 15 secs learning to forwarding)
• on ports that only connect to workstations or single devices, bridging loops should never be possible
• portfast shortens the listening and learning states to a negligible amount of time, meaning the port immediately moves to the forwarding state
• STP loop detection is still in place though, meaning the port will be put in the blocking state if a loop is detected
• disabled by default
• can be configured as a global default, meaning that all ports that are configured for access mode (non trunking) will have portfast automatically enabled:
• Switch(config)# spanning-tree portfast default
• can be enabled or disabled on specific switch ports:
• Switch(config-if)# [no] spanning-tree portfast
• portfast = TCN BPDU’s are not sent when a switch port in portfast mode goes up or down
• to display the current portfast status:
• Switch# show spanning-tree interface type mod/num portfast

UplinkFast

• enables fast uplink failover on an access layer switch when dual uplinks are connected into the distribution layer
• gets rid of the up to 50 second delay that would occur if an access layer switch has redundant uplink connections to 2 distribution layers switches and one of the links fails
• enables switches at the ends of spanning-tree branches to have a functioning root port while keeping one or more redundant or potential root ports in blocking mode.  when the primary root port uplink fails, another port immediately is brought up for use
• Swtich(config)# spanning-tree uplinkfast [max-update-rate pkts-per-second]
• when enabled, it is for the whole switch and all VLAN’s
• keeps track of possible paths to the root bridge, so the command cannot be issued on the root bridge
• modifications are made to the local switch to ensure it doesn’t become the root bridge or a transit switch to the root bridge:
• bridge priority is raised to 49152
• port costs of all local switch ports raised by 3000
• the max-update-rate parameter allows the local switch to notify other upstream switches that stations downstream can be reached over the newly activated link.  this is done by sending the local switch sending dummy multicast frames to destination 0100.0ccd.cdcd on behalf of the stations contained in it’s MAC address table.  These frames are sent out a rate specified by this parameter, to control the amount of bandwidth used.
• display the current status of STP uplink fast:
• Switch# show spanning-tree uplinkfast

BackboneFast

• enables fast convergence in the network backbone or core layer switches after a spanning-tree topology change occurs
• a switch actively determines whether alternative paths exist to the root bridge, in case the switch detects an indirect link failure (a when a link that is not directly connected to a switch fails
• indirect link failures are detected when a switch receives inferior BPDU’s from it’s designated bridge on either it’s root port or a blocked port
• normally a switch must wait for the max-age timer to expire before responding to the inferior BPDU’s
• simple to configure
• short-circuits the max-age timer when needed
• can reduce the maximum convergence delay only from 50 to 30 seconds
• configure with the following command: Switch(config)# spanning-tree backbonefast
• should be enabled on all switches in the network because it requires the use of the RLQ request and reply mechanism
• disabled by default
• verify with the following command: Switch# show spanning-tree backbone fast