NX-OS Overview
Source:
Storage Area Network (SAN) has been around for ages. During the early days, physical SANs are used to separate SAN traffic of different requirements, and from isolating storage traffic from users traffic. Moreover, the idea is to segregate access to LUN targets of different groups such as separating Accounting data from Marketing or marking bandwidths between disparate VMWare ESXi host bus adapters (HBAs).
Recently, Virtual SANs (VSANs) have been a standard of practice, whereby multiple SANs are configured within a set of VSAN capable switches. Another way to understand this concept is use switches that can assign VSAN traffic per port instead of per switch.
Within a VSAN, Zones can be configured. Zones are always contained within a single VSAN and will not span between VSANs. These are useful as labeling targets by the node name for administering and troubleshooting purposes. Furthermore, multiple zones can be grouped together as a zone set.
For the purpose of this article, VMWare VSAN is out of scope of the document. There are design considerations of that technology that is very different from a Cisco implementation. Hence that topic could be covered in another essay.
By design:
– A SAN does not provide file abstraction, only block-level operations
– SAN does not operate on TCP/IP; thus, its blocks contain less overhead compared to TCP/IP packets.
– VSAN 1 should not be used for production
– VSAN 4094 is called isolated VSAN. Its purpose is to host interface members of deleted VSANs and to act as a default holder for non-trunking ports
– User VSANs range from 2 to 4093
This statement displays interfaces that are not configured with a user-defined VSAN
IRV-SAN-SW01# show vsan 4094 membership
vsan 4094(isolated_vsan) interfaces:
[empty]
Creating a VSAN
config t #enter config mode
vsan database #enter vsan database edit mode
vsan 10 #use vsan 10
vsan 10 name VSAN10 #name it
end #exit config mode
Assigning Ports to VSAN
config t
vsan database
vsan 10
vsan 10 interface fc1/1 #set fiber channel module 1 slot 1 as member of vsan 10
end
Load Balancing
config t
vsan database
vsan 10
vsan 10 loadbalancing src-dst-ox-id #using sourceID, destID, and OX_ID(default) in selection process
# vsan 10 loadbalancing src-dst-id #only using source and destination ID in selection process
end
Check Load Balancing Config
IRV-SAN-SW01# show vsan 10
vsan 10 information
name:VSAN0010 state:active
interoperability mode:default
loadbalancing:src-id/dst-id/oxid
operational state:up
Dynamic Port VSAN Membership (DPVM)
Enabling DPVM
config t
feature dpvm
DPVM has “autolearn” features that will sense device pWWNs being connected F ports (FL ports are unsupported) and populate its database
Enabling Auto-Learn
config t
dpvm auto-learn
Display association between interfaces, VSAN numbers, and Port Names:
IRV-SAN-SW01# show flogi database
--------------------------------------------------------------------------------
INTERFACE VSAN FCID PORT NAME NODE NAME
--------------------------------------------------------------------------------
fc1/2 10 dddd dd:dd:dd:dd:dd:dd:dd:dd dd:dd:dd:dd:dd:dd:dd:dd
fc1/2 10 ...Information ommitted...
fc2/7 1
Enable Name Server Proxies:
config t
fcns proxy-port 00:00:00:00:00:00:00:xx vsan 10 #Configures a proxy port for the specified VSAN.
Rejecting Duplication pWWNs to ensure integrity of existing pWWN in the database. This overrides the default behavior.
config t
fcns reject-duplicate-pwwn vsan 10
Display Name Server Database Statistical Information
IRV-SAN-SW01# show fcns database
VSAN 1:
--------------------------------------------------------------------------
FCID TYPE PWWN (VENDOR) FC4-TYPE:FEATURE
--------------------------------------------------------------------------
0x8301a0 N ..Information Omitted.. (HP) scsi-fcp:target
Total number of entries = 1
VSAN 10:
--------------------------------------------------------------------------
FCID TYPE PWWN (VENDOR) FC4-TYPE:FEATURE
--------------------------------------------------------------------------
0xxxxxxx N ..Information Omitted.. scsi-fcp:target
0xxxxxxx N ..Information Omitted.. scsi-fcp:init
0xxxxxxx N ..Information Omitted.. (Emulex) scsi-fcp:init
0xxxxxxx N ..Information Omitted.. scsi-fcp:init
0xxxxxxx N ..Information Omitted.. scsi-fcp:init
0xxxxxxx N ..Information Omitted.. scsi-fcp:init
0xxxxxxx N ..Information Omitted.. scsi-fcp:init
0xxxxxxx N ..Information Omitted.. scsi-fcp:init
0xxxxxxx N ..Information Omitted.. scsi-fcp:init 248
0xxxxxxx N ..Information Omitted.. scsi-fcp:init
0xxxxxxx N ..Information Omitted.. scsi-fcp:init
0xxxxxxx N ..Information Omitted.. scsi-fcp:init
0xxxxxxx N ..Information Omitted.. scsi-fcp:init
0xxxxxxx N ..Information Omitted.. scsi-fcp:init 248
0xxxxxxx N ..Information Omitted.. scsi-fcp:target
0xxxxxxx N ..Information Omitted.. scsi-fcp:target
0xxxxxxx N ..Information Omitted.. scsi-fcp:target
0xxxxxxx N ..Information Omitted.. scsi-fcp:target
0xxxxxxx N ..Information Omitted.. (NetApp) scsi-fcp:init 248
0xxxxxxx N ..Information Omitted.. (NetApp) scsi-fcp:init 248
0xxxxxxx N ..Information Omitted.. (Emulex) scsi-fcp:init
0xxxxxxx N ..Information Omitted.. (Emulex) scsi-fcp:init
0xxxxxxx N ..Information Omitted.. (Emulex) scsi-fcp:init
0xxxxxxx N ..Information Omitted.. (Emulex) scsi-fcp:init
0xxxxxxx N ..Information Omitted.. (Emulex) ipfc scsi-fcp:init
0xxxxxxx N ..Information Omitted.. (HP) scsi-fcp:target
0xxxxxxx N ..Information Omitted.. (HP) scsi-fcp:target
0xxxxxxx N ..Information Omitted.. (Emulex) scsi-fcp:init
0xxxxxxx N ..Information Omitted.. (Cisco) npv
0xxxxxxx N ..Information Omitted.. (Emulex) scsi-fcp:init
0xxxxxxx N ..Information Omitted.. (Cisco) npv
0xxxxxxx N ..Information Omitted.. (Cisco) npv
0xxxxxxx N ..Information Omitted.. (Cisco) npv
0xxxxxxx N ..Information Omitted.. (Cisco) npv
0xxxxxxx N ..Information Omitted.. (Emulex) scsi-fcp:init
0xxxxxxx N ..Information Omitted.. (Cisco) npv
0xxxxxxx N ..Information Omitted.. (Cisco) npv
0xxxxxxx N ..Information Omitted.. (HP) scsi-fcp:target
0xxxxxxx N ..Information Omitted.. (HP) scsi-fcp:target
0xxxxxxx N ..Information Omitted.. (HP) scsi-fcp:target
0xxxxxxx N ..Information Omitted.. (HP) scsi-fcp:target
0xxxxxxx N ..Information Omitted.. (HP) scsi-fcp:target
0xxxxxxx N ..Information Omitted.. (HP) scsi-fcp:target
0xxxxxxx N ..Information Omitted.. (HP) scsi-fcp:target
0xxxxxxx N ..Information Omitted.. (HP) scsi-fcp:target
0xxxxxxx N ..Information Omitted.. (HP) scsi-fcp:target
0xxxxxxx N ..Information Omitted.. (HP) scsi-fcp:target
Total number of entries = 47
Fabric-Device Management Interface (FDMI) can display this information about attached HBAs:
- Manufacturer, model, and serial number
- Node name and node symbolic name
- Hardware, driver, and firmware versions
- Host operating system (OS) name and version number
IRV-SAN-SW01# show fdmi database
Registered HBA List for VSAN 10
xx:xx:xx:xx:xx:xx:xx:xx
xx:xx:xx:xx:xx:xx:xx:xx
IRV-SAN-SW01# show fdmi database detail
Registered HBA List for VSAN 10
-------------------------------
HBA-ID: ..Information Omitted..
-------------------------------
Node Name :..Information Omitted..
Manufacturer :QLogic Corporation
Serial Num :U15309
Model :QLE2562
Model Description:PCI-Express Dual Channel 8Gb Fibre Channel HBA
Hardware Ver :
Driver Ver :8.05.00.03.06.0-k
ROM Ver :0.00
Firmware Ver :8.04.00 (9095)
Port-id: xx:xx:xx:xx:xx:xx:xx:xx
Supported FC4 types:scsi-fcp
Supported Speed :1G 2G 4G
Current Speed :Unknown
Maximum Frame Size :2048
OS Device Name :qla2xxx
-------------------------------
HBA-ID: ..Information Omitted..
-------------------------------
Node Name :..Information Omitted..
Manufacturer :QLogic Corporation
Serial Num :P11473
Model :QLE2662
Model Description:QLogic QLE2662 Dual Port FC16 HBA
Hardware Ver :
Driver Ver :8.05.00.03.06.0-k
ROM Ver :0.00
Firmware Ver :8.04.00 (d0d5)
Port-id: xx:xx:xx:xx:xx:xx:xx:xx
Supported FC4 types:scsi-fcp
Supported Speed :1G
Current Speed :Unknown
Maximum Frame Size :2048
OS Device Name :qla2xxx
Display registering devices
VSAN01# show rscn scr-table vsan 10
SCR table for VSAN: 10
---------------------------------------------
FC-ID REGISTERED FOR
---------------------------------------------
0xxxxxxx fabric and nport detected rscns
0xxxxxxx fabric and nport detected rscns
0xxxxxxx fabric and nport detected rscns
0xxxxxxx fabric and nport detected rscns
0xxxxxxx fabric and nport detected rscns
0xxxxxxx fabric and nport detected rscns
0xxxxxxx fabric and nport detected rscns
0xxxxxxx fabric and nport detected rscns
Total number of entries = 8
IRV-SAN-SW01# show rscn statistics vsan 10
Statistics for VSAN: 10
-------------------------
Number of SCR received = 717
Number of SCR ACC sent = 717
Number of SCR RJT sent = 0
Number of RSCN received = 0
Number of RSCN sent = 6771
Number of RSCN ACC received = 6766
Number of RSCN ACC sent = 0
Number of RSCN RJT received = 0
Number of RSCN RJT sent = 0
Number of SW-RSCN received = 1057
Number of SW-RSCN sent = 100
Number of SW-RSCN ACC received = 100
Number of SW-RSCN ACC sent = 1057
Number of SW-RSCN RJT received = 0
Number of SW-RSCN RJT sent = 0
Show RSCN Timer
VSAN01# show rscn event-tov vsan 10
Event TOV : 2000 ms
Show RSCN Time Configuration Distribution
VSAN01# show cfs application name rscn
Enabled : No
Timeout : 20s
Merge Capable : Yes
Scope : Logical
Use Case
Enough academic background information. Let’s dive into a real-world Use Case.
Scenario:
– 2 Data Center Locations that are geographically divided. Let’s call them California and Florida.
– At the Florida location, there are 2 DS-C9513 VSAN enclosures
– Each enclosure connects to one of the four 3Par 7400C Controllers
– Each controller shall have 1 port dedicated to VSAN 10 and 1 port to VSAN 20
– Each ESXi host shall have 2 Fiber Optic connections going to FABRIC01 and FABRIC02
– Each ESXi host shall have 2 Ethernet links to the core Cisco Nexus 5548UP switches
– The Core switches shall have fiber connectivity to the SAN Fabric switches
Physical SAN switches in Florida:
Credits: I’ve pulled the Cisco object drawings from here http://www.visiocafe.com/downloads/vsdfx/VSDfx-Cisco.zip
Note: the drawing has omitted a link between this Florida site with the California site. If you will, imagine a High Bandwidth MPLS line between these sites to connect from FABRIC01 & FABRIC02 (Florida Data Center) with FABRIC03 & FABRIC04 (California Data Center) in a mesh configuration. VSAN10 / VSAN20 will be trunked into this MPLS link. On the WAN aspects, ISR routers need to be configured to use vlan based L2TPv3 for the L2 point to point transport service. IPSec is optional as this is MPLS rather than VPN. Drawing this would exceed the succinctness of this article.
Sample Configuration of Adding ESXi Host pWWN into the Fabric
Checking configuration of FABRIC01
FLO-SAN-SW01# show ver
Cisco Nexus Operating System (NX-OS) Software
TAC support: https://www.cisco.com/tac/
Documents: https://www.cisco.com/c/en/us/products/index.html
s_home.html
Copyright (c) 2002-2014, Cisco Systems, Inc. All rights reserved.
The copyrights to certain works contained herein are owned by
other third parties and are used and distributed under license.
Some parts of this software are covered under the GNU Public
License. A copy of the license is available at
http://www.gnu.org/licenses/gpl-3.0.html.
Software
BIOS: version 1.0.19
loader: version N/A
kickstart: version 5.2(8d)
system: version 5.2(8d)
BIOS compile time: 02/01/10
kickstart image file is: bootflash:///m9200-s2ek9-kickstart-mz.5.2.8d.bin
kickstart compile time: 12/25/2020 12:00:00 [04/09/2014 06:42:37]
system image file is: bootflash:///m9200-s2ek9-mz.5.2.8d.bin
system compile time: 2/19/2014 14:00:00 [04/09/2014 08:15:36]
Hardware
cisco MDS 9222i ("4x1GE IPS, 18x1/2/4Gbps FC/Sup2")
Motorola, e500v2 with 1036300 kB of memory.
Processor Board ID XXXXXXXXXX
Device name: FLO-SAN-SW01
bootflash: 1000440 kB
Kernel uptime is 1494 day(s), 4 hour(s), 9 minute(s), 13 second(s)
Last reset at 345795 usecs after Fri Mar 27 18:42:26 2015
Reason: Reset due to upgrade
System version: 5.0(4b)
Service:
Check Existing VSAN Config
FLO-SAN-SW01# show vsan 20
vsan 20 not configured
FLO-SAN-SW01# show vsan 10
vsan 10 information
name:VSAN0010 state:active
interoperability mode:default
loadbalancing:src-id/dst-id/oxid
operational state:up
Configuring a new ESXi host into VSAN
## FLO-SAN-SW01 VSAN10 WPPN1 ##
config t #Enter config mode
fcalias name ESX007-PWWN1 VSAN 10 #Create fiber channel alias with a meaningful name and associate it with VLAN 10
member pwwn xx:xx:xx:xx:xx:xx:xx:xx #Include the pwwn of ESX007-PWWN1 as a member
exit
zone name ESX007-PWWN1-FABRIC01_011 VSAN 10 #Create a dedicated zone in VSAN10 for ESX007-PWWN1 for
member fcalias ESX007-PWWN1 #Include ESX007-PWWN1 into this zone
member fcalias FABRIC01_011 #Include Fabric01_controller:index0-bus:1-port:1 into this zone
exit
zone name ESX007-PWWN1-FABRIC01_111 VSAN 10
member fcalias ESX007-PWWN1
member fcalias FABRIC01_111 #Controller2
exit
zone name ESX007-PWWN1-FABRIC01_211 VSAN 10
member fcalias ESX007-PWWN1
member fcalias FABRIC01_211 #Controller3
exit
zone name ESX007-PWWN1-FABRIC01_311 VSAN 10
member fcalias ESX007-PWWN1
member fcalias FABRIC01_311 #Controller4
exit
zoneset name ZoneSet01 VSAN 10 #Create a zoneset named ZoneSet01 in VSAN 10
member ESX007-PWWN1-FABRIC01_011 #Add zone names as members of this ZoneSet01
member ESX007-PWWN1-FABRIC01_111
member ESX007-PWWN1-FABRIC01_211
member ESX007-PWWN1-FABRIC01_311
exit
zoneset activate name ZoneSet01 VSAN 10 #Set ZoneSet01 as active
zone commit VSAN 10 #Commit configuration changes
copy running-config startup-config fabric #Write to startup config
## FABRIC02 VSAN20 WPPN2 ##
config t
fcalias name ESX007-PWWN2 VSAN 20
member pwwn xx:xx:xx:xx:xx:xx:xx:xx
exit
zone name ESX007-PWWN2-FABRIC02_011 VSAN 20
member fcalias ESX007-PWWN2
member fcalias FABRIC02_011
exit
zone name ESX007-PWWN2-FABRIC02_111 VSAN 20
member fcalias ESX007-PWWN2
member fcalias FABRIC02_111
exit
zone name ESX007-PWWN2-FABRIC02_211 VSAN 20
member fcalias ESX007-PWWN2
member fcalias FABRIC02_211
exit
zone name ESX007-PWWN2-FABRIC02_311 VSAN 20
member fcalias ESX007-PWWN2
member fcalias FABRIC02_311
exit
zoneset name ZoneSet01 VSAN 10
member ESX007-PWWN2-FABRIC02_011
member ESX007-PWWN2-FABRIC02_111
member ESX007-PWWN2-FABRIC02_211
member ESX007-PWWN2-FABRIC02_311
exit
zoneset activate name ZoneSet01 VSAN 20
zone commit VSAN 20
copy running-config startup-config fabric
Explanations:
The term fabric, in this context, means back-plane connectivity. Allowable bandwidth of these MDS 9222i back-planes are extremely high. Any module installed on the same switch chassis will have access to this fabric. Nowadays, this also includes fiber optic channels or specialized 8-channel data direct connections to adjacent network devices. According to this Use Case, FLO-SAN-SW01 & FLO-SAN-SW02 are two separate fabrics. Although they are connected with two High Availability links, those are not configured as such mesh has been unnecessary. Each new ESXi host shall be confined within its zone set so that its HBA will not see LUN intended for other hosts. Each zone set has 4 paths being made accessible for each of the 2 pWWNs of an ESXi host, so there are a total of eight paths to the serial attached storage array.