Configuration of an SDN Lab with HP Switches (5700, 5130, 3500, 2920): Part 1

I build a small data center based on mini fat-tree topology to investigate performance of the SDN-based networks. The experiment plan is very rich, it contains scenarios including: SDN, ECMP, heterogeneous switches, 1/10/40Gbps. At my university, we have a handful of fresh HP switches, so I want to play with them a bit and measure their performance. But first things first: configuration.

Goals for Part 1

As the Rome wasn't built in a day so I also start with basics.

  • First, I need a VLAN that is always there whatever happens = the blue one, VLAN 1.
  • Second, I need the experiment network that we will play with = the cisco-red one, VLAN 100.
  • Third, I don't want to touch too much the servers (C##), so the IP configuration runs over DHCP.
  • Finally, I want to play with the red network without the need to run to the data center (it may rain or something).

Step 1: Topology Diagram

Briefly speaking, the blue, red, and green are the vlan numbers. The number in the color rounded box attached to a cable is the physical port number. So, switch SW00 is connected to the SW42 with a cable that connects ports (SW10)-35 <--> (SW42)-28.

Switches SW00 and SW10 (HPE 5700) are connected to the SW4x's using a 10Gbit SPF+ DAC cable. VLAN 4 uses 40Gbit QSFP+ DAC cable. The rest is just a plain 1Gbit Cat6 Ethernet.

I use are the following models of Switches:

  • HP 2920-24G (J9726A), firmware WB.15.12.0015,
  • HP 3500yl-24G (J8692A), firmware K.15.17.0007,
  • HPE FF 5700-32XGT-8XG-2QSFP+ (JG898A), firmware 2422P01,
  • HPE 5130-24G-4SFP+ EI (JG932A), firmware 3111P03.

Testbed topology Click for larger version

Step 2: DHCP servers

This needs to be done in a time-effective manner. So I install webmin, the dhcp server, and apply a basic configuration. My servers run Debian or Ubuntu and here is the configuration outline.

Install webmin as described on the webmin page.

wget http://prdownloads.sourceforge.net/webadmin/webmin_1.801_all.deb  
sudo dpkg --install webmin_1.801_all.deb  
sudo apt-get -f install  

Install the ISC DHCP Server and point the interface where it listens to the requests.

sudo apt-get install isc-dhcp-server  
sudo dpkg-reconfigure isc-dhcp-server # I pick eth1 for C16  

Let me apply a staring config for the server C16

sudo vim /etc/dhcp/dhcpd.conf  

and the following minimal configuration. It should be enough just to start.

default-lease-time 600;  
max-lease-time 7200;  
log-facility local7;  
subnet 10.100.1.0 netmask 255.255.255.0 {  
range 10.100.1.200 10.100.1.254;  
    option subnet-mask 255.255.255.0;
    option broadcast-address 10.0.0.254;
    host SW00 {
        option host-name "SW00";
       hardware ethernet 5c:8a:38:21:83:1b;
       fixed-address 10.100.1.100;
    }
}

Now start the dhcp server and verify that is is running.

sudo service isc-dhcp-server restart  

This should be enough for the Switch SW00 to get the fixed IP address.

Step 3: Switch Sw00 (HP 5700)

Now, I configure the switch. The switch SW00 runs Comware operating system from HP.

sys  
vlan 1  
name production  
# no ports as this is default vlan
quit  
#
interface Vlan-interface 1  
dhcp client identifier ascii SW00  
ip address dhcp-alloc  
quit  
#
vlan 100  
name experiment  
port Ten-GigabitEthernet1/0/16  # C16 SDN controller  
port Ten-GigabitEthernet 1/0/33  
port Ten-GigabitEthernet 1/0/34  
port Ten-GigabitEthernet 1/0/35  
port Ten-GigabitEthernet 1/0/36  
port Ten-GigabitEthernet 1/0/15 # this is only 1Gbps but the switch names it ten.  
quit  
#
interface Vlan-interface 100  
dhcp client identifier ascii SW00  
ip address dhcp-alloc  
quit  
#
vlan 4  
name future  
port FortyGigE1/0/41  
#
interface Vlan-interface 4  
ip address 10.40.1.100 255.255.255.0  
undo shutdown  
quit  
save  

Now verify if the switch has got the right IP addresses using command display ip interface brief. You should see the output:

<SW00> display ip interface brief  
*down: administratively down
(s): spoofing
Interface                   Physical Protocol IP Address      Description  
Vlan-interface1             up       up       10.1.1.16  
Vlan-interface100           up       up       10.100.1.100  

Step 4: Switches SW40 - SW43 (HP 5130)

Now, I configure the 5130 switches

sys  
vlan 1  
name production  
# no ports as this is default vlan
quit  
#
interface Vlan-interface 1  
dhcp client identifier ascii SW40 # change it for other switches!  
ip address dhcp-alloc  
quit  
#
vlan 100  
name experiment  
port Ten-GigabitEthernet 1/0/28  
port GigabitEthernet 1/0/10  
port GigabitEthernet 1/0/12  
quit  
#
interface vlan 100  
ip address dhcp-alloc  
dhcp client identifier ascii SW40 # change it for other switches!  
quit  
#
interface Ten-GigabitEthernet 1/0/27  
shutdown  
quit  
interface Ten-GigabitEthernet 1/0/28  
undo shutdown  
quit  
display ip interface brief  
display interface vlan 100 | include "IP packet" # to write down the MAC address for static DHCP configuration  
save  

Now you should be able to ping the DHCP server from every switch and server assuming that both server interfaces eth0 and eth1 are configured for dhcp.

user@C36# ping 10.1.0.1 # using VLAN 1  
user@C36# ping 10.100.1.16 # using VLAN 100  
user@C36# ping 10.100.1.142 # SW42  
user@C36# ping 10.100.1.100 # SW00  

Or use nmap to discover active hosts

user@C16# nmap -n -sP 10.100.1.0/24  

and the sample output you may see:

Starting Nmap 6.40 ( http://nmap.org ) at 2016-06-09 20:05 CEST  
Nmap scan report for 10.100.1.10  
Host is up (0.000082s latency).  
MAC Address: 50:65:F3:6E:55:95 (Unknown)  
Nmap scan report for 10.100.1.11  
Host is up (0.000051s latency).  
MAC Address: 9C:B6:54:75:C2:41 (Unknown)  
Nmap scan report for 10.100.1.12  
Host is up (0.000097s latency).  
MAC Address: 9C:B6:54:75:BE:E5 (Unknown)  
Nmap scan report for 10.100.1.13  
Host is up (-0.10s latency).  
MAC Address: 9C:B6:54:75:BF:6D (Unknown)  
Nmap scan report for 10.100.1.17  
Host is up (0.000086s latency).  
MAC Address: 9C:B6:54:75:BE:F5 (Unknown)  
Nmap scan report for 10.100.1.36  
Host is up (0.000086s latency).  
MAC Address: 14:02:EC:04:8B:F9 (Unknown)  
Nmap scan report for 10.100.1.37  
Host is up (-0.10s latency).  
MAC Address: 14:02:EC:04:5C:71 (Unknown)  
Nmap scan report for 10.100.1.38  
Host is up (0.000081s latency).  
MAC Address: 14:02:EC:04:6B:85 (Unknown)  
Nmap scan report for 10.100.1.39  
Host is up (0.000089s latency).  
MAC Address: 14:02:EC:04:2C:AD (Unknown)  
Nmap scan report for 10.100.1.100  
Host is up (0.0048s latency).  
MAC Address: 5C:8A:38:21:83:1B (Unknown)  
Nmap scan report for 10.100.1.135  
Host is up (0.0037s latency).  
MAC Address: 74:46:A0:5F:1E:80 (Hewlett Packard)  
Nmap scan report for 10.100.1.140  
Host is up (-0.092s latency).  
MAC Address: 2C:23:3A:BE:37:92 (Unknown)  
Nmap scan report for 10.100.1.141  
Host is up (-0.092s latency).  
MAC Address: 2C:23:3A:BE:33:A2 (Unknown)  
Nmap scan report for 10.100.1.142  
Host is up (-0.094s latency).  
MAC Address: 2C:23:3A:B3:A7:0C (Unknown)  
Nmap scan report for 10.100.1.143  
Host is up (-0.082s latency).  
MAC Address: 2C:23:3A:B3:64:AC (Unknown)  
Nmap scan report for 10.100.1.16  
Host is up.  
Nmap done: 256 IP addresses (16 hosts up) scanned in 4.68 seconds  

In the output you can see MAC addresses that need to be added to the DHCP server configuration if you want to use static IPs for the servers.

Possible problems

  • Make sure to disable dhcp server on all switches as this may lead to a serious mess in the network. For SW00, SW10, SW40, SW41, SW42, SW43, SW3500-1 run command: sys and then undo dhcp enable.
  • Can't ssh to a switch? TODO

The Rest

There is a lot of stuff that needs to be configured yet to support all planned experiment scenarios. I will cover further configuration in the next parts. Some of the tasks for the next parts include:

  • OpenFlow
  • SW10
  • ECMP (+routing)

I will update this post with the current links to the next parts once they appear!

Appendix: Switch config dumps

#
 version 7.1.045, Release 3109P05
#
 sysname SW43
#
 irf mac-address persistent timer
 irf auto-update enable
 undo irf link-delay
 irf member 1 priority 1
#
 lldp global enable
#
 password-recovery enable
#
vlan 1  
#
vlan 100  
 name experiment
#
 stp global enable
#
interface NULL0  
#
interface Vlan-interface1  
 ip address dhcp-alloc
 dhcp client identifier ascii SW43
#
interface Vlan-interface100  
 ip address dhcp-alloc
 dhcp client identifier ascii SW43
#
interface GigabitEthernet1/0/1  
#
interface GigabitEthernet1/0/2  
#
interface GigabitEthernet1/0/3  
#
interface GigabitEthernet1/0/4  
#
interface GigabitEthernet1/0/5  
#
interface GigabitEthernet1/0/6  
#
interface GigabitEthernet1/0/7  
#              
interface GigabitEthernet1/0/8  
#
interface GigabitEthernet1/0/9  
#
interface GigabitEthernet1/0/10  
 port access vlan 100
#
interface GigabitEthernet1/0/11  
#
interface GigabitEthernet1/0/12  
 port access vlan 100
#
interface GigabitEthernet1/0/13  
#
interface GigabitEthernet1/0/14  
#
interface GigabitEthernet1/0/15  
#
interface GigabitEthernet1/0/16  
#
interface GigabitEthernet1/0/17  
#
interface GigabitEthernet1/0/18  
#
interface GigabitEthernet1/0/19  
#
interface GigabitEthernet1/0/20  
#
interface GigabitEthernet1/0/21  
#
interface GigabitEthernet1/0/22  
#
interface GigabitEthernet1/0/23  
#
interface GigabitEthernet1/0/24  
#
interface Ten-GigabitEthernet1/0/25  
#
interface Ten-GigabitEthernet1/0/26  
#
interface Ten-GigabitEthernet1/0/27  
 shutdown
#
interface Ten-GigabitEthernet1/0/28  
 port access vlan 100
#              
 scheduler logfile size 16
#
line class aux  
 user-role network-admin
#
line class vty  
 user-role network-operator
#
line aux 0  
 user-role network-admin
#
line vty 0 4  
 authentication-mode scheme
 user-role network-admin
 protocol inbound ssh
#
line vty 5 63  
 user-role network-operator
#
 ssh server enable
#
radius scheme system  
 user-name-format without-domain
#              
domain system  
#
 domain default enable system
#
role name level-0  
 description Predefined level-0 role
#
role name level-1  
 description Predefined level-1 role
#
role name level-2  
 description Predefined level-2 role
#
role name level-3  
 description Predefined level-3 role
#
role name level-4  
 description Predefined level-4 role
#
role name level-5  
 description Predefined level-5 role
#
role name level-6  
 description Predefined level-6 role
#
role name level-7  
 description Predefined level-7 role
#
role name level-8  
 description Predefined level-8 role
#
role name level-9  
 description Predefined level-9 role
#
role name level-10  
 description Predefined level-10 role
#
role name level-11  
 description Predefined level-11 role
#
role name level-12  
 description Predefined level-12 role
#
role name level-13  
 description Predefined level-13 role
#              
role name level-14  
 description Predefined level-14 role
#
user-group system  
#
local-user manager class manage  
 service-type ssh
 authorization-attribute user-role network-admin
#
local-user user class manage  
 service-type http
 authorization-attribute user-role network-admin
 authorization-attribute user-role network-operator
#
 ip http enable
 ip https enable
#
return

#
 version 7.1.045, Release 2422P01
#
 sysname SW00
#
 irf mac-address persistent timer
 irf auto-update enable
 undo irf link-delay
 irf member 1 priority 1
 irf mode normal
#
 ip redirects enable
#
 lldp global enable
#
 system-working-mode StandardBridge
 password-recovery enable
#
vlan 1  
 name production
#
vlan 4  
 name fourtyG
#              
vlan 100  
 name experiment
#
 stp instance 0 root primary
#
interface NULL0  
#
interface Vlan-interface1  
 ip address dhcp-alloc
 dhcp client identifier ascii SW00
#
interface Vlan-interface4  
 ip address 10.40.1.100 255.255.255.0
 dhcp client identifier ascii SW00-VLAN4
#
interface Vlan-interface100  
 ip address dhcp-alloc
 dhcp client identifier ascii SW00
#
interface FortyGigE1/0/41  
 port access vlan 4
#
interface FortyGigE1/0/42  
#
interface M-GigabitEthernet0/0/0  
#
interface Ten-GigabitEthernet1/0/1  
#
interface Ten-GigabitEthernet1/0/2  
#
interface Ten-GigabitEthernet1/0/3  
#
interface Ten-GigabitEthernet1/0/4  
#
interface Ten-GigabitEthernet1/0/5  
#
interface Ten-GigabitEthernet1/0/6  
#
interface Ten-GigabitEthernet1/0/7  
#
interface Ten-GigabitEthernet1/0/8  
#
interface Ten-GigabitEthernet1/0/9  
#
interface Ten-GigabitEthernet1/0/10  
#              
interface Ten-GigabitEthernet1/0/11  
#
interface Ten-GigabitEthernet1/0/12  
#
interface Ten-GigabitEthernet1/0/13  
#
interface Ten-GigabitEthernet1/0/14  
#
interface Ten-GigabitEthernet1/0/15  
 port access vlan 100
#
interface Ten-GigabitEthernet1/0/16  
 port access vlan 100
#
interface Ten-GigabitEthernet1/0/17  
#
interface Ten-GigabitEthernet1/0/18  
#
interface Ten-GigabitEthernet1/0/19  
#
interface Ten-GigabitEthernet1/0/20  
#
interface Ten-GigabitEthernet1/0/21  
#
interface Ten-GigabitEthernet1/0/22  
#
interface Ten-GigabitEthernet1/0/23  
#
interface Ten-GigabitEthernet1/0/24  
#
interface Ten-GigabitEthernet1/0/25  
#
interface Ten-GigabitEthernet1/0/26  
#
interface Ten-GigabitEthernet1/0/27  
#
interface Ten-GigabitEthernet1/0/28  
#
interface Ten-GigabitEthernet1/0/29  
#
interface Ten-GigabitEthernet1/0/30  
#
interface Ten-GigabitEthernet1/0/31  
#
interface Ten-GigabitEthernet1/0/32  
#              
interface Ten-GigabitEthernet1/0/33  
 port access vlan 100
#
interface Ten-GigabitEthernet1/0/34  
 port access vlan 100
#
interface Ten-GigabitEthernet1/0/35  
 port access vlan 100
#
interface Ten-GigabitEthernet1/0/36  
 port access vlan 100
#
interface Ten-GigabitEthernet1/0/37  
#
interface Ten-GigabitEthernet1/0/38  
#
interface Ten-GigabitEthernet1/0/39  
#
interface Ten-GigabitEthernet1/0/40  
#
 scheduler logfile size 16
#
line class aux  
 user-role network-admin
#
line class vty  
 user-role network-operator
#
line aux 0  
 user-role network-admin
#
line vty 0 4  
 authentication-mode scheme
 user-role network-operator
 protocol inbound ssh
#
line vty 5 63  
 user-role network-operator
#
 ssh server enable
#
 sntp enable
 sntp unicast-server 85.214.194.162
#
radius scheme system  
 user-name-format without-domain
#
domain system  
#
 domain default enable system
#
role name level-0  
 description Predefined level-0 role
#
role name level-1  
 description Predefined level-1 role
#
role name level-2  
 description Predefined level-2 role
#
role name level-3  
 description Predefined level-3 role
#
role name level-4  
 description Predefined level-4 role
#
role name level-5  
 description Predefined level-5 role
#              
role name level-6  
 description Predefined level-6 role
#
role name level-7  
 description Predefined level-7 role
#
role name level-8  
 description Predefined level-8 role
#
role name level-9  
 description Predefined level-9 role
#
role name level-10  
 description Predefined level-10 role
#
role name level-11  
 description Predefined level-11 role
#
role name level-12  
 description Predefined level-12 role
#
role name level-13  
 description Predefined level-13 role
#
role name level-14  
 description Predefined level-14 role
#
user-group system  
#
local-user manager class manage  
 service-type ssh
 authorization-attribute user-role network-admin
#
return  

Running configuration:

; J8692A Configuration Editor; Created on release #K.15.17.0007
; Ver #08:02.ff.f7.fc.7f.ff.3f.ef:ae
hostname "SW3500-1"  
module 1 type j86xxa  
console idle-timeout serial-usb 0  
logging 10.0.1.1  
timesync sntp  
no stack  
ip default-gateway 10.1.0.1  
ip dns dhcp  
no ip icmp unreachable  
ip route 0.0.0.0 0.0.0.0 10.0.1.1  
interface 1  
   name "to-production"
   exit
interface 2  
   name "to-sw00-p15"
   exit
snmp-server community "public" unrestricted  
snmp-server host 10.0.1.1 community "public"  
snmpv3 enable  
snmpv3 restricted-access  
snmpv3 user "initial"  
no lldp config 1-24 dot3TlvEnable poe_config  
vlan 1  
   name "default"
   no untagged 2,10
   untagged 1,3-9,11-24
   ip address dhcp-bootp
   exit
vlan 100  
   name "VLAN100"
   untagged 2,10
   ip address dhcp-bootp
   exit
no tftp server  
no autorun  
no dhcp config-file-update  
no dhcp image-file-update