Multipathing and Multiple Connections Per Session - Two sides of the same iSCSI coin?

One again a record breaking title for a post! lets hope my google-fu is not
effected by long titles...or I'm in real trouble ;)

So I was working today on something that envolved me testing iSCSI functionality with Windows Server 2008.
While I was waiting for the VM to come up, I set about testing the iSCSI initiator within Windows 7.

What interested me most was a feature called "MCS" which stands for Multiple Connections Per Session and is defined within RFC-3720 and as such a a protocol level feature that allows features we have previously seen with MPIO.

Here is how to get there:

Load the iscsi software from Control Panel->Administative Tools->iSCSI Initiator:
Pic1:




Select the Target from the list click "properties"
Pic2:




Select the MCS policy you wish to have, I selected "fail over only" which is the same
as "fixed" in MPIO world.

Pic3:




You probably will only have one session at the moment, therfore click "add"
Dont click "connect"!
Pic4:



Click "Advanced"
Here is where you pick the other iSCSI target portal.
Pic5:






And thats great! we have a redundant path to our iSCSI targets..but notice this button:
Pic6:




Hmm MPIO is not avalible within Windows 7, which is fine as MCS pretty much gets us to the same place (Inface some say MCS is better) however with Windows Server 2008 we have the option of MPIO so lets give it a go!

First thing to remember is that MPIO is a driver thing so if you have an EMC,3par,netapp,Dell etc device they all have MPIO driver for Windows 2008 so you need to follow their instructions (and look for DSM instructions), here we are using Windows 2008 Software iSCSI Initiator and Windows Server 2008 native MPIO driver.

When you install/start iscsi on windows server 2008 it asks you to install MPIO, if you said no..or just forgot install MPIO like this:

From the "Add features Wizard"
Pic1:



Once installed select MPIO from Control Panel click "Add support for iSCSI devices"
then reboot (p.s. here is where you would add the 3rd Party DSM drive btw)
Pic2:



Go Back to the iscsi Initiator (within Administrative tools)
Pic3:



Select the target click properties
Pic4:


Highlight the sessions click "Devices..."
Pic5:



Click MPIO and select the Policy you want
Pic6:





Hope that helps someone out there!


Sources:
http://www.ietf.org/rfc/rfc3720.txt

http://www.windowsitpro.com/article/virtualization2/Q-With-iSCSI-what-s-the-difference-between-Multipath-I-O-MPIO-and-multiple-connections-per-session-MCS-.aspx

Thank You VMware - vExpert for Roggy!

A certain sense of disbelief hit me when I received the email from John Troyer letting me know that I had become a vExpert 2010.
It is an honour to be given this award especially considering the company I am in and the sites they have created:
Duncan Epping Yellow Bricks
Edwin Friesen Thinstall Guru
Eric Sloof NTPRO.NL
These sites are not only unique but also technically outstanding and if you have not bookmarked them already I suggest you do!
So thank you VMware and thank you John Troyer for making giving back to the community so easy

Basic BGP - Path Selection with Vyatta

There is actually very little BGP documentation out there on Vyatta, which is strange as if there is one real strength of Vyatta it is BGP.
This set of videos is all about BGP and if it proves popular I will do some more with some more advanced features.

Here is the Picture:




Basic BGP - Path Selection with Vyatta -Part 1
General Setup

Basic BGP - Path Selection with Vyatta -Part 1 from Roggy on Vimeo.

Basic BGP - Path Selection with Vyatta -Part 2
Checking BGP peering
Adding Next-hop-self

Basic BGP - Path Selection with Vyatta -Part 2 from Roggy on Vimeo.

Basic BGP - Path Selection with Vyatta -Part 3
Creating ACLs
Creating Prefix Lists
Creating Route-maps
Setting Local Pref
Setting Med
Clearing a Peer

Basic BGP - Path Selection with Vyatta -Part 3 from Roggy on Vimeo.

Although it can seem a little boring, I always like to include the full configs:
R1

firewall {
all-ping enable
broadcast-ping disable
conntrack-table-size 32768
conntrack-tcp-loose enable
ip-src-route disable
ipv6-receive-redirects disable
ipv6-src-route disable
log-martians enable
name ALLOW_ESTABLISHED {
default-action drop
rule 10 {
action accept
state {
established enable
}
}
}
receive-redirects disable
send-redirects enable
source-validation disable
syn-cookies enable
}
interfaces {
ethernet eth0 {
address 172.12.123.1/24
description R1-R2-R3
duplex auto
hw-id 00:0c:29:fe:17:2d
smp_affinity auto
speed auto
}
ethernet eth1 {
duplex auto
hw-id 00:0c:29:fe:17:37
smp_affinity auto
speed auto
}
ethernet eth2 {
address dhcp
duplex auto
hw-id 00:0c:29:fe:17:41
smp_affinity auto
speed auto
}
loopback lo {
address 5.5.5.5/24
address 6.6.6.6/24
address 7.7.7.7/24
address 8.8.8.8/24
}
}
policy {
}
protocols {
bgp 1 {
neighbor 172.12.123.2 {
remote-as 234
}
neighbor 172.12.123.3 {
remote-as 234
}
redistribute {
connected {
}
}
}
}
service {
dns {
forwarding {
cache-size 150
listen-on eth1
name-server 208.67.222.222
name-server 208.67.220.220
}
}
nat {
rule 10 {
outbound-interface eth0
source {
address 192.168.10.0/24
}
type masquerade
}
}
ssh {
allow-root
port 22
protocol-version v2
}
}
system {
host-name R1
login {
user root {
authentication {
encrypted-password $1$ORKO400D$9GoL/vifapZLo3p.sLkUs/
plaintext-password ""
}
level admin
}
user vyatta {
authentication {
encrypted-password $1$Z9oMjC/m$r.T2vNILnVuZnIwkKhg58.
}
level admin
}
}
ntp-server 0.vyatta.pool.ntp.org
package {
auto-sync 1
repository community {
components main
distribution stable
password ""
url http://packages.vyatta.com/vyatta
username ""
}
repository lenny {
components main
distribution lenny
password ""
url http://packages.vyatta.com/debian/
username ""
}
repository VC6 {
components main
distribution VC6.0
password ""
url http://packages.vyatta.com/vyatta/
username ""
}
}
syslog {
global {
facility all {
level notice
}
facility protocols {
level debug
}
}
}
time-zone GMT
}


/* Warning: Do not remove the following line. */
/* === vyatta-config-version: "cluster@1:dhcp-relay@1:dhcp-server@4:firewall@3:ipsec@2:nat@3:quagga@1:system@3:vrrp@1:wanloadbalance@2:webgui@1:webproxy@1" === */
/* Release version: VC6.0-2010.03.22 */



R2

firewall {
all-ping enable
broadcast-ping disable
conntrack-table-size 32768
conntrack-tcp-loose enable
ip-src-route disable
ipv6-receive-redirects disable
ipv6-src-route disable
log-martians enable
name ALLOW_ESTABLISHED {
default-action drop
rule 10 {
action accept
state {
established enable
}
}
}
receive-redirects disable
send-redirects enable
source-validation disable
syn-cookies enable
}
interfaces {
ethernet eth0 {
address 172.12.123.2/24
description R1-R2-R3
duplex auto
hw-id 00:0c:29:fa:84:8d
smp_affinity auto
speed auto
}
ethernet eth1 {
address 172.12.234.2/24
description Inside
duplex auto
hw-id 00:0c:29:fa:84:97
smp_affinity auto
speed auto
}
ethernet eth2 {
address dhcp
description DMZ
duplex auto
hw-id 00:0c:29:fa:84:a1
smp_affinity auto
speed auto
}
loopback lo {
}
}
policy {
access-list 100 {
rule 10 {
action permit
destination {
any
}
source {
any
}
}
}
access-list 150 {
rule 10 {
action permit
destination {
any
}
source {
inverse-mask 0.0.0.255
network 172.12.234.0
}
}
}
route-map SET-LOCAL-PREF {
rule 10 {
action permit
match {
ip {
address {
access-list 100
}
}
}
set {
local-preference 301
}
}
}
route-map SET-MED {
rule 10 {
action permit
match {
ip {
address {
access-list 150
}
}
}
set {
metric 201
}
}
rule 20 {
action permit
match {
ip {
address {
access-list 100
}
}
}
}
}
}
protocols {
bgp 234 {
neighbor 172.12.123.1 {
remote-as 1
route-map {
export SET-MED
}
}
neighbor 172.12.234.4 {
nexthop-self
remote-as 234
route-map {
export SET-LOCAL-PREF
}
}
redistribute {
connected {
}
}
}
}
service {
dns {
forwarding {
cache-size 150
listen-on eth1
name-server 208.67.222.222
name-server 208.67.220.220
}
}
ssh {
allow-root
port 22
protocol-version v2
}
}
system {
host-name R2
login {
user root {
authentication {
encrypted-password $1$ORKO400D$9GoL/vifapZLo3p.sLkUs/
plaintext-password ""
}
level admin
}
user vyatta {
authentication {
encrypted-password $1$Z9oMjC/m$r.T2vNILnVuZnIwkKhg58.
}
level admin
}
}
ntp-server 0.vyatta.pool.ntp.org
package {
auto-sync 1
repository community {
components main
distribution stable
password ""
url http://packages.vyatta.com/vyatta
username ""
}
repository lenny {
components main
distribution lenny
password ""
url http://packages.vyatta.com/debian/
username ""
}
repository VC6 {
components main
distribution VC6.0
password ""
url http://packages.vyatta.com/vyatta/
username ""
}
}
syslog {
global {
facility all {
level notice
}
facility protocols {
level debug
}
}
}
time-zone GMT
}


/* Warning: Do not remove the following line. */
/* === vyatta-config-version: "cluster@1:dhcp-relay@1:dhcp-server@4:firewall@3:ipsec@2:nat@3:quagga@1:system@3:vrrp@1:wanloadbalance@2:webgui@1:webproxy@1" === */
/* Release version: VC6.0-2010.03.22 */



R3

firewall {
all-ping enable
broadcast-ping disable
conntrack-table-size 32768
conntrack-tcp-loose enable
ip-src-route disable
ipv6-receive-redirects disable
ipv6-src-route disable
log-martians enable
name ALLOW_ESTABLISHED {
default-action drop
rule 10 {
action accept
state {
established enable
}
}
}
receive-redirects disable
send-redirects enable
source-validation disable
syn-cookies enable
}
interfaces {
ethernet eth0 {
address 172.12.123.3/24
description R1-R2-R3
duplex auto
hw-id 00:0c:29:21:bd:6f
smp_affinity auto
speed auto
}
ethernet eth1 {
address 172.12.234.3/24
description R2-R3-R4
duplex auto
hw-id 00:0c:29:21:bd:79
smp_affinity auto
speed auto
}
ethernet eth2 {
address dhcp
description DMZ
duplex auto
hw-id 00:0c:29:21:bd:83
smp_affinity auto
speed auto
}
loopback lo {
}
}
policy {
access-list 150 {
rule 10 {
action permit
destination {
any
}
source {
inverse-mask 0.0.0.255
network 172.12.234.0
}
}
}
prefix-list ALL-ROUTES {
rule 10 {
action permit
le 32
prefix 0.0.0.0/0
}
}
route-map SET-LOCAL-PREF {
rule 10 {
action permit
match {
ip {
address {
prefix-list ALL-ROUTES
}
}
}
set {
local-preference 201
}
}
}
route-map SET-MED {
rule 10 {
action permit
match {
ip {
address {
access-list 150
}
}
}
set {
metric 101
}
}
rule 20 {
action permit
match {
ip {
address {
prefix-list ALL-ROUTES
}
}
}
}
}
}
protocols {
bgp 234 {
neighbor 172.12.123.1 {
remote-as 1
route-map {
export SET-MED
}
}
neighbor 172.12.234.4 {
nexthop-self
remote-as 234
route-map {
export SET-LOCAL-PREF
}
}
redistribute {
connected {
}
}
}
}
service {
dns {
forwarding {
cache-size 150
listen-on eth1
name-server 208.67.222.222
name-server 208.67.220.220
}
}
ssh {
allow-root
port 22
protocol-version v2
}
}
system {
host-name R3
login {
user root {
authentication {
encrypted-password $1$ORKO400D$9GoL/vifapZLo3p.sLkUs/
plaintext-password ""
}
level admin
}
user vyatta {
authentication {
encrypted-password $1$Z9oMjC/m$r.T2vNILnVuZnIwkKhg58.
}
level admin
}
}
ntp-server 0.vyatta.pool.ntp.org
package {
auto-sync 1
repository community {
components main
distribution stable
password ""
url http://packages.vyatta.com/vyatta
username ""
}
repository lenny {
components main
distribution lenny
password ""
url http://packages.vyatta.com/debian/
username ""
}
repository VC6 {
components main
distribution VC6.0
password ""
url http://packages.vyatta.com/vyatta/
username ""
}
}
syslog {
global {
facility all {
level notice
}
facility protocols {
level debug
}
}
}
time-zone GMT
}


/* Warning: Do not remove the following line. */
/* === vyatta-config-version: "cluster@1:dhcp-relay@1:dhcp-server@4:firewall@3:ipsec@2:nat@3:quagga@1:system@3:vrrp@1:wanloadbalance@2:webgui@1:webproxy@1" === */
/* Release version: VC6.0-2010.03.22 */



R4

firewall {
all-ping enable
broadcast-ping disable
conntrack-table-size 32768
conntrack-tcp-loose enable
ip-src-route disable
ipv6-receive-redirects disable
ipv6-src-route disable
log-martians enable
name ALLOW_ESTABLISHED {
default-action drop
rule 10 {
action accept
state {
established enable
}
}
}
receive-redirects disable
send-redirects enable
source-validation disable
syn-cookies enable
}
interfaces {
ethernet eth0 {
address 172.12.234.4/24
duplex auto
smp_affinity auto
speed auto
}
ethernet eth2 {
address dhcp
duplex auto
smp_affinity auto
speed auto
}
loopback lo {
}
}
protocols {
bgp 234 {
neighbor 172.12.234.2 {
remote-as 234
}
neighbor 172.12.234.3 {
remote-as 234
}
redistribute {
connected {
}
}
}
}
service {
dhcp-server {
disabled false
shared-network-name POOL1 {
authoritative disable
subnet 192.168.10.0/24 {
default-router 192.168.10.1
dns-server 192.168.10.1
domain-name Vyatta.local
lease 86400
start 192.168.10.10 {
stop 192.168.10.200
}
}
}
}
dns {
forwarding {
cache-size 150
listen-on eth1
name-server 208.67.222.222
name-server 208.67.220.220
}
}
nat {
rule 10 {
outbound-interface eth0
source {
address 192.168.10.0/24
}
type masquerade
}
}
ssh {
allow-root
port 22
protocol-version v2
}
}
system {
host-name R4
login {
user root {
authentication {
encrypted-password $1$ORKO400D$9GoL/vifapZLo3p.sLkUs/
plaintext-password ""
}
level admin
}
user vyatta {
authentication {
encrypted-password $1$Z9oMjC/m$r.T2vNILnVuZnIwkKhg58.
}
level admin
}
}
ntp-server 0.vyatta.pool.ntp.org
package {
auto-sync 1
repository community {
components main
distribution stable
password ""
url http://packages.vyatta.com/vyatta
username ""
}
repository lenny {
components main
distribution lenny
password ""
url http://packages.vyatta.com/debian/
username ""
}
repository VC6 {
components main
distribution VC6.0
password ""
url http://packages.vyatta.com/vyatta/
username ""
}
}
syslog {
global {
facility all {
level notice
}
facility protocols {
level debug
}
}
}
time-zone GMT
}


/* Warning: Do not remove the following line. */
/* === vyatta-config-version: "cluster@1:dhcp-relay@1:dhcp-server@4:firewall@3:ipsec@2:nat@3:quagga@1:system@3:vrrp@1:wanloadbalance@2:webgui@1:webproxy@1" === */
/* Release version: VC6.0-2010.03.22 */



VMware Storage Alphabet Soup and Making the Most of VMwares Multipathing

Having recently moved into an enviroment where the storage is a little alien to me, I thought would be helpful to buff up on some storage knowledge and thought it might help some readers too.
Here is a diagram of a midrange san:

(Thanks Virtualgeek for this picture)


See the two items list as "Data Processor(head) A" and "Data Processor(head) B"?
Traditionally if you are using Active/Active Processor array you should use "Fixed" as the Multipathing method and In an Active/Passive array use "MRU".

However this changed with:
ALUA:Symmetric Logical Unit Access
Essentially in midrange san enviroments (EMC Clariion etc), this allows an unoptimized and an optimized path to a lun through different heads.

ESX(4) the HBA is aware of optimized and unoptimized paths as it knows which head has control of the LUN!
Suddenly we can use MRU with Active/Active heads.

MRU Most recently used:Use the Optimized Path unless it is not avalible then use the Unoptimized path (ESX 4.0/vSphere only)

Fixed: Always use this LUN unless it is unavalible.

NMP:Native MultiPath Driver:

MMP:Multipath Plugin (EMC Powerpath)

Round Robin: Within ESX server's iSCSI HBA it sends 4000 IO blocks down one path then moves to the next path.

Custom Policy:
Use the following commmand to tweak the iSCSI HBA
esxcfg-mpath --lun vmhba32:0:8 --policy custom --custom-hba-policy any --custom-max-blocks 1024 --custom-max-commands 50 --custom-target-policy any


References:
http://www.vmware.com/pdf/vi3_35/esx_3/r35/vi3_35_25_iscsi_san_cfg.pdf
http://www.vmware.com/pdf/vi3_35_25_roundrobin.pdf
http://virtualgeek.typepad.com/virtual_geek/2009/09/a-couple-important-alua-and-srm-notes.html
http://virtualgeek.typepad.com/virtual_geek/2008/08/celerra-virtual.html

Having problems connecting Outlook 2007 to Exchange 2003? SPN might be to blame

Afternoon,

I was having some problems today connection Outlook 2007 to an Exchange 2003 SP2 box today.

Here is the Error message that was being recieved:

The connection to Microsoft Exchange is unavailable. Outlook must be online or connected to complete this action.


After a couple of hours playing around I managed to narrow this down to Outlook 2007 now authenticating with kerberos and not being able to find the Service Prinical Name for the global catalog server.

Authentication via Outlook 2003 (NTLM) was perfect, as was HTTPS (OWA) this was just affecting Kerberos.

Here is how I fixed it:
1) Install Windows Server 2003 Support tools:SUPPTOOLS.MSI
2)Run setspn -L ExchangeServerName you will see something like this:

Registered ServicePrincipalNames for CN=,CN=Computers,DC=example,DC=com:
exchangeAB/
exchangeAB/.example.com
exchangeMDB/
exchangeMDB/.example.com
exchangeRFR/
exchangeRFR/.example.com
SMTPSVC/
SMTPSVC/.example.com
HOST/
HOST/.example.com


exchangeAB/
exchangeAB/.example.com

The above line is the one we are interested in. We need to change it:

setspn -D exchangeAB/ExchangeServerName ExchangeServerName
setspn -D exchangeAB/ExchangeServerName.example.com ExchangeServerName


Then re-add the details:

setspn -A exchangeAB/GlobalCatalogServerName GlobalCatalogServerName
setspn -A exchangeAB/GlobalCatalogServerName.example.com GlobalCatalogServerName


The output from setspn should now be:


Registered ServicePrincipalNames for CN=,CN=Computers,DC=example,DC=com:
exchangeMDB/
exchangeMDB/.example.com
exchangeRFR/
exchangeRFR/.example.com
SMTPSVC/
SMTPSVC/.example.com
HOST/
HOST/.example.com


Note the ExchangeAB SPNs are gone as they are now pointing to the domain controller (GC)


Reference List:
http://support.microsoft.com/kb/927612/en-us

Using VMware View with Network cards as Removeable Devices

This is only going to be a quick on hopefully :)

For those that do not know vmware presents most of their nics as removeable/USB devices PCNET(Vlance),Intel e1000 and vmxnet,vmxnet2,vmxnet3 this is to allow the feature of "hot add" which is a great way off adding hardware to a VM without powering the machine off.

Today whilst playing around it was highlighted to me that some "adventurous" VM View users that had USB enabled used that oppurtunity to disable the network card. Forcing the intervention of the admin from the cloud to re-add the vnic

Heres a couple of ways around it:

1) Use the configuration options to add "devices.hotplug" = "false" like this:


*Also does not affect cpu/mem hotplug

2) Edit the *.vmx file and add:

devices.hotplug = "false"


3) Hide the "safelty remove hardware option" like this:



4) Use the "NoDisplayClass" to customize the driver *yuck!


Hope that helps someone out there!



Source:http://kb.vmware.com/selfservice/microsites/search.do?cmd=displayKC&docType=kc&externalId=1012225&sliceId=1&docTypeID=DT_KB_1_1&dialogID=64244650&stateId=0%200%2067543541
*NB:Using VMXnet3 does not disable hot plug as indicated in the KB

Using HSRP,CARP and VRRP within VMware

Hi all,

Just a quick note incase it helps anyone else out there.

I was recently in a lab moving from a vSS (Virtual Standard Switch) over to a vDS (Virtual distributed switch) when I came across and issue with a Pfsense box.

I had two VMs using CARP (which is like HSRP or VRRP) to push about a highly avalible IP address.
Now most vm admins out there will work out that for VRRP or CARP to work you need to enable "Promiscuous Mode" within the vSS or within the port group.


The issue I came across was with Promiscuous Mode and one uplink port (vDS or vSS) CARP was working perfectly however on adding the second uplink port to the vDS, pings to the CARP address were dropping.

Therefore - lesson learned was: When load balaning across multiple nics (or uplink ports in vDS terminology) you need to also be using the load balancing method of "Route Based on IP hash" (with accompanying switch config) if you plan to use CARP or VRRP else it will not work!