Wednesday 27 January 2010

VMware and Active Directory Replication

Just thought I would drop a note for those that use VMware for tier 2 server roles (most people) and probably are using snapshots/clone etc.

I have been troubleshooting a vMotion issue with a client recently where One particular VM (a domain controller) would Vmotion fine, yet after a fine hours would BSOD, however all the other VMs hosted on this host were fine.

After a couple of days troubleshooting we managed to work out the issue was a faulty bank of RAM when going over 8GB+.
This meant that if you close a few VMs bring over a new VM, as long as total utilization was under 8GB you were fine, however once you went over 8GB with a VM, that VM was the one to suffer!

During the troubleshooting process this particular VM was migrated in various ways storage then host, then storage and host in one go,cloned,snapshoted etc etc only once the VM was stable and the RAM replaced the fun with AD then started.

The troubleshooting within VMware had caused a little issue with AD.
Here is the main message (amongst a fair few):


Event Type: Error
Event Source: NTDS General
Event Category: Service Control
Event ID: 2103
Date: 26/01/2010
Time: 20:37:18
User: NT AUTHORITY\ANONYMOUS LOGON
Computer: DCV2
Description:
The Active Directory database has been restored using an unsupported restoration procedure.

Active Directory will be unable to log on users while this condition persists. As a result, the Net Logon service has paused.

User Action
See previous event logs for details.

For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.


And for those that like pictures :)



MS knows this as a "USN rollback condition" and talks about it endlessly here

So what was the fix?
Well the VM did have some FSMOs roles, so after "Seizing the roles" ran this command on the DC I was getting the above eventids on:

repadmin /options DC_Name -disable_inbound_repl -disable_outbound_repl


then ran dcpromo (to demote the controller) rebooted and ran dcpromo again (to promote the controller)and all was back to normal.

Although this issue was not directly related to vmware (could of just as easily happened with SAN snapshots or norton ghost) it is something to look out for when snapshoting/cloning and troubleshooting VM issues where the VM is looking after a tier 2 distributed app.

Thursday 21 January 2010

Vyatta - Example of OpenVPN infront of Microsoft ISA Server

Another day another lab :)



This scenrio was given to me by someone who stopped by the blog and wondered if it was possible to swap out some of the kit infront of his ISA box with Vyatta...the answer of course was yes!
Here is the diagram:


Here are the videos:
Part1:
Initial Setup and Testing

Vyatta - Example of OpenVPN infront of Microsoft ISA Server - Part 1 from Richard Vimeo on Vimeo.



Part2:
Second part of the lab setup

Vyatta - Example of OpenVPN infront of Microsoft ISA Server - Part 2 from Richard Vimeo on Vimeo.



Part 3
Load Balancing
Certificate Setup
OpenVPN Site to Site setup


Vyatta - Example of OpenVPN infront of Microsoft ISA Server - Part 3 from Richard Vimeo on Vimeo.




Part 4:
Certificate Setup/signing/installing etc
OpenVPN Site to Site setup continued..
OpenVPN Remote Access setup (+Client)


Vyatta - Example of OpenVPN infront of Microsoft ISA Server - Part 4 from Richard Vimeo on Vimeo.




Part 5
DMZ Setup
DMZ Routing & NAT
Testing!


Vyatta - Example of OpenVPN infront of Microsoft ISA Server - Part 5 from Richard Vimeo on Vimeo.




Configs!
R1

/**********************************************************************\
firewall {
all-ping enable
broadcast-ping disable
conntrack-table-size 32768
conntrack-tcp-loose enable
ip-src-route disable
ipv6-receive-redirects disable
ipv6-src-route disable
log-martians enable
name ALLOW_ESTABLISHED {
default-action drop
rule 10 {
action accept
state {
established enable
}
}
}
receive-redirects disable
send-redirects enable
source-validation disable
syn-cookies enable
}
interfaces {
ethernet eth0 {
address 98.63.88.81/29
address 98.63.88.82/29
address 98.63.88.83/29
address 98.63.88.84/29
address 98.63.88.85/29
description ISP1
duplex auto
hw-id 00:0c:29:7f:b2:7d
smp_affinity auto
speed auto
}
ethernet eth1 {
address 75.93.91.193/29
address 75.93.91.194/29
address 75.93.91.195/29
address 75.93.91.196/29
description ISP2
duplex auto
hw-id 00:0c:29:7f:b2:87
smp_affinity auto
speed auto
}
ethernet eth2 {
address 10.0.12.1/24
description R1-ISA
duplex auto
hw-id 00:0c:29:7f:b2:91
smp_affinity auto
speed auto
}
ethernet eth3 {
address 10.0.2.1/24
description R1-DMZ
duplex auto
hw-id 00:0c:29:7f:b2:9b
smp_affinity auto
speed auto
}
loopback lo {
}
openvpn vtun0 {
local-host 98.63.88.81
mode server
replace-default-route {
}
server {
subnet 10.1.8.0/24
}
tls {
ca-cert-file /etc/openvpn/ca.crt
cert-file /etc/openvpn/r1.crt
dh-file /etc/openvpn/dh1024.pem
key-file /etc/openvpn/r1.key
}
}
openvpn vtun1 {
local-address 10.1.9.1
local-host 75.93.91.193
mode site-to-site
remote-address 10.1.9.2
remote-host 213.123.123.10
tls {
ca-cert-file /etc/openvpn/ca.crt
cert-file /etc/openvpn/r1.crt
dh-file /etc/openvpn/dh1024.pem
key-file /etc/openvpn/r1.key
role passive
}
}
}
load-balancing {
wan {
flush-connections
interface-health eth0 {
failure-count 2
nexthop 98.63.88.86
success-count 1
test 10 {
ping
resp-time 5
target 98.63.88.86
}
}
interface-health eth1 {
failure-count 1
nexthop 75.93.91.198
success-count 1
test 10 {
ping
resp-time 5
target 75.93.91.198
}
}
rule 10 {
destination {
address !10.0.0.0/16
}
inbound-interface eth2
interface eth0 {
weight 1
}
interface eth1 {
weight 1
}
protocol all
}
rule 20 {
destination {
address !10.0.0.0/16
}
inbound-interface eth3
interface eth0 {
weight 1
}
interface eth1 {
weight 1
}
protocol all
}
rule 30 {
destination {
address !10.0.0.0/16
}
inbound-interface vtun0
interface eth0 {
weight 1
}
interface eth1 {
weight 1
}
protocol all
}
}
}
protocols {
static {
interface-route 10.0.10.0/24 {
next-hop-interface vtun1 {
}
}
route 0.0.0.0/0 {
next-hop 75.93.91.198 {
}
next-hop 98.63.88.86 {
}
}
route 10.0.0.0/24 {
next-hop 10.0.12.2 {
}
}
route 10.0.1.0/24 {
blackhole {
}
}
}
}
service {
dhcp-server {
disabled false
shared-network-name POOL1 {
authoritative disable
subnet 192.168.10.0/24 {
default-router 192.168.10.1
dns-server 192.168.10.1
domain-name Vyatta.local
lease 86400
start 192.168.10.10 {
stop 192.168.10.200
}
}
}
}
dns {
forwarding {
cache-size 150
listen-on eth2
listen-on eth3
listen-on vtun0
name-server 208.67.222.222
name-server 208.67.220.220
}
}
nat {
rule 10 {
destination {
address 75.93.91.194
port 25
}
inbound-interface eth1
inside-address {
address 10.0.2.10
port 25
}
protocol tcp
type destination
}
}
ssh {
allow-root true
port 22
protocol-version v2
}
}
system {
host-name R1
login {
user root {
authentication {
encrypted-password $1$ORKO400D$9GoL/vifapZLo3p.sLkUs/
plaintext-password ""
}
level admin
}
user vyatta {
authentication {
encrypted-password $1$Z9oMjC/m$r.T2vNILnVuZnIwkKhg58.
}
level admin
}
}
ntp-server 0.vyatta.pool.ntp.org
package {
auto-sync 1
repository community {
components main
distribution stable
password ""
url http://packages.vyatta.com/vyatta
username ""
}
repository kenwood {
components main
distribution kenwood
password ""
url http://packages.vyatta.com/vyatta-dev/kenwood/unstable/
username ""
}
repository lenny {
components main
distribution lenny
password ""
url http://packages.vyatta.com/debian/
username ""
}
}
syslog {
global {
facility all {
level notice
}
facility protocols {
level debug
}
}
}
time-zone GMT
}


/* Warning: Do not remove the following line. */
/* === vyatta-config-version: "cluster@1:dhcp-relay@1:dhcp-server@4:firewall@3:ipsec@1:nat@3:quagga@1:system@1:vrrp@1:wanloadbalance@1:webgui@1" === */
/* Release version: VC6_a2 */




/**********************************************************************\



R2

/**********************************************************************\
firewall {
all-ping enable
broadcast-ping disable
conntrack-table-size 32768
conntrack-tcp-loose enable
ip-src-route disable
ipv6-receive-redirects disable
ipv6-src-route disable
log-martians enable
name ALLOW_ESTABLISHED {
default-action drop
rule 10 {
action accept
state {
established enable
}
}
}
receive-redirects disable
send-redirects enable
source-validation disable
syn-cookies enable
}
interfaces {
ethernet eth0 {
address 213.123.123.10/24
description Outside
duplex auto
hw-id 00:0c:29:f5:c1:84
smp_affinity auto
speed auto
}
ethernet eth1 {
address 10.0.10.1/24
description Inside
duplex auto
hw-id 00:0c:29:f5:c1:8e
smp_affinity auto
speed auto
}
ethernet eth2 {
description DMZ
duplex auto
hw-id 00:0c:29:f5:c1:98
smp_affinity auto
speed auto
}
loopback lo {
}
openvpn vtun1 {
local-address 10.1.9.2
mode site-to-site
remote-address 10.1.9.1
remote-host 75.93.91.193
tls {
ca-cert-file /etc/openvpn/ca.crt
cert-file /etc/openvpn/r2.crt
key-file /etc/openvpn/r2.key
role active
}
}
}
protocols {
static {
interface-route 10.0.0.0/24 {
next-hop-interface vtun1 {
}
}
interface-route 10.0.2.0/24 {
next-hop-interface vtun1 {
}
}
interface-route 10.0.12.0/24 {
next-hop-interface vtun1 {
}
}
route 0.0.0.0/0 {
next-hop 213.123.123.1 {
}
}
}
}
service {
dhcp-server {
disabled false
shared-network-name POOL1 {
authoritative disable
subnet 192.168.10.0/24 {
default-router 192.168.10.1
dns-server 192.168.10.1
domain-name Vyatta.local
lease 86400
start 192.168.10.10 {
stop 192.168.10.200
}
}
}
}
dns {
forwarding {
cache-size 150
listen-on eth1
name-server 208.67.222.222
name-server 208.67.220.220
}
}
nat {
rule 10 {
destination {
address !10.0.0.0/16
}
outbound-interface eth0
source {
address 10.0.10.0/24
}
type masquerade
}
}
ssh {
allow-root true
port 22
protocol-version v2
}
}
system {
host-name R2
login {
user root {
authentication {
encrypted-password $1$ORKO400D$9GoL/vifapZLo3p.sLkUs/
plaintext-password ""
}
level admin
}
user vyatta {
authentication {
encrypted-password $1$Z9oMjC/m$r.T2vNILnVuZnIwkKhg58.
}
level admin
}
}
ntp-server 0.vyatta.pool.ntp.org
package {
auto-sync 1
repository community {
components main
distribution stable
password ""
url http://packages.vyatta.com/vyatta
username ""
}
repository kenwood {
components main
distribution kenwood
password ""
url http://packages.vyatta.com/vyatta-dev/kenwood/unstable/
username ""
}
repository lenny {
components main
distribution lenny
password ""
url http://packages.vyatta.com/debian/
username ""
}
}
syslog {
global {
facility all {
level notice
}
facility protocols {
level debug
}
}
}
time-zone GMT
}


/* Warning: Do not remove the following line. */
/* === vyatta-config-version: "cluster@1:dhcp-relay@1:dhcp-server@4:firewall@3:ipsec@1:nat@3:quagga@1:system@1:vrrp@1:wanloadbalance@1:webgui@1" === */
/* Release version: VC6_a2 */

/**********************************************************************\




ISA Screenshots:







Enjoy!