GOPAS TECHED 2012 Ing Ondej eveek GOPAS a

GOPAS TECHED 2012 Ing. Ondřej Ševeček | GOPAS a. s. | MCM: Directory Services | MVP: Enterprise Security | ondrej@sevecek. com | www. sevecek. com | ACTIVE DIRECTORY REPLICATION ISSUES AND TROUBLESHOOTING

Active Directory Replication Issues and Troubleshooting NETWORK SERVICES

Central Database LDAP – Lightweight Directory Access Protocol database query language, similar to SQL TCP/UDP 389, SSL TCP 636 Global Catalog (GC) – TCP/UDP 3268, SSL TCP 3269 D/COM Dynamic TCP – Replication D/COM Dynamic TCP – NSPI Kerberos UDP/TCP 88 Windows NT 4. 0 SAM SMB/CIFS TCP 445 (or Net. BIOS) �password resets, SAM queries SMB/DCOM Dynamic TCP �NTLM pass-through �Kerberos PAC validation

Design Considerations Distributed system DCs disconnected for very long times several months Multimaster replication with some FSMO roles

Design Considerations Example: Caribean cruises, DC/IS/Exchange on board with tens of workstations and users, some staff hired during journey. No or bad satelite connectivity only. DCs synced after ship is berthed at main office. Challenge: Must work independently for long time periods. Different independent cruiseliners/DCs can accomodate changes to user accounts, email addresses, Exchange settings. Cannot afford lost of any one.

Database Microsoft JET engine JET Blue common with Microsoft Exchange used by DHCP, WINS, COM+, WMI, CA, CS, RDS Broker %WINDIR%NTDS. DIT ESENTUTL Opened by LSASS. EXE

Installed services LSASS Security Accounts Manager TCP 445 SMB + Named Pipes D/COM Dynamic TCP Kerberos Key Distribution Center UDP, TCP 88 Kerberos Active Directory Domain Services UDP, TCP 389 LDAP NTDS. DIT

Installed services LSASS NT 4. 0 TCP 445 SMB + Named Pipes SAM NTLM Pass-through PAC validation D/COM Dynamic TCP Connect to domain KDC UDP, TCP 88 Kerberos Windows 2000+ NTDS UDP, TCP 389, . . . LDAP/ADSI Client NTDS Replication FIM/DRS API Client

Uninstallation DCPROMO requires working replication connectivity with other DCs DCPROMO /forceremoval does not access network at all can run in DS Restore Mode

NTDSUTIL Metadata Cleanup Connection Connect to server srv 2. idtt. local Quit Select operation target List sites Select site 0 List domains in site Select domain 0 List servers in site Select server 0 Quit Remove selected server

Metadata Cleanup

Active Directory Replication Issues and Troubleshooting TOPOLOGY

Knowledge Consistency Checker (KCC) runs 5 minutes after boot Repl topology update delay (secs) runs every 15 minutes periodically Repl topology update period (secs)

Intrasite Replication Topology DC 1 DC 4 DC 2 DC 3

Originating Updates and Notifications DC 1 15 sec DC 4 DC 2 3 sec DC 3

Notification and Replication I have got some changes DCOM TCP Random Kerberos Authenticated DC 1 DC 2 Give me your replica DCOM TCP Random Kerberos Authenticated

Intrasite Replication – 3 Hops max. DC 1 DC 4 DC 2 DC 3 DC 5 DC 7 DC 6

Intersite Replication (no Bridgeheads) DC 1 DC 5 DC 2 DC 3 DC 4 DC 6 DC 7

Intersite Replication (no Bridgeheads) 15 sec DC 1 DC 5 DC 2 DC 3 3 sec schedule 3 sec DC 4 3 sec DC 7 DC 6 3 sec

Intersite Replication with a Bridgehead 15 sec DC 1 DC 5 schedule DC 2 3 sec DC 3 3 sec DC 4 3 sec DC 7 DC 6 3 sec

Intrasite Replication Uses notifications by default (originating/received) 300/30 sec on Windows 2000 15/3 sec on Windows 2003 Occurs every hour as scheduled n. TDSSite. Settings At this frequency KCC detects unavailable partners HKLMSystemCCSServicesNTDSParameters Replicator notify pause after modify (secs) Replicator notify pause between DSAs (secs)

Intrasite Replication notification 15 sec random TCP download changes DC 1 DC 2 random TCP download changes schedule random TCP

Intersite Replication DC 1 DC 2 download changes schedule random TCP

Intersite Replication Does not use notifications by default site. Link: options = USE_NOTIFY (1) Compression used site. Link: options = DISABLE_COMPRESSION (4) Bridge all site links

Site Link Design

Site Link Design (Better? ) Olomouc London Roma Cyprus Paris Berlin

Site Link Design (Worse? ) Olomouc Paris London Roma Cyprus Berlin

Static TCP for Replication HKLMSystemCurrent. Control. SetServices NTDSParameters TCP/IP Port = DWORD Replication + NSPI NetlogonParameters DCTcpip. Port = DWORD LSASS (Pass-through) NTFRSParameters RPC TCP/IP Port Assignment = DWORD DFSRDIAG Static. RPC /port: xxx /Member: dc 1

Urgent Replication (Notification) Intrasite only intersite also if notification enabled Do not wait for delay (15/3 sec) In the case of account lockout password and lockout policy RID FSMO owner change DC password or trust account password change

Immediate Replication (Notification) Password changes from DCs to PDC Regardless of site boundaries PDC downloads only the single user object all changed attributes but only single object From DC/PDC further with normal replication

Example Replication Traffic Atomic replication of a single object with a one byte attribute change Notification + replication intersite compressed Overall 7536 B 30 packets ~10 round trips 50 ms round trip means 500 ms transfer time consumption at 120 kbps Useful data ~80 B

Bridge All Site Links On A Olomouc A B A Prague London A Paris B Roma site links are transitive Cyprus can be disabled on IP transport A

Bridge All Site Links Off A Olomouc A B A Prague London A Paris B Roma site links are not transitive Cyprus partition is cut off A

GC Replication A GC Olomouc A GC A Prague London GC Paris B Roma Cyprus A one-way: from the source NC into the nearest GC two-way: GCs between themselves A

GC Replication A Olomouc A A London Prague B GC Paris Roma Cyprus A B one-way: from the source NC into the nearest GC two-way: GCs between themselves A

Subnetting in AD (Apps) DC 1 DC 5 Exchange DC 2 DC 3 Exchange 10. 0. 248 / 29 DC 4 10. x. x / 16

Subnetting in AD (Recovery) DC 1 DC 5 Recovery Site 10. 0. 7 / 32 DC 3 DC 4 10. x. x / 16

Rebuilding After Failure

Rebuilding After Failure Inter-site Intersite. Failures. Allowed Max. Failure. Time. For. Intersite. Link (secs) Intra-site (immediate neighbors) Critical. Link. Failures. Allowed Max. Failure. Time. For. Critical. Link Intra-site (optimalization for non-critical) Non. Critical. Link. Failures. Allowed Max. Failure. Time. For. Non. Critical. Link

Active Directory Replication Issues and Troubleshooting MODIFICATIONS

Modification operations Create new object Modify attributes change/delete value change distinguished. Name = rename Rename container all subobjects renamed as well

Replication Metadata REPADMIN /Show. Obj. Meta all attributes when originating DC

Replication conflicts The later action wins if no one is later then random (USN) Attribute modified on two DCs “simultaneously” only one change wins Linked multivalue attribute modified merged (on 2003+ forest level) Object/container deleted and object modified deleted Object moved into a deleted container CN=lost and found Two objects with the same s. AMAccount. Name, cn or user. Principal. Name created object renamed, logins duplicit

Linked Multi-values

Replication 11: 05 Kamil 10: 00 Helen 11: 00 DC 1 9: 00 DC 2

Replication Basics 11: 30 Kamil 10: 00 Helen 11: 00 DC 1 11: 30 Kamil 10: 00 Helen 11: 00 DC 2

Replication Basics 12: 05 Kamil 10: 00 Helen 11: 00 Judith DC 1 12: 00 DC 1 11: 30 Kamil 10: 00 Helen 11: 00 DC 2

Replication Basics 12: 30 DC 1 12: 30 Kamil 10: 00 Helen 11: 00 Judith 12: 00 DC 1 DC 2

Replication Basics 12: 30 Kamil 10: 00 Helen 11: 00 DC 1 12: 30 Kamil 10: 00 DC 1 Helen 11: 00 DC 1 Judith 12: 00 DC 1 DC 2 Marie Judith 12: 00 DC 1 DC 3 11: 00 Me

Replication Basics 12: 30 Kamil 10: 00 Helen 11: 00 DC 1 12: 30 Kamil 10: 00 DC 1 Helen 11: 00 DC 1 Judith 12: 00 DC 1 DC 2 Marie Judith 11: 00 12: 00 DC 1 Kamil DC 3 DC 1 10: 30 DC 2 7: 00 10: 00 DC 1 Me

Replication Basics 13: 30 Kamil 10: 00 Helen 11: 00 DC 1 12: 30 Kamil 10: 00 DC 1 Helen 11: 00 DC 1 Judith 12: 00 DC 1 DC 2 Marie Judith 11: 00 12: 00 DC 1 Kamil DC 3 DC 1 10: 30 DC 2 7: 00 10: 00 DC 1 Me

Replication Basics 13: 30 Kamil 10: 00 Helen 11: 00 DC 1 12: 30 Kamil 10: 00 DC 1 Helen 11: 00 DC 1 Judith 12: 00 DC 1 DC 2 Marie Judith 11: 00 12: 00 DC 1 Kamil DC 3 DC 1 12: 30 DC 2 13: 30 10: 00 DC 1 Me

Replication Basics 14: 15 Kamil 10: 00 Helen 11: 00 Judith 12: 00 DC 1 Kamil 10: 00 DC 1 Helen 11: 00 DC 1 Judith 12: 00 DC 1 Marie 11: 00 DC 2 DC 3 DC 1 12: 30 DC 2 13: 30

USN Each object modification increments USN for that object and for the whole DC Each DC remembers USNs of its replication partners repadmin /showutdvec

USN 2 USN 5001 1 USN 1001 2 5001 3 3001 1 1001 3 3001 3 USN 3001 1 1001 2 5001

USN Kamil 1002 John 1003 2 USN 5001 1 USN 1003 2 5001 3 3001 1 1001 3 3001 3 USN 3001 1 1001 2 5001

USN Kamil 1002 John 1003 1 USN 1003 2 5001 3 3001 2 USN 5001 Notify Give me 1 1001 3 3001 1002, 3 3 USN 3001 1 1001 2 5001

USN Kamil 1002 John 1003 Kamil 5002 John 5003 1 USN 1003 2 5001 3 3001 2 USN 5003 1 1003 3 3001 3 USN 3001 1 1001 2 5001

USN Kamil 1002 John 1003 Kamil 5002 John 5003 Maria 1 USN 1003 2 5001 3 3001 2 USN 5004 1 1003 3 3001 3 USN 3001 1 1001 2 5001

USN Kamil 1002 John 1003 Kamil 5002 John 5003 Maria 1 USN 1003 2 5001 3 3001 2 USN 5004 1 1003 3 3001 3 USN 3004 1 1003 2 5004 Kamil 3002 John 3003 Maria 3004

USN 1 Kamil 1002 1 John 1003 1 Kamil 5002 1 John 5003 2 Maria 1 USN 1003 2 5001 3 3001 2 USN 5004 1 1003 3 3001 3 1 USN 30041 Kamil 3002 John 3003 Maria 3004 1 2 1003 2 5004

USN 1 Kamil 1002 1 John 1003 1 Kamil 5002 1 John 5003 2 Maria 1 USN 1003 2 5001 3 3004 2 Maria 2 USN 5004 1 1003 3 3001 3 1 USN 30041 Kamil 3002 John 3003 Maria 3004 1 2 1003 2 5004

Active Directory Replication Issues and Troubleshooting REPLICATION PROBLEMS

The Three Problems Single DC offline for a long time not so long as tombstone! authentication problem Tombstone lifetime two separate DC zones not a “business” consistency problem USN rollback restore from snapshot, image, manual backup total inconsistency!

DC Offline for Long Time Month 0 PWD 21 OLD PWD - DC 2 DC 1 DC 2 PWD 21 DC 3 PWD 31 My. PWD 11 PWD 31 OLD PWD - DC 3

DC Offline for Long Time Month 1 PWD 22 OLD PWD 21 DC 2 DC 1 DC 2 PWD 21 DC 3 PWD 31 My. PWD 11 PWD 32 OLD PWD 31 DC 3

DC Offline for Long Time Month 2 PWD 23 OLD PWD 22 DC 1 DC 2 PWD 21 DC 3 PWD 31 My. PWD 11 PWD 33 OLD PWD 32 DC 3

DC Offline for Long Time Month 3 PWD 23 OLD PWD 22 PWD 21 Kerberos KDC DC 2 TGS Ticket DC 1 DC 2 PWD 21 DC 3 PWD 31 My. PWD 11 PWD 33 OLD PWD 32 DC 3

DC Offline for Long Time Month 3 PWD 23 OLD PWD 22 PWD 23 KDC Disabled DC 2 TGS Ticket Kerberos KDC DC 1 DC 2 PWD 21 DC 3 PWD 31 My. PWD 11 PWD 33 OLD PWD 32 DC 3

DC Isolated for Long Time Month 3 DC 1 PWD 13 KDC Disabled PWD 11 DC 2 TGT Ticket Kerberos KDC DC 1 My. PWD 13 DC 1 PWD DC 3 11

DC Isolated for Long Time Month 3 KDC Disabled NETDOM RESETPWD DC 1 PWD 14 DC 2 TGT Ticket DC 1 My. PWD 14 DC 1 PWD DC 3 14

Lingering Objects When DC didn’t replicate during the tombstone. Lifetime, it halts replication Can be restored by Allow Replication with Divergent and Corrupt Partner HKLMSystemCCSServicesNTDSParameters turn on, replicate, turn off

Objects and Tombstones Frank DC 1 Stan Tania Frank DC 3 Stan Tania Frank DC 2 Stan Tania Frank DC 4 Stan Tania

Objects and Tombstones Frank DC 1 Stan Tania Frank DC 3 Stan Tania Frank DC 2 Stan Tania Frank DC 4 Stan Tania

Objects and Tombstones Frank DC 1 Stan Tania Frank DC 3 Stan Tania Frank DC 2 Stan Tania Frank DC 4 Stan Tania

Objects and Tombstones Frank DC 1 Stan Tania Frank DC 3 Stan Tania Frank DC 2 Stan Tania Frank DC 4 Stan Tania

Garbage Collection 1/day Frank DC 1 Frank DC 3 Tania Stan Tania Frank DC 2 Stan Tania Frank DC 4 Tania

Garbage Collection 1/day Frank DC 1 Frank DC 3 Tania Frank DC 2 Frank DC 4 Tania

Lingering Objects Frank DC 1 Stan Tania Frank DC 3 Stan Tania Frank DC 2 Stan Tania Frank DC 4 Stan Tania

Lingering Objects Frank DC 1 Stan Tania Frank DC 3 Stan Tania Frank DC 2 Stan Tania Frank DC 4 Stan Tania

Lingering Objects Frank DC 1 Frank DC 3 Tania Stan Tania Frank DC 2 Frank DC 4 Tania Stan Tania

Lingering Objects Frank DC 1 Frank DC 3 Tania Stan Tania Frank DC 2 Frank DC 4 Tania Stan Tania

Possible Problems Inconsistent distributed database Proliferation of partial objects after modification of some attributes Allow Replication with Divergent and Corrupt Partner blocks replication after tombstone lifetime Strict Replication Consistency detects partial objects if replication allowed

Lingering Objects

Lingering Objects Strict Replication Consistency HKLMSystemCCSServicesNTDSParameters 1 – do not replicate 0 – request full copy from source By default only on new Windows 2003+ installations

Automatic Repair Philosphy? Business logic says “deleted already” should we investigate? Metadata cleanup? we may need some data from the vesel Remove lingering objects

Removing Lingering Objects REPADMIN /Remove. Lingering. Objects target source. GUID DN /advisory_mode source. GUID – healthy DC’s GUID (without {}) target – suspected DC’s name with lingering objects DN – naming context DN /advisory_mode just logs the found objects (on the ill DC)

Lingering Object found/deleted

Correct Registry Settings Long term normal operation Strict consistency = 1 Allow divergent partner = 0 Temporary repair operation Strict consistency = 1 Allow divergent partner = 1

USN Rollback May or may not be detected Cannot be repaired not always lingering objects! DC must be denoted/repromoted unplug network DCPROMO /forceremoval NTDSUTIL Roles NTDSUTIL Metadata Cleanup

USN Rollback DC 1 1001 Snapshot 2 USN 5001 1 1001 3 3001

USN Rollback DC 1 1001 Kamil 1002 John 1003 Judith 1004 Helen 1005 Eva 1006 Snapshot 2 USN 5001 1 1001 3 3001

USN Rollback DC 1 1001 Snapshot Kamil 1002 John 1003 Judith 1004 Helen 1005 Eva 1006 2 USN 5001 1 1006 3 3001

Restore DC 1 1001 Restore Kamil 1002 John 1003 Judith 1004 Helen 1005 Eva 1006 2 USN 5001 1 1006 3 3001

USN Rollback (Detectable) DC 1 1001 Restore Kamil 1002 John 1003 Judith 1004 Helen 1005 Eva 1006 2 USN 5001 1 1006 3 3001

USN Rollback (Detectable) DC 1 1001 Restore Kamil 1002 Frank 1002 John 1003 Stan 1003 Judith 1004 Helen 1005 Eva 1006 2 USN 5001 1 1006 3 3001

USN Rollback (Detectable)

USN Rollback (Detectable)

USN Rollback (Detectable)

USN Rollback (Detectable)

USN Rollback (Non-detect. ) DC 1 1001 Restore Kamil 1002 Frank 1002 John 1003 Stan 1003 Judith 1004 Tania 1004 Helen 1005 Mark 1005 Eva 1006 Martin 1006 Victor 1007 Leo 1008 2 USN 5001 1 1006 3 3001

USN Rollback (Non-detect. ) DC 1 1001 Frank 1002 Stan 1003 Tania 1004 Mark 1005 Martin 1006 Victor 1007 Leo 1008 Restore Kamil 1002 John 1003 Judith 1004 Helen 1005 Eva 1006 Victor 1007 Leo 1008 2 USN 5001 1 1008 3 3001

Restoring VM Snapshots Restore offline HKLMSystemCurrent. Control. SetServicesNTDS Database Restored from Backup = DWORD = 1 Restart NTDS service changes Invocation. ID of the database instance

GOPAS TECHED 2012 Ing. Ondřej Ševeček | GOPAS a. s. | MCM: Directory Services | MVP: Enterprise Security | ondrej@sevecek. com | www. sevecek. com | THANK YOU!