UCSD 6 5 Bare Metal Automation Windows Server
UCSD 6. 5 Bare Metal Automation Windows Server 2012 R 2 with SAN Boot Lauri Toropainen Technical Solutions Architect, ltoropai@cisco. com Updated July 2017
Solution Overview © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
Cisco UCS Director - Overview Infrastructure Automation and Private Cloud Foundation Infrastructure Automation and Orchestration Physical Compute, Network, Storage, Hypervisor Day 0 bring up of Infrastructure Single Pane Management Converged Infrastructure (Flex. Pod, Vblock, . . etc. ) Private Cloud Foundation Virtualization Physical Servers Secure Multi-Tenancy Application Infrastructure Blueprint Resource Management Self-Service Portal Metering and Showback Improve IT Operational Efficiency Reduce Opex & Decrease Service Delivery time © 2017 Reduce Capex Increased Visibility Cisco and/or its affiliates. rights reserved. Cisco on Confidential All Increase focus value-add services Network Storage UCS Director
UCS Director – Bare Metal as a Service What is BMaa. S ? OS Only OS + Application Cisco UCS • Automate OS & Application provisioning on Bare Metal Servers • Self-Service Delivery, Access and Management • Secure Isolation between Tenants • Chargeback/Showback © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
Bare Metal Server Provisioning Architecture OS image library • Windows • Linux • Hypervisors Bare Metal Provisioning Workflow UCS Director Bare Metal Agent (BMA) 2 1 UCS Director 1 Management Network 1 PXE Network 2 UCS Service Profile 4 Customer Network 3 1) 2) 3) 4) Infrastructure management network PXE network for OS installation Storage network Customer network © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Confidential Supported storage (such as Net. App, EMC, or IBM) FC/FCo. E/i. SCSI boot LUNs (block storage)
UCSD 6. 5 BMA Initial Setup © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
Pre-requisites for UCSD 6. 5 BMA • Deploy supported converged infrastructure stack (such as Flex. Pod) with UCS, Nexus/MDS, and 3 rd party storage • Install UCSD 6. 5 based on this guide • Add infrastructure element managers in UCSD as physical and virtual accounts based on these guides © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
Pre-requisites for UCSD 6. 5 BMA • Prepare separate management and PXE networks in the converged infrastructure • Allocate DHCP pool in the PXE network • Management and PXE networks should be added as tagged VLANs on the UCS v. NICs connected to the infrastructure ESXi hosts • Management and PXE networks should be configured with VLAN ID in v. Sphere • PXE VLAN should have active uplinks only to a single fabric in UCS, depending on which fabric (A or B) BM hosts will connect to during the provisioning © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
UCSD 6. 5 BMA Installation • Install UCSD 6. 5 BMA based on this guide • Make sure that the BMA is connected to management and PXE networks • During the first time login, change the root password from the default (pxeboot) • Verify that the BMA VM is completely booted up by launching /opt/infra/bin/shelladmin © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
Add BMA Account to UCSD • Enable Samba service for Windows • Set Samba password, which will be used later in Windows BM provisioning (startnet. cmd script) • The default Samba password for smbuser account is cisco 123 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
Configure DHCP pool for the BMA Account © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
Start BMA Services and Verify Service Status © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
Adding Windows Server 2012 R 2 OS Image to UCSD 6. 5 BMA © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
Pre-requisites for Windows Server 2012 R 2 Install ADK • Deploy a Windows VM • Download and install Windows Assessment and Deployment Kit (ADK) version 10 that supports Windows Server 2012 R 2. You may also use ADK 8. 1 • Navigate to Windows Preinstallation Environment folder and modify the following lines in copype. cmd • set SOURCE=C: Program Files (x 86)Windows Kits10Assessment and Deployment KitWindows Preinstallation Environment%WINPE_ARCH% • set FWFILESROOT=C: Program Files (x 86)Windows Kits10Assessment and Deployment KitDeployment Tools%WINPE_ARCH%Oscdimg • NOTE: it looks like copype. cmd script will not work without these changes © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
Pre-requisites for Windows Server 2012 R 2 Create Win. PE. wim • Go to C: Program Files (x 86)Windows Kits10Assessment and Deployment KitWindows Preinstallation Environment and create Win. PE files • copype. cmd amd 64 C: Win. PE_amd 64 • Create directory C: Win 2 k 12 R 2 x 64 • Copy boot. wmi from C: Win. PE_amd 64mediasources to C: Win 2 k 12 R 2 x 64Win. PE. wim and mount it • Dism /mount-wim /wimfile: C: Win 2 k 12 R 2 x 64Win. PE. wim /index: 1 /mountdir: C: Win 2 k 12 R 2 x 64mount © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
Pre-requisites for Windows Server 2012 R 2 Edit startnet. cmd • Edit C: Win 2 k 12 R 2 x 64mountwindowssystem 32startnet. cmd according to your BMA setup • wpeinit • wpeutil. exe Initialize. Network • wpeutil. exe disablefirewall • net use R: \192. 168. 1. 1bits /user: smbuser cisco 123 • R: PXE interface in BMA Samba password in BMA • @echo • Powershell. exe -Non. Interactive -No. Logo -Execution. Policy bypass -command R: En. Route. ps 1 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
Pre-requisites for Windows Server 2012 R 2 Add UCS drivers to Win. PE. wim • Download UCS drivers for Windows from software. cisco. com, for example �ucs-bxxx-drivers-windows. 3. 1. 3 b. iso • Extract drivers from the ISO file to a directory • Add drivers to Win. PE. wim • Dism /image: C: Win 2 k 12 R 2 x 64mount /Add-Driver /driver: C: Win 2 k 12 R 2 x 64ucs-b-drivers-w 2 k 12 r 2 /Recurse /forceunsigned • Verify that drivers were added correctly • Dism /Get-Drivers /Image: C: Win 2 k 12 R 2 x 64mount © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
Pre-requisites for Windows Server 2012 R 2 Add the required Win. PE Power. Shell packages to Win. PE. wim (1) • Dism /Add-Package /Image: "C: Win 2 k 12 R 2 x 64mount" /Package. Path: "C: Program Files (x 86)Windows Kits10Assessment and Deployment KitWindows Preinstallation Environmentamd 64Win. PE_OCsWin. PEWMI. cab" • Dism /Add-Package /Image: "C: Win 2 k 12 R 2 x 64mount" /Package. Path: "C: Program Files (x 86)Windows Kits10Assessment and Deployment KitWindows Preinstallation Environmentamd 64Win. PE_OCsen-usWin. PEWMI_en-us. cab" • Dism /Add-Package /Image: "C: Win 2 k 12 R 2 x 64mount" /Package. Path: "C: Program Files (x 86)Windows Kits10Assessment and Deployment KitWindows Preinstallation Environmentamd 64Win. PE_OCsWin. PENet. FX. cab" • Dism /Add-Package /Image: "C: Win 2 k 12 R 2 x 64mount" /Package. Path: "C: Program Files (x 86)Windows Kits10Assessment and Deployment KitWindows Preinstallation Environmentamd 64Win. PE_OCsen-usWin. PENet. FX_en-us. cab" • Dism /Add-Package /Image: "C: Win 2 k 12 R 2 x 64mount" /Package. Path: "C: Program Files (x 86)Windows Kits10Assessment and Deployment KitWindows Preinstallation Environmentamd 64Win. PE_OCsWin. PEScripting. cab" • Dism /Add-Package /Image: "C: Win 2 k 12 R 2 x 64mount" /Package. Path: "C: Program Files (x 86)Windows Kits10Assessment and Deployment KitWindows Preinstallation Environmentamd 64Win. PE_OCsen-usWin. PEScripting_en-us. cab" © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
Pre-requisites for Windows Server 2012 R 2 Add the required Win. PE Power. Shell packages to Win. PE. wim (2) • Dism /Add-Package /Image: "C: Win 2 k 12 R 2 x 64mount" /Package. Path: "C: Program Files (x 86)Windows Kits10Assessment and Deployment KitWindows Preinstallation Environmentamd 64Win. PE_OCsWin. PEPower. Shell. cab" • Dism /Add-Package /Image: "C: Win 2 k 12 R 2 x 64mount" /Package. Path: "C: Program Files (x 86)Windows Kits10Assessment and Deployment KitWindows Preinstallation Environmentamd 64Win. PE_OCsen-usWin. PEPower. Shell_en-us. cab" • Dism /Add-Package /Image: "C: Win 2 k 12 R 2 x 64mount" /Package. Path: "C: Program Files (x 86)Windows Kits10Assessment and Deployment KitWindows Preinstallation Environmentamd 64Win. PE_OCsWin. PEStorage. WMI. cab" • Dism /Add-Package /Image: "C: Win 2 k 12 R 2 x 64mount" /Package. Path: "C: Program Files (x 86)Windows Kits10Assessment and Deployment KitWindows Preinstallation Environmentamd 64Win. PE_OCsen-usWin. PEStorage. WMI_en-us. cab" • Dism /Add-Package /Image: "C: Win 2 k 12 R 2 x 64mount" /Package. Path: "C: Program Files (x 86)Windows Kits10Assessment and Deployment KitWindows Preinstallation Environmentamd 64Win. PE_OCsWin. PEDism. Cmdlets. cab" • Dism /Add-Package /Image: "C: Win 2 k 12 R 2 x 64mount" /Package. Path: "C: Program Files (x 86)Windows Kits10Assessment and Deployment KitWindows Preinstallation Environmentamd 64Win. PE_OCsen-usWin. PEDism. Cmdlets_en-us. cab" © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
Pre-requisites for Windows Server 2012 R 2 Unmount Win. PE. wim and upload it to BMA • Unmount Win. PE. wim committing the changes • Dism /Unmount-Image /Mount. Dir: C: Win 2 k 12 R 2 x 64mount /Commit • Upload the modified Win. PE. wim to UCSD BMA /opt/cnsaroot/Boot, replacing the version that already exists there © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
Pre-requisites for Windows Server 2012 R 2 Prepare Windows installation files • Upload Windows Server 2012 R 2 installation files (ISO) to BMA /tmp directory • Mount and copy the files from the ISO image to the existing Samba directory for Windows Server 2012 R 2 on the BMA • mkdir /mnt/iso • mount -o loop /tmp/en_windows_server_2012_r 2_with_update_x 64_dvd. iso • • /mnt/iso cd /mnt/iso cp -r. /samba/Win 2 k 12 R 2 x 64/ cd /tmp umount /mnt/iso © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
Pre-requisites for Windows Server 2012 R 2 Add UCS drivers to install. wim (1) • On the Windows VM, mount Windows Server 2012 R 2 ISO image • Copy sourcesinstall. wim from the mounted drive to C: Win 2 k 12 R 2 x 64 • Check the available OS edition names in install. wim • Get-Windows. Image -Image. Path C: Win 2 k 12 R 2 x 64install. wim • Image. Index : 1 • Image. Name : Windows Server 2012 R 2 SERVERSTANDARDCORE • Image. Index : 2 • Image. Name : Windows Server 2012 R 2 SERVERSTANDARD • Image. Index : 3 • Image. Name : Windows Server 2012 R 2 SERVERDATACENTERCORE • Image. Index : 4 • Image. Name : Windows Server 2012 R 2 SERVERDATACENTER © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
Pre-requisites for Windows Server 2012 R 2 Add UCS drivers to install. wim (2) • Mount the OS edition you are planning to use later with BMA • Dism /mount-wim /wimfile: C: Win 2 k 12 R 2 x 64install. wim /index: 2 /mountdir: C: Win 2 k 12 R 2 x 64mount • Add the previously downloaded UCS Windows drivers to the OS edition • Dism /image: C: Win 2 k 12 R 2 x 64mount /Add-Driver /driver: C: Win 2 k 12 R 2 x 64ucs-b-drivers-w 2 k 16 /Recurse • Unmount the OS edition committing the changes • Dism /Unmount-Image /Mount. Dir: C: Win 2 k 12 R 2 x 64mount /Commit • Repeat these steps for all other OS editions you are planning to use © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
Pre-requisites for Windows Server 2012 R 2 Upload install. wim to BMA • Upload the modified install. wim to UCSD BMA /samba/Win 2 k 12 R 2 x 64/sources, replacing the version that already exists there © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
UCS Service Profile Setup for EMC SAN Boot © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
Pre-requisites for UCS Service Profiles • Create pools required for the service profile template • MAC address pools for v. NICs • WWPN pools for v. HBAs (Fabric A and B) • UUID pool • WWNN pool • Server pool with unassociated servers © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
Pre-requisites for UCS Service Profiles • Create v. NIC templates required for the service profile template Management v. NIC with PXE VLAN (native) – used during BM provisioning • Management v. NIC with Management VLAN (native) – used after BM provisioning • • Create v. HBA templates required for the service profile template v. HBA for Fabric A SAN boot • v. HBA for Fabric B SAN boot (multipathing) • • Create boot policies required during and after the BM provisioning Initial boot policy with a single FC/FCo. E path and PXE LAN interface • Post-provisioning boot policy with FC/FCo. E multipathing for fabric A and B • • Always use initial templates, in order to allow changes in the individual service profiles during the provisioning © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
Pre-requisites for UCS Service Profiles Management v. NIC template with PXE VLAN Enable fabric failover with the same default fabric, which is used by the PXE VLAN in v. Sphere © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
Pre-requisites for UCS Service Profiles Management v. NIC template with Management VLAN Enable fabric failover to simplify NIC teaming in Windows OS © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
Pre-requisites for UCS Service Profiles Fabric A v. HBA template © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
Pre-requisites for UCS Service Profiles Fabric B v. HBA template © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
Pre-requisites for UCS Service Profiles Boot policy for BM provisioning SAN disk (with a single path) must be placed before the PXE LAN interface, because Windows will reboot multiple times during the installation © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Confidential Add EMC Storage Processor A WWPN as the primary target
Pre-requisites for UCS Service Profiles Boot policy for post-BM provisioning Add EMC Storage Processor A and B WWPNs as the primary and secondary targets © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
Pre-requisites for UCS Service Profiles EMC VNX Settings Verify that Storage Processors are reachable in both fabrics © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
Create UCS Service Profile Template © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
Create UCS Service Profile Template © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
Create UCS Service Profile Template Use Management v. NIC template with PXE VLAN for eth 0 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
UCSD Workflow for Windows Server 2012 R 2 BM Provisioning © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
Windows Server 2012 R 2 BM Provisioning Workflow © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
Windows Server 2012 R 2 BM Provisioning Workflow Execute workflow This workflow has user inputs for the hostname and administrator password. The IP configuration will be provided by DHCP. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
Windows Server 2012 R 2 BM Provisioning Workflow © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
Windows Server 2012 R 2 BM Provisioning Workflow At this point we’ll use boot policy with PXE LAN interface © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
Windows Server 2012 R 2 BM Provisioning Workflow n 5548 -lab 1# show zone name BM-W 2 k 12 R 2 -Demo 2 -vhba 0 -to-vnx-spa 4 vsan 201 pwwn 20: 00: 25: b 5: 02: 0 a: 00 [bm_w 2 k 12 r 2_demo 2_fc 0_A] pwwn 50: 06: 01: 64: 3 e: a 0: 11: a 0 [vnx 5500_SPA_4] n 5548 -lab 1# show zoneset name hellab-ucs-vnx vsan 201 zone name BM-W 2 k 12 R 2 -Demo 2 -vhba 0 -to-vnx-spa 4 vsan 201 pwwn 20: 00: 25: b 5: 02: 0 a: 00 [bm_w 2 k 12 r 2_demo 2_fc 0_A] pwwn 50: 06: 01: 64: 3 e: a 0: 11: a 0 [vnx 5500_SPA_4] Adding zoning for a single path in fabric A © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
Windows Server 2012 R 2 BM Provisioning Workflow Service profile is associated with a server in a pool © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
Windows Server 2012 R 2 BM Provisioning Workflow At this point open KVM console for monitoring the provisioning tasks © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
Windows Server 2012 R 2 BM Provisioning Workflow The server name will be used in the EMC object names Task creates a 100 GB LUN for Windows © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
Windows Server 2012 R 2 BM Provisioning Workflow © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
Windows Server 2012 R 2 BM Provisioning Workflow © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
Windows Server 2012 R 2 BM Provisioning Workflow Add initiators from both fabrics to the storage group © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
Windows Server 2012 R 2 BM Provisioning Workflow © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
Windows Server 2012 R 2 BM Provisioning Workflow © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Confidential Verify in UCSD that PXE environment has been set up
Windows Server 2012 R 2 BM Provisioning Workflow Configuration files for this request are located here in BMA: /samba/as-repository/00 -25 -b 5 -02 -00 -00 Task is using this PXE configuration template: /opt/cnsaroot/templates/Win 2 k 12 R 2 x 64 -DHCP/autounattend. xml © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
Windows Server 2012 R 2 BM Provisioning Workflow Verify that server is able to access the EMC SAN target © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
Windows Server 2012 R 2 BM Provisioning Workflow Verify that server gets an IP address and PXE configuration from BMA © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
Windows Server 2012 R 2 BM Provisioning Workflow Verify that server starts downloading Win. PE. wim © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
Windows Server 2012 R 2 BM Provisioning Workflow When startnet. cmd executes, server should be able to mount R: from BMA and launch Power. Shell script En. Route. ps 1, which will locate the unattend answer file for this PXE request (00 -2 b-b 5 -02 -00 -00. xml) © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
Windows Server 2012 R 2 BM Provisioning Workflow Setup will download and install Windows Server 2012 R 2 from BMA © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
Windows Server 2012 R 2 BM Provisioning Workflow Windows will restart multiple times during the installation. SAN disk must be the first device in the UCS boot policy, otherwise PXE task will start again © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
Windows Server 2012 R 2 BM Provisioning Workflow Setup will add device drivers to Windows and restart again © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
Windows Server 2012 R 2 BM Provisioning Workflow Setup will execute Windows First Logon commands from the answer file and restart one more time © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
Windows Server 2012 R 2 BM Provisioning Workflow When PXE request is removed, the status will change to “Archived” © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
Windows Server 2012 R 2 BM Provisioning Workflow Server is powered off to allow changes in boot policy and v. NIC configuration © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
Windows Server 2012 R 2 BM Provisioning Workflow n 5548 -lab 2# show zone name BM-W 2 k 12 R 2 -Demo 2 -vhba 1 -to-vnx-spb 4 vsan 202 pwwn 20: 00: 25: b 5: 02: 0 b: 00 [bm_w 2 k 12 r 2_demo 2_fc 1_B] pwwn 50: 06: 01: 6 c: 3 e: a 0: 11: a 0 [vnx 5500_SPB_4] n 5548 -lab 2# show zoneset name hellab-ucs-vnx vsan 202 zone name BM-W 2 k 12 R 2 -Demo 2 -vhba 1 -to-vnx-spb 4 vsan 202 pwwn 20: 00: 25: b 5: 02: 0 b: 00 [bm_w 2 k 12 r 2_demo 2_fc 1_B] pwwn 50: 06: 01: 6 c: 3 e: a 0: 11: a 0 [vnx 5500_SPB_4] Adding zoning for the second path in fabric B (multipathing) © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
Windows Server 2012 R 2 BM Provisioning Workflow In the final setup PXE LAN interface is removed and the second SAN path is added © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
Windows Server 2012 R 2 BM Provisioning Workflow v. NIC eth 0 with PXE VLAN is removed v. NIC eth 0 with Management VLAN is added © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
Windows Server 2012 R 2 BM Provisioning Workflow Server is powered on and should be reachable with RDP © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
- Slides: 67