UCSD 6 5 Bare Metal Automation Windows Server

UCSD 6. 5 Bare Metal Automation Windows Server 2016 with i. SCSI Boot Lauri Toropainen Technical Solutions Architect, ltoropai@cisco. com Updated July 2017

Solution Overview © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Confidential

Cisco UCS Director - Overview Infrastructure Automation and Private Cloud Foundation Infrastructure Automation and Orchestration Physical Compute, Network, Storage, Hypervisor Day 0 bring up of Infrastructure Single Pane Management Converged Infrastructure (Flex. Pod, Vblock, . . etc. ) Private Cloud Foundation Virtualization Physical Servers Secure Multi-Tenancy Application Infrastructure Blueprint Resource Management Self-Service Portal Metering and Showback Improve IT Operational Efficiency Reduce Opex & Decrease Service Delivery time © 2017 Reduce Capex Increased Visibility Cisco and/or its affiliates. rights reserved. Cisco on Confidential All Increase focus value-add services Network Storage UCS Director

UCS Director – Bare Metal as a Service What is BMaa. S ? OS Only OS + Application Cisco UCS • Automate OS & Application provisioning on Bare Metal Servers • Self-Service Delivery, Access and Management • Secure Isolation between Tenants • Chargeback/Showback © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Confidential

Bare Metal Server Provisioning Architecture OS image library • Windows • Linux • Hypervisors Bare Metal Provisioning Workflow UCS Director Bare Metal Agent (BMA) 2 1 UCS Director 1 Management Network 1 PXE Network 2 UCS Service Profile 4 Customer Network 3 1) 2) 3) 4) Infrastructure management network PXE network for OS installation Storage network Customer network © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Confidential Supported storage (such as Net. App, EMC, or IBM) FC/FCo. E/i. SCSI boot LUNs (block storage)

UCSD 6. 5 BMA Initial Setup © 2017 Cisco and/or its affiliates. All rights

Pre-requisites for UCSD 6. 5 BMA • Deploy supported converged infrastructure stack (such as Flex. Pod) with UCS, Nexus/MDS, and 3 rd party storage • Install UCSD 6. 5 based on this guide • Add infrastructure element managers in UCSD as physical and virtual accounts based on these guides © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Confidential

Pre-requisites for UCSD 6. 5 BMA • Prepare separate management and PXE networks in the converged infrastructure • Allocate DHCP pool in the PXE network • Management and PXE networks should be added as tagged VLANs on the UCS v. NICs connected to the infrastructure ESXi hosts • Management and PXE networks should be configured with VLAN ID in v. Sphere • PXE VLAN should have active uplinks only to a single fabric in UCS, depending on which fabric (A or B) BM hosts will connect to during the provisioning © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Confidential

UCSD 6. 5 BMA Installation • Install UCSD 6. 5 BMA based on this guide • Make sure that the BMA is connected to management and PXE networks • During the first time login, change the root password from the default (pxeboot) • Verify that the BMA VM is completely booted up by launching /opt/infra/bin/shelladmin © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Confidential

Add BMA Account to UCSD • Enable Samba service for Windows • Set Samba password, which will be used later in Windows BM provisioning (startnet. cmd script) • The default Samba password for smbuser account is cisco 123 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Confidential

Configure DHCP pool for the BMA Account © 2017 Cisco and/or its affiliates. All

Start BMA Services and Verify Service Status © 2017 Cisco and/or its affiliates. All

Adding Windows Server 2016 OS Image to UCSD 6. 5 BMA © 2017 Cisco

Pre-requisites for Windows Server 2016 Install ADK • Deploy a Windows VM • Download and install Windows Assessment and Deployment Kit (ADK) version 10 that supports Windows Server 16 • Navigate to Windows Preinstallation Environment folder and modify the following lines in copype. cmd • set SOURCE=C: Program Files (x 86)Windows Kits10Assessment and Deployment KitWindows Preinstallation Environment%WINPE_ARCH% • set FWFILESROOT=C: Program Files (x 86)Windows Kits10Assessment and Deployment KitDeployment Tools%WINPE_ARCH%Oscdimg • NOTE: it looks like copype. cmd script will not work without these changes © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Confidential

Pre-requisites for Windows Server 2016 Create Win. PE. wim • Go to C: Program Files (x 86)Windows Kits10Assessment and Deployment KitWindows Preinstallation Environment and create Win. PE files • copype. cmd amd 64 C: Win. PE_amd 64 • Create directory C: Win 2 k 16 x 64 • Copy boot. wmi from C: Win. PE_amd 64mediasources to C: Win 2 k 16 x 64Win. PE. wim and mount it • Dism /mount-wim /wimfile: C: Win 2 k 16 x 64Win. PE. wim /index: 1 /mountdir: C: Win 2 k 16 x 64mount © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Confidential

Pre-requisites for Windows Server 2016 Edit startnet. cmd • Edit C: Win 2 k 16 x 64mountwindowssystem 32startnet. cmd according to your BMA setup • wpeinit • wpeutil. exe Initialize. Network • wpeutil. exe disablefirewall • net use R: \192. 168. 1. 1bits /user: smbuser cisco 123 • R: PXE interface in BMA Samba password in BMA • @echo • Powershell. exe -Non. Interactive -No. Logo -Execution. Policy bypass -command R: En. Route. ps 1 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Confidential

Pre-requisites for Windows Server 2016 Add UCS drivers to Win. PE. wim • Download UCS drivers for Windows from software. cisco. com, for example �ucs-bxxx-drivers-windows. 3. 1. 3 b. iso • Extract drivers from the ISO file to a directory • Add drivers to Win. PE. wim • Dism /image: C: Win 2 k 16 x 64mount /Add-Driver /driver: C: Win 2 k 16 x 64ucs-b -drivers-w 2 k 16 /Recurse /forceunsigned • Verify that drivers were added correctly • Dism /Get-Drivers /Image: C: Win 2 k 16 x 64mount © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Confidential

Pre-requisites for Windows Server 2016 Add the required Win. PE Power. Shell packages to Win. PE. wim (1) • Dism /Add-Package /Image: "C: Win 2 k 16 x 64mount" /Package. Path: "C: Program Files (x 86)Windows Kits10Assessment and Deployment KitWindows Preinstallation Environmentamd 64Win. PE_OCsWin. PEWMI. cab" • Dism /Add-Package /Image: "C: Win 2 k 16 x 64mount" /Package. Path: "C: Program Files (x 86)Windows Kits10Assessment and Deployment KitWindows Preinstallation Environmentamd 64Win. PE_OCsen-usWin. PEWMI_en-us. cab" • Dism /Add-Package /Image: "C: Win 2 k 16 x 64mount" /Package. Path: "C: Program Files (x 86)Windows Kits10Assessment and Deployment KitWindows Preinstallation Environmentamd 64Win. PE_OCsWin. PENet. FX. cab" • Dism /Add-Package /Image: "C: Win 2 k 16 x 64mount" /Package. Path: "C: Program Files (x 86)Windows Kits10Assessment and Deployment KitWindows Preinstallation Environmentamd 64Win. PE_OCsen-usWin. PENet. FX_en-us. cab" • Dism /Add-Package /Image: "C: Win 2 k 16 x 64mount" /Package. Path: "C: Program Files (x 86)Windows Kits10Assessment and Deployment KitWindows Preinstallation Environmentamd 64Win. PE_OCsWin. PEScripting. cab" • Dism /Add-Package /Image: "C: Win 2 k 16 x 64mount" /Package. Path: "C: Program Files (x 86)Windows Kits10Assessment and Deployment KitWindows Preinstallation Environmentamd 64Win. PE_OCsen-usWin. PEScripting_en-us. cab" © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Confidential

Pre-requisites for Windows Server 2016 Add the required Win. PE Power. Shell packages to Win. PE. wim (2) • Dism /Add-Package /Image: "C: Win 2 k 16 x 64mount" /Package. Path: "C: Program Files (x 86)Windows Kits10Assessment and Deployment KitWindows Preinstallation Environmentamd 64Win. PE_OCsWin. PEPower. Shell. cab" • Dism /Add-Package /Image: "C: Win 2 k 16 x 64mount" /Package. Path: "C: Program Files (x 86)Windows Kits10Assessment and Deployment KitWindows Preinstallation Environmentamd 64Win. PE_OCsen-usWin. PEPower. Shell_en-us. cab" • Dism /Add-Package /Image: "C: Win 2 k 16 x 64mount" /Package. Path: "C: Program Files (x 86)Windows Kits10Assessment and Deployment KitWindows Preinstallation Environmentamd 64Win. PE_OCsWin. PEStorage. WMI. cab" • Dism /Add-Package /Image: "C: Win 2 k 16 x 64mount" /Package. Path: "C: Program Files (x 86)Windows Kits10Assessment and Deployment KitWindows Preinstallation Environmentamd 64Win. PE_OCsen-usWin. PEStorage. WMI_en-us. cab" • Dism /Add-Package /Image: "C: Win 2 k 16 x 64mount" /Package. Path: "C: Program Files (x 86)Windows Kits10Assessment and Deployment KitWindows Preinstallation Environmentamd 64Win. PE_OCsWin. PEDism. Cmdlets. cab" • Dism /Add-Package /Image: "C: Win 2 k 16 x 64mount" /Package. Path: "C: Program Files (x 86)Windows Kits10Assessment and Deployment KitWindows Preinstallation Environmentamd 64Win. PE_OCsen-usWin. PEDism. Cmdlets_en-us. cab" © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Confidential

Pre-requisites for Windows Server 2016 Unmount Win. PE. wim and upload it to BMA • Unmount Win. PE. wim committing the changes • Dism /Unmount-Image /Mount. Dir: C: Win 2 k 16 x 64mount /Commit • Upload the modified Win. PE. wim to UCSD BMA /opt/cnsaroot/Boot, replacing the version that already exists there © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Confidential

Pre-requisites for Windows Server 2016 Prepare Windows installation files • Upload Windows Server 2016 installation files (ISO) to BMA /tmp directory • Mount and copy the files from the ISO image to the existing Samba directory for Windows Server 2016 on the BMA • mkdir /mnt/iso • mount -o loop /tmp/en_windows_server_2016_x 64_dvd_9718492. iso /mnt/iso • cd /mnt/iso • cp -r. /samba/Win 2 k 16 x 64/ • cd /tmp • umount /mnt/iso © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Confidential

Pre-requisites for Windows Server 2016 Add UCS drivers to install. wim (1) • On the Windows VM, mount Windows Server 2016 ISO image • Copy sourcesinstall. wim from the mounted drive to C: Win 2 k 16 x 64 • Check the available OS edition names in install. wim • Get-Windows. Image -Image. Path C: Win 2 k 16 x 64install. wim • Image. Index : 1 • Image. Name : Windows Server 2016 SERVERSTANDARDCORE • Image. Index : 2 • Image. Name : Windows Server 2016 SERVERSTANDARD • Image. Index : 3 • Image. Name : Windows Server 2016 SERVERDATACENTERCORE • Image. Index : 4 • Image. Name : Windows Server 2016 SERVERDATACENTER © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Confidential

Pre-requisites for Windows Server 2016 Add UCS drivers to install. wim (2) • Mount the OS edition you are planning to use later with BMA • Dism /mount-wim /wimfile: C: Win 2 k 16 x 64install. wim /index: 2 /mountdir: C: Win 2 k 16 x 64mount • Add the previously downloaded UCS Windows drivers to the OS edition • Dism /image: C: Win 2 k 16 x 64mount /Add-Driver /driver: C: Win 2 k 16 x 64ucs-b -drivers-w 2 k 16 /Recurse • Unmount the OS edition committing the changes • Dism /Unmount-Image /Mount. Dir: C: Win 2 k 16 x 64mount /Commit • Repeat these steps for all other OS editions you are planning to use © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Confidential

Pre-requisites for Windows Server 2016 Upload install. wim to BMA • Upload the modified install. wim to UCSD BMA /samba/Win 2 k 16 x 64/sources, replacing the version that already exists there © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Confidential

Pre-requisites for UCS Service Profiles • Create pools required for the service profile template • MAC address pools for v. NICs • IP address pools for i. SCSI initiators on both fabrics (i. SCSI- A, i. SCSI-B) • UUID pool • IQN pool for i. SCSI initiators • Server pool with unassociated servers © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Confidential

Pre-requisites for UCS Service Profiles • Create v. NIC templates required for the service profile template Management v. NIC with PXE VLAN (native) – used during BM provisioning • Management v. NIC with Management VLAN (native) – used after BM provisioning • i. SCSI-A v. NIC for i. SCSI boot • i. SCSI-B v. NIC for i. SCSI boot (multipathing) • • Create boot policies required during and after the BM provisioning Initial boot policy with a single i. SCSI path and PXE LAN interface • Post-provisioning boot policy with i. SCSI multipathing for fabric A and B • • Always use initial templates, in order to allow changes in the individual service profiles during the provisioning © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Confidential

Pre-requisites for UCS Service Profiles Management v. NIC template with PXE VLAN Enable fabric failover with the same default fabric, which is used by the PXE VLAN in v. Sphere © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Confidential

Pre-requisites for UCS Service Profiles Management v. NIC template with Management VLAN Enable fabric failover to simplify NIC teaming in Windows OS © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Confidential

Pre-requisites for UCS Service Profiles i. SCSI-A v. NIC template © 2017 Cisco and/or

Pre-requisites for UCS Service Profiles i. SCSI-B v. NIC template © 2017 Cisco and/or

Pre-requisites for UCS Service Profiles Boot policy for BM provisioning i. SCSI disk (with a single path) must be placed before the PXE LAN interface, because Windows will reboot multiple times during the installation © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Confidential

Pre-requisites for UCS Service Profiles Boot policy for post-BM provisioning i. SCSI disk only

Pre-requisites for UCS Service Profiles Net. App SVM Settings Verify that i. SCSI is enabled with target addresses in both fabrics © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Confidential

Create UCS Service Profile Template Use Management v. NIC template with PXE VLAN for eth 0 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Confidential Place i. SCSI v. NICs before eth 0 NOTE: v. NIC eth 0 used for the PXE request will be created 3 rd as the result

Create UCS Service Profile Template Add the Net. App i. SCSI address in fabric A as the target interface. Similarly, add target interfaces in post-provisioning boot policy BM_ISCSI_BOOT for both fabrics (not shown here) © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Confidential

UCSD Workflow for Windows Server 2016 BM Provisioning © 2017 Cisco and/or its affiliates.

Windows Server 2016 BM Provisioning Workflow © 2017 Cisco and/or its affiliates. All rights

Windows Server 2016 BM Provisioning Workflow Execute workflow This workflow has user inputs for the hostname and administrator password. The IP configuration will be provided by DHCP. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Confidential

Windows Server 2016 BM Provisioning Workflow At this point we’ll use boot policy with

Windows Server 2016 BM Provisioning Workflow The server name will be used in the

Windows Server 2016 BM Provisioning Workflow Initiator name is taken from task 2 output (Create UCS Service Profile from Template) © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Confidential

Windows Server 2016 BM Provisioning Workflow Task creates a 100 GB i. SCSI LUN

Windows Server 2016 BM Provisioning Workflow Configuration files for this request are located here in BMA: /samba/as-repository/00 -25 -b 5 -02 -00 -00 Task is using this PXE configuration template: /opt/cnsaroot/templates/Win 2 k 16 x 64 -DHCP/autounattend. xml © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Confidential

Windows Server 2016 BM Provisioning Workflow Verify that server is able to access the

Windows Server 2016 BM Provisioning Workflow Verify that server gets an IP address and

Windows Server 2016 BM Provisioning Workflow Verify that server starts downloading Win. PE. wim

Windows Server 2016 BM Provisioning Workflow When startnet. cmd executes, server should be able to mount R: from BMA and launch Power. Shell script En. Route. ps 1, which will locate the unattend answer file for this PXE request (00 -2 b-b 5 -02 -00 -00. xml) © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Confidential

Windows Server 2016 BM Provisioning Workflow Setup will download and install Windows Server 2016

Windows Server 2016 BM Provisioning Workflow Windows will restart multiple times during the installation. i. SCSI disk must be the first device in the UCS boot policy, otherwise PXE task will run in a loop © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Confidential

Windows Server 2016 BM Provisioning Workflow Setup will add device drivers to Windows and

Windows Server 2016 BM Provisioning Workflow Setup will execute Windows First Logon commands from the answer file and restart one more time © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Confidential

Windows Server 2016 BM Provisioning Workflow When PXE request is removed, the status will

Windows Server 2016 BM Provisioning Workflow In the final setup PXE LAN interface is removed and the second i. SCSI path is added © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Confidential

Windows Server 2016 BM Provisioning Workflow v. NIC eth 0 with PXE VLAN is removed v. NIC eth 0 with Management VLAN is added © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Confidential

Windows Server 2016 BM Provisioning Workflow Server is powered on and should be reachable

Post-Provisioning © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Confidential

Windows Server 2016 Post-Provisioning Multipath I/O (MPIO) is enabled by default during the installation. However, support for i. SCSI devices should be added separately. This can be done manually on the host or by adding this Power. Shell command in the answer file template. If MPIO support for i. SCSI devices is disabled, Windows will see two copies of the i. SCSI disk, when both paths are in use. If enabled, a single disk should be visible. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Confidential