Cisco UCS Energy Efficient BIOS ProfilePolicy M 5

Cisco UCS – Energy Efficient BIOS Profile/Policy M 5 Generation Servers with Skylake Processors CSPG UCS Product Management and Data Center Solutions Engineering Released: November 2017 Jeffrey Metcalf – Technical Marketing Engineer

Introduction © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 2

Introduction • Performance-per-watt efficiency improves approximately 20% each tick (microarchitecture) and tock (manufacturing process) release • Maximum server power consumption is steadily increasing, where actual power consumption is based on (1) hardware configuration, (2) BIOS settings, (3) OS power management settings and (4) workload • Cisco Integrated Management Controller (CIMC) and UCS-Manager (UCSM-M) policy based approach assist the data center administrator in these critical management tasks using CLI, UI or XML API with Power. Shell, Python or Ruby SDK © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 3

Primary BIOS Settings Impacting Power Consumption © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 4

Tokens Impacting Power Consumption (1 of 8) • Turbo Mode: Intel® Turbo Boost Technology 2. 0 accelerates processor and graphics performance for peak loads, automatically allowing processor cores to run faster than the rated operating frequency • Availability and frequency upside depends upon a number of factors including: (1) type of workload, (2) number of active cores, (3) estimated current consumption, (4) estimated power consumption and (5) processor temperature • Recommend disabling to minimize peak power consumption © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 5

Tokens Impacting Power Consumption (2 of 8) • Enhanced Intel Speed. Step Technology: Allows the system to dynamically adjust processor voltage and core frequency (referred to as P-states), decreasing average power consumption and heat production • Processor power consumption is a function of capacitance, frequency and voltage where: P = CV 2 f • Recommend enabling to minimize power consumption during periods that require fewer compute cycles, which balances performance and power savings © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 6

Tokens Impacting Power Consumption (3 of 8) • Processor P-state Coordination: There are three modes of P-state resolution defined by ACPI: SW_ANY, SW_ALL and HW_ALL • In SW_ANY mode, the OS Power Manager (OSPM) is responsible for coordinating the P-state among logical processors with dependencies (all Logical Processors in a package) and may initiate the transition on any of those Logical Processors in the domain • Recommend setting to SW_ANY to minimize power consumption during periods that require fewer compute cycles, which balances performance and power savings © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 7

Tokens Impacting Power Consumption (4 of 8) • Power Performance Tuning: Intel processors support Energy Efficiency Policy where the OS can provide hints to the Power Control Unit (PCU). The PCU uses these hints to manipulate the internal behavior of the processor and determine whether to target higher performance or power savings. • If set to BIOS, PCU will only honor the energy policy selected in BIOS • Recommend setting to OS with an OSPM policy of power save to minimize power consumption, where an OSPM policy allows in-band policy changes – a reboot is NOT required to implement a policy change, (e. g. a time-of-day policy) © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 8

Tokens Impacting Power Consumption (5 of 8) • To minimize the idle power consumption of the processor, multiple low power idle states (C-states) are implemented, where the highest state results in the highest power savings • The interface describing the low power states provided on the processor and how they are used by the operating system is described via the ACPI (Advanced Configuration and Power Interface) specification • The move to a multi-core architecture with significant logic shared between cores has resulted in the need to manage idle power consumption at the core and package (shared logic) © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 9

Tokens Impacting Power Consumption (6 of 8) • Processor C 6 Report and C 1 E: Core C-states controlled by the OS directed configuration and OSPM as defined by the ACPI specification • OS tells the threads in a core to go to a particular C-state (using MWAIT(Cx) instruction), and cores (HW) do the thread-level coordination to get each of the threads in the correct core Cx state and then the boot strap processor (BSP) thread takes the core to the correct Cx state • Recommend to enable core C-states to minimize power consumption during idle periods © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 10

Tokens Impacting Power Consumption (7 of 8) • Package C State: A function of the individual C-states that the cores enter, which impacts the state of the mesh, the LLC, the core's power planes, the uncore PLL, and indirectly the state of the integrated peripherals • The deepest sleep state is C 6 Retention, where the package operates at a retention voltage, rather than Pn voltage • Recommend setting to C 6 Retention © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 11

Tokens Impacting Power Consumption (8 of 8) • Additional BIOS settings can reduce server power consumption, e. g. Active State Power Management (ASPM), Link Speeds, Prefetchers, Execute Disable Bit, Corrected Machine Check Interrupt (CMCI), etc. . • However, these BIOS settings could adversely impact typical enterprise workloads – therefore, BIOS settings impacting power consumption are limited to those described earlier © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 12

OS Power Management Policy © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 13

Linux (RHEL) Operating System • RHEL includes separate profiles for specific use cases, together with the tuned-adm tool, that allows easy switching between • server-powersave is a power-saving profile directed at server systems: (1) enables ALPM power saving, (2) disables CD-ROM polling and (3) activates the CPU and disk plugins of tuned • cpufreq_powersave forces the CPU to use the lowest possible clock frequency, that will be statically set. This particular governor offers maximum power savings at the cost of the lowest CPU performance. • RHEL Power Management Guide © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 14

VMware ESXi Hypervisor Operating System • High performance – this power policy maximizes performance, using no power management features • Balanced policy (default in ESXi 6. 0) is designed to reduce host power consumption while having little or no impact on performance • Low power policy is designed to more aggressively reduce host power consumption at the risk of reduced performance • Custom policy starts out the same as Balanced, but allows for the modification of individual parameters • Performance Best Practices for VMware v. Sphere © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 15

Windows and Hyper-V Operating System • Starting with Windows Server 2008, Windows Server provides three power plans: Balanced, High Performance, and Power Saver • Balanced power plan is the default choice that aims to give the best energy efficiency for a set of typical server workloads • Power Saver plan is provides maximize power savings; however, server workload performance may be adversely impacted • Performance Tuning Guidelines for Windows Server © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 16

Configure BIOS – Standalone © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 17

Using CIMC CLI STEP 1: Toggle Navigation and Select Compute STEP 2: Select Processor Tab to Change BIOS Parameters STEP 3: Check Reboot Immediately and Click Save to Immediately Apply BIOS Changes © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 18

Using CIMC CLI Step Command or Action Purpose 1 Server# scope bios Enters BIOS command mode 2 Server /bios# scope processor Enters processor settings command mode 3 Server /bios/processor# show detail Display current processor settings 4 Server /bios/processor# set ? Display all processor object names 5 Server /bios/processor# set object_name ? Display selectable property values for particular processor object 6 Server /bios/processor# set object_name property_value Specify property value to assign particular processor object 7 Server /bios/processor *# commit Commit transaction(s) © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 19

Example Using CLI to Disable Turbo Boost Display Object Name for Each BIOS Token Name Display Current Value, Display Available Values and Change Value from Enabled to Disabled © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 20

Using Python SDK for CIMC XML API (1 of 6) 1. Create an Imc. Handle object and login to the IMC, at the prompt type Request: from imcsdk. imchandle import Imc. Handle handle = Imc. Handle(”ip_address_or_hostname", ”username", ”password") handle. login() Response: True 2. Query the server’s Turbo Boost bios token, determine how many objects where returned Request: bios_turbo_boost_mo = handle. query_classid("Bios. Vf. Intel. Turbo. Boost. Tech") len(bios_turbo_boost_mo) Response: 2 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 21

Using Python SDK for CIMC XML API (2 of 6) 3. Query returned two objects, print variable to display available property values for Turbo Boost Request: for i in range(2): print bios_turbo_boost_mo[i] Response: Managed Object : Bios. Vf. Intel. Turbo. Boost. Tech -------child_action : None dn : sys/rack-unit-1/bios-settings/Intel-Turbo-Boost-Tech rn : Intel-Turbo-Boost-Tech status : None vp_intel_turbo_boost_tech : disabled Managed Object : Bios. Vf. Intel. Turbo. Boost. Tech -------vp_intel_turbo_boost_tech : enabled Available Turbo Boost BIOS Token Property Values Are Disabled and Enabled © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 22

Using Python SDK for CIMC XML API (3 of 6) 4. Query Turbo Boost DN and assign object to variable Request: bios_turbo_boost_token_mo = handle. query_dn(“sys/rack-unit-1/bios-settings/Intel. Turbo-Boost-Tech”) print bios_turbo_boost_token_mo Response: Managed Object : Bios. Vf. Intel. Turbo. Boost. Tech -------child_action : None dn : sys/rack-unit-1/bios-settings/Intel-Turbo-Boost-Tech rn : Intel-Turbo-Boost-Tech status : None vp_intel_turbo_boost_tech : disabled © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 23

Using Python SDK for CIMC XML API (4 of 6) 5. Change value for vp_intel_turbo_boost_tech from disabled to enabled Request: bios_turbo_boost_token_mo. vp_intel_turbo_boost_tech = “enabled” print bios_turbo_boost_token_mo Response: Managed Object : Bios. Vf. Intel. Turbo. Boost. Tech -------child_action : None dn : sys/rack-unit-1/bios-settings/Intel-Turbo-Boost-Tech rn : Intel-Turbo-Boost-Tech status : None vp_intel_turbo_boost_tech : enabled © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 24

Using Python SDK for CIMC XML API (5 of 6) 6. Commit change to IMC, and turn on XML dump feature to see XML request and response Request: handle. set_dump_xml() handle. set_mo(bios_turbo_boost_token_mo) Response: {time_stamp}, 410 - imc - DEBUG - https: //{ip_address}: 443 ====> <config. Conf. Mo cookie=”{cookie}" dn="sys/rack-unit-1/bios-settings/Intel-Turbo-Boost-Tech" in. Hierarchical="false"><in. Config><bios. Vf. Intel. Turbo. Boost. Tech dn="sys/rack-unit-1/biossettings/Intel-Turbo-Boost-Tech" status="modified" vp. Intel. Turbo. Boost. Tech="enabled" /></in. Config></config. Conf. Mo> (time_stamp}- imc - DEBUG - https: //{ip_address}: 443 <==== <config. Conf. Mo dn="sys/rackunit-1/bios-settings/Intel-Turbo-Boost-Tech" cookie=”{cookie}" response="yes"> <out. Config><bios. Vf. Intel. Turbo. Boost. Tech dn="sys/rack-unit-1/bios-settings/Intel-Turbo. Boost-Tech" vp. Intel. Turbo. Boost. Tech="enabled" status="modified" ></bios. Vf. Intel. Turbo. Boost. Tech></out. Config></config. Conf. Mo> © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 25

Using Python SDK for CIMC XML API (6 of 6) 7. Immediate power cycle server to apply committed BIOS change Request: handle. unset_dump_xml() from imcsdk. mometa. compute. Compute. Rack. Unit import Compute. Rack. Unit. Consts server_mo = handle. query_dn(“sys/rack-unit-1”) mo_class = Compute. Rack. Unit. Consts server_mo. admin_power = mo_class. ADMIN_POWER_CYCLE_IMMEDIATE handle. set_mo(server_mo) Response: © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 26

Using BIOS Profile Configuring a BIOS profile helps you to utilize pre-configured token files with the right combination of the token values. Some of the pre-configured profiles that are available are virtualization, high-performance, low power, and so on. You can download a pre -configured profile from the Cisco UCS community website and apply it to the servers. © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 27

BIOS Profile Schema { "name": "{name}" "description": "{description}" "tokens": { "{key}": "{value}" } } Vars: name = (str) unique name assigned to bios profile description = (str) unique description assigned to bios profile key = (dictionary) bios token object name value = (dictionary) property value assigned to bios token object © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 28

Sample Energy Efficient BIOS Profile General purpose energy efficiency BIOS profile { "name": "Energy. Efficiency", "description": "A general purpose energy efficient profile (Energy. Efficiency. json)", "tokens": { "Intel. Turbo. Boost. Tech": "Disabled", "Enhanced. Intel. Speed. Step": "Enabled", "Psd. Coord. Type": "SW ANY", "Pwr. Perf. Tuning": "OS, " "Processor. C 6 Report": "Enabled", "Package. Cstate. Limit": "C 6 Retention", "Processor. C 1 E": "Enabled" } } NOTE: ONLY BIOS Token Values Included in Profile are Evaluated, All Other Tokens Maintain Their Current Value © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 29

Using CIMC CLI (1 of 2) STEP 1: Toggle Navigation and Select Compute STEP 2: Select Configure BIOS Profile © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 30

Using CIMC CLI (2 of 2) STEP 3: Select Upload from Remote Server or Browser Client STEP 4: Select BIOS Profile and Activate NOTE: After BIOS Profile Activation, User Prompted to Reboot Host and Apply Changes © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 31

Using CIMC CLI Step Command or Action Purpose 1 Server# scope bios Enters BIOS command mode 2 Server /bios # scope bios-profile Enters BIOS profile command mode 3 Server /bios-profile # install protocol server_address path Upload BIOS profile, followed by username and password prompt 4 Server /bios-profile # show detail Display validation status 5 Server /bios-profile # activate profile_name Activate BIOS profile, followed by prompt to back up BIOS configuration 6 Prompted to reboot system to apply changes to the BIOS setup parameters Applies changes and initiates immediate host reboot if Enter [y] © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 32

Using Python SDK for CIMC XML API (1 of 6) 1. Using module that provides APIs for BIOS related activities, upload BIOS profile Request: from imcsdk. apis. server import bios_profile_upload(handle, remote_server='ip_address', remote_file='path', protocol='scp', user='username', pwd='password') Response: imcsdk. mometa. upload. Upload. Bios. Profile object at 0 x 1069 f 3 c 50 2. Using module that provides APIs for BIOS related activities, activate BIOS profile Request: bios_profile_activate = (handle, name='Energy. Efficiency', backup_on_activate=True, reboot_on_activate=True) Response: imcsdk. mometa. bios. Bios. Profile object at 0 x 1069 f 32 d 0 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 33

Configure BIOS – Integrated © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 34

Server BIOS Settings • Cisco UCS provides two methods for making global modifications to the BIOS settings on servers in an Cisco UCS domain Create one or more BIOS policies that include a specific grouping of BIOS settings that match the needs of a server or set of servers 2) Use the default BIOS settings for a specific server platform 1) • Cisco UCS abstracts hardware into more than 100 configuration variables, that fully specify each server – stateless computing • BIOS token support varies by processor, e. g. some settings, such as, Turbo Boost are not support across all servers • Recommend verifying BIOS settings are supported © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 35

Verify Supported BIOS Tokens UCSM UI STEP 1: Using Navigation Pane Select Server STEP 2: Using Work Pane Select Inventory Tab and Motherboard Tab STEP 4: Verify BIOS Token(s) Presence STEP 3: Expand BIOS Menu to View BIOS Tokens Passed by UCSM to CIMC © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 36

Verify Supported BIOS Tokens UCSM CLI Step Command or Action Purpose 1 a UCS-A# scope server chassis-id / server-id Enters chassis server mode for the specified blade server 1 b UCS-A# scope server-num Enters server mode for the specified rack-mount server 2 UCS-A /server # scope bios Enter server bios mode 3 UCS-A /server # scope bios-settings Enters server defaults BIOS settings mode for the server 4 UCS-A /server # scope detail Display supported BIOS tokens © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 37

Create/Associate Profile UCSM UI (1 of 2) STEP 1: Using Navigation Pane Select Server and BISO Polices STEP 2: Select Add, Enter BIOS Policy: Name, Description and Reboot Setting, and Select OK © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 38

Create/Associate Profile UCSM UI (2 of 2) STEP 3: Using Navigation Pane Select Service Profile STEP 4: Select Policies Tab and Select Desired BIOS Policy from Available Polices STEP 6: Select Save Changes © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 39

Create/Associate Profile UCSM CLI (1 of 2) Step Command or Action Purpose 1 UCS-A# scope org-name Enters org mode for the specified organization. To enter the default org mode, type / as the org-name 2 UCS-A /org # scope create bios-policy name Creates a BIOS policy with the specified policy name, and enters org BIOS policy mode 3 UCS-A /org/bios-policy # set bios-feature-name bios-parameter-value Configure BIOS settings 3 a UCS-A /org/bios-policy # set description “Energy Efficient BIOS policy” Set BIOS policy description 3 b UCS-A /org/bios-policy* # set reboot-on-update yes—server is rebooted according to the service profile maintenance policy © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 40

Create/Associate Profile UCSM CLI (2 of 2) Step Command or Action Purpose 3 c UCS-A /org/bios-policy* # set processor-c-state -config c-state enabled Enable processor core C-state, which allows server to enter power saving mode during idle periods 3 d Configure remaining BIOS tokens 4 UCS-A /org/bios-policy* # commit-buffer Commits the transaction to the system configuration 5 UCS-A# scope org-name Enters org mode 6 UCS-A /org # scope service-profile sp-name Enters service profile configuration mode for the specified service profile 7 UCS-A /org/service-profile # set biospolicy-name Associates the specified BIOS policy with the service profile 8 UCS-A /org/service-profile* # commit-buffer Commits the transaction © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 41

Create/Associate Profile UCSM Python SDK 1. Create an Ucs. Handle object and login to the UCSM, at the prompt type Request: from ucsmsdk. ucshandle import Ucs. Handle handle = Ucs. Handle("ip_address_or_hostname", ”username", "password", secure=False) handle. login() Response: True 2. Create BOIS policy Request: from ucsmsdk. mometa. bios. Bios. VProfile import Bios. VProfile mo = Bios. VProfile(parent_mo_or_dn='org-root', name='energyefficiency', descr='General purpose BIOS profile for energy efficiency', reboot_on_update='yes') handle. add_mo(mo) handle. commit() © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 42

Create/Associate Profile UCSM Python SDK 3. Define BIOS token values for BIOS policy Request: from ucsmsdk. mometa. bios. Bios. Vf. Energy. Performance. Tuning import Bios. Vf. Energy. Performance. Tuning from ucsmsdk. mometa. bios. Bios. Vf. Enhanced. Intel. Speed. Step. Tech import Bios. Vf. Enhanced. Intel. Speed. Step. Tech from ucsmsdk. mometa. bios. Bios. Vf. Intel. Turbo. Boost. Tech import Bios. Vf. Intel. Turbo. Boost. Tech from ucsmsdk. mometa. bios. Bios. Vf. PSTATECoordination import Bios. Vf. PSTATECoordination from ucsmsdk. mometa. bios. Bios. Vf. Package. CState. Limit import Bios. Vf. Package. CState. Limit from ucsmsdk. mometa. bios. Bios. Vf. Processor. CState import Bios. Vf. Processor. CState from ucsmsdk. mometa. bios. Bios. Vf. Processor. C 1 E import Bios. Vf. Processor. C 1 E from ucsmsdk. mometa. bios. Bios. Vf. Processor. C 6 Report import Bios. Vf. Processor. C 6 Report from ucsmsdk. mometa. bios. Bios. Vf. CPUPower. Management import Bios. Vf. CPUPower. Management © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 43

Create/Associate Profile UCSM Python SDK 3. Define BIOS token values for BIOS policy (continued) mo_1 = Bios. VProfile(parent_mo_or_dn='org-root', name='energyefficiency') mo_2 = Bios. Vf. Energy. Performance. Tuning(parent_mo_or_dn=mo_1, vp_pwr_perf_tuning='os') mo_3 = Bios. Vf. Enhanced. Intel. Speed. Step. Tech(parent_mo_or_dn=mo_1, vp_enhanced_intel_speed_step_tech='enabled') mo_4 = Bios. Vf. Intel. Turbo. Boost. Tech(parent_mo_or_dn=mo_1, vp_intel_turbo_boost_tech='disabled') mo_5 = Bios. Vf. PSTATECoordination(parent_mo_or_dn=mo_1, vp_pstate_coordination='sw-any') mo_6 = Bios. Vf. Package. CState. Limit(parent_mo_or_dn=mo_1, vp_package_c_state_limit='c 6 -retention') mo_7 = Bios. Vf. Processor. CState(parent_mo_or_dn=mo_1, vp_processor_c_state='enabled') mo_8 = Bios. Vf. Processor. C 1 E(parent_mo_or_dn=mo_1, vp_processor_c 1_e='enabled') © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 44

Create/Associate Profile UCSM Python SDK 3. Define BIOS token values for BIOS policy (continued) mo_9 = Bios. Vf. Processor. C 6 Report(parent_mo_or_dn=mo_1, vp_processor_c 6_report='enabled') mo_10 = Bios. Vf. CPUPower. Management(parent_mo_or_dn=mo_1, vp_cpu_power_management='custom') handle. add_mo(mo_1, True) handle. commit() Response: © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 45

Create/Associate Profile UCSM Python SDK 4. Associate BIOS policy previously created Request: from ucsmsdk. mometa. ls. Ls. Server import Ls. Server mo_11 = Ls. Server(parent_mo_or_dn="org-root", name=”service-profile", bios_profile_name="energyefficiency") handle. add_mo(mo_11, True) handle. commit() Response: © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 46