32 bit Virtual Memory Constraints in Windows an

32 -bit Virtual Memory Constraints in Windows: an Update Mark Friedman Demand Technology 1020 Eighth Avenue South, Suite 6, Naples, FL 34102 USA phone: (239) 261 -8945 fax: (239) 261 -5456 e-mail: markf@demandtech. com http: //www. demandtech. com ã 2005 Demand Technology Software, Inc.

Outline n Topics ¨ ¨ ¨ Virtual Memory concepts Virtual Memory constraints in 32 -bit Windows /3 GB option Physical Address Extension (PAE) Address Windowing Extensions (AWE) 64 -bit relief ã 2005 Demand Technology, Inc. 32 -bit VM Constraints 2

Intel x 86 3

Intel x 86 in PAE mode ã 2005 Demand Technology, Inc. 32 -bit VM Constraints 4

Virtual Memory Manager n 4 GB virtual per process address space ¨ Lower 2 GB - Private n n n ¨ Page 0 reserved Code pages Heap Upper 2 GB - System n n n System code Shared dlls System cache ã 2005 Demand Technology, Inc. 32 -bit VM Constraints 5

Virtual Memory Manager ¨ ¨ ¨ All address spaces share the virtual addresses from the same upper 2 GB of the System area All virtual addresses in the lower 2 GB User area are unique Shared memory in the System range can be used for IPC ã 2005 Demand Technology, Inc. 32 -bit VM Constraints 6

Virtual Memory map n Paged ¨ ¨ ¨ n ¨ 32 -bit VM Constraints File Server IIS (html, jpg, gif) Page tables; kernel thread stacks non. Paged: ¨ ã 2005 Demand Technology, Inc. Memory mapped files Shared DLLs I/O buffers used by device drivers TCP Session data 7

Virtual Memory Manager n Commit Limit ¨ ¨ A hard limit on the number of Virtual Memory pages that the system will allocate Sizeof (RAM) + paging file(s) Memory allocations start to fail as % Committed Bytes in Use 100% For smaller systems (< 2 GB RAM), Committed Bytes tends to become exhausted prior to running out of VM ã 2005 Demand Technology, Inc. 32 -bit VM Constraints 8

Monitoring Committed Bytes ã 2005 Demand Technology, Inc. 32 -bit VM Constraints 9

Monitoring Process(*)Virtual Bytes ã 2005 Demand Technology, Inc. 32 -bit VM Constraints 10

Virtual Memory Manager ¨ Commit Limit n But page files are extendible! n Should you or shouldn’t you? ã 2005 Demand Technology, Inc. 32 -bit VM Constraints 11

Paging file extension ã 2005 Demand Technology, Inc. 32 -bit VM Constraints 12

System memory pools n n No hard limits on the sizes of the system pools One reserved area of virtual memory can fill up faster than the others; ã 2005 Demand Technology, Inc. Area Max Size file cache 960 MB PTEs 660 MB Paged pool 512 MB non. Paged Pools 256 MB 32 -bit VM Constraints 13

Monitoring the System Pools ã 2005 Demand Technology, Inc. 32 -bit VM Constraints 14

Virtual Memory map n Paged Pool n Cache PTEs non. Paged Pool ã 2005 Demand Technology, Inc. 32 -bit VM Constraints n Virtual memory map is dynamic, subject to adjustment by the OS One reserved area of virtual memory can fill up faster than the others When any one area fills, VMM routines can empty the file cache to acquire more vm for the paged/nonpaged pools 15

Default Paged and Nonpaged pool size can be overridden: ã 2005 Demand Technology, Inc. 32 -bit VM Constraints 16

Default Paged and Nonpaged pool sizes can be overridden: Dynamic adjustment of system virtual memory when one pool is exhausted. n Workloads most vulnerable: n ¨ ¨ n Terminal server with an excess of processes & threads Servers booted with the /3 GB option Non. Paged. Pool. Size, and System. Pages can be set explicitly. ¨ Max pool size is only available from the debugger! ã 2005 Demand Technology, Inc. 32 -bit VM Constraints 17

18

Extended Virtual Addressing n Boot. ini /3 GB switch ¨ ¨ n Physical Address Extension (PAE) ¨ n Virtual storage constraint relief for some applications Squeezes the System VM into 1 GB Supports 37 -bit real addresses on Xeon processors Address Windowing Extensions (AWE) ¨ Permits processes to address real memory > 4 GB ã 2005 Demand Technology, Inc. 32 -bit VM Constraints 19

Virtual Memory Manager n Boot. ini /3 GB switch ¨ Virtual storage constraint relief for some server applications n n n ¨ SQL Server Exchange 2000 Etc. /userva=Size. In. MB subparameter, where Sizein. MB can be any value between 2048 and 3072 ã 2005 Demand Technology, Inc. 32 -bit VM Constraints 20

Exchange 2003 memory tuning n Exchange Server 2003 recommendations /3 GB /Userva = 3030 System. Pages = -1 Heap. De. Commit. Free. Block. Threshold = 0 x 00040000 ms. Exch. ESEParam. Cache. Size. Max = 311296 (1. 2 GB) n See KB #815372 ã 2005 Demand Technology, Inc. 32 -bit VM Constraints 21

Virtual Memory constraints n Try the /3 GB switch ¨ 1 GB system area is not enough virtual memory for some applications n n Possible shortage of Free System Page Table Entries Possible shortage of Nonpaged Pool – ¨ Where Session data from TCP connections is stored Due to fragmentation, it may not be possible to failover a 2 GB+ private address space (e. g. , SQL Server, MS Exchange database – store. exe) using Microsoft Cluster Server (MCS) n During address space recovery on the standby node, the entire virtual memory allocation is acquired at one time ã 2005 Demand Technology, Inc. 32 -bit VM Constraints 22

Physical Address Extension n PAE is supported on most recent Intel servers ¨ ¨ n n New format Page Table Entries (PTEs) are 64 -bits Process address spaces are still limited to 4 GB. How can they exploit 64 GB of RAM? ¨ ¨ ¨ n Physical addresses are 37 -bits Up to 128 GB of RAM can be installed Expand sideways, using multiple address spaces Large Memory Enabled (LME) device drivers Address Windowing Extensions (AWE) Boot. ini /pae switch ¨ Limitation: system addresses can be no higher than 16 GB ã 2005 Demand Technology, Inc. 32 -bit VM Constraints 23

Intel x 86 in PAE mode ã 2005 Demand Technology, Inc. 32 -bit VM Constraints 24

Physical Address Extension n Expand sideways, using multiple address spaces ¨ ã 2005 Demand Technology, Inc. 32 -bit VM Constraints e. g. , MS SQL Server 25

IIS 6. 0 (expand sideways) ã 2005 Demand Technology, Inc. 32 -bit VM Constraints 26

Configuring IIS 6. 0 Application Pools ã 2005 Demand Technology, Inc. 32 -bit VM Constraints 27

Address Windowing Extensions (AWE) n API that allows processes to address real memory locations outside their 4 GB virtual addressing range ¨ ¨ Create and manage memory Overlays Used in conjunction with PAE Allocate. User. Physical. Pages Virtual Alloc Map. User. Physical. Pages ã 2005 Demand Technology, Inc. 32 -bit VM Constraints 28

AWE Note: Frequent unmapping and remapping of Physical Memory blocks is expensive! ã 2005 Demand Technology, Inc. 32 -bit VM Constraints 29

AWE support n MS SQL Server 2000: /3 GB, PAE, & AWE ¨ ¨ ¨ n Oracle: /3 GB, PAE, & AWE ¨ ¨ n Multiple process instances awe enabled = 1 AWE activity counters in the Buffer Manager object AWE_WINDOW_MEMORY Also Set max server Oraclememory SAS: /3 GB, PAE ¨ Work library can be placed in PAE memory ã 2005 Demand Technology, Inc. 32 -bit VM Constraints 30

System Virtual Memory tuning HKLMSYSTEMCurrent. Control. SetControlSession ManagerMemory Management settings Nonpaged. Pool. Si ze Defaults based on the size of RAM, up to 256 MB. Can be set explicitly. -1 extends Nonpaged. Pool. Size to its maximum Paged. Pool. Size Defaults based on the size of RAM, Up to 650 MB. Can be set explicitly; -1 extends Paged. Pool. Size to its maximum System. Pages Defaults based on the size of RAM. Can be set explicitly; -1 extends System. Pages to its maximum Large. System. Cac he 960 MB Zero value allocates minimum sized file cache ã 2005 Demand Technology, Inc. 32 -bit VM Constraints 31

Default Paged and Nonpaged pool size can be overridden: ã 2005 Demand Technology, Inc. 32 -bit VM Constraints 32

Default Paged and Nonpaged pool size can be overridden (Win 2003): n System Pages = x’ffff’ 16 or -1 ¨ ¨ n Paged. Pool. Size = x’ffff’ 16 or -1 ¨ ¨ n Maximizes the number of PTEs that can be built Potentially useful with /PAE or Terminal Services Allows the OS maximum flexibility to determine the size of the Paged Pool Or Paged. Pool. Size and Non. Paged. Pool. Size can be set explicitly Large. System. Cache controls the maximum virtual address range of the file cache ¨ 512 MB – 1 GB ã 2005 Demand Technology, Inc. 32 -bit VM Constraints 33

n !vm Debugger command runs the Page Frame Number (PFN) database Available Pages: 4920 Res. Avail Pages: 358 Locked IO Pages: 251 Free System PTEs: 204387 Free NP PTEs: 28645 Free Special NP: 0 Modified Pages: 596 Modified PF Pages: 660 Non. Paged. Pool Usage: 2750 Non. Paged. Pool Max: 33768 Paged. Pool 0 Usage: 3544 Paged. Pool 1 Usage: 1359 Paged. Pool 2 Usage: 1340 Paged. Pool Usage: 6243 Paged. Pool Maximum: 138240 Shared Commit: 6842 Special Pool: 0 Shared Process: 3688 Paged. Pool Commit: 6398 Driver Commit: 1630 Committed pages: 211846 Commit limit: 320257 ã 2005 Demand Technology, Inc. ( 19680 Kb) ( 1432 Kb) ( 1004 Kb) ( 817548 Kb) ( 114580 Kb) ( 2384 Kb) ( 2640 Kb) ( 11000 Kb) ( 135072 Kb) ( 14176 Kb) ( 5436 Kb) ( 5360 Kb) ( 24972 Kb) ( 552960 Kb) ( 27368 Kb) ( 0 Kb) ( 14752 Kb) ( 25592 Kb) ( 6520 Kb) ( 847384 Kb) ( 1281028 Kb) 32 -bit VM Constraints 34

!poolused Debugger command accounts for all system pool allocations (plus pooltags. txt documentation) lkd> !poolused 2 Sorting by Non. Paged Pool Consumed Pool Used: Tag LSwi NV File Mm. Cm LSwr Devi Thre Pc. Nw Irp Non. Paged Allocs Used 1 2576384 287 1379120 2983 504920 16 435248 128 406528 267 377472 452 296512 12 278880 669 222304 ã 2005 Demand Technology, Inc. Paged Allocs Used 0 0 14 55272 0 0 0 0 32 -bit VM Constraints 35

Poolmon utility (from DDK) ã 2005 Demand Technology, Inc. 32 -bit VM Constraints 36

64 -bit Virtual Memory Relief ã 2005 Demand Technology, Inc. 32 -bit VM Constraints 37

64 -bit Relief n Windows supports two flavors of 64 -bit hardware ¨ ¨ IA-64 (Itanium) AMD 64 and Intel x 64 (aka EMT) n n ã 2005 Demand Technology, Inc. Runs 32 -bit applications with almost no performance penalty 32 -bit applications can allocate their entire 4 GB virtual memory range 32 -bit VM Constraints 38

Questions ? ã 2005 Demand Technology, Inc. 32 -bit VM Constraints 39