Too Deep Too Wide Using Soft Limits To

Too Deep & Too Wide Using Soft Limits To Avoid Hard Crashes NY/NJ Websphere MQ Family User Group IBM 590 Madison Ave, New York City Tuesday March 25, 2003 NY/NJ MQ Family Users’ Group T. Robert Wyatt, Vice President Bank Of America Differentiated Service Websphere MQ Support 1

Objectives • Demonstrate why the default queue depth and message length values pose a potential risk • Discuss one approach to mitigating that risk March 25, 2003 NY/NJ MQ Family Users’ Group 2

Scenario For Disaster Unrealistic requirements: “Need to be able to queue up an entire day’s worth of large XML messages” – Increase MAXDEPTH – Increase MAXMSGL March 25, 2003 NY/NJ MQ Family Users’ Group 3

Whoops! One day the channel goes down and the system performs exactly as designed: – Queues begin to fill up – Underlying filesystem fills – MQ begins to cut FDC files – MQ and possibly the OS are halted March 25, 2003 NY/NJ MQ Family Users’ Group 4

Recovery • Free up disk space any way possible – Move any diagnostic files – Delete non-essential files – Move older MQ logs if linear March 25, 2003 NY/NJ MQ Family Users’ Group 5

Recovery When (or if) QMgr becomes responsive again: • Stop all channels • RCDMQIMG if linear • Shut down the QMgr March 25, 2003 NY/NJ MQ Family Users’ Group 6

Recovery Restarting the QMgr: Delete files or add space – Clean up the file system – Remount the filesystem to larger disk and copy files Results in extended outage, high probability of client impact March 25, 2003 NY/NJ MQ Family Users’ Group 7

What Happened? 5, 000 x 4 mb messages = 20 GB 5 k x 4 mb persistent msgs > 40 GB Def Dead. Q 640, 000 x 4 mb = 2. 5 TB! When the capacity specified by soft limits exceeds the physical limits, then the soft limits do not exist. March 25, 2003 NY/NJ MQ Family Users’ Group 8

Hard Limits Application constrained only by limits of physical disk space – Failure is sudden – Failure may be catastrophic • Affects OS & co-resident apps – Slow manual recovery – High client impact March 25, 2003 NY/NJ MQ Family Users’ Group 9

Soft Limits Application constrained by MQ – System degrades gracefully – Failure isolated to specific apps within MQ – Recovery is automated – normal operation of MQ restores service – Minimal client impact March 25, 2003 NY/NJ MQ Family Users’ Group 10

Tuning Discipline Insure soft limits < physical limits – Reduce MAXDEPTH, MAXMSGL or both – Reduce settings on SYSTEM. DEF* as well as application queues – Reduce Dead. Q Settings! March 25, 2003 NY/NJ MQ Family Users’ Group 11

Tuning Discipline Disk Space Available* MAXMSGL MAXDEPTH DSA = MAXMSGL x MAXDEPTH = DSA / MAXMSGL = DSA / MAXDEPTH *Use ½ DSA for persistent msgs March 25, 2003 NY/NJ MQ Family Users’ Group 12

Tuning Discipline Know your messages! – Soft limits must be appropriate to the business to be enforceable – Standardize, preferably automate – Verify on each machine – No exceptions Too shallow? Too narrow? Buy more disk! March 25, 2003 NY/NJ MQ Family Users’ Group 13

Apply System Wide System-wide tuning parameters – Channel MAXMSGL – QMgr MAXMSGL – Max uncommitted messages – Log file size and extents – Use separate, preferably dedicated, filesystems for /var/mqm/qmgrs and /var/mqm/logs March 25, 2003 NY/NJ MQ Family Users’ Group 14

Resources IBM Websphere MQ Manuals http: //www-3. ibm. com/software/ts/mqseries/library/manualsa/ – Quick Beginnings Guide – System Administration Manual Support Pacs http: //www-3. ibm. com/software/integration/supportpacs/ – Capacity planning pacs – Performance reports March 25, 2003 NY/NJ MQ Family Users’ Group 15

Resources MQSeries. net: http: //www. mqseries. net Vienna List Server: http: //www-3. ibm. com/software/ts/mqseries/support/usergroups/listserver. html Download presentation: http: //www. t-rob. net/Downloads/2 Deep 2 Wide. ppt T. Rob: March 25, 2003 T. [email protected] net NY/NJ MQ Family Users’ Group 16