IP Platforms Best Practices for Performance Pierre Lamy

Intro and revision history This document describes methods and techniques that users can apply on various Check Point IP Security Appliances to achieve optimal performance. Ø Version 1. 0 October 2009 Word format Ø Version 1. 1 January 2010 Word + PPT, minor revisions Ø Version 1. 2 April 2010 Updates © 2010 Check Point Software Technologies Ltd. | [Unrestricted] For everyone | 2 2

General Performance Best Practices n These guidelines are Appliance independent and do not require any special tuning. n Always use the latest versions of Check Point products. Always upgrade to the most recent HFA (Hot. Fix Accumulator) for a given version. n Create a small block of rules near the top of your rulebase, containing the most heavily used rules. These rules should be fully accelerated with Secure. XL. n Keep the rulebase simple and small. Reduce the number of rules by combining similar rules together. Rules which disable Secure. XL acceleration should be placed very low in the rulebase. n If not using VPN (encryption) on the module, make sure the VPN product is disabled on that module. © 2010 Check Point Software Technologies Ltd. | [Unrestricted] For everyone | 3 3

General Performance Best Practices n Do not use Qo. S from IPSO or Floodgate. n Avoid using Domain Objects. DNS lookup takes additional CPU cycles. n Avoid using UFP URI Filtering Protocol as this is resource intensive. n Use Networks instead of address ranges for Network Address Translation. n Keep logging to a minimum. Only business critical rules which will be analyzed should have logging enabled. Drop rules, Accept rules, Stealth rules, Cleanup rules and Implied rules should not log unless there is a clear business case and the customer intends on analyzing the logs on a regular basis. Otherwise logging should only be used for debug purposes. n IP Cluster members should have exactly the same package lists, having dissimilar packages can cause state sync issues resulting in a performance reduction. © 2010 Check Point Software Technologies Ltd. | [Unrestricted] For everyone | 4 4

General Performance Best Practices n When you install an ADP interface module in an appliance, the network processor in the card performs all VPN encryption and decryption, even for VPN packets that ingress or egress through non-ADP interfaces. The built-in Nokia encryption accelerator continues to accelerate IKE traffic but does not perform any other processing. n If VPN traffic ingresses or egresses through a non-ADP interface, throughput is negatively affected because the packets must transit the appliance’s backplane to reach the network processor in the ADP module. It is recommended that one configure VPNs to use only ADP interfaces to avoid this performance loss. © 2010 Check Point Software Technologies Ltd. | [Unrestricted] For everyone | 5 5

General Performance Best Practices n Uniprocessor systems (IP 152, IP 292, IP 395, IP 565, IP 1265) should use IPSO 4. 2 (this is correct in April 2010) while Multiprocessor systems (IP 695, IP 1285, IP 2455) should use IPSO 6. 2. The latest build of any major release should always be used. n Multiprocessor systems should have Check Point R 70 installed to take full advantage of Core. XL technology. sk 40465 has some more details about this. n Use R 70 + IPSO 6. 2 on uniprocessor systems where there is a need for a specific feature n Do not persist in the use of IPSO 4. 2 or R 65 once support is no longer offered for these products. © 2010 Check Point Software Technologies Ltd. | [Unrestricted] For everyone | 6 6

General Performance Best Practices n Using interface flow control can reduce network throughput on busy interfaces and we do not suggest it be enabled. n Avoid using Smart. View Monitor to constantly monitor system performance or collect historical data, as Smart. View Monitor itself has an impact on performance. n Avoid using custom scripts on systems which have performance issues, as the scripts will incur CPU resources. © 2010 Check Point Software Technologies Ltd. | [Unrestricted] For everyone | 7 7

General Performance Best Practices n A CST from the IPSO system as well as a cpinfo from the management station, are CRITICAL to provide to Check Point Support when opening a case for assistance in troubleshooting performance issues. Without at least those files, Check Point Support will be unable to assist the customer. For systems with extremely high CPU, or where running CST may cause problems, it is recommended to run it with the following syntax: “nice +20 cst”. The command may take hours to complete but will not divert critical system resources from processing traffic in a live environment. n Any cpinfo that is provided to Check Point Support MUST be generated using the latest cpinfo tool downloaded from the Support site. This requires uninstalling the old cpinfo, and then installing the latest one. Providing old cpinfo output to Check Point Support will delay Support response. © 2010 Check Point Software Technologies Ltd. | [Unrestricted] For everyone | 8 8

General Performance Best Practices n Avoid using Standalone installations. Separate the management station from the enforcement point by running the management station on another system. n Use the default settings in the capacity optimization tab of the enforcement point’s properties, changing only the total connections number. n General recommendations for these platforms are to use onboard quad ports for: n Security Gateway State Synchronization traffic n Cluster protocol network traffic n Policy and Appliance management traffic n A path from the enforcement point to the Check Point Log server n Secure. XL options should be matched between SXL and the Security Gateway settings; for example Sequence Validation and Delayed Notifications. © 2010 Check Point Software Technologies Ltd. | [Unrestricted] For everyone | 9 9

System Resources n High performance relies on the availability of key system resources: CPU, memory, and network interface bandwidth. n Tuning involves better using the current hardware, not simply upgrading. © 2010 Check Point Software Technologies Ltd. | [Unrestricted] For everyone | 10 10

System Resources - CPU n Small packet size traffic: The amount of traffic any network device can process is not determined by byte throughput numbers, but rather by the packets per second. A small packet uses as many resources as a large full size ethernet packet. n CPU utilization is incurred on a per-packet level, rather than per-byte. Therefore it is critical to note that a system that is processing a large number of small packets, works as hard as a system processing the same number of large packets. There may be a very large difference in the apparent byte throughput between the systems. n State synchronization particularly demands high CPU cycles because the nodes in a cluster perform synchronization for every connection they encounter. This ensures high availability but causes high CPU usage. This is especially important to consider, when deciding whether or not to synchronize short lived connection types like DNS when using a VRRP pair – the failover time of VRRP exceeds the DNS timeout. It would not be advisable to sync DNS connections in that scenario. © 2010 Check Point Software Technologies Ltd. | [Unrestricted] For everyone | 11 11

System Resources - CPU n High number of logging rules affects CPU. Logging uses CPU cycles and is discouraged where there is no need. n The Active Log feature in Smart. View Tracker will severely compromise the ability of an enforcement point to process traffic, and for performance reasons should not be used. n Accounting Logging: By default, accounting logging produces two kinds of log tracking (one in Log Viewer and one in Account Log View) for the same connection. Alerts as a tracking option, also have a significant negative impact on performance. © 2010 Check Point Software Technologies Ltd. | [Unrestricted] For everyone | 12 12

System Resources - CPU n Any configuration which disables Secure. XL or forces traffic to use Slowpath affects CPU. This includes rule configurations, Smart. Defense protections, and Floodgate. n NAT traffic incurs slightly more CPU impact that non-NATed traffic. NAT in an ADP environment is strongly discouraged as connection establishment rate acceleration does not work on NAT traffic. n Traffic which is not connection-rate accelerated, uses more CPU resources than traffic which is. Connection establishment and teardown uses (relatively) a lot of CPU resources, even when all Secure. XL acceleration is in use. © 2010 Check Point Software Technologies Ltd. | [Unrestricted] For everyone | 13 13

System Resources – Memory n Check Point recommends upgrading any Appliance’s memory to its full capacity to improve performance. n The main factors that demand high RAM usage are: n Concurrent connections n Concurrent VPN tunnels n NAT connections n Security Servers n Use the Web User Interface to determine how much memory the Appliance has installed. In the Web User Interface navigation tree, select Monitor --> System Utilization --> CPUMemory Live Utilization. Look for the “Total Real Memory” value. The top command can also be used on recent IPSO versions to view the information on the command line. n The amount of memory, allocated to the Check Point Security Gateway to process network traffic, is determined under the capacity optimization tab of the gateway object properties. The Automatic values should always be used, and the manual setting of memory allocation should NEVER be used unless directed to by Check Point Support. n You should always set the capacity optimization to the maximum values (connections) supported by the platform and memory configuration as detailed in the release notes for IPSO. There is no drawback to doing this, but leaving the low defaults in place will results in insufficient memory on busy systems. © 2010 Check Point Software Technologies Ltd. | [Unrestricted] For everyone | 14 14

System Resources – Memory n Two factors can reduce the number of Security Gateway connections that can be supported Ø Concurrent VPN tunnels are dependent upon the amount of memory available in the Appliance. As you add more VPN tunnels, the number of Security Gateway connections an Appliance can support will decrease. Ø Security Servers will reduce the maximum number of supported connections as they write to temporary files and use 8 entries in the connections table. © 2010 Check Point Software Technologies Ltd. | [Unrestricted] For everyone | 15 15

System Resources - Network Interface Bandwidth n When the Security Gateway performance reaches the limitation of the interface bandwidth and the CPU is still not fully utilized then the bottleneck is the interface. One option to increase performance in this case is to use more ports via Link Aggregation to achieve the maximum performance. n The limitation of the network interface is determined by the amount of packets per second it can process. Assuming 1518 byte frames, a 100 megabit NIC port can sustain ~8234 packets per second (pps ) in each direction (full duplex). A 1 gb NIC port can sustain ~82340 pps in each direction (full duplex). n While the network port may sustain much higher numbers of pps than these – and this is often seen in the field and QA – there is no guarantee that it WILL support more than these standard pps numbers for a given link speed. © 2010 Check Point Software Technologies Ltd. | [Unrestricted] For everyone | 16 16

System Resources - Network Interface Bandwidth n If the number of packets per second exceeds those numbers, using Link Aggregation to combine 2 or more links together is possible, thus increasing the amount of bandwidth in pps that can be processed on that logical network interface. n IPSO Sync should not need more than a 1 gb interface, and there can be problems when IPSO Sync is run over a Link Aggregation group. The speed of the IPSO and Check Point sync interface should be as fast as the fastest NIC port on the system. In a 10 gb environment, typically 1 gb for sync is sufficient. State sync should always be on an isolated VLAN or network segment. Note that it is NOT supported to run Check Point state sync over any interface VLAN other than 0 / 1 / untagged. Any VLAN’ing done on the switch access port is fine, but trunking is not supported as Check Point state sync is not designed to support the extra frame sizes. n Note: Security Gateway state sync link aggregated interfaces should be directly connected using cross-over cables, unless you are using 3 or more cluster members. IPSO and Check Point state sync should always be on an isolated VLAN or network segment. © 2010 Check Point Software Technologies Ltd. | [Unrestricted] For everyone | 17 17

IP Clustering n IP Clustering provides both high availability and scalability. IP Clustering is useful when the performance of one system alone is insufficient to provide the desired level of performance. For example, when an Appliance CPU reaches ~30%, it would be recommended to add another Appliance to form a two-member cluster that can scale the Security Gateway performance. This is a capacity planning exercise that Check Point Sales engineers can help with. The 30% number is considered an industry standard measurement or indicator that suggests more capacity should be added. n IP Clustering is especially beneficial when using Smart. Defense features. With all Smart. Defense features enabled, a twomember cluster’s HTTP transaction rate is about 40% higher than a Standalone Appliance. © 2010 Check Point Software Technologies Ltd. | [Unrestricted] For everyone | 18 18

IP Clustering n Use dedicated interfaces for cluster protocol networks and state synchronization; do not share interfaces with the production traffic. n It is strongly recommended to use separate interfaces for cluster protocol network and Security Gateway synchronization traffic so that they are separate Broadcast domains. n Use a bandwidth of at least 100 Mbps full duplex for IPSO sync interface(s). 1 gb is recommended. n Use switches, not hubs, and never use crossover cables for IP Clustering protocol networks. © 2010 Check Point Software Technologies Ltd. | [Unrestricted] For everyone | 19 19

IP Clustering n Do not use IP Clustering Forwarding mode, when performance is a concern. Unicast and Multicast provide better performance and less latency. Forwarding mode is a fallback mode, for when feature-poor network switches are in use. n If IGMP snooping is in use on the switch, disable it or configure static CAMs in order to allow Multicast traffic on specific ports. n Use dynamic cluster work assignment for optimum load balancing. This allows the cluster to periodically rebalance the load by moving active connections between nodes. n Use delayed synchronization if your system processes many short-lived connections, you are in VRRP or Standalone, and SXL templates are in use. A 30 second delay in synchronizing connections can boost the performance by about 20%. If you use Check Point delayed notifications, you must also enable Secure. XL delayed notifications. © 2010 Check Point Software Technologies Ltd. | [Unrestricted] For everyone | 20 20

ADP n Addition of ADP will increase performance of the appliance with some limitations explained below. The decision to purchase ADP add on cards should be made in consultation with Check Point Sales. ADP should be considered if the performance improvement desired falls in to one of the following categories: Ø Ø Ø Ø Packet throughput performance, specifically for small packets. Performance improvements for packet streams with mixed packet sizes. Encrypted traffic (VPN) forwarding. Long-lived connections performance. ► For example data transfer rates for protocols like ftp, http etc. NAT Performance only for long-lived NAT connections. (ADP accelerates NAT throughput. Connection-rate Acceleration is not currently supported for NAT connections. XMC cards should be used instead for high NAT & CPS. ) Latency for both un-encrypted and encrypted traffic. Multicast throughput performance. © 2010 Check Point Software Technologies Ltd. | [Unrestricted] For everyone | 21 21

ADP n Performance issues with mixing ADP with non-ADP interfaces: n The best performance one can get is by not mixing traffic between ADP and non-ADP interfaces. Running in mixed mode will have performance impacts. When run in dual-mode – having separate ADP traffic flows and separate non-ADP flows one can see the Appliance performance scale as opposed to not using any ADP interfaces at all. © 2010 Check Point Software Technologies Ltd. | [Unrestricted] For everyone | 22 22

ADP Benefits n Throughput Acceleration: n The first packet in a connection is sent up the stack to the Security Gateway that validated the packet based on the defined rule base. Once validated the Security Gateway application tells IPSO via the Secure. XL API to handle future packets in the same connection. IPSO then instructs the ADP sub-system to create a bi-directional flow for that connection. All future packets for that connection will now be processed by the ADP sub-system. n The following protocols benefit from Secure. XL & ADP Throughput acceleration: Ø TCP, UDP, & traffic carried over those protocols Ø IPSec VPN acceleration Ø Multicast forwarding Ø PIM (from IPSO 3. 9 for IP 2250 & IP 2255; from IPSO 4. 2 for all platforms) Ø GRE & ESP © 2010 Check Point Software Technologies Ltd. | [Unrestricted] For everyone | 23 23

ADP Benefits n Connection Rate Acceleration n The first packet in a connection is validated by the Security Gateway application. Once validated, the Security Gateway instructs IPSO to create a template so that IPSO can validate future connections where only the Source Port differs. A template consists of the following attributes: Src. Addr, Src. Port, Proto, Dest. Addr, & Dest-Port. IPSO compares the first packet in the next connection to its template table. If the packet matches a template then IPSO adds the connection to its table, then instructs ADP to create a bidirectional flow for the connection and lastly informs the Security Gateway about the new connection. All future packets are processed by the ADP module. n The following protocols benefit from Secure. XL & ADP Connection-rate acceleration: Ø Unencrypted TCP, UDP, & traffic carried over those protocols Ø Particularly effective on HTTP 1. 1 traffic Ø Even more effective on HTTP 1. 0 traffic Ø HTTP 1. 0: Separate connection for each HTTP component © 2010 Check Point Software Technologies Ltd. | [Unrestricted] For everyone | 24 24

ADP Best Practices n Configure traffic to flow in/out of the same ADP subsystem, since traversing to another ADP subsystem or worse to a non-ADP interface will negatively impact throughput performance. There is a 10 gb full duplex bandwidth over the crossbar between 2 ADP subsystems. n Do not use ports connected to the ADP subsystems for cluster protocol network or for Security Gateway state synchronization. Use onboard ports for Security Gateway state synchronization; this will guarantee that the synchronization data goes to its own channel and will avoid sync packets being lost. This also prevents the sync data from disrupting the data passing between the ADPs and the main CPU. Likewise do not log to the management station via the ADP interfaces. © 2010 Check Point Software Technologies Ltd. | [Unrestricted] For everyone | 25 25

ADP Best Practices n Note that the backplane connecting the ADP subsystems to the main CPU has limited bandwidth, and this bottleneck will impact throughput performance when there is a lot of non-accelerated traffic. n Do not combine non-ADP ports and ADP ports in a link aggregation group or redundancy group. n Do not include interfaces on different ADP I/O cards in the same link aggregation or redundancy group. IP Security Appliances do not support cross ADP link aggregation. n Secure. XL is enabled by default. Note that it’s critical not to disable Secure. XL because Secure. XL is required for ADPs to function. © 2010 Check Point Software Technologies Ltd. | [Unrestricted] For everyone | 26 26

ADP Best Practices n Avoid performing tcpdump or fw monitor on ADP interfaces when the interfaces are under heavy load. Performing a tcpdump or fw monitor on an ADP interface forces all traffic received or transmitted by the ADP subsystem, to be copied and piped to IPSO through the backplane. Secure. XL will still be used, and the ADP will still accelerate, but the backplane will be choked with data. This causes a significant degradation in performance due to constricted backplane capacity. © 2010 Check Point Software Technologies Ltd. | [Unrestricted] For everyone | 27 27

Limitations of ADP n Traffic that is not throughput or connection rate accelerated will not benefit from ADP acceleration. All limitations of Secure. XL apply to ADP. n Transparent mode will accelerate traffic normally, however a special design consideration is that there must be routes pointing out of the xmode interfaces, as Secure. XL depends on caching route table lookups. © 2010 Check Point Software Technologies Ltd. | [Unrestricted] For everyone | 28 28

Limitations of ADP n Enabling Sequence verifier: this solution requires enabling sequence verifier option on IPSO as well as in Smart. Dashboard. n This solution was suggested after analyzing the CST, where it was observed that most (60 million out of 80 million) of the TCP connections were getting closed with RSTs instead of the usual 3 -way handshake for terminating TCP connections. As part of the stateful inspection, the Security Gateway needs to monitor all TCP RSTs if sequence verifier is not turned ON, as they are categorized as untrusted RSTs. This behavior of terminating TCP connections causes additional load on ADP backplane interfaces, where the packet drops were observed. n By turning ON sequence verifier ADP will perform sequence verification on all TCP connections thereby validating even RSTs that are used to terminate the TCP connections. Once the TCP RST is validated by ADP and is accepted there is no need to send this packet to the Security Gateway, thereby reducing the backplane traffic and the overhead of the Security Gateway having to inspect these packets. © 2010 Check Point Software Technologies Ltd. | [Unrestricted] For everyone | 29 29

Limitations of ADP n Once the sequence verification is turned ON, you should see significant reduction in packets going over the backplane to the Security Gateway. This can be monitored by executing the following command n n ipsctl –i net: dev: adp: if: stats with an “r” option to monitor the rate of tcp_rst As a result of the reduction in TCP RSTs going over the backplane, we should observe less drops of data packets on backplane which can be monitored by executing the following command. n ipsctl –i net: dev: bp: msg: stats with an “r” option to monitor the rate of rx_fc_drops © 2010 Check Point Software Technologies Ltd. | [Unrestricted] For everyone | 30 30

Limitations of ADP n Too many control messages queued up on eth 1, will result in data loss on the data channels eth 2 -4. The queue depth for the control channel is tunable, the default is 64 in IPSO 6. 2. Potential values are 128 and 256: n n You should see a decrease in the rate of dropped packets on data channel. This can be monitored by executing the following command: n n ipsctl -w net: dev: bp: msg: delay_drop_limit 128 ipsctl –i net: dev: bp: msg: stats with an “r” option to monitor the rate of rx_fc_drops If this solution does not yield the desired result then the delay_drop_limit can be easily set back to its default value of 64. Setting of these ipsctl variables takes effect immediately and is non-intrusive. © 2010 Check Point Software Technologies Ltd. | [Unrestricted] For everyone | 31 31

Limitations of ADP n Turning off the delay_drop variable: this solution requires changing the default value of delay_drop, an ipsctl tunable in IPSO pro-actively drops data packets, when control channel is congested. This option can be completely turned off by executing the following command n ipsctl –w net: dev: bp: msg: delay_drop 0 n The option of dropping data packets when control channel is congested was developed under certain performance benchmarking conditions, where the box is tested for limits and the aggressive load conditions persist for an extended period of time. This is the reason for aggressive drop_delay_limit to 64 by default. n Unfortunately, this condition also comes into effect when there is transient congestion on the control channel. By turning off this feature, we do not drop the data packets pre-maturely. n Current congestion level and the max congestion level on the control channel can be monitored by executing the following command: n n ipsctl –i net: dev: bp: msg: stats with an “r” option to monitor the rate of bms_scheds and bms_scheds_max If this solution does not yield the expected result, then you can revert back to default behavior immediately by executing the following command. n ipsctl –w net: dev: bp: msg: delay_drop 1 © 2010 Check Point Software Technologies Ltd. | [Unrestricted] For everyone | 32 32

Limitations of ADP The PSL acceleration feature should be enabled on multicore systems using Secure. XL and Core. XL, with or without ADP. The ipsctl tunable can be found using ipsctl –a net: sxl PSL acceleration allows full acceleration of all but the last packet containing the application level Protocol Data Unit IPS / Smart. Defense takes care of a go/no-go to drop denied connections. © 2010 Check Point Software Technologies Ltd. | [Unrestricted] For everyone | 33 33

Security Gateway Performance Tuning n NAT n IP Appliances do not support Network Address Translation (NAT) connection acceleration. The first packets of the first connection on the same service are forwarded to the Security Gateway application. Then a “template” of that connection is created so that subsequent TCP establishments on the same service, where only the source port is different, will be accelerated by Secure. XL. NAT connections setup and teardown cannot be accelerated because NAT templates are not supported. n While each connection uses two entries in the flows table, connections involving NAT use four entries. NAT connections use more CPU and memory resources compared to ordinary connections. © 2010 Check Point Software Technologies Ltd. | [Unrestricted] For everyone | 34 34

Security Gateway Performance Tuning n Rulebase Size n Although there is no limit to the number of rules in a Security Gateway database, there is a performance impact as the number of rules grows. The more rules an Appliance has, the more it will cost the Appliance in compilation time and runtime efficiency. n Rulebase size affects connections rate performance. n Rulebase order is important and can affect performance. Use the following guidelines for organizing the rulebase: Ø The rulebase should be as simple as possible. With fewer rules the rulebase will be more efficient and less error prone. Ø When creating a rule, be specific. Narrow down the source, destination, and service. Ø Avoid using “Any” in the service field. Ø The most active NAT rules should be at the top of the NAT rulebase. Ø Defining Group Objects for networks allows the policy compiler to superset traffic in the actual rule for a performance gain. Ø Anti spoofing should be configured for all the Security Gateway interfaces. n Avoid using “negate” in the rulebase (For example, a network exclusion) © 2010 Check Point Software Technologies Ltd. | [Unrestricted] For everyone | 35 35

Performance Troubleshooting n Follow the guidelines to troubleshoot performance issues and use the best practices outlined above to optimize the Appliance overall performance. n Do NOT use fw monitor for performance troubleshooting. Connecting a span port via a switch, is preferred to tcpdump. Traffic captures sent to Check Point Support should not exceed 80 mb uncompressed. Particular lines within a packet capture should be referenced by the customer as needed. n Check free disk space using the “df –k” command line tool. © 2010 Check Point Software Technologies Ltd. | [Unrestricted] For everyone | 36 36

Performance Troubleshooting n Check currently used CPU statistics using the “vmstat 1” tool. The last 3 columns are significant, customers should never concern themselves with the other columns. The very last column is CPU Idle time, this is the amount of free CPU cycles, in percentage, since the last vmstat iteration. The second-last column is System CPU usage in percentage, this includes IPSO and Check Point kernels as well as interrupts. The third-from-last column is User CPU utilization, this is usually due to Policy Installation, Smart. Defense, Security Servers, and user scripts or commands. © 2010 Check Point Software Technologies Ltd. | [Unrestricted] For everyone | 37 37

Performance Troubleshooting n The “top” command line utility is used on recent IPSO versions, this utility will provide per-process CPU and memory utilization, as well as global statistics, and more granular CPU statistics such as % of interrupts. The percentage of interrupts includes both software interrupts and hardware interrupts. Hardware interrupts are virtually never the cause of performance issues, performance issues are virtually always caused by software interrupts. n If it can be shown that the performance problem is due to a high ratio of interrupts compared to overall CPU utilization, the fix is always first to properly tune Secure. XL. © 2010 Check Point Software Technologies Ltd. | [Unrestricted] For everyone | 38 38

Performance Troubleshooting n Secure. XL Acceleration statistics can be verified using the following commands: Ø Ø n fwaccel stats –s fwaccel templates –s These commands should provide a good overview of how much Secure. XL is in use. The Secure. XL and Nokia IPSO Guide (http: //downloads. checkpoint. com/dc/download. htm? ID=10036) should be used to help tune the rulebase, and ensure that as many connections and packets are accelerated as possible. © 2010 Check Point Software Technologies Ltd. | [Unrestricted] For everyone | 39 39

Performance Troubleshooting n If dropped packets are a concern, they can be checked as a snapshot in IPSO by running the command “ipsctl –a ifphys | grep qdrop”. This will provide an indication of which interfaces are dropping traffic. Please note that these counters are incremental since system boot time. For more information about qdrops, consult sk 39462. n To view realtime statistics for a particular counter, run ipsctl –i <full-counterpath> use the “r” command to toggle the rate per second counter refresh. n Any qdrop which is logged will have a corresponding reason code incremented, in n ifphys: <interface_name>: errors n ifphys: <interface_name>: stats n Common drops are rx_mpc, which represents the Receive, Missed Packet Count, and symerrs, Symbol Errors. © 2010 Check Point Software Technologies Ltd. | [Unrestricted] For everyone | 40 40

Performance Troubleshooting n rx_mpc is due to the operating system not being able to flush the receive buffer for the interface fast enough. The receive buffer queues up incoming data which has been received on the physical media, and is flushed from the queue every time there is an interrupt. The interrupt is triggered under 2 conditions – when the rx_ring has reached ¼ full, or after a timer is hit. More information about these processed is detailed in sk 39176. There advanced tunable variables that can be used under direction of Check Point Support to influence this behavior. n Symbol Errors are due to a bad fiber network cable, dirty or dusty NIC port or fiber connectors. It can also be due to a bad NIC card or switch port but this is very uncommon. Symbol errors only increment for received data for the local side, not sent data. More information about symerrs is detailed in sk 39733. © 2010 Check Point Software Technologies Ltd. | [Unrestricted] For everyone | 41 41

Performance Troubleshooting n n A lot of short lived connections transiting the enforcement point can cause slowdowns, as connection establishment and teardown incurs CPU utilization. This can be partly mitigated, by ensuring templates can be created for the most heavily used rules. You may also be able to use the Fast Expire Secure. XL feature. Fast Expire should be used primarily for short lived connections such as DNS. © 2010 Check Point Software Technologies Ltd. | [Unrestricted] For everyone | 42 42

Performance Troubleshooting n Use ifconfig to verify that no interfaces have the PROMISC flag set. An interface in promiscuous mode forwards all frames seen on the physical media, to the operating system for Layer 2 filtering. An interface in nonpromiscuous mode, uses a MAC chip to filter Layer 2 frames to ensure that only frames which are destined for the local machine are passed to the operating system. This is determined based on Unicast/Broadcast/Multicast MAC Address lists in the Receive Address High and Low registers. PROMISC is set for Transparent Mode and this is normal behavior. n Customers may wish to view the Security Gateway connection tables in human readable format to help with rules optimization. Check Point Support has internal-only tools to read the output of “fw tab –u”. Customers may be interested in an unsupported third-party script which can be found at http: //www. fw-1. de/aerasec/download/fw 1 -tool. pl n Check Point makes no guarantees about this product, and provides this information as reference only. © 2010 Check Point Software Technologies Ltd. | [Unrestricted] For everyone | 43 43

Advanced debugging and tuning n Advanced debugging and tuning should only be carried out under the direction of the Escalations group, or Development. © 2010 Check Point Software Technologies Ltd. | [Unrestricted] For everyone | 44 44