Lassen Performance Tuning

We need to apply the performance tuning parameters on Tomcat and HTTP server.

Tomcat Server Tuning

Jasper Configuration

Tomcat JSP compiler Jasper has some recommended configuration for production environment or rather non-development environment. They are described below and are made to web.xml

http://tomcat.apache.org/tomcat-6.0-doc/jasper-howto.html#Production%20Configuration

<servlet>
<servlet-name>jsp</servlet-name>
<servlet-class>org.apache.jasper.servlet.JspServlet</servlet-class>
<init-param>
<param-name>fork</param-name>
<param-value>false</param-value>
</init-param>
<init-param>
<param-name>xpoweredBy</param-name>
<param-value>false</param-value>
</init-param>

<!-- development Is Jasper used in development mode? If true, ---->
<!-- the frequency at which JSPs are checked for ---->
<!-- modification may be specified via the ---->
<!-- modificationTestInterval parameter. [true] ---->
<!-- -->

<init-param> <param-name>development</param-name> <param-value>false</param-value> </init-param>

<!-- mappedfile Should we generate static content with one ---->
<!-- print statement per input line, to ease ---->
<!-- debugging? [true] ---->
<!-- -->

<init-param>
<param-name>mappedfile</param-name>
<param-value>false</param-value>
</init-param>

<!-- modificationTestInterval ---->
<!-- Causes a JSP (and its dependent files) to not ---->
<!-- be checked for modification during the ---->
<!-- specified time interval (in seconds) from the ---->
<!-- last time the JSP was checked for ---->
<!-- modification. A value of 0 will cause the JSP ---->
<!-- to be checked on every access. ---->
<!-- Used in development mode only. [4] ---->
<!-- -->

<init-param>
<param-name>modificationTestInterval</param-name>
<param-value>3600</param-value>
</init-param>

<!-- genStrAsCharArray Should text strings be generated as char ---->
<!-- arrays, to improve performance in some cases? ---->
<!-- [false] ---->
<!-- -->

<init-param>
<param-name>genStringAsCharArray</param-name>
<param-value>true</param-value>
</init-param>

<load-on-startup>3</load-on-startup> </servlet>

Server Configuration

The following changes were made regarding the tomcat server made to server.xml

Reference: http://tomcat.apache.org/tomcat-6.0-doc/config/http.html

<Connector port="8080" protocol="HTTP/1.1"
enableLookups="false"
maxThreads="500"
minSpareThreads="100"
maxKeepAliveRequests="1"
acceptCount="200"
connectionTimeout="20000"
redirectPort="8443" />

maxThreads

It is the maximum number of request processing threads to be created by this Connector, which therefore determines the maximum number of simultaneous requests that can be handled. The number were set to 500 as there were requests rejected giving connection refused error when load applied on the server was about 400 user. There is thinktime applied in the test which should reduce the concurrency in the server but at certain point of time, there were few requests kept alive if the keep-alive timeout is too high.

maxThreads were increased to 1200, to match the number of users/node. This wasn't necessary but to solve the 40 min problem so that the connections are not refused by tomcat. Since maxThreads were increased, minSpareThreads were also increased to 500.

maxKeepAliveRequests

The maximum number of HTTP requests which can be pipelined until the connection is closed by the server. Setting this attribute to 1 will disable HTTP/1.0 keep-alive, as well as HTTP/1.1 keep-alive and pipelining. Setting this to -1 will allow an unlimited amount of pipelined or keep-alive HTTP requests. If not specified, this attribute is set to 100. There are redirects for each request so its good to have keep alive so that each redirected request is handled by the same thread.

Keep Alive on the tomcat server has been disabled, which probably solves the 502 Bad proxy gateway error

acceptCount

The maximum queue length for incoming connection requests when all possible request processing threads are in use. Any requests received when the queue is full will be refused. The default value is 10. It is set high so that we dont have any requests that is getting refused by the server.

MySQL Server tuning

The limit on connections made to MySQL server is specified by the max_client parameter. The default value is 100, so if we have connection pool size of 100 on the app servers, the limit on mysql server throws "com.mysql.jdbc.exceptions.MySQLNonTransientConnectionException: Too many connections".

The max connections were set to 800 which were eventually increased to 3300 (200 per node and 100 extra)

Load Balancer Tuning

The number of connections were increased from the default to handle the load.

<IfModule worker.c>
ServerLimit 50

#initial number of server processes to start
#http://httpd.apache.org/docs/2.2/mod/mpm_common.html#startservers
StartServers 3

#minimum number of worker threads which are kept spare
#http://httpd.apache.org/docs/2.2/mod/mpm_common.html#minsparethreads
MinSpareThreads 5000

#maximum number of worker threads which are kept spare
#http://httpd.apache.org/docs/2.2/mod/mpm_common.html#maxsparethreads
MaxSpareThreads 5000

#upper limit on the configurable number of threads per child process
#http://httpd.apache.org/docs/2.2/mod/mpm_common.html#threadlimit
ThreadLimit 200

#maximum number of simultaneous client connections
#http://httpd.apache.org/docs/2.2/mod/mpm_common.html#maxclients
MaxClients 10000

#number of worker threads created by each child process
#http://httpd.apache.org/docs/2.2/mod/mpm_common.html#threadsperchild
ThreadsPerChild 200

#maximum number of requests a server process serves
#http://httpd.apache.org/docs/2.2/mod/mpm_common.html#maxrequestsperchild
MaxRequestsPerChild 100000
</IfModule>

Linux tuning

We observed "too many open files" error on the linux machines. The hard limit for the "open files" parameter shown by ulimit -aH was set too low. We increased the ulimit on all the linux machines to the max (65536). This number also determines the number of socket that can be opened on a linux machine.

java.net.SocketException: Too many open files in system
 at java.net.Socket.createImpl(Socket.java:388)
 at java.net.Socket.(Socket.java:362)
 at java.net.Socket.(Socket.java:240)
 at org.apache.commons.httpclient.protocol.DefaultProtocolSocketFactory.createSocket(DefaultProtocolSocketFactory.java:80)
 at org.apache.commons.httpclient.protocol.DefaultProtocolSocketFactory.createSocket(DefaultProtocolSocketFactory.java:122)
 at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707)
 at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387)
 at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171)
 at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397)
 at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323)
 at org.apache.jmeter.protocol.http.sampler.HTTPSampler2.sample(HTTPSampler2.java:838)
 at org.apache.jmeter.protocol.http.sampler.HTTPSamplerBase.sample(HTTPSamplerBase.java:1021)
 at org.apache.jmeter.protocol.http.sampler.HTTPSamplerBase.sample(HTTPSamplerBase.java:1007)
 at org.apache.jmeter.threads.JMeterThread.run(JMeterThread.java:290)
 at java.lang.Thread.run(Thread.java:619)