Thursday, December 18, 2008

Load Testing Examinator: JMeter


Yes, you can load test Examinator yourself. We tested it with 19,200 users and got really good response times.

Scenario

The scenario is that the users will login to the application and choose an exam. They will start taking exam, answer some of the questions, select few of them for review, can go back in mid of test to any other question or any question selected for review. The scenario wants to simulate the real-world situation where all users would be taking a particular exam for the exam duration. Each user waits for around 45 secs to answer the next question. The users start ramping up in 2 minutes.

Overview
# of User Sessions: 19,200
Thinktime: 45 secs
User Arrival Rate: 160 users/sec
Ramp-up Time: 120 secs
Throughput: 445 pages/sec
Response Time: 5 ms

Duration of Exam: 110 mins
# of Questions in the Exam: 100
No. of Choices/Question: 5

Load Testing Tool: JMeter

# of Tomcat nodes: 16
# of Terracotta nodes: 1
# of JMeter instances: 16

Terracotta Monitoring: Terracotta Admin Console
System Monitoring: nmon
Load Balancer Monitoring: mod_status Module
Tomcat Monitoring: JConsole
GC Monitoring: verboseGC
JMeter can be really a pain-in-ass if we are trying to load 20,000 users from a single JMeter instance or distributed JMeter instances. Distributed JMeter uses RMI to control the instances running remotely creating an overhead on JMeter. JMeter goes into series of Full GC for high number of users for a large test plan. The Examinator test plan includes a lot of HTTP Samples, around 280 HTTP request.
Tips to use JMeter for load testing Examinator:
  • Avoid using GUI for load testing
  • Avoid using distributed JMeter instances
  • Reduce I/O by reducing data being saved under load per sample using bin/jmeter.properties
To make it happen, we ran 16 different JMeter instances after which we consolidated the results from each. Shell scripts was a great help to make it happen. Now we have a problem, each JMeter instance can't use same datapool (user login/passwords). We created copies of the JMeter Test plan and divided datapool of 19,200 into 16 files, each having 1200 users.

A small shell script does this for me:

if [ $# -lt 1 ];then
echo "Usage: "
exit
fi
name=`echo $1 | cut -d '.' -f 1`
while [ $i -lt $2 ]
do
echo "Creating ${name}${i}.jmx..."
cp $1 ${name}${i}.jmx
`sed -i 's/userList.csv/userList'$i'.csv/g' ${name}${i}.jmx`
done


Thus, each JMeter instances would be using a separate set of users and simulation 1200 users.
Posted by at No comments:

5 ms Response Time for Online tests - Isn't it awesome!!!

With Terracotta, it's all possible. We can get a response time of 5 ms for a online exam application under high load of 20,000 users. We have used best technologies and are able to get 5 ms response time and reduce the load on database.
The key fact to make it happen is to keep all the intermediate data in the memory and we have memory of 16 JVM clustered using Terracotta to use.
No need of slow access database to cluster the session data. Get more inside of it from http://www.terracotta.org/web/display/orgsite/Web+App+Reference+Implementation

Examinator is live!!!
http://reference.terracotta.org/examinator
The sources are available to download.
Posted by at No comments:

Thursday, December 4, 2008

502 Error: Bad Proxy

In the process of performance testing the examinator reference application of Terracotta, we faced a lots of 502 errors. During the performance testing few of the virtual users were receiving 502 error from the load balancer.

Load balancer: Apache HTTP server, mod_proxy_balancer module
App Server: Tomcat 6

The error received were

<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">

<html><head>

<title>502 Proxy Error</title>

</head><body>

<h1>Proxy Error</h1>

<p>The proxy server received an invalid response from an upstream server.<br />

The proxy server could not handle the request <em><a href="/examinator/exam/list.do">GET&nbsp;/examinator/exam/list.do</a></em>.<p>

Reason: <strong>Error reading from remote server</strong></p></p>

<hr>

<address>Apache/2.2.3 (Linux/SUSE) Server at xyz.abc.lan Port 80</address>

</body></html>

By disabling keepAlive on tomcat server, we were able to reduce these errors. http://tomcat.apache.org/tomcat-6.0-doc/config/http.html

<Connector port="8080" protocol="HTTP/1.1"
maxKeepAliveRequests="1"
redirectPort="8443" />
Posted by at No comments:

Response Time Tracking Method

The method described below records the server latency only i.e. the time spent in the server after a request is been received and response is sent back. It does not includes the network latency, browser rendering time, etc. The average time can also be printed in the logs (responseTime.log)

ResponseTimeTrackingValve

Apache Tomcat has a component valve which can be used to track down the response time for each page. A Valve element represents a component that will be inserted into the request processing pipeline for the associated Catalina container (Engine, Host, or Context). Each request passes through the enabled valves to the web container and response follows the same path.

The code can be downloaded from https://svn.terracotta.org/repo/forge/projects/exam-perf-test/ResponseTimeTrackingValve

The purpose of Customized Valve (ResponseTimeTrackingValve) is to record the response time for the pages and store the response time statistics for the page. ResponseTimeTrackingValve uses the AccessLogValve code as the base.

ResponseTimeTrackingValve needs to extend org.apache.catalina.valves.ValveBase located in Catalina.jar file. The main method to override is
invoke (org.apache.catalina.connector.Request request, org.apache.catalina.connector.Response response)
request is passed to next valve by
getNext().invoke(request, response);
The time taken by the server is calculated and saved in a hashtable. Key to the hashtable remains the URI of the request and following stats are being stored in it -

1. Average of all requests
2. Minimum Response Time
3. Maximum Response Time
4. Standard Deviation
5. Total Count of each Page Request made to that server

To calculate standard deviation, the sum of squares of the values is being stored, as we get a new value, square of the value gets added to it. Thus, standard deviation is calculated using these values. Same algo applies to other metrics. The response time for each page is not being stored, only the required metrics are being stored.


Where does valve stand in the HTTP request stack?


Installing the valve

To install the customized valve, deploy it as a JAR file and place it inside the Tomcats server/lib directory.

Register the Valve. Insert the following statement into Tomcat's server.xml file, right before the closing tag:

<Valve classname="org.tc.performance.ResponseTimeTrackingValve" exclude="*.png,*.css,*.js,*.gif,*.ico,/,index.html" prnCount="1000" suffix=".log" prefix="responseTime" lastPageToken="e1s31"/>

The valve keeps track of each and every request that has been made including the js, gifs, css, etc.

  1. We can specify the extensions of the files which we don't want to keep track of in the exclude parameter.
  2. prnCount is set to 1000, i.e. the average time will be printed each 1000 request.
  3. lastPageToken is the executionId of the page at which the users finish exams (this can be checked at the time of JMeter test creation). As in the test created by us, has url "/examinator/flow/takeexam.do?execution=e1s31". This will print each request response time. We can disable it by removing the token.

ResponseTime Tracking Servlet

This servlet serves the current data from the hashtable to the browser in the form of JSON. The data is shown on a web page which updates the tables with new data after specified interval. Servlet provides the method to reset the values in the hashtable. The statistics are shown in a table for each Request URI. This provides more detailed response time page-wise breakup.

The code can be downloaded from https://svn.terracotta.org/repo/forge/projects/exam-perf-test/TrackingWAR
Posted by at 2 comments:

Load Balancer for Tomcat Servers

Apache HTTP server mod_balancer module can be used as load balancer for tomcat servers. [http://httpd.apache.org/docs/2.2/mod/mod_proxy_balancer.html]
We can also configure the apache HTTP server for sticky sessions

Apache HTTP server supports prefork and worker module. worker module is recommended for less memory usage.
1. Configuring Apache Load Balancer

  • Install Apache HTTP Server
  • Enable following modules in Apache HTTP Server in conf/httpd.conf
LoadModule proxy_module modules/mod_proxy.so
LoadModule proxy_balancer_module modules/mod_proxy_balancer.so
LoadModule proxy_http_module modules/mod_proxy_http.so
LoadModule rewrite_module modules/mod_rewrite.so
  • Add the following code to httpd.conf
<VirtualHost *>
RewriteEngine On
UseCanonicalName On

# To reduce the logging on the http server

ErrorLog "/var/log/apache2/rewrite.log"
LogLevel error
CustomLog /dev/null combined

# Sticky session is maintained using JSESSIONID
# No failover

ProxyPass /examinator balancer://mycluster/ stickysession=JSESSIONID nofailover=On maxattempts=0 timeout=1800
ProxyPassReverse / balancer://cluster/
ProxyPreserveHost On
ProxyTimeout 1800

<Proxy balancer://mycluster>
BalancerMember http://perf21:8080 route=151
BalancerMember http://perf22:8080 route=152
</Proxy>

</VirtualHost>
  • To make it sticky sesion, we have to add the route ids to tomcat server config file.
  • Add route id as jvmRoute param in conf/server.xml of tomcat server. This id needs to be unique for each tomcat server. So we are using the last token of the IP address for each node. (eg . perf21 : 10.0.4.151)
This can be done by the following script

// Setting jvmRoute to the last digits of IP address in server.xml
// jvmRoute is specific to sticky session apache load balancer.
// See Load Balancer Configuration

IP=`/sbin/ifconfig | grep inet | grep '10.0' | cut -d '.' -f 4 | cut -d ' ' -f 1`
echo "Changing jvmRoute to $IP"
sed -i 's/IPCONFIG/'"$IP"'/g' $/conf/server.xml
  • This id would get appended to Cookies sent by this tomcat server. The load balancer will route based on this id to the respective server.
Posted by at No comments:
Subscribe to: Posts (Atom)