tag:blogger.com,1999:blog-89281818203693032702024-02-08T10:06:25.125+05:30Billions & TerabytesThe scale we work on!Anonymoushttp://www.blogger.com/profile/00021983633501930224noreply@blogger.comBlogger23125tag:blogger.com,1999:blog-8928181820369303270.post-45350089383417899712012-10-07T16:49:00.000+05:302013-03-19T19:57:33.831+05:30BigMemory Go - Fast Restartable Store: Kickass!!<div dir="ltr" style="text-align: left;" trbidi="on">
<script type="text/javascript">var dzone_url = 'http://performanceterracotta.blogspot.in/2012/10/bigmemory-go-latest-product-offering.html';
var dzone_title = 'Kickass BigMemory Go!!';
var dzone_blurb = 'BigMemory Go - Fast Restartable Store';
var dzone_style = '2';</script><script language="javascript" src="http://widgets.dzone.com/links/widgets/zoneit.js"></script><br />
<div dir="ltr" style="text-align: left;" trbidi="on">
<a href="http://terracotta.org/products/bigmemorygo" target="_blank">BigMemory Go</a>, the latest product offering from Terracotta. BigMemory Go lets you keep all of your application's data instantly available in your server's memory. Your application will respond promptly when you have your data closest to the application. <br />
<br />
A general application will have a database storing GBs of data, which makes the application slow. BigMemory Go provides a brand new reliable, durable, crash resilient and, above all, fast storage, called Fast Restartable Store. <a href="http://terracotta.org/documentation/bigmemorygo/configuration/fast-restart" target="_blank">Fast Restart</a> feature provides the option to store a fully consistent copy of the in-memory data on the local disk at all times. <br />
<br />
We generally use Database for bookkeeping, state-of-record, persistent data. With BigMemory Go, we provide durable persistent data store, which can handle any kind of shutdown - planned or unplanned. The next time your application starts up, all of the data that was in memory is still available and very quickly accessible.<br />
<br />
We are here comparing the performance of mysql db (tuned to my best knowledge) with Ehcache hibernate 2nd level cache (w/ DB backend) and BigMemory store altogether. The sample application used here is classic Spring's PetClinic app (using hibernate), modified for simplicity and shed spring webflow, converted to standalone Java performance test. This app has a number of Owners which have Pets and corresponding Visits to the Clinic. The test initially creates a number of Owners with pets and visits and puts them to the storage. Then, it executes 10% read/write test for 30 mins.<br />
<br />
The three cases considered here are <br />
<ol style="text-align: left;">
<li>No Hibernate caching with DB</li>
<li>Ehcache with BigMemory used as Hibernate 2nd level cache with DB backend.</li>
<li>BigMemory Go Fast Restartable Store (FRS)</li>
</ol>
<div style="text-align: left;">
Performance comparison chart shows that BigMemory Go FRS outnumbers the competitors. With no hibernate and db transaction managements, it just runs faster than ever.</div>
<div style="text-align: left;">
<br /></div>
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi5p1WxZlHMDHZlIY5VG6mLBfaDgmlaqybd9b2uj-z-LW5Rf_EQrPWtx7vQPDiRqE8XI3pvdi01ndCM6LHiozFKjYxKtGe77ow8JiV5rKgnvfRb10WxItnyLWtVqxMUqDyNLO7s6LIYmqQ/s1600/db.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="278" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi5p1WxZlHMDHZlIY5VG6mLBfaDgmlaqybd9b2uj-z-LW5Rf_EQrPWtx7vQPDiRqE8XI3pvdi01ndCM6LHiozFKjYxKtGe77ow8JiV5rKgnvfRb10WxItnyLWtVqxMUqDyNLO7s6LIYmqQ/s320/db.png" width="320" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">DB vs Ehcache h2lc vs BigMemory FRS</td></tr>
</tbody></table>
<br />
BigMemory Go FRS provides consistent 50 µs latency 99.99% of the time compared to 3 ms average latency for ehcache h2lc and 60 ms latency for MySQL db.With hibernate 2nd level cache we are surely making reads faster but still we have too much of hibernate and JPA transaction overhead to it.<br />
<div style="text-align: left;">
<br />
BigMemory Go <a href="http://terracotta.org/documentation/bigmemorygo/configuration/fast-restart" target="_blank">FRS </a>is new-age data storage which is durable, reliable and super-fast. ;)<br />
<br />
<br />
Test Sources (Maven Project <a href="https://docs.google.com/open?id=0Bw08t-P-Cf42QmdJWEcxQl82aUk" target="_blank">Download</a>)</div>
</div>
</div>
Anonymoushttp://www.blogger.com/profile/00021983633501930224noreply@blogger.com2tag:blogger.com,1999:blog-8928181820369303270.post-38273323296214673072012-09-21T14:22:00.002+05:302012-10-06T08:18:54.689+05:30NUMA & Java<div dir="ltr" style="text-align: left;" trbidi="on"><script type="text/javascript">var dzone_url = 'http://performanceterracotta.blogspot.in/2012/09/numa-java.html';
var dzone_title = '[title]';
var dzone_blurb = 'NUMA & Java';
var dzone_style = '2';</script><br />
<script language="javascript" src="http://widgets.dzone.com/links/widgets/zoneit.js"></script> <br />
<div dir="ltr" style="text-align: left;" trbidi="on">Time to deploy your application, looking forward to procure hardware that suits best to the load requirements. Boxes with 40 cores or 80 cores are pretty common these days. General conception is more cores, more processing power, more throughput. But I have seen a little contrary results, showing a small cpu-intensive test run performs slower on 80 core box than smaller 40 core box. <br />
<br />
These boxes with huge cores comes with Non-Uniform Memory Access (NUMA) Architecture. NUMA is an architecture which boosts the performance of memory access for local nodes. These new hardware boxes are divided into different zones called Nodes. These nodes have a certain number of cores alloted with a portion of memory alloted to them. So for the box with 1 TB RAM and 80 Cores, we have 4 nodes each having 20 cores and 256 GB of memory alloted.<br />
<br />
<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg1psvt5Rw-mxrELjjf3DuILCw95c6Y9dyvpWZNfeuv6OtFGOTSFoGPwvWRVw17iXtRFxAu6ynjD1oH6p69PmwfdgNgTXOsany2d1r63slYSaOOzFVMOEANcs05isHLF2CHSlRmpAglC_k/s1600/numa.PNG" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="320" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg1psvt5Rw-mxrELjjf3DuILCw95c6Y9dyvpWZNfeuv6OtFGOTSFoGPwvWRVw17iXtRFxAu6ynjD1oH6p69PmwfdgNgTXOsany2d1r63slYSaOOzFVMOEANcs05isHLF2CHSlRmpAglC_k/s320/numa.PNG" width="295" /></a></div><br />
You can check that using command, <span style="font-family: "Courier New",Courier,monospace;">numactl --hardware</span><br />
<blockquote class="tr_bq"><span style="font-family: "Courier New",Courier,monospace;">>numactl --hardware<br />
available: 4 nodes (0-3)<br />
node 0 size: 258508 MB<br />
node 0 free: 186566 MB<br />
node 1 size: 258560 MB<br />
node 1 free: 237408 MB<br />
node 2 size: 258560 MB<br />
node 2 free: 234198 MB<br />
node 3 size: 256540 MB<br />
node 3 free: 237182 MB<br />
node distances:<br />
node 0 1 2 3 <br />
0: 10 20 20 20 <br />
1: 20 10 20 20 <br />
2: 20 20 10 20 <br />
3: 20 20 20 10 </span></blockquote><br />
When JVM starts it starts thread which are scheduled on the cores in some random nodes. Each thread uses its local memory to be fastest as possible. Thread might be in WAITING state at some point and gets rescheduled on CPU. This time its not guaranteed that it will be on same node. Now this time, it has to access a remote memory location which adds latency. Remote memory access is slower, because the instructions has to traverse a interconnect link which introduces additional hops.<br />
<br />
Linux command numactl provides a way to bind the process to certain nodes only. It locks a process to a specific node both for execution and memory allocation. If a JVM instance is locked to a single node then the inter node traffic is removed and all memory access will happen on the fast local memory. <br />
<blockquote><span style="font-family: "Courier New",Courier,monospace;">numactl <br />
--cpunodebind=nodes, -c nodes <br />
Only execute process on the CPUs of nodes. </span></blockquote>Created a small test which tries to serialize a big object and calculates the transactions per sec and latency.<br />
<br />
To execute a java process bound to one node execute<br />
<blockquote class="tr_bq"><span style="font-family: "Courier New",Courier,monospace;">numactl --cpunodebind=0 java -Dthreads=10 -jar serializationTest.jar </span></blockquote><br />
Ran this test on two different boxes.<br />
<br />
<b>Box A </b><br />
4 CPU x 10 cores x 2 (hyperthreading) = Total of 80 cores<br />
Nodes: 0,1,2,3<br />
<br />
<b>Box B</b><br />
2 CPU x 10 cores x 2 (hyperthreading) = Total of 40 cores<br />
Nodes: 0,1<br />
<br />
CPU Speed : 2.4 Ghz for both.<br />
Default settings are too use all nodes available on boxes.<br />
<!--[if gte mso 9]><xml> <o:OfficeDocumentSettings> <o:AllowPNG/> </o:OfficeDocumentSettings> </xml><![endif]--><br />
<!--[if gte mso 9]><xml> <w:WordDocument> <w:View>Normal</w:View> <w:Zoom>0</w:Zoom> <w:TrackMoves/> <w:TrackFormatting/> <w:PunctuationKerning/> <w:ValidateAgainstSchemas/> <w:SaveIfXMLInvalid>false</w:SaveIfXMLInvalid> <w:IgnoreMixedContent>false</w:IgnoreMixedContent> <w:AlwaysShowPlaceholderText>false</w:AlwaysShowPlaceholderText> <w:DoNotPromoteQF/> <w:LidThemeOther>EN-US</w:LidThemeOther> <w:LidThemeAsian>X-NONE</w:LidThemeAsian> <w:LidThemeComplexScript>X-NONE</w:LidThemeComplexScript> <w:Compatibility> <w:BreakWrappedTables/> <w:SnapToGridInCell/> <w:WrapTextWithPunct/> <w:UseAsianBreakRules/> <w:DontGrowAutofit/> <w:SplitPgBreakAndParaMark/> <w:EnableOpenTypeKerning/> <w:DontFlipMirrorIndents/> <w:OverrideTableStyleHps/> </w:Compatibility> <m:mathPr> <m:mathFont m:val="Cambria Math"/> <m:brkBin m:val="before"/> <m:brkBinSub m:val="--"/> <m:smallFrac m:val="off"/> <m:dispDef/> <m:lMargin m:val="0"/> <m:rMargin m:val="0"/> <m:defJc m:val="centerGroup"/> <m:wrapIndent m:val="1440"/> <m:intLim m:val="subSup"/> <m:naryLim m:val="undOvr"/> </m:mathPr></w:WordDocument> </xml><![endif]--><!--[if gte mso 9]><xml> <w:LatentStyles DefLockedState="false" DefUnhideWhenUsed="true"
DefSemiHidden="true" DefQFormat="false" DefPriority="99"
LatentStyleCount="267"> <w:LsdException Locked="false" Priority="0" SemiHidden="false"
UnhideWhenUsed="false" QFormat="true" Name="Normal"/> <w:LsdException Locked="false" Priority="9" SemiHidden="false"
UnhideWhenUsed="false" QFormat="true" Name="heading 1"/> <w:LsdException Locked="false" Priority="9" QFormat="true" Name="heading 2"/> <w:LsdException Locked="false" Priority="9" QFormat="true" Name="heading 3"/> <w:LsdException Locked="false" Priority="9" QFormat="true" Name="heading 4"/> <w:LsdException Locked="false" Priority="9" QFormat="true" Name="heading 5"/> <w:LsdException Locked="false" Priority="9" QFormat="true" Name="heading 6"/> <w:LsdException Locked="false" Priority="9" QFormat="true" Name="heading 7"/> <w:LsdException Locked="false" Priority="9" QFormat="true" Name="heading 8"/> <w:LsdException Locked="false" Priority="9" QFormat="true" Name="heading 9"/> <w:LsdException Locked="false" Priority="39" Name="toc 1"/> <w:LsdException Locked="false" Priority="39" Name="toc 2"/> <w:LsdException Locked="false" Priority="39" Name="toc 3"/> <w:LsdException Locked="false" Priority="39" Name="toc 4"/> <w:LsdException Locked="false" Priority="39" Name="toc 5"/> <w:LsdException Locked="false" Priority="39" Name="toc 6"/> <w:LsdException Locked="false" Priority="39" Name="toc 7"/> <w:LsdException Locked="false" Priority="39" Name="toc 8"/> <w:LsdException Locked="false" Priority="39" Name="toc 9"/> <w:LsdException Locked="false" Priority="35" QFormat="true" Name="caption"/> <w:LsdException Locked="false" Priority="10" SemiHidden="false"
UnhideWhenUsed="false" QFormat="true" Name="Title"/> <w:LsdException Locked="false" Priority="1" Name="Default Paragraph Font"/> <w:LsdException Locked="false" Priority="11" SemiHidden="false"
UnhideWhenUsed="false" QFormat="true" Name="Subtitle"/> <w:LsdException Locked="false" Priority="22" SemiHidden="false"
UnhideWhenUsed="false" QFormat="true" Name="Strong"/> <w:LsdException Locked="false" Priority="20" SemiHidden="false"
UnhideWhenUsed="false" QFormat="true" Name="Emphasis"/> <w:LsdException Locked="false" Priority="59" SemiHidden="false"
UnhideWhenUsed="false" Name="Table Grid"/> <w:LsdException Locked="false" UnhideWhenUsed="false" Name="Placeholder Text"/> <w:LsdException Locked="false" Priority="1" SemiHidden="false"
UnhideWhenUsed="false" QFormat="true" Name="No Spacing"/> <w:LsdException Locked="false" Priority="60" SemiHidden="false"
UnhideWhenUsed="false" Name="Light Shading"/> <w:LsdException Locked="false" Priority="61" SemiHidden="false"
UnhideWhenUsed="false" Name="Light List"/> <w:LsdException Locked="false" Priority="62" SemiHidden="false"
UnhideWhenUsed="false" Name="Light Grid"/> <w:LsdException Locked="false" Priority="63" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Shading 1"/> <w:LsdException Locked="false" Priority="64" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Shading 2"/> <w:LsdException Locked="false" Priority="65" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium List 1"/> <w:LsdException Locked="false" Priority="66" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium List 2"/> <w:LsdException Locked="false" Priority="67" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Grid 1"/> <w:LsdException Locked="false" Priority="68" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Grid 2"/> <w:LsdException Locked="false" Priority="69" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Grid 3"/> <w:LsdException Locked="false" Priority="70" SemiHidden="false"
UnhideWhenUsed="false" Name="Dark List"/> <w:LsdException Locked="false" Priority="71" SemiHidden="false"
UnhideWhenUsed="false" Name="Colorful Shading"/> <w:LsdException Locked="false" Priority="72" SemiHidden="false"
UnhideWhenUsed="false" Name="Colorful List"/> <w:LsdException Locked="false" Priority="73" SemiHidden="false"
UnhideWhenUsed="false" Name="Colorful Grid"/> <w:LsdException Locked="false" Priority="60" SemiHidden="false"
UnhideWhenUsed="false" Name="Light Shading Accent 1"/> <w:LsdException Locked="false" Priority="61" SemiHidden="false"
UnhideWhenUsed="false" Name="Light List Accent 1"/> <w:LsdException Locked="false" Priority="62" SemiHidden="false"
UnhideWhenUsed="false" Name="Light Grid Accent 1"/> <w:LsdException Locked="false" Priority="63" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Shading 1 Accent 1"/> <w:LsdException Locked="false" Priority="64" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Shading 2 Accent 1"/> <w:LsdException Locked="false" Priority="65" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium List 1 Accent 1"/> <w:LsdException Locked="false" UnhideWhenUsed="false" Name="Revision"/> <w:LsdException Locked="false" Priority="34" SemiHidden="false"
UnhideWhenUsed="false" QFormat="true" Name="List Paragraph"/> <w:LsdException Locked="false" Priority="29" SemiHidden="false"
UnhideWhenUsed="false" QFormat="true" Name="Quote"/> <w:LsdException Locked="false" Priority="30" SemiHidden="false"
UnhideWhenUsed="false" QFormat="true" Name="Intense Quote"/> <w:LsdException Locked="false" Priority="66" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium List 2 Accent 1"/> <w:LsdException Locked="false" Priority="67" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Grid 1 Accent 1"/> <w:LsdException Locked="false" Priority="68" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Grid 2 Accent 1"/> <w:LsdException Locked="false" Priority="69" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Grid 3 Accent 1"/> <w:LsdException Locked="false" Priority="70" SemiHidden="false"
UnhideWhenUsed="false" Name="Dark List Accent 1"/> <w:LsdException Locked="false" Priority="71" SemiHidden="false"
UnhideWhenUsed="false" Name="Colorful Shading Accent 1"/> <w:LsdException Locked="false" Priority="72" SemiHidden="false"
UnhideWhenUsed="false" Name="Colorful List Accent 1"/> <w:LsdException Locked="false" Priority="73" SemiHidden="false"
UnhideWhenUsed="false" Name="Colorful Grid Accent 1"/> <w:LsdException Locked="false" Priority="60" SemiHidden="false"
UnhideWhenUsed="false" Name="Light Shading Accent 2"/> <w:LsdException Locked="false" Priority="61" SemiHidden="false"
UnhideWhenUsed="false" Name="Light List Accent 2"/> <w:LsdException Locked="false" Priority="62" SemiHidden="false"
UnhideWhenUsed="false" Name="Light Grid Accent 2"/> <w:LsdException Locked="false" Priority="63" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Shading 1 Accent 2"/> <w:LsdException Locked="false" Priority="64" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Shading 2 Accent 2"/> <w:LsdException Locked="false" Priority="65" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium List 1 Accent 2"/> <w:LsdException Locked="false" Priority="66" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium List 2 Accent 2"/> <w:LsdException Locked="false" Priority="67" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Grid 1 Accent 2"/> <w:LsdException Locked="false" Priority="68" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Grid 2 Accent 2"/> <w:LsdException Locked="false" Priority="69" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Grid 3 Accent 2"/> <w:LsdException Locked="false" Priority="70" SemiHidden="false"
UnhideWhenUsed="false" Name="Dark List Accent 2"/> <w:LsdException Locked="false" Priority="71" SemiHidden="false"
UnhideWhenUsed="false" Name="Colorful Shading Accent 2"/> <w:LsdException Locked="false" Priority="72" SemiHidden="false"
UnhideWhenUsed="false" Name="Colorful List Accent 2"/> <w:LsdException Locked="false" Priority="73" SemiHidden="false"
UnhideWhenUsed="false" Name="Colorful Grid Accent 2"/> <w:LsdException Locked="false" Priority="60" SemiHidden="false"
UnhideWhenUsed="false" Name="Light Shading Accent 3"/> <w:LsdException Locked="false" Priority="61" SemiHidden="false"
UnhideWhenUsed="false" Name="Light List Accent 3"/> <w:LsdException Locked="false" Priority="62" SemiHidden="false"
UnhideWhenUsed="false" Name="Light Grid Accent 3"/> <w:LsdException Locked="false" Priority="63" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Shading 1 Accent 3"/> <w:LsdException Locked="false" Priority="64" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Shading 2 Accent 3"/> <w:LsdException Locked="false" Priority="65" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium List 1 Accent 3"/> <w:LsdException Locked="false" Priority="66" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium List 2 Accent 3"/> <w:LsdException Locked="false" Priority="67" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Grid 1 Accent 3"/> <w:LsdException Locked="false" Priority="68" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Grid 2 Accent 3"/> <w:LsdException Locked="false" Priority="69" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Grid 3 Accent 3"/> <w:LsdException Locked="false" Priority="70" SemiHidden="false"
UnhideWhenUsed="false" Name="Dark List Accent 3"/> <w:LsdException Locked="false" Priority="71" SemiHidden="false"
UnhideWhenUsed="false" Name="Colorful Shading Accent 3"/> <w:LsdException Locked="false" Priority="72" SemiHidden="false"
UnhideWhenUsed="false" Name="Colorful List Accent 3"/> <w:LsdException Locked="false" Priority="73" SemiHidden="false"
UnhideWhenUsed="false" Name="Colorful Grid Accent 3"/> <w:LsdException Locked="false" Priority="60" SemiHidden="false"
UnhideWhenUsed="false" Name="Light Shading Accent 4"/> <w:LsdException Locked="false" Priority="61" SemiHidden="false"
UnhideWhenUsed="false" Name="Light List Accent 4"/> <w:LsdException Locked="false" Priority="62" SemiHidden="false"
UnhideWhenUsed="false" Name="Light Grid Accent 4"/> <w:LsdException Locked="false" Priority="63" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Shading 1 Accent 4"/> <w:LsdException Locked="false" Priority="64" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Shading 2 Accent 4"/> <w:LsdException Locked="false" Priority="65" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium List 1 Accent 4"/> <w:LsdException Locked="false" Priority="66" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium List 2 Accent 4"/> <w:LsdException Locked="false" Priority="67" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Grid 1 Accent 4"/> <w:LsdException Locked="false" Priority="68" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Grid 2 Accent 4"/> <w:LsdException Locked="false" Priority="69" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Grid 3 Accent 4"/> <w:LsdException Locked="false" Priority="70" SemiHidden="false"
UnhideWhenUsed="false" Name="Dark List Accent 4"/> <w:LsdException Locked="false" Priority="71" SemiHidden="false"
UnhideWhenUsed="false" Name="Colorful Shading Accent 4"/> <w:LsdException Locked="false" Priority="72" SemiHidden="false"
UnhideWhenUsed="false" Name="Colorful List Accent 4"/> <w:LsdException Locked="false" Priority="73" SemiHidden="false"
UnhideWhenUsed="false" Name="Colorful Grid Accent 4"/> <w:LsdException Locked="false" Priority="60" SemiHidden="false"
UnhideWhenUsed="false" Name="Light Shading Accent 5"/> <w:LsdException Locked="false" Priority="61" SemiHidden="false"
UnhideWhenUsed="false" Name="Light List Accent 5"/> <w:LsdException Locked="false" Priority="62" SemiHidden="false"
UnhideWhenUsed="false" Name="Light Grid Accent 5"/> <w:LsdException Locked="false" Priority="63" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Shading 1 Accent 5"/> <w:LsdException Locked="false" Priority="64" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Shading 2 Accent 5"/> <w:LsdException Locked="false" Priority="65" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium List 1 Accent 5"/> <w:LsdException Locked="false" Priority="66" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium List 2 Accent 5"/> <w:LsdException Locked="false" Priority="67" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Grid 1 Accent 5"/> <w:LsdException Locked="false" Priority="68" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Grid 2 Accent 5"/> <w:LsdException Locked="false" Priority="69" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Grid 3 Accent 5"/> <w:LsdException Locked="false" Priority="70" SemiHidden="false"
UnhideWhenUsed="false" Name="Dark List Accent 5"/> <w:LsdException Locked="false" Priority="71" SemiHidden="false"
UnhideWhenUsed="false" Name="Colorful Shading Accent 5"/> <w:LsdException Locked="false" Priority="72" SemiHidden="false"
UnhideWhenUsed="false" Name="Colorful List Accent 5"/> <w:LsdException Locked="false" Priority="73" SemiHidden="false"
UnhideWhenUsed="false" Name="Colorful Grid Accent 5"/> <w:LsdException Locked="false" Priority="60" SemiHidden="false"
UnhideWhenUsed="false" Name="Light Shading Accent 6"/> <w:LsdException Locked="false" Priority="61" SemiHidden="false"
UnhideWhenUsed="false" Name="Light List Accent 6"/> <w:LsdException Locked="false" Priority="62" SemiHidden="false"
UnhideWhenUsed="false" Name="Light Grid Accent 6"/> <w:LsdException Locked="false" Priority="63" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Shading 1 Accent 6"/> <w:LsdException Locked="false" Priority="64" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Shading 2 Accent 6"/> <w:LsdException Locked="false" Priority="65" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium List 1 Accent 6"/> <w:LsdException Locked="false" Priority="66" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium List 2 Accent 6"/> <w:LsdException Locked="false" Priority="67" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Grid 1 Accent 6"/> <w:LsdException Locked="false" Priority="68" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Grid 2 Accent 6"/> <w:LsdException Locked="false" Priority="69" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Grid 3 Accent 6"/> <w:LsdException Locked="false" Priority="70" SemiHidden="false"
UnhideWhenUsed="false" Name="Dark List Accent 6"/> <w:LsdException Locked="false" Priority="71" SemiHidden="false"
UnhideWhenUsed="false" Name="Colorful Shading Accent 6"/> <w:LsdException Locked="false" Priority="72" SemiHidden="false"
UnhideWhenUsed="false" Name="Colorful List Accent 6"/> <w:LsdException Locked="false" Priority="73" SemiHidden="false"
UnhideWhenUsed="false" Name="Colorful Grid Accent 6"/> <w:LsdException Locked="false" Priority="19" SemiHidden="false"
UnhideWhenUsed="false" QFormat="true" Name="Subtle Emphasis"/> <w:LsdException Locked="false" Priority="21" SemiHidden="false"
UnhideWhenUsed="false" QFormat="true" Name="Intense Emphasis"/> <w:LsdException Locked="false" Priority="31" SemiHidden="false"
UnhideWhenUsed="false" QFormat="true" Name="Subtle Reference"/> <w:LsdException Locked="false" Priority="32" SemiHidden="false"
UnhideWhenUsed="false" QFormat="true" Name="Intense Reference"/> <w:LsdException Locked="false" Priority="33" SemiHidden="false"
UnhideWhenUsed="false" QFormat="true" Name="Book Title"/> <w:LsdException Locked="false" Priority="37" Name="Bibliography"/> <w:LsdException Locked="false" Priority="39" QFormat="true" Name="TOC Heading"/> </w:LatentStyles> </xml><![endif]--><!--[if gte mso 10]> <style>
/* Style Definitions */
table.MsoNormalTable
{mso-style-name:"Table Normal";
mso-tstyle-rowband-size:0;
mso-tstyle-colband-size:0;
mso-style-noshow:yes;
mso-style-priority:99;
mso-style-parent:"";
mso-padding-alt:0in 5.4pt 0in 5.4pt;
mso-para-margin-top:0in;
mso-para-margin-right:0in;
mso-para-margin-bottom:10.0pt;
mso-para-margin-left:0in;
line-height:115%;
mso-pagination:widow-orphan;
font-size:11.0pt;
font-family:"Calibri","sans-serif";
mso-ascii-font-family:Calibri;
mso-ascii-theme-font:minor-latin;
mso-hansi-font-family:Calibri;
mso-hansi-theme-font:minor-latin;
mso-bidi-font-family:"Times New Roman";
mso-bidi-theme-font:minor-bidi;}
table.MsoTableLightShadingAccent5
{mso-style-name:"Light Shading - Accent 5";
mso-tstyle-rowband-size:1;
mso-tstyle-colband-size:1;
mso-style-priority:60;
mso-style-unhide:no;
border-top:solid #4BACC6 1.0pt;
mso-border-top-themecolor:accent5;
border-left:none;
border-bottom:solid #4BACC6 1.0pt;
mso-border-bottom-themecolor:accent5;
border-right:none;
mso-padding-alt:0in 5.4pt 0in 5.4pt;
mso-para-margin:0in;
mso-para-margin-bottom:.0001pt;
mso-pagination:widow-orphan;
font-size:11.0pt;
font-family:"Calibri","sans-serif";
mso-ascii-font-family:Calibri;
mso-ascii-theme-font:minor-latin;
mso-hansi-font-family:Calibri;
mso-hansi-theme-font:minor-latin;
mso-bidi-font-family:"Times New Roman";
mso-bidi-theme-font:minor-bidi;
color:#31849B;
mso-themecolor:accent5;
mso-themeshade:191;}
table.MsoTableLightShadingAccent5FirstRow
{mso-style-name:"Light Shading - Accent 5";
mso-table-condition:first-row;
mso-style-priority:60;
mso-style-unhide:no;
mso-tstyle-border-top:1.0pt solid #4BACC6;
mso-tstyle-border-top-themecolor:accent5;
mso-tstyle-border-left:cell-none;
mso-tstyle-border-bottom:1.0pt solid #4BACC6;
mso-tstyle-border-bottom-themecolor:accent5;
mso-tstyle-border-right:cell-none;
mso-tstyle-border-insideh:cell-none;
mso-tstyle-border-insidev:cell-none;
mso-para-margin-top:0in;
mso-para-margin-bottom:0in;
mso-para-margin-bottom:.0001pt;
line-height:normal;
mso-ansi-font-weight:bold;
mso-bidi-font-weight:bold;}
table.MsoTableLightShadingAccent5LastRow
{mso-style-name:"Light Shading - Accent 5";
mso-table-condition:last-row;
mso-style-priority:60;
mso-style-unhide:no;
mso-tstyle-border-top:1.0pt solid #4BACC6;
mso-tstyle-border-top-themecolor:accent5;
mso-tstyle-border-left:cell-none;
mso-tstyle-border-bottom:1.0pt solid #4BACC6;
mso-tstyle-border-bottom-themecolor:accent5;
mso-tstyle-border-right:cell-none;
mso-tstyle-border-insideh:cell-none;
mso-tstyle-border-insidev:cell-none;
mso-para-margin-top:0in;
mso-para-margin-bottom:0in;
mso-para-margin-bottom:.0001pt;
line-height:normal;
mso-ansi-font-weight:bold;
mso-bidi-font-weight:bold;}
table.MsoTableLightShadingAccent5FirstCol
{mso-style-name:"Light Shading - Accent 5";
mso-table-condition:first-column;
mso-style-priority:60;
mso-style-unhide:no;
mso-ansi-font-weight:bold;
mso-bidi-font-weight:bold;}
table.MsoTableLightShadingAccent5LastCol
{mso-style-name:"Light Shading - Accent 5";
mso-table-condition:last-column;
mso-style-priority:60;
mso-style-unhide:no;
mso-ansi-font-weight:bold;
mso-bidi-font-weight:bold;}
table.MsoTableLightShadingAccent5OddColumn
{mso-style-name:"Light Shading - Accent 5";
mso-table-condition:odd-column;
mso-style-priority:60;
mso-style-unhide:no;
mso-tstyle-shading:#D2EAF1;
mso-tstyle-shading-themecolor:accent5;
mso-tstyle-shading-themetint:63;
mso-tstyle-border-left:cell-none;
mso-tstyle-border-right:cell-none;
mso-tstyle-border-insideh:cell-none;
mso-tstyle-border-insidev:cell-none;}
table.MsoTableLightShadingAccent5OddRow
{mso-style-name:"Light Shading - Accent 5";
mso-table-condition:odd-row;
mso-style-priority:60;
mso-style-unhide:no;
mso-tstyle-shading:#D2EAF1;
mso-tstyle-shading-themecolor:accent5;
mso-tstyle-shading-themetint:63;
mso-tstyle-border-left:cell-none;
mso-tstyle-border-right:cell-none;
mso-tstyle-border-insideh:cell-none;
mso-tstyle-border-insidev:cell-none;}
</style> <![endif]--> <br />
<table border="1" cellpadding="0" cellspacing="0" class="MsoTableLightShadingAccent5" style="border-collapse: collapse; border: none; mso-border-bottom-alt: solid #4BACC6 1.0pt; mso-border-bottom-themecolor: accent5; mso-border-top-alt: solid #4BACC6 1.0pt; mso-border-top-themecolor: accent5; mso-padding-alt: 0in 5.4pt 0in 5.4pt; mso-yfti-tbllook: 1184; width: 379px;"><tbody>
<tr style="height: 15.0pt; mso-yfti-firstrow: yes; mso-yfti-irow: -1;"> <td nowrap="" style="border-bottom: solid #4BACC6 1.0pt; border-left: none; border-right: none; border-top: solid #4BACC6 1.0pt; height: 15.0pt; mso-border-bottom-themecolor: accent5; mso-border-top-themecolor: accent5; padding: 0in 5.4pt 0in 5.4pt; width: 33.45pt;" valign="bottom" width="45"><div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0in; mso-yfti-cnfc: 5;"><b><span style="color: black; mso-ascii-font-family: Calibri; mso-bidi-font-family: Calibri; mso-fareast-font-family: "Times New Roman"; mso-hansi-font-family: Calibri;">Box</span></b></div></td> <td nowrap="" style="border-bottom: solid #4BACC6 1.0pt; border-left: none; border-right: none; border-top: solid #4BACC6 1.0pt; height: 15.0pt; mso-border-bottom-themecolor: accent5; mso-border-top-themecolor: accent5; padding: 0in 5.4pt 0in 5.4pt; width: 110.15pt;" valign="bottom" width="147"><div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0in; mso-yfti-cnfc: 1;"><b><span style="color: black; mso-ascii-font-family: Calibri; mso-bidi-font-family: Calibri; mso-fareast-font-family: "Times New Roman"; mso-hansi-font-family: Calibri;">NUMA policy</span></b></div></td> <td nowrap="" style="border-bottom: solid #4BACC6 1.0pt; border-left: none; border-right: none; border-top: solid #4BACC6 1.0pt; height: 15.0pt; mso-border-bottom-themecolor: accent5; mso-border-top-themecolor: accent5; padding: 0in 5.4pt 0in 5.4pt; width: 41.35pt;" valign="bottom" width="55"><div align="right" class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0in; mso-yfti-cnfc: 1; text-align: right;"><b><span style="color: black; mso-ascii-font-family: Calibri; mso-bidi-font-family: Calibri; mso-fareast-font-family: "Times New Roman"; mso-hansi-font-family: Calibri;">TPS</span></b></div></td> <td nowrap="" style="border-bottom: solid #4BACC6 1.0pt; border-left: none; border-right: none; border-top: solid #4BACC6 1.0pt; height: 15.0pt; mso-border-bottom-themecolor: accent5; mso-border-top-themecolor: accent5; padding: 0in 5.4pt 0in 5.4pt; width: 51.45pt;" valign="bottom" width="69"><div align="right" class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0in; mso-yfti-cnfc: 1; text-align: right;"><b><span style="color: black; mso-ascii-font-family: Calibri; mso-bidi-font-family: Calibri; mso-fareast-font-family: "Times New Roman"; mso-hansi-font-family: Calibri;">Latency</span></b></div><div align="right" class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0in; mso-yfti-cnfc: 1; text-align: right;"><b><span style="color: black; mso-ascii-font-family: Calibri; mso-bidi-font-family: Calibri; mso-fareast-font-family: "Times New Roman"; mso-hansi-font-family: Calibri;">(Avg)</span></b></div></td> <td nowrap="" style="border-bottom: solid #4BACC6 1.0pt; border-left: none; border-right: none; border-top: solid #4BACC6 1.0pt; height: 15.0pt; mso-border-bottom-themecolor: accent5; mso-border-top-themecolor: accent5; padding: 0in 5.4pt 0in 5.4pt; width: 48.0pt;" valign="bottom" width="64"><div align="right" class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0in; mso-yfti-cnfc: 1; text-align: right;"><b><span style="color: black; mso-ascii-font-family: Calibri; mso-bidi-font-family: Calibri; mso-fareast-font-family: "Times New Roman"; mso-hansi-font-family: Calibri;">Latency</span></b></div><div align="right" class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0in; mso-yfti-cnfc: 1; text-align: right;"><b><span style="color: black; mso-ascii-font-family: Calibri; mso-bidi-font-family: Calibri; mso-fareast-font-family: "Times New Roman"; mso-hansi-font-family: Calibri;">(Min)</span></b></div></td> </tr>
<tr style="height: 15.0pt; mso-yfti-irow: 0;"> <td nowrap="" style="background: #D2EAF1; border: none; height: 15.0pt; mso-background-themecolor: accent5; mso-background-themetint: 63; padding: 0in 5.4pt 0in 5.4pt; width: 33.45pt;" valign="top" width="45"><div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0in; mso-yfti-cnfc: 68;"><b><span style="color: black; mso-ascii-font-family: Calibri; mso-bidi-font-family: Calibri; mso-fareast-font-family: "Times New Roman"; mso-hansi-font-family: Calibri;">A</span></b></div></td> <td nowrap="" style="background: #D2EAF1; border: none; height: 15.0pt; mso-background-themecolor: accent5; mso-background-themetint: 63; padding: 0in 5.4pt 0in 5.4pt; width: 110.15pt;" valign="top" width="147"><div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0in; mso-yfti-cnfc: 64;"><span style="color: black; mso-ascii-font-family: Calibri; mso-bidi-font-family: Calibri; mso-fareast-font-family: "Times New Roman"; mso-hansi-font-family: Calibri;">Default</span></div></td> <td nowrap="" style="background: #D2EAF1; border: none; height: 15.0pt; mso-background-themecolor: accent5; mso-background-themetint: 63; padding: 0in 5.4pt 0in 5.4pt; width: 41.35pt;" valign="top" width="55"><div align="right" class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0in; mso-yfti-cnfc: 64; text-align: right;"><span style="color: black; mso-ascii-font-family: Calibri; mso-bidi-font-family: Calibri; mso-fareast-font-family: "Times New Roman"; mso-hansi-font-family: Calibri;">261</span></div></td> <td nowrap="" style="background: #D2EAF1; border: none; height: 15.0pt; mso-background-themecolor: accent5; mso-background-themetint: 63; padding: 0in 5.4pt 0in 5.4pt; width: 51.45pt;" valign="top" width="69"><div align="right" class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0in; mso-yfti-cnfc: 64; text-align: right;"><span style="color: black; mso-ascii-font-family: Calibri; mso-bidi-font-family: Calibri; mso-fareast-font-family: "Times New Roman"; mso-hansi-font-family: Calibri;">37</span></div></td> <td nowrap="" style="background: #D2EAF1; border: none; height: 15.0pt; mso-background-themecolor: accent5; mso-background-themetint: 63; padding: 0in 5.4pt 0in 5.4pt; width: 48.0pt;" valign="top" width="64"><div align="right" class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0in; mso-yfti-cnfc: 64; text-align: right;"><span style="color: black; mso-ascii-font-family: Calibri; mso-bidi-font-family: Calibri; mso-fareast-font-family: "Times New Roman"; mso-hansi-font-family: Calibri;">18</span></div></td> </tr>
<tr style="height: 15.0pt; mso-yfti-irow: 1;"> <td nowrap="" style="border: none; height: 15.0pt; padding: 0in 5.4pt 0in 5.4pt; width: 33.45pt;" valign="top" width="45"><div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0in; mso-yfti-cnfc: 4;"><b><span style="color: black; mso-ascii-font-family: Calibri; mso-bidi-font-family: Calibri; mso-fareast-font-family: "Times New Roman"; mso-hansi-font-family: Calibri;">B</span></b></div></td> <td nowrap="" style="border: none; height: 15.0pt; padding: 0in 5.4pt 0in 5.4pt; width: 110.15pt;" valign="top" width="147"><div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0in;"><span style="color: black; mso-ascii-font-family: Calibri; mso-bidi-font-family: Calibri; mso-fareast-font-family: "Times New Roman"; mso-hansi-font-family: Calibri;">Default</span></div></td> <td nowrap="" style="border: none; height: 15.0pt; padding: 0in 5.4pt 0in 5.4pt; width: 41.35pt;" valign="top" width="55"><div align="right" class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0in; text-align: right;"><span style="color: black; mso-ascii-font-family: Calibri; mso-bidi-font-family: Calibri; mso-fareast-font-family: "Times New Roman"; mso-hansi-font-family: Calibri;">387</span></div></td> <td nowrap="" style="border: none; height: 15.0pt; padding: 0in 5.4pt 0in 5.4pt; width: 51.45pt;" valign="top" width="69"><div align="right" class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0in; text-align: right;"><span style="color: black; mso-ascii-font-family: Calibri; mso-bidi-font-family: Calibri; mso-fareast-font-family: "Times New Roman"; mso-hansi-font-family: Calibri;">25</span></div></td> <td nowrap="" style="border: none; height: 15.0pt; padding: 0in 5.4pt 0in 5.4pt; width: 48.0pt;" valign="top" width="64"><div align="right" class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0in; text-align: right;"><span style="color: black; mso-ascii-font-family: Calibri; mso-bidi-font-family: Calibri; mso-fareast-font-family: "Times New Roman"; mso-hansi-font-family: Calibri;">5</span></div></td> </tr>
<tr style="height: 15.0pt; mso-yfti-irow: 2;"> <td nowrap="" style="background: #D2EAF1; border: none; height: 15.0pt; mso-background-themecolor: accent5; mso-background-themetint: 63; padding: 0in 5.4pt 0in 5.4pt; width: 33.45pt;" valign="top" width="45"><div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0in; mso-yfti-cnfc: 68;"><b><span style="color: black; mso-ascii-font-family: Calibri; mso-bidi-font-family: Calibri; mso-fareast-font-family: "Times New Roman"; mso-hansi-font-family: Calibri;">A</span></b></div></td> <td nowrap="" style="background: #D2EAF1; border: none; height: 15.0pt; mso-background-themecolor: accent5; mso-background-themetint: 63; padding: 0in 5.4pt 0in 5.4pt; width: 110.15pt;" valign="top" width="147"><div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0in; mso-yfti-cnfc: 64;"><span style="color: black; mso-ascii-font-family: Calibri; mso-bidi-font-family: Calibri; mso-fareast-font-family: "Times New Roman"; mso-hansi-font-family: Calibri;">--cpunodebind=0,1</span></div></td> <td nowrap="" style="background: #D2EAF1; border: none; height: 15.0pt; mso-background-themecolor: accent5; mso-background-themetint: 63; padding: 0in 5.4pt 0in 5.4pt; width: 41.35pt;" valign="top" width="55"><div align="right" class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0in; mso-yfti-cnfc: 64; text-align: right;"><span style="color: black; mso-ascii-font-family: Calibri; mso-bidi-font-family: Calibri; mso-fareast-font-family: "Times New Roman"; mso-hansi-font-family: Calibri;">405</span></div></td> <td nowrap="" style="background: #D2EAF1; border: none; height: 15.0pt; mso-background-themecolor: accent5; mso-background-themetint: 63; padding: 0in 5.4pt 0in 5.4pt; width: 51.45pt;" valign="top" width="69"><div align="right" class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0in; mso-yfti-cnfc: 64; text-align: right;"><span style="color: black; mso-ascii-font-family: Calibri; mso-bidi-font-family: Calibri; mso-fareast-font-family: "Times New Roman"; mso-hansi-font-family: Calibri;">23</span></div></td> <td nowrap="" style="background: #D2EAF1; border: none; height: 15.0pt; mso-background-themecolor: accent5; mso-background-themetint: 63; padding: 0in 5.4pt 0in 5.4pt; width: 48.0pt;" valign="top" width="64"><div align="right" class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0in; mso-yfti-cnfc: 64; text-align: right;"><span style="color: black; mso-ascii-font-family: Calibri; mso-bidi-font-family: Calibri; mso-fareast-font-family: "Times New Roman"; mso-hansi-font-family: Calibri;">3</span></div></td> </tr>
<tr style="height: 15.0pt; mso-yfti-irow: 3;"> <td nowrap="" style="border: none; height: 15.0pt; padding: 0in 5.4pt 0in 5.4pt; width: 33.45pt;" valign="top" width="45"><div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0in; mso-yfti-cnfc: 4;"><b><span style="color: black; mso-ascii-font-family: Calibri; mso-bidi-font-family: Calibri; mso-fareast-font-family: "Times New Roman"; mso-hansi-font-family: Calibri;">B</span></b></div></td> <td nowrap="" style="border: none; height: 15.0pt; padding: 0in 5.4pt 0in 5.4pt; width: 110.15pt;" valign="top" width="147"><div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0in;"><span style="color: black; mso-ascii-font-family: Calibri; mso-bidi-font-family: Calibri; mso-fareast-font-family: "Times New Roman"; mso-hansi-font-family: Calibri;">--cpunodebind=0</span></div></td> <td nowrap="" style="border: none; height: 15.0pt; padding: 0in 5.4pt 0in 5.4pt; width: 41.35pt;" valign="top" width="55"><div align="right" class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0in; text-align: right;"><span style="color: black; mso-ascii-font-family: Calibri; mso-bidi-font-family: Calibri; mso-fareast-font-family: "Times New Roman"; mso-hansi-font-family: Calibri;">1,613</span></div></td> <td nowrap="" style="border: none; height: 15.0pt; padding: 0in 5.4pt 0in 5.4pt; width: 51.45pt;" valign="top" width="69"><div align="right" class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0in; text-align: right;"><span style="color: black; mso-ascii-font-family: Calibri; mso-bidi-font-family: Calibri; mso-fareast-font-family: "Times New Roman"; mso-hansi-font-family: Calibri;">5</span></div></td> <td nowrap="" style="border: none; height: 15.0pt; padding: 0in 5.4pt 0in 5.4pt; width: 48.0pt;" valign="top" width="64"><div align="right" class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0in; text-align: right;"><span style="color: black; mso-ascii-font-family: Calibri; mso-bidi-font-family: Calibri; mso-fareast-font-family: "Times New Roman"; mso-hansi-font-family: Calibri;">3</span></div></td> </tr>
<tr style="height: 15.0pt; mso-yfti-irow: 4; mso-yfti-lastrow: yes;"> <td nowrap="" style="background: #D2EAF1; border-bottom: solid #4BACC6 1.0pt; border: none; height: 15.0pt; mso-background-themecolor: accent5; mso-background-themetint: 63; mso-border-bottom-themecolor: accent5; padding: 0in 5.4pt 0in 5.4pt; width: 33.45pt;" valign="top" width="45"><div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0in; mso-yfti-cnfc: 68;"><b><span style="color: black; mso-ascii-font-family: Calibri; mso-bidi-font-family: Calibri; mso-fareast-font-family: "Times New Roman"; mso-hansi-font-family: Calibri;">A</span></b></div></td> <td nowrap="" style="background: #D2EAF1; border-bottom: solid #4BACC6 1.0pt; border: none; height: 15.0pt; mso-background-themecolor: accent5; mso-background-themetint: 63; mso-border-bottom-themecolor: accent5; padding: 0in 5.4pt 0in 5.4pt; width: 110.15pt;" valign="top" width="147"><div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0in; mso-yfti-cnfc: 64;"><span style="color: black; mso-ascii-font-family: Calibri; mso-bidi-font-family: Calibri; mso-fareast-font-family: "Times New Roman"; mso-hansi-font-family: Calibri;">--cpunodebind=0</span></div></td> <td nowrap="" style="background: #D2EAF1; border-bottom: solid #4BACC6 1.0pt; border: none; height: 15.0pt; mso-background-themecolor: accent5; mso-background-themetint: 63; mso-border-bottom-themecolor: accent5; padding: 0in 5.4pt 0in 5.4pt; width: 41.35pt;" valign="top" width="55"><div align="right" class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0in; mso-yfti-cnfc: 64; text-align: right;"><span style="color: black; mso-ascii-font-family: Calibri; mso-bidi-font-family: Calibri; mso-fareast-font-family: "Times New Roman"; mso-hansi-font-family: Calibri;">1,619</span></div></td> <td nowrap="" style="background: #D2EAF1; border-bottom: solid #4BACC6 1.0pt; border: none; height: 15.0pt; mso-background-themecolor: accent5; mso-background-themetint: 63; mso-border-bottom-themecolor: accent5; padding: 0in 5.4pt 0in 5.4pt; width: 51.45pt;" valign="top" width="69"><div align="right" class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0in; mso-yfti-cnfc: 64; text-align: right;"><span style="color: black; mso-ascii-font-family: Calibri; mso-bidi-font-family: Calibri; mso-fareast-font-family: "Times New Roman"; mso-hansi-font-family: Calibri;">5</span></div></td> <td nowrap="" style="background: #D2EAF1; border-bottom: solid #4BACC6 1.0pt; border: none; height: 15.0pt; mso-background-themecolor: accent5; mso-background-themetint: 63; mso-border-bottom-themecolor: accent5; padding: 0in 5.4pt 0in 5.4pt; width: 48.0pt;" valign="top" width="64"><div align="right" class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0in; mso-yfti-cnfc: 64; text-align: right;"><span style="color: black; mso-ascii-font-family: Calibri; mso-bidi-font-family: Calibri; mso-fareast-font-family: "Times New Roman"; mso-hansi-font-family: Calibri;">3</span></div></td> </tr>
</tbody></table><br />
So we can infer that the default settings on Box A with more Nodes is performing low on a "CPU-intesive" test compared to default setting on 2-node Box B. But as we bind the process to only 2 nodes, it performs equally better. Probably, because it has lesser nodes to hop and probability of threads getting rescheduled on same is increased to 50%.<br />
<br />
With <span style="font-family: "Courier New",Courier,monospace;">--cpunodebind=0,</span> it just outperforms all the cases. <br />
<br />
NOTE: Above test was run with 10 threads on 10 core.<br />
<br />
Test Jar: <a href="https://docs.google.com/open?id=0Bw08t-P-Cf42enlwRWNkQmtJd1k" target="_blank">download</a><br />
Test Sources: <a href="https://docs.google.com/open?id=0Bw08t-P-Cf42dndXaUk2aWJ0Mlk" target="_blank">download</a> </div></div>Anonymoushttp://www.blogger.com/profile/00021983633501930224noreply@blogger.com5tag:blogger.com,1999:blog-8928181820369303270.post-77808008255653246592012-05-31T18:37:00.000+05:302012-10-06T08:21:34.207+05:30Fast, Predictable & Highly-Available @ 1 TB/Node<div dir="ltr" style="text-align: left;" trbidi="on"><script type="text/javascript">var dzone_url = 'http://performanceterracotta.blogspot.in/2012/05/fast-predictable-highly-available-1.html';
var dzone_title = 'Fast, Predictable & Highly-Available @ 1 TB/Node';
var dzone_blurb = '';
var dzone_style = '2';</script><br />
<script language="javascript" src="http://widgets.dzone.com/links/widgets/zoneit.js"></script> <br />
<br />
<div><div dir="ltr" style="text-align: left;" trbidi="on">The world is pushing huge amounts of data to applications every second, from mobiles, the web, and various gadgets. More applications these days have to deal with this data. To preserve performance, these applications need fast access to the data tier.<br />
<br />
RAM prices have crumbled over the past few years and we can now get hardware with a Terabyte of RAM much more cheaply. OK, got the hardware, now what? We generally use virtualization to create smaller virtual machines to meet applications scale-out requirements, as having a Java application with a terabyte of heap is impractical. JVM Garbage Collection will slaughter your application right away. Ever imagined how much time will it take to do a single full garbage collection for a terabyte of heap? It can pause an application for hours, making it unusable.<br />
<br />
<a href="http://terracotta.org/products/bigmemory" style="font-weight: bold; text-decoration: none;" target="_blank"> BigMemory</a> is the key to access terabytes of data with milliseconds of latency, with no maintenance of disk/raid configurations/databases. <br />
<br />
<blockquote style="color: orange;"><b>BigMemory = Big Data + In-memory </b></blockquote><br />
<a href="http://terracotta.org/products/bigmemory" style="font-weight: bold; text-decoration: none;" target="_blank"> BigMemory</a> can utilize your hardware to the last byte of RAM. <a href="http://terracotta.org/products/bigmemory" style="font-weight: bold; text-decoration: none;" target="_blank"> BigMemory</a> can store up to a terabyte of data in single java process.<br />
<br />
<blockquote style="color: orange;"><b>BigMemory provides "fast", "predictable" and "highly-available" data at 1 terabytes per node.</b></blockquote><br />
The following test uses two boxes, each with a terabyte of RAM. Leaving enough room for the OS, we were able to allocate 2 x 960 GB of BigMemory, for a total of 1.8+ TB of data. Without facing the problems of high latencies, huge scale-out architectures ... just using the hardware as it is.<br />
<br />
Test results: 23K readonly transactions per second with 20 ms latency.<br />
Graphs for test throughput and periodic latency over time.<br />
<br />
<br />
<br />
<div style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiFhlNeOECCXPyOdKlVZ6dDp9sRtg8BlvMzJhwgOZ0hUHSPKt6t9ETryaB2JUAPAmx1-JaO3YlN9RNVckOZy63LGZaKwL-IWf9ZhNtnrLzIhLaq53_NZr-LuAA6fpGP3KZYtTWHN8g1oDw/s1600/tps.png" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiFhlNeOECCXPyOdKlVZ6dDp9sRtg8BlvMzJhwgOZ0hUHSPKt6t9ETryaB2JUAPAmx1-JaO3YlN9RNVckOZy63LGZaKwL-IWf9ZhNtnrLzIhLaq53_NZr-LuAA6fpGP3KZYtTWHN8g1oDw/s1600/tps.png" /></a></div><div style="text-align: center;"><b>Readonly Periodic Throughput Graph</b></div><br />
<br />
<div class="separator" style="clear: both; text-align: center;"></div><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgkY2mOddrHyYq3cnLtjOlN12pTIoUxzT0JF3t0zIJHAbhqOR2TgQ5Kvytmi09WIW2xRy-Zrir7OQ40UtZUoDxV-fJqaH2KcO1hhdluXOleBbtrcALx-MxTXVArBmI02QWyEzO94OWR6qc/s1600/latency.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgkY2mOddrHyYq3cnLtjOlN12pTIoUxzT0JF3t0zIJHAbhqOR2TgQ5Kvytmi09WIW2xRy-Zrir7OQ40UtZUoDxV-fJqaH2KcO1hhdluXOleBbtrcALx-MxTXVArBmI02QWyEzO94OWR6qc/s1600/latency.png" /></a></div><div style="text-align: center;"><b>Readonly Periodic Latency Graph</b></div><br />
</div></div></div>Anonymoushttp://www.blogger.com/profile/00021983633501930224noreply@blogger.com3Gurgaon, Haryana, India28.4594965 77.026638328.3478185 76.8687098 28.5711745 77.1845668tag:blogger.com,1999:blog-8928181820369303270.post-65864961129855978952012-05-11T19:25:00.001+05:302012-10-06T08:22:34.121+05:30Billions of Entries and Terabytes of Data - BigMemory<div dir="ltr" style="text-align: left;" trbidi="on"><script type="text/javascript">var dzone_url = 'http://performanceterracotta.blogspot.in/2012/05/billions-of-entries-and-terabytes-of.html';
var dzone_title = ' Billions of Entries and Terabytes of Data - BigMemory ';
var dzone_blurb = '';
var dzone_style = '2';</script><br />
<script language="javascript" src="http://widgets.dzone.com/links/widgets/zoneit.js"></script> <br />
<br />
<div class="MsoNormal"><span style="font-size: small;"><i><b>Combine BigMemory and Terracotta Server Array for Performance and Scalability</b></i></span> <br />
<br />
The age of <i style="mso-bidi-font-style: normal;"><a href="http://en.wikipedia.org/wiki/Big_data" target="_blank">Big Data</a> </i>is upon us. With ever expanding data sets, and the requirement to minimize latency as much as possible, you need a solution that offers the best in reliability and performance. Terracotta’s <a href="http://www.terracotta.org/products/bigmemory?src=performanceterracotta.blogspot.com" target="_blank">BigMemory </a>caters to the world of big data, giving your application access to literally terabytes of data, in-memory, with the highest performance and controlled latency. </div><div class="MsoNormal">At Terracotta, while working with customers and testing our releases, we continuously experiment with huge amounts of data. This blog illustrates how we were blown away by the results of a test using <a href="http://www.terracotta.org/products/bigmemory?src=performanceterracotta.blogspot.com" target="_blank">BigMemory </a>and the Terracotta Server Array (TSA) to cluster and distribute data over multiple computers. </div><h1> <span style="font-size: large;"> Test Configuration</span></h1><div class="MsoNormal">Our goal was to take four large servers, each with 1TB of RAM, and push them to the limit with <a href="http://www.terracotta.org/products/bigmemory?src=performanceterracotta.blogspot.com" target="_blank">BigMemory </a>in terms of data set size, as well as performance while reading and writing to this data set. As shown in Figure 1, all four servers were configured using the Terracotta Server Array to act as one distributed but uniform data cache. </div><div class="separator" style="clear: both; text-align: center;"><a aiotarget="false" aiotitle="" href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhHWFKsU5mOuvoVvW1jP_QQKUg23cU_WvG_rgn3GlFwSveM5IrExpGeFx0Rln1oUs5cYGe0yj9MhXEONw4FJuk-n1Y6l5-nYPqd3XDHU5fqVHcXaIgBfx8qHHFgeKdUpZJTdernlYH-tN4/s1600/cluster.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="182" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhHWFKsU5mOuvoVvW1jP_QQKUg23cU_WvG_rgn3GlFwSveM5IrExpGeFx0Rln1oUs5cYGe0yj9MhXEONw4FJuk-n1Y6l5-nYPqd3XDHU5fqVHcXaIgBfx8qHHFgeKdUpZJTdernlYH-tN4/s320/cluster.png" width="320" /></a></div><div class="MsoNormal" style="page-break-after: avoid;"><span style="font-family: Helvetica,sans-serif; font-size: 12pt; line-height: 115%;"></span></div><div class="MsoCaption" style="text-align: center;">Figure 1 - Terracotta Server Array with a ~4TB BigMemory Cache</div><div class="MsoCaption" style="text-align: center;"><br />
</div><div class="MsoNormal">We then configured 16 instances of BigMemory across the four servers in the Terracotta Server Array, where each server had 952GB of BigMemory in-memory cache allocated. This left enough free RAM available to the OS on each server. With Terracotta Server Array, you can configure large data caches with high-availability, with your choice of data striping and/or mirroring across the servers in the array (for more information on this, read here <a href="http://terracotta.org/documentation/terracotta-server-array/configuration-guide">http://terracotta.org/documentation/terracotta-server-array/configuration-guide</a> or watch this video <a href="http://blip.tv/terracotta/terracotta-server-array-striping-2865283">http://blip.tv/terracotta/terracotta-server-array-striping-2865283</a>. The end result was 3.8 terabytes of BigMemory available to our sample application for its in-memory data needs.</div><div class="MsoNormal">Next, we ran 16 instances of our test application, each on its own server, to load data and then perform <i style="mso-bidi-font-style: normal;">read</i> and <i style="mso-bidi-font-style: normal;">write</i> operations. Additionally, the Terracotta Developer Console (see Figure 2) makes it quick and simple to view and test your in-memory data performance while your application is running. Note that we could have configured BigMemory on these servers as well, thereby forming a Level 1 (L1) cache layer for even lower latency access to hot-set data. However, in this blog we decided to focus on the near-linear scalability of BigMemory as configured on the four stand-alone servers. We’ll cover L1 and L2 cache hierarchies and hot-set data in a future blog.</div><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiLlqq-eI73EcLMEOurkubdNuA7b7nxloBUk6hUH6lkrZytqsqUn5-32w2MBs0ZjIY-WbAdNWbl1Vs_QTVqZyxp7EFX1DElkg8fQUK77hAcUmglaPPl8HRSeJ6L71ecJPiP1ASBrN43-2w/s1600/ehcache-panel.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="229" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiLlqq-eI73EcLMEOurkubdNuA7b7nxloBUk6hUH6lkrZytqsqUn5-32w2MBs0ZjIY-WbAdNWbl1Vs_QTVqZyxp7EFX1DElkg8fQUK77hAcUmglaPPl8HRSeJ6L71ecJPiP1ASBrN43-2w/s320/ehcache-panel.png" width="320" /></a></div><div class="MsoNormal" style="page-break-after: avoid;"></div><div class="MsoCaption" style="text-align: center;">Figure 2 - The developer console helps configure your application's in-memory data topology.</div><h1> <span style="font-size: large;"> Data Set Size (2 billion entries; 4TB total)</span></h1><div class="MsoNormal">Now that we had our 3.8 terabyte BigMemory up and running, we loaded close to 4 terabytes of data into the Terracotta Server Array at a rate of about 450 gigabytes per hour. The sample objects loaded were data arrays, each 1700 bytes in size. To be precise, we loaded 2 billion of these entries with 200GB left over for key space.</div><div class="MsoNormal">The chart below outlines the data set configuration, as well as the test used:</div><div align="center"><table border="1" cellpadding="0" cellspacing="0" class="MsoTableLightShadingAccent5" style="border-collapse: collapse; border: none; mso-border-bottom-alt: solid #4BACC6 1.0pt; mso-border-bottom-themecolor: accent5; mso-border-top-alt: solid #4BACC6 1.0pt; mso-border-top-themecolor: accent5; mso-padding-alt: 0in 5.4pt 0in 5.4pt; mso-yfti-tbllook: 1184;"><tbody>
<tr style="mso-yfti-firstrow: yes; mso-yfti-irow: -1;"> <td style="border-bottom: solid #4BACC6 1.0pt; border-left: none; border-right: none; border-top: solid #4BACC6 1.0pt; mso-border-bottom-themecolor: accent5; mso-border-top-themecolor: accent5; padding: 0in 5.4pt 0in 5.4pt; width: 154.1pt;" valign="top" width="205"><div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0in; mso-yfti-cnfc: 5;"><b><span style="color: #31849b;">Test Configuration</span></b></div></td> <td style="border-bottom: solid #4BACC6 1.0pt; border-left: none; border-right: none; border-top: solid #4BACC6 1.0pt; mso-border-bottom-themecolor: accent5; mso-border-top-themecolor: accent5; padding: 0in 5.4pt 0in 5.4pt; width: 248.05pt;" valign="top" width="331"><div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0in; mso-yfti-cnfc: 1;"><br />
</div></td> </tr>
<tr style="mso-yfti-irow: 0;"> <td style="background: #D2EAF1; border: none; mso-background-themecolor: accent5; mso-background-themetint: 63; padding: 0in 5.4pt 0in 5.4pt; width: 154.1pt;" valign="top" width="205"><div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0in; mso-yfti-cnfc: 68;"><b><span style="color: #31849b;">Test:</span></b></div></td> <td style="background: #D2EAF1; border: none; mso-background-themecolor: accent5; mso-background-themetint: 63; padding: 0in 5.4pt 0in 5.4pt; width: 248.05pt;" valign="top" width="331"><div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0in; mso-yfti-cnfc: 64;"><span style="color: #31849b;">Offheap-test</span></div></td> </tr>
<tr style="mso-yfti-irow: 1;"> <td style="border: none; padding: 0in 5.4pt 0in 5.4pt; width: 154.1pt;" valign="top" width="205"><div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0in; mso-yfti-cnfc: 4;"><b><span style="color: #31849b;">svn url:</span></b></div></td> <td style="border: none; padding: 0in 5.4pt 0in 5.4pt; width: 248.05pt;" valign="top" width="331"><div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0in;"><span style="color: #31849b;"><a href="https://svn.terracotta.org/repo/forge/offHeap-test">https://svn.terracotta.org/repo/forge/offHeap-test</a> </span></div></td> </tr>
<tr style="mso-yfti-irow: 2;"> <td style="background: #D2EAF1; border: none; mso-background-themecolor: accent5; mso-background-themetint: 63; padding: 0in 5.4pt 0in 5.4pt; width: 154.1pt;" valign="top" width="205"><div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0in; mso-yfti-cnfc: 68;"><b><span style="color: #31849b;">Element # </span></b></div></td> <td style="background: #D2EAF1; border: none; mso-background-themecolor: accent5; mso-background-themetint: 63; padding: 0in 5.4pt 0in 5.4pt; width: 248.05pt;" valign="top" width="331"><div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0in; mso-yfti-cnfc: 64;"><span style="color: #31849b;">2 Billion</span></div></td> </tr>
<tr style="mso-yfti-irow: 3;"> <td style="border: none; padding: 0in 5.4pt 0in 5.4pt; width: 154.1pt;" valign="top" width="205"><div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0in; mso-yfti-cnfc: 4;"><b><span style="color: #31849b;">Value Size </span></b></div></td> <td style="border: none; padding: 0in 5.4pt 0in 5.4pt; width: 248.05pt;" valign="top" width="331"><div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0in;"><span style="color: #31849b;">1700 bytes (simple byte arrays)</span></div></td> </tr>
<tr style="mso-yfti-irow: 4;"> <td style="background: #D2EAF1; border: none; mso-background-themecolor: accent5; mso-background-themetint: 63; padding: 0in 5.4pt 0in 5.4pt; width: 154.1pt;" valign="top" width="205"><div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0in; mso-yfti-cnfc: 68;"><b><span style="color: #31849b;">Terracotta Server # </span></b></div></td> <td style="background: #D2EAF1; border: none; mso-background-themecolor: accent5; mso-background-themetint: 63; padding: 0in 5.4pt 0in 5.4pt; width: 248.05pt;" valign="top" width="331"><div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0in; mso-yfti-cnfc: 64;"><span style="color: #31849b;">16</span></div></td> </tr>
<tr style="mso-yfti-irow: 5;"> <td style="border: none; padding: 0in 5.4pt 0in 5.4pt; width: 154.1pt;" valign="top" width="205"><div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0in; mso-yfti-cnfc: 4;"><b><span style="color: #31849b;">Stripes # </span></b></div></td> <td style="border: none; padding: 0in 5.4pt 0in 5.4pt; width: 248.05pt;" valign="top" width="331"><div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0in;"><span style="color: #31849b;">16 (1 Terracotta Server per Mirror Group)</span></div></td> </tr>
<tr style="mso-yfti-irow: 6;"> <td style="background: #D2EAF1; border: none; mso-background-themecolor: accent5; mso-background-themetint: 63; padding: 0in 5.4pt 0in 5.4pt; width: 154.1pt;" valign="top" width="205"><div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0in; mso-yfti-cnfc: 68;"><b><span style="color: #31849b;">Application Node # </span></b></div></td> <td style="background: #D2EAF1; border: none; mso-background-themecolor: accent5; mso-background-themetint: 63; padding: 0in 5.4pt 0in 5.4pt; width: 248.05pt;" valign="top" width="331"><div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0in; mso-yfti-cnfc: 64;"><span style="color: #31849b;">16</span></div></td> </tr>
<tr style="mso-yfti-irow: 7;"> <td style="border: none; padding: 0in 5.4pt 0in 5.4pt; width: 154.1pt;" valign="top" width="205"><div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0in; mso-yfti-cnfc: 4;"><b><span style="color: #31849b;">Terracotta Server Heap </span></b></div></td> <td style="border: none; padding: 0in 5.4pt 0in 5.4pt; width: 248.05pt;" valign="top" width="331"><div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0in;"><span style="color: #31849b;">3 GB</span></div></td> </tr>
<tr style="mso-yfti-irow: 8;"> <td style="background: #D2EAF1; border: none; mso-background-themecolor: accent5; mso-background-themetint: 63; padding: 0in 5.4pt 0in 5.4pt; width: 154.1pt;" valign="top" width="205"><div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0in; mso-yfti-cnfc: 68;"><b><span style="color: #31849b;">Application Node Heap </span></b></div></td> <td style="background: #D2EAF1; border: none; mso-background-themecolor: accent5; mso-background-themetint: 63; padding: 0in 5.4pt 0in 5.4pt; width: 248.05pt;" valign="top" width="331"><div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0in; mso-yfti-cnfc: 64;"><span style="color: #31849b;">2 GB</span></div></td> </tr>
<tr style="mso-yfti-irow: 9;"> <td style="border: none; padding: 0in 5.4pt 0in 5.4pt; width: 154.1pt;" valign="top" width="205"><div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0in; mso-yfti-cnfc: 4;"><b><span style="color: #31849b;">Cache Warmup Threads </span></b></div></td> <td style="border: none; padding: 0in 5.4pt 0in 5.4pt; width: 248.05pt;" valign="top" width="331"><div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0in;"><span style="color: #31849b;">100</span></div></td> </tr>
<tr style="mso-yfti-irow: 10;"> <td style="background: #D2EAF1; border: none; mso-background-themecolor: accent5; mso-background-themetint: 63; padding: 0in 5.4pt 0in 5.4pt; width: 154.1pt;" valign="top" width="205"><div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0in; mso-yfti-cnfc: 68;"><b><span style="color: #31849b;">Test Threads </span></b></div></td> <td style="background: #D2EAF1; border: none; mso-background-themecolor: accent5; mso-background-themetint: 63; padding: 0in 5.4pt 0in 5.4pt; width: 248.05pt;" valign="top" width="331"><div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0in; mso-yfti-cnfc: 64;"><span style="color: #31849b;">100</span></div></td> </tr>
<tr style="mso-yfti-irow: 11;"> <td style="border: none; padding: 0in 5.4pt 0in 5.4pt; width: 154.1pt;" valign="top" width="205"><div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0in; mso-yfti-cnfc: 4;"><b><span style="color: #31849b;">Read Write %age</span></b></div></td> <td style="border: none; padding: 0in 5.4pt 0in 5.4pt; width: 248.05pt;" valign="top" width="331"><div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0in;"><span style="color: #31849b;">10</span></div></td> </tr>
<tr style="mso-yfti-irow: 12; mso-yfti-lastrow: yes;"> <td style="background: #D2EAF1; border-bottom: solid #4BACC6 1.0pt; border: none; mso-background-themecolor: accent5; mso-background-themetint: 63; mso-border-bottom-themecolor: accent5; padding: 0in 5.4pt 0in 5.4pt; width: 154.1pt;" valign="top" width="205"><div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0in; mso-yfti-cnfc: 68;"><b><span style="color: #31849b;">Terracotta Server BigMemory </span></b></div></td> <td style="background: #D2EAF1; border-bottom: solid #4BACC6 1.0pt; border: none; mso-background-themecolor: accent5; mso-background-themetint: 63; mso-border-bottom-themecolor: accent5; padding: 0in 5.4pt 0in 5.4pt; width: 248.05pt;" valign="top" width="331"><div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0in; mso-yfti-cnfc: 64;"><span style="color: #31849b;">238 GB/Stripe. Total: 3.8 TB</span></div></td> </tr>
</tbody></table></div><div class="MsoNormal"><br />
</div><h1> <span style="font-size: large;"> The Test Results</span></h1><div class="MsoNormal">As mentioned above, we ran 16 instances of our test application, each loading data at a rate of 4,000 transactions per second (tps), per server, reaching a total of 64,000 tps. At this rate, we were limited mostly by our network, as 64K tps of our data sample size translates to around 110 MB per second, which is almost 1 Gigabit per second (our network’s theoretical maximum). Figure 2 graphs the average latency measured while loading the data.</div><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjCYY7o638PpcFdfb6aMvg7cNrKx9EdA10-1F7N8UXBGJNpm2zeaXlyN2Xm_6ZelpQ4BWRE_fXCz8AHLPBYXCSCsoUYzjWV27tV5962y_oPhyphenhyphen5yjK4xXJCA4o0uPktgKZNtm6GBmJ__Ifw/s1600/periodic+latency.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="167" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjCYY7o638PpcFdfb6aMvg7cNrKx9EdA10-1F7N8UXBGJNpm2zeaXlyN2Xm_6ZelpQ4BWRE_fXCz8AHLPBYXCSCsoUYzjWV27tV5962y_oPhyphenhyphen5yjK4xXJCA4o0uPktgKZNtm6GBmJ__Ifw/s320/periodic+latency.png" width="320" /></a></div><div class="MsoNormal" style="page-break-after: avoid;"></div><div class="MsoCaption" style="text-align: center;">Figure 3 - Average latency, in millisecond, during the load phase.</div><div class="MsoCaption" style="text-align: center;"><br />
</div><div class="MsoNormal">The test phase consisted of operations distributed as 10% writes and 90% reads on randomly accessed keys over our data set of 2 billion entries. The chart below summarizes the incredible performance, measured in tps, of our sample application running with BigMemory.<br />
<br />
<br />
<div align="center"><table border="1" cellpadding="0" cellspacing="0" class="MsoTableLightShadingAccent5" style="border-collapse: collapse; border: none; mso-border-bottom-alt: solid #4BACC6 1.0pt; mso-border-bottom-themecolor: accent5; mso-border-top-alt: solid #4BACC6 1.0pt; mso-border-top-themecolor: accent5; mso-padding-alt: 0in 5.4pt 0in 5.4pt; mso-yfti-tbllook: 1184;"><tbody>
<tr style="mso-yfti-firstrow: yes; mso-yfti-irow: -1;"> <td style="border-bottom: solid #4BACC6 1.0pt; border-left: none; border-right: none; border-top: solid #4BACC6 1.0pt; mso-border-bottom-themecolor: accent5; mso-border-top-themecolor: accent5; padding: 0in 5.4pt 0in 5.4pt; width: 116.05pt;" valign="top" width="155"><div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0in; mso-yfti-cnfc: 5;"><b><span style="color: #31849b;">Test Results </span></b></div></td> <td style="border-bottom: solid #4BACC6 1.0pt; border-left: none; border-right: none; border-top: solid #4BACC6 1.0pt; mso-border-bottom-themecolor: accent5; mso-border-top-themecolor: accent5; padding: 0in 5.4pt 0in 5.4pt; width: 180.2pt;" valign="top" width="240"><div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0in; mso-yfti-cnfc: 1;"><br />
</div></td> </tr>
<tr style="mso-yfti-irow: 0;"> <td style="background: #D2EAF1; border: none; mso-background-themecolor: accent5; mso-background-themetint: 63; padding: 0in 5.4pt 0in 5.4pt; width: 116.05pt;" valign="top" width="155"><div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0in; mso-yfti-cnfc: 68;"><b><span style="color: #31849b;">Warmup Avg TPS </span></b></div></td> <td style="background: #D2EAF1; border: none; mso-background-themecolor: accent5; mso-background-themetint: 63; padding: 0in 5.4pt 0in 5.4pt; width: 180.2pt;" valign="top" width="240"><div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0in; mso-yfti-cnfc: 64;"><span style="color: #31849b;">4k / Application Node = 64 k total</span></div></td> </tr>
<tr style="mso-yfti-irow: 1;"> <td style="border: none; padding: 0in 5.4pt 0in 5.4pt; width: 116.05pt;" valign="top" width="155"><div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0in; mso-yfti-cnfc: 4;"><b><span style="color: #31849b;">Warmup Avg Latency </span></b></div></td> <td style="border: none; padding: 0in 5.4pt 0in 5.4pt; width: 180.2pt;" valign="top" width="240"><div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0in;"><span style="color: #31849b;">24 ms</span></div></td> </tr>
<tr style="mso-yfti-irow: 2;"> <td style="background: #D2EAF1; border: none; mso-background-themecolor: accent5; mso-background-themetint: 63; padding: 0in 5.4pt 0in 5.4pt; width: 116.05pt;" valign="top" width="155"><div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0in; mso-yfti-cnfc: 68;"><b><span style="color: #31849b;">Test Avg TPS </span></b></div></td> <td style="background: #D2EAF1; border: none; mso-background-themecolor: accent5; mso-background-themetint: 63; padding: 0in 5.4pt 0in 5.4pt; width: 180.2pt;" valign="top" width="240"><div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0in; mso-yfti-cnfc: 64;"><span style="color: #31849b;">4122 / Application Node = 66 k total</span></div></td> </tr>
<tr style="mso-yfti-irow: 3; mso-yfti-lastrow: yes;"> <td style="border-bottom: solid #4BACC6 1.0pt; border: none; mso-border-bottom-themecolor: accent5; padding: 0in 5.4pt 0in 5.4pt; width: 116.05pt;" valign="top" width="155"><div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0in; mso-yfti-cnfc: 4;"><b><span style="color: #31849b;">Test Avg Latency </span></b></div></td> <td style="border-bottom: solid #4BACC6 1.0pt; border: none; mso-border-bottom-themecolor: accent5; padding: 0in 5.4pt 0in 5.4pt; width: 180.2pt;" valign="top" width="240"><div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0in;"><span style="color: #31849b;">23 ms</span></div></td> </tr>
</tbody></table></div></div><div class="MsoNormal">We reached this level of performance with only four physical servers, each with 1 terabyte of RAM. With more servers and more RAM, we can easily scale up to 100 terabytes of data, almost linearly. This performance and scalability simply wouldn’t be possible without BigMemory and the Terracotta Server Array. </div><div class="MsoNormal"><br />
</div><div class="MsoNormal"><u><b>Check out the following links for more information:</b></u></div><div class="MsoListParagraphCxSpFirst" style="mso-list: l0 level1 lfo1; text-indent: -.25in;"><span style="font-family: Symbol;">·<span style="font-family: 'Times New Roman'; font-size: 7pt; font-style: normal; font-variant: normal; font-weight: normal; line-height: normal;"> </span></span>Terracotta Server Array: <a href="http://terracotta.org/documentation/terracotta-server-array/introduction">http://terracotta.org/documentation/terracotta-server-array/introduction</a></div><div class="MsoListParagraphCxSpLast" style="mso-list: l0 level1 lfo1; text-indent: -.25in;"><span style="font-family: Symbol;">·<span style="font-family: 'Times New Roman'; font-size: 7pt; font-style: normal; font-variant: normal; font-weight: normal; line-height: normal;"> </span></span>Terracotta BigMemory: <a href="http://terracotta.org/documentation/bigmemory/overview">http://terracotta.org/documentation/bigmemory/overview</a></div><div class="MsoNormal"><br />
</div><div class="MsoNormal"><br />
</div></div>Anonymoushttp://www.blogger.com/profile/00021983633501930224noreply@blogger.com6tag:blogger.com,1999:blog-8928181820369303270.post-71970817576573877762012-01-19T16:14:00.001+05:302012-10-06T08:23:34.095+05:30Performance Benefits of Pinned Cache<div dir="ltr" style="text-align: left;" trbidi="on"><script type="text/javascript">var dzone_url = 'http://performanceterracotta.blogspot.in/2012/01/performance-benefits-of-pinned-cache.html';
var dzone_title = 'Performance Benefits of Pinned Cache';
var dzone_blurb = '';
var dzone_style = '2';</script><br />
<script language="javascript" src="http://widgets.dzone.com/links/widgets/zoneit.js"></script> <br />
<div dir="ltr" style="text-align: left;" trbidi="on"><div dir="ltr" style="text-align: left;" trbidi="on"><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgAuqTstTsc0oeL4Yu4zLrtyUpPWj3NA6IsMGbpP_R2Khr7LeE85_TDMcp1DRTf26tkn2mVHvoIkLh9mRT6t03TSFUL5F8j82-euqS2hyphenhyphenfdNcNfWfKgFkfSFRi5KLYMeUvXmjdsEcMrtxQ/s1600/unpinned.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><br />
</a></div>Terracotta is a Tiered Data Management platform that enables applications to keep as much of it's important data as close to where it needs to be. It does so automatically and can help keep data retrieval times at the micro-second level for up to <a href="http://performanceterracotta.blogspot.com/2011/11/from-terrabytes-o-to-just-terrabytes.html" style="font-weight: bold; text-decoration: none;" target="_blank">a Terrabyte of data. </a><br />
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiZ_owP41ac3AtQXsqkNyhzmu0q4BK1IVxlz_IhKTPZ9V1Xqo6FAWPWZdwkcNv-ovcIvyyUxePjXoPHj4StLP3PjzBjWFgMA6ayU6DrFqyjf3KKyRd34Pnip89IQPnFUSXLzv5LQaqT79s/s1600/tiered.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="274" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiZ_owP41ac3AtQXsqkNyhzmu0q4BK1IVxlz_IhKTPZ9V1Xqo6FAWPWZdwkcNv-ovcIvyyUxePjXoPHj4StLP3PjzBjWFgMA6ayU6DrFqyjf3KKyRd34Pnip89IQPnFUSXLzv5LQaqT79s/s320/tiered.png" width="320" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Terracotta Server Array tiered architecture</td></tr>
</tbody></table><br />
<br />
With<a href="http://terracotta.org/products/enterprise-ehcache/automatic-resource-control" style="font-weight: bold; text-decoration: none;" target="_blank"> Automatic Resource Control</a>, we try to utilize the local cache in an efficient manner keeping the most accessed data nearest to the application. More the hits, more its locally cached. An application which accesses all the data (readonly & readwrite) equally, then ARC will try to distribute the local cache space equally to all the caches. Read more in <a href="http://performanceterracotta.blogspot.com/2011/11/efficient-ehcache-automatic-resource.html" style="font-weight: bold; text-decoration: none;" target="_blank">my last blog.</a><br />
<br />
In some applications, we are concerned about throughput (or minimum latencies) accessing one of the data set, which can be certainly achieved if we have it in local cache. Sometimes the developer/admin knows some specific things about an application that allow them to make specific performance decisions about subsets of data. The admin may know that by keeping the "US States Cache" in local heap one can keep a system running at maximum speed. <br />
<br />
With Cache Pinning, we can pin the small caches locally(which can fit in local heap) while other caches are controlled by <a href="http://terracotta.org/products/enterprise-ehcache/automatic-resource-control" style="font-weight: bold; text-decoration: none;" target="_blank">ARC</a>. The latencies for the pinned cache would be minimum at micro-second level.<br />
<br />
Here is a test which tries to simulate this kind of scenario and brings out the performance benefits of the Pinned Caches.<br />
<br />
The test has 4 caches out of which one cache is a readonly cache. Each cache loads around 250MB of data, total of 1GB data. With 512MB of local cache, ARC tries to equally distribute the resources. Since the local heap is not enough to hold all the data, we store the data at lower layer.<br />
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgAuqTstTsc0oeL4Yu4zLrtyUpPWj3NA6IsMGbpP_R2Khr7LeE85_TDMcp1DRTf26tkn2mVHvoIkLh9mRT6t03TSFUL5F8j82-euqS2hyphenhyphenfdNcNfWfKgFkfSFRi5KLYMeUvXmjdsEcMrtxQ/s1600/unpinned.png" style="margin-left: auto; margin-right: auto;"><img border="0" height="222" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgAuqTstTsc0oeL4Yu4zLrtyUpPWj3NA6IsMGbpP_R2Khr7LeE85_TDMcp1DRTf26tkn2mVHvoIkLh9mRT6t03TSFUL5F8j82-euqS2hyphenhyphenfdNcNfWfKgFkfSFRi5KLYMeUvXmjdsEcMrtxQ/s400/unpinned.png" width="400" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Period Throughput Chart - All caches are Unpinned</td></tr>
</tbody></table>The above chart shows that all the caches are getting accessed equally achieving almost same throughput. ARC is working properly but readonly cache is mostly reading from the lower layers as the some of the data is being moved to lower layer i.e. Terracotta, by other caches.<br />
<br />
After pinning the readonly cache locally, the applicaion can access the data in readonly cache super-fast way. Readwrite caches will anyway be storing the updated values in Terracotta Server Array.<br />
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a aiotarget="false" aiotitle="" href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiUkzsLoerFkj9MnlbMkPUcd9cetgP5wsrbVbhmwBMFcPdWS8MltE11OFy9u2gCiya6KumrjebPH06-Unm_9sOhG9bvQ4o7euxbBuaYlmsMkJDZER2clM0aXLkjJhQIUlRnUkF12hg7EZ8/s1600/pinned.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="221" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiUkzsLoerFkj9MnlbMkPUcd9cetgP5wsrbVbhmwBMFcPdWS8MltE11OFy9u2gCiya6KumrjebPH06-Unm_9sOhG9bvQ4o7euxbBuaYlmsMkJDZER2clM0aXLkjJhQIUlRnUkF12hg7EZ8/s400/pinned.png" width="400" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Period Throughput Chart - Readonly Pinned Cache/ReadWrite Unpinned Caches</td></tr>
</tbody></table><br />
The above chart shows that with cache pinning, readonly cache gets a huge boost from somewhere around 1000 tps is now touching 500k tps. As we keep the data closer to the application, it performs better. Now ReadOnly cache is holding up 250 MB (total data) out of total 512 MB of local heap.<br />
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiS-KZ4WWtTuFmD5wU_UcI1sitw-EDR7eq6aORPHEGSwJMqPX2CSCueyheMnDdeK58CmdhPecbnIuVu6aFl8BDckPwf_tiIBHtxHNlv7xu9F9IdJqzayAcNyLsoWpAK-bW7TVG8EnJrtrI/s1600/app.png" style="margin-left: auto; margin-right: auto;"><img border="0" height="222" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiS-KZ4WWtTuFmD5wU_UcI1sitw-EDR7eq6aORPHEGSwJMqPX2CSCueyheMnDdeK58CmdhPecbnIuVu6aFl8BDckPwf_tiIBHtxHNlv7xu9F9IdJqzayAcNyLsoWpAK-bW7TVG8EnJrtrI/s400/app.png" width="400" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Application Throughput</td></tr>
</tbody></table>The total throughput of the app also gets a huge boost with cache pinning as ReadOnly cache is performing faster with low latencies and high throughput.<br />
<br />
To enable cache pinning add following to cache config. Read more at <a href="http://ehcache.org/documentation/configuration/data-life#pinning-data" style="font-weight: bold; text-decoration: none;" target="_blank">ehcache.org.</a><br />
<blockquote class="tr_bq" style="font-family: "Courier New",Courier,monospace;"><pinning store="localmemory|localcache|incache"> </blockquote> Example: <br />
<pre style="background-image: URL(https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjFOCEBwfMO3Nyxk5xGuoJuBFSB-cINcZmnIN6HfVOmMuPNu51dqsy0T9T8AomCvRGV_xe0M4lKsg708_RAsEpRkLtBh3HCvHVO_vW0sQ4Byl44pB9ZLkvJoYAH2rXDNsGBCmQkAw_ER8iJ/s320/codebg.gif); background: #f0f0f0; border: 1px dashed #CCCCCC; color: black; font-family: arial; font-size: 12px; height: auto; line-height: 20px; overflow: auto; padding: 0px; text-align: left; width: 99%;"><code style="color: black; word-wrap: normal;">
<ehcache
maxBytesLocalHeap="300m">
<defaultCache/>
<cache
name="readonly-cache"
eternal="true">
<pinning
store="localmemory"/>
</cache>
<cache
name="readwrite-cache-2"
eternal="true">
</cache>
<cache
name="readwrite-cache-3"
eternal="true">
</cache>
<cache
name="readwrite-cache-1"
eternal="true">
</cache>
<terracottaConfig
url="localhost:9510"/>
</ehcache>
</code></pre><br />
</div></div>To download the test <a href="https://docs.google.com/open?id=0Bw08t-P-Cf42ZjBkYmRhM2MtNmMwNi00YmZhLTg2OGMtNGJiYmViYWM4MTc3">click here.</a><br />
<br />
</div>Anonymoushttp://www.blogger.com/profile/00021983633501930224noreply@blogger.com1tag:blogger.com,1999:blog-8928181820369303270.post-24822341080897377782011-11-24T12:04:00.001+05:302012-06-06T20:02:23.319+05:30From "Terrabytes :O" to "Just Terrabytes!!"<div dir="ltr" style="text-align: left;" trbidi="on">
<br />
My first computer had 64MB of RAM, since then technology has improved and got a lot cheaper. We can get Terrabytes of RAM easily.<br />
<br />
But when i talk about storing TB of data on a single java application/process,<br />
I get reaction like are you insane or what!! TB of data on Java application, it wont even start and if it gets into GC (Garbage Collection), you can go and have a coffee at Starbucks even then it wont finish.<br />
<br />
Then I say, <a href="http://www.terracotta.org/products/bigmemory?src=performanceterracotta.blogspot.com" style="font-weight: bold; text-decoration: none;" target="_blank">BigMemory</a> is the saviour you don't have to worry about GCs any more.<br />
But still can <a href="http://www.terracotta.org/products/bigmemory?src=performanceterracotta.blogspot.com" style="font-weight: bold; text-decoration: none;" target="_blank">BigMemory</a> store TBs of Data without any performance degradation?<br />
<br />
Here is my experiment, i tried loading 1 TB of data on a single JVM with <a href="http://www.terracotta.org/products/bigmemory?src=performanceterracotta.blogspot.com" style="font-weight: bold; text-decoration: none;" target="_blank">BigMemory</a>.<br />
Tried loading 1 Trillion (yes, you read it correctly its thousand times a billion, which we call as trillion) elements of around 850 bytes of payload each. Total data is ~900G, hit the hardware limitaions, but we can sure make it more than TB if hardware is available.<br />
<br />
Came across a huge box with 1TB of RAM, which made this happen. To reduce any GC issues, reduce the JVM Heap to 2G.<br />
<br />
The test create an Ehcache and loads the data onto it. ehcache configuration used for the test.<br />
<br />
<pre style="background-image: URL(https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjFOCEBwfMO3Nyxk5xGuoJuBFSB-cINcZmnIN6HfVOmMuPNu51dqsy0T9T8AomCvRGV_xe0M4lKsg708_RAsEpRkLtBh3HCvHVO_vW0sQ4Byl44pB9ZLkvJoYAH2rXDNsGBCmQkAw_ER8iJ/s320/codebg.gif); background: #f0f0f0; border: 1px dashed #CCCCCC; color: black; font-family: arial; font-size: 12px; height: auto; line-height: 20px; overflow: auto; padding: 0px; text-align: left; width: 99%;"><code style="color: black; word-wrap: normal;"><ehcache
name="cacheManagerName_0"
maxBytesLocalOffHeap="990g">
<cache
name="mgr_0_cache_0"
maxEntriesLocalHeap="3000"
overflowToOffHeap="true"/>
</ehcache>
</code></pre>
<br />
Here is the graph of period (=4 secs) warmup thoughput over time. Secondary Axis of the chart show the total data stored.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjFYdoyXtQPtgs8_GD4kHRkEhGPqo0bKWhcOWDRPWFrrk979iLAP5Apnkeq6rM7HXqernlt3v5zduhSwq9mDqL5IvzuWUZgnABV8Y-_UPmfMo6CiCY-vYhKbnAMlE17zcqBO8wjFYY59gQ/s1600/Warmup-TPS.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="179" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjFYdoyXtQPtgs8_GD4kHRkEhGPqo0bKWhcOWDRPWFrrk979iLAP5Apnkeq6rM7HXqernlt3v5zduhSwq9mDqL5IvzuWUZgnABV8Y-_UPmfMo6CiCY-vYhKbnAMlE17zcqBO8wjFYY59gQ/s320/Warmup-TPS.png" width="320" /></a></div>
<br />
<br />
There are few slight dips in the chart these are when <a href="http://www.terracotta.org/products/bigmemory?src=performanceterracotta.blogspot.com" style="font-weight: bold; text-decoration: none;" target="_blank">BigMemory</a> is expanding to store more data. The Throughput is above 200,000 all the time with an average of 350,000 txns/sec.<br />
<br />
The latencies are also a big concern for the applications.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgv_zKQPCBnGz_KQYPoYb7hYRwaye5ooRtEDpy1JSHngXmWRpNYuynGuBXp2BPiMOMdsnvLIYPrew58aQq0HloiuU-HtvmOOuyA72PDQao0a-4_EfYqluBL-dRUpAoryYkCwNj_aeiwA5E/s1600/Warmup-Latency.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="179" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgv_zKQPCBnGz_KQYPoYb7hYRwaye5ooRtEDpy1JSHngXmWRpNYuynGuBXp2BPiMOMdsnvLIYPrew58aQq0HloiuU-HtvmOOuyA72PDQao0a-4_EfYqluBL-dRUpAoryYkCwNj_aeiwA5E/s320/Warmup-Latency.png" width="320" /></a></div>
<br />
Its below a 1 ms and average being 300 <span style="background-color: white; color: #222222; font-family: arial,sans-serif; line-height: 19px;">µs.</span><br />
<br />
Okay, we have loaded TB of data, now what. Does it even work?<br />
<br />
Yes, it does. The test phase does read write operation over the data set. Randomly selects an elements updates it and put it back to the cache.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi_5WcJvVZaZT8A5SSCnUb4Olvs4Um-NzMYAKITL0IRqtgvew95-fR4-W1Fxq3KW5r09wC9xyFb7AZoAmeylz6w2FOjg1wWOv9JlbEOlamoNji33CjAWLzjw0Tt10gicjiVvQMtpJYxIpw/s1600/test-tps.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="178" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi_5WcJvVZaZT8A5SSCnUb4Olvs4Um-NzMYAKITL0IRqtgvew95-fR4-W1Fxq3KW5r09wC9xyFb7AZoAmeylz6w2FOjg1wWOv9JlbEOlamoNji33CjAWLzjw0Tt10gicjiVvQMtpJYxIpw/s320/test-tps.png" width="320" /></a></div>
<div class="separator" style="clear: both; text-align: center;">
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEijoAFOeNBasKNJkKB-HcEbV0l31fkdHwfSw5FSy3sUaSccOf1Cj0Toaoy_dgGpLEwS5hfMPK6SUdI9vt7aTDoFdAXoRrYFxRcSBbKQcGVR9GfVjRW180aB22rGPgw96f0SSc7CmKuzgjw/s1600/Test-Latency.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="178" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEijoAFOeNBasKNJkKB-HcEbV0l31fkdHwfSw5FSy3sUaSccOf1Cj0Toaoy_dgGpLEwS5hfMPK6SUdI9vt7aTDoFdAXoRrYFxRcSBbKQcGVR9GfVjRW180aB22rGPgw96f0SSc7CmKuzgjw/s320/Test-Latency.png" width="320" /></a></div>
<br />
<br />
I will say throughput and latencies are not that bad :)<br />
The spikes are due to JVM GC, even with 2GB heap we will have few GCs, but the pause time is not even 2 secs. So we get the max latency for the test to be around 2 secs but the 99 percentile is around 500 <span style="background-color: white; color: #222222; font-family: arial,sans-serif; line-height: 19px;">µs</span><br />
<br />
So if you application is slowed down by database or you are spending thousands of dollars on maintaining a database.<br />
Get <a href="http://www.terracotta.org/products/bigmemory?src=performanceterracotta.blogspot.com" style="font-weight: bold; text-decoration: none;" target="_blank">BigMemory</a> and offload your database!<br />
<br />
There would be concerns about searching this huge data, we have <a href="http://ehcache.org/documentation/apis/search" style="font-weight: bold; text-decoration: none;" target="_blank">ehcache search</a> that makes it happen.<br />
<br />
<br />
<br /></div>Anonymoushttp://www.blogger.com/profile/00021983633501930224noreply@blogger.com2tag:blogger.com,1999:blog-8928181820369303270.post-71267025511954730172011-11-04T20:16:00.001+05:302011-11-04T20:20:01.475+05:30New efficient Ehcache Automatic Resource Control (ARC)<div dir="ltr" style="text-align: left;" trbidi="on">
To speed up an application, the most common technique used is caching and <a href="http://www.ehcache.org/?src=performanceterracotta.blogspot.com" style="font-weight: bold; text-decoration: none;" target="_blank">Ehcache</a> is most commonly used in Java world. <a href="http://www.terracotta.org/products/bigmemory?src=performanceterracotta.blogspot.com" style="font-weight: bold; text-decoration: none;" target="_blank">BigMemory</a> <a href="http://www.ehcache.org/?src=performanceterracotta.blogspot.com" style="font-weight: bold; text-decoration: none;" target="_blank">Ehcache</a> with <a href="http://www.terracotta.org/?src=performanceterracotta.blogspot.com" style="font-weight: bold; text-decoration: none;" target="_blank">Terracotta </a>can cache terrabytes of data without any GC issues, bringing data closer to the clustered application in efficient manner. <a href="http://www.terracotta.org/?src=performanceterracotta.blogspot.com" style="font-weight: bold; text-decoration: none;" target="_blank">Terracotta </a>Server stores whole data in <a href="http://www.terracotta.org/products/bigmemory?src=performanceterracotta.blogspot.com" style="font-weight: bold; text-decoration: none;" target="_blank">BigMemory</a> and provides the required data to the clustered application. With new feature, <a href="http://www.terracotta.org/products/bigmemory?src=performanceterracotta.blogspot.com" style="font-weight: bold; text-decoration: none;" target="_blank">BigMemory</a> at application level (<a href="http://www.terracotta.org/?src=performanceterracotta.blogspot.com" style="font-weight: bold; text-decoration: none;" target="_blank">Terracotta </a>client or Layer1/L1), we are bringing cached data more closer to the application.<br />
<br />
In an application, we might need to cache different types data in different caches. At some point of time, we would be using one of the caches heavily with few hits to other caches. Before <a href="http://www.terracotta.org/?src=performanceterracotta.blogspot.com" style="font-weight: bold; text-decoration: none;" target="_blank">Terracotta </a>3.6.0 release, we can specify the number of elements to be cached per cache, but with limited heap/<a href="http://www.terracotta.org/products/bigmemory?src=performanceterracotta.blogspot.com" style="font-weight: bold; text-decoration: none;" target="_blank">BigMemory</a> it gets tough to allocate space for each cache in efficient manner. The memory allocated to each cache is fixed so even if its not used much, the data will reside with application.<br />
<br />
With the new feature, Automatic Resource Control (ARC), <a href="http://www.ehcache.org/?src=performanceterracotta.blogspot.com" style="font-weight: bold; text-decoration: none;" target="_blank">Ehcache</a> will manage the heap/<a href="http://www.terracotta.org/products/bigmemory?src=performanceterracotta.blogspot.com" style="font-weight: bold; text-decoration: none;" target="_blank">BigMemory</a> allocated depending on the cache usage. With increased usage of cache_0, <a href="http://www.ehcache.org/?src=performanceterracotta.blogspot.com" style="font-weight: bold; text-decoration: none;" target="_blank">Ehcache</a> will try to allocate more memory to cache_0 and as usage of cache_1 increase, it tries to manage the space between both the caches.<br />
<br />
<span style="font-weight: bold; text-decoration: none;">Test Case</span><br />
<br />
Here is a small test which creates two caches, loads up the data to both the caches. During the test phase, threads access both the caches, say cache_0 & cache_1, but for one of the caches, cache_0, introduced a small delay after each transaction reducing the hits to the cache.<br />
<br />
Both cache can store 1.5GB of data, total of 3GB of data, while <a href="http://www.terracotta.org/products/bigmemory?src=performanceterracotta.blogspot.com" style="font-weight: bold; text-decoration: none;" target="_blank">BigMemory</a> allocated at the application level is 2GB only. Its enough to store all cached data for one of the cache but not both.<br />
<br />
<span style="font-weight: bold; text-decoration: none;">Access Pattern</span><br />
<ol>
<li>2 mins, both the caches are being used </li>
<li>Next 10 mins cache_0 is being used more often </li>
<li>Again for 2 mins both the caches are being used</li>
<li>Next 10 mins cache_1 is used heavily</li>
<li>Repeat</li>
</ol>
<br />
First, will like to discuss the case without ARC.<br />
<br />
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi-sG5wq_FxzKYChJCzI6Vk2Bwc7efLIBIN9s4F19QAqai_8K8Vq6Yn3KhzWVbQurgldCoEu6OZ7OWyb_c4CmOMPzKy04q66xP2fqtH45RD9mUgbbbOU8WuCnmlwGysUgVf4DwzquFlVlw/s1600/TPS-No-ARC.png"><img alt="" border="0" id="BLOGGER_PHOTO_ID_5671144276066670466" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi-sG5wq_FxzKYChJCzI6Vk2Bwc7efLIBIN9s4F19QAqai_8K8Vq6Yn3KhzWVbQurgldCoEu6OZ7OWyb_c4CmOMPzKy04q66xP2fqtH45RD9mUgbbbOU8WuCnmlwGysUgVf4DwzquFlVlw/s400/TPS-No-ARC.png" style="cursor: hand; cursor: pointer; display: block; height: 210px; margin: 0px auto 10px; text-align: center; width: 400px;" /></a><br />
<br />
<br />
The tps remains almost constant even if other cache is not being used. We can see if both the caches are being used then also application throughput is same.<br />
<br />
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhmtxDeqCAltZqsPWRJ2i5NypYqcLYP0s9yt5Q1XOEyvMGPipQ2vUwm9BaERJezEoxGW6Ml6Al1chTD9iuQnwobeD3sivLZqDEy83oNx8ET2iZinb2VEOCwJRMh8My53jGXCSxIW_YWhnk/s1600/L1BM-No-ARC.png"><img alt="" border="0" id="BLOGGER_PHOTO_ID_5671144300406695458" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhmtxDeqCAltZqsPWRJ2i5NypYqcLYP0s9yt5Q1XOEyvMGPipQ2vUwm9BaERJezEoxGW6Ml6Al1chTD9iuQnwobeD3sivLZqDEy83oNx8ET2iZinb2VEOCwJRMh8My53jGXCSxIW_YWhnk/s400/L1BM-No-ARC.png" style="cursor: hand; cursor: pointer; display: block; height: 210px; margin: 0px auto 10px; text-align: center; width: 400px;" /></a><br />
<br />
The L1 <a href="http://www.terracotta.org/products/bigmemory?src=performanceterracotta.blogspot.com" style="font-weight: bold; text-decoration: none;" target="_blank">BigMemory</a> usage remains constant throughout the test.<br />
<br />
Now we should praise the benefits of ARC<br />
<br />
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhCzI89UEazXhxf_6Cp6qRfSnFwi9quS56JkDTQLv7r5DMW9Uu7kuFU9RTB3h6dli3Oze-UWP0Aoj_VZ4mxUBTIeWm7N4YHH57pfBukwgXSJpdbi0D_tP-8K3DEs4TTWwC-JgKCIP40vBA/s1600/TPS-ARC.png"><img alt="" border="0" id="BLOGGER_PHOTO_ID_5671144273861612978" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhCzI89UEazXhxf_6Cp6qRfSnFwi9quS56JkDTQLv7r5DMW9Uu7kuFU9RTB3h6dli3Oze-UWP0Aoj_VZ4mxUBTIeWm7N4YHH57pfBukwgXSJpdbi0D_tP-8K3DEs4TTWwC-JgKCIP40vBA/s400/TPS-ARC.png" style="cursor: hand; cursor: pointer; display: block; height: 210px; margin: 0px auto 10px; text-align: center; width: 400px;" /></a><br />
<br />
<br />
Now with ARC, we can see if full cached data is at application <a href="http://www.terracotta.org/products/bigmemory?src=performanceterracotta.blogspot.com" style="font-weight: bold; text-decoration: none;" target="_blank">BigMemory</a>, the throughput gets a boost and touches 140k txn/sec. With both caches being used the tps is almost same as with and without ARC.<br />
<br />
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgsNhbWsJ3AoSxqXt2M5GyG22_vEYCEB9qSvm_dJdeAznzDdSqmrx0jgABDFgxGmdMthUrZGCNRhxyThVcfI66N_wLgIX9gUfElDtL_ragGJ2zg-l6Q4IZWKl35ejcFLGVaqAy6pRcxS-g/s1600/L1BM-ARC.png"><img alt="" border="0" id="BLOGGER_PHOTO_ID_5671144288245420754" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgsNhbWsJ3AoSxqXt2M5GyG22_vEYCEB9qSvm_dJdeAznzDdSqmrx0jgABDFgxGmdMthUrZGCNRhxyThVcfI66N_wLgIX9gUfElDtL_ragGJ2zg-l6Q4IZWKl35ejcFLGVaqAy6pRcxS-g/s400/L1BM-ARC.png" style="cursor: hand; cursor: pointer; display: block; height: 210px; margin: 0px auto 10px; text-align: center; width: 400px;" /></a><br />
<br />
The throughput variation can be understood by the graph of L1 <a href="http://www.terracotta.org/products/bigmemory?src=performanceterracotta.blogspot.com" style="font-weight: bold; text-decoration: none;" target="_blank">BigMemory</a> usage. We can see that the L1 <a href="http://www.terracotta.org/products/bigmemory?src=performanceterracotta.blogspot.com" style="font-weight: bold; text-decoration: none;" target="_blank">BigMemory</a> usage for the cache_0 increasing, as it is heavily used. Overtime, it uses most of the memory for cache_0. As cache_1 usage increases, the memory usage for it also increases giving boost to the throughput.<br />
<br />
To enable ARC, we just need to provide maxBytesLocalOffheap at Cache Manager level.<br />
<br />
here is a sample ehcache.xml<br />
<br />
<pre style="background-image: URL(https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjFOCEBwfMO3Nyxk5xGuoJuBFSB-cINcZmnIN6HfVOmMuPNu51dqsy0T9T8AomCvRGV_xe0M4lKsg708_RAsEpRkLtBh3HCvHVO_vW0sQ4Byl44pB9ZLkvJoYAH2rXDNsGBCmQkAw_ER8iJ/s320/codebg.gif); background: #f0f0f0; border: 1px dashed #CCCCCC; color: black; font-family: arial; font-size: 12px; height: auto; line-height: 20px; overflow: auto; padding: 0px; text-align: left; width: 99%;"><code style="color: black; word-wrap: normal;"> <ehcache
name="cacheManagerName_0"
maxBytesLocalHeap="512m"
maxBytesLocalOffHeap="2g">
<defaultCache/>
<cache
name="cache_1"
overflowToOffHeap="true">
<terracotta/>
</cache>
<cache
name="cache_0"
overflowToOffHeap="true">
<terracotta/>
</cache>
<terracottaConfig
url="localhost:9510"/>
</ehcache>
</code></pre>
<br />
This is 1 client attached to Terracotta Server to keep the testcase simpler. With multiple nodes, it will bring out the benefits more. :)<br />
<br />
The above testcase is with only two caches, just picture if we have 10s of caches and tuning each cache would be a problem. With <a href="http://www.ehcache.org/?src=performanceterracotta.blogspot.com" style="font-weight: bold; text-decoration: none;" target="_blank">Ehcache</a> ARC, its <a href="http://www.ehcache.org/?src=performanceterracotta.blogspot.com" style="font-weight: bold; text-decoration: none;" target="_blank">Ehcache</a> responsibility to manage the data efficiently giving maximum throughput out of the application.</div>Anonymoushttp://www.blogger.com/profile/00021983633501930224noreply@blogger.com0tag:blogger.com,1999:blog-8928181820369303270.post-5077361655836092942010-11-09T13:13:00.016+05:302010-11-24T09:50:46.028+05:30"Terracotta Fairy" brings BigMemory for Java usersI used to discuss about the Java goodness with my friends working on native platforms. But they always used to crib about Java., its slower, max latency is high, Garbage Collection(GC) ruins the user experience and tuning GC is a NIGHTMARE! Didn't had anything to defend on these points :(, as its the fact.<br /><blockquote style="font-weight: bold; font-style: italic; color: rgb(255, 204, 0);">Garbage Collection kills the Java Application.</blockquote><br />Being in Java world, loading 100G of data on java heap sounds crazy. I did some experiments to load 100G of data on single JVM. Even for Read-only case (no writes/updates to reduce the GC problems), it wont fit in 150G of heap.<br /><br />On tuning GC, reducing Young Gen Space, reducing Survivor Ratio, etc, didnt helped much. The test just gets into back-to-back full GCs, killing the application throughput and latency. To get it working had to give 200 GB of heap and it performed so badly.<br /><br />Then I wished, wont it be really nice to fit whole data, without any GC problems. <span style="font-weight: bold;">"</span><a style="font-weight: bold;" href="http://terracotta.org/" target="_blank">Terracotta</a> <span style="font-weight: bold;">Fairy</span>" listened to us and here we have <a href="http://www.terracotta.org/bigmemory?src=performanceterracotta.blogspot.com" target="_blank">BigMemory Ehcache</a>. BigMemory is a GC murder weapon from <a href="http://terracotta.org/" target="_blank">Terracotta</a>, like an AA12 ShotGun.<br /><blockquote style="font-weight: bold; font-style: italic; color: rgb(255, 204, 0);">Now, we can store 350GB of data with no GC .<br />Can you believe this !! Literally NO GC!!</blockquote><br /><span style="font-weight: bold; font-style: italic;">Wanna see it with your eyes, here are the charts from the battle of Troy - On-Heap vs BigMemory.</span><br /><br />The following charts shows the <a href="http://ehcache.org/" target="_blank">Ehcache </a>use case which I thought would be fair enough. Ehcache being most widely used Java cache already outperforms other available cache. Didn't wanted to choose best use case for BigMemory i.e. 100% writes NOR the best for On-Heap i.e. Read-only. The read/write ratio is 50% reads and 50% writes. The hot set is that 90% of the time cache.get() will access 10% of the key set. This is representative of the the familiar Pareto distribution that is very commonly observed. The Test loads up full data in the <a href="http://ehcache.org/" target="_blank">Ehcache </a>and then starts doing r/w operations on it.<br /><br />These test machine was a Cisco UCS box running with Intel(R) Xeon(R) Processors. It had 6 quad-core 2.93Ghz Xeon(R) cpus for a total of 24 cores, with 378GB (<!--[if gte mso 9]><xml> <w:worddocument> <w:view>Normal</w:View> <w:zoom>0</w:Zoom> <w:trackmoves/> <w:trackformatting/> <w:punctuationkerning/> <w:validateagainstschemas/> <w:saveifxmlinvalid>false</w:SaveIfXMLInvalid> <w:ignoremixedcontent>false</w:IgnoreMixedContent> <w:alwaysshowplaceholdertext>false</w:AlwaysShowPlaceholderText> <w:donotpromoteqf/> <w:lidthemeother>EN-US</w:LidThemeOther> <w:lidthemeasian>X-NONE</w:LidThemeAsian> <w:lidthemecomplexscript>X-NONE</w:LidThemeComplexScript> <w:compatibility> <w:breakwrappedtables/> <w:snaptogridincell/> <w:wraptextwithpunct/> <w:useasianbreakrules/> <w:dontgrowautofit/> <w:splitpgbreakandparamark/> <w:enableopentypekerning/> <w:dontflipmirrorindents/> <w:overridetablestylehps/> </w:Compatibility> <w:browserlevel>MicrosoftInternetExplorer4</w:BrowserLevel> <m:mathpr> <m:mathfont val="Cambria Math"> <m:brkbin val="before"> <m:brkbinsub val="--"> <m:smallfrac val="off"> <m:dispdef/> <m:lmargin val="0"> <m:rmargin val="0"> <m:defjc val="centerGroup"> <m:wrapindent val="1440"> <m:intlim val="subSup"> <m:narylim val="undOvr"> </m:mathPr></w:WordDocument> </xml><![endif]--><!--[if gte mso 9]><xml> <w:latentstyles deflockedstate="false" defunhidewhenused="true" defsemihidden="true" defqformat="false" defpriority="99" latentstylecount="267"> <w:lsdexception locked="false" priority="0" semihidden="false" unhidewhenused="false" qformat="true" name="Normal"> <w:lsdexception locked="false" priority="9" semihidden="false" unhidewhenused="false" qformat="true" name="heading 1"> <w:lsdexception locked="false" priority="9" qformat="true" name="heading 2"> <w:lsdexception locked="false" priority="9" qformat="true" name="heading 3"> <w:lsdexception locked="false" priority="9" qformat="true" name="heading 4"> <w:lsdexception locked="false" priority="9" qformat="true" name="heading 5"> <w:lsdexception locked="false" priority="9" qformat="true" name="heading 6"> <w:lsdexception locked="false" priority="9" qformat="true" name="heading 7"> <w:lsdexception locked="false" priority="9" qformat="true" name="heading 8"> <w:lsdexception locked="false" priority="9" qformat="true" name="heading 9"> <w:lsdexception locked="false" priority="39" name="toc 1"> <w:lsdexception locked="false" priority="39" name="toc 2"> <w:lsdexception locked="false" priority="39" name="toc 3"> <w:lsdexception locked="false" priority="39" name="toc 4"> <w:lsdexception locked="false" priority="39" name="toc 5"> <w:lsdexception locked="false" priority="39" name="toc 6"> <w:lsdexception locked="false" priority="39" name="toc 7"> <w:lsdexception locked="false" priority="39" name="toc 8"> <w:lsdexception locked="false" priority="39" name="toc 9"> <w:lsdexception locked="false" priority="35" qformat="true" name="caption"> <w:lsdexception locked="false" priority="10" semihidden="false" unhidewhenused="false" qformat="true" name="Title"> <w:lsdexception locked="false" priority="1" name="Default Paragraph Font"> <w:lsdexception locked="false" priority="11" semihidden="false" unhidewhenused="false" qformat="true" name="Subtitle"> <w:lsdexception locked="false" priority="22" semihidden="false" unhidewhenused="false" qformat="true" name="Strong"> <w:lsdexception locked="false" priority="20" semihidden="false" unhidewhenused="false" qformat="true" name="Emphasis"> <w:lsdexception locked="false" priority="59" semihidden="false" unhidewhenused="false" name="Table Grid"> <w:lsdexception locked="false" unhidewhenused="false" name="Placeholder Text"> <w:lsdexception locked="false" priority="1" semihidden="false" unhidewhenused="false" qformat="true" name="No Spacing"> <w:lsdexception locked="false" priority="60" semihidden="false" unhidewhenused="false" name="Light Shading"> <w:lsdexception locked="false" priority="61" semihidden="false" unhidewhenused="false" name="Light List"> <w:lsdexception locked="false" priority="62" semihidden="false" unhidewhenused="false" name="Light Grid"> <w:lsdexception locked="false" priority="63" semihidden="false" unhidewhenused="false" name="Medium Shading 1"> <w:lsdexception locked="false" priority="64" semihidden="false" unhidewhenused="false" name="Medium Shading 2"> <w:lsdexception locked="false" priority="65" semihidden="false" unhidewhenused="false" name="Medium List 1"> <w:lsdexception locked="false" priority="66" semihidden="false" unhidewhenused="false" name="Medium List 2"> <w:lsdexception locked="false" priority="67" semihidden="false" unhidewhenused="false" name="Medium Grid 1"> <w:lsdexception locked="false" priority="68" semihidden="false" unhidewhenused="false" name="Medium Grid 2"> <w:lsdexception locked="false" priority="69" semihidden="false" unhidewhenused="false" name="Medium Grid 3"> <w:lsdexception locked="false" priority="70" semihidden="false" unhidewhenused="false" name="Dark List"> <w:lsdexception locked="false" priority="71" semihidden="false" unhidewhenused="false" name="Colorful Shading"> <w:lsdexception locked="false" priority="72" semihidden="false" unhidewhenused="false" name="Colorful List"> <w:lsdexception locked="false" priority="73" semihidden="false" unhidewhenused="false" name="Colorful Grid"> <w:lsdexception locked="false" priority="60" semihidden="false" unhidewhenused="false" name="Light Shading Accent 1"> <w:lsdexception locked="false" priority="61" semihidden="false" unhidewhenused="false" name="Light List Accent 1"> <w:lsdexception locked="false" priority="62" semihidden="false" unhidewhenused="false" name="Light Grid Accent 1"> <w:lsdexception locked="false" priority="63" semihidden="false" unhidewhenused="false" name="Medium Shading 1 Accent 1"> <w:lsdexception locked="false" priority="64" semihidden="false" unhidewhenused="false" name="Medium Shading 2 Accent 1"> <w:lsdexception locked="false" priority="65" semihidden="false" unhidewhenused="false" name="Medium List 1 Accent 1"> <w:lsdexception locked="false" unhidewhenused="false" name="Revision"> <w:lsdexception locked="false" priority="34" semihidden="false" unhidewhenused="false" qformat="true" name="List Paragraph"> <w:lsdexception locked="false" priority="29" semihidden="false" unhidewhenused="false" qformat="true" name="Quote"> <w:lsdexception locked="false" priority="30" semihidden="false" unhidewhenused="false" qformat="true" name="Intense Quote"> <w:lsdexception locked="false" priority="66" semihidden="false" unhidewhenused="false" name="Medium List 2 Accent 1"> <w:lsdexception locked="false" priority="67" semihidden="false" unhidewhenused="false" name="Medium Grid 1 Accent 1"> <w:lsdexception locked="false" priority="68" semihidden="false" unhidewhenused="false" name="Medium Grid 2 Accent 1"> <w:lsdexception locked="false" priority="69" semihidden="false" unhidewhenused="false" name="Medium Grid 3 Accent 1"> <w:lsdexception locked="false" priority="70" semihidden="false" unhidewhenused="false" name="Dark List Accent 1"> <w:lsdexception locked="false" priority="71" semihidden="false" unhidewhenused="false" name="Colorful Shading Accent 1"> <w:lsdexception locked="false" priority="72" semihidden="false" unhidewhenused="false" name="Colorful List Accent 1"> <w:lsdexception locked="false" priority="73" semihidden="false" unhidewhenused="false" name="Colorful Grid Accent 1"> <w:lsdexception locked="false" priority="60" semihidden="false" unhidewhenused="false" name="Light Shading Accent 2"> <w:lsdexception locked="false" priority="61" semihidden="false" unhidewhenused="false" name="Light List Accent 2"> <w:lsdexception locked="false" priority="62" semihidden="false" unhidewhenused="false" name="Light Grid Accent 2"> <w:lsdexception locked="false" priority="63" semihidden="false" unhidewhenused="false" name="Medium Shading 1 Accent 2"> <w:lsdexception locked="false" priority="64" semihidden="false" unhidewhenused="false" name="Medium Shading 2 Accent 2"> <w:lsdexception locked="false" priority="65" semihidden="false" unhidewhenused="false" name="Medium List 1 Accent 2"> <w:lsdexception locked="false" priority="66" semihidden="false" unhidewhenused="false" name="Medium List 2 Accent 2"> <w:lsdexception locked="false" priority="67" semihidden="false" unhidewhenused="false" name="Medium Grid 1 Accent 2"> <w:lsdexception locked="false" priority="68" semihidden="false" unhidewhenused="false" name="Medium Grid 2 Accent 2"> <w:lsdexception locked="false" priority="69" semihidden="false" unhidewhenused="false" name="Medium Grid 3 Accent 2"> <w:lsdexception locked="false" priority="70" semihidden="false" unhidewhenused="false" name="Dark List Accent 2"> <w:lsdexception locked="false" priority="71" semihidden="false" unhidewhenused="false" name="Colorful Shading Accent 2"> <w:lsdexception locked="false" priority="72" semihidden="false" unhidewhenused="false" name="Colorful List Accent 2"> <w:lsdexception locked="false" priority="73" semihidden="false" unhidewhenused="false" name="Colorful Grid Accent 2"> <w:lsdexception locked="false" priority="60" semihidden="false" unhidewhenused="false" name="Light Shading Accent 3"> <w:lsdexception locked="false" priority="61" semihidden="false" unhidewhenused="false" name="Light List Accent 3"> <w:lsdexception locked="false" priority="62" semihidden="false" unhidewhenused="false" name="Light Grid Accent 3"> <w:lsdexception locked="false" priority="63" semihidden="false" unhidewhenused="false" name="Medium Shading 1 Accent 3"> <w:lsdexception locked="false" priority="64" semihidden="false" unhidewhenused="false" name="Medium Shading 2 Accent 3"> <w:lsdexception locked="false" priority="65" semihidden="false" unhidewhenused="false" name="Medium List 1 Accent 3"> <w:lsdexception locked="false" priority="66" semihidden="false" unhidewhenused="false" name="Medium List 2 Accent 3"> <w:lsdexception locked="false" priority="67" semihidden="false" unhidewhenused="false" name="Medium Grid 1 Accent 3"> <w:lsdexception locked="false" priority="68" semihidden="false" unhidewhenused="false" name="Medium Grid 2 Accent 3"> <w:lsdexception locked="false" priority="69" semihidden="false" unhidewhenused="false" name="Medium Grid 3 Accent 3"> <w:lsdexception locked="false" priority="70" semihidden="false" unhidewhenused="false" name="Dark List Accent 3"> <w:lsdexception locked="false" priority="71" semihidden="false" unhidewhenused="false" name="Colorful Shading Accent 3"> <w:lsdexception locked="false" priority="72" semihidden="false" unhidewhenused="false" name="Colorful List Accent 3"> <w:lsdexception locked="false" priority="73" semihidden="false" unhidewhenused="false" name="Colorful Grid Accent 3"> <w:lsdexception locked="false" priority="60" semihidden="false" unhidewhenused="false" name="Light Shading Accent 4"> <w:lsdexception locked="false" priority="61" semihidden="false" unhidewhenused="false" name="Light List Accent 4"> <w:lsdexception locked="false" priority="62" semihidden="false" unhidewhenused="false" name="Light Grid Accent 4"> <w:lsdexception locked="false" priority="63" semihidden="false" unhidewhenused="false" name="Medium Shading 1 Accent 4"> <w:lsdexception locked="false" priority="64" semihidden="false" unhidewhenused="false" name="Medium Shading 2 Accent 4"> <w:lsdexception locked="false" priority="65" semihidden="false" unhidewhenused="false" name="Medium List 1 Accent 4"> <w:lsdexception locked="false" priority="66" semihidden="false" unhidewhenused="false" name="Medium List 2 Accent 4"> <w:lsdexception locked="false" priority="67" semihidden="false" unhidewhenused="false" name="Medium Grid 1 Accent 4"> <w:lsdexception locked="false" priority="68" semihidden="false" unhidewhenused="false" name="Medium Grid 2 Accent 4"> <w:lsdexception locked="false" priority="69" semihidden="false" unhidewhenused="false" name="Medium Grid 3 Accent 4"> <w:lsdexception locked="false" priority="70" semihidden="false" unhidewhenused="false" name="Dark List Accent 4"> <w:lsdexception locked="false" priority="71" semihidden="false" unhidewhenused="false" name="Colorful Shading Accent 4"> <w:lsdexception locked="false" priority="72" semihidden="false" unhidewhenused="false" name="Colorful List Accent 4"> <w:lsdexception locked="false" priority="73" semihidden="false" unhidewhenused="false" name="Colorful Grid Accent 4"> <w:lsdexception locked="false" priority="60" semihidden="false" unhidewhenused="false" name="Light Shading Accent 5"> <w:lsdexception locked="false" priority="61" semihidden="false" unhidewhenused="false" name="Light List Accent 5"> <w:lsdexception locked="false" priority="62" semihidden="false" unhidewhenused="false" name="Light Grid Accent 5"> <w:lsdexception locked="false" priority="63" semihidden="false" unhidewhenused="false" name="Medium Shading 1 Accent 5"> <w:lsdexception locked="false" priority="64" semihidden="false" unhidewhenused="false" name="Medium Shading 2 Accent 5"> <w:lsdexception locked="false" priority="65" semihidden="false" unhidewhenused="false" name="Medium List 1 Accent 5"> <w:lsdexception locked="false" priority="66" semihidden="false" unhidewhenused="false" name="Medium List 2 Accent 5"> <w:lsdexception locked="false" priority="67" semihidden="false" unhidewhenused="false" name="Medium Grid 1 Accent 5"> <w:lsdexception locked="false" priority="68" semihidden="false" unhidewhenused="false" name="Medium Grid 2 Accent 5"> <w:lsdexception locked="false" priority="69" semihidden="false" unhidewhenused="false" name="Medium Grid 3 Accent 5"> <w:lsdexception locked="false" priority="70" semihidden="false" unhidewhenused="false" name="Dark List Accent 5"> <w:lsdexception locked="false" priority="71" semihidden="false" unhidewhenused="false" name="Colorful Shading Accent 5"> <w:lsdexception locked="false" priority="72" semihidden="false" unhidewhenused="false" name="Colorful List Accent 5"> <w:lsdexception locked="false" priority="73" semihidden="false" unhidewhenused="false" name="Colorful Grid Accent 5"> <w:lsdexception locked="false" priority="60" semihidden="false" unhidewhenused="false" name="Light Shading Accent 6"> <w:lsdexception locked="false" priority="61" semihidden="false" unhidewhenused="false" name="Light List Accent 6"> <w:lsdexception locked="false" priority="62" semihidden="false" unhidewhenused="false" name="Light Grid Accent 6"> <w:lsdexception locked="false" priority="63" semihidden="false" unhidewhenused="false" name="Medium Shading 1 Accent 6"> <w:lsdexception locked="false" priority="64" semihidden="false" unhidewhenused="false" name="Medium Shading 2 Accent 6"> <w:lsdexception locked="false" priority="65" semihidden="false" unhidewhenused="false" name="Medium List 1 Accent 6"> <w:lsdexception locked="false" priority="66" semihidden="false" unhidewhenused="false" name="Medium List 2 Accent 6"> <w:lsdexception locked="false" priority="67" semihidden="false" unhidewhenused="false" name="Medium Grid 1 Accent 6"> <w:lsdexception locked="false" priority="68" semihidden="false" unhidewhenused="false" name="Medium Grid 2 Accent 6"> <w:lsdexception locked="false" priority="69" semihidden="false" unhidewhenused="false" name="Medium Grid 3 Accent 6"> <w:lsdexception locked="false" priority="70" semihidden="false" unhidewhenused="false" name="Dark List Accent 6"> <w:lsdexception locked="false" priority="71" semihidden="false" unhidewhenused="false" name="Colorful Shading Accent 6"> <w:lsdexception locked="false" priority="72" semihidden="false" unhidewhenused="false" name="Colorful List Accent 6"> <w:lsdexception locked="false" priority="73" semihidden="false" unhidewhenused="false" name="Colorful Grid Accent 6"> <w:lsdexception locked="false" priority="19" semihidden="false" unhidewhenused="false" qformat="true" name="Subtle Emphasis"> <w:lsdexception locked="false" priority="21" semihidden="false" unhidewhenused="false" qformat="true" name="Intense Emphasis"> <w:lsdexception locked="false" priority="31" semihidden="false" unhidewhenused="false" qformat="true" name="Subtle Reference"> <w:lsdexception locked="false" priority="32" semihidden="false" unhidewhenused="false" qformat="true" name="Intense Reference"> <w:lsdexception locked="false" priority="33" semihidden="false" unhidewhenused="false" qformat="true" name="Book Title"> <w:lsdexception locked="false" priority="37" name="Bibliography"> <w:lsdexception locked="false" priority="39" qformat="true" name="TOC Heading"> </w:LatentStyles> </xml><![endif]--><!--[if gte mso 10]> <style> /* Style Definitions */ table.MsoNormalTable {mso-style-name:"Table Normal"; mso-tstyle-rowband-size:0; mso-tstyle-colband-size:0; mso-style-noshow:yes; mso-style-priority:99; mso-style-parent:""; mso-padding-alt:0in 5.4pt 0in 5.4pt; mso-para-margin:0in; mso-para-margin-bottom:.0001pt; mso-pagination:widow-orphan; font-size:10.0pt; font-family:"Times New Roman","serif";} </style> <![endif]--><span style="font-size: 11pt; font-family: "Calibri","sans-serif"; color: rgb(31, 73, 125);">396191024 kB</span>) RAM, running RHEL5.1 with Sun JDK 1.6.0_21 in 64 bit mode.<br /><br />For BigMemory testcase, had just 2G of java heap even when loading 350 GB of data while for OnHeap testcases, had java heap of twice the data size.<br /><br /><br /><br /><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh9DUbY-hKsJTwPz5atHlHe6VOPxmO5skTSqWJrq-rtNsu1mfSuD1YUs_VumVWm6CHVNzX7dbEbUwfJOhBHZAapjOPaZrvAj01RaKYCsm6PLTbb9KpJluIGuCcT2-BEFHJoZKhCHufXXQs/s1600/Full.PNG" target="_blank"><img style="display: block; margin: 0px auto 10px; text-align: center; cursor: pointer; width: 400px; height: 268px;" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh9DUbY-hKsJTwPz5atHlHe6VOPxmO5skTSqWJrq-rtNsu1mfSuD1YUs_VumVWm6CHVNzX7dbEbUwfJOhBHZAapjOPaZrvAj01RaKYCsm6PLTbb9KpJluIGuCcT2-BEFHJoZKhCHufXXQs/s400/Full.PNG" alt="" id="BLOGGER_PHOTO_ID_5537444316544282738" border="0" /></a><br /><br />This chart compares the Largest Full GC duration occurred during the full run of the test. The numbers are taken from verbose GC logs.<br />If you take a microscope, then you can see there is a small green bar beside huge Al-burj tower types red bars. Those are GC duration for BigMemory :). Merely going above 1.2 sec, <a href="http://www.terracotta.org/bigmemory?src=performanceterracotta.blogspot.com" target="_blank">BigMemory</a> surely kills Garbage Collection and removes the stigma on the Java.<br /><br /><a aiotarget="false" aiotitle="" href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjQ2JVpJv828wrbx1_TFEOV3ymKAaMA6Ii-o91TiuX9bhjAJ1cbaGBjJeuTEtoLeXXqVFgN25nqH-aZpED73wHgEbyVTL-ft44uHGTHbTWTmqV3R75Olg4pRDltlQZLbsR7R8dxdGJaKIQ/s1600/maxLt.PNG" target="_blank"><img style="display: block; margin: 0px auto 10px; text-align: center; cursor: pointer; width: 400px; height: 270px;" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjQ2JVpJv828wrbx1_TFEOV3ymKAaMA6Ii-o91TiuX9bhjAJ1cbaGBjJeuTEtoLeXXqVFgN25nqH-aZpED73wHgEbyVTL-ft44uHGTHbTWTmqV3R75Olg4pRDltlQZLbsR7R8dxdGJaKIQ/s400/maxLt.PNG" alt="" id="BLOGGER_PHOTO_ID_5537444700225430610" border="0" /></a><br /><br />This charts compares the Max Latency during the test run. As expected, this should be equal to the Max Full GC duration, since GC just blocks the application. <a href="http://www.terracotta.org/bigmemory?src=performanceterracotta.blogspot.com" target="_blank">BigMemory</a> fairly defeats OnHeap here. Anyway who will like to have 4-5 mins of pause in his application, not me atleast !<br /><br /><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhYbdYhSneFBwBbdMtzVh6-0_SWWNzx1eOKb42WV97wRuW_TqG6ii-2FIWvIed2mfK0vZrC7gS_lNJovEiVJgJ-SNkPNULWjUHwI5_VMYlozUVmvBJ4NfmlAAodP9gDDO_aSCpKRz5r2tQ/s1600/10_tps.PNG" target="_blank"><img style="display: block; margin: 0px auto 10px; text-align: center; cursor: pointer; width: 400px; height: 270px;" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhYbdYhSneFBwBbdMtzVh6-0_SWWNzx1eOKb42WV97wRuW_TqG6ii-2FIWvIed2mfK0vZrC7gS_lNJovEiVJgJ-SNkPNULWjUHwI5_VMYlozUVmvBJ4NfmlAAodP9gDDO_aSCpKRz5r2tQ/s400/10_tps.PNG" alt="" id="BLOGGER_PHOTO_ID_5537444699308239618" border="0" /></a><br /><br />Lets see how <a href="http://www.terracotta.org/bigmemory?src=performanceterracotta.blogspot.com" target="_blank">BigMemory</a> throughput behaves with increase in the data size. The chart above shows that after certain point the throughput remains unaffected by the data size. Also, did a run with 350G of data and the tps/latency was constant. (Did we ever thought of caching 350G of data in an application :O). The drop in tps from 512M - 4G of data is because for smaller data size, ehcache stores the entries on heap (remember test is 10% hotset, so till hotset is on Heap & SMALL enough to fit, its faster). And we don't have much GC occurring for smaller data size.<br /><br /><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjB0c-yi4Mq3bfeI5Y7oOQO1WIRqrwbqUkO9EaRIPnpI1BHnwlnTFwvD-d9Yx40i5hm6L-ksZw1238GruJENiWwHodzdz6p4JC8VFhSqLW_VXMwvaXJjTMIRTh9AXDVUkjtxh9qyuF8hkw/s1600/MEAN-LATECNY.PNG" target="_blank"><img style="display: block; margin: 0px auto 10px; text-align: center; cursor: pointer; width: 400px; height: 270px;" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjB0c-yi4Mq3bfeI5Y7oOQO1WIRqrwbqUkO9EaRIPnpI1BHnwlnTFwvD-d9Yx40i5hm6L-ksZw1238GruJENiWwHodzdz6p4JC8VFhSqLW_VXMwvaXJjTMIRTh9AXDVUkjtxh9qyuF8hkw/s400/MEAN-LATECNY.PNG" alt="" id="BLOGGER_PHOTO_ID_5537444708455478962" border="0" /></a><br /><br />Latency, the most worried about factor for user experience. We don't want our users to wait for 5 secs, because first impression is the last impression. The charts show the mean latency for the tests. Note that all those numbers are in micro-seconds. So they are anyway less than 0.5 secs, meeting your deadliest SLAs. <a href="http://www.terracotta.org/bigmemory?src=performanceterracotta.blogspot.com" target="_blank">BigMemory</a> wins undoubtedly.<br /><br /><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjvMR1CYjeoIGMGhzOaV_PS-YGgzi-cP13F6DWVsqnhyphenhyphengbYmw7c5uSrk9inzbdz1yr9ImTER-1MVc7JZbLsrQ3wxS9U2wcsQ9xyAam80ozXkRV9_cF4ZHlGTiiRa4DAY9WKHXdVxjVgVP0/s1600/tps-compare.PNG" target="_blank"><img style="display: block; margin: 0px auto 10px; text-align: center; cursor: pointer; width: 400px; height: 270px;" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjvMR1CYjeoIGMGhzOaV_PS-YGgzi-cP13F6DWVsqnhyphenhyphengbYmw7c5uSrk9inzbdz1yr9ImTER-1MVc7JZbLsrQ3wxS9U2wcsQ9xyAam80ozXkRV9_cF4ZHlGTiiRa4DAY9WKHXdVxjVgVP0/s400/tps-compare.PNG" alt="" id="BLOGGER_PHOTO_ID_5537444712031459250" border="0" /></a><br /><br />Here comes biggest test for <a aiotitle="Big Memory" href="http://www.terracotta.org/bigmemory?src=performanceterracotta.blogspot.com" target="_blank">BigMemory</a>. Why will someone use BigMemory if it doesn't performs good enough as On Heap. We can't just ignore throughput for latency. We can see the <a href="http://www.terracotta.org/bigmemory?src=performanceterracotta.blogspot.com">BigMemory</a> throughput numbers outperforms On Heap numbers here also. On-Heap throughput just goes on decreasing as full GCs would be killing the test. Imagine pausing for 4 mins during the test, will surely reduce the average throughput significantly.<br />Note: The test which I ran is 50% writes, so we might be overshadowing the onHeap throughput but 10% writes throughput were also comparable.<br /><br /><a aiotarget="false" aiotitle="" href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjlkwc5b_-bLGoC5i6LUsDZNeEcv9L1K0TpmV7C70oXpXmhFPkZZe7yKk_MUFMeGCCc8rhc09UZ5qkG4hzw4lhJl62ZRWrr7zKaM9EHzEWRwV2Fjw1CABvnXj4xs55iu93S7xP2vDUnLwU/s1600/late-compare.PNG" target="_blank"><img style="display: block; margin: 0px auto 10px; text-align: center; cursor: pointer; width: 400px; height: 270px;" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjlkwc5b_-bLGoC5i6LUsDZNeEcv9L1K0TpmV7C70oXpXmhFPkZZe7yKk_MUFMeGCCc8rhc09UZ5qkG4hzw4lhJl62ZRWrr7zKaM9EHzEWRwV2Fjw1CABvnXj4xs55iu93S7xP2vDUnLwU/s400/late-compare.PNG" alt="" id="BLOGGER_PHOTO_ID_5537444707791378610" border="0" /></a><br /><br />Mean latency graphs also says the same story which I have been saying all over my blog. <a href="http://www.terracotta.org/bigmemory?src=performanceterracotta.blogspot.com" target="_blank">BigMemory</a> outperforms OnHeap :)<br /><br /><br /><span style="font-weight: bold;">Do-it-yourself:</span> Here is the <a href="https://svn.terracotta.org/repo/forge/offHeap-test/">svn link</a> to the test, a Maven-based performance comparisons between different store configurations.<br /><span style="font-weight: bold;">Note: </span>You will need to get a <a href="http://www.terracotta.org/bigmemory?src=performanceterracotta.blogspot.com" target="_blank">demo license key</a> and install it as discussed above to run the test.<br /><br />So bottom line:<br />If you are fed-up of GCs, check <a href="http://www.terracotta.org/bigmemory?src=performanceterracotta.blogspot.com" target="_blank">BigMemory</a>.<br />If you want to cache 350GB of data, check <a href="http://www.terracotta.org/bigmemory?src=performanceterracotta.blogspot.com" target="_blank">BigMemory</a>.<br />If you want to use most <span style="font-weight: bold; font-style: italic;">AWESOME </span>java cache ever made, check <a href="http://www.terracotta.org/bigmemory?src=performanceterracotta.blogspot.com" target="_blank">BigMemory</a>.<br /><br /><br />If you like the post, vote it up on <a href="http://www.dzone.com/links/terracotta_fairy_brings_big_memory_for_java_users.html">dzone.</a> :)Anonymoushttp://www.blogger.com/profile/00021983633501930224noreply@blogger.com1tag:blogger.com,1999:blog-8928181820369303270.post-45707640783491712322009-09-30T00:01:00.001+05:302009-09-30T00:01:12.356+05:30Fire up FirefoxMake you firefox rock and change the way you browse. Optimize your bandwidth. Get what you want, exclude unwanted. Add-ons that change firefox<br /><ol><li><a href="https://addons.mozilla.org/en-US/firefox/addon/1865">Adblock Plus</a> - This one rocks than any other damn plugin. Blocks all the ads and you can put your own regex to block </li><li><a href="https://addons.mozilla.org/en-US/firefox/addon/12">All-in-One Gestures</a> - Make gestures on Firefox to do what you want. Make a gesture zoom-in a image, open tab, much more.</li><li><a href="https://addons.mozilla.org/en-US/firefox/addon/327">Tweak Network</a> - Have bandwidth still firefox downloading pages slowly. Install this and change to power mode. </li><li><a href="https://addons.mozilla.org/en-US/firefox/addon/1455">Tiny Menu </a>- Want more screen space for webpage than menus, etc.</li><li><a href="https://addons.mozilla.org/en-US/firefox/addon/220">Flashgot </a>- Strictly for heavy downloaders. Configure to use Leechget and fire up your downloading.</li><li><a href="https://addons.mozilla.org/en-US/firefox/addon/4810">Speed Dial </a>- Speed dial your fav sites, instead of typing and wasting time.</li><li><a href="https://addons.mozilla.org/en-US/firefox/addon/2324">Session Manager </a>- Did you ever wanted to recover your last session, get your searches and pages back. Try this.</li><li><a href="https://addons.mozilla.org/en-US/firefox/addon/5972">RAM Back </a>- Firefox eating up your RAM. Get your RAM back.</li><li><a href="https://addons.mozilla.org/en-US/firefox/addon/60">Web Developer </a>- Using GPRS/slow connection, dont want to waste your bandwidth on images etc, block them using this. Clear cookies, and more for a web developer.</li><li><a href="https://addons.mozilla.org/en-US/firefox/addon/1419">IE Tab </a>- Few of damned sites work on IE only, use this to get rid of them. Configure to use IE rendering engine everytime you visit those sites.</li><li><a href="https://addons.mozilla.org/en-US/firefox/addon/1843">Firebug </a>- Like developing/inspecting web applications, wanted to know whats slowing your page. Check this out!!!</li><li><a href="https://addons.mozilla.org/en-US/firefox/addon/1117">Fox Clocks </a>- Boss in different continent. Get this to get time there.</li><li><a href="https://addons.mozilla.org/en-US/firefox/addon/5579">Cooliris </a>- iPhone interface to browse images</li><li><a href="https://addons.mozilla.org/en-US/firefox/addon/6076">Better Gmail2</a> - Gmail users try this out. Nice one!!!</li><li><a href="https://addons.mozilla.org/en-US/firefox/addon/398">ForecastFox</a> - Get forecast on your firefox.<br /></li></ol>Anonymoushttp://www.blogger.com/profile/00021983633501930224noreply@blogger.com0tag:blogger.com,1999:blog-8928181820369303270.post-34108475860596713002009-09-29T11:05:00.014+05:302009-09-30T00:47:26.864+05:30Glassfish with Standalone Terracotta EhcacheWhile running standalone ehcache sample app (<a href="https://svn.terracotta.org/repo/forge/projects/tim-ehcache/branches/tc-3.1/tim-ehcache-1.7-sample">https://svn.terracotta.org/repo/forge/projects/tim-ehcache/branches/tc-3.1/tim-ehcache-1.7-sample</a>) with glassfish server, glassfish client was able to make connection to tc server but fails with java.nio.channels.CancelledKeyException.<br /><br /><br /><span style="font-weight: bold;">terracotta-server.logs</span><br /><span style="font-family: courier new;font-size:85%;" ><br />2009-09-25 12:34:56,037 [L2_L1:TCComm Main Selector Thread (listen 0:0:0:0:0:0:0:0:9510)] INFO com.tc.net.protocol.transport.ServerMessageTransport - ConnectionID(14.e02630feb1904743b5782b6324fa26a2): Attaching new connection: com.tc.net.core.TCConnectionJDK14@24008846: connected: true, closed: false local=10.0.4.158:9510 remote=10.0.4.158:53243 connect=[Fri Sep 25 12:34:56 PDT 2009] idle=1ms [28 read, 0 write]<br />2009-09-25 12:34:56,038 [L2_L1:TCComm Main Selector Thread (listen 0:0:0:0:0:0:0:0:9510)] WARN com.tc.net.core.CoreNIOServices - Skipping modifyInterest - WRITE on com.tc.net.core.TCConnectionJDK14@24008846: connected: true, closed: false local=10.0.4.158:9510 remote=10.0.4.158:53243 connect=[Fri Sep 25 12:34:56 PDT 2009] idle=0ms [28 read, 0 write]<br />2009-09-25 12:34:56,038 [L2_L1:TCComm Main Selector Thread (listen 0:0:0:0:0:0:0:0:9510)] INFO com.tc.net.core.TCWorkerCommManager - Selecting [L2_L1:TCWorkerComm # 0, wt:0] from [[L2_L1:TCWorkerComm # 0, wt:0], [L2_L1:TCWorkerComm # 1, wt:0]]<br />2009-09-25 12:35:06,041 [L2_L1:TCWorkerComm # 0] WARN com.tc.net.core.CoreNIOServices - Exception trying to shutdown socket output: Transport endpoint is not connected<br />2009-09-25 12:35:06,042 [L2_L1:TCWorkerComm # 0] WARN com.tc.net.protocol.transport.ServerMessageTransport - ConnectionID(14.e02630feb1904743b5782b6324fa26a2): CLOSE EVENT : com.tc.net.core.TCConnectionJDK14@24008846: connected: false, closed: true local=10.0.4.158:9510 remote=10.0.4.158:53243 connect=[Fri Sep 25 12:34:56 PDT 2009] idle=10004ms [81 read, 81 write]. STATUS : START<br />2009-09-25 12:35:06,042 [L2_L1:TCWorkerComm # 0] WARN com.tc.net.protocol.transport.ServerMessageTransport - ConnectionID(14.e02630feb1904743b5782b6324fa26a2): closing down connection - com.tc.net.core.TCConnectionJDK14@24008846: connected: false, closed: true local=10.0.4.158:9510 remote=10.0.4.158:53243 connect=[Fri Sep 25 12:34:56 PDT 2009] idle=10004ms [81 read, 81 write]<br /></span><br /><span style="font-weight: bold;">terracotta-client.logs</span><br /><span style="font-size:85%;"><span style="font-family: courier new;"><br />2009-09-25 12:34:56,035 [httpSSLWorkerThread-8081-0] INFO com.tc.net.protocol.transport.ClientMessageTransport - ConnectionID(-1.ffffffffffffffffffffffffffffffff): Attaching new connection: com.tc.net.core.TCConnectionJDK14@13859800: connected: true, closed: false local=10.0.4.158:53243 remote=10.0.4.158:9510 connect=[Fri Sep 25 12:34:56 PDT 2009] idle=1ms [0 read, 0 write]</span><br /><span style="font-family: courier new;">2009-09-25 12:35:06,044 [L1_L2:TCComm Main Selector Thread (listen 0:0:0:0:0:0:0:0:50565)] WARN com.tc.net.protocol.transport.ClientMessageTransport - ConnectionID(-1.ffffffffffffffffffffffffffffffff): CLOSE EVENT : com.tc.net.core.TCConnectionJDK14@13859800: connected: false, closed: true local=10.0.4.158:53243 remote=10.0.4.158:9510 connect=[Fri Sep 25 12:34:56 PDT 2009] idle=10009ms [0 read, 81 write]. STATUS : SYN_SENT</span><br /><span style="font-family: courier new;">2009-09-25 12:35:06,044 [L1_L2:TCComm Main Selector Thread (listen 0:0:0:0:0:0:0:0:50565)] WARN com.tc.net.protocol.transport.ClientMessageTransport - ConnectionID(-1.ffffffffffffffffffffffffffffffff): closing down connection - com.tc.net.core.TCConnectionJDK14@13859800: connected: false, closed: true local=10.0.4.158:53243 remote=10.0.4.158:9510 connect=[Fri Sep 25 12:34:56 PDT 2009] idle=10009ms [0 read, 81 write]</span><br /><span style="font-family: courier new;">2009-09-25 12:35:06,044 [httpSSLWorkerThread-8081-0] WARN com.terracottatech.console - Timeout connecting to server: Timeout of 10000 milliseconds occured</span><br /><span style="font-family: courier new;">2009-09-25 12:35:06,044 [L1_L2:TCComm Main Selector Thread (listen 0:0:0:0:0:0:0:0:50565)] WARN com.tc.net.core.CoreNIOServices - java.nio.channels.CancelledKeyException occured</span><br /><br /></span><span style="font-weight: bold;">glassfish logs<br /><br /></span><span style="font-family: courier new;font-size:85%;" >[#|2009-09-25T12:37:21.157-0700|WARNING|sun-appserver9.1|javax.enterprise.system.stream.err|_ThreadID=15;_ThreadName=httpSSLWorkerThread-8081-0;_RequestID=f34b40b7-5245-48d5-9df8-03fb63e09a7c;|2009-09-25 12:37:21,157 WARN - Timeout connecting to server: Timeout of 10000 milliseconds occured<br />[#|2009-09-25T12:37:36.169-0700|WARNING|sun-appserver9.1|javax.enterprise.system.stream.err|_ThreadID=15;_ThreadName=httpSSLWorkerThread-8081-0;_RequestID=f34b40b7-5245-48d5-9df8-03fb63e09a7c;|2009-09-25 12:37:36,169 WARN - Timeout connecting to server: Timeout of 10000 milliseconds occured<br /></span><br /><br /><br />This could be solved by adding the following jvm option to glassfish domain.xml.<br /><br /><pre><jvm-options>-Dcom.sun.enterprise.server.ss.ASQuickStartup=false</jvm-options></pre>Anonymoushttp://www.blogger.com/profile/00021983633501930224noreply@blogger.com0tag:blogger.com,1999:blog-8928181820369303270.post-40136752966660015082009-09-29T10:51:00.002+05:302009-09-29T10:59:25.319+05:30Multiple Instances of Weblogic on same nodeWhile QAing terracotta standalone ehcache and terracottta standalone hibernate, we need to test the kit with different containers such as tomcat, jetty, glassfish, weblogic, jboss. It is good if we have all setup in same machine. When trying to install weblogic server on same machine, we can start a managed server on the same machine or create another domain and use terracotta server to cluster it.<br /><br />I came across a helpful blog to create new domains and a PDF file attached to it.<br /><blockquote><a href="http://blogs.oracle.com/learnwithpavan/2008/09/any_application_server_need_to.html"> http://blogs.oracle.com/learnwithpavan/2008/09/any_application_server_need_to.html</a><br /><a href="http://blogs.oracle.com/learnwithpavan/mydocs/weblogic/Creating%20a%20new%20Weblogic%20Domain.pdf">http://blogs.oracle.com/learnwithpavan/mydocs/weblogic/Creating%20a%20new%20Weblogic%20Domain.pdf</a><br /></blockquote>On step 7 as mentioned in the pdf, we can change the listen port to any other unused port like 8001 and you are good to go.<br /><br />We can start two different domain weblogic servers on same machine now.Anonymoushttp://www.blogger.com/profile/00021983633501930224noreply@blogger.com0tag:blogger.com,1999:blog-8928181820369303270.post-53710583623793631602009-09-29T10:09:00.013+05:302009-09-29T10:51:03.373+05:30Getting JAR version programmaticallyWhile QAing Ehcache, just to make sure that we have correct JAR loaded and tested, we tried to get the version of JAR from the pom.properties in ehcache-core-xxx.jar. Following code help us accessing the jar file for a particular class, and subsequently the files in it.<br /><span style="font-size:100%;"><br /></span><pre><span style="font-size:100%;"><span style="font-family:courier new;"> public String getEhcacheVersion(){</span><br /></span><span><span><span style="font-size:100%;"><span style="font-family:courier new;"> </span></span></span></span>String <span><span><span style="font-size:100%;"><span style="font-family:courier new;">ehcacheVersion</span></span></span></span><span><span><span style="font-size:100%;"><span style="font-family:courier new;"> = null;<br /> </span></span></span></span><span style="font-size:100%;"><span style="font-family:courier new;">JarInputStream jarIn = null;</span><br /><span style="font-family:courier new;"> File file = null;</span><br /><span style="font-family:courier new;"> try {</span><br /><span style="font-family:courier new;"> file = new File(Ehcache.class.getProtectionDomain()<br /></span></span><span style="font-size:100%;"><span><span style="font-family:courier new;"> </span></span></span><span style="font-size:100%;"><span><span style="font-family:courier new;"> </span></span></span><span style="font-size:100%;"><span><span style="font-family:courier new;"> </span></span></span><span style="font-size:100%;"><span><span style="font-family:courier new;"> </span></span></span><span style="font-size:100%;"><span style="font-family:courier new;">.getCodeSource().getLocation().toURI());</span><br /></span><span style="font-size:100%;"><span><span style="font-family:courier new;"> </span></span></span><span style="font-size:100%;"><span style="font-family:courier new;">jarIn = new JarInputStream(new FileInputStream(file));</span><br /></span><span style="font-size:100%;"><span><span style="font-family:courier new;"> </span></span></span><span style="font-size:100%;"><span style="font-family:courier new;">JarEntry entry;</span><br /><br /><span style="font-family:courier new;"> while ((entry = jarIn.getNextJarEntry()) != null) {</span><br /><span style="font-family:courier new;"> if (entry.getName().toUpperCase().equalsIgnoreCase("META-INF/maven/"<br /> +"net.sf.ehcache/ehcache-core/pom.properties")){</span><br /><span style="font-family:courier new;"> BufferedReader reader = new BufferedReader(new InputStreamReader(jarIn));</span><br /><span style="font-family:courier new;"> String line = null;</span><br /><span style="font-family:courier new;"> while ((line = reader.readLine()) != null) {</span><br /><span style="font-family:courier new;"> if(line.startsWith("version")){</span><br /><span style="font-family:courier new;"> ehcacheVersion=line;<br /></span></span><span><span><span style="font-size:100%;"><span style="font-family:courier new;"> </span></span></span></span><span style="font-size:100%;"><span style="font-family:courier new;">break;</span><br /><span style="font-family:courier new;"> }</span><br /><span style="font-family:courier new;"> }</span><br /><span style="font-family:courier new;"> }</span><br /><span style="font-family:courier new;"> }</span><br /><span style="font-family:courier new;"> }</span><br /><span style="font-family:courier new;"> catch (URISyntaxException e1) {e1.printStackTrace();}</span><br /><span style="font-family:courier new;"> catch (IOException e) {e.printStackTrace();}</span><br /><span style="font-family:courier new;"> finally {</span><br /><span style="font-family:courier new;"> if (jarIn != null)</span><br /><span style="font-family:courier new;"> try { jarIn.close(); } catch (Exception e) {}</span><br /><span style="font-family:courier new;"> }</span><br /> return </span><span><span><span style="font-size:100%;"><span style="font-family:courier new;">ehcacheVersion;<br /></span></span></span></span><span style="font-size:100%;"><span style="font-family:courier new;">}</span></span><br /></pre>mvn puts data into META-INF/maven/net.sf.ehcache/ehcache-core/pom.properties<br /><br /><pre>#Generated by Maven<br />#Thu Sep 24 12:20:41 PDT 2009<br />version=xxx<br />groupId=net.sf.ehcache<br />artifactId=ehcache-core<br /></pre><br />We can also use MANIFEST.MF to get required data.Anonymoushttp://www.blogger.com/profile/00021983633501930224noreply@blogger.com0tag:blogger.com,1999:blog-8928181820369303270.post-5961453930969230062009-09-18T01:31:00.006+05:302009-09-29T10:41:59.251+05:30Hibernate 3 with Weblogic results in ClassNotFoundException HqlTokenWhen using Hibernate 3 with Weblogic Server 10.3, I was getting the following exception when deploying Terracotta h2lc tutorial application.<br /><br /><pre>org.hibernate.QueryException: ClassNotFoundException: org.hibernate.hql.ast.HqlToken [from org.hibernate.tutorial.domain.Person p]<br />at org.hibernate.hql.ast.HqlLexer.panic(HqlLexer.java:80)<br />at antlr.CharScanner.setTokenObjectClass(CharScanner.java:340)<br />at org.hibernate.hql.ast.HqlLexer.setTokenObjectClass(HqlLexer.java:54)<br />at antlr.CharScanner.(CharScanner.java:51)<br />at antlr.CharScanner.(CharScanner.java:60)<br />at org.hibernate.hql.antlr.HqlBaseLexer.(HqlBaseLexer.java:56)<br />at org.hibernate.hql.antlr.HqlBaseLexer.(HqlBaseLexer.java:53)<br />at org.hibernate.hql.antlr.HqlBaseLexer.(HqlBaseLexer.java:50)<br />at org.hibernate.hql.ast.HqlLexer.(HqlLexer.java:49)<br />at org.hibernate.hql.ast.HqlParser.getInstance(HqlParser.java:67)<br />at org.hibernate.hql.ast.QueryTranslatorImpl.parse(QueryTranslatorImpl.java:268)<br />at org.hibernate.hql.ast.QueryTranslatorImpl.doCompile(QueryTranslatorImpl.java:182)<br />at org.hibernate.hql.ast.QueryTranslatorImpl.compile(QueryTranslatorImpl.java:136)<br />at org.hibernate.engine.query.HQLQueryPlan.(HQLQueryPlan.java:101)<br />at org.hibernate.engine.query.HQLQueryPlan.(HQLQueryPlan.java:80)<br />at org.hibernate.engine.query.QueryPlanCache.getHQLQueryPlan(QueryPlanCache.java:94)<br />at org.hibernate.impl.AbstractSessionImpl.getHQLQueryPlan(AbstractSessionImpl.java:156)<br />at org.hibernate.impl.AbstractSessionImpl.createQuery(AbstractSessionImpl.java:135)<br />at org.hibernate.impl.SessionImpl.createQuery(SessionImpl.java:1651)<br />at org.hibernate.tutorial.web.EventManagerServlet.init(EventManagerServlet.java:28)<br />at javax.servlet.GenericServlet.init(GenericServlet.java:241)<br />at weblogic.servlet.internal.StubSecurityHelper$ServletInitAction.run(StubSecurityHelper.java:283)<br />at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:321)<br />at weblogic.security.service.SecurityManager.runAs(Unknown Source)<br />at weblogic.servlet.internal.StubSecurityHelper.createServlet(StubSecurityHelper.java:64)<br />at weblogic.servlet.internal.StubLifecycleHelper.createOneInstance(StubLifecycleHelper.java:58)<br />at weblogic.servlet.internal.StubLifecycleHelper.(StubLifecycleHelper.java:48)<br />at weblogic.servlet.internal.ServletStubImpl.prepareServlet(ServletStubImpl.java:521)<br />at weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:235)<br />at weblogic.servlet.internal.TailFilter.doFilter(TailFilter.java:26)<br />at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:42)<br />at org.hibernate.tutorial.web.SessionInterceptor.doFilter(SessionInterceptor.java:41)<br />at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:42)<br />at weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:3496)<br />at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:321)<br />at weblogic.security.service.SecurityManager.runAs(Unknown Source)<br />at weblogic.servlet.internal.WebAppServletContext.securedExecute(WebAppServletContext.java:2180)<br />at weblogic.servlet.internal.WebAppServletContext.execute(WebAppServletContext.java:2086)<br />at weblogic.servlet.internal.ServletRequestImpl.run(ServletRequestImpl.java:1406)<br />at weblogic.work.ExecuteThread.execute(ExecuteThread.java:201)<br />at weblogic.work.ExecuteThread.run(ExecuteThread.java:173)<br /></pre><br />This seems to be a known issue <a href="https://www.hibernate.org/250.html#A25">https://www.hibernate.org/250.html#A25</a><br /><br /><span style="font-size:130%;"><span style="font-weight: bold;">Solution:</span></span><br /><br />For WAR files, Add WEB-INF/weblogic.xml<br /><br /><pre><?xml version="1.0" encoding="ISO-8859-1"?><br /><!DOCTYPE weblogic-web-app PUBLIC "-//BEA Systems, Inc.//DTD Web Application 8.1//EN"<br /> "http://www.bea.com/servers/wls810/dtd/weblogic810-web-jar.dtd"><br /><weblogic-web-app><br /> <container-descriptor><br /> <prefer-web-inf-classes>true</prefer-web-inf-classes><br /> </container-descriptor><br /></weblogic-web-app><br /><br /></pre><blockquote><span style="color: rgb(51, 102, 255);">Note: All the dependencies should be in the WAR</span><br /><span style="color: rgb(255, 0, 0);">Warning: The above code can lead you to some ClassCastExceptions which is due to conflicting classes in WEB-INF/lib and system libs.</span><br /></blockquote>For me, it threw<br /><pre>java.lang.ClassCastException: weblogic.xml.jaxp.RegistryXMLReader cannot be cast to org.xml.sax.XMLReader <br />at org.xml.sax.helpers.XMLReaderFactory.loadClass(Unknown Source)<br />at org.xml.sax.helpers.XMLReaderFactory.createXMLReader(Unknown Source)<br />at org.dom4j.io.SAXHelper.createXMLReader(SAXHelper.java:83)<br />at org.dom4j.io.SAXReader.createXMLReader(SAXReader.java:894)<br />at org.dom4j.io.SAXReader.getXMLReader(SAXReader.java:715)<br /><br /></pre>so needed to remove xml-apis jar from WEB-INF/lib.<br /><br />The above solution is for Web Archives only. For other applications, we might need to add the following code. Didnt tested this though :)<br /><br /><pre><?xml version="1.0" encoding="ISO-8859-1"?><br /><weblogic-application><br /> <prefer-application-packages><br /> <package-name>antlr.*</package-name><br /> </prefer-application-packages><br /></weblogic-application><br /><br /></pre>Anonymoushttp://www.blogger.com/profile/00021983633501930224noreply@blogger.com0tag:blogger.com,1999:blog-8928181820369303270.post-23942487237510503312009-03-02T22:27:00.002+05:302009-03-02T22:30:46.059+05:30Examinator 16 Nodes - Test Run ResultsBelow is the result of the script for 16 tomcat server runs with Terracotta server. The MySQL server has been cleaned and restarted before the test run.<span style="font-weight: bold;"><br /><br /></span><blockquote><span style="font-weight: bold;">Analyzing Jmeter logs ...</span><br />==============================<br />502 Error = 0<br />503 Error = 0<br />DEV-1929 = 0<br /><br /><span style="font-weight: bold;">Response Time</span><br />==============================<br />Average JMeter response time [e1s31] = 96.8721<br />Average JMeter response time = 35.7428<br /><br /><span style="font-weight: bold;">Server Latency Stats</span><br />=====================<br />Average = 4.76148 ms<br />Minimum = 0 ms<br />Maximum = 7036 ms<br />Std. Deviation = 59.4633<br />Total # of Requests = 5407000<br />Nodes = 16<br /><br /><span style="font-weight: bold;">L2 Configuration</span><br />==============================<br />TC_OPTS=-Xms4g -Xmx4g -XX:+DisableExplicitGC -XX:-TraceClassUnloading -XX:TargetSurvivorRatio=90 -Xss128k -XX:+AggressiveHeap -Dcom.tc.l2.objectmanager.fault.logging.enabled=true -Dcom.tc.l2.cachemanager.logging.enabled=true -Dcom.tc.l2.berkeleydb.je.maxMemoryPercent=15 -XX:SurvivorRatio=12<br /><br /><span style="font-weight: bold;">L1 Configuration</span><br />==============================<br />CATALINA_OPTS=-Duse.async.processing=true -Dasync.concurrency=5 -Dcom.tc.hibernate.useFineGrainedLocking=false -Dcom.tc.l1.cachemanager.enabled=false -Duse.pojoizer=true -Xms1024m -Xmx1024m -server -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -verbose:gc -XX:TargetSurvivorRatio=90 -Xss128k -XX:NewSize=512m -XX:MaxNewSize=512m <br /><br /><span style="font-weight: bold;">Object DB Size</span><br />==============================<br />26G /export1/bench/perfTests/terracotta/server/data/objectdb<br /><br /><span style="font-weight: bold;">L2 VerboseGC log Analysis</span><br /><br /><span style="font-weight: bold;">GC Report</span><br />================<br />Avg Full GC Duration = 6.12302 secs<br />Avg Full GC Interval = 99.5817 secs<br />Total Full GC Time = 385.75 secs<br /><br />No. of Full GC = 63<br />Avg Young GC Duration = 0.234307 secs<br />Avg Young GC Interval = 10.8526 secs<br />No. of Young GC = 589<br />Total Young GC Time = 136.44 secs<br /><br />Avg Young GC Occurence b/w Full GC = 9.34921<br />Avg Young GC Time b/w Full GC = 2.16571 secs</blockquote>Anonymoushttp://www.blogger.com/profile/00021983633501930224noreply@blogger.com0tag:blogger.com,1999:blog-8928181820369303270.post-27470932885894948292009-02-20T20:42:00.004+05:302009-02-20T20:58:30.320+05:30Initial Lassen Performance ResultsThe below are the test results for the 8 tomcat cluster with terracotta server. It is the result form the <span style="font-weight: bold; font-style: italic;">masterRun.sh </span>script.<br /><blockquote><span style="font-weight: bold;">Analyzing Jmeter logs ...</span><br /><span style="font-weight: bold;">==============================</span><br />502 Error = 0<br />503 Error = 0<br />DEV-1929 = 0<br />//This includes network latency, etc.<br /><span style="font-weight: bold;">Response Time </span><br /><span style="font-weight: bold;">==============================</span><br />Average JMeter response time [Last Page/e1s31] = 144.109 ms<br />Average JMeter response time = 98.2516 ms<br /><br />//This is server latency printed by the apache tomcat servlet engine<br /><span style="font-weight: bold;">Server Latency Stats </span><br /><span style="font-weight: bold;">=====================</span><br />Average = 9.90304 ms<br />Minimum = 0 ms<br />Maximum = 5851 ms<br />Std. Deviation = 40.5956<br />Total # of Requests = 2704000<br />Nodes = 8<br /><br /><span style="font-weight: bold;">L2 Configuration</span><br /><span style="font-weight: bold;">==============================</span><br />TC_OPTS=-Xms4g -Xmx4g -XX:+DisableExplicitGC -XX:-TraceClassUnloading -XX:TargetSurvivorRatio=90 -Xss128k -XX:+AggressiveHeap -Dcom.tc.l2.objectmanager.fault.logging.enabled=true -Dcom.tc.l2.cachemanager.logging.enabled=true -Dcom.tc.l2.berkeleydb.je.maxMemoryPercent=15 -XX:SurvivorRatio=12<br /><br /><span style="font-weight: bold;">L1 Configuration</span><br /><span style="font-weight: bold;">==============================</span><br />CATALINA_OPTS=-Duse.async.processing=true -Dasync.concurrency=5 -Dcom.tc.hibernate.useFineGrainedLocking=false -Dcom.tc.l1.cachemanager.enabled=false -Duse.pojoizer=true -Xms1024m -Xmx1024m -server -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -verbose:gc -XX:TargetSurvivorRatio=90 -Xss128k -XX:NewSize=512m -XX:MaxNewSize=512m<br /><br />// We have async TIM installed which let the saving the results to database asynchronously.<br /><span style="font-weight: bold;">MySQL DB (perf28) Entries in EXAM_RESULT</span><br /><span style="font-weight: bold;">=========================================</span><br />COUNT(*)<br />4547<br /><br /><span style="font-weight: bold;">MySQL DB (perf28) results completion time</span><br /><span style="font-weight: bold;">==========================================</span><br />min(START_TIME) max(END_TIME) completionsecs<br />2009-02-20 04:40:12 2009-02-20 06:21:41 6089<br /><br /><span style="font-weight: bold;">Object DB Size</span><br /><span style="font-weight: bold;">==============================</span><br />13G /export1/bench/perfTests/terracotta/server/data/objectdb<br /><br />L2 VerboseGC log Analysis<br /><br /><span style="font-weight: bold;">GC Report</span><br /><span style="font-weight: bold;">================</span><br />Avg Full GC Duration = 5.50871 secs<br />Avg Full GC Interval = 247.74 secs<br />Total Full GC Time = 137.718 secs<br /><br />No. of Full GC = 25<br />Avg Young GC Duration = 0.230732 secs<br />Avg Young GC Interval = 25.8137 secs<br />No. of Young GC = 246<br />Total Young GC Time = 56.76 secs<br /><br />Avg Young GC Occurence b/w Full GC = 9.84<br />Avg Young GC Time b/w Full GC = 2.2704 secs</blockquote><br />The page response time is less than 10 ms.<br /><br />The above response time includes Servlet Initialization time. All the servlets/application initializes when first request hits the tomcat server which could be bit high.<br /><br />After 5 mins from the test run completion<br /><br /><blockquote>mysql -u root -h perf28 -e 'SELECT COUNT(*) FROM exam.EXAM_RESULT'<br />+----------+<br />| COUNT(*) |<br />+----------+<br />| 9600 |<br />+----------+</blockquote>Anonymoushttp://www.blogger.com/profile/00021983633501930224noreply@blogger.com0tag:blogger.com,1999:blog-8928181820369303270.post-59656233080385076252009-02-20T17:21:00.005+05:302009-02-20T17:45:54.880+05:30Lassen Performance TuningWe need to apply the performance tuning parameters on Tomcat and HTTP server.<br /><br /><h2 id="HTomcatServerTuning"><span>Tomcat Server Tuning</span></h2><p> </p><h3 id="HJasperConfiguration"><span>Jasper Configuration</span></h3><p> Tomcat JSP compiler Jasper has some recommended configuration for production environment or rather non-development environment. They are described below and are made to web.xml</p><span class="nobr"></span><a aiotarget="false" aiotitle="http://tomcat.apache.org/tomcat-6.0-doc/jasper-howto.html#Production%20Configuration" href="http://tomcat.apache.org/tomcat-6.0-doc/jasper-howto.html#Production%20Configuration">http://tomcat.apache.org/tomcat-6.0-doc/jasper-howto.html#Production%20Configuration</a><br /><br /><blockquote><br /><span style="font-family:courier new;"><br /><servlet><br /> <servlet-name>jsp</servlet-name><br /> <servlet-class>org.apache.jasper.servlet.JspServlet</servlet-class><br /> <init-param><br /> <param-name>fork</param-name><br /> <param-value>false</param-value><br /> </init-param><br /> <init-param><br /> <param-name>xpoweredBy</param-name><br /> <param-value>false</param-value><br /> </init-param><br /><br /><!-- development Is Jasper used in development mode? If true, ----><br /><!-- the frequency at which JSPs are checked for ----><br /><!-- modification may be specified via the ----><br /><!-- modificationTestInterval parameter. [true] ----><br /><!-- --><br /><br /><init-param> <param-name>development</param-name> <param-value>false</param-value> </init-param><br /><br /><!-- mappedfile Should we generate static content with one ----><br /><!-- print statement per input line, to ease ----><br /><!-- debugging? [true] ----><br /><!-- --><br /><br /><init-param><br /> <param-name>mappedfile</param-name><br /> <param-value>false</param-value><br /></init-param><br /><br /><!-- modificationTestInterval ----><br /><!-- Causes a JSP (and its dependent files) to not ----><br /><!-- be checked for modification during the ----><br /><!-- specified time interval (in seconds) from the ----><br /><!-- last time the JSP was checked for ----><br /><!-- modification. A value of 0 will cause the JSP ----><br /><!-- to be checked on every access. ----><br /><!-- Used in development mode only. [4] ----><br /><!-- --><br /><br /><init-param><br /> <param-name>modificationTestInterval</param-name><br /> <param-value>3600</param-value><br /></init-param><br /><br /><!-- genStrAsCharArray Should text strings be generated as char ----><br /><!-- arrays, to improve performance in some cases? ----><br /><!-- [false] ----><br /><!-- --><br /><br /><init-param><br /> <param-name>genStringAsCharArray</param-name><br /> <param-value>true</param-value><br /></init-param><br /><br /><load-on-startup>3</load-on-startup> </servlet><br /><br /></span></blockquote><br /><h3 id="HServerConfiguration"><span>Server Configuration</span></h3><p> The following changes were made regarding the tomcat server made to server.xml</p><p> Reference: <span class="nobr"><a href="http://tomcat.apache.org/tomcat-6.0-doc/config/http.html"><span class="nobr"></span></a><a href="http://tomcat.apache.org/tomcat-6.0-doc/config/http.html">http://tomcat.apache.org/tomcat-6.0-doc/config/http.html</a></span></p><p><span style="font-family:courier new;"></span></p><blockquote><span style="font-family:courier new;"><Connector port="8080" protocol="HTTP/1.1" </span><br /><span style="font-family:courier new;"> enableLookups="false" </span><br /><span style="font-family:courier new;"> maxThreads="500" </span><br /><span style="font-family:courier new;"> minSpareThreads="100" </span><br /><span style="font-family:courier new;"> maxKeepAliveRequests="1" </span><br /><span style="font-family:courier new;"> acceptCount="200" </span><br /><span style="font-family:courier new;"> connectionTimeout="20000" </span><br /><span style="font-family:courier new;"> redirectPort="8443" /></span></blockquote><span style="font-family:courier new;"></span><br /><p></p><h4 id="HmaxThreads"><span>maxThreads</span></h4><p> It is the maximum number of request processing threads to be created by this Connector, which therefore determines the maximum number of simultaneous requests that can be handled. The number were set to 500 as there were requests rejected giving connection refused error when load applied on the server was about 400 user. There is thinktime applied in the test which should reduce the concurrency in the server but at certain point of time, there were few requests kept alive if the keep-alive timeout is too high.</p><p> </p><div style="color: rgb(51, 102, 255);" class="infomessage"><blockquote>maxThreads were increased to 1200, to match the number of users/node. This wasn't necessary but to solve the 40 min problem so that the connections are not refused by tomcat. Since maxThreads were increased, minSpareThreads were also increased to 500. </blockquote></div><p> </p><h4 id="HmaxKeepAliveRequests"><span>maxKeepAliveRequests</span></h4><p> The maximum number of HTTP requests which can be pipelined until the connection is closed by the server. Setting this attribute to 1 will disable HTTP/1.0 keep-alive, as well as HTTP/1.1 keep-alive and pipelining. Setting this to -1 will allow an unlimited amount of pipelined or keep-alive HTTP requests. If not specified, this attribute is set to 100. There are redirects for each request so its good to have keep alive so that each redirected request is handled by the same thread.</p><p> </p><div style="color: rgb(51, 102, 255);" class="infomessage"><blockquote>Keep Alive on the tomcat server has been disabled, which probably solves the 502 Bad proxy gateway error</blockquote></div><p> </p><h4 id="HacceptCount"><span>acceptCount</span></h4><p> The maximum queue length for incoming connection requests when all possible request processing threads are in use. Any requests received when the queue is full will be refused. The default value is 10. It is set high so that we dont have any requests that is getting refused by the server.</p><h2 id="HMySQLServertuning"><span>MySQL Server tuning</span></h2><p> The limit on connections made to MySQL server is specified by the max_client parameter. The default value is 100, so if we have connection pool size of 100 on the app servers, the limit on mysql server throws "com.mysql.jdbc.exceptions.MySQLNonTransientConnectionException: Too many connections".</p><p> </p><div class="infomessage"><blockquote style="color: rgb(51, 102, 255);">The max connections were set to 800 which were eventually increased to 3300 (200 per node and 100 extra)</blockquote><h2 id="HLoadBalancerTuning"><span>Load Balancer Tuning</span></h2><p> The number of connections were increased from the default to handle the load. </p><span style="font-family:courier new;"></span><blockquote><span style="font-family:courier new;"><IfModule worker.c></span><br /><span style="font-family:courier new;">ServerLimit 50</span><br /><br /><span style="font-family:courier new;">#initial number of server processes to start </span><br /><span style="font-family:courier new;">#http://httpd.apache.org/docs/2.2/mod/mpm_common.html#startservers </span><br /><span style="font-family:courier new;">StartServers 3</span><br /><br /><span style="font-family:courier new;">#minimum number of worker threads which are kept spare </span><br /><span style="font-family:courier new;">#http://httpd.apache.org/docs/2.2/mod/mpm_common.html#minsparethreads </span><br /><span style="font-family:courier new;">MinSpareThreads 5000</span><br /><br /><span style="font-family:courier new;">#maximum number of worker threads which are kept spare </span><br /><span style="font-family:courier new;">#http://httpd.apache.org/docs/2.2/mod/mpm_common.html#maxsparethreads </span><br /><span style="font-family:courier new;">MaxSpareThreads 5000</span><br /><br /><span style="font-family:courier new;">#upper limit on the configurable number of threads per child process </span><br /><span style="font-family:courier new;">#http://httpd.apache.org/docs/2.2/mod/mpm_common.html#threadlimit </span><br /><span style="font-family:courier new;">ThreadLimit 200</span><br /><br /><span style="font-family:courier new;">#maximum number of simultaneous client connections </span><br /><span style="font-family:courier new;">#http://httpd.apache.org/docs/2.2/mod/mpm_common.html#maxclients </span><br /><span style="font-family:courier new;">MaxClients 10000</span><br /><br /><span style="font-family:courier new;">#number of worker threads created by each child process </span><br /><span style="font-family:courier new;">#http://httpd.apache.org/docs/2.2/mod/mpm_common.html#threadsperchild </span><br /><span style="font-family:courier new;">ThreadsPerChild 200</span><br /><br /><span style="font-family:courier new;">#maximum number of requests a server process serves </span><br /><span style="font-family:courier new;">#http://httpd.apache.org/docs/2.2/mod/mpm_common.html#maxrequestsperchild </span><br /><span style="font-family:courier new;">MaxRequestsPerChild 100000</span><br /><span style="font-family:courier new;"></IfModule></span></blockquote><span style="font-family:courier new;"></span><br /><h2 id="HLinuxtuning"><span>Linux tuning</span></h2><p> We observed "too many open files" error on the linux machines. The hard limit for the "open files" parameter shown by ulimit -aH was set too low. We increased the ulimit on all the linux machines to the max (65536). This number also determines the number of socket that can be opened on a linux machine. </p><div class="code" style="font-family:courier new;"><pre><blockquote>java.net.SocketException: Too many open files in system<br /> at java.net.Socket.createImpl(Socket.java:388)<br /> at java.net.Socket.<init>(Socket.java:362)<br /> at java.net.Socket.<init>(Socket.java:240)<br /> at org.apache.commons.httpclient.protocol.DefaultProtocolSocketFactory.createSocket(DefaultProtocolSocketFactory.java:80)<br /> at org.apache.commons.httpclient.protocol.DefaultProtocolSocketFactory.createSocket(DefaultProtocolSocketFactory.java:122)<br /> at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707)<br /> at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387)<br /> at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171)<br /> at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397)<br /> at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323)<br /> at org.apache.jmeter.protocol.http.sampler.HTTPSampler2.sample(HTTPSampler2.java:838)<br /> at org.apache.jmeter.protocol.http.sampler.HTTPSamplerBase.sample(HTTPSamplerBase.java:1021)<br /> at org.apache.jmeter.protocol.http.sampler.HTTPSamplerBase.sample(HTTPSamplerBase.java:1007)<br /> at org.apache.jmeter.threads.JMeterThread.run(JMeterThread.java:290)<br /> at java.lang.<span class="java-object">Thread</span>.run(<span class="java-object">Thread</span>.java:619)</init></init></blockquote></pre></div><br /></div><p><span class="nobr"></span> </p>Anonymoushttp://www.blogger.com/profile/00021983633501930224noreply@blogger.com0tag:blogger.com,1999:blog-8928181820369303270.post-69208796965578863682009-02-20T16:49:00.006+05:302009-02-20T17:19:33.529+05:30Monitoring Lassen Performance RunsHere is a brief descriptions of the logs.<br /><br /><span style="font-weight: bold;">1. server.<server_name>.log [ Dir:LassenPerfFramework/logs ] </server_name></span><br />This contains the startup logs for Terracotta Servers. If there are problems in the starting Terracotta Servers, check these logs first.<br /><br /><span style="font-weight: bold;">2. TCserver_GC.<server_name>.log [ Dir:LassenPerfFramework/logs ] </server_name></span><br />This contains the verbose GC output for the Terracotta Servers. If you want to work with different JVM GC settings, these logs contains the GC information.<br /><br /><span style="font-weight: bold;">3. log*.jtl [ Dir:LassenPerfFramework/logs ] </span><br />This contains the output from the JMeter test run. Each HTTP Response Logging from the Cluster goes into this. This does NOT contains the response body for the successful responses (HTTP response Code:302 or 200). For errors/exceptions (HTTP response Code:500 ,502, 503, 404, etc), JMeter prints out the Response from the server. It helps in diagnosing the problems.<br /><br />Each line contains the response time for that sample and response code from the server.<br /><span style="font-family:courier new;"></span><blockquote><span style="font-family:courier new;"><httpSample t="5" lt="5" ts="1184177284608" s="true" lb="/examinator/" rc="200" rm="OK" tn="Thread Group 1-1" dt="text"/></span></blockquote>t="5" : 5 ms is the response time for this sample<br />lb="/examinator" : The URL hit by the HTTP request.<br />rc="200" : HTTP Response code from the server<br />tn="Thread Group 1-1" : Thread Group.<br /><br /><span style="font-weight: bold;">4. GC*.log [ Dir:LassenPerfFramework/logs ] </span><br />This contains the output from JMeter that how many threads have started plus the JVM verbose GC output if enabled.<br /><br /><span style="font-weight: bold;">5. responseTime.log [ Dir: ~/perfTests/tomcat/logs ]</span><br />This contains the output from the <a href="http://performanceterracotta.blogspot.com/2008/12/response-time-tracking-method.html">ResponseTimeTrackingValve </a>installed on the tomcat servers. It prints out the average, maximum, minimum server latency time per 1000 requests. This doesnt include the network latency.<br /><br /><span style="font-weight: bold;">6. catalina.out [ Dir: ~/perfTests/tomcat/logs ]</span><span style="font-weight: bold;"></span><br />These are the logs from the Tomcat servers<span style="font-weight: bold;"><span style="font-weight: bold;">. </span></span>Any tomcat related errors will be logged in these logs.<span style="font-weight: bold;"><span style="font-weight: bold;"><br /></span></span>Anonymoushttp://www.blogger.com/profile/00021983633501930224noreply@blogger.com0tag:blogger.com,1999:blog-8928181820369303270.post-88323259791937209212009-01-05T15:55:00.010+05:302009-02-20T16:47:59.623+05:30Lassen Performance Testing Framework<span style="font-weight: bold;">1. Overview</span><br /><br />Currently, The framework does the following<br /><br /><ol><li>Reads the properties file </li><li>Checks the machine availability, aborts if any other java process is found running</li><li>Starts the Terracotta server </li><li>Starts the tomcat servers</li><li>Checks that each server is started</li><li>Starts the examinator load test using Jmeters<br /></li><li>Collects the results</li></ol><blockquote>SVN URL :<a href="https://svn.terracotta.org/repo/forge/projects/exam-perf-test/LassenPerfFramework"> https://svn.terracotta.org/repo/forge/projects/exam-perf-test/LassenPerfFramework</a></blockquote><br /><span style="font-weight: bold;">2. lassen_perf.properties</span><br /><br />The following properties file is being used to configure the testing environment. Perf# are the machine hostnames.<br /><br /><span style="font-family:courier new;"></span><blockquote><span style="font-family:courier new;">#################### Notes #########################<br />- Avoid using "<br />- keep these scripts on a shared drive<br />####################################################<br /><br />BASE_DIR = /shares/perf/hsingh/lassen_perf<br />JAVA_HOME = /usr/java/default/<br /><br /># apache tomcat<br /># CATALINA_HOME : The installation path for apache tomcat<br /># CATALINA_OPTS : Set JVM arguments for tomcat servers<br /><br />CATALINA_HOME = /shares/perf/hsingh/apache-tomcat-6.0.18<br />CATALINA_OPTS = -Duse.async.processing=true -Dasync.concurrency=5 -Dcom.tc.l1.cachemanager.enabled=false -Duse.pojoizer=false -XX:NewSize=512m -XX:MaxNewSize=512m -Xms1024m -Xmx1024m -server -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -verbose:gc<br /><br /># terracotta server<br /># TC_CONFIG_PATH : The path to tc-config.xml file for terracotta server<br /># TC_HOME : The installation path for Terracotta server<br /># TC_OPTS : Set JVM arguments for terracotta server here<br /><br />TC_CONFIG_PATH = /shares/perf/hsingh/examinator/branches/tc-2.7/tc-config.xml<br />TC_HOME = /shares/perf/hsingh/terracotta-co/2.7/code/base/build/dist/terracotta-2.7.2-snapshot/<br />TC_OPTS = -Xms1g -Xmx1g -Dcom.tc.l2.objectmanager.fault.logging.enabled=true -Dcom.tc.l2.cachemanager.logging.enabled=true -Dcom.tc.l2.berkeleydb.je.maxMemoryPercent=15<br /><br /># Jmeter<br /># JMETER_DIR : The installation path for Jmeter<br /># JMETER_NODES : Set machines to be used to run JMeter. Syntax: <machine-name>:<no.>. Each JMeter consumes 4 GB of RAM<br /><br />JMETER_DIR = /shares/perf/hsingh/LassenPerfFramework/jakarta-jmeter-2.3.2<br />JMETER_NODES = perf01:3 perf37:5<br /><br /># list of servers to be used to set the testing environment<br /><br />L1_NODES = perf21 perf22 perf23 perf24 perf25 perf26 perf27 perf28 perf29 perf30 perf31 perf32 perf33 perf34 perf35 perf36<br />L2_NODE = perf02<br />MYSQL = perf28</no.></machine-name></span><br /></blockquote><span style="font-weight: bold;">3. Directory Listing</span><br /><table><br /><tbody><tr><td style="font-weight: bold;">Script</td><td style="font-weight: bold;">Directory</td><td style="font-weight: bold;">Comments</td></tr><tr><td>clean_mysql.sh </td><td> scripts/</td><td>Deletes the previous exam results from the mysql database</td></tr><tr><td>kill-server.sh </td><td> scripts/</td><td>Kills all the tomcat and terracotta server</td></tr><tr><td>lb-status.sh </td><td> scripts/</td><td> CL tool to check the number of concurrent requests being processed by the Load Balancer</td></tr><tr><td>machine-status.sh </td><td> scripts/</td><td>Checks the java process running on the machines</td></tr><tr><td>set-env.sh </td><td> scripts/</td><td> Reads and parses the Variables being used in the scripts</td></tr><tr><td>start-tomcat.sh </td><td> scripts/</td><td>Builts DSO boot jar and starts tomcat server</td></tr><tr><td>startAll-tomcat.sh </td><td> scripts/</td><td>Starts all the tomcat server</td></tr><tr><td>tc-server.sh </td><td> scripts/</td><td>starts terracotta server</td></tr><tr><td>GC </td><td> scripts/results/</td><td>Analyzes the verbose GC output</td></tr><tr><td>collectAll-tomcatlogs.pl </td><td> scripts/results/</td><td>collects all tomcat logs</td></tr><tr><td>latency.sh </td><td> scripts/results/</td><td>prints out the average server latency calculated using the response time tracking valve logs</td></tr><tr><td>results/.sh </td><td> scripts/results/</td><td>Master script for resutls</td></tr></tbody></table><br /><br /><span style="font-weight: bold;">4. How to run the test</span><br /><br /><ol><li>Add or modify the machines to be used for the test in the <span style="font-weight: bold;"> <span style="font-style: italic;">lassen_perf.properties. </span><br /></span></li><li>Configure the<span style="font-weight: bold;"> </span>Load balancer to use the tomcat machines specified in the properties file. Need to comment or uncomment specific tomcat servers<span style="font-weight: bold;"> </span>in the httpd.conf (<span style="font-style: italic;">perf.conf</span>)<span style="font-weight: bold;">.</span></li><li>Restart the Load Balancer.<span style="font-weight: bold;"> </span><span style="font-style: italic; font-weight: bold;">/sbin/service apache2 restart </span>on SLES 10<span style="font-weight: bold;">. </span>It requires root access.</li><li>Clean the MySQL db using <span style="font-style: italic; font-weight: bold;">"scripts/clean_mysql.sh ."</span><br /></li><li>Start the test using <span style="font-style: italic;">masterRun.sh</span></li><li>To abort the test stop the <span style="font-style: italic;">masterRun.sh </span>script and kill all servers using <span style="font-weight: bold; font-style: italic;">"scripts/kill-server.sh"</span></li></ol>Anonymoushttp://www.blogger.com/profile/00021983633501930224noreply@blogger.com0tag:blogger.com,1999:blog-8928181820369303270.post-69581982476974711342008-12-18T11:04:00.001+05:302008-12-18T11:59:44.022+05:30Load Testing Examinator: JMeter<a onblur="try {parent.deselectBloggerImageGracefully();} catch(e) {}" href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi_FwAvkVv6DjFogcDqKMUPsEz45JqB6pGBkRyQpwhOiatGIGGYK6vc1PmavJAXfFgYjcb7cjRwFE8BomqgRG3rmwHZL58Nqdg9GlsVe8Ad_4_69d3-Yg8fdSZP2LK8t0vVRm5TySd6XJI/s1600-h/msg+c.png"><img style="margin: 0pt 0pt 10px 10px; float: right; cursor: pointer; width: 228px; height: 126px;" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi_FwAvkVv6DjFogcDqKMUPsEz45JqB6pGBkRyQpwhOiatGIGGYK6vc1PmavJAXfFgYjcb7cjRwFE8BomqgRG3rmwHZL58Nqdg9GlsVe8Ad_4_69d3-Yg8fdSZP2LK8t0vVRm5TySd6XJI/s320/msg+c.png" alt="" id="BLOGGER_PHOTO_ID_5281013485469669890" border="0" /></a><br />Yes, you can load test <a href="http://www.terracotta.org/web/display/orgsite/Web+App+Reference+Implementation">Examinator</a> yourself. We tested it with 19,200 users and got really good response times.<br /><br /><span style="font-weight: bold;">Scenario</span><br /><br />The scenario is that the users will login to the application and choose an exam. They will start taking exam, answer some of the questions, select few of them for review, can go back in mid of test to any other question or any question selected for review. The scenario wants to simulate the real-world situation where all users would be taking a particular exam for the exam duration. Each user waits for around 45 secs to answer the next question. The users start ramping up in 2 minutes.<br /><br /><span style="font-weight: bold;">Overview</span><br /><blockquote># of User Sessions: 19,200<br />Thinktime: 45 secs<br />User Arrival Rate: 160 users/sec<br />Ramp-up Time: 120 secs<br />Throughput: 445 pages/sec<br />Response Time: 5 ms<br /><br />Duration of Exam: 110 mins<br /># of Questions in the Exam: 100<br />No. of Choices/Question: 5<br /><br />Load Testing Tool: <a href="http://jakarta.apache.org/site/downloads/downloads_jmeter.cgi">JMeter</a><br /><br /># of Tomcat nodes: 16<br /># of Terracotta nodes: 1<br /># of JMeter instances: 16<br /><br />Terracotta Monitoring: <a href="http://www.terracotta.org/confluence/display/docs1/Admin+Console+Guide">Terracotta Admin Console</a><br />System Monitoring: <a href="http://www.ibm.com/developerworks/wikis/display/WikiPtype/nmon">nmon</a><br />Load Balancer Monitoring: <a href="http://httpd.apache.org/docs/2.2/mod/mod_status.html">mod_status </a>Module<br />Tomcat Monitoring: <a href="http://java.sun.com/developer/technicalArticles/J2SE/jconsole.html">JConsole</a><br />GC Monitoring: verboseGC<br /></blockquote>JMeter can be really a pain-in-ass if we are trying to load 20,000 users from a single JMeter instance or distributed JMeter instances. Distributed JMeter uses RMI to control the instances running remotely creating an overhead on JMeter. JMeter goes into series of Full GC for high number of users for a large test plan. The Examinator test plan includes a lot of HTTP Samples, around 280 HTTP request.<br /><blockquote>Tips to use JMeter for load testing Examinator:<br /><ul><li>Avoid using GUI for load testing</li><li>Avoid using distributed JMeter instances<br /></li><li>Reduce I/O by reducing data being saved under load per sample using bin/jmeter.properties</li></ul></blockquote>To make it happen, we ran 16 different JMeter instances after which we consolidated the results from each. Shell scripts was a great help to make it happen. Now we have a problem, each JMeter instance can't use same datapool (user login/passwords). We created copies of the JMeter Test plan and divided datapool of 19,200 into 16 files, each having 1200 users.<br /><br />A small shell script does this for me:<br /><span style="font-size:100%;"></span><blockquote><span style="font-size:100%;"><br /><span style="font-family:courier new;">if [ $# -lt 1 ];then<br />echo "Usage: <script_name> <jmeter_testplan_name>"<br />exit<br />fi<br />name=`echo $1 | cut -d '.' -f 1`</jmeter_testplan_name></script_name></span><br /><span style="font-family:courier new;">while [ $i -lt $2 ]</span><br /><span style="font-family:courier new;">do </span><br /><span style="font-family:courier new;">echo "Creating ${name}${i}.jmx..."</span><br /><span style="font-family:courier new;">cp $1 ${name}${i}.jmx</span><br /><span style="font-family:courier new;">`sed -i 's/userList.csv/userList'$i'.csv/g' ${name}${i}.jmx`</span><br /><span style="font-family:courier new;">done</span></span></blockquote><br /><br />Thus, each JMeter instances would be using a separate set of users and simulation 1200 users.Anonymoushttp://www.blogger.com/profile/00021983633501930224noreply@blogger.com0tag:blogger.com,1999:blog-8928181820369303270.post-13974956075853715752008-12-18T10:49:00.002+05:302009-01-05T16:47:10.725+05:305 ms Response Time for Online tests - Isn't it awesome!!!With Terracotta, it's all possible. We can get a response time of 5 ms for a online exam application under high load of 20,000 users. We have used best technologies and are able to get 5 ms response time and reduce the load on database.<br /><blockquote>The key fact to make it happen is to keep all the intermediate data in the memory and we have memory of 16 JVM clustered using Terracotta to use.<br /></blockquote>No need of slow access database to cluster the session data. Get more inside of it from <a href="http://www.terracotta.org/web/display/orgsite/Web+App+Reference+Implementation">http://www.terracotta.org/web/display/orgsite/Web+App+Reference+Implementation</a><br /><br /><blockquote>Examinator is live!!!<br /><a href="http://reference.terracotta.org/examinator">http://reference.terracotta.org/examinator</a><br /></blockquote>The sources are available to download.Anonymoushttp://www.blogger.com/profile/00021983633501930224noreply@blogger.com0tag:blogger.com,1999:blog-8928181820369303270.post-30416031390949038312008-12-04T15:24:00.000+05:302008-12-04T15:34:18.241+05:30502 Error: Bad ProxyIn the process of performance testing the <a href="http://terracotta.org/web/display/orgsite/Web+App+Reference+Implementation">examinator reference application</a> of <a href="http://terracotta.org/">Terracotta</a>, we faced a lots of 502 errors. During the performance testing few of the virtual users were receiving 502 error from the load balancer.<br /><br />Load balancer: Apache HTTP server, mod_proxy_balancer module<br />App Server: Tomcat 6<br /><br />The error received were<br /><br /><blockquote><p class="MsoNormal"><i><span style="font-family:Tahoma;font-size:85%;"><span style="font-size: 10pt; font-style: italic; font-family: Tahoma;"><!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN"><o:p></o:p></span></span></i></p> <p class="MsoNormal"><i><span style="font-family:Tahoma;font-size:85%;"><span style="font-size: 10pt; font-style: italic; font-family: Tahoma;"><html><head><o:p></o:p></span></span></i></p> <p class="MsoNormal"><i><span style="font-family:Tahoma;font-size:85%;"><span style="font-size: 10pt; font-style: italic; font-family: Tahoma;"><title>502 Proxy Error</title><o:p></o:p></span></span></i></p> <p class="MsoNormal"><i><span style="font-family:Tahoma;font-size:85%;"><span style="font-size: 10pt; font-style: italic; font-family: Tahoma;"></head><body><o:p></o:p></span></span></i></p> <p class="MsoNormal"><i><span style="font-family:Tahoma;font-size:85%;"><span style="font-size: 10pt; font-style: italic; font-family: Tahoma;"><h1>Proxy Error</h1><o:p></o:p></span></span></i></p> <p class="MsoNormal"><i><span style="font-family:Tahoma;font-size:85%;"><span style="font-size: 10pt; font-style: italic; font-family: Tahoma;"><p>The proxy server received an invalid
<o:p></o:p>response from an upstream server.<br />
<o:p></o:p></span></span></i></p> <p class="MsoNormal"><i><span style="font-family:Tahoma;font-size:85%;"><span style="font-size: 10pt; font-style: italic; font-family: Tahoma;">The proxy server could not handle the request <em><a href="/examinator/exam/list.do">GET&nbsp;/examinator/exam/list.do</a></em>.<p><o:p></o:p></span></span></i></p> <p class="MsoNormal"><i><span style="font-family:Tahoma;font-size:85%;"><span style="font-size: 10pt; font-style: italic; font-family: Tahoma;">Reason: <strong>Error reading from remote server</strong></p></p><o:p></o:p></span></span></i></p> <p class="MsoNormal"><i><span style="font-family:Tahoma;font-size:85%;"><span style="font-size: 10pt; font-style: italic; font-family: Tahoma;"><hr><o:p></o:p></span></span></i></p> <p class="MsoNormal"><i><span style="font-family:Tahoma;font-size:85%;"><span style="font-size: 10pt; font-style: italic; font-family: Tahoma;"><address>Apache/2.2.3 (Linux/SUSE) Server at <st1:place st="on"><st1:placename st="on">xyz.abc.lan</st1:placename> <st1:placetype st="on">Port</st1:placetype></st1:place> 80</address><o:p></o:p></span></span></i></p> <p class="MsoNormal"><i><span style="font-family:Tahoma;font-size:85%;"><span style="font-size: 10pt; font-style: italic; font-family: Tahoma;"></body></html></span></span></i></p></blockquote><p class="MsoNormal"><i><span style="font-family:Tahoma;font-size:85%;"><span style="font-size: 10pt; font-style: italic; font-family: Tahoma;"><o:p></o:p></span></span></i></p>By disabling keepAlive on tomcat server, we were able to reduce these errors. <a aiotitle="http://tomcat.apache.org/tomcat-6.0-doc/config/http.html" href="http://tomcat.apache.org/tomcat-6.0-doc/config/http.html">http://tomcat.apache.org/tomcat-6.0-doc/config/http.html</a><br /><br /><span style="font-family: courier new;"></span><blockquote><span style="font-family: courier new;"><Connector port="8080" protocol="HTTP/1.1" </span><br /><span style="font-family: courier new;"> maxKeepAliveRequests="1" </span><br /><span style="font-family: courier new;"> redirectPort="8443" /></span></blockquote>Anonymoushttp://www.blogger.com/profile/00021983633501930224noreply@blogger.com0tag:blogger.com,1999:blog-8928181820369303270.post-79978701337101393812008-12-04T14:32:00.000+05:302008-12-04T15:02:45.249+05:30Response Time Tracking MethodThe method described below records the server latency only i.e. the time spent in the server after a request is been received and response is sent back. It does not includes the network latency, browser rendering time, etc. The average time can also be printed in the logs (responseTime.log)<br /><br /><span style="font-weight: bold;font-size:130%;" >ResponseTimeTrackingValve</span><br /><br />Apache Tomcat has a component valve which can be used to track down the response time for each page. A Valve element represents a component that will be inserted into the request processing pipeline for the associated Catalina container (Engine, Host, or Context). Each request passes through the enabled valves to the web container and response follows the same path.<br /><br /><span style="font-style: italic; color: rgb(51, 102, 255);"></span><blockquote><span style="font-style: italic; color: rgb(51, 102, 255);">The code can be downloaded from <a href="https://svn.terracotta.org/repo/forge/projects/exam-perf-test/ResponseTimeTrackingValve">https://svn.terracotta.org/repo/forge/projects/exam-perf-test/ResponseTimeTrackingValve</a></span> </blockquote><br />The purpose of Customized Valve (ResponseTimeTrackingValve) is to record the response time for the pages and store the response time statistics for the page. ResponseTimeTrackingValve uses the AccessLogValve code as the base.<br /><br />ResponseTimeTrackingValve needs to extend org.apache.catalina.valves.ValveBase located in Catalina.jar file. The main method to override is <blockquote>invoke (org.apache.catalina.connector.Request request, org.apache.catalina.connector.Response response)</blockquote> request is passed to next valve by<blockquote> getNext().invoke(request, response);</blockquote>The time taken by the server is calculated and saved in a hashtable. Key to the hashtable remains the URI of the request and following stats are being stored in it -<br /><br />1. Average of all requests<br />2. Minimum Response Time<br />3. Maximum Response Time<br />4. Standard Deviation<a onblur="try {parent.deselectBloggerImageGracefully();} catch(e) {}" href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEinrrUkLuw3xyClX5MIxZBV60p6MJ9M8ltiPky3LrK0zgPkbzppoUlkoegB0mcgGBsoj2B5MUHgMsDxHxwzfZvfQTZDNHpQ2EBgW6ICThoNYfg2XjsBeRqXafx8Pe1q3cg5E0BJhtMrIy8/s1600-h/deviation.png"><img style="margin: 0pt 0pt 10px 10px; float: right; cursor: pointer; width: 260px; height: 151px;" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEinrrUkLuw3xyClX5MIxZBV60p6MJ9M8ltiPky3LrK0zgPkbzppoUlkoegB0mcgGBsoj2B5MUHgMsDxHxwzfZvfQTZDNHpQ2EBgW6ICThoNYfg2XjsBeRqXafx8Pe1q3cg5E0BJhtMrIy8/s320/deviation.png" alt="" id="BLOGGER_PHOTO_ID_5275858164644770754" border="0" /></a><br />5. Total Count of each Page Request made to that server<br /><br />To calculate standard deviation, the sum of squares of the values is being stored, as we get a new value, square of the value gets added to it. Thus, standard deviation is calculated using these values. Same algo applies to other metrics. The response time for each page is not being stored, only the required metrics are being stored.<br /><br /><br />Where does valve stand in the HTTP request stack?<br /><br /><a onblur="try {parent.deselectBloggerImageGracefully();} catch(e) {}" href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgtd5BspHIkEa2f6PZxWQq5V4SpglgCMRQ_o04tS0RX5S_blkAeLCCax7jot0ZTLmaRsvOeQp2zYJI4yjKJ7IsYld7E291FBmkmWnq631VF2qmVz6J2_lCjHHrwrlCI0KRnPgYqbXVcCzU/s1600-h/valve.JPG"><img style="margin: 0px auto 10px; display: block; text-align: center; cursor: pointer; width: 320px; height: 124px;" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgtd5BspHIkEa2f6PZxWQq5V4SpglgCMRQ_o04tS0RX5S_blkAeLCCax7jot0ZTLmaRsvOeQp2zYJI4yjKJ7IsYld7E291FBmkmWnq631VF2qmVz6J2_lCjHHrwrlCI0KRnPgYqbXVcCzU/s320/valve.JPG" alt="" id="BLOGGER_PHOTO_ID_5275859516424934226" border="0" /></a><br /><h4 id="HInstallingthevalve"><span>Installing the valve</span></h4><p> To install the customized valve, deploy it as a JAR file and place it inside the Tomcats server/lib directory.</p><p> Register the Valve. Insert the following statement into Tomcat's server.xml file, right before the closing tag:</p><p style="font-family: courier new;"></p><blockquote style="font-family: courier new;"><span style="font-size:100%;"><Valve classname="org.tc.performance.ResponseTimeTrackingValve" exclude="*.png,*.css,*.js,*.gif,*.ico,/,index.html" prnCount="1000" suffix=".log" prefix="responseTime" lastPageToken="e1s31"/></span></blockquote><p></p><p>The valve keeps track of each and every request that has been made including the js, gifs, css, etc. </p><ol><li>We can specify the extensions of the files which we don't want to keep track of in the exclude parameter.</li><li>prnCount is set to 1000, i.e. the average time will be printed each 1000 request.</li><li>lastPageToken is the executionId of the page at which the users finish exams (this can be checked at the time of JMeter test creation). As in the test created by us, has url "/examinator/flow/takeexam.do?execution=e1s31". This will print each request response time. We can disable it by removing the token.</li></ol><br /><h3 id="HResponseTimeTrackingServlet"><span>ResponseTime Tracking Servlet</span></h3><p> This servlet serves the current data from the hashtable to the browser in the form of JSON. The data is shown on a web page which updates the tables with new data after specified interval. Servlet provides the method to reset the values in the hashtable. The statistics are shown in a table for each Request URI. This provides more detailed response time page-wise breakup.</p><p> </p><div class="infomessage"><blockquote style="font-style: italic; color: rgb(51, 102, 255);">The code can be downloaded from <span class="wikiexternallink"><a href="https://svn.terracotta.org/repo/forge/projects/exam-perf-test/TrackingWAR"><span class="nobr"></span></a><a href="https://svn.terracotta.org/repo/forge/projects/exam-perf-test/TrackingWAR"><span class="nobr"></span></a><a aiotitle="https://svn.terracotta.org/repo/forge/projects/exam-perf-test/TrackingWAR" href="https://svn.terracotta.org/repo/forge/projects/exam-perf-test/TrackingWAR">https://svn.terracotta.org/repo/forge/projects/exam-perf-test/TrackingWAR</a></span></blockquote><span class="wikiexternallink"><a aiotitle="https://svn.terracotta.org/repo/forge/projects/exam-perf-test/TrackingWAR" href="https://svn.terracotta.org/repo/forge/projects/exam-perf-test/TrackingWAR"></a></span></div>Anonymoushttp://www.blogger.com/profile/00021983633501930224noreply@blogger.com2tag:blogger.com,1999:blog-8928181820369303270.post-43416536320061765762008-12-04T13:22:00.002+05:302008-12-18T12:12:16.591+05:30Load Balancer for Tomcat ServersApache HTTP server mod_balancer module can be used as load balancer for tomcat servers. [<a href="http://httpd.apache.org/docs/2.2/mod/mod_proxy_balancer.html">http://httpd.apache.org/docs/2.2/mod/mod_proxy_balancer.html</a>]<br />We can also configure the apache HTTP server for sticky sessions<br /><blockquote><br /><span style="font-style: italic; color: rgb(51, 102, 255);">Apache HTTP server supports prefork and worker module. worker module is recommended for less memory usage.</span></blockquote>1. Configuring Apache Load Balancer<br /><br /><ul><li> Install Apache HTTP Server</li><li>Enable following modules in Apache HTTP Server in conf/httpd.conf</li></ul><blockquote>LoadModule proxy_module modules/mod_proxy.so<br />LoadModule proxy_balancer_module modules/mod_proxy_balancer.so<br />LoadModule proxy_http_module modules/mod_proxy_http.so<br />LoadModule rewrite_module modules/mod_rewrite.so</blockquote><ul><li>Add the following code to httpd.conf</li></ul><blockquote style="font-family:courier new;"><span style="font-size:100%;"><span style="font-family: courier new;"><VirtualHost *></span><br /><span style="font-family: courier new;">RewriteEngine On</span><br /><span style="font-family: courier new;">UseCanonicalName On</span><br /><br /><span style="font-family: courier new;"># To reduce the logging on the http server</span><br /><br /><span style="font-family: courier new;">ErrorLog "/var/log/apache2/rewrite.log"</span><br /><span style="font-family: courier new;">LogLevel error</span><br /><span style="font-family: courier new;">CustomLog /dev/null combined</span><br /><br /><span style="font-family: courier new;"># Sticky session is maintained using JSESSIONID</span><br /><span style="font-family: courier new;"># No failover</span><br /><br /><span style="font-family: courier new;">ProxyPass /examinator balancer://mycluster/ stickysession=JSESSIONID nofailover=On maxattempts=0 timeout=1800</span><br /><span style="font-family: courier new;">ProxyPassReverse / balancer://cluster/</span><br /><span style="font-family: courier new;">ProxyPreserveHost On</span><br /><span style="font-family: courier new;">ProxyTimeout 1800</span><br /><br /><span style="font-family: courier new;"><Proxy balancer://mycluster></span><br /><span style="font-family: courier new;">BalancerMember http://perf21:8080 route=151</span><br /><span style="font-family: courier new;">BalancerMember http://perf22:8080 route=152</span><br /><span style="font-family: courier new;"></Proxy></span><br /><br /><span style="font-family: courier new;"></VirtualHost></span><br /></span></blockquote><ul><li>To make it sticky sesion, we have to add the route ids to tomcat server config file.</li><li>Add route id as jvmRoute param in conf/server.xml of tomcat server. This id needs to be unique for each tomcat server. So we are using the last token of the IP address for each node. (eg . perf21 : 10.0.4.151)</li></ul>This can be done by the following script<br /><br /><blockquote style="font-family:courier new;"><span style="font-family: courier new;font-size:100%;" >// Setting jvmRoute to the last digits of IP address in server.xml<br />// jvmRoute is specific to sticky session apache load balancer.<br />// See Load Balancer Configuration<br /><br />IP=`/sbin/ifconfig | grep inet | grep '10.0' | cut -d '.' -f 4 | cut -d ' ' -f 1`<br />echo "Changing jvmRoute to $IP"<br />sed -i 's/IPCONFIG/'"$IP"'/g' $/conf/server.xml</span><br /></blockquote><ul><li>This id would get appended to Cookies sent by this tomcat server. The load balancer will route based on this id to the respective server.<br /></li></ul><engine defaulthost="perf21" name="Catalina" jvmroute="151"><br /></engine>Anonymoushttp://www.blogger.com/profile/00021983633501930224noreply@blogger.com0