About 2 years ago, I introduced you to the <a href="/en/knowledge-hub/blog/the-centerdevice-cloud-architecture">architecture of CenterDevice</a> , and it is now time for an update.A quick refresher for those who do not want to read that, now outdated, article: CenterDevice is a startup by codecentric which provides document storage in the cloud. It really shines for documents like invoices, orders, project management, presentations etc. where the powerful search engine finds you, what you are looking for without the need for any manually maintained structures. It provides plenty of means to share documents within or outside your organisation. All documents are encrypted and stored in Germany (if that matters to you).TL;DR: In November 2014 we released version 2 of our API, relaunched all clients and moved our datacenter (virtually and physically). Some tech changed, some stayed the same. Better continue reading 🙂 <h2>Our Clients</h2>CenterDevice offers many different clients to its users. With the relaunch we finalized the migration to a new look and feel and introduced a new frequently asked for concept called “collection”. Before “collections” we assumed organizing people and documents in a “group” would be sufficient, but it turned out that those are two different things. Now administrators can organize people in “groups”, and everybody can put documents into “collections”.<h3>Android</h3>Among a unified look across platforms, we added powerful PDF viewing functionality to the Android app. The older version used preview images for each page, but now the app just downloads the PDF to display it. This brings increased performance, as well as added functionality. <a href="https://play.google.com/store/apps/details?id=de.centerdevice.android"> </a><div> <img src="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/Ju0a7Zi0TXu5fODHyOSS" alt="centerdevice-android2" width="168" height="300" /><img loading="lazy" src="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/GC5JbKQhGGVRMGX26Tcg" alt="centerdevice-android1" width="168" height="300" /><img loading="lazy" src="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/OTtqHwdGR12O9DMJaEFt" alt="centerdevice-android3" width="168" height="300" /></div> When we went to implement mobile apps 3 years ago, we decided to go with native apps, rather than using a crosscompiler like Phonegap or Apache Cordova or even HTML5 apps. At least at that time it was not clear how good features like certificate pinning, local storage, camera access and preview integration for different file formats could have worked. We stuck to the plan and still have no intentions to rewrite the apps in any non-native way. Getting the app into the play store never was a problem. The shared usage of library code between Android, desktop and web apps is a plus. The downside with Android development is still the slow emulator, but it is slightly compensated by the easier distribution of test builds. The PDF viewing technology used is the commercial Qoppa PDF viewer.<h3>iOS</h3>The iOS app now supports phones and tablets with a common look and feel. Feature wise the most important addition is the added sharing functionality. <a href="https://itunes.apple.com/de/app/centerdevice/id933144073"> </a><div> <img loading="lazy" src="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/V2tyhK2tRjCWl3lQbEpg" alt="centerdevice-ios0" width="168" height="300" /><img loading="lazy" src="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/Y6AaCHkITNKWiLimKvsW" alt="centerdevice-ios1" width="168" height="300" /><img loading="lazy" src="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/UH4moqb7RQKOVM9iIHng" alt="centerdevice-ios2" width="168" height="300" /></div> Apple gave us a lot of headaches with the release of our relaunch app, which was in development for a year. We planned with plenty of headroom for the approval process, but it took much longer. The main problem was, that this is now a new “app”: It has a new AppID because it is an universal app, rather than a iPad only app. This caused the big review process to kick in, which applied new arbitrary checks, which seemed ok two years ago. In the end we needed an expedited approval to be “only one week late”, which is after 4 weeks of review. Due to the slow adoption of iOS8, this app supports iOS7 and 8 and does not yet use any iOS8 specifics. Distribution of test builds is still a mess, sorry Apple. For viewing PDFs we are experimenting with the open source vfr/Reader as an alternative to the commercial pspdfkit.<h3>Desktop</h3>A new member of our client family is the desktop application. Frequently requested, it seems that working with files is still a desktop and offline thing. The application will keep local copies of your documents in sync with what is on the server, so you always have access to documents even when you are offline. It will support monitoring certain local folders for automatic file upload soon. You can get it at <a href="https://www.centerdevice.de/download/">www.centerdevice.de/download</a> .<a href="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/5CzgkgFRDqYVWz6XDBdk"><img loading="lazy" src="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/5CzgkgFRDqYVWz6XDBdk" alt="centerdevice-desktop" width="300" height="201" /></a> The Desktop Client is a JavaFX8 Application which comes with a bundled JRE. After trying various installer solutions we settled with a custom mechanism to allow separate updates of the JRE and the application code. JavaFX8 is finally a usable platform and was very easy to create the UI with. We used a minimalistic, slightly adopted version of Adam Biens “afterburner.fx” and Google Guice for dependency injection. Some of the data queried from the server is stored in a local Derby database, while the downloaded documents will reside as files in a directory. For now we decided to hide that directory and discourage manual modifications, because there are many tricky edge cases involved when the Desktop Client is actually not aware that you are about to modify a file.<h3>Web</h3>The biggest feature set of all apps is still in web hand. Some of the more administrative workflows are only available here. The left hand side navigation is now hosting collections, while groups and users are on the right hand side. There are a few view modes for you to choose, and you can resize it to your liking.<a href="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/45cxeefDRfCjlfA0gn0P"><img loading="lazy" src="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/45cxeefDRfCjlfA0gn0P" alt="centerdevice-web" width="300" height="160" /></a> Being a complex web application, it is unfortunately also the slowest of our apps. Especially in Internet Explorer. If you really need IE, I feel sorry for you. We use Vaadin 7.3 and a customized Valo Sass Theme. After multiple years of debugging and hotfixing, we have finally given up on using Vaadin push. If it works for you, you are lucky. It did not work for us, with all the potential network proxies and browsers our end customers use. We are now using a 5 second polling, which is “good enough” for us. If we need to poll faster we switch the interval dynamically. That is why we are looking forward to Vaadin 7.4, where <a href="http://dev.vaadin.com/ticket/12936">polling no longer causes layout phases</a> . Still I think Vaadin is a good choice for the type of application we have here at our hand, it allows a very easy integration into a Java stack, and using Node or Angular would require more work on that end. However abstractions come at a cost and debugging Vaadin might not be your cup of tea 🙂 For viewing all types of PDF files, we incorporate the open source mozilla/pdf.js viewer.<h2>Our API</h2>And there are a few third party clients already using the CenterDevice API. Unfortunately there is none which I can talk about, but if you are interested, we have published our API, so you could get started developing a custom extension: <div><div><div>Click on the button to load the content from public.centerdevice.de.<a href="#">Load content</a> </div></div><div></div></div> (<a href="https://public.centerdevice.de/dcebab04-d000-432b-83ad-285e32070cbc">https://public.centerdevice.de/dcebab04-d000-432b-83ad-285e32070cbc</a> ) The API lives at https://api.centerdevice.de/v2 but without valid auth tokens you will not get far 🙂 It is still implemented using Jersey. Versioning is implemented using a master class for each version which knows all valid resources. This pattern allows us to either reuse the same Resource classes for different API versions, or do customization by composition or inheritance. Its pretty flexible, but also difficult to judge when to apply which pattern for differences in versions. Being backwards compatible is a great challenge everybody should go through.<h2>Our Backend</h2><a href="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/b0YvVmcStGTIaPyUPCTm"><img loading="lazy" src="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/b0YvVmcStGTIaPyUPCTm" alt="centerdevice2" width="300" height="240" /></a> If you compare this picture to the last published architecture, you see a few changes, but nothing major. We still have the separation between Web Servers (called tomcat-centerdevice in the picture) and REST Servers (named tomcat-rest). The Web Servers host the server side of the Vaadin applications, as well as a few other pages and admin interfaces. The main difference from the REST Servers is that they maintain state, and require session stickyness. Plans are there to put the sessions into memcached, but so far were not at priority. The REST Servers serve our REST API. All our clients use the above linked public API, with only a few exceptions for private management functionality, which uses a private REST API. As you can see in the picture, there is no direct access to any data store from the frontends, which increases security and allows us to scale better.There are 3 data sources for the rest server:<ul><li>Elasticsearch for all kinds of search related queries.</li><li>MongoDB for all metadata and user data.</li><li>Ceph as the storage for all documents and various previews.</li></ul>Elasticsearch replaced Apache Solr. Elasticsearch is very easy to maintain and fast. It finds its cluster members automatically and even if it doesn’t, a simple restart solves most of the issues. We had some problems when cluster members died, but it never affected production and was straightforward to resolve. Another nice thing about Elasticsearch is that it allows many index related operations on the fly. Like changing the schema. Christian has written a great <a href="/en/knowledge-hub/blog/elasticsearch-zero-downtime-reindexing-problems-solutions">blog describing our index handling strategy</a> . We have a few <a href="/en/knowledge-hub/blog?q=elasticsearch">more blog posts about Elasticsearch</a> in case you are curious.MongoDB is still going strong, but when we moved our cluster (more below) we noticed again that it was not built for administration 🙁 The schema free data storage is great, but for example taking and restoring a backup takes days (!) when authentication is enabled. Perhaps we were the only ones on the planet to run with mongo auth. Who knows. Besides that, my colleagues documented a lot of <a href="/wissens-hub/blog?q=mongodb">best practices in other blog posts</a> .Ceph is our replacement for Gluster. It is a distributed key-value store designed to hold binary artefacts. You could use it as file system, however that is not recommended. We use it as Swift compatible API using RadosGW. Whatever is stored to Ceph (mainly your original documents and preview images/pdfs of them) is encrypted using ChaCha20 256Bit. ChaCha20 is faster than AES if no hardware acceleration is used, and it is an open, crypto analyzed mechanism, in contrast to AES, which is still not proven to not contain a backdoor. Ceph performs really good, as you can find in <a href="/en/knowledge-hub/blog/ceph-object-storage-fast-gets-benchmarking-ceph ">Lukas Benchmarking Post</a> . It is really surprising to figure out that a networked file system is actually much faster than local discs. However, Ceph is quite resource intensive during cleanup, maintenance or failover, so even when it looks like it is disc only, it actually requires some amount of CPU and is best placed on machines dedicated to “being the file system”.Whenever a new document is uploaded, the REST Server sends the Document Server a message to start processing the document. The actual tasks executed on a document depend very much on its mimetype, the most important ones are:<ul><li>Apache Tika for text extraction.</li><li>Tesseract OCR if Tika was unable to find text.</li><li>LibreOffice to create PDFs out of document formats.</li><li>ffmpeg to convert various video formats.</li><li>Imagemagick + Ghostscript to create preview images out of almost anything.</li></ul>All of those tools really work great, but are really tricky to set up and avoid regressions when some magic command line flags change. Sometimes the queue to the document server fills up a bit (it is a Rabbit MQ beneath), so we implemented a mechanism which will prefer processing requests from other users over requests from the same user over and over again, so everbody gets a fair share of processing power.A new piece in the infrastructure is the Import Server. Users can add a Dropbox oAuth Token via the web interface (that is why we talk to dropbox from there) and the Import Server will upload selected documents asynchronously. The Import Server is designed to work with any third party data provider. We have prototypes for google drive and instagram, but they are not productified yet. Another part not visible on the picture are e-Mail servers, which handle incoming mail uploads. You can generate a mail upload alias in the Web UI, to which you can mail attachments to. These attachments get extracted and uploaded to the REST server from the mail servers. The same mail servers are also responsible for sending out notification/subscription e-mails.<h2>Our Infrastructure</h2>We also moved now to a completely virtualized infrastructure. But of course a virtualized infrastructure needs to sit on physical infrastructure. For that we have a mostly active-active HA setup for all networking and management hardware:<ul><li>Firewalls: 2x Dell Sonicwall NSA 3600.</li><li>Switches: 4x Dell Networking N2024, 1x Dell Power Connect 5524.</li><li>Management Server: 2x Dell PowerEdge R420.</li><li>Worker Server: 7X PowerEdge R510, 24CPU, 128GB RAM, 12x4TB HDD, 6x1GBit Networking.</li></ul><div> <a href="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/9zBSrap6QXC9eT1oeenb"><img loading="lazy" src="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/9zBSrap6QXC9eT1oeenb" alt="centerdevice-fw" width="150" height="150" /></a> <a href="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/l3R5N4WgS9WbzaR1kR3a"><img loading="lazy" src="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/l3R5N4WgS9WbzaR1kR3a" alt="centetrdevice-switches" width="150" height="150" /></a> <a href="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/0B01ZZfLRRia308tlqAP"><img loading="lazy" src="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/0B01ZZfLRRia308tlqAP" alt="centerdevice-nodes" width="150" height="150" /></a> <a href="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/4jtc27MJQZC85AzPrrFp"><img loading="lazy" src="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/4jtc27MJQZC85AzPrrFp" alt="centerdevice-infrastructure" width="150" height="150" /></a> </div> On top of that we run OpenStack as virtualization platform: <a href="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/HZCl3zMCQcP8GpYv7KOi"><img loading="lazy" src="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/HZCl3zMCQcP8GpYv7KOi" alt="centerdevice-virtual" width="286" height="300" /></a> We run right now 4 “all in one machines”, which each come with 2 Tomcats, Import and Document Server, MongoDB and Elasticsearch. Everything is set up using Ansible, which is comparable to Chef or Puppet, but with reduced abstraction layers to be closer to shell commands operations people know. We like that simplicity a lot. (Colleagues have written <a href="/en/knowledge-hub/blog?q=ansible">more blogs about Ansible</a> ). This is our “old” setup, which we plan to separate out into virtual machines in the next step. Besides that there are servers for e-Mail, AppDynamics monitoring and an admin gateway. You can find that “4” being mentioned in the AppDynamics screenshot above a few times. Two HAProxy loadbalancers terminate SSL traffic and balance the internal and external traffic onto the worker nodes. HAProxy is powerful and allows plenty of configuration options. For example <a href="/en/knowledge-hub/blog/haproxy-http-header-rate-limiting">easy rate limiting</a> , as described by my colleague Daniel.<img loading="lazy" src="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/4yu2x7tSiKfFgBCNPVO8" alt="centerdevice-ssl" width="300" height="163" />We take pride in running a <a href="https://www.ssllabs.com/ssltest/analyze.html?d=api.centerdevice.de ">A+ rated SSL setup</a> . All our apps use certificate pinning. They only work if the receive the certificate our server should serve. This eliminates any potential for man in the middle attacks, as attackers might be able to forge a trusted certificate, but it would never be identical to the ones baked into the apps. Apps using certificate pinning are guaranteed to have a secure connection with the intended server.<h2>Our Plans</h2>Next step will be to containerize components like mail server and document server, so that we can scale them even easier. While it looks like that “4” is a hardcoded number in many places it is actually not. For example starting a new document server would just work due to the way it communicates via RabbitMQ. Similarly a new Elasticsearch node would just work. Our local development environments already run Docker, so hopefully this is an easy step (TM). New hardware is already available, as seen in the pictures above, and is currently being provisioned.

The CenterDevice Cloud Architecture Revisited

On Monday 29th at the JavaOne conference I was proudly presenting my talk about profilers and how they work for the first time in English to a packed Yosemite B/C room. For everybody who was turned away, have no fear, the session was recorded and will be made available on parleys.com later. I will give an update once that is available, but if you cannot wait until then, please check out the slides below at speakerdeck. Looking forward to presenting again at JavaOne next year! If you did attend my session, please give feedback via the JavaOne app or session builder to let me know how I can improve.

Does My Profiler Tell The Truth? #javaone

Strings consume a lot of memory in any application. Especially the char[] containing the individual UTF-16 characters is contributing to most of the memory consumption of a JVM by each character eating up two bytes. It is not uncommon to find 30% of the memory consumed by Strings, because not only are Strings the best format to interact with humans, but also popular HTTP APIs use lots of Strings. With Java 8 Update 20 we now have access to a new feature called String Deduplication, which requires the G1 Garbage Collector and is turned off by default. String Deduplication takes advantage of the fact that the char arrays are internal to strings and final, so the JVM can mess around with them. Various strategies for String Duplication have been considered, but the one implemented now follows the following approach: Whenever the garbage collector visits String objects it takes note of the char arrays. It takes their hash value and stores it alongside with a weak reference to the array. As soon as it finds another String which has the same hash code it compares them char by char. If they match as well, one String will be modified and point to the char array of the second String. The first char array then is no longer referenced anymore and can be garbage collected.This whole process of course brings some overhead, but is controlled by tight limits. For example if a string is not found to have duplicates for a while it will be no longer checked.So how does this work in practice? First you need the Java 8 Update 20 which was just recently released.Then you can run the following code with: <code>-Xmx256m -XX:+UseG1GC</code><div>

```java5
public class LotsOfStrings {
 
 private static final LinkedList<String> LOTS_OF_STRINGS = new LinkedList<>();
 
 public static void main(String[] args) throws Exception {
 int iteration = 0;
 while (true) {
 for (int i = 0; i < 100; i++) {
 for (int j = 0; j < 1000; j++) {
 LOTS_OF_STRINGS.add(new String("String " + j));
 }
 }
 iteration++;
 System.out.println("Survived Iteration: " + iteration);
 Thread.sleep(100);
 }
 }
}
```

</div>This code will run and terminate after 30 iterations with an OutOfMemoryError.Now run it with String Deduplication enabled: <code>-Xmx256m -XX:+UseG1GC -XX:+UseStringDeduplication -XX:+PrintStringDeduplicationStatistics</code>It will now run significantly longer and terminate after 50 iterations.The JVM now also prints what it does, so lets take a look:<div>

```bash
[GC concurrent-string-deduplication, 4658.2K->0.0B(4658.2K), avg 99.6%, 0.0165023 secs]
 [Last Exec: 0.0165023 secs, Idle: 0.0953764 secs, Blocked: 0/0.0000000 secs]
 [Inspected: 119538]
 [Skipped: 0( 0.0%)]
 [Hashed: 119538(100.0%)]
 [Known: 0( 0.0%)]
 [New: 119538(100.0%) 4658.2K]
 [Deduplicated: 119538(100.0%) 4658.2K(100.0%)]
 [Young: 372( 0.3%) 14.5K( 0.3%)]
 [Old: 119166( 99.7%) 4643.8K( 99.7%)]
 [Total Exec: 4/0.0802259 secs, Idle: 4/0.6491928 secs, Blocked: 0/0.0000000 secs]
 [Inspected: 557503]
 [Skipped: 0( 0.0%)]
 [Hashed: 556191( 99.8%)]
 [Known: 903( 0.2%)]
 [New: 556600( 99.8%) 21.2M]
 [Deduplicated: 554727( 99.7%) 21.1M( 99.6%)]
 [Young: 1101( 0.2%) 43.0K( 0.2%)]
 [Old: 553626( 99.8%) 21.1M( 99.8%)]
 [Table]
 [Memory Usage: 81.1K]
 [Size: 2048, Min: 1024, Max: 16777216]
 [Entries: 2776, Load: 135.5%, Cached: 0, Added: 2776, Removed: 0]
 [Resize Count: 1, Shrink Threshold: 1365(66.7%), Grow Threshold: 4096(200.0%)]
 [Rehash Count: 0, Rehash Threshold: 120, Hash Seed: 0x0]
 [Age Threshold: 3]
 [Queue]
 [Dropped: 0]
```

</div>For our convenience we do not need to add up all data ourselves but can use the handy totals calculation. The above snippet is the forth execution of String Deduplication, it took 16ms and looked at about 120k Strings. All of them are new, meaning not yet looked at. These numbers look different in real apps, where strings are passed multiple times, thus some might be skipped or have a hashcode already (as you might know the hash code of a String is computed lazy). In above case all strings could be deduplicated, removing 4.5MB of data from memory. The Table section gives statistics about the internal tracking table, and the Queue one lists how many requests for deduplication have been dropped due to load, which is one part of the overhead reduction mechanism.So how does this compare to String Interning? I blogged about <a href="/en/knowledge-hub/blog/save-memory-by-using-string-intern-in-java">how great String Interning is for memory efficiency</a> . In fact the String Deduplication is almost like interning with the exception that interning reuses the whole String instance, not just the char array.The argument the creators of <a href="http://openjdk.java.net/jeps/192">JDK Enhancement Proposal 192</a> make is that often developers do not know where the right place to intern strings would be, or that this place is hidden behind frameworks. As I wrote, you need some knowledge where you typically encounter duplicates (like country names). String Deduplication also benefits duplicate Strings across applications inside the same JVM and thus also includes stuff like XML Schemas, urls, jar names etc which one normally would assume not appear multiple times.It also adds no runtime overhead as it is performed asynchronously and concurrent during garbage collection, while String Interning happens in the application thread. This now also explains the reason we find that <code>Thread.sleep()</code> in above code. Without the sleep there would be too much pressure on GC, so String Deduplication would not run at all. But this is a problem only for such sample code. Real applications usually find a few ms spare time to run String Deduplication.

String Deduplication – A new feature in Java 8 Update 20

At codecentric we like to use the Java and GWT framework Vaadin for creation of web applications following the MVP pattern. While it features a AWT/SWT like programming model, it is however lacking built-in support for a CDI framework, which is strongly recommended to inject view components and to send events to presenters using an event bus. Additionally Vaadin applications are usually deployed on Jetty and Tomcat, web servers which do not come with built-in CDI support as well.There are two parts needed to enable dependency injection and events in Vaadin:<ul><li>Add CDI support to the containers not having it.</li><li>Add CDI support to Vaadin.</li></ul><h2>CDI and Tomcat</h2>Adding CDI to containers is easy. In this post we are using the <a href="http://weld.cdi-spec.org/">JBoss Weld</a> , which is the CDI reference implementation and built into JBoss AS. It can be easily added to a servlet container using this maven fragment:<div>

```xml
<dependency>
	<groupId>org.jboss.weld</groupId>
	<artifactId>weld-core</artifactId>
	<version>2.2.3.Final</version>
</dependency>
<dependency>
	<groupId>org.jboss.weld.servlet</groupId>
	<artifactId>weld-servlet-core</artifactId>
	<version>2.2.3.Final</version>
</dependency>
```

</div>Weld comes with quite a few dependencies, so you might want to check them and for example exclude JSF or Guava.Adding the dependency alone will not be sufficient. While CDI initializes on servlet container startup, it does not do anything for us yet. We will take care of that later.<h2>CDI and Vaadin</h2>Next we want to add CDI Support to Vaadin. What does that actually mean? After adding Weld, we could use standard CDI annotations already. Vaadin has the concept of a UI. A user can have multiple UIs in her browser session. This means that @SessionScoped would not work, because then there could be only one UI per session, or these UIs would influence each other unexpectedly. To solve that problem, Vaadin offers the <a href="https://github.com/vaadin/cdi/tree/master/vaadin-cdi">vaadin-cdi plugin</a> which is currently in alpha state. It ships the @UIScoped annotation which will make sure each UI can have its own set of components and other injected @UIScoped dependencies.<h2>CDI and MVP in Vaadin</h2>To ease development of MVP style applications the <a href="https://github.com/tomivirkki/cdi-mvp">cdi-mvp plugin</a> can be used. It comes with base classes for views and presenters, as well as event support. Like <code>vaadin-cdi</code>, the <code>cdi-mvp</code> plugin is right now in alpha stage and unfortunately neither does see active development. However they only consist of good ideas and little code. For our project we have combined the codebase of both and removed stuff we did not need. We also needed to fix the <code>BeanManager</code> lookup which in Tomcat needs to look like this:<div>

```java5
InitialContext initialContext = new InitialContext();
beanManager = (BeanManager) initialContext.lookup("java:comp/env/BeanManager");
```

</div><h2>Making Weld Active</h2>When you would run the application now, you would get a message like “No active context for session scope found”. As said above, Weld has started its stuff but is not actually doing anything with servlet requests. The <a href="http://docs.jboss.org/weld/reference/latest/en-US/html/environments.html#d0e5583">Weld documentation</a> tells you that you need to add a servlet listener which will work as long as you do not use any push functionality. But if you use push, do not use the servlet listener! It assumes that a request starts and ends, like it did in the olden days. With push this is no longer true and you can get the strangest of errors from NullPointerExceptions to some users data showing up in another session! Luckily Vaadin allows us to hook into code which is guaranteed to be invoked before and after any Vaadin activity, either sync or async.<h2>Hooking into VaadinServletService</h2>You most likely already have extended VaadinServlet to do custom initialisation. To activate Weld we need to override <code>createServletService()</code>.<div>

```java5
protected VaadinServletService createServletService(DeploymentConfiguration deploymentConfiguration)
 throws ServiceException {
 return new WeldVaadinServletService(this, deploymentConfiguration);
}
```

</div>The code of WeldVaadinServletService is trivial. It activates and deactivates the WeldContexts on request start and request end.<div>

```java5
public final class WeldVaadinServletService extends VaadinServletService {
 
 public WeldVaadinServletService(VaadinServlet servlet, DeploymentConfiguration deploymentConfiguration)
 throws ServiceException {
 super(servlet, deploymentConfiguration);
 init();
 }
 
 @Override
 public void requestStart(VaadinRequest request, VaadinResponse response) {
 VaadinWeldListener.initWeldForRequest(request);
 super.requestStart(request, response);
 }
 
 @Override
 public void requestEnd(VaadinRequest request, VaadinResponse response, VaadinSession session) {
 try {
 super.requestEnd(request, response, session);
 } finally {
 VaadinWeldListener.clearWeldForRequest(request);
 }
 }
}
```

</div>Now you need to be strong. VaadinWeldListener contains a few hacks to activate the Weld contexts.<div>

```java5
import static org.jboss.weld.environment.servlet.Listener.LISTENER_USED_ATTRIBUTE_NAME;
import static org.jboss.weld.servlet.ConversationFilter.CONVERSATION_FILTER_REGISTERED;
 
public class VaadinWeldListener implements ServletContextListener {
 
 private static WeldInitialListener WELD_LISTENER;
 private static ServletContext CTX;
 
 @Override
 public void contextInitialized(ServletContextEvent sce) {
 CTX = sce.getServletContext();
 // Do not use conversation filter. So claim its registered.
 CTX.setAttribute(CONVERSATION_FILTER_REGISTERED, Boolean.TRUE);
 // Claim that a listener is existing and start registering it ourselves.
 CTX.setAttribute(LISTENER_USED_ATTRIBUTE_NAME, Boolean.TRUE);
 
 WELD_LISTENER = new WeldInitialListener();
 WELD_LISTENER.contextInitialized(sce);
 }
 
 public static void initWeldForRequest(VaadinRequest vaadinRequest) {
 WELD_LISTENER.requestInitialized(new ServletRequestEvent(CTX, (VaadinServletRequest) vaadinRequest));
 }
 
 public static void clearWeldForRequest(VaadinRequest vaadinRequest) {
 WELD_LISTENER.requestDestroyed(new ServletRequestEvent(CTX, (VaadinServletRequest) vaadinRequest));
 }
 
 @Override
 public void contextDestroyed(ServletContextEvent sce) {
 WELD_LISTENER.contextDestroyed(sce);
 }
 
}
```

</div>This does similar stuff to what the weld listener would have done. It registers a single instance of itself and makes sure that <code>requestInitialized()</code> and <code>requestDestroyed()</code> are invoked on a <code>WeldInitialListener</code>. It also sets attributes to prevent other parts of Weld doing that initialisation. The main difference is that this code will now be invoked before and after each Vaadin activity, while the Weld listener would only listen to <code>requestInitialized(ServletRequestEvent sre)</code> and <code>requestDestroyed(ServletRequestEvent sre)</code> from <code>ServletListener</code>, which is however not guaranteed to be invoked for push. Also Weld does some assumption that both methods are invoked on the same worker thread. With push that is also not guaranteed anymore. Non blocking IO allows connections to suspend and be picked up by different worker threads later on.<h2>Vaadin, Push and CDI live happily together ever after</h2>Developing a Vaadin application with CDI is much easier than without. Using the container built in CDI or Weld and the plugins however is not trivial and requires understanding about the request lifecycles and scopes. Unfortunately, even one year after the release of official support of push in Vaadin, there are still many problems to be found. If you decide to use push for more than trivial cases, you will end up debugging a lot.

Using Weld CDI with Vaadin and Push

What do you do when your Garbage Collection runs all the time? Well usually you would try to introduce some magic GC parameters to fix your issue. But let me tell you: Excessive garbage production cannot be fixed by parameters! You use the parameters to tune an “ok” GC to become a “good” GC. In complex applications it is very easy to lose control over your garbage. I have seen logs where gigabytes of garbage is produced in a second. Visualized it looks like this:<img src="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/QoxG7KfRKeKmgk4FbGPw" alt="lot-of-garbage" width="552" height="523" />But usually it happens only in production, or, if you are lucky, maybe even in load testing, but the main problem remains: How can you figure out what kind garbage is produced? Garbage leaves no trace.There are a few possible ways to do that:<ul><li>Take a heap dump. Use a tool like <a href="http://www.eclipse.org/mat/">Eclipse MAT</a> to inspect the references of garbage. This works. But taking a heap dump will stop your machine for a while, so you do not want to do that in production. Also, it is limited in what it sees. Maybe your timing is bad and you take the heap dump right after a GC when there is no significant garbage in memory. Even if there is garbage, you will never really know if that is really the biggest contributor.</li><li>Use a profiler like <a href="http://www.ej-technologies.com/products/jprofiler/overview.html">JProfiler</a> . Allocation profiling however is actually quite expensive and requires a lot of tuning to bring the overhead down. Even then it is risky to run in production. Using it on a development machine works, but still you might be unsure if that object creation you are looking at is really creating the garbage you can see from your GC logs.</li><li>Add logging. This is for the desperate. The amount of logging and additional bookkeeping usually creates a lot of additional pressure on memory and GC as well as log file pollution. Also it requires deployment of many temporary debugging versions.</li></ul>I did not like any of those solutions, so I decided to create my own tool. Maybe somebody else has done this already, but in that case I was unable to find it.<h3>I created and open sourced <a href="https://github.com/codecentric/allocation-tracker">Allocation Tracker</a> . </h3>Allocation Tracker is a performance and production optimized object creation counter. All it does is to add a counter to each <code> call of a class matching a given prefix. It is very fast on startup, where it does load time byte code manipulation. And it is very fast on object creation, where it only adds a counter per class name. It is also designed to consume very little memory. After startup it does nothing until turned on via JMX. Eventually you want to look at what has been recorded. That functionality is also exposed via JMX.</code><h3>Usage Example</h3>At codecentric we use the Vaadin Framework a lot. We also contribute back features and optimizations. When starting our application with <code>-javaagent:/tmp/allocation-tracker-agent.jar=com.vaadin</code> all <code>com.vaadin</code> classes will get instrumented. After letting the tracker work for about a minute while working with the application, we can generate a report of the most frequently created instances:

```

com/vaadin/server/EncodeResult 102800
com/vaadin/server/AbstractClientConnector$AllChildrenIterable 9442
com/vaadin/server/AbstractClientConnector$CombinedIterator 9442
com/vaadin/util/CurrentInstance 3507
com/vaadin/event/ConnectorEvent 3445
com/vaadin/shared/communication/SharedState 1703
com/vaadin/shared/AbstractComponentState 1693
com/vaadin/server/ClientConnector$AttachEvent 1598
com/vaadin/server/ClientConnector$DetachEvent 1460

```

Do not get confused by the slashes, that is bytecode speak. As a developer you usually use dots instead. So Vaadin produces a lot of <code>EncodeResults</code>. No surprise for me. I have contributed a <a href="http://dev.vaadin.com/ticket/13814">few</a> <a href="http://dev.vaadin.com/ticket/13545">patches</a> already on this, because the encoding showed up as a CPU hotspot in profiling. Version 7.1.15 used here for profiling does not have them merged yet. What was a surprise for me, however, is the sheer amount of <code>Iterables</code> created. The code doing that can be seen in AbstractClientConnector#L582.Whenever a component is attached, detached or marked as dirty, it will propagate the event to its children. To iterate over its children, the two classes mentioned above are created in line 331. It turns out, this is not needed in most cases and can be easily changed to much more efficient code, as I have done in commit a3578d3ac293442b64cbc5f5feb4cb9106799fdb. Creating <code>Iterables</code> does not really seem to be the correct solution here, but because <code>getAllChildrenIterable()</code> is a public API it needs to be fulfilled.After applying the change, a recoding of approximately the same actions results in this report:

```

com/vaadin/server/EncodeResult 108604
com/vaadin/event/ConnectorEvent 3654
com/vaadin/util/CurrentInstance 2998
com/vaadin/shared/communication/SharedState 1768
com/vaadin/shared/AbstractComponentState 1757
com/vaadin/server/ClientConnector$AttachEvent 1720
com/vaadin/server/ClientConnector$DetachEvent 1579

```

The inner classes (with the change now also anonymous) now appear quite at the end:

```

com/vaadin/server/AbstractClientConnector$1 32
com/vaadin/server/AbstractClientConnector$1$1 32

```

So without any noticeable impact the allocation tracker helped us find excessive object creation and we were able to craft and apply a fix.<h3>Limitations</h3>Not everything is perfect. Because of the high performance design, the agent does have a few limitations:<ul><li>It does not record allocation stacks. While a profiler can tell you where in code you allocate objects, or a heap dump can tell you where objects are referenced (but maybe not so nicely for garbage, which by its nature is no longer referenced), this tracker cannot do that. The reason is that with a high frequency of instantiations it would create a lot of stack traces, and need to store and compare them. That would be a lot of overhead. The reason why the allocation tracker still helps is that in most applications there are very limited places where a “hot” constructor is invoked. The Vaadin Iterables mentioned above were only created in one place. Same usually applies to any object typical programs use. If it really happens to be a class that is used everywhere, you know at least which it is and can maybe build your own counter into that specific class to identify the hot path.</li><li>It records superclass construction. As you can see in above report it is listing <code>com/vaadin/server/ClientConnector$AttachEvent</code>, <code>com/vaadin/server/ClientConnector$DetachEvent</code> and <code>com/vaadin/event/ConnectorEvent</code>. If you consult Vaadin code, you will see, and maybe have even guessed, that <code>AttachEvent</code> and <code>DetachEvent</code> both subclass <code>ConnectorEvent</code>. In Java constructors are always invoked up the chain until <code>java.lang.Object</code>. The agent has instrumented the constructor of <code>ConnectorEvent</code>, without looking at the class hierarchy. Figuring out if a class is abstract, or non-abstract but never instantiated directly at load time is a bit of work, especially considering dynamic class loading. The other option would be to suppress tracking superclass creations at runtime. However this would be quite a lot of extra code. In the end the output of superclasses is not that problematic, as a developer can easily recognize and filter them. Or maybe their total count can give some extra hints.</li><li>It is not really suitable for JVM classes. The agent will produce a warning when the instrumented prefix is “java” and maybe in future will get a set of blacklisted JVM classes. Messing with them is not really safe. Also the results would not be really useful. There are plenty of instances created in dark areas of the JVM or application servers, so the agent’s use would be quite limited. If you are a JVM or app server developer wanting to use the agent, you for sure will be able to modify it to suit you needs 🙂</li></ul><h3>Conclusion</h3>If you want to reduce your object allocation, most likely motivated from frequent or long GCs, the allocation tracker will be a easy to use and safe tool for you. It will be maintained by codecentric and we will merge any performance improving pull requests. Feature wise it will stay mostly as it is right now, solving the limitations it has would affect its performance to a large extend and defeat its primary purpose. All my plans for it are documented in <a href="https://github.com/CodingFabian/allocation-tracker/issues">Github issues</a> . If you found any issues, fixed problems or improved your code using the allocation tracker, please let us know or create a Pull Request!

Fixing Garbage Collection Issues Easily

Every Java developer with more than a few months of coding experience has written code like this before:<div>

```java5
try {
 "Hello World".getBytes("UTF-8");
} catch (UnsupportedEncodingException e) {
 // Every implementation of the Java platform is required to support UTF-8
 // Why the $!?% do I have to catch an exception which can never happen
}
```

</div>What I realized just recently is that Java 7 already provided a fix for this ugly code, which not many people have adopted:<div>

```java5
"Hello World".getBytes(StandardCharsets.UTF_8);
```

</div>Yay! No Exception! But it is not only nicer, it is also faster! You will be surprised to see by how much! Let us first look at the implementations for both getBytes() calls:<div>

```java5
return StringCoding.encode(charset, value, 0, value.length);
```

</div>Not exciting. We shall dig on:<div>

```java5
static byte[] encode(String charsetName, char[] ca, int off, int len)
 throws UnsupportedEncodingException
{
 StringEncoder se = deref(encoder);
 String csn = (charsetName == null) ? "ISO-8859-1" : charsetName;
 if ((se == null) || !(csn.equals(se.requestedCharsetName())
 || csn.equals(se.charsetName()))) {
 se = null;
 try {
 Charset cs = lookupCharset(csn);
 if (cs != null)
 se = new StringEncoder(cs, csn);
 } catch (IllegalCharsetNameException x) {}
 if (se == null)
 throw new UnsupportedEncodingException (csn);
 set(encoder, se);
 }
 return se.encode(ca, off, len);
}
```

</div>and<div>

```java5
static byte[] encode(Charset cs, char[] ca, int off, int len) {
 CharsetEncoder ce = cs.newEncoder();
 int en = scale(len, ce.maxBytesPerChar());
 byte[] ba = new byte[en];
 if (len == 0)
 return ba;
 boolean isTrusted = false;
 if (System.getSecurityManager() != null) {
 if (!(isTrusted = (cs.getClass().getClassLoader0() == null))) {
 ca = Arrays.copyOfRange(ca, off, off + len);
 off = 0;
 }
 }
 ce.onMalformedInput(CodingErrorAction.REPLACE)
 .onUnmappableCharacter(CodingErrorAction.REPLACE)
 .reset();
 if (ce instanceof ArrayEncoder) {
 int blen = ((ArrayEncoder)ce).encode(ca, off, len, ba);
 return safeTrim(ba, blen, cs, isTrusted);
 } else {
 ByteBuffer bb = ByteBuffer.wrap(ba);
 CharBuffer cb = CharBuffer.wrap(ca, off, len);
 try {
 CoderResult cr = ce.encode(cb, bb, true);
 if (!cr.isUnderflow())
 cr.throwException();
 cr = ce.flush(bb);
 if (!cr.isUnderflow())
 cr.throwException();
 } catch (CharacterCodingException x) {
 throw new Error(x);
 }
 return safeTrim(ba, bb.position(), cs, isTrusted);
 }
}
```

</div>Wooha. Well it looks like the one taking a <code>Charset</code> is more complicated, right? Wrong. The last line of <code>encode(String charsetName, char[] ca, int off, int len)</code> is <code>se.encode(ca, off, len)</code>, and the source of that looks mostly like the source of <code>encode(Charset cs, char[] ca, int off, int len)</code>. Very much simplified, this makes the whole code from <code>encode(String charsetName, char[] ca, int off, int len)</code> basically just overhead. Worth noting is the line <code>Charset cs = lookupCharset(csn);</code> which in the end will do this:<div>

```java5
private static Charset lookup(String charsetName) {
 if (charsetName == null)
 throw new IllegalArgumentException("Null charset name");
 
 Object[] a;
 if ((a = cache1) != null && charsetName.equals(a[0]))
 return (Charset)a[1];
 // We expect most programs to use one Charset repeatedly.
 // We convey a hint to this effect to the VM by putting the
 // level 1 cache miss code in a separate method.
 return lookup2(charsetName);
}
 
private static Charset lookup2(String charsetName) {
 Object[] a;
 if ((a = cache2) != null && charsetName.equals(a[0])) {
 cache2 = cache1;
 cache1 = a;
 return (Charset)a[1];
 }
 
 Charset cs;
 if ((cs = standardProvider.charsetForName(charsetName)) != null ||
 (cs = lookupExtendedCharset(charsetName)) != null ||
 (cs = lookupViaProviders(charsetName)) != null)
 {
 cache(charsetName, cs);
 return cs;
 }
 
 /* Only need to check the name if we didn't find a charset for it */
 checkName(charsetName);
 return null;
}
```

</div>Wooha again. Thats quite impressive code. Also note the comment <code>// We expect most programs to use one Charset repeatedly.</code>. Well thats not exactly true. We need to use charsets when we have more than one and need to convert between them. But yes, for most internal usage this will be true.Equipped with this knowledge, I can easily write a <a href="http://openjdk.java.net/projects/code-tools/jmh/">JMH</a> benchmark that will nicely show the performance difference between these two <code>String.getBytes()</code> calls. The benchmark can be found in <a href="https://gist.github.com/CodingFabian/11261644">this gist</a> . On my machine it produces this result:

```
Benchmark Mean Mean error Units
preJava7CharsetLookup 3956.537 144.562 ops/ms
postJava7CharsetLookup 7138.064 179.101 ops/ms

```

The whole result can be found in the gist, or better: obtained by running the benchmark yourself. But the numbers already speak for themselves: By using the StandardCharsets, you not only do not need to catch a pointless exception, but also almost double the performance of the code 🙂

Faster and Cleaner Code since Java 7

Working with time and date in any programming language is a mess. To get an idea how much of an mess it actually is, I can recommend many reads, but the following video by Tom Scott nicely sums it up:<div><div><div> <img src="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/3zZog0F7RWldFeAUvUTg" alt="YouTube" /><div>By loading the video, you agree to YouTube's privacy policy. <a href="https://policies.google.com/privacy?hl=en&gl=en" target="_blank">Learn more</a> <a href="#">Load video</a> Always unblock YouTube</div></div></div><div></div></div>Java had two tries to come up with a good date and time implementation: java.util.Date and java.util.Calendar, but both failed. Then <a href="https://twitter.com/jodastephen">Stephen Colebourne</a> came along and invented <a href="http://www.joda.org/joda-time/">joda-time</a> . Joda-Time is one of my favourite libraries and is without any doubt the best implementation of time and date handling you can find for Java. It was so good, that it became the foundation of <a href="https://jcp.org/en/jsr/detail?id=310">JSR-310</a> , the newest attempt at a core Java API for date management which is now shipped with Java 8, in many aspects. While profiling some of our code recently, I finally saw the opportunity to say “thank you, Stephen” in my own way: by contributing patches. One of the key features in Joda-Time is thread safety. It allows formatters and chronologies to be reused by multiple threads. But how exactly they are configured is up to the developer. It is possible to come up with any format pattern (like, “dd. hh:mm – yy”) and use it. Because the creation of formatters is quite expensive, Joda Time will automatically cache them. In fact the caching guarantees are pretty strong: For all possible configurations of formatters and chronologies, only a single instance will ever exist and you always will get that one back.To achieve this, the caches were “synchronized”.Here is one example:<div>

```java5
Map<String, DateTimeFormatter> cCache = new HashMap<String, DateTimeFormatter>()
 
private DateTimeFormatter getFormatter(Locale locale) {
 locale = (locale == null ? Locale.getDefault() : locale);
 String key = Integer.toString(iType + (iDateStyle << 4) + (iTimeStyle << 8)) + locale.toString();
 DateTimeFormatter f = null;
 synchronized (cCache) {
 f = cCache.get(key);
 if (f == null) {
 String pattern = getPattern(locale);
 f = DateTimeFormat.forPattern(pattern);
 cCache.put(key, f);
 }
 }
 return f;
}
```

</div>This code is typical for cache implementations. It is safe programming. In fact I am glad that the code is already safe, which unfortunately a lot of caching code is not. It is, however, not up to date to “modern” Java programming standards.Here is how it works: After having computed a cache key, which involves Locale.toString() (a method which is surprisingly expensive), all multithreaded access is synchronized on the cache instance. Then the entry for the key is looked up. If it is not found, the entry is computed and added to the cache. And finally the synchronization ends, allowing other threads to look into the cache.The main reason why this is inefficient is that threads need to wait, even if they would access a different value. Some people believe that the JVM optimizes the synchronized block. In fact it does, when it sees access by a single thread only. But in many caching cases, this is just not true. The cache is accessed from plenty of threads and this involves the object monitor overhead.So the obvious idea is to not perform locking/synchronization when just reading from the cache. Thankfully Doug Lea already developed a Map which exactly does that: java.util.concurrent.ConcurrentHashMap. It is a general purpose concurrent map, which does not lock on reads. While it is possible to write even faster or more efficient data structures, it performs really well as a drop in replacement for the above code.The rewritten code looks like this:<div>

```java5
private DateTimeFormatter getFormatter(Locale locale) {
 locale = (locale == null ? Locale.getDefault() : locale);
 StyleFormatterCacheKey key = new StyleFormatterCacheKey(iType, iDateStyle, iTimeStyle, locale);
 DateTimeFormatter f = cCache.get(key);
 if (f == null) {
 f = DateTimeFormat.forPattern(getPattern(locale));
 DateTimeFormatter oldFormatter = cCache.putIfAbsent(key, f);
 if (oldFormatter != null) {
 f = oldFormatter;
 }
 }
 return f;
}
```

</div>Whats different? Well first of all, I introduced class called StyleFormatterCacheKey, which gets the values previously used to compute the cache key. Why is that? Well the java.util.HashMap and java.util.concurrent.ConcurrentHashMap both will perform equals() and hashCode() calls on the key. It is more efficient to delegate these calls to the original data structures, instead of creating a java.lang.String which will than perform its own computations. As you can see cChache.get() is no longer synchronized. Gets can run through incredibly fast.But what happens on a miss? What happens here is that possibly multiple threads create a new org.joda.time.format.DateTimeFormat and try to put it to the cache. The tricky part is putIfAbsent(), which ensures that only the first entry added to the map is actually used. Any later formatters by other threads are not added to the cache. Here now the strong caching guarantee comes into play. If our code shall never see two different formatters, we have to then discard our own computed formatter and return the one which was the first one added to the cache. If your cache has less strict requirements, you could even get away with ignoring the return value or replacing the value in the map by calling just put().So, you might wonder, how this actually performs? I have done a quite simplistic benchmark using the great <a href="http://openjdk.java.net/projects/code-tools/jmh/">JMH benchmarking framework</a> . It assumes the fact that the same pattern is used all the time (so we have concurrency on insertion and read) by multiple threads. What I did not tell so far is that the above getFormatter() call is happening behind the scenes whenever you print or parse dates, so the benchmark is actually doing that.Here its result I got when running with Java 8 on my MacBookPro: Old code single threaded<blockquote>

```
Mode Samples Mean Mean error Units
thrpt 20 2022.227 46.700 ops/ms

```

</blockquote>Old code multi threaded (3 threads)<blockquote>

```
Mode Samples Mean Mean error Units
thrpt 20 5806.483 133.980 ops/ms

```

</blockquote>New code single threaded<blockquote>

```
Mode Samples Mean Mean error Units
thrpt 20 2324.084 30.112 ops/ms

```

</blockquote>New code multi threaded (3 threads)<blockquote>

```
Mode Samples Mean Mean error Units
thrpt 20 6228.543 174.374 ops/ms

```

</blockquote>(full original benchmark at: <a href="https://gist.github.com/CodingFabian/9088631">https://gist.github.com/CodingFabian/9088631</a> )So, it’s not exciting, is it? Well, it shows that coming up with an appropriate benchmark is hard. It is a “good case” benchmark, where we do not have many writes, which would prolong the synchronized block. Still the java.util.concurrent.ConcurrentHashMap usage is 10% faster with similar characteristics with regards to scaling with threads.So, should we care? Hmm, maybe. Let’s look at a different example. This time we want to use a org.joda.time.chrono.BuddhistChronology to calculate buddhistic dates:<div>

```java5
BuddhistChronology.getInstance(DateTimeZone.getDefault());
```

</div>Lets do a benchmark first, shall we?Single threaded:<blockquote>

```
Mode Samples Mean Mean error Units
thrpt 20 26315.317 278.507 ops/ms

```

</blockquote>Multi threaded using 3 Threads:<blockquote>

```
Mode Samples Mean Mean error Units
thrpt 20 13817.764 264.348 ops/ms

```

</blockquote>(original full benchmark at: <a href="https://gist.github.com/CodingFabian/9058922">https://gist.github.com/CodingFabian/9058922</a> )Wait, what? Running multiple threads actually drastically reduces the performance? Here is the code:<div>

```java5
public static synchronized BuddhistChronology getInstance(DateTimeZone zone) {
 if (zone == null) {
 zone = DateTimeZone.getDefault();
 }
 BuddhistChronology chrono;
 synchronized (cCache) {
 chrono = cCache.get(zone);
 if (chrono == null) {
 // First create without a lower limit.
 chrono = new BuddhistChronology(GJChronology.getInstance(zone, null), null);
 // Impose lower limit and make another BuddhistChronology.
 DateTime lowerLimit = new DateTime(1, 1, 1, 0, 0, 0, 0, chrono);
 chrono = new BuddhistChronology(LimitChronology.getInstance(chrono, lowerLimit, null), "");
 cCache.put(zone, chrono);
 }
 }
 return chrono;
}
```

</div>This looks awfully like the previous example, does it? Again cCache is a java.util.HashMap. Notice that the whole method is synchronized. Here is what i changed it to:<div>

```java5
public static BuddhistChronology getInstance(DateTimeZone zone) {
 if (zone == null) {
 zone = DateTimeZone.getDefault();
 }
 BuddhistChronology chrono = cCache.get(zone);
 if (chrono == null) {
 // First create without a lower limit.
 chrono = new BuddhistChronology(GJChronology.getInstance(zone, null), null);
 // Impose lower limit and make another BuddhistChronology.
 DateTime lowerLimit = new DateTime(1, 1, 1, 0, 0, 0, 0, chrono);
 chrono = new BuddhistChronology(LimitChronology.getInstance(chrono, lowerLimit, null), "");
 BuddhistChronology oldChrono = cCache.putIfAbsent(zone, chrono);
 if (oldChrono != null) {
 chrono = oldChrono;
 }
 }
 return chrono;
}
```

</div>not much difference. I removed the synchronized keywords and used a java.util.concurrent.ConcurrentHashMap.Eager to see what effect it had? Lets run the same bench again:Single threaded:<blockquote>

```
Mode Samples Mean Mean error Units
thrpt 20 198646.049 2716.604 ops/ms

```

</blockquote>Multi threaded using 3 threads:<blockquote>

```
Mode Samples Mean Mean error Units
thrpt 20 554412.528 21280.949 ops/ms

```

</blockquote>No, I am not cheating. This is real. We get a ten times increase in performance even in single threaded usage! And it suddenly scales in a linear manner rather than crash and burn.<h2>Conclusion</h2>synchronized is a Java keyword, and the JVM can do arcane tricks to optimize it. But sometimes the JVM or the developer fails with it. java.util.concurrent.ConcurrentHashMap is a much better alternative, because its semantics and performance are defined by its API. It should be the go-to solution for building multithreaded accessed maps.<h3>References</h3>1. Commit optimizing Chronology caches: <a href="https://github.com/JodaOrg/joda-time/commit/37ce3704787a4967a6238e3f3f56894f7554daef">https://github.com/JodaOrg/joda-time/commit/37ce3704787a4967a6238e3f3f56894f7554daef</a> related benchmark <a href="https://gist.github.com/CodingFabian/9058922">https://gist.github.com/CodingFabian/9058922</a> 2. Commit optimizing Formatter caches: <a href="https://github.com/JodaOrg/joda-time/commit/acfec45d0324e41b8159cf2e63071390893e2037">https://github.com/JodaOrg/joda-time/commit/acfec45d0324e41b8159cf2e63071390893e2037</a> Related benchmark <a href="https://gist.github.com/CodingFabian/9088631">https://gist.github.com/CodingFabian/9088631</a>

Writing better Caches in Action: Improving Joda Time

<img src="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/OfuKEXo0TzuQWLsUNKfp" alt="tabletopday1" width="400" height="544" /> codecentric participated this year again at the international TableTopDay, where we celebrate geekdom by playing board games all day. <a href="/en/knowledge-hub/blog/international-tabletopday-at-codecentric">As last year</a> , the event was open to everybody. Over the course of the day, about 30 people played in our Solingen office. We started at 10am and had planned for 8-10 hours of gaming, but in fact the last players left shortly after midnight. I would like to thank everybody attending for making this an awesome event and introducing others to new games. If you did not attend, but are interested in the games we played, Ill give a short summary, so that you can prepare for next year. Until then: Play more games (as <a href="https://twitter.com/wilw">@wilw</a> says). <hr /><h3>Dominion</h3><img loading="lazy" src="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/I1BWir6sQs2kUhSFJAQb" alt="tabletopday2" width="600" height="490" /> Dominion is a deck building card game by Donald X. Vaccarino. The goal is to accumulate as many point cards as possible, but because players use all their cards to play, getting points early, can actually prevent playing effectively in the later game. It is up to the player to find a good mix of money, action and point cards to stay ahead. The game has numerous expansions with plenty of alternative cards, giving it huge replayability. No game will ever be the same.<hr /><h3>Lords of Waterdeep</h3><img loading="lazy" src="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/9FPDpnjwT2KcSayz09PG" alt="tabletopday3" width="600" height="512" /> Lords of Waterdeep is a role playing strategy game by Peter Lee and Rodney Thompson. Players assume roles of the Lords of Waterdeep and play according to their own plans. A key factor is that the identity is not known until the end, leaving room for misdirection and intrigues. Due to many rules, this game is a bit more complicated to get started with. Watching the <a href="https://www.youtube.com/watch?v=D7DvEIfrP84">TableTop episode</a> with host Wil Wheaton and guests Felicia Day, Pat Rothfuss and Brandon Laatsch can help to get you started.<hr /><h3>Kingsburg</h3><img loading="lazy" src="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/H4zdAV9kRnSKwly8fHJc" alt="tabletopday4" width="600" height="484" /> Kingsburg is a dice rolling game by Andrea Chiarvesio and Luca Iennaco. It involved more strategy than you would expect, but it is also possible to get screwed by dice. With the dice roll results players obtain various resources. Once a player takes it with a roll, it is blocked for others until the end of the turn. This adds quite a competitive aspect to the game. Resources then can be spent on buildings which give access to special abilities. Due to the high amount of luck and blocking strategies, I would only recommend to play with people you like 🙂<hr /><h3>Voll auf die Glocke</h3><img loading="lazy" src="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/shvpok8VQuWbe1tzWnG2" alt="tabletopday5" width="600" height="512" /> Voll auf die Glocke is a simple card trading game where players are teamed up and trade cards around. When a player completes a set, she has to sign her teammate to ring the bell (German: “Glocke”). But if the other team notices the sign, they can hit the bell on their own even with not having the set completed.<hr /><h3>Tac</h3><img loading="lazy" src="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/6K3rShdMRKmsx39VnO8h" alt="tabletopday7" width="600" height="539" /> Tac is an Parcheesi like board and card game by Kolja Sparrer, where players have to get their balls across the board by using cards instead of dice, reducing the luck aspect of similar games which use dice for that.<hr /><h3>Small World</h3><img loading="lazy" src="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/X6G3OedLS2iwgGKlJYUH" alt="tabletopday8" width="600" height="495" /> Small World is an area control game by Philippe Keyaerts. it has a lot of similarities to Risk, but with the twist, that players control fantasy races like halflings or merfolk. Every race has special abilities which can be modified by additional attributes. How about flying orcs? The second twist is that players should play a race only for a limited time and then switch to a different one. The game is easy to learn and the interesting race and ability combinations give some amount of replayability. Trivia: <a href="https://www.youtube.com/watch?v=X9QtdiRJYro">The TableTop Episode</a> featuring Host Wil Wheaton and guests Jenna Busch, Grant Imahara and Sean “Day9” Plott was the first episode aired and the first to reach over one million views.<hr /><h3>Robo Rally</h3><img loading="lazy" src="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/ILGwcOOySjKuPogedBiL" alt="tabletopday6" width="600" height="579" /> Robo rally is a sequence planning game by Richard Garfield. Every player controls a robot and needs to get that robot to pass multiple checkpoints across the map. The map is filled with obstacles like conveyer belts and lazers. To make the robot run, players need to program it with random movement cards they get. After all robots are programmed the program is executed and robots and the board moves. Due to unexpected player interaction, things can get hilarious quickly. Many robots have ben crushed, fallen of the board or lasered into pieces in our play.<hr /><h3>Forbidden Island</h3><img loading="lazy" src="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/R8PMDCpRzmj9eLXW8NKV" alt="tabletopday9" width="600" height="541" /> Forbidden Island is an objective focused cooperative game by Matt Leacock, where players have to gather treasures from an island which is sinking. It has multiple scenarios, and different player character abilities, which are required to get all objectives before they are lost to the water.<hr /><h3>Zombicide</h3><img loading="lazy" src="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/LmzZ0jOHTPGPxsRJav38" alt="tabletopday12" width="600" height="530" /> Zombicide is a cooperative survival game by Raphaël Guiton, Jean-Baptiste Lullien and Nicolas Raoult. Players play together using their different survivor abilities to complete scenarios. Depending on the scenario things escalate quickly and zombies fill the board. The game gets increasingly difficult every single turn, which requires the players to discuss optimal movements a lot. The game is a huge time sink. We completed our scenario after 6 hours, and it was very close. <img loading="lazy" src="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/ut4xFjaATiCCwWvrUGOL" alt="tabletopday13" width="600" height="391" /><hr /><h3>Heckmeck am Bratwurmeck</h3><img loading="lazy" src="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/Wclci6bTo21Usmd6JE2W" alt="tabletopday10" width="600" height="493" /> Heckmeck am Bratwurmeck is a dice rolling game by Reiner Knizia. It requires the player to re-roll her dices to possibly get more points, which adds a strategic element to an otherwise heavy luck based game.<hr /><h3>Cafe International</h3><img loading="lazy" src="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/Q9SkjlZROK8RHj8jXXRo" alt="tabletopday11" width="600" height="568" /> Cafe International is a tile laying game by Rudi Hoffmann, where players have to balance the tables in a restaurant. Seating guests has to follow some rules and has differs in scoring.<hr /><h3>Agricola</h3><img loading="lazy" src="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/2miWFJqfRlWD1SntBjg7" alt="tabletopday14" width="600" height="536" /> Agricola is a farming themed resource management strategy game by Uwe Rosenberg. Players build their farms by gathering resources. Players need to balance the needs of farmers and animals to provide housing and food, as well as to improve the farm. It is a fairly complex game, because it involves complicated action chains: Fo example, to provide bread to farmers, players have to plow, sow, wait for wheat to grow, reap and bake. All requiring multiple turns to complete.<hr /><h3>One-Player-Special: Rubiks Cube</h3><a href="https://twitter.com/mirko_novakovic">Mirkos</a> son Luca discovered a rubiks cube we had as decoration. He was busy for a while trying to solve it. After a while he came back with it to his dad, asking him to solve it. The picture shows Mirko trying to use a tutorial on youtube to achieve that. <img loading="lazy" src="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/mfoavWnRTqKFRayTtYL5" alt="tabletopday-special" width="730" height="914" />

TableTopDay 2014 at codecentric

When I run the following Java program, which consumes all free memory with artificial data structures, something interesting happens. Something very similar happens to all real applications: <a href="https://gist.github.com/CodingFabian/8708393">https://gist.github.com/CodingFabian/8708393</a> 

```

[node04] ~ ➜ jdk1.7.0_51/bin/java -Xms31g -Xmx31g -Xmn50m Memory
Total Memory (in bytes): 33279705088
Free Memory (in bytes): 33278908064
Max Memory (in bytes): 33279705088
Elements created and added to LinkedList: 587889429

```

The parameters cause the JVM heap to be 31GB big, and the space used for object allocation only 50MB. Lets try a higher max memory setting of 32GB and see what happens:

```

[node04] ~ ➜ jdk1.7.0_51/bin/java -Xms32g -Xmx32g -Xmn50m Memory
Total Memory (in bytes): 34353446912
Free Memory (in bytes): 34352649800
Max Memory (in bytes): 34353446912
Elements created and added to LinkedList: 385481085

```

What is happening here? With more memory, we can create less elements? Oops! <h2>Oop = Ordinary object pointer</h2> Yes, Oops are actually the reason. An oop is the memory address of an object. In the 32 bit world, this can go from 0 to 2^32-1. Which happens to be 4G, which means every byte of a 4GB heap can be addressed. To manage larger amounts of memory the JVM actually needs to run with 64 bit addresses, which can manage 2^64-1 bytes which are about 18.5 Exabyte (a lot). What this means is that for a twice as big address, a much larger memory area can be addressed. But there are not that many applications that really need 18.5 Exabyte of memory (let alone there are no servers which have so much memory). When people switched from a 32bit JVM to a 64bit JVM they noticed the loss of available memory due to larger addresses and complained. The JVM developers invented a feature called CompressedOops to fight it. Because the JVM memory layout uses a 8byte addressing scheme, which means that objects can be at address 0, 8, 16, 24… but not at 2, 7 or any other non multiple of 8, compressed oops just address the virtual positions 0, 1, 2, 3 instead of the real ones 0, 8, 16, 24. To get from the compressed address to the real one, the JVM needs just to left shift it 3 times. Easy. But, as you might imagine, this fails for heaps larger 32GB. The trick would actually work on a 32bit VM and enable addressing 8 times the memory which is 4×8 = 32GB. But sadly the 32 bit operating systems can’t do that. As soon as the JVM needs to manage 32GB+ heaps, it cannot do the CompressedOops trick anymore. All object addresses become 64 bit. When going from 31GB, to 32GB in my example, about 200 million less objects can fit into the heap. Yuk. To accommodate the same amount of objects one would need at least a 48GB heap:

```

[node04] ~ ➜ jdk1.7.0_51/bin/java -Xms48g -Xmx48g -Xmn50m Memory
Total Memory (in bytes): 51533316096
Free Memory (in bytes): 51532518984
Max Memory (in bytes): 51533316096
Elements created and added to LinkedList: 578932717

```

We can turn the compression off, which shows us how much it actually helps us:

```

[node04] ~ ➜ jdk1.7.0_51/bin/java -XX:-UseCompressedOops -Xms31g -Xmx31g -Xmn50m Memory
Total Memory (in bytes): 33279705088
Free Memory (in bytes): 33278907976
Max Memory (in bytes): 33279705088
Elements created and added to LinkedList: 373400520

```

As you can see the difference is dramatic. My demo application, as well as most real life applications consist of a large amount of references. Due to that most applications will be unable to utilize the memory between 32GB and mid 40’s GB. Bummer!<h4>Appendix</h4><ul><li>Compressed Oops explained by Oracle: <a href="https://wikis.oracle.com/display/HotSpotInternals/CompressedOops">https://wikis.oracle.com/display/HotSpotInternals/CompressedOops</a> </li><li>Server used is a Dell Power Edge 510 24 core (2xE5645), 128GB, running Debian Wheezy and JDK 7.51, part of our <a href="https://www.centerdevice.de">centerdevice.de</a> infrastructure.</li></ul>

Why 35GB Heap is Less Than 32GB – Java JVM Memory Oddities

There have been plenty of talks on JavaOne 2013 about the release of Mission Control and Flight Recorder with Java 7 update 40. The release was anticipated for a long time (since 2010 actually) and now was finally done. But why do people get so excited about it? Well, Flight Recorder and Mission Control were tools produced by BEA (then Oracle) for the JRockit JVM. For a long time they have been an important differentiator for JRockit, because they were adding monitoring and profiling capabilities to the JVM which were unmatched by Sun or IBM. After Oracle aquired Sun it was announced that the product lines will merge eventually. Now the convergence between JRockit and HotSpot finally came to a point where these two tools are available in the HotSpot JVM as well. Before discussing my point of view on that further, please head over to my <a href="http://www.infoq.com/news/2013/10/misson-control-flight-recorder">article on Mission Control and Flight Recorder on InfoQ</a> and come back when you have read it 🙂 So, you are back? Good, now that you know what these tools are about, I can continue with my personal opinion about them. First off, you might have noticed that they are not only “tools”, but actually are integrated deep into the JVM. Without the event generating code they could not display anything VisualVM could not display. And while there is an API to add additional events, there is no API to actually read the data. This is a closed system! When Sun was in charge, they added the JVMTI and JavaAgent interfaces for monitoring and profiling which were open interfaces. They could be used by third party vendors to build powerful tools on top. As codecentric we have always recommended the best monitoring and profiling tools available to our customers. All of the tools were either using JVMTI, like JProfiler does, or JavaAgent, like the APM Solution AppDynamics. There has been a lot of competition on the tooling market, but it was based on fair ground. All vendors did have the same basis to build on. But now the situation has changed. From now on, there is a JVM internal interface and a Oracle proprietary tool, which claim to offer “very low overhead always on profiling and monitoring” nobody else will be able to provide. If the claim holds true, and I am currently evaluating that, this is a big setback for the third party tools which do not have access to the new functionality, which for example contains the ability to do sampling which does not involve thread cooperation via safepoints. There is little hope, that Oracle will make the technology available to third parties, which means that the best tools might in future only come from Oracle. That would be sad, because the openness of the Java ecosystem is considered to be the main advantage over .Net. But for the situation right now, it seems that licensing will be a relevant topic. Even with a seemingly superior technology, the production license of Flight Recorder comes at 5,000 or 15,000 USD. And that is per processor. JProfiler costs about 500 USD per installation which allows profiling any system (but usually not suitable for production) and AppDynamics is at about 3,000 per production JVM (not processor). So even when the profiling capabilities no longer match what Oracle is now able to deliver, the price will make a difference.

Finally – Mission Control and Flight Recorder for HotSpot

<img src="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/nb5WeaBKRai0Xwt094xH" alt="TableTopDay_200x200" width="200" height="200" />All the geeks at codecentric were pretty excited about TableTopDay, a table top board gaming holiday invented by the fine folks at GeekAndSundry, Felica Day and Wil Wheaton. As response to their extremly successful <a href="http://www.youtube.com/playlist?list=PL4F80C7D2DC8D9B6C">YouTube Show “TableTop”</a> , which accumulated already over 8.5 million views, they proclaimed March 30th to be International TableTopDay. Over 2500 groups registered their event on tabletopday.com and so did we. While I expected only codecentrics to attend, we ended with one guy working for one of our customers, one random guy who registered on our Google+ Event page and 4 strangers who showed up without any notice. But also all the codecentrics brought their families. It was successful and fun event, for us and for all board game nerds worldwide, as proven by the hashtag <a href="https://twitter.com/search?q=%23TableTopDay&src=hash">#TableTopDay</a> , which was trending on Twitter the whole Saturday, and many people posting pictures or even livestreaming their event. Games we played:<ul><li><a href="http://www.amazon.de/gp/product/B001BAUGW0/ref=as_li_ss_tl?ie=UTF8&camp=1638&creative=19454&creativeASIN=B001BAUGW0&linkCode=as2&tag=exfuror">Pandemic</a> </li><li><a href="http://www.amazon.de/gp/product/B001REM4KC/ref=as_li_ss_tl?ie=UTF8&camp=1638&creative=19454&creativeASIN=B001REM4KC&linkCode=as2&tag=exfuror">Small World</a> </li><li><a href="http://www.amazon.de/gp/product/B0014LETVU/ref=as_li_ss_tl?ie=UTF8&camp=1638&creative=19454&creativeASIN=B0014LETVU&linkCode=as2&tag=exfuror">Stone Age</a> </li><li><a href="http://www.amazon.de/gp/product/B000KY536C/ref=as_li_ss_tl?ie=UTF8&camp=1638&creative=19454&creativeASIN=B000KY536C&linkCode=as2&tag=exfuror">Fearsome Floors</a> </li><li><a href="http://www.amazon.de/gp/product/B002IUFSPM/ref=as_li_ss_tl?ie=UTF8&camp=1638&creative=19454&creativeASIN=B002IUFSPM&linkCode=as2&tag=exfuror">Castle Panic</a> </li><li><a href="http://www.amazon.de/gp/product/B0036X00EE/ref=as_li_ss_tl?ie=UTF8&camp=1638&creative=19454&creativeASIN=B0036X00EE&linkCode=as2&tag=exfuror">Hansa Teutonica</a> </li><li><a href="http://www.amazon.de/gp/product/B004LWF076/ref=as_li_ss_tl?ie=UTF8&camp=1638&creative=19454&creativeASIN=B004LWF076&linkCode=as2&tag=exfuror">Zombie Dice</a> </li><li><a href="http://www.amazon.de/gp/product/B006WY1X1S/ref=as_li_ss_tl?ie=UTF8&camp=1638&creative=19454&creativeASIN=B006WY1X1S&linkCode=as2&tag=exfuror">Munchkin</a> </li></ul>And maybe others; I was busy winning, could not watch all games being played 🙂 Thanks to all people attending making this possible, and special thanks to codecentric for hosting this event! Looking forward to next year, or earlier whenever possible![slideshow id=22 w=700 h=400]Play more games!

International #TableTopDay at codecentric

In this post I want to give you an insight into the architecture of <a href="https://www.centerdevice.de">CenterDevice</a> , a document management and collaboration tool for the enterprise hosted in our own cloud datacenter in Germany. CenterDevice is a startup of codecentric, with a few codecentrics working full or part time on it.Let me start with a screenshot from our AppDynamics monitoring. <img src="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/LyS0qMi9Sw6MiVo7gYBE" alt="centerdevice-architecture" width="699" height="316" /> The green boxes are all Java 7 based services monitored with AppDynamics. The number “4” within the blue circle indicates that at the moment there are currently 4 instances each running. This number can change when services are scaled up and down. We currently run a mix of KVM virtualized and non-virtualized CentOS instances on high end Dell Machines with a total of about 148 cores, 600GB memory and 150TB disc space. The lines connecting the boxes indicate call directions, their average response times and how often they were made in that time interval. What is really great about AppDynamics is that the services, and how they talk to each other is automatically detected. In the middle of the picture is the heart of our architecture “tomcat-rest”. When you talk to

```
api.centerdevice.de/v1
```

 you arrive there (and see a message that tells you you are missing our oAuth authorization – we are planning to publish our API soon). It hosts all of our REST services, which are implemented with Jersey. We chose Jersey, because it is very easy to implement REST Services with it, is a proven piece of software and also easily testable in unit and integration tests as outlined in this excellent blog post by Michael Lex about <a href="/en/knowledge-hub/blog/writing-lightweight-rest-integration-tests-with-the-jersey-test-framework">Writing lightweight REST integration tests with the Jersey Test Framework</a> . Services accessing any kind of data use our MongoDB backend, visualized by AppDynamics as blue boxes on the top. As a scalable database with easy access and painless schema changes (because there is no schema), it suits our performance requirements, as well as the changeability we require for a young product with frequent releases. If you want to know more about MongoDB I recommend reading Thomas Jaspers <a href="/en/knowledge-hub/blog/mongodb-tutorial">MongoDB tutorial</a> . We talk to it via the Java Driver, have implemented replicasets and shard to multiple nodes. The major challenge here is to accept the eventual consistency. Even code which writes data might not yet be able to read it again in the next line. Uploaded documents are stored on the Gluster backed XFS file system. All data is encrypted using a per user 256 AES key before it is persisted on the file system. Gluster then takes care of replicating the data to all servers. For clients, we developed two native clients: <a href="https://itunes.apple.com/de/app/centerdevice/id933144073?mt=8">CenterDevice for iPad</a> and <a href="https://play.google.com/store/apps/details?id=de.centerdevice.android&hl=en">CenterDevice for Android Phone</a> . The reason for native apps is that some emulated or cross compiled platforms have plenty of issues. Mark Zuckerberg also recently had a widely acknowledged talk about the <a href="http://techcrunch.com/2012/09/11/mark-zuckerberg-our-biggest-mistake-with-mobile-was-betting-too-much-on-html5/">problems HTML5 mobile clients have</a> at the moment. And because iOS dominates the tablet market and Android rules the phone market, that is what we started with. <img loading="lazy" src="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/yw4i9G4SRcWHspIHReh9" alt="centerdevice-mobile-clients" width="696" height="361" />Note that our screenshots show cute baby animals, instead of boring documents, because they are more adorable. Besides our native iPad and Android Phone application, and a few third party clients, the main user of the REST Server is our Vaadin 6 based web client. In fact we have two of them

```
app.centerdevice.de
```

 and

```
public.centerdevice.de
```

Both are logically hosted on “tomcat-centerdevice”. Vaadin, as a framework, allows us to implement complex rich web interfaces easily, using SWT Style Java code. It compiles to GWT JavaScript which is then delivered to the browser. The bulk of the work can be easily done in Vaadin, which we combine with CDI-utils, a plugin that allows implementing the MVP pattern easily using Weld as CDI library. Developing complex components however requires developing GWT Widgets yourself sometimes, which is not that easy. For easy copy to clipboard functionality, Henning Treu wrote a <a href="/en/knowledge-hub/blog/copy-to-clipboard-with-vaadin">Copy to clipboard vaadin addon</a> , which we open sourced. <img loading="lazy" src="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/F0TDVqt1S4Geg7yR8c6F" alt="centerdevice-web-app" width="690" height="382" /> Communication between the web application servers and the rest server is only unidirectional, but sometimes we want to send back notifications (like new documents somebody just shared with you) to the webserver. That is where RabbitMQ comes into play. The application map from AppDynamics shows our 3 usages of messaging:<ul><li>Sending Notifications from REST to Web.</li><li>Sending requests to send e-Mails (currently sent from and consumed by the REST Server).</li><li>Sending processing requests from REST server to doc-server.</li></ul>RabbitMQ is set up using HA queues and worked flawlessly so far. Tobias Trelle wrote an <a href="/wissens-hub/blog/amqp-messaging-with-rabbitmq">introduction into RabbitMQ with Spring, which provides more background about Rabbit MQ and AMQP.</a> The document processing done by “doc-server”, which has multiple tasks depending on the type of input document:<ul><li>Generating PDF representations</li><li>Generating preview images for different sizes</li><li>Performing fulltext extraction</li><li>Performing OCR</li><li>Getting page count</li><li>Obtaining additional metadata</li><li>Detecting language</li></ul>There are basically two types of documents that we use as basis: PDF and Images. If we are getting any other format, we use Libre Office to convert it to PDF, or Imagemagick to convert it to images. ImageMagick is then also used to generate preview images in various sizes. Depending on the type of document we can use Apache Tika to get the fulltext from the document. For documents where Tika cannot find a fulltext, we resort to OCR running on “tomcat-ocr”. OCR will be done using tesseract. Further metadata is extracted using Tika and custom detectors. Search capabilities are provided by Apache Solr 3, running on “tomcat-solr”. Solr is running in Master Slave mode. One neat feature of CenterDevice is that is performs super fast search on everything we can extract from documents or their metadata.<h3>Performance</h3> As you can see in the screenshot, the overall performance is quite nice. However it largely depends on the documents uploaded for individual processing requests. AppDynamics automatically learns the normal behavior and alerts us when there is deviation. For uploads and downloads however, we turned this learning off. Depending on the document and the clients connectivity, there is just nothing like a normal time a client takes to upload or download a document. We however gather diagnostic data for extraordinary slow up/downloads to investigate in case of customer complaints. A similar story applies to the document processing. While in most cases also the learned baselines are good, sometimes they do not match. That is why in the screenshot the connection to our external processing services are red. Some heavy document processing was going on at the time the screenshot was taken. When performance degrades, AppDynamics captures important metrics and provides us code level insight into the root cause. So far the most issues were typical like too many queries, too much API calls, inefficient indicies etc.<h3>ToDos</h3> We always have ideas for improving stuff and are moving fast, so the architecture will change. While currently we do not have a pressing need, these changes will be most likely coming in future:* Introduce WebSession replication for failover using memcache or redis. (currently we loose web session data on failure (not happened so far), and deployments (happening during nights when no sessions are alive)). The major challenge so far seems to get it integrated into the Servlet 3 async pushing we do. * Switch from Gluster to Hadoop File System (We were bitten by lots of Gluster problems, like running on ext-4 64bit, which we now changed to xfs) * Switch from Solr 3 to Elastic Search (the master-slave failover just does not work as nicely and does not scale, plus changes to Solr 3 need downtime) * Add a reverse proxy / loadbalancer layer to perform green / blue deployments, redirect specific users to certain versions of the software.<h3>Join the team!</h3> I hope you enjoyed the overview on architecture and software we use. If you are interested in helping us build and grow this stack, we have good news for you: CenterDevice and codecentric are hiring! We are especially looking for a dev-opsy Linux and Hardware enthusiast to build out what I described above. Get in touch!

The CenterDevice Cloud Architecture

During Attila Szegedis talk about “<a href="http://gotocon.com/dl/goto-amsterdam-2011/slides/AttilaSzegedi_JVMPerformanceOptimizationsAtTwittersScale.pdf">lessons learned about the JVM</a> “, I was surprised that he emphasized the importance of knowing the amount of data you store in memory. It is not common to be concerned about object size in enterprise Java programming, but he gave a good example of what they had to do Twitter.<h2>Recap: Memory Footprint of Data</h2>Question: How much memory does the String “Hello World” consume? Answer: 62/86 Bytes (32/64 bit Java)! This breaks down into 8/16 (Object Header for String) + 11 * 2 (characters) + [8/16 (Object Header char Array) + 4 (array length) padded to 16/24] + 4 (Offset) + 4 (Count) + 4 (HashCode) + 4/8 (Reference to char Array). [On 64Bit the size of String Object is padded to 40].<h2>The Problem</h2>Imagine you have a lot of Locations attached to tweets in your data store. The implementation of the location as a Java class could look like this<div>

```java5
class Location {
	String city;
	String region;
	String countryCode;
	double long;
	double lat;
}
```

</div>So if you load all the locations of tweets ever made, it is quite obvious that you load a lot of String objects, and at the scale Twitter has, there are for sure a lot of duplicate Strings. Attila said that this data did not fit into a 32 GB heap. So the question is: can we reduce memory consumption, so that all Locations fit into memory? Let us have a look at two possible solutions, which can even augment each other.<h2>Attilas Solution</h2>There is a, more or less, hidden dependency between the data stored in the Location class, which, once realized, will solve the problem in an elegant way, non-technical way. We can just apply normalization to the Object and split it into two:<div>

```java5
class SharedLocation {
	String city;
	String region;
	String countryCode;
}
class Location {
	SharedLocation sharedLocation;
	double long;
	double lat;
}
```

</div>This is actually a neat solution, because cities rarely change the region and country they reside in. The combination of those Strings is unique. And it is flexible, so that even violations of those uniqueness can be handled. This makes always sense for data which could be user input. By doing so, multiple Tweets from “Solingen, NRW, DE” just consume one SharedLocation. But still “Ratingen, NRW, DE” would store 3 additional String in memory, rather than just the new one “Ratingen”. By refactoring the data model like that the amount of data for that Twitter research project dropped to about 20GB.<h2>String interning</h2>But what to do when you do not want to, or simply cannot refactor the data model? Or the researcher at twitter would not have had a 20GB Heap? The answer is String interning, which keeps every String only once in memory. But there is huge confusion about String interning. Many people ask if it speeds up equals checks, because equal strings are actually identical Strings when using interning. Yes, it might do that (like all objects should)

```

// java.lang.String
public boolean equals(Object anObject) {
 if (this == anObject) {
 return true;
 }
 //...
}

```

But equals performance is not the reason you should do interning. String interning is intended to reuse String objects to save memory.<blockquote>Only use String.intern() on Strings you know are occurring multiple times, and only do it to save memory</blockquote>How effective interning is, is determined by the ratio of duplicate/unique strings. And it depends on whether it is easy to change code at string generating places.<h3>So how does it work?</h3>String interning takes a String instance (so it already exists in the Heap) and checks if an identical copy exists already in a StringTable. That StringTable is basically a HashSet that stores the String in the Permanent Generation. The only purpose of that Table is to keep a single instance of the String alive. If it is in there, the instance is returned. If its not, its added to the String Table:<div>

```c
// OpenJDK 6 code
JVM_ENTRY(jstring, JVM_InternString(JNIEnv *env, jstring str))
 JVMWrapper("JVM_InternString");
 JvmtiVMObjectAllocEventCollector oam;
 if (str == NULL) return NULL;
 oop string = JNIHandles::resolve_non_null(str);
 oop result = StringTable::intern(string, CHECK_NULL);
 return (jstring) JNIHandles::make_local(env, result);
JVM_END
 
oop StringTable::intern(Handle string_or_null, jchar* name,
 int len, TRAPS) {
 unsigned int hashValue = hash_string(name, len);
 int index = the_table()->hash_to_index(hashValue);
 oop string = the_table()->lookup(index, name, len, hashValue);
 
 // Found
 if (string != NULL) return string;
 
 // Otherwise, add to symbol to table
 return the_table()->basic_add(index, string_or_null, name, len,
 hashValue, CHECK_NULL);
}
```

</div>As a result, this specific instance of a String only exists once.<h3>How to use interning</h3>The right place to use String interning is where you read from data store and add the Objects/String to a larger scope. Note that ALL Strings which are hardcoded (as constant or anywhere in code) are automatically interned by the compiler. An example would be:<div>

```java
String city = resultSet.getString(1);
String region = resultSet.getString(2);
String countryCode = resultSet.getString(3);
double city = resultSet.getDouble(4);
double city = resultSet.getDouble(5);
 
Location location = new Location(city.intern(), region.intern(), countryCode.intern(), long, lat);
allLocations.add(location);
```

</div>All newly created location objects will use the interned string. The temporary strings read from the database will be garbage collected.<h3>How to find out how effective would be string interning</h3>You are best off taking a heap dump of a quite full heap. It can even be collected at an OutOfMemoryError. Open it in MAT and select the <code>java.lang.String</code> from the histogram. On that one use “Java Basics” and “Group By Value” <img src="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/yRA2soALRoeR4IVdSLkV" alt title="Mat-find-duplicate-strings" width="752" height="475" /> Depending on the heap size, this may take a long time. In the end the result will be like this. Sorting either along retained heap or number of objects will reveal interesting things: <img loading="lazy" src="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/yr6iXCVhSU2IxpW0qrfD" alt title="Mat-duplicate-strings" width="728" height="476" /> From this screenshot we can see that empty Strings take a lot of memory! 2 million empty Strings take a total of 130MB. Then we see some amount of JavaScript that is loaded, a few more technical strings like the keys, which are used for localization. And we can see some Strings related to business logic. These business logic strings are probably the easiest to intern, because we might know where they are loaded into memory. For the other ones we need to use “Merge shortest Path to GC Root” to identify where they are stored, which may or may not reveal to use where it is created.<h3>Tradeoffs</h3>So why not always do String interning? Because it slows your code down! Here a small example:<div>

```java5
private static final int MAX = 40000000;
public static void main(String[] args) throws Exception {
	long t = System.currentTimeMillis();
	String[] arr = new String[MAX];
	for (int i = 0; i < MAX; i++) {
		arr[i] = new String(DB_DATA[i % 10]);
		// and: arr[i] = new String(DB_DATA[i % 10]).intern();
	}
	System.out.println((System.currentTimeMillis() - t) + "ms");
	System.gc();
	System.out.println(arr[0]);
}
```

</div>The code uses a String array to keep a strong reference to the String objects. and we print the first element in the end to avoid removal of the structure due to optimization. Then we load 10 different Strings from the database. I used new String() here to illustrate the temp String allocation you always do when you read from storage. At the end I do a GC, so that the results are correct and no leftovers are included. This was run on a 64bit Windows, JDK 1.6.0_27, i5-2520M with 8GB Ram. Run with <code>-XX:+PrintGCDetails -Xmx6G -Xmn3G</code> to log all GC activity. Here is the output: Without intern()<div>

```bash
1519ms
[GC [PSYoungGen: 2359296K->393210K(2752512K)] 2359296K->2348002K(4707456K), 5.4071058 secs] [Times: user=8.84 sys=1.00, real=5.40 secs] 
[Full GC (System) [PSYoungGen: 393210K->392902K(2752512K)] [PSOldGen: 1954792K->1954823K(1954944K)] 2348002K->2347726K(4707456K) [PSPermGen: 2707K->2707K(21248K)], 5.3242785 secs] [Times: user=3.71 sys=0.20, real=5.32 secs] 
DE
Heap
 PSYoungGen total 2752512K, used 440088K [0x0000000740000000, 0x0000000800000000, 0x0000000800000000)
 eden space 2359296K, 18% used [0x0000000740000000,0x000000075adc6360,0x00000007d0000000)
 from space 393216K, 0% used [0x00000007d0000000,0x00000007d0000000,0x00000007e8000000)
 to space 393216K, 0% used [0x00000007e8000000,0x00000007e8000000,0x0000000800000000)
 PSOldGen total 1954944K, used 1954823K [0x0000000680000000, 0x00000006f7520000, 0x0000000740000000)
 object space 1954944K, 99% used [0x0000000680000000,0x00000006f7501fd8,0x00000006f7520000)
 PSPermGen total 21248K, used 2724K [0x000000067ae00000, 0x000000067c2c0000, 0x0000000680000000)
 object space 21248K, 12% used [0x000000067ae00000,0x000000067b0a93e0,0x000000067c2c0000)
```

</div>With intern()<div>

```bash
4838ms
[GC [PSYoungGen: 2359296K->156506K(2752512K)] 2359296K->156506K(2757888K), 0.1962062 secs] [Times: user=0.69 sys=0.01, real=0.20 secs] 
[Full GC (System) [PSYoungGen: 156506K->156357K(2752512K)] [PSOldGen: 0K->18K(5376K)] 156506K->156376K(2757888K) [PSPermGen: 2708K->2708K(21248K)], 0.2576126 secs] [Times: user=0.25 sys=0.00, real=0.26 secs] 
DE
Heap
 PSYoungGen total 2752512K, used 250729K [0x0000000740000000, 0x0000000800000000, 0x0000000800000000)
 eden space 2359296K, 10% used [0x0000000740000000,0x000000074f4da6f8,0x00000007d0000000)
 from space 393216K, 0% used [0x00000007d0000000,0x00000007d0000000,0x00000007e8000000)
 to space 393216K, 0% used [0x00000007e8000000,0x00000007e8000000,0x0000000800000000)
 PSOldGen total 5376K, used 18K [0x0000000680000000, 0x0000000680540000, 0x0000000740000000)
 object space 5376K, 0% used [0x0000000680000000,0x0000000680004b30,0x0000000680540000)
 PSPermGen total 21248K, used 2725K [0x000000067ae00000, 0x000000067c2c0000, 0x0000000680000000)
 object space 21248K, 12% used [0x000000067ae00000,0x000000067b0a95d0,0x000000067c2c0000)
```

</div>We can see that the difference is significant. It took 3.3 seconds longer to run with interning. But the memory saved was enormous. When the code finished, the program using interning used 253472K(250M) of Memory. The other one used 2397635K (2.4G). That is quite a difference and illustrates nicely the tradeoffs when using String interning.

Save Memory by Using String Intern in Java

I was at a customer recently to further improve their application monitoring for which they were using AppDynammics. When I arrived, they told me: “Good that you came Fabian, we have something interesting to show”. Usually operations guys are very concerned about something when they say this to me, but in this case, they were happy. “We had a big outage when we took our new software live on Saturday!”. Ooops?? Nobody likes production outage on Saturdays! Why are they so happy about it? “Well it was a long Saturday, but thanks to our monitoring we knew what was going on”.So let me walk you through what they did, and show you the problem that killed their server.<h2>An Overview of the Situation</h2>Here is what the situation looked like on Saturday 8 in the morning. <img src="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/CaJu5zmQVimPDE76lgXW" alt title="thread-pool-exhausted" width="700" height="475" /> If you don’t know AppDynamics, here a short introduction:<ul><li>The biggest area is the application map. AppDynamics autodiscovers and monitors all systems and their interaction.</li><li>On the right side there are some statistics. We can see some Stalls and Abnormal Slow requests. AppDynamics found out that something is not right.</li><li>On the bottom is the historical view. Because this is historical data now, we only see hourly data. During the incident, the data was more fine grained.</li></ul>The historical data is of most interest for us, looking back. We can see the server start on the left hand side (blue icons on top of green bar). The load on the system climbed rapidly (green bar). But then the system broke down. The response time increased massively (blue bar). And requests decreased. Soon the server crashed and had to be restarted. But the restart did not help, the system did not recover. Luckily it did later after the problem was fixed 🙂<h2>What went wrong</h2>The system crashed with an OutOfMemory Error. In our <a href="/en/knowledge-hub/blog/java-outofmemoryerror-a-tragedy-in-seven-acts">OutOfMemoryError Series</a> , we already discussed some of them, and here is another: “Unable to create native thread”.This error is actually an interesting fact. Outages with this kind of Error are usually not created by inefficient code, but either by infrastructural problems or buggy threading code. So lets look at the number of threads on each server recorded by AppDynamics:<img loading="lazy" src="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/clka5Yq2QvC9LtKiz0hl" alt title="thread-pool-exhausted-2" width="700" height="433" />Ouch. When it crashed, the system had already created 1800 threads. What was producing these threads? I have seen this before at customers, so I immediately knew this. And also my client was able to find out by seeing a few snapshots of stalled calls. JSP Compilation was the culprit. All application servers have an option to precompile JSPs on startup, or when requested for the first time. But both settings are problematic when the server is started under high load. The problem was resolved on Sunday Morning 4AM. But the ops team had not to be awake those 20 hours. They had reported the qualified issue to Oracle around noon on Saturday. The Oracle provided “fix” was simple, but not documented anywhere. Or at least not where you can find it if you do not know it. Here is the screenshot from AppDynamics discovering the property change and restart at 4AM. <img loading="lazy" src="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/VWpuxtzCRzWqaXqQ8PHd" alt title="thread-pool-exhausted-3" width="601" height="419" /><h2>Limiting JSP Compiler Threads on Weblogic</h2>The “secret” switch was an environment variable you have to set:

```

BEA_COMPILER_NUM_THREADS = 1

```

While 1 is for sure very pessimistic, it seems to be a way better setting than unlimited, which is the default. No one should ever create an unlimited amount of threads. If you think in a classical view, you should only have as much threads as you have number of cores. modern JVMs can easily handle like ten times that much threads, without much starvation. But in this case we had about 100 Compiler Threads per CPU.

Analyzing Production Outage Caused by Weblogic Compiler Threads

In Act 5 of our OutOfMemoryError series, I talked about the lack of tool support for finding memory leaks at runtime. I got some negative feedback on dzone, because the tool I discussed was commercial 🙁 Shortly after I published that post, I got notified of a product called “plumbr”. Plumbr is in beta phase, which means it is free at the moment, and they even have a public pricing discussion. But still we need to accept, that there is no free solution. Its not my fault, so don’t be angry 🙂 In this short update, I will show you how plumber will find the very same memory leak, which AppDynamics found in my last article. <h3>Installation</h3>Plumbr is as easy to install as AppDynamics. Your downloaded package contains files for various platforms, so this is an indication, that plumbr is using a JVMTI native agent in addition to the javaagent. After unzipping, I needed to add this to my Tomcat 7, which will be running the memory leak app:<div>

```bash
SET PATH=C:\Tools\plumbr\plumbr\win\64;%PATH%
set CATALINA_OPTS=-agentlib:plumbr -javaagent:C:\Tools\plumbr\plumbr\plumbr.jar
```

</div><h3>Finding Memory Leaks Fast</h3>Plumbr is working very fast. I deployed my leaking application and started the load script. After only a few minutes this messaged appeared: <img src="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/N2RcZxrQIqQh9XjVOqfS" alt title="plumbr-found-leak" width="677" height="234" />So the only thing I needed to do was to open the generated report: <img loading="lazy" src="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/vxIS0wvYQJEy95HsdYCS" alt title="plumbr-found-leak-2" width="700" height="748" />Tadaa! The leak was correctly identified. And it also lists where the leak comes from, which is very important:

```

The objects are created at
 de.codecentric.memleak.leak.StringQueueLeak.leak(de.codecentric.memleak.domain.Bookmark):58

```

When compared to AppDynamics, this is slightly less information, because the call stack is shorter. And it also does not give invocation counts. However this is still great information, which helps us to identify and fix the leak. And it is presented in a dead simple way.<h3>Further improvements</h3>You might wonder why there are 3 leaks identified, while AppDynamics did show only one of them. The second and third are classes used by the Derby database: <img loading="lazy" src="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/78OgUJmQWyiZtUuOTdxA" alt title="plumbr-finalizer" width="700" height="460" />So plumbr currently assumes that a reference from java.lang.ref.Finalizer is a memory leak. There is some controversy (for example on <a href="http://stackoverflow.com/questions/8355064/is-memory-leak-why-java-lang-ref-finalizer-eat-so-much-memory">stackoverflow</a> ) about this being possible leaks. So I mailed this observation to the plumber developers and got a quick answer: While they think Finalizers are problematic, they want to change the handling, so there will be less likely false positive reportings.I think plumbr is a really interesting solution. It solves the very specific problem of finding memory leaks in an elegant way. It reports fast and did not add significant overhead to my application, but I still have to trial it in a real application server environment to make an educated statement.

Find Memory Leaks at Runtime – Addendum

On November 8th and 9th, the first Velocity Europe conference took place in Berlin. About 500 people from the Web Performance and Operations Community came together to share their experience about creating and operating web sites that are really fast. Because “being fast” drives the codecentric performance solutions group, we of course attended as well. Our partner <a href="http://www.appdynamics.com/free">AppDynamics</a> was so kind to co-host us in their booth.There were plenty of interesting discussion and sessions. While I found most of the Web Performance track sessions to be a browser vendor competition, the Operations and Culture track clearly provided a lot of useful insights. The only session I did not understand was the rant-fest by Arthur Bergman. But I could read from the reactions of Steve Souders, who was hosting the conference, that he did not understand it as well 🙂I head the pleasure to meet Tobias Baldauf, who is bootstrapping a web performancce optimization user group in cologne. Of course I will attend the very first meeting taking place on Friday, 2nd of December. One day before the global <a href="https://github.com/coreyhaines/coderetreat/wiki/Cologne">CodeRetreatDay, Kurt Häusler organizes in Cologne</a> . codecentric is sponsoring that day, so we are looking foward to meet you on either of those dates.We are happy to see the importance ob web performance grow and are proud to be a part of such a great community.

Web Performance Conference Velocity Europe in Berlin

The <a href="http://markmail.org/message/sxjtefpayanbqfe5">voting</a> is over: <a href="http://harmony.apache.org/">Apache Harmony</a> , an open Java implementation, will be discontinued.After IBM stopped supporting Harmony and moving over to <a href="http://openjdk.java.net/">Open JDK</a> , interest in Harmony vanished. Additionally there have been political argues about the Technology Compatibility Kit (TCK) and also about Googles Dalvik VM, which is based on Harmony. Seems like Harmony never had a real chance.While it is sad that Harmony is now dead, it is good to see consolidated work being done on the Open JDK.

Apache Harmony is Dead – Long live Open JDK

When we began thinking about a relaunch of our homepage 3 months ago, we already knew a few key facts about what we wanted:<ul><li>use high quality cloud infrastructure</li><li>run with high standard performance</li><li>use a flexible standard software stack</li><li>provide great ease of use for editors</li></ul>While we had used <a href="http://www.opencms.org/en/">OpenCMS</a> for the homepage in the past, we decided to go for <a href="http://wordpress.org/">WordPress</a> for our new homepage. We have been using WordPress for our blog for years and it provides the ease of use we want. It is dead simple, yet powerful. Even with its default configuration, it offers sufficient functionality, and it can be easily extended with custom or off-the-shelf plugins.But this post is not about our decision to go with WordPress, but how we achieved to run a medium traffic homepage and blog for free, without compromising speed.Did I say free? – Yes!Amazon offers new customers a free micro tier for a year: <a href="http://aws.amazon.com/free/">aws.amazon.com/free</a> . new customers are just new accounts. But, of course, it is quite restricted: You get a virtual CPU, 613MB memory and have to use Amazon Linux.After I started the micro instance with its 8GB EBS, I installed the basic services required for WordPress:<div>

```bash
sudo yum install httpd mysql mysqld php php-mysql
```

</div>After installing and configuring WordPress everything looked fine, but performance degraded fast, when multiple users were accessing the site. The first action I took was to install APC, a bytecode cache and optimizer. You should always use APC when deploying a PHP site:<div>

```bash
sudo yum install php-pecl-apc
```

</div>The situation improved quite a bit. But I would need to improve it even more to make the site reliably serve multiple users at the same time.When I worked with the free micro instance I learned two things<ul><li>“Steal Time”! Never seen this before:<div>

```bash
Cpu(s): 2.0%us, 0.3%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.0%hi, 0.0%si, 97.7%st
```

</div>That means, the system wants to use more CPU time, but does not get it from the hypervisor.</li><li>EBS is slow! But others noticed that, too.</li></ul>The process suffering most from those findings was mysqld, so I considered how to improve that.I applied the <a href="http://www.codinghorror.com/blog/files/matt-mullenweg-wordpress-mysql-recommendations.txt">WordPress mysql tuning recommendations</a> by Matt Mullenweg, but this alone did not show any significant impact. It was just not possible to get good performance out of mysqld due to CPU and IO limitations. I couldn’t bypass IO completely by caching everything due to memory constraints.So I looked for a good plugin for WordPress to avoid database access through clever caching of some stuff. And indeed there are a few plugins for that purpose:<img title="wp-cache" src="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/kD9CePqpTn20Yci23XEO" alt width="582" height="309" />I ended up installing Quick Cache (which no longer exists, an alternative would be Total Cache), and was pleased to see server load drop significantly.It did not work for many people editing, as the caching has no effect. Amazon does limit the CPU to short bursts. many people editing can sometimes result in 30 seconds steal time due to that. To fix this, I limited the mysqld cpu usage:<div>

```bash
wget http://centos.alt.ru/repository/centos/5/i386/cpulimit-1.2-1.el5.i386.rpm
rpm -i cpulimit-1.2-1.el5.i386.rpm
nohup sudo cpulimit -v --exe mysqld --limit 50 &
```

</div>Now that I had optimized most CPU and IO issues with clever caching, next up was tuning the httpd.conf to make sure the server handles the right number of users at any time. Because we use mod_prefork, we had to look carefully at the amount of memory each Apache process was using. To bring down memory consumption of httpd, I removed most loaded modules, but still every Apache was eating up between 25 and 35MB of memory. mysqld was taking around 80MB, so I decided to go for about 15 worker instances, which would utilize almost all available memory.<div>

```bash
StartServers 15
MinSpareServers 15
MaxSpareServers 15
ServerLimit 15
MaxClients 15
MaxRequestsPerChild 500
```

</div>MaxServersPerChild is important to recycle instances in case of leaking memory.15 concurrent connections is quite a low limit, so as a next step I had to make sure server communication was reduced to a minimum.<h2>Web Performance Optimization</h2>With the server having a solid setup, the low hanging fruit are now on the browser side. The default output of a WordPress theme with multiple plugins installed is very verbose. Because you do not want to modify the theme a lot and have little chance to tweak code generated by plugins, we needed a better way to optimize html.This can be easily done by the fabulous mod_pagespeed httpd module from Google.<div>

```bash
sudo rpm -i mod-pagespeed-*.rpm
sudo service httpd restart
```

</div>The impact was again very significant. The aggressive cache settings cut down most of the sever requests, removing strain from the limited amount of httpd processes. Also, the reduced image sizes were very noticeable. Especially in a content management system, you have limited control over the images used by editors and authors. mod_pagespeed automatically reencodes them and stores them on disk. But mod_pagespeed does not create the cache out of thin air. It checks the image every time and just skips the reencoding if the cached version is valid:<div>

```bash
46.137.99.225 - - [07/Aug/2011:03:54:37 +0200] "GET /wp-content/uploads/2011/07/wanted-memory.png HTTP/1.1" 200 116616 "-" "Serf/0.7.2 mod_pagespeed/0.9.18.6-886"
```

</div>this is where mod_expires comes into play:<div>

```bash
ExpiresActive On
ExpiresDefault "access plus 10 days"
```

</div>What time you choose depends on the type of images. If you can guarantee that the image will never change go for years. A week is generally a good point to start. As this setting is only affecting mod_pagespeed on the server, you can even go a bit more aggressive and clean the cache after updates.To further clean up the html beyond what mod_pagespeed can do, I removed most of blog stuff by adding this to functions.php:<div>

```php
remove_action('wp_head', 'rsd_link');
remove_action('wp_head', 'wlwmanifest_link');
remove_action('wp_head', 'feed_links_extra', 3);
remove_action('wp_head', 'feed_links', 2);
remove_action('wp_head', 'remote_login_js_loader');
function remove_generator() { return ''; }
add_filter('the_generator', 'remove_generator');
```

</div>WordPress ships with a jquery javascript library, but to benefit from caching the most and saving more server traffic, we should use the jquery hosted on the Google Content Delivery Network. Additionally we stop loading the l10n.js which we do not need:<div>

```php
function init_scripts() {
wp_deregister_script( 'jquery' );
wp_register_script( 'jquery', 'http://ajax.googleapis.com/ajax/libs/jquery/1.6.2/jquery.min.js');
wp_deregister_script('l10n');
}
 
if (!is_admin()) {
add_action('init', 'init_scripts');
}
```

</div>Optimization is done mainly for unauthenticated users. To prevent issues for administrators in the complex WordPress admin interface, most changes are not done for logged in users.As a result of this, www.codecentric.de now has a pagespeed rating of 95/100A single user should experience page loads in less than a second, for most pages in less than 300ms.So how did it work out at scale? I decided to run a benchmark against the old dedicated machine and the new EC2 micro. 4 real cpus vs 1 virtual one, 2GB of memory vs 613MB. The comparison is not completely fair, as I did not tune and tweak the old blog to the extreme. I ran the benchmark using <a href="http://httpd.apache.org/docs/2.0/programs/ab.html">apachebench</a> , with 5 concurrent users and 50 requests hitting the homepage.old:<div>

```bash
Requests per second: 3.07 [#/sec] (mean)
Connection Times (ms)
 min mean[+/-sd] median max
Connect: 9 71 424.8 9 3014
Processing: 989 1513 461.3 1352 3420
Waiting: 597 865 344.2 768 2861
Total: 1004 1584 603.1 1373 4287
```

</div>new:<div>

```bash
Requests per second: 18.04 [#/sec] (mean)
Connection Times (ms)
 min mean[+/-sd] median max
Connect: 38 44 4.4 44 52
Processing: 183 224 47.3 220 493
Waiting: 58 89 41.4 80 335
Total: 222 268 49.5 265 544
```

</div>even with the connect taking longer due to geographical distance to the Amazon Dublin datacenter, the results are amazing.Because I know I can handle 15 concurrent requests, I tried running 45 requests in parallel, each doing 20 loads.<div>

```bash
Requests per second: 50.00 [#/sec] (mean)
Connection Times (ms)
 min mean[+/-sd] median max
Connect: 39 42 4.8 43 78
Processing: 300 839 116.7 869 1181
Waiting: 179 711 115.2 742 1049
Total: 340 881 116.8 912 1225
```

</div>Even with higher load, response times are still below a second. Cool!<h3>Is micro enough?</h3>I am quite confident that micro serves us well for a long time. But there was an issue, which lead me to this tweet: <a href="http://twitter.com/#!/CodingFabian/status/100558375905001472"><img loading="lazy" title src="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/rsJ2nYqvRdisO7zAolZM" alt width="499" height="220" /></a> The moment we pushed the DNS update was scary. The site was slowing to a crawl, but why? I had checked everything! But we unknowingly caused a distributed denial of service (DDoS) attack. I checked the access_log and saw what was causing the trouble: All the search engines I knew were crawling our new site and blog, and additionally tons of bots I did not even know to that date. To make it worse, we posted the relaunch on internal chat and mail, causing over a hundred codecentric employees check out the new site. On top of that, all caches were cold.<h3>Cloud to the rescue!</h3>Because this is Amazon, I could resolve the situation easily. I stopped the instance, changed the type to m1.small instead of t1.micro and hit start. After the machine came back and reassigned the virtual IP itself, I reconfigured apache to run up to 50 processes. The whole “upscaling” process took less than 30 seconds, and was not noticeable for users who already were complaining about slow response. The machine was still heavily loaded, but response times were at an acceptable 1-2 second level. 4 hours later it was all over. Crawling bots and codecentrics were gone, the site was calm. Another 30 seconds later, the machine was back at t1.micro, and free. Spendings for this exercise: 16 centsI am quite sure that there will be never again such a load on the system as during launch. And in case there will be, we can scale up in less than a minute. For example on Fridays, when many codecentric employees blog or publish content on the homepage, we just scale up.I hope you enjoyed reading our experience. If you consider switching to the cloud, or want to talk to us about improving performance in the web or on your sever, feel free to contact us.

Creating a High Performance Cloud Hosted Homepage

In a <a href="http://www.slideshare.net/fabianlange/web-performance-optimization-web-tech-conference-2011-talk">recent talk</a> about Web Performance Optimization during the Web Tech Conference I recommended mod_pagespeed as a cool automatic tool to reduce requests to CSS and JavaScripts and, by that, improve website load performance. But I noticed while watching stats for www.codecentric.de that mod_pagespeed does not work correctly.It should combine a few CSS files, but I found them separate in HTML:<div>

```html4strict
<link rel='stylesheet' id='login-css' href='wp-admin/css/login.css,qver=20110610.pagespeed.cf.ZqFxSZoD1J.css' type='text/css' media='all'/>
<link rel='stylesheet' id='colors-fresh-css' href='wp-admin/css/colors-fresh.css,qver=20110703.pagespeed.cf.ldpZyBlEpG.css' type='text/css' media='all'/>
```

</div>Ok, this is at least a minified version of the CSS files. Without pagespeed it would have looked like this:<div>

```html4strict
<link rel='stylesheet' id='login-css' href='wp-admin/css/login.css?ver=20110610' type='text/css' media='all'/>
<link rel='stylesheet' id='colors-fresh-css' href='wp-admin/css/colors-fresh.css?ver=20110703' type='text/css' media='all'/>
```

</div>Those two lines should have been combined by the “combine_css” filter. But indeed the documentation has a few notes on when this does not happen:<blockquote>The CSS Combine filter operates within the scope of a “flush window”. Specifically, large, or dynamically generated HTML files may be “flushed” by the resource generator before they are complete. When the CSS combiner encounters a flush, it will emit all CSS combinations seen up to the point of the flush. After the flush, it will begin collecting a new CSS combination.</blockquote>and<blockquote>IE Directives containing CSS links form a “barrier” for the CSS combiner. Multiple CSS elements found before an IE directive are combined together immediately before the IE directive. Multiple CSS elements found after are also combined, but the combination does not span across the IE directive, as that would affect the order that the browser sees the CSS elements.</blockquote>However, here we do not have IE directives, nor are both lines in different “flush-window”s. So why doesn’t it work? There is another, not documented, constraint:<blockquote>Tags can only be combined when no information will get lost.</blockquote>In this simple example it is easy to see what this lost information would be: The identifier of the link tags, “login-css” and “colors-fresh-css”, would no longer exist. It would be theoretically possible to use these identifiers from JavaScript code to retrieve and modify the css.But where does the ID come from? Do we actually need them, or can we get rid of them?WordPress uses a function called “<a href="http://codex.wordpress.org/Function_Reference/wp_enqueue_style">wp_enqueue_style</a> ” which can be used by plugins or themes to add styles dynamically to the head-section. wp_enqueue_style takes a mandatory first parameter called hande. And this handle is what WordPress will use to generate the ID. Usually you do not need the ID. There is actually no code in standard WordPress which would use it, and one would rarely ever write code using is on stylesheet links yourself. WordPress is just adding the id for any case. To make it possible for mod_pagespeed to combine the CSS files, we need to remove the ID. This is easily possible using a built-in WordPress filter. The “style_loader_tag” filter is called on the final link, and we can hook into it to remove the id.<div>

```php
function remove_style_id($link) {
 return preg_replace("/id='.*-css'/", "", $link);
}
add_filter('style_loader_tag', 'remove_style_id');
```

</div>This would result in the following HTML:<div>

```html4strict
<link rel='stylesheet' href='wp-admin/css/login.css?ver=20110610' type='text/css' media='all'/>
<link rel='stylesheet' href='wp-admin/css/colors-fresh.css?ver=20110703' type='text/css' media='all'/>
```

</div>And indeed the tags are combined as expected:<div>

```html4strict
<link rel="stylesheet" type="text/css" href="wp-admin/css/login.css,,qver==20110610+colors-fresh.css,,qver==20110703,Mcc.35EZfaJvqU.css.pagespeed.cf.I5odih6TFY.css" media="all"/>
```

</div>I should also mention that this behaviour is not restricted to the ID. It is valid for other attributes as well, especially “media”. If you serve different styles for different medias, they of course cannot be combined. I would recommend to check that you use the medias correctly, or, if you do not need the distinction between medias, to remove the media attribute as well.

WordPress and Mod_Pagespeed – why combine_css does not work

Deadlocks are a nasty thing for programmers. They usually occur never in development environments, but only in production. Also, because it is affecting production, operation teams resolve those issues quickly by restarting the server, destroying all evidence for developers.So developers almost never have a chance to see a deadlock in action.Wouldn’t it cool to have such information on hand? Something looking like this: <img title="deadlocks" src="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/vU9SCMlxSLyhtRhnqNZp" alt width="611" height="164" /> But good news! I did not make this picture up. It is part of <a href="http://www.appdynamics.com/">AppDynamics</a> , which will automatically detect deadlocks in production environments and provide all the information needed to fix them.For the following quick demo screencast I used the “Dining Philosophers” problem, described in an alternative variant by Dr. Heinz Kabutz in his <a href="http://www.javaspecialists.eu/archive/Issue194.html">recent newsletter</a> . Heinz told me of this recently, when he taught his Java Specialists Master Course at codecentric in Düsseldorf. I had planned to do a demo of the deadlock detection feature, but did not have time to write a nice deadlock yet. Thanks Heinz 🙂<div> </div>When AppDynamics has an agent in a live JVM, it will monitor thread activity and check for deadlocks automatically. This works without any configuration and with the usual very low overhead AppDynamics takes.Because it preserves the information for developers, it is safe to restart the machine. The information what caused the deadlock will be still available for debugging. This has proven quite useful during our Java Performance troubleshooting gigs.<img loading="lazy" title="monitors" src="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/prlXYJnkQhispckX6WIf" alt width="633" height="473" />I think the deadlock detection is a very useful feature, which is not very obvious in AppDynamics, because it has no button to activate. Of course that’s good, because it works fully automatic.

Automatic Java Deadlock Detection

During the second act of this <a href="/en/knowledge-hub/blog/java-outofmemoryerror-a-tragedy-in-seven-acts">series</a> we explained <a href="/en/knowledge-hub/blog/java-memory-leaks-et-al">Java Memory Leaks</a> . We elaborated how Memory Leaks can occur in Java and how they differ from traditional Memory Leaks. Todays post is going to explain how to have a closer look at all the objects residing in the Java heap. This includes references between the objects, which are required to identify memory leaks. The most accessible tool to approach analyzing memory issues are so called Java heapdumps, which can be created automatically in the event of an OutOfMemoryError by passing the JVM command line parameter <code>-XX:+HeapDumpOnOutOfMemoryError</code>. In this case the JVM will write the dump right before it exits. Analyzing these kind of dumps is also called “Post mortem”, as the JVM is already dead. But of course one can also create dumps manually at any point during runtime. More on that later. While we take on heapdumps in this post, we will see other alternatives for hunting lost memory during the next act.Java Heapdumps A heapdump is a textual or binary representation of the Java heap which is usually written to a file. Various tools can reconstruct the objects and their references from the information stored in the heap dump file. The only information missing is the heap area the object was residing in. This is quite sad, as we just learned from <a href="/en/knowledge-hub/blog/java-memory-configuration-and-monitoring-3rd-act">act 3</a> the importance and meaning of the heap areas. However, this information is not required to locate Java Memory Leaks.Wanted: Dead or alive <img title="Wanted Memory" src="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/MjyaXvknSA26RtvH7HOU" alt width="199" height="267" />Heapdumps can contain to kinds of objects:<ul><li>Alive objects only (those are objects, which can be reached via a GC root reference)</li><li>All objects (including the no longer referenced, but not yet garbage collected objects)</li></ul>Because the live objects can be easily determined from various VM internal mechanisms, a heapdump containing only live objects can be created acceptable fast. Creating a full heap dump with dead objects as well takes much longer and also consumes more disk space. However, the live objects are sufficient to search for Memory Leaks. Unreferenced objects are only of interest in respect to Object Cycling or GC Thrashing. In those cases one wants to find unnecessary or excessive object allocation.Creating Heapdumps As said initially, heap dumps can be created in 2 ways. But the creation of a dump will halt the JVM and all running processes and applications. This is required to be able to dump a consistent state of the memory. Due to this halt, it is not recommended to use this functionality on a production system very often, if at all.<ol><li>When using the JVM parameter <code>-XX:+HeapDumpOnOutOfMemoryError</code> the JVM will create a heap dump whenever it encounters an OutOfMemoryError before it quits. There is no reason not to use this parameter. The JVM has died and any negative impact on performance is irrelevant. But the size of the dumps can be an issue for certain servers. As the dump is very often the only point of information after a crash it should be created and preserved by the server administrator to allow a “post-mortem” analysis. Unfortunately they are very often simple deleted.</li><li>Using the JVM tool jmap, heapdumps can be created from the command line:

```
jmap -dump:live,format=b,file=<filename> <PID>
```

The option live is important, as it will restrict the dump to live objects only. If you want to have also dead objects, simply omit this option. Format “b” (for “binary”) is recommended. While there is an ASCII version as well, manually analyzing it is almost impossible due to the amount of data.</li></ol>Heap – inside out There are a few tools for analyzing heapdumps offline. But detailing a comparison here is not worth the effort, as there is a clear winner: <a href="http://www.eclipse.org/mat/">Eclipse Memory Analyzer Toolkit (Eclipse MAT)</a> Eclipse MAT can read, parse and analyze dumps of various formats and sizes. It has an easy to use, but sometimes a bit too powerful, user interface and is completely free to use. While older tools required to load the whole dump into memory, requiring very often 1.5 times the size of the dump as free memory, MAT will index the dump initially which allows fast analysis using just very few system resources. This enables users to work even with very large heapdumps.<img loading="lazy" title="Common Heap Objects" src="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/MMs7JK7TdqcujCN7CAYw" alt width="332" height="480" />After opening a dump with Eclipse MAT following objects are usually listed very often:<ul><li>char[]</li><li>java.lang.Object[]</li><li>java.lang.Class</li><li>java.lang.String</li><li>byte[]</li><li>java.util.HashMap</li><li>java.util.HashMap$Entry</li><li>short[]</li><li>int[]</li><li>java.util.ArrayList</li><li>java.lang.Integer</li></ul>Those objects are in every dump in large amounts. But are those objects the issue? How can I find the real root cause for a memory leak? Extensive Leaks When looking at an OutOfMemoryError induced heapdump post mortem, one can usually assume that the Memory Leak filled the majority of the memory. But how big is the leak? While the dump contains all object instances, Eclipse MAT shows in its Class Histogramm only class names, amount of instances and the sum of their shallow size. Shallow Size is basically the sum of the sizes of all contained primitive data types, as well as the size of the references to other objects. But the so called Retained Heap is more important for finding leaks, as it is the total size of all objects being kept alive by this dominator. To get this number, MAT needs to calculate the Retained Size by following all object references. As sorting by Retained Heap is very helpful for finding leaks, MAT displays the largest retained heaps in a pie chart. A prominent example for a lot of Retained Heap in web applications is usually the

```
org.apache.catalina.session.StandardManager
```

The purpose is clear: keep user data for the time of their session. But very often frameworks store a lot of data here as well. We already blogged about <a href="/en/knowledge-hub/blog/richfaces-sessions-eating-memory-analysis-of-a-memory-leak">Ajax4JSF consuming a lot memory</a> in the session. But also custom written, and unfortunately often incorrect, caches appear very often in the pie charts. This makes it very easy to locate memory leaks and their cause quickly.<img loading="lazy" title="Large Objects in Heap" src="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/PDx6Lg96SQGv9P6LolYX" alt width="578" height="378" /> To determine who is creating these objects, or find out what the purpose of some structures is, the actual instances with their incoming and outgoing references are required. To get them, choose List Objects with incoming References from the context menu. Now a tree structure is displayed, showing all instances with all incoming references. Those references keep the object alive and prevented them from being garbage collected. Outgoing References are interesting as well, because they show the actual contents of the instances, helping to find out their purpose. They are especially interesting when the actual dominator, like the SessionManagers, should not be removed, but the excessive objects referenced by it should.<img loading="lazy" title="instance-details" src="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/OfWUbQUfSrqQnyEixRhx" alt width="526" height="405" /> Sneaking Memory Leaks Finding Memory Leaks long before an OutOfMemoryError occurs is possible by creating multiple snapshots with the previously described method. A single snapshot alone can seldom help deciding whether the amount of objects is ok or not. But slowly growing object counts can hint at leaks. Eclipse MAT allows comparing two heapdump snapshots and finding growing structures and instance counts. But this requires two well timed dumps. With some amount of load they should be minutes apart, with few load hours or days. Having a greater time lag helps in separating normal jitter from real issues. The following example nicely depicts the obvious growing amount of XML related classes:<img loading="lazy" title="Heap Delta" src="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/JTP3C6y8S9uaaG653N5b" alt width="491" height="480" /> Something better But comparing snapshots is cumbersome and in some cases even misleading. And we actually never want an OutOfMemoryError to happen in production. Even worse, heapdumps do not tell us about the lines of code actually creating objects and not removing them. There has to be a better tool than heapdumps! And there are: Profiler and some APM tools allow recoding statistics and access pathes during runtime. The information provided by those tools is often more significant and easier to act on than heapdumps. For those reasons, we are going to have a closer look at those tools during our next installment of our OutOfMemoryError tragedy.

Create and Understand Java Heapdumps (Act 4)

I usually encourage to upgrade early. For me it keeps the overall upgrade effort lower than big bang upgrades. It also brings early access to performance improvements. But of course you always need to measure! And this is what I did for a customer, planning to upgrade from Drools 5.0.1 to <a href="http://www.jboss.org/drools/">Drools 5.2</a> . To my surprise, I noticed that loading and executing rulebases is much slower in 5.2. That customer has a cache, which contains a StatelessSession per rules file. The rules are loaded on startup to provide a smooth experience on the first call. But loading all the files was already causing concerns at startup time. So I hoped for improvements. But initial unit test execution showed degraded performance. Tests took 3 times longer to execute. While the measurements were done quite simple by checking junit execution time, there is no need to discuss precision. So I attached <a href="http://www.ej-technologies.com/products/jprofiler/overview.html">JProfiler</a> to a unit test which loads and executes 21 <a href="http://docs.jboss.org/drools/release/5.2.0.Final/drools-expert-docs/html/ch06.html#d0e6651">Drools Decisiontables based on XLS</a> . The results were quite interesting: [singlepic id=348 w=320 h=240 float=center] It is obvious that we have a cause for our performance problem. ClassLoading seems to be the bottleneck. While a single call is quite fast, it is executed a lot. But there is another interesting fact on these hot spots. Can you spot it?Yes: System.gc() is invoked 42 times (I like that number). JProfiler says it is accountable for 8% of the total time. But we know Garbage Collection can have side effects. Why do we have so much Garbage Collection at all? In fact this is not the total amount of Garbage Collection activity, but the number of explicit calls. I have seen it many times already. Invoking <a href="/en/knowledge-hub/blog/invoking-system-gc-can-have-serious-impact">System.gc() is bad</a> (most of the times).[singlepic id=350 w=320 h=240 float=center]So why is it invoked? Which is the code guilty of invoking Sytem.gc()? JProfiler can tell us easily:[singlepic id=352 w=320 h=240 float=center]I have seen this before as well and for me this is a flaw in the frequently used <a href="http://jexcelapi.sourceforge.net/">Java Excel API</a> . It does invoke <a href="http://download.oracle.com/javase/6/docs/api/java/lang/System.html#gc%28%29">System.gc()</a> after each time it has read bytes from disk. I really think you shouldn’t do that. And this case again proves why. But at least there is a flag in jxl and it is explained in WorkbookSettings:

```
/**
 * Flag to indicate whether the system hint garbage collection
 * is enabled or not.
 * As a rule of thumb, it is desirable to enable garbage collection
 * when reading large spreadsheets from a batch process or from the
 * command line, but better to deactivate the feature when reading
 * large spreadsheets within a WAS, as the calls to System.gc() not
 * only garbage collect the junk in JExcelApi, but also in the
 * webservers JVM and can cause significant slowdown
 * GC deactivated using -Djxl.nogc=true on the JVM command line
 * Activated by default or by using -Djxl.nogc=false on the JVM
 * command line
 */
```

Ok, lets add this sytem property. Lets have a look if that did the trick?[singlepic id=349 w=320 h=240 float=center]and here the GC activity after activating the nogc flag[singlepic id=351 w=320 h=240 float=center]While it did not fix the obvious problem with classloading, it did prove to be a lowhanging fruit. Reducing GC activity and total runtime of this test from 338 seconds to 315 (both in profiled mode). While it is not impressive on a unit test, which by nature has a different JVM setting than a server, it also worked flawlessly in the integration test system. Cache init time was reduced from about 750 seconds to 240 seconds, just by activating this switch.While the system property made things better for almost no effort, Ill have a look into the classloading madness by drools in a second post.

Java Performance: Drools 5.2 with Decisiontables – Part I

In the previous act, we learned about <a href="/en/knowledge-hub/blog/java-memory-leaks-et-al">java memory leaks and how memory is consumed</a> . From the first act we learned about the <a href="/en/knowledge-hub/blog/the-java-memory-architecture-1-act">separation of the java heap into multiple areas</a> . Equipped with that knowledge, we can now have a look at the memory consumption during runtime and learn how to configure the JVM. While there are multiple memory regions managed by the JVM for different purposes, the one that is of most interest is the heap. In this post, I will describe the key elements of the heap and show how to have a look at it during runtime, using various JVM commands and especially the VisualVM.<h2>Heap size: Minimum and Maximum</h2>The memory management functionality is actually quite clever. It tries to avoid allocating too much memory from the operating system, but on the other hand it also tries to ensure that enough memory is available to the application. To illustrate the behavior, I created a small Java program which wastes memory:<div>

```java5
ArrayList list = new ArrayList();
Thread.sleep(5000); // 5s to allow starting visualvm
for (long l = 0; l < Long.MAX_VALUE; l++) {
	list.add(new Long(l));
	for (int i = 0; i < 5000; i++) {
		// busy wait, 'cause 1ms sleep is too long
		if (i == 5000) break;
	}
}
```

</div>Eventually this leads to: <code>Exception in thread "main" java.lang.OutOfMemoryError: Java heap space</code>Below are two graphs of the heap consumption monitored by VisualVM. The yellow area shows the available memory while the blue area shows the currently used memory. The graph sometimes decreases, because garbage collection frees some temporary objects used to increase the ArrayList size. The graph on the left hand side results from the application using the default settings of the JVM. You can clearly see that initially the JVM uses very few system resources and then starts to request more memory from the operating system as it is needed, finally ending at 128 megabytes. In the graph on the right hand side we see that the yellow area already covers 128MB from the start. This is because I added a JVM flag to change the default behavior. I will show you the flags in a second.<div><img title="Adaptive Heap Size" src="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/JdibrHKUTuKngi63ra8j" alt width="225" height="236" />Adaptive Heap Size</div><div><img loading="lazy" title="Fixed Heap Size" src="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/IBxzU3tJSDSjrI8eQXb3" alt width="226" height="230" />Fixed Heap Size</div> One interesting thing to notice is: We get the OutOfMemoryError before the memory is fully depleted. You might wonder why. We will find out later, but start thinking about it. Now, let us return to the flags I used. There are two flags to control the overall size of the JVM heap: <ul><li><code>-Xmx128m</code> will set the maximum heap size to 128 megabytes</li><li><code>-Xms128m</code> will set the initial/minimum heap size to 128 megabytes</li></ul>While we may have an obvious reason for setting the maximum size to a value higher than the default (which varies based on the platform), there are two reasons for setting the minimum as well:<ol><li>Most Java applications are server type of applications, which run exclusively on a machine and are intended to use all the memory available. This reduces the minor overhead a dynamic resizing of the heap will impose.</li><li>More important: Turning dynamic heap sizing off makes it much easier to understand and influence the internals of the heap, which I will talk about next. It also makes tuning effects more consistent over time and easier to perform.</li></ol><h2>Heap areas: Tenured, Young, and Permanent</h2>The heap itself is separated into three areas: Tenured (also called Old), Young (also called New), and Permanent (Perm). As already outlined in our first act, the idea behind that is to make garbage collection more efficient. Permanent should never require collection (although you can configure it), Old should be rarely collected, and Young will be collected a lot. <img loading="lazy" title="Survivor Age Histogram" src="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/Z0HfpYgDT1uEPnC4aBja" alt width="299" height="197" />The Young area usually shows a lot of activity, so most garbage collection algorithms are designed to optimize collections in that area. The graph on the right hand side (taken from <a href="http://www.oracle.com/technetwork/java/gc-tuning-5-138395.html#1.1.%20Generations|outline">Oracle documentation</a> ) shows how the age of objects is typically distributed. While most objects only live for a short time, there are also quite a few objects that stay alive a bit longer than 2-3 Young generation garbage collections (called minor collections). So the Young collection will be performed by saving the ones that survive in so-called survivor spaces. Objects stay in the survivor spaces until they have survived long enough to be promoted into the Old generation. Due to the way GC works, two Survivor Spaces are used and one of them is always empty. The remaining part of the Young generation is called Eden.<h2>Gathering heap metrics</h2>So how is the total memory distributed onto the different areas? The JVM offers us to keep track of a lot of information on memory via JMX. For example, there is a nice command line utility which shows you the usage of all your spaces:<div>

```bash
jstat
```

</div>For my running Eclipse it looks like this:<div>

```bash
C:\Users\fla>jstat -gcutil 4188
 S0 S1 E O P YGC YGCT FGC FGCT GCT
 0,00 0,00 1,61 50,89 99,90 35 0,304 109 28,229 28,532
```

</div>Ok, looks like the JVM is quite idle. Eden is almost empty (very low percentage), Old is filled about 50%. Perm is pretty full. The GC/GCT measurements are not important at the moment. But what is the size of those spaces? <code>jstat -gc</code> reveals this:<div>

```bash
C:\Users\fla>jstat -gc 4188
S0C S1C S0U S1U EC EU OC OU PC PU YGC YGCT FGC FGCT GCT
1984,0 1984,0 0,0 0,0 16256,0 263,6 40388,0 20553,3 55808,0 55752,7 35 0,304 133 34,558 34,861
```

</div>Lots of numbers, all in kilobytes. Survivor Spaces are 2MB each, Eden is 16MB, Old is 40MB, Perm is 55MB. This comes close to the overall memory consumption of 135MB. The gap are the non heap memory areas.All options and columns are explained in great detail in the <a href="http://download.oracle.com/javase/1.5.0/docs/tooldocs/share/jstat.html#gc_option">official jstat documentation</a> .<h2>Configuring the heap</h2>What can we configure, now that we know the occupation and sizes of the spaces? While the sizes offer a lot of configuration functionality, they are usually only touched when looking at garbage collection. All heap areas are equally good for storing data, but for garbage collection you might want to change some.Without configuration, a client JVM uses this calculation for the spaces:

```
Heap = Tenured + Young
Tenured = 2 x Young
Young = Survivor x 2 + Eden
```

A common change is increasing the maximum Permanent generation size using <code>-XX:MaxPermSize=128m</code>. The initial size can be set with <code>-XX:PermSize=64m</code>. Depending on the application, the New size can also be adjusted. This can be done as ratio <code>-XX:NewRatio=2</code> (which is recommended, as it adjusts automatically), or as fixed size <code>-XX:NewSize=128m</code> or <code>-Xmn128m</code> (which is less flexible but easier to understand). All ratios are configured as “one of me – N of the other”, where N is the value provided to the flag. For example, <code>-XX:NewRatio=2</code> means 33% of the heap are for New and the remaining 66% are for Old.Young space can be configured by using <code>-XX:SurvivorRatio</code> to control the size of the survivor spaces, but this is often not very effective. A typical configuration for a web application could look like this:<div>

```bash
-Xms2g -Xmx2g -XX:NewRatio=4 -XX:MaxPermSize=512m -XX:SurvivorRatio=6
```

</div>The total memory used by Java would be slightly over 2.5GB. Old would be 1.6GB, Eden 300MB, and each Survivor Space 50MB.<h2>Monitoring the Heap</h2>Because the JVM is providing a lot of information on the heap, command line output is not the most convenient way to read it. In the screencast below, I will show you three tools for looking at the heap:<ul><li><a href="http://java.sun.com/developer/technicalArticles/J2SE/jconsole.html">JConsole</a> , basic UI to read the JMX metrics provided by the JVM</li><li><a href="http://java.sun.com/performance/jvmstat/visualgc.html">Visual GC</a> , a great GC visualizer also available as plugin for VisualVM – suited best for on the spot analysis</li><li><a href="http://appdynamics.com/free">AppDynamics</a> , providing long term monitoring for memory statistics</li></ul><div> Unable to display content. Adobe Flash is required.</div><h2>Why does OutOfMemory occur with free memory available?</h2>You should now be able to answer the question we brought up in the beginning. Let’s have a look at the last snapshot of the Visual GC output: <img loading="lazy" title="Visual GC" src="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/wDc7SAs4QoqfZt6558og" alt width="579" height="298" /> The Old generation is full. Survivor Spaces are empty, and Eden has only 30%. Which means that we get an OutOfMemoryError even when we have still almost 30MB free memory in our Xmx limit of 128MB. So why do we get the error at all? While we do not know exactly what’s going on, the code creates bigger and bigger arrays to hold the Long objects created in the loop. A new Array is about to be created right here: <code>at java.util.Arrays.copyOf(Arrays.java:2760)</code>. Now suppose that at some point there is not enough space in Eden for doing this (which means that we can assume that the array is larger than 22MB). A garbage collection should kick in and make some space available. However, there is no space left in Old, and the existing Eden objects cannot be moved anywhere. That’s when OutOfMemoryError is thrown. To conclude, when you see an OutOfMemoryError it doesn’t mean that your whole heap memory has been exhausted. But you can be sure that at least one of the spaces is filled, preventing creation or movement of objects there.In the next act, we will have a look at the actual objects stored in the heap by creating heap dumps.

Java Memory Configuration and Monitoring (3rd Act)

When I listened to Martin Fowlers Talk last week at Java User Group Cologne, I felt the urge to talk about one of my projects at codecentric. Martin said that DSLs are very useful because the business people can actually read the code programmers produce. This is in fact more powerful than the often brought argument, that business people can write code. Martin did a fair amount of ranting on the structure of Java code. He said that you need to decode it like a detective, so he built his own language with a parser. But noting on that, he made a great quote:<blockquote>Some people think that writing parsers hard, because they did not attend a compiler class at college. And those who did attend such a class know that writing a parser is hard.</blockquote> And because of that, he showed that his custom built DSL is pretty much valid Ruby code. But the point is that once you have your concepts laid out, the actual formatting doesn’t matter that much and you can write your DSL in a host language like Ruby, or, as we did, in <a href="http://www.jboss.org/drools">Drools</a> . Drools offers a Rule Engine. That means the semantics behind it are already optimized for evaluating stuff. We needed three DSLs for the project:<ol><li>A recommendation engine, which uses profile data to come up with recommended products</li><li>A validation engine, which checks the profile for consistency</li><li>A calculation engine, which uses profile data to calculate prices</li></ol>All three were similar enough to consider using one technology for them, but they are in practice quite different. The recommendation engine would check for certain properties and come up with a recommendation, while the calculation needs formulas or tabular data. The validation works similar to the recommendation, but returns errors instead.With knowing that, JBoss Drools came immediately to our mind. It was a perfect fit from technology and we only had to work on a nice way integrating our domain into the Drools DSL.All three need to work on what we call a profile. We already have a domain object representing that, but invoking plain getters was not wat we wanted. For example we wanted to check for a age but only had a birthdate in our profile. So we came up with the idea of creating a wrapper around our domain object. This wrapper takes then all accessors needed to formulate the DSL from a business perspective. Output is handled by global objects in Drools where you can set values on, so we used a nicely named output object in the global space. The rest of the excercise was just to tweak it, so that it became readable. And in fact this worked out for us. Our customer can read recommendation and validation rules to verify them, and they can fill in the pricing data. Judge for yourself. Do you like how it looks? Yes, that is a DSL. And not even a fancy one.Here, take a look at our validation DSL using Drools:<div>

```java5
global ValididationErrors errors;
 
rule "age needs to be less than 101"
	when
	 Profile( age > 100 )
	then
	 errors.add("If you are older than 100, you need to contact your local sales consultant.", Ids.BIRTHDAY);
end
 
rule "Birthday needs to be in the past"
	when
	 Profile( birthday > today )
	then
	 errors.add("We can only provide a calculation for already born people.", Ids.BIRTHDAY);
end
```

</div>And here our recommendation DSL:<div>

```java5
global Productportfolio products;
 
rule "children"
when
	Profile( age <= 15 )
then
	products.recommend( "PRODUCT_A", 5 );
	products.recommend( "PRODUCT_B", 0 );
	products.recommend( "ANOTHER_PRODUCT", 3 );
end
```

</div>and a screenshot of our calculation engine. You can even fold the spreadsheet to hide more implementation details:<a href="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/oNfzxwgS3yBl5mQgCYXP"><img title="drools-pricing" src="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/oNfzxwgS3yBl5mQgCYXP" alt width="540" height="222" /></a> We had great success with that, as we can hand those DSLs and spreadsheets to business people, who can validate that the “programmed logic” is what they want to see.

Pragmatic DSL in Practice, with Java and Drools

During the load tests of a project I noticed the overall response time going down rapidly when more and more users were accessing the application. The application was built using the Struts 2 Java framework, accessing the database via Hibernate. It was quite a simple application, so I was wondering why it was going down so fast. When looking at the <a href="http://www.appdynamics.com/free">AppDynamics monitoring</a> , I realized quickly, that there has to be an issue with Freemarker, the templating engine you can use in Struts2:<a href="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/A4ka7v5KQI68JCRkZt7P"><img src="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/A4ka7v5KQI68JCRkZt7P" alt width="553" height="320" /></a> All hot spots originate from freemarker.cache.URLTemplateLoader.getURL() which is the abstract definition of a method implemented by the Struts org.apache.struts2.views.freemarker.StrutsClassTemplateLoader.Lets debug further and find out why this code is hanging: AppDynamics collected a few thread dumps for us, which reveal a few blocked threads on sun.misc.URLClassPath.getLoader().<a href="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/dkv4cYfNRUa1Ergp77VK"><img loading="lazy" src="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/dkv4cYfNRUa1Ergp77VK" alt width="540" height="382" /></a> Looking at the source code, this is synchronized on the URLClassPath instance.

```
private synchronized Loader getLoader(int index) {
```

 I did not expect this to be a bottleneck, but it turned out that there are additional threads holding/wanting this lock for classloading. This however was paused by native code for memory allocation.There is nothing wrong with the Freemarker/Struts way of loading the templates, however it does seem to create a lot of unnecessary access to resources.Usually templates do not change often, so it would be great if there was some kind of caching facility. And indeed Freemarker offers something:

```
template_update_delay=60000
```

This is a setting you can pipe through a struts file called freemarker.properties into the freemarker engine. <a href="http://struts.apache.org/getting-started/">According to the documentation</a> , this takes a number of seconds a template has to be in the cache before it will be reloaded. So one should set this to a high value.Struts folks also advise you to set a property called struts.freemarker.templatesCache to true, as they claim that the last modification cannot be obtained in a stable manner from the class path. However, as the last modification already worked well for my tested modern JDKs, I doubt this will bring further improvement.Using the template_update_delay worked for the project, so I am fine with that, yet I still wonder why there should be a need at all to reload the files. Ok, if you hot-hack them, you need to reload them, but on real production systems, you should load them once. There is quite some code in freemarker.cache.TemplateCache which would not be required then.While you are at tuning the Freemarker related code, you might want to provide a custom implementation of the class  org.apache.struts2.views.freemarker.FreemarkerManager, which is the connection between Struts2 and Freemarker. On closer look, it reveals to you that it tries to load the template twice from the file system before resorting to the classpath. This gives you some flexibility when developing your application, but comes with a price you might not want to pay.After changing the “cache” setting, the HotSpots disappeared which gave the application more room to breathe. Classloader became a much less contested resource.

Freemarker Template Caching – Analyzing a Struts2 Performance Problem

While TV stations record and broadcast their review of the year’s events already in November or December, we follow the agile tradition and thus look back only after the year (2010) has actually ended.Changes in 2010We continued to grow steadily during 2010 and by now employ over a hundred IT specialists. But we did not just grow in numbers, but also expanded our reach geographically by adding 3 more sites to our map.In <a href="http://de.wikipedia.org/wiki/Novi_Sad" target="_blank">Novi Sad</a> , Serbia, we ramped up an additional agile nearshoring location. The main responsibility of our “Mobile Competence Center” is the research and development of mobile “Apps” for iOS and Android Smartphones and Tablets. During December our first customer made their app available in the AppStore, making the short development cycle a real success. While the app does not offer much for non-german speaking people, you can still have a look at it here: <a href="http://itunes.apple.com/de/app/vgh-automobil/id404553250?mt=8">VGH autoMobil</a> .Mid-year we additionally founded new offices in <a href="http://de.wikipedia.org/wiki/Dordrecht" target="_blank">Dordrecht</a> reaching for the benelux market with our Agile Software Factory and Services of Performance Solutions.Our newly created unit Performance Solutions received a major boost with our new office in Munich. Additionally, we were able to partner with the highly successful startup company <a href="http://appdynamics.com/">AppDynamics</a> , which offers a state-of-the-art APM solution with astonishing ease of use. 50 new customers during the first 6 months after the product went public – that undeniably proves the need for it.Our <a href="/en/services">Agile Software Factory</a> has been working for customers in many projects and still found time to innovate. As one artifact of this innovation, codecentric became Program Development Partner and host of the Professional Scrum Developer Course of scrum.org.With similar efforts, codecentric employees contributed to plenty of open source projects, with special focus on the test automation toolkit <a href="http://code.google.com/p/robotframework/">robotframework</a> , which we use in our projects for automated acceptance tests.Our project teams experimented with different kind of <a href="http://twitpic.com/2mpij0">Extreme Feedback Devices</a> to always stay informed on the state of their project. They work pretty well, I think.Moving applications into the cloud was technology-wise the major challenge. Some projects were already started, like with our customer Eismann, who is running on Amazon EC2 now. In 2011 more will most likely follow.To further exchange and improve our agile knowledge, we created the Agile Meetup in Dusseldorf in fall 2010. We had 3 meetups already, and got great feedback.<a href="/en/services">Document Solutions</a> finalized a major project in 2010, which consolidated scanning and processing of subsidy claims for health insurance for civil servants. The 3 federal states Rhineland-Palatinate, Hesse, and North Rhine-Westphalia cooperated on this project. 100 new jobs were created by the increased processing volume for North Rhine-Westphalia in Detmold.In 2010 we also had plenty of first class events, hosted by our meet the experts. Those included the Certified Scrum Master Training by Boris Gloger, an Acceptance Test Workshop by Elizabeth Hendrickson, and the famous Java Specialists Master Course by Dr. Heinz Kabutz.KnowledgeAn essential value of codecentric is the knowledge of its employees. Therefore, we continued to invest into the 4+1 week, but also added softskill and language trainings for everybody.As usual, our employees had lots of opportunities to celebrate, either in small team organized events, or company-wide events like at our canoe-tour or the Christmas party, which brought us some remarkable speeches. A special codecentric tradition.We demonstrated our competence at several fairs and conferences in 2010, including a keynote at the W-JAX and various sessions at XP-Days, OOP, or gearconf. Of course we are planning to continue with this in 2011 and are looking forward to meeting you live. A soon to come opportunity is the JAX, where we are proudly hosting the <a href="http://jax.de/2011/specialdays/">Performance Day</a> . If you cannot attend or meet us in person, we are always very happy to discuss with you via our new hompage and of course this blog.Apropos BlogWe are very happy to see you all reading our blog! During 2010 we had about 50 thousand readers. Those numbers are for the web alone, some more people read our blog via its RSS feed. During 2010 we wrote and published 95 posts in German and 73 in English, numbers that we hope to topple in 2011.As a small side project for our blog, we started the codecentric podcast, which has already 5 episodes out on itunes. We got nice feedback for those episodes as well as for our newly launched <a href="http://www.facebook.com/more4fi">Facebook page for IT specialists</a> , running under the brand apprenticeship program “more4fi”.So there is a lot to do and to achieve this year. Let’s get started!

A Retrospective on 2010

The last area I want to examine in my series on WPO is the client side. Currently many people look at the browser as the only client. But I think clients like Apps can benefit from the same ideas as they are built with the same concepts nowadays, some even in HTML.One especially interesting fact about browser optimization is that it is mostly built on guessing. This is a nice commonality with SEO for which this is even more true, as nobody really knows how search engines operate. As a result of this, the majority of WPO tools use a term called “grade”. That’s right. Like in school you don’t know exactly what to do to get better. Just try some more. However there are a several tools available that analyze your sites and can give you pointers as to what you might want to fix or change to improve your grade. Two examples are <a href="http://developer.yahoo.com/yslow/">YSlow</a> by Yahoo and <a href="http://code.google.com/speed/page-speed/">PageSpeed</a> by Google which both use a catalog of best practices to check your sites.You can also use an aggregator like <a href="http://www.webpagetest.org">webpagetest.org</a> or <a href="http://www.showslow.com">showslow.org</a> to run various tests for you. They all offer check lists to let you know how your site is violating best practices and how to fix it. However, what you do not get is an estimate how much your changes will affect page performance. <h2>Requests waste time</h2>Despite the fact that we unfortunately do not know exactly how a site’s behavior will change when applying best practices, it is very clear that extra requests are to be avoided at all costs. Requests cost time and money. There are two aspectes of requests that can be looked at:<ol><li>Connection time</li><li>Payload transfer time</li></ol>Of course payload transfer never comes without connection. Avoiding payload transfer can be achieved easily by two practices<ol><li>Compression</li><li>Caching</li></ol><h3>Compression</h3>Compression is piece of cake, but very often neglected. It comes from a time long back in history when early versions of IE could not handle compressed responses. Adding compression varies but is usually very easy. I recommend to let your container do this (rather than to code it into your application). For Apache this is as simple as:

```
<IfModule mod_deflate.c>
AddOutputFilterByType DEFLATE text/html text/plain text/xml text/css
 text/javascript application/x-javascript application/javascript
 application/json application/xml text/x-component
</IfModule>
```

Of course making images as small as possible is a sensible idea too. A few days ago Kornel Lesinski published a neat tutorial on <a href="http://calendar.perfplanet.com/2010/png-that-works/">PNGs that work</a> , which also takes care about the transparency issues with IE 6.<h3>Caching</h3>Activating Caching is also easy, but requires a bit of thinking to avoid delivering outdated content. While you can try to cache all your HTML, most people do not need this, but should only cache really static content. So if you have a folder managed by your CMS like “export” or a place where all your images go, the whole folder should have an expiry defined for a long time. In Apache this is easy as well:

```
<IfModule mod_expires.c>
 ExpiresActive On
 ExpiresDefault "access plus 10 years"
</IfModule>
```

The only problem is that if you are going to replace an image, it has to have a new name, otherwise it will not get downloaded (we declared the old version to be good for 10 years). The usual practice for solving this is to include a version or timestamp in the file name. Problem solved. The process can be automated partially or fully, as discussed later.<h3>Connection</h3>Regarding connection time, I already described in my second post how this can be cut down using the slow-start trick. But bundling resources can be even more efficient and also avoid multiple requests, especially if they have just a small content. This is usually employed for bundling JavaScripts into a single file and for so called CSS-sprites, a technique from early computer games, where one large image file contains all images and just a section of it is displayed. Of course the best advice is to not include anything on your pages that you do not need.<h2>JavaScripts waste time</h2>Browsers perform multiple duties, but the one that takes most time is JavaScript code parsing and execution. This is also the reason why browser makers keep improving their JS engines constantly. As a consequence, I do not know how long benchmarks like the great JavaScript Loop Benchmark by Greg Reimer will hold true. The main issue I personally see is the amount of really bad JavaScript code on the net. Developers are tempted to copy and paste parts from the internet without actually programming it. JavaScript is a powerful language, but it’s shocking to see how badly it gets used (I did this myself).The above linked loop benchmark shows how easily you can (or at least could) mess it up. A smart use of language might let the loop take 15ms, a incorrect usage 3203ms (Example: HTML Collection (length=1000, looped 100 times), IE7). Additionally, most JavaScript execution is blocking page rendering. This is not very desirable and often seen with advertisements. Unobtrusive JavaScript is the answer to that, but rarely used. The way to go is to move all JavaScript to the end of the page, removing all inline JavaScript. However, a change like this is seldom realistic.What we can do is profile JavaScript code. The timing specs of the <a href="http://www.w3.org/2010/webperf/">W3C-Performance Working Group</a> are still in the works, but Firefox already has a profiler included in the fantastic FireBug extension. Chrome Developer tools are a close follower of Firebug. They both allow us to watch code execution and find hotspots to fix.<h2>Automatic fix</h2>Recently Google released an Apache Module called <a href="http://code.google.com/speed/page-speed/docs/module.html">mod_pagespeed</a> . The idea behind it is that most best practices just should be followed and not cause any issues. However, some are hard to do upfront, but can be easily applied at runtime. This is the job of mod_pagespeed. It will fix your HTML, links, and cache config. It works best on unoptimized sites. However, it is extra code that is getting executed, so it might slow optimized sites down a bit. My usual advice: measure your results. Besides mod_pagespeed, there are multiple commercial solutions available, which also include CDN solutions.Another kind of automatic fix is the <a href="http://html5boilerplate.com/">HTML5 Boilerplate</a> , a web site template that covers all the best and proven configuration preconfigured and documented. I can highly recommend it for its ideas, especially for the CSS part of it.<h2>Summary</h2>I do believe that employing best practices makes your webpages faster, but finding the 20% change that will improve your page by 80% is not that easy. Tooling support is still a bit limited, so it is left to us to experiment with ideas and check their effects. Browsers are getting faster day by day, so cheating and voodoo has a short lifespan. Also for the web, we need a clean and simple design. If you want some expert consulting, feel free to contact us. We can find out what makes your pages slow and what will give you the boost with the best return on investment on all layers: infrastructure, server software and client side.I hope you have enjoyed my short introduction on Web Performance Optimization. Happy Holidays!My WPO series:<ol><li><a href="/en/knowledge-hub/blog/web-performance-optimization-is-the-new-seo">Introduction into Web Performance Optimization</a> </li><li><a href="/en/knowledge-hub/blog/web-performance-optimization-the-infrastructure">Web Performance Optimization: The Infrastructure</a> </li><li><a href="/en/knowledge-hub/blog/web-performance-optimization-serverside-software">Web Performance Optimization: Serverside Software</a> </li><li>Web Performance Optimization: Client Side</li></ol>

Web Performance Optimization: Client Side

This part of my series covers the most traditional part of WPO: The (custom written) software that runs on the server. So optimization also includes all our software design decisions and coding. Examples might be biased to Java, as this is my main language.<h2>Build Scalable Designs</h2>Before joining codecentric, projects I worked on were usually ending up in terrible troubles and coming late (they were big enough so that this could not only be my fault). As a result all systems and scalability tests were canceled. Actually that didn’t matter that much, as they were planned at the end of the projects, where design bugs are impossible to fix. At codecentric we are working on improving our agile process every day to make our projects successful. We need to think about a scalable design from day one. The question “What do we need to change to add 2 servers?” should ideally be answered with “nothing“. So how to get there? Some of the factors for a scalable design have already been mentioned in my post about <a href="/en/knowledge-hub/blog/web-performance-optimization-the-infrastructure">infrasturcture-oriented WPO</a> . Agile methodologies really help here, but even without you should run experiments with scalability early on. You should as well do at least basic load testing to understand load patterns in your application. I really would like to see this integrated into definitions of done and executed all the time, but I do believe doing this at all early on will bring big improvements. But just testing scalability does not realize it magically. It has to be designed. Gernot Starke talked at our last Meet The Experts about the dilemma that architects are facing when trying to learn about good design. His argument was that even with big companies like <a href="http://www.facebook.com/note.php?note_id=39391378919">Facebook</a> , <a href="http://highscalability.com/scaling-twitter-making-twitter-10000-percent-faster">Twitter</a> , <a href="http://www.infoq.com/articles/ebay-scalability-best-practices">Ebay</a> , Amazon and Google talking about their software designs quite often, those are rarely applicable to software most of us design and write every day. I think he is right. And I think he is wrong. Yes we might not have hundred thousands of data queries every second, but the design allowing this might scale and work better than what I would have come up with, even for my mid size customers application. So it is of course good to learn those patterns.<h2>Upgrade third party early</h2>Not just our coding and design make up the system performance. There are also plenty of other software products involved. I am assuming that we use at least a dozen external software products when building our applications. Thats not a bad thing. We do not need to write that code, which saves us a lot of time. But perhaps even more important: We do not need to be experts. We do not need to be experts on rule-systems, managing network connections, caching, encoding, compression and a lot more. We (almost) can concentrate on building our solution. So when we trust them building good components to build upon, why don’t we upgrade often? In the recent years, more and more software makers put “performance” in their release notes. Almost every release of every software brings performance improvements, but we usually do not take them. For me there are two reasons for that<ol><li>Fear of backwards incompatible changes</li><li>Chaos in third party management with a inflexible process</li></ol>Both are valid reasons. However the first one just gets worse. After a certain point the changes required to upgrade have accumulated to a big pile that nobody wants to touch. So I recommend to upgrade often to benefit from all the performance improvements your external experts make. From my experience there is an interesting correlation between fear and (performance) gain. Upgrading Application Server, Databases, UI Frameworks, Rule Engines usually gives much better performance than changing a version of Apache commons-lang. But they are feared more. I think the reason for that is that those parts are huge and complex, which is exactly the reason for them offering so much optimization potential. But when fearing problems, how could you decide to use it at all? The second one however is difficult to solve. Many people think throwing Maven at the problem will solve it, but I think the real reason is often a messy design or just ignoring the issue, which technologies cannot rescue. On the technical side things like maven for build management and dependency declaration and for example OSGi for managing this at runtime are really helpful, but can never even out design issues. I do believe this can be managed in one way or another.<h2>Choose the fastest communication protocol</h2>Communication protocols are very important in a distributed system. However we often fail at doing good decisions there. With all the SOA hype we all build software that use SOAP Webservices. Which is worst protocol at all – performance wise. Also often services are either too small or designed incorrectly, so that a lot of data has to be transferred or many remote invocations have to be made. But assuming a good design, the protocol can make a difference. There are public available benchmarks, like the <a href="http://docs.jboss.org/jbossremoting/docs/benchmark/performance_benchmark.html">JBoss Remoting Benchmark</a> , or a Java benchmark by Daniel Gredler as well as many other you can google for. But ideally you do run your own benchmark for your use case. With respect to AJAX there are only 3 formats which all work over the HTTP connection.<ol><li>XML – But I think nobody is using that. Too much data overhead.</li><li>JSON – Protocol of choice for most developers. Often plain key value pairs. Fast, as it can be translated directly to JavaScript Objects.</li><li>JavaScript – Instead of just JS Objects some people transport code that will be executed. Allows fancy tricks, but is an indicator for a too generic interface.</li></ol>An interesting addition to that will be <a href="http://dev.w3.org/html5/websockets/">WebSockets</a> , which is coming with next browser releases and already somewhat supported through solutions like Kaazing.<h2>Gain insight into your application performance</h2>The server side application is a big black box for WPO, but unfortunately a main contributer to the lack of great performance. You cannot compensate a slow serverside response with WPO tricks, but you need to investigate why the response is slow. For that I recommend an application performance monitoring solution. Unlike traditional systems monitoring, it opens up the black box and can look inside. Those APM solutions usually support just a single programming language, and we think that for Java <a href="http://www.appdynamics.com/">AppDynamics</a> is the best solution on the market, which is why it is in our portfolio. When you have a monitoring solution running on your production, you will get a fast pointer to code and design areas that cause your application to slow down. When you are working to fix those issues, you usually use a profiler on a developer machine to capture every tiny detail of the transaction you are trying to improve. For Java I can recommend <a href="http://www.yourkit.com/">YourKit</a> and <a href="http://www.ej-technologies.com/products/jprofiler/overview.html">JProfiler</a> .The server side software is usually not looked in detail at by many WPO folks. The reason for that is that this area is now new, but still an important factor. At codecentric we have a lot of experience in solving those performance issues, both on design and on deep-inside-frameworkcode level. In my last episode of this series, I will talk about the most hyped area of WPO. The clients, meaning browsers, and the tuning potential they offer.My WPO series:<ol><li><a href="/en/knowledge-hub/blog/web-performance-optimization-is-the-new-seo">Introduction into Web Performance Optimization</a> </li><li><a href="/en/knowledge-hub/blog/web-performance-optimization-the-infrastructure">Web Performance Optimization: The Infrastructure</a> </li><li>Web Performance Optimization: Serverside Software</li><li><a href="/en/knowledge-hub/blog/web-performance-optimization-client-side">Web Performance Optimization: Client Side</a> </li></ol>

Web Performance Optimization: Serverside Software

In my previous blog I described three key areas for WPO, one of them being infrastructure with all topics around server setup. In this blog I am going to describe this in detail.<h2>About Content Delivery Networks</h2>Wouldn’t it be great to have somebody hosting your content, close to your customers? That’s what Content Delivery Networks (CDN) are useful for. So far only large companies with worldwide customers have used them, but they can be just as useful locally, too. They do have the fastest network connection possible and can reduce your IT spendings. But you can also create CDNs yourself. Hosting images on a different subdomain reduces data transferred, as less headers and cookies are sent with the request for each image. Those subdomains could be also pointing to specialized servers. As a simple example: a httpd could serve images from a RAM disk, instead of having a Tomcat server generating them from inside an archive file. And you can utilize public CDNs like Google. A very good example is including JavaScript Libraries from Google. Not only are they hosted on excellent network and server capacity that you can use for free. They are also cached with maximum effect, because you share your cache with other big sites resulting in even faster cache primes. A comprehensive discussion on the advantages of using Google as CDN by Dave Ward covers this in great detail.<h2>Distributed Memory Caches are fast</h2>Content hosting is one part of Infrastructure. The other part is running your application with your business logic. You cannot prepare readymade responses, but will have to make sure all requests can be answered, even when thousands of users are hitting your site. Once you outgrow a single server things get complicated. But this can be fought with simple designs. A common problem involved with scaling Java applications is that session data is held per node. So you cannot shift users easily to different servers. This is called “sticky session”. An attempt to fix this was to introduce session replication, which copies session data to other nodes which then can take over the user. But I strongly advise not to do this. It just causes to much trouble and effort with a minimal advantage. It would be much better, however, to have a stateless server. This allows ramping up and down computation power with ease. The big question is: Where should the state go. We need state. Looking back, state was put into the session, because the central data storage called “database” was just too slow and did not scale easily either. But I am not mandating to put session state in traditional databases. I am proposing to remove them as well from your architecture. The state of the art solution for this dilemma are so called NoSQL databases, which work in a distributed way and store data in key value pairs. It’s not rocket science, but simple. And current developments prove that this simple solution works out much better than the traditional RDBMS. Big Java NoSQL databases are Hadoop and Cassandra. And session information should be kept in memory on a distributed memory cache solution like MemCache. A great compilation of solution ideas can be found on <a href="http://nosql-database.org/">nosql-database.org </a> The reason for making your application stateless is that it allows easy scaling. When load goes up usually some limits on existing infrastructure are reached that would in fact scale, but no longer linearly. In those situations it is advisable to start additional servers. Ideally decoupled ones, so you can either start API servers, Presentation servers, or Logic servers. Dynamically starting servers, also close to your customers, is the real added value of “the cloud”. This is not possible with complicated replication and failover solutions mostly deployed on companies’ internal systems.<h2>Protocol Finetuning</h2>Of course fast networking equipment and sensible physical distribution of servers also make sense, but offer only little tuning potential. However WPO leaders like Google started to create new networking protocols like <a href="http://www.chromium.org/spdy/spdy-whitepaper">SPDY</a> , build custom network adapters or bend the rules set by RFCs to make a fast experience. One example of these is the so called slow-start feature of TCP. As many other RFCs, TCP was defined in the early days of networking and is still used. At that time clients had very bad connectivity to the server, so the idea was to protect the server from bad clients and vice versa, so they start sending data only when the clients accept them. The amount of data that can be sent is called initial window and is described in <a href="http://www.rfc-editor.org/rfc/rfc3390.txt">RFC 3390</a> . But actually sending more data saves roundtrip time, allowing to get below 100ms page loads. A good start into this discussion can be found in Ben Strong’s Blog about Cheating on slow start.Those are just a few ideas how operations could help improving the performance of web pages. While some aspects are limited by the application architecture, others could be offered as a premium service by the hoster to their customers. While the area of infrastructure is not our key competence at codecentric, we can help designing applications that get the full benefit from infrastructure optimizations and can speak the language of operation teams to get performance improved on all sides. One of them also being software architecture, which I will discuss in the next installment of this blog.My WPO series:<ol><li><a href="/en/knowledge-hub/blog/web-performance-optimization-is-the-new-seo">Introduction into Web Performance Optimization</a> </li><li>Web Performance Optimization: The Infrastructure</li><li><a href="/en/knowledge-hub/blog/web-performance-optimization-serverside-software">Web Performance Optimization: Serverside Software</a> </li><li><a href="/en/knowledge-hub/blog/web-performance-optimization-client-side">Web Performance Optimization: Client Side</a> </li></ol>

Web Performance Optimization: The Infrastructure

<h2>What is Web Performance optimization?</h2>SEO is the way to make your website successful? Not anymore. Search Engine Optimization was useful 10 years ago, when search engines needed help indexing your content. But today the only tricky part is to provide good content and search engines will magically find it. It is especially interesting to notice that even sites which have a “bad” SEO but provide good content are preferred over SEO optimized sites with bad content. So, because finding something on the Internet is not an issue anymore, it is even more important to keep customers satisfied with their experience on your site or they will move to your competitor. The key factor for customer satisfaction is website performance. So we need to make sure that websites are fast, which is called Web Performance Optimization = WPO. WPO means addressing various performance challenges to deliver your webpages as fast as possible to keep customers satisfied and generate more revenue. Studies have proven that users will more likely come to sites that load fast, especially on mobile connectivity. 2-3 seconds is the average time users are willing to wait before going to alternatives. With a strong trend to even less than 2 seconds. As a side effect of improved performance, also operative costs can be cut with reduced power and bandwidth consumption, resulting in optimized cost per sale and the overall bottom line.<h2>Can we prove the effect of WPO on revenue and user experience?</h2>Those promises and numbers indeed exist, as multiple companies already have proven by measuring the impact of performance optimizations.One of the best examples is Google. Google went into the search engine market which was owned by bigger players. However, Google understood WPO and has been always working on delivering the fastest website. In just a few months Google became the number one search engine with just good and fast results. Today, Google continues to push for faster web performance and just recently released Google Instant: search results that appear before you finished typing for them.But also besides Google other key Internet companies build their success on improving performance. Here are some interesting numbers.<ul><li>Amazon: 100 ms delay caused a <a href="http://glinden.blogspot.com/2006/11/marissa-mayer-at-web-20.html">drop in revenue</a> .</li><li>Google: 400 ms delay caused a 0.59% decrease in search requests per user.</li><li>Yahoo!: 400 ms delay caused a <a href="http://www.slideshare.net/stoyan/dont-make-me-wait-or-building-highperformance-web-applications">5-9% decrease in traffic</a> .</li><li>Bing: 2 seconds delay caused a 4.3% drop in revenue per user.</li><li>Mozilla made their download page 2.2 seconds faster and was rewarded with an <a href="http://blog.mozilla.com/metrics/category/website-optimization/">increase of 15.4% in downloads</a> .</li><li>Google Maps reduced the file volume by 30% and observed <a href="http://news.cnet.com/8301-10784_3-9954972-7.html">a 30% increase in map requests</a> .</li><li>Netflix, which uses Appdynamics to monitor their performance, enabled gzip on the server; simply by this single action pages became 13-25% faster and saved 50% of traffic volume!</li><li>Shopzilla succeeded in reducing the loading time from 7 down to 2 seconds, whereby the conversion rate increased by 7-12, they observed a 25% increase in page requests, they were able to <a href="http://www.stevesouders.com/blog/2008/03/06/how-green-is-your-web-page/">retire 50% of their servers</a> , thus saving energy costs.</li><li>AOL observed the number of page views on several websites. While the fastest users requested 7-8 pages, the slowest only viewed 3-4.</li><li>Edmunds got down from over 10 seconds to less than 2, adding 17% page views and increasing revenue by 3%.</li></ul><h2>How to implement WPO?</h2>There are 3 main parties which can contribute to improving web performance<h3>Infrastructure and Hosting Team</h3>Usually some kind of scaling and loadbalancing is used when high load is expected, however there are multiple ways to set up the various servers. On the Internet, many architectures try to achieve a shared nothing principle, while in enterprises usually failover and replication architectures are used. Also the usage of content delivery networks close to your customers is part of this area. Usually companies outsource this to hosting provider, allowing the hosters to create unique solutions for their customers. However not all optimizations can be done by the hoster alone and also the devops trend is closing the gap.<h3>System and Software Architects</h3>We are talking about the server side software here which includes optimizing the actual application code, which is generating webpages and ajax responses. This area is hughe and contains most of the software design issues as well as a lot of coding. At codecentric we have a long history in solving performance issues inside and around Java serverside software. In this area even third party upgrades like Application Server or utilized frameworks can give a better performance. Also the correct protocol and caching solution choice and its options do matter.<h3>Web Developers</h3>When working for web one usually assumes that the client is thin. But this is only partially true for modern Web Applications. Parsing and executing markup, Javascript and CSS can be a heavy process. Complex layouts and scripts can stress browser engines which were not prepared for this load. In recent days browser vendors did improve their engines, however optimizing can still yield vast improvements. This especially holds true for mobile devices, which are still at the low end of performance. Developers also have to think about the actual data transfer to clients. Payload needs to be made small and fast, avoiding re-requesting data unnecessarily.In the next installments of this series, I will talk about those three areas and give examples of best practices or typical Antipattern. You will see that there is a grey area and some overlap between these responsibilities. There are also different troubleshooting and monitoring tools available for each area.Credits go to <a href="http://twitter.com/coachwei">Coach Wei</a> for his <a href="http://blog.yottaa.com/2010/11/secret-sauce-for-successful-web-site-web-performance-optimization-wpo/">inspiring blog post about WPO</a> , where I took some facts from.My WPO series:<ol><li>Introduction into Web Performance Optimization</li><li><a href="/en/knowledge-hub/blog/web-performance-optimization-the-infrastructure">Web Performance Optimization: The Infrastructure</a> </li><li><a href="/en/knowledge-hub/blog/web-performance-optimization-serverside-software">Web Performance Optimization: Serverside Software</a> </li><li><a href="/en/knowledge-hub/blog/web-performance-optimization-client-side">Web Performance Optimization: Client Side</a> </li></ol>

Web Performance Optimization is the new SEO

The following example shows how easy it is to combine a performance monitoring solution with a profiler. On a regular patrol through our <a href="http://www.appdynamics.com/free">AppDynamics</a> monitoring on our continuously integrated projects, I found this interesting HotSpot in iText. <a href="http://itextpdf.com/">iText</a> , a previously free but now commercial java library, allows to parse PDFs easily by just using a PDFReader:<div>

```java
PdfReader reader = new PdfReader(filename);
```

</div>Thats really easy! But in fact this does more than expected as you easily can discover on the following screenshot showing code HotSpots in AppDynamics: [singlepic id=336 w=560 float=]About 20% of the whole transaction is spend on opening 2 PDFs. I was getting curious. What is happening there? Is there a chance for slow code?The Invocation Trace shows that inside the Constructor of the PdfReader a PRTokeniser is created, which itself creates a RandomAccessFileOrArray. This then again opens a RandomAccessFile or a MappedRandomAccessFile. Latter uses a FileChannel to read the file into memory. Looks like a simple approach, but with a lot delegation, so there should be a way to look for alternatives to play with.So I wrote a little microbenchmark for our code which reads fields from PDF files. After3000 warmup calls, I ran 10000 benchmarked calls, which totaled to 12.3 seconds on my machine (Win 7 32bit, Java 1.6.20, -server). To understand what the code was doing I used <a href="http://www.yourkit.com/">YourKit as Profiler</a> . After the code was run, it produces this report: <img src="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/cpATh8lpT6qfFaGlaoRN" alt title="YourKit sees iText issue" width="568" height="218" />I found an alternative Constructor, which was taking a RandomAccessFileOrArray directly. So i tried this to be able to substitute the method for reading the file. So to approach this in baby steps, I did what the other constructor would have done:<div>

```java5
PdfReader reader = new PdfReader(new RandomAccessFileOrArray(filename), null);
```

</div>But I got surprised! I found out that this code took a lot less time for the 10k loops: 5.8 seconds. By just doing the same stuff as before? Really? The JavaDoc revealed what was going differently here:<div>

```java5
/**
* Reads and parses a pdf document. Contrary to the other constructors only the xref is read
* into memory. The reader is said to be working in "partial" mode as only parts of the pdf
* are read as needed.
**/
```

</div>Interesting! This could be performing a lot better for my use cases, which only covers reading form fields.YouKit proved this. No more traces of expensive constructors: <img loading="lazy" src="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/kG01qymZSZqQkgiqfTnO" alt title="YourKit verifies changed performance profile" width="570" height="203" />So this second variant is faster. And I was not even looking for this. But when working on performance, I can recommend to reach for lowhanging fruits. And very often you find that you need to change a slightly different place than the one exposing the performance issue. Since I did this improvement in our PDFService code it never showed up in performance reports.For me this is a very good example of how you should work towards improved performance<ol><li>Monitor performance in production and test.</li><li>Analyze anomalies.</li><li>Evaluate different approaches to solving the problem using a profiler.</li><li>Prefer doing small simple changes with large effect.</li><li>Continue observing performance.</li></ol>

Optimizing iText performance using AppDynamics and YourKit

Recently at a customer, I saw massive amounts of garbage generated, causing many garbage collections, as well as a huge slowdown inside Hibernate code. I browsed through the slow transactions recorded in production by AppDynamics, and was wondering why the DB access was super fast, while some other code in Hibernate was slow. I actually didn’t expect that a well used framework like Hibernate could be so slow in some unrelated code. But when I looked closer and drilled down into the snapshot data, I noticed that the time was actually spent in org.hibernate.pretty.Printer. Oh! So that explains the tons of garbage produced and massive slowdowns in hibernate: It was generating debug sql/object information. But why did the customer not see that before. No log did show the information at all. The disc also must have been full after minutes under production load.The answer is simple, and quite shocking. The following sample Log4J config might already reveal it.<div>

```xml
<log4j:configuration xmlns:log4j="http://jakarta.apache.org/log4j/">
 <appender name="console" class="org.apache.log4j.ConsoleAppender">
 <param name="Threshold" value="Error"/> 
 <param name="Target" value="System.out"/> 
 <layout class="org.apache.log4j.PatternLayout"> 
 <param name="ConversionPattern" value="%-5p %c{1} - %m%n"/> 
 </layout> 
 </appender> 
 
 <root> 
 <priority value ="debug" /> 
 <appender-ref ref="console" /> 
 </root> 
</log4j:configuration>
```

</div>Now have a look at the following strangely named class.<div>

```java5
package de.codecentric;
 
import org.apache.log4j.Logger;
 
public class PhantomLogger {
 
	private static final Logger LOG = Logger.getLogger(PhantomLogger.class);
 
	public static void main(String[] args) {
		System.out.println("Doing stuff");
		if (LOG.isDebugEnabled()) {
			LOG.debug(getDebugMessage());
		}
 
	}
 
	private static String getDebugMessage() {
		System.out.println("Expensive log generation here");
		return "Debug message";
	}
 
}
```

</div>The code was running with debug logging defined by the root category – so LOG.isDebugEnabled() also returned true. The heavy lifting did occur. Strings were mashed, garbage produced. Then passed on to the poor Log4J which knew that debug logging had to be accepted. It looked for the appender, found it and sent it there. The appender itself took the debug information, said “No thanks” and threw it away, as its threshold was not met which was set to Error.The lesson learnt from this is: Try to avoid using the threshold of the appender, or make sure that all categories are at maximum at the lowest level used by an appender they are appending to.

Phantom java logger causing major performance problems

<img src="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/PPw4Y9HRRlGEVegrymDD" alt title="gearconf" width="319" height="81" />The GearConf is a growing german speaking conference for professional software development. While it is focused on tools, the usual agile methodologies also found their place. Hosted by Infaktum, it took place this year on 11th and 12th October in the friendly City Hostel in Düsseldorf, directly at the shore of the Rhine. codecentric sponsored the conference with a booth and two talks, by Andreas and me. Andreas talked about the effective use of JIRA. Unfortunately he had some minor technical difficulties, so he promised to follow up with a series of videos. I have already seens ome of them, and I am already looking forward to his blog post about them. My talk was about performance. I was asking myself how to place optimization for best performance fits into the agile development processes. For me there was an easy answer: Its part of the definition of done. Of course you might ask how to achive this. My answer for that was: use a great tool like AppDynamics, which we also showed both days to interested visitors of our booth. But no matter what tool you use, you should be able to efficiently take care of all aspects of your software, which is fully in line with the spirit of Software Craftsmanship.So here my slides for that talk:<div><a href="http://www.slideshare.net/fabianlange/agile-development-of-high-performance-applications">Agile Development of High Performance Applications</a> <div>View more <a href="http://www.slideshare.net/">presentations</a> from <a href="http://www.slideshare.net/fabianlange">Fabian Lange</a> .</div></div>To summarize to days of gearconf: Good conference, lots of great talks and in common an interest in improving the standards of software development.

Report from the GearConf – Good Tools matter

AppDynamics is the rising star in the Application Performance Management sky.Mirko gives a really good description why AppDynamics delivers the right solutions for todays distributed architectures in his Post “<a href="/en/knowledge-hub/blog/troubleshoot-java-in-production-introducing-appdynamics-lite">Troubleshoot Java in production – introducing AppDynamics Lite</a> “. If you have not read it yet, head over to it now.The key promise of AppDynamics is simplicity. Easy installation, easy operation and very fast results. In fact this sounds uncommon and hard to believe. But this is indeed the case, as proven by the following uncut screencasts. Installation of AppDynamics is fast and easy like pie. In this screencast I will be showing to you how to perform the installation in less than 3 minutes (there is a full screen mode in the lower right corner): Directly after the installation you can start investigating your performance issues. In the following screencast I show you the basic functionality and a possible workflow for finding the root cause of a slow response (there is a full screen mode in the lower right corner): AppDynamics Lite is suitable for Development and for Production. It can be installed in almost no time, does not require any complex configuration and delivers valuable results within the first minutes. You can download the free lite edition on <a href="http://www.appdynamics.com/free">appdynamics.com/free</a> .

Easy Performance Analysis with AppDynamics Lite

When developing a site in JSF many people like to use the <a href="http://www.jboss.org/richfaces">Richfaces Framework</a> . It is a pretty solid framework that comes wit a lot of functionality and nice components. It also comes with an Ajax framework called <a href="http://en.wikipedia.org/wiki/Ajax4jsf">Ajax4JSF</a> (also called a4j). When deploying this site then facing the Internet to production, many people start finding out that their applications eat a lot of memory, leading to unresponsive systems or OutOfMemoryError crashes. This is due to a design issue in JSF / A4J and cannot be easily fixed, but worked around. But lets start with an analysis of whats wrong with our otherwise nice application. Finding the culpritTo find out whats wrong the best is to create a heap dump at peak usage to show whats consuming all the memory. After opening that dump in the wonderful <a href="http://www.eclipse.org/mat/">Eclipse MAT</a> you might get a picture like this:<img title="richfaces-sessions" src="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/PrUv6UURguqkBrkysica" alt width="538" height="441" />Ouch. 1.8 GB out of 2GB are consumed by sessions. I usually then filter by “StandardSession” for apache sessions to be able to easily browse around the sessions that have high amount of retained heap.<img loading="lazy" title="richfaces-sessions-per-session" src="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/5KgIDVSSf2zc3ikVPReD" alt width="533" height="170" />Ouch again… 10MB per session, this cannot scale. Now it could of course be possible that some careless programmer put in much of our data into the session, but thats simply not true, as we can easily find out:<img loading="lazy" src="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/BeZWSJMkQZm4z1pkfYLA" alt title="richfaces-session-detail" width="590" height="435" />Thats interesting. Almost all of our memory is consumed by AjaxStateHolders. So what does this actually do?How JSF and A4J workI try to keep it simple. Before JSF actually renders HTML to the users browser, it builds an internal representation. For each page (or view) this internal representation (called component tree) is created and run through the lifefcycle. Any user events are processed. If there is a component where a user can select one of 3 items, those 3 items are loaded and a string representation put onto the page. The component keeps track of which item is selected.Now we do not want the user to submit the whole page to select another item, which would cause the component tree to be recreated and a different item to be selected. We want Ajax! For that A4J remembers the state of the components when the page was displayed. So it knows the component tree and those 3 items, and which one was selected. On AJAX actions the component is looked up, the state modified by selecting another item and the partial HTML representation sent back. This is also called partial page rendering.So how does the remembering work? Well you might have guessed it: It creates a AjaxStateHolder in the user session and attaches the component tree to it.How this can get bigOk, this might be the current page. I can have big views but it is only one. No thats unfortunately not true. A4J does store more than one view. But why? Look at your browser, you are quite likely to find a “back” button somewhat in the to-left corner. When you click that you see the last page. But the browser did not send a request, so the server does not know that you are on that page. Imagine you click AJAX functionality: Where should the server get the component tree from for that time you have been on the page? Easy answer: This is as well in the AjaxStateHolder. By default this tracks back 16 views. and because you might come across the same view in your browsing history more than once there are up to 16 variants of a view.As you can see this can get big. And remember, this is per user.How to fixUnfortunately there is no fix. This is how JSF works, and how A4J works.There is a ticket on the Richfaces bug tracker which describes this: RF-3878 – Session memory leak. And the best answer is: reduce the number of views you store.<div>

```xml
<context-param>
 <description></description>
 <param-name>com.sun.faces.numberOfViewsInSession</param-name>
 <param-value>1</param-value>
 </context-param>
<context-param>
 <description></description>
 <param-name>com.sun.faces.numberOfLogicalViews</param-name>
 <param-value>1</param-value>
</context-param>
```

</div>You will loose the browser back button functionality, but gain a lot of memory. Second way to fix is to reduce the size of the component tree, which is most likely affected by big lists of data and complex structures. However this is easier said than done.

Richfaces Sessions Eating Memory – Analysis of a Memory Leak

There are many ways to do a good sprint retrospective, so we decided to try a new one every now and then. This time we took the role of a painter, painting out impression of the last sprint into a formidable piece of art. It might look strange at the beginning, but it quite nicely captures the impressions and feelings about the last iteration and provokes different thoughts than a simple list of items which could be improved. Lets see if we will use this method some time again. As an example I am going to present you my painting. The ones from my team mates are interesting as well, but I would not dare to show them without permission 🙂 <img title="a_different_reflection" src="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/yLkbptXRSy2D7Re4hNhe" alt width="565" height="419" />You can see the long way to our goal. But we are close to it. You can see that I would like to get a reward for the fantastic sprints we did, so I painted a little cake right after the finish line. Below the path leading to the finish, I depicted our architecture and the actual goal of the project. I really like how well the architecture evolves and works our for us, and also like that our customers start to realize that their confusion was not necessary and they can realize their ideas wit it. At the top left, there is a bad test stability. It really can be improved and we actually started already addressing this. Right below that you can see our burndown for that iteration, which is not bad, but should have run below the line instead above it. It seems that I was as well pretty pleased with the amount of features we did create and the value (meaning money) it can generate for our customers. last but not least I draw the great sprint demo we did for our customers the day before.So how do you like my picture (or doing this kind of retrospective at all)?

A Different Take on Sprint Retrospectives

Not taking JavaDoc seriously can easily happen to any developer. Or maybe you haven’t read that specific part of it, you should better have read. If you did and cannot see a problem locally, you might be tempted to ignore it.Here is an often ignored part from java.text.SimpleDateFormat JavaDoc:<div>

```plain
 * Date formats are not synchronized.
 * It is recommended to create separate format instances for each thread.
 * If multiple threads access a format concurrently, it must be synchronized
 * externally.
```

</div>Perhaps that warning isn’t strong enough, because it reads: “It is recommended”. This leads to this highly dangerous line of code I have seen in almost every project and is likely to produce hangs in concurrently accessed code.<div>

```java5
private static final SimpleDataFormat DATE_FORMAT = new SimpleDateFormat("yyyy.MM.dd");
```

</div>But it gets worse with System.gc(). There is no warning in the JavaDoc:<div>

```plain
* Calling the gc method suggests that the Java Virtual
* Machine expend effort toward recycling unused objects in order to
* make the memory they currently occupy available for quick reuse.
* When control returns from the method call, the Java Virtual
* Machine has made a best effort to reclaim space from all discarded
* objects.
```

</div>Doesn’t sound too bad, does it? You should wonder what “suggests” and “best effert” really means. Perhaps you are tempted to use System.gc(), so you might look on the internet for other people using this, hoping for an explanation what in reality happens. You might find <a href="http://stackoverflow.com/questions/2414105/why-is-it-a-bad-practice-to-call-system-gc">good insights on Stackoverflow</a> . But what exactly this does nobody seems to know.I think a picture tells more than thousand words: <img title="avoid_system_gc" src="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/kJJPaZ0jQXK8imfjXk0V" alt width="594" height="363" />We are looking at a pretty big system of one of our customers. It has 14GB of heap and we helped them to bring down the memory consumtion so that the app ran quite well. However there were still pretty long pause times (so called stop the world time). The graph was created from a Garbage Collection log and points us to the cause of the pause times. The blue line shows Garbage Collection activity. So why are there exactly 13 spikes with up to 35 seconds? All the other activity seems to be really fast. The reason for those 13 spikes is simple: They all were caused by an invocation of System.gc().So what happens? Calling System.gc() seems to effectively stop any optimization the GC is doing, forcing a full collection to start, rather than the optimized CMS Algorithm used.<blockquote>Note: This system is using a Sun 1.6.0.18 64 bit HotSpot Server VM. GC is set to -XX:+UseConcMarkSweepGC and -XX:+UseParNewGC. System.gc() does not have to produce results like that. Your results might vary. However also <a href="http://stackoverflow.com/questions/2414105/why-is-it-a-bad-practice-to-call-system-gc/2415234#2415234">others have reported</a> , that when using CMS and System.gc() a Full GC is always performed.</blockquote>If you cannot find yourself invoking System.gc(), they might hide deeply in a used library. To fix this, you can use the the server argument “-XX:+DisableExplicitGC”. This argument is also recommended by the SUN/Oracle performance tuning guide.

Invoking System.gc() can have serious impact

We at codecentric have hundreds of automatic builds run every day, sometimes they … fail. This post is not about lame excuses. “nah the build shouldn’t fail, that was a trivial change…” does not count. But there are situations where a build fails because … well nobody really knows.Some people say: cosmic rays! But we know that is not true. To efficiently utilize a CI system without the need to troubleshoot a long time here some common issues we encountered and ideas how to mitigate them. <ul><li>A test might do some time calculation and either in the test or in the code under test time is taken twice. While most of the time there is no difference, there might be a microsecond sometimes. A good indicator for this is a message like: Time was 23:30:00 but expected 23:30:00 (note microseconds are not shown)</li><li>Code, Testcode or Testrunner/CI code might leave files behind. Sometimes these are log files, sometimes files produced as test output. Take the time to search the whole server for all file writes made during a test run and take care to have a cleanup in place. Don’t forget to add disk space monitoring, because build machines have hard discs that tend to get full. (Hudson can do both)</li><li>Users logging into the CI system might lock resources, like files or ports, or do anything bad to the machine. You should not allow user logons.  All “analysis” should be made read only. Note that also read access or parallel tests can lock files.</li><li>It is not necessarily a bug when your code or tests do not run when the system date is 1th of January 1970. It could be, but you should make sure the system always uses the current time. Set up a ntp daemon. If you must use specific points in time for testing, you should be able to set a time source for all of your code, like a spring bean called TimeProvider which normally resorts to the system time.</li><li>If your tests need to apply evil hacks to test your software (which might be required. If not get rid of the hacks) it is often safer to let test execution fork, so tests cannot introduce side effects via the JVM (like setting System properties). Code coverage tools using bytecode manipulation count as hacks.</li><li>If you have multiple build machines ensure that they are as similar as possible. If you can afford it, you can set up a farm of build machines with defined differences, so you can spread testing on varying hardware in case this cannot be simulated. You do not want any surprise differences on which you spend hours to find out.</li><li>Consider setting up the system under test for integration tests nightly from scratch. Those tests tend to get messed up by exploratory testing and ad hoc demos. It is easier to automate such stuff than one would think, though it takes some time.</li><li>If you do automatic deployment, you need to at least stop the server, copy changed artifacts and then restart the server. Any kind of “hot deployment” is unfortunately just to fragile for reliable results.</li><li>After doing any change to configuration or infrastructure of your CI system trigger a build manually. If not done the next normal developer checkin will cause a failing build and will leave the dev wondering how that change could break this stuff.</li><li>If you find your tests hanging in your code, especially if multiple tests were run at the same time, take a heap and thread dump of the JVM before restarting the tests. You might be lucky that you found by accident a real concurrency issue inside your code. You should be grateful for that because you hardly can deliberately test this.</li></ul>Yes it is possible that the build breaks without any issues in your software, but it wastes a lot of time on investigation. Know the weaknesses of your system and try to fix them or at least document them.We have an issue with one of our integration test suites, which connects to an external service. Sometimes this just hangs and results in a connection timeout. Until a while ago this always “broke” the build. The result of this was that every engineer had a look at the build, the logs and eventually found out it was a timeout on the external system- We discussed this and decided to stop wasting time on investigating this. So we added a mechanic that in this special timeout case the build does not turn red. It stays green, but creates a tag “timeouted” on the test. This of course has a problem. A green build with “timouted” can have a problem with the computation of the extern call results. It might be really red. But we cannot know this. Real green build are not allowed to have “timeouted” tests. But the important part is that we want “real red” builds, which turn only red when there is a issue we can fix. In RobotFramework you can <a href="http://robotframework.org/robotframework/2.9b2/RobotFrameworkUserGuide.html">define a third state for “noncritical failures”</a> . Decide yourself if this is something for you.But the most important takeaway is: If a broken build is not caused by test or production code, you must find the reason and address it. You cannot say: “cosmic rays” because that will lead everybody to say “broken build – cosmic rays” and you will have much less successful builds because eventually no one will care. Red should always mean: Team take action!

Top Excuses Why Automatic Builds Suddenly Fail

I admit, this post is a bit “off-topic”. Recently we migrated this blog from using qTranslate to WPML for publishing in German and English. Main reasons were much better updates and a cleaner separation. But one feature was missing because of that: We want to have comments from both languages below the postings (which are now two posts, where they have been one before). WPML doesn’t support this out of the box, but we were able to do this easily ourselves. Because we believe in Open Source, here the sourcecode of our comment-merging plugin. If required one could also make languages configurable, but we did not need that, and it would have introduced some more complexity.<a href="http://wordpress.org/extend/plugins/wpml-comment-merging/">WPML comment merging plugin on wordpress.org</a> <div>

```php
<?php
/*
Plugin Name: WPML comment merging
Plugin URI: http://wordpress.org/extend/plugins/wpml-comment-merging/
Description: This plugin merges comments from all translations of the posts and pages, so that they all are displayed on each other. Comments are internally still attached to the post or page they were made on.
Version: 1.3
Author: Fabian Lange
Author URI: http://blog.codecentric.de/en/2010/06/wordpress-wpml-comments-filter-plugin/
License: MIT
*/
 
function sort_merged_comments($a, $b) { 
	return $a->comment_ID - $b->comment_ID;
}
 
function merge_comments($comments, $post_ID) {
	// get all the languages for which this post exists
	$languages = icl_get_languages('skip_missing=1');
	$type = is_page($post_ID) ? 'page' : 'post';
	foreach($languages as $l) {
		// in $comments are already the comments from the current language
		if(!$l['active']) {
			$otherID = icl_object_id($post_ID, $type, false, $l['language_code']);
			$othercomments = get_comments( array('post_id' => $otherID, 'status' => 'approve', 'order' => 'ASC') );
			$comments = array_merge($comments, $othercomments);
		}
	}
	if ($languages) {
		// if we merged some comments in we need to reestablish an order
		usort($comments, 'sort_merged_comments');
	}
	return $comments;
}
 
function merge_comment_count($count, $post_ID) {
	// get all the languages for which this post exists
	$languages = icl_get_languages('skip_missing=1');
	$type = is_page($post_ID) ? 'page' : 'post';
 
	foreach($languages as $l) {
		// in $count is already the count from the current language
		if(!$l['active']) {
			$otherID = icl_object_id($post_ID, $type, false, $l['language_code']);
			if($otherID) {
				// cannot use call_user_func due to php regressions
				if ($type == 'page') {
					$otherpost = get_page($otherID);
				} else {
					$otherpost = get_post($otherID);
				}
				if ($otherpost) {
					// increment comment count using translation post comment count.
					$count = $count + $otherpost->comment_count;
				}
			}
		}
	}
	return $count;
}
 
add_filter('comments_array', 'merge_comments', 100, 2);
add_filter('get_comments_number', 'merge_comment_count', 100, 2);
```

</div>

WordPress WPML Comments Filter Plugin

When configuring mule, you might find out that some of your services need to call external parties. Depending on your deployment scenario there needs to be a proxy server involved. However this might change for various test stages, some need, some don’t, some need them only for specific domains and like that. To reduce configuration overhead we have created an automatic proxy selector used for our services. Because I think its neat, I want to share it with you. <div>

```xml
<https:connector name="autoProxy">
 <spring:property name="proxyHostname">
 <spring:bean class="org.springframework.jndi.JndiObjectFactoryBean">
 <spring:property name="jndiName" value="/jndi/config/proxyHost" />
 <spring:property name="defaultObject" value="" />
 </spring:bean>
 </spring:property>
 <spring:property name="proxyPort">
 <spring:bean class="org.springframework.jndi.JndiObjectFactoryBean">
 <spring:property name="jndiName" value="/jndi/config/proxyPort" />
 <spring:property name="defaultObject" value="0" />
 </spring:bean>
 </spring:property>
 <service-overrides dispatcherFactory="de.codecentric.mule.HttpAutoProxyMessageDispatcherFactory" />
 <https:tls-key-store path="/mule.keystore" keyPassword="changeit" storePassword="changeit"/>
 <https:tls-server path="/mule.keystore" storePassword="changeit"/>
</https:connector>
```

</div>As you can see we configured a Connector which creates this MessageDispatcherFactory, it passes the configuration obtained from JNDI to it and also adds a keystore for ssl connections to it (with the great default password changeit :-)) Using it is then straightforward:<div>

```xml
 <outbound-endpoint address="https://external.service/endpoint" synchronous="true" connector-ref="autoProxy">
```

</div>The Factory itself is dead simple:<div>

```java5
public class HttpAutoProxyMessageDispatcherFactory
 extends AbstractMessageDispatcherFactory {
 public MessageDispatcher create(OutboundEndpoint endpoint) throws MuleException {
 return new HttpAutoProxyMessageDispatcher(endpoint);
 }
}
```

</div>The HttpAutoProxyMessageDispatcher implementation is easy was well. And contains a few hardcoded hosts that should not be proxied. Feel free to improve that part:<div>

```java5
public class HttpAutoProxyMessageDispatcher
 extends HttpClientMessageDispatcher {
 
 private final boolean hasProxy;
 private final String proxyHost;
 private final int proxyPort;
 
 public HttpAutoProxyMessageDispatcher(OutboundEndpoint endpoint) {
 super(endpoint);
 this.proxyHost = ((HttpConnector) endpoint.getConnector()).getProxyHostname();
 this.proxyPort = ((HttpConnector) endpoint.getConnector()).getProxyPort();
 this.hasProxy = StringUtils.isNotBlank(proxyHost);
 }
 
 @Override
 protected HostConfiguration getHostConfig(URI uri) throws URISyntaxException {
 String host = uri.getHost();
 HostConfiguration config = new HostConfiguration();
 config.setHost(host, uri.getPort(), Protocol.getProtocol(uri.getScheme().toLowerCase()));
 if (hasProxy && !isLocalhost(host)) {
 config.setProxy(proxyHost, proxyPort);
 }
 return config;
 }
 
 private boolean isLocalhost(String host) {
 return "localhost".equals(host) || "127.0.0.1".equals(host);
 }
 
}
```

</div>When you applied this pattern you only need to make sure JNDI information regarding the proxy is correct on each environment, and the mule configuration will automatically adapt to it.<h3>Update: Mule 3.2</h3>Recent Versions of Mule made it even easier to implement the HttpAutoProxyMessageDispatcher:<div>

```java5
public class HttpAutoProxyMessageDispatcher extends
 HttpClientMessageDispatcher implements MessageDispatcher {
 
 public HttpAutoProxyMessageDispatcher(OutboundEndpoint endpoint) {
 super(endpoint);
 }
 
 @Override
 protected HostConfiguration getHostConfig(URI uri) throws Exception {
 Protocol protocol = Protocol.getProtocol(uri.getScheme().toLowerCase());
 String host = uri.getHost();
 int port = uri.getPort();
 HostConfiguration config = new HostConfiguration();
 config.setHost(host, port, protocol);
 if (!isLocalhost(host) && StringUtils.isNotBlank(connector.getProxyHostname())) {
 config.setProxy(connector.getProxyHostname(), connector.getProxyPort());
 }
 return config;
 }
 
 private boolean isLocalhost(String host) {
 return "localhost".equals(host) || "127.0.0.1".equals(host); 
 }
}
```

</div>

Automatic Proxy Selection for Mule ESB Webservices

In this blog post, I want to show you how we at codecentric are using Mule ESB and Apache CXF in our projects to create webservices very easily, and what you can do to make them faster, because they are pretty slow out of the box. So why a webservice at all? This is a good question, and perhaps the most critical for performance. Webservices are good if you want to make your Interface or Service public, or want to use them internally where other transports (like RMI) are not available, either due to firewall configuration or cross programming language environments. People struggling with a good setup might often not be able to change this at all, so lets take it for granted. We use the Mule Enterprise Service Bus in some projects, but it might not be your product of choice. Documentation is only available when registering on their site, and their release policy is a bit unclear. I am not that happy with it, but it works pretty well when you found your hooks and patched a <a href="http://www.mulesoft.org/jira/browse/MULE-4095">few</a> <a href="http://www.mulesoft.org/jira/browse/MULE-4634">places</a> . To expose Webservices you can handcode them, use <a href="http://ws.apache.org/axis/">Apache Axis</a> or <a href="http://cxf.apache.org/">Apache CXF</a> . I prefer CXF because I find its API and generated code cleaner, also it is still alive and maintained by mule people and default for mule as well. For the service component we use plain spring pojos.So lets have a look at our Webservice Interface. To make it a little more interesting, we make a nontrival service, which takes and returns domain objects (but be aware of the http request size on large object trees)<div>

```java5
@WebService
public interface RecommendationService {
	@WebMethod
	public Products recommendProducts(
		@WebParam(name="user")
		User user,
		@WebParam(name="genre")
		Genre genre
	);
}
```

</div>There is an implementation for this service existing as well. The next step is then to wire it up in mule. First we need to configure mule to accept webservice calls. Because we start mule in an WAR file, we use the servlet connector, but you can use the jetty connector as well when running standalone:<div>

```xml
<servlet:connector name="servletConnector" 
 servletUrl="http://localhost:8080/mule/services/recommendation?wsdl" />
```

</div>Next up is the config of the service itself using cxf:<div>

```xml
<model name="recommendationServiceModel">
	<service name="recommendation">
		<inbound>
			<cxf:inbound-endpoint address="servlet://recommendation" synchronous="true" />
		</inbound>
		<component>
			<spring-object bean="RecommendationService" />
		</component>
	</service>
</model>
```

</div>And of course the service:<div>

```xml
<spring:bean id="RecommendationService"
 class="de.codecentric.RecommendationServiceImpl" />
```

</div>While you can mash all into one file, I recommend to split your mule and component configuration to multiple files, so that you do not get lost in the masses of xml. You could separate them by type (service, component, mule config) or by service. That is it already for the mule configuration part, so lets try to invoke it. Because we do not have an easy way to pass the domain objects yet, we just try to read the wsdl to veryify it is working.<div>

```bash
http://localhost:8080/mule/services/recommendation?wsdl
```

</div>Watch out for any s which tell you that the listing is not complete, but is available on a seperate url geven as attribute in the import.Generating a java client for accessing the service is very easy using the wsdl2java command from CXF:<div>

```bash
wsdl2java -client -d src/main/java -p de.codecentric.client 
 http://localhost:8080/mule/services/recommendation?wsdl
```

</div>If you would be an external party, you now could work with the stuff that has been generated. Internally however you most likely would prefer continue working with your domain objects User, Products and Genre. This will help you dealing with updates happening in your developmentcycle and provide domain methods you implemented on the model, but are not generated. Because CXF is really smart we can just delete the following generated classes:<ul><li>Genre</li><li>ObjectFactory</li><li>package-info</li><li>Products</li><li>User</li></ul>Fix the imports by using your domain objects instead and delete the @XmlSeeAlso({ObjectFactory.class}) reference.This should leave you with the interface and implementation of the service and two wrapper objects for request and response, and a dummy client. Running the dummy client (with CXF on the classpath) should now invoke the webservice.What it does behind the scenes when you are using<div>

```java5
RecommendationServiceImplService ss = new RecommendationServiceImplService(wsdlURL, SERVICE_NAME);
RecommendationService port = ss.getRecommendationServiceImplPort();
```

</div>is that it creates a dynamic proxy using reflection from the remote wsdl.We could stop here now. We have a dynamic Webservice client that uses the domain objects. All is fine, but the performance really sucks.The WSDL is read over the wire, and translated to the proxy class. We could add the WSDL locally, but that would require downloading it everytime the domain objects change. External clients of course need to do that, but we want to be less affected by the changes incremental development introduces. Also still the proxy class generation would be slow. We measured the wall time spent in the whole stack, and proxy generation by far outnumbered every other code.To improve this we create a pool, using commons pool GenericObjectPool.<div>

```java5
private final GenericObjectPool recommendationServicePool;
 
RecommendationServiceFactory recommendationServiceFactory = new RecommendationServiceFactory();
recommendationServicePool = new GenericObjectPool(recommendationServiceFactory, new Config());
```

</div>So the pool needs a factory to make instances and a configuration. The confguration can be tweaked, but defaults should be fine for now.The factory implementation is straightforward:<div>

```java5
public class RecommendationServiceFactory implements PoolableObjectFactory {
public Object makeObject() throws Exception {
 RecommendationServiceImplService service = new RecommendationServiceImplService();
 RecommendationService port = service.getRecommendationServiceImplPort();
 return port;
}
public boolean validateObject(Object arg0) {
 // consider all controllers valid objects
 return true;
}
public void destroyObject(Object arg0) throws Exception {}
public void activateObject(Object arg0) throws Exception {}
public void passivateObject(Object arg0) throws Exception {}
}
```

</div>Now we can invoke our service like that:<div>

```java5
RecommendationService port = (RecommendationService) recommendationServicePool.borrowObject();
try {
 Products products = port.recommendProducts(user, genre);
} finally {
 recommendationServicePool.returnObject(port);
}
```

</div>Please do not forget to return the Service you have borrowed.Wrap up We used MuleESB to configure and deploy a Spring component based Webservice which uses Domain objects. We exposed the service using Apache CXF and used it to generate a client as well. Afterwards we tweaked the generated client to use our domain objects instead of generated clients. Then we introduced an object pool to avoid creating the proxy classes over and over again.perhaps you want to know if there was a real performance gain. I strongly advise to profile it yourself. The simplest method is to measure milliseconds orund our last codeblock, executing it more than once. The very first invocation of the whole stack took 360ms on my machine. All subsequent calls were down to 4-6ms. This is a 100x improvement. Remember, that this time includes a lot: Calling over HTTP (even on localhost) a ESB running inside a WAR on JBoss, finding the correct service endpoint based on the URL, invoking an instance of that Service, and all the way back.Do not judge premature. Web Services can be pretty fast. And thanks to a lot of frameworks, also easy to setup.

Simple and Fast Webservices with Mule ESB and Apache CXF

Last week I had the pleasure attended Heinz Kabutz <a href="http://www.javaspecialists.eu/courses/master.jsp">Java Specialists Master course</a> to sharpen my Java skills. Java Champion Heinz, is a great trainer who manages to combine anecdotes, hard facts and deep Java knowledge with engaging exercises to a well done course. The scope was the whole spectrum of Java, but focusing on the details you normally do not use, or know how to use. Some of the material he already published as part of <a href="http://www.javaspecialists.eu/archive/archive.jsp">his newsletters</a> , which are read all around the world.Let me share my impressions on the course with you in this day by day review… Day 1The course started with discussing Threads, and how we should use them. Quite a complex topic for early morning. We played with a ThreadGroup that became a custom Thread pool. ThreadGroup is not the best designed class. Heinz called it a child’s drawing from the early years of Java. I found using the <a href="http://java.sun.com/j2se/1.5.0/docs/api/java/util/concurrent/locks/package-summary.html">java.util.concurrent Locks</a> really easy. Finally gone are the days of synchronized(). Before the lunch break, Heinz showed us his laws on concurrency, which he also presented in an abbreviated form at our meet the experts – performance event. We came across this code:<div>

```java5
private boolean running = true;
public void dojob() {
 while(running) {
 // do something useful
 }
}
public void shutdown() {
 running = false;
}
```

</div>When running in a Server VM (with java -server), this might never stop, due to optimizing running would be inlined by HotSpot with being true all the time. To avoid this you would have to make it volatile. Because I asked about debugging, we tried stopping there and the debugger showed us: running = false. Still the code continued to executed, because the debugger sees the correct field value, but the running code doesn’t. It gets more interesting with this code:<div>

```java5
public void doJob() {
 boolean myRunning = running;
 while(running){
 // do something useful
 myRunning = running;
 }
}
```

</div>When looking with the debugger we saw this:<div>

```java5
running = false; myrunning = true;
```

</div>however the loop still looped. But when forcing the line to execute via F7, code terminated. This can be a nightmare to debug, so it is good to know what you should take care of when writing multithreaded programs.Also something to remember is to check<div>

```java5
if (Thread.interrupted()) {
 throw new InterruptedException()
}
```

</div>as first code in all methods declaring an InterruptedException.We learned about <a href="http://java.sun.com/j2se/1.5.0/docs/api/java/util/concurrent/CompletionService.html">CompletionService</a> looks like an interesting interface for mass processing of asynchronous work. So, who needs closures? 🙂Day 2We started with Java new (yet another new?) IO, which brings quite a lot of new features, but somehow is not as widely used as it should be. One reason might be that it can easily get much more complicated to use. Perhaps one should try easy examples before writing an Async nonblocking server (which you can after attending the course :-)).<div>

```java5
FileChannel fc = new RandomAccessFile("test.txt", "r").getChannel();
MappedByteBuffer buffer = fc.map(FileChannel.MapMode.READ_ONLY, 0, fc.size());
```

</div>Second half was on understanding Java memory management. This of course is less often really applicable, but understanding it is quite critical to solving problems. We had a look at stuff like caching and pooling, and why this creates leaks or loitering objects. This reminds me of my older <a href="/wissens-hub/blog/jsp-tag-pooling-memory-leaks">post about tag pooling</a> in servlet containers. I never understood really why tags are pooled, and even worse, do not have proper lifecycle methods to detect this when using an IDE. We used jVisualVm and <a href="http://www.javaperformancetuning.com/tools/hpjmeter/index.shtml">HPjMeter</a> to watch the GC at work.Day 3On day 3, I learned some interesting inner mechanics of Collection classes I did not use before (like <a href="http://java.sun.com/j2se/1.5.0/docs/api/java/util/PriorityQueue.html">PriorityQueue</a> ), as well as some nasty tricks on classloading. Heinz explained java.lang.reflect.Proxies really well, and once understood, using them was not that difficult. Actually the best instruction is in the JavaDoc, but you must know how to read it:<div>

```java5
Foo f = (Foo) Proxy.newProxyInstance(
		Foo.class.getClassLoader(), new Class[] { Foo.class },
		new InvocationHandler() {
		 public Object invoke(Object foo, Method method, Object[] arguments) throws Throwable {
		 return method.invoke(foo, arguments);
		 }
		});
```

</div>In the afternoon we discussed about Exceptions, and I made up my mind on checked vs unchecked exceptions. Personally I will use unchecked exceptions for developer/programming errors. Catching them is not required, app may crash – Developers should fix this. However everything which is related to the Environment the app runs ins should work with checked exceptions. Ideally they provide sensible information, not just a message. Also quite important: Just rethrow Exceptions! Found yourself not able to decide what to do with an InterruptedException? Well just rethrow it 🙂 And handle it in the Thread code (calling interrupted() and exiting a loop). I never did that often, because I don’t like polluting my method signature, but it should be considered. Do not be afraid of retrowing Exceptions.Day 4The last day of the course started with a tough performance optimization exercise. The tough part was that we were not allowed to improve the code until we had all the numbers written down and eliminated any overhead of testing code. I especially liked this, because it thought me how eager I am sometimes fixing issues that I forget proving them first. As kind of side note, we discussed the various modes the JVM can run in and found out how slow java -Xint is. After having sped up the code down to 10% of its initial runtime, we moved on to Date and Time, which was a bit short chapter. I can recommend using <a href="http://joda-time.sourceforge.net/">jodatime</a> and <a href="http://site.icu-project.org/">icu4j</a> , and try to stay away from java.util.Date. Before the end of the course we covered logging including some nifty tricks. The most important lesson about logging is that you need to use code guards (which was not new to me, but I like the term, I never heard before):<div>

```java5
if (log.isDebugEnabled()){
 log.debug(complexObject.toString() + expensive.toString());
}
```

</div>Wrap-UpI can wholeheartedly recommend this course. 4 days packed with lots of information and exercises that are well done so that they are a challenge for every participant. You should have worked with Java already some time. This is definitely not a beginners course. You will rush past topics which you can only grasp when you have experienced the problem before. I can also recommend taking this training in German, because Heinz has a really funny accent 🙂

Java Specialist Master Course Field Report

<a href="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/pks3D8VdTjq13giXYGsL"><img title="Dr_codecentric_und_seine_kranken_Pfleger" src="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/pks3D8VdTjq13giXYGsL" alt width="297" height="183" /></a> “Dr. codecentric und seine kranken Pfleger”, (codecentric, M.D. and his sick attendants) the codecentric board game team, Andreas Ebbert-Karroum, Torsten Rodemann, Marc Clemens and Fabian Lange (left to right) competed in Dinslakenhighly motivated for the qualification for the national board game championship this Saturday. After having been quite successful last year at our first attempt, our expectations were high, despite the lack of proper training.Every player had to play one round of each of the games: <a href="http://www.amazon.de/gp/product/B0010B5472?ie=UTF8&tag=exfuror&linkCode=as2&camp=1638&creative=19454&creativeASIN=B0010B5472">Agricola</a> , a pretty complex strategy game, <a href="http://www.amazon.de/gp/product/B0014LETVU?ie=UTF8&tag=exfuror&linkCode=as2&camp=1638&creative=19454&creativeASIN=B0014LETVU">Stoneage</a> , a round based resource gathering game, <a href="http://www.amazon.de/gp/product/B001EWE4EG?ie=UTF8&tag=exfuror&linkCode=as2&camp=1638&creative=19454&creativeASIN=B001EWE4EG">Dominion</a> , an interesting card game, in which players compile a custom card deck according to the available cards and their strategy, and <a href="http://www.amazon.de/gp/product/B0007M18XY?ie=UTF8&tag=exfuror&linkCode=as2&camp=1638&creative=19454&creativeASIN=B0007M18XY">Heckmeck am Bratwurmeck</a> , a pretty simple dice-rolling game, which trains you calculating probabilities. <ul><li> First round of the Tournament, sporting Dominion went really well for our Team. 2 first places and a second one were great. Only one game was lost due to the high amount of curses played by witches.</li><li>Agricola strained our nerves for over two hours. It was pretty intense, and we were unable to oppose our co-players, who were well prepared. We tried both, the farmer and rancher strategy, but only achieved a second, 2 third and a fourth place.</li><li>After a short snack-break, we had some relaxing dice rolling activities with Heckmeck am Bratwurmeck. But it seems gambling is not our profession. 3 times a fourth place was really sub-par. The other third place could not compensate those losses.</li><li>Last game of the day was Stoneage. 3 of us tried the “starvation” strategy, which was working fairly well, but only brought us 2 third and a fourth place. The one victory was done with some more mainstream farming and building strategy. Overheard discussions tought us that most players already have overcome “starvation” and came up with something better.</li></ul>Despite the not very successful results, we had a fun day! And lots of motivation for the coming year. Perhaps we should then run a board game training bootcamp? 🙂

codecentric playing at german board game championship

I will be attending the <a href="/wissens-hub/webinare">course by Dr. Heinz Kabutz</a> next week from 2nd-5th of March in Düsseldorf.I have pretty high expectations, as I already read his newsletter since quite some time and also my impressions of Heinz, when he gave his talk at our meet the experts, were very positive. He really knows all the nasty details about Java, and I hope I can get some of those out of his and into my brain. <a href="http://www.javaspecialists.eu/courses/master.jsp">The course promises “Extreme Java”</a> . I am looking forward to very esotheric subtleties without any practical value. And of course to many caveates that you indeed would find in everyday work.I am also looking forward to meeting other Java geeks to exchange knowledge and have a good week. Will you be there?(note that the course is in German)

Looking forward to the Java Specialist Master Course

Automated builds and tests already have a long tradition at codecentric, but we never managed to put up build radiators in our new offices. Till today. Developers could have looked up the status in the past, but getting it pushed to you while enjoying a hot coffee or tea helps. This is because when you forget checking for your build status it is most likely red.<a href="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/nJ4Ym0qBTgSnj6nflhjr"><img src="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/nJ4Ym0qBTgSnj6nflhjr" alt title="coofeeandbuilds" width="468" height="276" /></a>

Hot Coffee and Green Builds

<img src="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/8naxZy4tTieIirH3JBS9" alt="scrumday düsseldorf" title="scrumday düsseldorf" width="200" height="150" /><a href="http://www.scrum-day.de/">scrumdays</a> are local smaller sized events where SCRUM communities can meet and exchange their experience. As we are also very active in the agile community, codecentric of course had also a booth at todays scrumday in Düsseldorf. From the talks I was able to pick up two trending topics, which are quite interesting. They are not dealing with what scrum is and why it is great. They were talking about adaptions to the process to make it fit for large and small companies, which is a bit worrying, because I think the idea is “plan do check act” or “apply, inspect, adapt”. So please try to do SCRUM by the book for at least 3 iterations (the standard length <a href="http://borisgloger.com/">Boris Gloger</a> recommends 🙂 ) before adapting the process. I know its hard. But you really should try this to get the full effect. The presentations should not give the impression that you have to do scrum wrong to get it working correctly. I think that should have been made clearer.The second topic is much more interesting: Ok, we have Scrum, but how does this affect our organization? Do we need different management levels? Do we need different payment models? What motivates people to work in a SCRUM environment? My colleague Andreas talked about the science behind motivation and why SCRUM is really good at motivating people. I can really recommend his talk, so I included it below from slideshare. Also <a href="http://chcrudy.wordpress.com/">Christiane Philipps</a> gave some interesting insights. I think these ideas need to be explored further, because SCRUM reveals people issues which traditional models try to suppress.However the event as such was a bit disappointing. It felt like the main audience were the speakers presenting each other and to the exhibitors. Not that many “regular guests”. Time between sessions was spot on, but the sessions itself felt a bit short with only 45 minutes. I think 60 would have been better. I personally liked more our meet the experts instead, even though (or perhaps because?) there is just one track, excellent speakers and a much more familiar atmosphere in our premises rather than the anonymous hotel lobby, which is not that suited for having interesting discussions.As promised here Andreas slides. All credits for them please attribute to him 🙂<div><a href="http://www.slideshare.net/AndreasEK/macht-scrum-zufriedener">Macht Scrum zufriedener?</a> <div>View more <a href="http://www.slideshare.net/">presentations</a> from <a href="http://www.slideshare.net/AndreasEK">Andreas Ebbert-Karroum</a> .</div></div>

#scrumday in düsseldorf

<img src="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/lOrJSdeeQkSVnqqTAoi4" alt="mte-architektur-jar-openspace" title="mte-architektur-jar-openspace" width="200" height="150" />This post shall sum up the results from our fruitful discussion on friday evenig. The idea for the open space discussion was sparked by Stefan Zörner who talked about modularity and what happens when you have no control over modularity. This post will not try to repeat the discussion but merely conserve the results: <ul><li>OSGi is a great module concept that is missing in pure Java. It is somewhat sad that the average enterprise developer will not be able to use it for a while. The main problem seems to be the migration and interoperability of existing non OSGi code, and the changed runtime environment, which is new for the standard operations department.</li><li>You shall make many small modules for your own software. There is a benefit in having them although it cost slightly more work to manage them (for example in eclipse).</li><li>It is mandatory to have somebody watching over external dependencies.</li><li>Many third party dependencies cause various troubles: Classloading, legal, filesize, deployment time, runtime memory</li><li>Maven or Ivy do not solve the problem but they help a lot</li><li>Evaluate the usage of third party code on a regular basis. Consider removing dependencies that are only used for very few code</li></ul>Thanks to all attendees. Over twenty people joining the discussion shows that there is a need for good ideas.

Meet The Experts Architecture – Open Space: Managing the JAR Chaos

After upgrading Java to version 1.6.0.17 our CXF based webservices running on Mule ESB did not work anymore. <div>

```java5
Caused by: javax.xml.stream.XMLStreamException: java.io.UnsupportedEncodingException: "utf-8"
	at com.ctc.wstx.stax.WstxOutputFactory.createSW(WstxOutputFactory.java:257)
	at com.ctc.wstx.stax.WstxOutputFactory.createXMLStreamWriter(WstxOutputFactory.java:124)
	at org.apache.cxf.interceptor.StaxOutInterceptor.handleMessage(StaxOutInterceptor.java:67)
	... 25 more
Caused by: java.io.UnsupportedEncodingException: "utf-8"
	at sun.nio.cs.StreamEncoder.forOutputStreamWriter(StreamEncoder.java:42)
	at java.io.OutputStreamWriter.<init>(OutputStreamWriter.java:83)
	at com.ctc.wstx.stax.WstxOutputFactory.createSW(WstxOutputFactory.java:253)
	... 27 more
```

</div>That looks strange, because utf-8 should be a supported encoding. But a closer look reveals that the encoding passed contains the quotes. And that does not work. According to the specification quotes are allowed around the charset, but that is not very common. The quotes were introduced in JAX-WS 2.1.2.luckily the problem is easy to patch when you know this:<div>

```java5
if (encoding != null && encoding.startsWith("\"") && encoding.endsWith("\"")) {
 encoding = encoding.substring(1, encoding.length() - 1);
}
```

</div>insert this code at two places:<div>

```java5
org.apache.cxf.interceptor.StaxInInterceptor#handleMessage()
after
String encoding = (String)message.get(Message.ENCODING);
 
and
org.apache.cxf.interceptor.StaxInInterceptor#getEncoding()
before
return encoding;
```

</div>The patched class needs to be on the classpath before the original one. When doing so, the webservices worked again. A small unit test verifies this behavior and ensures that after a potential Mule/CXF upgrade the quoted encodings work.

Mule+CXF struggle with quoted encodings

A hot topic here at the Devoxx were the upcoming Java editions with their features and changes in the language syntax. While it is nice that you will be able to switch() on Strings, have a modularized platform and other cool stuff, one thing is a bit odd: the hype <a href="http://en.wikipedia.org/wiki/Closure">Closures</a> get. Java 7 will be delayed mainly by this. But what is the hype? <blockquote>Closures are tricky to explain. I am not going to try it. But it is basically a piece of code that gets its own life. Closures can reference state that is no longer existing outside their scope.</blockquote>Closures are neat. no doubt. I like programming with them in JavaScript. And I was pretty much attracted by <a href="http://clojure.org">Clojure</a> when Howard Lewis Ship demoed it yesterday in his session. When talking about languages like Clojure it is crystal clear that this is a new language; it comes with its very own concepts you need to understand before you can develop. Knowing how to develop in OO Java does not help for developing in Clojure, even though it has interop with Java. I do wonder where Java is heading with adding closures. The average Java developer does not understand it, or even when he does, has no need for it. You need a different mental model to successfully implement this.This new model is <a href="http://en.wikipedia.org/wiki/MapReduce">map&reduce</a> . Well, ok, map and reduce are not the mental model behind Clojure, but understanding it allows developing in Java and using closures efficiently. Map&reduce was almost in every session, and it seems to be the answer to the scaling question. It simplifies a lot, because it only works when you simplify. To quote Gregor Hohpe: “if we would have rebuilt a relational database, just to make it faster, we would have failed”. This is so true. The inherent complexity of an relational model / database does not get any faster. If it would be possible, Oracle certainly would have done this already.So we have to simplify, but how? Treat data as key value pairs, like in the good old Cobol times. Dump relations, what makes them important? They just cause troubles.When you have made it to this with you mental transition you are ready to go with any language. With or without closures. you can do map and reduce and paralleliz as you like. The link between those is that you accept to handle one item at a time, you don’t have global state or side effects, and you invoke your closures as workers on data which return results.Gregor showed a few patterns they apply at Google. I especially liked Sawzall, which seems not to be open sourced by Google yet. It is a nice descriptive and concise calculation instruction per data item. Its actually pretty much a single closure:

```
errorcount table sum of int
--------
if contains input 'error'
 emit errorcount <- 1
```

It is basically a call to think outside the box. We cannot solve problems without, because we cannot improve existing solutions anymore. They are highly optimized. We need something new to solve our challenges.

#devoxx 09: map&reduce and closures

As every year, codecentric Developers are attending the Devoxx. Devoxx in Antwerp is among the top conferences for Java in Europe, known for its hand picked Speakers and excellent topics. No surprise that the depicted 7 gents in codecentric Shirts did not want to miss this conference.<img title="codecentric_at_devoxx" src="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/KDHfQTy8Q6OvKuiBkgJh" alt="codecentric_at_devoxx" width="548" height="411" /> Already during the fist keynote, <a href="http://twitter.com/#search?q=%23devoxx">#devoxx</a> grew to a trending topic on twitter. Partially due to the still working wifi, which is withstanding the enormous activity quite well.Antwerp and the Devoxx crew offer a great sideshow as well. We have our traditional steak dinner, Belgium beer and the sponsored <a href="http://www.imdb.com/title/tt1190080/">2012</a> movie here at the Metropolis cinema, where also all the sessions and the exhibition take place.I am pretty sure we will have one or another post on interesting sessions or topics as well.

codecentric Crew visiting #Devoxx 2009

Having a presentation slot at a Java User Group is always special. Its an audience who cares, or is there just for the buffet. No kidding, todays evening was great. Besides my talk on RAP for which i attach the slides, there were new insights on what purpose programming languages were made for: solving problems. Not problems we introduced with programming languages, but other problems like calculations. Andreas Schreiber showed that the scientists at DLR use many languages (which suit their use cases best) and how to integrate them with Java. Dr. Simon Wiest showed afterwards how you can utilize bear lamps (or lava lamps) to monitor your CI server and what can be done to speed builds up. The evening ended with discussions how we can utilize our Nabaztags for that better than just rotating the ears 🙂<div><div><a href="http://www.slideshare.net/fabianlange/eclipse-rich-ajax-platform">Eclipse Rich Ajax Platform</a> <div>View more <a href="http://www.slideshare.net/">documents</a> from <a href="http://www.slideshare.net/fabianlange">Fabian Lange</a> .</div></div></div>

JUG Cologne – 5th October – Slides on Eclipse RAP

Andreas pointed me at a nice library, <a href="http://projectlombok.org/">lombok</a> . It enhances Java compilation so that classes need less clutter to work.<a href="http://projectlombok.org/"><img title="Project Lombok" src="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/h1kIeAkXSCqyU8GRXcLq" alt="Project Lombok" width="267" height="233" /></a> For me, the features are awesome. Just by adding @Data to a class, it will generate all the getters and setters, toString() and the hashCode() and equals() methods. This is pretty much as in groovy the magic accessors. I like that. Because I can focus on the important stuff, rather than scrolling through hundreds of clutter lines. I also like that nobody plays around with my getters and setters and introduces side effects. It also fits to agile process: Eliminate <a href="http://en.wikipedia.org/wiki/Muda_%28Japanese_term%29">muda</a> .I really like @SneakyThrows because i hate the UnsupportedEncodingException whenever if specify “UTF-8” which really never ever can happen (as long as the parameter passed into is a constant, not a dynamic param). I also like how this is implemented: lombok hooks into the Java compilation and just generates code for the appropriate annotations. The Eclipse plugin takes care that navigation through code does not suffer.It is just cool and for sure will grow to include more useful featues.But the downside of this is that we introduce (more) magic. Magic, which is hard to predict. I feel that the learning curve for the average developer is too steep, so in the end, the gain, which is actually just a bit cleaner code, is not really paid off when there is confusion around.Also think of maintenance: You now would use lombok, but in 2 years it is no longer cool and the project is dead (which I don’t hope; just assume). When now an additional year later somebody has to work with this code, the “clean” code strikes back, and the maintainer wonders where the getters and setters are.<blockquote>Sidenote on Getters and Setters: I believe you actually do not need them, and most often just generate because Framework X requires them. I think Getters and Setters should not introduce side effects and just set the value (that is done by lombok as well). But why not just make the field public? There is no added value in having stupid Getters and Setters. You should have domain methods manipulating internal data. Thats it, no Lombok and no boilerplate code required at all 🙂</blockquote>It is a nice toy for the experienced programmer. And its powerful. I certainly will use it for private projects.What do you think about the “value / confusion” ratio? Is the ratio constant? Meaning valuable libraries cause always a linear amount of confusion?

Project Lombok – Cool, but too much Magic?

Back after having a great evening at todays <a href="http://www.rheinjug.de/">RheinJUG</a> talk held by Neal Ford. It was almost a perfect fit for our upcoming <a href="/wissens-hub/blog/meet-the-experts-architektur-27-november-2009">Meet the Experts – Architecture</a> . Because Neal has the <a href="http://github.com/nealford/presentations/tree/master/Emergent%20Design%20and%20Evolutionary%20Architecture%20(Neal%20Ford).pdf">slides on his github</a> , I just want to briefly summarize my personal takeaways. <ol><li>Neal confirms our credo here at codecentric: The code is the real deal. Code is design, code is architecture. Everything non-code (including comments) is not important because it is not part of the plan that the compiler converts into our product. Its also the idea behind executable specifications, <a href="/wissens-hub/blog/agiles-testen-das-herzstuck-agiler-softwareentwicklung">Thomas wrote about</a> a few days back. He also stated the importance of cleaning up code and fixing “broken windows”. Because code should get stronger over time… and not weaker.</li><li>Neal presented architecture as basically the same as design, but architecture is more persistent and unlikely to change, while design adopts to the business problem that changes more frequently.</li><li>He classifies two kinds of complexity. There is inherent complexity of a problem. This is actually the complexity we have to master to solve the problem. But we introduce accidental complexity by using or creating overly generic solutions. Accidental complexity can be fought by either preventing decisions made too early, and choosing simpler solutions first, because you can get more complex easily, but the way back is almost impossible.</li></ol>There are a lot more ideas in his talk, most of them I can agree to. And because Neals presentation skills are so advanced I could have listened to him much longer 🙂In case you did also attend, what did you take away from it?PS: Pac Man will never be anymore as it used to 🙂

Neal Ford at RheinJUG: Emergent Design & Evolutionary Architecture

One of the biggest problems in agile development teams is “effort”. Of course it is always about effort, because effort is money and we all like our money. In planning we can cope with effort quite easily: “oh that’s a week effort”, but when it comes down to “what to do today” we start struggling. And how can we complete the “one week effort” if we struggle with our daily work?My proposal for solving this issue is: At the end of the day, whatever you have done this day, if it is not complete, throw it away! How can I do such a statement? I believe that every single task can be done in a day. Many agile and iterative processes take care that tasks are small. But none does enforce that a task has to be finished in one day, but allow that tasks can run longer. The additional pain involved in throwing away work takes care that everybody gets more aware of what value they create a day. If a single task takes longer than a day this can have three main reasons:The task is way too bigThis is obvious, and sounds contradicting to the assumption that every task can be done in a day, but on closer look it is just about the correct definition of a task. Throw away what has been done and split the task. Note that whatever has been done is not wasted. Because it was not clear in the beginning that this task would take longer than a day, there has been too less knowledge about this task. At the end of the day there is enough knowledge gained to make smaller tasks. Be reminded: There is always a unit of work that can be done each day.Progressing with the task is blocked by othersDo not sit down and wait. Take another task. Resolving the impediment can be done earliest next day, and the solution will be available earliest in the day after. During that time two other tasks could have been finished. Ideally find out if there is a unit of work that you can finish before moving on.A simple task takes too longMy favorite. It is very much possible that the developer is heading into the wrong direction. Throw the futile coding away and restart the next day. Although managers might not believe in this, I see many developers nodding, knowing that when you are stuck its easier to restart. In most cases code quality and efficiency is also better.Is this wasting time? No it is learning! Learning to create manageable tasks. It also will lead to more accurate task definitions. Even when there will be a higher rate of “revert” after implementing this philosophy, it will rapidly decrease. Awareness of the actual tasks, the value they contribute and the quality of the result are easier to check. And the effort gets manageable, because the “oh a week of effort” could be identified as 3 days effort.There are also a lot more aspects on this. By either committing code every afternoon, or throwing it away it is made sure that the codebase for every developer is identical each morning. Merging code is easier, and incompatible changes are less likely. Also it is less likely that two or more people solve the same issue. Besides these technical advantages this also frees developers mind for a relaxed evening. And there is no need to remember all the details from the day before.Why not?The most common counter argument against this is, that it is impossible to define a whatever complicated task so that it can be done in a day. This cannot be true, because there has to be a starting point somewhere, and after sitting down 4 hours, something has been done. Even after 10 minutes already something has been done. A “unsplittable” 2 week task, can and has always to be split to 10 working days. The point is that not the average developer has to do this split in his head, and deliver after 2 weeks, but all involved people do this. And thus results are visible next day already. And it enables also less skilled developers to contribute resolving more complicated user stories.This article has been written on a single day, reviewed on another, translated on the third and published on the fourth day. At the end of each day something was done.

Commit every day, or revert – Be agile, every day

JSP custom tags were once widely used, but even still nowadays they find their way into projects. Not to mention the masses of production code using them. And almost all projects I have looked at using custom tags had the same issue. When writing JSP custom tags you have to remember the lifecycle model of a custom tag, because the container will typically pool tag instances. This is allowed and recommended by the specification, but can create lots of trouble when the tag is incorrectly written. Storing big objects in the tag instance will create a memory leak that can make your server go boom (or nothing happens because your tag pool is small and the instance is almost al time in use).Typically this sneaks by unnoticed in development environment. The code causing this problem looks typically like this one here:<div>

```java5
public class MyTag extends TagSupport {
 public Object leak;
 public int doStartTag(){
 leak = new BigObject();
 }
}
```

</div>This is a problem, because the lifecycle of a pooled tag looks like this:<ol><li>load class</li><li>create instance</li><li>invoke setPageContext()</li><li>invoke setters</li><li>call doStartTag</li><li>call possibly other tag methods depending on the tag type and return values</li><li>call doEndTag()</li><li>put instance in pool</li></ol>when the same tag is re-requested it may start at step 3. If in the above example this tag is pooled lets say with 10 instances and 10 simultaneous requests, 10 instances are created in the pool. But after that only a few requests come in. But still 10 instances of the tag are in the pool and contain reference to the BigObject. This is a memory leak.To avoid this always null out “transient” instance variables of the class and reload them either in setPageContext() or doStartTag(). Also note that the code inside the constructor might only be run once per tag, even when the tag is used on hundreds of pages. The amount of tag instances created (and thus also the amount of constructors invoked) depends on the container and pool settings and on the serverload.<div>

```java5
public class MyTag extends TagSupport {
 public Object noLeak;
 
 public void setPageContext(PageContext pc){
 noLeak = new BigObject();
 }
 
 public int doStartTag(){
 }
 
 public void doEndTag(){
 noLeak = null;
 }
}
```

</div>Other alternatives are to improve code design and make variables as local as possible. A relates problem is that you might find values in your tag that you don’t expect. Lets say a property is set when a certain parameter has a specific value. Now that condition was true, so the property holds an object that is not nulled out. The next usage of that tag will find the object even though the condition on the parameter is not set. This can really mess up your logic.

JSP Tag Pooling Memory Leaks

Because searching for “Convert InputStream to String” still brings up solutions involving StringBuffer, byte[] or something like that, developers still produce large amounts of different implementations of the same conversion in their projects. In an attempt to reduce the variations here the simplest solution using no custom code at all:<div>

```java5
 InputStream stream = getClass().getResourceAsStream("/classpath/resource.xml");
 String xml = org.apache.commons.io.IOUtils.toString(stream);
```

</div>Please spread the word, use apache commons 🙂

Convert InputStream to String

To prove that I can do short posts as well, here a quick guide to SVN in latest Eclipse release. This was not that easy in previous releases, but now it works like a charm:<ol><li>Help > Install New Software… > Galileo – http://download.eclipse.org/releases/galileo > Collaboration > Subversive SVN Team Provider (Incubation)</li><li>Help > Install New Software… > Add > Polarion – http://community.polarion.com/projects/subversive/download/eclipse/2.0/galileo-site/</li><li>> Subversive SVN Connectors > Subversive SVN Connectors</li><li>> Subversive SVN Connectors > JavaHL 1.6.0 Win32 Binaries (Optional)</li></ol>Dont get confused by “optional”. Step 1 gets us the SVN integration for Eclipse. Step 2 creates the Polarion update site. Step 3 installs the interface between Eclipse and the Polarion implementation, and step 4 chooses that implementation.

Eclipse Galileo and SVN

“Can I change this code?” sounds like a normal question, but in my opinion it expresses a problem in agile development that needs addressing.Foremost: This is a very good question, because it shows a noble intent: Make code you found better. Following the boyscout rule “always leave the campground cleaner than you found it” is among the best practices for coding. But the question indicates that there is something in the way of doing it. In Scrum you call this an impediment which calls for resolution, but also non-Scrum projects should take care of it. Also it should be said that solving this on a case by case basis doesn’t work. Effective development is only possible when this question is never asked again. Here some reasons I found most common for asking the question and possible solutions:<ul><li>The code in question is not accessible, or politically tabu ⇒ Convince your teams that common code ownership brings benefit for everybody and is not about undermining special assignments of teams. Most developers will be very careful when touching others code, so the risk is neglectable.</li><li>The code refactoring will break existing API ⇒ For many projects this problem is not as big as usually perceived. Usually the code is used internally only, and perhaps the new API will be much more pleasant to use. Even when this API is used externally, there is nothing wrong with changing API in a controlled manner.</li><li>The developer is uncertain of her skills ⇒ Our industry demands for continuous learning and actually many developers like working on stuff that’s new for them. Its okay to feel unsure, but new ideas could make old code better. Let the developer pair program with the original author or one of the more experienced folks on the team. If everything fails there should be tests…</li><li>Nonexistent tests make the change risky ⇒ Invest time in writing tests that check the current code behavior. If the change is intentionally changeing behavior, find out if thats okay, by talking to an architect, or by checking who uses the code you want to change.</li><li>The actual usage of the code is unknown ⇒ try to get a big (architecture) picture and as much as code from this or dependent projects as possible. Create a big refactoring setup for developers to use in case of this kind of code changes. IDEs can find many possible usages of code and CI servers can also run on refactoring branches.</li><li>There is no backup ⇒ Still running without version control? Do your homework and get one.</li></ul>Changing code is good, because we all want to make it better or nicer. It helps keeping solutions clean and simple. If you are not going to change the cause (a bug, design flaw or whatever), people start to introduce workarounds, with is not nice, not clean, not simple and just a waste of bytes.My call to developers is: “Don’t be afraid of changing other peoples code with good intend.”My call to project managers and SCRUM master is: “Make changing others code a pleasant experience”To answer the question in this postings title with a prevalent quote: “Yes, you can!“

Can I change this Code?

On Tuesday I attended Neal Fords session on <a href="http://en.wikipedia.org/wiki/Domain-specific_language">DSLs</a> and what they do for us. He showed that Java language capabilities for making or using DSLs are limited. But as we at codecentric have a lot Java, let’s have a deeper look to what extend it is possible. While some people already adopted the Builder pattern from Josh Bloch using a fluid interface, this has a problem, called the finishing problem. Because the <a href="http://www.javaspecialists.eu/archive/Issue163.html">Builder pattern</a> works as chained method calls, some stuff might not be initialized when required. Neal showed an example, which he took from <a href="http://www.jmock.org/" target="_blank">jMock</a> , how to solve this problem in Java. This impressed me, so I came up with my own idea, which in fact is already done by some GWT or Swing developers. My Java DSL Example<div>

```java5
Session ormPitfalls = new Session() {{
	speaker("Mirko");
	attending("Fabian");
	attending("a lot of other people");
	at("Rheingoldhalle");
	at(new JaxDate() {{
		day2();
		session2();
	}});
}};
```

</div>At the first glance this doesn’t look strange.. but… Whoa.. double curly braces?? The trick that is applied here, is that an anonymous class extending Session (and later JaxDate as well) is created. After the default constructor has run the initializer block runs. This initializer, written between that extra pair of curly braces, now contains method calls using the domain language of Session.Is this useful?Well, lets start that way: It has a problem. Each instance has its own class definition because they are anonymous classes. Which means slightly higher construction times, slightly higher memory usage and a different getClass().getName().<img title="Performance of Anonymous Classes" src="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/Q3cRyjzlRba01onweCCQ" alt="Performance of Anonymous Classes" width="445" height="139" />And I also think that we are not there yet: We don’t want business people to write production code in a Java DSL. Do we?But…… there is a great place for this: Testing! We all want to include business related people in testing, however they find it difficult to setup testcases due the amount of object bootstrapping required. But the above shown creation of that Session object is very easy to understand. Testers can use the domain methods, rather than, for example, fiddling around with creating multiple different Date Instances using the <a href="http://java.sun.com/j2se/1.5.0/docs/api/java/util/Calendar.html" target="_blank">Java Calendar API</a> . In fact this approach creates much better code than the old one. Currently the method being tested will calculate a date and the testcase will calculate it as well. But there is no proof that the testcase calculation is correct. Using the above code will solve this, because date calculation is implemented only once, so it is less error prone and also easier to understand.Side note on CalendarWhile rating at the Calendar; How do you currently set a date to, lets say, noon tomorrow? Like this?<div>

```java5
Calendar today = Calendar.getInstance();
today.setTime(calendar.getTime());
today.add(Calendar.DAY_OF_YEAR, 1);
today.set(Calendar.HOUR_OF_DAY, 12);
today.set(Calendar.MINUTE, 0);
today.set(Calendar.SECOND, 0);
today.set(Calendar.MILLISECOND, 0);
```

</div>This does not only look ugly, but also has incorrect semantics. Better semantics would be:<div>

```java5
Calendar calendar = GregorianCalendar.getInstance();
Calendar today = calendar.clone();
today.setTime(calendar.getTime());
Calendar tomorrow = today.clone();
tomorrow.add(Calendar.DAY_OF_YEAR, 1);
Calendar tomorrowNoon = tomorrow.clone();
tomorrowNoon.set(Calendar.HOUR_OF_DAY, 12);
tomorrowNoon.set(Calendar.MINUTE, 0);
tomorrowNoon.set(Calendar.SECOND, 0);
tomorrowNoon.set(Calendar.MILLISECOND, 0);
```

</div>uhh.. well at least it tries to give all states in between a correct name. If this would be all in a method called tomorrowNoon() on our CustomCalendar/Date object this would be much better (it works only on “this.“) and this kind of ugly code is hidden from the user.ConclusionDSLs are worth looking at, but for many projects this is not the time already. However a pseudo DSL notation can be a powerful tool for testing and creating less error prone code, but short and speaking one. As the side note shows, creating methods with a domain meaning does help always, so just do it.

Pragmatic Domain Specific Languages in Java

InnoDB has been the most popular <a href="http://mysql.com/">MySQL</a> Storage Engine as the MyISAM Engine was not usable for most use cases due to its limitations. MySQL AB was working on <a href="http://www.mysql.com/why-mysql/white-papers/">Falcon</a> , the successor of InnoDB and MyISAM, for quite some time already. After MySQL <a href="http://web.archive.org/web/20150218172446/http://www.mysql.com/news-and-events/sun-to-acquire-mysql.html">has been aquired by Sun</a> and the Falcon chief architect left the team it didn’t look too good for MySQL already. But after the announcement of <a href="http://www.sun.com/aboutsun/pr/2009-04/sunflash.20090420.1.xml">Oracle going to buy Sun</a> everything is open again. Personally I think its looking quite bad for MySQL, especially after InnoDB has been published as <a href="http://www.innodb.com/wp/products/embedded-innodb/">standalone database</a> . Looking at the whole product portfolio of Oracle in the database segment now is very confusing. While it has been discussed the last days that “MySQL” could survive as “small Oracle” I think that the situation is much clearer now: There is nothing that is on the upside for MySQL. Many “MySQL” installations already run exclusively with InnoDB. It just fits better into the strategic portfolio of Oracle, making it the ideal small business solution. There is no need for MySQL to be the small brother of the big “real Oracle”.But still doubts remain and we will have to sit and wait the future brings. But while sitting and waiting we can have a look at our existing installations and evaluate what requirements they fulfill and what features of MySQL we really use. Then we are prepared for whatever Oracle will decide.

The End of MySQL?

Today I noticed that the Apache Commons libs are lacking a method to camelize Strings.Because I needed to convert such an XML name MY_TINY_PROPERTY into a Java property name myTinyProperty, I ended up writing a simple camelizer myself.<div>

```java5
private String toCamelCase(String value, boolean startWithLowerCase) {
	String[] strings = StringUtils.split(value.toLowerCase(), "_");
	for (int i = startWithLowerCase ? 1 : 0; i < strings.length; i++){
		strings[i] = StringUtils.capitalize(strings[i]);
	}
	return StringUtils.join(strings);
}
```

</div>I guess this helpers covers most of the requirements. At least it is covering all that I need. As an alternative one could use the WordUtils#capitalizeFully() and postprocess the result.The way back is even more difficult, because the delimiters for split() are a bit harder to determine. While many other programming languages feature a camelize and underscore method, Java (and Apache Commons) does not. Why? Should I propose a patch?

StringUtils.camelize(String);

I would like to discuss an issue one can likely experience with agile development processes and systems which data needs to be maintened during upgrades:<blockquote>A customer care application for a PC retailer was developed so far and the software is running pretty well. Because warranty fraud is increasing, management decides to put in a check with the hardware supplier to check the validity of the serial number. But when checking the numbers against the webservice of the supplier almost all existing data gets rejected, because the serial numbers were not at all or incorrectlly entered so far. The supplier provides a validation algorithm which is added to the application. But after the next release of the software many employees complain about the system rejecting changes or updates of existing customer data. Also a third party financial system no longer is able to update the payment status.</blockquote>This scenario describes that existing data is very likely to cause trouble when adding further validation rules. One could limit the data validation to the user interface, which would help the financial system, but still call center agents would no longer be able to update the invalid data.Automatic data upgrade/conversion/fixing procedures are no promising candidates, because often the information is just missing and need to be obtained manually (e.g. from the customer).Here some ideas to limit the effect of this problem:<ol><li>Before adding new rules, generate a list of existing data which will be affected and let them be fixed on beforehand</li><li>Show rules for a given time only as warning, not as error</li><li>Use additional information to limit validation rules to new data only</li><li>When adding new data fields, think about at least a basic validation</li></ol>Have you made similar experience? Do you have any better ideas to fight this issues? Or even a procedure to avoid them?

Data Validation Alongside Agile Development

We at codecentric are not only leaders in Java performance, but support our clients on a variety of software development challenges.I posted in a previous entry about <a href="/wissens-hub/blog/vergleich-von-java-und-php-fur-webanwendungen" target="_blank">Comparison of PHP and Java</a> . I said that PHP mainly lacks appropriate tooling to develop debug and maintain. To be fair with PHP this posting will today describe how to profile PHP applications to tune performance. We will see how this tooling compares to the Java ones.For Java developers <a href="http://www.ej-technologies.com/products/jprofiler/overview.html" target="_blank">JProfiler</a> should be a well known tool for profiling and analyzing performance of Java applications. De equivalent in PHP is called Xdebug Xdebug + CacheGrind. <a href="http://xdebug.org/docs/profiler" target="_blank">Xdebug</a> is recording the calls to the PHP scripts, while <a href="http://sourceforge.net/projects/wincachegrind/">WinCacheGrind</a> is then used afterwards to evaluate the collected Data. In this example we are goung to profile a command line call of the <a href="http://www.symfony-project.org/about">symfony Framworks</a> . Because I am commiter for this framework, I use Xdebug from time to time to ensure a good nonfunctional quality.First you need to download the appropriate ‘.dll’ (or ‘.so’ for Linux) extension from the Xdebug website and put it to the other extensions into the ‘php/ext’ directory. To enable tat extension you need to add Xdebug to the ‘php.ini’ Xdebug and tell it where it should put the profiling results:<div>

```none
zend_extension_ts="d:\dev\php\ext\php_xdebug-2.0.3-5.2.5.dll"
xdebug.profiler_output_dir="c:\xdebug-profiling"
xdebug.profiler_enable=1
```

</div>Take care to edit the correct ‘php.ini’ file. There is one for the command line calls usually found in the PHP directory and another one in the Apache directory for the PHP processes spawned by Apache. Additionally you need to create the target directory first because Xdebug will not create it if it is not existing yet.Invoking ‘php -m’ from the command line should output something including:<div>

```none
[Zend Modules]
Xdebug
```

</div>Now we are going to profile a simple call. For example one to the symfony sandbox to create an application:<div>

```none
php symfony generate:app codecentric
```

</div>After this call has ended an 8MB ‘cachegrind.out.4896’ been created for us. It now can be loaded using WinCacheGrind, which looks like this:<a href="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/HQZHGQ6xR5a0JQX07kJC"><img title="WinCacheGrind loaded PHP profiling data" src="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/HQZHGQ6xR5a0JQX07kJC" alt width="300" height="239" /></a> Unfortunately WinCacheGrind has only limited functionality. <a href="http://sourceforge.net/projects/kcachegrind/" target="_blank">KCacheGrind</a> allows some more settings and graphical representation of the data, but also is not as complete as the Java tools we usually use. It also does not contain a memory view, which might be due to the fact that the PHP memory management is pretty messy. Quite useful though is is the timing information displayed.On the ‘Overall’ tab you can find code which took in sum the most time of your application. This is usually the code where you can make the easiest improvements. For the symfony task we can find here the following:<div>

```none
Function : sfFinder->search_in
Avg. Self : 0,2ms
Avg. Cum. : 5,1ms
Total Self: 127ms
Total Cum.: 3214ms
Calls : 629
```

</div>you can see that the search_in function takes an average of 0,2ms, but methods invoked by it add another 5,1ms in average. Because this function is called 629 times, this gives you the total time spent. A good candidate for checking if it really should be called that often, and if yes, if a single call can be made slightly faster.As you can see, also without <a href="/en/services" target="_blank">a lot of expertise</a> for performance issues, one can use above described tools to optimize PHP application performance. Perhaps in some time in future PHP and its tools will be as sophisticated as it is already today status quo in Java.

Profiling PHP Applications

From the 20th to 22nd of October the 6th Ajax World Conference took place in the sunny San Jose, CA. I was there those 3 days as delegate of codecentric to catch up with the newest trends and developments in Ajax and RIA.I tried to collect and write down all the impressions I got during the conference in the attached slideset. I really hope I managed to write everything down in a comprehensive way. In general the wholistic presentation on RIA, AJAX and web 2.0 was very interesting. Not only the technical talks, but also very enterprise business oriented sessions were a pleasure to attend. It really shaped my understanding of the state of the art today.I met many interesting and friendly people there, learned a lot about JavaScript and frameworks and also collected good arguments for implementing features in “new” technologies.In case of any questions on the conference, my slides or RIA in general, do not hesitate to contact me.

Ajax World Conference in San Jose, CA

Today I want to write about an issue that occurred to me with Struts2, but it is also valid for other scenarios. Ill first give an example:Elections in the States are coming close, so we are going to make a favorite poll.First implement a service that has three methods:<div>

```java5
public interface ElectionFavorite {
 public List getFavs(int userId);
 public List getChoices();
 public void setFavs(int userId, List);
}
```

</div>Looks pretty straightforward. Our choices are a list of top candidates, including Homer Simpson, Dilbert and Captain Kirk for possible presidents.To make our poll we write a small Struts2 based application and use the CheckboxList to display the choices. We use the two getters from the an EJB implementation of this interface to obtain the possible choices and the made selections in case the guest is changing its vote.The application works fine, we make our choice and save.<div>

```java5
NoClassDefFoundException: com.opensymphony.xwork.util.XWorkList not found
```

</div>How come? Well Struts2 does some conversion magic. It converts the returned selections from the page, which are HTML arrays, into this type, for whatever reason they think it is appropriate. But when we passing this List instance to the EJB that List implementation is most likely not on the classpath. Depending on your classloading setup when trying this, it might work, but when the EAR the EJB is in has no access to struts/xwork jars from the Web Application you will be getting this error.To fix this there are some options on the client side. One might be to implement a proxy that converts the List:<div>

```java5
setFavs(List favs){
 this.favs = new ArrayList();
 this.favs.addAll(favs);
}
```

</div>And now the general learning from this exercise: Using interfaces in APIs is great, because it allows the usage of different implementations. But when the chosen implementation is not found on the receiver’s classpath bad things can happen. There are three options to resolve this:<ol><li>Make sure that all client classes are available on the service classpath</li><li>Make a project convention that only JDK implementations are passed</li><li>Don’t use Interfaces in Service APIs</li></ol>Neither one is an easy pick, I must admit. Feel free to post better ideas into the comments.(Option 3 doesnt work in Struts2 due the magic involved when converting request parameters)

Struts2 Interface and EJB issues

Peter Roßbach, core developer at the Tomcat project, held a talk at <a href="https://www.google.de/url?sa=t&rct=j&q=&esrc=s&source=web&cd=1&cad=rja&uact=8&ved=0ahUKEwjerb-A79TPAhXGKsAKHYkXBlcQFggeMAA&url=http%3A%2F%2Fwww.rheinjug.de%2F&usg=AFQjCNEiK46P8FcgAdlpTiw7AqniYB3Ncg&sig2=F2ut-BAkZDQcyqsUkjBBgQ">RheinJUG in Düsseldorf</a> on Thursday. It was very delightful mixture on Tomcat architecture, best practices, Open Source community as well as current and future developments, rather than a dry technical talk. Tuning Tomcat PerformanceRegarding performance, he mentioned that the main negative factor is to use development mode for production environments. Well have a look at a few parameters and which do affect Tomcat performance.As of tomcat 6 there is the improved protocol handler available:<div>

```apache
<Connector port="8080" protocol="org.apache.coyote.http11.Http11NioProtocol"
 connectionTimeout="20000" />
```

</div>additionally Java NIO powers a new improved file download option. Servlets just have to set sendfile request parameters and not output any more data. That file will then be pushed by tomcat to the user:<div>

```text
 org.apache.tomcat.sendfile.filename: Canonical filename of the file which will be sent as a String
 org.apache.tomcat.sendfile.start: Start offset as a Long
 org.apache.tomcat.sendfile.end: End offset as a Long
```

</div>production servers should not deploy automatically:<div>

```xml
<Host name="localhost" autoDeploy="false" deployOnStartup="false" deployXML="false">
```

</div>The Jasper JSP compiler should be tuned as well:<div>

```xml
<servlet>
 <servlet-name>jsp</servlet-name>
 <servlet-class>org.apache.jasper.servlet.JspServlet</servlet-class>
 <init-param>
<param-name>development</param-name>
<param-value>false</param-value>
 </init-param>
 <init-param>
<param-name>genStringAsCharArray</param-name>
<param-value>true</param-value>
 </init-param>
 <init-param>
<param-name>trimSpaces</param-name>
<param-value>true</param-value>
 </init-param>
</servlet>
```

</div>Using mod_jk for loadbalancingAs a second main topic of his talk, he presented the module “mod_jk”, which in connection with a Apache Httpd and the Tomcat AJP protocol enables distributing java applications on multiple Tomcat servers.He explained the different loadbalancing setups and that it might make even sense to user a Tomcat server per application, as this allows better memory management and improves overall high-availibility.Tomcat Configuration:<div>

```xml
<Connector port="8009" protocol="AJP/1.3"/>
<Engine name="Catalina" defaultHost="localhost" jvmRoute="node01">
```

</div>mod_jk Configuration inside Httpd:<div>

```apache
<IfModule !mod_jk.c>
 LoadModule jk_module "modules/mod_jk.so"
</IfModule>
JkShmFile "logs/mod_jk.shm"
 
JKWorkerProperty worker.list=loadbalancer
JKWorkerProperty worker.node01.port=8009
JKWorkerProperty worker.node01.host=localhost
JKWorkerProperty worker.node01.type=ajp13
JKWorkerProperty worker.loadbalancer.type=lb
JKWorkerProperty worker.loadbalancer.connection_pool_minsize=0
JKWorkerProperty worker.loadbalancer.connect_timeout=30000
JKWorkerProperty worker.loadbalancer.prepost_timeout=10000
JKWorkerProperty worker.loadbalancer.balance_workers=node01
JKWorkerProperty worker.loadbalancer.method=Request
JKWorkerProperty worker.loadbalancer.retries=2
JKWorkerProperty worker.loadbalancer.recovery_options=7
JKMount /myapps* loadbalancer
```

</div>Further documentation on mod_jk worker can be found here:<a href="http://tomcat.apache.org/connectors-doc/reference/workers.html">http://tomcat.apache.org/connectors-doc/reference/workers.html</a> In total a very interesting talk, with lots of stuff to learn from, let it be the Tomcat project or the Tomcat architecture, or how Open Source “lives” and what one as developer or architect should consider when developing a Java web application.

Peter Roßbachs Tomcat 6 Talk at RheinJUG

Today <a href="https://www.codecentric.de/standorte">Christian</a> and I fought hard with a misterious connection problem. As a part of a longer process a web service call was failing from time to time. Actually it failed only under load. Due to that we did just debugging outputs, rather than breakpoints, as we were afraid of breaking some runtime behaviour by stopping the threads. But that was not so trivial as we would liked it to have: The webservice client is based upon the Apache Commons <a href="http://hc.apache.org/httpclient-3.x/index.html" target="_blank">HTTPclient</a> . The webserive itself was realized in Struts2 using its <a href="https://struts.apache.org/maven/struts2-core/apidocs/org/apache/struts2/dispatcher/mapper/Restful2ActionMapper.html" target="_blank">Restful2ActionMappers</a> . Both got several services injected via Spring.Initially we were after the unusual large size of the request and response, but after some analysis it seemed ok, however we still got flooded by variations of:<div>

```java5
java.io.IOException: CRLF expected at end of chunk: 72/84
java.io.IOException: Bad chunk size: somexml
```

</div>As we realized that this has to happen on the HTTP transport level, we had a look around and found following spring config for our bean:<div>

```xml
<bean name="restClient" class="de.codecentric.framework.RestClient">
 <property name="httpClient">
 <bean class="org.apache.commons.httpclient.HttpClient" />
 </property>
</bean>
```

</div>Doesn’t look bad, but by default the HttpClient has a ConnectionManager which will return always the same Connection instance. Even to different threads. Once you get to know this you will find immediately<div>

```java
org.apache.commons.httpclient.MultiThreadedHttpConnectionManager
```

</div>which unfortunately did not help much. In addition we now go an<div>

```java
java.io.IOException: connection closed
```

</div>What might suprise you is that <a href="http://hc.apache.org/httpclient-3.x/threading.html#MultiThreadedHttpConnectionManager" target="_blank">MultiThreadedHttpConnectionManager</a> is multithreading capable, but does only allow 2 connections per host. After we did increase that out application was running fine again. Even under load.The Spring config does look similar to this now:<div>

```xml
<bean name="restClient" class="de.codecentric.framework.RestClient">
 <property name="httpClient">
 <bean class="org.apache.commons.httpclient.HttpClient">
 <property name="httpConnectionManager">
 <bean class="org.apache.commons.httpclient.MultiThreadedHttpConnectionManager" destroy-method="shutdown">
 <property name="params">
 <bean class="org.apache.commons.httpclient.params.HttpConnectionManagerParams">
 <property name="defaultMaxConnectionsPerHost" value="20" />
 </bean>
 </property>
 </bean>
 </property>
 </bean>
 </property>
</bean>
```

</div>Leasson learnt: When multiple threads are using the same UrlConnection, it can go fine, but more likely, some very obscure things (exceptions) can (will) happen 🙂

Connection Sharing Sideeffects

No other language has been causing controversial discussions for a long time as PHP. The codecentric GmbH has specialized in Java, so we get some requests for migration of PHP applications.This involves often the question of whether Java is better than PHP, which is actually not the main problem. Both in Java as well as in PHP, there are frameworks, designed to create Web applications. Frameworks can of course offset drawbacks of language, but also deny benefits of languages.To understand the comparison of Java and PHP we must go back in time to about the year 2000. Java brought with Servlets and Struts first concepts for Web applications, but to create, configure and deploy them was very complicated. With the boom of the Internet, a new developer community grew, which quickly learned HTML. But pure HTML limits the possibility of interaction and CGI-Perl scripts were cumbersome and difficult. PHP, however, offered an elegant and simple way, if we wanted a date in a web page, we renamed “.html” to “.php” and inserted where wanted <?php echo date ()?> . On <a href="http://httpd.apache.org/" target="_blank">Apache Webserver</a> , which already was prepared for PHP, the new file worked out of the box.Although in Java Server Pages the ability to use scriptlets also did exist, this was frowned upon as unclean. Instead, the Java community advocated the use of components. In my opinion, a critical factor for the categorization of Java as “Enterprise”.For Internet applications a beautiful design is more important than a functional, as it attracts more customers. While HTML built in Dreamweaver or Front Page by the designer could be easily expandable with dynamic functionality by PHP developers, Java component oriented frameworks could not really work with it. PHP could enrich design with functionality. In Java, however, one had to beautify the functionality.But in recent years both sides improved. Java reduced the complexity, frameworks like <a href="http://tapestry.apache.org/" target="_blank">Tapestry</a> or <a href="http://code.google.com/webtoolkit/" target="_blank">GWT</a> , permitted by templates created by designers. PHP learned with version 5 useful object orientation and frameworks such as <a href="http://framework.zend.com/" target="_blank">Zend</a> or <a href="http://www.symfony-project.org/" target="_blank">symfony</a> brought design concepts to PHP developers. Also additional libraries of Java found correlation to PHP. For example the PHP ORMs Propel and Doctrine.From today’s point of view also offer Java and PHP similar functionality. Nevertheless, other aspects are to consider:<ul><li>Stability PHP has in my opinion, significant weaknesses. The procedural backward compatibility, no real deprecation mechanism, a mess semi platform independent libraries and functionality are just some of the issues the PHP. PHP lacks a clean cut, which the PHP planned to do with version 6. Java, however, has a clean platform independence and a fairly well-defined number of core libraries with appropriate quality standards.</li><li>Performance While Java was formerly often described as slow, today’s JVMs are highly optimized for speed, while the script languages, including PHP, still struggle with this. For example a first usable garbage collector will be shipped with PHP 5.3. Also other optimizations were moving very slowly into PHP Runtimes. This might be due to the fact that PHP in contrast to Java restarts the VM after each request, which of course brings additional performance problems. For example for each request session data has to be read from disk. Although there are solutions in PHP (MemCache, APC) these are rarely and partly still heavily in development. Interestingly, this drawback makes scaling of PHP applications fairly simple. As completely separate requests can be processed, additional hardware results in relatively linear improvements in the capacity of the server. On the Web, the focus is rather on the number of requests, not directly on the exact duration of an individual requests.</li><li>Choice Ideally, you never will re-invent the wheel. It makes sense to look for already existing solutions. Both in PHP, as well as in Java, there is a lot of modular software, partly with free, partly with non-free licenses. However, PHP modules expose significantly more problems than those written in Java. For example, some PHP module developers invented own concepts (e.g. Zend Loader was created by Zend as a substitute for packages) or the modules are only optimized for a framework (like symfony plug-ins). Java is, especially through the “complicated concepts” such as Class Loading and packages, better prepared for modularization. Due to better tool support (Ant / Maven, Javadoc, JUnit) Java Frameworks have easier to install, better documented and tested artifacts. However PHP tools for these tasks are also on the rise (pake / phing, PHPDocumentor, PHPUnit / lime).</li><li>Integration Integration is certainly the strength of Java. On the one hand, Java itself is almost “Industry Standard”, on the other hand, there are many standards implementations in Java. If a PHP Web application should communicate with a specific protocol, the selection of libraries is rather limited. Even worse, implementations are either only partially implemented or very rudimentary (such as Zend OpenID). Integration of PHP applications with other services usually happens through the database layer.</li><li>Developer know-how Even 20 years ago, Frederic Brooks searched for the “Silver Bullet” and did not find it. In his article he came to the conclusion that software design, problem formulation and the capabilities of the developers are far more important than tools or languages. Therefore, it is certainly a good idea to implement a website by a designer with knowledge of PHP with a state of the art PHP Framework. If it would be a Web front-end of a Java EE backend application Java would be the obvious choice.</li></ul>However developer knowledge should not hide the fact that some technological hurdles can be overcome only with certain technologies or other languages. So consulting an expert with specific skills makes much more sense than to give it a try with inadequate tools.So I conclude that Java is still the better choice for many projects. For smaller projects that can be isolated scripting languages might reach the target faster. As a compromise you might give Groovy Grails a try?

Comparison of Java and PHP for Web Applications

My talk about profilers and their inner workings from JavaOne is now viewable for free in its entire length on Parleys. Watch it and let me know if you liked it 🙂<img src="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/pZq5KASlQaoklIeuLYLi" width="640" height="482" />

My talk about profilers and their inner workings from JavaOne is now viewable for free in its entire length on Parleys. Watch it and let me know if you liked it 🙂

Video: Is My Profiler Telling The Truth?

My talk about profilers and their inner workings from JavaOne is now viewable for free in its entire length on Parleys. Watch it and let me know if you liked it 🙂<img src="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/pZq5KASlQaoklIeuLYLi" width="640" height="482">

Logo der codecentric AG, einem in Deutschland führenden IT-Consulting Unternehmen

Logo der codecentric AG einem in Deutschland führenden IT-Consulting Unternehmen

Data Effectiveness

Digital Products & Solutions

Digital sovereignty

IT Security

Artificial intelligence

Software Modernization

Services from A to Z ...

Services

How we work

References

Locations

Blog

Knowledge Hub

Careers

About us

News

Contact us

Do you have any questions or would you like to exchange ideas? Happy to help – just enter your contact details.

[ENG] Contact Us | Sub-Navi-Kontakt oder Zero Search Hits

Kontakt aufnehmen

Marketing & Community Management

Konntest du keine Einträge zu deinem Thema finden?  

Aktuelles

Thank you for your request. I will get back to you.

Thank you for your registration. A confirmation email is on its way to you.

This page is a preview. \nClick here\n to exit preview mode.

Sorry, we did not find any matching blog articles to your search. 

Do you still have questions? Just send me a message.

You have questions or want to exchange ideas with me? Gladly - just enter your contact details.

I have read the \nprivacy statement\n of codecentric AG and confirm this by submitting the form.*

Discover exciting further topics and let the codecentric world inspire you.

I hereby consent to the collection and processing of the above data for the purpose of receiving the newsletter of codecentric AG by e-mail. You can revoke your consent at any time by sending an e-mail to datenschutz@codecentric.de, by clicking on the link in the information e-mail itself, or by sending an e-mail to the contact details listed in the \nimprint\n. I have taken note of the \nprivacy policy\n of codecentric AG and confirm this by sending the form.*

Job application form field placeholder 10

Thank you for taking part in our free online IT security check 🙏.

To receive your result, we still need your email address

Karriere Podcast Page Info CTA Paragraph Mobile

Would you like to hear more quotes from our colleagues? What is it like to work in IT consulting at codecentric? Click here for an overview of all culture & career podcasts.

Softwerkercast Podcast Page InfoCTA Paragraph Mobile

Softwerkercast Podcast Page InfoCTA Paragraph

Softwerkercast Podcast Page InfoCTA Title

Softwerkercast Podcast Page InfoCTA Button

Sorry, we couldn't find any results for your search query.

Maybe the popular search entries will help you:

Apply now as [HEADING_TITLE] at codecentric

Video: Is My Profiler Telling The Truth?

Was this post helpful?

Blog author