Production-ready standalone REST using Dropwizard

Leave a reply

There has been a large movement to REST interfaces because of the shift to new mediums, such as mobile, as well as the overall shift to more client-side, in-browser functionality using HTML5/JS. While the server side processing of views still have their place, more and more users are wanting to see less full request-response interfaces, and more ajax and push functionality in their web applications. This is almost creating a new era of a two tiered architecture, shifting more of the domain and business processes closer to the user and leaving data access and other pluggable functionality on the server. If you squint your eyes, you can see the parallels to the Powerbuilder/stored procedure paradigms but the difference now if these technologies are incredibly scalable.

Thanks to the large movement to open source technology used in the big internet shops (Facebook, Twitter, blah-blah-blah), the lowly developer doesn’t have to look far to grab a good starter kit for building some pretty cool applications. One of the latest ones I found is Dropwizard. Per the website:

“Developed by Yammer to power their JVM-based backend services, Dropwizard pulls together stable, mature libraries from the Java ecosystem into a simple, light-weight package that lets you focus on getting things done. Dropwizard has out-of-the-box support for sophisticated configuration, application metrics, logging, operational tools, and much more, allowing you and your team to ship a production-quality HTTP+JSON web service in the shortest time possible.”

The cool thing about Dropwizard is the fact the services are turned inside out. Instead of the software being embedded in a container, the services usually associated with containers are embedded in the application. It’s a J2SE-based HTTP/JSON/REST framework that doesn’t require the deployment and installation of containers or web servers. The reason it’s cool is scalability part of it. If you want more services, you can horizontally scale on the same machine by simply starting some additional instances.

The other cool thing is that their documentation to get started is damn good. While there are tons of blogs out there that really serve a great need in further refining the existing documentation, Dropwizard needs no help. So I am not going to try to reinvent the wheel here. Go get started: http://dropwizard.codahale.com/getting-started/

After doing some development with the platform, here are some observations:

1) It’s really good for REST based services. But if you need to serve up any UI stuff outside of a pure HTML5/JS model, then you are going to have to jump through some hoops to integrate any view technologies into the stack such as JSF. Sure it has some cool capabilities to add assets to the web container to be served up, but once you start going there, there are some hurdles and fine grained knowledge needed to get things to fit right.

2) Security model requires some internal development especially if you want to do something other than Basic HTTP authentication and can get hairy. I tried multiple authentication/authorization models and even tried extending some of the existing Framework classes. While it can be done, it’s not easy.

3) No built in Spring integration and no Java EE CDI stuff. Not hard to initialize and get it into the model.

4) The stack is a best of compilation of some of my favorite things: Jackson, Hibernate Validator, and others. I also got introduced to some new stuff like JDBI. If anything, Dropwizard gets you to possibly take a look at some other ways to do things and perhaps might even change your development stack.

Overall, it’s worth a weekend for the tinkerer and a POC after that at your shop, especially if you are deploying a mobile app with a lot of users.

Go follow the developer @Coda

Couple of Dropwizard links I found helpful:

http://brianoneill.blogspot.com/2012/05/dropwizard-and-spring-sort-of.html

https://speakerdeck.com/jacek99/dropwizard-and-spring-the-perfect-java-rest-server-stack

http://www.gettingcirrius.com/2013/02/integrating-spring-security-with.html

Putting software inside other software

Leave a reply

I’ve lived most of my technical life in a web based application. Because I am a java guy and don’t like Swing development, when I build something, it usually had a web based interface. There are two really great things about web UIs: anyone can use them and they are horizontally scalable.

When you live in web applications, you are also mostly living in application servers. There are so many names and flavors you can get lost really fast. I came to age during the popularization of J2EE and did quite a bit of work on the behemoth platforms like Weblogic and Websphere, which I don’t think anyone can still justify why we used them. Cutting your teeth on EJB 2.0 CMP entity beans is not the most pleasant experience. Couple that with my experience with CORBA before that, and you’ll understand why I love Spring.

Recently I have been diving back into the technology, taking a particularly longer look at JEE technologies. I’ve been using JSF for years but always spring-backed. I’ve taken a liking to the JBoss AS 7 capabilities and even the TomEE server. But the question still begs to be answered: why do we need these bloated application servers anyway?

With REST and simple embeddable web servers such as Jetty, you can get some of the goodness of a J2SE environment and none of the snowflake configurations required by the “standard” app servers. There is even some open source tools at your disposal, most notably Dropwizard, which allows you to have some pretty robust standalone web resources without the containers (well, the containers are still there but they aren’t as instrusive). These platforms do have their drawbacks, requiring some pretty extensive work to handle security and if you need a full blown UI using JSF, it can get painful.

What the industry needs is an easily deployable standalone application model that doesn’t require extensive custom configuration. I need to build an application, deploy it over there, create some network rules and start it. And if I need it to cluster, it can be self aware of it’s clustering environment (Hazelcast has spoiled me). When your application developer is spending more time building plumbing and security rather than domain objects and service layers, it’s a problem. Don’t get me wrong, there are plenty of useful and relavent cases where large application server infrastructure is not only necessary, but can be a strategic advantage in terms of scalability and ease of deployment. I just wish there were more choices.

Are you ready for the cloud?

Leave a reply

The “Cloud” is one of the most overused and misunderstood buzzwords in technology today. For most people who have watched too many Microsoft commercials touting their cloud integrated operating systems, it’s this abstract resource in the sky that everything magically connects to and things happen. Most people do not understand that it’s simply an external network of computation resources (computers). Just like you have your server at work with all your spreadsheets on it (the ubiquitous S: drive or something), the public cloud is simply a set of computers located at a data center that you have access to.

But this is where the true cloud conversation starts, and there are so many other aspects to the cloud people do not understand. To understand cloud computing, especially public cloud such as Amazon EC2, you must first understand virtualization. To understand virtualization, you must need to understand operating systems and how they work with your computer. Essentially, a computer is made up of a bunch of different components (hard drive, memory, network card) that talk to each other through a common physical pathway(motherboard). But there is another pathway required for these communications and that is the operating system (OS). Each component, such as the hard drive, only knows about itself and has an interface in which it allows software to interact with it via the physical connection, and this software is typically the OS. When most people buy a computer, it has a single OS on it that single handedly controls all the hardware interactions.

Virtualization simply adds an additional layer to allow multiple OS access to the same hardware. Instead of having one OS running on a computer, you can have two or more running as a virtual machine (VM). The virtualization software acts as a traffic cop, taking instructions and other interactions from each OS and ensuring they all get access to the hardware devices they need, but also make sure that each of the devices get the needed information back to the correct OS. There are lots of examples of virtualization software, most notably VMWare and Virtualbox, that will allow users to run multiple OS on their machines. You can run an instance of Ubuntu on Windows 7. There are huge benefits to be gained from this, but when it comes to public cloud computing, the main benefit is shared hardware and abstraction of computing resources away from physical hardware.

Once you understand how virtualization works, it’s not a big leap to realize the public cloud is simply an automated virtualized environment allowing users to create new VMs on demand. The user doesn’t have to worry about having enough hardware available or even where that VM is located. They simply create an instance of a VM by specifying attributes of how they want the machine configured, such as processor speed or memory capacity, and don’t care about where or how. The public cloud is simply a manifestation of a concept that has been maturing for quite a while – turning computation resources into a homogeneous commodity rather than a specialized product.

This is the point to where light bulbs start turning on above the heads of executives. They start looking at these opportunities to use generic, commoditized computing resources and remove the risks and costs associated with maintaining and managing data centers. All of the hardware that sits unused for weeks at a time because it’s only needed for special cases, like emergency production support releases, can be sunset. We can build performance testing environments in a single day and then delete the entire thing by the time we go home. The possibilities are endless.

But let’s be clear about something. There is something about public cloud infrastructure that makes it special. It’s not the virtualization software. It’s not the hardware used. It’s the people.

Public clouds like Amazon EC2 have the best and brightest operations engineers on the planet, creating automated processes behind the scenes so that users like us just need to click a button. It’s not easy. Their environment is a hurricane of requests and issues that most people can’t dream of. They manage a half-a-million linux servers. 500,000 servers. Most people ask how they do it and the answer is simple: they automate everything they can. Implementing that simple answer is where most people start to run into issues. Luckily, Amazon hires the best ops people in the world and probably pays them a small fortune to do this, both of which is simply not available to most businesses. Public cloud is about standing on the shoulders of the best operations talent in the world and taking advantage of their automation procedures.

Remember those light bulbs I referred to earlier above the heads of executives? Let’s take a sledgehammer to them with a single statement: All your data that you put into the public cloud is on the shared hardware. Your CRM data that you have in your cloud database? It could be sitting on the same drive and the same platter as your competitor. Your super-secret computation process could be sharing processing time with Reddit on the same CPU. The public cloud means we all share the same hardware. While there are quite a few security measures to make sure the virtualized segregation stays in place, we all know what typically happens – all secure data is deemed not cloud worthy and our virtualized hopes and dreams die with a whimper.

Until someone says the words “Private Cloud”.

Here’s the problem with “Private Cloud”. The public clouds are good because of the people and the culture. It’s not the tools. In fact, I would be willing to bet that 99% of the tools used by Amazon could easily be purchased or found via open source. Most organizations simply don’t have the resources, the processes, or the intestinal fortitude to engage in such an endeavor. You need three things to be successful: a culture of automation, talented hybrid operations and developers willing to blur the lines between coding and deployment and tools to automate. You can buy the tools. You can buy the people. You can’t buy culture.

Let’s get past the limitations I stated earlier and create a hypothetical company with it’s own private cloud. Unless your software is built for the cloud, your not really buying anything. When I say built for the cloud, imagine being able to scale an application horizontally (think web application) by adding new web servers on demand. Here’s a basic list of how to create a VM from scratch and take it into a production cluster:

  1. Create a new virtual machine.
  2. Install the correct image/OS.
  3. Automatically configure the VM to the specifications needed for the application (install Java, run patches, etc).
  4. Install any containers (JBoss, Tomcat, whatever) and configure them automatically (JMS, DB Connections, Clustering with IP addresses, etc).
  5. Deploy an EAR or WAR from your internal build repository and have it automatically join any existing cluster seamlessly.
  6. Configure the network and other appliances to recognize and route requests to the new server automatically.
  7. Do everything just stated in reverse.

Unless you can do something like this automatically, then you aren’t cloud. You are virtualized. Not that simply being a company that utilizes virtualization isn’t a great first step. There are many uses of virtualization that could provide many benefits to a company quickly, such as having virtualized QA environments that can quickly be initialized to run emergency production support releases. Virtualization is a great thing by itself, but virtualization is not cloud.

Secondly, the fourth and fifth bullet points are where most software shops get caught up. Their applications are simply not built for it. When the architects laid the groundwork for the application, they didn’t think about having the ability to quickly add new instances of the application to the cluster such as not using DNS to handle routing requests. There are some decisions that are made or even not even addressed that are core to an application that can’t handle these types of environments. It’s a square peg in a round hole and while some applications can be retrofit to handle the automation, some will need to be rearchitected. There are even some applications who don’t make sense in the cloud.

I encourage everyone to give virtualization a look. Every organization who has multiple environments for development, QA and UAT would benefit from virtualizing them. There are many software packages and platforms out there that are easy to use and some that are even open source. But before you start down the cloud path, make sure you do your due diligence. Are you ready for the cloud? Sometimes the correct answer is no. And that’s OK too.

DevOps, Open Communication and Family

Leave a reply

I recently had a incident with a family member that didn’t work out as well as I hoped it would. Over the past six months we had both unintentionally slighted each other in different ways which lead to increased tension in our relationship. With any relationship, personal or professional, if there is tension in the air and stress around conversations and interactions, it will result in a reduction of contact. As humans, we tend to seek the easy path, and are typically conflict adverse, which would naturally lead us to not engaging with others who add stress to our lives. But with family, you really can’t eliminate the communication and unfortunately, we had both reduced communication down to the point the frustrations bubbled over and we both unleashed the issues on each other in one huge brain dump. It didn’t go well.

Here’s why it didn’t go well.

  • We had stopped communicating.
  • Instead of expressing our displeasure about perceived slights, we bottled them up.
  • The stories had been discussed with other family members, which gave each of us additional positive feedback that “we were right and they were wrong”.
  • Misinformation

Sound familiar?

This story reflects some common occurrences of dysfunction across technology teams. As technology continues to progress and business requirements lead to decreased cycle time to bring requirements to production, two things are happening. First, there is an overall increased tension added to the workday. Agile teams, with short delivery time frames and high expectations, are expected to deliver production ready working software in sometimes two weeks. Operations and QA teams also have an increased expectation to help facilitate and deliver these releases in a more rapid manner. I happen to believe this tension is actually a positive thing, and leads to both increased productivity and employee satisfaction, but for some organizations, it’s added fuel to an already burning fire.

In addition to the increased tension, these delivery requirements are bringing the development, QA and operations teams together. In a traditionally siloed organization, these teams can have fundamentally different cultures. Just think of the stereotypes associated with the teams: Developers are cowboys who rarely test and don’t care about infrastructure; QA teams are just there to create bugs that don’t exist and make more work for themselves; ops are a bunch of keyboard jockeys who think everything that doesn’t have a command line interface is crap. Developers cause issues for QA, QA causes issues for developers. Ops causes issues for QA teams and developers, while all the deployment issues are blamed on ops. While this characterization is a bit comical, in my experience there tends to be some truth to it. The teams have different concerns – and sometimes these concerns are in direct conflict with each other.

DevOps is the manifestation of these realities. DevOps is most importantly a cultural movement bringing these independent teams together to create shared concerns, and stressing collaboration and communication. It’s about identifying the walls that exist between teams and bringing those walls down. There are some things that can help organizations become more collaborative  such as automation tools for building, testing and deployment, but DevOps tools are not the silver bullet. You can have all the latest and greatest tools at your disposal and carte blanche from management to make infrastructure decisions, but if your development team still thinks code deployment is not their problem, then you will create bottlenecks in your delivery pipeline and increase your cycle time.

Using the reasons listed above, let’s look at how DevOps can be applied to reduce dysfunction in your organization.

  • We had stopped communicating – DevOps helps blends the teams and bubble concerns up to a single set of common cultural norms, centered around delivery of business functionality. For example, DevOps culture helps QA teams work directly with developers during sprints, creating testing scenarios in parallel to development efforts. These efforts not only help the teams deliver higher quality code, but it increases the number of requirement interpretations on the team and flushing out missing or incomplete understanding of requirements. Simply put, more eyes on requirements at earlier stages of implementation reduces work associated with late changes, which tend to be more risky in their nature.
  • Instead of expressing our displeasure about perceived slights, we bottled them up – You have to be able to sit down with any colleague and have a discussion about perceived slights in a positive way. There was a recent episode of DevOps Cafe which described an organization that actually documented a scale they could use during any conversation to give the other person a heads up that they are entering into “getting personal” territory, without creating additional conflict. A team member could simply say “You are getting pretty close to a 3, Bob” during a conversation and Bob knew that he needed to tone down his communication, or restate his issue in a more objective manner. It’s a great idea because it immediately puts the other person on notice about how their statements are being interpreted without resorting to similar tactics. Escalation is what is to be avoided. And sometimes, you just need a referee. DevOps specific people can be great at this, as they can put on each hat (Dev, QA, Ops) independently.
  • The stories had been discussed with other family members, which gave each of us additional positive feedback that “we were right and they were wrong” – When you think something is right, that’s one thing. But go to a group of like minded individuals for verification on your thoughts and you’ll get the dreaded confirmation bias. Your opinions about situations need to be expressed to those who don’t necessarily hold your exact ideals to get a true reflection. Ask a bunch of developers why a deployment went wrong and you’ll rarely get them to say it’s because of the code. It’s like going to the Republican national convention to get confirmation of why raising taxes is bad. The other thing is does is reinforce the confirmation, which makes you “dig in deeper”. We should all stay nimble and prepare to be flip-floppers. There  is nothing wrong with changing your mind, especially when new information is available.
  • Misinformation – A quarter of things that I was arguing about with my family member were just flat out false. Until we had the discussion, we had assumed things were a certain way or that certain things had occurred when they didn’t. The truth doesn’t do any good if all parties involved don’t have all the facts. Make any conflict first an exercise in fact finding. Answer the questions of what, why, how and when. Leave the who out of it for now.

Although the conflict with the family member was unfortunate, I am glad it finally happened. The occasion gave us both a chance to clear the air, sync up regarding misinformation and create a new set of norms and expectations moving forward. The biggest realization about the event was that we both needed to work harder in the future to nurture our relationship more. Family takes work. Relationships take work.

Although most people separate their family relationships from their working relationships, so much of the dynamics are the same. In fact, I spend more time in closer proximity and talk more with my colleagues with anyone in my family besides my wife and kid. The dynamics end up being very similar to family – they end up being very close to you, share many of your life’s highs and lows, and you have to have a relationship with them. These types of things are not easy for any organization. People take things personally, get hurt, they get angry. They are human. But with all of these negatives, the positives are equally awesome – the creativity of your development team bringing unbelievable functionality to your organization; the rock of your QA team giving you the confidence to push out quality code without worry; knowing you have an ops team ready to kill any rogue issues at a moment’s notice.

In the end, it’s all about communication, shared ideals and being family.

Dodd Frank Implementation – Great Case for Agile Development

Leave a reply

The Dodd-Frank Act (DFA) has been a major disruptor in 2012, especially in the energy industry. For those of you who are not familiar, DFA has created a set of pretty extensive external reporting requirements, both for trade life cycle events but also for data aggregations. For some organizations who are Swap Dealers, these reporting requirements are a major burden, requiring external reporting of trade life cycle events, such as execution and confirmation, in time frames as short as 15 minutes. In the financial services world, these reporting burdens are not as big of a leap, as the infrastructure and generally accepted practices of external, real time communication to facilitate straight through processing is prevalent. The energy and commodities world is not on the same level of sophistication, both because commodities trading tends to require much more bespoke and complex deal modeling, but also simply because they never needed to report events externally real-time.

In addition to these requirements, there has been volatility in the rules. Certain rules, such as Position Reporting, have been vacated (for now) leaving many projects in mid-flight. Other rules, such as Swap Data Repository reporting (Part 45), had data interfaces and workflow definitions offloaded on multiple vendors (p 2139), resulted in a very fragmented ecosystem where many-to-many data mappings and formats were required for different asset classes. Additionally, SDRs are implementing their systems during these different rule changes and clarifications, resulting in a fairly unstable integration task. This type of work is perfect for agile development.

  • Short sprints would allow you to push out functionality in short time frames, giving the team a natural checkpoint to make sure the functionality was still in-line with the latest legal opinion or CFTC change (Physical Options, anyone?). Every two weeks, the team can stop, demo the functionality they have built and receive feedback for the next sprint. Volatile requirements require a tight, frequent feedback loop. If you are building a huge technical spec document for a DFA implementation, you are toast.
  • Code demos naturally push end-to-end testing, giving the users a look at the real implementation rather than waiting until the last minute. The users can make adjustments at an earlier stage, reducing project risk and increasing customer satisfaction.

I would highly encourage all the companies who haven’t started your DFA efforts to look to agile to manage the project. Your developers and your users will thank you for it.

Demo Code: Create Persistence Jar using JPA

Leave a reply

I love keeping my repository code in a single jar, isolated from all other code. Persistence code should be portable and reusable as a library specific to a database or even a schema. This wasn’t always the easiest thing to do, especially in an ecosystem where the library may run in a Spring based webapp, a swing gui and a Java EE EJB application. Here’s the template code for how to get that ability.

First, let’s look at the basic EntityManager usage pattern. There are much more sophisticated ways of doing this but I’ll keep it simple for my own sake.

//Get the correct persistence unit and EntityManagerFactory
EntityManagerFactory entityManagerFactory = Persistence.createEntityManagerFactory("demoManager");
EntityManager entityManager = entityManagerFactory.createEntityManager();
entityManager.getTransaction().begin();
//Create an object and save it
entityManager.persist(new ApplicationUser());
//We are just testing so roll that back
entityManager.getTransaction().rollback();
//Close it down.
entityManager.close();
entityManagerFactory.close();

JPA Persistence is driven from a file in your classpath, located at META-INF/persistence.xml. Essentially, when creating the EntityManagerFactory, the Persistence class will go look for the persistence.xml file at that location. No file? You get a INFO: HHH000318: Could not find any META-INF/persistence.xml file in the classpath error. Eclipse users: Sometimes you gotta clean to get the file to show up for the junit test. Here’s a simple persistence.xml that shows how to use JPA outside of a container. 

<?xml version="1.0" encoding="UTF-8"?>
<persistence xmlns="http://java.sun.com/xml/ns/persistence" 
	xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
	xsi:schemaLocation="http://java.sun.com/xml/ns/persistence http://java.sun.com/xml/ns/persistence/persistence_2_0.xsd" 
	version="2.0">

	<persistence-unit name="demoManager" transaction-type="RESOURCE_LOCAL">
		<class>com.stephennimmo.demo.jpa.ApplicationUser</class>
		<properties>
			<property name="javax.persistence.jdbc.driver" value="org.hsqldb.jdbcDriver" />
			<property name="javax.persistence.jdbc.user" value="sa" />
			<property name="javax.persistence.jdbc.password" value="" />
			<property name="javax.persistence.jdbc.url" value="jdbc:hsqldb:." />
			<property name="hibernate.dialect" value="org.hibernate.dialect.HSQLDialect" />
			<property name="hibernate.hbm2ddl.auto" value="create-drop" />
		</properties>
	</persistence-unit>

</persistence>

Notice when you create the EntityManagerFactory you need to give it the name of the persistence-unit. Rest is pretty vanilla, but if you need some additional explanation.

Next, let’s look at the basic JPA object. 

@Entity(name="APPLICATION_USER")
public class ApplicationUser implements Serializable {
	
	private static final long serialVersionUID = -4505032763946912352L;

	@Id
	@GeneratedValue(strategy=GenerationType.IDENTITY)
	@Column(name="APPLICATION_USER_UID")
	private Long uid;
	
	@Column(name="LOGIN")
	private String login;

//Getters and Setters omitted for brevity sake

}

And if you want to use the JPA in a container, here’s a simple example of how the persistence.xml would change. 

<?xml version="1.0" encoding="UTF-8"?>
<persistence xmlns="http://java.sun.com/xml/ns/persistence" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
	xsi:schemaLocation="http://java.sun.com/xml/ns/persistence http://java.sun.com/xml/ns/persistence/persistence_2_0.xsd" version="2.0">

	<persistence-unit name="demoManager" transaction-type="JTA">
		<provider>org.hibernate.ejb.HibernatePersistence</provider>
		<jta-data-source>java:/DefaultDS</jta-data-source>
		<class>com.stephennimmo.demo.jpa.ApplicationUser</class>
		<properties>
			<property name="hibernate.dialect" value="org.hibernate.dialect.HSQLDialect" />
			<property name="hibernate.hbm2ddl.auto" value="create-drop" />
		</properties>
	</persistence-unit>
	
</persistence>

That’s basically it. The most painful point comes if you are trying to use a persistence jar inside EJB jar, it REQUIRES you to list out the classes in the persistence.xml.

As always, demo code available at my public repository.

http://code.google.com/p/stephennimmo

DevOps culture can create a strategic advantage for your company

Leave a reply

I’ve recently stumbled across a great podcast called DevOps Cafe. These guys basically push out about a show a week, interviewing different people from the DevOps community or simply discussing DevOps culture and why it’s great. DevOps is a software development culture that stresses communication, collaboration and integration between software developers and operations. DevOps blurs the line between development, deployment and system administration creating a more team based approach rather than throwing things over the proverbial wall. DevOps is something that has existed for many years now, but like so many other things, there is finally a nomenclature and language being built around it because of its adoption into the enterprise arena.

What are some of the things this solves? If any of these ring true in your organization, you might want to take a look at DevOps.
  • Does it take weeks to deploy a new build to production?
  • Does the code deployment require manual intervention such as ops updating a properties file?
  • Is your QA team manually testing some production support patch when they should be busy working on regression tests for the next build?
  • After any release, do you must have a period of  ”all hands on deck” because you aren’t sure if something is going to go wrong?

What does DevOps culture bring to your organization that can create an advantage?

  • Agile development – small sprints create small deliverables that provide small, manageable change. Lots of small changes rather than huge changes can lower the risks of deployments and help get truly needed bug fixes and enhancements out the door. Hey trading organizations – this should be something you are very interested in as you typically have lots of small changes!
  • Continuous Delivery – your dev teams should be able to push the latest code to production at all times. There are no more huge branch and merge operations, because development work is broken down into many small chunks and pushed continuously and tested continuously. I can’t tell you how many shops I have worked in that take weeks to deliver to production already completed code, only to have that code be obsolete or need changes prior to going live. Moving to production could be a weekly thing and something you do every week should be easy, right?
  • Automation – if you do something more than twice, you need to automate it. Developers should be writing code, not creating builds. QA teams should be creating repeatable regression tests, not manually testing the UI again. Operations should be writing new optimized deployment scripts, not manually patching servers. And they should all be working together to push the product to production – a failure on any part is a failure on the whole. We should all be in this together.

How do you get started?

  1. DevOps is a culture, not a tool set  A new paradigm needs to be established and continuously reinforced – we are all in this together.
  2. From a development side, stop branching and starting checking in on the trunk. Create a new build on every check-in and have that build fully regression tested. EVERY TIME you check in, the code should be production ready.
  3. From the ops side, start automating the environment by using virtualization. Ops should be able to add a new node to a cluster with the push of a button. Ops should be able to deliver builds to a host with the push of a button.
  4. Your QA team needs to refocus their efforts on producing repeatable regression tests on the builds. Again, this should be repeatable.

Here’s some of my favorite tools for the jobs (Remember, I am a java guy).

  • Build with Maven.
  • Manage your artifacts with Sonatype Nexus. I also like Apache Archiva.
  • Automate your testing with TestNG, JUnit. Create regression testing for webapps using Selenium.
  • Deploy using Puppet. I also like Jenkins. Depends on how fancy your infrastructure is. Puppet does a lot with infrastructure management as well.

Performance Killers for Low Latency Java Applications

Leave a reply

There are fundamental differences in writing low latency programs which require much more planning and detailed execution than your normal web application or basic workflow application. When optimizing for low latency, here are some of the touch points to be addressed.

  • The first and most deadly is IO. IO of any kind includes network, disk or anything outside the CPU and memory (and sometimes anything outside the CPU). This is the toughest issue to manage because low latency applications also tend to have additional non-functional requirements around failover and guaranteed delivery of messages. We want things to be fast, but losing information along the way is a non-starter. What are the typical ways to reduce the chances of lost data? That’s right, almost all of them involve IO. This answer usually comes down to expectations around performance. How fast is fast enough and does that requirement make redundancy or guaranteed messaging mutually-exclusive?
  • Garbage collection is one of the greatest gifts and largest curses to Java developers. The first time I realized I could instantiate an object without worry about it’s size or having to specifically release the object, it was a great weight off my shoulders. That was until the first time a 3 second garbage collection hit in the middle of processing a thousand messages a second. When building low latency applications, why you create objects becomes important again and object reuse strategies can make a huge difference in your overall application performance. If you are a java guy, you need to be able to discuss GC at a very low level. http://www.oracle.com/technetwork/java/javase/gc-tuning-6-140523.html
  • Non-essential functionality needs to be removed. AOP pointcuts around every service call for logging or whatever adds method calls and object creation to the stack. Unless your application needs it, get rid of it. Especially the logging and do the logging right if you need it! http://www.javalobby.org/java/forums/t18758.html
  • Hardware makes a difference. You have a low latency application and your running in the cloud? Virtualization adds network hops. Using commodity hardware? Those absolutely necessary IO calls would perform much better on a fibre. Spinning disks instead of SSD? Get out of here.

Once these other issues are tackled, then you can actually start coding for low latency. See, you could spend thousands of hours pouring over every detail of your code, optimizing which Collections object you use and reducing three lines of code to two, but if you do all of that and run the app on EC2 and use file-based SAN for your messaging redundancy persistence, you are toast. Tackle the big questions first before you tackle the little questions.

Here’s some tools to help:

Metrics - http://metrics.codahale.com

JAMon - http://jamonapi.sourceforge.net

JProfiler – FREE! - http://java.dzone.com/articles/jprofiler-your-java-code-could

VisualVM - http://visualvm.java.net/index.html

Yourkit Profiler – http://www.yourkit.com/overview/index.jsp

Demo Code: Java Concurrency Framework, Multithreading and Race Conditions

Leave a reply

This demo template was used to demonstrate not only the capabilities around multithreading using Java Concurrency Framework, specifically the executors. Instead of doing the “Hello World” type of tutorials which have already been done very well by others, I chose to use the time to demonstrate how to extend the framework to solve an issue.

The use case in question is related to FIX message processing, or actually dependent transaction processing in general. For example, let’s say someone sends an equities NewOrderSingle to buy 100 shares of $GOOG, then immediately realizes they don’t want the order and send a OrderCancelRequest. These two messages will be related and need to be processed in order. Processing the cancel prior to the new order would result in two errors – DK for the cancel and would leave an open order, which is bad. For these messages, they will have an identifier (ClOrdID) that is common between them and can be used to identify them.

If you have a multithreaded system, taking messages in and processing them, you could have a race condition where one thread picks up the order and another thread (on possibly a different machine) pick up the cancel. This results in the possibility of the cancel thread finishing first. We can’t have that.

To simplify the use case, we simply the logic. We want the same thread to process both messages. We will accomplish this by mapping the identifier (ClOrdID) to a thread using a local concurrent map, and routing any subsequent messages into the same thread by looking in the map for the key. No key, find the least used queue. Key is there, send it into the same queue. But wait, these chains could have no end because of a FIX message called OrderCancelReplace which will allow you to simultaneously cancel an existing order and replace it with a new order, usually with a different quantity or price. With these messages, you could be a one-to-many relationship for a thread. Here’s an example.

New -> 1000
CanRepl -> 1000, 1001
CanRepl -> 1001, 1002
Cancel -> 1002

Yes, that means that any messages related to these 3 different order ids need to roll into the same queue if they are all still in process. So if you have 10 threads running and the new 1000 goes to thread 5, then all of the order ids (1000, 1001, 1002) need to go to thread 5 if their related messages are still processing. Fun, eh? Try building this logic with 10 fix engines routing messages into an ESB with a cluster of message driven beans listening to the queues. It gets complicated. To the code!

First I needed to extend the Runnable interface to create the ability to see what my current and dependent keys are. For messages like NewOrderSingle, there is no dependent key, but the ClOrderID is the key. For OrderCancelReplaceRequest, the ClOrdID is the key, and the OrigClOrdID will be the dependent key. 

public interface KeyedRunnable<T> extends Runnable {
	
	T getKey();
	
	T getDependentKey();

}

Then there are two phases to the handling of these messages. The first is handling and routing inbound messages to the right thread. This service creates a set of ThreadPoolExecutor objects, each with a single thread. Yes, a pool of single threaded thread pools is a thread pool. Why did I do it this way? Because the executors do a bunch of stuff that you have to hand code if building your own Thread objects. 

public class KeyedExecutorService<T> {

	private int numberOfExecutors;
	private Random random = new Random();
	private ExecutorService unorderedExecutorService;
	private ConcurrentMap<T, UUID> uuidMap = new ConcurrentHashMap<T, UUID>();
	private ConcurrentMap<UUID, KeyedThreadPoolExecutor<T>> threadPoolExecutorMap = new ConcurrentHashMap<UUID, KeyedThreadPoolExecutor<T>>();

	public KeyedExecutorService(int numberOfExecutors) {
		unorderedExecutorService = Executors.newCachedThreadPool();
		this.numberOfExecutors = numberOfExecutors;
		for (int i = 0; i < numberOfExecutors; i++) {
			UUID uuid = UUID.randomUUID();
			KeyedThreadPoolExecutor<T> threadPoolExecutor = new KeyedThreadPoolExecutor<T>(uuidMap, 1, 1, Long.MAX_VALUE, TimeUnit.SECONDS, new LinkedBlockingQueue<Runnable>());
			threadPoolExecutorMap.put(uuid, threadPoolExecutor);
		}
	}

	public void execute(KeyedRunnable<T> command) {
		if (command.getKey() == null && command.getDependentKey() == null){
			unorderedExecutorService.execute(command);
		}
		synchronized (uuidMap) {
			if (command.getDependentKey() == null){
				System.out.println("No Dependent Key, sending to least used");
				executeInLeastActivePool(command);
			} else {
				if (uuidMap.containsKey(command.getDependentKey())){
					UUID uuid = uuidMap.get(command.getDependentKey());
					uuidMap.put(command.getKey(), uuid);
					System.out.println("Dependent Key Found, sending to " + uuid);
					threadPoolExecutorMap.get(uuid).execute(command);
				} else {
					System.out.println("No Dependent Key Found, sending to least used");
					executeInLeastActivePool(command);
				}
			}
		}
	}
	
	private void executeInLeastActivePool(KeyedRunnable<T> command){
//This could be round-robined if you have a fast processing system.
		Set<Entry<UUID, KeyedThreadPoolExecutor<T>>> entrySet = threadPoolExecutorMap.entrySet();
		for (Entry<UUID, KeyedThreadPoolExecutor<T>> entry : entrySet) {
			if (entry.getValue().getQueue().size() == 0) {
				uuidMap.put(command.getKey(), entry.getKey());
				entry.getValue().execute(command);
				return;
			}
		}
		List<UUID> uuidList = new ArrayList<UUID>(threadPoolExecutorMap.keySet());
		UUID uuid = uuidList.get(random.nextInt(numberOfExecutors));
		uuidMap.put(command.getKey(), uuid);
		threadPoolExecutorMap.get(uuid).execute(command);
	}
	
	public void shutdown(){
		unorderedExecutorService.shutdown();
		for (Iterator<UUID> iterator = threadPoolExecutorMap.keySet().iterator(); iterator.hasNext();) {
			threadPoolExecutorMap.get(iterator.next()).shutdown();
		}
	}

}

So once the message get to the right thread, after it executes, we need to remove that key from the mapping to make sure it opens back up. If a Cancel comes in for ClOrdID 1000, but the NewOrderSingle is already processed, then it can be processed anywhere. Notice the two classes both have a reference to the ConcurrentMap that stores the UUIDs for thread mapping/routing. 

public class KeyedThreadPoolExecutor<T> extends ThreadPoolExecutor {
	
	private ConcurrentMap<T, UUID> uuidMap;

	public KeyedThreadPoolExecutor(ConcurrentMap<T, UUID> uuidMap, int corePoolSize, int maximumPoolSize, long keepAliveTime, TimeUnit unit, BlockingQueue<Runnable> workQueue) {
		super(corePoolSize, maximumPoolSize, keepAliveTime, unit, workQueue);
		this.uuidMap = uuidMap;
	}
	
	@SuppressWarnings("unchecked")
	@Override
	protected void afterExecute(Runnable command, Throwable t) {
		super.afterExecute(command, t);
		if (command instanceof KeyedRunnable<?>){
			KeyedRunnable<T> keyedRunnable = (KeyedRunnable<T>)command;
			if (keyedRunnable.getKey() == null && keyedRunnable.getDependentKey() == null){
				return;
			}
			synchronized (uuidMap) {
				uuidMap.remove(keyedRunnable.getKey());
			}
		}
		
	}

}

Here’s how you would use the API. 

@Test
	public void runKeyedExecutionTest(){
		KeyedExecutorService<String> threadMultipoolExecutor = new KeyedExecutorService<String>(10);
		int total = 0;
		for (int i = 1; i < 100000; i++) {
			for (int j = 1; j < 100; j++) {
				String aString = UUID.randomUUID().toString();
				StringPrinter sp = new StringPrinter(Integer.toString(j), Integer.toString(j - 1), aString);
				threadMultipoolExecutor.execute(sp);
				total++;
			}
		}
		System.out.println("Done " + total);
		threadMultipoolExecutor.shutdown();
	}

This algorithm can be expounded to a distributed system, however any consumption still needs to be single threaded on the backend! Realistically, just process it in the local threads and scale your fix farm out horizontally.

Demo Code Repository – http://code.google.com/p/stephennimmo/
Quickfix/J – http://www.quickfixj.org

Scrum without metrics is useless

Leave a reply

You can read all day about the benefits and weaknesses involved in agile project delivery. In my opinion, the most important aspect is fairly simple: I want my team to deliver more code with less defects in an accurate time frame.

You can spend all day deciding prioritization of backlogs, argue about whether or not scope changes should result in sprint cancellations, or whether or not the ideal workday has 6 or 7 hours of capacity but if your team isn’t getting better at doing it’s job, your missing out on the real power of scrum. Here’s a few steps you can take to help reap the benefits of scrum.

  • It all starts with planning. Have a sprint planning session prior to kickoff. Have your entire team sit down with the selected sprint backlog items and provide estimates for development. But don’t just throw numbers out there. Make sure everyone is giving independent estimations and then compare everyone’s estimates. If you have 3 team members give a 2 day estimation and the 4th member says it’s ten days, then something is wrong. Stop. Discuss and get everyone to buy in to the estimate. Write down these planning numbers and have those numbers available at the end of the sprint – regardless of the mid-sprint changes.
  • At the end of the sprint, take the actual hours and compare them against the estimations. Create a tolerance level and for any estimations outside the tolerance, have the team answer for and discuss the reasons for divergence. While these are considered estimations, your developers should be held accountable for them, not because they need to be disciplined but because they should WANT to get better at providing estimations.
  • Demo your code. Do it for anyone in the company. If no one shows up, still keep doing it. It’s just as much for your development team as it is for the product customers. Show off!
  • Don’t be afraid to change your sprint’s underlying attributes to better suit the product backlog. If your demo creation is a big ordeal, then moving to 3 week sprints from 2 week sprints would give you more bandwidth.

Agile isn’t the silver bullet. It won’t solve your problems but it will make your problems very painful, very fast.