Dropwizard and Quartz Part 1: Scheduled Java Jobs In A Nice Package

ColinM_1-thumbIt's very common to take simple,  regular, or automated tasks, cook them into your favorite executable and add another entry to crontab, and off you go, because you have bigger fish to fry. And this is fine, sort of.. It will work well, and your job(s) may continue to run for quite some time without errors. But what happens when you have lots of  jobs? Jobs that run on multiple schedules? Jobs that must not run concurrently?  Jobs that rely on flaky 3rd party services that fail every now and then? How are you going to manage and keep track of that easily? Over time it can become a nasty mess. The problem is that you have out-grown CRON and need something a little more sophisticated. There are a number of solutions out there: some are free, others can be expensive; some may be too simple, others too complex or cumbersome, and in the end  they may be decent but not exactly what you want. You don't want to reinvent the wheel with a grassroots solution, but you will need something that is flexible and malleable enough for your needs. If, and this may be a big IF, you are a Java shop, or have jobs/tasks that are platform/language agnostic, Dropwizard + Quartz could be a great solution for you.

Quartz is a Java scheduler, that is in principle very similar to good-old CRON, but with a lot more bells and whistles. Dropwizard is a well thought-out web services framework which will provide an excellent wrapper for managing and keeping tabs on your scheduled jobs.

A little bit about Dropwizard

From dropwizard.codahale.com:
 Dropwizard is a Java framework for developing ops-friendly, high-performance, RESTful web services. Dropwizard pulls together stable, mature libraries from the Java ecosystem into a simple, light-weight package that lets you focus on getting things done.

That pretty much says it all. Using Jetty, Jersey, and Jackson, among other things, it makes building a web service very very simple, giving you a number of nice features out of the box like configuration and health checking which we will discuss briefly below.

A little bit about Quartz

 From quartz-scheduler.org:
Quartz is a full-featured, open source job scheduling service that can be integrated with, or used along side virtually any Java EE or Java SE application - from the smallest stand-alone application to the largest e-commerce system. Quartz can be used to create simple or complex schedules for executing tens, hundreds, or even tens-of-thousands of jobs; jobs whose tasks are defined as standard Java components that are programmed to fulfill the requirements of your application. The Quartz Scheduler includes many enterprise-class features, such as JTA transactions and clustering. 

Quartz does everything CRON can do and much much more. Jobs can be stateful or stateless, monitored throughout every step of their life-cycle, and it comes with all of the Java error and event handling goodness you need. There are 3 main components to Quartz: the scheduler (of which you can have many, be we'll be using one), triggers, and jobs. As mentioned above Quartz has a clustered mode of operation where a schedules and jobs can be shared among multiple instances, but we haven't played around with that (yet!).

What we want out of this

The goal here is to have a centralized system that:

  • Runs our jobs exactly when and as often we want (Flexible)
  • Handle temporary failures (Robust)
  • Sends notifications of critical/permanent failures (Reliable)
  • Handles complex jobs using (almost) any 3rd party Java lib or service (Useful)
  • Allows non-tech personnel to see what's going on (User Friendly)
  • Testable and maintainable (Quality)

To do this we use some of Dropwizard's and Quartz's awesomesauce:

  • Create a managed instance of a Quartz scheduler for graceful start-up, shutdown, etc
  • Use a Dropwizard Health check to watch our Quartz Scheduler
  • Quartz Scheduler and  Job listeners
    • To track the current state of the system
    • Handle errors
    • Retry jobs that failed due to temporary issues (locked resources, timeouts, etc)
  • Dropwizard Configuration to set up our Quartz Scheduler
  • Quartz Job XML files to instantiate our jobs and triggers
  • Add web resources that interact with out managed Quartz Scheduler.

Dropwizard + Quartz: The Nitty Gritty

The code samples below provide a skeleton to get Quartz up and running within a Dropwizard web service. In this post we will be breaking down the most basic parts we need to give us a simple foundation to build upon in later posts. The first step is to create your main Dropwizard Service class that kicks everything off.

[gist]http://gist.github.com/3121212.js[/gist]

Our Dropwizard managed Quartz class is responsible for starting, stopping and checking in on our Scheduler and its jobs.

[gist]http://gist.github.com/3121315[/gist]

Configuring Dropwizard & Quartz

Dropwizard has a straightforward  configuration mechanism that uses YAML or JSON configuration files, making it easy to set environment and initialization parameters. We will be making use of this to set our Quartz Scheduler properties. This could be done in a separate quartz.properties file, but it in most cases it is better to keep all of your environment settings in one place.

[gist]http://gist.github.com/3121429.js[/gist]

Dropwizard YAML Confiuration for Quartz

The YAML configuration is pulled  in when the service is kick off using a simple command line argument. For example:

java -jar myDropwizardQuartzService.jar server production.yml

[gist]http://gist.github.com/3121351[/gist]

Dropwizard - Quartz Healthcheck

Dropwizard has a simple metrics and health check system that makes keeping tabs on your services or service features very straight forward. As our managed Quartz Manager / error handling gets more complex, its state/health can be completely encapsulated such that the health check doesn't need to be altered.

[gist]http://gist.github.com/3123161.js[/gist]

Creating Jobs Through XML

In this example we are instantiating/scheduling our jobs by listing XML files that described the jobs themselves, any data we want to pass in, and their triggers. It is possible to have multiple XML files, as seen in the configuration example above. Each XML file can contain multiple jobs and triggers.

[gist]http://gist.github.com/3121364.js[/gist]

For more about Quartz jobs and triggers take a look at the examples and tutorials.

In later posts we will cover and go into some more detail on the following topics:

  • More about  jobs, passing data, using  job and scheduler contexts
  • Scheduler and Job listeners
  • Handling and Retrying a job when it fails
  • Web resources and interface for the Quartz Service

Stay tuned!