It's very common to take simple, regular, or automated tasks, cook them into your favorite executable and add another entry to crontab, and off you go, because you have bigger fish to fry. And this is fine, sort of.. It will work well, and your job(s) may continue to run for quite some time without errors. But what happens when you have lots of jobs? Jobs that run on multiple schedules? Jobs that must not run concurrently? Jobs that rely on flaky 3rd party services that fail every now and then? How are you going to manage and keep track of that easily? Over time it can become a nasty mess. The problem is that you have out-grown CRON and need something a little more sophisticated. There are a number of solutions out there: some are free, others can be expensive; some may be too simple, others too complex or cumbersome, and in the end they may be decent but not exactly what you want. You don't want to reinvent the wheel with a grassroots solution, but you will need something that is flexible and malleable enough for your needs. If, and this may be a big IF, you are a Java shop, or have jobs/tasks that are platform/language agnostic, Dropwizard + Quartz could be a great solution for you.
Quartz is a Java scheduler, that is in principle very similar to good-old CRON, but with a lot more bells and whistles. Dropwizard is a well thought-out web services framework which will provide an excellent wrapper for managing and keeping tabs on your scheduled jobs.
A little bit about Dropwizard
From dropwizard.codahale.com: Dropwizard is a Java framework for developing ops-friendly, high-performance, RESTful web services. Dropwizard pulls together stable, mature libraries from the Java ecosystem into a simple, light-weight package that lets you focus on getting things done.
That pretty much says it all. Using Jetty, Jersey, and Jackson, among other things, it makes building a web service very very simple, giving you a number of nice features out of the box like configuration and health checking which we will discuss briefly below.
A little bit about Quartz
From quartz-scheduler.org:Quartz is a full-featured, open source job scheduling service that can be integrated with, or used along side virtually any Java EE or Java SE application - from the smallest stand-alone application to the largest e-commerce system. Quartz can be used to create simple or complex schedules for executing tens, hundreds, or even tens-of-thousands of jobs; jobs whose tasks are defined as standard Java components that are programmed to fulfill the requirements of your application. The Quartz Scheduler includes many enterprise-class features, such as JTA transactions and clustering.
Quartz does everything CRON can do and much much more. Jobs can be stateful or stateless, monitored throughout every step of their life-cycle, and it comes with all of the Java error and event handling goodness you need. There are 3 main components to Quartz: the scheduler (of which you can have many, be we'll be using one), triggers, and jobs. As mentioned above Quartz has a clustered mode of operation where a schedules and jobs can be shared among multiple instances, but we haven't played around with that (yet!).
What we want out of this
The goal here is to have a centralized system that:
Runs our jobs exactly when and as often we want (Flexible)
Handle temporary failures (Robust)
Sends notifications of critical/permanent failures (Reliable)
Handles complex jobs using (almost) any 3rd party Java lib or service (Useful)
Allows non-tech personnel to see what's going on (User Friendly)
Testable and maintainable (Quality)
To do this we use some of Dropwizard's and Quartz's awesomesauce:
Create a managed instance of a Quartz scheduler for graceful start-up, shutdown, etc
Use a Dropwizard Health check to watch our Quartz Scheduler
Quartz Scheduler and Job listeners
To track the current state of the system
Handle errors
Retry jobs that failed due to temporary issues (locked resources, timeouts, etc)
Dropwizard Configuration to set up our Quartz Scheduler
Quartz Job XML files to instantiate our jobs and triggers
Add web resources that interact with out managed Quartz Scheduler.
Dropwizard + Quartz: The Nitty Gritty
The code samples below provide a skeleton to get Quartz up and running within a Dropwizard web service. In this post we will be breaking down the most basic parts we need to give us a simple foundation to build upon in later posts. The first step is to create your main Dropwizard Service class that kicks everything off.
Our Dropwizard managed Quartz class is responsible for starting, stopping and checking in on our Scheduler and its jobs.
Configuring Dropwizard & Quartz
Dropwizard has a straightforward configuration mechanism that uses YAML or JSON configuration files, making it easy to set environment and initialization parameters. We will be making use of this to set our Quartz Scheduler properties. This could be done in a separate quartz.properties file, but it in most cases it is better to keep all of your environment settings in one place.
Dropwizard YAML Confiuration for Quartz
The YAML configuration is pulled in when the service is kick off using a simple command line argument. For example:
java -jar myDropwizardQuartzService.jar server production.yml
Dropwizard - Quartz Healthcheck
Dropwizard has a simple metrics and health check system that makes keeping tabs on your services or service features very straight forward. As our managed Quartz Manager / error handling gets more complex, its state/health can be completely encapsulated such that the health check doesn't need to be altered.
Creating Jobs Through XML
In this example we are instantiating/scheduling our jobs by listing XML files that described the jobs themselves, any data we want to pass in, and their triggers. It is possible to have multiple XML files, as seen in the configuration example above. Each XML file can contain multiple jobs and triggers.
For more about Quartz jobs and triggers take a look at the examples and tutorials.
In later posts we will cover and go into some more detail on the following topics:
More about jobs, passing data, using job and scheduler contexts
Scheduler and Job listeners
Handling and Retrying a job when it fails
Web resources and interface for the Quartz Service
Stay tuned!