It's very common to take simple, regular, or automated tasks, cook them into your favorite executable and add another entry to crontab, and off you go, because you have bigger fish to fry. And this is fine, sort of.. It will work well, and your job(s) may continue to run for quite some time without errors. But what happens when you have lots of jobs? Jobs that run on multiple schedules? Jobs that must not run concurrently? Jobs that rely on flaky 3rd party services that fail every now and then? How are you going to manage and keep track of that easily? Over time it can become a nasty mess. The problem is that you have out-grown CRON and need something a little more sophisticated. There are a number of solutions out there: some are free, others can be expensive; some may be too simple, others too complex or cumbersome, and in the end they may be decent but not exactly what you want. You don't want to reinvent the wheel with a grassroots solution, but you will need something that is flexible and malleable enough for your needs. If, and this may be a big IF, you are a Java shop, or have jobs/tasks that are platform/language agnostic, Dropwizard + Quartz could be a great solution for you.
Quartz is a Java scheduler, that is in principle very similar to good-old CRON, but with a lot more bells and whistles. Dropwizard is a well thought-out web services framework which will provide an excellent wrapper for managing and keeping tabs on your scheduled jobs.
A little bit about Dropwizard
From dropwizard.codahale.com: Dropwizard is a Java framework for developing ops-friendly, high-performance, RESTful web services. Dropwizard pulls together stable, mature libraries from the Java ecosystem into a simple, light-weight package that lets you focus on getting things done.
That pretty much says it all. Using Jetty, Jersey, and Jackson, among other things, it makes building a web service very very simple, giving you a number of nice features out of the box like configuration and health checking which we will discuss briefly below.
A little bit about Quartz
From quartz-scheduler.org: Quartz is a full-featured, open source job scheduling service that can be integrated with, or used along side virtually any Java EE or Java SE application - from the smallest stand-alone application to the largest e-commerce system. Quartz can be used to create simple or complex schedules for executing tens, hundreds, or even tens-of-thousands of jobs; jobs whose tasks are defined as standard Java components that are programmed to fulfill the requirements of your application. The Quartz Scheduler includes many enterprise-class features, such as JTA transactions and clustering.
Quartz does everything CRON can do and much much more. Jobs can be stateful or stateless, monitored throughout every step of their life-cycle, and it comes with all of the Java error and event handling goodness you need. There are 3 main components to Quartz: the scheduler (of which you can have many, be we'll be using one), triggers, and jobs. As mentioned above Quartz has a clustered mode of operation where a schedules and jobs can be shared among multiple instances, but we haven't played around with that (yet!).
What we want out of this
The goal here is to have a centralized system that:
- Runs our jobs exactly when and as often we want (Flexible)
- Handle temporary failures (Robust)
- Sends notifications of critical/permanent failures (Reliable)
- Handles complex jobs using (almost) any 3rd party Java lib or service (Useful)
- Allows non-tech personnel to see what's going on (User Friendly)
- Testable and maintainable (Quality)
To do this we use some of Dropwizard's and Quartz's awesomesauce:
- Create a managed instance of a Quartz scheduler for graceful start-up, shutdown, etc
- Use a Dropwizard Health check to watch our Quartz Scheduler
- Quartz Scheduler and Job listeners
- To track the current state of the system
- Handle errors
- Retry jobs that failed due to temporary issues (locked resources, timeouts, etc)
- Dropwizard Configuration to set up our Quartz Scheduler
- Quartz Job XML files to instantiate our jobs and triggers
- Add web resources that interact with out managed Quartz Scheduler.
Dropwizard + Quartz: The Nitty Gritty
The code samples below provide a skeleton to get Quartz up and running within a Dropwizard web service. In this post we will be breaking down the most basic parts we need to give us a simple foundation to build upon in later posts. The first step is to create your main Dropwizard Service class that kicks everything off.
Our Dropwizard managed Quartz class is responsible for starting, stopping and checking in on our Scheduler and its jobs.