Task Scheduler

At time we need to run a bunch of scheduled jobs to do some work. Crontab is a a great utility for running such scheduled jobs but it has its limitations. Lets see what all things could make a better scheduler

Requirement checklist 📝#

Ability to run a job manually without using the entire crontab command Check the status of the scheduled job(Completed/Failed) Jobs which would be parameterized Have a UI to manage scheduled jobs. We should be able to perform operations like - scheduling it, creating a new job, assigning the time when it should run

Tech stack based on the requirement#

Based on the requirement our candidate to manage these scheduled jobs or tasks is celery.

Here is the tech stack which we will be using

• Django - To provide a simple interface to manage the scheduler
• Celery - To run and manage tasks
• Redis - Our message broker
• Postgres - To store task relevant data(status, scheduler, etc). "The World's Most Advanced Open Source Relational Database"

High level Dataflow diagram#

Setting up 🎉#

Github Project#

• You can clone/fork this project to get started. I have configured all the below mentioned plugins and you can just run the project using docker-compose. Project Link

Resources#

• You can use this guide to setup django + celery
• We will use django-celery-beat to store the schedule of our tasks in database.
• django-celery-results is another plugin which help us in storing the task results in database and make it accessible by the django admin page directly (You can use flower for checking up the results)

# Storing the celery tasks results
CELERY_RESULT_BACKEND = "django-db"

• We can define the redis server details like this

# Settings for celery and redis configurations
CELERY_BROKER_URL = 'redis://redis:6379'  

Screenshots of the project 📸#

Django Admin Home#

List of Tasks#

Add a Task#

Check Tasks results#

Revising the requirements#

Ability to run a job manually without using the entire crontab command

Check the status of the scheduled job(Completed/Failed)

Jobs which would be parameterized

Have a UI to manage scheduled jobs. We should be able to perform operations like - scheduling it, creating a new job, assigning the time when it should run