A general purpose architecture for background processes

Introduction

The purpose of this article is to present a general purpose architecture for background processes. In this page, background process is used interchangeable with job.

Like any general purpose architecture, it cannot fullfil every single case since the requirements can be different across the different applications.

This architecture tries to address the requirements listed below and it should be adjusted accordingly whereas the specific case has a different set of constraint, requirements and wishes.

Requirements

In order to have a general purpose architecture that can be applied in a high number of scenarios, the following requirements should be satisfied.

Separation of concern

The business logic performed by the job must be loose coupled from the launcher logic.

The laucher component is responsible just to invoke a specific job.

The launcher can be either a scheduler that automatically triggers the job according to its configuration or manually through an explicit trigger action

Multiple Execution Environment

A general purpose architecture should allow to run the jobs either in a standalone environment or in a container.

Cluster awareness

In case jobs are deployed and launched within a container (e.g. within a web application), they must support those scenario where the deployment is performed in a cluster environment.

If a scheduler is used as a launcher component to trigger a job, it must keep in account that the job, according to its scheduling, must be triggered only once and not multiple times because of the multiple nodes.

 Description

The architecture is based on a launcher (trigger) component that is responsible to run the job.

Since there could be different way how a job can be launched, the launcher component should contains multiple implemetations such as

  1. REST API: If the launcher is running in a container a REST endpoind or any other protocol could be used to trigger the job remotely
  2. Command Line Interface (CLI): A command line interface could be useful to trigger a job (either on a local machine or remotely) using shell commands
  3. Internal Java API: Once the launcher component is deployed along with an enterprise application, it could be useful to expose the launcher API so that other Java components will have the capability to trigger jobs

All different launcher implementations finally could rely on the popular Quartz Scheduler to actually trigger a job.

Among many other features, it supports deployments in cluster environments where the scheduler runs on multiple nodes but still the jobs are triggered only once.

The job itself can be either a typical Spring Batch job or any generic background process.

GeneralPurposeArchitecture

 

 

 

 

 

 

Networking at home

Sometime you need to do things just to have fun and not becuase there is a real need.

This is the case of my home network that I’ve over complicated ON PURPOSE.

network-diagram

  • Two different private LAN (192.168.1.0/24, 10.0.0.0/24)
  • 8 Wifi networks
  • Internal DNS service to resolve hostname under subdomain (intranet.renatodelgaudio.com)
  • Internal DHCP service (Disabled the one provided by the router) that automatically update the entries in the DNS). Picture below
  • The two DNS machines are behind a clustered load balancer that redirect in Round Robin fashion the client’s  DNS queries.
  • Two NAS devices with over 9 TB available storage
  • Home server running 24×7 hosting around 20 VM with APC battery

 

Cluster of loadbalancer to server DNS queries (This is really over engineering and made only for fun!! )

DNS-cluster-loadbalancer