How to create asynchronous and iterable methods with failover support

When developing applications, we need to make some processing more robust and less fault-tolerant, especially when requesting remote services that may be down for a long period of time.

In this article, we’ll introduce a new framework that aims to provide declarative non-blocking retry support for methods in Spring-based applications using annotations.

There are two main implementations of this framework:

  1. thread pool task-based implementation: it is based on implementation ThreadPoolTaskExecutor without keeping the executor thread busy during the entire retry processing, as opposed to concatenation @Async (see “Spring Boot – Async Methods”) and @Retryable (See “Retry Handling with Spring-Retry”). Indeed, when using the traditional spring retry Annotations, the thread that runs the method performs the full retry policy, including the waiting period, and remains busy until the end. For example, if ThreadPoolTaskExecutor There are 10 threads with a retry policy which can take 5 minutes, and the application receives 30 requests, only 10 requests can be processed simultaneously and the others will be blocked for the full 5 minutes. So the execution of 30 requests may take 15 minutes.
  2. Quartz job based implementation: This implementation is based on the Quartz library. If configured with JDBC JobStore it supports load-balancing, failover, and persistence. This means that even if one node in the cluster is down, others can handle the operation and retry.

basic concept

Comment

To make a method iterable, you can annotate it with @AsyncRetryable and specify the following attributes:

  • retryPolicy (Mandatory): Policy bean name that defines the next retry time if an exception is thrown during method execution
  • retryFor (Optional): List of exceptions that should be retried
  • noRetryFor (Optional): List of exceptions that should not be retried
  • retryListener (Optional): The listener bean name that triggers events related to the annotated method execution

The following example shows how to use @AsyncRetryable In a declarative style:

@Bean
public class Service {
    
    @AsyncRetryable(retryPolicy = "fixedWindowRetryableSchedulingPolicy", 
                    retryFor = IOException.class, 
                    noRetryFor={ArithmeticException.class}, 
                    retryListener = "retryListener")
    public void method(String arg1, Object arg2){
        // ...do something
    }
}

In this example, if type. exception of IOException method is thrown during perform() Execution will be retried as per policy fixedWindowRetryableSchedulingPolicy, if exception of type ArithmeticException is thrown, no retry will be made.

All events that occurred during the method call are notified to the bean listener. retryListener,

retry policy

The retry policy defines the next execution time for each failed execution. Indeed, when the annotated method throws a retryable exception, the retry policy bean is called to obtain the waiting period before calling the method again.

This framework provides three basic implementations:

  1. FixedWindowRetryableSchedulingPolicy: This policy is used for retrying with a fixed waiting period and maximum attempt limit.
  2. StaticAsyncRetryableSchedulingPolicy: This policy accepts a series of waiting periods for each attempt.
  3. LinearAsyncRetryableSchedulingPolicy: This policy each time multiplies the previous waiting period by a factor to extend the duration of the next one. The coefficient default value is 2.

The following example shows how to configure a FixedWindowRetryableSchedulingPolicy which will trigger the annotated method the first time in 10 seconds, then do 3 retries within a waiting period of 20 seconds each.

@Configuration
public class AsyncRetryConfiguration {
    
    ...

    @Bean
    public FixedWindowRetryableSchedulingPolicy fixedWindowRetryableSchedulingPolicy() {
        return new FixedWindowRetryableSchedulingPolicy(10000,3,20000);
    }
}

CommentIt is possible to customize the retry policy by implementing the :interface AsyncRetryableSchedulingPolicy,

try again listener

The retry listener is used to detect events during the retry processing lifecycle. AsyncRetryableListener The interface is defined as below:

public interface AsyncRetryableListener<T> {

    void beforeRetry(Integer retryCount, Object[] args);

    void afterRetry(Integer retryCount,T result, Object[] args, Throwable e);

    void onRetryEnd(Object[] args, Throwable e);

}

Methods beforeRetry() And afterRetry() are triggered respectively before and after the call of the annotated method. process onRetryEnd() The retry is triggered at the end of the process. The methods defined above are called neater, whether the annotated method succeeds or fails.

The characteristics of the methods are:

  • retryCount: current retry number
  • result: the value returned by the annotated method in case of success
  • args:annotated method arguments values ​​in the same order
  • e: Exception thrown during the execution of annotated method; This value is null if the method is executed successfully.

how to use it

thread pool task-based implementation

To use the asynchronous retry feature based on the Spring Thread Pool Task Scheduler, all you need to do is add the following dependencies:

<dependency>
     <artifactId>async-retry-spring-scheduler</artifactId>
     <groupId>org.digibooster.retryable</groupId>
     <version>1.0.2</version>
</dependency>

add annotation EnableThreadPoolBasedAsyncRetry For a configuration class, and finally, define the retry policy bean as follows:

@Configuration
@EnableThreadPoolBasedAsyncRetry
public class AsyncRetryConfiguration {

    /**
    * the annotated method will be triggered the first time after 1 second and will
    * perform 2 retries eatch 20 seconds in case of failure
    */
    @Bean
    public FixedWindowRetryableSchedulingPolicy fixedWindowRetryableSchedulingPolicy() {
        return new FixedWindowRetryableSchedulingPolicy(10000,3,20000);
    }
}

Quartz-based implementation (in memory)

This configuration uses RAM to store retry tasks. It is not permanent and does not support load-balancing and failover. So the retry will be lost when the server is restarted.

To use this implementation, add the following dependencies:

<dependency>
	<groupId>org.springframework.boot</groupId>
	<artifactId>spring-boot-starter-quartz</artifactId>
</dependency>
<dependency>
	<groupId>org.quartz-scheduler</groupId>
	<artifactId>quartz</artifactId>
</dependency>
<dependency>
	<artifactId>async-retry-quartz-scheduler</artifactId>
	<groupId>org.digibooster.retryable</groupId>
	<version>1.0.2</version>
</dependency>

Add a Configuration class that extends DefaultQuartzBasedAsyncRetryableConfigAdapter,

@Configuration
public class RetryAsyncQuartzInMemoryConfiguration extends DefaultQuartzBasedAsyncRetryableConfigAdapter {

    @Bean
    public FixedWindowRetryableSchedulingPolicy fixedWindowRetryableSchedulingPolicy() {
        return new FixedWindowRetryableSchedulingPolicy(10000,3,20000);
    }

    @Bean("schedulerFactoryBean")
    public SchedulerFactoryBean schedulerFactoryBean(@Autowired QuartzSchedulerJobFactory quartzSchedulerJobFactory,
                                                     @Autowired QuartzProperties quartzProperties) {
        SchedulerFactoryBean factory = new SchedulerFactoryBean();
        Properties properties = new Properties();
        properties.putAll(quartzProperties.getProperties());
        factory.setQuartzProperties(properties);
        factory.setJobFactory(quartzSchedulerJobFactory);
        return factory;
    }

}

Finally, add the following lines to the application.yml file:

spring:
  quartz:
    auto-startup: true
    job-store-type: memory

Quartz-Based Implementation (JDBC)

This configuration uses a database to store retry tasks. It is persistent and supports load-balancing and failover, so retries will not be lost when the server is restarted.

To use this implementation, add the following dependencies:

<dependency>
	<groupId>org.springframework</groupId>
	<artifactId>spring-tx</artifactId>
</dependency>
<dependency>
	<groupId>org.springframework.boot</groupId>
	<artifactId>spring-boot-starter-quartz</artifactId>
</dependency>
<dependency>
	<groupId>org.quartz-scheduler</groupId>
	<artifactId>quartz</artifactId>
</dependency>
<dependency>
	<artifactId>async-retry-quartz-scheduler</artifactId>
	<groupId>org.digibooster.retryable</groupId>
	<version>1.0.2</version>
</dependency>

Add a Configuration class that extends QuartzDBBasedAsyncRetryableConfigAdapter as follows:

@Configuration
public class ConfigurationClass extends QuartzDBBasedAsyncRetryableConfigAdapter {

    @Autowired
    PlatformTransactionManager transactionManager;

    @Override
    public PlatformTransactionManager getTransactionManager() {
        return transactionManager;
    }

    @Bean
    public FixedWindowRetryableSchedulingPolicy fixedWindowRetryableSchedulingPolicy() {
        return new FixedWindowRetryableSchedulingPolicy(10000,3,20000);
    }

    @Bean("schedulerFactoryBean")
    public SchedulerFactoryBean schedulerFactoryBean(@Autowired QuartzSchedulerJobFactory quartzSchedulerJobFactory,
                                                     @Autowired QuartzProperties quartzProperties,
                                                     @Autowired DataSource dataSource) {
        SchedulerFactoryBean factory = new SchedulerFactoryBean();
        Properties properties = new Properties();
        properties.putAll(quartzProperties.getProperties());
        factory.setQuartzProperties(properties);
        factory.setJobFactory(quartzSchedulerJobFactory);
        factory.setDataSource(dataSource);
        return factory;
    }

}

Finally, add the following configuration to the application.yml file:

spring:
  quartz:
    auto-startup: true
    job-store-type: jdbc
    properties:
      org.quartz.jobStore.isClustered: true
      org.quartz.scheduler.instanceName: RetryInstance # optional
      org.quartz.scheduler.instanceId: AUTO # optional
    jdbc:
      initialize-schema: always # optional

Comment: When using Quartz with a database, a delay will be caused by the Quartz implementation. To reduce the delay, you can change the property value org.quartz.jobStore.clusterCheckinInterval, The framework source code is published on GitHub.

Leave a Comment