Building Custom Readers and Writers in Spring Batch

In Spring Batch, the custom readers and writers are the components that you could create to read and write data in a selected way consistent with the application's requirements. These components are used in batch processing jobs to deal with the input and output of records.

• Custom Reader: A custom reader in Spring Batch is responsible for analyzing data from a data source and imparting it to the batch process for further processing. To create a custom reader, we need to implement the ItemReader interface.
  
• Custom Writer: A custom writer in Spring Batch is responsible for writing processed data to a destination. To create a custom writer, you need to implement the ItemWriter interface.

Key Differences between Custom Reader and Custom Writer

Example of Custom Reader and Custom Writer

Scenario: Imagine you are building a Spring Batch application to process articles from GeeksforGeeks. The articles are received in a custom JSON format from an external source. Your task is to read and process these articles, extract relevant information, and then store it in a database.

• Use of Custom ReaderProblem: The JSON format used by the external source for articles is custom and doesn't match the standard JSON reader provided by Spring Batch.
• Solution with Custom Reader: Create a custom reader that understands the custom JSON format and implements logic to read and parse the articles correctly.Implement any necessary data transformation or validation logic within the custom reader.
• Use of Custom WriterProblem: The processed article data needs to be stored in a database, and the standard JDBC writer may not provide the required flexibility.
• Solution with Custom Writer: Develop a custom writer that takes the processed article data and writes it to the database using a specific schema or business rules. Incorporate any additional steps, such as updating related tables or performing post-processing tasks within the custom writer.

Code Implementation for the above discussed example:

Custom Json Article Reader:

// CustomJsonArticleReader.java
import com.fasterxml.jackson.databind.ObjectMapper;
import org.springframework.batch.item.ItemReader;

import java.io.IOException;
import java.util.Iterator;
import java.util.List;

public class CustomJsonArticleReader implements ItemReader<Article> {

    private final Iterator<String> jsonArticlesIterator;
    private final ObjectMapper objectMapper = new ObjectMapper();

    public CustomJsonArticleReader(List<String> jsonArticles) {
        this.jsonArticlesIterator = jsonArticles.iterator();
    }

    @Override
    public Article read() {
        if (jsonArticlesIterator.hasNext()) {
            String jsonArticle = jsonArticlesIterator.next();

            try {
                // Custom logic to parse the JSON and convert it to an Article object
                return objectMapper.readValue(jsonArticle, Article.class);
            } catch (IOException e) {
                // Handle JSON parsing errors
                e.printStackTrace();
            }
        }
        return null;
    }
}

In this given code:

• CustomJsonArticleReader reads data from a list of JSON strings representing articles.
• It uses Jackson's ObjectMapper to deserialize the JSON into an Article object.
• Handle any necessary data transformation or validation logic within the read method.

Custom Database Article Writer:

// CustomDatabaseArticleWriter.java
import org.springframework.batch.item.ItemWriter;

import java.util.List;

public class CustomDatabaseArticleWriter implements ItemWriter<Article> {

    private final ArticleRepository articleRepository; // Assuming you have a Spring Data JPA repository for articles

    public CustomDatabaseArticleWriter(ArticleRepository articleRepository) {
        this.articleRepository = articleRepository;
    }

    @Override
    public void write(List<? extends Article> articles) throws Exception {
        for (Article article : articles) {
            // Custom logic to store the article in the database
            articleRepository.save(article);

            // Additional business logic or post-processing steps if needed
            // For example, updating related tables or triggering events
        }
    }
}

In the given code:

• CustomDatabaseArticleWriter implements the ItemWriter interface for processing and storing Article objects in a database.
• It uses a Spring Data JPA repository (ArticleRepository) to save the articles to the database.
• You can include additional business logic or post-processing steps within the write method.

Integrating Custom Readers and Writers in Spring Batch Job Configuration:

• Configure Dependencies: Ensure that you have the necessary dependencies in your project, including Spring Batch and any other required libraries. Also, make sure to have the appropriate database configuration if you are using a database.
• Create Custom Reader: Implement a custom reader (e.g., CustomJsonArticleReader) that understands the custom JSON format of GeeksforGeeks articles and reads data from an external source.
• Create Custom Writer: Implement a custom writer (e.g., CustomDatabaseArticleWriter) to handle the storage of processed articles, applying any necessary business logic or transformations before persisting the data.
• Configure Spring Batch Job: Create a Spring Batch job configuration class (e.g., BatchConfiguration) and define beans for your custom reader and writer. Inject any required dependencies, such as a database repository. Let's integrate the CustomJsonArticleReader and CustomDatabaseArticleWriter into a Spring Batch job configuration.

import org.springframework.batch.core.Job;
import org.springframework.batch.core.Step;
import org.springframework.batch.core.configuration.annotation.EnableBatchProcessing;
import org.springframework.batch.core.configuration.annotation.JobBuilderFactory;
import org.springframework.batch.core.configuration.annotation.StepBuilderFactory;
import org.springframework.batch.core.launch.support.RunIdIncrementer;
import org.springframework.batch.item.ItemReader;
import org.springframework.batch.item.ItemWriter;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;

@Configuration
@EnableBatchProcessing
public class BatchConfiguration {

    private final JobBuilderFactory jobBuilderFactory;
    private final StepBuilderFactory stepBuilderFactory;

    public BatchConfiguration(JobBuilderFactory jobBuilderFactory, 
                              StepBuilderFactory stepBuilderFactory) {
        this.jobBuilderFactory = jobBuilderFactory;
        this.stepBuilderFactory = stepBuilderFactory;
    }

    @Bean
    public ItemReader<Article> customJsonArticleReader() {
        List<String> jsonArticles = Arrays.asList(
                "{\"title\":\"Article 1\",\"content\":\"Content 1\"}",
                "{\"title\":\"Article 2\",\"content\":\"Content 2\"}"
        );
        return new CustomJsonArticleReader(jsonArticles);
    }

    @Bean
    public ItemWriter<Article> customDatabaseArticleWriter(ArticleRepository articleRepository) {
        return new CustomDatabaseArticleWriter(articleRepository);
    }

    @Bean
    public Step processArticlesStep(ItemReader<Article> customJsonArticleReader, ItemWriter<Article> customDatabaseArticleWriter) {
        return stepBuilderFactory.get("processArticlesStep")
                .<Article, Article>chunk(10)
                .reader(customJsonArticleReader)
                .writer(customDatabaseArticleWriter)
                .build();
    }

    @Bean
    public Job processArticlesJob(Step processArticlesStep) {
        return jobBuilderFactory.get("processArticlesJob")
                .incrementer(new RunIdIncrementer())
                .flow(processArticlesStep)
                .end()
                .build();
    }
}

• Define Processing Step: Configure a processing step within the job that uses the custom reader and writer. Set up the necessary chunk size and specify any additional processing logic if needed.
• Run the Job: Run the Spring Batch job, either through a scheduler, command-line interface, or any other triggering mechanism. Monitor the job execution to ensure that articles are processed and stored correctly.
• Test and Debug: Thoroughly test the integration, checking for proper data retrieval, processing, and storage. Debug any issues that arise during the job execution.