Multiple Data sources in Spring batch

Creating multiple data sources using Spring Batch:  Use case is read data from one data source and write data in another data source.  I believe Spring-batch is very good tool for implementing data pipeline such as ETL batch job.  I had this use case of running an ETL batch job nightly and Spring-boot-batch is one of the best option because we can do following:

  • Extract data from source data source
  • Transform Data
  • Load data in destination data source
  • Expose the transformed data as REST service
  • Schedule multiple ETL jobs in one application
Taking example of two data sources:  Make entry of two data sources in application.properties:

Create a database configuration class: I have named it DatabaseConfig:
@Configuration
public class DatabaseConfig {

    @ConfigurationProperties(prefix = "spring.datasource")
    @Bean(name= "sourceDb")
    public DataSource sourceDataSource() {
        return DataSourceBuilder.create().build();
    }
   
    @Bean(name = "destinationDb")
    @Primary
    @ConfigurationProperties(prefix="spring.batch.datasource")
    public DataSource destinationDataSource(){
      return DataSourceBuilder.create().build();
    }
}

The Data source marked with primary will be used to create the meta tables.  Spring framework creates meta tables in DB to create, manage, track, etc. batch jobs. 
public ItemReader itemReader(@Qualifier("sourceDb"final DataSource dataSource) {
             JdbcCursorItemReader<SourceTable> reader = new JdbcCursorItemReader<>();
            
             String FETCH_SQL_QUERY = " SELECT query"
              reader.setSql(FETCH_SQL_QUERY);
              reader.setDataSource(dataSource);
             // more code
}

public ItemWriter itemWriter(@Qualifier("destinationDb"final DataSource dataSource,
                    ItemPreparedStatementSetter<DestinationTable> setter) {
             JdbcBatchItemWriter<MessageUsage> writer = new JdbcBatchItemWriter<>();

             String INSERT_QUERY = "INSERT query";
            
             writer.setSql(INSERT_QUERY);
             writer.setDataSource(dataSource);
}

Comments

Popular Posts