May 21, 2018

Multiple Data sources in Spring batch

Creating multiple data sources using Spring Batch:  Use case is read data from one data source and write data in another data source.  I believe Spring-batch is very good tool for implementing data pipeline such as ETL batch job.  I had this use case of running an ETL batch job nightly and Spring-boot-batch is one of the best option because we can do following:

  • Extract data from source data source
  • Transform Data
  • Load data in destination data source
  • Expose the transformed data as REST service
  • Schedule multiple ETL jobs in one application
Taking example of two data sources:  Make entry of two data sources in application.properties:

Create a database configuration class: I have named it DatabaseConfig:
@Configuration
public class DatabaseConfig {

    @ConfigurationProperties(prefix = "spring.datasource")
    @Bean(name= "sourceDb")
    public DataSource sourceDataSource() {
        return DataSourceBuilder.create().build();
    }
   
    @Bean(name = "destinationDb")
    @Primary
    @ConfigurationProperties(prefix="spring.batch.datasource")
    public DataSource destinationDataSource(){
      return DataSourceBuilder.create().build();
    }
}
public ItemReader itemReader(@Qualifier("sourceDb"final DataSource dataSource) {
             JdbcCursorItemReader<SourceTable> reader = new JdbcCursorItemReader<>();
            
             String FETCH_SQL_QUERY = " SELECT query"
              reader.setSql(FETCH_SQL_QUERY);
              reader.setDataSource(dataSource);
             // more code
}

public ItemWriter itemWriter(@Qualifier("destinationDb"final DataSource dataSource,
                    ItemPreparedStatementSetter<DestinationTable> setter) {
             JdbcBatchItemWriter<MessageUsage> writer = new JdbcBatchItemWriter<>();

             String INSERT_QUERY = "INSERT query";
            
             writer.setSql(INSERT_QUERY);
             writer.setDataSource(dataSource);
}
Share:

No comments

Post a Comment

Comment

© Technology Development with Java background | All rights reserved.
Blogger Template Crafted by pipdig