May 21, 2018

Multiple Data sources in Spring batch

Creating multiple data sources using Spring Batch:  Use case is read data from one data source and write data in another data source.  I believe Spring-batch is very good tool for implementing data pipeline such as ETL batch job.  I had this use case of running an ETL batch job nightly and Spring-boot-batch is one of the best option because we can do following:

  • Extract data from source data source
  • Transform Data
  • Load data in destination data source
  • Expose the transformed data as REST service
  • Schedule multiple ETL jobs in one application
Taking example of two data sources:  Make entry of two data sources in

Create a database configuration class: I have named it DatabaseConfig:
public class DatabaseConfig {

    @ConfigurationProperties(prefix = "spring.datasource")
    @Bean(name= "sourceDb")
    public DataSource sourceDataSource() {
        return DataSourceBuilder.create().build();
    @Bean(name = "destinationDb")
    public DataSource destinationDataSource(){
      return DataSourceBuilder.create().build();
public ItemReader itemReader(@Qualifier("sourceDb"final DataSource dataSource) {
             JdbcCursorItemReader<SourceTable> reader = new JdbcCursorItemReader<>();
             String FETCH_SQL_QUERY = " SELECT query"
             // more code

public ItemWriter itemWriter(@Qualifier("destinationDb"final DataSource dataSource,
                    ItemPreparedStatementSetter<DestinationTable> setter) {
             JdbcBatchItemWriter<MessageUsage> writer = new JdbcBatchItemWriter<>();

             String INSERT_QUERY = "INSERT query";

No comments

Post a Comment


© Technology Strategy, Architecture, and Development | All rights reserved.
Blogger Template Crafted by pipdig