Spring Batch 2.0 - Part II - Flat File To Database

Part I of my Spring Batch blog ran through an example of a basic Spring Batch job. Now lets put together one that reads 200,000 rows from a flat file and inserts them into the database. The entire process took around 1 minute and 10 seconds to execute. That is pretty good time for Java-based batch processing. In all fairness I must point out that, this is relatively fast since, I am using a local MySQL database and there is no processing related logic being performed during the entire process.

The file is a comma separated file with format => receiptDate,memberName,checkNumber,checkDate,paymentType,depositAmount,paymentAmount,comments

The DDL for the database table:

 create table ledger (
 ID INT NOT NULL AUTO_INCREMENT,
 rcv_dt date,
 mbr_nm VARCHAR(100) not null,
 chk_nbr VARCHAR(10) not null,
 chk_dt date,
 pymt_typ VARCHAR(50) not null,
 dpst_amt double,
 pymt_amt double,
 comments VARCHAR(100),
 PRIMARY KEY (ID)
)



Here is the spring application context xml file...

<?xml version="1.0"encoding="UTF-8"?>

<beans xmlns="http://www.springframework.org/schema/beans"

      xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"xmlns:aop="http://www.springframework.org/schema/aop"

      xmlns:tx="http://www.springframework.org/schema/tx"xmlns:batch="http://www.springframework.org/schema/batch"

      xmlns:context="http://www.springframework.org/schema/context"

 

      xsi:schemaLocation="http://www.springframework.org/schema/beans www.springframework.org/schema/beans/spring-beans-2.0.xsd

class="MsoNormal" style="margin-bottom: 0.0001pt; line-height: normal;">    http://www.springframework.org/schema/tx www.springframework.org/schema/tx/spring-tx-2.0.xsd

    http://www.springframework.org/schema/aop www.springframework.org/schema/aop/spring-aop-2.0.xsd

class="MsoNormal" style="margin-bottom: 0.0001pt; line-height: normal;">    http://www.springframework.org/schema/batch  size="1">www.springframework.org/schema/batch/spring-batch-2.0.xsd

    http://www.springframework.org/schema/context href="http://www.springframework.org/schema/context/spring-context-2.5.xsd">www.springframework.org/schema/context/spring-context-2.5.xsd" >

 

      <!-- 1) USE ANNOTATIONS TO CONFIGURE SPRING BEANS -->

      <context:component-scan base-package="com.batch"/>

 

      <!-- 2)DATASOURCE, TRANSACTION MANAGER AND JDBC TEMPLATE  -->

      <bean id="dataSource"

            class="org.springframework.jdbc.datasource.DriverManagerDataSource">

            <property name="driverClassName"value="com.mysql.jdbc.Driver" />

            <property name="url"value="jdbc:mysql://localhost/seamdb" />

            <property name="username"value="root" />

            <property name="password"value="root" />

      </bean>

 

      <bean id="transactionManager"

            class="org.springframework.jdbc.datasource.DataSourceTransactionManage">

            <property name="dataSource"ref="dataSource" />

      </bean>

      <tx:annotation-driven transaction-manager="transactionManager"/>

 

      <bean id="jdbcTemplate"class="org.springframework.jdbc.core.JdbcTemplate">

            <property name="dataSource"ref="dataSource" />

      </bean>

 

 

      <!-- 3) JOBREPOSITORY - WE USE IN-MEMORY REPOSITORY FOR OUR EXAMPLE -->

      <bean id="jobRepository"

            class="org.springframework.batch.core.repository.support.MapJobRepositoryFactoryBean">

            <property name="transactionManager"ref="transactionManager" />

      </bean>

 

      <!-- 4)LAUNCH JOBS FROM A REPOSITORY -->

      <bean id="jobLauncher"

            class="org.springframework.batch.core.launch.support.SimpleJobLauncher">

            <property name="jobRepository"ref="jobRepository" />

      </bean>

 

      <!--

            5) Define the job and its steps. In our case I use onestep. Configure

            its readers and writers

      -->

      <batch:job id="simpleJob">

            <batch:listeners>

                  <batch:listener ref="appJobExecutionListener"/>

            </batch:listeners>

            <batch:step id="step1">

                  <batch:tasklet>

                        <batch:listeners>

                              <batch:listener ref="itemFailureLoggerListener"/>

                        </batch:listeners>

                        <batch:chunk reader="itemReader"writer="itemWriter"

                              commit-interval="1000" />

                  </batch:tasklet>

            </batch:step>

      </batch:job>

 

 

      <!--======================================================= -->

      <!--  6) READER -->

      <!--======================================================= -->

      <bean id="itemReader"class="org.springframework.batch.item.file.FlatFileItemReader">

            <property name="resource"value="classpath:com/batch/todb/ledger.txt"/>

            <!--property name="linesToSkip" value="1" /-->

            <property name="lineMapper">

                  <bean class="org.springframework.batch.item.file.mapping.DefaultLineMapper">

                        <property name="lineTokenizer">

                              <bean class="org.springframework.batch.item.file.transform.DelimitedLineTokenizer">

                                    <property name="names"

                                          value="receiptDate,memberName,checkNumber,checkDate,paymentType,depositAmount,paymentAmount,comments"/>

                              </bean>

                        </property>

                        <property name="fieldSetMapper"ref="ledgerMapper" />

                  </bean>

            </property>

      </bean>

 

      <bean id="inputFile"class="org.springframework.core.io.ClassPathResource">

            <constructor-arg value="com/batch/todb/ledger.txt"/>

      </bean>

</beans>



  • 1 through 4 are the same as the previous blog.
  • 5 - Defines the job and its steps. Also registers a job listener and a step listener
  • 6 - The reader used to read the flat file with comma separated columns. The FlatFileItemReader will read the rows from the flat file and pass it to the writer to persist to the database..
  • 8 - Not shown here is the item writer. It is configured using annotations and the class is LedgerWriter.
Now for some of the Java code.


//========================================================

// Ledger BEAN - Bean representing a single ledger

//========================================================

 

package com.batch.todb;

import java.util.Date;

public class Ledger {
    private int id;
    private Date receiptDate;
    private String memberName;
    private String checkNumber;
    private Date checkDate;
    private String paymentType;
    private double depositAmount;
    private double paymentAmount;
    private String comments;

    public int getId() {
        return id;
    }

    public void setId(int id) {
        this.id = id;
    }

    public Date getReceiptDate() {
        return receiptDate;
    }

    public void setReceiptDate(Date receiptDate) {
        this.receiptDate = receiptDate;
    }

    public String getMemberName() {
        return memberName;
    }

    public void setMemberName(String memberName) {
        this.memberName = memberName;
    }

    public String getCheckNumber() {
        return checkNumber;
    }

    public void setCheckNumber(String checkNumber) {
        this.checkNumber = checkNumber;
    }

    public Date getCheckDate() {
        return checkDate;
    }

    public void setCheckDate(Date checkDate) {
        this.checkDate = checkDate;
    }

    public String getPaymentType() {
        return paymentType;
    }

    public void setPaymentType(String paymentType) {
        this.paymentType = paymentType;
    }

    public double getDepositAmount() {
        return depositAmount;
    }

    public void setDepositAmount(double depositAmount) {
        this.depositAmount = depositAmount;
    }

    public double getPaymentAmount() {
        return paymentAmount;
    }

    public void setPaymentAmount(double paymentAmount) {
        this.paymentAmount = paymentAmount;
    }

    public String getComments() {
        return comments;
    }

    public void setComments(String comments) {
        this.comments = comments;
    }
}

 

//========================================================

// Ledger DAO - Used to persist ledgers to the ledger table

//========================================================

package com.batch.todb;

import java.sql.PreparedStatement;
import java.sql.SQLException;

import javax.sql.DataSource;

import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.jdbc.core.JdbcTemplate;
import org.springframework.jdbc.core.PreparedStatementSetter;
import org.springframework.stereotype.Component;
import org.springframework.transaction.annotation.Propagation;
import org.springframework.transaction.annotation.Transactional;

@Component
public class LedgerDAOImpl extends JdbcTemplate implements LedgerDAO {

    @Autowired
    public void setDataSource(DataSource dataSource) {
        super.setDataSource(dataSource);
    }

    @Transactional(propagation = Propagation.REQUIRED)
    public void save(final Ledger item) {
        super
                .update(
                        "insert into ledger (rcv_dt, mbr_nm, chk_nbr, chk_dt, pymt_typ, dpst_amt, pymt_amt, comments) values(?,?,?,?,?,?,?,?)",
                        new PreparedStatementSetter() {
                            public void setValues(PreparedStatement stmt)
                                    throws SQLException {
                                stmt.setDate(1, new java.sql.Date(item
                                        .getReceiptDate().getTime()));
                                stmt.setString(2, item.getMemberName());
                                stmt.setString(3, item.getCheckNumber());
                                stmt.setDate(4, new java.sql.Date(item
                                        .getCheckDate().getTime()));
                                stmt.setString(5, item.getPaymentType());
                                stmt.setDouble(6, item.getDepositAmount());
                                stmt.setDouble(7, item.getPaymentAmount());
                                stmt.setString(8, item.getComments());
                            }
                        });
    }
}

 

//========================================================

// Ledger WRITER - Performs db operations on a given list of ledger objects

//========================================================

package com.batch.todb;

import java.util.Iterator;
import java.util.List;

import org.springframework.batch.item.ItemWriter;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Component;

@Component("itemWriter")
public class LedgerWriter implements ItemWriter {

    @Autowired
    private LedgerDAO itemDAO;

    public void write(List items) throws Exception {
        for (Iterator<Ledger> iterator = items.iterator(); iterator.hasNext() ) {
            Ledger item = iterator.next();
            itemDAO.save(item);
        }
    }
}

 

//========================================================

// Ledger MAPPER - Maps a set of fields for a single record to the Ledger bean

//========================================================

package com.batch.todb;

import java.text.DecimalFormat;
import java.text.ParseException;

import org.springframework.batch.item.file.mapping.FieldSetMapper;
import org.springframework.batch.item.file.transform.FieldSet;
import org.springframework.stereotype.Component;
import org.springframework.stereotype.Service;

@Component("ledgerMapper")
public class LedgerMapper implements FieldSetMapper {
    private final static String DATE_PATTERN = "mm/DD/yy";
    private final static String DOLLAR_PATTERN = "$###,###.###";

    public Object mapFieldSet(FieldSet fs) {
        Ledger item = new Ledger();
        int idx = 0;
        item.setReceiptDate(fs.readDate(idx++, DATE_PATTERN));
        item.setMemberName(fs.readString(idx++));
        item.setCheckNumber(fs.readString(idx++));
        item.setCheckDate(fs.readDate(idx++, DATE_PATTERN));
        item.setPaymentType(fs.readString(idx++));

        // deposit amount
        try {
            DecimalFormat fmttr = new DecimalFormat(DOLLAR_PATTERN);
            Number number = fmttr.parse(fs.readString(idx++));
            item.setDepositAmount(number.doubleValue());
        } catch (ParseException e) {
            item.setDepositAmount(0);
        }

        // payment amount
        try {
            DecimalFormat fmttr = new DecimalFormat(DOLLAR_PATTERN);
            Number number = fmttr.parse(fs.readString(idx++));
            item.setPaymentAmount(number.doubleValue());
        } catch (ParseException e) {
            item.setPaymentAmount(0);
        }

        //
        return item;
    }
}

 



The test driver again is a JUnit class.

package com.batch.todb;

import org.apache.log4j.Logger;
import org.apache.log4j.PropertyConfigurator;
import org.junit.Before;
import org.junit.Test;
import org.junit.runner.RunWith;
import org.springframework.batch.core.Job;
import org.springframework.batch.core.JobParameters;
import org.springframework.batch.core.launch.JobLauncher;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.test.context.ContextConfiguration;
import org.springframework.test.context.junit4.AbstractTransactionalJUnit4SpringContextTests;
import org.springframework.test.context.junit4.SpringJUnit4ClassRunner;
import org.springframework.test.context.transaction.TransactionConfiguration;
import org.springframework.util.StopWatch;

@ContextConfiguration(locations = "classpath:com/batch/todb/contextToDB.xml")
@RunWith(SpringJUnit4ClassRunner.class)
@TransactionConfiguration(transactionManager = "transactionManager", defaultRollback = false)
public class ToDBBatchTestCase extends
        AbstractTransactionalJUnit4SpringContextTests {

    private final static Logger logger = Logger
            .getLogger(ToDBBatchTestCase.class);

    @Autowired
    private JobLauncher launcher;

    @Autowired
    private Job job;
    private JobParameters jobParameters = new JobParameters();

    @Before
    public void setup() {
        PropertyConfigurator
                .configure("c:/mathew/springbatch2/src/com/batch/log4j.properties");
    }

    @Test
    public void testLaunchJob() throws Exception {
        StopWatch sw = new StopWatch();
        sw.start();
        launcher.run(job, jobParameters);
        sw.stop();
        logger.info(">>> TIME ELAPSED:" + sw.prettyPrint());

    }

    @Autowired
    public void setLauncher(JobLauncher bootstrap) {
        this.launcher = bootstrap;
    }

    @Autowired
    public void setJob(Job job) {
        this.job = job;
    }
}



Running the test case will insert approx 200k rows into the ledger table. It took roughly 1:12 seconds for the entire process to execute.

INFO ToDBBatchTestCase:46 - >>> TIME ELAPSED:StopWatch '': running time (millis) = 71678

Next move over to Spring Batch - Part III - From Database to Flat File

Please see Part III to download entire project file with dependencies
 
 
  
  

 del.icio.us  Stumbleupon  Technorati  Digg 

Posted by Mathew Thomas at 2/2/2009 11:16 PM
Categories: Spring
 

What did you think of this article?




Trackbacks
Trackback specific URL for this entry
  • Trackbacks are closed for this entry.
Comments
Display comments as (Linear | Threaded)

  • 8/28/2009 12:55 PM Exult wrote:
    You tutorial is excellent.

    Can you please out the required jar files and NoteaDAO interface?

    I having bigtime trouble running the code

    Thanks
    Reply to this
    1. 8/30/2009 7:58 AM Mathew Thomas wrote:
      Please refer to Part III of the series. I have linked in a jar file with the entire project including libraries. Note that I am using an older version of Spring Batch. I have yet to update this with the latest version. Maybe tonight

      Reply to this
      1. 10/23/2009 5:39 AM fatih wrote:
        I downloaded all code but could not find where ledger.txt ? Can you help?
        Reply to this
    2. 10/23/2009 5:49 AM fatih wrote:
      I found it This sample is excellent.Thank you very much
      Reply to this
Leave a comment

Submitted comments will be subject to moderation before being displayed.

 Enter the above security code (required)

 Name

 Email (will not be published)

 Website

Your comment is 0 characters limited to 3000 characters.