Pillai’s

Technology and Everything…..

Batch Processing Using Oracle BPEL

Posted by Shibu on December 2, 2007


Ever needed to create a BPEL process where you have to do batch processing? I will explain how you can create a simple BPEL process which will process a customer file and once all customer records are processed invoke next process. Figure on the left shows the overall process diagram.
Input is a simple csv file which has three fields, Customer Name, Address1 and Address2. Just create a simple file which has three records.
Now Create a partner link using File Reader Adapter. You should enable the “Files Contain Multiple Messages”. Choose a very high number (999999999) in the “Publish Messages in Batches of” field. like shown below

Now we have to create two new variables. One to hold the total number of records(count) and another one to keep track of the iterations (iterator). You can get the total record count using countNodes function. Use this function for you customer node (ora:countNodes(‘receiveInput_Read_InputVariable_1′,’Root-Element’,’/ns2:Root-Element/ns2:customer’)
Assign a constan value 1 to the iteration function. Next you have to create a while process. Use the expression ‘iterator'<=’count’ as your while condition. Now in the while process, you can assign each file record field to input variable of the next invoke process. Remember that when we enabled multiple messages to be loaded, BPEL will load all records into memmory array. So now you have to use iterator as index of the array to process record by record. You can use the getElement function for this purpose (ora:getElement(‘receiveInput_Read_InputVariable_1′,’Root-Element’,’/ns2:Root-Element/ns2:customer/ns2:Name’,
bpws:getVariableData(‘iterator’)))
See how I am using iterator as index value. This expression fetches Name element from the first node of Customer. Once all elements from the Customer nodes are copied to destination variables, increment ‘iterator’ by one. Now you can add your next process after the while loop, so that will be started after processing all file records.

2 Responses to “Batch Processing Using Oracle BPEL”

  1. Iswarya said

    Hi Shibu

    Just came across your blog on batch processing.

    I am attempting to do the same in a BPEL process and using SQL DB Adpater.

    It will be great if you can throw some light on it.

    I am trying to read a file which has some 2000 records and inserting into three different tables using flow activitiy.

    Its inserting all the records successfully into the tables but its just slow. I mean all the instances goes to ‘Manual Recovery’ and then appears in the Instance tab after some 5 mins.

    Can you please help me
    Thanks
    ISwarya

  2. Shibu said

    Iswarya,

    You can try one thing (this is what we did when we wanted to process more than 20000 records)

    We sued ESB to read the file and put records inot JMS Queue. We susbscribed queue in BPEL and processed records. Each record we had transaction_id that way we were able to identify they are from the same file.

    Another thing is debatching with “N” number of records. Say for example debatch with 50 records. If your file has 500 records then BPEL will create 10 instances.

    Hope this helps you.

    –Shibu

Leave a comment