Java client to publish 5 million events to Universal Messaging channel\IS Publishable document

This is a continuation to the previous blog where I have shown how to publish a protobuf event from Java to IBM Universal Messaging channel which is also a publishable document on IBM webMethods Integration Server.

This whole process took about 4 minutes to publish 5 million records to the UM channel with the specifications that I have described in specification section. On processing end, the processing time for 5 million records will depend on the number of threads you have configured on the webMethods messaging trigger and the number of wM Integration servers in a cluster. I have tested this with a 2 node webMethods Integration Server cluster, and it took about 5 minutes to process the 5 million records.

The more webMethods Integration Servers you have in the cluster and more the number of threads working in parallel processing the messages faster.

You can Download the PDF with detailed step-by-step instructions along with the full code.

Note: Feel free to reach out to aarremreddy@silverspringtech.com for any questions or inquiries.

Proto file: message test_Akshith_docs_SalesDataDoc { optional test_Akshith_docs_SalesDataDoc_salesData salesData = 1; optional pub_publish_envelope _env = 536870910; } message test_Akshith_docs_SalesDataDoc_salesData { optional string region = 1; optional string country = 2; optional string itemType = 3; optional string salesChannel = 4; optional string orderPriority = 5; optional string orderDate = 6; optional string orderId = 7; optional string shipDate = 8; optional string unitsSold = 9; optional string unitPrice = 10; optional string unitCost = 11; optional string totalRevenue = 12; optional string totalCost = 13; optional string totalProfit = 14; } message pub_publish_envelope { optional string activation = 1; optional int32 appLastSeqn = 2; optional string appPassword = 3; optional int32 appSeqn = 4; optional string appUserName = 5; optional int32 priority = 6; optional string businessContext = 7; repeated int32 controlLabel = 8; optional string errorsTo = 9; optional string errorRequestsTo = 10; optional string locale = 11; optional int32 maxResults = 12; optional string replyTo = 13; optional int32 runLevel = 14; optional bytes signature = 15; optional string signatureType = 16; optional int32 startResult = 17; optional int32 tag = 18; optional string trackId = 19; optional string transactionId = 20; optional string transformState = 21; optional int32 age = 22; optional string connectionIntegrity = 23; optional string destId = 24; optional int64 enqueueTime = 25; optional string logBroker = 26; optional string logHost = 27; optional string pubDistinguishedName = 28; optional string pubId = 29; optional bytes pubNetAddr = 30; optional int64 pubSeqn = 31; repeated int32 pubLabel = 32; optional int64 recvTime = 33; repeated pub_publish_envelope_route route = 34; optional string uuid = 35; optional string status = 36; } message pub_publish_envelope_route { optional string broker = 1; optional int64 recvTime = 2; optional int64 enqueueTime = 3; } //Read File Line By Line while ((strLine = br.readLine()) != null) { tempData=new SalesData(); String[] tempList=strLine.split(","); tempData.setRegion(tempList[0]); tempData.setCountry(tempList[1]); tempData.setItemType(tempList[2]); tempData.setSalesChannel(tempList[3]); tempData.setOrderPriority(tempList[3]); tempData.setOrderDate(tempList[4]); tempData.setOrderId(tempList[5]); tempData.setShipDate(tempList[6]); tempData.setUnitsSold(tempList[7]); tempData.setUnitPrice(tempList[8]); tempData.setUnitCost(tempList[9]); tempData.setTotalRevenue(tempList[10]); tempData.setTotalCost(tempList[11]); tempData.setTotalProfit(tempList[12]); nProtobufEvent evt=UMUtilities.SalesDataToProtoEvt(descriptorFileBytes,tempData); myChannel.publish(evt); evt=null; }