Spring Batch OOM error - out-of-memory

I am using Spring Batch framework for reading the huge file using Partitioner framework. The sample code is attached.
If the file records are going more than 1000 then I am getting Outofmemory error due to ConcurrentHashMap handling in execution context.
public Map<String, ExecutionContext> partition(int gridSize) {
System.out.println("inside Partition" +inboundDir);
// Map<String, ExecutionContext> partitionMap = new HashMap<String, ExecutionContext>();
File file = new File(inboundDir);
System.out.println("is directory"+file.isDirectory());
Map<String, ExecutionContext> queue = new HashMap<>();
try {
List<List<String>> trunks = new ArrayList<>();
System.out.println("Inside File"+file);
// read and store data to a list of trunk
int chunkSize = 1;
int count = 1;
int totalCount=0;
try (BufferedReader br = new BufferedReader(new FileReader(file))) {
String line;
List items = new ArrayList();
System.out.println("chunkSize"+chunkSize);
System.out.println("count"+count);
while ((line = br.readLine()) != null) {
totalCount=totalCount+1;
System.out.println("line---------"+count % chunkSize);
if (count % chunkSize == 0) {
trunks.add(items);
items = new ArrayList();
System.out.println("trunks---------"+trunks.size());
}
System.out.println("items---------"+line);
items.add(line);
count++;
System.out.println("count---------"+count);
}
if(items.size()>0){
trunks.add(items);
}
System.out.println("inside total count"+totalCount);
for (int i=2; i<trunks.size(); i++) {
ExecutionContext value = new ExecutionContext();
value.put("fileResource", trunks.get(i));
value.put("totalCount", totalCount);
queue.put("trunk"+i+file, value);
}
}catch(FileNotFoundException e1){
e1.printStackTrace();
}
}
catch (Exception e) {
e.printStackTrace();
}
return queue;
}

Related

How to save objects in a proper way with the stream writer?

In the program.cs the user is asked if he wanna read the data, if he types y then the method Doc.ReadDoc starts is there any proper way:
class Program
{
static void Main(string[] args)
{
do
{
var path = "C:\\Users\\ks\\Desktop\\C#";
string fileName = path + #"\TestFile.txt";
Console.WriteLine("Do you want to read it? y/n");
string yesorno = Console.ReadLine();
if (yesorno=="y")
{
Console.Clear();
Doc.ReadDoc();
}
Console.WriteLine("Which type of vehicle");
string type = Console.ReadLine();
Console.WriteLine("how many tires");
int raeder = Convert.ToInt32( Console.ReadLine());
var Vehicle = new Used_Cars(type, raeder);
Doc.Write(Vehicle);
} while (true);
}
}
The Class with the methods (Read, Write):
public static List<string> ReadDoc()
{
var list = new List<string>();
var pfad = "C:\\Users\\ks\\Desktop\\C#";
string fileName = path+ #"\TestFile.txt";
try
{
using (StreamReader sr = new StreamReader(fileName))
{
Console.WriteLine("Data found");
string line;
Console.WriteLine(sr.ReadToEnd());
}
}
catch (Exception e)
{
// Let the user know what went wrong.
Console.WriteLine("Data not found");
Console.WriteLine(e.Message);
list = null;
}
return list;
}
And the last Method is the Write method, is this a good code to save properties in a file? How could i stop the program with ESC or smth like that, so if the user presses ESC it should stop.
public static void Write(Used_Cars vehicle)
{
var pfad = "C:\\Users\\ks\\Desktop\\C#";
string fileName = path+ #"\TestFile.txt";
Console.WriteLine("Is it correct?");
Console.WriteLine("y/n");
string yeahorno= Console.ReadLine();
if (jaodernein == "y")
{
try
{
using (StreamWriter writer = new StreamWriter(fileName))
{
writer.WriteLine(vehicle.Vehicle);
writer.WriteLine(vehicle.Wheels);
Console.WriteLine();
}
}
catch (Exception exp)
{
Console.Write(exp.Message);
}
}
}

NumberFormatException when using FileReader and StringTokenizer

The issue I am having is with the "Office" String. I get this: java.lang.NumberFormatException: For input string: "Office:" I feel like I need to do something else with the parseline down in the tokenizer section? Am I on the right track? I am basically trying to read from a file than I will calculate total sales and write to another file. This is the error I get even when I try to display my people.txt file to an output screen on my GUI. I just need a little advice to where to look to fix this. I have looked up many things but have not came close.
public class PersonReader {
public static void main(String args[]) throws IOException {
PersonReader reader = new PersonReader();
List<person> people = reader.readPeople("people.txt");
System.out.println(people);
}
public List<person> readPeople(String filename) throws IOException {
File f = new File(filename);
FileReader reader = new FileReader(f);
BufferedReader breader = new BufferedReader(reader);
List<person> people = new ArrayList<person>();
String line = breader.readLine();
while (line != null) {
person p = null;
try {
p = parseLine(line);
} catch (Exception e) {
e.printStackTrace();
}
if (p == null) {
System.out.println("This row is bad." + line);
} else {
people.add(p);
}
line = breader.readLine();
}
return people;
}
private static person parseLine(String line) {
int repID;
String firstName;
String lastName;
double books;
double paper;
double office;
String district;
String contact;
String next;
StringTokenizer st = new StringTokenizer(line, ", ");
repID = Integer.parseInt(st.nextToken().trim());
firstName = st.nextToken().trim();
lastName = st.nextToken().trim();
books = Double.parseDouble(st.nextToken().trim());
parseLine(line);
paper = Double.parseDouble(st.nextToken().trim());
parseLine(line);
office = Double.parseDouble(st.nextToken().trim());
parseLine(line);
district = st.nextToken().trim();
parseLine(line);
contact = st.nextToken().trim();
parseLine(line);
if (repID < 1) {
return null;
}
if (firstName.length() == 0) {
return null;
}
if (lastName.length() == 0) {
return null;
}
if (books < 1) {
return null;
}
if (paper < 1) {
return null;
}
if (office < 1) {
return null;
}
if (district.length() == 0) {
return null;
}
if (contact.length() == 0) {
return null;
}
person p = new person();
p.setRepID(repID);
p.setFirstName(firstName);
p.setLastName(lastName);
p.setBooks(books);
p.setPaper(paper);
p.setOffice(office);
p.setDistrict(district);
p.setContact(contact);
return p;
}
}

how to refer to a non-declared variables in setResultConverter

I'd like to re-use a dialog class for data manipulation. The data will be retrieved from database. It depends on which table the class retrieve the data from, the size of the table columns is not fixed, so I can't declare column variables. After users update data, I would like to convert the input data using setResultConverter but do not know how to refer to the variable, since the program generates TextFields dynamically. Please help. Here is the the code.
public class AddDialog {
private Dialog<DBtable> dialog = new Dialog<DBtable>();
private ButtonType saveBtn;
//database variables
private Connection connect; // = null;
private String dbTblName;
//gridpane content variables
private GridPane contentPane = new GridPane();
private HashMap<String, TextField> fieldMap =
new HashMap<String, TextField>();
private ArrayList<String> dataList = new ArrayList<String>();
public AddDialog (String title, String header, String dbTable) {
this.dbTblName = dbTable;
dialog.setTitle(title);
dialog.setHeaderText(header);
saveBtn = new ButtonType("Save", ButtonData.OK_DONE);
dialog.getDialogPane().getButtonTypes().addAll(saveBtn,
ButtonType.CANCEL);
dialog.getDialogPane().setContent(getLayout(dbTable));
Optional<DBtable> result = dialog.showAndWait();
result.ifPresent(data -> {
System.out.println(" data="+data+" 0="+data.getID()+
" 1="+data.getField1());
});
} // constructor ends
public GridPane getLayout(String dbTable) {
String sql = "select column_name, description ";
sql += "from syscolumn_description ";
sql += "where table_name = \'" + dbTable + "\'";
String fieldLabel, fieldCol;
ResultSet ds = null;
// retrieve meta data from database
connect = DBConnect.getConnect(connect);
try {
Statement labelStmnt = connect.createStatement();
ds = labelStmnt.executeQuery(sql);
int row = 0;
while (ds.next()) {
row += 2;
//label....column=0 row=row+2;
fieldLabel = ds.getString("DESCRIPTION");
contentPane.add(new Text(fieldLabel), 0, row);
//textField...column=1 row=row+2;
contentPane.add(new TextField(), 1, row);
fieldCol = ds.getString("COLUMN_NAME");
fieldMap.put(fieldCol, new TextField());
} // while result set loop ends
} catch (Exception e) {
e.printStackTrace();
} finally {
try {if(ds != null) ds.close();} catch (Exception e) {};
}
// convert result
dialog.setResultConverter(dialogButton -> {
if (dialogButton == saveBtn) {
int i=0;
for (Map.Entry<String, TextField> e : fieldMap.entrySet()) {
dataList.add(e.getValue().getText());
i++;
System.out.println("col="+e.getKey()+
" data="+e.getValue().getText());
} // map loop end
return new DBtable(dataList, i);
}
return null;
});
return contentPane;
} //getLayout ends
} // AddDialog ends

Calling Stored procedure inside a thread to update multiple records

I am trying to calla stored procedure for various unique entities . The stored procedure for a single entity takes about 33 secs. So I decided to call it using threads.
Here are some of trials I have done :
public bool ExecuteTaxRateLinkingParallel(int mapID, int createdBy)
{
try
{
int snapshotID = (int)(HttpContext.Current.Session[GlobalConstant.snapShotID]);
List<TaxEntity> taxEntities = new List<TaxEntity>();
List<Task> tasks = new List<Task>();
using (var ctx = new TopazDbContainer())
{
taxEntities = ctx.TaxEntities.AsParallel().Where(t => t.IsActive == true).ToList<TaxEntity>();
}
Parallel.ForEach<TaxEntity>(taxEntities, (entity) =>
{
//SqlConnection connection; SqlTransaction trans; SqlCommand command;
// break this into pieces of 5
var task = Task.Factory.StartNew(() =>
{
using (var pctx = new TopazDbContainer())
{
try
{
int taxEntityID = entity.TaxEntityID;
pctx.CommandTimeout = 5000;
//string connectionString = System.Configuration.ConfigurationManager.ConnectionStrings["TOPAZDBConnectionStringParallel"].ConnectionString;
//connection = new SqlConnection(connectionString);
//command = new SqlCommand("dbo.[Usp_TaxRatesLinkingParallel]", connection);
//trans = connection.BeginTransaction();
//command.CommandType = CommandType.StoredProcedure;
//command.Parameters.AddWithValue("#MapID", mapID);
//command.Parameters.AddWithValue("#UserID", createdBy);
//command.Parameters.AddWithValue("#TaxEntityID", taxEntityID);
//command.Parameters.AddWithValue("#SnapshotID", snapshotID);
//connection.Open();
//command.CommandTimeout = 5000;
//command.ExecuteReader().AsParallel();
pctx.ContextOptions.LazyLoadingEnabled = true;
//pctx.ExecuteStoreCommand("Exec [Usp_TaxRatesLinkingParallel] #MapID={0},#UserID={1},#TaxEntityID={2},#SnapshotID{3}", new SqlParameter("MapID", mapID), new SqlParameter("UserID", createdBy), new SqlParameter("TaxEntityID", taxEntityID), new SqlParameter("SnapshotID", snapshotID));
var param = new DbParameter[] { new SqlParameter("UserID", createdBy), new SqlParameter("TaxEntityID", taxEntityID), new SqlParameter("SnapshotID", snapshotID) };
pctx.ExecuteStoreCommand("Exec [Usp_TaxRatesLinkingParallel] #MapID,#UserID,#TaxEntityID,#SnapshotID", param);
//var result = output.FirstOrDefault();
}
catch (TaskCanceledException tx)
{
}
catch (Exception e)
{
}
finally
{
pctx.SaveChanges();
pctx.Connection.Close();
}
}
}, TaskCreationOptions.PreferFairness);
tasks.Add(task);
try
{
Task.WaitAll(tasks.ToArray());
}
catch (AggregateException ae)
{
ae.Handle((x) =>
{
if (x is UnauthorizedAccessException)
{
return true;
}
else
{
return false;
}
});
}
catch (Exception ex)
{
throw ex;
}
});
return true;
}
catch (Exception ex)
{
TopazErrorLogs.AddTopazErrorLogBL(ex, 1, 1);
throw new TopazCustomException(GlobalConstant.errorMessage);
}
}
For some the above statements the SP seems like it runs fine but when I check from the application or from backend the records doesn't get updated.
Need help!
If you are not on .NET 4.5 yet, you can use these extension methods to execute your commands async.
using System.Diagnostics.Contracts;
using System.Threading.Tasks;
using System.Xml;
namespace System.Data.SqlClient
{
public static class SqlCommandExtensions
{
public static Task<SqlDataReader> ExecuteReaderAsync(this SqlCommand command)
{
Contract.Requires(command != null);
return ExecuteReaderAsync(command, null);
}
public static Task<SqlDataReader> ExecuteReaderAsync(this SqlCommand command, object state)
{
Contract.Requires(command != null);
return Task.Factory.FromAsync<SqlDataReader>(command.BeginExecuteReader, command.EndExecuteReader, state);
}
public static Task<XmlReader> ExecuteReaderXmlAsync(this SqlCommand command)
{
Contract.Requires(command != null);
return ExecuteReaderXmlAsync(command, null);
}
public static Task<XmlReader> ExecuteReaderXmlAsync(this SqlCommand command, object state)
{
Contract.Requires(command != null);
return Task.Factory.FromAsync<XmlReader>(command.BeginExecuteXmlReader, command.EndExecuteXmlReader, state);
}
public static Task<int> ExecuteNonQueryAsync(this SqlCommand command)
{
Contract.Requires(command != null);
return ExecuteNonQueryAsync(command, null);
}
public static Task<int> ExecuteNonQueryAsync(this SqlCommand command, object state)
{
Contract.Requires(command != null);
return Task.Factory.FromAsync<int>(command.BeginExecuteNonQuery, command.EndExecuteNonQuery, state);
}
}
}
It is not an asynchronous database query that you are doing here. Please have a look:
Asynchronous Database Calls With Task-based Asynchronous Programming Model (TAP) in ASP.NET MVC 4
Here is an example of an asynchronous database call with new async / await features:
public async Task<IEnumerable<Car>> GetCarsAsync() {
var connectionString =
ConfigurationManager.ConnectionStrings["CarGalleryConnStr"].ConnectionString;
var asyncConnectionString = new SqlConnectionStringBuilder(connectionString) {
AsynchronousProcessing = true
}.ToString();
using (var conn = new SqlConnection(asyncConnectionString)) {
using (var cmd = new SqlCommand()) {
cmd.Connection = conn;
cmd.CommandText = selectStatement;
cmd.CommandType = CommandType.Text;
conn.Open();
using (var reader = await cmd.ExecuteReaderAsync()) {
return reader.Select(r => carBuilder(r)).ToList();
}
}
}
}
You may find the detailed info inside the blog post.

Thread safety in parameters passed to a static method

Assuming a static method like below is called from ASP.NET page,
can a different thread(b) overwrite the value of s1 after the first line is executed by thread(a)?
If so, can assigning parameters to local variables before manipulation solve this?
public static string TestMethod(string s1, string s2, string s3)
{
s1 = s2 + s3;
....
...
return s1;
}
Is there are a simple way to recreate such thread safety related issues?
Thanks.
No, the parameters are local variables - they're independent of any other threads. As strings are also immutable, you're safe. If these were mutable - e.g. a parameter of StringBuilder s1 - then although the value of s1 (a reference) couldn't be changed, the object that the parameter referred to could change its contents.
ref and out parameters could potentially have issues, as they can alias variables which are shared between threads.
I had same confusion too and here is my test code. Just sharing it ...
public partial class _default : System.Web.UI.Page
{
protected void Page_Load(object sender, EventArgs e)
{
ThreadTest b = new ThreadTest();
Thread t = new Thread(new ParameterizedThreadStart(ThreadTest.sum));
Thread t1 = new Thread(new ParameterizedThreadStart(ThreadTest.sum));
t.Start(10);
t1.Start(12);
}
}
class ThreadTest
{
public static void sum(object XX)
{
int x = (int)XX;
for (int i = 0; i < x; i++)
{
System.Diagnostics.Debug.WriteLine("max : " + x + " --- " + i.ToString());
}
}
}
... Now if you run this you will see that int x is safe. so local non static variables are safe for a process and can not crippled by multiple thread
Yes, under some condition, as seen by the example code.
public static class ConsoleApp {
public static void Main() {
Console.WriteLine("Write something.");
var str = Console.ReadLine();
if (String.IsNullOrEmpty(str))
return;
new Thread(() => TestMethod(null, str, "")).Start();
// Allow TestMethod to execute.
Thread.Sleep(100);
unsafe {
// Grab pointer to our string.
var gcHandle = GCHandle.Alloc(str, GCHandleType.Pinned);
var strPtr = (char*)gcHandle.AddrOfPinnedObject().ToPointer();
// Change it, one character at a time, wait a little more than
// TestMethod for dramatic effect.
for (int i = 0; i < str.Length; ++i) {
strPtr[i] = 'x';
Thread.Sleep(1100);
}
}
// Tell TestMethod to quit.
_done = true;
Console.WriteLine("Done.");
Console.ReadLine();
}
private static Boolean _done;
public static void TestMethod(String x, String y, String z) {
x = y + z;
while (!_done) {
Console.WriteLine("{0} {1}", DateTime.Now.ToLongTimeString(), x);
Thread.Sleep(1000);
}
}
}
Requirements (afaik)
Unsafe context to use pointers.
Use String.Concat(String str0, String str1) which is optimized for cases where str0 == String.Empty or str1 == String.Empty, which returns the non-empty string. Concatenating three or more strings would create a new string which blocks this.
Here's a fixed version of your modified one.
public static class ConsoleApp {
private static Int32 _counter = 10;
public static void Main() {
for (var i = 0; i < 10; i++) {
var str = GetString();
Console.WriteLine("Input: {0} - {1}", DateTime.Now.ToLongTimeString(), str);
new Thread(() => TestMethod(str)).Start();
unsafe {
var gcHandle = GCHandle.Alloc(str, GCHandleType.Pinned);
var strPtr = (char*)gcHandle.AddrOfPinnedObject().ToPointer();
strPtr[0] = 'A';
strPtr[1] = 'B';
strPtr[2] = 'C';
strPtr[3] = 'D';
strPtr[4] = 'E';
}
}
Console.WriteLine("Done.");
Console.ReadLine();
}
private static String GetString() {
var builder = new StringBuilder();
for (var i = _counter; i < _counter + 10; i++)
builder.Append(i.ToString());
_counter = _counter + 10;
return builder.ToString();
}
public static void TestMethod(Object y) {
Thread.Sleep(2000);
Console.WriteLine("Output: {0} {1}", DateTime.Now.ToLongTimeString(), y);
}
}
This still works because Object.ToString() is overriden in String to return this, thus returning the exact same reference.
Thanks Simon, here is the my evaluation on this.
In the following code, i spawn threads simply using Thread.Start and the output becomes inconsistent.
This is proving that string passed on to a method can be modified.
If otherwise please explain!
public static class ConsoleApp{
[ThreadStatic]
private static int counter = 10;
public static void Main()
{
string str;
object obj = new object();
// Change it, one character at a time, wait a little more than
// TestMethod for dramatic effect.
for (int i = 0; i < 10; i++)
{
lock (obj)
{
str = GetString();
Console.WriteLine(DateTime.Now.ToLongTimeString());
//ThreadPool.QueueUserWorkItem(TestMethod, str);
new Thread(() => TestMethod(str)).Start();
}
}
Console.WriteLine("Done.");
Console.ReadLine();
}
private static string GetString()
{
object obj = new object();
lock (obj)
{
StringBuilder sb = new StringBuilder();
int temp = 0;
for (int i = counter; i < counter + 10; i++)
{
sb.Append(i.ToString());
temp = i;
}
counter = temp;
return sb.ToString();
}
}
public static void TestMethod(object y)
{
Thread.Sleep(2000);
Console.WriteLine("{0} {1}", DateTime.Now.ToLongTimeString(), y.ToString());
}
}
Thanks.

Resources