Database update without data loss - sqlite

there is a table with 3 columns: id, title, poem, favorite.
So, the favorite is written 1 or 0, the default is 0, it changes if the user adds a line to the Favorites.
The problem is that if I update the database (add more rows), all Favorites will disappear from the user. How to transfer the Favorites from the old database to the new one in the onUpgrade method?
DbHelper Class Code
public class PoemsDbHelper extends SQLiteOpenHelper {
private static String DB_NAME = "brodsky.db";
private static String DB_PATH = "";
private static final int DB_VERSION = 3;
private SQLiteDatabase db;
private final Context context;
private boolean needUpdate = false;
public PoemsDbHelper(Context context) {
super(context, DB_NAME, null, DB_VERSION);
if (android.os.Build.VERSION.SDK_INT >= 17)
DB_PATH = context.getApplicationInfo().dataDir + "/databases/";
else
DB_PATH = "/data/data/" + context.getPackageName() + "/databases/";
this.context = context;
copyDataBase();
this.getReadableDatabase();
}
public void updateDataBase() throws IOException {
if (needUpdate) {
File dbFile = new File(DB_PATH + DB_NAME);
if (dbFile.exists())
dbFile.delete();
copyDataBase();
needUpdate = false;
}
}
private boolean checkDataBase() {
File dbFile = new File(DB_PATH + DB_NAME);
return dbFile.exists();
}
private void copyDataBase() {
if (!checkDataBase()) {
this.getReadableDatabase();
this.close();
try {
copyDBFile();
} catch (IOException mIOException) {
throw new Error("ErrorCopyingDataBase");
}
}
}
private void copyDBFile() throws IOException {
InputStream input = context.getAssets().open(DB_NAME);
//InputStream input = context.getResources().openRawResource(R.raw.info);
OutputStream output = new FileOutputStream(DB_PATH + DB_NAME);
byte[] buffer = new byte[1024];
int length;
while ((length = input.read(buffer)) > 0)
output.write(buffer, 0, length);
output.flush();
output.close();
input.close();
}
public boolean openDataBase() throws SQLException {
db = SQLiteDatabase.openDatabase(DB_PATH + DB_NAME, null, SQLiteDatabase.CREATE_IF_NECESSARY);
return db != null;
}
#Override
public synchronized void close() {
if (db != null)
db.close();
super.close();
}
#Override
public void onCreate(SQLiteDatabase db) {
}
#Override
public void onUpgrade(SQLiteDatabase db, int oldVersion, int newVersion) {
if (newVersion > oldVersion)
needUpdate = true;
}
}

In this case it seems you are overwriting your "old" database with a new, clean, empty database. I suggest trying copying the old database file to a temporary backup, overwrite old database file with new, clean, empty database file.
Then you must define an upgrade strategy, i.e. how are fields moved from v. 1 to v. 2. What happens if you removed a field? What happens if you added a new field? Has some of your data been restructured? This is an exercise that is best sketched out on paper.
You can now attach the old, backed up database to the new, clean database (see SQL command ATTACH), and insert new records into the new database from the old, following the strategy you sketched out above.
If anything fails, you can always revert to your backup file and notify the user, instead of loosing all their data.

Related

SQLite Database for Commenting function

There is no errors with the codes. However, I am trying to build a comment section where the name and comments will be saved in the database(SQLite).
1) It does not stay in the Textview after i come back from another page.
2) Example, i commented once, and the comment shows. I comment again, it actually re-displays the previous and the new comments together.
public class BuyerHome extends AppCompatActivity {
DatabaseHelper2 myDB;
//EditText name,comment;
EditText nameIn;
EditText commentIn;
TextView viewComment;
Button postComment;
#Override
protected void onCreate(Bundle savedInstanceState) {
super.onCreate(savedInstanceState);
setContentView(R.layout.activity_buyer_home);
nameIn = (EditText) findViewById(R.id.nameInput);
commentIn = (EditText) findViewById(R.id.commentText);
viewComment = (TextView) findViewById(R.id.viewCommentText);
myDB = new DatabaseHelper2(this);
postComment = (Button) findViewById(R.id.buttonComment);
postComment.setOnClickListener(new View.OnClickListener() {
#Override
public void onClick(View v) {
String name = nameIn.getText().toString();
String comment = commentIn.getText().toString();
if(nameIn.length()!=0 && commentIn.length()!=0){
AddData(name,comment);
nameIn.setText("");
commentIn.setText("");
}else{
Toast.makeText(BuyerHome.this, "Insert fields",
Toast.LENGTH_SHORT).show();
}
Cursor data = myDB.getContents();
if(data.getCount()==0){
Toast.makeText(BuyerHome.this, "Database empty",
Toast.LENGTH_SHORT).show();
}
else{
while(data.moveToNext()){
viewComment.append(data.getString(1));
viewComment.append(data.getString(2)+"\n");
}
}
//viewComment.append("\n"+name+":"+comment);
}
});
}
public void AddData(String name,String comment){
boolean insertData = myDB.insertData(name,comment);
if(insertData==true){
Toast.makeText(this, "Success", Toast.LENGTH_SHORT).show();
}else{
Toast.makeText(this, "Fail", Toast.LENGTH_SHORT).show();
}
}
}
1) I would like the past comments to be there as long as the page is available.
2) I would like only the latest comment to be displayed(on top of the past comments), and not the past comments(re-displaying again) with the new comment.
I'd suggest perhaps using a second view for the older comments. You would also need a means of determining the latest as opposed to older comments.
In the example below the second view is a Listview (this would enable you to click on a specific message, perhaps quoting it, perhaps extracting the name).
The example caters for two ways of determining the last posted comment via the id column (note _id is actually used as the column name as this then allows a Cursor Adapter to be used for the ListView which can be advantageous).
To facilitate showing the comments after returning the onResume method has been overridden inn the activity.
here's the code :-
DatabaseHelper2.java
public class DatabaseHelper2 extends SQLiteOpenHelper {
private final static String DBNAME = "buyerdb";
private final static int DBVERSION = 1;
public final static String TBL_COMMENT = "comment";
public final static String COL_COMMENT_ID = BaseColumns._ID;
public final static String COL_COMMENT_NAME = "name";
public final static String COL_COMMENT_COMMENT = "comment";
public final static String COl_COMMENT_TIMESTAMP = "timestamp";
private String crt_tbl_comment = "CREATE TABLE IF NOT EXISTS " + TBL_COMMENT + "(" +
COL_COMMENT_ID + " INTEGER PRIMARY KEY, " +
COL_COMMENT_NAME + " TEXT," +
COL_COMMENT_COMMENT + " TEXT, " +
COl_COMMENT_TIMESTAMP + " TEXT DEFAULT CURRENT_TIMESTAMP" +
")";
SQLiteDatabase mDB;
public DatabaseHelper2(Context context) {
super(context, DBNAME, null, DBVERSION);
mDB = this.getWritableDatabase();
}
#Override
public void onCreate(SQLiteDatabase db) {
db.execSQL(crt_tbl_comment);
}
#Override
public void onUpgrade(SQLiteDatabase db, int oldVersion, int newVersion) {
}
public long addData(String name, String comment) {
ContentValues cv = new ContentValues();
cv.put(COL_COMMENT_NAME,name);
cv.put(COL_COMMENT_COMMENT,comment);
return mDB.insert(TBL_COMMENT,null,cv);
}
public Cursor getLatestComment() {
return mDB.query(TBL_COMMENT,null,null,null,null,null,COl_COMMENT_TIMESTAMP + " DESC","1");
}
public Cursor getAllButLatestComment() {
String whereclause = COL_COMMENT_ID + " < (SELECT max(" +
COL_COMMENT_ID +
") FROM " + TBL_COMMENT +
")";
return mDB.query(TBL_COMMENT,null,whereclause,null,null,null,COl_COMMENT_TIMESTAMP + " DESC");
}
}
Note that the timestamp column hasn't been utilised but could be, as when a row (a comment) is added the current timestamp will automatically be used to set the column.
Note the getLatestComment and the getAllButLatestComment methods return a Cursor with the appropriate rows.
BuyerHome.java
public class BuyerHome extends AppCompatActivity {
EditText nameIn;
EditText commentIn;
TextView viewComment;
Button postComment;
ListView olderComments; //<<<<<<<<<< Added (Listview needs to be added to the layout)
DatabaseHelper2 myDB;
Cursor oldermessages, latestmessage; //<<<<<<<<<< Added
SimpleCursorAdapter sma; //<<<<<<<<<< Added adapter for the ListView
#Override
protected void onCreate(Bundle savedInstanceState) {
super.onCreate(savedInstanceState);
setContentView(R.layout.activity_buyer_home);
nameIn = (EditText) findViewById(R.id.nameInput);
commentIn = (EditText) findViewById(R.id.commentText);
viewComment = (TextView) findViewById(R.id.viewCommentText);
olderComments = (ListView) findViewById(R.id.olderComments); //<<<<<<<<<< Added
myDB = new DatabaseHelper2(this);
postComment = (Button) findViewById(R.id.buttonComment);
postComment.setOnClickListener(new View.OnClickListener() {
#Override
public void onClick(View v) {
String name = nameIn.getText().toString();
String comment = commentIn.getText().toString();
if (nameIn.length() != 0 && commentIn.length() != 0) {
myDB.addData(name, comment); //<<<<<<<<<< method starts changed lower case a
nameIn.setText("");
commentIn.setText("");
} else {
Toast.makeText(BuyerHome.this, "Insert fields",
Toast.LENGTH_SHORT).show();
}
displayComments(); //<<<<<<<<<< added to refresh the display according to the latest comments
}
});
}
//<<<<<<<<<< ADDED so will display comments whenever called including returning from another activity
#Override
protected void onResume() {
super.onResume();
displayComments();
}
//<<<<<<<<<< ADDED entire method for displaying comments
private void displayComments() {
latestmessage = myDB.getLatestComment();
if (latestmessage.moveToFirst()) {
viewComment.setText(latestmessage.getString(latestmessage.getColumnIndex(DatabaseHelper2.COL_COMMENT_NAME)) +
" said :- " +
latestmessage.getString(latestmessage.getColumnIndex(DatabaseHelper2.COL_COMMENT_COMMENT)));
} else {
viewComment.setText("");
}
oldermessages = myDB.getAllButLatestComment();
if (sma == null) {
sma = new SimpleCursorAdapter(
this,
android.R.layout.simple_list_item_2,oldermessages,
new String[]{DatabaseHelper2.COL_COMMENT_NAME,DatabaseHelper2.COL_COMMENT_COMMENT},
new int[]{android.R.id.text1,android.R.id.text2},
0
);
olderComments.setAdapter(sma);
} else {
sma.swapCursor(oldermessages);
}
}
}
Result
The highlighted line is the latest comment.
The latest comment is followed by the list of all other comments.

Netty: TCP Client Server File transfer: Exception TooLongFrameException:

I am new to netty and I am trying to design a solution as below for transfer of file from Server to Client over TCP:
1. Zero copy based file transfer in case of non-ssl based transfer (Using default region of the file)
2. ChunkedFile transfer in case of SSL based transfer.
The Client - Server file transfer works in this way:
1. The client sends the location of the file to be transfered
2. Based on the location (sent by the client) the server transfers the file to the client
The file content could be anything (String /image /pdf etc) and any size.
Now, I get this TooLongFrameException: at the Server side, though the server is just decoding the path received from the client, for running the code mentioned below (Server/Client).
io.netty.handler.codec.TooLongFrameException: Adjusted frame length exceeds 65536: 215542494061 - discarded
at io.netty.handler.codec.LengthFieldBasedFrameDecoder.fail(LengthFieldBasedFrameDecoder.java:522)
at io.netty.handler.codec.LengthFieldBasedFrameDecoder.failIfNecessary(LengthFieldBasedFrameDecoder.java:500)
Now, My question is:
Am I wrong with the order of Encoders and Decoders and its configuration? If so, what is the correct way to configure it to receive a file from the server?
I went through few related StackOverflow posts SO Q1,SO Q2 , SO Q3, SO Q4. I got to know about the LengthFieldBasedDecoder, but I didn't get to know how to configure its corresponding LengthFieldPrepender at the Server (Encoding side). Is it even required at all?
Please point me into the right direction.
FileClient:
public final class FileClient {
static final boolean SSL = System.getProperty("ssl") != null;
static final int PORT = Integer.parseInt(System.getProperty("port", SSL ? "8992" : "8023"));
static final String HOST = System.getProperty("host", "127.0.0.1");
public static void main(String[] args) throws Exception {
// Configure SSL.
final SslContext sslCtx;
if (SSL) {
SelfSignedCertificate ssc = new SelfSignedCertificate();
sslCtx = SslContextBuilder.forServer(ssc.certificate(), ssc.privateKey()).build();
} else {
sslCtx = null;
}
// Configure the client
EventLoopGroup group = new NioEventLoopGroup();
try {
Bootstrap b = new Bootstrap();
b.group(group)
.channel(NioSocketChannel.class)
.option(ChannelOption.SO_KEEPALIVE, true)
.handler(new ChannelInitializer<SocketChannel>() {
#Override
public void initChannel(SocketChannel ch) throws Exception {
ChannelPipeline pipeline = ch.pipeline();
if (sslCtx != null) {
pipeline.addLast(sslCtx.newHandler(ch.alloc(), HOST, PORT));
}
pipeline.addLast("frameDecoder", new LengthFieldBasedFrameDecoder(64*1024, 0, 8));
pipeline.addLast("frameEncoder", new LengthFieldPrepender(4));
pipeline.addLast(new ObjectDecoder(ClassResolvers.cacheDisabled(null)));
pipeline.addLast(new ObjectEncoder());
pipeline.addLast( new FileClientHandler()); }
});
// Start the server.
ChannelFuture f = b.connect(HOST,PORT).sync();
// Wait until the server socket is closed.
f.channel().closeFuture().sync();
} finally {
// Shut down all event loops to terminate all threads.
group.shutdownGracefully();
}
}
}
FileClientHandler:
public class FileClientHandler extends ChannelInboundHandlerAdapter{
#Override
public void channelActive(ChannelHandlerContext ctx) {
String filePath = "/Users/Home/Documents/Data.pdf";
ctx.writeAndFlush(Unpooled.wrappedBuffer(filePath.getBytes()));
}
#Override
public void channelRead(ChannelHandlerContext ctx, Object msg) throws Exception {
System.out.println("File Client Handler Read method...");
}
#Override
public void exceptionCaught(ChannelHandlerContext ctx, Throwable cause) {
cause.printStackTrace();
}
}
FileServer:
/**
* Server that accept the path of a file and echo back its content.
*/
public final class FileServer {
static final boolean SSL = System.getProperty("ssl") != null;
static final int PORT = Integer.parseInt(System.getProperty("port", SSL ? "8992" : "8023"));
public static void main(String[] args) throws Exception {
// Configure SSL.
final SslContext sslCtx;
if (SSL) {
SelfSignedCertificate ssc = new SelfSignedCertificate();
sslCtx = SslContextBuilder.forServer(ssc.certificate(), ssc.privateKey()).build();
} else {
sslCtx = null;
}
// Configure the server.
EventLoopGroup bossGroup = new NioEventLoopGroup(1);
EventLoopGroup workerGroup = new NioEventLoopGroup();
try {
ServerBootstrap b = new ServerBootstrap();
b.group(bossGroup, workerGroup).channel(NioServerSocketChannel.class)
.option(ChannelOption.SO_KEEPALIVE, true).handler(new LoggingHandler(LogLevel.INFO))
.childHandler(new ChannelInitializer<SocketChannel>() {
#Override
public void initChannel(SocketChannel ch) throws Exception {
ChannelPipeline pipeline = ch.pipeline();
if (sslCtx != null) {
pipeline.addLast(sslCtx.newHandler(ch.alloc()));
}
pipeline.addLast("frameDecoder",new LengthFieldBasedFrameDecoder(64*1024, 0, 8));
pipeline.addLast("frameEncoder", new LengthFieldPrepender(4));
pipeline.addLast(new ObjectDecoder(ClassResolvers.cacheDisabled(null)));
pipeline.addLast(new ObjectEncoder());
pipeline.addLast(new ChunkedWriteHandler());
pipeline.addLast(new FileServerHandler());
}
});
// Start the server.
ChannelFuture f = b.bind(PORT).sync();
// Wait until the server socket is closed.
f.channel().closeFuture().sync();
} finally {
bossGroup.shutdownGracefully();
workerGroup.shutdownGracefully();
}
}
}
FileServerHandler:
public class FileServerHandler extends ChannelInboundHandlerAdapter {
#Override
public void channelRead(ChannelHandlerContext ctx, Object obj) throws Exception {
RandomAccessFile raf = null;
long length = -1;
try {
ByteBuf buff = (ByteBuf)obj;
byte[] bytes = new byte[buff.readableBytes()];
buff.readBytes(bytes);
String msg = new String(bytes);
raf = new RandomAccessFile(msg, "r");
length = raf.length();
} catch (Exception e) {
ctx.writeAndFlush("ERR: " + e.getClass().getSimpleName() + ": " + e.getMessage() + '\n');
return;
} finally {
if (length < 0 && raf != null) {
raf.close();
}
}
if (ctx.pipeline().get(SslHandler.class) == null) {
// SSL not enabled - can use zero-copy file transfer.
ctx.writeAndFlush(new DefaultFileRegion(raf.getChannel(), 0, length));
} else {
// SSL enabled - cannot use zero-copy file transfer.
ctx.writeAndFlush(new ChunkedFile(raf));
}
}
#Override
public void exceptionCaught(ChannelHandlerContext ctx, Throwable cause) {
cause.printStackTrace();
System.out.println("Exception server.....");
}
}
I referred Netty In Action and code samples from here
There are multiple things wrong with your server/client. First thing the SSL, for the client you don't need to initialize a SslContext for a server instead you would do something like this:
sslCtx = SslContextBuilder.forClient().trustManager(InsecureTrustManagerFactory.INSTANCE).build();
On the server side of things you use a SelfSignedCertificate which in itself isn't wrong but would like to remind you that it should only be used for debugging purposes and not in production. In addition you use the ChannelOption.SO_KEEPALIVE which isn't recommended since the keepalive interval is OS-dependent. Furthermore you added Object En-/Decoder to your pipeline which in your case don't do anything useful so you can remove them.
Also you configured your LengthFieldBasedFrameDecoder wrong due to an incomplete and wrong parameter list. In the netty docs you need the version of the constructor which defines the lengthFieldLength and initialBytesToStrip. Besides the not stripping the length field you also defined the wrong lengthFieldLength which should be the same as your LengthFieldPrepender's lengthFieldLength which is 4 bytes. In conlusion you could use the constructor like this:
new LengthFieldBasedFrameDecoder(64 * 1024, 0, 4, 0, 4)
In both your handler you don't specify a Charset when en-/decoding your String which could lead to problems because if no ´Charset´ is defined the systems default will be used which could vary. You could do something like this:
//to encode the String
string.getBytes(StandardCharsets.UTF_8);
//to decode the String
new String(bytes, StandardCharsets.UTF_8);
Additionally you tried to use the DefaultFileRegion if no SslHandler was added to the pipeline which would have been fine if you didn't added the LengthFieldHandler since they would need a memory copy of the byte[] to send to added the length field. Moreover I would recommend using the ChunkedNioFile instead of the ChunkedFile because it's nonblocking which is always a good thing. You would do this like that:
new ChunkedNioFile(randomAccessFile.getChannel())
One final thing on how to decode a ChunkedFile as it's split in chunks you can simply assamble them tougether with a simple OutputStream. Here's an old file handler of mine:
public class FileTransferHandler extends SimpleChannelInboundHandler<ByteBuf> {
private final Path path;
private final int size;
private final int hash;
private OutputStream outputStream;
private int writtenBytes = 0;
private byte[] buffer = new byte[0];
protected FileTransferHandler(Path path, int size, int hash) {
this.path = path;
this.size = size;
this.hash = hash;
}
#Override
protected void channelRead0(ChannelHandlerContext ctx, ByteBuf byteBuf) throws Exception {
if(this.outputStream == null) {
Files.createDirectories(this.path.getParent());
if(Files.exists(this.path))
Files.delete(this.path);
this.outputStream = Files.newOutputStream(this.path, StandardOpenOption.CREATE, StandardOpenOption.APPEND);
}
int size = byteBuf.readableBytes();
if(size > this.buffer.length)
this.buffer = new byte[size];
byteBuf.readBytes(this.buffer, 0, size);
this.outputStream.write(this.buffer, 0, size);
this.writtenBytes += size;
if(this.writtenBytes == this.size && MurMur3.hash(this.path) != this.hash) {
System.err.println("Received file has wrong hash");
return;
}
}
#Override
public void channelInactive(ChannelHandlerContext ctx) throws Exception {
if(this.outputStream != null)
this.outputStream.close();
}
}

How do you print TextArea to a USB Thermal Printer 58mm?(JAVAFX)

So I'm trying to make a billing system in which I want to print a receipt.I was able to do it with some code that I found online,but the font size is too big to print in the 58mm wide paper.I'm not able to adjust the font size.Any kind of help with this issue will be highly appreciated.Thank You.
Here is The Code :
public class PrinterService implements Printable {
public List<String> getPrinters(){
DocFlavor flavor = DocFlavor.BYTE_ARRAY.AUTOSENSE;
PrintRequestAttributeSet pras = new HashPrintRequestAttributeSet();
PrintService printServices[] = PrintServiceLookup.lookupPrintServices(
flavor, pras);
List<String> printerList = new ArrayList<String>();
for(PrintService printerService: printServices){
printerList.add( printerService.getName());
}
return printerList;
}
#Override
public int print(Graphics g, PageFormat pf, int page)
throws PrinterException {
if (page > 0) { /* We have only one page, and 'page' is zero-based */
return NO_SUCH_PAGE;
}
/*
* User (0,0) is typically outside the imageable area, so we must
* translate by the X and Y values in the PageFormat to avoid clipping
*/
Graphics2D g2d = (Graphics2D) g;
g2d.translate(pf.getImageableX(), pf.getImageableY());
/* Now we perform our rendering */
g.setFont(new Font("Roman", 0, 8));
g.drawString("Hello world !", 0, 10);
return PAGE_EXISTS;
}
public void printString(String printerName, String text) {
// find the printService of name printerName
DocFlavor flavor = DocFlavor.BYTE_ARRAY.AUTOSENSE;
PrintRequestAttributeSet pras = new HashPrintRequestAttributeSet();
PrintService printService[] = PrintServiceLookup.lookupPrintServices(
flavor, pras);
PrintService service = findPrintService(printerName, printService);
DocPrintJob job = service.createPrintJob();
try {
byte[] bytes;
// important for umlaut chars
bytes = text.getBytes("CP437");
Doc doc = new SimpleDoc(bytes, flavor, null);
job.print(doc, null);
} catch (Exception e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
public void printBytes(String printerName, byte[] bytes) {
DocFlavor flavor = DocFlavor.BYTE_ARRAY.AUTOSENSE;
PrintRequestAttributeSet pras = new HashPrintRequestAttributeSet();
PrintService printService[] = PrintServiceLookup.lookupPrintServices(
flavor, pras);
PrintService service = findPrintService(printerName, printService);
DocPrintJob job = service.createPrintJob();
try {
Doc doc = new SimpleDoc(bytes, flavor, null);
job.print(doc, null);
} catch (Exception e) {
e.printStackTrace();
}
}
private PrintService findPrintService(String printerName,
PrintService[] services) {
for (PrintService service : services) {
if (service.getName().equalsIgnoreCase(printerName)) {
return service;
}
}
return null;
}
}
#FXML
public void printit(ActionEvent actionEvent)
{
PrinterService printerService = new PrinterService();
System.out.println(printerService.getPrinters());
//print some stuff
printerService.printString("POS-58-Series", area.getText());
}

Issue with Custom Pipeline component

The Custom Pipeline component developed reads the incoming stream to a folder and pass only some meta data through the MessageBox.I am using the one already availaible in Code Project
using System;
using System.Collections.Generic;
using System.Text;
using Microsoft.BizTalk.Message.Interop;
using Microsoft.BizTalk.Component.Interop;
using System.IO;
namespace SendLargeFilesDecoder
{
[ComponentCategory(CategoryTypes.CATID_PipelineComponent)]
[ComponentCategory(CategoryTypes.CATID_Decoder)]
[System.Runtime.InteropServices.Guid("53fd04d5-8337-42c2-99eb-32ac96d1105a")]
public class SendLargeFileDecoder : IBaseComponent,
IComponentUI,
IComponent,
IPersistPropertyBag
{
#region IBaseComponent
private const string _description = "Pipeline component used to save large files to disk";
private const string _name = "SendLargeFileDecoded";
private const string _version = "1.0.0.0";
public string Description
{
get { return _description; }
}
public string Name
{
get { return _name; }
}
public string Version
{
get { return _version; }
}
#endregion
#region IComponentUI
private IntPtr _icon = new IntPtr();
public IntPtr Icon
{
get { return _icon; }
}
public System.Collections.IEnumerator Validate(object projectSystem)
{
return null;
}
#endregion
#region IComponent
public IBaseMessage Execute(IPipelineContext pContext, IBaseMessage pInMsg)
{
if (_largeFileLocation == null || _largeFileLocation.Length == 0)
_largeFileLocation = Path.GetTempPath();
if (_thresholdSize == null || _thresholdSize == 0)
_thresholdSize = 4096;
if (pInMsg.BodyPart.GetOriginalDataStream().Length > _thresholdSize)
{
Stream originalStream = pInMsg.BodyPart.GetOriginalDataStream();
string srcFileName = pInMsg.Context.Read("ReceivedFileName", "http://schemas.microsoft.com/BizTalk/2003/file-properties").ToString();
string largeFilePath = _largeFileLocation + System.IO.Path.GetFileName(srcFileName);
FileStream fs = new FileStream(largeFilePath, FileMode.Create);
// Write message to disk
byte[] buffer = new byte[1];
int bytesRead = originalStream.Read(buffer, 0, buffer.Length);
while (bytesRead != 0)
{
fs.Flush();
fs.Write(buffer, 0, buffer.Length);
bytesRead = originalStream.Read(buffer, 0, buffer.Length);
}
fs.Flush();
fs.Close();
// Create a small xml file
string xmlInfo = "<MsgInfo xmlns='http://SendLargeFiles'><LargeFilePath>" + largeFilePath + "</LargeFilePath></MsgInfo>";
byte[] byteArray = System.Text.Encoding.UTF8.GetBytes(xmlInfo);
MemoryStream ms = new MemoryStream(byteArray);
pInMsg.BodyPart.Data = ms;
}
return pInMsg;
}
#endregion
#region IPersistPropertyBag
private string _largeFileLocation;
private int _thresholdSize;
public string LargeFileLocation
{
get { return _largeFileLocation; }
set { _largeFileLocation = value; }
}
public int ThresholdSize
{
get { return _thresholdSize; }
set { _thresholdSize = value; }
}
public void GetClassID(out Guid classID)
{
classID = new Guid("CA47347C-010C-4B21-BFCB-22F153FA141F");
}
public void InitNew()
{
}
public void Load(IPropertyBag propertyBag, int errorLog)
{
object val1 = null;
object val2 = null;
try
{
propertyBag.Read("LargeFileLocation", out val1, 0);
propertyBag.Read("ThresholdSize", out val2, 0);
}
catch (ArgumentException)
{
}
catch (Exception ex)
{
throw new ApplicationException("Error reading PropertyBag: " + ex.Message);
}
if (val1 != null)
_largeFileLocation = (string)val1;
if (val2 != null)
_thresholdSize = (int)val2;
}
public void Save(IPropertyBag propertyBag, bool clearDirty, bool saveAllProperties)
{
object val1 = (object)_largeFileLocation;
propertyBag.Write("LargeFileLocation", ref val1);
object val2 = (object)_thresholdSize;
propertyBag.Write("ThresholdSize", ref val2);
}
#endregion
}
}
The issue here is the LargeFileLocation is configurable in the receive pipeline. If I give a location for the first time for example E:\ABC\ the files are sent to the location.
But if I change the location to E:\DEF\ the files are still being sent to the previous location E:\ABC. I tried to create a new biztalk application deleting the old one but still I get the files dropped in to the old location E:\ABC\ not sure why.
Most likely the issue is with respect to Property definition of LargeFileLocation and its implementation and usage in IPersistPropertyBag interfaces. You can try following things:
Check if you have added E:\ABC path in Pipeline at design time. If
yes remove it from there and set in Admin console for first time
also and see how it behaves, my feeling is it will take temp path
location.
Change the Properties and IPersistPropertyBag implementation to use property with declaration such as public string LargeFileName {get;set;} i.e. no local variables _largeFileName.
Have you deleted the dll in %BizTalkFolder%\Pipeline Components\ ?
To refresh the pipeline component, you need delete the old dll file/remove the item in VS toolbox. then restart the VS, deploy it again.
and for this LargeFileLocation , I suggest you make it as a property so you can config it.

Hadoop: the Mapper didn't read files from multiple input paths

The Mapper didn't manage to read a file from multiple directories. Could anyone help?
I need to read one file in each mapper. I've added multiple input paths and implemented the custom WholeFileInputFormat, WholeFileRecordReader. In the map method, I don't need the input key. I make sure that each map can read a whole file.
Command line: hadoop jar AutoProduce.jar Autoproduce /input_a /input_b /output
I specified two input path----1.input_a; 2.input_b;
Run method snippets:
Job job = new Job(getConf());
job.setInputFormatClass(WholeFileInputFormat.class);
FileInputFormat.setInputPaths(job, new Path(args[0]), new Path(args[1]));
FileOutputFormat.setOutputPath(job, new Path(args[2]));
map method snippets:
public void map(NullWritable key, BytesWritable value, Context context){
FileSplit fileSplit = (FileSplit) context.getInputSplit();
System.out.println("Directory :" + fileSplit.getPath().toString());
......
}
Custom WholeFileInputFormat:
class WholeFileInputFormat extends FileInputFormat<NullWritable, BytesWritable> {
#Override
protected boolean isSplitable(JobContext context, Path file) {
return false;
}
#Override
public RecordReader<NullWritable, BytesWritable> createRecordReader(
InputSplit split, TaskAttemptContext context) throws IOException,
InterruptedException {
WholeFileRecordReader reader = new WholeFileRecordReader();
reader.initialize(split, context);
return reader;
}
}
Custom WholeFileRecordReader:
class WholeFileRecordReader extends RecordReader<NullWritable, BytesWritable> {
private FileSplit fileSplit;
private Configuration conf;
private BytesWritable value = new BytesWritable();
private boolean processed = false;
#Override
public void initialize(InputSplit split, TaskAttemptContext context)
throws IOException, InterruptedException {
this.fileSplit = (FileSplit) split;
this.conf = context.getConfiguration();
}
#Override
public boolean nextKeyValue() throws IOException, InterruptedException {
if (!processed) {
byte[] contents = new byte[(int) fileSplit.getLength()];
Path file = fileSplit.getPath();
FileSystem fs = file.getFileSystem(conf);
FSDataInputStream in = null;
try {
in = fs.open(file);
IOUtils.readFully(in, contents, 0, contents.length);
value.set(contents, 0, contents.length);
} finally {
IOUtils.closeStream(in);
}
processed = true;
return true;
}
return false;
}
#Override
public NullWritable getCurrentKey() throws IOException,InterruptedException {
return NullWritable.get();
}
#Override
public BytesWritable getCurrentValue() throws IOException,InterruptedException {
return value;
}
#Override
public float getProgress() throws IOException {
return processed ? 1.0f : 0.0f;
}
#Override
public void close() throws IOException {
// do nothing
}
}
PROBLEM:
After setting two input paths, all map tasks read files from only one directory..
Thanks in advance.
You'll have to use MultipleInputs instead of FileInputFormat in the driver. So your code should be as:
MultipleInputs.addInputPath(job, new Path(args[0]), <Input_Format_Class_1>);
MultipleInputs.addInputPath(job, new Path(args[1]), <Input_Format_Class_2>);
.
.
.
MultipleInputs.addInputPath(job, new Path(args[N-1]), <Input_Format_Class_N>);
So if you want to use WholeFileInputFormat for the first input path and TextInputFormat for the second input path, you'll have to use it the following way:
MultipleInputs.addInputPath(job, new Path(args[0]), WholeFileInputFormat.class);
MultipleInputs.addInputPath(job, new Path(args[1]), TextInputFormat.class);
Hope this works for you!

Resources