Vaadin 7: File Upload - vaadin7

I have a Upload component in which I´m supposed to import a xml file in order to parse it.
I´m trying to use the File.createTempFile method to create the file phisically,but something weird is going on.
For example,if I take the file named "test.xml" and use the createTempFile method to create it on the disk,the name of the generate file becomes something like 'test.xml13234xml'.How can I create the file the correct way?

This is expected when using i.e. createTempFile method as it implicitly creates a file with random prefix:
// a part of createTempFile method
private static final SecureRandom random = new SecureRandom();
static File generateFile(String prefix, String suffix, File dir) {
long n = random.nextLong();
if (n == Long.MIN_VALUE) {
n = 0; // corner case
} else {
n = Math.abs(n);
}
return new File(dir, prefix + Long.toString(n) + suffix);
}
which should give something like 'test.xml13234xml'.
If you want to create a file with the correct name and keep it for later use you can rename/move it within uploadSucceeded method.
public class ExampleUpload implements Upload.Receiver, Upload.SucceededListener {
private Upload xmlUpload;
private File tempFile;
public ExampleUpload() {
this.xmlUpload = new Upload("Upload:", this);
this.xmlUpload.addSucceededListener(this);
}
#Override
public OutputStream receiveUpload(String filename, String mimeType) {
try {
tempFile = File.createTempFile(filename, "xml");
tempFile.deleteOnExit();
return new FileOutputStream(tempFile);
} catch (IOException e) {
e.printStackTrace();
}
return null;
}
#Override
public void uploadSucceeded(SucceededEvent event) {
try {
File destinationFile = new File("c:\\" + event.getFilename());
FileUtils.moveFile(tempFile, destinationFile));
// TODO read and parse destinationFile
} catch (IOException e) {
e.printStackTrace();
}
}
}

Related

How to scan dtable drool file if there is any changes in file and load it again using kiescanner drool version 7.4..final

I am working on drool dtable xls file with spring.
i have implemented the business rules in xls file using external location and then with the help of kie services i am executing rules.
Following is the code snippet that's how i am loading rules in engine.
at the start of spring initialization i am calling init() method
see below spring configuration.
<bean id="droolsService" class="com.example.drools.DroolsServiceImpl" init-method="init">
Java Code
public void init() {
LOG.info("inside init");
KieSession kieSession;
for (RequestType type : droolsMap.keySet()) {
try {
kieSession = getKieSession(this.getDroolsMap().get(type));
droolsRules.put(type, kieSession);
} catch (Exception e) {
LOG.error("Failed to load kiesession:", e);
throw new RuntimeException(e);
}
}
}
private KieSession getKieSession(final String file) throws DroolsParserException, IOException, BiffException {
KieServices kieServices = KieServices.Factory.get();
KieFileSystem kfs = kieServices.newKieFileSystem();
InputStream stream = null;
String drl = null;
String RULE_PATH = "src/main/resources/";
SpreadsheetCompiler converter = new SpreadsheetCompiler();
//Workbook workbook = Workbook.getWorkbook(DroolsServiceImpl.class.getResourceAsStream(file));
Workbook workbook = Workbook.getWorkbook(new FileInputStream(file));
LOG.info("Loading rule file " + file);
for (Sheet sheet : workbook.getSheets()) {
LOG.info("Loading Sheet " + sheet.getName());
stream = new FileInputStream(file);
drl = converter.compile(stream, sheet.getName());
//StringReader reader = new StringReader(drl);
String DRL_FILE = RULE_PATH + sheet.getName() + ".drl";
System.out.println("Drool file added ::: " + DRL_FILE);
kfs.write(DRL_FILE, ResourceFactory.newReaderResource(new StringReader(drl)));
stream.close();
}
KieBuilder kieBuilder = kieServices.newKieBuilder(kfs).buildAll();
KieContainer kieContainer = kieServices.newKieContainer(kieServices.getRepository().getDefaultReleaseId());
KieSessionConfiguration conf = SessionConfiguration.newInstance();
KieSession ksession = kieContainer.newKieSession(conf);
if (kieBuilder.getResults().hasMessages(Message.Level.ERROR)) {
List<Message> errors = kieBuilder.getResults().getMessages(Message.Level.ERROR);
StringBuilder sb = new StringBuilder("Errors:");
for (Message msg : errors) {
sb.append("\n " + msg);
}
try {
throw new Exception(sb.toString());
} catch (Exception e) {
e.printStackTrace();
} finally {
if (stream != null)
stream.close();
if (workbook != null)
workbook.close();
}
}
return ksession;
}
Everything working perfect but the problem is i am not able to scan the file changes. If files is modified then i have to restart the server in order to sync the changes.
I have tried listener to load specific init() method after xls dtable has any changes but its not working , same old result is coming.
I have tried kiescanner but i am not able to get the concept.
KieScanner is loading maven kjar so how do i suppose to create kjar.
I just wanted to kie api scan if any changes in the drool file and try to reload whole changes in kiecontainer without server restarting.
Found the answer myself, Posting because it will help someone who needed.
What I did , I have used apache VFS File Monitor-
DefaultFileMonitor fm = new DefaultFileMonitor(new CustomFileListener());
When file will modified , create or get deleted it will call CustomFileListener.
Following is the implementation of CustomFileListener.
import org.apache.commons.vfs2.FileChangeEvent;
import org.apache.commons.vfs2.FileListener;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.beans.factory.support.DefaultListableBeanFactory;
import org.springframework.web.context.ContextLoader;
import org.springframework.web.context.support.XmlWebApplicationContext;
public class CustomFileListener implements FileListener {
private static final Logger LOG = LoggerFactory.getLogger(CustomFileListener.class);
#Override
public void fileCreated(FileChangeEvent fileChangeEvent) throws Exception {
}
#Override
public void fileDeleted(FileChangeEvent fileChangeEvent) throws Exception {
}
#Override
public void fileChanged(FileChangeEvent fileChangeEvent) throws Exception {
LOG.debug(" Under FileChanged Method");
LOG.debug(" File has been changed hence reinitializing init method = " + fileChangeEvent.getFile().getName().getPath());
XmlWebApplicationContext xmlWebApplicationContext =
(XmlWebApplicationContext) ContextLoader.getCurrentWebApplicationContext();
DefaultListableBeanFactory defaultListableBeanFactory =
(DefaultListableBeanFactory) xmlWebApplicationContext.getBeanFactory();
DroolsServiceImpl droolsService = (DroolsServiceImpl) defaultListableBeanFactory.getBean("droolsService");
droolsService.init();
}
}
What i did when the file will change, It will call fileChanged method.
In that i have fetched cached bean(DroolServiceImpl) from ContextLoader.getCurrentWebApplicationContext(); and called its init() method.
So this it will reload whole process and reinitialize the KieModule,KieRepository.

Can Storm's HdfsBolt flush data after a timeout as well?

We are using Storm to process streaming data and store into HDFS. We have got everything to work but have one issue. I understand that we can specify the number of tuples after which the data gets flushed to HDFS using SyncPolicy, something like this below:
SyncPolicy syncPolicy = new CountSyncPolicy(Integer.parseInt(args[3]));
The question I have is can the data also be flushed after a timeout? For e.g. we have set the SyncPolicy above to 1000 tuples. If for whatever reason we get 995 tuples and then the data stops coming in for a while is there any way that storm can flush the 995 records to HDFS after a specified timeout (5 seconds)?
Thanks in advance for any help on this!
Shay
Yes, if you send a tick tuple to the HDFS bolt, it will cause the bolt to try to sync to the HDFS file system. All this happens in the HDFS bolt's execute function.
To configure tick tuples for your topology, in your topology config. In Java, to set that to every 300 seconds the code would look like:
Config topologyConfig = new Config();
topologyConfig.put(Config.TOPOLOGY_TICK_TUPLE_FREQ_SECS, 300);
StormSubmitter.submitTopology("mytopology", topologyConfig, builder.createTopology());
You'll have to adjust that last line depending on your circumstances.
There is an alternative solution for this problem,
First, lets clarify about sync policy, If your sync policy is 1000 ,then HdfsBolt only sync the data from 1000 tuple by calling hsync() method in execute() means it only clears the buffer by pushing data to disk, but for faster write disk may uses its cache and not writing to file directly.
The data is written to the file only when the size of data matches your rotation policy that need to specify at the time of bolt creation.
FileRotationPolicy rotationPolicy = new FileSizeRotationPolicy(100.0f, Units.KB);
So for flushing the record the to file after timeout, Seperate your tick tuple from normal tuples in excecute method and calculate the time difference of both tuple, If the diff is greater than timeout period then write the data to file.
By handling tick tuple differently you can also avoid the tick tuple frequency written to your file.
See the below code for better understanding:
public class CustomHdfsBolt1 extends AbstractHdfsBolt {
private static final Logger LOG = LoggerFactory.getLogger(CustomHdfsBolt1.class);
private transient FSDataOutputStream out;
private RecordFormat format;
private long offset = 0L;
private int tickTupleCount = 0;
private String type;
private long normalTupleTime;
private long tickTupleTime;
public CustomHdfsBolt1() {
}
public CustomHdfsBolt1(String type) {
this.type = type;
}
public CustomHdfsBolt1 withFsUrl(String fsUrl) {
this.fsUrl = fsUrl;
return this;
}
public CustomHdfsBolt1 withConfigKey(String configKey) {
this.configKey = configKey;
return this;
}
public CustomHdfsBolt1 withFileNameFormat(FileNameFormat fileNameFormat) {
this.fileNameFormat = fileNameFormat;
return this;
}
public CustomHdfsBolt1 withRecordFormat(RecordFormat format) {
this.format = format;
return this;
}
public CustomHdfsBolt1 withSyncPolicy(SyncPolicy syncPolicy) {
this.syncPolicy = syncPolicy;
return this;
}
public CustomHdfsBolt1 withRotationPolicy(FileRotationPolicy rotationPolicy) {
this.rotationPolicy = rotationPolicy;
return this;
}
public CustomHdfsBolt1 addRotationAction(RotationAction action) {
this.rotationActions.add(action);
return this;
}
protected static boolean isTickTuple(Tuple tuple) {
return tuple.getSourceComponent().equals(Constants.SYSTEM_COMPONENT_ID)
&& tuple.getSourceStreamId().equals(Constants.SYSTEM_TICK_STREAM_ID);
}
public void execute(Tuple tuple) {
try {
if (isTickTuple(tuple)) {
tickTupleTime = Calendar.getInstance().getTimeInMillis();
long timeDiff = normalTupleTime - tickTupleTime;
long diffInSeconds = TimeUnit.MILLISECONDS.toSeconds(timeDiff);
if (diffInSeconds > 5) { // specify the value you want.
this.rotateWithOutFileSize(tuple);
}
} else {
normalTupleTime = Calendar.getInstance().getTimeInMillis();
this.rotateWithFileSize(tuple);
}
} catch (IOException var6) {
LOG.warn("write/sync failed.", var6);
this.collector.fail(tuple);
}
}
public void rotateWithFileSize(Tuple tuple) throws IOException {
syncHdfs(tuple);
this.collector.ack(tuple);
if (this.rotationPolicy.mark(tuple, this.offset)) {
this.rotateOutputFile();
this.offset = 0L;
this.rotationPolicy.reset();
}
}
public void rotateWithOutFileSize(Tuple tuple) throws IOException {
syncHdfs(tuple);
this.collector.ack(tuple);
this.rotateOutputFile();
this.offset = 0L;
this.rotationPolicy.reset();
}
public void syncHdfs(Tuple tuple) throws IOException {
byte[] e = this.format.format(tuple);
synchronized (this.writeLock) {
this.out.write(e);
this.offset += (long) e.length;
if (this.syncPolicy.mark(tuple, this.offset)) {
if (this.out instanceof HdfsDataOutputStream) {
((HdfsDataOutputStream) this.out).hsync(EnumSet.of(SyncFlag.UPDATE_LENGTH));
} else {
this.out.hsync();
}
this.syncPolicy.reset();
}
}
}
public void closeOutputFile() throws IOException {
this.out.close();
}
public void doPrepare(Map conf, TopologyContext topologyContext, OutputCollector collector) throws IOException {
LOG.info("Preparing HDFS Bolt...");
this.fs = FileSystem.get(URI.create(this.fsUrl), this.hdfsConfig);
this.tickTupleCount = 0;
this.normalTupleTime = 0;
this.tickTupleTime = 0;
}
public Path createOutputFile() throws IOException {
Path path = new Path(this.fileNameFormat.getPath(),
this.fileNameFormat.getName((long) this.rotation, System.currentTimeMillis()));
this.out = this.fs.create(path);
return path;
}
}
You can directly use this class in your project.
Thanks,

Spring MVC Multipart file upload random FileNotFoundException

I built a web application using spring MVC, everything is working fine except the file upload in which I got random FileNotFoundExceptions. I found some solutions online like using a different tmp folder but I keep getting random error.
My code is:
#RequestMapping(value="/upload", method=RequestMethod.POST)
public #ResponseBody String handleFileUpload(#RequestParam("file") final MultipartFile multipartFile,
#RequestHeader("email") final String email, #RequestHeader("password") String password){
if (authenticateUser(email, password)) {
if (!multipartFile.isEmpty()) {
System.out.println("Start processing");
Thread thread = new Thread(){
public void run(){
ProcessCSV obj = new ProcessCSV();
try {
File file = multipartToFile(multipartFile);
if(file !=null) {
obj.extractEvents(file, email, cluster, session);
}
else {
System.out.println("null File");
}
} catch (IOException e) {
System.out.println("File conversion error");
e.printStackTrace();
}
}
};
thread.start();
return "true";
} else {
return "false";
}
}
else {
return "false";
}
}
and:
public File multipartToFile(MultipartFile multipartFile) throws IOException {
File uploadFile = null;
if(multipartFile != null && multipartFile.getSize() > 0) {
uploadFile = new File("/tmp/" + multipartFile.getOriginalFilename());
FileOutputStream fos = null;
try {
uploadFile.createNewFile();
fos = new FileOutputStream(uploadFile);
IOUtils.copy(multipartFile.getInputStream(), fos);
} catch (FileNotFoundException e) {
System.out.println("File conversion error");
e.printStackTrace();
} catch (IOException e) {
System.out.println("File conversion error");
e.printStackTrace();
} finally {
if (fos != null) {
try {
fos.close();
} catch (IOException e) {
System.out.println("File conversion error");
e.printStackTrace();
}
}
}
}
else {
System.out.println("null MultipartFile");
}
return uploadFile;
}
and the configuration file:
multipart.maxFileSize: 100MB
multipart.maxRequestSize: 100MB
multipart.location = ${user.home}
server.port = 8090
I used different versions of the multipartToFile function, one was using multipartfile.transferTo() but I was getting the same random error. Any advice?
Thank you
EDIT stack trace:
java.io.IOException: java.io.FileNotFoundException: /Users/aaa/upload_07720775_4b37_4b86_b370_40280388f3a4_00000003.tmp (No such file or directory)
at org.apache.catalina.core.ApplicationPart.write(ApplicationPart.java:121)
at org.springframework.web.multipart.support.StandardMultipartHttpServletRequest$StandardMultipartFile.transferTo(StandardMultipartHttpServletRequest.java:260)
at main.RESTController.multipartToFile(RESTController.java:358)
at main.RESTController$1.run(RESTController.java:241)
Caused by: java.io.FileNotFoundException: /Users/aaa/upload_07720775_4b37_4b86_b370_40280388f3a4_00000003.tmp (No such file or directory)
at java.io.FileInputStream.open0(Native Method)
at java.io.FileInputStream.open(FileInputStream.java:195)
at java.io.FileInputStream.<init>(FileInputStream.java:138)
at org.apache.tomcat.util.http.fileupload.disk.DiskFileItem.write(DiskFileItem.java:392)
at org.apache.catalina.core.ApplicationPart.write(ApplicationPart.java:119)
... 3 more
I had just had a night of terror with this error. I found out that MultiPartFile is only recognisable to and by the #Controller class. So if you pass it to another bean which is not a controller, Spring will not be able to help you. It somewhat makes sense that the #Controller is tightly bound to the front screen (communication from the browser to the system - Controllers are the entry point from the browser). So any conversation must happen there in the Controller.
In my case, I did something like the following:
#Controller
public class FileUploadingController{
#PostMapping("/uploadHistoricData")
public String saveUploadedDataFromBrowser(#RequestParam("file") MultipartFile file) {
try {
String pathToFile = "/home/username/destination/"
new File(pathToFile).mkdir();
File newFile = new File(pathToFile + "/uploadedFile.csv");
file.transferTo(newFile); //transfer the uploaded file data to a java.io.File which can be passed between layers
dataService.processUploadedFile( newFile);
} catch (IOException e) {
//handle your exception here please
}
return "redirect:/index?successfulDataUpload";
}
}`
I had the same problem, it looks like MultipartFile is using different current dir internally, so all not absolute paths are not working.
I had to convert my path to an absolute path and then it worked.
It is working inside #RestController and in other beans too.
Path path = Paths.get(filename).toAbsolutePath();
fileToImport.transferTo(path.toFile());
fileToImport is MultipartFile.

Hadoop: the Mapper didn't read files from multiple input paths

The Mapper didn't manage to read a file from multiple directories. Could anyone help?
I need to read one file in each mapper. I've added multiple input paths and implemented the custom WholeFileInputFormat, WholeFileRecordReader. In the map method, I don't need the input key. I make sure that each map can read a whole file.
Command line: hadoop jar AutoProduce.jar Autoproduce /input_a /input_b /output
I specified two input path----1.input_a; 2.input_b;
Run method snippets:
Job job = new Job(getConf());
job.setInputFormatClass(WholeFileInputFormat.class);
FileInputFormat.setInputPaths(job, new Path(args[0]), new Path(args[1]));
FileOutputFormat.setOutputPath(job, new Path(args[2]));
map method snippets:
public void map(NullWritable key, BytesWritable value, Context context){
FileSplit fileSplit = (FileSplit) context.getInputSplit();
System.out.println("Directory :" + fileSplit.getPath().toString());
......
}
Custom WholeFileInputFormat:
class WholeFileInputFormat extends FileInputFormat<NullWritable, BytesWritable> {
#Override
protected boolean isSplitable(JobContext context, Path file) {
return false;
}
#Override
public RecordReader<NullWritable, BytesWritable> createRecordReader(
InputSplit split, TaskAttemptContext context) throws IOException,
InterruptedException {
WholeFileRecordReader reader = new WholeFileRecordReader();
reader.initialize(split, context);
return reader;
}
}
Custom WholeFileRecordReader:
class WholeFileRecordReader extends RecordReader<NullWritable, BytesWritable> {
private FileSplit fileSplit;
private Configuration conf;
private BytesWritable value = new BytesWritable();
private boolean processed = false;
#Override
public void initialize(InputSplit split, TaskAttemptContext context)
throws IOException, InterruptedException {
this.fileSplit = (FileSplit) split;
this.conf = context.getConfiguration();
}
#Override
public boolean nextKeyValue() throws IOException, InterruptedException {
if (!processed) {
byte[] contents = new byte[(int) fileSplit.getLength()];
Path file = fileSplit.getPath();
FileSystem fs = file.getFileSystem(conf);
FSDataInputStream in = null;
try {
in = fs.open(file);
IOUtils.readFully(in, contents, 0, contents.length);
value.set(contents, 0, contents.length);
} finally {
IOUtils.closeStream(in);
}
processed = true;
return true;
}
return false;
}
#Override
public NullWritable getCurrentKey() throws IOException,InterruptedException {
return NullWritable.get();
}
#Override
public BytesWritable getCurrentValue() throws IOException,InterruptedException {
return value;
}
#Override
public float getProgress() throws IOException {
return processed ? 1.0f : 0.0f;
}
#Override
public void close() throws IOException {
// do nothing
}
}
PROBLEM:
After setting two input paths, all map tasks read files from only one directory..
Thanks in advance.
You'll have to use MultipleInputs instead of FileInputFormat in the driver. So your code should be as:
MultipleInputs.addInputPath(job, new Path(args[0]), <Input_Format_Class_1>);
MultipleInputs.addInputPath(job, new Path(args[1]), <Input_Format_Class_2>);
.
.
.
MultipleInputs.addInputPath(job, new Path(args[N-1]), <Input_Format_Class_N>);
So if you want to use WholeFileInputFormat for the first input path and TextInputFormat for the second input path, you'll have to use it the following way:
MultipleInputs.addInputPath(job, new Path(args[0]), WholeFileInputFormat.class);
MultipleInputs.addInputPath(job, new Path(args[1]), TextInputFormat.class);
Hope this works for you!

How to import an xquery module using Saxon

I am having some troubles running an Xquery with Saxon9HE, which has a reference to an external module.
I would like Saxon to resolve the module with a relative path rather absolute.
the module declaration
module namespace common = "http://my-xquery-utils";
from the main xquery
import module namespace common = "http://my-xquery-utils" at "/home/myself/common.xquery";
from my java code
public class SaxonInvocator {
private static Processor proc = null;
private static XQueryEvaluator xqe = null;
private static DocumentBuilder db = null;
private static StaticQueryContext ctx = null;
/**
* Utility for debug, should not be called outside your IDE
*
* #param args xml, xqFile, xqParameter
*/
public static void main(String[] args) {
XmlObject instance = null;
try {
instance = XmlObject.Factory.parse(new File(args[0]));
} catch (XmlException ex) {
Logger.getLogger(SaxonInvocator.class.getName()).log(Level.SEVERE, null, ex);
} catch (IOException ex){
Logger.getLogger(SaxonInvocator.class.getName()).log(Level.SEVERE, null, ex);
}
System.out.print(transform(instance, args[1], args[2]));
}
public static String transform(XmlObject input, String xqFile, String xqParameter) {
String result = null;
try {
proc = new Processor(false);
proc.getUnderlyingConfiguration().getOptimizer().setOptimizationLevel(0);
ctx = proc.getUnderlyingConfiguration().newStaticQueryContext();
ctx.setModuleURIResolver(new ModuleURIResolver() {
#Override
public StreamSource[] resolve(String moduleURI, String baseURI, String[] locations) throws XPathException {
StreamSource[] modules = new StreamSource[locations.length];
for (int i = 0; i < locations.length; i++) {
modules[i] = new StreamSource(getResourceAsStream(locations[i]));
}
return modules;
}
});
db = proc.newDocumentBuilder();
XQueryCompiler comp = proc.newXQueryCompiler();
XQueryExecutable exp = comp.compile(getResourceAsStream(xqFile));
xqe = exp.load();
ByteArrayInputStream bais = new ByteArrayInputStream(input.xmlText().getBytes("UTF-8"));
StreamSource ss = new StreamSource(bais);
XdmNode node = db.build(ss);
xqe.setExternalVariable(
new QName(xqParameter), node);
result = xqe.evaluate().toString();
} catch (SaxonApiException e) {
e.printStackTrace();
} catch (UnsupportedEncodingException e) {
e.printStackTrace();
} catch (IOException e) {
e.printStackTrace();
}
return result;
}
public static InputStream getResourceAsStream(String resource) {
InputStream stream = SaxonInvocator.class.getResourceAsStream("/" + resource);
if (stream == null) {
stream = SaxonInvocator.class.getResourceAsStream(resource);
}
if (stream == null) {
stream = SaxonInvocator.class.getResourceAsStream("my/project/" + resource);
}
if (stream == null) {
stream = SaxonInvocator.class.getResourceAsStream("/my/project/" + resource);
}
return stream;
}
}
If a change it into a relative path like
import module namespace common = "http://my-xquery-utils" at "common.xquery";
I get
Error on line 22 column 1
XQST0059: java.io.FileNotFoundException
I am not sure how the ModuleURIResolver should be used.
Saxon questions are best asked on the Saxon forum at http://saxonica.plan.io - questions asked here will probably be noticed eventually but sometimes, like this time, they aren't our first priority.
The basic answer is that for the relative URI to resolve, the base URI needs to be known, which means that you need to ensure that the baseURI property in the XQueryCompiler is set. This happens automatically if you compile the query from a File, but not if you compile it from an InputStream.
If you don't know a suitable base URI to set, the alternative is to write a ModuleURIResolver, which could for example fetch the module by making another call on getResourceAsStream().

Resources