As of now i am downloading the file from SFTP to local and then adding into the database.I want to remove the extra step that is to download the file to machine.
The Text file input step is based on Apache vfs, which can read from a sftp server. So the solution is to define the Filename/Directory with the appropriate syntax:
sftp://[ username[: password]#] hostname[: port][ relative-path]
Supported file systems are on the Apache Common VFS web page.
Related
I have a remote MonetDB server running and I want to bulk upload a csv file as it is much faster.
Based on the params in MonetDB.R, there is a csvdump=TRUE option but I don't think it works when you are trying to do this against a remote server. The server has to be local.
https://rdrr.io/github/MonetDB/monetdb-r/man/dbWriteTable.html
First, am I correct that I can't do this and if not, is there a workaround? I have a dataframe with +5M rows so it takes a long time with insert statements rather than using COPY INTO.
When I try using csvdump=TRUE against the remote server, it can't find the csv file because it is local to computer that called the dbWriteTable command.
I think you are right. As a workaround either use explicit COPY INTO ON CLIENT SQL statements or first use some file transfer tool to copy the file to the remote server before calling dbWriteTable.
It reads from MonetDB's documentation on COPY INTO:
FROM files ON SERVER
With ON SERVER, which is the default, the file name must be an
absolute path on the system on which the database server (mserver5) is
running. ...
Interestingly enough pymonetdb, the Python driver for MonetDB, uses ON CLIENT for bulk loads. From the pymonetdb's doc:
File Uploads and Downloads
Classes related to file transfer requests as used by COPY INTO ON
CLIENT.
You might want to file an issue for the MonetDB R-driver project to have similar behavior as pymonetdb.
I am beginner to WebDAV/HTTP protocol. I want to transfer files across network using WebDAV. The source received camera image raw data files has to be placed in a webDAV server (a folder). These files are then copied by a remote client via webDAV. my questions below
Is it necessary to use webDAV/HTTP GET/POST/PUT method calls on the server machine to copy images
or just a normal Linux/QNX command shall work?
If we do not need to go via WebDAV just to copy files then how can we attach properties to these
files like file name and size? are these automatically supplied to remote client by webDAV using
some OS file system calls?
Thank you in advance for your answers
I want to move files from local folder to remote URL in a scheduled way using WebDAV.
I found this URL useful but this shows the script to transfer only single file, instead I want to transfer all files from a local folder to remote URL through winscp using WebDAV protocol:
http://winscp.net/eng/docs/guide_automation
Any pointers for this would be helpful.
Use the following WinSCP script:
open http://user:password#example.com/
put d:\path\* /home/user/
close
Read about file masks.
If you really need to move files (as opposite to copy), add the -delete switch to the put command:
put -delete d:\path\* /home/user/
For scheduling, see the WinSCP guide to scheduling file transfers.
My scenario as follows:
One java program is updating some random files to a SFTP location.
My requirement is as soon as a file is uploaded by the previous java program, using java I need to download the file. The files can be of size 100MB. I am searching for some java API which is helpful in this way. Here I even don't know the name of files. But I can keep a regular expression for this. A same file can be uploaded by previous program periodically. Since file size is high I need to wait until the complete file to be uploaded.
I used Jsch to download files, but I am not getting how to poll using jsch.
Polling
All you can do is to keep listing remote directory periodically, until you find a new file. There's no better way with SFTP. For that you obviously use ChannelSftp.ls().
Regarding selecting files matching certain pattern, see:
JSch ChannelSftp.ls - pass match patterns in java
Waiting until the upload is complete
Again, there's no support for this in widespread implementations of SFTP.
For details, see my answer at:
SFTP file lock mechanism.
I'm developing an application using Adobe Flex 4.5 SDK, in which the user would be able to export multiple files bundled in one zip file. I was thinking that I must need to take the following steps in order for performing this task:
Create a temporary folder on the server for the user who requested the download. Since it is an anonymous type of user, I have to read Sate/Session information to identify the user.
Copy all the requested files into the temporary folder on the server
Zip the copied file
Download the zip file from the server to the client machine
I was wondering if anybody knows any best-practice/sample-code for the task
Thanks
The ByteArray class has some methods for compressing, but this is more for data transport, not for packaging up multiple files.
I don't like saying things are impossible, but I will say that this should be done on the server-side. Depending on your server architecture I would suggest sending the binary files to a server script which could package the files for you.
A quick google search for your preferred server-side language and zipping files should give you some sample scripts to get you started.