I am using SSHJ SFTP library to get file list from SFTP-server.
The connection to server is very slow and there are tens of thousands of files in directory. Often getting file list will end in various timeout / socket errors.
Is there possibility to tell the client to retrieve file list only from eg. ".zip" files so that it would have positive impact on the performance? Pseudo command: sftpClient.ls("*.zip")
I know there is a method List<RemoteResourceInfo> net.schmizz.sshj.sftp.SFTPClient.ls(String path, RemoteResourceFilter filter) which will filter the list, but from what I understand, the filtering would happen only in client side? ie. the client would still receive whole file list and just after then it would be filtered.
Is there any way to achieve this so that server would only return the names requested? Does the SFTP-protocol even support this?
Indeed, the SFTP protocol does not have a way to provide a list of files matching any criteria. It does not matter, what SFTP library you are using.
You would have to use another interface/API if you need the filtered list. If you have a shell access, you might use shell command ls *.zip.
Or build you own (REST?) API.
I would like to copy a file from my computer to a remote server via SCP using R.
I have found 2 functions that appear to satisfy this partially.
1.
Using ssh.utils
ssh.utils::cp.remote(path.src="~/myfile.txt",
remote.dest="username#remote",
remote.src="", path.dest="~/temp", verbose=TRUE)
I've noticed that with this method, if I need to enter a password (when remote doesn't have my public key), the function produces an error.
2.
Using RCurl:
RCurl appears to have more robust functionality in the scp() function, but, from what I can tell, it is only for copying a file from a remote to my local machine. I would like to do the opposite.
Is there another way to use these functions or is there another function that would be able to copy a file from my local machine to a remote machine via SCP?
One approach to address the need to enter a password interactively is to use sshpass (see https://stackoverflow.com/a/13955428/6455166) in a call to system, e.g.
system('sshpass -p "password" scp ~/myfile.txt username#remote.com:/some/remote/path')
See the linked answer above for more details, including options to avoid embedding the password in the command.
I'm working on a little e2 application that needs to put/get data
to/from riak. I apparently have a misunderstanding
about some things.
I can successfully put and get key-value pairs to a bucket using the
erlang client. However, when I try to curl the
data out, I get nothing. In fact, doing:
curl -X GET
"http://riak1:8098/riak?buckets=true" returns '{"buckets":[]}'.
Ditto for curl -X GET "http://192.168.29.11:8098/buckets?buckets=true"
In my application, though, I can do this:
8> Object = riakc_obj:new(<<"testbucket">>, <<"testkey">>,
<<"testdata">>). {riakc_obj,<<"testbucket">>,<<"testkey">>,undefined,[],
undefined,<<"testdata">>}
9> riakc_pb_socket:put(Pid, Object).
ok
10> riakc_pb_socket:get(Pid, <<"testbucket">>, <<"testkey">>).
{ok,{riakc_obj,<<"testbucket">>,<<"testkey">>,
<<107,206,97,96,96,96,204,96,202,5,82,28,202,156,255,126,
238,185,94,62,53,131,41,...>>,
[{{dict,2,16,16,8,80,48,
{[],[],[],[],[],[],[],[],[],[],[],[],...},
{{[],[],[],[],[],[],[],[],[],[],...}}},
<<"testdata">>}],
undefined,undefined}}
11> riakc_pb_socket:list_buckets(Pid).
{ok,[<<"tb1">>,<<"testbucket">>]}
So what is the difference between the two? How can I use curl (or any
other client) to see the buckets, and retrieve
data?
--
Note that this was also sent to the riak-user mailing list
I figured out what the problem was. I had another instance of Riak running on the machine where my application was running, and the app was defaulting to "localhost" when connecting to Riak, rather than connecting to my test cluster.
The data was written correctly, just to the wrong instance.
Is it possible to make the send port change output location based on a promoted property?
We have an interface that needs to send it to a different port based on the client. But we add clients on a regular basis, so adding a new send port (both in the administrator and orchestration) will require a lot of maintenance, while the only thing that happens is a directory change
The folders are like this ...
\\server\SO\client1\Out
\\server\SO\client2\Out
\\server\SO\client3\Out
I tried using the SourceFilename to create a file name like client1\Out\filename.xml but this doesn't work.
Is there any way to do this with a single send port?
It is possible to set the OutboundTransportLocation property in context. This property contains the full path/name of the file that will be output by the file adapter. So in your case I guess you could do something along the line (if it had to be done in a pipeline component):
message.Context.Write(
OutboundTransportLocation.Name,
OutboundTransportLocation.Namespace,
string.format(#"\\server\SO\{0}\Out", client));
Of course you can do a similar thing in your orchestration.
No need of a dynamic port...
I have multiple isolated environments to setup with SaltStack. I have created some base states and custom states for each environment. For the moment, the only way I can identify an environment is by requesting a TXT record on the DNS server.
Is there a way I can select the right environment in SaltStack.
How can I put this information in a pillar or a grain?
Salt's dig module might help you here. You can use it to query information from DNS records. It needs the command line dig tool to be installed.
Use a command line:
salt-call dig.TXT google.com
to produce an output like this:
local:
- "v=spf1 include:_spf.google.com ~all"
Use a salt state to put it into a grain:
# setupgrain.sls
mygrainname:
grains.present:
- value: {{ salt['dig.TXT']('google.com') }}
Once you have the information in a grain you can select salt nodes on the grain information using matchers.