Websockets in R - r

I managed to establish a connection in R to Mtgox websocket with following specs:
url: https://socketio.mtgox.com/mtgox?Currency=USD
port: 80
specs: https://en.bitcoin.it/wiki/MtGox/API/Streaming
I used the improved R library "websocket" downloaded from https://github.com/zeenogee/R-Websockets:
require("websockets")
con = websocket("https://socketio.mtgox.com/mtgox?Currency=USD")
and the connection was successfully established. However, it seems that the socket is not broadcasting. I made an easy function f
f = function(con) {
Print("Test Test!", con)
}
set_callback("receive", f, con)
while(TRUE)
{
service(con)
Sys.sleep(0.05)
}
which should print some text whenever some data are received from the websocket. But the websocket doesnt seem to trigger the "receive" method and nothing is displayed. Code ended up with infinite loop with no output.
I know that the websocket is working so there must be a mistake in the code. Do I have to "ping" the socket somehow to start broadcasting? Anyone has anidea how to get it working?
Thanks!

Firstly, you have an infinite loop, because you have defined an infinite loop:
While(TRUE)
It is worth noting, numerous R websocket implementations leverage this loop, so may not be a bug but rather an implementation detail causing what you are seeing.
It would appear that you need to subscribe to the 'message' event not 'receive' (
https://en.bitcoin.it/wiki/MtGox/API/Streaming).
In JavaScript (from MtGox Spec):
conn.on('message', function(data) {
// Handle incoming data object.
});
Or in R:
set_callback('message',f,con)
Failing that...
I would also comment to say, that maybe the stream is returning you data that you are not able to implicitly print in R Print function.
Sample:
{
"op":"remark",
"message":<MESSAGE FROM THE SERVER>,
"success":<boolean>
}
If the data follows this format as defined in spec, you may which to examine how that data is being parsed, and the "op" which is being returned.

Related

Adding a column to MongoDB from R via mongolite gives persistent error

I want to add a column to a MongoDB collection via R. The collection has tabular format and is already relatively big (14000000 entries, 140 columns).
The function I am currently using is
function (collection, name, value)
{
mongolite::mongo(collection)$update("{}", paste0("{\"$set\":{\"",
name, "\": ", value, "}}"), multiple = TRUE)
invisible(NULL)
}
It does work so far. (It takes about 5-10 Minutes, which is ok. Although, it would be nice if the speed could be improved somehow).
However, it also gives me persistently the following error that interrupts the execution of the rest of the script.
The error message reads:
Error: Failed to send "update" command with database "test": Failed to read 4 bytes: socket error or timeout
Any help on resolving this error would be appreciated. (If there are ways to improve the performance of the update itself I'd also be more than happy for any advices.)
the default socket timeout is 5 minutes.
You can override the default by setting sockettimeoutms directly in your connection URI:
mongoURI <- paste0("mongodb://", user,":",pass, "#", mongoHost, ":", mongoPort,"/",db,"?sockettimeoutms=<something large enough in milliseconds>")
mcon <- mongo(mongoCollection, url=mongoURI)
mcon$update(...)

R Package IBrokers placeOrder() function fails

I'm using the package: IBrokers. It works well for me when I request historical data. Also the call to reqAccountUpdates() works well.
I am having problems with this script:
# myscript.r
.libPaths("rpackages")
library(IBrokers)
tws2 = twsConnect(2)
print('Attempting BUY')
mytkr = twsFuture("ES","GLOBEX","201412")
myorderid = sample(1001:3001, 1)
IBrokers:::.placeOrder(tws2, mytkr, twsOrder(myorderid, "BUY", "1", "MKT"))
twsDisconnect(tws2)
Sometimes the above script works okay. Usually though it fails. When it fails, it seems to connect okay.
Then I see this in my TWS console:
03:47:45:581 JTS-EServerSocket-290: [2:47:71:1:0:0:0:ERR] Message type -1. Socket I/O error -
03:47:45:581 JTS-EServerSocket-290: Anticipated error
jextend.d: Socket I/O error -
at jextend.sc.b(sc.java:364)
at jextend.ch.sb(ch.java:1534)
at jextend.ch.run(ch.java:1390)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.net.SocketException: Connection reset
at java.net.SocketInputStream.read(SocketInputStream.java:196)
at java.net.SocketInputStream.read(SocketInputStream.java:122)
at java.net.SocketInputStream.read(SocketInputStream.java:210)
at jextend.xh.d(xh.java:45)
at jextend.sc.c(sc.java:579)
at jextend.sc.r(sc.java:227)
at jextend.af.a(af.java:232)
at jextend.sc.f(sc.java:650)
at jextend.pd.a(pd.java:822)
at jextend.sc.b(sc.java:358)
... 3 more
03:47:45:583 JTS-EServerSocket-290: [2:47:71:1:0:0:0:ERR] Socket connection for client{2} has closed.
03:47:45:583 JTS-EWriter14-291: [2:47:71:1:0:0:0:ERR] Unable write to socket client{2} -
03:47:45:584 JTS-EServerSocketNotifier-288: Terminating
Can you offer any ideas on how you would wrestle with this issue?
One other piece of info:
I think the call to reqIds() may be necessary. Sometimes reqIds() would return an id not high enough. Then, I'd use it and placeOrder() would fail. So, I call reqIds() but then use Sys.time() to give me an id which is larger than the last ID I used.
Another problem may have been some code-text I copied out of a PowerPoint. Some of the code-characters may have been corrupt.
The main problem was orderid.
I need to be careful how I generate orderid.
Also I may have moused in bad characters from a powerpoint preso which had an example.
I posted some code which works in a comment in this thread.
Dan

Show all open RODBC connections

Does anyone know how to do this? showConnections won't list any open connections from odbcConnect.
You can narrow down your search in the following way, which will return all variables of your environment currently of the class RODBC.
envVariables<-ls()
bools<-sapply(envVariables, function(string){
class(get(string))=="RODBC"
})
rodbcObj<-envVariables[bools]
Closed connections are still of the class RODBC though, so there is still a little work to be done here.
We can define a function, using trycatch, that will try to get the connection info of the associated RODBC object. If it is an open connection, then this command will run fine, and we return the string of the variable name.
If the RODBC object is not an open connection, this will throw an error, which we catch and, in the way I've implemented, return NA. You could return any number of things here.
openConns<-function(string){
tryCatch({
result<-odbcGetInfo(get(string))
string
}, error = function(e){
NA
})
}
We then remove the return value that corresponds to the error. In my case, NA, so I do na.omit on the return.
na.omit(sapply(rodbcObj, openConns))
Or alternatively
result<-sapply(rodbcObj, openConns)
result[!is.na(result)]
Any questions or comments on it let me know
-DMT

Implementation of simple polling of results file

For one of my dissertation's data collection modules, I have implemented a simple polling mechanism. This is needed, because I make each data collection request (one of many) as SQL query, submitted via Web form, which is simulated by RCurl code. The server processes each request and generates a text file with results at a specific URL (RESULTS_URL in code below). Regardless of the request, URL and file name are the same (I cannot change that). Since processing time for different data requests, obviously, is different and some requests may take significant amount of time, my R code needs to "know", when the results are ready (file is re-generated), so that it can retrieve them. The following is my solution for this problem.
POLL_TIME <- 5 # polling timeout in seconds
In function srdaRequestData(), before making data request:
# check and save 'last modified' date and time of the results file
# before submitting data request, to compare with the same after one
# for simple polling of results file in srdaGetData() function
beforeDate <- url.exists(RESULTS_URL, .header=TRUE)["Last-Modified"]
beforeDate <<- strptime(beforeDate, "%a, %d %b %Y %X", tz="GMT")
<making data request is here>
In function srdaGetData(), called after srdaRequestData()
# simple polling of the results file
repeat {
if (DEBUG) message("Waiting for results ...", appendLF = FALSE)
afterDate <- url.exists(RESULTS_URL, .header=TRUE)["Last-Modified"]
afterDate <- strptime(afterDate, "%a, %d %b %Y %X", tz="GMT")
delta <- difftime(afterDate, beforeDate, units = "secs")
if (as.numeric(delta) != 0) { # file modified, results are ready
if (DEBUG) message(" Ready!")
break
}
else { # no results yet, wait the timeout and check again
if (DEBUG) message(".", appendLF = FALSE)
Sys.sleep(POLL_TIME)
}
}
<retrieving request's results is here>
The module's main flow/sequence of events is linear, as follows:
Read/update configuration file
Authenticate with the system
Loop through data requests, specified in configuration file (via lapply()),
where for each request perform the following:
{
...
Make request: srdaRequestData()
...
Retrieve results: srdaGetData()
...
}
The issue with the code above is that it doesn't seem to be working as expected: upon making data request, the code should print "Waiting for results ..." and then, periodically checking the results file for being modified (re-generated), print progress dots until the results are ready, when it prints confirmation. However, the actual behavior is that the code waits long time (I intentionally made one request a long-running), not printing anything, but then, apparently retrieves results and prints both "Waiting for results ..." and " Ready" at the same time.
It seems to me that it's some kind of synchronization issue, but I can't figure out what exactly. Or, maybe it's something else and I'm somehow missing it. Your advice and help will be much appreciated!
In a comment to the question, I believe MrFlick solved the issue: the polling logic appears to be functional, but the problem is that the progress messages are out of synch with current events on the system.
By default, the R console output is buffered. This is by design: to speed things up and avoid the distracting flicker that may be associated with frequent messages etc. We tend to forget this fact, particularly after we've been using R in a very interactive fashion, running various ad-hoc statement at the console (the console buffer is automatically flushed just before returning the > prompt).
It is however possible to get message() and more generally console output in "real time" by either explicitly flushing the console after each critical output statement, using the flush.console() function, or by disabling buffering at the level of the R GUI (right-click when on the console, see Buffered output Ctrl W item. This is also available in the Misc menu)
Here's a toy example of the explicit use of flush.console. Note the use of cat() rather than message() as the former doesn't automatically add a CR/LF to the output. The latter however is useful however because its messages can be suppressed with suppressMessages() and the like. Also as shown in the comment you can cat the "\b" (backspace) character to make the number overwrite one another.
CountDown <- function() {
for (i in 9:1){
cat(i)
# alternatively to cat(i) use: message(i)
flush.console() # <<<<<<< immediate ouput to console.
Sys.sleep(1)
cat(" ") # also try cat("\b") instead ;-)
}
cat("... Blast-off\n")
}
The output is the following, what is of course not evident in this print-out is that it took 10 seconds overall with one number printed every second, before the final "Blast off"; do remove the flush.console() statement and the output will come at once, after 10 seconds, i.e. when the function terminates (unless console is not buffered at the level of the GUI).
CountDown()
9 8 7 6 5 4 3 2 1 ... Blast-off

Rmpi send/receive hanging?

I am writing the code for a population MCMC. I will try to provide as much information I think could help, so please bear with me.
I am using tempered distributions and I want to perform exchange moves, i.e a moves that propose to swap the value of two chains.
What I have done (exchange happening in the master)
I had written this initially by
letting each chain mutate for a specified number of iterations n.
at every n-th iteration, I would send the slaves' results to the master and there attempt an exchange of parameters between chains.
Then send updated values back to slaves and repeat the process.
What I want to achieve (exchange directly between slaves)
This is working fine, but I wanted to clean up my code and remove unnecessary communication between master and slave . That is, let the slaves communicate directly between them.
So assuming I am spawning 10 slaves,
at iteration n, I want to let slave1-slave2, slave3-slave4, ...., slave9-slave10 communicate between them
at iteration 2*n, I want to let slave2-slave3, slave4-slave5, ...slave8-slave9 communicate between them
and so on, so that I let samples travel through the temperature ladder.
The problem
And this is where I am facing a problem.
I think I am managing to send a value from one slave to another (my print statement "Succesfully sent" gets printed in the log files of the slaves that are sending) but this doesn't seem to be received (my "Succesfully received" statement doesn't get printed in the log of the partner slaves).
And the program just hangs. I think maybe I have caused a deadlock, but I am not sure what I have done wrong?
Could you please advise? I have used as guide this Parallel Tempering R code
http://www.lindonslog.com/mathematics/parallel-tempering-r-rmpi/
Please see below my code
Many thanks!
Sofia
ind <- mpi.comm.rank()
oddFlag<-0 ### object to flag code suitable for odd/even numbered slaves.
for (i in 1:TotalIter) {
##### normal MCMC move (single chain mutation) - logL.current
if ( i%%exchangeInterval == 0 ){ ### every nth (right now 5th) iteration, attempt an exchange
message("\n\nAttempt an exchange move")
oddFlag<-oddFlag+1
exchange<-0
logL.partner<-0
if (ind%%2 == oddFlag%%2) { ###when oddFlag even , the following code concerns even-numbered slaves. When odd number, it concerns odd-numbered slaves.
ind.partner<-ind+1
if (0<ind.partner && ind.partner<(noChains+1)){
message("This is the slave: ", ind, " and its partner is: ", ind.partner)
message("The tag for receiving logL.partner is: ", ind.partner)
logL.partner<-mpi.recv.Robj(source=ind.partner,tag=ind.partner) #### receive the logL of partner
message("Succesfully received")
message("This is the logL.partner: ", logL.partner)
exchanges.attempted<-exchanges.attempted+1
if (runif(1)< min(1, exp((logL.partner - estimatorSelf)*(temper[ind] - temper[ind.partner] )))) { ############# exp((chain2 - chain1)*(T1 - T2))
message("I exchanged the values")
exchange<-1
print(exchange)
exchanges.accepted<-exchanges.accepted+1
}
mpi.send.Robj(obj=exchange,dest=ind.partner,tag=15*ind)
}
if (exchange==1){
### exchange parameters with mpi.send.Robj/mpi.recv.Robj functions
}
} else { ##### ###when oddFlag even , the following code concerns odd-numbered slaves. For oddFlag odd, it concerns even-numbered slaves.
ind.partner<-ind-1
if (0<ind.partner && ind.partner<(noChains+1)){
message("This is the slave: ", ind, " and its partner is: ", ind.partner)
message("The tag for sending logL.current is: ", ind)
mpi.send.Robj(obj=logL.current,dest=ind.partner,tag=ind) ### send logL to partner
message("Succesfully sent")
exchange<-mpi.recv.Robj(source=ind.partner, tag=15*ind.partner)
message("I received the exchange message")
}
if (exchange==1){
### exchange parameters send/receive functions
}
}
}
}
This seems very strange. To my knowledge the send and receive commands are locking so for the send to work without the receive to work strikes me as a bit odd. Have you tried the code on the guide? If you like I can send you the entire R-code just to see if it works on your system. If the code on the guide works then you can safely say that it isn't a problem with your MPI setup.
One thing you might try is for the receive command instead of using ind.partner you could use mpi.any.source() (i think is the correct one) which will accept a message of any tag. If this resolves your deadlock, it could be a problem with the tags (but by eye there doesn't seem anything wrong to me).
Another thing you might try is removing the "source=" and the "tag=" on the receive commands. I notice in my code that I don't have any there whilst I do for the send commands. Perhaps that was causing me too problems, but I can't quite remember. Let me know how it goes, I hope it all works out.

Resources