I am currently creating a REST API with restrserve.
While trying to add a logger to my application I ran into this problem.
By setting the log level as: application$logger$set_log_level("error"),
my console output on error has a JSON structure like:
{
"timestamp":"2020-09-22 18:25:37.425642",
"level":"ERROR",
"name":"Application",
"pid":38662,
"msg":"",
"context":{
"request_id":"alphanumeric string",
"message":{ here is the description of the error }
}
}
I set up a printer function following the instructions on the site, but among the available fields (timestamp, level, logger_name, pid, message), "context" is not present,so basically, by just printing the message my logs are empty files.
This is my printer function:
application$logger$set_printer(FUN = function(timestamp, level, logger_name, pid, message, ...){
write(message, file = paste("/Log/Monitor_", pid,
timestamp, ".txt", sep = ""))
# attempting to print "context" raises an error!
})
Is there a way to print the "context" field present in the console output to a file?
Thanks in advance, any suggestions would be very helpful
It is always a good idea to take a look to the source. There is extra ... argument where context is passed.
Generally it is advisable to use lgr package for user-level logging.
Related
I want to add a column to a MongoDB collection via R. The collection has tabular format and is already relatively big (14000000 entries, 140 columns).
The function I am currently using is
function (collection, name, value)
{
mongolite::mongo(collection)$update("{}", paste0("{\"$set\":{\"",
name, "\": ", value, "}}"), multiple = TRUE)
invisible(NULL)
}
It does work so far. (It takes about 5-10 Minutes, which is ok. Although, it would be nice if the speed could be improved somehow).
However, it also gives me persistently the following error that interrupts the execution of the rest of the script.
The error message reads:
Error: Failed to send "update" command with database "test": Failed to read 4 bytes: socket error or timeout
Any help on resolving this error would be appreciated. (If there are ways to improve the performance of the update itself I'd also be more than happy for any advices.)
the default socket timeout is 5 minutes.
You can override the default by setting sockettimeoutms directly in your connection URI:
mongoURI <- paste0("mongodb://", user,":",pass, "#", mongoHost, ":", mongoPort,"/",db,"?sockettimeoutms=<something large enough in milliseconds>")
mcon <- mongo(mongoCollection, url=mongoURI)
mcon$update(...)
I am not an admin so I can't change the scopes. I can send slackr_bot messages to a channel I set up in the creation of the app in UI but doing the below does not work. Has anyone found a solution to this?
I created a txt file called: test.txt
Within that txt file it looks like this:
api_token: xxxxxxxxxxxx
channel: #channel_name
username: myusername
incoming_webhook_url: https://hooks.slack.com/services/xxxxxxxxxxx/xxxxxxxxxxxxx
Then I want to simply send a message but eventually I would like to run the function
ggslackr(qplot(mpg, wt, data=mtcars))
slackr_setup(config_file = "test.txt")
my_message <- paste("I'm sending a Slack message at", Sys.time(), "from my R script.")
slackr_msg(my_message, channel = "#channel_name", as_user=F)
Here is the error message:
Error: Join columns must be present in data.
x Problem with `id`.
Run `rlang::last_error()` to see where the error occurred.
In addition: Warning message:
In structure(vars, groups = group_vars, class = c("dplyr_sel_vars", :
Calling 'structure(NULL, *)' is deprecated, as NULL cannot have attributes.
Consider 'structure(list(), *)' instead.
Edit #2:
Okay, I learned some things regarding packages. If I had to do this over, I'd have gone to their github repo and read the issue tracker.
The reason is that it appears that slackr has a few issues related to changes in Slack's API.
And also since there has been a large updating of R (version 4.x) a lot of packages got broken.
My sense is that our issue is with a line of code inside a slackr function (slackr_util.r--iirc) that calls a dplyr join that is looking for a particular id that does not exist.
So, I'm going to watch the issue tracker and see what comes of it.
Edit: Try slackr_bot(my_message,channel = "#general")
worked as advertised!
But ggslackr continues to fail.
I'm having the same issue. I've found in another thread a debugging start:
`rlang::last_error()`
When I run that,
Backtrace:
1. slackr::slackr_msg(my_message, channel = "#general")
5. slackr::slackr_chtrans(channel)
6. slackr::slackr_ims(api_token)
8. dplyr:::left_join.data.frame(users, ims, by = "id", copy = TRUE)
9. dplyr:::join_mutate(...)
10. dplyr:::join_cols(...)
11. dplyr:::standardise_join_by(by, x_names = x_names, y_names = y_names)
12. dplyr:::check_join_vars(by$y, y_names)
So, step 8 there is a join effort by id, which I suppose this implies that 'id' is missing.
yet, if I run from github issue tracker : slackr::slackrSetup(echo=TRUE) I get the following:
{
"SLACK_CHANNEL": ["#general"],
"SLACK_USERNAME": ["slackr_brian"],
"SLACK_ICON_EMOJI": ["NA"],
"SLACK_INCOMING_URL_PREFIX": ["https://hooks.xxxxxxx"],
"SLACK_API_TOKEN": ["token secret"]
}
I'm not sure where to go from here as the issue tracker conversation makes mention of confirming webhooks going to the correct channel and becomes very user specific.
So, that's as far as I have gotten.
I'm trying to extract abstract from Scopus, using rscopus and with the functions that I got from https://github.com/christopherBelter/scopusAPI
I have the API key using my university account, but when trying to save the data in xml, with:
theXML <- searchByString(string = query, outfile = "testdata.xml")
I got an error:
Error in searchByString(string = query, outfile = "testdata.xml") : Unauthorized (HTTP 401).
3. stop(http_condition(x, "error", task = task, call = call))
2. httr::stop_for_status(theURL) at scopusAPI.R#12
1. searchByString(string = query, outfile = "testdata.xml")
Does something wrong with my API key since ": Unauthorized (HTTP 401)"?
Have you tried validating your string outside of the script?
The Elsevier Developer Portal would be a good place to try out your string to confirm it works or to refine it so it works
"https://dev.elsevier.com/scopus.html"
For one of my dissertation's data collection modules, I have implemented a simple polling mechanism. This is needed, because I make each data collection request (one of many) as SQL query, submitted via Web form, which is simulated by RCurl code. The server processes each request and generates a text file with results at a specific URL (RESULTS_URL in code below). Regardless of the request, URL and file name are the same (I cannot change that). Since processing time for different data requests, obviously, is different and some requests may take significant amount of time, my R code needs to "know", when the results are ready (file is re-generated), so that it can retrieve them. The following is my solution for this problem.
POLL_TIME <- 5 # polling timeout in seconds
In function srdaRequestData(), before making data request:
# check and save 'last modified' date and time of the results file
# before submitting data request, to compare with the same after one
# for simple polling of results file in srdaGetData() function
beforeDate <- url.exists(RESULTS_URL, .header=TRUE)["Last-Modified"]
beforeDate <<- strptime(beforeDate, "%a, %d %b %Y %X", tz="GMT")
<making data request is here>
In function srdaGetData(), called after srdaRequestData()
# simple polling of the results file
repeat {
if (DEBUG) message("Waiting for results ...", appendLF = FALSE)
afterDate <- url.exists(RESULTS_URL, .header=TRUE)["Last-Modified"]
afterDate <- strptime(afterDate, "%a, %d %b %Y %X", tz="GMT")
delta <- difftime(afterDate, beforeDate, units = "secs")
if (as.numeric(delta) != 0) { # file modified, results are ready
if (DEBUG) message(" Ready!")
break
}
else { # no results yet, wait the timeout and check again
if (DEBUG) message(".", appendLF = FALSE)
Sys.sleep(POLL_TIME)
}
}
<retrieving request's results is here>
The module's main flow/sequence of events is linear, as follows:
Read/update configuration file
Authenticate with the system
Loop through data requests, specified in configuration file (via lapply()),
where for each request perform the following:
{
...
Make request: srdaRequestData()
...
Retrieve results: srdaGetData()
...
}
The issue with the code above is that it doesn't seem to be working as expected: upon making data request, the code should print "Waiting for results ..." and then, periodically checking the results file for being modified (re-generated), print progress dots until the results are ready, when it prints confirmation. However, the actual behavior is that the code waits long time (I intentionally made one request a long-running), not printing anything, but then, apparently retrieves results and prints both "Waiting for results ..." and " Ready" at the same time.
It seems to me that it's some kind of synchronization issue, but I can't figure out what exactly. Or, maybe it's something else and I'm somehow missing it. Your advice and help will be much appreciated!
In a comment to the question, I believe MrFlick solved the issue: the polling logic appears to be functional, but the problem is that the progress messages are out of synch with current events on the system.
By default, the R console output is buffered. This is by design: to speed things up and avoid the distracting flicker that may be associated with frequent messages etc. We tend to forget this fact, particularly after we've been using R in a very interactive fashion, running various ad-hoc statement at the console (the console buffer is automatically flushed just before returning the > prompt).
It is however possible to get message() and more generally console output in "real time" by either explicitly flushing the console after each critical output statement, using the flush.console() function, or by disabling buffering at the level of the R GUI (right-click when on the console, see Buffered output Ctrl W item. This is also available in the Misc menu)
Here's a toy example of the explicit use of flush.console. Note the use of cat() rather than message() as the former doesn't automatically add a CR/LF to the output. The latter however is useful however because its messages can be suppressed with suppressMessages() and the like. Also as shown in the comment you can cat the "\b" (backspace) character to make the number overwrite one another.
CountDown <- function() {
for (i in 9:1){
cat(i)
# alternatively to cat(i) use: message(i)
flush.console() # <<<<<<< immediate ouput to console.
Sys.sleep(1)
cat(" ") # also try cat("\b") instead ;-)
}
cat("... Blast-off\n")
}
The output is the following, what is of course not evident in this print-out is that it took 10 seconds overall with one number printed every second, before the final "Blast off"; do remove the flush.console() statement and the output will come at once, after 10 seconds, i.e. when the function terminates (unless console is not buffered at the level of the GUI).
CountDown()
9 8 7 6 5 4 3 2 1 ... Blast-off
I've written some PROGRESS code that outputs some data to a user defined file. The data itself isn't important, the output process works fine. It's basically
DEFINE VARIABLE filePath.
UPDATE filePath /*User types in something like C:\UserAccount\New.txt */
OUTPUT TO (VALUE) filePath.
Which works fine, a txt file is created in the input directory. My question is:
Does progress have any functionality that would allow me to check if an input
file path is valid? (Specifically, if the user has input a valid directory, and if they have permission to create a file in the directory they've chosen)
Any input or feedback would be appreciated.
FILE-INFO
Using the system handle FILE-INFO gives you a lot of information. It also works on directories.
FILE-INFO:FILE-NAME = "c:\temp\test.p".
DISPLAY
FILE-INFO:FILE-NAME
FILE-INFO:FILE-CREATE-DATE
FILE-INFO:FILE-MOD-DATE
FILE-INFO:FILE-INFO
FILE-INFO:FILE-MOD-TIME
FILE-INFO:FILE-SIZE
FILE-NAME:FILE-TYPE
FILE-INFO:FULL-PATHNAME
WITH FRAME f1 1 COLUMN SIDE-LABELS.
A simple check for existing directory with write rights could be something like:
FUNCTION dirOK RETURNS LOGICAL (INPUT pcDir AS CHARACTER):
FILE-INFO:FILE-NAME = pcDir.
IF INDEX(FILE-INFO:FILE-TYPE, "D") > 0
AND INDEX(FILE-INFO:FILE-TYPE, "W") > 0 THEN
RETURN TRUE.
ELSE
RETURN FALSE.
END FUNCTION.
FILE-NAME:FILE-TYPE will start with a D for directories and a F for plain files. It also includes information about reading and writing rights. Check the help for more info. If the file doesn't exist basically all attributes except FILE-NAME will be empty or unknown (?).
Edit: it seems that FILE-TYPE returns W in some cases even if there's no actual writing rights in that directory so I you might need to handle this through error processing instead
ERROR PROCESSING
OUTPUT TO VALUE("f:\personal\test.txt").
PUT UNFORMATTED "Test" SKIP.
OUTPUT CLOSE.
CATCH eAnyError AS Progress.Lang.ERROR:
/* Here you could check for specifically error no 98 indicating a problem opening the file */
MESSAGE
"Error message and number retrieved from error object..."
eAnyError:GetMessage(1)
eAnyError:GetMessageNum(1) VIEW-AS ALERT-BOX BUTTONS OK.
END CATCH.
FINALLY:
END FINALLY.
SEARCH
When checking for a single file the SEARCH command will work. If the file exists it returns the complete path. It does however not work on directory, only files. If you SEARCH without complete path e g SEARCH("test.p") the command will search through the directories set in the PROPATH environment variable and return the first matching entry with complete path. If there's no match it will return unknown value (?).
Syntax:
IF SEARCH("c:\temp\test.p") = ? THEN
MESSAGE "No such file" VIEW-AS ALERT-BOX ERROR.
ELSE
MESSAGE "OK" VIEW-AS ALERT-BOX INFORMATION.
SYSTEM-DIALOG GET-FILE character-field has an option MUST-EXIST if you want to use a dailogue to get filename/dir from user. Example from manual
DEFINE VARIABLE procname AS CHARACTER NO-UNDO.
DEFINE VARIABLE OKpressed AS LOGICAL INITIAL TRUE.
Main:
REPEAT:
SYSTEM-DIALOG GET-FILE procname
TITLE "Choose Procedure to Run ..."
FILTERS "Source Files (*.p)" "*.p",
"R-code Files (*.r)" "*.r"
MUST-EXIST
USE-FILENAME
UPDATE OKpressed.
IF OKpressed = TRUE THEN
RUN VALUE(procname).
ELSE
LEAVE Main.
END.