I am streaming strings and parsing them into floats/ints from a .dat. Everything is going well until I work with the last item in the string, which returns an error.
Example Data:
x = "15505052|B|Other|2.22|250022|2776|09/2019|01/2004|"
y = split(x, "|") # <-- one inner loop of the loop
if isblank(y[9]) == false
floater = parse(Float64, y[9])
end
This is the error I receive:
ArgumentError("float64(String): invalid number format")
while loading In[42], in expression starting on line 56
in float64 at string.jl:1613
in parse at /Users/laptop/.julia/v0.3/Compat/src/Compat.jl:217
in loan_aqsn at In[42]:36
I have visually inspected the last column in the above data strings and determined that the data should be parseable into a Float64, and indeed every other column works just fine.
What I have found is that I can individually parse lines from the .dat just fine (for testing purposes), but when I move to streaming the file Julia throws the error. The only thing I can figure out about this error is that it has to due with working with the final record in the string y[9], which makes no sense to me.
Any help with this frustrating problem is much appreciated.
You're testing whether y is blank, when it is in fact an empty string:
x = "15505052|B|Other|2.22|250022|2776|09/2019|01/2004|"
y = split(x)
y[9] # ""
isblank(y[9]) # false, since we have an empty string
float(y[9]) # ERROR: ArgumentError("float64(String): invalid number format")
You should modify your code to use isempty() instead
using Compat # for parse(Float, String)
if !isempty(y[9])
floater = parse(Float64, y[9])
end
# do stuff
Relevant manual section (for v0.3)
Which julia version are you using? Mine doesn't recognize parse(fmt,str) as a valid command (v 0.3.10).
Try really parsing the numbers, not just converting them. That way the divisions are properly executed (like 01/2004 in the second x).
strs = ["12", "12.1", "12/3"]
for i in strs
a::Float64 = eval(parse(i))
println(a)
end
this works fine in my julia REPL, returning 12.0,12.1 and 4.0.
Note that I am "executing" those numbers, opening a security concern if I use this code for other thing than a quick script.
Related
I'm in the process of working on a coding project in a pluto notebook. It's early days but I'm curious why the value of 1000 doesn't show beside quadorder in the screenshot above. It seems to output for everything else!
It is the ; in your code that triggers that. However, I don't think it is supposed to be expected behavior. It is supposed to be a comment, but the parser probably sees the ; and thinks that it is part of the code.
If you put it between quoting marks then this doesn't happen.
I.e. this should work as expected:
quadorder = 1000 # like python Julia doesn't require ';'
Otherwise, it would usually also work if you just don't put it at the end of the line.
The Julia REPL evaluates the semicolon wrongly and it is a know issue github.com/JuliaLang/julia/issues/28743.
You can actually trick it into some weird behavior that suppresses output. For example this returns no output:
julia> ";#"
julia> a = ["; #", "hi", 3]
julia> a = "223" #;
The reason is that the REPL parser looks for the last semicolon in the line and if there is any # or if it is at the end of the line (whitespaces don't matter), then it suppresses any output.
I've written a prime-generating function generatePrimes (full code here) that takes input bound::Int64 and returns a Vector{Int64} of all primes up to bound. After the function definition, I have the following code:
println("Generating primes...")
println("Last prime: ", generatePrimes(10^7)[end])
println("Primes generated.")
which prints, unexpectedly,
Generating primes...
9999991
Primes generated.
This output misses the "Last prime: " segment of the second print statement. The output does work as expected for smaller inputs; any input at least up to 10^6, but somehow fails for 10^7. I've tried several workarounds for this (e.g. assigning the returned value or converting it to a string before calling it in a print statement, combining the print statements, et cetera) and discovered some other weird behaviour: if the "Last prime", is removed from the second print statement, for input 10^7, the last prime doesn't print at all and all I get is a blank line between the first and third print statements. These issues are probably related, and I can't seem to find anything online about why some print statements wouldn't work in Julia.
Thanks so much for any clarification!
Edit: Per DNF's suggestion, following are some reductions to this issue:
Removing the first and last print statements doesn't change anything -- a blank line is always printed in the case I outlined and each of the cases below.
println(generatePrimes(10^7)[end]) # output: empty line
Calling the function and storing the last index in a variable before calling println doesn't change anything either; the cases below work exactly the same either way.
lastPrime::Int = generatePrimes(10^7)[end]
println(lastPrime) # output: empty line
If I call the function in whatever form immediately before a println, an empty line is printed regardless of what's inside the println.
lastPrime::Int = generatePrimes(10^7)[end]
println("This doesn't print") # output: empty line
println("This does print") # output: This does print
If I call the function (or print the pre-generated-and-stored function result) inside a println, anything before the function call (that's also inside the println) isn't printed. The 9999991 and anything else there may be after the function call is printed only if there is something else inside the println before the function call.
# Example 1
println(generatePrimes(10^7)[end]) # output: empty line
# Example 2
println("This first part doesn't print", generatePrimes(10^7)[end]) # output: 9999991
# Example 3
println("This first part doesn't print", generatePrimes(10^7)[end], " prints") # output: 9999991 prints
# Example 4
println(generatePrimes(10^7)[end], "prime doesn't print") # output: prime doesn't print
I could probably list twenty different variations of this same thing, but that probably wouldn't make things any clearer. In every single case version of this issue I've seen so far, the issue only manifests if there's that function call somewhere; println prints large integers just fine. That said, please let me know if anyone feels like they need more info. Thanks so much!
Most likely you are running this code from Atom Juno which recently has some issues with buffering standard output (already reported by others and I also sometimes have this problem).
One thing you can try to do is to flush your standard output
flush(stdout)
Like with any unstable bug restarting Atom Juno also seems to help.
I had the same issue. For me, changing the terminal renderer (File -> Settings -> Packages -> julia-client -> Terminal Options) from webgl to canvas (see pic below) seems to solve the issue.
change terminal renderer
I've also encountered this problem many times. (First time, it was triggered after using the debugger. It is probably unrelated but I have been using Julia+Juno for 2 weeks prior to this issue.)
In my case, the code before the println statement needed to have multiple dictionary assignation (with new keys) in order to trigger the behavior.
I also confirmed that the same code ran in Command Prompt (with same Julia interpreter) prints fine. Any hints about how to further investigate this will be appreciated.
I temporarily solve this issue by printing to stderr, thinking that this stream has more stringent flush mechanism: println(stderr, "hello!")
So, I'm brushing up on how to work with data frames in R and I came across this little bit of code from https://cloud.r-project.org/web/packages/data.table/vignettes/datatable-intro.html:
input <- if (file.exists("flights14.csv")) {
"flights14.csv"
} else {
"https://raw.githubusercontent.com/Rdatatable/data.table/master/vignettes/flights14.csv"
}
Apparently, this assigns the strings (character vectors?) in the if and else statements to input based on the conditional. How is this working? It seems like magic. I am hoping to find somewhere in the official R documentation that explains this.
From other languages I would have just done:
if (file.exists("flights14.csv")) {
input <- "flights14.csv"
} else {
input <- "https://raw.githubusercontent.com/Rdatatable/data.table/master/vignettes/flights14.csv"
}
or in R there is ifelse which also seems designed to do exactly this, but somehow that first example also works. I can memorize that this works but I'm wondering if I'm missing the opportunity to understand the bigger picture about how R works.
From the documentation on the ?Control help page under "Value"
if returns the value of the expression evaluated, or NULL invisibly if none was (which may happen if there is no else).
So the if statement is kind of like a function that returns a value. The value that's returned is the result of either evaulating the if or the then block. When you have a block in R (code between {}), the brackets are also like a function that just return the value of the last expression evaluated in the block. And a string literal is a valid expression that returns itself
So these are the same
x <- "hello"
x <- {"hello"}
x <- {"dropped"; "hello"}
x <- if(TRUE) {"hello"}
x <- if(TRUE) {"dropped"; "hello"}
x <- if(TRUE) {"hello"} else {"dropped"}
And you only really need blocks {} with if/else statements when you have more than one expression to run or when spanning multiple lines. So you could also do
x <- if(TRUE) "hello" else "dropped"
x <- if(FALSE) "dropped" else "hello"
These all store "hello" in x
You are not really missing anything about the "big picture" in R. The R if function is atypical compared both to other languages as well as to R's typical behavior. Unlike most functions in R which do require assignment of their output to a "symbol", i.e a proper R name, if allows assignments that occur within its consequent or alternative code blocks to occur within the global environment. Most functions would return only the final evaluation, while anything else that occurred inside the function body would be garbage collected.
The other common atypical function is for. R for-loops only
retain these interior assignments and always return NULL. The R Language Definition calls these atypical R functions "control structures". See section 3.3. On my machine (and I suspect most Linux boxes) that document is installed at: http://127.0.0.1:10731/help/doc/manual/R-lang.html#Control-structures. If you are on another OS then there is probably a pulldown Help menu in your IDE that will have a pointer to it. Thew help document calls them "control flow constructs" and the help page is at ?Control. Note that it is necessary to quote these terms when you wnat to access that help page using one of those names since they are "reserved words". So you would need ?'if' rather than typing ?if. The other reserved words are described in the ?Reserved page.
?Control
?'if' ; ?'for'
?Reserved
# When you just type:
?if # and hit <return>
# you will see a "+"-sign which indicateds an incomplete expression.
# you nthen need to hit <escape> to get back to a regular R interaction.
In R, functions don't need explicit return. If not specified the last line of the function is automatically returned. Consider this example :
a <- 5
b <- 1
result <- if(a == 5) {
a <- a + 1
b <- b + 1
a
} else {b}
result
#[1] 6
The last line in if block was saved in result. Similarly, in your case the string values are "returned" implicitly.
I am trying to figure out if it is possible, with a sane amount of programming, to create a certain debugging function by using R's metaprogramming features.
Suppose I have a block of code, such that each line uses as all or part of its input the output from thee line before -- the sort of code you might build with pipes (though no pipe is used here).
{
f1(args1) -> out1
f2(out1, args2) -> out2
f3(out2, args3) -> out3
...
fn(out<n-1>, args<n>) -> out<n>
}
Where for example it might be that:
f1 <- function(first_arg, second_arg, ...){my_body_code},
and you call f1 in the block as:
f1(second_arg = 1:5, list(a1 ="A", a2 =1), abc = letters[1:3], fav = foo_foo)
where foo_foo is an object defined in the calling environment of f1.
I would like a function I could wrap around my block that would, for each line of code, create an entry in a list. Each entry would be named (line1, line2) and each line entry would have a sub-entry for each argument and for the function output. the argument entries would consist, first, of the name of the formal, to which the actual argument is matched, second, the expression or name supplied to that argument if there is one (and a placeholder if the argument is just a constant), and third, the value of that expression as if it were immediately forced on entry into the function. (I'd rather have the value as of the moment the promise is first kept, but that seems to me like a much harder problem, and the two values will most often be the same).
All the arguments assigned to the ... (if any) would go in a dots = list() sublist, with entries named if they have names and appropriately labeled (..1, ..2, etc.) if they are assigned positionally. The last element of each line sublist would be the name of the output and its value.
The point of this is to create a fairly complete record of the operation of the block of code. I think of this as analogous to an elaborated version of purrr::safely that is not confined to iteration and keeps a more detailed record of each step, and indeed if a function exits with an error you would want the error message in the list entry as well as as much of the matched arguments as could be had before the error was produced.
It seems to me like this would be very useful in debugging linear code like this. This lets you do things that are difficult using just the RStudio debugger. For instance, it lets you trace code backwards. I may not know that the value in out2 is incorrect until after I have seen some later output. Single-stepping does not keep intermediate values unless you insert a bunch of extra code to do so. In addition, this keeps the information you need to track down matching errors that occur before promises are even created. By the time you see output that results from such errors via single-stepping, the matching information has likely evaporated.
I have actually written code that takes a piped function and eliminates the pipes to put it in this format, just using text manipulation. (Indeed, it was John Mount's "Bizarro pipe" that got me thinking of this). And if I, or we, or you, can figure out how to do this, I would hope to make a serious run on a second version where each function calls the next, supplying it with arguments internally rather than externally -- like a traceback where you get the passed argument values as well as the function name and and formals. Other languages have debugging environments like that (e.g. GDB), and I've been wishing for one for R for at least five years, maybe 10, and this seems like a step toward it.
Just issue the trace shown for each function that you want to trace.
f <- function(x, y) {
z <- x + y
z
}
trace(f, exit = quote(print(returnValue())))
f(1,2)
giving the following which shows the function name, the input and output. (The last 3 is from the function itself.)
Tracing f(1, 2) on exit
[1] 3
[1] 3
I am running a data set (in the example, "data object ") through several different functions in R and concatenating the numeric results at the end. See:
a<-median((function1(x=1,dataobject,reps=500)),na.rm=TRUE)
b<-median((function2(x=1,dataobject,reps=500)),na.rm=TRUE)
c<-median((function3(x=1,dataobject,reps=500)),na.rm=TRUE)
d<-median((function4(x=1,dataobject,reps=500)),na.rm=TRUE)
e<-median((function5(x=1,dataobject,reps=500)),na.rm=TRUE)
f<-median((function6(x=1,dataobject,reps=500)),na.rm=TRUE)
c(a,b,c,d,e,f)
However, some of the functions cannot be run with the data set I am using, and so they return an error; e.g. "function3" can't be run so when it gets to the concatenation step it gives "Error: object 'e' not found" and does not return anything. Is there any way to tell R at the concatenation step to assign a value of "NA" to an object that is not found and continue to run the rest of the code instead of stopping? So that the return would be
[1] 99.233 75.435 77.782 92.013 NA 97.558
A simple question, but I could not find any other instances of it being asked. I originally tried to set up a function to run everything and output the concatenated results, but ran into the same problem (when a function can't be run, the entire wrapper function stops as well and I don't know how to tell R to skip something it can't compute).
Any thoughts are greatly appreciated! Thanks!
A couple of solutions I can think of,
Initialize all the variables you plan to use, so they have a default value that you want.
a = b = c = d = e = NA
then run your code. If an error pops up, you will have NA in the variable.
Use "tryCatch". If you are unaware what this is, I recommend reading on it. It lets you handle errors.
Here is an example from your code,
tryCatch({
a<-median((function1(x=1,dataobject,reps=500)),na.rm=TRUE)
},
error = function(err){
print("Error in evaluating a. Initializing it to NA")
a <<- NA
})