How to feed data properly in tensorflow - dictionary

I have been learning Tensorflow and understanding feed_dict has been a challenge. Take for example the following piece of code i am working on
p=0
self.sequence_length=25
with tf.Session() as sess:
init.run()
char_to_ix={ch:ix for ix,ch in enumerate(self.words)}
ix_to_char={ix:ch for ix,ch in enumerate(self.words)}
words_in_input=self.data[p:p+self.sequence_length]
inputs=[char_to_ix[ix] for ix in words_in_input]
words_in_target=self.data[p+1:p+self.sequence_length+1]
targets=[char_to_ix[ix] for ix in words_in_target]
onex=sess.run([selected_next_letter],feed_dict={self.X:inputs,self.y:targets})
p=p+1
This gives the error: Shapes of all inputs must match: values[0].shape = [25] != values[1].shape = []
However, when I edit the code to
with tf.Session() as sess:
init.run()
char_to_ix={ch:ix for ix,ch in enumerate(self.words)}
ix_to_char={ix:ch for ix,ch in enumerate(self.words)}
words_in_input=self.data[p:p+self.sequence_length]
inputs=[char_to_ix[ix] for ix in words_in_input]
words_in_target=self.data[p+1:p+self.sequence_length+1]
targets=[char_to_ix[ix] for ix in words_in_target]
for x,y in zip(inputs,targets):
onex=sess.run([selected_next_letter],feed_dict={self.X:x,self.y:y})
It executes.
My questions is: Is it possible to feed the whole list such as inputs and targets in the feed_dict or must I input it through a loop one by one. I ask this because the tutorials I have been reading, I see a whole list being passed in a feed_dict such as
loss_val = sess.run([train_op, loss_mean], feed_dict={
images_batch:images_batch_val,
labels_batch:labels_batch_val
})

Usually the reason for that error is because your input array(x) isn’t the same size as your labels array(y). As the error states it looks like your labels array is empty. Before doing anything tensorflowy make sure both x and y arrays have values in them and that they are of the same size.
To answer your question, yes you can use lists when training and is the preferred way of using tensorflow.

Related

OpenMDAO Optional Error on Unconnected Input

Is there any way to force OpenMDAO to raise an error if a given input is unconnected? I know for many inputs, defaults can be provided such that the input doesn't need to be connected, however is there a way to tell OpenMDAO to automatically raise an error if certain key inputs are unconnected?
This is not built into OpenMDAO, as of V3.17. However, it is possible to do it. The only caveat is that i had to use some non public APIs to make it work (notice the use of the p.model._conn_global_abs_in2out). So those APIs are subject to changer later.
This code should give you the behavior your want. You could augment things with the use of variable tagging if you wanted a solution that didn't require you to give a list of variable names to the validate function. The list_inputs method can accept tags to filter by instead if you prefer that.
import openmdao.api as om
def validate_connections(prob, force_connected):
# make sure its a set and not a generic iterator (i.e. array)
force_connected_set = set(force_connected)
model_inputs = prob.model.list_inputs(out_stream=None, prom_name=True)
#gets the promoted names from the list of inputs
input_set = set([inp[1]['prom_name'] for inp in model_inputs])
# filter the inputs into connected and unconnected sets
connect_dict = p.model._conn_global_abs_in2out
unconnected_inputs = set()
connected_inputs = set()
for abs_name, in_data in model_inputs:
if abs_name in connect_dict and (not 'auto_ivc' in connect_dict[abs_name]):
connected_inputs.add(in_data['prom_name'])
else:
unconnected_inputs.add(in_data['prom_name'])
# now we need to check if there are any unconnected inputs
# in the model that aren't in the spec
illegal_unconnected = force_connected_set.intersection(unconnected_inputs)
if len(illegal_unconnected) > 0:
raise ValueError(f'unconnected inputs {illegal_unconnected} are are not allowed')
p = om.Problem()
###############################################################################################
# comment and uncomment these three lines to change the error you get from the validate method
###############################################################################################
# p.model.add_subsystem('c0', om.ExecComp('x=3*a'), promotes_outputs=['x'])
# p.model.add_subsystem('c1', om.ExecComp('b=a+17'))
# p.model.connect('c1.b', 'c2.b')
p.model.add_subsystem('c2', om.ExecComp('y=2*x+b'), promotes_inputs=['x'])
p.model.add_subsystem('c3', om.ExecComp('z=x**2+y'))
p.setup()
p.final_setup()
If this is a feature you think should be added to OpenMDAO proper, then feel free to submit a POEM proposing how a formal feature and its API might look.

Accessing API with for-loop randomly has encoding error, which breaks loop in R

I'm trying to access an API from iNaturalist to download some citizen science data. I'm using the package rinat to get this done (see vignette). The loop below is, essentially, pulling all observations for one species, in one state, in one year iteratively on a per-month basis, then summing the number of observations for that year (input parameters subset from my actual script for convenience).
require(rinat)
state_ids <- c(18, 14)
bird_ids <- c(14886,1409)
months <- c(1:12)
final_nums <- vector()
for(i in 1:length(state_ids)){
total_count <- vector()
for(j in 1:length(months)){
monthly <- get_inat_obs(place_id=state_ids[i],
taxon_id=bird_ids[i],
year=2019,
month = months[j])
total_count <- append(total, length(monthly$scientific_name))
print(paste("done with month", months[j], "in state", state_ids[i]))
}
final_nums <- append(final_nums, sum(total_count))
print(paste("done with state", state_ids[i]))
}
Occasionally, and seemingly randomly, I get the following error:
No encoding supplied: defaulting to UTF-8.
Error in if (!x$headers$`content-type` == "text/csv; charset=utf-8") { :
argument is of length zero
This ends up breaking the loop or makes the loop run without actually pulling any real data. Is this an issue with my script, or the API, or something else? I've tried manually supplying encoding information to the get_inat_obs() function, but it doesn't accept that as an argument. Thank you in advance!
I don't believe this is an error in your script. The issue is with the api most likely.
the error argument is of length zero is a common error when you try to make a comparison that has no length. For example:
if(logical(0) == "TEST") print("WORKED!!")
#Error in if (logical(0) == "TEST") print("WORKED!!") :
# argument is of length zero
I did some a few greps on their source code to see where this if statement is and it seems to be within inat_handle line 211 in get_inate_obs.R
This would suggest that the authors did not expect for
!x$headers$`content-type` == 'text/csv; charset=utf-8'
to evaluate to logical(0), but more specifically
x$headers$`content-type`
to be NULL.
I would suggest making a bug report on their GitHub and recommend they change the specified line to:
if(is.null(x$headers$`content-type`) || !x$headers$`content-type` == 'text/csv; charset=utf-8'){
Suggesting a bug is usually more well received if you have a reproducible example.
Also, you could totally make this change yourself locally by cloning out the git repository, editing the file, rebuild the package, and then confirm if you no longer get an error in your code.

Search list of keywords in elastic using R elastic package

I am making a shiny app where a user can input an excel file of terms and search them on elastic holdings. The excel file contains one column of keywords and is read into a list, then what I am trying to do is have the each item in the list searched using Search(). I easily did this in Python with a for loop over the terms and then the search connection inside the for loop and got accurate results. I understand that it isn't that easy in R, but I cannot get to the right solution. I am using the R elastic package and have been trying different versions of Search() for over a day. I have not used elastic before so my apologies for not understanding the syntax much. I know that I need to do something with aggs for a list of terms..
Essentially I want to search on the source.body_ field and I want to use match_phrase for searching my terms.
Here is the code I have in Python that works, but I need everything in R for the shiny app and don't want to use reticulate.
queries = list()
for term in my_terms:
search_result = es.search(index="cars", body={"query": {"match_phrase": {'body_':term}}}, size = 5000)
search_result.update([('term', term)])
queries.append(search_result)
I established my elastic connection as con and made sure it can bring back accurate matches on just one keyword with:
match <- {"query": {"match_phrase" : {"body_" : "mustang"}}}
search_results <- Search(con, index="cars", body = match, asdf = TRUE)
That worked how I expected it to with just one keyword explicitly defined.
So after that, here is what I have tried for a list of my_terms:
aggs <- '{"aggs":{"stats":{"terms":{"field":"my_terms"}}}}'
queries <- data.frame()
for (term in my_terms) {
final <- Search(con, index="cars", body = aggs, asdf = TRUE, size = 5000)
rbind(queries, final)
}
When I run this, it just brings back everything in my elastic. I also tried running that with the for loop commented out and that didn't work.
I also have tried to embed my_terms inside my match list from the single term search at the beginning of this post like so:
match <- '{"query": {"match_phrase" : {"body_": "my_terms"}}}'
Search(con, index="cars", body = match, asdf = TRUE, size = 5000)
This returns nothing. I have tried so many combinations of the aggs list and match list but nothing returns what I'm looking for. Any advice would be much appreciated, I feel like I've read just about everything so far and now I'm just confused.
UPDATE:
I figured it out with
p <- data.frame()
for (t in the_terms$keyword) {
result <- Search(con, index="cars", body = paste0('{"query": {"match_phrase" : {"body_":', '"', t, '"', '}}}'), asdf = TRUE, size = 5000)
p <- rbind(p, result$hit$hit)
}

window function R code

fine people of stackoverflow. I have become trapped on a rather simple part of my program and was wondering if you guys could help me.
library(nonlinearTseries)
tt<-c(0,500,1000)
mm<-rep(0,2)
for (j in 1:2){mm[j]=estimateEmbeddingDim(window(rnorm(1000), start=tt[j],end=tt[j+1]), number.points=(tt[j+1]-tt[j]),do.plot=FALSE)}
Warning message:
In window.default(rnorm(1000), start = tt[j], end = tt[j + 1]) :
'start' value not changed
If I plug in the values directly (tt[1], tt[2], tt[3]), it works but I also get a warning
estimateEmbeddingDim(window(rnorm(1000), start=tt[1],end=tt[2]), number.points=(tt[2]-tt[1]),do.plot=FALSE)
[1] 9
Warning message:
In window.default(rnorm(1000), start = tt[1], end = tt[2]) :
'start' value not changed
Thanks, Matt.
The problem seems to be with the
window(rnorm(1000), start=tt[j],end=tt[j+1])
lines. First of all, window is only meant to be used with a time series object (class=="ts"). In this case, rnorm(1000) simply returns a numeric vector, there are no dates associated with this object. So i'm not sure what you think this function does. Did you only want to extract the values that were between 0-500 and 500-1000? If so that seems a bit because with a standard normal variable, the max of 1000 samples isn't likely to be much over 4 let alone 500.
So be sure to use a proper "ts" object with dates and everything to get this to work.

Retrieving Variable Declaration

How can I find how did I first declare a certain variable when I am a few hundred
lines down from where I first declared it. For example, I have declared the following:
a <- c(vectorA,vectorB,vectorC)
and now I want to see how I declared it. How can I do that?
Thanks.
You could try using the history command:
history(pattern = "a <-")
to try to find lines in your history where you assigned something to the variable a. I think this matches exactly, though, so you may have to watch out for spaces.
Indeed, if you type history at the command line, it doesn't appear to be doing anything fancier than saving the current history in a tempfile, loading it back in using readLines and then searching it using grep. It ought to be fairly simple to modify that function to include more functionality...for example, this modification will cause it to return the matching lines so you can store it in a variable:
myHistory <- function (max.show = 25, reverse = FALSE, pattern, ...)
{
file1 <- tempfile("Rrawhist")
savehistory(file1)
rawhist <- readLines(file1)
unlink(file1)
if (!missing(pattern))
rawhist <- unique(grep(pattern, rawhist, value = TRUE,
...))
nlines <- length(rawhist)
if (nlines) {
inds <- max(1, nlines - max.show):nlines
if (reverse)
inds <- rev(inds)
}
else inds <- integer()
#file2 <- tempfile("hist")
#writeLines(rawhist[inds], file2)
#file.show(file2, title = "R History", delete.file = TRUE)
rawhist[inds]
}
I will assume you're using the default R console. If you're on Windows, you can File -> Save history and open the file in your fav text browser, or you can use function savehistory() (see help(savehistory)).
What you need to do is get a (good) IDE, or at least a decent text editor. You will benevit from code folding, syntax coloring and much more. There's a plethora of choices, from Tinn-R, VIM, ESS, Eclipse+StatET, RStudio or RevolutionR among others.
You can run grep 'a<-' .Rhistory from terminal (assuming that you've cdd to your working directory). ESS has several very useful history-searching functions, like (comint-history-isearch-backward-regexp) - binded to M-r by default.
For further info, consult ESS manual: http://ess.r-project.org/Manual/ess.html
When you define a function, R stores the source code of the function (preserving formatting and comments) in an attribute named "source". When you type the name of the function, you will get this content printed.
But it doesn't do this with variables. You can deparse a variable, which generates an expression that will produce the variable's value but this doesn't need to be the original expression. For example when you have b <- c(17, 5, 21), deparse(b) will produce the string "c(17, 5, 21)".
In your example, however, the result wouldn't be "c(vectorA,vectorB,vectorC)", it would be an expression that produces the combined result of your three vectors.

Resources