RAmazonS3 broken pipe when upload files - r

I'm trying to use RAmazonS3 to upload a local file to S3 storage, but I keep getting a broken pipe error.
require(RAmazonS3)
options(AmazonS3 = c('xxx' = "xxx")) #login and secret
setwd('[local directory]/reports') #set working directory to location of "polarity.png"
addFile("polarity.png", "umusergen", "destination.png",type="image/png",meta = c(foo = 123, author = "Duncan Temple Lang"))
Send failure: Broken Pipe
It works fine if I just try to upload content
addFile(I("This is a test"), "umusergen", "destination.png",type="text",meta = c(foo = 123, author = "Duncan Temple Lang"))

addFile() is quite simplistic and is focused on text content unless told otherwise.
Use
content = readBin("polarity.png", raw(), file.info("polarity.png")[1, "size"])
and
addFile(content, "umusergen", "destination.png", type = "image/png")
I'll update the addFile() function to allow one to indicate this is binary or content
or to use the (MIME) type.

Related

R/exams d2l multiple choice question doesn't select correct answer

I use the following to create a D2L exam from the "capital.Rmd" example (I converted the question to schoice)
exams2blackboard("capitals.Rmd", n =3, name = "testquiz" )
After I upload the testquiz.zip file, I notice that the correct answer must be manually chosen on the D2L platform.
I was wondering if there is a workaround.
Many Thanks,
Umut
If you want the correct solution to be selected, do not use the Import option from the Question Library or from the Quiz itself. Use the Import/Export/Copy Components under the Course Admin tab.
If you import the questions through the following steps, BrightSpace correctly picks the right solution. It’s a bit longer but seems to correctly choose the solution.
Under the Course Admin tab of your course, go to
'Import/Export/Copy Components' -> ‘Import Components’ -> Start -> (drag and drop the ZIP file)
Click ‘Advanced Options…’
This step will take a few minutes for large files; if you do not click
Advanced Options, then the import will automatically import the
questions into the 'Question Library' and will generate a Quiz with the
imported questions; you do not want this.
-> Continue -> Continue -> at this point choose 'Question Library' from the section 'Select Components to Import'
I would not choose ‘Quizzes’ because it automatically creates a quiz
and makes it available to students. It has the unfortunate side-effect
of making ALL the questions available, which means all the versions of
various dynamic questions; this is not something we want.
-> Continue -> Continue. This stage takes a few minutes for large
imports.
Now the Questions are available in the Question Library and can be used to generate new quizzes. Each question has the correct answer selected already. This works for ‘schoice’ and ‘mchoice’ versions of questions. Currently, plots are not imported, though, still trying to figure out why.
This problem is new to me. In earlier versions of Brightspace/D2L the import of single-choice and multiple-choice exercises via exams2blackboard() worked well. Possibly, D2L changed in the meantime given that neither the current release version from CRAN nor the development version from R-Forge work for you.
D2L also supports other import formats and we did play around with some of these. See the following discussions in the R/exams forum on R-Forge:
https://R-Forge.R-project.org/forum/forum.php?thread_id=33404&forum_id=4377&group_id=1337
https://R-Forge.R-project.org/forum/forum.php?thread_id=33657&forum_id=4377&group_id=1337
Notably we tried to use the XML-based QTI 2.1 format that seems to be employed by D2L internally. However, D2L apparently uses a particular custom flavor of QTI 2.1. It should be possible to reverse engineer that and improve exams2qti21() correspondingly but so far (to the best of my knowledge) no one put the time and effort into this that would be needed.
For simple single/multiple choice questions a CSV-based exchange format can also be used. I have put together a very basic exams2d2l() function that was posted in the threads above and that I'm also including below. It can set up the CSV file for a single exercise like the capitals.Rmd exercise that you use above. For plain text exercises like that it seems to work well but not for more complex elements (graphics, code, math, etc.).
exams2d2l <- function(file, dir = ".", ## n = 1L, nsamp = NULL disabled for now
name = NULL, quiet = TRUE, edir = NULL, tdir = NULL, sdir = NULL, verbose = FALSE,
resolution = 100, width = 4, height = 4, svg = FALSE,
encoding = "", converter = NULL, ...)
{
## for Rnw exercises use "ttm" converter otherwise "pandoc" converter
if(any(tolower(tools::file_ext(unlist(file))) == "rmd")) {
if(is.null(converter)) converter <- "pandoc"
} else {
if(is.null(converter)) converter <- "ttm"
}
## output directory or display on the fly
## output name processing
if(is.null(name)) name <- tools::file_path_sans_ext(basename(file))
## set up .html transformer and writer function
htmltransform <- make_exercise_transform_html(converter = converter, ...)
## create exam with HTML text
rval <- xexams(file,
driver = list(sweave = list(quiet = quiet, pdf = FALSE, png = !svg, svg = svg,
resolution = resolution, width = width, height = height, encoding = encoding),
read = NULL, transform = htmltransform, write = NULL),
dir = dir, edir = edir, tdir = tdir, sdir = sdir, verbose = verbose)
## currently: only a single exercise
rval <- rval[[1L]][[1L]]
## put together CSV
cleanup <- function(x) gsub('"', '""', paste(x, collapse = "\n"), fixed = TRUE)
rval <- c(
'NewQuestion,MC,,,',
sprintf('ID,"%s",,,', cleanup(rval$metainfo$file)),
sprintf('Title,"%s",,,', cleanup(rval$metainfo$name)),
sprintf('QuestionText,"%s",,,', cleanup(rval$question)),
sprintf('Points,%s,,,', if(is.null(rval$metainfo$points)) 1 else rval$metainfo$points),
'Difficulty,1,,,',
'Image,,,,',
paste0('Option,', ifelse(rval$metainfo$solution, 100, 0), ',"', cleanup(rval$questionlist), '",,"', cleanup(rval$solutionlist), '"'),
'Hint,,,,',
sprintf('Feedback,"%s",,,', cleanup(rval$solution))
)
writeLines(rval, file.path(dir, paste0(name, ".csv")))
invisible(rval)
}

R download.file downloading corrupted and smaller file

I'm trying to create a script to download a daily updated dataset. But the final file seems corrupted and is only 9~10kb while the original one is 95kb. Below is my code.
#Getting dataset from website daily
dir = getwd()
dash= "/"
date = Sys.Date()
date = gsub("-","",date)
filename = "CoronaDataSet"
extension = ".zip"
save = paste(dir,dash,date," - ",filename,extension, sep = "")
save
url = "https://www.kaggle.com/imdevskp/corona-virus-report/download"
download.file(url,save)
My console returns the following:
trying URL 'https://www.kaggle.com/imdevskp/corona-virus-report/download'
Content type 'text/html; charset=utf-8' length unknown
downloaded 8957 bytes
I also tried mode = "wb","a","ab". None downloaded the full size file.
Method = "auto","libcurl","curl" returns and invalid argument.
cannot open destfile 'C:/Users/Name/Documents/R/Course/Download file from url/20200323 - CoronaDataSet.zip', reason 'Invalid argument'
What may I be missing in my code? Any help is much appreciated.

Google Earth Engine download problems, is this caused by immutable server side objects?

I have a function that will download an image collection as a TFrecord or a geotiff.
Heres the function -
def download_image_collection_to_drive(collection, aois, bands, limit, export_format):
if collection.size().lt(ee.Number(limit)):
bands = [band for band in bands if band not in ['SCL', 'QA60']]
for aoi in aois:
cluster = aoi.get('cluster').getInfo()
geom = aoi.bounds().getInfo()['geometry']['coordinates']
aoi_collection = collection.filterMetadata('cluster', 'equals', cluster)
for ts in range(1, 11):
print(ts)
ts_collection = aoi_collection.filterMetadata('interval', 'equals', ts)
if ts_collection.size().eq(ee.Number(1)):
image = ts_collection.first()
p_id = image.get("PRODUCT_ID").getInfo()
description = f'{cluster}_{ts}_{p_id}'
task_config = {
'fileFormat': export_format,
'image': image.select(bands),
'region': geom,
'description': description,
'scale': 10,
'folder': 'output'
}
if export_format == 'TFRecord':
task_config['formatOptions'] = {'patchDimensions': [256, 256], 'kernelSize': [3, 3]}
task = ee.batch.Export.image.toDrive(**task_config)
task.start()
else:
logger.warning(f'no image for interval {ts}')
else:
logger.warning(f'collection over {limit} aborting drive download')
It seems whenever it gets to the second aoi it fails, Im confused by this as if ts_collection.size().eq(ee.Number(1)) confirms there is an image there so it should manage to get product id from it.
line 24, in download_image_collection_to_drive
p_id = image.get("PRODUCT_ID").getInfo()
File "/lib/python3.7/site-packages/ee/computedobject.py", line 95, in getInfo
return data.computeValue(self)
File "/lib/python3.7/site-packages/ee/data.py", line 717, in computeValue
prettyPrint=False))['result']
File "/lib/python3.7/site-packages/ee/data.py", line 340, in _execute_cloud_call
raise _translate_cloud_exception(e)
ee.ee_exception.EEException: Element.get: Parameter 'object' is required.
am I falling foul of immutable server side objects somewhere?
This is a server-side value, problem, yes, but immutability doesn't have to do with it — your if statement isn't working as you intend.
ts_collection.size().eq(ee.Number(1)) is a server-side value — you've described a comparison that hasn't happened yet. That means that doing any local operation like a Python if statement cannot take the comparison outcome into account, and will just treat it as a true value.
Using getInfo would be a quick fix:
if ts_collection.size().eq(ee.Number(1)).getInfo():
but it would be more efficient to avoid using getInfo more than needed by fetching the entire collection's info just once, which includes the image info.
...
ts_collection_info = ts_collection.getInfo()
if ts_collection['features']: # Are there any images in the collection?
image = ts_collection.first()
image_info = ts_collection['features'][0] # client-side image info already downloaded
p_id = image_info['properties']['PRODUCT_ID'] # get ID from client-side info
...
This way, you only make two requests per ts: one to check for the match, and one to start the export.
Note that I haven't actually run this Python code, and there might be some small mistakes; if it gives you any trouble, print(ts_collection_info) and examine the structure you actually received to figure out how to interpret it.

Arugment Error in Google Sheets Custom Function

I created a custom script in Google Sheets using Retrieving location address based on place name and city in Google Spreadsheet. Its is awesome! I wanted to modify the syntax to use the lat long instead of the city and state. I came up with the following:
function mapAddress(SiteName,LatLong) {
var Radius = '100';
var API_KEY = 'XXXXXX';
var url = 'https://maps.googleapis.com/maps/api/place/textsearch/json?query=' +
SiteName + '&location=' + LatLong + '&radius=' + Radius + '&key=' + API_KEY;
var response = UrlFetchApp.fetch(url);
var json = response.getContentText();
obj = JSON.parse(json);
addr = obj.results[0].formatted_address;
return addr;
}
Unfortunately I get an invalid argument in line 6 even though it creates a url that I can post in a browser and get valid results.
Invalid argument:
https://maps.googleapis.com/maps/api/place/textsearch/json?query=BurgerKIng&location= 35.221997,-101.831297&radius=100&key=XXXXXX
(line 6).
I have tried encoding the url but nothing helps. I am a newbie coder but this is about to drive me nuts. Would you be kind enough to help?
For posterity and to close the question: you noted in a comment that there was a space in the source data, resulting in a space after the = in the URL. Removing this resolved the issue.
The answer was "could it be theres a space in location= 35.." as I mentioned in my comment. You can accept this as the answer for others to see.

CSV file (with special characters) upload encoding issue

I am trying to upload a CSV file that has special characters using ServletFileUpload of apache common. But the special characters present in the CSV are being stored as junk characters in the database. The special characters I have are Trademark, registered etc. Following is the code snippet.
ServletFileUpload upload = new ServletFileUpload();
FileItemIterator iter = upload.getItemIterator(request);
while (iter.hasNext()) {
FileItemStream item = iter.next();
String name = item.getFieldName();
InputStream stream = item.openStream();
if (item.isFormField()) {
System.out.println("Form field " + name + " with value "
+ Streams.asString(stream, "UTF-8") + " detected.");
}
}
I have tried reading it using BufferendReader, used request.setCharacterEncoding("UTF-8"), tried upload.setHeaderEncoding("UTF-8") and also checked with IOUtils.copy() method, but none of them worked.
Please advice how to get rid of this issue and where it needs to be addressed? Is there anything I need to do beyond servlet code?
Thanks
What database are using? What character set is database using? Characters can be malformed in the database rather than in Java code.

Resources