R, GET and GZ compression - r

Question

I am building clients onto RESTful APIs. Some links let me download attachments (files) from the server, and in the best case these are .txt. I only mention the RESTful part since it means that I have to send some headers and potentially body with each post - the standard R 'filename'=URL logic won't work.
Sometimes people bundle many txts into a zip. These are awkward since I don't know what they contain until I download many of them.
For the moment, I am unpackaging these, gzipping the files (adds the .gz extension) and re-uploading them. They can then be indexed and downloaded.
I'm using Hadley's cute httr package, but I can't see an elegant way to decompress the gz files.
When using read.csv or similar any files with a gz ending are automatically decompressed (convenient!). What's the equivalent when using httr or curl?
content(GET("http://glimmer.rstudio.com/alexbbrown/gz/sample.txt.gz"))
[1] 1f 8b 08 08 4e 9e 9b 51 00 03 73 ...
That looks nice, a compressed byte stream with the correct header (1f 8b). Now I need the text contents, so I tried using memDecompress, which says it should do this:
memDecompress(content(GET("http://glimmer.rstudio.com/alexbbrown/gz/sample.txt.gz")),type="gzip")
Error in memDecompress(content(GET("http://glimmer.rstudio.com/alexbbrown/gz/sample.txt.gz")), :
internal error -3 in memDecompress(2)
What's the proper solution here?
Also, is there a way to get R to pull the INDEX of a remote .zip file without downloading all of it?

Answer 1

The following works, but seems a little convoluted:
> scan(gzcon(rawConnection(content(GET("http://glimmer.rstudio.com/alexbbrown/gz/sample.txt.gz")))),"",,,"\n")
Read 1 item
[1] "These are not the droids you are looking for"

Answer 2

You can add a parser to handle the mime type. Look at ?content and the line You can add new parsers by adding appropriately functions to httr:::parser
ls(httr:::parsers)
#[1] "application/json" "application/x-www-form-urlencoded" #"image/jpeg"
#[4] "image/png" "text/html" #"text/plain"
#[7] "text/xml"
we can add one to handle gz content. I dont have a better answer at this point then you gave so you can incorporate your function.
assign("application/octet-stream", function(x, ...) {scan(gzcon(rawConnection(x)),"",,,"\n")},envir = httr:::parsers)
content(GET("http://glimmer.rstudio.com/alexbbrown/gz/sample.txt.gz"), as = "parsed")
Read 1 item
[1] "These are not the droids you are looking for"
>
EDIT:
I hacked together an alternative:
assign("application/octet-stream", function(x, ...) {f <- tempfile(); writeBin(x,f);untar(f);readLines(f, warn = FALSE)},envir = httr:::parsers)
content(GET("http://glimmer.rstudio.com/alexbbrown/gz/sample.txt.gz"), as = "parsed")
#[1] "These are not the droids you are looking for"
With regards to listing the files in the archive maybe you can adjust the function somewhat.
If we try to get the httr source files. They have a mime type "application/x-gzip"
assign("application/x-gzip", function(x, ...) {
f <- tempfile();
writeBin(x,f);
if(!is.null(list(...)$list)){
if(list(...)$list){
return(untar(f, list = TRUE))
}else{
untar(f, ...);
readLines(f)
}
}else{
untar(f, ...);
readLines(f)
}
}, envir = httr:::parsers)
content(GET("http://cran.r-project.org/src/contrib/httr_0.2.tar.gz"), as = "parsed", list = TRUE)
# > head(content(GET("http://cran.r-project.org/src/contrib/httr_0.2.tar.gz"), as = "parsed", list = TRUE))
#[1] "httr/" "httr/MD5" "httr/tests/"
#[4] "httr/tests/test-all.R" "httr/README.md" "httr/R/"