I am downloading global data from tiles by NASA MODIS satellite using MODIStsp 2.0.9 in R. This would give me a single stitched TIFF file for the entire glob.
I am getting difference in the resolution of the image when I run the same code in Windows 10 vs linux.
PS: I am not able to understand why the resolution of the stitched global image should vary when the input parameters to the function call don't vary at all. The difference in large between the two OS.
MODIStsp(gui = FALSE,
out_folder = dropbox,
out_folder_mod = dropbox,
selprod = 'Surf_Temp_Daily_1Km (M*D11A1)',
bandsel = "LST_Day_1km", # daily surface temp
sensor = "Terra",
# your username for NASA http server
user = "user" ,
# your password for NASA http server
password = "pass",
start_date = '2002.01.01',
end_date = '2002.01.01',
#end_date = '2020.12.31',
verbose = TRUE,
spatmeth = "bbox",
bbox = c(-180.00,-90.00,180.00,90.00),
out_format = 'GTiff',
compress = 'None',
out_projsel = 'User Defined',
output_proj = 4326,
delete_hdf = FALSE,
parallel = TRUE,
reprocess = FALSE
)
Related
Question
It seems the knitr cache becomes invalidated by copying the relevant files (.rmd script and cache directory) to another computer.
Why is that so and
how can I work around this?
Details
I do various lengthy calculations on two computers. I thought the following procedure could work:
Knit a first version of a report on machine A. (includes some lengthy calculations)
Copy the files created, i.e. the script and the cache directory, to machine B.
Continue editing the report on machine B (without recalculations because everything is cached).
This does not work, after copying the files to B, "knit" performs a full recalculation. This is even the case before any editing of the script was performed, i.e. just the act of copying from A to B seems enough to invalidate the cache.
Why is a full recalculation on B performed? As I understood it the caching mechanism boils down to creating and comparing a hash. I had hoped that after copying the hash would remain unchanged.
Is there something else I should copy in addition? Or is there any other way I can make the procedure above work?
Example
Any trivial script works as an example such as the one below:
```{r setup, include=FALSE}
knitr::opts_chunk$set(cache = TRUE)
```
Bla Bla
```{r test}
tmp = sort(runif(1e7))
```
I don't know the details of why that happens, but the workaround is easy: save values to files explicitly, and read them back in. You can use
saveRDS(x, "x.rds")
to save the variable x to a file named x.rds, and then
x <- readRDS("x.rds")
to read it back in. If you want to get fancy, you can check for the existence of x.rds using file.exists("x.rds") and do the full calculation followed by saveRDS if that returns FALSE, otherwise just read the data.
EDITED TO ADD: If you really want to know the answer to your first question, one possible approach would be to copy the folder back from the 2nd computer to the 1st, and see if it works back there. If not, do a binary compare of the original and twice copied directories and see what has changed.
If it does work, it might simply be different RNGkind() settings on the two computers: it's pretty common to have the buggy sample.kind = "Rounding" saved. Not sure that caching would use this. Or perhaps different package versions or R versions: when I updated knitr the cache was invalidated.
MORE additions:
If you want to see what has changed, then turn on debugging on the digest::digest function, and call knitr::knit("src.Rmd"). digest() is called for each cached chunk, and passed a large list in its object argument. It should return the same hash value if the list is the same, so you'll want to save those objects, and compare them between the two computers. For example, with your toy example above, I get this passed as object:
list(eval = TRUE, echo = TRUE, results = "markup", tidy = FALSE,
tidy.opts = NULL, collapse = FALSE, prompt = FALSE, comment = "##",
highlight = TRUE, size = "normalsize", background = "#F7F7F7",
strip.white = TRUE, cache = 3, cache.path = "cache/", cache.vars = NULL,
cache.lazy = TRUE, dependson = NULL, autodep = FALSE, fig.keep = "high",
fig.show = "asis", fig.align = "default", fig.path = "figure/",
dev = "png", dev.args = NULL, dpi = 72, fig.ext = NULL, fig.width = 7,
fig.height = 7, fig.env = "figure", fig.cap = NULL, fig.scap = NULL,
fig.lp = "fig:", fig.subcap = NULL, fig.pos = "", out.width = NULL,
out.height = NULL, out.extra = NULL, fig.retina = 1, external = TRUE,
sanitize = FALSE, interval = 1, aniopts = "controls,loop",
warning = TRUE, error = TRUE, message = TRUE, render = NULL,
ref.label = NULL, child = NULL, engine = "R", split = FALSE,
purl = TRUE, label = "test", code = "tmp = sort(runif(1e7))",
75L)
I would like to download a grib2 file data in a range, as done in this Python notebook:
https://nbviewer.org/github/microsoft/AIforEarthDataSets/blob/main/data/noaa-hrrr.ipynb
(see cell 5)
I have tried the following code, but it seems to download the whole GRIB file instead of the range:
using HTTP
url = "https://noaahrrr.blob.core.windows.net/hrrr/hrrr.20210513/conus/hrrr.t12z.wrfsfcf01.grib2"
range_start = 38448330
range_end = 39758083
grib2_bytes = HTTP.request("GET", url; headers = Dict("Range" => Dict("bytes" => [range_start; range_end]) ) );
# save bytes to file
io = open("variable.grib2", "w");
write(io, grib2_bytes); # I can see the file is too big (148 MB)
close(io)
# rest of the code is just to read the data
# The downloaded file subset is a valid GRIB2 file.
using GRIB
f = GribFile("variable.grib2")
msg = Message(f)
To mimic the python code you should use string interpolation:
range_start = 38448330
range_end = 39758083
headers = Dict(
"Range" => "bytes=$(range_start)-$(range_end)"
)
I want to deploy influxDB with collectd.
For previous version ie 1.8 of influxDB it was supported.
Do we have support for collectd in the latest version of influxdb i.e 2.0?
This functionality was moved to telegraf. in /etc/telegraf/telegraf.conf you have to define a collectd input and an influxdb output, then it works. I haven't found an example so I had to piece it together from different sources of documentation:
[agent]
interval = "10s"
round_interval = true
metric_batch_size = 1000
metric_buffer_limit = 10000
collection_jitter = "0s"
flush_interval = "10s"
flush_jitter = "0s"
precision = ""
hostname = ""
omit_hostname = false
[[inputs.socket_listener]]
service_address = "udp://:25826"
data_format = "collectd"
collectd_auth_file = "/etc/collectd/collectd.auth"
collectd_security_level = "encrypt"
collectd_typesdb = ["/usr/share/collectd_types.db"]
collectd_parse_multivalue = "split"
[[outputs.influxdb_v2]]
urls = ["http://influxdb:8086"]
token = "$DOCKER_INFLUXDB_INIT_ADMIN_TOKEN"
organization = "$DOCKER_INFLUXDB_INIT_ORG"
bucket = "$DOCKER_INFLUXDB_INIT_BUCKET"
I am trying to use to sen2r() function (Package sen2r_1.3.2) with default parameters but getting the following error:
Error in paste(c(...), collapse = sep) : argument is missing, with no default.
I know the error wants me to fill in some parameters, but the source manual clearly says that the default should work, and the parameters can be set subsequently upon launching the GUI.
Using the s2_gui() launches the shiny app, but keeps hanging when I try to "Save and Close"
R version 3.6.3 (2020-02-29)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: Ubuntu 18.04.4 LTS
Also, can someone with a 'higher reputation' please create a sen2r tag, for easier subsequent communications?
Here is the traceback...
sen2r()
Error in paste(c(...), collapse = sep) :
argument is missing, with no default
> traceback()
7: paste(c(...), collapse = sep)
6: strsplit(paste(c(...), collapse = sep), "\n")
5: unlist(strsplit(paste(c(...), collapse = sep), "\n"))
4: strwrap(unlist(strsplit(paste(c(...), collapse = sep), "\n")),
width = width, indent = indent, exdent = exdent, prefix = prefix,
initial = initial)
3: print_message(type = "waiting", "It seems you are running this package for the first time. ",
"Do you want to verify/install the required dependencies using a GUI (otherwise, an\n automatic check will be performed)? (y/n) ",
)
2: .sen2r(param_list = param_list, pm_arg_passed = pm_arg_passed,
gui = gui, preprocess = preprocess, s2_levels = s2_levels,
sel_sensor = sel_sensor, online = online, order_lta = order_lta,
apihub = apihub, downloader = downloader, overwrite_safe = overwrite_safe,
rm_safe = rm_safe, step_atmcorr = step_atmcorr, sen2cor_use_dem = sen2cor_use_dem,
sen2cor_gipp = sen2cor_gipp, max_cloud_safe = max_cloud_safe,
timewindow = timewindow, timeperiod = timeperiod, extent = extent,
extent_name = extent_name, s2tiles_selected = s2tiles_selected,
s2orbits_selected = s2orbits_selected, list_prods = list_prods,
list_rgb = list_rgb, list_indices = list_indices, index_source = index_source,
rgb_ranges = rgb_ranges, mask_type = mask_type, max_mask = max_mask,
mask_smooth = mask_smooth, mask_buffer = mask_buffer, clip_on_extent = clip_on_extent,
extent_as_mask = extent_as_mask, reference_path = reference_path,
res = res, res_s2 = res_s2, unit = unit, proj = proj, resampling = resampling,
resampling_scl = resampling_scl, outformat = outformat, rgb_outformat = rgb_outformat,
index_datatype = index_datatype, compression = compression,
rgb_compression = rgb_compression, overwrite = overwrite,
path_l1c = path_l1c, path_l2a = path_l2a, path_tiles = path_tiles,
path_merged = path_merged, path_out = path_out, path_rgb = path_rgb,
path_indices = path_indices, path_subdirs = path_subdirs,
thumbnails = thumbnails, parallel = parallel, processing_order = processing_order,
use_python = use_python, tmpdir = tmpdir, rmtmp = rmtmp,
log = log, globenv = sen2r_env, .only_list_names = FALSE)
1: sen2r()
I ran s2_gui() as is...no parameters specified. But i am running the dependency check now, I suspect that should clear things up even for the GUI.
This error was due to a code bug, which was fixed (see the GitHub issue 292).
Until the sen2r CRAN version will be updated, the bug can be:
solved installling the sen2r GitHub version (remotes::install_github("ranghetti/sen2r")), or
bypassed launching check_gdal() before running sen2r().
This is a bug in the original code.
In the traceback that you provided, it included:
3: print_message(type = "waiting", "It seems you are running this package for the first time. ",
"Do you want to verify/install the required dependencies using a GUI (otherwise, an\n automatic check will be performed)? (y/n) ",
)
Notably, I'll truncate most of the strings:
3: print_message(type = "waiting", "It seems ... time. ",
"Do you ... performed)? (y/n) ",
) # ^-- extra comma, invalid R syntax
Notice how it ends with a comma and then a right-paren? Yup, that's a syntax error in R. (This has been submitted as issue 292 on the original repo.)
I am testing a program executed partially on a MPC603 and partially on a MPC555.
I have to verify that some data is correctly "moved" from one processor to the other via a DPRAM.
I am guessing that at some point "someone" makes a conversion but I don't know how to find what kind of conversion is done.
Here are some examples:
Pt_Dpram->acq1 at 0x8D00008 = 0x3EB2
acq1 = (0xA010538) = 1182451712 = 0x467AC800
Pt_Dpram->acq2 at 0x8D0000A = 0x5528
acq2 = (0xA010540) = 1185566720 = 0x46AA5000
Pt_Dpram->acq3 at 0x8D0000C = 0x416E
acq3 = (0xA010548) = 1107552036 = 0x4203E724
Pt_Dpram->acq4 at 0x8D0000E = 0x413C
acq4 = (0xA010550) = 1107526232 = 0x42038258
I got my answers from a collegue : the values in acqX are in Motorola binary format : http://en.wikipedia.org/wiki/SREC_(file_format)
Here is a small software that does the conversion : http://www.hexworkshop.com/onlinehelp/500/html/idhelp_baseconv.htm