I would like to download a grib2 file data in a range, as done in this Python notebook:
https://nbviewer.org/github/microsoft/AIforEarthDataSets/blob/main/data/noaa-hrrr.ipynb
(see cell 5)
I have tried the following code, but it seems to download the whole GRIB file instead of the range:
using HTTP
url = "https://noaahrrr.blob.core.windows.net/hrrr/hrrr.20210513/conus/hrrr.t12z.wrfsfcf01.grib2"
range_start = 38448330
range_end = 39758083
grib2_bytes = HTTP.request("GET", url; headers = Dict("Range" => Dict("bytes" => [range_start; range_end]) ) );
# save bytes to file
io = open("variable.grib2", "w");
write(io, grib2_bytes); # I can see the file is too big (148 MB)
close(io)
# rest of the code is just to read the data
# The downloaded file subset is a valid GRIB2 file.
using GRIB
f = GribFile("variable.grib2")
msg = Message(f)
To mimic the python code you should use string interpolation:
range_start = 38448330
range_end = 39758083
headers = Dict(
"Range" => "bytes=$(range_start)-$(range_end)"
)
Related
Currently using httr package
Files <- c("mypath.png", “test.png”)
R <- POST(url, body = list(x = upload_file(#FIles),b = “test” ))
Trying to post multiple files with same key
Similar to the script below from python
files = [('file', open('report.xls', 'rb')), ('file', open('report2.xls', 'rb'))]
Need to be done in curl
#Workaround in curl library.
params=list()
for (file in embImgList)
{params <- append(params, list(FILES= form_file(path = file)))}
I found a solution that worked
httr::POST(url = upload_url,
add_headers(`Authorization` = authorization),
body = list(files = upload_file(file_location[1]),
files = upload_file(file_location[2]),
files = upload_file(file_location[3]),
files = upload_file(file_location[4]),
files = upload_file(file_location[5])),
encode = "multipart",
verbose()
)
I want to cut large audio file into different segments and store them in WAV format using pyaudio. I basically need to listen to audio and then cut the file from starting point to where i want to cut,and again start recording and cut another portion, but i am not sure how can i do it with pyaudio. Am i looking for an alternate library ?
I am new to python, any sort of help would be appreciable.
This is code, i have experimented with:
import pyaudio
import wave
import time
CHUNK = 1024
FORMAT = pyaudio.paInt16
CHANNELS = 2
RATE = 44100
WAVE_OUTPUT_FILENAME = "output.wav"
wf = wave.open("A001017001_Edited.wav", 'rb')
p = pyaudio.PyAudio()
stream = p.open(format=FORMAT,
channels=CHANNELS,
rate=RATE,
input=True,
frames_per_buffer=CHUNK)
check = True;
While(check):
start = input("Do you wish to start recording?,then press ENTER")
if (start == 13):
try:
stream.start_stream()
p = time.time()
kdata = wf.readframes(CHUNK)
while len(kdata) > 0:
stream.write(kdata)
kdata = wf.readframes(CHUNK)
except KeyboardInterrupt:
q = time.time()
RECORD_SECONDS = (q-p); #gets time since wave file is played
frames = []
for i in range(0, int(RATE / CHUNK * RECORD_SECONDS)):
data = stream.read(CHUNK)
frames.append(data)
print(int(RATE / CHUNK * RECORD_SECONDS))
print("stopped recording")
stream.stop_stream()
wf = wave.open(WAVE_OUTPUT_FILENAME, 'wb')
wf.setnchannels(CHANNELS)
wf.setsampwidth(p.get_sample_size(FORMAT))
wf.setframerate(RATE)
wf.writeframes(b''.join(frames))
#compare if the whole audio is listened
#or not and
#if yes return false
stream.close()
p.terminate()
wf.close()
I want to prepare my own image data for training in torch.
I tried to find a good source for this but could not find.
They have given reference to data that has been already prepared in .lua or .t7 formats.
Can you please explain the procedure of preparing raw image data for torch? (training, validation and test sets)
Thanks
you may try to write your own data loader class. store your image paths in a table, read image using
require 'image'
YOUR_RGB_FILE_PATH = "/home/username/image.png"
img = image.load(YOUR_RGB_FILE_PATH, 3)
Write your lua code in a iTorch notebook, it helps you debug quickly.
if you do not know how to start, you can refer to the project here wrote with lua torch.
require 'io'
require 'torch'
require 'image'
------------------------------ Parameters ---------------------------------
file_name = '.../train.txt'
save_name = '.../train.t7'
num_images = 10000*3
num_channels = 3
width = 51
height = 51
---------------------------------------------------------------------------
file = io.open(file_name, 'rb')
data = torch.Tensor(num_images, num_channels, width, height):byte()
label = torch.Tensor(num_images):byte()
counter = 1
for line in file:lines() do
print(counter)
image_name, image_label = line:split(' ')[1], line:split(' ')[2]
data[counter] = image.load(image_name, num_channels, 'byte')
label[counter] = image_label
counter = counter + 1
end
torch.save(save_name, {data = data, label = label})
library(tuneR)
Awaaz <- readWave("C:/Users/Ram/Downloads/XC312210 - Mallee Emu-wren - Stipiturus mallee.mp3" ,
from = 1, to = Inf,"samples", header = FALSE, toWaveMC = NULL)
Getting below error message...
Error in while (!(RIFF == "RIFF" && WAVE == "WAVE")) { :
missing value where TRUE/FALSE needed
TuneR
readWave: reading Wave files
readMP3: read an MPEG-2 layer 3 file into a Wave object
readWave reads Wave files, and readMP3 reads MP3 files. Simple enough.
I'm using Python 3.4.2 and PyPDF2 1.24 (also using reportlab 3.1.44 in case that helps) on windows 7.
I recently upgraded from Python 2.7 to 3.4, and am in the process of porting my code. This code is used to create a blank pdf page with links embedded in it (using reportlab) and merge it (using PyPDF2) with an existing pdf page. I had an issue with reportlab in that saving the canvas used StringIO which needed to be changed to BytesIO, but after doing that I ran into this error:
Traceback (most recent call last):
File "C:\cms_software\pdf_replica\builder.py", line 401, in merge_pdf_files
input_page.mergePage(link_page)
File "C:\Python34\lib\site-packages\PyPDF2\pdf.py", line 2013, in mergePage
self.mergePage(page2)
File "C:\Python34\lib\site-packages\PyPDF2\pdf.py", line 2059, in mergePage
page2Content = PageObject._pushPopGS(page2Content, self.pdf)
File "C:\Python34\lib\site-packages\PyPDF2\pdf.py", line 1973, in _pushPopGS
stream = ContentStream(contents, pdf)
File "C:\Python34\lib\site-packages\PyPDF2\pdf.py", line 2446, in __init
stream = BytesIO(b_(stream.getData()))
File "C:\Python34\lib\site-packages\PyPDF2\generic.py", line 826, in getData
decoded._data = filters.decodeStreamData(self)
File "C:\Python34\lib\site-packages\PyPDF2\filters.py", line 326, in decodeStreamData
data = ASCII85Decode.decode(data)
File "C:\Python34\lib\site-packages\PyPDF2\filters.py", line 264, in decode
data = [y for y in data if not (y in ' \n\r\t')]
File "C:\Python34\lib\site-packages\PyPDF2\filters.py", line 264, in
data = [y for y in data if not (y in ' \n\r\t')]
TypeError: 'in <string>' requires string as left operand, not int
Here is the line and the line above where the traceback mentions:
link_page = self.make_pdf_link_page(pdf, size, margin, scale_factor, debug_article_links)
if link_page != None:
input_page.mergePage(link_page)
Here are the relevant parts of that make_pdf_link_page function:
packet = io.BytesIO()
can = canvas.Canvas(packet, pagesize=(size['width'], size['height']))
....# left out code here is just reportlab specifics for size and url stuff
can.linkURL(url, r1, thickness=1, color=colors.green)
can.rect(x1, y1, width, height, stroke=1, fill=0)
# create a new PDF with Reportlab that has the url link embedded
can.save()
packet.seek(0)
try:
new_pdf = PdfFileReader(packet)
except Exception as e:
logger.exception('e')
return None
return new_pdf.getPage(0)
I'm assuming it's a problem with using BytesIO, but I can't create the page with reportlab with StringIO. This is a critical feature that used to work perfectly with Python 2.7, so I'd appreciate any kind of feedback on this. Thanks!
UPDATE:
I've also tried changing from using BytesIO to just writing to a temp file, then merging. Unfortunately I got the same error.
Here is tempfile version:
import tempfile
temp_dir = tempfile.gettempdir()
temp_path = os.path.join(temp_dir, "tmp.pdf")
can = canvas.Canvas(temp_path, pagesize=(size['width'], size['height']))
....
can.showPage()
can.save()
try:
new_pdf = PdfFileReader(temp_path)
except Exception as e:
logger.exception('e')
return None
return new_pdf.getPage(0)
UPDATE:
I found an interesting bit of information on this. It seems if I comment out the can.rect and can.linkURL calls it will merge. So drawing anything on a page, then trying to merge it with my existing pdf is causing the error.
After digging in to PyPDF2 library code, I was able to find my own answer. For python 3 users, old libraries can be tricky. Even if they say they support python 3, they don't necessarily test everything. In this case, the problem was with the class ASCII85Decode in filters.py in PyPDF2. For python 3, this class needs to return bytes. I borrowed the code for this same type of function from pdfminer3k, which is a port for python 3 of pdfminer. If you exchange the ASCII85Decode() class for this code, it will work:
import struct
class ASCII85Decode(object):
def decode(data, decodeParms=None):
if isinstance(data, str):
data = data.encode('ascii')
n = b = 0
out = bytearray()
for c in data:
if ord('!') <= c and c <= ord('u'):
n += 1
b = b*85+(c-33)
if n == 5:
out += struct.pack(b'>L',b)
n = b = 0
elif c == ord('z'):
assert n == 0
out += b'\0\0\0\0'
elif c == ord('~'):
if n:
for _ in range(5-n):
b = b*85+84
out += struct.pack(b'>L',b)[:n-1]
break
return bytes(out)