I am trying to get a stream of 24-bit audio samples from some USB audio device using Python.
I already searched for some solutions and found this thread that is using PyAudio stream with a format of pyaudio.paInt24, unfortunately I still got 16-bit samples. I tried to record with PyAudio stream and save with WAVE, sample code:
def initialize(self):
self.p = pyaudio.PyAudio()
self.stream = self.p.open(format=self.format, # pyaudio.paInt24
channels=self.channels, # 1
rate=self.fs, # 16000
input=True,
input_device_index=self.input_device_index,
frames_per_buffer=self.chunk) # 1024
def run(self):
frames = []
for _ in range(0, int(self.fs / self.chunk * self.duration)):
data = self.stream.read(self.chunk)
frames.append(data)
self.stream.stop_stream()
self.stream.close()
self.p.terminate()
# write recording to a WAV file
wf = wave.open(self.recorded_file, 'wb')
wf.setnchannels(self.channels)
wf.setsampwidth(self.p.get_sample_size(self.format))
wf.setframerate(self.fs)
wf.writeframes(b''.join(frames))
wf.close()
I also tried the same the SoundDevice, sample code:
def initialize(self):
sd.default.samplerate = self.fs # 16000
sd.default.channels = self.channels # 1
sd.default.device = self.input_device_index
def run(self):
data = sd.rec(int(self.duration * self.fs), dtype=np.int32)
sd.wait()
# Save file
wf = wave.open(self.recorded_file, 'wb')
wf.setnchannels(self.channels)
wf.setsampwidth(3)
wf.setframerate(self.fs)
wf.writeframes(b''.join(data))
In order to make sure that the problem occurs when receiving the samples and not while saving them, I made sure that a 24-bit audio file can indeed be saved with WAVE, using this sample code:
import wave
with wave.open(path_to_file_to_read, mode='rb') as rb:
white_noise = rb.readframes(rb.getframerate())
with wave.open(path_to_file_to_save+'_save_with_wave.wav', mode='wb') as wb:
wb.setnchannels(rb.getnchannels())
wb.setsampwidth(rb.getsampwidth())
wb.setnframes(rb.getnframes())
wb.setframerate(rb.getframerate())
wb.writeframesraw(white_noise)
Also, I put a break point right after the recording was finished to take a look on the recorded samples and I see that it looks like 16-bit:
I will note that in order to make sure that the USB audio device does send me the 24-bit samples I did a recording using Audacity and indeed I got 24-bit, for some reason I can not do anything similar with Python.
Is there a simple way to record 24-bit samples using Python code?
Both PyAudio and SoundDevice use portaudio bindings between some windows sound architecture and python.
PortAudio does not suppport full depth 24 bit, and the result is 16 bit of audio with additional 8 zero bits.
SoundCard supports 24 bit - https://soundcard.readthedocs.io/en/latest/
In order to get real 24 bit samples, you can use the following:
import soundcard as sc
import soundfile as sf
FS = 16000
TIME_IN_SEC = 5
FRAMES = FS * TIME_IN_SEC
DEVICE_NAME = 'default'
FILE_PATH = 'abc.wav'
# get device
audio_device = sc.get_microphone(DEVICE_NAME)
# receive samples
data = audio_device.record(samplerate=FS, numframes=FRAMES)
# write recording to a WAV file
sf.write(FILE_PATH, data, FS, subtype='PCM_24')
Related
I've designed a tachometer using an arduino setup and I get the output values(rpm), but since I don't have an wifi module, I've connected my arduino and raspberry pi 4 using usb. I can read the rpm value in the pi terminal. But now I need to send these data to an adafruit io page. How do I write the code to read the data from the usb port of my pi in real-time? I've written a script which can print it on the webpage but each time I've to write a value. It would be really helpful if i can get the answers. I'm new to coding and just exploring these.
from Adafruit_IO import*
ADAFRUIT_IO_USERNAME = '******'
ADAFRUIT_IO_KEY = '**********************'
aio = Client(ADAFRUIT_IO_USERNAME,ADAFRUIT_IO_KEY)
try:
test = aio.feeds('test')
except RequestError:
test_feed = Feed(name='test')
test_feed = aio.create_feed(test_feed)
val = 4
aio.send('test',val)
The below code is an example of what will work in Python, assuming your USB is connected at /dev/tty.usbmodem14201:
import serial
ser = serial.Serial('/dev/tty.usbmodem14201', baudrate=9600) # NB set your baudrate to the one you are using!
ser.flushInput()
while True: #constant loop to get readings in real time
ser_bytes = ser.readline() #read the incoming message
decoded_bytes = ser_bytes[0:len(ser_bytes)-2].decode("utf-8") #decode it
print(decoded_bytes) # print out what you got or, alternatively, make a web call to Adrafruit
I am trying to put together some code in Python 3.6 to help test the computer hardware that passes through my hands as an IT tech.
I'd like to have a script that plays a simple sine wave tone on the left speaker, then the right and then the both speakers together.
I have found a potentially helpful script over at Pyaudio How to get sound on only one speaker but some of the code to actually run it is missing - chiefly the code for making sin wave tones. I have looked around online and have tried reverse-engineering this back into the code on that page but the maths is a little high-level for me! Sorry.
Thanks,
Will
Update:
I think I have found a partial (albeit long-winded solution) with 'sounddevice' for python 3
#!/usr/bin/env python3
import argparse
import logging
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("filename", help="audio file to be played back")
parser.add_argument("-d", "--device", type=int, help="device ID")
args = parser.parse_args()
try:
import sounddevice as sd
import soundfile as sf
data, fs = sf.read(args.filename, dtype='float32')
sd.play(data, fs, device=args.device, blocking=True, mapping=[1])
status = sd.get_status()
if status:
logging.warning(str(status))
except BaseException as e:
# This avoids printing the traceback, especially if Ctrl-C is used.
raise SystemExit(str(e))
The main chunk of code is repeat twice more but with "mapping = [1]" changed to "mapping = [2]" to test the right speaker and finally with "mapping = [?]" removed in the final block to test both speakers.
I found this over at https://python-sounddevice.readthedocs.io/en/0.2.1/examples.html.
Of course, if anyone knows a quicker and graceful way of getting this done, please share!
You could generate the sine tone directly in Python instead of loading it from a file. I've written some tutorials about creating simple sine tones:
https://nbviewer.jupyter.org/github/mgeier/python-audio/blob/master/simple-signals.ipynb
http://nbviewer.jupyter.org/github/spatialaudio/communication-acoustics-exercises/blob/master/intro.ipynb
Those tutorials use NumPy, because it makes manipulating the audio buffers very easy. But you can of course also do it in pure Python, if you prefer.
Here's an example:
#!/usr/bin/env python3
import math
import sounddevice as sd
sd.default.device = None
sd.default.samplerate = samplerate = 48000
duration = 1.5
volume = 0.3
frequency = 440
# fade time in seconds:
fade_in = 0.01
fade_out = 0.3
buffer = memoryview(bytearray(int(duration * samplerate) * 4)).cast('f')
for i in range(len(buffer)):
buffer[i] = volume * math.cos(2 * math.pi * frequency * i / samplerate)
fade_in_samples = int(fade_in * samplerate)
for i in range(fade_in_samples):
buffer[i] *= i / fade_in_samples
fade_out_samples = int(fade_out * samplerate)
for i in range(fade_out_samples):
buffer[-(i + 1)] *= i / fade_out_samples
for mapping in ([1], [2], [1, 2]):
sd.play(buffer, blocking=True, mapping=mapping)
sd.sleep(500)
Note that this code is using 32-bit floating point numbers (each one using 4 bytes), that's why we reserve 4 times more bytes in our bytearray than the required number of samples.
Basically I am working on a python project where I download and index files from the sec edgar database. The problem however, is that when using the requests module, it take a very long time to save the text in a variable (between ~130 and 170 seconds for one file).
The file roughly has around 16 million characters, and I wanted to see if there was any way to easily lower the time it takes to retrieve the text. -- Example:
import requests
url ="https://www.sec.gov/Archives/edgar/data/0001652044/000165204417000008/goog10-kq42016.htm"
r = requests.get(url, stream=True)
print(r.text)
Thanks!
What I found is in the code for r.text, specifically when no encoding was given ( r.encoding == 'None' ). The time spend detecting the encoding was 20 seconds, I was able to skip it by defining the encoding.
...
r.encoding = 'utf-8'
...
Additional details
In my case, my request was not returning an encoding type. The response was 256k in size, the r.apparent_encoding was taking 20 seconds.
Looking into the text property function. It tests to see if there is an encoding. If there is None, it will call the apperent_encoding function which will scan the text to autodetect the encoding scheme.
On a long string this will take time. By defining the encoding of the response ( as described above), you will skip the detection.
Validate that this is your issue
in your above example :
from datetime import datetime
import requests
url = "https://www.sec.gov/Archives/edgar/data/0001652044/000165204417000008/goog10-kq42016.htm"
r = requests.get(url, stream=True)
print(r.encoding)
print(datetime.now())
enc = r.apparent_encoding
print(enc)
print(datetime.now())
print(r.text)
print(datetime.now())
r.encoding = enc
print(r.text)
print(datetime.now())
of course the output may get lost in the printing, so I recommend you run the above in an interactive shell, it may become more aparent where you are losing the time even without printing datetime.now()
From #martijn-pieters
Decoding and printing 15MB of data to your console is often slower than loading data from a network connection. Don't print all that data. Just write it straight to a file.
For learning purposes, I wrote a small C Python module that is supposed to perform an IPC cuda memcopy to transfer data between processes. For testing, I wrote equivalent programs: one using theano's CudaNdarray, and the other using pycuda. The problem is, even though the test programs are nearly identical, the pycuda version works while the theano version does not. It doesn't crash: it just produces incorrect results.
Below is the relevant function in the C module. Here is what it does: every process has two buffers: a source and a destination. Calling _sillycopy(source, dest, n) copies n elements from each process's source buffer to the neighboring process's dest array. So, if I have two processes, 0 and 1, processes 0 will end up with process 1's source buffer and processes 1 will end up with process 0's source buffer.
Note that to transfer cudaIpcMemHandle_t values between processes, I use MPI (this is a small part of a larger project which uses MPI). _sillycopy is called by another function, "sillycopy" which is exposed in Python by the standard Python C API methods.
void _sillycopy(float *source, float* dest, int n, MPI_Comm comm) {
int localRank;
int localSize;
MPI_Comm_rank(comm, &localRank);
MPI_Comm_size(comm, &localSize);
// Figure out which process is to the "left".
// m() performs a mod and treats negative numbers
// appropriately
int neighbor = m(localRank - 1, localSize);
// Create a memory handle for *source and do a
// wasteful Allgather to distribute to other processes
// (could just use an MPI_Sendrecv, but irrelevant right now)
cudaIpcMemHandle_t *memHandles = new cudaIpcMemHandle_t[localSize];
cudaIpcGetMemHandle(memHandles + localRank, source);
MPI_Allgather(
memHandles + localRank, sizeof(cudaIpcMemHandle_t), MPI_BYTE,
memHandles, sizeof(cudaIpcMemHandle_t), MPI_BYTE,
comm);
// Open the neighbor's mem handle so we can do a cudaMemcpy
float *sourcePtr;
cudaIpcOpenMemHandle((void**)&sourcePtr, memHandles[neighbor], cudaIpcMemLazyEnablePeerAccess);
// Copy!
cudaMemcpy(dest, sourcePtr, n * sizeof(float), cudaMemcpyDefault);
cudaIpcCloseMemHandle(sourcePtr);
delete [] memHandles;
}
Now here is the pycuda example. For reference, using int() on a_gpu and b_gpu returns the pointer to the underlying buffer's memory address on the device.
import sillymodule # sillycopy lives in here
import simplempi as mpi
import pycuda.driver as drv
import numpy as np
import atexit
import time
mpi.init()
drv.init()
# Make sure each process uses a different GPU
dev = drv.Device(mpi.rank())
ctx = dev.make_context()
atexit.register(ctx.pop)
shape = (2**26,)
# allocate host memory
a = np.ones(shape, np.float32)
b = np.zeros(shape, np.float32)
# allocate device memory
a_gpu = drv.mem_alloc(a.nbytes)
b_gpu = drv.mem_alloc(b.nbytes)
# copy host to device
drv.memcpy_htod(a_gpu, a)
drv.memcpy_htod(b_gpu, b)
# A few more host buffers
a_p = np.zeros(shape, np.float32)
b_p = np.zeros(shape, np.float32)
# Sanity check: this should fill a_p with 1's
drv.memcpy_dtoh(a_p, a_gpu)
# Verify that
print(a_p[0:10])
sillymodule.sillycopy(
int(a_gpu),
int(b_gpu),
shape[0])
# After this, b_p should have all one's
drv.memcpy_dtoh(b_p, b_gpu)
print(c_p[0:10])
And now the theano version of the above code. Rather than using int() to get the buffers' address, the CudaNdarray way of accessing this is via the gpudata attribute.
import os
import simplempi as mpi
mpi.init()
# select's one gpu per process
os.environ['THEANO_FLAGS'] = "device=gpu{}".format(mpi.rank())
import theano.sandbox.cuda as cuda
import time
import numpy as np
import time
import sillymodule
shape = (2 ** 24, )
# Allocate host data
a = np.ones(shape, np.float32)
b = np.zeros(shape, np.float32)
# Allocate device data
a_gpu = cuda.CudaNdarray.zeros(shape)
b_gpu = cuda.CudaNdarray.zeros(shape)
# Copy from host to device
a_gpu[:] = a[:]
b_gpu[:] = b[:]
# Should print 1's as a sanity check
print(np.asarray(a_gpu[0:10]))
sillymodule.sillycopy(
a_gpu.gpudata,
b_gpu.gpudata,
shape[0])
# Should print 1's
print(np.asarray(b_gpu[0:10]))
Again, the pycuda code works perfectly and the theano version runs, but gives the wrong result. To be precise, at the end of the theano code, b_gpu is filled with garbage: neither 1's nor 0's, just random numbers as though it were copying from a wrong place in memory.
My original theory regarding why this was failing had to do with CUDA contexts. I wondered if it was possible theano was doing something with them that meant that the cuda calls made in sillycopy were run under a different CUDA context than had been used to create the gpu arrays. I don't think this is the case because:
I spent a lot of time digging deep in theano's code and saw no funny business being played with contexts
I would expect such a problem to result in a bad crash, not an incorrect result, which is not what happens.
A secondary thought is whether this has to do the fact that theano spawns several threads, even when using a cuda backend, which can be verified this by running "ps huH p ". I don't know how threads might affect anything, but I have run out of obvious things to consider.
Any thoughts on this would be greatly appreciated!
For reference: the processes are launched in the normal OpenMPI way:
mpirun --np 2 python test_pycuda.py
Although they resemble files, objects in Amazon S3 aren't really "files", just like S3 buckets aren't really directories. On a Unix system I can use head to preview the first few lines of a file, no matter how large it is, but I can't do this on a S3. So how do I do a partial read on S3?
S3 files can be huge, but you don't have to fetch the entire thing just to read the first few bytes. The S3 APIs support the HTTP Range: header (see RFC 2616), which take a byte range argument.
Just add a Range: bytes=0-NN header to your S3 request, where NN is the requested number of bytes to read, and you'll fetch only those bytes rather than read the whole file. Now you can preview that 900 GB CSV file you left in an S3 bucket without waiting for the entire thing to download. Read the full GET Object docs on Amazon's developer docs.
The AWS .Net SDK only shows only fixed-ended ranges are possible (RE: public ByteRange(long start, long end) ). What if I want to start in the middle and read to the end? An HTTP range of Range: bytes=1000- is perfectly acceptable for "start at 1000 and read to the end" I do not believe that they have allowed for this in the .Net library.
get_object api has arg for partial read
s3 = boto3.client('s3')
resp = s3.get_object(Bucket=bucket, Key=key, Range='bytes={}-{}'.format(start_byte, stop_byte-1))
res = resp['Body'].read()
Using Python you can preview first records of compressed file.
Connect using boto.
#Connect:
s3 = boto.connect_s3()
bname='my_bucket'
self.bucket = s3.get_bucket(bname, validate=False)
Read first 20 lines from gzip compressed file
#Read first 20 records
limit=20
k = Key(self.bucket)
k.key = 'my_file.gz'
k.open()
gzipped = GzipFile(None, 'rb', fileobj=k)
reader = csv.reader(io.TextIOWrapper(gzipped, newline="", encoding="utf-8"), delimiter='^')
for id,line in enumerate(reader):
if id>=int(limit): break
print(id, line)
So it's an equivalent of a following Unix command:
zcat my_file.gz|head -20