I was hoping to build an application that streams audio (mp3, ogg, etc.) from my microphone to a web browser.
I think I can use the html5 audio tag to read/play the stream from my server.
The area I'm really stuck on is how to setup the streaming http endpoint. What technologies will I need, and how should my server be structured to get the live audio from my mic and accessible from my server?
For example, for streaming mp3, do I constantly respond with mp3 frames as they are recorded?
Thanks for any help!
First off, let's split this problem up into a few parts. You have the audio capture (recording), the encoding/codec, the server, and the receiving clients.
Capture -> Codec -> Server -> Several Clients
For audio capture, you will need to use the Web Audio API along with getUserMedia. This will allow you to get 32-bit floating point PCM samples from the recording device. This data stream takes up a ton of bandwidth... a few megabit for a stereo stream. This stream is not directly playable in an HTML5 audio tag, and while you could play it on the receiving end with the Web Audio API, it takes up too much bandwidth to be useful. You need to use a codec to get the bandwidth usage down.
The codecs you want to look at include MP3, AAC (and its variants such as HE-AAC), and Opus. Not all browsers support all codecs. MP3 is the most widely compatible but AAC provides better quality for a given bitrate. Opus is a free and open codec but still doesn't have the greatest client adoption. In any case, there isn't yet a codec that you can run in-browser with any real stability. (Although it's being worked on! There are a lot of test projects made with Emscripten.) I solved this problem by reducing the bit depth of my samples to 16-bit signed integers and sending this PCM stream to a server to do the codec, over a binary websocket.
This encoding server took the PCM stream and ran it through a codec server-side. Here you can use whatever you'd like, such as a licensed codec binary or a tool like FFmpeg which encapsulates multiple codecs.
Next, this server streamed the data to a real streaming media server like Icecast. SHOUTcast and Icecast servers take the encoded stream and relay it to many clients over an HTTP-like connection. (Icecast is HTTP compliant whereas SHOUTcast is close but not quite there which can cause compatibility issues.)
Once you have your streaming server set up, it's as simple as referencing the stream URL in your <audio> tag.
Hopefully that gets you started. Depending on your needs, you might also look into WebRTC which does all of this for you but doesn't give you options for quality and also doesn't scale beyond a few users.
Related
As i starting to work with video streaming, i've got a question:
Video streaming is the process of breaking video file into small data packages that are sent over network. But where do they stored and what happen with it after streaming was finished? I am asking because unlike from download, streaming does not keep the file locally, that's how it described in internet. What is the process of handling stream buffers under the hood. Can someone point me into right direction?
Any help appreciated
Thanks
Most video streams are actually HTTP request and response based - i.e. he client (player) request the video chunk by chunk and then plays it as it receives each chunk.
To answer your question what happens to the chunks when they are downloaded, this will depend on the player and the device. In general the chunks will be rebuilt into the particular video container that is being used, e.g. mp4, and then played.
How long they are stored will depend on the device and the players caching rules and capacity.
I was capturing youtube video packets using wireshark. I saw it was http tunneled over tcp packet. (Even in case of youtube live streaming).
But whatever I know is that youtube uses flash video technology and html5. Again in some websites they mention about DASH protocol.
My question is, what is the exact protocol used by youtube? And how we can interpret the data that I have captured in wireshark? In the capture it is shown as just 'Data'. There is nothing mention as video data or any other things like that.
YouTube primarily uses the VP9 and H.264/MPEG-4 AVC video formats, and the Dynamic Adaptive Streaming over HTTP protocol.
By January 2019, YouTube had begun rolling out videos in AV1 format.
For mobile - Sometimes Youtube servers are sending data using RTSP which is an Application Layer Protocol.
On the transport layer RTSP uses both TCP and UDP.
If you want to parse youtube data from wireshark you will have to store it and run it inside a flashplayer. The video is sent as a flash object embedded into the HTML Page that is sent to you via https.
Source:
https://en.wikipedia.org/wiki/YouTube#Features
The exact protocol is tcp; although YouTube has been switching over to UDP as of late. The inability to interpret packet data is intentional, the way YouTube breaks up streaming data prevents capture apps like Wireshark from exposing anything about the data being transferred. To interpret the data you are going to need capture the data from a substantial number of packets and compile it to form a part of the file be sent. It’s best to just take the source IP from the pocket sender and use DNS to resolve it to the Domain name, then do research on what type of data that can be expected from that domain, but obviously this is extremely unreliable.
After reasearching for a few days, i m still lost with this issue:
I have a webcam connected over WiFi to my Android device.
I wrote an Android app to connect to a specified Socket of the webcam (IP and port). From this Socket i get an InputStream which is already encoded in H.264. Then i redirect this InputStream from the android device to my server, where i managed to decode it to images/frame by using Xuggler.
I would like to stream my webcam live to the internet to a flash player or something.
I know i have to use Wowza, FMS or RED5 for this.
My problem is, that i dont understand how to proceed with the InputStream i have. All examples i ve read need a mp4/flv or other container file to stream from... but i have a continuous live InputStream.
Some other examples consider using Flash Encoder. But my InputStream is already encoded in H.264.
This is a general understanding question. Please advise me on how to solve this.
Thank you
you have following options -
Encode in flv container. Yes you can transmit live stream using using flv container. You can set the 'duration' field in the header to be arbitrary long. e.g youtube use this trick for live streaming.
you can encode the stream into RTMP. ffmpeg has code for rtmp code which can be used for understand, or i believe there are other opensource rtmp muxers available.
convert the stream into HLS, there are flash based HLS player available.
why flash if I may ask, hope you know that HTML5 video tag now directly accepts h264 encoded videos.
While using RTMP if the request is tunneled through HTTP, how different it is from a HTTP request?
What would be the performance implications of tunneling while using RTMP?
The advantage of RTMP streams over the casual HTTP based progressive downloading is far too realistic to ignore
You can serve Flash Video over the Internet using RTMP, a special protocol for real-time server applications ranging from instant messaging to collaborative data sharing to video streaming. Whereas HTTP-delivered Flash Video is referred to as progressive download video, RTMP-delivered Flash Video is called streaming video. However, because the term streaming is so often misused, I prefer the term real-time streaming video.
One of the benefits of RTMP delivery for the viewer is near-instantaneous playback of video, provided the Flash Video file is encoded with a bitrate appropriate to the viewer's connection speed. Real-time streaming video can also be seeked to any point in the content. This feature is particularly advantageous for long-duration content because the viewer doesn't have to wait for the video file to load before jumping ahead, as is the case for HTTP-delivered video.
http://www.cisco.com/en/US/prod/collateral/video/ps11488/ps11791/ps11802/white_paper_c11-675935.html
is http streaming possible without using any streaming servers?
Of course. You can output and flush, it gets to client before you end the script, thus it's streaming.
For live streaming, only segmented, like Apple HLS, other variants of segmented HLS (like OSMF) are not widely supported at the moment.
IIS from microsoft can also do smooth streaming (and Apple HLS as well).
Apple HLS can be supported on any web server when you pre-segment stream to chunks and just upload to web server path.
For VoD streaming, there is lot's of modules for all web servers.
Yes, although libraries have varying level of support. What needs to be used is "http chunking", such that lib does not try to buffer the whole request/response in memory (to computed the content length header) and instead indicate content comes in chunks.
Yes,not only its possible but has been implemented by various media server companies, only reason they still make usage of servers because of commercial purpose. Basically the content you want to stream should be divided into chunks/packets and then client machine can request those chunks via simple HTTP Get Requests.
Well if you have WebSockets available you can actually get quite low-latency streaming for low-fps scenarios by sending video frames as jpegs.
You can also send audio separately and play it using WebAudio on your browser. I imagine it could work for scenarios where you do not require perfect audio-video sync.
Another approach is to stream MPEG chunks through WebSockets, decode them in JS using jsmpeg and render to a canvas. You can find more here (video-only):
http://phoboslab.org/log/2013/09/html5-live-video-streaming-via-websockets
yes, the answer to your problem with http streaming is MPEG-DASH tech