I seek to use the http live streaming standard with video. I'd like to eliminate any delay while a user is working with our app, but the current architecture requires fully encoding audio with any new or removed video clips.
Is there an incremental encoding approach to http live streaming so that I can
keep the audio track separate, but playback seamlessly with the video stream
allows .ts chunks to be independently encoded and streamed back to a user faster than re-encoding an entire video
References:
https://datatracker.ietf.org/doc/html/draft-pantos-http-live-streaming
https://developer.apple.com/streaming/
You could re-encode the required segments fairly easily -- there's no need to have the entire stream encoded before playing it (otherwise live events would be impossible). You have to be careful with the timestamps in the TS packets if you want it to be truly seamless. But what might be easiest is to use EXT-X-DISCONTINUITY markers around the re-created portions.
As for audio, there's no need to re-encode it. You should be able to just copy the encoded audio from one TS container to another. For example, if you're using ffmpeg, you would use -acodec copy to take it from the original ts.
Related
How can I save the video after live streaming of nginx-rtmp-module and play it back with hls . I use record to save to flv and then convert flv to m3u8, it takes a lot of time if the video is large. If I use hls_cleanup off, I can't actively choose to turn the record on or off. What is the correct way to save and play back using hls ? Please or show me if you know . Thanks very much
For small video file, both DVR-FLV or HLS are OK.
For large video file, as you mentioned, HLS is better. You need to manage each ts file and its duration, to generate the m3u8 index when streaming finished.
If you need to merge multiple publish stream to one stream, HLS is also better, for example, if need to adjust the encoder, use another encoder, or reconnect to server for network fail. If use DVR-FLV, there will be more than one FLV file and it's hard to merge them(need to covert to ts, concat them, then transcoding).
Furthermore, HLS is much better for producing during streaming, like sport programs, you may need to produce many VoD files during live streaming, and we can't wait streaming end:
encoder ---RTMP---> Server --HLS--> VoD During Streaming
I have a server (not internet connected) that hosts a webpage with company data on an internal website. The server also contains videos (thousands of them) in a defined directory structure.
When a client connects I can display the videos to them on the internal website. The problem is some of the video files are 1Gb or larger and the connection to some clients is rather slow; the browser seems to be trying to download them in order to play them rather than stream them.
Is there a video streaming server that I could send a file path to and it would serve the video back to the client as a stream?
I guess this is essentially transcoding the video that I need done. I'm not sure if PLEX or something like that is able to do it dynamically as there are hundreds of videos and new videos added all the time.
Sorry if i'm not being clear on my need. Send me a question if I haven't been clear on a point.
...the browser seems to be trying to download them in order to play them rather than stream them.
To echo what #Offbeatmammal said in the comments, if you're using MP4 files, you need to ensure the MOOV atom is at the beginning of the file. Without it, the browser doesn't know what byte offsets to request.
Ideally, encode your video files as fragmented. In FFmpeg:
ffmpeg -i ... -f mp4 -movflags frag_keyframe+empty_moov output.mp4
See also: https://stackoverflow.com/a/9734251/362536
That should allow the client to stream the MP4 files from any web server that supports HTTP/1.1 range requests. (Most all do, unless configured otherwise.)
However, there is another point to address:
The problem is some of the video files are 1Gb or larger and the connection to some clients is rather slow...
While fixing the streaming issue means the clients won't have to download the whole file first, they still need the bandwidth to keep up with the stream. If it's possible they won't, you'll want to implement some sort of transcoder.
I would recommend using an existing segmented streaming method such as DASH or HLS. HLS is currently the most compatible, thanks to Apple's platform policies. Either will enable adaptive bitrate switching, which will allow slow clients to automatically switch to a lower bitrate stream that they can smoothly keep up with. That way, slower clients can still see the video, albeit a lower quality one, while fast clients can get the full quality video.
You can use FFmpeg to do the transcoding and HLS playlist creation.
I'm not sure if PLEX or something like that is able to do it dynamically as there are hundreds of videos and new videos added all the time.
As for when you do this transcode, I suppose it depends on how much load you're looking at. If this is just one or two people viewing the file, you can transcode on demand if your servers can keep up. Ideally, you have at least a couple stream variants around for less popular files, and add more later if needed.
If you're doing this live, I'd recommend doing all of your transcoding up front. You can always prune old files/variants if you need the storage back.
I understand in a client-server model gRPC can do a bidirectional streaming of data.
I have not tried yet, but want to know will it be possible to stream audio and video data from a source to cloud server using gRPC and then broadcast to multiple client, all in real time ?
TLDR: I would not recommend video over gRPC. Primarily because it wasn't designed for it, so doing it would take a lot of hacking. You should probably take a look at WebRTC + a specific video codec.
More information below:
gRPC has no video compression
When sending video, we want to send things efficiently because sending it raw could require 1GB/s connectivity.
So we use video compression / video encoding. For example, H.264, VP8 or AV1.
Understand how video compression works (eg saving bandwidth by minimising similar data shared between frames in a video)
There is no video encoder for protobufs (the format used by gRPC).
You could then try image compression and save the images in a bytes field (e.g. bytes image_frame = 1;, but this is less efficient and definitely takes up unnecessary space for videos.
It's probably possible to encode frames into protobufs using a video encoder (e.g. H.264) and then decode them to play in applications. However, it might take a lot of hacking/engineering effort. This use case is not what gRPC/protobufs is designed for and not commonly done. Let me know if you hack something together, I would be curious.
gRPC is reliable
gRPC uses TCP (not UDP), which is reliable.
At a glance, reliability might be handy, to avoid corrupting data or lost data. However, depending on the use case (realtime video or audio calls), we may prefer to skip frames if they are dropped or delayed. The losses may be unnoticeable or painless to the user.
If the packet is delayed, it will wait for the packet before playing the rest. (aka. out of order delivery)
If the packet is dropped, it will resend it (aka. packet loss)
Therefore, video conferencing apps usually use WebRTC/RTP (configured to be unreliable)
Having mentioned that, looks like Zoom was able to implement Video-over-WebSockets, which is also a reliable transport (over TCP). So it's not "game-over", just highly not recommended and a lot more effort. They have moved over to WebRTC though.
Data received on the WebSockets goes into a WebAssembly (WASM) based decoder. Audio is fed to an AudioWorklet in browsers that support that. From there the decoded audio is played using the WebAudio “magic” destination node.
I want to setup a video on demand server which support Http protocol. It is like Youtube, which hosts a lot of videos, and end users could play them from browser (by using Flash or Html 5).
Two quick questions,
For the big video files, shall I put them on disk or in memory? How Youtube or other big video site did it? Not sure if put all video in memory is too expensive, and put video on disk is too slow?
Is there any open source video hosting server for my purpose? If steaming is supported, it will be great.
thanks in advance,
George
If you just want to have an HTML page that links to your video files - no problem, but most browsers will download the entire file before you system even considers playing it.
If you want to stream the files (like YouTube and others do) then you aren't actually using HTTP for the video itself. HTTP is used to get the information about the stream so your player can stream and play directly without having to download the entire file first.
Streaming video uses RTSP (or some other streaming protocol) for the audio and video data.
The closest HTTP protocol can get to "streaming" video is to use Server-Push of individual image frame with each frame flagged to replace the previous frame. Not all browsers can handle this directly, but might need an ActiveX control or Java Applet. The original QuickTime did this before the streaming protocols were implemented at the servers.
re: how does YouTube deal with big video file
I suspect they are on disk until they are needed. Moved into memory only as needed. Flushed from memory when no longer needed.
re: is there an open source video server for my purpose
YES! Check out http://www.videolan.org/
-Jesse
another approach is to use HTTP Live Streaming - HLS - the web server is simply a standard httpd server - video/audio is preprocessed on server side into a set of bitrate playlists.
The logic is on the client side to retrieve the media as a series of 6 second files, based on bandwidth appropriate playlist.
So :
- use files not memory
- there are open source HLS segmentators (ffmpeg)
I'm trying to record a DVB-Channel with a DVB-T Tuner.
I already did much research on this topic but I don't get really "information" what to do.
Basically I'm already able to create a own Graph with the default GraphEdit, make a tune request and watch a channel. Converting the Graph to C# Code with the DirectShowLib or to C++ isn't a big problem for me.
But what I don't know, what is the right approach to record the movie. (Without decode it to mpeg / avi and so on.)
The most important parts of the graph are some tuning related filters, they connect to the demultiplexer (demux), and the demux will output a video and audio stream.
The easiest way to get the mpeg stream is putting a filter before the demux. For example a samplegrabber. There you will receive the complete transport stream as it is broadcasted. But that normally contains multiple programs which are multiplexed on the same frequency. If you only need one program, you need to filter the other programs out of the stream.
If you only need a single program, it is probably easier to directly connect the audio and video stream coming out of the demultiplexer, to a multiplexer, and write it's output to a file. You need to make sure there is no decoder or any other filter between the demux and the mux. The problem is that you need to find a directshow multiplexer, as windows does not contain a standard multiplexer. I don't know any free multiplexer.
What you also can do is write the audio and video directly to a file. (again without decoding, or anything else). Then use for example ffmpeg to join the audio and video to a single file.
C:\> ffmpeg -i input.m2v -i input.mp2 -vcodec copy -acodec copy output.mpg
You probably also need to delay the audio or video stream to get them in sync.
One addition, of course you can also use ffmpeg to convert the multi program transport stream to a single program stream.