I'm trying to process a remote m3u8 playlist containing (a possibly encrypted) HLS stream. The output I'm looking for is a mp4 container with MPEG-4 inside.
The playlist is a result of an ended live stream and may contain EXT-X-DISCONTINUITY tags. As I understand, there's no "built-in" way to process it and indeed, there are plenty of warnings like "Non-monotonous DTS in output stream" and the resulting file always has some playback issues.
There are a couple of options to "glue" it. On an unencrypted stream, I found concat demuxer to produce the result with the least playback problems. The command is:
LIST=chunks.list; ffmpeg -loglevel 'debug' -f concat -safe 0 -protocol_whitelist "file,http,https,tcp,tls,crypto" -i $LIST -c copy -movflags frag_keyframe -y output_concat.mp4
where chunks.list is something like:
file 'https://www.example.org/chunk1.ts'
file 'https://www.example.org/chunk2.ts'
file 'https://www.example.org/chunk3.ts'
Now, I'm trying to use concat demuxer to process encrypted chunks. I've tried passing -key and -iv options in different places and changing chunks.list to be like file 'crypto+https...' but it won't pick up encryption key:
Opening an input file: chunks.list.
[concat # 0x7f9fb6800c00] Opening 'chunks.list' for reading
[NULL # 0x7f9fb6007e00] Opening 'crypto+https://www.example.org/chunk1.ts' for reading
[crypto # 0x7f9fb5700a00] decryption key not set
[concat # 0x7f9fb6800c00] Impossible to open 'crypto+https://www.example.org/chunk1.ts'
[AVIOContext # 0x7f9fb5700780] Statistics: 5094 bytes read, 0 seeks
chunks.list: Invalid argument
Docs mention encryption options for crypto protocol, so it looks it's just a matter of passing these in a proper way.
When, instead of using concat demuxer, I try to process and decrypt only one chunk like:
ffmpeg -i crypto+https://www.example.org/chunk1.ts -key <my_hex_key> -iv <my_iv> chunk1.ts it works fine. There's nothing wrong with the key itself, I'm able to decrypt it with other tools (openssl etc).
Is it possible for concat demuxer to handle decryption? If so, where should I pass key and iv options?
Related
I have a large (8GB) packet capture (.pcap) that has generated a number of Snort alerts.
I suspect that some data may have been exfiltrated while encoded. Is there an easy way to determine if anything has gone out as Base64 encoded without having to sift through Wireshark to find the problem, perhaps using the terminal?
My worry is that an actor could conduct lateral movement within the system and extract files to a 3rd party system which wouldn't be recognized by Snort alerts.
I've tried parsing the large .pcap into 200MB files to allow closer examination (my VM has memory limitations).
This inspired me to write pdml2flow-base64strings a plugin for pdml2flow. Using the power of pdml2flow the plugin searches in all fields known by wireshark/tshark for valid base64 encoded data. If it finds base64 strings it then decodes and prints the raw data for you. You can use the --minlength switch, if you want to limit the search only for data of a certain size. It also allows you to narrow down your search to only ascii (--ascii) or utf-8 (--utf8).
For example if you want to extract all utf-8 strings encoded in any field known by wireshark/tshark you could use pdml2flow with the plugin:
tshark -r dump.cap -Tpdml | pdml2flow +base64strings --utf8
or if you don't want flow aggregation use pdml2frame
tshark -r dump.cap -Tpdml | pdml2frame +base64strings --utf8
I hope this helps. Any input is welcome, thank you.
Disclosure: I am the author of pdml2flow and pdml2flow-base64strings
Not an easy task but you could try using tsharkon the command line.
Generated base64 payload:
echo "base64 encoded payload" | base64
YmFzZTY0IGVuY29kZWQgcGF5bG9hZAo=
Setup a listening socket:
netcat -l -p 8090
POST to it
curl -X POST -d 'YmFzZTY0IGVuY29kZWQgcGF5bG9hZAo=' http://localhost:8090
^C
Parsing on command line after capturing with wireshark:
tshark -r ~/tmp/base64.pcapng -T fields -e http.file_data | grep -E '^([A-Za-z0-9+/]{4})*([A-Za-z0-9+/]{3}=|[A-Za-z0-9+/]{2}==)?$'
Result (plus empty lines for non matches, not shown for simplicity's sake):
YmFzZTY0IGVuY29kZWQgcGF5bG9hZAo=
There is one file I want to encrypt with GnuPG by
gpg2 --homedir=~/.gnupg --always-trust=true --recipient="BlahBlah" --encrypt=/path/to/file --output=/path/to/output_file
However this command seems to hang forever and never return. Interestingly, after I interrupt process, there is indeed /path/to/output_file created , however the bytes written there is much bigger than raw payload (for example my /path/to/file is only of 5 bytes but it turns out there are nearly 200 bytes written to /path/to/output_file).
There must be something wrong, but I really couldn't figure out what is it.
I have in advance imported the key for BlahBlah by gpg --import key.asc. It happens both for GnuPG 1 and GnuPG 2.
You're applying --encrypt in a wrong way. --encrypt does not expect any parameters, the file(s) to be worked on are passed as very last arguments. Additionally, following the documentation you should pass --output /path/to/output_file instead of --output=/path/to/output_file. Finally, GnuPG distinguishes between options and commands, and options should precede commands.
What you observe is that GnuPG starts writing header information, but then waits for input from STDIN (until interrupted).
The GnuPG command line you're looking for is
gpg2 --homedir=~/.gnupg --always-trust=true --recipient="BlahBlah" --output /path/to/output_file --encrypt /path/to/file
One last hint: the combination of --always-trust=true and resolving a recipient by user ID is a very bad idea, as any other key with the same user ID in the local keyring might be used. Pass the full key's fingerprint instead, which specifically selects a distinct key (using short key IDs is not secure, either).
I'm currently doing a stream that is supposed to display correctly within Flowplayer.
First I send it to another PC via RTP. Here, I also checked with VLC that the codec etc. arrive correctly, which they do.
Now I want to expose this stream to Flowplayer as a file, so it can be displayed, via something I used in VLC:
http://localhost:8080/test.mp4
for example.
The full line I got is: ffmpeg -i input -f mp4 http://localhost:8080/test.mp4
However, no matter how I try to do this, I only get an input/output error. Is this only possible with something like ffserver or another?
What I think is this doesn't work because ffmpeg can't act as a server; on VLC it works since it can. (Though VLC ruins the codecs I set and it can't be read afterwards for some reason)
A (sort of) workaround I can use is saving the RTP stream to a file, and then letting flowplayer load it. This, however, only works once the file is not accessed anymore; I get a codec error otherwise.
To have FFmpeg act as an HTTP server, you need to pass the -listen 1 option. Additionally, -f mp4 will result in a non-fragmented MP4, which is not suitable for streaming. You can get a fragmented MP4 with -movflags frag_keyframe+empty_moov. A full working command line is:
ffmpeg -i input -listen 1 -f mp4 -movflags frag_keyframe+empty_moov http://localhost:8080
Other options you may find helpful are -re to limit the streaming speed to the input framerate, -stream_loop -1 to loop the input, and -c copy to avoid reencoding.
you need this command line
ffmpeg -f v4l2 -s 320x240 -r 25 -i /dev/video0 -f alsa -ac 1 -i hw:0 http://localhost:8090/feed1.ffm
make sure that your feed name ends with ".ffm" and if it's not the case, then add "-f ffm" before your feed URL, to manually specify the output format (because ffmpeg won't be able to figure it out automatically any more), like this "-f ffm http://localhost:8090/blah.bleh".
I trying to decrypt AES-128 encrypted m3u8 video files such as this one :
the m3u8 file :
#EXTM3U
#EXT-X-MEDIA-SEQUENCE:0
#EXT-X-ALLOW-CACHE:NO
#EXT-X-VERSION:2
#EXT-X-FAXS-CM:MII6lAYJKoZIhvcNAQcCoII6hTCCOoECAQExCzAJBgUrDgMCGgUAM... very long key...
#EXT-X-KEY:METHOD=AES-128,URI="faxs://faxs.adobe.com",IV=0X99b74007b6254e4bd1c6e03631cad15b
#EXT-X-TARGETDURATION:8
#EXTINF:8,
video.mp4Frag1Num0.ts
#EXTINF:8,
video.mp4Frag1Num1.ts
...
I've tried with openssl :
openssl aes-128-cbc -d -kfile key.txt -iv 99b74007b6254e4bd1c6e03631cad15b -nosalt -in video_enc.ts -out video_dec.ts
key.txt contains the very long key
-->
bad decrypt
1074529488:error:06065064:digital envelope routines:EVP_DecryptFinal_ex:bad decrypt:evp_enc.c:539:
What am-I doing wrong ?
This might be a bit of a hack, but given a URL to an .m3u8 file, it will download and decrypt the files that make up the stream:
#!/usr/bin/env bash
curl "$1" -s | awk 'BEGIN {c=0} $0 ~ "EXT-X-KEY" {urlpos=index($0,"URI=")+5; ivpos=index($0,"IV="); keyurl=substr($0, urlpos, ivpos-urlpos-2); iv=substr($0, ivpos+5); print "key=`curl -s '\''"keyurl"'\'' | hexdump -C | head -1 | sed \"s/00000000//;s/|.*//;s/ //g\"`"; print "iv="iv} $0 !~ "-KEY" && $0 ~ "http" {printf("curl -s '\''"$0"'\'' | openssl aes-128-cbc -K $key -iv $iv -d >seg%05i.ts\n", c++)}' | bash
This script generates a second script that extracts keys and initialization vectors and uses them to decrypt while downloading. It needs curl, awk, hexdump, sed, and openssl to run. It'll probably choke on an unencrypted stream, or on a stream that uses something other than AES-128 (is any other encryption supported?).
You'll get a bunch of files: seg00000.ts, seg00001.ts, etc. Use tsMuxeR (https://www.videohelp.com/software/tsMuxeR) to merge these into a single file (simple concatenation didn't work for me...it's what I tried first):
(echo "MUXOPT --no-pcr-on-video-pid --new-audio-pes --vbr --vbv-len=500"; (echo -n "V_MPEG4/ISO/AVC, "; for i in seg*.ts; do echo -n "\"$i\"+"; done; echo ", fps=30, insertSEI, contSPS, track=258") | sed "s/+,/,/"; (echo -n "A_AAC, "; for i in seg*.ts; do echo -n "\"$i\"+"; done; echo ", track=257") | sed "s/+,/,/") >video.meta
tsMuxeR video.meta video.ts
(Track IDs and framerate may need adjustment...get the values to use by passing one of the downloaded files to tsMuxeR.)
Then use ffmpeg to remux to something a bit more widely understood:
ffmpeg -i video.ts -vcodec copy -acodec copy video.m4v
In order to decrypt encrypted video stream you need encryption key.
This key is not part of the stream. It should be obtained separately.
EXT-X-FAXS-CM header contains DRM meta-data and not the key.
This is excert from Adobe Media Server developer guide:
The Adobe Access Server protected variant playlist also needs to include the #EXT-X-FAXS-CM tag. The value of #EXT-X-FAXS-CM tag in variant playlist is the relative URI referring to the DRM metadata of one of the individual streams.At the client, the #EXT-X-FAXS-CM tag in variant playlist will be used to create the DRM session. The same DRM session will be used for all encrypted M3U8 files inside the variant playlist.
Full guide can be found here:
http://help.adobe.com/en_US/adobemediaserver/devguide/WS5262178513756206-4b6aabd1378392bb59-7fe8.html
There is also mention that faxs://faxs.adobe.com URI is for local key serving.
So key obtained locally from a device.
While some of the bash scripts in the existing answers get you part (or even all) of the way, depending which site you're trying to download from, you might hit other obstacles (different auth method, custom license server mount, etc.)
I've found streamlink to be the most robust solution for this, which also lets you stream directly (rather than download), if that's what you're after, and it has all the site-specific work already done for you for a long list of sites (see plugins section, but keep in mind it's under active development and the latest release was in June, so for some of the newer ones you'll have to git clone and install from source).
In many cases, VLC will happily convert an .m3u8 video to an unencrypted .ts or .mp4. In the VLC graphical interface, go to Media > Convert/Save.
Even through this file includes AES encrypted data, openssl don't know the m3u8 format. However FFmpeg might be able to handle it.
I have a single key encrypted HTTP Live Stream which decodes fine in Quicktime and iPhone. I'm trying to create a simple client application to do the decryption of the ts files. Right now I've used openssl to decrypt. I believe I have the correct arguments to openssl and I'm inserting the key and the IV properly. I can successfully decrypt the first .ts file in the stream but fail to decrypt after. I'm using the following script which I found in the archives here and I reversed to do decryption:
#!/bin/sh
hexKey=$(cat encryption.key | hexdump-e '16/1 "%02x"')
hexIV='00000000000000000000000000000001'
openssl aes-128-cbc -d -in ./multi_2.ts -out ./clear.ts -p-nosalt -iv ${hexIV} -K ${hexKey}
where:
encryption.key is the key file I retrieved from the M3u8 stream file.
multi_2.ts is the second .ts file in the m3u8 stream file.
Any help would be appreciated.
The subsequent .ts files are probably fine, you just need to join them into one file in order to be playable:
cat clear*.ts > joined.ts
hexIV='00000000000000000000000000000001'
should be :
hexIV='00000000000000000000000000000002'
as multi_2.ts is the second .ts file in the m3u8 stream file.