I have an audio file (.m4a) and want to split it into smaller pieces using ffmpeg. All answers I have seen do something along one of the following two possibilities:
ffmpeg -i inputFile -map 0 -f segment -segment_time 60 -c copy "$output%03d.m4a
or
ffmpeg -i inputFile -acodec copy -ss start_time -to end_time outputFile
With 1. the first file is fine. From the second file on, I just get a minute of silence in quicktime, and in VLC the file plays but the timing is odd: for example the second file should have second 0 equals to second 60 in the original file. However in vlc it starts playing on second 60 and goes on to second 120.
With 2. I have to set start times and end for each file, unfortunately I notice a small jump when I play one after the other, so it seems as if some miliseconds are lost.
There are definitely a few old questions asked around this, but none of them actually helped me with this.
Quicktime probably doesn't like files starting with a timestamp other than 0.
Use
ffmpeg -i inputFile -map 0 -f segment -segment_time 60 -reset_timestamps 1 -c copy "$output%03d.m4a
Related
I have a huge log file. I know I can tar it at end but I want the file to get zipped after every 10K line and also ensure that no data is lost.
The final goal is stop the increasing size of the file and keep it at specific limit.
Just a sample code :--
sh script.sh > log1.log &
Now, I want keep zipping log1.log so that it never crosses specific size limit.
Regards,
Abhay
let file be file.txt, then you can do :-
x=$(wc -l file.txt|cut -f 1 -d " ")
if [[ $x >> 10000 ]]
then
sed '1,10000d' file.txt > file2.txt
fi
After that just zip file2.txt and remove file2.txt
Consider using the split command. It can split by lines, bytes, pattern, etc.
split -l 10000 log1.log `date "+%Y%m%d-%H%M%S-"`
This will split the file named "log1.log" into one or more files. Each file will contain no more than 10,000 lines. These files will be named something like 20180327-085711-aa, 20180327-085711-ab, etc. You can use split's -a argument for really large log files so that it will use more than two characters in the file suffix.
The tricky part is that your shell script is still writing to the file. After the contents are split, the log must be truncated. Note that there is a small time slice between splitting the file and truncating it, so some logging data might be lost.
This example splits into 50,000 line files:
$ wc log.text
528193 1237600 10371201 log.text
$ split -l 50000 log.text `date "+%Y%m%d-%H%M%S-"` && cat /dev/null > log.text
$ ls
20180327-090530-aa 20180327-090530-ae 20180327-090530-ai
20180327-090530-ab 20180327-090530-af 20180327-090530-aj
20180327-090530-ac 20180327-090530-ag 20180327-090530-ak
20180327-090530-ad 20180327-090530-ah log.text
$ wc 20180327-090530-aa
50000 117220 982777 20180327-090530-aa
If you only want to truncate the file if it reaches a certain size (number of lines), wrap this split command in a shell script that gets run periodically (such as through cron). Here's an example of checking file size:
if (( `wc -l < log.text` > 1000000 ))
then
echo time to truncate
fi
I'm trying to test the amount of information lost with some different video codecs. I've got a python script which uses PyPNG to write a series of 8 bit RGB images. I then encode it using avconv, for instance
avconv -r 1 -i ../frames/data%03d.png -c:v ffv1 -qscale:v 0 -r 1
outffv1.avi
I then decode this back into pngs like so
avconv -r 1 -i outffv1.avi -r 1 ./outffv1/frame%03d.png
But when I compare the images before and after the video compression, they are different (mean absolute error of ~~15%). The thing that is confusing me is that this is true (give or take) independent of the codec.
For instance, I get similar answers for libtheora for a range of qscale values.
The png encoding i.e. write to png, and immediately load back in without and video compression step, is lossless.
UPDATE - more precise worked example:
Single input frame here: https://www.dropbox.com/s/2utk1xs2t8heai9/data001.png?dl=0
Encoded to video like this: avconv -r 1 -i ./frames/data%03d.png -c:v ffv1 -qscale:v 0 -r 1 outffv1.avi
resultant video here: https://www.dropbox.com/s/g1babae2a41v914/outffv1.avi?dl=0
decoded to a png again here: https://www.dropbox.com/s/8i8zg1qn7dxsgat/out001.png?dl=0
using this command: avconv -r 1 -i outffv1.avi -qscale:v 31 -r 1 out%03d.png
and image magick differenced like this
compare out001.png ./frames/data001.png diff.png
to give this (non-zero) diff
https://www.dropbox.com/s/vpouk54p0dieqif/diff.png?dl=0
Your video file most likely uses the YUV color format. PNG uses RGB. The conversion of the color is not a lossless process.
I'm trying to come up with a unix pipeline of commands that will allow me to log only the most recent n lines of a program's output to a text file.
The text file should never be more than n lines long. (it may be less when it is first filling the file)
It will be run on a device with limited memory/resources, so keeping the filesize small is a priority.
I've tried stuff like this (n=500):
program_spitting_out_text > output.txt
cat output.txt | tail -500 > recent_output.txt
rm output.txt
or
program_spitting_out_text | tee output.txt | tail -500 > recent_output.txt
Obviously neither works for my purposes...
Anyone have a good way to do this in a one-liner? Or will I have to write a script/utility?
Note: I don't want anything to do with dmesg and must use standard BSD unix commands. The "program_spitting_out_text" prints out about 60 lines/second, every second.
Thanks in advance!
If program_spitting_out_text runs continuously and keeps it's file open, there's not a lot you can do.
Even deleting the file won't help since it will still continue to write to the now "hidden" file (data still exists but there is no directory entry for it) until it closes it, at which point it will be really removed.
If it closes and reopens the log file periodically (every line or every ten seconds or whatever), then you have a relatively easy option.
Simply monitor the file until it reaches a certain size, then roll the file over, something like:
while true; do
sleep 5
lines=$(wc -l <file.log)
if [[ $lines -ge 5000 ]]; then
rm -f file2.log
mv file.log file2.log
touch file.log
fi
done
This script will check the file every five seconds and, if it's 5000 lines or more, will move it to a backup file. The program writing to it will continue to write to that backup file (since it has the open handle to it) until it closes it, then it will re-open the new file.
This means you will always have (roughly) between five and ten thousand lines in the log file set, and you can search them with commands that combine the two:
grep ERROR file2.log file.log
Another possibility is if you can restart the program periodically without affecting its function. By way of example, a program which looks for the existence of a file once a second and reports on that, can probably be restarted without a problem. One calculating PI to a hundred billion significant digits will probably not be restartable without impact.
If it is restartable, then you can basically do the same trick as above. When the log file reaches a certain size, kill of the current program (which you will have started as a background task from your script), do whatever magic you need to in rolling over the log files, then restart the program.
For example, consider the following (restartable) program prog.sh which just continuously outputs the current date and time:
#!/usr/bin/bash
while true; do
date
done
Then, the following script will be responsible for starting and stopping the other script as needed, by checking the log file every five seconds to see if it has exceeded its limits:
#!/usr/bin/bash
exe=./prog.sh
log1=prog.log
maxsz=500
pid=-1
touch ${log1}
log2=${log1}-prev
while true; do
if [[ ${pid} -eq -1 ]]; then
lines=${maxsz}
else
lines=$(wc -l <${log1})
fi
if [[ ${lines} -ge ${maxsz} ]]; then
if [[ $pid -ge 0 ]]; then
kill $pid >/dev/null 2>&1
fi
sleep 1
rm -f ${log2}
mv ${log1} ${log2}
touch ${log1}
${exe} >> ${log1} &
pid=$!
fi
sleep 5
done
And this output (from an every-second wc -l on the two log files) shows what happens at the time of switchover, noting that it's approximate only, due to the delays involved in switching:
474 prog.log 0 prog.log-prev
496 prog.log 0 prog.log-prev
518 prog.log 0 prog.log-prev
539 prog.log 0 prog.log-prev
542 prog.log 0 prog.log-prev
21 prog.log 542 prog.log-prev
Now keep in mind that's a sample script. It's relatively intelligent but probably needs some error handling so that it doesn't leave the executable running if you shut down the monitor.
And, finally, if none of that suffices, there's nothing stopping you from writing your own filter program which takes standard input and continuously outputs that to a real ring buffer file.
Then you would simply do:
program_spitting_out_text | ringbuffer 4096 last4k.log
That program could be a true ring buffer in that it treats the 4k file as a circular character buffer but, of course, you'll need a special marker in the file to indicate the write-point, along with a program that can turn it back into a real stream.
Or, it could do much the same as the scripts above, rewriting the file so that it's always below the size desired.
Since apparently this basic feature (circular file) does not exist on GNU/Linux, and because I needed it to track logs on my Raspberry Pi with limited storage, I just wrote the code as suggest above!
Behold: circFS
Unlike other tools quoted on this post and other similar, the maximum size is arbitrary and only limited by the actual available storage.
It does not rotate with several files, all is kept in the single file, which is rewritten on "release".
You can have as many log files as needed in the virtual directory.
It is a single C file (~600 lines including comments), and it builds with a single compile line after having installed fuse development dependencies.
This first version is very basic (see the README), if you want to improve it with some of the TODOs (see the TODO) be welcome to submit pull requests.
As a joke, this is my first "write only" fuse driver! :-)
Basically I have a folder with hundreds of video files(*.avi) each one with more or less an hour long. What I would like to achieve is a piece of code that could go through each one of those videos and randomly select two or three frames from each file and then stitch it back together or in alternative save the frames in a folder as jpegs.
Initially I thought I could do this using R but quickly I've realised that I would need something else possible working together with R.
Is it possible to call FFMPEG from R to do the task above?
I've trawled the internet looking for things that could help me start but most of what I've found is too specific and really applicable to what I need to do.
Could anyone please help me out or simply point me in the right direction.
Many thanks
I had a related question here recently, and found it was more straightforward to do this in bash, if you're using a Unix system.
I might get downvoted to oblivion for posting this answer here as it's not related to R, but I hope it helps. Something like this worked for me:
#!/bin/bash
for i in *.avi
do TOTAL_FRAMES=$(ffmpeg -i $i -vcodec copy -acodec copy -f null /dev/null 2>&1 | grep frame | cut -d ' ' -f 1 | sed s/frame=//)
FPS=ffmpeg -i 18_1*avi -vcodec copy -acodec copy -f null /dev/null 2>&1 | grep fps | cut -d ' ' -f 2 | sed s/fps=//
for j in {1..3}
do RANDOM_FRAME=$[RANDOM % TOTAL_FRAMES]
TIME=$((RANDOM_FRAME/FPS))
ffmpeg -ss $TIME -i $i -frames:v 1 ${i}_${j}.jpg
done
done
Basically, for each .avi in the directory, the number of frames and FPS is calculated. Then, three random frames are extracted as a .jpg using the $RANDOM function in bash and feeding the random frame into ffmpeg as a hh:mm:ss time by dividing RANDOM_FRAME by FPS.
You could always do these calculations from inside R with system() calls if you're not familiar with bash lingo.
Try this, worked for me in many format.
Written in bash ofcrs. I just randomised the seconds, minutes and hours independently in a range of duration of video by applying constraints over it and combined it to form a proper timestamp. All you need was a random timestamp in a video duration range.
#!/bin/bash
##This script will create timestamps.
NO_OF_FRAMES=10
DURATION=$(ffprobe -select_streams v -show_streams input.avi 2>/dev/null | grep DURATION | sed -e 's/TAG\:DURATION=//')
HOUR=$(echo $DURATION | cut -d ':' -f 1)
MIN=$(echo $DURATION | cut -d ':' -f 2)
SEC=$(echo $DURATION | cut -d ':' -f 3 | cut -d '.' -f 1)
for s in $(shuf -i 0-$SEC -n$NO_OF_FRAMES)
do
m=$(shuf -i 0-$MIN -n1)
h=$(shuf -i 0-$HOUR -n1)
time=$(echo $h:$m:$s)
echo $time
done
Now you can just pass it through a another loop from another script to grab your frames, or just make a function 🙃.
#!/bin/bash
time=$(bash time_stamps.sh | sort)
for TIME in $time
do
ffmpeg -ss $TIME -i input.avi -frames:v 1 sample_$TIME.jpg
done
I am using the following command
ffmpeg -i Apple.ts -map 0:0 -map 0:1 -c copy -f segment -segment_time 10 -segment_list test.m3u8 -segment_format ts 'fileSequence%d.ts'
The files do get segmented but the values are not precise. see in the .m3u8 generated below the target duration is 12 but i specified 10
#EXTM3U
#EXT-X-VERSION:3
#EXT-X-MEDIA-SEQUENCE:0
#EXT-X-ALLOWCACHE:1
#EXTINF:11.478100,
fileSequence0.ts
#EXTINF:10.410400,
fileSequence1.ts
#EXTINF:11.745067,
fileSequence2.ts
#EXTINF:7.841167,
fileSequence3.ts
#EXTINF:8.024678,
fileSequence4.ts
#EXT-X-TARGETDURATION:12
#EXT-X-ENDLIST
Also if i dont want the floting point duration how do i do it?
To get strict duration you need to ensure your input file has been encoded with a strict I picture rate. Ensure you have set keyframe interval to 10 seconds and sc_threshold to 0 while encoding the original mp4.with ffmpeg. Then run your segmentation command. This will give you exact 10 second segments.
Otherwise ffmpeg will try and cut around the 10 second mark ( or whatever duration you have given ) at the closes I picture it can find.