[hw] strange stream behaviour

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
5 messages Options
hw
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

[hw] strange stream behaviour

hw
Hi,

there are video streams showing a strange behaviour when recording them
with ffmpeg in that ffmpeg will record the stream for an apparently
random amount of time, ranging from one minute to ten minutes, and then
stop recording.  Once it stops recording, ffmpeg continues to run
indefinitely, not receiving and not recording anything.

I have programmed my recording application that uses ffmpeg to overcome
this problem such that it monitors the size of the recorded file.  If
the size doesn´t change for at least eight seconds, the ffmpeg process
supposed to record it is being killed and a new one is started to record
the same stream.

When the recording is finished, I have a number of files (chunks) that
were recorded, and I can use ffmpeg to concatenate these files.  When
watching the concatenated version of the recording, the video that has
been recorded shows "backjumps": That means there are a few seconds that
were at the end of chunk N which also are at the beginning of chunk
N+1.  How many seconds are overlapping varies.

This is, of course, annoying.  It is also puzzling because I used to
think that the recording stops because there sometimes is insufficient
bandwidth to continue to record.  This doesn´t seem to be the case,
though:


The stream being recorded is an hls stream accessed via the http URL of
an m3u8 playlist.

There are such streams that can be played with ffplay without
interruption for hours.  (I haven´t tried recording those yet.  Since
the streams that can not be recorded continuously also cut off when
being played with ffplay, I can assume it would be possible to record
the streams that play uninterrupted as continuously as they play.)


Now I´m assuming that the problem isn´t insufficient bandwidth or an
issue with ffmepg, but more likely an incompatible or troubled server.
That hls provides the "stream" (I wouldn´t call that a stream because
"stream" implies a continuous flow) in small chunks can explain the
annoying overlaps I am observing:

When the recording is restarted, it starts with a chunk that already has
been received and continues with the next chunks provided, some of which
also have already been recorded.

It can take about 10 seconds for ffmpeg to be restarted.  I have not
observed a single missing chunk, not even with a two hour movie
scattered over thirty single files.  There have only been overlaps.

If there was a bandwidth problem, I would expect chunks to be missing.
Since no chunks are missing even after a rather lengthy break in
recording, all chunks seem to be available on the server for more then
ten seconds.

So what if the time between the server telling ffmpeg 'use this next
chunk' and the availability of the chunk on the server is somewhat long?
Is there a time window within which ffmpeg tries to get the next chunk,
and is that window too short?  Is there an option to make that window
longer, or a place in the source where I could make it longer?

What else might cause the interruptions?

Why is ffmpeg unaware that recording has ceased and doesn´t quit when
that happens?  It is silly that the control program needs to poll the
size of the output file every eight seconds to figure out if something
is still being recorded or not.  Doing that isn´t really feasible,
either, because it is not guaranteed that eight seconds is always the
right polling interval.

Why doesn´t ffmpeg quit after it has been recording for the amount of
time specified with the '-t' option?  For example:


ffmpeg -n -loglevel 8 -i http://example.com/foo.m3u8 -codec copy -t 00:10:00 -f mpegts example


Ffmpeg does not stop after the ten minutes have passed when the
recording has stopped before that.  That doesn´t make any sense because
ffmpeg obviously either figures it is still recording --- in which case
it needs to stop after ten minutes --- or it figures it is not
recording, in which case it should quit before the ten minutes have
passed, or at least when they have passed.  Having it run indefinitely
is not the right choice.

So what can I do to record without interruptions, or at least without
the annoying overlaps?
_______________________________________________
ffmpeg-user mailing list
[hidden email]
http://ffmpeg.org/mailman/listinfo/ffmpeg-user

To unsubscribe, visit link above, or email
[hidden email] with subject "unsubscribe".
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [hw] strange stream behaviour

Bhikkhu Mettavihari
Greetings

I note you did not have a reply to this long and exhaustive mail

if you have solved it I would be happy to know the solution
If not try to put -report into the command line.

ffmpeg -n -loglevel 8 -report -i http://example.com/foo.m3u8 -codec
copy -t 00:10:00 -f mpegts example


and post the output of the log file
you may then get a response which will be useful for me also

with best regards
Mettavihari

On Sun, Aug 6, 2017 at 9:03 PM, hw <[hidden email]> wrote:

> Hi,
>
> there are video streams showing a strange behaviour when recording them
> with ffmpeg in that ffmpeg will record the stream for an apparently
> random amount of time, ranging from one minute to ten minutes, and then
> stop recording.  Once it stops recording, ffmpeg continues to run
> indefinitely, not receiving and not recording anything.
>
> I have programmed my recording application that uses ffmpeg to overcome
> this problem such that it monitors the size of the recorded file.  If
> the size doesn´t change for at least eight seconds, the ffmpeg process
> supposed to record it is being killed and a new one is started to record
> the same stream.
>
> When the recording is finished, I have a number of files (chunks) that
> were recorded, and I can use ffmpeg to concatenate these files.  When
> watching the concatenated version of the recording, the video that has
> been recorded shows "backjumps": That means there are a few seconds that
> were at the end of chunk N which also are at the beginning of chunk
> N+1.  How many seconds are overlapping varies.
>
> This is, of course, annoying.  It is also puzzling because I used to
> think that the recording stops because there sometimes is insufficient
> bandwidth to continue to record.  This doesn´t seem to be the case,
> though:
>
>
> The stream being recorded is an hls stream accessed via the http URL of
> an m3u8 playlist.
>
> There are such streams that can be played with ffplay without
> interruption for hours.  (I haven´t tried recording those yet.  Since
> the streams that can not be recorded continuously also cut off when
> being played with ffplay, I can assume it would be possible to record
> the streams that play uninterrupted as continuously as they play.)
>
>
> Now I´m assuming that the problem isn´t insufficient bandwidth or an
> issue with ffmepg, but more likely an incompatible or troubled server.
> That hls provides the "stream" (I wouldn´t call that a stream because
> "stream" implies a continuous flow) in small chunks can explain the
> annoying overlaps I am observing:
>
> When the recording is restarted, it starts with a chunk that already has
> been received and continues with the next chunks provided, some of which
> also have already been recorded.
>
> It can take about 10 seconds for ffmpeg to be restarted.  I have not
> observed a single missing chunk, not even with a two hour movie
> scattered over thirty single files.  There have only been overlaps.
>
> If there was a bandwidth problem, I would expect chunks to be missing.
> Since no chunks are missing even after a rather lengthy break in
> recording, all chunks seem to be available on the server for more then
> ten seconds.
>
> So what if the time between the server telling ffmpeg 'use this next
> chunk' and the availability of the chunk on the server is somewhat long?
> Is there a time window within which ffmpeg tries to get the next chunk,
> and is that window too short?  Is there an option to make that window
> longer, or a place in the source where I could make it longer?
>
> What else might cause the interruptions?
>
> Why is ffmpeg unaware that recording has ceased and doesn´t quit when
> that happens?  It is silly that the control program needs to poll the
> size of the output file every eight seconds to figure out if something
> is still being recorded or not.  Doing that isn´t really feasible,
> either, because it is not guaranteed that eight seconds is always the
> right polling interval.
>
> Why doesn´t ffmpeg quit after it has been recording for the amount of
> time specified with the '-t' option?  For example:
>
>
> ffmpeg -n -loglevel 8 -i http://example.com/foo.m3u8 -codec copy -t 00:10:00 -f mpegts example
>
>
> Ffmpeg does not stop after the ten minutes have passed when the
> recording has stopped before that.  That doesn´t make any sense because
> ffmpeg obviously either figures it is still recording --- in which case
> it needs to stop after ten minutes --- or it figures it is not
> recording, in which case it should quit before the ten minutes have
> passed, or at least when they have passed.  Having it run indefinitely
> is not the right choice.
>
> So what can I do to record without interruptions, or at least without
> the annoying overlaps?
> _______________________________________________
> ffmpeg-user mailing list
> [hidden email]
> http://ffmpeg.org/mailman/listinfo/ffmpeg-user
>
> To unsubscribe, visit link above, or email
> [hidden email] with subject "unsubscribe".



--
Streaming video from http://learntv.lk
_______________________________________________
ffmpeg-user mailing list
[hidden email]
http://ffmpeg.org/mailman/listinfo/ffmpeg-user

To unsubscribe, visit link above, or email
[hidden email] with subject "unsubscribe".
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [hw] strange stream behaviour

Moritz Barsnick
On Sat, Aug 12, 2017 at 12:15:01 +0530, Mettavihari D wrote:
> if you have solved it I would be happy to know the solution
> If not try to put -report into the command line.
>
> ffmpeg -n -loglevel 8 -report -i http://example.com/foo.m3u8 -codec
> copy -t 00:10:00 -f mpegts example
>
> and post the output of the log file
> you may then get a response which will be useful for me also

I agree. At least to understand the "-t" issue, we need to see the
complete, uncut console output from ffmpeg.

Regarding the apparent stalling: I have no idea. ffmpeg with a high
enough loglevel (probably default nowadays) reports which m3u8 segments
it is opening. Does the progress line show any progress? Increase in
number of frames, in size, does the time increase?

For m3u8, it might also be worth finding a tool which actually
downloads the segments without de-/remuxing them. youtube-dl can
basically do this, though I haven't figured out how with live m3u8 and
with plain m3u8 URLs. Otherwise, write a downloader of your own (I have
done it for non-live URLs) and see if the server does something
incorrectly - like not providing the segments or stalling on them or
something.

Moritz
_______________________________________________
ffmpeg-user mailing list
[hidden email]
http://ffmpeg.org/mailman/listinfo/ffmpeg-user

To unsubscribe, visit link above, or email
[hidden email] with subject "unsubscribe".
hw
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [hw] strange stream behaviour

hw
In reply to this post by Bhikkhu Mettavihari
Mettavihari D <[hidden email]> writes:

> Greetings
>
> I note you did not have a reply to this long and exhaustive mail
>
> if you have solved it I would be happy to know the solution

I don´t have a solution with ffmpeg yet, so I´m planning to write
something myself that figures out which snippets to get from the server
and downloads them.  Once the snippets have been saved, I can use ffmpeg
to concatenate them as I do now, the only difference being that I´ll
have more snippets than otherwise --- and no annoying jumps in the
recordings when I get all snippets.

They do have about half an hour of video available on the server at any
given time, and since I´m not watching life, it doesn´t matter in which
order the snippets are being downloaded.  I only need to concatenate
them in the right order once I have them.

It´s simply ffmpeg failing to download the snippets.

> If not try to put -report into the command line.
>
> ffmpeg -n -loglevel 8 -report -i http://example.com/foo.m3u8 -codec
> copy -t 00:10:00 -f mpegts example

Hm, thanks, that seems to confirm what I already suspected:

Ffmpeg just sits there, waiting indefinitely for an answer to its GET
request from the server that never arrives.  What it needs to do is time
out the GET requests and retry them if no timely answer arrives.

Perhaps that isn´t beeing done because there´s some assumption that the
stream needs to be recorded life so that snippets are not re-requested
once their assumed lifetime is over.  Re-requesting them would also
involve some sort of cache which ffmpeg doesn´t seem to have.

However, mplayer does have a cache and doesn´t do any better.

I doubt that this ever will be fixed.


You could try something like this:


for i in $(curl 'http://example.m3u8' | sed -e 's/#EXTINF:10.000,//'); do ./wg.sh $i; done


wg.sh:


#!/bin/sh

wget -T 7 "$1" &
exit


If you do that over and over again, you´ll end up with all the snippets
you need.  It just needs to be refined so that the snippets are being
downloaded in a more controlled manner, and not all of them at once.

That definitely works where ffmpeg fails to get the snippets.  I can
concatenate them and play the resulting video, which plays flawless for
about half an hour.  That´s how I know that they do have half an hour of
video on the server all the time.

As for the programming, what I´m having trouble with is doing the
required multitasking in perl without forking lots of processes.  But
I´ll find a way.
_______________________________________________
ffmpeg-user mailing list
[hidden email]
http://ffmpeg.org/mailman/listinfo/ffmpeg-user

To unsubscribe, visit link above, or email
[hidden email] with subject "unsubscribe".
hw
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [hw] strange stream behaviour

hw
In reply to this post by Moritz Barsnick
Moritz Barsnick <[hidden email]> writes:

> On Sat, Aug 12, 2017 at 12:15:01 +0530, Mettavihari D wrote:
>> if you have solved it I would be happy to know the solution
>> If not try to put -report into the command line.
>>
>> ffmpeg -n -loglevel 8 -report -i http://example.com/foo.m3u8 -codec
>> copy -t 00:10:00 -f mpegts example
>>
>> and post the output of the log file
>> you may then get a response which will be useful for me also
>
> I agree. At least to understand the "-t" issue, we need to see the
> complete, uncut console output from ffmpeg.
>
> Regarding the apparent stalling: I have no idea. ffmpeg with a high
> enough loglevel (probably default nowadays) reports which m3u8 segments
> it is opening. Does the progress line show any progress? Increase in
> number of frames, in size, does the time increase?

The server does not deliver, and when it does, it does not deliver what
could be considered a "stream".  I´ve got my recording software mostly
finished now, so I can see very well what the server does:


+ requests for the playlist sometimes time out (30s timeout)

+ maybe half the requests for segments listed in the playlist time out,
  and they can time out multiple times (The timeout I´m using for them
  varies; it can take several minutes to receive a segment in full.)

+ segments that appear later in the playlist can be duplicates of
  segments that appear earlier in the playlist, and there is no way to
  tell whether a segment is a duplicate or not before all desired
  segments have been received


I can only say: That sucks badly.

It´s amazing that the ppl running the server are even well paid for.

In any case, I can see how ffmpeg is unable to handle this.  Still I
think it´s a bug that ffmpeg sits around indefinitely trying to record
data that will never arrive.

> For m3u8, it might also be worth finding a tool which actually
> downloads the segments without de-/remuxing them. youtube-dl can
> basically do this, though I haven't figured out how with live m3u8 and
> with plain m3u8 URLs. Otherwise, write a downloader of your own (I have
> done it for non-live URLs) and see if the server does something
> incorrectly - like not providing the segments or stalling on them or
> something.

Yes, that´s what I did, and it´s what the server does.

I need to make a recording over a longer period of time to see if I am
still able to receive all segments, or if I get so far behind that some
will be missing, no matter what I do.


Is it normal to receive 62 duplicate segments when recording over a time
span of 10 minutes?  That is more than half of the segments, i. e. after
unlinking the 62 duplicates, 59 files remain.

But well, I put them together with cat, and the resulting video is
flawless.
_______________________________________________
ffmpeg-user mailing list
[hidden email]
http://ffmpeg.org/mailman/listinfo/ffmpeg-user

To unsubscribe, visit link above, or email
[hidden email] with subject "unsubscribe".
Loading...