Understanding/preventing buffer overruns

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

Understanding/preventing buffer overruns

Simon
Hi all, I have a recording configuration going that's grabbing three
1080 30p inputs and one 720 30p input simultaneously, and seems to be
capable of either prores or libx264 crf 23 compressions. The resulting
video appears to be smooth, and when I include a clock in one of the
streams, it maintains good time over a couple of minutes of test.

But... I get almost continuous buffer overruns after about a minute.
Can someone tell me where to start learning about how I might fix this
(I presume it can't be good that it's doing this even though the
output seems to be OK, right?)

One negative thing I notice is that when I play this back with VLC it
complains about timestamps (and some other complaints about recursion
and decoder hardware that I don't think are related to this)

[00007fc4251af680] main decoder error: Could not convert timestamp
4637816048 for FFmpeg
[00007fc425099d20] main decoder error: Timestamp conversion failed
(delay 1000000, buffering 100000, bound 9000000)
[00007fc425099d20] main decoder error: Could not convert timestamp
4637849382 for FFmpeg
[00007fc4251a19f0] main decoder error: Timestamp conversion failed
(delay 1000000, buffering 100000, bound 9000000)

these repeat extensively

Here's a sample of the messages during recording (which also repeat
essentially continuously)

[decklink @ 0x55b2249e8580] Decklink input buffer overrun!
    Last message repeated 1 times
[decklink @ 0x55b2249a4040] Decklink input buffer overrun!944kB
time=00:00:52.30 bitrate=549308.8kbits/s dup=35 drop=4048 speed=0.999x
[decklink @ 0x55b2249e8580] Decklink input buffer overrun!
    Last message repeated 13 times

Can anyone point me at what part of the documentation might address
this, or better still give me hints?

TIA,
Simon
_______________________________________________
ffmpeg-user mailing list
[hidden email]
https://ffmpeg.org/mailman/listinfo/ffmpeg-user

To unsubscribe, visit link above, or email
[hidden email] with subject "unsubscribe".
Reply | Threaded
Open this post in threaded view
|

Fwd: Understanding/preventing buffer overruns

Simon
[...snip, mostly successful scenario describing simultaneous recording
of four video channels...]

> But... I get almost continuous buffer overruns after about a minute.
> Can someone tell me where to start learning about how I might fix this
> (I presume it can't be good that it's doing this even though the
> output seems to be OK, right?)
[...snip...]
> Here's a sample of the messages during recording (which also repeat
> essentially continuously)
>
> [decklink @ 0x55b2249e8580] Decklink input buffer overrun!
>     Last message repeated 1 times
> [decklink @ 0x55b2249a4040] Decklink input buffer overrun!944kB
> time=00:00:52.30 bitrate=549308.8kbits/s dup=35 drop=4048 speed=0.999x
> [decklink @ 0x55b2249e8580] Decklink input buffer overrun!
>     Last message repeated 13 times

Poking around, it seems the universal answer to all questions of this
kind is "your computer can't handle the workload". However, I'm
absolutely certain this is not the case for me since if I run a single
ffmpeg process to capture these four video plus four audio channels, I
get high rates of buffer overrun reports, but, if I run five *separate
but simultaneous* ffmpeg processes, each one capturing/compressing a
single video stream, and a fifth that captures the audio (which is
actually coming from a four channel USB device), the recordings
proceed at 1x, with no buffer overruns, and no complaints of any kind
at the default logging level. Also, the CPU load during this
five-ffmpegs-at-once totals just about one-half the total available
CPU power of the machine.

The only real downside to this approach that I perceive is that I have
a tiny (perhaps one or two frames??) synchronization error between the
relative files, but it seems to indicate that perhaps ffmpeg's
internal buffer management might either allow control that I don't
know about that would fix this (I'd love to hear about it if this is
the case) or that there is a significant opportunity for improving the
tool in respect of higher performance modern machines and this kind of
capture (I'm only capturing standard 1080/30p and increasingly it
seems that 4K is growing in popularity.)

So, to that end, I now have two new questions (unless someone can tell me how to
tune the buffers or whatever and make this work well as a single process)

1) how would I report this to the developers (the ffmpeg-devel list
seems to say that it's for discussion of development, not for
reporting issues/bugs/rfes)?

2) What pointers for how I might be able to participate in trying to
work on the improvement?

On the latter, I have to admit that it's a longshot as, a) it's been
25 years since I wrote any
C++ (I still code, but other languages), b) I would have to start
understanding this software from the ground up which would likely take
ages, and c) I have severely limited time that is "spare".
_______________________________________________
ffmpeg-user mailing list
[hidden email]
https://ffmpeg.org/mailman/listinfo/ffmpeg-user

To unsubscribe, visit link above, or email
[hidden email] with subject "unsubscribe".
Reply | Threaded
Open this post in threaded view
|

Re: Fwd: Understanding/preventing buffer overruns

Marton Balint


On Mon, 20 Jul 2020, Simon Roberts wrote:

> [...snip, mostly successful scenario describing simultaneous recording
> of four video channels...]
>
>> But... I get almost continuous buffer overruns after about a minute.
>> Can someone tell me where to start learning about how I might fix this
>> (I presume it can't be good that it's doing this even though the
>> output seems to be OK, right?)
> [...snip...]
>> Here's a sample of the messages during recording (which also repeat
>> essentially continuously)
>>
>> [decklink @ 0x55b2249e8580] Decklink input buffer overrun!
>>     Last message repeated 1 times
>> [decklink @ 0x55b2249a4040] Decklink input buffer overrun!944kB
>> time=00:00:52.30 bitrate=549308.8kbits/s dup=35 drop=4048 speed=0.999x
>> [decklink @ 0x55b2249e8580] Decklink input buffer overrun!
>>     Last message repeated 13 times
>
> Poking around, it seems the universal answer to all questions of this
> kind is "your computer can't handle the workload". However, I'm
> absolutely certain this is not the case for me since if I run a single
> ffmpeg process to capture these four video plus four audio channels, I
> get high rates of buffer overrun reports, but, if I run five *separate
> but simultaneous* ffmpeg processes, each one capturing/compressing a
> single video stream, and a fifth that captures the audio (which is
> actually coming from a four channel USB device), the recordings
> proceed at 1x, with no buffer overruns, and no complaints of any kind
> at the default logging level. Also, the CPU load during this
> five-ffmpegs-at-once totals just about one-half the total available
> CPU power of the machine.

Output IO speed can also limit the ability of the process to read the
input fast enough.

>
> The only real downside to this approach that I perceive is that I have
> a tiny (perhaps one or two frames??) synchronization error between the
> relative files, but it seems to indicate that perhaps ffmpeg's
> internal buffer management might either allow control that I don't
> know about that would fix this (I'd love to hear about it if this is
> the case) or that there is a significant opportunity for improving the
> tool in respect of higher performance modern machines and this kind of
> capture (I'm only capturing standard 1080/30p and increasingly it
> seems that 4K is growing in popularity.)

1-2 frames difference is unavoidable, even if you capture multiple inputs
in the same process, syncrhronized capture is not guaranteed at all.

>
> So, to that end, I now have two new questions (unless someone can tell me how to
> tune the buffers or whatever and make this work well as a single process)
>
> 1) how would I report this to the developers (the ffmpeg-devel list
> seems to say that it's for discussion of development, not for
> reporting issues/bugs/rfes)?

This is mailing list is the right place. Or trac.ffmpeg.org if you are
confident this is a bug (I think it is not).

>
> 2) What pointers for how I might be able to participate in trying to
> work on the improvement?

Complete command line you are using and the console output is a good
start.

>
> On the latter, I have to admit that it's a longshot as, a) it's been
> 25 years since I wrote any
> C++ (I still code, but other languages), b) I would have to start
> understanding this software from the ground up which would likely take
> ages, and c) I have severely limited time that is "spare".

As we all.

Regards,
Marton
_______________________________________________
ffmpeg-user mailing list
[hidden email]
https://ffmpeg.org/mailman/listinfo/ffmpeg-user

To unsubscribe, visit link above, or email
[hidden email] with subject "unsubscribe".
Reply | Threaded
Open this post in threaded view
|

Re: Fwd: Understanding/preventing buffer overruns

Simon
> > [...snip, mostly successful scenario describing simultaneous recording
> > of four video channels...]
> >
> >> But... I get almost continuous buffer overruns after about a minute.
> >> Can someone tell me where to start learning about how I might fix this
> >> (I presume it can't be good that it's doing this even though the
> >> output seems to be OK, right?)
> > [...snip...]
> >> Here's a sample of the messages during recording (which also repeat
> >> essentially continuously)
> >>
> >> [decklink @ 0x55b2249e8580] Decklink input buffer overrun!
> >>     Last message repeated 1 times
> >> [decklink @ 0x55b2249a4040] Decklink input buffer overrun!944kB
> >> time=00:00:52.30 bitrate=549308.8kbits/s dup=35 drop=4048 speed=0.999x
> >> [decklink @ 0x55b2249e8580] Decklink input buffer overrun!
> >>     Last message repeated 13 times
> >
> > Poking around, it seems the universal answer to all questions of this
> > kind is "your computer can't handle the workload". However, I'm
> > absolutely certain this is not the case for me since if I run a single
> > ffmpeg process to capture these four video plus four audio channels, I
> > get high rates of buffer overrun reports, but, if I run five *separate
> > but simultaneous* ffmpeg processes, each one capturing/compressing a
> > single video stream, and a fifth that captures the audio (which is
> > actually coming from a four channel USB device), the recordings
> > proceed at 1x, with no buffer overruns, and no complaints of any kind
> > at the default logging level. Also, the CPU load during this
> > five-ffmpegs-at-once totals just about one-half the total available
> > CPU power of the machine.
>
> Output IO speed can also limit the ability of the process to read the
> input fast enough.

Undoubtedly true, but hypothesis that such a problem is the cause of
my observations is inconsistent with the machine's demonstrated
ability to capture, compress, and write these streams into five files,
on the same storage medium, concurrently, simply by using five
concurrent ffmpeg processes. This hardware performs the work
successfully, but when ffmpeg tries to do it as a single process (with
many threads), it gets into trouble (or at least reports that it
does--I'm still not sure if the warning messages actually represent a
real problem or not).

As a side note, I'm writing to M.2 PCIe storage that can sink 2.4
GBytes/second continuously (yes, I did run this for long enough to
overflow the caches many times over), and the output of these captures
don't come even close; the total bandwidth for them runs less than 90
MBytes/second.

> > The only real downside to this approach that I perceive is that I have
> > a tiny (perhaps one or two frames??) synchronization error between the
> > relative files, but it seems to indicate that perhaps ffmpeg's
> > internal buffer management might either allow control that I don't
> > know about that would fix this (I'd love to hear about it if this is
> > the case) or that there is a significant opportunity for improving the
> > tool in respect of higher performance modern machines and this kind of
> > capture (I'm only capturing standard 1080/30p and increasingly it
> > seems that 4K is growing in popularity.)
>
> 1-2 frames difference is unavoidable, even if you capture multiple inputs
> in the same process, syncrhronized capture is not guaranteed at all.

Fair enough. Good to know; I shan't waste time trying to pursue that further.

> > So, to that end, I now have two new questions (unless someone can tell me how to
> > tune the buffers or whatever and make this work well as a single process)
> >
> > 1) how would I report this to the developers (the ffmpeg-devel list
> > seems to say that it's for discussion of development, not for
> > reporting issues/bugs/rfes)?
>
> This is mailing list is the right place. Or trac.ffmpeg.org if you are
> confident this is a bug (I think it is not).

I agree that it's not a bug. It's an opportunity for enhancement.
ffmpeg is *functionally* correct in all respects related to the
captures that I'm performing. There is, however, a significant
opportunity for improving the *performance* (or more precisely the
capacity/throughput).

It might be that there's tuning available at the command line that can
rectify this, but nothing has been proposed to this point, and, as I
mentioned, every other question on these lines that I've seen has
ended with "your hardware can't do that much work". I have
demonstrated that in this situation, that is not the case; my hardware
has demonstrated that it *can* do it, end-to-end.

So, is this the right place for an enhancement request/proposal? Is
there a standard form for the email header?

trac.ffmpeg.org Thanks. this looks like I shall need to poke around in
the developer docs there and see if I can begin to understand how this
thing's architecture hangs together.

> > 2) What pointers for how I might be able to participate in trying to
> > work on the improvement?
>
> Complete command line you are using and the console output is a good
> start.

I was unclear, I'm contemplating whether I might try to contribute a
fix to the source code, to put some effort into improving ffmpeg.

However, this is the command that generates buffer overruns:

ffmpeg  -f decklink -i 'DeckLink Quad HDMI Recorder (1)'   -f decklink
-raw_format argb -i 'DeckLink Quad HDMI Recorder (4)'   -f decklink -i
'DeckLink Quad HDMI Recorder (2)'   -f decklink -raw_format argb -i
'DeckLink Quad HDMI Recorder (3)' -f alsa -channels 4 -i plughw:1
-map 0:v -map 1:v -map 2:v -map 3:v   -r 30 -c:v prores_ks -profile:v
3 -q:v 4 -channels 4  -map 4:a -channel_layout quad -c:a copy
-metadata title=DeckLinkCapture  -y  capture-4chan-audiocopy.mov

Splitting this into five separate commands, generating five separate
output files, running them in background simultaneously, works without
any overruns, dropped frames, or any other complaints.

> > On the latter, I have to admit that it's a longshot as, a) it's been
> > 25 years since I wrote any
> > C++ (I still code, but other languages), b) I would have to start
> > understanding this software from the ground up which would likely take
> > ages, and c) I have severely limited time that is "spare".
>
> As we all.

Ain't that the truth, and please know I appreciate everyone's input!
_______________________________________________
ffmpeg-user mailing list
[hidden email]
https://ffmpeg.org/mailman/listinfo/ffmpeg-user

To unsubscribe, visit link above, or email
[hidden email] with subject "unsubscribe".
Reply | Threaded
Open this post in threaded view
|

Re: Fwd: Understanding/preventing buffer overruns

Marton Balint


On Mon, 20 Jul 2020, Simon Roberts wrote:

>>> [...snip, mostly successful scenario describing simultaneous recording
>> > of four video channels...]
>> >
>> >> But... I get almost continuous buffer overruns after about a minute.
>> >> Can someone tell me where to start learning about how I might fix this
>> >> (I presume it can't be good that it's doing this even though the
>> >> output seems to be OK, right?)
>> > [...snip...]
>> >> Here's a sample of the messages during recording (which also repeat
>> >> essentially continuously)
>> >>
>> >> [decklink @ 0x55b2249e8580] Decklink input buffer overrun!
>> >>     Last message repeated 1 times
>> >> [decklink @ 0x55b2249a4040] Decklink input buffer overrun!944kB
>> >> time=00:00:52.30 bitrate=549308.8kbits/s dup=35 drop=4048 speed=0.999x
>> >> [decklink @ 0x55b2249e8580] Decklink input buffer overrun!
>> >>     Last message repeated 13 times
>> >
>> > Poking around, it seems the universal answer to all questions of this
>> > kind is "your computer can't handle the workload". However, I'm
>> > absolutely certain this is not the case for me since if I run a single
>> > ffmpeg process to capture these four video plus four audio channels, I
>> > get high rates of buffer overrun reports, but, if I run five *separate
>> > but simultaneous* ffmpeg processes, each one capturing/compressing a
>> > single video stream, and a fifth that captures the audio (which is
>> > actually coming from a four channel USB device), the recordings
>> > proceed at 1x, with no buffer overruns, and no complaints of any kind
>> > at the default logging level. Also, the CPU load during this
>> > five-ffmpegs-at-once totals just about one-half the total available
>> > CPU power of the machine.
>>
>> Output IO speed can also limit the ability of the process to read the
>> input fast enough.
>
> Undoubtedly true, but hypothesis that such a problem is the cause of
> my observations is inconsistent with the machine's demonstrated
> ability to capture, compress, and write these streams into five files,
> on the same storage medium, concurrently, simply by using five
> concurrent ffmpeg processes. This hardware performs the work
> successfully, but when ffmpeg tries to do it as a single process (with
> many threads), it gets into trouble (or at least reports that it
> does--I'm still not sure if the warning messages actually represent a
> real problem or not).
>
> As a side note, I'm writing to M.2 PCIe storage that can sink 2.4
> GBytes/second continuously (yes, I did run this for long enough to
> overflow the caches many times over), and the output of these captures
> don't come even close; the total bandwidth for them runs less than 90
> MBytes/second.
>
>> > The only real downside to this approach that I perceive is that I have
>> > a tiny (perhaps one or two frames??) synchronization error between the
>> > relative files, but it seems to indicate that perhaps ffmpeg's
>> > internal buffer management might either allow control that I don't
>> > know about that would fix this (I'd love to hear about it if this is
>> > the case) or that there is a significant opportunity for improving the
>> > tool in respect of higher performance modern machines and this kind of
>> > capture (I'm only capturing standard 1080/30p and increasingly it
>> > seems that 4K is growing in popularity.)
>>
>> 1-2 frames difference is unavoidable, even if you capture multiple inputs
>> in the same process, syncrhronized capture is not guaranteed at all.
>
> Fair enough. Good to know; I shan't waste time trying to pursue that further.
>
>> > So, to that end, I now have two new questions (unless someone can tell me how to
>> > tune the buffers or whatever and make this work well as a single process)
>> >
>> > 1) how would I report this to the developers (the ffmpeg-devel list
>> > seems to say that it's for discussion of development, not for
>> > reporting issues/bugs/rfes)?
>>
>> This is mailing list is the right place. Or trac.ffmpeg.org if you are
>> confident this is a bug (I think it is not).
>
> I agree that it's not a bug. It's an opportunity for enhancement.
> ffmpeg is *functionally* correct in all respects related to the
> captures that I'm performing. There is, however, a significant
> opportunity for improving the *performance* (or more precisely the
> capacity/throughput).
>
> It might be that there's tuning available at the command line that can
> rectify this, but nothing has been proposed to this point, and, as I
> mentioned, every other question on these lines that I've seen has
> ended with "your hardware can't do that much work". I have
> demonstrated that in this situation, that is not the case; my hardware
> has demonstrated that it *can* do it, end-to-end.
>
> So, is this the right place for an enhancement request/proposal? Is
> there a standard form for the email header?
>
> trac.ffmpeg.org Thanks. this looks like I shall need to poke around in
> the developer docs there and see if I can begin to understand how this
> thing's architecture hangs together.
>
>> > 2) What pointers for how I might be able to participate in trying to
>> > work on the improvement?
>>
>> Complete command line you are using and the console output is a good
>> start.
>
> I was unclear, I'm contemplating whether I might try to contribute a
> fix to the source code, to put some effort into improving ffmpeg.
>
> However, this is the command that generates buffer overruns:
>
> ffmpeg  -f decklink -i 'DeckLink Quad HDMI Recorder (1)'   -f decklink
> -raw_format argb -i 'DeckLink Quad HDMI Recorder (4)'   -f decklink -i
> 'DeckLink Quad HDMI Recorder (2)'   -f decklink -raw_format argb -i
> 'DeckLink Quad HDMI Recorder (3)' -f alsa -channels 4 -i plughw:1
> -map 0:v -map 1:v -map 2:v -map 3:v   -r 30 -c:v prores_ks -profile:v
> 3 -q:v 4 -channels 4  -map 4:a -channel_layout quad -c:a copy
> -metadata title=DeckLinkCapture  -y  capture-4chan-audiocopy.mov

I'd try to find out if capturing without audio makes any difference.

Regards,
Marton
_______________________________________________
ffmpeg-user mailing list
[hidden email]
https://ffmpeg.org/mailman/listinfo/ffmpeg-user

To unsubscribe, visit link above, or email
[hidden email] with subject "unsubscribe".
Reply | Threaded
Open this post in threaded view
|

Re: Fwd: Understanding/preventing buffer overruns

Carl Zwanzig
In reply to this post by Simon
On 7/20/2020 9:57 AM, Simon Roberts wrote:
> Splitting this into five separate commands, generating five separate
> output files, running them in background simultaneously, works without
> any overruns, dropped frames, or any other complaints.

This more suggests either a threading/process management issue within ffmpeg
or that the underlying OS is starving the ffmpeg process for resources.
(IIRC, some codecs/etc are single-threaded.)

Which OS?

What are the CPU/memory usages in each running scenario?

Try capturing uncompressed video to the null device; then try encoded to a
null device; that should show if there's an issue with the encoder or
elsewhere. Try using a different encoder, too, for test.


And please give us the cmd output (again)- at least with the headers, stream
info, and a few error messages.

Later,

z!
_______________________________________________
ffmpeg-user mailing list
[hidden email]
https://ffmpeg.org/mailman/listinfo/ffmpeg-user

To unsubscribe, visit link above, or email
[hidden email] with subject "unsubscribe".