Lavfi video generator slow performance

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
16 messages Options
Reply | Threaded
Open this post in threaded view
|

Lavfi video generator slow performance

spookybathtub
Hello.  I'm trying to use ffmpeg to generate test images, and I'm not
getting very fast performance.  Is there any way to improve speed?  If
Lavfi is the problem, then is there any other high-performance virtual
input I can use instead?  I would actually prefer to use something more
complex, like mandelbrot, but I just changed to solid red color for this
test.

Example command:

ffmpeg -f lavfi -i color=color=red -t 60 -pix_fmt rgb48le -y -s 3840x2160
ffmpeg\%4d.tiff
ffmpeg version N-90610-g2accdd3871

This runs at 6fps.  My CPU usage is only about 1% and my disk has plenty of
bandwidth (3000 MBps).

Thanks,
Elliott
_______________________________________________
ffmpeg-user mailing list
[hidden email]
http://ffmpeg.org/mailman/listinfo/ffmpeg-user

To unsubscribe, visit link above, or email
[hidden email] with subject "unsubscribe".
Reply | Threaded
Open this post in threaded view
|

Re: Lavfi video generator slow performance

Carl Eugen Hoyos-2
2018-04-05 0:56 GMT+02:00, Elliott Balsley <[hidden email]>:

> ffmpeg -f lavfi -i color=color=red -t 60 -pix_fmt rgb48le -y
> -s 3840x2160 ffmpeg\%4d.tiff

Complete, uncut console output missing.

You get significantly higher performance if you tell
the color filter to create its output at the size you
wish instead of scaling later:
color=color=red:s=3840x2160

Carl Eugen
_______________________________________________
ffmpeg-user mailing list
[hidden email]
http://ffmpeg.org/mailman/listinfo/ffmpeg-user

To unsubscribe, visit link above, or email
[hidden email] with subject "unsubscribe".
Reply | Threaded
Open this post in threaded view
|

Re: Lavfi video generator slow performance

spookybathtub
Thank you, with that suggestion I do get approximately double the speed.
But still my CPU usage is very low.  Is there any way to get more
performance out of this?


ffmpeg.exe -f lavfi -i color=color=red:s=3840x2160 -t 60 -pix_fmt rgb48le
-y -s 3840x2160 TIFF\output%4d.tiff
ffmpeg version N-90615-g53688b62ca Copyright (c) 2000-2018 the FFmpeg
developers
  built with gcc 7.3.0 (GCC)
  configuration: --enable-gpl --enable-version3 --enable-sdl2
--enable-bzlib --enable-fontconfig --enable-gnutls --enable-iconv
--enable-libass --enable-libbluray --enable-libfreetype --enable-libmp3lame
--enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenjpeg
--enable-libopus --enable-libshine --enable-libsnappy --enable-libsoxr
--enable-libtheora --enable-libtwolame --enable-libvpx --enable-libwavpack
--enable-libwebp --enable-libx264 --enable-libx265 --enable-libxml2
--enable-libzimg --enable-lzma --enable-zlib --enable-gmp
--enable-libvidstab --enable-libvorbis --enable-libvo-amrwbenc
--enable-libmysofa --enable-libspeex --enable-libxvid --enable-libaom
--enable-libmfx --enable-amf --enable-ffnvcodec --enable-cuvid
--enable-d3d11va --enable-nvenc --enable-nvdec --enable-dxva2
--enable-avisynth
  libavutil      56. 13.100 / 56. 13.100
  libavcodec     58. 17.100 / 58. 17.100
  libavformat    58. 10.100 / 58. 10.100
  libavdevice    58.  2.100 / 58.  2.100
  libavfilter     7. 14.100 /  7. 14.100
  libswscale      5.  0.102 /  5.  0.102
  libswresample   3.  0.101 /  3.  0.101
  libpostproc    55.  0.100 / 55.  0.100
Input #0, lavfi, from 'color=color=red:s=3840x2160':
  Duration: N/A, start: 0.000000, bitrate: N/A
    Stream #0:0: Video: rawvideo (I420 / 0x30323449), yuv420p, 3840x2160
[SAR 1:1 DAR 16:9], 25 tbr, 25 tbn, 25 tbc
Stream mapping:
  Stream #0:0 -> #0:0 (rawvideo (native) -> tiff (native))
Press [q] to stop, [?] for help
[swscaler @ 0000023b208b59c0] No accelerated colorspace conversion found
from yuv420p to rgb48le.
Output #0, image2, to 'TIFF\output%4d.tiff':
  Metadata:
    encoder         : Lavf58.10.100
    Stream #0:0: Video: tiff, rgb48le, 3840x2160 [SAR 1:1 DAR 16:9],
q=2-31, 200 kb/s, 25 fps, 25 tbn, 25 tbc
    Metadata:
      encoder         : Lavc58.17.100 tiff
frame=  272 fps= 13 q=-0.0 Lsize=N/A time=00:00:10.88 bitrate=N/A
speed=0.536x
video:8817448kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB
muxing overhead: unknown
Exiting normally, received signal 2.

(I canceled it with Ctrl-C)

On Wed, Apr 4, 2018 at 4:08 PM, Carl Eugen Hoyos <[hidden email]> wrote:

> 2018-04-05 0:56 GMT+02:00, Elliott Balsley <[hidden email]>:
>
> > ffmpeg -f lavfi -i color=color=red -t 60 -pix_fmt rgb48le -y
> > -s 3840x2160 ffmpeg\%4d.tiff
>
> Complete, uncut console output missing.
>
> You get significantly higher performance if you tell
> the color filter to create its output at the size you
> wish instead of scaling later:
> color=color=red:s=3840x2160
>
> Carl Eugen
> _______________________________________________
> ffmpeg-user mailing list
> [hidden email]
> http://ffmpeg.org/mailman/listinfo/ffmpeg-user
>
> To unsubscribe, visit link above, or email
> [hidden email] with subject "unsubscribe".
_______________________________________________
ffmpeg-user mailing list
[hidden email]
http://ffmpeg.org/mailman/listinfo/ffmpeg-user

To unsubscribe, visit link above, or email
[hidden email] with subject "unsubscribe".
Reply | Threaded
Open this post in threaded view
|

Re: Lavfi video generator slow performance

Kieran O Leary
Hi

On Fri, 6 Apr 2018, 01:11 Elliott Balsley, <[hidden email]> wrote:

> Thank you, with that suggestion I do get approximately double the speed.
> But still my CPU usage is very low.  Is there any way to get more
> performance out of this?
>
>
> ffmpeg.exe -f lavfi -i color=color=red:s=3840x2160 -t 60 -pix_fmt rgb48le
> -y -s 3840x2160 TIFF\output%4d.tiff
>

Well CPU could be low cos your disk can't keep up? Each Ultra HD 16 bit RGB
frame will be about 47 megabytes. At 13 FPS,you're pushing over 600
megabytes a second.  Are you expecting your disk to perform faster than
this?

Best,

Kieran.
_______________________________________________
ffmpeg-user mailing list
[hidden email]
http://ffmpeg.org/mailman/listinfo/ffmpeg-user

To unsubscribe, visit link above, or email
[hidden email] with subject "unsubscribe".
Reply | Threaded
Open this post in threaded view
|

Re: Lavfi video generator slow performance

spookybathtub
Yes, this RAID can write at 3000MBps.

> On Apr 5, 2018, at 17:22, Kieran O Leary <[hidden email]> wrote:
>
> Hi
>
>> On Fri, 6 Apr 2018, 01:11 Elliott Balsley, <[hidden email]> wrote:
>>
>> Thank you, with that suggestion I do get approximately double the speed.
>> But still my CPU usage is very low.  Is there any way to get more
>> performance out of this?
>>
>>
>> ffmpeg.exe -f lavfi -i color=color=red:s=3840x2160 -t 60 -pix_fmt rgb48le
>> -y -s 3840x2160 TIFF\output%4d.tiff
>>
>
> Well CPU could be low cos your disk can't keep up? Each Ultra HD 16 bit RGB
> frame will be about 47 megabytes. At 13 FPS,you're pushing over 600
> megabytes a second.  Are you expecting your disk to perform faster than
> this?
>
> Best,
>
> Kieran.
> _______________________________________________
> ffmpeg-user mailing list
> [hidden email]
> http://ffmpeg.org/mailman/listinfo/ffmpeg-user
>
> To unsubscribe, visit link above, or email
> [hidden email] with subject "unsubscribe".
_______________________________________________
ffmpeg-user mailing list
[hidden email]
http://ffmpeg.org/mailman/listinfo/ffmpeg-user

To unsubscribe, visit link above, or email
[hidden email] with subject "unsubscribe".
Reply | Threaded
Open this post in threaded view
|

Re: Lavfi video generator slow performance

Carl Zwanzig
Permit me to be skeptical, I've spent a lot of time working on storage systems.

On 4/5/2018 6:40 PM, Elliott Balsley wrote:
> Yes, this RAID can write at 3000MBps.
That doesn't tell us too much. Is that a raw sequential workload or onto a
file system (and which one). (Have you benchmarked the system with a
representative workload?)

What interface (SAS, iSCSI, 10g ether, FC) and protocol? A 12g (bits per
second) SAS connection can only move 1500 MB/sec before you remove overhead,
so as a guess it's probably only good for maybe 800MB/s of data. Then
there's file-system overhead and other users. NFS over 10g ethernet will
come in slower, of course. And block size matters, too.

In short, I'd be looking very closely at the storage system and connections.
fio is a decent benchmarking tool, there are others.


And after I wrote all that above... does ffmpeg do synchronous writes? Even
with file system buffering, they're -slow-; most benchmarks use queued I/O
with long-ish queues (64/128). Also, what is ffmpeg's write buffer size? If
it's writing 512 byte blocks, it'll run a lot slower than if it's writing
64k blocks (or 128k). (I'm not diving into the code to check myself.)

Later,

z!

_______________________________________________
ffmpeg-user mailing list
[hidden email]
http://ffmpeg.org/mailman/listinfo/ffmpeg-user

To unsubscribe, visit link above, or email
[hidden email] with subject "unsubscribe".
Reply | Threaded
Open this post in threaded view
|

Re: Lavfi video generator slow performance

spookybathtub
Those are good questions, thank you.  I have 8 SSDs in RAID5 on a LSI 9361 SAS controller, formatted as NTFS.  I got the 3000 number from AJA System Test.  That test was writing a single file, so probably performance would drop when writing lots of small files.  I will benchmark again with small files.  I have no idea about ffmpeg’s write buffer size.

This machine has 28 real CPU cores, so I’m wondering if ffmpeg or lavfi is limited to one core?
Is there a way to verify whether disk write speed is the bottleneck by using ffplay?  When I run ffplay like this, I don’t see any fps readout, or maybe I don’t know where to look.

fifteen:~ elliott$ ffplay -f lavfi -i mandelbrot=s=3840x2160 -t 5
ffplay version 3.4.git Copyright (c) 2003-2018 the FFmpeg developers
  built with Apple LLVM version 9.0.0 (clang-900.0.39.2)
  configuration: --prefix=/opt/local --enable-swscale --enable-avfilter --enable-avresample --enable-libmp3lame --enable-libvorbis --enable-libopus --enable-librsvg --enable-libtheora --enable-libopenjpeg --enable-libmodplug --enable-libvpx --enable-libsoxr --enable-libspeex --enable-libass --enable-libbluray --enable-lzma --enable-gnutls --enable-fontconfig --enable-libfreetype --enable-libfribidi --disable-libjack --disable-libopencore-amrnb --disable-libopencore-amrwb --disable-indev=jack --enable-opencl --disable-outdev=xv --enable-audiotoolbox --enable-videotoolbox --enable-sdl2 --mandir=/opt/local/share/man --enable-shared --enable-pthreads --cc=/usr/bin/clang --arch=x86_64 --enable-x86asm --enable-libx265 --enable-gpl --enable-postproc --enable-libx264 --enable-libxvid --enable-nonfree --enable-libfdk-aac
  libavutil      56.  7.101 / 56.  7.101
  libavcodec     58. 11.101 / 58. 11.101
  libavformat    58.  9.100 / 58.  9.100
  libavdevice    58.  1.100 / 58.  1.100
  libavfilter     7. 12.100 /  7. 12.100
  libavresample   4.  0.  0 /  4.  0.  0
  libswscale      5.  0.101 /  5.  0.101
  libswresample   3.  0.101 /  3.  0.101
  libpostproc    55.  0.100 / 55.  0.100
Input #0, lavfi, from 'mandelbrot=s=3840x2160':KB sq=    0B f=0/0  
  Duration: N/A, start: 0.000000, bitrate: N/A
    Stream #0:0: Video: rawvideo (RGB[0] / 0x424752), rgb0, 3840x2160 [SAR 1:1 DAR 16:9], 25 tbr, 25 tbn, 25 tbc


_______________________________________________
ffmpeg-user mailing list
[hidden email]
http://ffmpeg.org/mailman/listinfo/ffmpeg-user

To unsubscribe, visit link above, or email
[hidden email] with subject "unsubscribe".
Reply | Threaded
Open this post in threaded view
|

Re: Lavfi video generator slow performance

Carl Zwanzig
A few things-

On 4/5/2018 9:23 PM, Elliott Balsley wrote:
> I have 8 SSDs in RAID5 on a LSI 9361 SAS controller, formatted as NTFS.
> I got the 3000 number from AJA System Test.  That test was writing a
> single file, so probably performance would drop when writing lots of
> small files.  I will benchmark again with small files.
Writing to RAID5 is really only as fast as a single drive since for every
block written, it has to write that data, read the entire stripe,
calculate/write the new parity. (And RAID5 is generally considered to be
slow...)

I wouldn't trust Aja's number; download a copy of 'fio' and use that (or one
of the several windows-based disk benchmarks). Since you're doing lots of
"small" files, the file system overhead will be really eat into the throughput.

> This machine has 28 real CPU cores, so I’m wondering if ffmpeg or lavfi
> is limited to one core?
Use SysInternals "process explorer" to watch the individual cores while
coding; it's much better than the windoze task manager.

All of these are readily find-able via google :).


Do you _really_ need a bunch of identical TIFF files? If they're all the
same, make one file and use a script to copy/rename it a bunch of times.
(And TIFFs are huge compared to other still image formats anyway.)

Later,

z!
_______________________________________________
ffmpeg-user mailing list
[hidden email]
http://ffmpeg.org/mailman/listinfo/ffmpeg-user

To unsubscribe, visit link above, or email
[hidden email] with subject "unsubscribe".
Reply | Threaded
Open this post in threaded view
|

Re: Lavfi video generator slow performance

Roberto Ragusa
On 04/06/2018 06:56 AM, Carl Zwanzig wrote:

> Do you _really_ need a bunch of identical TIFF files? If they're all the same, make one file and use a script to copy/rename it a bunch of times. (And TIFFs are huge compared to other still image formats anyway.)

On Linux one would use a single file and softlinks/hardlinks.
Can this be done on NTFS?


--
   Roberto Ragusa    mail at robertoragusa.it
_______________________________________________
ffmpeg-user mailing list
[hidden email]
http://ffmpeg.org/mailman/listinfo/ffmpeg-user

To unsubscribe, visit link above, or email
[hidden email] with subject "unsubscribe".
Reply | Threaded
Open this post in threaded view
|

Re: Lavfi video generator slow performance

Reindl Harald


Am 06.04.2018 um 14:47 schrieb Roberto Ragusa:
> On 04/06/2018 06:56 AM, Carl Zwanzig wrote:
>
>> Do you _really_ need a bunch of identical TIFF files? If they're all the same, make one file and use a script to copy/rename it a bunch of times. (And TIFFs are huge compared to other still image formats anyway.)
>
> On Linux one would use a single file and softlinks/hardlinks.
> Can this be done on NTFS?

https://msdn.microsoft.com/de-de/library/windows/desktop/aa365006(v=vs.85).aspx
_______________________________________________
ffmpeg-user mailing list
[hidden email]
http://ffmpeg.org/mailman/listinfo/ffmpeg-user

To unsubscribe, visit link above, or email
[hidden email] with subject "unsubscribe".
Reply | Threaded
Open this post in threaded view
|

Re: Lavfi video generator slow performance

Carl Eugen Hoyos-2
In reply to this post by spookybathtub
2018-04-06 2:04 GMT+02:00, Elliott Balsley <[hidden email]>:
> Thank you, with that suggestion I do get approximately double the speed.
> But still my CPU usage is very low.  Is there any way to get more
> performance out of this?
>
>
> ffmpeg.exe -f lavfi -i color=color=red:s=3840x2160 -t 60 -pix_fmt rgb48le

If you add " -f null -" to the above, you can test if the disk
is the bottleneck (see "fps" in the status line).
(The command line is supposed to occupy >= 8 cores)

Why do you need 48 bit output? You do realize that the scaler cannot
"invent" higher precision?

Do not top-post here, Carl Eugen
_______________________________________________
ffmpeg-user mailing list
[hidden email]
http://ffmpeg.org/mailman/listinfo/ffmpeg-user

To unsubscribe, visit link above, or email
[hidden email] with subject "unsubscribe".
Reply | Threaded
Open this post in threaded view
|

Re: Lavfi video generator slow performance

spookybathtub
> If you add " -f null -" to the above, you can test if the disk
> is the bottleneck (see "fps" in the status line).
> (The command line is supposed to occupy >= 8 cores)

Thanks for the advice everyone.  I was wrong about the RAID speeds, it does seem like disk write is the bottleneck in this case.

> Why do you need 48 bit output? You do realize that the scaler cannot
> "invent" higher precision?

To explain better:  My goal was to create an image sequence that I can use as an input in another program.  I don’t actually want a bunch of red frames, that was just an intermediate test.  What I really want is movement, something like mandelbrot or another fancy filter.  Now testing with null output, I get 3.6fps using mandelbrot.  Is there any faster way to do something like this?  Assuming I get much faster storage, I was hoping to get at least 100fps.

ffmpeg.exe -f lavfi -i mandelbrot=s=3840x2160 -t 10 -pix_fmt rgb48le -f null -
ffmpeg version N-90610-g2accdd3871 Copyright (c) 2000-2018 the FFmpeg developers
  built with gcc 7.3.0 (GCC)
  configuration: --enable-gpl --enable-version3 --enable-sdl2 --enable-bzlib --enable-fontconfig --enable-gnutls --enable-iconv --enable-libass --enable-libbluray --enable-libfreetype --enable-libmp3lame --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenjpeg --enable-libopus --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libtheora --enable-libtwolame --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxml2 --enable-libzimg --enable-lzma --enable-zlib --enable-gmp --enable-libvidstab --enable-libvorbis --enable-libvo-amrwbenc --enable-libmysofa --enable-libspeex --enable-libxvid --enable-libaom --enable-libmfx --enable-amf --enable-ffnvcodec --enable-cuvid --enable-d3d11va --enable-nvenc --enable-nvdec --enable-dxva2 --enable-avisynth
  libavutil      56. 13.100 / 56. 13.100
  libavcodec     58. 17.100 / 58. 17.100
  libavformat    58. 10.100 / 58. 10.100
  libavdevice    58.  2.100 / 58.  2.100
  libavfilter     7. 14.100 /  7. 14.100
  libswscale      5.  0.102 /  5.  0.102
  libswresample   3.  0.101 /  3.  0.101
  libpostproc    55.  0.100 / 55.  0.100
Input #0, lavfi, from 'mandelbrot=s=3840x2160':
  Duration: N/A, start: 0.000000, bitrate: N/A
    Stream #0:0: Video: rawvideo (RGB[0] / 0x424752), rgb0, 3840x2160 [SAR 1:1 DAR 16:9], 25 tbr, 25 tbn, 25 tbc
Stream mapping:
  Stream #0:0 -> #0:0 (rawvideo (native) -> wrapped_avframe (native))
Press [q] to stop, [?] for help
Output #0, null, to 'pipe:':
  Metadata:
    encoder         : Lavf58.10.100
    Stream #0:0: Video: wrapped_avframe, rgb48le, 3840x2160 [SAR 1:1 DAR 16:9], q=2-31, 200 kb/s, 25 fps, 25 tbn, 25 tbc
    Metadata:
      encoder         : Lavc58.17.100 wrapped_avframe
frame=   54 fps=3.5 q=-0.0 Lsize=N/A time=00:00:02.16 bitrate=N/A speed=0.14x
video:28kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: unknown
Exiting normally, received signal 2.
_______________________________________________
ffmpeg-user mailing list
[hidden email]
http://ffmpeg.org/mailman/listinfo/ffmpeg-user

To unsubscribe, visit link above, or email
[hidden email] with subject "unsubscribe".
Reply | Threaded
Open this post in threaded view
|

Re: Lavfi video generator slow performance

Carl Eugen Hoyos-2
2018-04-13 0:31 GMT+02:00, Elliott Balsley <[hidden email]>:

> ffmpeg.exe -f lavfi -i mandelbrot=s=3840x2160 -t 10 -pix_fmt rgb48le
> -f null -

Again:
Why do you convert to rgb48?
Do you understand that this cannot improve quality?

The mandelbrot filter is slower than the color filter, yes.
(asm optimization patch likely welcome.)

Carl Eugen
_______________________________________________
ffmpeg-user mailing list
[hidden email]
http://ffmpeg.org/mailman/listinfo/ffmpeg-user

To unsubscribe, visit link above, or email
[hidden email] with subject "unsubscribe".
Reply | Threaded
Open this post in threaded view
|

Re: Lavfi video generator slow performance

spookybathtub
> Why do you convert to rgb48?
> Do you understand that this cannot improve quality?

Yes of course.  I don’t care about the quality, but I need sample files in this format to test my other program.

> The mandelbrot filter is slower than the color filter, yes.
> (asm optimization patch likely welcome.)

I’m not a programmer, sorry.  I guess ffmpeg is not the right tool for this job.  Thanks.
_______________________________________________
ffmpeg-user mailing list
[hidden email]
http://ffmpeg.org/mailman/listinfo/ffmpeg-user

To unsubscribe, visit link above, or email
[hidden email] with subject "unsubscribe".
Reply | Threaded
Open this post in threaded view
|

Re: Lavfi video generator slow performance

Paul B Mahol
On 4/13/18, Elliott Balsley <[hidden email]> wrote:

>> Why do you convert to rgb48?
>> Do you understand that this cannot improve quality?
>
> Yes of course.  I don't care about the quality, but I need sample files in
> this format to test my other program.
>
>> The mandelbrot filter is slower than the color filter, yes.
>> (asm optimization patch likely welcome.)
>
> I'm not a programmer, sorry.  I guess ffmpeg is not the right tool for this
> job.  Thanks.

Have you ever noticed there are other video source filters? Like life, cellauto
and mptestsrc?
_______________________________________________
ffmpeg-user mailing list
[hidden email]
http://ffmpeg.org/mailman/listinfo/ffmpeg-user

To unsubscribe, visit link above, or email
[hidden email] with subject "unsubscribe".
Reply | Threaded
Open this post in threaded view
|

Re: Lavfi video generator slow performance

Moritz Barsnick
On Fri, Apr 13, 2018 at 08:17:41 +0200, Paul B Mahol wrote:
> > I'm not a programmer, sorry.  I guess ffmpeg is not the right tool for this
> > job.  Thanks.
>
> Have you ever noticed there are other video source filters? Like life, cellauto
> and mptestsrc?

Indeed, if all you need is frames with changing content...

In my setup(s), life is approx 3x fast than mandelbrot, and testsrc2 is
even 25x faster.

(mptestsrc only delivers 512x512, AFAICT, so it would need to be
rescaled again?)

Check
$ ffmpeg -filters | grep '|->V'
for more sources. I think the documentation even has a section of its
own, but I haven't checked whether all are even mentioned.

Cheers,
Moritz
_______________________________________________
ffmpeg-user mailing list
[hidden email]
http://ffmpeg.org/mailman/listinfo/ffmpeg-user

To unsubscribe, visit link above, or email
[hidden email] with subject "unsubscribe".