FFMPEG endlessly increases memory usage over time with HLS packaging.

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

FFMPEG endlessly increases memory usage over time with HLS packaging.

alexdjca
Hi All,

I would like to use ffmpeg as live HLS ABR packager and use /dev/shm to store playlist and chunks in RAMDISK.
But unfortunately I’ve noticed that ffmpeg constantly increase the memory used until it saturates it.

the script does basically this steps:

mkdir -p /dev/shm/live/test
cd /dev/shm/live/test
/usr/bin/ffmpeg -i udp://239.239.239.109:4000?fifo_size=1000000 -i udp:// 239.239.239.109:3000?fifo_size=1000000 -i udp:// 239.239.239.109:2000?fifo_size=1000000 -i udp:// 239.239.239.109:5000?fifo_size=1000000 -i udp:// 239.239.239.109:6000?fifo_size=1000000 -c copy -b:v:0 1400k -b:v:1 700k -b:v:2 300k -b:a:0 128k -b:a:1 64k -b:a:2 64k -b:v:3 2500k -b:v:4 3200k -b:a:3 128k -b:a:4 128k -map 0:v -map 0:a -map 1:v -map 1:a -map 2:v -map 2:a -map 3:v -map 3:a -map 4:v -map 4:a -f hls -master_pl_name index.m3u8 -var_stream_map v:0,a:0,name:SD v:1,a:1,name:SDh v:2,a:2,name:SDq v:3,a:3,name:HD v:4,a:4,name:FHD -hls_time 10 -hls_list_size 6 -hls_flags delete_segments -hls_segment_filename test-%v/chunk-%08d.ts test-%v/playlist.m3u8


Then, this is the result of the free command as soon I launch the script:
              total        used        free      shared  buff/cache   available
Mem:       32924468      670052    29991268        3788     2263148    31781984
Swap:       8388604           0     8388604


This is the same result after 4 minutes:
              total        used        free      shared  buff/cache   available
Mem:       32924468     1089804    29504504       70348     2330160    31295808
Swap:       8388604           0     8388604


This is after 7 minutes:
              total        used        free      shared  buff/cache   available
Mem:       32924468     1224096    29360344       78540     2340028    31153156
Swap:       8388604           0     8388604

This is after 10 minutes:
              total        used        free      shared  buff/cache   available
Mem:       32924468     1338492    29258888       64832     2327088    31052356
Swap:       8388604           0     8388604

Here after 14:
              total        used        free      shared  buff/cache   available
Mem:       32924468     1389880    29211896       60160     2322692    31005696
Swap:       8388604           0     8388604

Here after 30:
              total        used        free      shared  buff/cache   available
Mem:       32924468     1516036    28879424      111332     2529008    30825508
Swap:       8388604           0     8388604

Please bear in mind that as a matter of test, I’ve also tried to store the files in a SSD but the outcome was basically the same.

As you can see the memory usage constantly increases, and it ends up exhausting all the memory. The only way to release the memory in use is to kill the script and run it again.

Any suggestion on how to limit the memory usage? For now I’m killing each ffmpeg process when it reaches a specific amount of memory used, but obviously this interrupts the stream.

The executable is the one distributed with Ubuntu 20.04LTS, here its manifest:

ffmpeg version 4.2.2-1ubuntu1 Copyright (c) 2000-2019 the FFmpeg developers
  built with gcc 9 (Ubuntu 9.3.0-3ubuntu1)
  configuration: --prefix=/usr --extra-version=1ubuntu1 --toolchain=hardened --libdir=/usr/lib/x86_64-linux-gnu --incdir=/usr/include/x86_64-linux-gnu --arch=amd64 --enable-gpl --disable-stripping --enable-avresample --disable-filter=resample --enable-avisynth --enable-gnutls --enable-ladspa --enable-libaom --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libcdio --enable-libcodec2 --enable-libflite --enable-libfontconfig --enable-libfreetype --enable-libfribidi --enable-libgme --enable-libgsm --enable-libjack --enable-libmp3lame --enable-libmysofa --enable-libopenjpeg --enable-libopenmpt --enable-libopus --enable-libpulse --enable-librsvg --enable-librubberband --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libssh --enable-libtheora --enable-libtwolame --enable-libvidstab --enable-libvorbis --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx265 --enable-libxml2 --enable-libxvid --enable-libzmq --enable-libzvbi --enable-lv2 --enable-omx --enable-openal --enable-opencl --enable-opengl --enable-sdl2 --enable-libdc1394 --enable-libdrm --enable-libiec61883 --enable-nvenc --enable-chromaprint --enable-frei0r --enable-libx264 --enable-shared

And, finally, I’ve tried to run the same command line for 600 seconds with valgrind and it doesn’t seem to leak so much…. These are the results:

valgrind --leak-check=yes /usr/bin/ffmpeg -v quiet -i udp://239.5.208.109:4000?fifo_size=1000000 -i udp://239.5.208.109:3000?fifo_size=1000000 -i udp://239.5.208.109:2000?fifo_size=1000000 -i udp://239.5.208.109:5000?fifo_size=1000000 -i udp://239.5.208.109:6000?fifo_size=1000000 -c copy -b:v:0 1400k -b:v:1 700k -b:v:2 300k -b:a:0 128k -b:a:1 64k -b:a:2 64k -b:v:3 2500k -b:v:4 3200k -b:a:3 128k -b:a:4 128k -map 0:v -map 0:a -map 1:v -map 1:a -map 2:v -map 2:a -map 3:v -map 3:a -map 4:v -map 4:a -t 600 -f hls -master_pl_name master.m3u8 -var_stream_map "v:0,a:0,name:SD v:1,a:1,name:SDh v:2,a:2,name:SDq v:3,a:3,name:HD v:4,a:4,name:FHD" -hls_time 10 -hls_list_size 4 -hls_flags delete_segments -hls_segment_filename test-%v/chunk-%08d.ts test-%v/playlist.m3u8
==40833== Memcheck, a memory error detector
==40833== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.
==40833== Using Valgrind-3.15.0 and LibVEX; rerun with -h for copyright info
==40833== Command: /usr/bin/ffmpeg -v quiet -i udp://239.5.208.109:4000?fifo_size=1000000 -i udp://239.5.208.109:3000?fifo_size=1000000 -i udp://239.5.208.109:2000?fifo_size=1000000 -i udp://239.5.208.109:5000?fifo_size=1000000 -i udp://239.5.208.109:6000?fifo_size=1000000 -c copy -b:v:0 1400k -b:v:1 700k -b:v:2 300k -b:a:0 128k -b:a:1 64k -b:a:2 64k -b:v:3 2500k -b:v:4 3200k -b:a:3 128k -b:a:4 128k -map 0:v -map 0:a -map 1:v -map 1:a -map 2:v -map 2:a -map 3:v -map 3:a -map 4:v -map 4:a -t 600 -f hls -master_pl_name master.m3u8 -var_stream_map v:0,a:0,name:SD\ v:1,a:1,name:SDh\ v:2,a:2,name:SDq\ v:3,a:3,name:HD\ v:4,a:4,name:FHD -hls_time 10 -hls_list_size 4 -hls_flags delete_segments -hls_segment_filename test-%v/chunk-%08d.ts test-%v/playlist.m3u8
==40833==
==40833==
==40833== HEAP SUMMARY:
==40833==     in use at exit: 49,540 bytes in 253 blocks
==40833==   total heap usage: 3,694,888 allocs, 3,694,635 frees, 21,189,061,586 bytes allocated
==40833==
==40833== 16 bytes in 1 blocks are possibly lost in loss record 90 of 243
==40833==    at 0x483DD99: calloc (in /usr/lib/x86_64-linux-gnu/valgrind/vgpreload_memcheck-amd64-linux.so)
==40833==    by 0xA4F2D30: g_malloc0 (in /usr/lib/x86_64-linux-gnu/libglib-2.0.so.0.6400.2)
==40833==    by 0xA46FDBE: ??? (in /usr/lib/x86_64-linux-gnu/libgobject-2.0.so.0.6400.2)
==40833==    by 0xA473066: g_type_register_fundamental (in /usr/lib/x86_64-linux-gnu/libgobject-2.0.so.0.6400.2)
==40833==    by 0xA44EEDE: ??? (in /usr/lib/x86_64-linux-gnu/libgobject-2.0.so.0.6400.2)
==40833==    by 0xA4480B7: ??? (in /usr/lib/x86_64-linux-gnu/libgobject-2.0.so.0.6400.2)
==40833==    by 0x4011B89: call_init.part.0 (dl-init.c:72)
==40833==    by 0x4011C90: call_init (dl-init.c:30)
==40833==    by 0x4011C90: _dl_init (dl-init.c:119)
==40833==    by 0x4001139: ??? (in /usr/lib/x86_64-linux-gnu/ld-2.31.so)
==40833==    by 0x47: ???
==40833==    by 0x1FFF000472: ???
==40833==    by 0x1FFF000482: ???
==40833==
==40833== 16 bytes in 1 blocks are possibly lost in loss record 91 of 243
==40833==    at 0x483B723: malloc (in /usr/lib/x86_64-linux-gnu/valgrind/vgpreload_memcheck-amd64-linux.so)
==40833==    by 0x483E017: realloc (in /usr/lib/x86_64-linux-gnu/valgrind/vgpreload_memcheck-amd64-linux.so)
==40833==    by 0xA4F2D7F: g_realloc (in /usr/lib/x86_64-linux-gnu/libglib-2.0.so.0.6400.2)
==40833==    by 0xA46FD37: ??? (in /usr/lib/x86_64-linux-gnu/libgobject-2.0.so.0.6400.2)
==40833==    by 0xA473066: g_type_register_fundamental (in /usr/lib/x86_64-linux-gnu/libgobject-2.0.so.0.6400.2)
==40833==    by 0xA44EEDE: ??? (in /usr/lib/x86_64-linux-gnu/libgobject-2.0.so.0.6400.2)
==40833==    by 0xA4480B7: ??? (in /usr/lib/x86_64-linux-gnu/libgobject-2.0.so.0.6400.2)
==40833==    by 0x4011B89: call_init.part.0 (dl-init.c:72)
==40833==    by 0x4011C90: call_init (dl-init.c:30)
==40833==    by 0x4011C90: _dl_init (dl-init.c:119)
==40833==    by 0x4001139: ??? (in /usr/lib/x86_64-linux-gnu/ld-2.31.so)
==40833==    by 0x47: ???
==40833==    by 0x1FFF000472: ???
==40833==
==40833== 16 bytes in 1 blocks are possibly lost in loss record 92 of 243
==40833==    at 0x483DD99: calloc (in /usr/lib/x86_64-linux-gnu/valgrind/vgpreload_memcheck-amd64-linux.so)
==40833==    by 0xA4F2D30: g_malloc0 (in /usr/lib/x86_64-linux-gnu/libglib-2.0.so.0.6400.2)
==40833==    by 0xA46FDBE: ??? (in /usr/lib/x86_64-linux-gnu/libgobject-2.0.so.0.6400.2)
==40833==    by 0xA473066: g_type_register_fundamental (in /usr/lib/x86_64-linux-gnu/libgobject-2.0.so.0.6400.2)
==40833==    by 0xA44EF41: ??? (in /usr/lib/x86_64-linux-gnu/libgobject-2.0.so.0.6400.2)
==40833==    by 0xA4480B7: ??? (in /usr/lib/x86_64-linux-gnu/libgobject-2.0.so.0.6400.2)
==40833==    by 0x4011B89: call_init.part.0 (dl-init.c:72)
==40833==    by 0x4011C90: call_init (dl-init.c:30)
==40833==    by 0x4011C90: _dl_init (dl-init.c:119)
==40833==    by 0x4001139: ??? (in /usr/lib/x86_64-linux-gnu/ld-2.31.so)
==40833==    by 0x47: ???
==40833==    by 0x1FFF000472: ???
==40833==    by 0x1FFF000482: ???
==40833==
==40833== 16 bytes in 1 blocks are possibly lost in loss record 93 of 243
==40833==    at 0x483B723: malloc (in /usr/lib/x86_64-linux-gnu/valgrind/vgpreload_memcheck-amd64-linux.so)
==40833==    by 0x483E017: realloc (in /usr/lib/x86_64-linux-gnu/valgrind/vgpreload_memcheck-amd64-linux.so)
==40833==    by 0xA4F2D7F: g_realloc (in /usr/lib/x86_64-linux-gnu/libglib-2.0.so.0.6400.2)
==40833==    by 0xA46FD37: ??? (in /usr/lib/x86_64-linux-gnu/libgobject-2.0.so.0.6400.2)
==40833==    by 0xA473066: g_type_register_fundamental (in /usr/lib/x86_64-linux-gnu/libgobject-2.0.so.0.6400.2)
==40833==    by 0xA44EF41: ??? (in /usr/lib/x86_64-linux-gnu/libgobject-2.0.so.0.6400.2)
==40833==    by 0xA4480B7: ??? (in /usr/lib/x86_64-linux-gnu/libgobject-2.0.so.0.6400.2)
==40833==    by 0x4011B89: call_init.part.0 (dl-init.c:72)
==40833==    by 0x4011C90: call_init (dl-init.c:30)
==40833==    by 0x4011C90: _dl_init (dl-init.c:119)
==40833==    by 0x4001139: ??? (in /usr/lib/x86_64-linux-gnu/ld-2.31.so)
==40833==    by 0x47: ???
==40833==    by 0x1FFF000472: ???
==40833==
==40833== 16 bytes in 1 blocks are possibly lost in loss record 94 of 243
==40833==    at 0x483DD99: calloc (in /usr/lib/x86_64-linux-gnu/valgrind/vgpreload_memcheck-amd64-linux.so)
==40833==    by 0xA4F2D30: g_malloc0 (in /usr/lib/x86_64-linux-gnu/libglib-2.0.so.0.6400.2)
==40833==    by 0xA46FDBE: ??? (in /usr/lib/x86_64-linux-gnu/libgobject-2.0.so.0.6400.2)
==40833==    by 0xA473066: g_type_register_fundamental (in /usr/lib/x86_64-linux-gnu/libgobject-2.0.so.0.6400.2)
==40833==    by 0xA458FFF: ??? (in /usr/lib/x86_64-linux-gnu/libgobject-2.0.so.0.6400.2)
==40833==    by 0xA4480C1: ??? (in /usr/lib/x86_64-linux-gnu/libgobject-2.0.so.0.6400.2)
==40833==    by 0x4011B89: call_init.part.0 (dl-init.c:72)
==40833==    by 0x4011C90: call_init (dl-init.c:30)
==40833==    by 0x4011C90: _dl_init (dl-init.c:119)
==40833==    by 0x4001139: ??? (in /usr/lib/x86_64-linux-gnu/ld-2.31.so)
==40833==    by 0x47: ???
==40833==    by 0x1FFF000472: ???
==40833==    by 0x1FFF000482: ???
==40833==
==40833== 16 bytes in 1 blocks are possibly lost in loss record 95 of 243
==40833==    at 0x483B723: malloc (in /usr/lib/x86_64-linux-gnu/valgrind/vgpreload_memcheck-amd64-linux.so)
==40833==    by 0x483E017: realloc (in /usr/lib/x86_64-linux-gnu/valgrind/vgpreload_memcheck-amd64-linux.so)
==40833==    by 0xA4F2D7F: g_realloc (in /usr/lib/x86_64-linux-gnu/libglib-2.0.so.0.6400.2)
==40833==    by 0xA46FD37: ??? (in /usr/lib/x86_64-linux-gnu/libgobject-2.0.so.0.6400.2)
==40833==    by 0xA473066: g_type_register_fundamental (in /usr/lib/x86_64-linux-gnu/libgobject-2.0.so.0.6400.2)
==40833==    by 0xA458FFF: ??? (in /usr/lib/x86_64-linux-gnu/libgobject-2.0.so.0.6400.2)
==40833==    by 0xA4480C1: ??? (in /usr/lib/x86_64-linux-gnu/libgobject-2.0.so.0.6400.2)
==40833==    by 0x4011B89: call_init.part.0 (dl-init.c:72)
==40833==    by 0x4011C90: call_init (dl-init.c:30)
==40833==    by 0x4011C90: _dl_init (dl-init.c:119)
==40833==    by 0x4001139: ??? (in /usr/lib/x86_64-linux-gnu/ld-2.31.so)
==40833==    by 0x47: ???
==40833==    by 0x1FFF000472: ???
==40833==
==40833== 16 bytes in 1 blocks are possibly lost in loss record 96 of 243
==40833==    at 0x483DD99: calloc (in /usr/lib/x86_64-linux-gnu/valgrind/vgpreload_memcheck-amd64-linux.so)
==40833==    by 0xA4F2D30: g_malloc0 (in /usr/lib/x86_64-linux-gnu/libglib-2.0.so.0.6400.2)
==40833==    by 0xA46FDBE: ??? (in /usr/lib/x86_64-linux-gnu/libgobject-2.0.so.0.6400.2)
==40833==    by 0xA473066: g_type_register_fundamental (in /usr/lib/x86_64-linux-gnu/libgobject-2.0.so.0.6400.2)
==40833==    by 0xA453CC3: ??? (in /usr/lib/x86_64-linux-gnu/libgobject-2.0.so.0.6400.2)
==40833==    by 0xA4480C6: ??? (in /usr/lib/x86_64-linux-gnu/libgobject-2.0.so.0.6400.2)
==40833==    by 0x4011B89: call_init.part.0 (dl-init.c:72)
==40833==    by 0x4011C90: call_init (dl-init.c:30)
==40833==    by 0x4011C90: _dl_init (dl-init.c:119)
==40833==    by 0x4001139: ??? (in /usr/lib/x86_64-linux-gnu/ld-2.31.so)
==40833==    by 0x47: ???
==40833==    by 0x1FFF000472: ???
==40833==    by 0x1FFF000482: ???
==40833==
==40833== 16 bytes in 1 blocks are possibly lost in loss record 97 of 243
==40833==    at 0x483B723: malloc (in /usr/lib/x86_64-linux-gnu/valgrind/vgpreload_memcheck-amd64-linux.so)
==40833==    by 0x483E017: realloc (in /usr/lib/x86_64-linux-gnu/valgrind/vgpreload_memcheck-amd64-linux.so)
==40833==    by 0xA4F2D7F: g_realloc (in /usr/lib/x86_64-linux-gnu/libglib-2.0.so.0.6400.2)
==40833==    by 0xA46FD37: ??? (in /usr/lib/x86_64-linux-gnu/libgobject-2.0.so.0.6400.2)
==40833==    by 0xA473066: g_type_register_fundamental (in /usr/lib/x86_64-linux-gnu/libgobject-2.0.so.0.6400.2)
==40833==    by 0xA453CC3: ??? (in /usr/lib/x86_64-linux-gnu/libgobject-2.0.so.0.6400.2)
==40833==    by 0xA4480C6: ??? (in /usr/lib/x86_64-linux-gnu/libgobject-2.0.so.0.6400.2)
==40833==    by 0x4011B89: call_init.part.0 (dl-init.c:72)
==40833==    by 0x4011C90: call_init (dl-init.c:30)
==40833==    by 0x4011C90: _dl_init (dl-init.c:119)
==40833==    by 0x4001139: ??? (in /usr/lib/x86_64-linux-gnu/ld-2.31.so)
==40833==    by 0x47: ???
==40833==    by 0x1FFF000472: ???
==40833==
==40833== 96 bytes in 1 blocks are possibly lost in loss record 216 of 243
==40833==    at 0x483DD99: calloc (in /usr/lib/x86_64-linux-gnu/valgrind/vgpreload_memcheck-amd64-linux.so)
==40833==    by 0xA4F2D30: g_malloc0 (in /usr/lib/x86_64-linux-gnu/libglib-2.0.so.0.6400.2)
==40833==    by 0xA46F0C7: ??? (in /usr/lib/x86_64-linux-gnu/libgobject-2.0.so.0.6400.2)
==40833==    by 0xA46F26A: ??? (in /usr/lib/x86_64-linux-gnu/libgobject-2.0.so.0.6400.2)
==40833==    by 0xA447FDE: ??? (in /usr/lib/x86_64-linux-gnu/libgobject-2.0.so.0.6400.2)
==40833==    by 0x4011B89: call_init.part.0 (dl-init.c:72)
==40833==    by 0x4011C90: call_init (dl-init.c:30)
==40833==    by 0x4011C90: _dl_init (dl-init.c:119)
==40833==    by 0x4001139: ??? (in /usr/lib/x86_64-linux-gnu/ld-2.31.so)
==40833==    by 0x47: ???
==40833==    by 0x1FFF000472: ???
==40833==    by 0x1FFF000482: ???
==40833==    by 0x1FFF000485: ???
==40833==
==40833== 96 bytes in 1 blocks are possibly lost in loss record 217 of 243
==40833==    at 0x483DD99: calloc (in /usr/lib/x86_64-linux-gnu/valgrind/vgpreload_memcheck-amd64-linux.so)
==40833==    by 0xA4F2D30: g_malloc0 (in /usr/lib/x86_64-linux-gnu/libglib-2.0.so.0.6400.2)
==40833==    by 0xA46F0C7: ??? (in /usr/lib/x86_64-linux-gnu/libgobject-2.0.so.0.6400.2)
==40833==    by 0xA46F26A: ??? (in /usr/lib/x86_64-linux-gnu/libgobject-2.0.so.0.6400.2)
==40833==    by 0xA473058: g_type_register_fundamental (in /usr/lib/x86_64-linux-gnu/libgobject-2.0.so.0.6400.2)
==40833==    by 0xA44EEDE: ??? (in /usr/lib/x86_64-linux-gnu/libgobject-2.0.so.0.6400.2)
==40833==    by 0xA4480B7: ??? (in /usr/lib/x86_64-linux-gnu/libgobject-2.0.so.0.6400.2)
==40833==    by 0x4011B89: call_init.part.0 (dl-init.c:72)
==40833==    by 0x4011C90: call_init (dl-init.c:30)
==40833==    by 0x4011C90: _dl_init (dl-init.c:119)
==40833==    by 0x4001139: ??? (in /usr/lib/x86_64-linux-gnu/ld-2.31.so)
==40833==    by 0x47: ???
==40833==    by 0x1FFF000472: ???
==40833==
==40833== 96 bytes in 1 blocks are possibly lost in loss record 218 of 243
==40833==    at 0x483DD99: calloc (in /usr/lib/x86_64-linux-gnu/valgrind/vgpreload_memcheck-amd64-linux.so)
==40833==    by 0xA4F2D30: g_malloc0 (in /usr/lib/x86_64-linux-gnu/libglib-2.0.so.0.6400.2)
==40833==    by 0xA46F0C7: ??? (in /usr/lib/x86_64-linux-gnu/libgobject-2.0.so.0.6400.2)
==40833==    by 0xA46F26A: ??? (in /usr/lib/x86_64-linux-gnu/libgobject-2.0.so.0.6400.2)
==40833==    by 0xA473058: g_type_register_fundamental (in /usr/lib/x86_64-linux-gnu/libgobject-2.0.so.0.6400.2)
==40833==    by 0xA44EF41: ??? (in /usr/lib/x86_64-linux-gnu/libgobject-2.0.so.0.6400.2)
==40833==    by 0xA4480B7: ??? (in /usr/lib/x86_64-linux-gnu/libgobject-2.0.so.0.6400.2)
==40833==    by 0x4011B89: call_init.part.0 (dl-init.c:72)
==40833==    by 0x4011C90: call_init (dl-init.c:30)
==40833==    by 0x4011C90: _dl_init (dl-init.c:119)
==40833==    by 0x4001139: ??? (in /usr/lib/x86_64-linux-gnu/ld-2.31.so)
==40833==    by 0x47: ???
==40833==    by 0x1FFF000472: ???
==40833==
==40833== 96 bytes in 1 blocks are possibly lost in loss record 219 of 243
==40833==    at 0x483DD99: calloc (in /usr/lib/x86_64-linux-gnu/valgrind/vgpreload_memcheck-amd64-linux.so)
==40833==    by 0xA4F2D30: g_malloc0 (in /usr/lib/x86_64-linux-gnu/libglib-2.0.so.0.6400.2)
==40833==    by 0xA46F0C7: ??? (in /usr/lib/x86_64-linux-gnu/libgobject-2.0.so.0.6400.2)
==40833==    by 0xA46F26A: ??? (in /usr/lib/x86_64-linux-gnu/libgobject-2.0.so.0.6400.2)
==40833==    by 0xA473058: g_type_register_fundamental (in /usr/lib/x86_64-linux-gnu/libgobject-2.0.so.0.6400.2)
==40833==    by 0xA458FFF: ??? (in /usr/lib/x86_64-linux-gnu/libgobject-2.0.so.0.6400.2)
==40833==    by 0xA4480C1: ??? (in /usr/lib/x86_64-linux-gnu/libgobject-2.0.so.0.6400.2)
==40833==    by 0x4011B89: call_init.part.0 (dl-init.c:72)
==40833==    by 0x4011C90: call_init (dl-init.c:30)
==40833==    by 0x4011C90: _dl_init (dl-init.c:119)
==40833==    by 0x4001139: ??? (in /usr/lib/x86_64-linux-gnu/ld-2.31.so)
==40833==    by 0x47: ???
==40833==    by 0x1FFF000472: ???
==40833==
==40833== 96 bytes in 1 blocks are possibly lost in loss record 220 of 243
==40833==    at 0x483DD99: calloc (in /usr/lib/x86_64-linux-gnu/valgrind/vgpreload_memcheck-amd64-linux.so)
==40833==    by 0xA4F2D30: g_malloc0 (in /usr/lib/x86_64-linux-gnu/libglib-2.0.so.0.6400.2)
==40833==    by 0xA46F0C7: ??? (in /usr/lib/x86_64-linux-gnu/libgobject-2.0.so.0.6400.2)
==40833==    by 0xA46F26A: ??? (in /usr/lib/x86_64-linux-gnu/libgobject-2.0.so.0.6400.2)
==40833==    by 0xA473058: g_type_register_fundamental (in /usr/lib/x86_64-linux-gnu/libgobject-2.0.so.0.6400.2)
==40833==    by 0xA453CC3: ??? (in /usr/lib/x86_64-linux-gnu/libgobject-2.0.so.0.6400.2)
==40833==    by 0xA4480C6: ??? (in /usr/lib/x86_64-linux-gnu/libgobject-2.0.so.0.6400.2)
==40833==    by 0x4011B89: call_init.part.0 (dl-init.c:72)
==40833==    by 0x4011C90: call_init (dl-init.c:30)
==40833==    by 0x4011C90: _dl_init (dl-init.c:119)
==40833==    by 0x4001139: ??? (in /usr/lib/x86_64-linux-gnu/ld-2.31.so)
==40833==    by 0x47: ???
==40833==    by 0x1FFF000472: ???
==40833==
==40833== 132 bytes in 1 blocks are possibly lost in loss record 222 of 243
==40833==    at 0x483DD99: calloc (in /usr/lib/x86_64-linux-gnu/valgrind/vgpreload_memcheck-amd64-linux.so)
==40833==    by 0xA4F2D30: g_malloc0 (in /usr/lib/x86_64-linux-gnu/libglib-2.0.so.0.6400.2)
==40833==    by 0xA4700D4: ??? (in /usr/lib/x86_64-linux-gnu/libgobject-2.0.so.0.6400.2)
==40833==    by 0xA4730E9: g_type_register_fundamental (in /usr/lib/x86_64-linux-gnu/libgobject-2.0.so.0.6400.2)
==40833==    by 0xA44EEDE: ??? (in /usr/lib/x86_64-linux-gnu/libgobject-2.0.so.0.6400.2)
==40833==    by 0xA4480B7: ??? (in /usr/lib/x86_64-linux-gnu/libgobject-2.0.so.0.6400.2)
==40833==    by 0x4011B89: call_init.part.0 (dl-init.c:72)
==40833==    by 0x4011C90: call_init (dl-init.c:30)
==40833==    by 0x4011C90: _dl_init (dl-init.c:119)
==40833==    by 0x4001139: ??? (in /usr/lib/x86_64-linux-gnu/ld-2.31.so)
==40833==    by 0x47: ???
==40833==    by 0x1FFF000472: ???
==40833==    by 0x1FFF000482: ???
==40833==
==40833== 132 bytes in 1 blocks are possibly lost in loss record 223 of 243
==40833==    at 0x483DD99: calloc (in /usr/lib/x86_64-linux-gnu/valgrind/vgpreload_memcheck-amd64-linux.so)
==40833==    by 0xA4F2D30: g_malloc0 (in /usr/lib/x86_64-linux-gnu/libglib-2.0.so.0.6400.2)
==40833==    by 0xA4700D4: ??? (in /usr/lib/x86_64-linux-gnu/libgobject-2.0.so.0.6400.2)
==40833==    by 0xA4730E9: g_type_register_fundamental (in /usr/lib/x86_64-linux-gnu/libgobject-2.0.so.0.6400.2)
==40833==    by 0xA44EF41: ??? (in /usr/lib/x86_64-linux-gnu/libgobject-2.0.so.0.6400.2)
==40833==    by 0xA4480B7: ??? (in /usr/lib/x86_64-linux-gnu/libgobject-2.0.so.0.6400.2)
==40833==    by 0x4011B89: call_init.part.0 (dl-init.c:72)
==40833==    by 0x4011C90: call_init (dl-init.c:30)
==40833==    by 0x4011C90: _dl_init (dl-init.c:119)
==40833==    by 0x4001139: ??? (in /usr/lib/x86_64-linux-gnu/ld-2.31.so)
==40833==    by 0x47: ???
==40833==    by 0x1FFF000472: ???
==40833==    by 0x1FFF000482: ???
==40833==
==40833== 148 bytes in 1 blocks are possibly lost in loss record 225 of 243
==40833==    at 0x483DD99: calloc (in /usr/lib/x86_64-linux-gnu/valgrind/vgpreload_memcheck-amd64-linux.so)
==40833==    by 0xA4F2D30: g_malloc0 (in /usr/lib/x86_64-linux-gnu/libglib-2.0.so.0.6400.2)
==40833==    by 0xA46FEE8: ??? (in /usr/lib/x86_64-linux-gnu/libgobject-2.0.so.0.6400.2)
==40833==    by 0xA4730E9: g_type_register_fundamental (in /usr/lib/x86_64-linux-gnu/libgobject-2.0.so.0.6400.2)
==40833==    by 0xA458FFF: ??? (in /usr/lib/x86_64-linux-gnu/libgobject-2.0.so.0.6400.2)
==40833==    by 0xA4480C1: ??? (in /usr/lib/x86_64-linux-gnu/libgobject-2.0.so.0.6400.2)
==40833==    by 0x4011B89: call_init.part.0 (dl-init.c:72)
==40833==    by 0x4011C90: call_init (dl-init.c:30)
==40833==    by 0x4011C90: _dl_init (dl-init.c:119)
==40833==    by 0x4001139: ??? (in /usr/lib/x86_64-linux-gnu/ld-2.31.so)
==40833==    by 0x47: ???
==40833==    by 0x1FFF000472: ???
==40833==    by 0x1FFF000482: ???
==40833==
==40833== 148 bytes in 1 blocks are possibly lost in loss record 226 of 243
==40833==    at 0x483DD99: calloc (in /usr/lib/x86_64-linux-gnu/valgrind/vgpreload_memcheck-amd64-linux.so)
==40833==    by 0xA4F2D30: g_malloc0 (in /usr/lib/x86_64-linux-gnu/libglib-2.0.so.0.6400.2)
==40833==    by 0xA46FEE8: ??? (in /usr/lib/x86_64-linux-gnu/libgobject-2.0.so.0.6400.2)
==40833==    by 0xA4730E9: g_type_register_fundamental (in /usr/lib/x86_64-linux-gnu/libgobject-2.0.so.0.6400.2)
==40833==    by 0xA453CC3: ??? (in /usr/lib/x86_64-linux-gnu/libgobject-2.0.so.0.6400.2)
==40833==    by 0xA4480C6: ??? (in /usr/lib/x86_64-linux-gnu/libgobject-2.0.so.0.6400.2)
==40833==    by 0x4011B89: call_init.part.0 (dl-init.c:72)
==40833==    by 0x4011C90: call_init (dl-init.c:30)
==40833==    by 0x4011C90: _dl_init (dl-init.c:119)
==40833==    by 0x4001139: ??? (in /usr/lib/x86_64-linux-gnu/ld-2.31.so)
==40833==    by 0x47: ???
==40833==    by 0x1FFF000472: ???
==40833==    by 0x1FFF000482: ???
==40833==
==40833== 184 bytes in 1 blocks are possibly lost in loss record 228 of 243
==40833==    at 0x483DFAF: realloc (in /usr/lib/x86_64-linux-gnu/valgrind/vgpreload_memcheck-amd64-linux.so)
==40833==    by 0xA4F2D7F: g_realloc (in /usr/lib/x86_64-linux-gnu/libglib-2.0.so.0.6400.2)
==40833==    by 0xA46F043: ??? (in /usr/lib/x86_64-linux-gnu/libgobject-2.0.so.0.6400.2)
==40833==    by 0xA4732B4: g_type_register_static (in /usr/lib/x86_64-linux-gnu/libgobject-2.0.so.0.6400.2)
==40833==    by 0xA45AD12: g_param_type_register_static (in /usr/lib/x86_64-linux-gnu/libgobject-2.0.so.0.6400.2)
==40833==    by 0xA45D7EA: ??? (in /usr/lib/x86_64-linux-gnu/libgobject-2.0.so.0.6400.2)
==40833==    by 0xA4480CB: ??? (in /usr/lib/x86_64-linux-gnu/libgobject-2.0.so.0.6400.2)
==40833==    by 0x4011B89: call_init.part.0 (dl-init.c:72)
==40833==    by 0x4011C90: call_init (dl-init.c:30)
==40833==    by 0x4011C90: _dl_init (dl-init.c:119)
==40833==    by 0x4001139: ??? (in /usr/lib/x86_64-linux-gnu/ld-2.31.so)
==40833==    by 0x47: ???
==40833==    by 0x1FFF000472: ???
==40833==
==40833== LEAK SUMMARY:
==40833==    definitely lost: 0 bytes in 0 blocks
==40833==    indirectly lost: 0 bytes in 0 blocks
==40833==      possibly lost: 1,352 bytes in 18 blocks
==40833==    still reachable: 48,188 bytes in 235 blocks
==40833==                       of which reachable via heuristic:
==40833==                         newarray           : 1,536 bytes in 16 blocks
==40833==         suppressed: 0 bytes in 0 blocks
==40833== Reachable blocks (those to which a pointer was found) are not shown.
==40833== To see them, rerun with: --leak-check=full --show-leak-kinds=all
==40833==
==40833== For lists of detected and suppressed errors, rerun with: -s
==40833== ERROR SUMMARY: 18 errors from 18 contexts (suppressed: 0 from 0)
root@live-packager-pool01-main:/dev/shm/live/test#


Any idea?

Thanks in advance,

Alex

_______________________________________________
ffmpeg-user mailing list
[hidden email]
https://ffmpeg.org/mailman/listinfo/ffmpeg-user

To unsubscribe, visit link above, or email
[hidden email] with subject "unsubscribe".
Reply | Threaded
Open this post in threaded view
|

Re: FFMPEG endlessly increases memory usage over time with HLS packaging.

Carl Eugen Hoyos-2


> Am 25.06.2020 um 02:38 schrieb Alessandro Molon <[hidden email]>:
>
> I would like to use ffmpeg as live HLS ABR packager and use /dev/shm to store playlist and chunks in RAMDISK.
> But unfortunately I’ve noticed that ffmpeg constantly increase the memory used until it saturates it.

No.
At least both your „free“ and „valgrind“ output indicate the opposite (as does the word „saturate“ above).

Please read up about the columns „free“, „used“ and „available“ of free console output.
https://www.linuxatemyram.com/

Carl Eugen
_______________________________________________
ffmpeg-user mailing list
[hidden email]
https://ffmpeg.org/mailman/listinfo/ffmpeg-user

To unsubscribe, visit link above, or email
[hidden email] with subject "unsubscribe".
Reply | Threaded
Open this post in threaded view
|

Re: FFMPEG endlessly increases memory usage over time with HLS packaging.

alexdjca
Apparently you think that the constantly increasing amount of "Used" memory in my case falls in the case described as:

Memory that is: used, but can be made available
You’d call it: Free (or Available)
Linux calls it: Used (and Available)

Let's see what happens if I run 20 processes for few days...

Cheers,
Alex


-----Original Message-----
From: ffmpeg-user <[hidden email]> On Behalf Of Carl Eugen Hoyos
Sent: 25 June 2020 07:28
To: FFmpeg user questions <[hidden email]>
Subject: Re: [FFmpeg-user] FFMPEG endlessly increases memory usage over time with HLS packaging.



> Am 25.06.2020 um 02:38 schrieb Alessandro Molon <[hidden email]>:
>
> I would like to use ffmpeg as live HLS ABR packager and use /dev/shm to store playlist and chunks in RAMDISK.
> But unfortunately I’ve noticed that ffmpeg constantly increase the memory used until it saturates it.

No.
At least both your „free“ and „valgrind“ output indicate the opposite (as does the word „saturate“ above).

Please read up about the columns „free“, „used“ and „available“ of free console output.
https://www.linuxatemyram.com/

Carl Eugen
_______________________________________________
ffmpeg-user mailing list
[hidden email]
https://ffmpeg.org/mailman/listinfo/ffmpeg-user

To unsubscribe, visit link above, or email [hidden email] with subject "unsubscribe".
_______________________________________________
ffmpeg-user mailing list
[hidden email]
https://ffmpeg.org/mailman/listinfo/ffmpeg-user

To unsubscribe, visit link above, or email
[hidden email] with subject "unsubscribe".
Reply | Threaded
Open this post in threaded view
|

Re: FFMPEG endlessly increases memory usage over time with HLS packaging.

Carl Eugen Hoyos-2
Am Do., 25. Juni 2020 um 20:39 Uhr schrieb Alessandro Molon
<[hidden email]>:

> Let's see what happens if I run 20 processes for few days...

On most hardware, you cannot run 20 (real-life) FFmpeg
processes for a few days but if there is constantly increasing
memory footprint or a memory leak, one process should
be sufficient to show it.

Please find out what top-posting means and avoid it here,
Carl Eugen
_______________________________________________
ffmpeg-user mailing list
[hidden email]
https://ffmpeg.org/mailman/listinfo/ffmpeg-user

To unsubscribe, visit link above, or email
[hidden email] with subject "unsubscribe".
Reply | Threaded
Open this post in threaded view
|

Re: FFMPEG endlessly increases memory usage over time with HLS packaging.

alexdjca
Actually I have several machines in production (dual Xeon 5620 - 32GB - Nvidia P2000) that are running 14 ffmpeg process each, all decoding Full HD mpegts and transcoding them in 4 different profiles each (HD/SD/SDh/SDq) and to be honest they work like a sharm and are also very stable.

Based on this experience I actually expected, just to repacketize pre-encoded content in HLS, to run at least 40/45 (even more) process per machine....

Let me go ahead with my experiments, I'll be more than happy to produce the results.

Alex

-----Original Message-----
From: ffmpeg-user <[hidden email]> On Behalf Of Carl Eugen Hoyos
Sent: 25 June 2020 22:19
To: FFmpeg user questions <[hidden email]>
Subject: Re: [FFmpeg-user] FFMPEG endlessly increases memory usage over time with HLS packaging.

Am Do., 25. Juni 2020 um 20:39 Uhr schrieb Alessandro Molon
<[hidden email]>:

> Let's see what happens if I run 20 processes for few days...

On most hardware, you cannot run 20 (real-life) FFmpeg processes for a few days but if there is constantly increasing memory footprint or a memory leak, one process should be sufficient to show it.

Please find out what top-posting means and avoid it here, Carl Eugen _______________________________________________
ffmpeg-user mailing list
[hidden email]
https://ffmpeg.org/mailman/listinfo/ffmpeg-user

To unsubscribe, visit link above, or email [hidden email] with subject "unsubscribe".
_______________________________________________
ffmpeg-user mailing list
[hidden email]
https://ffmpeg.org/mailman/listinfo/ffmpeg-user

To unsubscribe, visit link above, or email
[hidden email] with subject "unsubscribe".