Command being killed in perceptual hashing / ffmpeg in general

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

Command being killed in perceptual hashing / ffmpeg in general

Aoife Fitzmaurice
Hey everyone,

I'm wondering does anyone have any experience with their command being "killed" while using perceptual hashing, or ffmpeg in general?
I am using a Mac Pro with OSX El Capitan.

I ran the following input: $ ffmpeg -i  DAY\ 1/Card\ 2/B0011888.mov  -i  DAY\ 1/Card\ 1/B0011888U01.mov  -filter_complex signature=detectmode=full:nb_inputs=2 -f null -

and after a while the process began to slow down, before eventually being "killed." We also tried running it and excluding audio:

$ ffmpeg -i  DAY\ 1/Card\ 2/B0011888.mov  -i / /DAY\ 1/Card\ 1/B0011888U01.mov -an  -filter_complex signature=detectmode=full:nb_inputs=2 -f null -

to no avail.

I've used the above command on smaller files and there was no problem. I'll post the output below.

Thanks in advance!

Aoife


ifi-mac-pro:~ admin$ ffmpeg - I DAY\ 1/Card\ 2/B0011888.mov  -i  DAY\ 1/Card\ 1/B0011888U01.mov -an  -filter_complex signature=detectmode=full:nb_inputs=2 -f null -

ffmpeg version 3.3.3 Copyright (c) 2000-2017 the FFmpeg developers

  built with Apple LLVM version 7.3.0 (clang-703.0.31)

  configuration: --prefix=/usr/local/Cellar/ffmpeg/3.3.3 --enable-shared --enable-pthreads --enable-gpl --enable-version3 --enable-hardcoded-tables --enable-avresample --cc=clang --host-cflags= --host-ldflags= --enable-ffplay --enable-libass --enable-libfreetype --enable-libmp3lame --enable-libtesseract --enable-libx264 --enable-libxvid --enable-opencl --enable-videotoolbox --disable-lzma --enable-libopenjpeg --disable-decoder=jpeg2000 --extra-cflags=-I/usr/local/Cellar/openjpeg/2.2.0/include/openjpeg-2.2 --enable-vda

  libavutil      55. 58.100 / 55. 58.100

  libavcodec     57. 89.100 / 57. 89.100

  libavformat    57. 71.100 / 57. 71.100

  libavdevice    57.  6.100 / 57.  6.100

  libavfilter     6. 82.100 /  6. 82.100

  libavresample   3.  5.  0 /  3.  5.  0

  libswscale      4.  6.100 /  4.  6.100

  libswresample   2.  7.100 /  2.  7.100

  libpostproc    54.  5.100 / 54.  5.100

Guessed Channel Layout for Input Stream #0.1 : mono

Guessed Channel Layout for Input Stream #0.2 : mono

Input #0, mov,mp4,m4a,3gp,3g2,mj2, from ' /DAY 1/Card 2/B0011888.mov':

  Metadata:

    creation_time   : 2011-05-23T14:20:19.000000Z

    com.apple.quicktime.information: 060a2b340101010501010d4313000000f04922a3005705ca0800460202290214

    com.sony.bprl.mxf.nrtmetadata: <?xml version="1.0" encoding="UTF-8" standalone="yes"?><NonRealTimeMeta xmlns="urn:schemas-professionalDisc:nonRealTimeMeta:ver.1.20" xmlns:lib="urn:schemas-professionalDisc:lib:ver.1.20" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" lastUpdate="2

    com.sony.bprl.mxf.umid: 060a2b340101010501010d4313000000f04922a3005705ca0800460202290214

    org.smpte.mxf.package.material.packageid: 0x060a2b340101010501010d4313000000f04922a3005705ca0800460202290214

    com.sony.professionaldisc.nonrealtimemeta.lastupdate: 2011-05-19 16:20:51 +0000

    com.sony.bprl.xdcamtransfer.version: 2.12.0.631

    com.apple.proapps.reel: Card 2

    com.apple.proapps.clipID: S060A2B340101010501010D4313000000F04922A3005705CA0800460202290214

    original_format : MPEG HD

    original_format-eng: MPEG HD

    comment         : 060a2b340101010501010d4313000000f04922a3005705ca0800460202290214

    comment-eng     : 060a2b340101010501010d4313000000f04922a3005705ca0800460202290214

    timecode        : 05:18:16:20

  Duration: 01:03:13.08, start: 0.000000, bitrate: 36492 kb/s

    Stream #0:0(eng): Video: mpeg2video (Main) (xdve / 0x65766478), yuv420p(tv, bt709, progressive), 1920x1080 [SAR 1:1 DAR 16:9], 34938 kb/s, 25 fps, 25 tbr, 2500 tbn, 50 tbc (default)

    Metadata:

      creation_time   : 2011-05-23T14:20:19.000000Z

      handler_name    : Apple Alias Data Handler

      encoder         : XDCAM EX 1080p25 (35 Mb/s VBR)

    Stream #0:1(eng): Audio: pcm_s16be (lpcm / 0x6D63706C), 48000 Hz, mono, s16, 768 kb/s (default)

    Metadata:

      creation_time   : 2011-05-23T14:20:19.000000Z

      handler_name    : Apple Alias Data Handler

    Stream #0:2(eng): Audio: pcm_s16be (lpcm / 0x6D63706C), 48000 Hz, mono, s16, 768 kb/s (default)

    Metadata:

      creation_time   : 2011-05-23T14:20:19.000000Z

      handler_name    : Apple Alias Data Handler

    Stream #0:3(eng): Data: none (tmcd / 0x64636D74), 0 kb/s (default)

    Metadata:

      creation_time   : 2011-05-23T14:20:19.000000Z

      handler_name    : Apple Alias Data Handler

      reel_name       : Card 2

      timecode        : 05:18:16:20

Guessed Channel Layout for Input Stream #1.1 : mono

Guessed Channel Layout for Input Stream #1.2 : mono

Input #1, mov,mp4,m4a,3gp,3g2,mj2, from '/DAY 1/Card 1/B0011888U01.mov':

  Metadata:

    creation_time   : 2011-05-23T14:20:08.000000Z

    com.apple.quicktime.information: 060a2b340101010501010d4313000000f04922a3005705ca0800460202290214

    com.sony.bprl.mxf.nrtmetadata: <?xml version="1.0" encoding="UTF-8" standalone="yes"?><NonRealTimeMeta xmlns="urn:schemas-professionalDisc:nonRealTimeMeta:ver.1.20" xmlns:lib="urn:schemas-professionalDisc:lib:ver.1.20" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" lastUpdate="2

    com.sony.bprl.mxf.umid: 060a2b340101010501010d4313000000f04922a3005705ca0800460202290214

    org.smpte.mxf.package.material.packageid: 0x060a2b340101010501010d4313000000f04922a3005705ca0800460202290214

    com.sony.professionaldisc.nonrealtimemeta.lastupdate: 2011-05-19 15:45:45 +0000

    com.sony.bprl.xdcamtransfer.version: 2.12.0.631

    com.apple.proapps.reel: Card 1

    com.apple.proapps.clipID: S060A2B340101010501010D4313000000F04922A3005705CA0800460202290214

    original_format : MPEG HD

    original_format-eng: MPEG HD

    comment         : 060a2b340101010501010d4313000000f04922a3005705ca0800460202290214

    comment-eng     : 060a2b340101010501010d4313000000f04922a3005705ca0800460202290214

    timecode        : 05:18:16:20

  Duration: 00:28:06.24, start: 0.000000, bitrate: 36460 kb/s

    Stream #1:0(eng): Video: mpeg2video (Main) (xdve / 0x65766478), yuv420p(tv, bt709, progressive), 1920x1080 [SAR 1:1 DAR 16:9], 34906 kb/s, 25 fps, 25 tbr, 2500 tbn, 50 tbc (default)

    Metadata:

      creation_time   : 2011-05-23T14:20:08.000000Z

      handler_name    : Apple Alias Data Handler

      encoder         : XDCAM EX 1080p25 (35 Mb/s VBR)

    Stream #1:1(eng): Audio: pcm_s16be (lpcm / 0x6D63706C), 48000 Hz, mono, s16, 768 kb/s (default)

    Metadata:

      creation_time   : 2011-05-23T14:20:08.000000Z

      handler_name    : Apple Alias Data Handler

    Stream #1:2(eng): Audio: pcm_s16be (lpcm / 0x6D63706C), 48000 Hz, mono, s16, 768 kb/s (default)

    Metadata:

      creation_time   : 2011-05-23T14:20:08.000000Z

      handler_name    : Apple Alias Data Handler

    Stream #1:3(eng): Data: none (tmcd / 0x64636D74), 0 kb/s (default)

    Metadata:

      creation_time   : 2011-05-23T14:20:08.000000Z

      handler_name    : Apple Alias Data Handler

      reel_name       : Card 1

      timecode        : 05:18:16:20

Stream mapping:

  Stream #0:0 (mpeg2video) -> signature:in0

  Stream #1:0 (mpeg2video) -> signature:in1

  signature -> Stream #0:0 (wrapped_avframe)

Press [q] to stop, [?] for help

Output #0, null, to 'pipe:':

  Metadata:

    timecode        : 05:18:16:20

    com.apple.quicktime.information: 060a2b340101010501010d4313000000f04922a3005705ca0800460202290214

    com.sony.bprl.mxf.nrtmetadata: <?xml version="1.0" encoding="UTF-8" standalone="yes"?><NonRealTimeMeta xmlns="urn:schemas-professionalDisc:nonRealTimeMeta:ver.1.20" xmlns:lib="urn:schemas-professionalDisc:lib:ver.1.20" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" lastUpdate="2

    com.sony.bprl.mxf.umid: 060a2b340101010501010d4313000000f04922a3005705ca0800460202290214

    org.smpte.mxf.package.material.packageid: 0x060a2b340101010501010d4313000000f04922a3005705ca0800460202290214

    com.sony.professionaldisc.nonrealtimemeta.lastupdate: 2011-05-19 16:20:51 +0000

    com.sony.bprl.xdcamtransfer.version: 2.12.0.631

    com.apple.proapps.reel: Card 2

    com.apple.proapps.clipID: S060A2B340101010501010D4313000000F04922A3005705CA0800460202290214

    original_format : MPEG HD

    original_format-eng: MPEG HD

    comment         : 060a2b340101010501010d4313000000f04922a3005705ca0800460202290214

    comment-eng     : 060a2b340101010501010d4313000000f04922a3005705ca0800460202290214

    encoder         : Lavf57.71.100

    Stream #0:0: Video: wrapped_avframe, yuv420p, 1920x1080 [SAR 1:1 DAR 16:9], q=2-31, 200 kb/s, 25 fps, 25 tbn, 25 tbc (default)

    Metadata:

      encoder         : Lavc57.89.100 wrapped_avframe

Killed: 925 fps=117 q=-0.0 size=N/A time=01:03:13.00 bitrate=N/A speed=4.67x

ifi-mac-pro:~ admin$





------
This email has been scanned for spam and malware by The Email Laundry.


_______________________________________________
ffmpeg-user mailing list
[hidden email]
http://ffmpeg.org/mailman/listinfo/ffmpeg-user

To unsubscribe, visit link above, or email
[hidden email] with subject "unsubscribe".
Reply | Threaded
Open this post in threaded view
|

Re: Command being killed in perceptual hashing / ffmpeg in general

Carl Eugen Hoyos-2
2017-10-04 18:49 GMT+02:00 Aoife Fitzmaurice <[hidden email]>:

>   Duration: 01:03:13.08, start: 0.000000, bitrate: 36492 kb/s

Does it also crash for a shorter file or a file with (much)
smaller dimensions?

Does it crash because of oom, or for other reasons?

Generally, a backtrace, disassembly and register dump are
required for crash reports (esp. if they are not easily
reproducible).

Carl Eugen
_______________________________________________
ffmpeg-user mailing list
[hidden email]
http://ffmpeg.org/mailman/listinfo/ffmpeg-user

To unsubscribe, visit link above, or email
[hidden email] with subject "unsubscribe".
Reply | Threaded
Open this post in threaded view
|

Re: Command being killed in perceptual hashing / ffmpeg in general

Aoife Fitzmaurice

>Generally, a backtrace, disassembly and register dump are required for crash reports (esp. if they are not easily reproducible).

_______________________________________________

Hi Carl,

Thanks for your response! I changed the dimensions to 320x240 and it worked as expected.  It may just be that the files are too large.

I was following the ffmpeg bug report documentation (https://www.ffmpeg.org/bugreports.html) to try and get any additional information on why the process crashed with the original file, but that process didn't work either.  

Input: ifi-mac-pro:~ admin$ gdb ffmpeg
Output: GNU gdb (GDB) 8.0.1
Copyright (C) 2017 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-apple-darwin15.6.0".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from ffmpeg...(no debugging symbols found)...done.

Input: (gdb) r ffmpeg -i Card\ 1/B0011888U01.mov  -i / Card\ 2/B0011888.mov   -filter_complex signature=detectmode=full:nb_inputs=2 -f null -loglevel verbose -
Output: Starting program: /usr/local/bin/ffmpeg ffmpeg -i Card\ 1/B0011888U01.mov  -i  Card\ 2/B0011888.mov   -filter_complex signature=detectmode=full:nb_inputs=2 -f null -loglevel verbose -
Unable to find Mach task port for process-id 63368: (os/kern) failure (0x5).
 (please check gdb is codesigned - see taskgated(8))
(gdb)

I looked (codesigned - see taskgated(8)) and it looks like it needs to circumvent some security certificate, which I wouldn't be confident doing on this workstation. Would there be another way to get the backtrace, disassembly and register dump for any issues in the future?

Thanks,

Aoife
_______________________________________________
ffmpeg-user mailing list
[hidden email]
http://ffmpeg.org/mailman/listinfo/ffmpeg-user

To unsubscribe, visit link above, or email
[hidden email] with subject "unsubscribe".
Reply | Threaded
Open this post in threaded view
|

Re: Command being killed in perceptual hashing / ffmpeg in general

Carl Eugen Hoyos-2
2017-10-06 16:19 GMT+02:00 Aoife Fitzmaurice <[hidden email]>:
>
>>Generally, a backtrace, disassembly and register dump are
> required for crash reports (esp. if they are not easily reproducible).

> Thanks for your response! I changed the dimensions to 320x240
> and it worked as expected.

Allow me to repeat my question that you cut:
Does it crash because of oom, or for other reasons?

I cannot answer that, you should be able to see if the
memory consumption is moderate or huge.

>  It may just be that the files are too large.
>
> I was following the ffmpeg bug report documentation (https://www.ffmpeg.org/bugreports.html) to try and get any additional information on why the process crashed with the original file, but that process didn't work either.
>
> Input: ifi-mac-pro:~ admin$ gdb ffmpeg

Could it be that you should use "lldb" instead of gdb?

In any case, only debugging "ffmpeg_g" helps, "ffmpeg"
should not contain any debug symbols.

Carl Eugen
_______________________________________________
ffmpeg-user mailing list
[hidden email]
http://ffmpeg.org/mailman/listinfo/ffmpeg-user

To unsubscribe, visit link above, or email
[hidden email] with subject "unsubscribe".
Reply | Threaded
Open this post in threaded view
|

Re: Command being killed in perceptual hashing / ffmpeg in general

Aoife Fitzmaurice
>Does it crash because of oom, or for other reasons?

Yes, it runs out of memory. The memory consumption is huge (80gbs) and swapping 60gbs over. A colleague of mine recreated the problem using test files :

ffmpeg -f lavfi -i testsrc=size=1920x1080:rate=25  -c:v mpeg2video -t 01:03:00 -b:v 35000k -minrate 35000k -maxrate 35000k  -pix_fmt yuv420p -g 12 aftest1.mov && ffmpeg -f lavfi -i testsrc=size=1920x1080:rate=25  -c:v mpeg2video -t 00:28:06 -b:v 35000k -minrate 35000k -maxrate 35000k  -pix_fmt yuv420p -g 12 aftest2.mov


>Could it be that you should use "lldb" instead of gdb?

Thank you, that worked!

Input: (using files created above)

ifi-mac-pro:ffmpeg admin$ lldb ./ffmpeg_g
(lldb) target create "./ffmpeg_g"
Current executable set to './ffmpeg_g' (x86_64).
(lldb) r -i aftest2.mov  -i  aftest1.mov -filter_complex signature=detectmode=full:nb_inputs=2 -f null -


Output:

(lldb) r -i /Users/admin/aftest2.mov  -i /Users/admin/aftest1.mov -filter_complex signature=detectmode=full:nb_inputs=2 -f null -
Process 10575 launched: './ffmpeg_g' (x86_64)
ffmpeg version N-87767-g14b1553 Copyright (c) 2000-2017 the FFmpeg developers
  built with Apple LLVM version 8.0.0 (clang-800.0.42.1)
  configuration: --enable-gpl
  libavutil      55. 77.101 / 55. 77.101
  libavcodec     57.106.104 / 57.106.104
  libavformat    57. 82.104 / 57. 82.104
  libavdevice    57.  9.102 / 57.  9.102
  libavfilter     6.106.101 /  6.106.101
  libswscale      4.  7.103 /  4.  7.103
  libswresample   2.  8.100 /  2.  8.100
  libpostproc    54.  6.100 / 54.  6.100
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from '/Users/admin/aftest2.mov':
  Metadata:
    major_brand     : qt  
    minor_version   : 512
    compatible_brands: qt  
    encoder         : Lavf57.71.100
  Duration: 00:28:06.00, start: 0.000000, bitrate: 35000 kb/s
    Stream #0:0(eng): Video: mpeg2video (Main) (xdve / 0x65766478), yuv420p(tv, progressive), 1920x1080 [SAR 1:1 DAR 16:9], 34999 kb/s, 25 fps, 25 tbr, 12800 tbn, 50 tbc (default)
    Metadata:
      handler_name    : DataHandler
      encoder         : Lavc57.89.100 mpeg2video
Input #1, mov,mp4,m4a,3gp,3g2,mj2, from '/Users/admin/aftest1.mov':
  Metadata:
    major_brand     : qt  
    minor_version   : 512
    compatible_brands: qt  
    encoder         : Lavf57.71.100
  Duration: 01:03:00.00, start: 0.000000, bitrate: 35000 kb/s
    Stream #1:0(eng): Video: mpeg2video (Main) (xdve / 0x65766478), yuv420p(tv, progressive), 1920x1080 [SAR 1:1 DAR 16:9], 34999 kb/s, 25 fps, 25 tbr, 12800 tbn, 50 tbc (default)
    Metadata:
      handler_name    : DataHandler
      encoder         : Lavc57.89.100 mpeg2video
Stream mapping:
  Stream #0:0 (mpeg2video) -> signature:in0
  Stream #1:0 (mpeg2video) -> signature:in1
  signature -> Stream #0:0 (wrapped_avframe)
Press [q] to stop, [?] for help
Output #0, null, to 'pipe:':
  Metadata:
    major_brand     : qt  
    minor_version   : 512
    compatible_brands: qt  
    encoder         : Lavf57.82.104
    Stream #0:0: Video: wrapped_avframe, yuv420p, 1920x1080 [SAR 1:1 DAR 16:9], q=2-31, 200 kb/s, 25 fps, 25 tbn, 25 tbc (default)
    Metadata:
      encoder         : Lavc57.106.104 wrapped_avframe
Process 10575 stopped=-0.0 size=N/A time=00:28:06.00 bitrate=N/A speed=2.97x    
* thread #1: tid = 0x18e8e, 0x00007fff8a63663f libsystem_malloc.dylib`tiny_free_reattach_region + 214, queue = 'com.apple.main-thread', stop reason = signal SIGKILL
    frame #0: 0x00007fff8a63663f libsystem_malloc.dylib`tiny_free_reattach_region + 214
libsystem_malloc.dylib`tiny_free_reattach_region:
->  0x7fff8a63663f <+214>: movq   %r12, %rdi
    0x7fff8a636642 <+217>: callq  0x7fff8a63247f            ; get_tiny_free_size
    0x7fff8a636647 <+222>: movw   %ax, %r15w
    0x7fff8a63664b <+226>: testw  %r15w, %r15w
(lldb)



Could it be a memory leak, or does perceptual hashing not suit large files at present?

Thank you,

Aoife
_______________________________________________

_______________________________________________
ffmpeg-user mailing list
[hidden email]
http://ffmpeg.org/mailman/listinfo/ffmpeg-user

To unsubscribe, visit link above, or email
[hidden email] with subject "unsubscribe".
Reply | Threaded
Open this post in threaded view
|

Re: Command being killed in perceptual hashing / ffmpeg in general

Carl Eugen Hoyos-2
2017-10-10 18:45 GMT+02:00 Aoife Fitzmaurice <[hidden email]>:
>>Does it crash because of oom, or for other reasons?
>
> Yes, it runs out of memory. The memory consumption is huge
> (80gbs) and swapping 60gbs over. A colleague of mine
> recreated the problem using test files :
>
> ffmpeg -f lavfi -i testsrc=size=1920x1080:rate=25  -c:v mpeg2video -t 01:03:00 -b:v 35000k -minrate 35000k -maxrate 35000k  -pix_fmt yuv420p -g 12 aftest1.mov && ffmpeg -f lavfi -i testsrc=size=1920x1080:rate=25  -c:v mpeg2video -t 00:28:06 -b:v 35000k -minrate 35000k -maxrate 35000k  -pix_fmt yuv420p -g 12 aftest2.mov

The problem is that the files have a significant difference in
length.

Does it make sense to use the signature filter on two
files with very different length?

Thank you, Carl Eugen
_______________________________________________
ffmpeg-user mailing list
[hidden email]
http://ffmpeg.org/mailman/listinfo/ffmpeg-user

To unsubscribe, visit link above, or email
[hidden email] with subject "unsubscribe".