FFmpeg & hashing algorithm speed

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

FFmpeg & hashing algorithm speed

Peter B.
Hi everyone!

Hashing for fixity and transfer-checking seems to be quite a bottleneck
for archiving big video/film files for archiving.

Therefore I was considering if using "simpler" algorithms (like CRC
instead of SHA256) could speed up the process.
I was surprised that e.g. MD5 vs CRC in Python as well as PHP seemed to
be equally "fast".

------------------------------------
So I was wondering:
Does FFmpeg have/do any speed optimization for hashing functionality
(framemd5, hash muxer, etc) in its code?
------------------------------------


I've noticed, for example that "-f hash" for MD5 uses only half, SHA256
just a quarter of my CPU. Input source was H.264/AAC/MP4, which I guess
decodes multi-threaded?

I'd be very grateful if anyone's interested to provide me some insights
or ideas in how FFmpeg's hashing could be made faster.



Thank you very much in advance,
Peter B.


PS:
I also found a possibly related claim in a Stackoverflow answer that the
hashing functions are not implemented as fast as they could be:
https://stackoverflow.com/questions/16122067/md5-vs-crc32-which-ones-better-for-common-use/35484131#comment67221384_35484131
_______________________________________________
ffmpeg-user mailing list
[hidden email]
https://ffmpeg.org/mailman/listinfo/ffmpeg-user

To unsubscribe, visit link above, or email
[hidden email] with subject "unsubscribe".
Reply | Threaded
Open this post in threaded view
|

Re: FFmpeg & hashing algorithm speed

Carl Eugen Hoyos-2
Am Mi., 6. Nov. 2019 um 23:40 Uhr schrieb Peter B. <[hidden email]>:

> Hashing for fixity and transfer-checking seems to be quite a bottleneck
> for archiving big video/film files for archiving.
>
> Therefore I was considering if using "simpler" algorithms (like CRC
> instead of SHA256) could speed up the process.

> I was surprised that e.g. MD5 vs CRC in Python as well as PHP
> seemed to be equally "fast".

But this is not related to FFmpeg, no?

> So I was wondering:
> Does FFmpeg have/do any speed optimization for hashing functionality
> (framemd5, hash muxer, etc) in its code?

Some of Giorgio's commits were meant to increase hash function
speed:
http://git.videolan.org/?p=ffmpeg.git;a=search;s=vazzana;st=author
There is an email with a script to compare the implementation's speeds,
I don't know when it was sent to the development mailing list though.

> I've noticed, for example that "-f hash" for MD5 uses only half, SHA256
> just a quarter of my CPU.

Command line and complete, uncut console output missing.

Carl Eugen
_______________________________________________
ffmpeg-user mailing list
[hidden email]
https://ffmpeg.org/mailman/listinfo/ffmpeg-user

To unsubscribe, visit link above, or email
[hidden email] with subject "unsubscribe".
Reply | Threaded
Open this post in threaded view
|

Re: FFmpeg & hashing algorithm speed

Nicolas George
Carl Eugen Hoyos (12019-11-07):
> There is an email with a script to compare the implementation's speeds,
> I don't know when it was sent to the development mailing list though.

If you mean tools/crypto_bench.c, it was not only sent to the list but
applied.

Regards,

--
  Nicolas George

_______________________________________________
ffmpeg-user mailing list
[hidden email]
https://ffmpeg.org/mailman/listinfo/ffmpeg-user

To unsubscribe, visit link above, or email
[hidden email] with subject "unsubscribe".

signature.asc (849 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: FFmpeg & hashing algorithm speed

Carl Eugen Hoyos-2
In reply to this post by Carl Eugen Hoyos-2
Am Do., 7. Nov. 2019 um 00:09 Uhr schrieb Carl Eugen Hoyos <[hidden email]>:

>
> Am Mi., 6. Nov. 2019 um 23:40 Uhr schrieb Peter B. <[hidden email]>:
>
> > Hashing for fixity and transfer-checking seems to be quite a bottleneck
> > for archiving big video/film files for archiving.
> >
> > Therefore I was considering if using "simpler" algorithms (like CRC
> > instead of SHA256) could speed up the process.
>
> > I was surprised that e.g. MD5 vs CRC in Python as well as PHP
> > seemed to be equally "fast".
>
> But this is not related to FFmpeg, no?
>
> > So I was wondering:
> > Does FFmpeg have/do any speed optimization for hashing functionality
> > (framemd5, hash muxer, etc) in its code?
>
> Some of Giorgio's commits were meant to increase hash function
> speed:
> http://git.videolan.org/?p=ffmpeg.git;a=search;s=vazzana;st=author
> There is an email with a script to compare the implementation's speeds,
> I don't know when it was sent to the development mailing list though.

This mail leads to a long thread:
http://mplayerhq.hu/pipermail/ffmpeg-devel/2013-May/143854.html
In the end a file tools/ffhash.c was created to compare hash
implementation speeds.

Carl Eugen
_______________________________________________
ffmpeg-user mailing list
[hidden email]
https://ffmpeg.org/mailman/listinfo/ffmpeg-user

To unsubscribe, visit link above, or email
[hidden email] with subject "unsubscribe".
Reply | Threaded
Open this post in threaded view
|

Re: FFmpeg & hashing algorithm speed

Peter B.
Thanks for the infos!

And "yes": My PHP/Python examples were not related to FFmpeg, but this
discovery led me to become curious how FFmpeg does it - and therefore
ask this question :)

Will read the Giorgio-Thread and maybe return with reproducible
(commandline + uncut console output) question scenarios.



Thank you very much!
Pb


On 07/11/2019 00:21, Carl Eugen Hoyos wrote:

> Am Do., 7. Nov. 2019 um 00:09 Uhr schrieb Carl Eugen Hoyos <[hidden email]>:
>> Am Mi., 6. Nov. 2019 um 23:40 Uhr schrieb Peter B. <[hidden email]>:
>>
>>> Hashing for fixity and transfer-checking seems to be quite a bottleneck
>>> for archiving big video/film files for archiving.
>>>
>>> Therefore I was considering if using "simpler" algorithms (like CRC
>>> instead of SHA256) could speed up the process.
>>> I was surprised that e.g. MD5 vs CRC in Python as well as PHP
>>> seemed to be equally "fast".
>> But this is not related to FFmpeg, no?
>>
>>> So I was wondering:
>>> Does FFmpeg have/do any speed optimization for hashing functionality
>>> (framemd5, hash muxer, etc) in its code?
>> Some of Giorgio's commits were meant to increase hash function
>> speed:
>> http://git.videolan.org/?p=ffmpeg.git;a=search;s=vazzana;st=author
>> There is an email with a script to compare the implementation's speeds,
>> I don't know when it was sent to the development mailing list though.
> This mail leads to a long thread:
> http://mplayerhq.hu/pipermail/ffmpeg-devel/2013-May/143854.html
> In the end a file tools/ffhash.c was created to compare hash
> implementation speeds.
>
> Carl Eugen
> _______________________________________________
> ffmpeg-user mailing list
> [hidden email]
> https://ffmpeg.org/mailman/listinfo/ffmpeg-user
>
> To unsubscribe, visit link above, or email
> [hidden email] with subject "unsubscribe".

_______________________________________________
ffmpeg-user mailing list
[hidden email]
https://ffmpeg.org/mailman/listinfo/ffmpeg-user

To unsubscribe, visit link above, or email
[hidden email] with subject "unsubscribe".