Verifying lossless rewrapping/transcoding in one step?

classic Classic list List threaded Threaded
15 messages Options
Reply | Threaded
Open this post in threaded view
|

Verifying lossless rewrapping/transcoding in one step?

Peter B.
Hi everyone :)

I was wondering if maybe there's a quicker way to verify lossless
transcoding or container rewrapping than these 3 steps:

  1. generate content hash during transcoding.
  2. generate content hash of output file.
  3. diff source-hash with output-hash (using whatever diff-tool).


I'm doing this in bash scripts, but I'm wondering if it would be
possible to tell ffmpeg something like this:

"Please generate the output and compare source and target's content hash
and only return exit code 0 if they match"


Is that already implemented, or would it be possible to do that?



Thank you very much in advance,
Peter B.
_______________________________________________
ffmpeg-user mailing list
[hidden email]
https://ffmpeg.org/mailman/listinfo/ffmpeg-user

To unsubscribe, visit link above, or email
[hidden email] with subject "unsubscribe".
Reply | Threaded
Open this post in threaded view
|

Re: Verifying lossless rewrapping/transcoding in one step?

Brainiarc7
On Fri, 26 Jul 2019 at 21:02, Peter B. <[hidden email]> wrote:

> Hi everyone :)
>
> I was wondering if maybe there's a quicker way to verify lossless
> transcoding or container rewrapping than these 3 steps:
>
>   1. generate content hash during transcoding.
>   2. generate content hash of output file.
>   3. diff source-hash with output-hash (using whatever diff-tool).
>
>
> I'm doing this in bash scripts, but I'm wondering if it would be
> possible to tell ffmpeg something like this:
>
> "Please generate the output and compare source and target's content hash
> and only return exit code 0 if they match"
>
>
> Is that already implemented, or would it be possible to do that?
>
>
>
> Thank you very much in advance,
> Peter B.
>
>
Hey there,

You could use a muxer such as framemd5:
https://ffmpeg.org/ffmpeg-formats.html#framemd5-1

See the link above for usage.


--
まさひろ はやし
_______________________________________________
ffmpeg-user mailing list
[hidden email]
https://ffmpeg.org/mailman/listinfo/ffmpeg-user

To unsubscribe, visit link above, or email
[hidden email] with subject "unsubscribe".
Reply | Threaded
Open this post in threaded view
|

Re: Verifying lossless rewrapping/transcoding in one step?

Peter B.
On 26/07/2019 22:54, Dennis Mungai wrote:
> You could use a muxer such as framemd5:
> https://ffmpeg.org/ffmpeg-formats.html#framemd5-1

Thanks for your reply, but I've been using framemd5 for years.
I meant, if there's a way that I don't have to code a script around the
ffmpeg call to do the actual validation?


Kind regards,
Peter B.
_______________________________________________
ffmpeg-user mailing list
[hidden email]
https://ffmpeg.org/mailman/listinfo/ffmpeg-user

To unsubscribe, visit link above, or email
[hidden email] with subject "unsubscribe".
Reply | Threaded
Open this post in threaded view
|

Re: Verifying lossless rewrapping/transcoding in one step?

Peter B.
Hi everyone :)

I assume no answer means there's no option to do that.

Would anyone be willing to implement it?
And how much could that approximately cost?


Thank you very much in advance,
Peter B.
_______________________________________________
ffmpeg-user mailing list
[hidden email]
https://ffmpeg.org/mailman/listinfo/ffmpeg-user

To unsubscribe, visit link above, or email
[hidden email] with subject "unsubscribe".
Reply | Threaded
Open this post in threaded view
|

Re: Verifying lossless rewrapping/transcoding in one step?

Moritz Barsnick
On Fri, Aug 02, 2019 at 14:51:36 +0200, Peter B. wrote:
> I assume no answer means there's no option to do that.

No, there isn't.

I believe in using the right tools for the right tasks. If it can be
glued together with a script, why worry about modifying an existing
program to do the same?

That said, regarding your request, I believe the showstopper is in the
concept: To re-calculate the checksum of the output, it needs to be
decoded again. That's not something ffmpeg can do within its binary, it
can't use the output and chain it as its own input. You *can* do that
by scripting (obviously), and you could construct some program with the
libav* libraries. But at the end of the day, you want the features of
ffmpeg (the command line tool), which you can't just reeinvent. So
basically, you need to prop something into the command line tool which
can chain encodes and decodes. And that's no small task, I believe,
unlike "yet another special muxer/demuxer".

That's my take on it: Way too much effort for the gain.

Cheers,
Moritz
_______________________________________________
ffmpeg-user mailing list
[hidden email]
https://ffmpeg.org/mailman/listinfo/ffmpeg-user

To unsubscribe, visit link above, or email
[hidden email] with subject "unsubscribe".
Reply | Threaded
Open this post in threaded view
|

Re: Verifying lossless rewrapping/transcoding in one step?

Peter B.
Dear Moritz,

Thank you for your explanation!


On 03/08/2019 14:07, Moritz Barsnick wrote:
> I believe in using the right tools for the right tasks. If it can be
> glued together with a script, why worry about modifying an existing
> program to do the same?

I completely agree, but... ;)

One reason (for me) is that I haven't found a generic way to script this
check for inhomogenous source material. I don't know how many A/V
streams, in which configuration, etc. I know I can ffprobe, then parse,
etc - but:

The 2nd reason (also for me) is, that I've now written the same
functionality in different scripts for slightly different use cases.

And the 3rd reason (for me...) is:
It's actually not-so-trivial to explain this validation-check approach
when I do FFmpeg workshops (for commandline newbies, and people who
don't know what a "diff" even is)


:D


> That said, regarding your request, I believe the showstopper is in the
> concept: To re-calculate the checksum of the output, it needs to be
> decoded again. That's not something ffmpeg can do within its binary, it
> can't use the output and chain it as its own input.

That's a very interesting insight!
Thank you very much. I was sure there's a reason, but now I finally
know. Thank you!


> That's my take on it: Way too much effort for the gain.

Hm...
I'm thinking about how (under the given circumstances you've mentioned),
this could still be made a bit "easier"...

Would it be possible to, for example, modify the hash muxer to output a
hash for each stream - in a way that one could easily parse that
textfile and know which hash belongs to which stream?

For example, instead of "CRC32=...." let's say:
"v:0:CRC32=...
v:1:CRC32=...
a:0:CRC32...
a:1:CRC32...
a:2...
"

Something like that.
Just an idea... :D


Thank you very much again!
Peter


_______________________________________________
ffmpeg-user mailing list
[hidden email]
https://ffmpeg.org/mailman/listinfo/ffmpeg-user

To unsubscribe, visit link above, or email
[hidden email] with subject "unsubscribe".
Reply | Threaded
Open this post in threaded view
|

Re: Verifying lossless rewrapping/transcoding in one step?

Paul B Mahol
On Mon, Aug 5, 2019 at 4:16 PM Peter B. <[hidden email]> wrote:

> Dear Moritz,
>
> Thank you for your explanation!
>
>
> On 03/08/2019 14:07, Moritz Barsnick wrote:
> > I believe in using the right tools for the right tasks. If it can be
> > glued together with a script, why worry about modifying an existing
> > program to do the same?
>
> I completely agree, but... ;)
>
> One reason (for me) is that I haven't found a generic way to script this
> check for inhomogenous source material. I don't know how many A/V
> streams, in which configuration, etc. I know I can ffprobe, then parse,
> etc - but:
>
> The 2nd reason (also for me) is, that I've now written the same
> functionality in different scripts for slightly different use cases.
>
> And the 3rd reason (for me...) is:
> It's actually not-so-trivial to explain this validation-check approach
> when I do FFmpeg workshops (for commandline newbies, and people who
> don't know what a "diff" even is)
>
>
> :D
>
>
> > That said, regarding your request, I believe the showstopper is in the
> > concept: To re-calculate the checksum of the output, it needs to be
> > decoded again. That's not something ffmpeg can do within its binary, it
> > can't use the output and chain it as its own input.
>
> That's a very interesting insight!
> Thank you very much. I was sure there's a reason, but now I finally
> know. Thank you!
>
>
> > That's my take on it: Way too much effort for the gain.
>
> Hm...
> I'm thinking about how (under the given circumstances you've mentioned),
> this could still be made a bit "easier"...
>
> Would it be possible to, for example, modify the hash muxer to output a
> hash for each stream - in a way that one could easily parse that
> textfile and know which hash belongs to which stream?
>
> For example, instead of "CRC32=...." let's say:
> "v:0:CRC32=...
> v:1:CRC32=...
> a:0:CRC32...
> a:1:CRC32...
> a:2...
> "
>

Hash muxer does not work like that. Why would you need hash muxer for
multiple streams?

You can use several maps to produce hash of more stream at once, each
output directed to separate file.

ffmpeg.exe -i fate-suite/mxf/C0023S01.mxf -vn -map 0:1 -f framecrc
first_audio_stream.framecrc -map 0:2 -f framecrc
second_audio_stream.framecrc
_______________________________________________
ffmpeg-user mailing list
[hidden email]
https://ffmpeg.org/mailman/listinfo/ffmpeg-user

To unsubscribe, visit link above, or email
[hidden email] with subject "unsubscribe".
Reply | Threaded
Open this post in threaded view
|

Re: Verifying lossless rewrapping/transcoding in one step?

Peter B.
Hi Paul!

On 05/08/2019 16:25, Paul B Mahol wrote:

> On Mon, Aug 5, 2019 at 4:16 PM Peter B. <[hidden email]> wrote:
>
>> [...]
>> For example, instead of "CRC32=...." let's say:
>> "v:0:CRC32=...
>> v:1:CRC32=...
>> a:0:CRC32...
>> a:1:CRC32...
>> a:2...
>> "
>>
> Hash muxer does not work like that.

Oh :(


> Why would you need hash muxer for multiple streams?

Because I want to verify that every stream was transferred losslessly
from source to target.
For both: audio and video.
(and actually also non-AV tracks, any time-based metadata, etc)


> You can use several maps to produce hash of more stream at once, each output directed to separate file.
>
> ffmpeg.exe -i fate-suite/mxf/C0023S01.mxf -vn -map 0:1 -f framecrc
> first_audio_stream.framecrc -map 0:2 -f framecrc
> second_audio_stream.framecrc

Thanks!

However, that's what I'm already doing (-f hash), but I'm writing for
fully-automated transcoding scenarios, so I need to parse the source
file to know how many streams (of which type), and then have another
part that puts together the right commandline recipe, and then write
code that generates names for each stream-hash output file, and then,
and then I need to keep track of all these files, and then ...

And I believe I'm neither the first nor the last person to write code
for automating this.
So I thought: wouldn't it be great to find a nicer way of validating
"output-fidelity"? :)

Any ideas?


What I don't understand:
If the hash muxer output can generate a single hash for the sums of all
streams, I presume it has access to all streams. Since it can also
output individual hashes to individual files, why isn't it possible to
do the same, but store the hashcodes in variables (associated with their
source stream), and then write that information in one single file?


I like puzzles :)
Thank you very much in advance!

Peter B.

_______________________________________________
ffmpeg-user mailing list
[hidden email]
https://ffmpeg.org/mailman/listinfo/ffmpeg-user

To unsubscribe, visit link above, or email
[hidden email] with subject "unsubscribe".
Reply | Threaded
Open this post in threaded view
|

Re: Verifying lossless rewrapping/transcoding in one step?

Paul B Mahol
On Mon, Aug 5, 2019 at 11:13 PM Peter B. <[hidden email]> wrote:

> Hi Paul!
>
> On 05/08/2019 16:25, Paul B Mahol wrote:
> > On Mon, Aug 5, 2019 at 4:16 PM Peter B. <[hidden email]> wrote:
> >
> >> [...]
> >> For example, instead of "CRC32=...." let's say:
> >> "v:0:CRC32=...
> >> v:1:CRC32=...
> >> a:0:CRC32...
> >> a:1:CRC32...
> >> a:2...
> >> "
> >>
> > Hash muxer does not work like that.
>
> Oh :(
>
>
> > Why would you need hash muxer for multiple streams?
>
> Because I want to verify that every stream was transferred losslessly
> from source to target.
> For both: audio and video.
> (and actually also non-AV tracks, any time-based metadata, etc)
>
>
> > You can use several maps to produce hash of more stream at once, each
> output directed to separate file.
> >
> > ffmpeg.exe -i fate-suite/mxf/C0023S01.mxf -vn -map 0:1 -f framecrc
> > first_audio_stream.framecrc -map 0:2 -f framecrc
> > second_audio_stream.framecrc
>
> Thanks!
>
> However, that's what I'm already doing (-f hash), but I'm writing for
> fully-automated transcoding scenarios, so I need to parse the source
> file to know how many streams (of which type), and then have another
> part that puts together the right commandline recipe, and then write
> code that generates names for each stream-hash output file, and then,
> and then I need to keep track of all these files, and then ...
>
> And I believe I'm neither the first nor the last person to write code
> for automating this.
> So I thought: wouldn't it be great to find a nicer way of validating
> "output-fidelity"? :)
>
> Any ideas?
>
>
> What I don't understand:
> If the hash muxer output can generate a single hash for the sums of all
> streams, I presume it has access to all streams. Since it can also
> output individual hashes to individual files, why isn't it possible to
> do the same, but store the hashcodes in variables (associated with their
> source stream), and then write that information in one single file?
>
>
> I like puzzles :)
> Thank you very much in advance!
>

If I understand correctly, basically need hash of each stream separately
(for framehash this already works).

That is quite possible to be implemented. Feel free to open feature request
on FFmpeg bug tracker.
_______________________________________________
ffmpeg-user mailing list
[hidden email]
https://ffmpeg.org/mailman/listinfo/ffmpeg-user

To unsubscribe, visit link above, or email
[hidden email] with subject "unsubscribe".
Reply | Threaded
Open this post in threaded view
|

Re: Verifying lossless rewrapping/transcoding in one step?

Moritz Barsnick
On Tue, Aug 06, 2019 at 10:07:40 +0200, Paul B Mahol wrote:
> If I understand correctly, basically need hash of each stream separately
> (for framehash this already works).
>
> That is quite possible to be implemented. Feel free to open feature request
> on FFmpeg bug tracker.

Just to show that it works, I have hacked up a quick proof of concept.
(Hacked up means: Possibly not clean, not enough reuse of code, ...)

From this for a video stream:
./ffmpeg_g -f lavfi -i testsrc -map 0 -t 1 -f hash -
[...]
SHA256=c747429bf5d240958db9545b184e2b226f826e67495ad5d46245794bc1b84078

and this for an audio stream:
./ffmpeg_g -f lavfi -i anoisesrc=seed=0 -map 0 -t 1 -f hash -
[...]
SHA256=4a3aa64bd7ef86ff2afb43c9346076792497cdcfcd741f10a1bea2e016294417

I can now produce this (note the new muxer):
./ffmpeg_g -f lavfi -i testsrc -f lavfi -i anoisesrc=seed=0 -map 0 -map 1 -t 1 -f streamhash -
SHA256=c747429bf5d240958db9545b184e2b226f826e67495ad5d46245794bc1b84078
SHA256=4a3aa64bd7ef86ff2afb43c9346076792497cdcfcd741f10a1bea2e016294417

I just couldn't quickly figure out how to prefix the stream specifiers
(e.g. 0:0, 0:1).

I'm not hapy with submitting a hack as a patch to ffmpeg-devel though.
;-) But I can provide the patch here or somewhere else.

Cheers,
Moritz
_______________________________________________
ffmpeg-user mailing list
[hidden email]
https://ffmpeg.org/mailman/listinfo/ffmpeg-user

To unsubscribe, visit link above, or email
[hidden email] with subject "unsubscribe".
Reply | Threaded
Open this post in threaded view
|

Re: Verifying lossless rewrapping/transcoding in one step?

Paul B Mahol
On Tue, Aug 6, 2019 at 12:14 PM Moritz Barsnick <[hidden email]> wrote:

> On Tue, Aug 06, 2019 at 10:07:40 +0200, Paul B Mahol wrote:
> > If I understand correctly, basically need hash of each stream separately
> > (for framehash this already works).
> >
> > That is quite possible to be implemented. Feel free to open feature
> request
> > on FFmpeg bug tracker.
>
> Just to show that it works, I have hacked up a quick proof of concept.
> (Hacked up means: Possibly not clean, not enough reuse of code, ...)
>
> From this for a video stream:
> ./ffmpeg_g -f lavfi -i testsrc -map 0 -t 1 -f hash -
> [...]
> SHA256=c747429bf5d240958db9545b184e2b226f826e67495ad5d46245794bc1b84078
>
> and this for an audio stream:
> ./ffmpeg_g -f lavfi -i anoisesrc=seed=0 -map 0 -t 1 -f hash -
> [...]
> SHA256=4a3aa64bd7ef86ff2afb43c9346076792497cdcfcd741f10a1bea2e016294417
>
> I can now produce this (note the new muxer):
> ./ffmpeg_g -f lavfi -i testsrc -f lavfi -i anoisesrc=seed=0 -map 0 -map 1
> -t 1 -f streamhash -
> SHA256=c747429bf5d240958db9545b184e2b226f826e67495ad5d46245794bc1b84078
> SHA256=4a3aa64bd7ef86ff2afb43c9346076792497cdcfcd741f10a1bea2e016294417
>
> I just couldn't quickly figure out how to prefix the stream specifiers
> (e.g. 0:0, 0:1).
>

Look how framehash does it?


>
> I'm not hapy with submitting a hack as a patch to ffmpeg-devel though.
> ;-) But I can provide the patch here or somewhere else.
>
_______________________________________________
ffmpeg-user mailing list
[hidden email]
https://ffmpeg.org/mailman/listinfo/ffmpeg-user

To unsubscribe, visit link above, or email
[hidden email] with subject "unsubscribe".
Reply | Threaded
Open this post in threaded view
|

Re: Verifying lossless rewrapping/transcoding in one step?

Moritz Barsnick
On Tue, Aug 06, 2019 at 12:21:58 +0200, Paul B Mahol wrote:
> > I just couldn't quickly figure out how to prefix the stream specifiers
> > (e.g. 0:0, 0:1).
> Look how framehash does it?

Yeah, too obvious. I was looking for "0:1", but the first digit is
meaningless, and the second one is just the stream index (which I use
to iterate anyway).

I'll try to rework it to give the "hash" muxer a "-per_stream" boolean
option, then I can (and must) reuse all the code.

Cheers,
Moritz
_______________________________________________
ffmpeg-user mailing list
[hidden email]
https://ffmpeg.org/mailman/listinfo/ffmpeg-user

To unsubscribe, visit link above, or email
[hidden email] with subject "unsubscribe".
Reply | Threaded
Open this post in threaded view
|

Re: Verifying lossless rewrapping/transcoding in one step?

Peter B.
On 06/08/2019 13:31, Moritz Barsnick wrote:
> Yeah, too obvious. I was looking for "0:1", but the first digit is
> meaningless, and the second one is just the stream index (which I use
> to iterate anyway).
> I'll try to rework it to give the "hash" muxer a "-per_stream" boolean
> option, then I can (and must) reuse all the code.

Wooow!
That'd be awesome!

Thank you soooo much in advance,
Peter



_______________________________________________
ffmpeg-user mailing list
[hidden email]
https://ffmpeg.org/mailman/listinfo/ffmpeg-user

To unsubscribe, visit link above, or email
[hidden email] with subject "unsubscribe".
Reply | Threaded
Open this post in threaded view
|

Re: Verifying lossless rewrapping/transcoding in one step?

Moritz Barsnick
On Wed, Aug 07, 2019 at 15:10:06 +0200, Peter B. wrote:
> > I'll try to rework it to give the "hash" muxer a "-per_stream" boolean
> > option, then I can (and must) reuse all the code.
>
> Wooow!
> That'd be awesome!

I sent a non-hackish implementation to Peter privately. I'll clean it
up and post it to ffmpeg-devel eventually, at the risk of its
usefulness or quality being doubted. Or improved.

Cheers,
Moritz
_______________________________________________
ffmpeg-user mailing list
[hidden email]
https://ffmpeg.org/mailman/listinfo/ffmpeg-user

To unsubscribe, visit link above, or email
[hidden email] with subject "unsubscribe".
Reply | Threaded
Open this post in threaded view
|

Re: Verifying lossless rewrapping/transcoding in one step?

Paul B Mahol
On Wed, Aug 7, 2019 at 4:26 PM Moritz Barsnick <[hidden email]> wrote:

> On Wed, Aug 07, 2019 at 15:10:06 +0200, Peter B. wrote:
> > > I'll try to rework it to give the "hash" muxer a "-per_stream" boolean
> > > option, then I can (and must) reuse all the code.
> >
> > Wooow!
> > That'd be awesome!
>
> I sent a non-hackish implementation to Peter privately. I'll clean it
> up and post it to ffmpeg-devel eventually, at the risk of its
> usefulness or quality being doubted. Or improved.
>

If you do not do that soon, I  will.


> Cheers,
> Moritz
> _______________________________________________
> ffmpeg-user mailing list
> [hidden email]
> https://ffmpeg.org/mailman/listinfo/ffmpeg-user
>
> To unsubscribe, visit link above, or email
> [hidden email] with subject "unsubscribe".
_______________________________________________
ffmpeg-user mailing list
[hidden email]
https://ffmpeg.org/mailman/listinfo/ffmpeg-user

To unsubscribe, visit link above, or email
[hidden email] with subject "unsubscribe".