Syncing clips based on audio waveform?

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|

Syncing clips based on audio waveform?

artwait
Hi all,

This is coming from out of left field I imagine, but I'm wondering if
anyone knows of a filter (or chain of filters) which would allow me to
align/sync two video clips based on matches in their audio waveforms.

I'm shooting events with multiple cameras and would like to either trim the
clips to a precise, matching starting frame or generate a metadata "offset"
marker indicating how far off one clip is from the other in time. Then I
can trim the clips to match.

Is this possible in ffmpeg (or anywhere else for that matter!)?

Thanks very much!

Art
_______________________________________________
ffmpeg-user mailing list
[hidden email]
http://ffmpeg.org/mailman/listinfo/ffmpeg-user
Reply | Threaded
Open this post in threaded view
|

Re: Syncing clips based on audio waveform?

Pavel Koshevoy
On 3/1/2013 9:38 AM, Arthur Wait wrote:

> Hi all,
>
> This is coming from out of left field I imagine, but I'm wondering if
> anyone knows of a filter (or chain of filters) which would allow me to
> align/sync two video clips based on matches in their audio waveforms.
>
> I'm shooting events with multiple cameras and would like to either trim the
> clips to a precise, matching starting frame or generate a metadata "offset"
> marker indicating how far off one clip is from the other in time. Then I
> can trim the clips to match.
>
> Is this possible in ffmpeg (or anywhere else for that matter!)?


I don't know if there is anything in ffmpeg that would accomplish what
you need right now, and I doubt implementing it as a filter is very
practical.  It could be implemented as a separate utility -- give it a
reference file and another file and it spits out time offset of the
second file relative to the reference file (if it can find a correlation).

Pavel.


_______________________________________________
ffmpeg-user mailing list
[hidden email]
http://ffmpeg.org/mailman/listinfo/ffmpeg-user
Reply | Threaded
Open this post in threaded view
|

Re: Syncing clips based on audio waveform?

Pavel Koshevoy
On 3/1/2013 3:17 PM, Pavel Koshevoy wrote:

> On 3/1/2013 9:38 AM, Arthur Wait wrote:
>> Hi all,
>>
>> This is coming from out of left field I imagine, but I'm wondering if
>> anyone knows of a filter (or chain of filters) which would allow me to
>> align/sync two video clips based on matches in their audio waveforms.
>>
>> I'm shooting events with multiple cameras and would like to either
>> trim the
>> clips to a precise, matching starting frame or generate a metadata
>> "offset"
>> marker indicating how far off one clip is from the other in time. Then I
>> can trim the clips to match.
>>
>> Is this possible in ffmpeg (or anywhere else for that matter!)?
>
>
> I don't know if there is anything in ffmpeg that would accomplish what
> you need right now, and I doubt implementing it as a filter is very
> practical.  It could be implemented as a separate utility -- give it a
> reference file and another file and it spits out time offset of the
> second file relative to the reference file (if it can find a
> correlation).

Now that I think about it, this feature probably should be implemented
in ffprobe.

     Pavel

_______________________________________________
ffmpeg-user mailing list
[hidden email]
http://ffmpeg.org/mailman/listinfo/ffmpeg-user
Reply | Threaded
Open this post in threaded view
|

Re: Syncing clips based on audio waveform?

Dave Rice-3

On Mar 1, 2013, at 6:00 PM, Pavel Koshevoy <[hidden email]> wrote:

> On 3/1/2013 3:17 PM, Pavel Koshevoy wrote:
>> On 3/1/2013 9:38 AM, Arthur Wait wrote:
>>> Hi all,
>>>
>>> This is coming from out of left field I imagine, but I'm wondering if
>>> anyone knows of a filter (or chain of filters) which would allow me to
>>> align/sync two video clips based on matches in their audio waveforms.
>>>
>>> I'm shooting events with multiple cameras and would like to either trim the
>>> clips to a precise, matching starting frame or generate a metadata "offset"
>>> marker indicating how far off one clip is from the other in time. Then I
>>> can trim the clips to match.
>>>
>>> Is this possible in ffmpeg (or anywhere else for that matter!)?
>>
>>
>> I don't know if there is anything in ffmpeg that would accomplish what you need right now, and I doubt implementing it as a filter is very practical.  It could be implemented as a separate utility -- give it a reference file and another file and it spits out time offset of the second file relative to the reference file (if it can find a correlation).
>
> Now that I think about it, this feature probably should be implemented in ffprobe.

+1 for that.
Dave Rice
_______________________________________________
ffmpeg-user mailing list
[hidden email]
http://ffmpeg.org/mailman/listinfo/ffmpeg-user
Reply | Threaded
Open this post in threaded view
|

Re: Syncing clips based on audio waveform?

Paul B Mahol
On 3/2/13, Dave Rice <[hidden email]> wrote:

>
> On Mar 1, 2013, at 6:00 PM, Pavel Koshevoy <[hidden email]> wrote:
>
>> On 3/1/2013 3:17 PM, Pavel Koshevoy wrote:
>>> On 3/1/2013 9:38 AM, Arthur Wait wrote:
>>>> Hi all,
>>>>
>>>> This is coming from out of left field I imagine, but I'm wondering if
>>>> anyone knows of a filter (or chain of filters) which would allow me to
>>>> align/sync two video clips based on matches in their audio waveforms.
>>>>
>>>> I'm shooting events with multiple cameras and would like to either trim
>>>> the
>>>> clips to a precise, matching starting frame or generate a metadata
>>>> "offset"
>>>> marker indicating how far off one clip is from the other in time. Then
>>>> I
>>>> can trim the clips to match.
>>>>
>>>> Is this possible in ffmpeg (or anywhere else for that matter!)?
>>>
>>>
>>> I don't know if there is anything in ffmpeg that would accomplish what
>>> you need right now, and I doubt implementing it as a filter is very
>>> practical.  It could be implemented as a separate utility -- give it a
>>> reference file and another file and it spits out time offset of the
>>> second file relative to the reference file (if it can find a
>>> correlation).
>>
>> Now that I think about it, this feature probably should be implemented in
>> ffprobe.
>
> +1 for that.

Nonsense...

> Dave Rice
> _______________________________________________
> ffmpeg-user mailing list
> [hidden email]
> http://ffmpeg.org/mailman/listinfo/ffmpeg-user
>
_______________________________________________
ffmpeg-user mailing list
[hidden email]
http://ffmpeg.org/mailman/listinfo/ffmpeg-user
Reply | Threaded
Open this post in threaded view
|

Re: Syncing clips based on audio waveform?

Bouke / VideoToolShed
In reply to this post by artwait
----- Original Message -----
From: "Arthur Wait" <[hidden email]>
To: <[hidden email]>
Sent: Friday, March 01, 2013 5:38 PM
Subject: [FFmpeg-user] Syncing clips based on audio waveform?


> Hi all,
>
> This is coming from out of left field I imagine, but I'm wondering if
> anyone knows of a filter (or chain of filters) which would allow me to
> align/sync two video clips based on matches in their audio waveforms.

I'm guessing you're after field recording syncing...
it is done in a commercial application called Plural Eyes.
The original developer has sold it, and now it's owned (and improved /
maintained) by Red Giant IIRC.
With improved, i mean it sucks less, but it still is dog slow, and has only
success when both sources are very much alike.
For double system shooting it is absolutely unreliable (It is now advertised
as a 'Fix it in post' product.)

From my (limited) point of view, waveform is not the best way, as noise will
drive you crazy.
Finding peaks would be better, but you'll find peaks in one source that is
not in the other.
(Like tapping on mics, other close sounds.)
Then, there is no guarantee both sources run at exactly the same speed.
A drift of some 20 Msecs over a long take is not uncommon.
Not really a problem for post, but you need to compensate in your filter.
Then there is the trouble of not knowing for sure what takes belong
together.
(Of course no problem in a lab, but for stuff coming out of the field,
Murphy has a strong case.)

hth,
Bouke (who is a timecode freak)

> I'm shooting events with multiple cameras and would like to either trim
> the
> clips to a precise, matching starting frame or generate a metadata
> "offset"
> marker indicating how far off one clip is from the other in time. Then I
> can trim the clips to match.
>
> Is this possible in ffmpeg (or anywhere else for that matter!)?
>
> Thanks very much!
>
> Art
> _______________________________________________
> ffmpeg-user mailing list
> [hidden email]
> http://ffmpeg.org/mailman/listinfo/ffmpeg-user
>
>
>


_______________________________________________
ffmpeg-user mailing list
[hidden email]
http://ffmpeg.org/mailman/listinfo/ffmpeg-user
Reply | Threaded
Open this post in threaded view
|

Re: Syncing clips based on audio waveform?

Emir
In reply to this post by artwait
yes ! you can do it with a new filter.
https://www.emirchouchane.com/async-v0.1-ffmpeg/



--
Sent from: http://www.ffmpeg-archive.org/
_______________________________________________
ffmpeg-user mailing list
[hidden email]
http://ffmpeg.org/mailman/listinfo/ffmpeg-user

To unsubscribe, visit link above, or email
[hidden email] with subject "unsubscribe".
Reply | Threaded
Open this post in threaded view
|

Re: Syncing clips based on audio waveform?

Carl Eugen Hoyos-2
2017-10-05 21:10 GMT+02:00 Emir <[hidden email]>:
> yes ! you can do it with a new filter.
> https://www.emirchouchane.com/async-v0.1-ffmpeg/

Is there a reason you don't send a patch (made with
git format-patch) with your new filter to the FFmpeg
development mailing list?

Thank you, Carl Eugen
_______________________________________________
ffmpeg-user mailing list
[hidden email]
http://ffmpeg.org/mailman/listinfo/ffmpeg-user

To unsubscribe, visit link above, or email
[hidden email] with subject "unsubscribe".