Smooth FPS change via ffmpeg

classic Classic list List threaded Threaded
13 messages Options
Reply | Threaded
Open this post in threaded view
|

Smooth FPS change via ffmpeg

RazrFalcon
Hi!

I have a 240FPS video made by GoPro and I'm looking for a way to gradually
change a video framerate. If I want to simply slow down a video I can use:

ffmpeg -i raw.mp4 -filter_complex "[v:0]setpts=8*PTS" -r 30 out.mp4

and it works fine. But when I'm trying to use a "variable" framerate like
this:

ffmpeg -i raw.mp4 -filter_complex "[v:0]setpts='lerp(2,8,T/5)*PTS'" -r 30
out.mp4

(5 is a video duration in seconds)

the resulting video stutters a lot.

Is there are a correct way to smoothly change the framerate using some
curve/function?

Thanks.
_______________________________________________
ffmpeg-user mailing list
[hidden email]
https://ffmpeg.org/mailman/listinfo/ffmpeg-user

To unsubscribe, visit link above, or email
[hidden email] with subject "unsubscribe".
Reply | Threaded
Open this post in threaded view
|

Re: Smooth FPS change via ffmpeg

Nicolas George
RazrFalcon (12019-09-23):

> and it works fine. But when I'm trying to use a "variable" framerate like
> this:
>
> ffmpeg -i raw.mp4 -filter_complex "[v:0]setpts='lerp(2,8,T/5)*PTS'" -r 30
> out.mp4
>
> (5 is a video duration in seconds)
>
> the resulting video stutters a lot.
>
> Is there are a correct way to smoothly change the framerate using some
> curve/function?
This is a math issue.

Imagine you want to slow down the second half of a 20 seconds video. The
formula for the first half is just t' = t. The formula for the second
half would, naively, be t' = 2*t. But try to plot this, for example in
gnuplot:

plot [0:20] x < 10 ? x : 2 * x

It does not match at the stitch point. Instead, the second formula needs
to be t'=2*t-10:

plot [0:20] x < 10 ? x : 2 * x - 10

If we neglect the fact that frames are discrete, we can say that the
frame rate is the speed of the frames timestamps. In mathematical terms,
the derivative. The derivative of a linear function is easy. Other kinds
are more tricky.

If you have a constant frame rate, the timestamps will be affine, but
you can still choose the constant offset. If you want to connect two
segments with constant frame rate, you need to adjust the offsets so
that they match. Hence the -10 in the example.

If the frame rate varies continuously, you need to do the calculus.
Fortunately, with a linear interpolation, the calculus is easy.

For the same reason:

        sin( (440 + t / 10 * 440) * t)

will NOT give a sine sound going smoothly from 440 Hz to 880 Hz.

Regards,

--
  Nicolas George

_______________________________________________
ffmpeg-user mailing list
[hidden email]
https://ffmpeg.org/mailman/listinfo/ffmpeg-user

To unsubscribe, visit link above, or email
[hidden email] with subject "unsubscribe".

signature.asc (849 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Smooth FPS change via ffmpeg

RazrFalcon
Thanks for a detailed explanation, but I'm still not sure how this should
be implemented.
I understand that I should accumulate/calculate the offset for each PTS,
but I do not understand how.

On Mon, 23 Sep 2019 at 17:06, Nicolas George <[hidden email]> wrote:

> RazrFalcon (12019-09-23):
> > and it works fine. But when I'm trying to use a "variable" framerate like
> > this:
> >
> > ffmpeg -i raw.mp4 -filter_complex "[v:0]setpts='lerp(2,8,T/5)*PTS'" -r 30
> > out.mp4
> >
> > (5 is a video duration in seconds)
> >
> > the resulting video stutters a lot.
> >
> > Is there are a correct way to smoothly change the framerate using some
> > curve/function?
>
> This is a math issue.
>
> Imagine you want to slow down the second half of a 20 seconds video. The
> formula for the first half is just t' = t. The formula for the second
> half would, naively, be t' = 2*t. But try to plot this, for example in
> gnuplot:
>
> plot [0:20] x < 10 ? x : 2 * x
>
> It does not match at the stitch point. Instead, the second formula needs
> to be t'=2*t-10:
>
> plot [0:20] x < 10 ? x : 2 * x - 10
>
> If we neglect the fact that frames are discrete, we can say that the
> frame rate is the speed of the frames timestamps. In mathematical terms,
> the derivative. The derivative of a linear function is easy. Other kinds
> are more tricky.
>
> If you have a constant frame rate, the timestamps will be affine, but
> you can still choose the constant offset. If you want to connect two
> segments with constant frame rate, you need to adjust the offsets so
> that they match. Hence the -10 in the example.
>
> If the frame rate varies continuously, you need to do the calculus.
> Fortunately, with a linear interpolation, the calculus is easy.
>
> For the same reason:
>
>         sin( (440 + t / 10 * 440) * t)
>
> will NOT give a sine sound going smoothly from 440 Hz to 880 Hz.
>
> Regards,
>
> --
>   Nicolas George
> _______________________________________________
> ffmpeg-user mailing list
> [hidden email]
> https://ffmpeg.org/mailman/listinfo/ffmpeg-user
>
> To unsubscribe, visit link above, or email
> [hidden email] with subject "unsubscribe".
_______________________________________________
ffmpeg-user mailing list
[hidden email]
https://ffmpeg.org/mailman/listinfo/ffmpeg-user

To unsubscribe, visit link above, or email
[hidden email] with subject "unsubscribe".
Reply | Threaded
Open this post in threaded view
|

Re: Smooth FPS change via ffmpeg

JackDesBwa
> Thanks for a detailed explanation, but I'm still not sure how this should
> be implemented.
> I understand that I should accumulate/calculate the offset for each PTS,
> but I do not understand how.
>

I do not fully understand neither, but I become interested in complex
animation curves too, thanks to your question.
Speeding-up or slowing-down gradually can be useful in other situations.

> If we neglect the fact that frames are discrete, we can say that the
> > frame rate is the speed of the frames timestamps. In mathematical terms,
> > the derivative. The derivative of a linear function is easy. Other kinds
> > are more tricky.
>

I think it is based on the fact that you define your transformation in
terms of framerate (frame per second) and you express it in terms of frame
position, and the two have integral/derivative relationship.

By "cheating", I would say that the integral of a linear curve is a
parabola [ T**2 ] that you want to start at 2 at the beginning [ T**2+2 ]
and finish at 8 at the end [ (8-2)*T**2+2 ] and stretch on 5 seconds [
(8-2)*T**2/5**2+2 ] but then I am not sure how to apply this.
Also, if it is the right formula, I do not understand (well I am tired now)
how to come to this in the general case from the expression of the
framerate curve (0.5-t*3/40)*original_framerate.

JackDesBwa
_______________________________________________
ffmpeg-user mailing list
[hidden email]
https://ffmpeg.org/mailman/listinfo/ffmpeg-user

To unsubscribe, visit link above, or email
[hidden email] with subject "unsubscribe".
Reply | Threaded
Open this post in threaded view
|

Re: Smooth FPS change via ffmpeg

Nicolas George
JackDesBwa (12019-09-23):
> Speeding-up or slowing-down gradually can be useful in other situations.

This is indeed a very frequent problem. I encountered it first when
doing tone slides.

> I think it is based on the fact that you define your transformation in
> terms of framerate (frame per second) and you express it in terms of frame
> position, and the two have integral/derivative relationship.

Exactly.

> By "cheating", I would say that the integral of a linear curve is a
> parabola [ T**2 ] that you want to start at 2 at the beginning [ T**2+2 ]
> and finish at 8 at the end [ (8-2)*T**2+2 ] and stretch on 5 seconds [
> (8-2)*T**2/5**2+2 ] but then I am not sure how to apply this.
> Also, if it is the right formula, I do not understand (well I am tired now)
> how to come to this in the general case from the expression of the
> framerate curve (0.5-t*3/40)*original_framerate.

That could work, but I really would discourage from using that kind of
trial-and-error techniques because they do not scale to more complex
cases. Better go for the proper formalism, it is not very complex, a
good high-school graduate with science major can do it.

Your question is: at time t, you want to know what frame you are f(t)
(or the other way around, it does not matter much).

With constant frame rate, i.e. constant speed v, that is easy:

        f(t) = v × t + f(0)

But if the speed is itself variable, v(t), then the equation is NOT:

        f(t) = v(t) × t + f(0)

The relation is in fact that:

        d/dt f(t) = v(t)

and it becomes:

        f(t) = ∫ v(t)

So, to compute the timestamp of a frame with variable speed:

* Express your frame rate as a complete formula: t → v

* Integrate it: t → f.

* Find the reciprocal: f → t.

I cannot help more without knowing the extent of your background in
calculus.

Also, I WILL NOT help somebody who neglected to read the guidelines of
this mailing-list and follow them, including the bit about top-posting,
looking it up if necessary.

Regards,

--
  Nicolas George

_______________________________________________
ffmpeg-user mailing list
[hidden email]
https://ffmpeg.org/mailman/listinfo/ffmpeg-user

To unsubscribe, visit link above, or email
[hidden email] with subject "unsubscribe".

signature.asc (849 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Smooth FPS change via ffmpeg

Michael Koch
Nicolas,

> So, to compute the timestamp of a frame with variable speed:
>
> * Express your frame rate as a complete formula: t → v

Is t the time in the input video, or the time in  the output video?

Michael

_______________________________________________
ffmpeg-user mailing list
[hidden email]
https://ffmpeg.org/mailman/listinfo/ffmpeg-user

To unsubscribe, visit link above, or email
[hidden email] with subject "unsubscribe".
Reply | Threaded
Open this post in threaded view
|

Re: Smooth FPS change via ffmpeg

Rodney Baker-2
In reply to this post by Nicolas George
Forgive the naive response but...
Isn't FPS mostly a matter of playback via a viewer (in earlier days mostly
a projector)?

What I mean by this is that one second is one second no matter how we cut
it so we have a finite number of frames we can place into that second.  If
the FPS is 24 we only get 24 frames to work with... 240 FPS gives us 240
frames to work with.  Apologies for stating the obvious here.

To me this seems to be the hint of a way forward.
We might for instance look at the playback software/device and ask, "How do
we manipulate the speed of the playback?"
Then the frames all stay the same... just the playback speed of those
frames changes.

This does still matter for actual frames of course in that if we start with
24 frames we can't pull out new.additional frames out of nowhere without
some method of creating those additional frames.  When starting with 240
frames it could be somewhat easier in that we just have to ignore or leave
out some selection of frames although there are potential issues with loss
of information from the discarded frames.  Where things get difficult is
when a selected frame doesn't contain the imagery we need.  For example
perhaps there are blurs present that we can't easily remove or... we need
blurs where none are present.

Just a recurring thought I have whenever topic of changing FPS arise.
Apologies if this is slightly off topic.

(First time posting but long time reading.  I enjoy all the information on
this mailing list!)
- Rodney
_______________________________________________
ffmpeg-user mailing list
[hidden email]
https://ffmpeg.org/mailman/listinfo/ffmpeg-user

To unsubscribe, visit link above, or email
[hidden email] with subject "unsubscribe".
Reply | Threaded
Open this post in threaded view
|

Re: Smooth FPS change via ffmpeg

Carl Eugen Hoyos-2
In reply to this post by RazrFalcon
Am Mo., 23. Sept. 2019 um 14:52 Uhr schrieb RazrFalcon <[hidden email]>:

> I have a 240FPS video made by GoPro and I'm looking for a way to gradually
> change a video framerate. If I want to simply slow down a video I can use:
>
> ffmpeg -i raw.mp4 -filter_complex "[v:0]setpts=8*PTS" -r 30 out.mp4
>
> and it works fine. But when I'm trying to use a "variable" framerate like
> this:
>
> ffmpeg -i raw.mp4 -filter_complex "[v:0]setpts='lerp(2,8,T/5)*PTS'" -r 30
> out.mp4

(Complete, uncut console output missing.)

"-r 30" forces constant framerate output.
Additionally, our mp4 muxer does not support variable frame rate,
you would have to use another muxer.

(This assumes you really want a variable framerate output, not
a "variable framerate")

Carl Eugen
_______________________________________________
ffmpeg-user mailing list
[hidden email]
https://ffmpeg.org/mailman/listinfo/ffmpeg-user

To unsubscribe, visit link above, or email
[hidden email] with subject "unsubscribe".
Reply | Threaded
Open this post in threaded view
|

Re: Smooth FPS change via ffmpeg

Michael Koch
In reply to this post by Nicolas George

> So, to compute the timestamp of a frame with variable speed:
>
> * Express your frame rate as a complete formula: t → v
>
> * Integrate it: t → f.
>
> * Find the reciprocal: f → t.

Let's give it a try. My input video has framerate=20 and length=10s.

Let's change the framerate linearly from 20 at the beginning to 10 at
the end:
v(t) = 20 - t

After integrating we get  f = 20 * t - 0.5 * t^2

The inverse function is  t = 20 - sqrt(400 - 2 * f)

ffmpeg -f lavfi -i testsrc2=size=vga:duration=10:rate=20 -lavfi
"setpts='(20-sqrt(400-2*N))/TB' " -y out.mp4

The resulting video gets slower towards the end, but the length is
18.95s and that can't be correct. With a constant framerate of 10 the
length would be 20s. The length should be somewhere wetween 10s and 20s,
but not so close to 20s. Something must be wrong.

This is an interesting question and I would highly appreciate if someone
could post a working example.

Michael
_______________________________________________
ffmpeg-user mailing list
[hidden email]
https://ffmpeg.org/mailman/listinfo/ffmpeg-user

To unsubscribe, visit link above, or email
[hidden email] with subject "unsubscribe".
Reply | Threaded
Open this post in threaded view
|

Re: Smooth FPS change via ffmpeg

Carl Eugen Hoyos-2
Am Mo., 23. Sept. 2019 um 23:04 Uhr schrieb Michael Koch
<[hidden email]>:

> ffmpeg -f lavfi -i testsrc2=size=vga:duration=10:rate=20 -lavfi
> "setpts='(20-sqrt(400-2*N))/TB' " -y out.mp4

(Does this syntax really work?)

As said before, our mp4 muxer does not (correctly) support
constant frame rate.

Carl Eugen
_______________________________________________
ffmpeg-user mailing list
[hidden email]
https://ffmpeg.org/mailman/listinfo/ffmpeg-user

To unsubscribe, visit link above, or email
[hidden email] with subject "unsubscribe".
Reply | Threaded
Open this post in threaded view
|

Re: Smooth FPS change via ffmpeg

Michael Koch
Am 23.09.2019 um 23:07 schrieb Carl Eugen Hoyos:
> Am Mo., 23. Sept. 2019 um 23:04 Uhr schrieb Michael Koch
> <[hidden email]>:
>
>> ffmpeg -f lavfi -i testsrc2=size=vga:duration=10:rate=20 -lavfi
>> "setpts='(20-sqrt(400-2*N))/TB' " -y out.mp4
> (Does this syntax really work?)
>
> As said before, our mp4 muxer does not (correctly) support
> constant frame rate.

you mean it doesn't support variable framerate, right?

Well, the syntax gives no error message, the resulting video has a
constant framerate of 20 and the speed changes from normal to slow
motion. But the length is not as expected.

Michael

_______________________________________________
ffmpeg-user mailing list
[hidden email]
https://ffmpeg.org/mailman/listinfo/ffmpeg-user

To unsubscribe, visit link above, or email
[hidden email] with subject "unsubscribe".
Reply | Threaded
Open this post in threaded view
|

Re: Smooth FPS change via ffmpeg

Carl Eugen Hoyos-2
Am Mo., 23. Sept. 2019 um 23:19 Uhr schrieb Michael Koch
<[hidden email]>:

>
> Am 23.09.2019 um 23:07 schrieb Carl Eugen Hoyos:
> > Am Mo., 23. Sept. 2019 um 23:04 Uhr schrieb Michael Koch
> > <[hidden email]>:
> >
> >> ffmpeg -f lavfi -i testsrc2=size=vga:duration=10:rate=20 -lavfi
> >> "setpts='(20-sqrt(400-2*N))/TB' " -y out.mp4
> > (Does this syntax really work?)
> >
> > As said before, our mp4 muxer does not (correctly) support
> > constant frame rate.
>
> you mean it doesn't support variable framerate, right?

Correct.

> Well, the syntax gives no error message,

True.

> the resulting video has a constant framerate of 20

This is enforced for the mp4 muxer.

> and the speed changes from normal to slow
> motion. But the length is not as expected.

Carl Eugen
_______________________________________________
ffmpeg-user mailing list
[hidden email]
https://ffmpeg.org/mailman/listinfo/ffmpeg-user

To unsubscribe, visit link above, or email
[hidden email] with subject "unsubscribe".
Reply | Threaded
Open this post in threaded view
|

Re: Smooth FPS change via ffmpeg

Michael Koch
In reply to this post by RazrFalcon
Am 23.09.2019 um 14:45 schrieb RazrFalcon:

> Hi!
>
> I have a 240FPS video made by GoPro and I'm looking for a way to gradually
> change a video framerate. If I want to simply slow down a video I can use:
>
> ffmpeg -i raw.mp4 -filter_complex "[v:0]setpts=8*PTS" -r 30 out.mp4
>
> and it works fine. But when I'm trying to use a "variable" framerate like
> this:
>
> ffmpeg -i raw.mp4 -filter_complex "[v:0]setpts='lerp(2,8,T/5)*PTS'" -r 30
> out.mp4
>
> (5 is a video duration in seconds)
>
> the resulting video stutters a lot.
>
> Is there are a correct way to smoothly change the framerate using some
> curve/function?

Have a look at this posting where a totally different approach is described:
http://ffmpeg.org/pipermail/ffmpeg-user/2017-June/036310.html

Michael

_______________________________________________
ffmpeg-user mailing list
[hidden email]
https://ffmpeg.org/mailman/listinfo/ffmpeg-user

To unsubscribe, visit link above, or email
[hidden email] with subject "unsubscribe".