Resuming ffmpeg encode after interrupt

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

Resuming ffmpeg encode after interrupt

jd1008
When the encoding takes a very long time (> 2 hours), I
often will stop it (^z), syn the FS, suspend the computer to disk (s2d),
and that will power it off.
Now /SOMETIMES/ when I resume, the computer decides to boot as if it was
NOT suspended to disk. This leaves the ffmpeg encode partially done.

So, instead of restarting from the very beginning, is there a way to resume
the encode based on the size of the output file?
I  mean that ffmpeg should be able to know the time offset of the very
last frame encoded. Is this true?
If true, then ffmpeg should be able to continue from the source video file
by starting at the frame (based on the frame's time offset) that follows
the last frame encoded.
I am not an audio video guru. I am just thinking out loud to try to
clarify my question.

Thanx for any help in this.


_______________________________________________
ffmpeg-user mailing list
[hidden email]
http://ffmpeg.org/mailman/listinfo/ffmpeg-user

To unsubscribe, visit link above, or email
[hidden email] with subject "unsubscribe".
Reply | Threaded
Open this post in threaded view
|

Re: Resuming ffmpeg encode after interrupt

Moritz Barsnick
On Thu, Jun 08, 2017 at 21:24:52 -0600, JD wrote:
> So, instead of restarting from the very beginning, is there a way to resume
> the encode based on the size of the output file?
> I  mean that ffmpeg should be able to know the time offset of the very
> last frame encoded. Is this true?

Well, for most common codecs, it isn't just about frame for frame.
Codec compression often refers to frames before and after, and there
are calculations going on all along (e.g. for average bitrate) which
would be lost. That said, one *could* probably come up with codec
implementations which can become largely stateless at some point,
needing to remember or recover only a few parameters (HW encoders
*probably* work that way), but I don't know about actual cut-points.
These implementations may or may not exist, but it's just not that
easy. And ffmpeg isn't capable of keeping its encoding state, as far as
I know. (Intra-only codecs may be different, as each frame is encoded
for itself. But you still need to know about metadata, syncing with
audio, and so on. Video only: You can just continue and attach where
you left off.)

> I am not an audio video guru. I am just thinking out loud to try to
> clarify my question.

What you are trying to do it to suspend and resume a process which
needs to actually *go away* inbetween. There do exist methods for that,
often called "checkpointing" (or "checkpoint/restore"). For Linux, the
tool "criu" comes to mind:

https://criu.org/

The tool is maintained and is actually supposed to work quite well. I
haven't used it in many years though.

Your use case is described here:
https://criu.org/Usage_scenarios#Snapshots_of_apps

I think you need to scan the docs and the Wiki for understanding how to
use it. Please do let us know how it works for you!

Cheers,
Moritz
_______________________________________________
ffmpeg-user mailing list
[hidden email]
http://ffmpeg.org/mailman/listinfo/ffmpeg-user

To unsubscribe, visit link above, or email
[hidden email] with subject "unsubscribe".
Reply | Threaded
Open this post in threaded view
|

Re: Resuming ffmpeg encode after interrupt

jd1008


On 06/09/2017 01:25 AM, Moritz Barsnick wrote:

> On Thu, Jun 08, 2017 at 21:24:52 -0600, JD wrote:
>> So, instead of restarting from the very beginning, is there a way to resume
>> the encode based on the size of the output file?
>> I  mean that ffmpeg should be able to know the time offset of the very
>> last frame encoded. Is this true?
> Well, for most common codecs, it isn't just about frame for frame.
> Codec compression often refers to frames before and after, and there
> are calculations going on all along (e.g. for average bitrate) which
> would be lost. That said, one *could* probably come up with codec
> implementations which can become largely stateless at some point,
> needing to remember or recover only a few parameters (HW encoders
> *probably* work that way), but I don't know about actual cut-points.
> These implementations may or may not exist, but it's just not that
> easy. And ffmpeg isn't capable of keeping its encoding state, as far as
> I know. (Intra-only codecs may be different, as each frame is encoded
> for itself. But you still need to know about metadata, syncing with
> audio, and so on. Video only: You can just continue and attach where
> you left off.)
>
>> I am not an audio video guru. I am just thinking out loud to try to
>> clarify my question.
> What you are trying to do it to suspend and resume a process which
> needs to actually *go away* inbetween. There do exist methods for that,
> often called "checkpointing" (or "checkpoint/restore"). For Linux, the
> tool "criu" comes to mind:
>
> https://criu.org/
>
> The tool is maintained and is actually supposed to work quite well. I
> haven't used it in many years though.
>
> Your use case is described here:
> https://criu.org/Usage_scenarios#Snapshots_of_apps
>
> I think you need to scan the docs and the Wiki for understanding how to
> use it. Please do let us know how it works for you!
>
> Cheers,
> Moritz
Thank you Moritz.
I did find and install criu. So, the scenario I would use it is as follows:

1. I want to suspend the system to disk (i.e. to swap space).
2. Before I do that, I stop the encode/decode with Control-Z
3. I run criu -t <PID>
4. I suspend the system to disk

Some hours later,

I power up and boot same kernel I was running.
USUALLY it will automagically see that a full memory image dump is saved
on the swap device.
It proceeds to decompress it and load it into RAM, and restores the
CONTEXTS of all CPUS to their
state at time of suspend.
Systems resumes where it left off.
I restore the process
criu restore -d -D <The_Dir_Where_criu_Files_Are> -t <PID>

Now I am not sure that the system will ALWAYS resume from suspend-to-disk.
Sometimes, for reasons totally opaque to me, it decided not to resume, but
do a normal boot. All processes that WERE running at time of system suspend,
are lost, and so are the data they were manipulating.
In such cases, I am not certain the criu will work, because the PID to
restore
may clash with that of a new process that is running but is a totally
different
process - such  as a browser, or an email client ....etc.

Thanx,

JD
_______________________________________________
ffmpeg-user mailing list
[hidden email]
http://ffmpeg.org/mailman/listinfo/ffmpeg-user

To unsubscribe, visit link above, or email
[hidden email] with subject "unsubscribe".