Output to S3

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
9 messages Options
Reply | Threaded
Open this post in threaded view
|

Output to S3

Bohdan Fihar
Hi,

How can I output a video file to AWS S3 either public bucket ot Presigned
url ?
I tried
*ffmpeg -i small.mkv -f matroska -movflags frag_keyframe+empty_moov -method
PUT https://[my-bucket-name].s3.us-east-2.amazonaws.com/video/output.mkv
<http://s3.us-east-2.amazonaws.com/video/output.mkv>*

bur it won't work, fails with

*av_interleaved_write_frame(): I/O errorError writing trailer of https...*


--
All the best,

Bogdan
_______________________________________________
ffmpeg-user mailing list
[hidden email]
https://ffmpeg.org/mailman/listinfo/ffmpeg-user

To unsubscribe, visit link above, or email
[hidden email] with subject "unsubscribe".
Reply | Threaded
Open this post in threaded view
|

Re: Output to S3

Moritz Barsnick
Hi Bohdan/Bogdan,

On Tue, Sep 08, 2020 at 09:39:26 +0300, Bohdan Fihar wrote:

> How can I output a video file to AWS S3 either public bucket ot Presigned
> url ?
> I tried
> *ffmpeg -i small.mkv -f matroska -movflags frag_keyframe+empty_moov -method
> PUT https://[my-bucket-name].s3.us-east-2.amazonaws.com/video/output.mkv
> <http://s3.us-east-2.amazonaws.com/video/output.mkv>*
>
> bur it won't work, fails with
>
> *av_interleaved_write_frame(): I/O errorError writing trailer of https...*

I have no experience with S3, but this Stackoverflow thread seems to
suggest you need to use the "aws s3 cp" command:

https://stackoverflow.com/questions/54736246/stream-ffmpeg-transcoding-result-to-s3

I *guess* ffmpeg assumes it can seek on HTTP PUT output URLs, which
doesn't work on AWS S3. Instead, when piping with above command, ffmpeg
doesn't use this assumption. But I may be wrong.

Please try this.

Cheers,
Moritz
_______________________________________________
ffmpeg-user mailing list
[hidden email]
https://ffmpeg.org/mailman/listinfo/ffmpeg-user

To unsubscribe, visit link above, or email
[hidden email] with subject "unsubscribe".
Reply | Threaded
Open this post in threaded view
|

Re: Output to S3

Bohdan Fihar
Hi Moritz. Thanks for the answer.
I can't rely on "aws s3 cp", since it can be not installed on a machine
where I will run ffmpeg.

Do you know what this error means ?
[image: image.png]

On Tue, 8 Sep 2020 at 12:10, Moritz Barsnick <[hidden email]> wrote:

> Hi Bohdan/Bogdan,
>
> On Tue, Sep 08, 2020 at 09:39:26 +0300, Bohdan Fihar wrote:
> > How can I output a video file to AWS S3 either public bucket ot Presigned
> > url ?
> > I tried
> > *ffmpeg -i small.mkv -f matroska -movflags frag_keyframe+empty_moov
> -method
> > PUT https://[my-bucket-name].s3.us-east-2.amazonaws.com/video/output.mkv
> > <http://s3.us-east-2.amazonaws.com/video/output.mkv>*
> >
> > bur it won't work, fails with
> >
> > *av_interleaved_write_frame(): I/O errorError writing trailer of
> https...*
>
> I have no experience with S3, but this Stackoverflow thread seems to
> suggest you need to use the "aws s3 cp" command:
>
>
> https://stackoverflow.com/questions/54736246/stream-ffmpeg-transcoding-result-to-s3
>
> I *guess* ffmpeg assumes it can seek on HTTP PUT output URLs, which
> doesn't work on AWS S3. Instead, when piping with above command, ffmpeg
> doesn't use this assumption. But I may be wrong.
>
> Please try this.
>
> Cheers,
> Moritz
> _______________________________________________
> ffmpeg-user mailing list
> [hidden email]
> https://ffmpeg.org/mailman/listinfo/ffmpeg-user
>
> To unsubscribe, visit link above, or email
> [hidden email] with subject "unsubscribe".


--
All the best,

Bohdan

_______________________________________________
ffmpeg-user mailing list
[hidden email]
https://ffmpeg.org/mailman/listinfo/ffmpeg-user

To unsubscribe, visit link above, or email
[hidden email] with subject "unsubscribe".

image.png (17K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Output to S3

Carl Zwanzig
On 9/8/2020 6:07 AM, Bohdan Fihar wrote:
> I can't rely on "aws s3 cp", since it can be not installed on a machine
> where I will run ffmpeg.

A couple of things-
Isn't or can not be?
Please just post the error text, not an image of it (actually, you ought to
be posting the complete/unedited command and output).

(and do not top-post on this list)

z!
_______________________________________________
ffmpeg-user mailing list
[hidden email]
https://ffmpeg.org/mailman/listinfo/ffmpeg-user

To unsubscribe, visit link above, or email
[hidden email] with subject "unsubscribe".
Reply | Threaded
Open this post in threaded view
|

Re: Output to S3

kumowoon1025
In reply to this post by Bohdan Fihar
Hi,
> I can't rely on "aws s3 cp", since it can be not installed on a machine
> where I will run ffmpeg.
Is setting up a sort of proxy server that will intermediate and upload to s3 an option?

> Do you know what this error means ?
> [image: image.png]

That is a very small window of view into the error you got, I'd try using the send_expect_100 option and confirming which ssl library is used, no real basis but what I might start with when I do the throw all of it on the wall and see what sticks routine.

Regards,
Ted Park

_______________________________________________
ffmpeg-user mailing list
[hidden email]
https://ffmpeg.org/mailman/listinfo/ffmpeg-user

To unsubscribe, visit link above, or email
[hidden email] with subject "unsubscribe".
Reply | Threaded
Open this post in threaded view
|

Re: Output to S3

Bohdan Fihar
>  Is setting up a sort of proxy server that will intermediate and upload
to s3 an option?
It might be the option, but it will be the last option I'd like to accept.
I'm seeking for more simple solution.
I'm afraid that I'll stuck on sending a video to that server too.

>  That is a very small window of view into the error you got, I'd try
using the send_expect_100 option and confirming which ssl library is used,
no real basis but what I might start with when I do the throw all of it on
the wall and see what sticks routine.
The error says:
Error in the push function.
av_interleaved_write_frame(): I/O error
Error writing trailer of
https://bogdan-public.s3.us-east-2.amazonaws.com/video/output.ts: I/O error
frame=  218 fps=184 q=10.9 Lsize=     404kB time=00:00:07.17 bitrate=
461.4kbits/s speed=6.06x
video:352kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB
muxing overhead: 14.843175%
[tls @ 000001b41de52940] The specified session has been invalidated for
some reason.
[tls @ 000001b41de52940] Error in the pull function.
[https @ 000001b41de52840] URL read error: I/O error
Conversion failed!

I tried several video formats with this command:
ffmpeg -i sample_640x360.ts -method PUT
https://bogdan-public.s3.us-east-2.amazonaws.com/video/output.ts

The bucket (its name is changed here) is public and I was able to send data
to it by Http Put request from the C# code sample.

On Tue, 8 Sep 2020 at 20:07, Edward Park <[hidden email]> wrote:

> Hi,
> > I can't rely on "aws s3 cp", since it can be not installed on a machine
> > where I will run ffmpeg.
> Is setting up a sort of proxy server that will intermediate and upload to
> s3 an option?
>
> > Do you know what this error means ?
> > [image: image.png]
>
> That is a very small window of view into the error you got, I'd try using
> the send_expect_100 option and confirming which ssl library is used, no
> real basis but what I might start with when I do the throw all of it on the
> wall and see what sticks routine.
>
> Regards,
> Ted Park
>
> _______________________________________________
> ffmpeg-user mailing list
> [hidden email]
> https://ffmpeg.org/mailman/listinfo/ffmpeg-user
>
> To unsubscribe, visit link above, or email
> [hidden email] with subject "unsubscribe".



--
All the best,

Bohdan
_______________________________________________
ffmpeg-user mailing list
[hidden email]
https://ffmpeg.org/mailman/listinfo/ffmpeg-user

To unsubscribe, visit link above, or email
[hidden email] with subject "unsubscribe".
Reply | Threaded
Open this post in threaded view
|

Re: Output to S3

Bohdan Fihar
In reply to this post by Carl Zwanzig
Hi Carl,
This is the whole output:

d:\Programs\ffmpeg-20200831-4a11a6f-win64-static\bin>
ffmpeg -i sample_640x360.ts -method PUT
https://bogdan-public.s3.us-east-2.amazonaws.com/video/output.ts
ffmpeg version git-2020-08-31-4a11a6f Copyright (c) 2000-2020 the FFmpeg
developers
  built with gcc 10.2.1 (GCC) 20200805
  configuration: --enable-gpl --enable-version3 --enable-sdl2
--enable-fontconfig --enable-gnutls --enable-iconv --enable-libass
--enable-libdav1d --enable-libbluray --enable-libfreetype
--enable-libmp3lame --enable-libopencore-amrnb --enable-libopencore-amrwb
--enable-libopenjpeg --enable-libopus --enable-libshine --enable-libsnappy
--enable-libsoxr --enable-libsrt --enable-libtheora --enable-libtwolame
--enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx264
--enable-libx265 --enable-libxml2 --enable-libzimg --enable-lzma
--enable-zlib --enable-gmp --enable-libvidstab --enable-libvmaf
--enable-libvorbis --enable-libvo-amrwbenc --enable-libmysofa
--enable-libspeex --enable-libxvid --enable-libaom --enable-libgsm
--enable-librav1e --enable-libsvtav1 --disable-w32threads --enable-libmfx
--enable-ffnvcodec --enable-cuda-llvm --enable-cuvid --enable-d3d11va
--enable-nvenc --enable-nvdec --enable-dxva2 --enable-avisynth
--enable-libopenmpt --enable-amf
  libavutil      56. 58.100 / 56. 58.100
  libavcodec     58.101.101 / 58.101.101
  libavformat    58. 51.101 / 58. 51.101
  libavdevice    58. 11.101 / 58. 11.101
  libavfilter     7. 87.100 /  7. 87.100
  libswscale      5.  8.100 /  5.  8.100
  libswresample   3.  8.100 /  3.  8.100
  libpostproc    55.  8.100 / 55.  8.100
Input #0, mpegts, from 'sample_640x360.ts':
  Duration: 00:00:13.35, start: 1.433367, bitrate: 387 kb/s
  Program 1
    Metadata:
      service_name    : Service01
      service_provider: FFmpeg
    Stream #0:0[0x100]: Video: mpeg2video (Main) ([2][0][0][0] / 0x0002),
yuv420p(tv, progressive), 640x360 [SAR 1:1 DAR 16:9], 29.97 fps, 29.97 tbr,
90k tbn, 59.94 tbc
    Side data:
      cpb: bitrate max/min/avg: 0/0/0 buffer size: 49152 vbv_delay: N/A
Stream mapping:
  Stream #0:0 -> #0:0 (mpeg2video (native) -> mpeg2video (native))
Press [q] to stop, [?] for help
Output #0, mpegts, to '
https://bogdan-public.s3.us-east-2.amazonaws.com/video/output.ts':
  Metadata:
    encoder         : Lavf58.51.101
    Stream #0:0: Video: mpeg2video (Main), yuv420p, 640x360 [SAR 1:1 DAR
16:9], q=2-31, 200 kb/s, 29.97 fps, 90k tbn, 29.97 tbc
    Metadata:
      encoder         : Lavc58.101.101 mpeg2video
    Side data:
      cpb: bitrate max/min/avg: 0/0/200000 buffer size: 0 vbv_delay: N/A
frame=   17 fps=0.0 q=7.6 size=      93kB time=00:00:00.46
bitrate=1622.7kbits/s speed=0.9frame=  148 fps=147 q=12.7 size=     310kB
time=00:00:04.83 bitrate= 525.4kbits/s speed=4.[tls @ 00000279bdb62940] Error
in the push function.
av_interleaved_write_frame(): I/O error
Error writing trailer of
https://bogdan-public.s3.us-east-2.amazonaws.com/video/output.ts: I/O error
frame=  177 fps=156 q=13.1 Lsize=     349kB time=00:00:05.80 bitrate=
491.9kbits/s speed= 5.1x
video:300kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB
muxing overhead: 16.271618%
[tls @ 00000279bdb62940] The specified session has been invalidated for
some reason.
[tls @ 00000279bdb62940] Error in the pull function.
[https @ 00000279bdb62840] URL read error: I/O error
Conversion failed!

I tried several video formats.
The video files I downloaded from https://filesamples.com/.
AWS will not be installed on the machine.

--
All the best,

Bohdan
_______________________________________________
ffmpeg-user mailing list
[hidden email]
https://ffmpeg.org/mailman/listinfo/ffmpeg-user

To unsubscribe, visit link above, or email
[hidden email] with subject "unsubscribe".
Reply | Threaded
Open this post in threaded view
|

Re: Output to S3

kumowoon1025
In reply to this post by Bohdan Fihar
Hi,

>> Is setting up a sort of proxy server that will intermediate and upload
>> to s3 an option?
> It might be the option, but it will be the last option I'd like to accept.
> I'm seeking for more simple solution.
> I'm afraid that I'll stuck on sending a video to that server too.
Is the aversion to using the official s3 utilities due to resources and lack of access, or the additional integration and associated learning curve that it brings? If it's the latter, and you can configure the server to your needs, I think you might find any of the solutions that mount s3-compatible buckets as network shares, or even local fuse filesystems to your liking. Performance is not the best, and you will see request graph spike like mad, but it is in my opinion the most simple solution. (Since the s3 storage will be just like any other local path on the machine) Worth some consideration imo.

>> That is a very small window of view into the error you got, I'd try
>> using the send_expect_100 option and confirming which ssl library is used,
>> no real basis but what I might start with when I do the throw all of it on
>> the wall and see what sticks routine.
> The error says:
> Error in the push function.
> av_interleaved_write_frame(): I/O error
> Error writing trailer of
> https://bogdan-public.s3.us-east-2.amazonaws.com/video/output.ts: I/O error
> frame=  218 fps=184 q=10.9 Lsize=     404kB time=00:00:07.17 bitrate=
> 461.4kbits/s speed=6.06x
> video:352kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB
> muxing overhead: 14.843175%
> [tls @ 000001b41de52940] The specified session has been invalidated for
> some reason.
> [tls @ 000001b41de52940] Error in the pull function.
> [https @ 000001b41de52840] URL read error: I/O error
> Conversion failed!
I am thinking that the "Error in the push function" is not from FFmpeg. I missed it earlier, the quoting was messed up in my email client, but the only problem FFmpeg seems to be reporting is "I/O error," and consistently before or after sending the main data body. (Let me know if I am wrong about this)

So in addition to the -send_expect_100 true, I would add -multiple_requests false to disable pipelining

> This is the whole output:
>
> d:\Programs\ffmpeg-20200831-4a11a6f-win64-static\bin>
> ffmpeg -i sample_640x360.ts -method PUT
> https://bogdan-public.s3.us-east-2.amazonaws.com/video/output.ts
But the problem is the invalidated session, it suggests that you successfully got a valid session authorized beforehand, if it was invalidated there's no way to know why unless we start from there. Is this command unabridged as well? I think you mentioned you altered the url, but there's no api key or anything I would expect to be in the request header that uploads a file to an endpoint.

> I tried several video formats with this command:
> ffmpeg -i sample_640x360.ts -method PUT
> https://bogdan-public.s3.us-east-2.amazonaws.com/video/output.ts
>
> The bucket (its name is changed here) is public and I was able to send data
> to it by Http Put request from the C# code sample.

I have a feeling the C# code sample was part of a whole AWS S3 C# SDK, handling all AAA with credentials hardcoded in a configuration file or something. Doing the same with a REST api will require more hands-on operation.

Regards,
Ted Park

_______________________________________________
ffmpeg-user mailing list
[hidden email]
https://ffmpeg.org/mailman/listinfo/ffmpeg-user

To unsubscribe, visit link above, or email
[hidden email] with subject "unsubscribe".
Reply | Threaded
Open this post in threaded view
|

Re: Output to S3

Bohdan Fihar
Hi Edward,

No, my C# code doesn't contain anything related ASW S3, it just sends a
file to Url:
        private const string filePath =
@"d:\Programs\ffmpeg-20200831-4a11a6f-win64-static\bin\01.05.flv";

        private static void UploadObject(string url)
        {
            var httpRequest = System.Net.WebRequest.Create(url) as
System.Net.HttpWebRequest;
            httpRequest.Method = "PUT";
            using (var dataStream = httpRequest.GetRequestStream())
            {
                var buffer = new byte[8000];
                using (var fileStream = new System.IO.FileStream(filePath,
System.IO.FileMode.Open, System.IO.FileAccess.Read))
                {
                    int bytesRead = 0;
                    while ((bytesRead = fileStream.Read(buffer, 0,
buffer.Length)) > 0)
                    {
                        dataStream.Write(buffer, 0, bytesRead);
                    }
                }
            }
            var response = httpRequest.GetResponse() as
System.Net.HttpWebResponse;
        }

No credentials required, since it's a public bucket.
By the way, does FFMPEG support sending to HttpS ?

I tried adding these parameters*:-send_expect_100 true* and *-multiple_requests
false,* Also I added Content-Length header:
ffmpeg -i sample_640x360.ts -send_expect_100 true  -multiple_requests false
 -loglevel trace  -method PUT -headers "Content-Length: 646720"
https://bogdan-public.s3.us-east-2.amazonaws.com/video/output.ts

The result is:
[tcp @ 00000205536a5340] Starting connection attempt to 52.219.84.232 port
443
[tcp @ 00000205536a5340] Successfully connected to 52.219.84.232 port 443
[https @ 00000205536a5000] request: PUT /video/output.ts HTTP/1.1
Transfer-Encoding: chunked
User-Agent: Lavf/58.51.101
Accept: */*
Expect: 100-continue
Connection: close
Host: bogdan-public.s3.us-east-2.amazonaws.com
Icy-MetaData: 1
Content-Length: 646720

[https @ 00000205536a5000] header='HTTP/1.1 501 Not Implemented'
[https @ 00000205536a5000] http_code=501
[https @ 00000205536a5000] HTTP error 501 Not Implemented
https://bogdan-public.s3.us-east-2.amazonaws.com/video/output.ts: Server
returned 5XX Server Error reply
[AVIOContext @ 00000205536a7100] Statistics: 577680 bytes read, 2 seeks

Looks like something is missing.


On Wed, 9 Sep 2020 at 12:41, Edward Park <[hidden email]> wrote:

> Hi,
>
> >> Is setting up a sort of proxy server that will intermediate and upload
> >> to s3 an option?
> > It might be the option, but it will be the last option I'd like to
> accept.
> > I'm seeking for more simple solution.
> > I'm afraid that I'll stuck on sending a video to that server too.
> Is the aversion to using the official s3 utilities due to resources and
> lack of access, or the additional integration and associated learning curve
> that it brings? If it's the latter, and you can configure the server to
> your needs, I think you might find any of the solutions that mount
> s3-compatible buckets as network shares, or even local fuse filesystems to
> your liking. Performance is not the best, and you will see request graph
> spike like mad, but it is in my opinion the most simple solution. (Since
> the s3 storage will be just like any other local path on the machine) Worth
> some consideration imo.
>
> >> That is a very small window of view into the error you got, I'd try
> >> using the send_expect_100 option and confirming which ssl library is
> used,
> >> no real basis but what I might start with when I do the throw all of it
> on
> >> the wall and see what sticks routine.
> > The error says:
> > Error in the push function.
> > av_interleaved_write_frame(): I/O error
> > Error writing trailer of
> > https://bogdan-public.s3.us-east-2.amazonaws.com/video/output.ts: I/O
> error
> > frame=  218 fps=184 q=10.9 Lsize=     404kB time=00:00:07.17 bitrate=
> > 461.4kbits/s speed=6.06x
> > video:352kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB
> > muxing overhead: 14.843175%
> > [tls @ 000001b41de52940] The specified session has been invalidated for
> > some reason.
> > [tls @ 000001b41de52940] Error in the pull function.
> > [https @ 000001b41de52840] URL read error: I/O error
> > Conversion failed!
> I am thinking that the "Error in the push function" is not from FFmpeg. I
> missed it earlier, the quoting was messed up in my email client, but the
> only problem FFmpeg seems to be reporting is "I/O error," and consistently
> before or after sending the main data body. (Let me know if I am wrong
> about this)
>
> So in addition to the -send_expect_100 true, I would add
> -multiple_requests false to disable pipelining
>
> > This is the whole output:
> >
> > d:\Programs\ffmpeg-20200831-4a11a6f-win64-static\bin>
> > ffmpeg -i sample_640x360.ts -method PUT
> > https://bogdan-public.s3.us-east-2.amazonaws.com/video/output.ts
> But the problem is the invalidated session, it suggests that you
> successfully got a valid session authorized beforehand, if it was
> invalidated there's no way to know why unless we start from there. Is this
> command unabridged as well? I think you mentioned you altered the url, but
> there's no api key or anything I would expect to be in the request header
> that uploads a file to an endpoint.
>
> > I tried several video formats with this command:
> > ffmpeg -i sample_640x360.ts -method PUT
> > https://bogdan-public.s3.us-east-2.amazonaws.com/video/output.ts
> >
> > The bucket (its name is changed here) is public and I was able to send
> data
> > to it by Http Put request from the C# code sample.
>
> I have a feeling the C# code sample was part of a whole AWS S3 C# SDK,
> handling all AAA with credentials hardcoded in a configuration file or
> something. Doing the same with a REST api will require more hands-on
> operation.
>
> Regards,
> Ted Park
>
> _______________________________________________
> ffmpeg-user mailing list
> [hidden email]
> https://ffmpeg.org/mailman/listinfo/ffmpeg-user
>
> To unsubscribe, visit link above, or email
> [hidden email] with subject "unsubscribe".



--
All the best,

Bohdan
_______________________________________________
ffmpeg-user mailing list
[hidden email]
https://ffmpeg.org/mailman/listinfo/ffmpeg-user

To unsubscribe, visit link above, or email
[hidden email] with subject "unsubscribe".