Error when reading file as stream

classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

Error when reading file as stream

FFmpeg-users mailing list
Hi you lovely people,

I have another question that I cannot find a resolution for, so I am hoping somebody can shed a little light on the subject.
When passing the source as a file into ffmpeg from regular storage, my ffmpeg commands are working correctly and I am happy with the output.
When passing the same source to ffmpeg as a stream, through pipe; I am getting errors. This is not for all files, eg mxf files containing DV50, XDCAMHD422 etc have not shown any issue, neither do many mp4 files.
All files are probed before the transcode and all come back as probe score 100.

The error I get complains about…
pipe:: Invalid data found when processing input
Cannot determine format of input stream 0:0 after EOF

This is the whole command and output…

/usr/src/app # aws s3 cp s3://blah-blah-blah/source/14249/20200504/abc-6a-test_mov_34_1588580170874.mov - --quiet | ffmpeg -y -i -  -filter_complex "[0:0]fifo[vout];[0:1]afifo[01_afifo]" -map "[vout]" -map "[01_afifo]" -pix_fmt yuv420p -c:v libx264 -s 1280x720 -profile:v
main -level:v 3.1  -refs:v 3 -keyint_min 50 -b:v 6436k -r 25 -ac 2 -bsf:a aac_adtstoasc -c:a libfdk_aac -ar 44100 -b:a 192k -flags +global_header -movflags +faststart -f tee "[f=mp4:use_fifo=1]/data/temporary/transcode_20200504-26984-c69lmq/mov_source_1725.mp4"
ffmpeg version 4.2 Copyright (c) 2000-2019 the FFmpeg developers
  built with gcc 6.4.0 (Alpine 6.4.0)
  configuration: --disable-debug --disable-doc --disable-ffplay --enable-shared --enable-avresample --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-gpl --enable-libass --enable-libfreetype --enable-libvidstab --enable-libmp3lame --enable-libopenjpeg --enable-libopus --enable-libtheora --enable-libvorbis --enable-libvpx --enable-libwebp --enable-libx265 --enable-libxvid --enable-libx264 --enable-nonfree --enable-openssl --enable-libfdk_aac --enable-libkvazaar --enable-libaom --extra-libs=-lpthread --enable-postproc --enable-small --enable-version3 --extra-cflags=-I/opt/ffmpeg/include --extra-ldflags=-L/opt/ffmpeg/lib --extra-libs=-ldl --prefix=/opt/ffmpeg
  libavutil      56. 31.100 / 56. 31.100
  libavcodec     58. 54.100 / 58. 54.100
  libavformat    58. 29.100 / 58. 29.100
  libavdevice    58.  8.100 / 58.  8.100
  libavfilter     7. 57.100 /  7. 57.100
  libavresample   4.  0.  0 /  4.  0.  0
  libswscale      5.  5.100 /  5.  5.100
  libswresample   3.  5.100 /  3.  5.100
  libpostproc    55.  5.100 / 55.  5.100
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x556d07ec4000] stream 1, offset 0x30: partial file
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x556d07ec4000] Could not find codec parameters for stream 0 (Video: h264 (avc1 / 0x31637661), none(bt709), 1024x576, 6241 kb/s): unspecified pixel format
Consider increasing the value for the 'analyzeduration' and 'probesize' options
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'pipe:':
  Metadata:
    major_brand     : qt  
    minor_version   : 537199360
    compatible_brands: qt  
    creation_time   : 2010-11-17T13:05:57.000000Z
  Duration: 00:00:30.00, start: 0.000000, bitrate: N/A
    Stream #0:0(eng): Video: h264 (avc1 / 0x31637661), none(bt709), 1024x576, 6241 kb/s, 25 fps, 25 tbr, 2500 tbn, 5k tbc (default)
    Metadata:
      creation_time   : 2010-11-17T13:05:57.000000Z
      handler_name    : Apple Video Media Handler
      encoder         : H.264
    Stream #0:1(eng): Audio: aac (mp4a / 0x6134706D), 48000 Hz, stereo, fltp, 250 kb/s (default)
    Metadata:
      creation_time   : 2010-11-17T13:05:57.000000Z
      handler_name    : Apple Sound Media Handler
Stream mapping:
  Stream #0:0 (h264) -> fifo
  Stream #0:1 (aac) -> afifo
  fifo -> Stream #0:0 (libx264)
  afifo -> Stream #0:1 (libfdk_aac)
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x556d07ec4000] stream 1, offset 0x30: partial file
pipe:: Invalid data found when processing input
Cannot determine format of input stream 0:0 after EOF
Error marking filters as finished
Conversion failed!

I did try adding  ffmpeg -y -analyzeduration 100 -probesize 10000000 -i -  -filter_complex
But the same error occurred.

This is probably something to do with the fact that I am streaming the input, but I am not sure how to resolve it.
Any ideas will be warmly received

Thanks in advance
Adam
_______________________________________________
ffmpeg-user mailing list
[hidden email]
https://ffmpeg.org/mailman/listinfo/ffmpeg-user

To unsubscribe, visit link above, or email
[hidden email] with subject "unsubscribe".
Reply | Threaded
Open this post in threaded view
|

Re: Error when reading file as stream

Carl Eugen Hoyos-2


> Am 04.05.2020 um 11:38 schrieb adam smith via ffmpeg-user <[hidden email]>:
>
> When passing the same source to ffmpeg as a stream, through pipe; I am getting errors. This is not for all files, eg mxf files containing DV50, XDCAMHD422 etc have not shown any issue, neither do many mp4 files.
> All files are probed before the transcode and all come back as probe score 100.

Many files, esp. some mov/mp4 files, require seeking. Seeking is impossible if you provide the input through a pipe, this is not a limitation of FFmpeg.

Use qtfaststart to “fix” mov files, so they don’t require seeking anymore.

Carl Eugen
_______________________________________________
ffmpeg-user mailing list
[hidden email]
https://ffmpeg.org/mailman/listinfo/ffmpeg-user

To unsubscribe, visit link above, or email
[hidden email] with subject "unsubscribe".
Reply | Threaded
Open this post in threaded view
|

Re: Error when reading file as stream

FFmpeg-users mailing list
Thanks Carl,
Thanks for stopping me going down a never ending rabbit hole.

> On 4 May 2020, at 13:18, Carl Eugen Hoyos <[hidden email]> wrote:
>
> Many files, esp. some mov/mp4 files, require seeking. Seeking is impossible if you provide the input through a pipe, this is not a limitation of FFmpeg.

Good to know, makes sense and confirms the whole input as a stream being the issue and why mxf wrapped media works and some mp4.
I will have to think about how I can most efficiently resolve this.

If I mount the S3 bucket using S3FS I can read the file as normal which works, but requires the server to run S3FS and so require bespoke set up - which I am trying to avoid.

Perhaps I could write some code to localise the file if the first attempt fails with the error message…
pipe:: Invalid data found when processing input
Cannot determine format of input stream 0:0 after EOF
Error marking filters as finished.
That would work as I would be able to stream source when possible and fall back when not possible,  but it's not very elegant as requires a fail first.

If the issue is missing information from the source, is there a way I can pass information into ffmpeg at start up that I have extracted using ffprobe that would resolve the issue maybe?


>
> Use qtfaststart to “fix” mov files, so they don’t require seeking anymore.

Thanks, I use -flags +global_header -movflags +faststart on mp4 files I create using ffmpeg, is that the same thing?
I add for that for instant playback in browsers.
but I need to be able to accept files for transcode from third party providers so cannot guarantee they will be formatted using the faststart flag.

>
> Carl Eugen
>



_______________________________________________
ffmpeg-user mailing list
[hidden email]
https://ffmpeg.org/mailman/listinfo/ffmpeg-user

To unsubscribe, visit link above, or email
[hidden email] with subject "unsubscribe".
Reply | Threaded
Open this post in threaded view
|

Re: Error when reading file as stream

Carl Eugen Hoyos-2
Am Mo., 4. Mai 2020 um 14:35 Uhr schrieb adam smith via ffmpeg-user
<[hidden email]>:

> > Use qtfaststart to “fix” mov files, so they don’t require seeking anymore.
>
> Thanks, I use -flags +global_header -movflags +faststart on mp4 files I
> create using ffmpeg, is that the same thing?

The result is very similar, but qtfaststart takes an existing mov file and
only moves the index in front of the file, leaving everything else exactly
identical while ffmpeg -i input -movflags +faststart out.mov writes a
new file (with different properties) that has the index already moved
to the beginning of the file.

Carl Eugen
_______________________________________________
ffmpeg-user mailing list
[hidden email]
https://ffmpeg.org/mailman/listinfo/ffmpeg-user

To unsubscribe, visit link above, or email
[hidden email] with subject "unsubscribe".
Reply | Threaded
Open this post in threaded view
|

Re: Error when reading file as stream

FFmpeg-users mailing list


> On 4 May 2020, at 16:23, Carl Eugen Hoyos <[hidden email]> wrote:
>
> The result is very similar, but qtfaststart takes an existing mov file and
> only moves the index in front of the file, leaving everything else exactly
> identical while ffmpeg -i input -movflags +faststart out.mov writes a
> new file (with different properties) that has the index already moved
> to the beginning of the file.

Thanks for confirming Carl.

I thought I was adding the correct flags to my mp4 files but after a lot of testing it seems that because I am output using the tee function the flags I am adding are ignored and the output inherits the flags of the source.
Tee is currently -f tee "[f=mp4:use_fifo=1]
I am hoping I can add flags too, I will have a look about I am sure I will find something online.

That will not resolve 3rd party files but at least I will be producing media encoded as intended :)

thanks
Adam

_______________________________________________
ffmpeg-user mailing list
[hidden email]
https://ffmpeg.org/mailman/listinfo/ffmpeg-user

To unsubscribe, visit link above, or email
[hidden email] with subject "unsubscribe".
Reply | Threaded
Open this post in threaded view
|

Re: Error when reading file as stream

Carl Eugen Hoyos-2
Am Di., 5. Mai 2020 um 14:19 Uhr schrieb adam smith via ffmpeg-user
<[hidden email]>:

> That will not resolve 3rd party files

What's wrong with using qtfaststart on them (or on all files)?

Carl Eugen
_______________________________________________
ffmpeg-user mailing list
[hidden email]
https://ffmpeg.org/mailman/listinfo/ffmpeg-user

To unsubscribe, visit link above, or email
[hidden email] with subject "unsubscribe".
Reply | Threaded
Open this post in threaded view
|

Re: Error when reading file as stream

FFmpeg-users mailing list

> On 5 May 2020, at 17:15, Carl Eugen Hoyos <[hidden email]> wrote:
>
> Am Di., 5. Mai 2020 um 14:19 Uhr schrieb adam smith via ffmpeg-user
> <[hidden email]>:
>
>> That will not resolve 3rd party files
>
> What's wrong with using qtfaststart on them (or on all files)?
>

The problem is that the platform running ffmpeg is a container that can accesses the source media stored in s3 object storage through the aws command line stream copy. This enables FFmpeg to start transcoding immediately from any S3 bucket it has the account details for.
I presumed that the qtfaststart running on the same platform would also run into the problem with the source only being accessible through the aws command line.

Testing using…

aws s3 cp s3://storage/source/abc-6a-test_mov_34_1588580170874.mov - --quiet | qt-faststart - /data/temporary/transcode_20200505-993-j7flu0/testqtfaststart.mov

was unsuccessful and resulted in -: No such file or directory.

So if I am going to need to localise the file to use qt-faststart I might as well go straight to ffmpeg transcode.
Thanks for your help, I have learnt about a new tool which looks really useful for batch fixing archive mp4 for example.

Thanks
Adam

_______________________________________________
ffmpeg-user mailing list
[hidden email]
https://ffmpeg.org/mailman/listinfo/ffmpeg-user

To unsubscribe, visit link above, or email
[hidden email] with subject "unsubscribe".