problems consuming live single file hls

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

problems consuming live single file hls

Daniel Oberhoff-3
Hello,


I previously reported on problems with consuming a live single file hls
stream. It was quite embedded in the application so i failed to make a
good reproducible case. Now i reduced it.


So i have a producer and a consumer. The producer just generates test
video like this:

ffmpeg -f lavfi -i testsrc=duration=100:size=1280x720:rate=30  -g 1
-hls_flags single_file -hls_list_size 0 -hls_segment_type fmp4 -f hls
test_single_file_mp4.m3u8

the consumer (which i start a few seconds in) just consumes the video
and discards the frames:

ffmpeg -i test_single_file_mp4.m3u8 -f null /dev/null

the consumer after a few seconds reports invalid nal unit sizes. I
attach producer and consumer logs.

Before i consumed the stream with my own c++ based program and saw that
i had to basically ignore the last segment in the file, because it often
was not fully written after it had been "published" via the m3u8. I did
not have this problem when using multi-file hls.

Any Ideas? Thanks for help! We really need single file hls in our
product (https://game-on-technologies.com/) since our file systems start
having problems with all the small files.


Best


Daniel Oberhoff



_______________________________________________
ffmpeg-user mailing list
[hidden email]
http://ffmpeg.org/mailman/listinfo/ffmpeg-user

To unsubscribe, visit link above, or email
[hidden email] with subject "unsubscribe".

consumer.log (15K) Download Attachment
producer.log (13K) Download Attachment
pEpkey.asc (5K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: problems consuming live single file hls

Carl Eugen Hoyos-2
2019-01-10 14:59 GMT+01:00, Daniel Oberhoff <[hidden email]>:

> I previously reported on problems with consuming a live single
> file hls stream. It was quite embedded in the application so i
> failed to make a good reproducible case. Now i reduced it.

Please test current FFmpeg git head.

Are you missing the "-re" option?
Iiuc, you trying to read as-fast-as-possible an input
that is just created: How is this supposed to work,
what behaviour do you expect?

Carl Eugen
_______________________________________________
ffmpeg-user mailing list
[hidden email]
http://ffmpeg.org/mailman/listinfo/ffmpeg-user

To unsubscribe, visit link above, or email
[hidden email] with subject "unsubscribe".
Reply | Threaded
Open this post in threaded view
|

Re: problems consuming live single file hls

Daniel Oberhoff-3

Am 10.01.19 um 18:04 schrieb Carl Eugen Hoyos:
> 2019-01-10 14:59 GMT+01:00, Daniel Oberhoff <[hidden email]>:
>
>> I previously reported on problems with consuming a live single
>> file hls stream. It was quite embedded in the application so i
>> failed to make a good reproducible case. Now i reduced it.
> Please test current FFmpeg git head.

same behavior (i attach logs again)

> Are you missing the "-re" option?
> Iiuc, you trying to read as-fast-as-possible an input
> that is just created: How is this supposed to work,
> what behaviour do you expect?

I dont think -re is applicable. Actually it may be the consumer is
slower than the producer, which is ok. Or it may be slower for some time
and later catch up. So, yes, i want to consume as fast as possible.
Since hls is a streaming format i would actually expect this to work. It
does for multi file. The rationale is simple: the producer produces a
m3u8 plalist, appending to it new segments as they are produced. i.e.
after they have been fully written. Then the consumer sees them (by
polling the m3u8) and consumes them. This is how i would expect it to
work over http also, i.e. the consumer polls the playlist and consumes
any new segments. We simply shortcut here and do it locally via the file
system, which should be ok. Am I missing something? As I said, for
multi-file hls we dont have these problems, and i am not sure why single
file should be different. The only difference really is that instead of
whole files the data is in a byte range of a file.

Best


Daniel



_______________________________________________
ffmpeg-user mailing list
[hidden email]
http://ffmpeg.org/mailman/listinfo/ffmpeg-user

To unsubscribe, visit link above, or email
[hidden email] with subject "unsubscribe".

consumer.log (9K) Download Attachment
producer.log (9K) Download Attachment
pEpkey.asc (5K) Download Attachment
pEpkey.asc (5K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: problems consuming live single file hls

Daniel Oberhoff-3
In reply to this post by Carl Eugen Hoyos-2
(sorry, i keep miss-sending this to your personal address)

>> Since hls is a streaming format i would actually expect this to work.
> It does work for "multi file" which is what hls is meant to be.

Byte-range is part of the standard (see
https://tools.ietf.org/html/draft-pantos-http-live-streaming-08#section-3.4.1),
so single file is as much "meant to be" as multi-file afaict.

> If you use single file, I don't see how the behaviour can be
> different, you can reproduce it using mpegts on your local
> file system.

What do you mean? The desire is for new entries in the playlist to
become visible only after the data is visible to open/read. You are
saying this is impossible? I cant imagine that to be honest. Would you
accept a patch if i can get this to work?


Best


Daniel



_______________________________________________
ffmpeg-user mailing list
[hidden email]
http://ffmpeg.org/mailman/listinfo/ffmpeg-user

To unsubscribe, visit link above, or email
[hidden email] with subject "unsubscribe".

pEpkey.asc (5K) Download Attachment
pEpkey.asc (5K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: problems consuming live single file hls

Carl Eugen Hoyos-2
2019-01-12 11:44 GMT+01:00, Daniel Oberhoff <[hidden email]>:

>>> Since hls is a streaming format i would actually expect this to work.
>> It does work for "multi file" which is what hls is meant to be.
>
> Byte-range is part of the standard (see
> https://tools.ietf.org/html/draft-pantos-http-live-streaming-08#section-3.4.1),
> so single file is as much "meant to be" as multi-file afaict.

But the playlist does not change or does it?

>> If you use single file, I don't see how the behaviour can be
>> different, you can reproduce it using mpegts on your local
>> file system.
>
> What do you mean? The desire is for new entries in the playlist to
> become visible only after the data is visible to open/read. You are
> saying this is impossible? I cant imagine that to be honest. Would you
> accept a patch if i can get this to work?

Patches are always preferred over bug reports (but bug reports
are very welcome), and there is no doubt that I may completely
misunderstand the issue.

Carl Eugen
_______________________________________________
ffmpeg-user mailing list
[hidden email]
http://ffmpeg.org/mailman/listinfo/ffmpeg-user

To unsubscribe, visit link above, or email
[hidden email] with subject "unsubscribe".
Reply | Threaded
Open this post in threaded view
|

Re: problems consuming live single file hls

Daniel Oberhoff-3
>>>> Since hls is a streaming format i would actually expect this to work.
>>> It does work for "multi file" which is what hls is meant to be.
>> Byte-range is part of the standard (see
>> https://tools.ietf.org/html/draft-pantos-http-live-streaming-08#section-3.4.1),
>> so single file is as much "meant to be" as multi-file afaict.
> But the playlist does not change or does it?

Ah, I think now I understand where we missed each other: Yes it does!
That is one of the central points in HLS: Video is produced in segments
(either in separate files or by appending to one file, but in either
case playable individually). After a segment is produced it is
"published" via the playlist by appending some lines to it. To do this
atomic the file is usually copied, appended to, and then "swapped" back
in via rename.

The single file hls implementation in ffmpeg somehow fails to properly
order these things, i.e. segments become visible in the playlist before
they are fully written, breaking the standard hls consumer pattern.

>
>>> If you use single file, I don't see how the behaviour can be
>>> different, you can reproduce it using mpegts on your local
>>> file system.
>> What do you mean? The desire is for new entries in the playlist to
>> become visible only after the data is visible to open/read. You are
>> saying this is impossible? I cant imagine that to be honest. Would you
>> accept a patch if i can get this to work?
> Patches are always preferred over bug reports (but bug reports
> are very welcome), and there is no doubt that I may completely
> misunderstand the issue.
I think to some extent that happened (see above where i try to
reconcile), but that my be entirely my fault. I will see what I can do.


Best


Daniel Oberhoff


_______________________________________________
ffmpeg-user mailing list
[hidden email]
http://ffmpeg.org/mailman/listinfo/ffmpeg-user

To unsubscribe, visit link above, or email
[hidden email] with subject "unsubscribe".

pEpkey.asc (5K) Download Attachment