MPEG TS+PS parser

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

MPEG TS+PS parser

Mark Filipak (ffmpeg)
The command below does a flat, 1st level parse of an MPEG TS+PS (plus a little more). As I expand
the parse further, I'm going to hit the limit of 'sed's capability (I think). For further parses
I'll need to conditionally include subpatterns based on, 1, lookaheads with branching, and 2,
bytesize values extracted by the parse (some of which will have to compute and remember index
offsets). Both are capabilities that may be beyond sed's capabilities (I think).
-If possible, I plan to descend all the way into macroblocks. I'm already doing all this stuff by
hand. Now I want to automate it.-

Got any suggestions how I should proceed?

BTW, give the command a try and let me know what you think.

Thanks,
Mark.

xxd -p -u VIDEOFILE \
| tr -d '\n' \
| sed -r 's/000001/\x0D\x0A&/g' \
| sed -r 's/^00000100/                 picture_start_code\: &/g' \
| sed -r 's/^000001(0[1-9A-F]|[1-9A][0-9A-F])/                   slice_start_code\: &/g' \
| sed -r 's/slice_start_code\: ......../&\x0D\x0A                              slice\: /g' \
| sed -r 's/^000001B0/                         (reserved)\: &/g' \
| sed -r 's/^000001B1/                         (reserved)\: &/g' \
| sed -r 's/^000001B2/               user_data_start_code\: &/g' \
| sed -r 's/^000001B3/               sequence_header_code\: &/g' \
| sed -r 's/^000001B4/                sequence_error_code\: &/g' \
| sed -r 's/^000001B50/                         (reserved)\: &/g' \
| sed -r 's/^000001B51/                 sequence_extension\: &/g' \
| sed -r 's/^000001B52/         sequence_display_extension\: &/g' \
| sed -r 's/^000001B53/             quant_matrix_extension\: &/g' \
| sed -r 's/^000001B54/                copyright_extension\: &/g' \
| sed -r 's/^000001B55/        sequence_scalable_extension\: &/g' \
| sed -r 's/^000001B56/                         (reserved)\: &/g' \
| sed -r 's/^000001B57/          picture_display_extension\: &/g' \
| sed -r 's/^000001B58/           picture_coding_extension\: &/g' \
| sed -r 's/^000001B59/ picture_spatial_scalable_extension\: &/g' \
| sed -r 's/^000001B5A/picture_temporal_scalable_extension\: &/g' \
| sed -r 's/^000001B5B/        camera_parameters_extension\: &/g' \
| sed -r 's/^000001B5C/                    ITU-T extension\: &/g' \
| sed -r 's/^000001B5D/                         (reserved)\: &/g' \
| sed -r 's/^000001B5E/                         (reserved)\: &/g' \
| sed -r 's/^000001B5F/                         (reserved)\: &/g' \
| sed -r 's/^000001B6/                         (reserved)\: &/g' \
| sed -r 's/^000001B7/                  sequence_end_code\: &/g' \
| sed -r 's/^000001B8/                   group_start_code\: &/g' \
| sed -r 's/^000001B9/              MPEG_program_end_code\: &/g' \
| sed -r 's/^000001BA/                    pack_start_code\: &/g' \
| sed -r 's/^000001BB/           system_header_start_code\: &/g' \
| sed -r 's/^000001BC/                 program_stream_map\: &/g' \
| sed -r 's/^000001BD/                   private_stream_1\: &/g' \
| sed -r 's/^000001BE/                     padding_stream\: &/g' \
| sed -r 's/^000001BF/                   private_stream_2\: &/g' \
| sed -r 's/^000001[CD]/                   pes_audio_header\: &/g' \
| sed -r 's/^000001E/                   pes_video_header\: &/g' \
| sed -r 's/pes_video_header\: ......../&\x0D\x0A                  pes_packet_length\: /g' \
| sed -r 's/pes_packet_length\: ........../&\x0D\x0A                                     /g' \
| sed -r 's/^000001F0/                         ECM_stream\: &/g' \
| sed -r 's/^000001F1/                         EMM_stream\: &/g' \
| sed -r 's/^000001F2/                       DSMCC_stream\: &/g' \
| sed -r 's/^000001F3/               ISO\/IEC_13522_stream\: &/g' \
| sed -r 's/^000001F4/              H.222.1 type A stream\: &/g' \
| sed -r 's/^000001F5/              H.222.1 type B stream\: &/g' \
| sed -r 's/^000001F6/              H.222.1 type C stream\: &/g' \
| sed -r 's/^000001F7/              H.222.1 type D stream\: &/g' \
| sed -r 's/^000001F8/              H.222.1 type E stream\: &/g' \
| sed -r 's/^000001F9/                   ancillary_stream\: &/g' \
| sed -r 's/^000001FA/       14496-1_SL-packetized_stream\: &/g' \
| sed -r 's/^000001FB/             14496-1_FlexMux_stream\: &/g' \
| sed -r 's/^000001FC/                    metadata stream\: &/g' \
| sed -r 's/^000001FD/                 extended_stream_id\: &/g' \
| sed -r 's/^000001FE/               reserved data stream\: &/g' \
| sed -r 's/^000001FF/           program_stream_directory\: &/g' \
 > VIDEOFILE.parsed.txt

--
What if you woke up and found yourself in a police state?
African-Americans wake up in a police state every day.
_______________________________________________
ffmpeg-user mailing list
[hidden email]
https://ffmpeg.org/mailman/listinfo/ffmpeg-user

To unsubscribe, visit link above, or email
[hidden email] with subject "unsubscribe".
Reply | Threaded
Open this post in threaded view
|

Re: MPEG TS+PS parser

Carl Zwanzig
On 10/15/2020 1:44 AM, Mark Filipak (ffmpeg) wrote:
> Got any suggestions how I should proceed?

I hate to say this, but you're rather reinventing the wheel (and building it
out of toothpicks).

There are a fair number of mpeg-ts parsers already out there in a variety of
languages, and the kind of structure you've used is going to be slow and
resource-intensive. If nothing else, if you're using the bash shell use a
case statement to separate out the translations (instead of passing
everything through one HUGE pipeline).

(This is not a task for pattern substitution.)

Later,

z!
_______________________________________________
ffmpeg-user mailing list
[hidden email]
https://ffmpeg.org/mailman/listinfo/ffmpeg-user

To unsubscribe, visit link above, or email
[hidden email] with subject "unsubscribe".
Reply | Threaded
Open this post in threaded view
|

Re: MPEG TS+PS parser

Mark Filipak (ffmpeg)
On 10/16/2020 10:14 PM, Carl Zwanzig wrote:
> On 10/15/2020 1:44 AM, Mark Filipak (ffmpeg) wrote:
>> Got any suggestions how I should proceed?
>
> I hate to say this, but you're rather reinventing the wheel (and building it out of toothpicks).

You're absolutely right, Carl.

> There are a fair number of mpeg-ts parsers already out there in a variety of languages ...

Really? Do you mean VOBEdit? All that I've seen are formatted as pretty pictures and all that I've
seen have errors. I want a plain text format but I can't paste it here because it's too wide for email.

>... and the kind
> of structure you've used is going to be slow and resource-intensive. If nothing else, if you're
> using the bash shell use a case statement to separate out the translations (instead of passing
> everything through one HUGE pipeline).
>
> (This is not a task for pattern substitution.)

Oh, I know that. I just wanted to get started. Looking at the output (such as it is), I could see
cases where, for example, bogus start_codes (\x00 00 01) are inside private streams (and elsewhere).

I used SNOBOL about 35 years ago to quickly write very powerful parsers. I'm looking at Unicon now.

--
Pandemic perspective, 17 October 2020 - early in the pandemic:
Yesterday, China: 24 new cases, S.Korea: 47, U.S.: 71,687.
Today, U.S.: 4% of world population, 21% of cases, 20% of deaths.
Today, U.S. dead, pandemic (8 mo): 223,644, WWII (4 yrs): 419,000.
Today, U.S.: Of 5,619,045 resolved cases, 96% lived, 4% died.
22 Jan: U.S. & S.Korea reported 1st cases on the same day.
6 Mar, S.Korea: 140,000 total tests; results in 4 hours.
6 Mar, U.S.: 2000 total tests; results in 1 to 2 weeks.
May: for the first time, U.S. care-homes are required to report.
1 Jun, S.Korea: 0 total care-home deaths, U.S.: 33,000 +/- 7,000.
5 Aug: U.S. tests still only 1/4 of number needed; 4 day results.
1 Sep: Over 60% of U.S. nurses report lack of protective equipment.
18 Sep: U.S. doctors & nurses still acutely lack PPE; 1200 dead.
?? 2021: U.S. gains herd immunity, 10 million dead, civil conflict.
_______________________________________________
ffmpeg-user mailing list
[hidden email]
https://ffmpeg.org/mailman/listinfo/ffmpeg-user

To unsubscribe, visit link above, or email
[hidden email] with subject "unsubscribe".
Reply | Threaded
Open this post in threaded view
|

Re: MPEG TS+PS parser

Carl Zwanzig
On 10/16/2020 11:54 PM, Mark Filipak (ffmpeg) wrote:
> On 10/16/2020 10:14 PM, Carl Zwanzig wrote:
> You're absolutely right, Carl.

BTW, I go by "z" (how I sign my email).

>> There are a fair number of mpeg-ts parsers already out there in a variety
>> of languages ...
>
> Really? Do you mean VOBEdit? All that I've seen are formatted as pretty
> pictures and all that I've seen have errors.

Acually, I didn't even think of vobedit. A quick search turned up-
https://github.com/jeoliva/mpegts-basic-parser
https://github.com/jeeb/mpegts
https://github.com/daniep01/MPEG-2-Transport-Stream-Packet-Analyser
https://www.serbod.com/--cc/mpeg-ts-parseranalyzer
https://tsduck.io/

One of them probably does what you want or can be modified to.


> I used SNOBOL about 35 years ago to quickly write very powerful parsers. I'm
> looking at Unicon now.

If you're going that direction, you may be better off with something like
Rust or java/javascript (assuming that c/c++ are off the table); definitely
something that already has some mpeg code available (rust does).

I'll take further discussion off the ffmpeg list because it's no longer
on-topic.

Later,

z!
_______________________________________________
ffmpeg-user mailing list
[hidden email]
https://ffmpeg.org/mailman/listinfo/ffmpeg-user

To unsubscribe, visit link above, or email
[hidden email] with subject "unsubscribe".