Unable to record VP8+Opus from RTP+SDP

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

Unable to record VP8+Opus from RTP+SDP

Juan Navarro
Hi,

I'd like to record an RTP stream with one VP8 video and one OPUS audio;
these are originated from Chrome and then distributed as plain RTP
streams. I've written a basic SDP file to use as input for ffmpeg, but
my results have been mixed:

1) Recording only the video works fine with a WEBM container (using
"-vcodec copy -an -f webm"). ffmpeg is able to automatically find out
properties of the video, such as the resolution, and it stores it fine.
Other players (VLC tested) are able to open the file.

2) Recording only the audio is possible, again if using WEBM (with
"-acodec copy -vn -f webm"), but seems that no Opus headers/metadata are
stored. While ffplay is able to play the recorded file, other players
complain about it. E.g. VLC says "opus error: cannot read Opus header",
"opus error: initial Opus header is corrupted". There is, however, no
error or warning message printed out from the debug FFmpeg output.

Interestingly enough, using other container, e.g. OGG (with "-f opus"
and a .ogg output file), then the recording won't start at all due to
missing extradata:

    [opus @ 0x5e5d380] No extradata present
    Could not write header for output file #0 (incorrect codec
    parameters ?): Invalid data found when processing input


3) Recording of both audio and video in WEBM or Matroska doesn't work,
ffmpeg complains about not being able to get the video resolution (but
it was able to do so when recording only video):

    [sdp @ 0x6094a00] Could not find codec parameters for stream 1
    (Video: vp8, 1 reference frame, yuv420p): unspecified size
    Consider increasing the value for the 'analyzeduration' and
    'probesize' options
    [webm @ 0x6098640] dimensions not set
    Could not write header for output file #0 (incorrect codec
    parameters ?): Invalid argument

Note that I've already tried setting analyzeduration and probesize to
10M, this takes a long time to start the ffmpeg command but in the end
doesn't really change the outcome.


This is the input SDP:

    v=0
    o=- 0 0 IN IP4 127.0.0.1
    s=-
    c=IN IP4 127.0.0.1
    t=0 0
    m=audio 5006 RTP/AVP 109
    a=rtpmap:109 opus/48000/2
    m=video 5004 RTP/AVP 120
    a=rtpmap:120 VP8/90000



And these are the commands:

# Audio recording

    ffmpeg -protocol_whitelist file,rtp,udp \
         -fflags +genpts -i input.sdp \
         -map 0:a:0 -acodec copy -vn \
         -f webm -flags +global_header -y recording.webm


# Video recording

    ffmpeg -protocol_whitelist file,rtp,udp \
         -fflags +genpts -i input.sdp \
         -map 0:v:0 -vcodec copy -an \
         -f webm -flags +global_header -y recording.webm


# Audio+Video recording, this is the final use case I'd like to end up
solving, although being able to run the previous two commands will help
me learn and understand more about how FFmpeg works:

    ffmpeg -protocol_whitelist file,rtp,udp -nostdin -loglevel debug \
         -fflags +genpts -i input.sdp \
         -map 0:a:0 -map 0:v:0 -acodec copy -vcodec copy \
         -f webm -flags +global_header -y recording.webm




Given how easy it is to break the recording of the audio (eg. when
trying OGG format, instead of WEBM), I'm inclined to think that these
problems are matter of finding the right options for some of the ffmpeg
commands, and make the streams play well with the container. I want to
avoid transcoding, so all codecs should be "copy". After all, the target
here is to just store incoming media from Chrome, not to reencode it...

Could anyone give me a hand here?


PS. If anyone has a solution and wants the karma, I posted this question
in SuperUser:
https://superuser.com/questions/1468786/rtp-ffmpeg-dimensions-not-set-only-when-there-is-audiovideo



----

Full ffmpeg debug output:

ffmpeg version 4.2-static https://johnvansickle.com/ffmpeg/ Copyright
(c) 2000-2019 the FFmpeg developers
   built with gcc 6.3.0 (Debian 6.3.0-18+deb9u1) 20170516
   configuration: --enable-gpl --enable-version3 --enable-static
--disable-debug --disable-ffplay --disable-indev=sndio
--disable-outdev=sndio --cc=gcc-6 --enable-fontconfig --enable-frei0r
--enable-gnutls --enable-gmp --enable-libgme --enable-gray
--enable-libaom --enable-libfribidi --enable-libass --enable-libvmaf
--enable-libfreetype --enable-libmp3lame --enable-libopencore-amrnb
--enable-libopencore-amrwb --enable-libopenjpeg --enable-librubberband
--enable-libsoxr --enable-libspeex --enable-libsrt --enable-libvorbis
--enable-libopus --enable-libtheora --enable-libvidstab
--enable-libvo-amrwbenc --enable-libvpx --enable-libwebp
--enable-libx264 --enable-libx265 --enable-libxml2 --enable-libdav1d
--enable-libxvid --enable-libzvbi --enable-libzimg
   libavutil      56. 31.100 / 56. 31.100
   libavcodec     58. 54.100 / 58. 54.100
   libavformat    58. 29.100 / 58. 29.100
   libavdevice    58.  8.100 / 58.  8.100
   libavfilter     7. 57.100 /  7. 57.100
   libswscale      5.  5.100 /  5.  5.100
   libswresample   3.  5.100 /  3.  5.100
   libpostproc    55.  5.100 / 55.  5.100
Splitting the commandline.
Reading option '-protocol_whitelist' ... matched as AVOption
'protocol_whitelist' with argument 'file,rtp,udp'.
Reading option '-nostdin' ... matched as option 'stdin' (enable or
disable interaction on standard input) with argument 0.
Reading option '-loglevel' ... matched as option 'loglevel' (set logging
level) with argument 'debug'.
Reading option '-fflags' ... matched as AVOption 'fflags' with argument
'+genpts'.
Reading option '-i' ... matched as input url with argument 'input.sdp'.
Reading option '-map' ... matched as option 'map' (set input stream
mapping) with argument '0:a:0'.
Reading option '-map' ... matched as option 'map' (set input stream
mapping) with argument '0:v:0'.
Reading option '-acodec' ... matched as option 'acodec' (force audio
codec ('copy' to copy stream)) with argument 'copy'.
Reading option '-vcodec' ... matched as option 'vcodec' (force video
codec ('copy' to copy stream)) with argument 'copy'.
Reading option '-f' ... matched as option 'f' (force format) with
argument 'webm'.
Reading option '-flags' ... matched as AVOption 'flags' with argument
'+global_header'.
Reading option '-y' ... matched as option 'y' (overwrite output files)
with argument '1'.
Reading option 'recording.webm' ... matched as output url.
Finished splitting the commandline.
Parsing a group of options: global .
Applying option nostdin (enable or disable interaction on standard
input) with argument 0.
Applying option loglevel (set logging level) with argument debug.
Applying option y (overwrite output files) with argument 1.
Successfully parsed a group of options.
Parsing a group of options: input url input.sdp.
Successfully parsed a group of options.
Opening an input file: input.sdp.
[NULL @ 0x6094a00] Opening 'input.sdp' for reading
[sdp @ 0x6094a00] Format sdp probed with size=2048 and score=50
[sdp @ 0x6094a00] audio codec set to: opus
[sdp @ 0x6094a00] audio samplerate set to: 48000
[sdp @ 0x6094a00] audio channels set to: 2
[sdp @ 0x6094a00] video codec set to: vp8
[udp @ 0x609d480] end receive buffer size reported is 131072
[udp @ 0x6096f80] end receive buffer size reported is 131072
[sdp @ 0x6094a00] setting jitter buffer size to 500
[udp @ 0x6097e80] end receive buffer size reported is 131072
[udp @ 0x6097600] end receive buffer size reported is 131072
[sdp @ 0x6094a00] setting jitter buffer size to 500
[sdp @ 0x6094a00] Before avformat_find_stream_info() pos: 157 bytes
read:157 seeks:0 nb_streams:2

[sdp @ 0x6094a00] max_analyze_duration 5000000 reached at 5000000
microseconds st:0
[sdp @ 0x6094a00] Could not find codec parameters for stream 1 (Video:
vp8, 1 reference frame, yuv420p): unspecified size
Consider increasing the value for the 'analyzeduration' and 'probesize'
options

[sdp @ 0x6094a00] After avformat_find_stream_info() pos: 157 bytes
read:157 seeks:0 frames:252
Input #0, sdp, from 'input.sdp':
   Metadata:
     title           : -
   Duration: N/A, start: 0.000000, bitrate: N/A
     Stream #0:0, 252, 1/48000: Audio: opus, 48000 Hz, stereo, fltp
     Stream #0:1, 0, 1/90000: Video: vp8, 1 reference frame, yuv420p,
90k tbr, 90k tbn, 90k tbc
Successfully opened the file.
Parsing a group of options: output url recording.webm.
Applying option map (set input stream mapping) with argument 0:a:0.
Applying option map (set input stream mapping) with argument 0:v:0.
Applying option acodec (force audio codec ('copy' to copy stream)) with
argument copy.
Applying option vcodec (force video codec ('copy' to copy stream)) with
argument copy.
Applying option f (force format) with argument webm.
Successfully parsed a group of options.
Opening an output file: recording.webm.
[file @ 0x60e7000] Setting default whitelist 'file,crypto'

Successfully opened the file.
[webm @ 0x6098640] dimensions not set
Could not write header for output file #0 (incorrect codec parameters
?): Invalid argument
Stream mapping:
   Stream #0:0 -> #0:0 (copy)
   Stream #0:1 -> #0:1 (copy)
     Last message repeated 1 times
[AVIOContext @ 0x6123800] Statistics: 0 seeks, 0 writeouts

[AVIOContext @ 0x6095480] Statistics: 157 bytes read, 0 seeks

_______________________________________________
ffmpeg-user mailing list
[hidden email]
https://ffmpeg.org/mailman/listinfo/ffmpeg-user

To unsubscribe, visit link above, or email
[hidden email] with subject "unsubscribe".
Reply | Threaded
Open this post in threaded view
|

Re: Unable to record VP8+Opus from RTP+SDP

Juan Navarro
Some updates: recording video actually works fine in both cases of only
video and video+audio. A mistake in the video router was channeling
audio frames to what should have been the video stream.

However, the issue persists when trying to record Opus audio; it is just
not possible to play it back, supposedly due to missing headers (but
then how is it possible to record incoming RTP Opus audio?)

Cheers
_______________________________________________
ffmpeg-user mailing list
[hidden email]
https://ffmpeg.org/mailman/listinfo/ffmpeg-user

To unsubscribe, visit link above, or email
[hidden email] with subject "unsubscribe".
Reply | Threaded
Open this post in threaded view
|

Re: Unable to record VP8+Opus from RTP+SDP

kumowoon1025
> Some updates: recording video actually works fine in both cases of only
> video and video+audio. A mistake in the video router was channeling
> audio frames to what should have been the video stream.

I think every instance of this working is because ffmpeg and the media players you tried are good at making educated guesses.

The RTP payload format spec for opus is a proposed standard (rfc7587 <https://datatracker.ietf.org/doc/rfc7587/>) and the one for vp8 (rfc7741 <https://datatracker.ietf.org/doc/rfc7741/>) uses a profile of RTP that SDP hasn’t been updated to implement yet.

So I’m not sure how ffmpeg parses SDP files or if it uses a library but when you step back and take a look, you haven’t put in any specifics for either codec besides from maybe their names.

>   v=0
>   o=- 0 0 IN IP4 127.0.0.1
>   s=-
>   c=IN IP4 127.0.0.1
>   t=0 0
>   m=audio 5006 RTP/AVP 109
>   a=rtpmap:109 opus/48000/2
>   m=video 5004 RTP/AVP 120
>   a=rtpmap:120 VP8/90000

You can try reading the rfc’s and filling in the extra codec info it needs, but if you’re the one setting up the session, why not just use a simpler protocol? (I’m assuming the localhost is just a stand-in)

> However, the issue persists when trying to record Opus audio; it is just
> not possible to play it back, supposedly due to missing headers (but
> then how is it possible to record incoming RTP Opus audio?)

Like, you told ffmpeg it was going to be 48khz, 2 chs it believed your copy codec and just dumped everything into a file. But when you open up the file, does it really have two channels? is the audio interleaved? Nobody knows.

_______________________________________________
ffmpeg-user mailing list
[hidden email]
https://ffmpeg.org/mailman/listinfo/ffmpeg-user

To unsubscribe, visit link above, or email
[hidden email] with subject "unsubscribe".
Reply | Threaded
Open this post in threaded view
|

Re: Unable to record VP8+Opus from RTP+SDP

Juan Navarro
 >> So I’m not sure how ffmpeg parses SDP files or if it uses a library
but when you step back and take a look, you haven’t put in any specifics
for either codec besides from maybe their names.
 >>
 >>   v=0
 >>   o=- 0 0 IN IP4 127.0.0.1
 >>   s=-
 >>   c=IN IP4 127.0.0.1
 >>   t=0 0
 >>   m=audio 5006 RTP/AVP 109
 >>   a=rtpmap:109 opus/48000/2
 >>   m=video 5004 RTP/AVP 120
 >>   a=rtpmap:120 VP8/90000
 >
 > You can try reading the rfc’s and filling in the extra codec info it
needs, but if you’re the one setting up the session, why not just use a
simpler protocol? (I’m assuming the localhost is just a stand-in)
 >
 >> However, the issue persists when trying to record Opus audio; it is just
 >> not possible to play it back, supposedly due to missing headers (but
 >> then how is it possible to record incoming RTP Opus audio?)
 >
 > Like, you told ffmpeg it was going to be 48khz, 2 chs it believed
your copy codec and just dumped everything into a file. But when you
open up the file, does it really have two channels? is the audio
interleaved? Nobody knows.
 >

I just want to have a working command to record RTP media received from
an RTP router, in this case it is MediaSoup, which receives a WebRTC
stream from Chrome, then redistributes it as plain RTP. I first tried
using plain "rtp://" URL, but the ffmpeg command itself told me that an
SDP input file would be needed to use dynamic Payload Types (which is
totally sensible).

This is a standard SDP Offer as generated by Chrome to create a WebRTC
session:

    v=0
    o=- 6323209318560568134 2 IN IP4 127.0.0.1
    s=-
    t=0 0
    a=group:BUNDLE 0 1
    a=msid-semantic: WMS XAb2pgeSAM6YZw6uCRj11AE5Raf3cKHPNGl0
    m=audio 9 UDP/TLS/RTP/SAVPF 111 103 104 9 0 8 106 105 13 110 112 113 126
    c=IN IP4 0.0.0.0
    a=rtcp:9 IN IP4 0.0.0.0
    a=ice-ufrag:Ao6s
    a=ice-pwd:h3NoIquJSZ8L/yQUSy7nme5Y
    a=ice-options:trickle
    a=fingerprint:sha-256
    21:D2:14:1E:49:95:70:0B:A6:0B:93:5C:E5:A9:35:B0:8A:48:3C:95:75:06:62:74:5A:2B:99:29:0F:D0:3E:FB
    a=setup:actpass
    a=mid:0
    a=extmap:1 urn:ietf:params:rtp-hdrext:ssrc-audio-level
    a=extmap:2
    http://www.ietf.org/id/draft-holmer-rmcat-transport-wide-cc-extensions-01
    a=extmap:3 urn:ietf:params:rtp-hdrext:sdes:mid
    a=extmap:4 urn:ietf:params:rtp-hdrext:sdes:rtp-stream-id
    a=extmap:5 urn:ietf:params:rtp-hdrext:sdes:repaired-rtp-stream-id
    a=sendrecv
    a=msid:XAb2pgeSAM6YZw6uCRj11AE5Raf3cKHPNGl0
    816c42db-7d97-4053-86a0-bae0e5010985
    a=rtcp-mux
    a=rtpmap:111 opus/48000/2
    a=rtcp-fb:111 transport-cc
    a=fmtp:111 minptime=10;useinbandfec=1
    a=rtpmap:103 ISAC/16000
    a=rtpmap:104 ISAC/32000
    a=rtpmap:9 G722/8000
    a=rtpmap:0 PCMU/8000
    a=rtpmap:8 PCMA/8000
    a=rtpmap:106 CN/32000
    a=rtpmap:105 CN/16000
    a=rtpmap:13 CN/8000
    a=rtpmap:110 telephone-event/48000
    a=rtpmap:112 telephone-event/32000
    a=rtpmap:113 telephone-event/16000
    a=rtpmap:126 telephone-event/8000
    a=ssrc:3169356447 cname:XZAw22pKY7Y9bCph
    a=ssrc:3169356447 msid:XAb2pgeSAM6YZw6uCRj11AE5Raf3cKHPNGl0
    816c42db-7d97-4053-86a0-bae0e5010985
    a=ssrc:3169356447 mslabel:XAb2pgeSAM6YZw6uCRj11AE5Raf3cKHPNGl0
    a=ssrc:3169356447 label:816c42db-7d97-4053-86a0-bae0e5010985
    m=video 9 UDP/TLS/RTP/SAVPF 96 97 98 99 100 101 102 122 127 121 125
    107 108 109 124 120 123
    c=IN IP4 0.0.0.0
    a=rtcp:9 IN IP4 0.0.0.0
    a=ice-ufrag:Ao6s
    a=ice-pwd:h3NoIquJSZ8L/yQUSy7nme5Y
    a=ice-options:trickle
    a=fingerprint:sha-256
    21:D2:14:1E:49:95:70:0B:A6:0B:93:5C:E5:A9:35:B0:8A:48:3C:95:75:06:62:74:5A:2B:99:29:0F:D0:3E:FB
    a=setup:actpass
    a=mid:1
    a=extmap:14 urn:ietf:params:rtp-hdrext:toffset
    a=extmap:13 http://www.webrtc.org/experiments/rtp-hdrext/abs-send-time
    a=extmap:12 urn:3gpp:video-orientation
    a=extmap:2
    http://www.ietf.org/id/draft-holmer-rmcat-transport-wide-cc-extensions-01
    a=extmap:11 http://www.webrtc.org/experiments/rtp-hdrext/playout-delay
    a=extmap:6
    http://www.webrtc.org/experiments/rtp-hdrext/video-content-type
    a=extmap:7 http://www.webrtc.org/experiments/rtp-hdrext/video-timing
    a=extmap:8 http://tools.ietf.org/html/draft-ietf-avtext-framemarking-07
    a=extmap:9 http://www.webrtc.org/experiments/rtp-hdrext/color-space
    a=extmap:3 urn:ietf:params:rtp-hdrext:sdes:mid
    a=extmap:4 urn:ietf:params:rtp-hdrext:sdes:rtp-stream-id
    a=extmap:5 urn:ietf:params:rtp-hdrext:sdes:repaired-rtp-stream-id
    a=sendrecv
    a=msid:XAb2pgeSAM6YZw6uCRj11AE5Raf3cKHPNGl0
    0c05ddde-b52e-4776-97f0-755ded96e15d
    a=rtcp-mux
    a=rtcp-rsize
    a=rtpmap:96 VP8/90000
    a=rtcp-fb:96 goog-remb
    a=rtcp-fb:96 transport-cc
    a=rtcp-fb:96 ccm fir
    a=rtcp-fb:96 nack
    a=rtcp-fb:96 nack pli
    a=rtpmap:97 rtx/90000
    a=fmtp:97 apt=96
    a=rtpmap:98 VP9/90000
    a=rtcp-fb:98 goog-remb
    a=rtcp-fb:98 transport-cc
    a=rtcp-fb:98 ccm fir
    a=rtcp-fb:98 nack
    a=rtcp-fb:98 nack pli
    a=fmtp:98 profile-id=0
    a=rtpmap:99 rtx/90000
    a=fmtp:99 apt=98
    a=rtpmap:100 VP9/90000
    a=rtcp-fb:100 goog-remb
    a=rtcp-fb:100 transport-cc
    a=rtcp-fb:100 ccm fir
    a=rtcp-fb:100 nack
    a=rtcp-fb:100 nack pli
    a=fmtp:100 profile-id=2
    a=rtpmap:101 rtx/90000
    a=fmtp:101 apt=100
    a=rtpmap:102 H264/90000
    a=rtcp-fb:102 goog-remb
    a=rtcp-fb:102 transport-cc
    a=rtcp-fb:102 ccm fir
    a=rtcp-fb:102 nack
    a=rtcp-fb:102 nack pli
    a=fmtp:102
    level-asymmetry-allowed=1;packetization-mode=1;profile-level-id=42001f
    a=rtpmap:122 rtx/90000
    a=fmtp:122 apt=102
    a=rtpmap:127 H264/90000
    a=rtcp-fb:127 goog-remb
    a=rtcp-fb:127 transport-cc
    a=rtcp-fb:127 ccm fir
    a=rtcp-fb:127 nack
    a=rtcp-fb:127 nack pli
    a=fmtp:127
    level-asymmetry-allowed=1;packetization-mode=0;profile-level-id=42001f
    a=rtpmap:121 rtx/90000
    a=fmtp:121 apt=127
    a=rtpmap:125 H264/90000
    a=rtcp-fb:125 goog-remb
    a=rtcp-fb:125 transport-cc
    a=rtcp-fb:125 ccm fir
    a=rtcp-fb:125 nack
    a=rtcp-fb:125 nack pli
    a=fmtp:125
    level-asymmetry-allowed=1;packetization-mode=1;profile-level-id=42e01f
    a=rtpmap:107 rtx/90000
    a=fmtp:107 apt=125
    a=rtpmap:108 H264/90000
    a=rtcp-fb:108 goog-remb
    a=rtcp-fb:108 transport-cc
    a=rtcp-fb:108 ccm fir
    a=rtcp-fb:108 nack
    a=rtcp-fb:108 nack pli
    a=fmtp:108
    level-asymmetry-allowed=1;packetization-mode=0;profile-level-id=42e01f
    a=rtpmap:109 rtx/90000
    a=fmtp:109 apt=108
    a=rtpmap:124 red/90000
    a=rtpmap:120 rtx/90000
    a=fmtp:120 apt=124
    a=rtpmap:123 ulpfec/90000
    a=ssrc-group:FID 3597912382 1560435023
    a=ssrc:3597912382 cname:XZAw22pKY7Y9bCph
    a=ssrc:3597912382 msid:XAb2pgeSAM6YZw6uCRj11AE5Raf3cKHPNGl0
    0c05ddde-b52e-4776-97f0-755ded96e15d
    a=ssrc:3597912382 mslabel:XAb2pgeSAM6YZw6uCRj11AE5Raf3cKHPNGl0
    a=ssrc:3597912382 label:0c05ddde-b52e-4776-97f0-755ded96e15d
    a=ssrc:1560435023 cname:XZAw22pKY7Y9bCph
    a=ssrc:1560435023 msid:XAb2pgeSAM6YZw6uCRj11AE5Raf3cKHPNGl0
    0c05ddde-b52e-4776-97f0-755ded96e15d
    a=ssrc:1560435023 mslabel:XAb2pgeSAM6YZw6uCRj11AE5Raf3cKHPNGl0
    a=ssrc:1560435023 label:0c05ddde-b52e-4776-97f0-755ded96e15d


If you notice, there is a lot of metadata about the RTP session (ssrc,
cname), some about the ICE protocol (for WebRTC), but there isn't much
information there about the specifics of the codecs. In this case, the
preferred audio payload type is 111, which corresponds to the OPUS
codec. The specification "opus/48000/2" is expected to already convey
enough information to let the other peer know what codec it is going to
receive, and it allows for some (much) variance in the actual encoding
parameters that are going to be used; the receiver is just able to
auto-detect those and decode the stream.

For the cases where the codec has really incompatible profiles, to the
point where trying to decode with the wrong profile would fail, then
that information is also transmitted in the SDP; this is the case of the
H.264 profile levels (Baseline, Main, High, etc.), which are then
accordingly transmitted in the SDP Offer as totally different Payload
Types. But you'll notice that other codecs, like VP8, don't really need
such distinction.

Of course, such an SDP Offer is not really missing any crucial detail;
another browser such as Chrome, Firefox, Safari, etc. will be able to
receive and decode the incoming audio and video. So what I'm really
trying to do is doing the same with FFmpeg... now if you compare
Chrome's SDP with mine, it turns out mine does already have all relevant
details that are included in a "full-blown" SDP message.

_______________________________________________
ffmpeg-user mailing list
[hidden email]
https://ffmpeg.org/mailman/listinfo/ffmpeg-user

To unsubscribe, visit link above, or email
[hidden email] with subject "unsubscribe".
Reply | Threaded
Open this post in threaded view
|

Re: Unable to record VP8+Opus from RTP+SDP

kumowoon1025
> If you notice, there is a lot of metadata about the RTP session (ssrc,
> cname), some about the ICE protocol (for WebRTC), but there isn't much
> information there about the specifics of the codecs.

I think that statement is rather specious… WebRTC relies on some other app (perhaps a browser webapp implemented in js) to negotiate those specifics.

> In this case, the
> preferred audio payload type is 111, which corresponds to the OPUS
> codec. The specification "opus/48000/2" is expected to already convey
> enough information to let the other peer know what codec it is going to
> receive, and it allows for some (much) variance in the actual encoding
> parameters that are going to be used; the receiver is just able to
> auto-detect those and decode the stream.

That’s just “opus/48000/2” being the only string you’re allowed to put for the subtype/clock/channel fields of rtpmap. The actual encoding might differ, but as you said, in most cases the receiver is able to use the in-band signaling to decode. But in this case you’re trying to codec copy using parameters ffmpeg determined solely from the SDP file.

>   v=0
>   o=- 6323209318560568134 2 IN IP4 127.0.0.1
>   s=-
>   t=0 0
>   a=group:BUNDLE 0 1
>   a=msid-semantic: WMS XAb2pgeSAM6YZw6uCRj11AE5Raf3cKHPNGl0
>   m=audio 9 UDP/TLS/RTP/SAVPF 111 103 104 9 0 8 106 105 13 110 112 113 126
...

>   a=mid:0
>   a=extmap:1 urn:ietf:params:rtp-hdrext:ssrc-audio-level
>   a=extmap:2
>   http://www.ietf.org/id/draft-holmer-rmcat-transport-wide-cc-extensions-01
>   a=extmap:3 urn:ietf:params:rtp-hdrext:sdes:mid
>   a=extmap:4 urn:ietf:params:rtp-hdrext:sdes:rtp-stream-id
>   a=extmap:5 urn:ietf:params:rtp-hdrext:sdes:repaired-rtp-stream-id
>   a=sendrecv
>   a=msid:XAb2pgeSAM6YZw6uCRj11AE5Raf3cKHPNGl0
>   816c42db-7d97-4053-86a0-bae0e5010985
>   a=rtcp-mux
>   a=rtpmap:111 opus/48000/2
>   a=rtcp-fb:111 transport-cc
>   a=fmtp:111 minptime=10;useinbandfec=1
...
>   a=ssrc:3169356447 cname:XZAw22pKY7Y9bCph
>   a=ssrc:3169356447 msid:XAb2pgeSAM6YZw6uCRj11AE5Raf3cKHPNGl0
>   816c42db-7d97-4053-86a0-bae0e5010985
>   a=ssrc:3169356447 mslabel:XAb2pgeSAM6YZw6uCRj11AE5Raf3cKHPNGl0
>   a=ssrc:3169356447 label:816c42db-7d97-4053-86a0-bae0e5010985


WebRTC is far more complex than simple RTP. A separate application signals metadata about the connection and whether streams/tracks change or are added.
The extmap attributes declare the mapping between the RTP header extensions and the corresponding parameters that will be used in the session, which may well include framing parameters (for VBR, DTX, etc.)

And the fmtp attribute actually does specify formatting info...

> For the cases where the codec has really incompatible profiles, to the
> point where trying to decode with the wrong profile would fail, then
> that information is also transmitted in the SDP; this is the case of the
> H.264 profile levels (Baseline, Main, High, etc.), which are then
> accordingly transmitted in the SDP Offer as totally different Payload
> Types. But you'll notice that other codecs, like VP8, don't really need
> such distinction.

As with opus, the “extradata” that ffmpeg complains is missing would be provided from a separate signaler to WebRTC.

>   m=video 9 UDP/TLS/RTP/SAVPF 96 97 98 99 100 101 102 122 127 121 125
>   107 108 109 124 120 123
...
>   a=rtpmap:96 VP8/90000
>   a=rtcp-fb:96 goog-remb
>   a=rtcp-fb:96 transport-cc
>   a=rtcp-fb:96 ccm fir
>   a=rtcp-fb:96 nack
>   a=rtcp-fb:96 nack pli
>   a=rtpmap:97 rtx/90000
>   a=fmtp:97 apt=96

Also, as this session uses the AVPF profile, it can do things, like make sure the timestamp increments are synchronized with the RTP clock, get a keyframe at the start, have the server retransmit lost packets, etc.

> Of course, such an SDP Offer is not really missing any crucial detail;
> another browser such as Chrome, Firefox, Safari, etc. will be able to
> receive and decode the incoming audio and video. So what I'm really
> trying to do is doing the same with FFmpeg... now if you compare
> Chrome's SDP with mine, it turns out mine does already have all relevant
> details that are included in a "full-blown" SDP message.

You mentioned mediasoup. Can you have it serve the sdp or get the parameters and use those to write a proper one? I don’t know how much access you have to the media soup router, is it actually on localhost?
_______________________________________________
ffmpeg-user mailing list
[hidden email]
https://ffmpeg.org/mailman/listinfo/ffmpeg-user

To unsubscribe, visit link above, or email
[hidden email] with subject "unsubscribe".