FFmpeg lib + UDP + Qt issues

classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view
|

FFmpeg lib + UDP + Qt issues

isyiwang
Hello,
First of all, thank you to this amazing community of developers!  I have received a tremendous amount of help both directly and indirectly thanks to you all.

I am in need of some help regarding reading encoded video udp packets and decoding them into a Qt display.  I have managed to read the video packets and display them in Qt but video playback 'stutters'.  Checking the FPS of frames transmitted via the FFMpeg library to Qt, I get ~30 frames per second, so I'm led to believe that FFMpeg itself is sending me the same frames multiple times a second.  Here are the steps I took:


    1. Broadcast an H.264 encoded video stream via VLC to udp:/238.38.38.38:1234

    2. Use FFMpeg lib to decode the stream.  Here is my function (apologies in advance, if there is a better way to display code, please let me know!):

bool QVideoDecoder::decodeSeekFrame(int after)
{
bool done=false;
while(!done)
{
// Read a frame
timeoutTimer.start();
if(av_read_frame(pFormatCtx, &packet)<0)
return false;                             // Frame read failed (e.g. end of stream)
timeoutTimer.stop();

if(packet.stream_index==videoStream)
{
int frameFinished;
avcodec_decode_video2(pCodecCtx,pFrame,&frameFinished,&packet);

// Did we get a video frame?
if(frameFinished)
{
ffmpeg::AVRational millisecondbase = {1, 1000};
int f = packet.dts;
int t = ffmpeg::av_rescale_q(packet.dts,pFormatCtx->streams[videoStream]->time_base,millisecondbase);
if(LastFrameOk==false)
{
LastFrameOk=true;
LastLastFrameTime=LastFrameTime=t;
LastLastFrameNumber=LastFrameNumber=f;
}
else
{
// If we decoded 2 frames in a row, the last times are okay
LastLastFrameTime = LastFrameTime;
LastLastFrameNumber = LastFrameNumber;
LastFrameTime=t;
LastFrameNumber=f;
}

// Is this frame the desired frame?
if(after==-1 || LastFrameNumber>=after)
{
// Convert the image format (init the context the first time)
int w = pCodecCtx->width;
int h = pCodecCtx->height;
img_convert_ctx = ffmpeg::sws_getCachedContext(img_convert_ctx,w, h, pCodecCtx->pix_fmt, w, h, ffmpeg::PIX_FMT_RGB24, SWS_BICUBIC, NULL, NULL, NULL);

if(img_convert_ctx == NULL)
{
printf("Cannot initialize the conversion context!\n");
return false;
}
ffmpeg::sws_scale(img_convert_ctx, pFrame->data, pFrame->linesize, 0, pCodecCtx->height, pFrameRGB->data, pFrameRGB->linesize);

// Convert the frame to QImage
LastFrame=QImage(w,h,QImage::Format_RGB888);

for(int y=0;y<h;y++)
memcpy(LastFrame.scanLine(y),pFrameRGB->data[0]+y*pFrameRGB->linesize[0],w*3);

// Set the time
DesiredFrameTime = ffmpeg::av_rescale_q(after,pFormatCtx->streams[videoStream]->time_base,millisecondbase);
LastFrameOk=true;

done = true;
}
}
}
av_free_packet(&packet);
}

return done;
}

I have a periodic timer that requests the latest frame from FFMpeg. If I set the timer to a low rate (20 FPS), I get the video played back but at a slow rate.  If I set the timer to a high rate (60 FPS), I get the frames as soon as they are ready, but the video stutters.  I figured that I should request the frames back as fast as possible..  Any insight on how to avoid the stuttering is appreciated.

    3. Send the QImage back to Qt to be displayed -- I have tested the frames received by my Qt callback per second and can confirm that I am receiving ~ 30 FPS which leads me to believe the stuttering is caused by FFMpeg lib.

    4. I also wrote an interrupting callback to be called by FFMpeg if any of the blocking functions (i.e. av_read_frame) took too long:

Timer timeoutTimer;
static int decode_interrupt_cb(void)
{
if (timeoutTimer.getElapsedTimeInMilliSec() > 1000)
{
qDebug() << "Video Timed Out!";
return 1;
}
return 0;
}

Although I sometimes do receive the "Video Timed Out!" call, the av_read_frame function still blocks...

Any help would be greatly appreciated!