ffmpeg video to opengl texture
I'm trying to render frames grabbed and converted from a video using ffmpeg to an OpenGL texture to be put on a quad. I've pretty much exhausted google and not found an answer, well I've found answers but none of them seem to have worked.
Basically, I am using avcodec_decode_video2()
to decode the frame and then sws_scale()
to convert the frame to RGB and then glTexSubImage2D()
to create an openGL texture from it but can't seem to get anything to work.
I've made sure the "destination" AVFrame has power of 2 dimensions in the SWS Context setup. Here is my code:
SwsContext *img_convert_ctx = sws_getContext(pCodecCtx->width,
pCodecCtx->height, pCodecCtx->pix_fmt, 512,
256, PIX_FMT_RGB24, SWS_BICUBIC, NULL,
NULL, NULL);
//While still frames to read
while(av_read_frame(pFormatCtx, &packet)>=0) {
glClear(GL_COLOR_BUFFER_BIT);
//If the packet is from the video stream
if(packet.stream_index == videoStream) {
//Decode the video
avcodec_decode_video2(pCodecCtx, pFrame, &frameFinished, &packet);
//If we got a frame then convert it and put it into RGB buffer
if(frameFinished) {
printf("frame finished: %in", number);
sws_scale(img_convert_ctx, pFrame->data, pFrame->linesize, 0, pCodecCtx->height, pFrameRGB->data, pFrameRGB->linesize);
glBindTexture(GL_TEXTURE_2D, texture);
//gluBuild2DMipmaps(GL_TEXTURE_2D, 3, pCodecCtx->width, pCodecCtx->height, GL_RGB, GL_UNSIGNED_INT, pFrameRGB->data);
glTexSubImage2D(GL_TEXTURE_2D, 0, 0,0, 512, 256, GL_RGB, GL_UNSIGNED_BYTE, pFrameRGB->data[0]);
SaveFrame(pFrameRGB, pCodecCtx->width, pCodecCtx->height, number);
number++;
}
}
glColor3f(1,1,1);
glBindTexture(GL_TEXTURE_2D, texture);
glBegin(GL_QUADS);
glTexCoord2f(0,1);
glVertex3f(0,0,0);
glTexCoord2f(1,1);
glVertex3f(pCodecCtx->width,0,0);
glTexCoord2f(1,0);
glVertex3f(pCodecCtx->width, pCodecCtx->height,0);
glTexCoord2f(0,0);
glVertex3f(0,pCodecCtx->height,0);
glEnd();
As you can see in that code, I am also saving the frames to .ppm files just to make sure they are actually rendering, which they are.
The file being used is a .wmv at 854x480, could this be the problem? The fact I'm just telling it to go 512x256?
P.S. I've looked at this Stack Overflow question but it didn't help.
Also, I have glEnable(GL_TEXTURE_2D)
as well and have tested it by just loading in a normal bmp.
EDIT
I'm getting an image on the screen now but it is a garbled mess, I'm guessing something to do with changing things to a power of 2 (in the decode, swscontext
and gluBuild2DMipmaps
as shown in my code). I'm usually nearly exactly the same code as shown above, only I've changed glTexSubImage2D
to gluBuild2DMipmaps
and changed the types to GL_RGBA
.
Here is what the frame looks like:
EDIT AGAIN
Just realised I haven't showed the code for how pFrameRGB is set up:
//Allocate video frame for 24bit RGB that we convert to.
AVFrame *pFrameRGB;
pFrameRGB = avcodec_alloc_frame();
if(pFrameRGB == NULL) {
return -1;
}
//Allocate memory for the raw data we get when converting.
uint8_t *buffer;
int numBytes;
numBytes = avpicture_get_size(PIX_FMT_RGB24, pCodecCtx->width, pCodecCtx->height);
buffer = (uint8_t *) av_malloc(numBytes*sizeof(uint8_t));
//Associate frame with our buffer
avpicture_fill((AVPicture *) pFrameRGB, buffer, PIX_FMT_RGB24,
pCodecCtx->width, pCodecCtx->height);
Now that I ahve changed the PixelFormat
in avgpicture_get_size
to PIX_FMT_RGB24
, I've done that in SwsContext
as well and changed GluBuild2DMipmaps
to GL_RGB
and I get a slightly better image but it looks like I'm still missing lines and it's still a bit stretched:
Another Edit
After following Macke's advice and passing the actual resolution to OpenGL I get the frames nearly proper but still a bit skewed and in black and white, also it's only getting 6fps now rather than 110fps:
P.S.
I've got a function to save the frames to image after sws_scale()
and they are coming out fine as colour and everything so something in OGL is making it B&W.
LAST EDIT
Working! Okay I have it working now, basically I am not padding out the texture to a power of 2 and just using the resolution the video is.
I got the texture showing up properly with a lucky guess at the correct glPixelStorei()
glPixelStorei(GL_UNPACK_ALIGNMENT, 2);
Also, if anyone else has the subimage()
showing blank problem like me, you have to fill the texture at least once with glTexImage2D()
and so I use it once in the loop and then use glTexSubImage2D()
after that.
Thanks Macke and datenwolf for all your help.
opengl ffmpeg textures video-processing render-to-texture
add a comment |
I'm trying to render frames grabbed and converted from a video using ffmpeg to an OpenGL texture to be put on a quad. I've pretty much exhausted google and not found an answer, well I've found answers but none of them seem to have worked.
Basically, I am using avcodec_decode_video2()
to decode the frame and then sws_scale()
to convert the frame to RGB and then glTexSubImage2D()
to create an openGL texture from it but can't seem to get anything to work.
I've made sure the "destination" AVFrame has power of 2 dimensions in the SWS Context setup. Here is my code:
SwsContext *img_convert_ctx = sws_getContext(pCodecCtx->width,
pCodecCtx->height, pCodecCtx->pix_fmt, 512,
256, PIX_FMT_RGB24, SWS_BICUBIC, NULL,
NULL, NULL);
//While still frames to read
while(av_read_frame(pFormatCtx, &packet)>=0) {
glClear(GL_COLOR_BUFFER_BIT);
//If the packet is from the video stream
if(packet.stream_index == videoStream) {
//Decode the video
avcodec_decode_video2(pCodecCtx, pFrame, &frameFinished, &packet);
//If we got a frame then convert it and put it into RGB buffer
if(frameFinished) {
printf("frame finished: %in", number);
sws_scale(img_convert_ctx, pFrame->data, pFrame->linesize, 0, pCodecCtx->height, pFrameRGB->data, pFrameRGB->linesize);
glBindTexture(GL_TEXTURE_2D, texture);
//gluBuild2DMipmaps(GL_TEXTURE_2D, 3, pCodecCtx->width, pCodecCtx->height, GL_RGB, GL_UNSIGNED_INT, pFrameRGB->data);
glTexSubImage2D(GL_TEXTURE_2D, 0, 0,0, 512, 256, GL_RGB, GL_UNSIGNED_BYTE, pFrameRGB->data[0]);
SaveFrame(pFrameRGB, pCodecCtx->width, pCodecCtx->height, number);
number++;
}
}
glColor3f(1,1,1);
glBindTexture(GL_TEXTURE_2D, texture);
glBegin(GL_QUADS);
glTexCoord2f(0,1);
glVertex3f(0,0,0);
glTexCoord2f(1,1);
glVertex3f(pCodecCtx->width,0,0);
glTexCoord2f(1,0);
glVertex3f(pCodecCtx->width, pCodecCtx->height,0);
glTexCoord2f(0,0);
glVertex3f(0,pCodecCtx->height,0);
glEnd();
As you can see in that code, I am also saving the frames to .ppm files just to make sure they are actually rendering, which they are.
The file being used is a .wmv at 854x480, could this be the problem? The fact I'm just telling it to go 512x256?
P.S. I've looked at this Stack Overflow question but it didn't help.
Also, I have glEnable(GL_TEXTURE_2D)
as well and have tested it by just loading in a normal bmp.
EDIT
I'm getting an image on the screen now but it is a garbled mess, I'm guessing something to do with changing things to a power of 2 (in the decode, swscontext
and gluBuild2DMipmaps
as shown in my code). I'm usually nearly exactly the same code as shown above, only I've changed glTexSubImage2D
to gluBuild2DMipmaps
and changed the types to GL_RGBA
.
Here is what the frame looks like:
EDIT AGAIN
Just realised I haven't showed the code for how pFrameRGB is set up:
//Allocate video frame for 24bit RGB that we convert to.
AVFrame *pFrameRGB;
pFrameRGB = avcodec_alloc_frame();
if(pFrameRGB == NULL) {
return -1;
}
//Allocate memory for the raw data we get when converting.
uint8_t *buffer;
int numBytes;
numBytes = avpicture_get_size(PIX_FMT_RGB24, pCodecCtx->width, pCodecCtx->height);
buffer = (uint8_t *) av_malloc(numBytes*sizeof(uint8_t));
//Associate frame with our buffer
avpicture_fill((AVPicture *) pFrameRGB, buffer, PIX_FMT_RGB24,
pCodecCtx->width, pCodecCtx->height);
Now that I ahve changed the PixelFormat
in avgpicture_get_size
to PIX_FMT_RGB24
, I've done that in SwsContext
as well and changed GluBuild2DMipmaps
to GL_RGB
and I get a slightly better image but it looks like I'm still missing lines and it's still a bit stretched:
Another Edit
After following Macke's advice and passing the actual resolution to OpenGL I get the frames nearly proper but still a bit skewed and in black and white, also it's only getting 6fps now rather than 110fps:
P.S.
I've got a function to save the frames to image after sws_scale()
and they are coming out fine as colour and everything so something in OGL is making it B&W.
LAST EDIT
Working! Okay I have it working now, basically I am not padding out the texture to a power of 2 and just using the resolution the video is.
I got the texture showing up properly with a lucky guess at the correct glPixelStorei()
glPixelStorei(GL_UNPACK_ALIGNMENT, 2);
Also, if anyone else has the subimage()
showing blank problem like me, you have to fill the texture at least once with glTexImage2D()
and so I use it once in the loop and then use glTexSubImage2D()
after that.
Thanks Macke and datenwolf for all your help.
opengl ffmpeg textures video-processing render-to-texture
1
Voting to close. IMO, this question is getting out of hand. You need more hand-holding than SO is meant to support here. There are many issues that needs covering, and re-editing the question constantly isn't very good. I suggest to try a mail-list or a forum, or the SO OpenGL chat-room.
– Macke
Jun 28 '11 at 10:27
Okay well the issues are done now thanks to glPixelStorei. And I've deleted the comments etc. that are not helpful to anyone else approaching this question with the same problem.
– Infiniti Fizz
Jun 28 '11 at 13:17
3
Also, as the question is about getting an ffmpeg frame to an opengl texture, I don't understand how the question got out of hand, given all the questions and answers were about uploading an ffmpeg frame to an opengl texture, no matter if it was about performance (gluBuild2DMipmaps(), glTexSubImage2D()), pixel storage (glPixelStorei() etc.) ffmpeg's execution (sws_scale, SwsContext) or anything else to do with the problem. And editing the question is useful for others with the same problem as they can easily see the steps taken to resolve the issue.
– Infiniti Fizz
Jun 28 '11 at 14:19
@InfinitiFizz: Hey I'm trying to achieve same thing by creating a browser plugin, however, all I get is a black screen, can you please guide me?
– Dipen Shah
Jul 19 '16 at 18:20
add a comment |
I'm trying to render frames grabbed and converted from a video using ffmpeg to an OpenGL texture to be put on a quad. I've pretty much exhausted google and not found an answer, well I've found answers but none of them seem to have worked.
Basically, I am using avcodec_decode_video2()
to decode the frame and then sws_scale()
to convert the frame to RGB and then glTexSubImage2D()
to create an openGL texture from it but can't seem to get anything to work.
I've made sure the "destination" AVFrame has power of 2 dimensions in the SWS Context setup. Here is my code:
SwsContext *img_convert_ctx = sws_getContext(pCodecCtx->width,
pCodecCtx->height, pCodecCtx->pix_fmt, 512,
256, PIX_FMT_RGB24, SWS_BICUBIC, NULL,
NULL, NULL);
//While still frames to read
while(av_read_frame(pFormatCtx, &packet)>=0) {
glClear(GL_COLOR_BUFFER_BIT);
//If the packet is from the video stream
if(packet.stream_index == videoStream) {
//Decode the video
avcodec_decode_video2(pCodecCtx, pFrame, &frameFinished, &packet);
//If we got a frame then convert it and put it into RGB buffer
if(frameFinished) {
printf("frame finished: %in", number);
sws_scale(img_convert_ctx, pFrame->data, pFrame->linesize, 0, pCodecCtx->height, pFrameRGB->data, pFrameRGB->linesize);
glBindTexture(GL_TEXTURE_2D, texture);
//gluBuild2DMipmaps(GL_TEXTURE_2D, 3, pCodecCtx->width, pCodecCtx->height, GL_RGB, GL_UNSIGNED_INT, pFrameRGB->data);
glTexSubImage2D(GL_TEXTURE_2D, 0, 0,0, 512, 256, GL_RGB, GL_UNSIGNED_BYTE, pFrameRGB->data[0]);
SaveFrame(pFrameRGB, pCodecCtx->width, pCodecCtx->height, number);
number++;
}
}
glColor3f(1,1,1);
glBindTexture(GL_TEXTURE_2D, texture);
glBegin(GL_QUADS);
glTexCoord2f(0,1);
glVertex3f(0,0,0);
glTexCoord2f(1,1);
glVertex3f(pCodecCtx->width,0,0);
glTexCoord2f(1,0);
glVertex3f(pCodecCtx->width, pCodecCtx->height,0);
glTexCoord2f(0,0);
glVertex3f(0,pCodecCtx->height,0);
glEnd();
As you can see in that code, I am also saving the frames to .ppm files just to make sure they are actually rendering, which they are.
The file being used is a .wmv at 854x480, could this be the problem? The fact I'm just telling it to go 512x256?
P.S. I've looked at this Stack Overflow question but it didn't help.
Also, I have glEnable(GL_TEXTURE_2D)
as well and have tested it by just loading in a normal bmp.
EDIT
I'm getting an image on the screen now but it is a garbled mess, I'm guessing something to do with changing things to a power of 2 (in the decode, swscontext
and gluBuild2DMipmaps
as shown in my code). I'm usually nearly exactly the same code as shown above, only I've changed glTexSubImage2D
to gluBuild2DMipmaps
and changed the types to GL_RGBA
.
Here is what the frame looks like:
EDIT AGAIN
Just realised I haven't showed the code for how pFrameRGB is set up:
//Allocate video frame for 24bit RGB that we convert to.
AVFrame *pFrameRGB;
pFrameRGB = avcodec_alloc_frame();
if(pFrameRGB == NULL) {
return -1;
}
//Allocate memory for the raw data we get when converting.
uint8_t *buffer;
int numBytes;
numBytes = avpicture_get_size(PIX_FMT_RGB24, pCodecCtx->width, pCodecCtx->height);
buffer = (uint8_t *) av_malloc(numBytes*sizeof(uint8_t));
//Associate frame with our buffer
avpicture_fill((AVPicture *) pFrameRGB, buffer, PIX_FMT_RGB24,
pCodecCtx->width, pCodecCtx->height);
Now that I ahve changed the PixelFormat
in avgpicture_get_size
to PIX_FMT_RGB24
, I've done that in SwsContext
as well and changed GluBuild2DMipmaps
to GL_RGB
and I get a slightly better image but it looks like I'm still missing lines and it's still a bit stretched:
Another Edit
After following Macke's advice and passing the actual resolution to OpenGL I get the frames nearly proper but still a bit skewed and in black and white, also it's only getting 6fps now rather than 110fps:
P.S.
I've got a function to save the frames to image after sws_scale()
and they are coming out fine as colour and everything so something in OGL is making it B&W.
LAST EDIT
Working! Okay I have it working now, basically I am not padding out the texture to a power of 2 and just using the resolution the video is.
I got the texture showing up properly with a lucky guess at the correct glPixelStorei()
glPixelStorei(GL_UNPACK_ALIGNMENT, 2);
Also, if anyone else has the subimage()
showing blank problem like me, you have to fill the texture at least once with glTexImage2D()
and so I use it once in the loop and then use glTexSubImage2D()
after that.
Thanks Macke and datenwolf for all your help.
opengl ffmpeg textures video-processing render-to-texture
I'm trying to render frames grabbed and converted from a video using ffmpeg to an OpenGL texture to be put on a quad. I've pretty much exhausted google and not found an answer, well I've found answers but none of them seem to have worked.
Basically, I am using avcodec_decode_video2()
to decode the frame and then sws_scale()
to convert the frame to RGB and then glTexSubImage2D()
to create an openGL texture from it but can't seem to get anything to work.
I've made sure the "destination" AVFrame has power of 2 dimensions in the SWS Context setup. Here is my code:
SwsContext *img_convert_ctx = sws_getContext(pCodecCtx->width,
pCodecCtx->height, pCodecCtx->pix_fmt, 512,
256, PIX_FMT_RGB24, SWS_BICUBIC, NULL,
NULL, NULL);
//While still frames to read
while(av_read_frame(pFormatCtx, &packet)>=0) {
glClear(GL_COLOR_BUFFER_BIT);
//If the packet is from the video stream
if(packet.stream_index == videoStream) {
//Decode the video
avcodec_decode_video2(pCodecCtx, pFrame, &frameFinished, &packet);
//If we got a frame then convert it and put it into RGB buffer
if(frameFinished) {
printf("frame finished: %in", number);
sws_scale(img_convert_ctx, pFrame->data, pFrame->linesize, 0, pCodecCtx->height, pFrameRGB->data, pFrameRGB->linesize);
glBindTexture(GL_TEXTURE_2D, texture);
//gluBuild2DMipmaps(GL_TEXTURE_2D, 3, pCodecCtx->width, pCodecCtx->height, GL_RGB, GL_UNSIGNED_INT, pFrameRGB->data);
glTexSubImage2D(GL_TEXTURE_2D, 0, 0,0, 512, 256, GL_RGB, GL_UNSIGNED_BYTE, pFrameRGB->data[0]);
SaveFrame(pFrameRGB, pCodecCtx->width, pCodecCtx->height, number);
number++;
}
}
glColor3f(1,1,1);
glBindTexture(GL_TEXTURE_2D, texture);
glBegin(GL_QUADS);
glTexCoord2f(0,1);
glVertex3f(0,0,0);
glTexCoord2f(1,1);
glVertex3f(pCodecCtx->width,0,0);
glTexCoord2f(1,0);
glVertex3f(pCodecCtx->width, pCodecCtx->height,0);
glTexCoord2f(0,0);
glVertex3f(0,pCodecCtx->height,0);
glEnd();
As you can see in that code, I am also saving the frames to .ppm files just to make sure they are actually rendering, which they are.
The file being used is a .wmv at 854x480, could this be the problem? The fact I'm just telling it to go 512x256?
P.S. I've looked at this Stack Overflow question but it didn't help.
Also, I have glEnable(GL_TEXTURE_2D)
as well and have tested it by just loading in a normal bmp.
EDIT
I'm getting an image on the screen now but it is a garbled mess, I'm guessing something to do with changing things to a power of 2 (in the decode, swscontext
and gluBuild2DMipmaps
as shown in my code). I'm usually nearly exactly the same code as shown above, only I've changed glTexSubImage2D
to gluBuild2DMipmaps
and changed the types to GL_RGBA
.
Here is what the frame looks like:
EDIT AGAIN
Just realised I haven't showed the code for how pFrameRGB is set up:
//Allocate video frame for 24bit RGB that we convert to.
AVFrame *pFrameRGB;
pFrameRGB = avcodec_alloc_frame();
if(pFrameRGB == NULL) {
return -1;
}
//Allocate memory for the raw data we get when converting.
uint8_t *buffer;
int numBytes;
numBytes = avpicture_get_size(PIX_FMT_RGB24, pCodecCtx->width, pCodecCtx->height);
buffer = (uint8_t *) av_malloc(numBytes*sizeof(uint8_t));
//Associate frame with our buffer
avpicture_fill((AVPicture *) pFrameRGB, buffer, PIX_FMT_RGB24,
pCodecCtx->width, pCodecCtx->height);
Now that I ahve changed the PixelFormat
in avgpicture_get_size
to PIX_FMT_RGB24
, I've done that in SwsContext
as well and changed GluBuild2DMipmaps
to GL_RGB
and I get a slightly better image but it looks like I'm still missing lines and it's still a bit stretched:
Another Edit
After following Macke's advice and passing the actual resolution to OpenGL I get the frames nearly proper but still a bit skewed and in black and white, also it's only getting 6fps now rather than 110fps:
P.S.
I've got a function to save the frames to image after sws_scale()
and they are coming out fine as colour and everything so something in OGL is making it B&W.
LAST EDIT
Working! Okay I have it working now, basically I am not padding out the texture to a power of 2 and just using the resolution the video is.
I got the texture showing up properly with a lucky guess at the correct glPixelStorei()
glPixelStorei(GL_UNPACK_ALIGNMENT, 2);
Also, if anyone else has the subimage()
showing blank problem like me, you have to fill the texture at least once with glTexImage2D()
and so I use it once in the loop and then use glTexSubImage2D()
after that.
Thanks Macke and datenwolf for all your help.
opengl ffmpeg textures video-processing render-to-texture
opengl ffmpeg textures video-processing render-to-texture
edited May 23 '17 at 11:47
Community♦
11
11
asked Jun 27 '11 at 15:53
Infiniti FizzInfiniti Fizz
97632037
97632037
1
Voting to close. IMO, this question is getting out of hand. You need more hand-holding than SO is meant to support here. There are many issues that needs covering, and re-editing the question constantly isn't very good. I suggest to try a mail-list or a forum, or the SO OpenGL chat-room.
– Macke
Jun 28 '11 at 10:27
Okay well the issues are done now thanks to glPixelStorei. And I've deleted the comments etc. that are not helpful to anyone else approaching this question with the same problem.
– Infiniti Fizz
Jun 28 '11 at 13:17
3
Also, as the question is about getting an ffmpeg frame to an opengl texture, I don't understand how the question got out of hand, given all the questions and answers were about uploading an ffmpeg frame to an opengl texture, no matter if it was about performance (gluBuild2DMipmaps(), glTexSubImage2D()), pixel storage (glPixelStorei() etc.) ffmpeg's execution (sws_scale, SwsContext) or anything else to do with the problem. And editing the question is useful for others with the same problem as they can easily see the steps taken to resolve the issue.
– Infiniti Fizz
Jun 28 '11 at 14:19
@InfinitiFizz: Hey I'm trying to achieve same thing by creating a browser plugin, however, all I get is a black screen, can you please guide me?
– Dipen Shah
Jul 19 '16 at 18:20
add a comment |
1
Voting to close. IMO, this question is getting out of hand. You need more hand-holding than SO is meant to support here. There are many issues that needs covering, and re-editing the question constantly isn't very good. I suggest to try a mail-list or a forum, or the SO OpenGL chat-room.
– Macke
Jun 28 '11 at 10:27
Okay well the issues are done now thanks to glPixelStorei. And I've deleted the comments etc. that are not helpful to anyone else approaching this question with the same problem.
– Infiniti Fizz
Jun 28 '11 at 13:17
3
Also, as the question is about getting an ffmpeg frame to an opengl texture, I don't understand how the question got out of hand, given all the questions and answers were about uploading an ffmpeg frame to an opengl texture, no matter if it was about performance (gluBuild2DMipmaps(), glTexSubImage2D()), pixel storage (glPixelStorei() etc.) ffmpeg's execution (sws_scale, SwsContext) or anything else to do with the problem. And editing the question is useful for others with the same problem as they can easily see the steps taken to resolve the issue.
– Infiniti Fizz
Jun 28 '11 at 14:19
@InfinitiFizz: Hey I'm trying to achieve same thing by creating a browser plugin, however, all I get is a black screen, can you please guide me?
– Dipen Shah
Jul 19 '16 at 18:20
1
1
Voting to close. IMO, this question is getting out of hand. You need more hand-holding than SO is meant to support here. There are many issues that needs covering, and re-editing the question constantly isn't very good. I suggest to try a mail-list or a forum, or the SO OpenGL chat-room.
– Macke
Jun 28 '11 at 10:27
Voting to close. IMO, this question is getting out of hand. You need more hand-holding than SO is meant to support here. There are many issues that needs covering, and re-editing the question constantly isn't very good. I suggest to try a mail-list or a forum, or the SO OpenGL chat-room.
– Macke
Jun 28 '11 at 10:27
Okay well the issues are done now thanks to glPixelStorei. And I've deleted the comments etc. that are not helpful to anyone else approaching this question with the same problem.
– Infiniti Fizz
Jun 28 '11 at 13:17
Okay well the issues are done now thanks to glPixelStorei. And I've deleted the comments etc. that are not helpful to anyone else approaching this question with the same problem.
– Infiniti Fizz
Jun 28 '11 at 13:17
3
3
Also, as the question is about getting an ffmpeg frame to an opengl texture, I don't understand how the question got out of hand, given all the questions and answers were about uploading an ffmpeg frame to an opengl texture, no matter if it was about performance (gluBuild2DMipmaps(), glTexSubImage2D()), pixel storage (glPixelStorei() etc.) ffmpeg's execution (sws_scale, SwsContext) or anything else to do with the problem. And editing the question is useful for others with the same problem as they can easily see the steps taken to resolve the issue.
– Infiniti Fizz
Jun 28 '11 at 14:19
Also, as the question is about getting an ffmpeg frame to an opengl texture, I don't understand how the question got out of hand, given all the questions and answers were about uploading an ffmpeg frame to an opengl texture, no matter if it was about performance (gluBuild2DMipmaps(), glTexSubImage2D()), pixel storage (glPixelStorei() etc.) ffmpeg's execution (sws_scale, SwsContext) or anything else to do with the problem. And editing the question is useful for others with the same problem as they can easily see the steps taken to resolve the issue.
– Infiniti Fizz
Jun 28 '11 at 14:19
@InfinitiFizz: Hey I'm trying to achieve same thing by creating a browser plugin, however, all I get is a black screen, can you please guide me?
– Dipen Shah
Jul 19 '16 at 18:20
@InfinitiFizz: Hey I'm trying to achieve same thing by creating a browser plugin, however, all I get is a black screen, can you please guide me?
– Dipen Shah
Jul 19 '16 at 18:20
add a comment |
2 Answers
2
active
oldest
votes
Is the texture initialized when you
callglTexSubImage2D
? You need to
callglTexImage2D
(not Sub) one
time to initialize the texture object.
Use NULL for the data pointer, OpenGL
will then initialize a texture without
copying data.
answered
EDIT
You're not supplying mipmaping levels. So did you disable mipmaping?
glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MIN_FILER, linear_interpolation ? GL_LINEAR : GL_NEAREST);
EDIT 2 the image being upside down is no suprise as most image formats have the origin in the upper left, while OpenGL places the texture image's origin in the lower left. That banding you see there looks like wrong row stride.
EDIT 3
I did this kind of stuff myself about a year ago. I wrote me a small wrapper for ffmpeg, I called it aveasy https://github.com/datenwolf/aveasy
And this is some code to put the data fetched using aveasy into OpenGL textures:
#include <stdlib.h>
#include <stdint.h>
#include <stdio.h>
#include <string.h>
#include <math.h>
#include <GL/glew.h>
#include "camera.h"
#include "aveasy.h"
#define CAM_DESIRED_WIDTH 640
#define CAM_DESIRED_HEIGHT 480
AVEasyInputContext *camera_av;
char const *camera_path = "/dev/video0";
GLuint camera_texture;
int open_camera(void)
{
glGenTextures(1, &camera_texture);
AVEasyInputContext *ctx;
ctx = aveasy_input_open_v4l2(
camera_path,
CAM_DESIRED_WIDTH,
CAM_DESIRED_HEIGHT,
CODEC_ID_MJPEG,
PIX_FMT_BGR24 );
camera_av = ctx;
if(!ctx) {
return 0;
}
/* OpenGL-2 or later is assumed; OpenGL-2 supports NPOT textures. */
glBindTexture(GL_TEXTURE_2D, camera_texture[i]);
glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MIN_FILTER, GL_LINEAR);
glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MAG_FILTER, GL_LINEAR);
glTexImage2D(
GL_TEXTURE_2D,
0,
GL_RGB,
aveasy_input_width(ctx),
aveasy_input_height(ctx),
0,
GL_BGR,
GL_UNSIGNED_BYTE,
NULL );
return 1;
}
void update_camera(void)
{
glPixelStorei( GL_UNPACK_SWAP_BYTES, GL_FALSE );
glPixelStorei( GL_UNPACK_LSB_FIRST, GL_TRUE );
glPixelStorei( GL_UNPACK_ROW_LENGTH, 0 );
glPixelStorei( GL_UNPACK_SKIP_PIXELS, 0);
glPixelStorei( GL_UNPACK_SKIP_ROWS, 0);
glPixelStorei( GL_UNPACK_ALIGNMENT, 1);
AVEasyInputContext *ctx = camera_av;
void *buffer;
if(!ctx)
return;
if( !( buffer = aveasy_input_read_frame(ctx) ) )
return;
glBindTexture(GL_TEXTURE_2D, camera_texture);
glTexSubImage2D(
GL_TEXTURE_2D,
0,
0,
0,
aveasy_input_width(ctx),
aveasy_input_height(ctx),
GL_BGR,
GL_UNSIGNED_BYTE,
buffer );
}
void close_cameras(void)
{
aveasy_input_close(camera_av);
camera_av=0;
}
I'm using this in a project and it works there, so this code is tested, sort of.
Yes sorry it is, I actually went back to glTexImage2D after using Sub to make sure that that wasn't the problem.
– Infiniti Fizz
Jun 27 '11 at 16:46
@Infinity Fizz: See my first edit for another possible cause of your problem.
– datenwolf
Jun 27 '11 at 16:50
I think it's hard to say, we need a bit more context. I assume you're just seeing back? What happens when you try a simple debug texture? What happens when you don't set a texture for rendering, does a colored rectangle turn up correctly? Where are your calls to glGetError?
– El Marcel
Jun 28 '11 at 1:46
Hi, I've changed it to use gluBuild2DMipmaps instead just to rule out the possibility of mipmapping not being disabled and now it works! Well, I get a texture but it is a garbled version of the actual frames but I can kind of pick out the image properly. I don't know what to do to get the frame correctly though. I'm outputting to RGBA and using GL_UNSIGNED_BYTE in gluBuild2DMipmaps(). I've tried GL_RGB and messing around with the height and width of the swscontext, the decode and the gluBuild2DMipmaps and nothing has worked. I have posted above in my original question what the image looks like
– Infiniti Fizz
Jun 28 '11 at 8:49
Ah that's really cool datenwolf, but I really want to try and do it myself but if worse comes to worse I'll look at that thanks :-)
– Infiniti Fizz
Jun 28 '11 at 10:18
add a comment |
The file being used is a .wmv at
854x480, could this be the problem?
The fact I'm just telling it to go
512x256?
Yes!
The striped pattern is an obvious indication that you're mismatching data sizes (row-size.). (Since the colors are correct, RGB vs BGR vs BGRA and n-components is correct.)
You're telling OpenGL that the texture you're uploading is 512x256 (which it isn't, AFAICT). Use the real dimensions (NPOT, your card ought to support it if it's not ancient).
Otherwise, resize/pad your data before uploading it as a 1024x512 texture.
Update
I'm more familiar with OpenGL that the other functions you're calling.
sxs_scale might to what you want (i.e. scaling the image down to a pot-size).
However, scaling each frame might be slow.
I'd use the padding instead (which means, copy a small image (your video) into a part of a big texture (opengl)
Some other tips:
- Do you really need mipmaps? Only generate them if you need to downscale your texture smoothly (usually only needed when it's on some 3d-geometry).
- Avoid mipmap generation at runtime if you're rendering a video (especially, don't use gluBuildMipMaps2D, as that might run in software. There are other ways that is faster, if you need mipmapping (such as using GL_GENERATE_MIPMAP texture parameter). Seee this thread for more info.
- Avoid calling glTexImage repeatedly, as that creates a new texture. glTexSubImage just updates a part of a texture, which might work be better for you.
- If you want to upload the texture in a single step (which is preferable for performance reasons), but the data doesn't quite fit, look into
glPixelStore
to set pixel and row strides. I suspect that the data given from sxs_scale/wmw has some padding at the end of each row (the black line). Probably so that each row starts on an even 8-16-32-byte boundary.
Ah okay, well that makes sense but I didn't really know what else to do. If I try and put 854x480 into gluBuild2DMipmaps I'm only getting 6fps where I was getting around 110fps before, guessing it's to do with the power of 2 thing, also the video now seems to be in B&W (see the image in the original question now). When you say "resize/pad your data before uploading" do you mean doing it at the sws_scale() stage then? As in I should change Sws_Context to only get say 512x256 of the image, or pad it to 1024x512, how would I go about doing this?
– Infiniti Fizz
Jun 28 '11 at 10:09
@Infinity Fizz: gluBuild2DMimaps2D goes through 2 bottlenecks: First it created a image pyramid of the input image (i.e. it downsamples it) and then it goes through the costly glTexImage2D instead of the faster glTexSubImage2D.
– datenwolf
Jun 28 '11 at 10:16
@datenwolf neither of those are the main bottleneck. The core issue is that gluBuild2DMimaps2D re-samples the texture to a power of two then constructs the image pyramid ON THE CPU.
– fintelia
Jul 17 '14 at 17:37
@fintelia: "Generating the image pyramid" (which is what I wrote) is the ON THE CPU operation. And glTexImage2D is a costly operation. Both of them are bottlenecks, the image pyramid generation being the larger one (by several oders of magnitude); glTexImage2D can not be neglected either.
– datenwolf
Jul 17 '14 at 17:54
add a comment |
protected by bummi Jan 4 '16 at 8:07
Thank you for your interest in this question.
Because it has attracted low-quality or spam answers that had to be removed, posting an answer now requires 10 reputation on this site (the association bonus does not count).
Would you like to answer one of these unanswered questions instead?
2 Answers
2
active
oldest
votes
2 Answers
2
active
oldest
votes
active
oldest
votes
active
oldest
votes
Is the texture initialized when you
callglTexSubImage2D
? You need to
callglTexImage2D
(not Sub) one
time to initialize the texture object.
Use NULL for the data pointer, OpenGL
will then initialize a texture without
copying data.
answered
EDIT
You're not supplying mipmaping levels. So did you disable mipmaping?
glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MIN_FILER, linear_interpolation ? GL_LINEAR : GL_NEAREST);
EDIT 2 the image being upside down is no suprise as most image formats have the origin in the upper left, while OpenGL places the texture image's origin in the lower left. That banding you see there looks like wrong row stride.
EDIT 3
I did this kind of stuff myself about a year ago. I wrote me a small wrapper for ffmpeg, I called it aveasy https://github.com/datenwolf/aveasy
And this is some code to put the data fetched using aveasy into OpenGL textures:
#include <stdlib.h>
#include <stdint.h>
#include <stdio.h>
#include <string.h>
#include <math.h>
#include <GL/glew.h>
#include "camera.h"
#include "aveasy.h"
#define CAM_DESIRED_WIDTH 640
#define CAM_DESIRED_HEIGHT 480
AVEasyInputContext *camera_av;
char const *camera_path = "/dev/video0";
GLuint camera_texture;
int open_camera(void)
{
glGenTextures(1, &camera_texture);
AVEasyInputContext *ctx;
ctx = aveasy_input_open_v4l2(
camera_path,
CAM_DESIRED_WIDTH,
CAM_DESIRED_HEIGHT,
CODEC_ID_MJPEG,
PIX_FMT_BGR24 );
camera_av = ctx;
if(!ctx) {
return 0;
}
/* OpenGL-2 or later is assumed; OpenGL-2 supports NPOT textures. */
glBindTexture(GL_TEXTURE_2D, camera_texture[i]);
glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MIN_FILTER, GL_LINEAR);
glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MAG_FILTER, GL_LINEAR);
glTexImage2D(
GL_TEXTURE_2D,
0,
GL_RGB,
aveasy_input_width(ctx),
aveasy_input_height(ctx),
0,
GL_BGR,
GL_UNSIGNED_BYTE,
NULL );
return 1;
}
void update_camera(void)
{
glPixelStorei( GL_UNPACK_SWAP_BYTES, GL_FALSE );
glPixelStorei( GL_UNPACK_LSB_FIRST, GL_TRUE );
glPixelStorei( GL_UNPACK_ROW_LENGTH, 0 );
glPixelStorei( GL_UNPACK_SKIP_PIXELS, 0);
glPixelStorei( GL_UNPACK_SKIP_ROWS, 0);
glPixelStorei( GL_UNPACK_ALIGNMENT, 1);
AVEasyInputContext *ctx = camera_av;
void *buffer;
if(!ctx)
return;
if( !( buffer = aveasy_input_read_frame(ctx) ) )
return;
glBindTexture(GL_TEXTURE_2D, camera_texture);
glTexSubImage2D(
GL_TEXTURE_2D,
0,
0,
0,
aveasy_input_width(ctx),
aveasy_input_height(ctx),
GL_BGR,
GL_UNSIGNED_BYTE,
buffer );
}
void close_cameras(void)
{
aveasy_input_close(camera_av);
camera_av=0;
}
I'm using this in a project and it works there, so this code is tested, sort of.
Yes sorry it is, I actually went back to glTexImage2D after using Sub to make sure that that wasn't the problem.
– Infiniti Fizz
Jun 27 '11 at 16:46
@Infinity Fizz: See my first edit for another possible cause of your problem.
– datenwolf
Jun 27 '11 at 16:50
I think it's hard to say, we need a bit more context. I assume you're just seeing back? What happens when you try a simple debug texture? What happens when you don't set a texture for rendering, does a colored rectangle turn up correctly? Where are your calls to glGetError?
– El Marcel
Jun 28 '11 at 1:46
Hi, I've changed it to use gluBuild2DMipmaps instead just to rule out the possibility of mipmapping not being disabled and now it works! Well, I get a texture but it is a garbled version of the actual frames but I can kind of pick out the image properly. I don't know what to do to get the frame correctly though. I'm outputting to RGBA and using GL_UNSIGNED_BYTE in gluBuild2DMipmaps(). I've tried GL_RGB and messing around with the height and width of the swscontext, the decode and the gluBuild2DMipmaps and nothing has worked. I have posted above in my original question what the image looks like
– Infiniti Fizz
Jun 28 '11 at 8:49
Ah that's really cool datenwolf, but I really want to try and do it myself but if worse comes to worse I'll look at that thanks :-)
– Infiniti Fizz
Jun 28 '11 at 10:18
add a comment |
Is the texture initialized when you
callglTexSubImage2D
? You need to
callglTexImage2D
(not Sub) one
time to initialize the texture object.
Use NULL for the data pointer, OpenGL
will then initialize a texture without
copying data.
answered
EDIT
You're not supplying mipmaping levels. So did you disable mipmaping?
glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MIN_FILER, linear_interpolation ? GL_LINEAR : GL_NEAREST);
EDIT 2 the image being upside down is no suprise as most image formats have the origin in the upper left, while OpenGL places the texture image's origin in the lower left. That banding you see there looks like wrong row stride.
EDIT 3
I did this kind of stuff myself about a year ago. I wrote me a small wrapper for ffmpeg, I called it aveasy https://github.com/datenwolf/aveasy
And this is some code to put the data fetched using aveasy into OpenGL textures:
#include <stdlib.h>
#include <stdint.h>
#include <stdio.h>
#include <string.h>
#include <math.h>
#include <GL/glew.h>
#include "camera.h"
#include "aveasy.h"
#define CAM_DESIRED_WIDTH 640
#define CAM_DESIRED_HEIGHT 480
AVEasyInputContext *camera_av;
char const *camera_path = "/dev/video0";
GLuint camera_texture;
int open_camera(void)
{
glGenTextures(1, &camera_texture);
AVEasyInputContext *ctx;
ctx = aveasy_input_open_v4l2(
camera_path,
CAM_DESIRED_WIDTH,
CAM_DESIRED_HEIGHT,
CODEC_ID_MJPEG,
PIX_FMT_BGR24 );
camera_av = ctx;
if(!ctx) {
return 0;
}
/* OpenGL-2 or later is assumed; OpenGL-2 supports NPOT textures. */
glBindTexture(GL_TEXTURE_2D, camera_texture[i]);
glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MIN_FILTER, GL_LINEAR);
glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MAG_FILTER, GL_LINEAR);
glTexImage2D(
GL_TEXTURE_2D,
0,
GL_RGB,
aveasy_input_width(ctx),
aveasy_input_height(ctx),
0,
GL_BGR,
GL_UNSIGNED_BYTE,
NULL );
return 1;
}
void update_camera(void)
{
glPixelStorei( GL_UNPACK_SWAP_BYTES, GL_FALSE );
glPixelStorei( GL_UNPACK_LSB_FIRST, GL_TRUE );
glPixelStorei( GL_UNPACK_ROW_LENGTH, 0 );
glPixelStorei( GL_UNPACK_SKIP_PIXELS, 0);
glPixelStorei( GL_UNPACK_SKIP_ROWS, 0);
glPixelStorei( GL_UNPACK_ALIGNMENT, 1);
AVEasyInputContext *ctx = camera_av;
void *buffer;
if(!ctx)
return;
if( !( buffer = aveasy_input_read_frame(ctx) ) )
return;
glBindTexture(GL_TEXTURE_2D, camera_texture);
glTexSubImage2D(
GL_TEXTURE_2D,
0,
0,
0,
aveasy_input_width(ctx),
aveasy_input_height(ctx),
GL_BGR,
GL_UNSIGNED_BYTE,
buffer );
}
void close_cameras(void)
{
aveasy_input_close(camera_av);
camera_av=0;
}
I'm using this in a project and it works there, so this code is tested, sort of.
Yes sorry it is, I actually went back to glTexImage2D after using Sub to make sure that that wasn't the problem.
– Infiniti Fizz
Jun 27 '11 at 16:46
@Infinity Fizz: See my first edit for another possible cause of your problem.
– datenwolf
Jun 27 '11 at 16:50
I think it's hard to say, we need a bit more context. I assume you're just seeing back? What happens when you try a simple debug texture? What happens when you don't set a texture for rendering, does a colored rectangle turn up correctly? Where are your calls to glGetError?
– El Marcel
Jun 28 '11 at 1:46
Hi, I've changed it to use gluBuild2DMipmaps instead just to rule out the possibility of mipmapping not being disabled and now it works! Well, I get a texture but it is a garbled version of the actual frames but I can kind of pick out the image properly. I don't know what to do to get the frame correctly though. I'm outputting to RGBA and using GL_UNSIGNED_BYTE in gluBuild2DMipmaps(). I've tried GL_RGB and messing around with the height and width of the swscontext, the decode and the gluBuild2DMipmaps and nothing has worked. I have posted above in my original question what the image looks like
– Infiniti Fizz
Jun 28 '11 at 8:49
Ah that's really cool datenwolf, but I really want to try and do it myself but if worse comes to worse I'll look at that thanks :-)
– Infiniti Fizz
Jun 28 '11 at 10:18
add a comment |
Is the texture initialized when you
callglTexSubImage2D
? You need to
callglTexImage2D
(not Sub) one
time to initialize the texture object.
Use NULL for the data pointer, OpenGL
will then initialize a texture without
copying data.
answered
EDIT
You're not supplying mipmaping levels. So did you disable mipmaping?
glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MIN_FILER, linear_interpolation ? GL_LINEAR : GL_NEAREST);
EDIT 2 the image being upside down is no suprise as most image formats have the origin in the upper left, while OpenGL places the texture image's origin in the lower left. That banding you see there looks like wrong row stride.
EDIT 3
I did this kind of stuff myself about a year ago. I wrote me a small wrapper for ffmpeg, I called it aveasy https://github.com/datenwolf/aveasy
And this is some code to put the data fetched using aveasy into OpenGL textures:
#include <stdlib.h>
#include <stdint.h>
#include <stdio.h>
#include <string.h>
#include <math.h>
#include <GL/glew.h>
#include "camera.h"
#include "aveasy.h"
#define CAM_DESIRED_WIDTH 640
#define CAM_DESIRED_HEIGHT 480
AVEasyInputContext *camera_av;
char const *camera_path = "/dev/video0";
GLuint camera_texture;
int open_camera(void)
{
glGenTextures(1, &camera_texture);
AVEasyInputContext *ctx;
ctx = aveasy_input_open_v4l2(
camera_path,
CAM_DESIRED_WIDTH,
CAM_DESIRED_HEIGHT,
CODEC_ID_MJPEG,
PIX_FMT_BGR24 );
camera_av = ctx;
if(!ctx) {
return 0;
}
/* OpenGL-2 or later is assumed; OpenGL-2 supports NPOT textures. */
glBindTexture(GL_TEXTURE_2D, camera_texture[i]);
glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MIN_FILTER, GL_LINEAR);
glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MAG_FILTER, GL_LINEAR);
glTexImage2D(
GL_TEXTURE_2D,
0,
GL_RGB,
aveasy_input_width(ctx),
aveasy_input_height(ctx),
0,
GL_BGR,
GL_UNSIGNED_BYTE,
NULL );
return 1;
}
void update_camera(void)
{
glPixelStorei( GL_UNPACK_SWAP_BYTES, GL_FALSE );
glPixelStorei( GL_UNPACK_LSB_FIRST, GL_TRUE );
glPixelStorei( GL_UNPACK_ROW_LENGTH, 0 );
glPixelStorei( GL_UNPACK_SKIP_PIXELS, 0);
glPixelStorei( GL_UNPACK_SKIP_ROWS, 0);
glPixelStorei( GL_UNPACK_ALIGNMENT, 1);
AVEasyInputContext *ctx = camera_av;
void *buffer;
if(!ctx)
return;
if( !( buffer = aveasy_input_read_frame(ctx) ) )
return;
glBindTexture(GL_TEXTURE_2D, camera_texture);
glTexSubImage2D(
GL_TEXTURE_2D,
0,
0,
0,
aveasy_input_width(ctx),
aveasy_input_height(ctx),
GL_BGR,
GL_UNSIGNED_BYTE,
buffer );
}
void close_cameras(void)
{
aveasy_input_close(camera_av);
camera_av=0;
}
I'm using this in a project and it works there, so this code is tested, sort of.
Is the texture initialized when you
callglTexSubImage2D
? You need to
callglTexImage2D
(not Sub) one
time to initialize the texture object.
Use NULL for the data pointer, OpenGL
will then initialize a texture without
copying data.
answered
EDIT
You're not supplying mipmaping levels. So did you disable mipmaping?
glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MIN_FILER, linear_interpolation ? GL_LINEAR : GL_NEAREST);
EDIT 2 the image being upside down is no suprise as most image formats have the origin in the upper left, while OpenGL places the texture image's origin in the lower left. That banding you see there looks like wrong row stride.
EDIT 3
I did this kind of stuff myself about a year ago. I wrote me a small wrapper for ffmpeg, I called it aveasy https://github.com/datenwolf/aveasy
And this is some code to put the data fetched using aveasy into OpenGL textures:
#include <stdlib.h>
#include <stdint.h>
#include <stdio.h>
#include <string.h>
#include <math.h>
#include <GL/glew.h>
#include "camera.h"
#include "aveasy.h"
#define CAM_DESIRED_WIDTH 640
#define CAM_DESIRED_HEIGHT 480
AVEasyInputContext *camera_av;
char const *camera_path = "/dev/video0";
GLuint camera_texture;
int open_camera(void)
{
glGenTextures(1, &camera_texture);
AVEasyInputContext *ctx;
ctx = aveasy_input_open_v4l2(
camera_path,
CAM_DESIRED_WIDTH,
CAM_DESIRED_HEIGHT,
CODEC_ID_MJPEG,
PIX_FMT_BGR24 );
camera_av = ctx;
if(!ctx) {
return 0;
}
/* OpenGL-2 or later is assumed; OpenGL-2 supports NPOT textures. */
glBindTexture(GL_TEXTURE_2D, camera_texture[i]);
glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MIN_FILTER, GL_LINEAR);
glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MAG_FILTER, GL_LINEAR);
glTexImage2D(
GL_TEXTURE_2D,
0,
GL_RGB,
aveasy_input_width(ctx),
aveasy_input_height(ctx),
0,
GL_BGR,
GL_UNSIGNED_BYTE,
NULL );
return 1;
}
void update_camera(void)
{
glPixelStorei( GL_UNPACK_SWAP_BYTES, GL_FALSE );
glPixelStorei( GL_UNPACK_LSB_FIRST, GL_TRUE );
glPixelStorei( GL_UNPACK_ROW_LENGTH, 0 );
glPixelStorei( GL_UNPACK_SKIP_PIXELS, 0);
glPixelStorei( GL_UNPACK_SKIP_ROWS, 0);
glPixelStorei( GL_UNPACK_ALIGNMENT, 1);
AVEasyInputContext *ctx = camera_av;
void *buffer;
if(!ctx)
return;
if( !( buffer = aveasy_input_read_frame(ctx) ) )
return;
glBindTexture(GL_TEXTURE_2D, camera_texture);
glTexSubImage2D(
GL_TEXTURE_2D,
0,
0,
0,
aveasy_input_width(ctx),
aveasy_input_height(ctx),
GL_BGR,
GL_UNSIGNED_BYTE,
buffer );
}
void close_cameras(void)
{
aveasy_input_close(camera_av);
camera_av=0;
}
I'm using this in a project and it works there, so this code is tested, sort of.
edited Jun 28 '11 at 10:17
answered Jun 27 '11 at 16:32
datenwolfdatenwolf
133k10135237
133k10135237
Yes sorry it is, I actually went back to glTexImage2D after using Sub to make sure that that wasn't the problem.
– Infiniti Fizz
Jun 27 '11 at 16:46
@Infinity Fizz: See my first edit for another possible cause of your problem.
– datenwolf
Jun 27 '11 at 16:50
I think it's hard to say, we need a bit more context. I assume you're just seeing back? What happens when you try a simple debug texture? What happens when you don't set a texture for rendering, does a colored rectangle turn up correctly? Where are your calls to glGetError?
– El Marcel
Jun 28 '11 at 1:46
Hi, I've changed it to use gluBuild2DMipmaps instead just to rule out the possibility of mipmapping not being disabled and now it works! Well, I get a texture but it is a garbled version of the actual frames but I can kind of pick out the image properly. I don't know what to do to get the frame correctly though. I'm outputting to RGBA and using GL_UNSIGNED_BYTE in gluBuild2DMipmaps(). I've tried GL_RGB and messing around with the height and width of the swscontext, the decode and the gluBuild2DMipmaps and nothing has worked. I have posted above in my original question what the image looks like
– Infiniti Fizz
Jun 28 '11 at 8:49
Ah that's really cool datenwolf, but I really want to try and do it myself but if worse comes to worse I'll look at that thanks :-)
– Infiniti Fizz
Jun 28 '11 at 10:18
add a comment |
Yes sorry it is, I actually went back to glTexImage2D after using Sub to make sure that that wasn't the problem.
– Infiniti Fizz
Jun 27 '11 at 16:46
@Infinity Fizz: See my first edit for another possible cause of your problem.
– datenwolf
Jun 27 '11 at 16:50
I think it's hard to say, we need a bit more context. I assume you're just seeing back? What happens when you try a simple debug texture? What happens when you don't set a texture for rendering, does a colored rectangle turn up correctly? Where are your calls to glGetError?
– El Marcel
Jun 28 '11 at 1:46
Hi, I've changed it to use gluBuild2DMipmaps instead just to rule out the possibility of mipmapping not being disabled and now it works! Well, I get a texture but it is a garbled version of the actual frames but I can kind of pick out the image properly. I don't know what to do to get the frame correctly though. I'm outputting to RGBA and using GL_UNSIGNED_BYTE in gluBuild2DMipmaps(). I've tried GL_RGB and messing around with the height and width of the swscontext, the decode and the gluBuild2DMipmaps and nothing has worked. I have posted above in my original question what the image looks like
– Infiniti Fizz
Jun 28 '11 at 8:49
Ah that's really cool datenwolf, but I really want to try and do it myself but if worse comes to worse I'll look at that thanks :-)
– Infiniti Fizz
Jun 28 '11 at 10:18
Yes sorry it is, I actually went back to glTexImage2D after using Sub to make sure that that wasn't the problem.
– Infiniti Fizz
Jun 27 '11 at 16:46
Yes sorry it is, I actually went back to glTexImage2D after using Sub to make sure that that wasn't the problem.
– Infiniti Fizz
Jun 27 '11 at 16:46
@Infinity Fizz: See my first edit for another possible cause of your problem.
– datenwolf
Jun 27 '11 at 16:50
@Infinity Fizz: See my first edit for another possible cause of your problem.
– datenwolf
Jun 27 '11 at 16:50
I think it's hard to say, we need a bit more context. I assume you're just seeing back? What happens when you try a simple debug texture? What happens when you don't set a texture for rendering, does a colored rectangle turn up correctly? Where are your calls to glGetError?
– El Marcel
Jun 28 '11 at 1:46
I think it's hard to say, we need a bit more context. I assume you're just seeing back? What happens when you try a simple debug texture? What happens when you don't set a texture for rendering, does a colored rectangle turn up correctly? Where are your calls to glGetError?
– El Marcel
Jun 28 '11 at 1:46
Hi, I've changed it to use gluBuild2DMipmaps instead just to rule out the possibility of mipmapping not being disabled and now it works! Well, I get a texture but it is a garbled version of the actual frames but I can kind of pick out the image properly. I don't know what to do to get the frame correctly though. I'm outputting to RGBA and using GL_UNSIGNED_BYTE in gluBuild2DMipmaps(). I've tried GL_RGB and messing around with the height and width of the swscontext, the decode and the gluBuild2DMipmaps and nothing has worked. I have posted above in my original question what the image looks like
– Infiniti Fizz
Jun 28 '11 at 8:49
Hi, I've changed it to use gluBuild2DMipmaps instead just to rule out the possibility of mipmapping not being disabled and now it works! Well, I get a texture but it is a garbled version of the actual frames but I can kind of pick out the image properly. I don't know what to do to get the frame correctly though. I'm outputting to RGBA and using GL_UNSIGNED_BYTE in gluBuild2DMipmaps(). I've tried GL_RGB and messing around with the height and width of the swscontext, the decode and the gluBuild2DMipmaps and nothing has worked. I have posted above in my original question what the image looks like
– Infiniti Fizz
Jun 28 '11 at 8:49
Ah that's really cool datenwolf, but I really want to try and do it myself but if worse comes to worse I'll look at that thanks :-)
– Infiniti Fizz
Jun 28 '11 at 10:18
Ah that's really cool datenwolf, but I really want to try and do it myself but if worse comes to worse I'll look at that thanks :-)
– Infiniti Fizz
Jun 28 '11 at 10:18
add a comment |
The file being used is a .wmv at
854x480, could this be the problem?
The fact I'm just telling it to go
512x256?
Yes!
The striped pattern is an obvious indication that you're mismatching data sizes (row-size.). (Since the colors are correct, RGB vs BGR vs BGRA and n-components is correct.)
You're telling OpenGL that the texture you're uploading is 512x256 (which it isn't, AFAICT). Use the real dimensions (NPOT, your card ought to support it if it's not ancient).
Otherwise, resize/pad your data before uploading it as a 1024x512 texture.
Update
I'm more familiar with OpenGL that the other functions you're calling.
sxs_scale might to what you want (i.e. scaling the image down to a pot-size).
However, scaling each frame might be slow.
I'd use the padding instead (which means, copy a small image (your video) into a part of a big texture (opengl)
Some other tips:
- Do you really need mipmaps? Only generate them if you need to downscale your texture smoothly (usually only needed when it's on some 3d-geometry).
- Avoid mipmap generation at runtime if you're rendering a video (especially, don't use gluBuildMipMaps2D, as that might run in software. There are other ways that is faster, if you need mipmapping (such as using GL_GENERATE_MIPMAP texture parameter). Seee this thread for more info.
- Avoid calling glTexImage repeatedly, as that creates a new texture. glTexSubImage just updates a part of a texture, which might work be better for you.
- If you want to upload the texture in a single step (which is preferable for performance reasons), but the data doesn't quite fit, look into
glPixelStore
to set pixel and row strides. I suspect that the data given from sxs_scale/wmw has some padding at the end of each row (the black line). Probably so that each row starts on an even 8-16-32-byte boundary.
Ah okay, well that makes sense but I didn't really know what else to do. If I try and put 854x480 into gluBuild2DMipmaps I'm only getting 6fps where I was getting around 110fps before, guessing it's to do with the power of 2 thing, also the video now seems to be in B&W (see the image in the original question now). When you say "resize/pad your data before uploading" do you mean doing it at the sws_scale() stage then? As in I should change Sws_Context to only get say 512x256 of the image, or pad it to 1024x512, how would I go about doing this?
– Infiniti Fizz
Jun 28 '11 at 10:09
@Infinity Fizz: gluBuild2DMimaps2D goes through 2 bottlenecks: First it created a image pyramid of the input image (i.e. it downsamples it) and then it goes through the costly glTexImage2D instead of the faster glTexSubImage2D.
– datenwolf
Jun 28 '11 at 10:16
@datenwolf neither of those are the main bottleneck. The core issue is that gluBuild2DMimaps2D re-samples the texture to a power of two then constructs the image pyramid ON THE CPU.
– fintelia
Jul 17 '14 at 17:37
@fintelia: "Generating the image pyramid" (which is what I wrote) is the ON THE CPU operation. And glTexImage2D is a costly operation. Both of them are bottlenecks, the image pyramid generation being the larger one (by several oders of magnitude); glTexImage2D can not be neglected either.
– datenwolf
Jul 17 '14 at 17:54
add a comment |
The file being used is a .wmv at
854x480, could this be the problem?
The fact I'm just telling it to go
512x256?
Yes!
The striped pattern is an obvious indication that you're mismatching data sizes (row-size.). (Since the colors are correct, RGB vs BGR vs BGRA and n-components is correct.)
You're telling OpenGL that the texture you're uploading is 512x256 (which it isn't, AFAICT). Use the real dimensions (NPOT, your card ought to support it if it's not ancient).
Otherwise, resize/pad your data before uploading it as a 1024x512 texture.
Update
I'm more familiar with OpenGL that the other functions you're calling.
sxs_scale might to what you want (i.e. scaling the image down to a pot-size).
However, scaling each frame might be slow.
I'd use the padding instead (which means, copy a small image (your video) into a part of a big texture (opengl)
Some other tips:
- Do you really need mipmaps? Only generate them if you need to downscale your texture smoothly (usually only needed when it's on some 3d-geometry).
- Avoid mipmap generation at runtime if you're rendering a video (especially, don't use gluBuildMipMaps2D, as that might run in software. There are other ways that is faster, if you need mipmapping (such as using GL_GENERATE_MIPMAP texture parameter). Seee this thread for more info.
- Avoid calling glTexImage repeatedly, as that creates a new texture. glTexSubImage just updates a part of a texture, which might work be better for you.
- If you want to upload the texture in a single step (which is preferable for performance reasons), but the data doesn't quite fit, look into
glPixelStore
to set pixel and row strides. I suspect that the data given from sxs_scale/wmw has some padding at the end of each row (the black line). Probably so that each row starts on an even 8-16-32-byte boundary.
Ah okay, well that makes sense but I didn't really know what else to do. If I try and put 854x480 into gluBuild2DMipmaps I'm only getting 6fps where I was getting around 110fps before, guessing it's to do with the power of 2 thing, also the video now seems to be in B&W (see the image in the original question now). When you say "resize/pad your data before uploading" do you mean doing it at the sws_scale() stage then? As in I should change Sws_Context to only get say 512x256 of the image, or pad it to 1024x512, how would I go about doing this?
– Infiniti Fizz
Jun 28 '11 at 10:09
@Infinity Fizz: gluBuild2DMimaps2D goes through 2 bottlenecks: First it created a image pyramid of the input image (i.e. it downsamples it) and then it goes through the costly glTexImage2D instead of the faster glTexSubImage2D.
– datenwolf
Jun 28 '11 at 10:16
@datenwolf neither of those are the main bottleneck. The core issue is that gluBuild2DMimaps2D re-samples the texture to a power of two then constructs the image pyramid ON THE CPU.
– fintelia
Jul 17 '14 at 17:37
@fintelia: "Generating the image pyramid" (which is what I wrote) is the ON THE CPU operation. And glTexImage2D is a costly operation. Both of them are bottlenecks, the image pyramid generation being the larger one (by several oders of magnitude); glTexImage2D can not be neglected either.
– datenwolf
Jul 17 '14 at 17:54
add a comment |
The file being used is a .wmv at
854x480, could this be the problem?
The fact I'm just telling it to go
512x256?
Yes!
The striped pattern is an obvious indication that you're mismatching data sizes (row-size.). (Since the colors are correct, RGB vs BGR vs BGRA and n-components is correct.)
You're telling OpenGL that the texture you're uploading is 512x256 (which it isn't, AFAICT). Use the real dimensions (NPOT, your card ought to support it if it's not ancient).
Otherwise, resize/pad your data before uploading it as a 1024x512 texture.
Update
I'm more familiar with OpenGL that the other functions you're calling.
sxs_scale might to what you want (i.e. scaling the image down to a pot-size).
However, scaling each frame might be slow.
I'd use the padding instead (which means, copy a small image (your video) into a part of a big texture (opengl)
Some other tips:
- Do you really need mipmaps? Only generate them if you need to downscale your texture smoothly (usually only needed when it's on some 3d-geometry).
- Avoid mipmap generation at runtime if you're rendering a video (especially, don't use gluBuildMipMaps2D, as that might run in software. There are other ways that is faster, if you need mipmapping (such as using GL_GENERATE_MIPMAP texture parameter). Seee this thread for more info.
- Avoid calling glTexImage repeatedly, as that creates a new texture. glTexSubImage just updates a part of a texture, which might work be better for you.
- If you want to upload the texture in a single step (which is preferable for performance reasons), but the data doesn't quite fit, look into
glPixelStore
to set pixel and row strides. I suspect that the data given from sxs_scale/wmw has some padding at the end of each row (the black line). Probably so that each row starts on an even 8-16-32-byte boundary.
The file being used is a .wmv at
854x480, could this be the problem?
The fact I'm just telling it to go
512x256?
Yes!
The striped pattern is an obvious indication that you're mismatching data sizes (row-size.). (Since the colors are correct, RGB vs BGR vs BGRA and n-components is correct.)
You're telling OpenGL that the texture you're uploading is 512x256 (which it isn't, AFAICT). Use the real dimensions (NPOT, your card ought to support it if it's not ancient).
Otherwise, resize/pad your data before uploading it as a 1024x512 texture.
Update
I'm more familiar with OpenGL that the other functions you're calling.
sxs_scale might to what you want (i.e. scaling the image down to a pot-size).
However, scaling each frame might be slow.
I'd use the padding instead (which means, copy a small image (your video) into a part of a big texture (opengl)
Some other tips:
- Do you really need mipmaps? Only generate them if you need to downscale your texture smoothly (usually only needed when it's on some 3d-geometry).
- Avoid mipmap generation at runtime if you're rendering a video (especially, don't use gluBuildMipMaps2D, as that might run in software. There are other ways that is faster, if you need mipmapping (such as using GL_GENERATE_MIPMAP texture parameter). Seee this thread for more info.
- Avoid calling glTexImage repeatedly, as that creates a new texture. glTexSubImage just updates a part of a texture, which might work be better for you.
- If you want to upload the texture in a single step (which is preferable for performance reasons), but the data doesn't quite fit, look into
glPixelStore
to set pixel and row strides. I suspect that the data given from sxs_scale/wmw has some padding at the end of each row (the black line). Probably so that each row starts on an even 8-16-32-byte boundary.
edited Jun 28 '11 at 10:21
answered Jun 28 '11 at 9:53
MackeMacke
19.2k56791
19.2k56791
Ah okay, well that makes sense but I didn't really know what else to do. If I try and put 854x480 into gluBuild2DMipmaps I'm only getting 6fps where I was getting around 110fps before, guessing it's to do with the power of 2 thing, also the video now seems to be in B&W (see the image in the original question now). When you say "resize/pad your data before uploading" do you mean doing it at the sws_scale() stage then? As in I should change Sws_Context to only get say 512x256 of the image, or pad it to 1024x512, how would I go about doing this?
– Infiniti Fizz
Jun 28 '11 at 10:09
@Infinity Fizz: gluBuild2DMimaps2D goes through 2 bottlenecks: First it created a image pyramid of the input image (i.e. it downsamples it) and then it goes through the costly glTexImage2D instead of the faster glTexSubImage2D.
– datenwolf
Jun 28 '11 at 10:16
@datenwolf neither of those are the main bottleneck. The core issue is that gluBuild2DMimaps2D re-samples the texture to a power of two then constructs the image pyramid ON THE CPU.
– fintelia
Jul 17 '14 at 17:37
@fintelia: "Generating the image pyramid" (which is what I wrote) is the ON THE CPU operation. And glTexImage2D is a costly operation. Both of them are bottlenecks, the image pyramid generation being the larger one (by several oders of magnitude); glTexImage2D can not be neglected either.
– datenwolf
Jul 17 '14 at 17:54
add a comment |
Ah okay, well that makes sense but I didn't really know what else to do. If I try and put 854x480 into gluBuild2DMipmaps I'm only getting 6fps where I was getting around 110fps before, guessing it's to do with the power of 2 thing, also the video now seems to be in B&W (see the image in the original question now). When you say "resize/pad your data before uploading" do you mean doing it at the sws_scale() stage then? As in I should change Sws_Context to only get say 512x256 of the image, or pad it to 1024x512, how would I go about doing this?
– Infiniti Fizz
Jun 28 '11 at 10:09
@Infinity Fizz: gluBuild2DMimaps2D goes through 2 bottlenecks: First it created a image pyramid of the input image (i.e. it downsamples it) and then it goes through the costly glTexImage2D instead of the faster glTexSubImage2D.
– datenwolf
Jun 28 '11 at 10:16
@datenwolf neither of those are the main bottleneck. The core issue is that gluBuild2DMimaps2D re-samples the texture to a power of two then constructs the image pyramid ON THE CPU.
– fintelia
Jul 17 '14 at 17:37
@fintelia: "Generating the image pyramid" (which is what I wrote) is the ON THE CPU operation. And glTexImage2D is a costly operation. Both of them are bottlenecks, the image pyramid generation being the larger one (by several oders of magnitude); glTexImage2D can not be neglected either.
– datenwolf
Jul 17 '14 at 17:54
Ah okay, well that makes sense but I didn't really know what else to do. If I try and put 854x480 into gluBuild2DMipmaps I'm only getting 6fps where I was getting around 110fps before, guessing it's to do with the power of 2 thing, also the video now seems to be in B&W (see the image in the original question now). When you say "resize/pad your data before uploading" do you mean doing it at the sws_scale() stage then? As in I should change Sws_Context to only get say 512x256 of the image, or pad it to 1024x512, how would I go about doing this?
– Infiniti Fizz
Jun 28 '11 at 10:09
Ah okay, well that makes sense but I didn't really know what else to do. If I try and put 854x480 into gluBuild2DMipmaps I'm only getting 6fps where I was getting around 110fps before, guessing it's to do with the power of 2 thing, also the video now seems to be in B&W (see the image in the original question now). When you say "resize/pad your data before uploading" do you mean doing it at the sws_scale() stage then? As in I should change Sws_Context to only get say 512x256 of the image, or pad it to 1024x512, how would I go about doing this?
– Infiniti Fizz
Jun 28 '11 at 10:09
@Infinity Fizz: gluBuild2DMimaps2D goes through 2 bottlenecks: First it created a image pyramid of the input image (i.e. it downsamples it) and then it goes through the costly glTexImage2D instead of the faster glTexSubImage2D.
– datenwolf
Jun 28 '11 at 10:16
@Infinity Fizz: gluBuild2DMimaps2D goes through 2 bottlenecks: First it created a image pyramid of the input image (i.e. it downsamples it) and then it goes through the costly glTexImage2D instead of the faster glTexSubImage2D.
– datenwolf
Jun 28 '11 at 10:16
@datenwolf neither of those are the main bottleneck. The core issue is that gluBuild2DMimaps2D re-samples the texture to a power of two then constructs the image pyramid ON THE CPU.
– fintelia
Jul 17 '14 at 17:37
@datenwolf neither of those are the main bottleneck. The core issue is that gluBuild2DMimaps2D re-samples the texture to a power of two then constructs the image pyramid ON THE CPU.
– fintelia
Jul 17 '14 at 17:37
@fintelia: "Generating the image pyramid" (which is what I wrote) is the ON THE CPU operation. And glTexImage2D is a costly operation. Both of them are bottlenecks, the image pyramid generation being the larger one (by several oders of magnitude); glTexImage2D can not be neglected either.
– datenwolf
Jul 17 '14 at 17:54
@fintelia: "Generating the image pyramid" (which is what I wrote) is the ON THE CPU operation. And glTexImage2D is a costly operation. Both of them are bottlenecks, the image pyramid generation being the larger one (by several oders of magnitude); glTexImage2D can not be neglected either.
– datenwolf
Jul 17 '14 at 17:54
add a comment |
protected by bummi Jan 4 '16 at 8:07
Thank you for your interest in this question.
Because it has attracted low-quality or spam answers that had to be removed, posting an answer now requires 10 reputation on this site (the association bonus does not count).
Would you like to answer one of these unanswered questions instead?
1
Voting to close. IMO, this question is getting out of hand. You need more hand-holding than SO is meant to support here. There are many issues that needs covering, and re-editing the question constantly isn't very good. I suggest to try a mail-list or a forum, or the SO OpenGL chat-room.
– Macke
Jun 28 '11 at 10:27
Okay well the issues are done now thanks to glPixelStorei. And I've deleted the comments etc. that are not helpful to anyone else approaching this question with the same problem.
– Infiniti Fizz
Jun 28 '11 at 13:17
3
Also, as the question is about getting an ffmpeg frame to an opengl texture, I don't understand how the question got out of hand, given all the questions and answers were about uploading an ffmpeg frame to an opengl texture, no matter if it was about performance (gluBuild2DMipmaps(), glTexSubImage2D()), pixel storage (glPixelStorei() etc.) ffmpeg's execution (sws_scale, SwsContext) or anything else to do with the problem. And editing the question is useful for others with the same problem as they can easily see the steps taken to resolve the issue.
– Infiniti Fizz
Jun 28 '11 at 14:19
@InfinitiFizz: Hey I'm trying to achieve same thing by creating a browser plugin, however, all I get is a black screen, can you please guide me?
– Dipen Shah
Jul 19 '16 at 18:20