iPhone – saving OpenGL ES content to the Photo Album
Thursday 15 January 2009 - Filed under Objective C + iPhone
Someone posted a sad comment the other day mourning the loss of Flash content on this blog. Sorry, I follow what I’m into. Sometimes that’s Flash, sometimes (very briefly) it was Silverlight. I think I even got into Python here for a while. The iPhone stuff may be a diversion, or it may be my future path. Time will tell.
Anyway, for an app I’m working on, I needed to take a screenshot of OpenGL ES rendered content. I assumed there was some built in function for that, but much searching led me to the conclusion that it’s a roll-your-own kind of thing. So, after a couple of days, I was finally able to piece together at least three or four different semi-working solutions from various forums and mailing lists, combined with some hacking about, to come up with a solution that actually works.
The first and last steps are easy.
First step, you read the GL data into a raw byte array with glReadPixels. Simple enough.
Last step, you save a UIImage to the Photo Album with UIImageWriteToSavedPhotosAlbum.
The tough part is getting that byte array into a UIImage. My first attempt was to use [UIImage imageFromData:data]. But the problem with that is that that method expects data to be in a file format of one of the supported image types of UIImage, whereas glReadPixels is just raw pixel data.
Digging around some more, I found [UIImage imageWithCGImage:imageRef]. You can get a CGImageRef with CGImageCreate.
CGImageCreate requires a CGDataProviderRef. And you can create one of those with CGDataProviderCreateWithData, using the results from glReadPixels! Finally, a path from one end to the other.
glReadPixels -> CGDataProviderCreateWithData -> CGImageCreate -> [UIImage imageWithCGImage:] -> UIImageWriteToSavedPhotosAlbum
Yay!
But wait. One more snag. OpenGL uses standard Cartesian coordinates. In other words, +Y is up, -Y is down. So the byte array you get with glReadPixels (and thus your final image) will be upside down. A bit of fancy bit-twiddling fixed that up. Here are the final methods, meant to be used within a UIView with a CAEAGLLayer class (just like the EAGL class in the OpenGL ES template file).
-(UIImage *) glToUIImage {
NSInteger myDataLength = 320 * 480 * 4;
// allocate array and read pixels into it.
GLubyte *buffer = (GLubyte *) malloc(myDataLength);
glReadPixels(0, 0, 320, 480, GL_RGBA, GL_UNSIGNED_BYTE, buffer);
// gl renders "upside down" so swap top to bottom into new array.
// there's gotta be a better way, but this works.
GLubyte *buffer2 = (GLubyte *) malloc(myDataLength);
for(int y = 0; y < 480; y++)
{
for(int x = 0; x < 320 * 4; x++)
{
buffer2[(479 - y) * 320 * 4 + x] = buffer[y * 4 * 320 + x];
}
}
// make data provider with data.
CGDataProviderRef provider = CGDataProviderCreateWithData(NULL, buffer2, myDataLength, NULL);
// prep the ingredients
int bitsPerComponent = 8;
int bitsPerPixel = 32;
int bytesPerRow = 4 * 320;
CGColorSpaceRef colorSpaceRef = CGColorSpaceCreateDeviceRGB();
CGBitmapInfo bitmapInfo = kCGBitmapByteOrderDefault;
CGColorRenderingIntent renderingIntent = kCGRenderingIntentDefault;
// make the cgimage
CGImageRef imageRef = CGImageCreate(320, 480, bitsPerComponent, bitsPerPixel, bytesPerRow, colorSpaceRef, bitmapInfo, provider, NULL, NO, renderingIntent);
// then make the uiimage from that
UIImage *myImage = [UIImage imageWithCGImage:imageRef];
return myImage;
}
-(void)captureToPhotoAlbum {
UIImage *image = [self glToUIImage];
UIImageWriteToSavedPhotosAlbum(image, self, nil, nil);
}
I'm pretty proud of myself for figuring this all out. And it works great. But for the love of God, if there's an easier way, please let me know. It really, really, really seems like there should be. And if you pros see any horrendous memory leaks or anything of the sort in there, let me know. For instance, I feel like i should be freeing those malloc'd buffers when I'm done, but if I do that, the thing crashes. I don't know me so much about malloc. Does it get freed when it goes out of scope anyway?
2009-01-15 » keith
16 January 2009 @ 3:56 am
A couple of posts at the Apple DevForums talks about the same:
https://devforums.apple.com/message/11625#11625
and
https://devforums.apple.com/message/23781#23781
16 January 2009 @ 6:12 am
Hi, just wanted to say I’m enjoying your iPhone posts, I’ve been thinking of doing some development on that platform and it’s nice to read about someone approaching it in a similar way to how I work, from a similar perspective. So thanks.
-TP
16 January 2009 @ 10:46 am
wow this is perfect! was looking for something like this not long ago and mostly all I could find was using UIGraphicsGetImageFromCurrentImageContext(), but that doesn’t work for opengl es views. I could only find snippets of information here and there regarding saving opengl, but you’ve got it all in one place and working perfectly! thanks!
You are right about your concern with freeing memory though, I’m no pro, but I think there may be a few leaks. Malloc doesn’t get freed automatically (unless you send it to an NSData which owns it). In your case, you need to free buffer (preferably immediately after filling buffer2, cos its not needed anymore). Also free buffer2 after the image has finished saving (best place is the callback for when the CGImage is released (see code below). You also need to free everything you created using CGxxxCreatexxx() functions after you’ve used them. So thats the CGDataProviderRef, CGColorSpaceRef, and CGImageRef. Finally I personally prefer to not use the convenience method for [UIImage imageWithCGImage], but manually alloc and initWithCGImage. With the former, the UIImage is automatically released (which is good in case you forget to release it, but it may hang around for a while). By manually allocing, you have the responsibility of releasing yourself, but you can release it immediately after it is no longer needed (in the UIImageWriteToSavedPhotosAlbum finished callback). That is just a personal preference (and generally recommended it seems especially on iphone).
hope that made sense!
(P.S. I posted a little memory management tut at http://www.memo.tv/memory_management_with_objective_c_cocoa_iphone though its more about cocoa, whereas all the CGxxx stuff is not, but the concepts about retaining and releasing are the same).
Here is the full code btw with the memory stuff and callbacks. Thanks again for putting it all together and posting, saved me a lot of time!
// callback for CGDataProviderCreateWithData
void releaseData(void *info, const void *data, size_t dataSize) {
NSLog(@”releaseData\n”);
free((void*)data); // free the
}
// callback for UIImageWriteToSavedPhotosAlbum
- (void)image:(UIImage *)image didFinishSavingWithError:(NSError *)error contextInfo:(void *)contextInfo {
NSLog(@”Save finished\n”);
[image release]; // release image
}
-(void)saveCurrentScreenToPhotoAlbum {
CGRect rect = [[UIScreen mainScreen] bounds];
int width = rect.size.width;
int height = rect.size.height;
NSInteger myDataLength = width * height * 4;
GLubyte *buffer = (GLubyte *) malloc(myDataLength);
GLubyte *buffer2 = (GLubyte *) malloc(myDataLength);
glReadPixels(0, 0, width, height, GL_RGBA, GL_UNSIGNED_BYTE, buffer);
for(int y = 0; y <height; y++) {
for(int x = 0; x <width * 4; x++) {
buffer2[int((height - 1 - y) * width * 4 + x)] = buffer[int(y * 4 * width + x)];
}
}
free(buffer); // YOU CAN FREE THIS NOW
CGDataProviderRef provider = CGDataProviderCreateWithData(NULL, buffer2, myDataLength, releaseData);
int bitsPerComponent = 8;
int bitsPerPixel = 32;
int bytesPerRow = 4 * width;
CGColorSpaceRef colorSpaceRef = CGColorSpaceCreateDeviceRGB();
CGBitmapInfo bitmapInfo = kCGBitmapByteOrderDefault;
CGColorRenderingIntent renderingIntent = kCGRenderingIntentDefault;
CGImageRef imageRef = CGImageCreate(width, height, bitsPerComponent, bitsPerPixel, bytesPerRow, colorSpaceRef, bitmapInfo, provider, NULL, NO, renderingIntent);
CGColorSpaceRelease(colorSpaceRef); // YOU CAN RELEASE THIS NOW
CGDataProviderRelease(provider); // YOU CAN RELEASE THIS NOW
UIImage *image = [[UIImage alloc] initWithCGImage:imageRef]; // change this to manual alloc/init instead of autorelease
CGImageRelease(imageRef); // YOU CAN RELEASE THIS NOW
UIImageWriteToSavedPhotosAlbum(image, self, (SEL)@selector(image:didFinishSavingWithError:contextInfo:), nil); // add callback for finish saving
}
16 January 2009 @ 10:50 am
makes sense that you can’t free buffer2 til after the save. i assumed that the data provider and other methods were copying over the data, but perhaps they are merely retaining a link to it. that’s why the crash. thanks!
16 January 2009 @ 10:51 am
also, i wasn’t aware that you had to release the cg stuff like that. good to know.
16 January 2009 @ 10:53 am
Keith,
I’m a bit curious about that too. For my latest one I’m working on I needed to do the standard-c mixed in with the Obj-C for multi-dim arrays like that. For me though I wasn’t re-using it several times, so it gets destroyed ( afaik anyway ) when the view goes away. I’m doing a decently large ( 50×50 array of custom structs ) array and I haven’t seemed to run into any serious memory leaks on it.
And yeah, I feel your pain. Sometimes it feels like things that should be straight-forward are very much an uphill battle – I think we’ve gotten spoiled from working with higher-level languages for so long!
19 January 2009 @ 6:51 pm
How about this:
UIGraphicsBeginImageContext(cv.bounds.size);
[cv.layer renderInContext:UIGraphicsGetCurrentContext()];
UIImage *viewImage = UIGraphicsGetImageFromCurrentImageContext();
UIGraphicsEndImageContext();
UIImageWriteToSavedPhotosAlbum(viewImage, self, nil, nil);
22 January 2009 @ 12:09 pm
Pressing the Home and Sleep/Wake buttons simultaneously, very briefly, will save a screen shot to the Photo Album.
19 February 2009 @ 5:33 pm
[...] OpenGL ES Screenshot by admin on February 19th, 2009 The Bit-101 Blog has an entry that shows how to take a screenshot when using OpenGL ES. I tested this in my much-delayed particle-generator app, and it works as [...]
19 February 2009 @ 5:50 pm
Thanks so much for this example. I’m having an issue with the glReadPixels actually and maybe someone here can help. I’ve ripped heavily from the GLPaint example from Apple and have a painting app where I’m adding undo/redo support. So, just like Apple’s example I build line segments as the user drags and once the user ends the touch I want to snap a screen shot of that canvas (I save that off as a buffer to recall if the user does an undo later.. I save a stack of 10 deep for undo). This works well using you’re code with one problem, the very last line segment stamped down isn’t saved in the screenshot. I’m guessing it’s some sort of buffer issue or a glflush type issue? I’ve tried multiple variations to try and get it to work with no luck. Has anyone seen anything like this? I can try and post a small example if more info is needed.
Thanks!
Daniel
19 February 2009 @ 6:15 pm
My version requires half the memory by being creative swapping the image vertically, and treats colors holistically as 4-byte ints.
void releaseScreenshotData(void *info, const void *data, size_t size) {
free((void *)data);
};
- (UIImage *)screenshotImage {
NSInteger myDataLength = backingWidth * backingHeight * 4;
// allocate array and read pixels into it.
GLuint *buffer = (GLuint *) malloc(myDataLength);
glReadPixels(0, 0, backingWidth, backingHeight, GL_RGBA, GL_UNSIGNED_BYTE, buffer);
// gl renders “upside down” so swap top to bottom into new array.
for(int y = 0; y < backingHeight / 2; y++) {
for(int x = 0; x < backingWidth; x++) {
//Swap top and bottom bytes
GLuint top = buffer[y * backingWidth + x];
GLuint bottom = buffer[(backingHeight - 1 - y) * backingWidth + x];
buffer[(backingHeight - 1 - y) * backingWidth + x] = top;
buffer[y * backingWidth + x] = bottom;
}
}
// make data provider with data.
CGDataProviderRef provider = CGDataProviderCreateWithData(NULL, buffer, myDataLength, releaseScreenshotData);
// prep the ingredients
const int bitsPerComponent = 8;
const int bitsPerPixel = 4 * bitsPerComponent;
const int bytesPerRow = 4 * backingWidth;
CGColorSpaceRef colorSpaceRef = CGColorSpaceCreateDeviceRGB();
CGBitmapInfo bitmapInfo = kCGBitmapByteOrderDefault;
CGColorRenderingIntent renderingIntent = kCGRenderingIntentDefault;
// make the cgimage
CGImageRef imageRef = CGImageCreate(320, 480, bitsPerComponent, bitsPerPixel, bytesPerRow, colorSpaceRef, bitmapInfo, provider, NULL, NO, renderingIntent);
CGColorSpaceRelease(colorSpaceRef);
CGDataProviderRelease(provider);
// then make the UIImage from that
UIImage *myImage = [UIImage imageWithCGImage:imageRef];
CGImageRelease(imageRef);
return myImage;
}
18 April 2009 @ 10:51 pm
Great work! I have sucessfully used your code to generate screenshots. One doubt – the saved PNGs do not have an alpha channel. If I try to display a saved PNG over another then I cannot see the image below at all (blend modes do not have any effect). If I do a CMD-i on the saved PNGs the info window shows that there is no alpha channel. Comments?
4 May 2009 @ 9:06 am
[...] Part of the Hologram app is drawing something on the screen. That means dynamically modifying a texture map based on touches on the device. This is done through framebuffers which allow you to render into that texture map. I’ve got that part working. But eventually, I’d like to save out those texture maps and that’s where it’s getting a bit tricky. I’ve almost got it working through a call to glReadPixels and something like this. And then I need to convert that data into an image using CGImageCreate and CGDataProviderCreateWithData doing something like this. [...]
27 June 2009 @ 5:53 pm
Has anyone ever worked out the alpha issue? If you’re trying to save an image, that contains alpha values, you only get shades of grey/black where your image should be transparent.
5 July 2009 @ 6:24 am
Try it!
CGBitmapInfo bitmapInfo = kCGImageAlphaPremultipliedLast;
15 July 2009 @ 5:29 pm
Hi,
Thanks for the code.
I just want to extract the pixels from a UIView without going through renderInContext which is really too slow. How can I use your code without building a CAEAGLLayer (I tried to use your code just like it is but I get a black image with noise on the top…)
Thanks
6 October 2009 @ 6:05 am
[...] I have found a very nice method saveCurrentScreenToPhotoAlbum to capture the OpenGL view screen here. And below is an implementation on how to capture the screen in iPhone Simulator. You just [...]
15 October 2009 @ 2:04 pm
On line 6 you wrote. glReadPixels(0, 0, 320, 480, GL_RGBA, GL_UNSIGNED_BYTE, buffer);
Where is the data being extracted from? Because I am in a similar situation, I have a RGB float buffer and I need to use it to convert it into a CGImage.
Can you please tell me how I can get to accomplish such a thing?
2 December 2009 @ 12:05 am
Very good example. I’ve been looking for this solution for the last couple days.
Thank you very much.
29 December 2009 @ 11:09 pm
[...] (注:本文改编自iPhone – saving OpenGL ES content to the Photo Albumï¼Œæ‡’å¾—å…¨æ–‡ç¿»è¯‘äº†ï¼ŒåªæŒ‘一下é‡ç‚¹åŠ ä¸Šè‡ªå·±çš„è¯è¨€æè¿°ä¸€ä¸‹ï¼‰ [...]
30 December 2009 @ 3:44 pm
hi,
not exactly fair, but it still might interest you;
UIApplication has some private methods to make screenshots.
- (struct CGImage *)_createDefaultImageSnapshot;
- (void)_writeApplicationDefaultPNGSnapshot;
using the first method, saving a screenshot to an arbitrairy location is as simple as
- (void)snap
{
// save-path
NSArray * dirs = NSSearchPathForDirectoriesInDomains(NSDocumentDirectory,NSUserDomainMask,YES);
NSString *path = [dirs objectAtIndex:0];
// make snapshot
CGImageRef snpsht = [self _createDefaultImageSnapshot];
// save image
NSData * pngdata = UIImagePNGRepresentation([UIImage imageWithCGImage:snpsht]);
[pngdata writeToFile:[path stringByAppendingPathComponent:@"screenshot.png"] atomically:YES];
[self _writeApplicationDefaultPNGSnapshot];
}
but again, keep in mind that these are private methods.
30 December 2009 @ 3:46 pm
appologies, that last line of code
[self _writeApplicationDefaultPNGSnapshot];wasn’t supposed to be there…
15 February 2010 @ 5:55 am
Thanks for the blog it helped me allot but how to store a landscape view into image…it does work for vertical that is 320 height and 480 width but what about the 480 width and 320 height pictures
30 March 2010 @ 2:02 am
Thanks for sharing. Short and sweet solution. You’ve opened my eyes to a few new tricks.
21 April 2010 @ 1:17 am
Hi,
Is there a way we can get the full size image from the openGL?
Thanks!
19 May 2010 @ 3:30 am
Hi KP,
It’s really very helpful for me in my project. It saved my effort. Thanks a lot to post such a nice Tutorial.
19 May 2010 @ 3:40 am
Pannag Sanket, the code given in this tutorial by KP, is already to capture full size image (320X480) for a Portrait iPhone screen.
26 May 2010 @ 1:40 pm
Thanks for figuring this out.
Other commenters are right: you have several pretty serious memory leaks in there. Every method or function whose name includes “new”, “alloc”, “Create” or “copy” requires a balancing release/free. So you need this after creating the UIImage, but before returning it:
CGImageRelease(imageRef);
CGColorSpaceRelease(colorSpaceRef);
CGDataProviderRelease(provider);
free(buffer);
free(buffer2);
26 May 2010 @ 3:05 pm
My previous post was incorrect on one point: the CGImage continues to rely on buffer2, so you can’t free it immediately.
(It also continues to rely on other things you provide, but they’re reference counted, so they don’t actually get trashed until the CGImage is done with them. But malloc/free has no reference counting.)
A simple solution is to create a callback function that frees the data:
void freeImageData(void *info, const void *data, size_t size) {
free((void*)data);
}
Then tell the CGDataProvider to call your callback when it’s done with the data:
CGDataProviderCreateWithData(NULL, buffer2, myDataLength, freeImageData);
This seems to work.
30 May 2010 @ 6:57 pm
going from screen -> buffer is cool, but I’m wondering how to write back from buffer -> screen? anyone know?
7 June 2010 @ 5:05 am
The tutorial above is really helpful. In my app, i needed to take a screen shot of my EAGLView and use it in the app as a UIImage.
It works well with the code. But when i get the image its a little bit scaled. The image if i save it to photos album and compare, there is a scaling happening to the image.
I use the screenshot as a UIImage at the time i take the screenshot. So there is a jumpy feeling..
So any one got any hint to solve this scaling issue