SAN DIEGO, Aug. 6, 2007 – Sometimes, the old adage that “two negatives yield a positive” goes beyond mathematics and enters another realm. This week, researchers from The Hong Kong University of Science and Technology and Microsoft Research will present how two negatives can indeed equal a positive for digital photographers at SIGGRAPH 2007, the Association for Computing Machinery’s 34th annual international conference on computer graphics and interactive techniques. SIGGRAPH is considered the premier international forum for disseminating new scholarly work in computer graphics and interactive techniques, showcasing a full range of the world’s most significant achievements in the field and illuminating new directions for future investigations. SIGGRAPH 2007 runs August 5 – 9 at the San Diego Convention Center.
Image Deblurring with Blurred/Noisy Image Pairs describes the process of reconstructing blurry and noisy images into a sharp image.
The paper, Image Deblurring with Blurred/Noisy Image Pairs, authored by Lu Yuan and Long Quan at The Hong Kong University of Science and Technology, and Jian Sun and Harry Shum of Microsoft Research Asia, was identified by SIGGRAPH as one of six “select highlights from the SIGGRAPH 2007 papers program.” This research will help everyday digital photographers avoid one of the more common frustrations of capturing satisfactory photos under low-light conditions using a handheld camera, which is that images are often blurred, or cluttered with noise. In the context of digital photography, “noise” refers to the speckled patterns that often look like film grain. The technology enables digital photographers to generate high quality images by combining a blurred image with a noisy one.
The Image Deblurring with Blurred/Noisy Image Pairs paper is just one of several papers Microsoft Research will present at SIGGRAPH this year, building on the company’s involvement in the conference for more than a decade. In addition to sharing their research, Microsoft researchers have dedicated their time on various ACM and SIGGRAPH committees and received some of the organizations most prestigious awards over the years as well.
“Microsoft Research has a history of driving innovation in graphics for the company and the entire industry,” says Rashid. “Some of that is evidenced by the number of papers we have accepted at SIGGRAPH year after year, but what’s key is the exciting research results described in these papers. One technology lets us combine blurred and noisy images to reconstruct a sharp image and another lets us easily and accurately cut out an object with fuzzy edges from one image and paste it into another image seamlessly. SIGGRAPH really showcases what we’re doing with graphics technology so people can make better pictures and more realistic animations.”
Microsoft researchers authored or co-authored 14 of the 108 papers accepted for the SIGGRAPH 2007 Paper Program – this accounts for 13 percent, more than any other organization. In its commitment to open collaboration and working with universities to drive the state of the art in graphics, 11 of the 14 papers from Microsoft were written with academic partners from around the globe.
“This is not the first, nor the last joint paper between Microsoft Research Asia and the Department of Computer Science and Engineering at Hong Kong University of Science and Technology,” says Professor Quan of The Hong Kong University of Science and Technology and co-author on The Image Deblurring with Blurred/Noisy Image Pairs paper. ”It reflects our long time partnership in Vision and Graphics research.”
Researchers from Microsoft’s Asia, Cambridge and Redmond labs will present their work at this year’s conference. The research covers a breadth of graphics innovation that includes image analysis and enhancement, image slicing and stretching, lighting, sketching 3D shapes, geometry processing, shape deformation, image-based modeling, graphics architecture and big images.
Photo Clip Art presents automatic algorithms for improving object segmentation and blending, estimating true 3D object size and orientation.
Photo Clip Art from Carsten Rother, John Winn, and Antonio Criminisi at Microsoft Research Cambridge and Jean-François Lalonde, Derek Hoiem and Alexei A. Efros at Carnegie Mellon University takes a novel approach to real-world clip art, and Soft Scissors: An Interactive Tool for Real-Time, High-Quality Matting, by Michael Cohen, Microsoft Research Redmond, Jue Wang University of Washington and Maneesh Agrawala, University of California, Berkeley, demonstrates a new interactive paradigm for extracting objects from one image so that they can be seamlessly inserted into a new one.
Research that Sheds Light on Darkness
The Image Deblurring with Blurred/Noisy Image Pairs research addresses a classic and extensively studied problem: blurred and noisy photographic images. “In this research, we presented an image deblurring approach to produce a high quality image by combining a blurred and noisy pair of images,” says Jian Sun, Microsoft Research Asia. “The result is that the reconstructed image is sharper than the blurred image, and clearer than the noisy one. Using a blurred/noisy image pair for input, we are able to obtain incredible image deblurring results that were hardly imaginable before.” And one of the best parts for digital photographers of all levels and varieties, says Sun, is that the approach doesn’t require any special hardware: “Our approach uses an off-the-shelf, hand-held camera,” says Sun.
Sun says the research will empower both average and professional photographers to take great photos in previously challenging environments, such as indoor scenes or outdoor night scenes. As a software-based solution, the photographer can upload photos on a PC and then easily and quickly perform the procedure. At the same time, the approach to reducing blur and noise can be implemented in hardware or firmware within the digital camera. Sun plans to extend the approach to other applications, including reducing blur in video sequences.
Research that Makes it Easier to Make it Real
Photo Clip Art takes a new approach to synthesizing ‘real’ images. “Our goal was to solve the grand challenge of computer graphics by creating realistic images for user-defined scenes,” says Carsten Rother, Microsoft Research Cambridge. “In contrast to graphics, which creates a three-dimensional world from scratch, we turned the problem upside-down by combining real images in such a way that we never break the realness of them.”
The research delivers a system for inserting new objects into existing photographs by querying a vast, image-based object library, pre-computed using a publicly available Internet object database. It also presents automatic algorithms for improving object segmentation and blending, estimating true 3D object size and orientation, and estimating scene lighting conditions.
“Our central goal was to shield the user from the arduous tasks typically involved in image compositing,” Rother says. “The user is asked to do only two simple things. First, pick a three-dimensional location in the scene to place a new object and then select an object to insert using a hierarchical menu. Instead of trying to manipulate the object to change factors such as its orientation and color distribution, we simply retrieve an object of a specified class that has all the required properties from our large object library.”
Perhaps best of all – at least for novices – the technology provides a user interface that makes object insertion fast, simple and painless, even for the artistically challenged.
Rother says the work has implications for several fields. “Most Office systems, such as Microsoft Word and PowerPoint, can accommodate graphics clip art,” he says. “But imagine being able to have realistic objects instead of clip art and being able to easily create real-world images with them. That’s the essence of our research.”
Professionals who could benefit from real-world graphics capabilities, he says, include artists, graphic designers and architects. “Architects could use this capability to truly personalize their architectural sketches by inserting ‘real’ people into them,” he says.
Future work with this technology includes plans to add more scenarios, more objects and more computer visual capabilities and also to improve some of the vision components such as color matching and shadow synthesis.
Research that Softens the Scissors
Michael Cohen in Microsoft’s Redmond research lab set out to help photographers of all stripes avoid painful cuts by taking the sharpness out of their scissors. “This work demonstrates a new interactive paradigm for easily extracting objects from one image so that they can be seamlessly inserted into a new image,” he says. “It works especially well with complex foreground objects such as furry animals that require detailed cutting.”
Soft Scissors allows a user to easily remove images from one photograph and insert them into another even if they don’t have clean edges like those of a building.
The result of the research Cohen and his colleagues conducted leverages work they recently presented at the IEEE Computer Society Conference on Computer Vision and Pattern Recognition that improved the quality of the offline algorithm. At SIGGRAPH they will demonstrate the ability to incrementally update the matte in an online, interactive setting. A matte – or matting – refers to a computer-implemented method for extracting a selected subject from its background by generating a matte signal that represents the opacity of the foreground subject even in semi-transparent subject areas.
“Our goal was to provide a tool that can generate high quality mattes in real time,” Cohen says. In the system they devised, he says, the user roughly specifies the foreground boundary using an intelligent paint stroke – or soft scissor. The system then automatically updates the matte and foreground colors according to the newly added information along the stroke to instantly reveal a local region of the final composite.
Cohen adds that this research has facilitated the first interactive tool for high quality image matting and compositing. “This is the first system to provide such an intuitive interface,” he says. “We feel strongly that this is how cutting out objects will be done in future image editing systems.”
One of the key ways that Soft Scissors differentiates itself from other systems is its user interface, which Cohen says, is one of the most novel parts of the system. It allows the user to easily remove images from one photograph and insert them into another even if they don’t have clean edges like those of a building. Examples of images that can be easily and intuitively transported include furry pets or people with curly hair. “In the past there were ways to move objects, but it required the user to draw a line around the object as if they were using a pair of scissors and then cut the objects out,” says Cohen. “That’s hard to do if the fuzzy boundary of the subject is a mix of background and foreground.”
To alleviate those challenges, the research team spent two years developing a system that automates the process of determining whether the image’s pixels – the small colored dots that make up the image – constitute foreground, background or a mixture of the two. Soft Scissors also automatically calculates the percentages within those pixels that are foreground, background or both.
“The system looks ahead of the user’s actions to see how fuzzy the subject is and adjusts how soft the scissors are to come up with a much clearer image,” says Cohen. “It just does the right thing automatically. Users can select the new background ahead of time and then, as they’re brushing, see the edge of a dog for example going into a new background. Soft Scissors draws a soft line, a line that gradually and softly separates the foreground and background.”
