Context Aware Image Re-Sizing

If you have a large image that you need to make smaller you can either scale it or crop it. Scaling makes everything smaller and cropping can cut out important content. This jaw-dropping video demonstrates an image re-sizing algorithm that is content-aware. I was amazed; I hope it’s real and coming to Firefox extension soon.

tags: ,
  • Wow. That is amazing. I’m looking forward to seeing this technology in action. I would love to have this in a photo editor. This would also be great for auto generating thumbnails that have to be the same size.

  • That’s pretty cool, but also kind of disturbing to have parts of the picture just disappear. (Or appear out of nothing when the picture grows larger.)

  • That is very impressive. I can definitely see a big use for this as a photoshop plugin and also programmatically in the likes of ImageMagick or GD (for use in php web applications).

  • I thought it was pretty cool too, so I decided to do my own implementation inspired by the video.

    If you want to have a play around, there is source code (GPL3) available:

    See:
    http://www.ultra-premium.com/b/photos/resize.html

    And:
    http://www.ultra-premium.com/b/news/source_party.html

    For my version. The code is very simple, so you should be able to modify it to suit your own purposes quite easily (in fact, you will be forced to, since it it doesn’t really have a user interface :)

  • Michael

    Where do I get it?

  • beautiful. I love that as a side note, it makes the equivalent of rubberstamping out pieces of an image a 2-second process.

  • Mind Numbing demo! It was almost as if the algorithm could think! The only time it let up was when it resized the baby’s face to make it look like a Japanese cartoon character. But then, they had a solution for that too.

  • In response to the author’s call for a Firefox plugin:

    As a designer, I’m not too keen on having a browser automatically altering my images. In web design, content is often separated from presentation. This tends to work for text, but I’m not sure it’s appropriate for pictures. If we are talking about *good* images, the content is the presentation.

    To the average person, the content of an automatically re-sized image probably remains the same. It still has the same number of big trees, a mountain, whatever. But the presentation, which is altered by the algorithm, is compromised. If so, is it really the same image? Proportions, cropping, etc. are all essential to the final quality of a picture. Changing those values from the designer’s original intent could easily ruin a carefully composed layout. Worse, users might not be aware that an image has been resized– they would never know what the original image is supposed to look like. What if an image of a map was resized in this manner, but the user wasn’t made aware? What I’m describing is not just an aesthetic issue, but a usability issue, as well.

    Ultimately, only a person can reliably resize an image for the content. To successfully automate the process for all cases would be next to impossible. Cropping and scaling are design tools, not just a means of fitting an image to a page.

    The technology certainly has its uses offline (here, I see design *possibilities*), but if it were somehow implemented, I hope it would remain just a plugin and that it would not become a standard behavior for any browser. Altering the way images behave online would be a tremendous headache for both designers and users.

  • apolodor.ln

    this is just one of those things that it will make everyone life better.

  • Nick Senske’s reluctance is well founded!
    If this technique is widely embedded, as an automatic feature, a lot of clever or creative efforts from the photographer or graphist will simply be obfuscated for most of the users.
    The same kind of degradation has occurred in the movie industry: it takes a lot of efforts to set a TV screen correctly and respect original formats when playing a DVD…
    The same goes with color correction on many media.
    In fact, “end users”, when they are expert enough in the management of their display device, can interact more with the document they get than has ever been the case.
    This situation will probably lead to new concepts of creation, where the creator has to let more liberty to his/her public.
    If the distribution canal is the only one to choose the editing, provided as an automatic feature, it becomes part of the editorial process.
    Consumers will have to be made aware of that.

  • What Nick said.

    The analogy they use of the resized web page is flawed. Tell me, if someone resized this page and your text had to be compressed, would you willingly have words dropped? This is really no different than having pieces of a photo ‘compressed’ out based on some algorithm.

    Resizing a photo is one thing. Chopping the heck out of it changes everything about the picture.

    This is a gee wiz effect. Might be fun to experiment with as a photo tool plug-in, but a Firefox extension? Only if I can detect such so I can pull the photos from the page. After all, using such a plug-in tells me you’re really not interested in the photos.

  • This is a very clever technology, but using it to resize pictures in the browser is silly. Just resize them automatically using a standard re-size algorithm, like Opera already does.

    However, this technology must have some useful applications. I just can’t think of them right now!

  • David M. Wilkins

    Isn’t the biggest value of this in cross-device presentation? While I agree with all the preceeding comments about orginal intent and the warping of same, it seems to me that users of Blackberry, iPhone, regular web-enabled phones etc… could really benefit from this as a web extension on their phones. No one using these devices expects the same user experience they would typically have on a larger screen, but they may still need to see particular images, even if they are warped. Nick made the point that the image is the content, yet in most web instances on alternate devices, that content is not presented at all or is all but unusable. This may provide an additional option to designers — provide no image by design, provide an alternate image by design, *or* provide an alternate image programmatically when it makes sense.

    Just a thought.

  • Richard Mann

    To everyone who thinks this is heresy against the beauty of the original image, why not just allow the designer to specify that a particular image cannot be resized, that way images that are not of central importance, but are more ‘mood pieces’ can be resized so as to fit with the general design, while images that ‘are’ the content won’t be.

  • So the ‘solution’ is to have designers specify programmatic instructions and alternates for all their images? How realistic is it to come up with the best possible design for all possible window combinations and not end up with something mediocre? How does this save work and improve user experience?

    We need to think more critically about the real costs and not just assume that this is a good idea. What David and Richard are proposing adds a lot of complexity to the designer’s job (as I alluded to in the final point of my first post). Web design has enough variables as is. Now you want to change one of the few consistent behaviors across all browsers?

    The technology is certainly interesting and has many potential uses, but I don’t think it’s the best solution to the browser-sizing problem.

    The solution is design, not more technology. You want a beautiful and usable mobile or wide-screen experience? Design one. One resized page does not fit all.

  • David Lewis

    I suppose the question is do the programmers of browsers or browser plug-ins have any responsibility for preserving the designers original intent. it seems that they don’t and there’s no reason why they should. their motivations are entirely different from those of web site designers if the users want this technology and the programmers are willing to provide it then there’s no argument that will prevent it’s implementation. if that’s the case then designers will have to shift to designs that are robust at any scale and accept that images will not be as effective of a method for control of page layout.

  • David– Where do you get the expectation that designers should have no say when it comes to browsers (or other technology)? And why do you think that programmers’ and designers’ goals are that much different? While it’s true that design is at the service of others, that does not mean it plays a passive role.

    As a designer and a researcher, it is my responsibility to educate both users and programmers about the consequences of their decisions. Design is not anti- or pro-technology; it is critical thinking. It is about taking into account. Oftentimes it seems as though neither the programmers nor the users think these things all the way through. The prevailing wisdom seems to be: “if we are able to do something, then why not?”.

    I’m not defending designers; I am defending what is in the best interest of users (although often the two are related). What we’re discussing here is not merely an aesthetic argument– though aesthetics are indeed critical. Nor is it that designers are afraid of extra work. It is that there are usability concerns and compatibility issues that a change like this would introduce. I’m arguing that the need for resizing is better solved through other means.

    To return to your first point, programmers should care about how easy or hard they make the designer’s job (for instance, by preserving intent) because it’s directly related to user experience. If programmers truly care about users, then they should implement technologies that enable designers to deliver the best experience– not hamstring them creatively and break usability in the process.

    My point is and has always been: just because it seems to work in a demo doesn’t mean it’s a good idea in practice. I think this is an example of a technology being asked to solve the wrong problem.

  • I was bothered as I watched the tricks the technology can do because I envisioned myself looking at a resized or rescaled picture of something familiar (some scene in nature that I love, for example, or a picture of friends) and not recognizing it or not being able to identify an occasion because elements have been rescaled or removed. It would be so easy to be dishonest with this technology, it seems to me, even if dishonesty were not the object of the resizing/rescaling.

  • Walt Edwards

    The most obvious benefit would be in generating consistently sized thumbnails of widely varying documents and images. Current techniques just don’t cut it, if it works as advertised, this one would.

  • dbt

    If you actually watch the video, what you’ll see is that they have a tool for authors to insert their own metadata into their own images to allow for resizing. Supporting the algorithm on author-tagged images would allow for some great flexibility in image design.

  • Hi,
    If you are looking for a software to try out seam carving, take a look at http://www.thegedanken.com/retarget

    The program that you can download there (for Windows and Linux, and free) is already highly optimized concerning speed, and apart from enlarging or decreasing image size you can also use masks to protect or delete certain parts of your image.

    Have fun,
    Irmgard

  • These are things web designers do to “present” the image as a designed part of web pages. Reducing image size, maintaining image clarity, and ensuring that the image adds to the content of the page.

    I would hate to see something built in that altered the overall presentation. Currently, it is not difficult to identify professionally designed pages from hacked up stuff. On the fly mods could cloud the horizon…