About scaling, you said that “The method does not take into account the contents of the image at all”, which also appears to be the opinion of Avidan and Shamir. Now, I agree that scaling usually looks horrible when it’s disproportionate, but I think that it takes the content into account quite well - better than seam carving, even; it simply is almost lossless. Shrinking the image by a coefficient of two, say, retains pretty much all content. I am not sure what the formulas for non-integer proportions are, but in my experience proportionate scaling usually results in pictures that look pretty much the same. The only real loss is the resolution of the details, but all of the features remain, whereas in seam carving there’s no resolution loss, but some features (granted, the least significant ones) completely disappear. I think losing resolution is preferable in most cases.

For example, consider the resized image of the two pagodas. While the scaled image isn’t good looking (as it is disproportionate), I don’t find the seam-carved image quite satisfactory. One of the lost details is that the pagodas are not, in fact, that close to each other, and I think that results in quite different content in the carved image.

Now, say that you need to shrink a 600*400 picture into a 200*200 one. Simply scaling results in a terribly deformed picture. If you seam-carve, you carve away 400 vertical lines and 200 horizontal lines, which are two thirds of the width and one half of the height; that’s probably a lot of lost content. Now, if you proportionally scale the image to 300*200 and then carve it down to 200*200 (or perhaps carve it down to 400*400 and then scale, the difference is probably not that great but the former might be faster), you only need to carve away a third of the vertical lines, which is quite an improvement.

I rather like this method of scale and carve. An important question would be when to stop scaling and start carving. Say that you have to resize the picture so that the new width is A*width and the new height is B*height. One of the options, used in the previous paragraph, is to scale it by the maximum of A and B, and then carve out what remains. This is perhaps an improvement to just carving, but it won’t improve the situation in cases such as the two pagodas, where one dimension stays fixed. It could also be potentially self defeating in cases where you need to increase one dimension.

Another method would be to scale it by the minimum of A and B, and then carve-in the remainder. I like this version better, because carving in is detail-lossless. However, you do lose even more resolution, you end up with the least significant details extended, and adding seams is a harder technique which also doesn’t always look well.

Finally, any other value between A and B can be used, and then you need to carve both in and out. My preferred value is the geometric mean of A and B; it seems the most natural choice to me… Although by two definition of optimization I used, the optimal choices were the maximum and the arithmetic mean - obviously, the definitions just weren’t good enough (:. I don’t think the maximum is optimal, but the arithmetic mean might be.

If you feel like implementing this, as it’s probably not hard to add this to the existing seam-carving implementation, I’d really like to see how some of these values deal with the picture of the two pagodas. And sorry for the length of this thing, it’s likely that this post could be improved with some heavy seam-carving.

Maybe you want to take a look at http://www.thegedanken.com/retarget/ where you can download a prototype for Windows to play around a bit, still work to do …

regards
Ramin

Of course scaling does take into account the contents of the image (different input images usually generate different output images ). What I meant is percisely that “unimportant” parts of the image get to occupy as much space in the output image as the important parts.

Now, obviously all the forms of image shrinking loose information (there are num_colors^(original_width*original_height) input images and num_colors^(output_width*output_height) output images). So clearly, assuming the output size is small than the input size, several different input images get mapped to the same output image, thereby information is lost.

Choosing what information to loose is a more of a philosophical question than a mathematical one. In some contexts, realizing that the two pagodas are not very close to each other is important, in others realizing the exact proportions of the base of the pagoda to its height are more important.

I agree that scaling is less “lossy” in the sense that given a scaled down image and the original dimensions of it, the scaling process can be reversed quite efficiently (i.e. a source image very similar to the original input image can be found easily). Seam carving, on the other hand, is much harder to reverse.

I do think that seam carving is extremely useful not only as an image resizing mechanism, but as an image processing tool. A tool that removes the insignificant parts of the image retaining only its essential bits (e.g. instead of using it as a function from one image to another we can use it as a function from one image to a set of images containing the important parts, etc.).

Your idea of mixing scaling and carving sounds really good! It actually fits an other idea I had on improving image quality (but it is too long to describe in a comment…). A very interesting thing about seam carving is that by comparing the energy of the seam to remove vs. the total energy of the image we can get a very good idea of whether the seam is indeed irrelevant or whether we already reached a limit after which removing seams will delete important parts of the image.

I.e. if we detect that the next seam to remove has a lot of energy vs. the whole image, then we might want to stop carving and start scaling.

Implementing your ideas is a matter of minutes and I will sure do so (and let you know of the results ) after I get back to Israel!

PS - Tonight I am flying to Prague to present an algorithm I developed in a conference. Be back in 2 weeks, and I will tell you more about it then.

I have made an open source java program of this algorithm. It can be found here : http://pierrefrancois.leon.free.fr/smart-resizing/.
(but with a little difference : i recalculate the energy function at each sean removal)

My implementation also recalculates the energy at each seam removal (I think the algorithm will perform much worse if this wasn’t done). This can actually be done much more efficiently than recomputing the Gradient Magnitude of the entire image again (as most of it remains the same).

Another thing that can be done, which I haven’t implemented yet, is instead of simply removing a seam, somehow averaging it with its surrounding seams. And of course there are all the suggestions of combining carving and scaling.

I will look your implementation when I’ll get back from Prague (in about a week and a half).

Merci pour la remarque!

I just got done making a version in C++, and tested it on a bunch of people on a beach. I was frustrated that the algorithm started removing seams containing people long before it had run out of empty beach. I realized that this was because the beach had many horizontal layers and therefore lots of energy i the y-direction. So I tried again removing vertical seams, but this time I used the magnitude of the x-component as energy function, which turned out great, removing all the background beach before attacking the foreground people.

I haven’t really tested if this holds for many other pictures, but it seems to me that giving more weight to the x-gradient when calculating vertical seams, and conversly, more weight to y-gradient when removing horizontal seams, could be a good strategy. Any views?

I, too, recalculate the energy map at each iteration (in fact I “heal” it since only two parallel seams in the energy map are affected by the removal, unless you approximate the derivative with a radius higher than 1, which does not seem to be necessary at all). The seam cost map, however, has to be entirely recomputed.

Interesting idea. I will give it some thought and get back to you with my conclusions (although it will probably take me a couple of weeks as I am really busy now).

Peter,

You are absolutely right about “healing” the energy map instead of completely recalculating it, and I did the same in my code.
I actually also “heal” the seam cost map, instead of entirely recomputing it. What I did is to propagate the changes forward in only the affected pixels. Of course this affects more pixels than are contained in 2 vertical seams but it still only affects part of the seam cost map.

Maybe this becomes clear if we think through an example: If you have an image of an evenly colored blue square, removing a seam or a column from that will not leave any marks since putting blue next to the same blue is seamless.

Let’s then repeat the experiment with an image of the same square, but filled with a linear horizontal gradient*, you will find that removing seams or columns will result in the gradient becoming “steeper”. To understand what is happening, let’s assume (without loss of generality) that we are removing columns rather than seams. The entire fill has the same energy at every point since the change Pfrom one column to the next is always the same since the gradient is linear. Now if you remove an arbitrary column, the gradient at the neighboring columns increases, since the step between them is now twice as large as before. That means that the seam carving algorithm is not going to select one of those neighboring columns in the next step, since there are plenty of other columns that still have a smaller energy. This means that in the end you have removed columns with an even spacing so that you end up with something that is similar to the result you would end up with had you used regular downsampling using a nearest neighbor algorithm.

* Please note that when I’m talking about a gradient here, I mean the thing you would draw with Photoshop’s gradient tool (i.e. interpolated colors), not the mathemathical gradient. For the sake of simplicity, let’s assume that no dithering was used to render it.

I hope that answers your qestion.
Peter

I disagree with your answer. Removing the seam with the lowest energy may induce artifacts to the image (the energy of the seam with the lowest energy may still be high!).

Consider the case of an extremely noisy image. Simply removing seams will generate a lot of artifacts and somehow averaging the removed seams back into the image might make the situation much better.

You may argue that the seam carving method is inadequate for extremely noisy images, but consider the case of the pagoda image above - the vertical seams do not have a lot of energy (as the blue sky takes up most of them) but each of them is forced to pass through the wall and the water at the bottom - areas of high energy. Averaging the removed seams may reduce the artifacts occuring at those locations.

I completely agree with you. My understanding of Amnon’s question was that he/she wanted to know how the algorithm proposed in the paper deals with the issue. My answer is that it doesn’t deal with it at all (at least as far as I remember) because artifacts are often not very noticeable since most images have a lot of low energy regions, and I attempted to explain why the algorithm does not cause many strong artifacts in those regions. I should have made that clearer, and I should definitely not have used the term “generally”. English is not my native language, so please bear with me. I am really sorry if my post was misleading and caused any misconceptions. Again, you are absolutely right.

Your suggestion to average the seams is a really good idea, although I would propose a weighted combination of the original values since they do not contribute equally to the resulting pixel. Say we are removing a vertical seam. If you have pixels ABC and you remove pixel B (which is on the seam) ending up with DE, my suggestion is that D = 2/3 * A + 1/3 * B; and E = 2/3 * C + 1/3 * B; You could think of it as splitting the removed pixel and assigning one half of it to each of the neighboring pixels. The results should be on par with conventional resampling in terms of quality, except for any rounding errors that might accumulate. The only problem is that it slows the algorithm down even more.

I don’t think this will be a great improvement, but maybe a small one. I think that in this way, the vertical gradient is less important, because colours are merged horizontally instead of simply removed. Still haven’t seen the results of my other suggestions, though. No pressure.

On your gradient technique: I have been studying this for awhile, and conversion to grayscale before edge detection is generally an acceptable technique. There really isn’t much loss in the edge values, especially if you do a RGB->YUV conversion, such as seen here: http://en.wikipedia.org/wiki/YUV

There are techniques to increase performance. One way is to avoid recalculating the gradients after every seam removal. I have some examples of this on my website. You can average the edge value back into the edge image much like you do the image. This has its limits, which some of my examples demonstrate.

Unfortunately, the energy map MUST be recalculated after each removal. Any single change in the energy map affects all values above it in a tree pattern. At most in most circumstances, only about 1/3-1/2 of the energy map will remain unchanged, but specializing the code to avoid those regions might make it unwieldy.

During removal, yes, I recommend averaging the removed pixel’s values back into its neighbors. This limits some of the harsh edges that are likely to develop. However, it is important NOT to average the results back in when an area is marked for removal, as this will average the unwanted content back into the two edges of the area.

One thing that hasn’t been discussed here yet is enlarging by seam insertion. The paper’s method appears to be lacking, which I demonstrate on my website. The paper describes calculating all seams at once, then inserting next to each seam from least energy to most. Well, the problem is is that so many of the seams merge as they propagate up, producing horrible stretching. My test image that is 1024 pixels wide only had 4 distinct paths by the time they reach the top. The only solution that I have come up with (thanks to Ramin, who commented above), is to add artificial weight to both the least energy seam and the new inserted seam. Then, the least energy seam has to be recalculated. With the correct choosing the the artificial weight it can produce very good results.

Other options not mentioned to increase quality are the transport map and frequency domain removal. Both of these would DRASTICALLY increase calculation time. The transport map, described in the paper, needs to calculate both the horizontal and vertical seam removal at each step, then determine which one removed the least amount of energy, and then go with that decision. This is entirely possible to implement, just that it would disable most of the optimizations that we have already discovered. The other quality improvement is frequency domain removal. This is in no way fast, as the frequency domain brings in a whole bunch of fun stuff like imaginary numbers. I have done some frequency domain filtering before, but I am uncertain on how to approach removal (the paper mentions another paper which I have only briefly looked over).

Thanks for the nice read, keep up the interesting posts..

The simplest method to achieve the same results is this:
1. Calculate vertical least-energy seam as normal. The energy of the seam is the bottom value in the energy map, where you started the seam from.
2. Repeat step one except for the horizontal direction.
3. Remove the seam from step 1 and 2 that has the least energy.

Repeat these steps until one dimension of the image has reached its desired value. Then, you would have to perform the standard seam removal for the other dimension.

http://danluong.com/2007/12/21/seam-carving-matlab-implementation-tutorial/

There’s a variety of functionality in it, and the documentation should be thorough enough for anyone trying to figure out how the algorithm works. I calculate a single gradient image, and recalculate the energymap on every seam removal, and also include seam insertion functionality.

by the way , does any body know how does GIMP software do seam insertion, coz they did a good job and i need the trick. thanks

You are of course right - thanks for your correction!