Smooth Edges

Channels

When you begin, make you you have a shape on your stage without any fancy-pancy layer-styles applied to it. If you are using layer-styles, you can always add them in the end when we're done. Why? Simply because we're not gonna keep using this shape and re-applying things is a waste of time.

While holding Ctrl, click the thumbnail of the layer of your shape which will select your shape. Now open your channels-tab. If you do not see it anywhere on your screen, go to Window>Channels to make it visible. I recommend dragging it next to the layer-tab if you have Photoshop 7 and up.

With the Channel-tab open, click on the Save selection as Channel-button which will add a new Alpha-channel. To open this, simply click on the name "Alpha 1".

Inside the Alpha

When you're inside the alpha-channel, you can see that everything is reduced to black and white. White being your shape and the black being nothing.

Go to Filter>Blur>Gaussian Blur and depending how smooth or round you want your edges to be, apply a certain amount. I chose 9, which will get you a feather-radius (roundness) of about 18.

Now press Ctrl+L to open up the levels-window. Click and drag the black and the white slider closer to each other. Make sure they're not actually touching, because that will result in very choppy almost pixelated edges. The further they're apart, the fuzzier your edges will get.

When you're done, press OK and Ctrl-click the layer again to select this shape.

Finalising the shape

Simply click on the name of the top Channel to to return to the previous state. If you've selected the channel successfully, you will now see your selection over your old shape.

We can discard that and fill the selection with a desired color on a new layer.

Conclusion

And when you're done, you can re-add those layer-styles like you wanted. Remember that if you wish to have rounder edges, all you need to do is blur it more. Usually it's a simple math-solution to find out how much your feather-radius will be.

The feather-amount of almost always the blur-amount times 2. That means if you chose 9 like me, you will get a feather-amount around 18.

Enjoy and remember what I said: practice this a lot and in time this can be done in mere seconds!