It is somehow counter-intuitive, but rounded corners in polygon is not that easy task. Here are two of many possible solutions of the problem presented as UIBezierPath extensions. Keep in mind that for both extension polygon is represented by array of points ordered as they appear on perimeter.

Chaikin’s algorithm:

1. Traverse triplets of polygon vertices.
2. Find midpoints for each side.
3. Use middle vertex as a control point for Quadratic BezierCurve

This is it. You have nice blobbish rounded polygons.

Here is the code:

Usual corner rounding

Regular corner rounding can be done using planar geometry techniques. The idea is well described in this stackoverflow answer.

Yet I made small addition. Rounded polygon may degenerate into circle in case when edges lengths are not sufficient for given corner radius. In this case it will be a circle inscribed into a triangle defined by the triplet of points.

The code is here: UIBezierPath+RoundedCorners.swift

Or you can test it right away - it is a part of my Experiments Playground project