Google today introduced new Adsense ad formats that have rounded corners - the ads are available in two option - slightly rounded and very rounded (the former has a slighly lower corner radius that the latter).

google adsense round ads

If you are displaying Adsense Ads with background color same as the border color (which is recommended for high CTR), this new format would be of little interest as the ads will essentially look the same.

In terms of Adsense implement,  the following variable will be added to your ad script depending on the version you select:

google_ui_features = "rc:6"; [slightly rounded]

google_ui_features = "rc:10"; [very rounded]

google_ui_features = "rc:0"; [default, square corners]

Probably the rc: numbers indicates the pixel radius.