These are some very common questions among web publishers especially those who have just gained admission to the University of Adsense. While the answer is to keep on experimenting, most Adsense experts employ a simple technique called "AB Split Testing" to optimize their Adsense ads.
Here's a sample scenario to help you determine what format works best for your site - 300x250 or 338x280 ?
Step 1: Create two custom Adsense channels - name them as 300x250 and 336x280.
Step 3: This is an important step, you will merge the two Adsense snippets in such a fashion that each makes an appearance on your web pages nearly 50% of the time. Here's a sample code:
Monitor the performance (CTR, eCPM) of the two Adsense channels for a week or two to know which of the two Ad formats are converting better. [Switch to Advanced reports, select the two Adsense for Content channels and group by both date and channel]
In the next iteration, you may try the same trick with two different color schemes but keeping the Ad format same - just create separate Adsense channels for each color scheme.
If you are displaying Adsense ads on the homepage of blogs or other places where content changes very frequently, it's a good idea to test between Text and Text+Image ads.
The contextual ads may not be very relevant on blog homepage or archives because of the dynamic content and therefore CPM based Image ads may bring in more revenue.
Related Reading: Google Adsense Tips and Tricks