Depends on how much relevant traffic your getting. You want enough to make a statistically valid decision.
Perry Marshall's free split testing tool can help you to determine a confidence factor. Although it was designed for Adwords PPC meterics, it will easily apply toward split testing as well.
:::SplitTester.com:::
Simply enter your unique page visitors where it says "number of clicks" and your conversion percentage where it says CTR. The tool will tell you whether you have enough information yet to determine how likely your test subjects are to have different long-term results. A confidence factor of 95% or higher is where I like to be. Anything less than 90% means wait a little longer and try again once you have more data.
John