So I just heard Bencivenga say that he used to get really disappointed whenever he lost a split-run test against someone else.

That said, how exactly do these split-run tests work?

More specifically: How on earth can I conduct one? I mean, am I just supposed to shell-out $25,000 to Fortin.... $25,000 to Kennedy.... and $25,000 to Carlton and see who wins???

I mean, is there any way to conduct a split-run test between multiple copywriters without spending a fortune?

Sure. You can test different sales letters against each other.

More likely, you can different headlines or other sales letter elements against each other.

Grab moderator John Ritz's free Profit DNA report and you'll be on the road to better understanding the science of testing.

First... you don't have to use a superstar. Savings right there...

Second... you use 1 writer and split test changes in the copy...

Split test the "control" headline against a new one. If the new one wins, keep it as the control and test another, etc.

When you get the best performing headline, you move down the letter to the next area... Maybe it's the story. Keep the winning headline and start split testing the original story against another. Or against no story. Or the original story at different lengths.

When that's perfected, you continue on... Offer... guarantee... PS's...

Deep pockets would be the only corps that could affor to split test Gary against Clayton against Carlton, etc... and, IMO that wouldn't be money well spent. If you've got Carlton... start split testing as I outlined above.

Or, better yet...

You might want to try multivariate testing to speed things up.

Excellent point, Vin.

I'll tell a story on myself here. First, no top writer would knowingly go up against another one (unless it was some kind of grudge match) in an initial mailing. That would be like pitting the best seeds against each other in the first round of a tournament. Not a good strategy.

In the weird social politics of the scribe hierarchy, it would also be a no-win situation (and I'll explain why in a moment).

(That said, top writers sometimes DO participate in the same test... but the people paying for their services don't let them know.) (I'm a good example of why they want it kept secret, too: Whenever I found out I'd beaten one of my colleagues, I was merciless in kidding them about it.) (It's a sign of respect, in an undignified, juvenile, "nyah, nyah" way.) (And it's something you don't do with non-veteran writers, because you want to be careful not to crush anyone's spirits.)

(Okay, one other sub-point here: Top writers will go up against a control previously written by a peer, happily. With guys like Bencivenga, that control will change, however, because he's always tinkering with it.

It's no fun going up against guys like him. You'd be foolish to try.

Most writers, however, don't want to touch a winner once it's established as the control, for fear of messing up the mojo. So you've got a static target.

If you know how to deconstruct the control, you can figure out why it won originally... and find a way to build on those elements, rather than come up with something totally original.

The writer with the control is rightly upset with sneaky usurpers standing on his shoulders to beat him like that (and steal his royalties)... and I've heard that some top writers have it written into their contracts that only "in-house" writers will compete with them in the first few rounds of mailing schedules.

This kind of endless competition is great for the client, however, as the baseline for results keeps getting raised. In theory, anyway.

Oh, it's brutal in that high-end game. And all's fair.

Still, on a first-time mailing where there's no control, veteran writers prefer an uneven playing field with no other top guns in sight. You've got enough trouble crafting "A" level copy, without having to worry about another writer beating you by a hair in the test.)

Second: I often consult, behind the scenes, with "A" list writers looking for an outside eye on their hooks and sales angles... and I know that other writers also do this, freely and without credit or payment.

So by pitting a top guy against another one, you might be interupting that exchange of info and free critiques.

You'd be paying other writers to withhold info from competitors, and maybe get three pieces from three top writers that were worse than what you'd have coaxed out of one, with unrestricted access to his peers. (As Vin pointed out.)

In fact, I lost an attempt to beat a hot control for Boardroom once... and learned afterward that the writer had written the control after getting multiple critiques from me, the previous year. For free.

So I kinda beat myself. (I thought that control copy looked familiar.)

Tangled webs and all that.

Third: Here's the story I tell on myself. It illustrates something important.

Many moons ago, a deep-pockets client persuaded me to write an ad for him. I didn't care if he was paying other writers to go up against me or not -- I was focused on the control ad, which had only recently started to fatigue after a decade or so.

This was definitely not "easy pickings".

So I went gonzo with the research, and dug up all sorts of great ammo to use.

However, I let the client dictate the main benefit of the ad (and thus the USP). I had a hunch his demographic was actually responding to a secondary benefit he offered (but did not have in the headline). I got lazy, and didn't feel it was appropriate to bully this guy.

So I went against my hunch, and wrote the ad the client wanted. He loved it. (If you've ever heard me talk about dealing with clients, you know that this was a VERY bad sign. I never feel good about an ad unless the client is NERVOUS about it. If he's happy, I missed something.)

What I didn't know at the time was that this BASTARD had also hired Gary Halbert to write a separate ad, for a straight-up test. If I had known this, you can bet I would have done a more thorough gut-check.

Gary actually went with a similar hunch. Didn't even blink or have a blush of doubt -- just went with it. And insisted that his ad go out as is.

The client laughed at him, and his staff openly ridiculed what Gary had written. They all thought sure that my ad would demolish his in the test.

Didn't happen. Gary's ad -- which was aimed directly at the psychological sweet spot of the target audience -- won hands down. It was brutal.

And a brilliant example of how even sizzling copy is ultimately subservient to a killer strategy.

I kicked myself for a very long time after that (as all self-respecting writers, from Bencivenga on down, do when they lose)... and made sure I learned that lesson one more time.

Get your info, do your research... and then go with your gut, "gun to the head" style.

Professional copywriting can be lonely. When you learn that your instincts -- though proven right time and again -- will be mocked and actively fought against, you've got to dig in. (I was blacklisted at Rodale for refusing to change a word in the first piece I wrote for them. It almost never got mailed. Just as important: There are executives who still hate me for being right -- every year that piece continued as the control galled them no end, and they threw a fortune at other writers to try to knock it off. Good times.)

Halbert went for the jugular in his ad. The numbers were spectacular... yet, curiously, the client didn't continue to use Gary's ad. It just rubbed him the wrong way.

Gary proved his point, though. Pissed me off, but I cherish the lesson anyway.

No writer out there is immune from getting his butt kicked by another writer, regardless of reputation, wisdom, or skills.

Sometimes, it all comes down to a single piece of inside information that gets overlooked (or ignored) by another skilled writer.

Sometimes, that piece of info is more about salesmanship psychology, than about anything obvious with the target audience.

And now that I'm remembering other situations, there have been many clients who refused to go with a winning ad... because it violated their worldview. Belief trumped results. So you were wrong by being right.

It's a wicked world out there...

Hope this helps a little bit.

John Carlton

PS: In case you're wondering, yes, I've beaten Halbert before, too. While working closely with him, we'd argue about pieces and -- though we heavily ghosted each other -- we kept meticulous track of who was "right" and who was "deluded and pathetic" about the direction or thrust of any ad.

But I never wanted any final battle to "settle" anything -- I enjoyed having The Big Ugly Guy around in all his self-aggrandizing glory, and happily defer to him in nearly every category. I would never dream of ever trying to upstage him (though I delighted in embarrassing him and one-upping him in insults... a separate talent he deeply appreciated).

He set a bar that I doubt anyone else will clear for a very long time.

That mold has done been broke.

PPS: One last thought...

When you're considering sneaky ways to beat someone with better copy, never forget that MOST of the game rests on traffic, the right list in the right mood (like, starving or desperate), and a better offer.

It's futile to hire a top writer, at top prices, until after you've got your traffic/list/offer ducks lined up.

Hey Vin, do you think MV testing's a good idea if a "marketer's" never run even a single split test? Seems you could get lost down the rabbit hole pretty quick, if you don't have someone who knows what they're doing.

Andy,

Truth is (and I've been pretty open about this) I'm not a fan of multivariate testing at all. I believe the old fashioned split testing is slower... but the way to go. (for me at least)

I just mentioned it in that post because after reading what I wrote I said to myself, "someone's gonna negate it all by bringing up multivariate testing. so let me head that off at the pass."

I just figured mention that it's an option, then move on.

And as for your question... what you say about getting lost down a rabbit hole is why I don't care for it. It's beyond the scope of my little brain to keep something like that going without big screwups.

I look at testing copy the say way Dell's Tech support used to test machines on the phone. they'd always tell me to change only one setting per boot. That way if something changes in the performance of the machine, you don't have to figure out what setting you touched, did it.

I'm right with ya on that. And going back to the original point, if you were running 2 or 3 different writers against each other - I'd think the worst thing you could do would be try and cobble together a sales letter based on what you thought was working in each attempt.

Like Cucarifa implied, that's where the "art" of the matter rests, being able break down what's been successful and run it through your internal incubator with the insight to reassemble it into something new and better.

Which reminds me again of a former life of mine.

My apologies to everyone that I keep going to the same well, but I spent a lot of years in TV and doing what your talking about is like how when networks test pilots in front of test audiences... then get their hands into the "creative end" and screw things up.

They'll fire the creator... bring in a new head writer... creative consultants... even some "suits" to "fix it."

In the end you wind up with some Frankenstein creation that doesn't resemble the original show at all.

A LOT of great lessons in that post, John. As always, thanks for telling it like it as!

Good points, Vin and company as well. For a major campaign, I almost always prefer split testing over MV, because, well, for me, I'm more involved in it. As Vin pointed out, it's slower, but in some cases that's a real benefit. It forces you to pay attention. Depending on how one particular test goes, you may think of additional things to test that you never would have if you just threw everything into a MV tester and let it spit out the results for you.

But that being said, if you're juggling many projects, you almost have 1 or 2 "top" ones you're babysitting, and several others that are less of a priority, but you still want to always try to beat your controls.

For those secondary projects, MV testing can make sense, because you can almost "set it and forget it" while you focus on the priorities.

Great thread!

John