How do I test email subject lines when my initial list is small?
I have been reading a lot about the importance of testing different email subject lines and messages. However, I am just starting out and my initial list is small. I have about 200 emails right now. Would I be able to get a good enough sample size by testing different subject lines? Or should I wait until I grow my list?
Expert Answers
Sort by Date
Sort by Votes
Jan 20, 2016
Hi Jane,
at this stage, I would recommend you to focus on growing your list size first.
I see too many companies who focus on subject optimization while their list is too small. What is your open rate? If it was 20%, with 200 emails you would have sample of 40 people. That is just not enough.
Use a common sense, the subject should not look spammy and it should be to the point and interesting. Try to transform into your recipient an answer the question to yourself: "Would I open this? Does it sound interesting?"
When the open rate is 20% once and then 23% later (or for another group of people) - does it mean that the second subject is better?
It does not need to be the truth necessarily. What if the people in the first group just did not read your email because of the current weather outside? What if the people in the second group are just more engaged, so they would open even on the first subject no matter what?
Statistic methods work better with larger sets.
With small sets, they can be misguiding.
We were analyzing our customer's data from Clever Sender (www.cleversender.com) and for small sets, spending too much time on subject testing too soon is a waste of time.
Jan 14, 2016
The statistical validity of an A/B test is determined by three factors.
1) Sample size
2) The magnitude of your baseline response rate
3) The magnitude of the difference between the response rates
In testing, size matters: you need some combination of big samples, big response rates or big differences to generate valid results.
Your sample size is small but, theoretically, you can still get valid test results if your response rate is very high and/or if the difference between the two response rates is relatively large.
For instance if you send 100 messages to each group in an A/B test, and the click-through rate is 40% to group A vs. a click-trhough ate of 50% for group B, you'll have 95% certainty of repeating that result, at least theoretically.
For comparison, if your click-through rate is only 3% in one group and 3.3% in the other group, you'll need to send about 12,500 messages to establish a similar level of confidence in the result.
So, if your baseline clickthrough rate is relatively high, start testing now. But make your subject lines substantially different, i.e., try to move the needle a lot, because you won't learn anything with small samples if the magnitude of the difference is small.
Jan 13, 2016
Jane,
MailChimp has a cool program enables you to test two headlines in the same mailing and will also give you feedback on your headline through some type of algorithm.
Good luck,
Jim
Jan 11, 2016
Hi Jane,
just like Craig's suggested, the CoSchedule headline analyzer is a good tool to use.
Don't wait to grow your list, start with what you currently have (I started with a list of 40 and grew from there).
You can split test your email campaigns 50/50. Send out an email with a particular subject line to 100 people in your list and another with a different subject to the other 100.
Another thing to consider is the email-marketing tool you're using. I use Active Campaign, and it allows me to send the one subject line to 10% of my list and the other subject line to another 10%.
When the winner is determined (i.e. based on the number of opens or clicks), the remaining 80% of the list will get that winning email. You have the option set it up to any percentage you want; I just prefer to do it this way.
The main thing is giving your list something of value that will propel them to take the action you want.
All the best.
Jan 11, 2016
We operate with a smaller list as well. (1000) I'm answering from a B-B perspective.
Yes, start testing now. A) Make sure you test day parts as well. I find late afternoon and early morning to work well. B) Make sure your Subject Lines relate to the email. Solve problem or solution. (C I find that least caps in the subject lines the better.
Jan 11, 2016
I've had good success using the headline analyzer by CoSchedule. It's meant for blog post headlines but it is a great tool for grading each of your subject lines too.
Headlines/subject lines are very important and slight tweaking of each word can make a huge difference. This tool gives you a grade for each one and you can try a whole bunch until you get the highest score.
Don't wait. Sending to 20 people is good and will help you improve the quality of your emails.
Remember that the more personal and relevant your email, the greater response rate you will get so make sure you know your audience and what would resonate with them in the content of your email.
Check it out.
http://coschedule.com/headline-analyzer
Jan 11, 2016
Test your subject line at http://www.subjectline.com/ They offer free tests to see the effectiveness of your subject line to help them go from SPam to Inbox.
Craig thanks for the tip on coschedule...I'll check it out.