Your Guide to A/B Testing: How it Works to Drive More Advocacy Actions

There is no better way to learn how to most effectively engage your advocacy list than to regularly conduct A/B tests. However, many organizations either don’t test or conduct tests that don’t provide actionable insights for improving their advocacy campaigns.

Following these simple tips will result in your organization gaining a better understanding of what motivates its volunteers or members, and sending emails that drive more actions.

What is A/B Testing?

A/B testing is an essential tool in advocacy campaigns. It allows organizations to refine and optimize their outreach strategies to effectively engage their audience. In advocacy, A/B testing can determine which messaging and communication tactics resonate most with supporters or volunteers, ultimately driving more significant engagement and action.

In essence, A/B testing in advocacy involves creating two versions (A and B) of a specific element within your campaign — whether it be email subject lines, call-to-action buttons, or fundraising appeals — and testing them with two random segments of your audience. By analyzing which version yields higher engagement metrics, such as click-through or action rates, advocacy organizations can gain actionable insights into their audience’s preferences and behaviors.

What Should Advocacy Teams A/B Test?

Nearly every component of your advocacy email can be tested. Here are specific elements that can drive major improvements in your action rates.

Subject Lines

• Short vs. long
• Urgent vs. non-urgent
• Direct vs. intriguing
• Serious vs. humorous

Email Content

• Including an image vs. no image
• Action button vs. only hyperlinked text
• Issue description focused on statistics vs. street-level impact
• Personal story vs. just the facts

The Voice You Use

• Grassroots director vs. lobbyist
• Staff vs. volunteer
• Formal vs. informal

Email Logistics

• Day of the week
• Weekday vs. weekend
• Time of day

A/B Testing Best Practices

Test One Thing at a Time

The most common mistake people make is testing multiple variables simultaneously. If you test both a new subject line and issue description in the same email, you won’t know which change drove the improved (or decreased) performance.

Keep it simple. Test one specific element and be intentional about the difference between your two versions. Then integrate what you learned into your next set of tests.

Choose Between a Full List or a Sample Test

You have two options for testing your list — sending it to everyone or testing a sample first.

With a full-list test, half your list receives version A and half receives version B. This works well when you’re under time pressure to send your action alert or if you have a smaller list (under 10,000 records).

For less time-sensitive advocacy alerts, test with a smaller segment first. Send version A to 20 percent of your list and version B to another 20 percent. After about four hours (or a full day if you have the time), send the winning version to the remaining 60 percent. This approach lets you send the optimized version to a far more significant percentage of your list.

Test Regularly and Early in Your Campaigns

It’s beneficial to test a concept at least twice before drawing a definitive conclusion. That will help account for any anomalies related to a particular issue or moment in time. If your two tests show different results, look back to ensure the concepts were consistent across both tests. If so, test two more times to determine which result should be considered the truth.

Don’t wait for a critical legislative moment to begin testing your emails. Conduct most of your testing during the early parts of your campaign so you can send optimized emails that drive the strongest response when the actions matter most. 

By building testing into your regular email schedule you’ll be sure to have the answers you need at the most important moments of your campaign. 

Track What Matters Most and Use What You Learn

When analyzing your test results, focus on the metrics that truly matter. For advocacy that typically means completed actions. 

While open rates can provide interesting data, the number of people who actually complete your advocacy action will determine your campaign’s success. This is true even for subject lines, as they can impact far more than just open rates. Your subject line sets the tone for your email content and can also impact your clickthrough and action-taking rates.

Create a simple tracking system that documents:

• What you tested
• The results for each version
• What you learned from the test

The insights from your tests are only valuable if you put them into action. Consult this information regularly when writing future emails and be sure to integrate your learnings.

5 Tips for Getting Started with A/B Testing

Ready to try A/B testing for yourself? Follow these simple tips to get started. 

1. Select two elements you want to test in your next campaign.
2. Before launching your test, write down the exact element you’re testing and what you hope to learn from the results.
3. Choose your testing approach based on your timeline and list size. If you have time, use the sample method. If you need quick results, test the full list.
4. Create a testing calendar for the next three months, identifying specific elements to test in each campaign.
5. Focus on the metrics that truly impact the results of your campaign and integrate those learnings when creating new campaigns and content. 

Brian Rubenstein, president of Rubenstein Impact Group, helps nonprofit organizations build powerful grassroots programs by increasing volunteer engagement in advocacy campaigns and developing sustainable volunteer leadership programs. You can follow him on LinkedIn or email him at brian@rubensteinimpact.com.