3 Easy Ways to Set Up an Effective Bot Trap with AVID Ad Server

All high volume marketers want clean, cheap traffic. However, this is a catch-22. Clean traffic is not cheap and cheap traffic is not clean (think about it, you get what you pay for). So, how do you get the best of both worlds? Most of the garbage that comes through cheap traffic is bots. Without going in to too much detail, a bot is a script that shady publishers write that automatically visits their site (millions of times per hour in some cases). Each visit by the bot registers as an impression in your DSP, so they charge you for those impressions, or if you’re running a CPC campaign, the bot simulates a click on an ad with the same result. The DSP sees a click and charges you for it. The DSP pays the publisher a percent of what you paid for the impression or click. The DSP makes money, the publisher makes money… and you don’t.

Obviously this is not a good business model for DSP’s. Most DSP’s already have systems in place to catch the most obvious bot traffic, but ultimately it’s you who ends up paying for it, so YOU need to know how to deal with it. The more effective bots are already getting through the filters set up by DSP’s. That means putting a robots.txt file in your page is probably not going to do anything. That’s because robots can ignore your robots.txt file. Especially malware robots that scan the web for security vulnerabilities, and email address harvesters used by spammers will pay no attention, and the /robots.txt file is a publicly available file. Anyone can see what sections of your server you don’t want robots to use.

DSP’s take reports of bot traffic seriously and will usually offer credits or refunds when traffic is proven to be bot traffic. Using an ad server is one of the best ways to set up bot traps. The ad server collects information on every impression and click that goes through your bot trap. That information can be used to prove to your DSP or traffic source that the traffic quality is garbage. You can also use that information to continually improve your bot traps. Here’s how to set one up.

3 Easy Bot Traps

The basic set up for each trap method is the same. In each placement you’ll create at least two ad groups. The first ad group has the ads you’re using in your campaign, so let’s call this your Default Ad Group. The second ad group, we’ll call this one Bot Traffic, may just have one ad image. It can be the same ad you’re using in your Default Ad Group, but with a dummy redirect URL that goes anywhere (e.g. https://en.wikipedia.org/wiki/Internet_bot). This set up works even if you’re using a link tracker placement to manage the URL redirects for your native, Facebook, or AdWords campaigns.

This simple set up will give you a quick breakdown of your traffic impressions and clicks. Clicking on the gear icon will open the Edit Ad Group page where we’ll set up targeting conditions to make sure the bot traffic is sent to the Bot Traffic Ad Group.

#1 Device Type Targeting

The first method uses AVID’s targeting by device type.

First, edit the settings in your Bot Traffic Ad Group by selecting Device Type as the Condition and Robot as the Device Type.

This will give you a targeting profile that looks like this:

Next, on your Default Ads Group, you’ll set up the opposite condition by clicking the “Exclude Condition” box. When using targeting on ads or ad groups, it’s important to set up the opposite condition to make sure there is an ad targeted to every condition to prevent wasted impressions.

Clicking “Add” will give you a targeting profile that looks like this:

With this targeting set up, your Default Ad Group (and the ads in the group) will be shown to all device types except “Robot.” The Bot Traffic ad group will only be shown to device types identified a bots.

This is a simple and easy set up, but has a similar problem as the robots.txt file we mentioned earlier. The device type field is self identified, so a bot would have to tell the ad server that it is indeed a bot, just like a smartphone has information that identifies it as a smartphone. This works fine for search engine bots, but bots that are trying to avoid detection would be programmed to hide or change the information that would self identify them as a bot.

#2 IP Type Targeting

How the next bot trap works is by using AVID’s IP Type Targeting. AVID looks up the IP address of the user and determines the IP Type. Common IP Types are commercial (.com), organization (.org), University/College/School (.edu), residential, and mobile. Bots usually hide behind proxy IPs which are classified as a Data Center/Web Hosting/Transit IP Type. Most real users would be classified as residential or mobile users, so we’ll use that for the Default Ad Group Targeting.

First, set up the Default Ad Group Targeting:

  1. Select Target IP Type as the condition for the Default Ad Group.
  2. Select Residential (Fixed Line) as the IP Type.
  3. Click Add.
  4. In a new Profile, select Target IP Type as the condition.
  5. Select Mobile as the IP Type.
  6. Click Add.

That should give you a profile that looks like this:

In AVID, having multiple profiles in targeting treats the conditions as OR statements. So our Default Ad Group will serve to residential OR mobile IP types.

Next, set up the Bot Traffic Targeting:

  1. Select Target IP Type as the condition for the Bot Traffic ad group.
  2. Check the “Exclude Condition” box.
  3. Select “Residential (Fixed Line)” as the IP Type.
  4. Click Add.
  5. Change the Profile to Profile 1, or whatever you named the first profile.
  6. Select Target IP as the condition.
  7. Make sure the “Exclude Condition” box is checked.
  8. Select “Mobile” as the IP Type.

Your targeting profile should look like this:

AVID Pro Tip: Adding multiple conditions to the same profile treats each condition as an AND statement.

The way this condition is set up the Bot Traffic ad group will show to all IP types EXCEPT residential and mobile. This set up is more reliable because it’s harder for bots to hide the IP type they’re using. Now, it’s harder, but not impossible. This method is not fool-proof, but it is the most reliable bot trap.

#3 Language Targeting

The other way you can set up a bot trap is through AVID’s language targeting. This is the newest targeting condition added by AVID Ad Server. It’s meant to be a way to target ads based on the language(s) enabled on a users browser, but it can also be an effective bot trap. Bots will either have no language, or have a language different than the local language. Take a look at some numbers from an advertiser currently running an ad campaign in France.

All languages enabled on the browser are recorded as ISO 639-1 codes with the preferred language being listed first and the rest in order of preference from left to right. One of the biggest things that jump out is the amount of impressions with no browser language. Most commonly used automatically set a preferred language and don’t allow a user to delete all preferred languages (although Firefox does). The other thing that jumps out is the amount of impressions coming from browsers that don’t have “fr” anywhere in the language string. It’s wrong to assume that these are all bots, but they are definitely outside of the target market for this campaign and raise a flag.

First, set up the Default Ad Group Targeting:

  1. Select Language as the condition for the Default Ad Group.
  2. Select preferred from the Language drop down menu and enter the ISO 639-1 code of the language you want to target.
  3. Click Add.

Your targeting profile should look like this:

Next, set up the Bot Traffic Targeting:

  1. Select Language as the condition for the Bot Traffic ad group.
  2. Check the “Exclude Condition” box.
  3. Select preferred from the Language drop down menu and enter the ISO 639-1 code of the language you want to target.
  4. Click Add.

Your targeting profile should look like this:

This will send all traffic whose preferred browser language is NOT fr-FR (French, France) to your bot trap ad group.

Be Prepared

Knowing how to deal with bot traffic is all it takes, sometimes, to turn a losing campaign into a winner. If the traffic source you’re running on has a lot of bot traffic, hit them up for a refund. It’s always easier to do when you have the data to back it up. The three traps we’ve set up are a great way to monitor and manage bot traffic. You can also be creative and combine the targeting techniques to further increase your bot detection. With an ad server like AVID, bot traffic just became as easy to detect and optimize for as setting up a new traffic source is.