Product
Privacy, automatic engagements, and Mailgun’s bot detection
Identify automatic engagements for a better understanding of your message events and engagement metrics.
PUBLISHED ON
Now more than ever, users are concerned about their data privacy and what steps they can take to protect their personal information. And that’s something big players in the tech space can’t ignore for much longer. As more consumers learn about how their data is used, different Mailbox Service Providers (MBP) and technology leaders are changing how they handle personal data. Just this past year, Apple changed how they managed opens and clicks by creating a blanket event that occurs anytime a new email lands in the inbox.
Privacy is important, and it is crucial for every single person who receives an email – senders too. There are solutions out there that benefit both parties, and the industry is moving forward with the ability to identify bot opens and clicks without compromising the privacy of the recipient.
As best practices change, so do we. That’s why today we’re proud to announce that Bot Detection is available in Mailgun.
Table of contents
Gmail
How do bot engagements work?
For Apple Mail, all unique images from an email message (delivered to the inbox) are prefetched, loaded, and cached onto a CDN. If you’re using open tracking, that prefetch results in triggering an open event for your ESP, which could be true or not. From a tracking perspective, this prefetching creates different issues, since the information you normally get from that open event – device info and location via IP address – is obfuscated: The User-Agent string returns as a generic `Mozilla/5.0` user-agent, and the IP address is that of the CDN node, as opposed to the recipient’s actual IP address.
Gmail does something similar through the use of its Google Image Proxy. Both user location and browser information are hidden through an automatic open and, in return, the `User-Agent` is `Mozilla/5.0 (Windows NT 5.1; rv:11.0)` `Gecko Firefox/11.0 (via ggpht.com GoogleImageProxy)`. While this might get confused with a user open, the geographic location and client info will not be accurate, and the result will return very quickly.
But where did this change leave email senders?
Open rates as a result rose of this change, but not due to recipients engaging with any emails. These inflated metrics leave senders with an inaccurate analysis of their engagement and bring into question just how much they should rely on open rates when making decisions regarding email marketing. To say it left the email marketing space shaken would be an understatement, but these changes didn’t completely kill email.
After months of tracking the impact of these bot engagements, one thing is clear: Email’s not dead, but open rates definitely aren’t the most pristine metric to begin with when it comes to recipient engagement.
Bot opens and clicks on the recipient side have been inflating metrics for years, and while that can be remedied with a clean list, automated engagements on the MBP side needed a different solution.
Enter Bot Detection
Automatic opens tell on themselves in the name alone – automatic. These bot MBP opens return results at the same speed, if not faster, than the `250 OK` return. Human beings have lives outside of their email inbox, and seldom are they going to open an email the second it hits the inbox – and if an organic open was to be mistaken for a bot with our solution, the recipient would have to open it within less than a second of receipt. How fast is your trigger finger? Even if it’s a password reset or a package delivered receipt, real people are always going to need a little extra time to open and read an email.
Since bot opens are easily identifiable, all the email service provider has to do is create detection around that factor. At Mailgun, we have added a field to the events payload that will inform on whether or not a message is automatic, as well as where the bot open originated from as far as MBPs are concerned. It will appear towards the top of the payload, as exampled here:
The `bot` field is where you will see whether or not an open or click is automatic from Apple, Gmail, or an unknown bot engagement which might be an edge device like a firewall or anti-spam software that fetches the message’s tracking image and registers as a click/open event. The bot field gives that additional insight onto what could be affecting your engagement rates, and allow you to understand your overall email program performance that much better.
Apple Mail Privacy Protection (AMMP)
If the User-Agent is exactly Mozilla/5.0 it’s mostly likely AMMP, which gives us an accurate indicator for detecting Apple’s bot specifically.
Gmail
Gmail obfuscates their user location and browser data using a proxy image, so we can consider opens detected with Google’s user agent to be the result of an organic user open, but can’t trust the accuracy of the accompanying geo data.
Like with some of our unknown edge cases, Google has been known to pre-fetch the tracking image upon delivery. If it’s Google, we are able to identify based on the unique user agent, and the fact that the click rate is under 1 second.
Next steps
The worlds of email and privacy are constantly changing for the better. It’s up to email service providers to move with them, and at Mailgun we’re ready to do just that. We can effectively separate humans from bots using the User-Agent and timestamps but in the future, we’ll look at adding in additional MBP bot engagement parameters as they enter the privacy space, and continue to createstronger bot detection features for our customers.
Have an idea for Bot Detection or Mailgun overall as a whole? Send us a message on UserVoice – our best services and features are your ideas.
