Email has revolutionized business and personal communications. Unfortunately the power and cost effectiveness of email has been utilized and abused by email spamming companies. To combat illegal and unethical spamming operations, ISPs, corporations and individual users have adopted a variety of techniques to screen unsolicited and unwanted mail. Because these techniques are imperfect, legitimate and completely legal business or personal communications are at risk of being filtered or blocked. Quickpivot is committed to responsible email marketing practices. Our clients all agree to abide both by legislation and by best practices for email communications. Our commitment in return to our clients is to ensure that their legitimate mailings are not inappropriately caught or blocked by the filters and blocking mechanisms that target spam.
This document outlines a number of the processes and systems that are employed by Quickpivot to monitor and resolve email blocking issues that impact our clients. The Summary of Blocking Techniques is intended to provide a perspective on the many ways that a message can be blocked. The remaining sections outline systems and procedures that Quickpivot uses to monitor and manage the entire landscape of blocking issues on behalf of our clients.
Systems that block according to the source of the message, typically identify the sending server using one or more of the following:
IP Addresses – The IP address of the specific mail delivery server has been identified as a known (or suspected) source of spam. Sometimes the blocking is applied to an entire sub-net of IP addresses to hinder the movement by spammers among multiple IP addresses. Public lists of such addresses are referred to as RBLs (Real-time Block Lists).
Domain names – Blocking any server that sends messages using a domain name for an identified source of spam.
Non-compliant servers – The sending server is not configured properly according to Internet guidelines. This might include open relays, incorrect DNS lookups and other indicators of “rogue behavior”.
Unknown senders – Some of the most aggressive solutions will only allow messages from known sources. These systems check email address directories to compare inbound message sources to lists of known senders. Other solutions use what is termed “challenge-response” systems to refuse all unknown sources of messages until the source provides a response that requires human intervention.
Message Size – The total size of a message, including attachments, can trigger filtering software. This type of filtering is also done to avoid the delivery of virus attachments as well as to protect network performance. For this reason it is best not to send attachments with bulk emailing but rather to reference the documents using a download link.
Keyword/Phrase Filtering – The frequency in which certain combinations of text are likely to show up in messages that are spam as determined through the use of using statistical modeling (Bayesian/Markovian). Keyword filtering is perhaps the most dangerous in that it is extremely difficult to guarantee that innocent usage of certain terms does not cause an important personal email message to be filtered. Beyond individual keywords the conjunction of certain words and phrases can trigger filters. Ironically, the legal requirement for an opt-out link (“Click here to be removed from this list”) can be the exact language that is used to trigger a filter, though email being flagged as spam due to a single phrase is rare. Keyword filtering is less significant than it was historically, at ISPs and larger organizations, because of improved filtering approaches and the likelihood of false positives.
HTML Usage – The type and amount of HTML formatting can be a strong indicator that a message is commercial. Unfortunately, email newsletters that are requested by recipients are also likely to make extensive use of HTML formatting.
Usage of Links – Most spam messages include one or more “calls to action” in the form of links to web sites. The inclusion of links can trigger spam filters, particularly if the link itself has been associated with a website operated by a known spammer. When a link in a message has been associated with a high frequency of complaints, that link may be added to a private or public SURBL (Spam URI Real-Time Block List). Thereafter any message that is associated with the link (or even the derivative domain used in the link) may be subject to filtering.
Reputation Based Filtering – In the last decade ISPs have started to develop more intelligent and robust email filtering systems which monitor historical message sending patterns, and user engagement with email messages. These sorts of systems will detect events such as sudden changes in message volume, message open, click, and complaint rates to build a reputation for a sender at both the domain and IP level. These types of systems are quite beneficial for legitimate senders as they make it less likely that messages are filtered based on content, such as HTML or text to image ratio, when they have a high engagement rate and a low complaint rate.
Domain-based Message Authentication – Another recent development in email deliverability is DMARC which was designed to identify spoofed messages, which are messages that masquerade as if they’re sent from a different domain. DMARC consists of two mechanisms which were developed to sign and authenticate the originating domain for messages called SPF (Sender Policy Framework) and DKIM (Domain Keys Identified Mail) which are published on the public DNS and can be used for reverse lookups based on the FROM address of an incoming message.
The likelihood of any of these indicators being used to filter a legitimate email message varies according to the settings that are used by the individual filtering systems and those who manage those systems. The same message that is flagged and filtered by one company may be perfectly acceptable to another.
A common component of sophisticated filtering systems is the use of “spam-traps”. These are email addresses that are not associated with any normal email usage. The original spam-traps were set up solely for the use of trapping spammers by listing the addresses on web sites that were likely to be “harvested” by web crawlers that were used to find email addresses on web sites. As these email addresses never really belonged to a person who would use it to opt-in to any lists, the assumption by filtering systems is that any messages sent to these addresses must have come from spammers.
A later evolution of this approach is to include long dormant addresses as spam traps. While these addresses may once have been used for normal email purposes, they have not been active for several years. The filtering systems make the assumption that any messages sent to these email addresses are from senders who have not maintained their lists and are therefore more likely to be spammers.
Oftentimes the inclusion of a spam trap in an otherwise “clean” email list is the result of an email append process that inadvertently adds a spam trap address to the list. Any time email addresses are appended to a list, those addresses should be marked as appends and tested carefully.
QuickPivot manages dedicated domain names and IP addresses for its clients. These domains and IP addresses are only used for mailings sent by a particular client and have minimal risk of blockage due to the activities of another client. Unlike many email service providers, Quickpivot uses its dedicated domains for its mail sending servers as well as for the “visible from” address and the domain links within messages. This ensures a completely distinct identity and reputation for each of our clients. We also regularly monitor MTA logs to ensure that the delivery rate to specific ISPs matches with their expectation for inbound messages per hour, and proactively scrutinize aggregate server level exchanges to identify issues before they happen.
Additionally Quickpivot provides Spamassassin content scoring directly in the tool and has partnered with ReturnPath and Litmus to help identify inbox placement at most major ISPs to discover content issues before they happen.
The technical aspects, though, are only half the equation. Arguably the most important factors for maintaining a good reputation and high deliverability are the business processes related to messaging customers, collecting addresses, and following industry best practices will help to avoid issues like high complaint rates or spamtraps. The second part of this series will cover in detail.
Read the accompanying article Responsible Email Practices: Spam Blocking