Technology

E-mail is completely broken and unreliable thanks to big players like Google, Microsoft and Facebook. Shortly after the NSA spying revelations, I decided to move off of Gmail and back onto my own e-mail server. It wasn’t for privacy, as e-mail is often transmitted plain-text and has no more security than a postcard, but just a general desire to distance myself from Google services. I had run an e-mail server in the past using Postfix and Courier-IMAP back around 2005 (as well as Amavis-new, Spamassassin and ClamAV for spam and viruses). When I attempted to setup an e-mail server in 2013, the stack was pretty much identical except Dovecot now replaces Courier and additional tools such as DKIM, DMARC and SPF are now necessary for outgoing e-mail validation. However the largest challenge I faced wasn’t from my own technology stack, but with my e-mails becoming unreliable against both Google’s and Microsoft’s over-aggressive spam filters.

Google, Facebook and Closed Communication

Google has always been at ends with Facebook as they began to seriously compete with each other over communication services. In 2010, Google blocked Facebook from importing Gmail contacts to build an initial friend community in preparation for the launch of Google+ in 2011¹. Since that time the two networks have always been separate, with no ability to import contacts or friends from one to the other.

Google’s primary communication system was built on top of e-mail, an open, federated, and standard communication system. E-mail allows anyone to setup a point of communication on their own domain. The word federated in this sense means that independent systems are allowed to communicate by means of standard addresses. Telephone systems are somewhat federated in the sense that many providers communicate with each other using a standard addressing system based on phone numbers and international calling codes. Postal mail is an analogue type of federation as each country can establish a post office and send items to each other using a standardized address. Although each system can have its own internal structure, implementation, sorting routines and technology, there is an agreed upon set of standards for communicating with others in the same domain.

When it comes to e-mail, messages from one provider to another are sent via Simple Mail Transport Protocol (SMTP). Facebook tried to integrate e-mail into its own messaging service, giving all users their own @facebook.com e-mail address based on their username. Facebook also silently replaced everyone’s public e-mail address on their profile with an @facebook.com address, forcing people further into their close communication system. The service, which started in November of 2012, was plagued with problems and was rarely used. Eventually, Facebook shut down the service in early 2014².

I really hated using Facebook’s messaging system. Facebook offered an XMPP interface, another open federated standard for sending Instant Messages, however the reliability of their implementation was atrocious. Many of my messages simply failed to send with no notification of failure. I’d often have to login to Facebook to ensure my messages were actually getting through. Even reliability within the web interface was inferior to every other proprietary Instant Messenger at the time including AOL Instant Messenger (AIM), Yahoo Messenger and Windows Live Messenger (MSN). Although it is considerably more reliable today, it took nearly a decade for it to catch up with its counterparts.

Eventually, both of these giants would abandon most standardized federated protocols. Google dropped support for federated XMPP in GTalk (now Google Hangouts) in 2013³. Prior to this, people could communicate with contacts on Google’s GTalk service from their own servers. Google silently removed this feature before publicly announcing it. XMPP could still be used with Google Hangouts for person-to-person IMs within their service, but group chats and video chats are now only available using their proprietary Hangouts application.

Facebook never had federated XMPP support, but even their basic XMPP interface was shutdown in early 2015⁴. This forces users into only being able to use Facebook’s web interface or mobile app. Without a replacement API, developers who want to integrate 3rd party applications with Facebook’s messaging service must now reverse engineer their proprietary protocol.

Overaggressive Spam Filters

In 2007, Google purchased Postini, a company specializing in spam filtering software⁵. At the time, I worked for a company that used Postini internally and it worked fairly well. In 2012 I was complaining to a friend about how I didn’t like Gmail’s user interface. Defending Gmail as a service, he made the point that Google’s spam filters were ahead of other services, preventing any spam from getting through. Later I would learn this really isn’t an advantage. Not only does Gmail’s spam filter prevent spam from reaching its users, but it also blocks an incredible amount of non-spam e-mails.

“Earlier this year I moved my personal email from Google Apps to a self-hosted server, with hopes of launching a paid mail service à la Fastmail on the same infrastructure. I’ve done this before, and this server was configured perfectly: not on any blacklists, reverse DNS set up, SPF, DKIM and DMARC policies in place, etcetera…I had no issues sending to other servers running Postfix or Exim; SpamAssassin happily gave me a 0.0 score, but most big services and corporate mail servers were rejecting my mail, or flagging it as spam: Outlook.com accepted my email, but discarded it. GMail flagged me as spam…” -The Hostile Email Landscape. Jody Ribton⁶.

I’ve often run into Ribton’s issues as well. Even prior to leaving Gmail, I had e-mail I’d send to friends that would end up in their spam folder. Even internally, their spam filter is horribly over-aggressive. I still use my university e-mail accounts, outsourced to Google and Microsoft, for sending e-mail to schools and professors. In my own testing, e-mail from those accounts tends to get flagged as spam, especially if I include PDF attachments.

Microsoft’s fares no better. A few months ago, I sent an e-mail to a friend who I e-mail several times a year. Out of nowhere I receive the following response:

This is the mail system at host **removed**.

I'm sorry to have to inform you that your message could not
be delivered to one or more recipients. It's attached below.

For further assistance, please send mail to postmaster.

If you do so, please include this problem report. You can
delete your own text from the attached returned message.

The mail system

<**removed**@hotmail.com>: host mx3.hotmail.com[65.54.188.72] said: 550 SC-001
(BAY004-MC1F57) Unfortunately, messages from x.x.x.x weren't sent.
Please contact your Internet service provider since part of their network
is on our block list. You can also refer your provider to
http://mail.live.com/mail/troubleshooting.aspx#errors. (in reply to MAIL
FROM command)

I contacted my ISP to see if there were any issues with spammers on the subnet my server was hosted on, or if they had any network operations specialists with communication channels with Microsoft. They said they were unable to communicate with Microsoft about IP blacklists, and the only solution they had was to assign me a different IP address. I took it upon myself to file a problem with Microsoft, which resulted in the following response:

Conditionally mitigated
x.x.x.x/32
Our investigation has determined that the above IP(s) qualify for conditional mitigation. These IP(s) have been unblocked, but may be subject to low daily email limits until they have established a good reputation.

Please note that mitigating this issue does not guarantee that your email will be delivered to a user’s inbox.

Ongoing complaints from users will result in removal of the mitigation.

Mitigation may take 24 - 48 hours to replicate completely throughout our system.

If you feel your issue is not yet resolved, please reply to this email and one of our support team members will contact you for further investigation.

Following this, I attempted to resend the e-mail, which resulted in getting the same response again. I verified my servers were fully patched and checked my logs to ensure no one had found an exploit to use my server to send spam. I came up with nothing. Eventually I broke out of this loop and my e-mail was delivered. At least in this instance, I got a notice. Typically e-mail is dropped without any indication to the sender.

From my own e-mail server, even if I send an e-mail with no links, images or profanity, it will still end up in the receiver’s spam folder or get discarded silently. SPF, DKIM and DMARC are all domain verification systems for validating e-mail’s origin to prevent spam. I have all three records set in DNS records for all the domains I send e-mail from, verified they were correct using testing tools, and I still get flagged as spam.

I almost always must use a second channel of communication such as Facebook, Google Hangouts, SMS/Text or even Reddit messages, telling the receiver to check his or her spam folder. Sure enough, once they do and mark that message as not being spam, subsequent messages get through fine.

The Connection Problem

Facebook has taken an entirely different approach to spam prevention and messaging. If we look back at the MySpace days, one of the features Facebook supported was two-way confirmation process for friends. People have control over which individuals decide to add them, unlike other services such as Twitter. This allows Facebook to build a network, and use the links between individuals to determine the potential for a message to be spam.

Messages sent to an individual outside of close links (friends, friends of friends, and so fourth) would often go to a folder marked other, more recently renamed to filtered messages. At one time, Facebook even attempted charging people a fee to bypass the filter to prevent spam. The fee varied and was most likely based on an internal/proprietary algorithm, with Facebook founder Mark Zuckerberg having a fee of $100 USD for sending a message to him⁷.

Marking a message as not-spam is essentially making the same type of two-way approval for communication. The major problem is that people don’t often check their spam folders, which can be filled with thousands of messages at any one time.

Google attempts to build a similar hierarchy/friend network with their Google+/Hangouts services. Recently Google integrated Gmail into this system, allowing people to send e-mail to people they were connected to on Google+/Hangouts without knowing their e-mail address⁸. These messages aren’t really e-mail, but they appear alongside other messages in Gmail, further pushing communication into a closed-wall system that only works through proprietary, non-federated, commercial systems.

Decline of E-mail

In 2009, a Nielsen survey found that people used social networking far more than they used e-mail⁹. Many people today only use e-mail to sign up for other services. It becomes a bucket of notifications that are never checked. The inbox has turned into the spam folder and Google’s attempts of adding priority e-mail and automatic sorting seem to have come too little and too late.

The simple fact is that today, e-mail has become completely unreliable. A letter sent through the post office is more likely to get to the intended recipient than an e-mail sent to someone who doesn’t have you listed as a contact. Facebook and Google’s war over market share of the Internet have caused people to flock to their services as primary communication mechanism.

In November 2015, Facebook began blocking all communication mentioning the new social networking service Tsu. One of Tsu’s selling points seems to be a means to share advertising revenue with users of their service. Facebook removed all posts with links to the site and even news posts commenting on post removal¹⁰. It is possible that Tsu was spamming Facebook, or that the massive interest by people triggered automated spam processes, however it’s also likely the blocking was intentional. Just as in the Google and Facebook war, connection maps of individuals are an important asset. When one for-profit company controls the communication medium, they set the rules and can easily stamp out competitors to their monopoly, in the name of spam prevention.

E-mail was once the pillar of the Internet as a truly distributed, standards-based and non-centralized means to communication with people across the planet. Today, an increasing number of services people rely on are losing federation and interoperability by companies who need to keep people engaged on their for-profit services. Much of the Internet’s communication is moving to these walled gardens, leaving those who want to run their own services in an increasingly hostile communication landscape.