Improving email deliverability with DNS records

As always, every day brings up new challenges, and today I faced one of my greatest fears - dealing with mailing DNS records.

Up until today, I was only familiar with A Record, CNAME Record, and TXT Record. A while ago, I used the MX Record while setting up my mailbox for my domain, but I don't quite remember its purpose, but enough about me.

Mail eXchangers, such as Gmail, Yahoo, Outlook, check the validity of incoming email messages. As part of the validation process, the Mail eXchanger queries the DNS records of the sender and evaluates the sender's reputation.

The sender's reputation affects how the Mail eXchanger classifies the message and whether or not it should be marked as spam before it is delivered to the receiver. In some cases, the message is blocked entirely by the Mail eXchanger, and the receiver is unaware of it.

The Reputation

There are many algorithms out there for evaluating an email sender's reputation. To put it on a scale, I classified the reputation level into four levels - Very Low, Low, Standard, and Best.

Very Low

Blocked completely without even bothering the receiver, here are two common error messages that may appear in the ESP's logs if the sender's email has sent too many emails in a short period.

421 4.7.0 [TS01] Messages from <1.2.3.4> temporarily deferred due to user complaints
<1.2.3.4> ;see http://postmaster.yahoo.com/421-ts01.html

553 5.7.1 [BL21] Connections will not be accepted from 1.2.3.4,
    because the ip is in Spamhaus's list; 
see http://postmaster.yahoo.com/550-bl23.html

Above snippets from - glockapps.com - How To Remove Your IP From Yahoo Blacklist – Yahoo Blacklist Checker

If there's a need to send large volumes of email messages, it is recommended to purchase a dedicated IP with a warm-up mechanism. The subject is broadly explained in the provided link, though in short, it's another method of identifying yourself as a trusted sender, by saying:

"I own a unique IP address because I'm confident that Mail eXchangers won't block me. I know that it's easy to block an IP address, so yeah, I'm that confident."

Low Reputation

Emails are marked as spam due to the low authenticity of the sender or previous reports of other users. But, most of the time, it's because the content of the message is not specific enough to the receiver.

For example, Willy is a Gmail customer, and he is interested in computer science and surfing. Then, out of the blue, Willy receives an email message about a great body lotion for women. Though Willy may have subscribed to some cosmetics stores websites, if the message isn't specific enough, like "Hi Willy," then it might be marked as spam.

Send the relevant content to the appropriate audience to avoid getting here.

Standard Reputation

Email messages are getting to the client's mailbox as they should, though, in some Mail eXchangers such as Gmail, messages could move to the Promotions tab, while it wasn't intended.

"... Gmail will recommend emails based on previous user engagement, regardless of whether annotations are present."

Best Reputation

Email messages are getting to the client's mailbox as the sender intended.

Some businesses may want to get on Gmail's Promotions tab on purpose; it all depends on the sender's needs.

Are you ready to explore the dark depths of mailing DNS records? Let's go!

Baseline

Previously, I mentioned that I'm familiar with a few DNS records, but that doesn't mean that we're on the same page. So line up!

Plot

I need to improve the email deliverability of outgoing emails. The request first came from the marketing expert of one of my customers, and he told me "We need you for the DNS part."

Ok, I'm up for it; let's set up the DNS records and assist my customer in reaching out to his end-users.

But how can I make sure that I succeed? I mean, doing/learning stuff is fun, it really is, but how will I know if my changes impacted my email sender reputation? How do I know if there's a better open/click rate for my emails?

The above questions are widespread when the need to send emails arises. That usually comes with many requirements, such as "unsubscribe mechanism", "group contacts by segments", "statistics of open/click", and the list goes on, depending on your marketing manager 😉

Developing a system that can measure open/click rate and handle email/contacts management will take ages to build...

So maybe I should ...

Use professional help

Here comes ESP into play! ESP stands for Email Service Provider, and that means there are cloud providers out there who are willing to send emails on my behalf and enable me to manage the whole email delivery system in a one-stop-shop.

There are many ESPs out there such as Google Workspace, SendGrid and Mailchimp. Each provider has its pros and cons, but the common ground is - they all send emails on behalf of your domain.

The difference usually comes when an ESP, like SendGrid, offers comprehensive services like Automated Security which enabled you to purchase a dedicated IP as a service (we'll get to that).

We covered ESPs, moving on to getting familiar with industry standards for improving email deliverability.

NOTE: I might mention SendGrid a lot during this blog post, though I'm not biased towards any ESP; SendGrid is simply the service I've been using recently, so it's easier for me to make references. Most ESPs offer a free/trial plan so you can explore their features and then decide, and I recommend evaluating at least two ESPs before picking one.

Standards For Better Email Delivery

According to Google, at least four standards should be implemented (in this order)

1. SPF
2. DKIM
3. DMARC
4. (Optional) BIMI

And, I'm adding A dedicated IP to the mix.

Mail eXchangers check if the sender meets industry standards by querying its DNS record, and according to that, evaluate the sender's reputation. The sender evaluation process includes other methods, such as looking in https://multirbl.valli.org/ to see if the source domain (eventually IP) of the sender is marked as an "official spammer", that all it depends on the Mail eXchanger. The more standards the sender meets, the higher the chances the sender's email message will arrive at its target audience (receiver).

The illustration below is for the SPF record. Still, the same communication process is the same for all standards, where the Receiving Email Server (Mail eXchanger) validates the email sender's authenticity and reputation.

Image Source: https://twilio-cms-prod.s3.amazonaws.com/original_images/spf_mail_flow.jpeg

SPF

• Definition: Sender Policy Framework
• Remember with: S for "Sending emails"

This record allows ESPs such as SendGrid, Gmail, Mailchimp to send emails on behalf of your domain. So if my domain name is meirg.co.il and I would like to send emails with Google Workspace, I need to add Google's SPF record to my domain meirg.co.il.

And the funny thing about an SPF Record, that it's actually a TXT Record. Let me remind you that TXT Records are for "Domain validation and authenticity," so it's nothing more than adding a TXT Record and adding a value that is specific for the SPF standard.

Here's an example for setting up two SPF records, one for Google Workspace and the other for SendGrid's SPF.

# Google Workspace
Type: TXT
Name: meirg.co.il
Value: v=spf1 include:_spf.google.com ~all

# Sendgrid
Type: TXT
Name: em123.meirg.co.il
Value: u12345678.wl123.sendgrid.net

Notice the weird thing? Google Workspace has an SPF expression, while SendGrid provides a CNAME as a value in the TXT record. My gut told me that the CNAME should eventually resolve to an SPF expression, so I checked it with Authentication @ tools.wordtothewise.com.

I tested it by navigating to https://tools.wordtothewise.com/dns/txt/em123.meirg.co.il, which resolved in ... Drums ... u12345678.wl123.sendgrid.net as a TXT record which contains the following expression.

# SendGrid's CNAME resolves to SPF expression
v=spf1 ip4:122.122.122.122 -all

It is quite a wonder that eventually, it's ip4:122.122.122.122 and not a CNAME, as we saw in Google Workspace include:_spf.google.com. This wonder is called a dedicated IP.

Declaring an SPF Record, or any other record that is related to mailing, usually comes with a complete guide on how to create it, check Google Workspace Basic Setup for SPF which also explains how to include multiple ESPs in a single SPF Record.

When should I use a single SPF expression?

Getting back to my previous example of Google Workspace and SendGrid; Luckily, SendGrid provides a particular TXT Record, which includes a unique subdomain, em123; this makes the setup is really nice since you don't modify existing with the fear of breaking something.

Assuming your ESP*s* required the TXT Record of SPF to be present in the root domain, e.g., meirg.co.il, you'll have to edit your existing SPF expression and add additional ESPs. For example, here's how you would write an SPF expression when using both Google Workspace and Mailgun.

# Google Workspace + Mailgun
Type: TXT
Name: meirg.co.il
Value: v=spf1 include:_spf.google.com include:mailgun.org ~all

NOTE: Regarding which ESP you should choose (Google/SendGrid/Mailchimp/Mailgun, etc.), it depends on your needs. I use Google Workspace for receiving/sending emails on behalf of human beings who use @meirg.co.il and SendGrid for sending automated emails with my web application @meirg.co.il. By the way, Mailchimp is also excellent, I used it a long time ago, and I truly enjoyed it.

Recap on SPF

A TXT record that contains an SPF expression or a CNAME that is eventually resolved to an SPF expression. The SPF expression contains an authorization policy about which service can send emails on your domain's behalf.

IMPORTANT Make sure you send emails from the same address that you've authenticated in the ESP (SendGrid, Google), for example, DO NOT TRY THE FOLLOWING, sending emails from no-reply@meirg.com. In contrast, my authenticated domain address is meirg.co.il. That can easily happen if you're using SendGrid - API keys to send emails since the FROM field is not protected, the API call can "endure" anything you shove it.

DKIM

• Definition: DomainKeys Identified Mail (Signatures)
• Remember with: K for Keys.

This one is going to be very short compared to SPF since we've already covered all basics,

I like to think about DKIM as the HTTPS of emails. So DKIM is a way for an email sender to increase authenticity by adding a signature to the headers of an email message.

"...DKIM adds an encrypted signature to the header of all outgoing messages. Email servers (Mail eXchangers) that get signed messages use DKIM to decrypt the message header and verify the message was not changed after it was sent."

Here's how the DKIM record looks for my domain meirg.co.il.

Type: TXT
Name: google._domainkey.meirg.co.il
Value: v=DKIM1; k=rsa; p=Ultra-Long-Key-PUBLIC-KEY-392-chars

The subdomain google._domainkey was generated by Google for my domain meirg.co.il. After generating the DKIM record, I followed the instructions and authenticated my domain by adding a TXT record with the value provided by Google.

If I intend to use other ESPs, I can generate a subdomain per ESP, so for SendGrid, it'll be s1._domainkey.

As you can see, the *._domainkey is the critical value, as that's the one that the Mail eXchanger is checking. So if a DNS record, such as TXT or CNAME, is named with _domainkey, you can classify it safely as DKIM. The *._domainkey prefix, such as google and s1, is called a selector. One domain can have multiple DKIM selectors when using various ESPs.

NOTE: Curious about my public key? Dig for it! dig TXT google._domainkey.meirg.co.il

DMARC - Here comes D-MARC tu du du du

• Definition: Domain-based Message Authentication, Reporting & Conformance
• Remember with: DM "Domain Monitoring" and D-MARC like "the man(ager)"

We covered SPF and DKIM, but how do we know that our email messages are authenticated with SPF and signed with DKIM? Is there a way to track down it down with numbers? There is.

DMARC is an email authentication standard - Meeting up this standard means your SPF and DKIM domain records are legit. How does it mean that they're legit, you ask?

• Once a day, a DMARC Report is sent to a predefined mailbox. The report contains pass/fail attempts of SPF and DKIM.
• DMARC has the reject policy, so if a sender attempts to send an email and both SPF and DKIM checks didn't pass, the receiver (recipient) would not get the message.
• Furthermore, DMARC helps to authenticate the sender's identity since SPF requires a TXT record to exist in the sender's domain containing valid domains or IPs (usually of an ESP), and a TXT (or CNAME) record for DKIM which includes a public key that was given to sender by the ESP.

Here's how a DMARC Record looks like

Type: TXT
Name: _dmarc.meirg.co.il
Value: v=DMARC1; p=none; rua=mailto:meir+dmarc_agg@meirg.co.il, mailto:dmarc_agg@vali.email

The p=none is the declared policy, which is nothing. Having no policy at all is almost the same as not having a DMARC record. A proper DMARC record would be p=reject.

So why am I using p=none, you ask? Because I'm still not 100% sure that my outgoing email messages pass both SPF and DKIM by Mail eXchangers. Having p=reject would block my messages completely, so for testing purposes, start with p=none, and once you're sure that all your emails pass both SPF and DKIM, set the policy to p=reject.

Sounds great, but how can you check all that? It sounds tough.

See the rua key in the code snippet above; I added +dmarc_agg to my email address and created a filter in Gmail, so any message coming from meirg+dmarc_agg@meirg.co.il is tagged as dmarc-report. Remember that I mentioned that a daily DMARC report is sent to a "predefined email address"? that is it.

But do I want to analyze DMARC reports on my own? Yup, you guessed it, there's an excellent service that can do that for free, and it's called ValiMail. ValiMail gets the daily report and analyzes it by checking how many SPF and DKIM successes/failures occurred in the past 24 hours and presents the summary in a friendly dashboard.

ValiMail are partnered with many known organizations, such as Microsoft, Google, and SendGrid, so I consider them a trusted vendor.

NOTE: The silent dmarc_agg@vali.email in the rua key points to ValiMail's mailbox.

BIMI - Requires DMARC

• Definition: Brand Indicators for Message Identification
• Remember with: B for Bragging to our audience/receivers that we're a top-notch validated sender.

Reminding you that DMARC has the "reject policy", so if an email sender sets it to p=reject, pct=100 (or p=quarantine), ALL none qualifying emails (SPF+DKIM) will not be delivered. So, cool, who enforces it? BIMI!

Qualifying for BIMI requires registering a trademark in a known patent agency and then purchasing a Verified Mark Certificate (VMC) from DigiCert or Entrust So, meeting the BIMI standard is not that simple.

And of course, it is possible to partially meet BIMI for non-trademarked logos. However, since BIMI is still not enforced or widely recognized by Mail eXchangers, I consider it a nice-to-have, and I'll get to it once I'll implement it once I get enough data from the DMARC reports.

To my feeling, it'll be easier to qualify today for BIMI than it will be in a few years. If you spot a fully certified BIMI email sender, it means it is ... Genuine and can be trusted.

Mysterious MX

• Definition: Mail Exchanger
• Remember with: Gmail is a Mail Exchanger, Google Workspace is an ESP

Finally, we got to MX; in case you haven't noticed, I wrote "Mail eXchangers" each time I had to mention a mail exchanger. You've just learned that MX Record stands for Mail e*X*changers that are allowed to receive emails on your behalf.

Since MX Records are for receiving emails and not sending, see my stackoverflow answer on how to configure MX Records for AWS Simple Email Service (SES) and the things that I've learned during the process.

To see how an MX Record looks like, check mine with mxtoolbox.com.

Gmail is a Mail eXchanger

How do you send emails from your Gmail account if it's only receiving messages with a Mail eXchanger?

As part of being Gmail's client, you can send emails on behalf of @gmail.com. That is set behind the scenes by Gmail's engineers, so each time you send an email using Gmail's UI, or use Gmail's API, the emails that you send are signed with Gmail's signature, which includes SPF, DKIM, DMARC. That is why your emails are not marked as spam when you email your friends and family since most Mail Exchangers treat @gmail.com as a high reputation sender.

Strengthen your knowledge

Name all the standards and provide a short description

All records refer to the email sender's domain.

1. SPF - TXT record, which contains the allowed sources, domains or IPs, to send emails on behalf of an email sender.
2. DKIM - TXT/CNAME record, which contains a key generated by an ESP. I consider DKIM an HTTPS mechanism for emails, which uses the public-key cryptography to authenticate messages.
3. DMARC - TXT record, which contains the receiver of the daily reports. This record is not just for setting the receiver, and it also means that Mail eXchangers will check this record and act according to the rules in it. So if SPF and DKIM is not set, some email messages might not arrive at their destination.
4. BIMI - TXT record, which contains a URL to an SVG logo of the email sender, and a URL of the Verified Mark Certificate (VMC) that the email sender purchased. And of course, this is how email senders can brag that they're legit in front of their customers.

How do I check if my domain is a trusted email sender?

Check if your DMARC record is valid with https://tools.wordtothewise.com.

https://tools.wordtothewise.com/dmarc/check/DOMAIN_NAME

Hooray for Google Workspace customers - there's a dedicated online tool for checking whether your mailing DNS records are correctly set for Google Workspace ESP; here are my results meirg.co.il - Google Check MX Status

IMPORTANT: Do not be fooled; Google Workspace's tool is valid only if you're checking Google as an ESP. If you use this service on a Sendgrid domain, it will show many errors.

How do I read a DMARC expression?

The syntax of how to read/write a DMARC expression can be found in RFC7489 - Domain-based Message Authentication, Reporting, and Conformance (DMARC). Another excellent source for understanding the syntax is Google's tutorial on the DMARC record format

Let's inspect @gmail.com's DMARC record together

v=DMARC1; p=none; sp=quarantine; rua=mailto:mailauth-reports@google.com

• p=none - all emails, regardless if they're validated with SPF+DKIM, can get to the receiver

• sp=quarantine - all emails sent from subdomains, e.g. subdomain.gmail.com and are not signed with SPF+DKIM are marked as spam in the receiver's mailbox. sp stands for subdomain-policy and overrides p rules for subdomains.

• rua=mailto - It appears that Gmail is handling their DMARC reports internally since the reporting address belongs to @google.com. So I guess they have their own "analyzing and monitoring DMARC reports" system, kewl.

Inpsecting @yahoo.com's DMARC record

v=DMARC1; p=reject; pct=100; rua=mailto:d@rua.agari.com; mailto:d@ruf.agari.com;

• p=reject - all emails that are not signed with SPF+DKIM are rejected.

• pct=100 - the percentage of messages to apply the p policy. In this case, if it's not signed with SPF+DKIM, it's not getting to the receiver at all, very restrictive

It's fascinating to see how different enterprise-grade email senders are setting up their DMARC records. It might look like Gmail is very permissive with their p=none policy, but I'm sure they have a dedicated backend that filters out spam messages before they arrive at their customers' mailboxes.

Bonus - Can Gmail customers check the sender's validity in the UI?

Yes, yes, YES!

Eventually, an email message contains text, and this text instructs the Mail eXchanger how to validate the message's sender, process referenced images to be shown in the UI, etc.

Gmail made it very easy to check, click the ellipsis, and then Show Original.

That results in a very long page that contains the whole email message, though Gmail made it very easy to track the important things, such as if DMARC is valid.

After reading this blog post, you've just learned that checking DMARC is enough since it relies on both SPF and DKIM.

Dedicated IP

Psst! Glad to see that you've read it all; This blog post is long enough, so here's a great resource that can help you decide whether to purchase a dedicated IP or not - 3.4 Shared vs. Dedicated IPs, M3AAWG Sender Best Common Practices v3.0, Feb 2015.

In short, if you're sending high volumes, hundreds of thousands of emails a month, then you should purchase a dedicated IP. I hate it when it all ends up with "it depends on your needs", but hey, that's true.

Final Words

I admit that it was tough and felt like there's a lot to memorize, but once you truly understand how each components works, it's pretty nice and very not intimidating as it was. So enjoy setting up your mailing DNS records. That would be all.