This post will take you through setting up all the necessary DNS settings for your mail server and an additional server on a separate IP. The mail server will reside on a subdomain such as webmail.example.com while the main server will reside on example.com.
I highly recommend using DTDNS.com as they have provide an excellent service. I have a dynamic IP for one server and they automatically update the DNS records when the IP address changes. They also provide robust tools for managing advanced DNS settings including editing the file directly.
“‘A’ records, or Address records, are the most widely used entries in the DNS. These records define the host name to IP address mappings used by virtually every application on the Internet. You may have an unlimited number of ‘A’ records in a zone.” (DTDNS A Record Page).
This means that going to www.example.com or example.com will take you to the address listed in the table, for example your main website. Then if you go to mail.example.com it will take you to the other address listed in the table, for example your webmail login.
“‘MX’ records, or Mail Exchanger records, are used to define the hostname of the mail server that will handle incoming e-mail for a given host. For example, if someone sends e-mail to email@example.com then there will need to be an ‘MX’ record for domain.com. You may have an unlimited number of ‘MX’ records in a zone.” (DTDNS A Record Page).
The MX record defines the mail servers for the domain and you can have many mail servers located on different addresses. The preference settings determines which mail server will be used first.
SPF (Sender Policy Framework)
This is basically an anti-spam technique, it allows a receiving mail server to check that the sending mail server is an authorised host for sending mail from that domain. The SPF is set in the DNS records using a special txt entry.
IN TXT "v=spf1 ip4:126.96.36.199 ~all"
The IP Address is the IP Address of the mail server.
PTR Record, Reverse DNS Lookup
This is needed when hosting an email server as it adds validity to your mail server, some mail server will not accept email from an IP Address that does not have a PTR record set. This should be possible from your DNS provider or through your hosting provider. I use Digital Ocean to host my VPS on a static IP Address and they handle this, under the DNS tab is a section labelled PTR Records.
The way to set the PTR record if hosting a VPS through Digital Ocean is setting the hostname through the control panel on their website. You need to set the hostname to the hostname set in your DNS settings, in this example it will be mail.example.com
More information can be found in the links below.
Further Reading/To Dos
You may want to configure DKIM signing keys for your emails, this gives you the best chance of your emails getting through and not caught in anti-spam filters.