The most popular name server is ISCs BIND. It’s been the market leader in name servers since the very beginning and has proven to be a reliable workhorse with robust features and reliable updates. It also has some issues that are affecting Hawk Host.

The Issue

We initially started using cPanel with a DNS cluster for our primary name servers. As we grew and we started hosting a large number of websites off this cluster we began to notice a trend – when reloading BIND (which includes updates, a rndc reload, transfers, etc) or anything of that nature it will use a large amount of CPU – to the point that it crippled most processors. Another fun gotcha is when reloading zones it does not respond to requests. The result being one of our name servers would be unresponsive until a full reload took place.

This was clearly a large issue especially if there was a security release related with BIND or if a server crashed / needed to be rebooted for any reason. We weren’t the only ones to notice this. So we attempted to mitigate this issue…

Mitigating issues with BIND. Part 1.

The only graceful solution we came up with was to start deploying additional clusters and start using those while letting our initial cluster “relax” a bit. While this let us grow for another 1-2 years it was more of a kludge fix that wasn’t nearly as ideal. In addition to this as the additional clusters grew we noticed this wasn’t going to scale nicely.

We evaluated throwing more hardware at the issue which indeed helped but it still wasn’t ideal. We looked into an anycast sort of solution however it wasn’t economical nor easy with our current network.

Enter cPanel release 11.32 with MyDNS support.

Mitigating issues with BIND. Part 2.

While we loved the idea of using MyDNS as our primary DNS server we still weren’t terribly comfortable. It’s had a futile existence for sometime going into a long hiatus and not being maintained to being forked (MyDNS-NG, which cPanel is supporting). The development however is still a bit spotty for our liking.

This however did provide us with an easy, seamless way to get our DNS zones into MySQL thanks to cPanels conversion script. Now we just needed to find a way to use it with software we were comfortable with.

While looking into customizing PowerDNS queries using their gMySQL backend a colleague of ours noticed that PowerDNS actually had a MyDNS backend which allowed PowerDNS to query and understand the MyDNS database schema! Unfortunately this isn’t the “best” solution as PowerDNS has to do numerous queries per request however we felt much better with PowerDNS ability to handle a high stress environment as it’s been tested in large distributed deployments for sometime.

Using PowerDNS with cPanel and MyDNS

Now we get to the part of setting it up so PowerDNS can serve your DNS cluster. This will be a “barebones” approach however you’ll quickly realize the flexibility this offers if you’re familiar with PowerDNS.

Compiling and Installing PowerDNS

Here at Hawk Host we like to take a managed approach to our software and servers. This means instead of simply compiling software and being happy with it we build RPM’s for our servers. This ensures consistency across the board, easy management of the software and makes automation much easier.

It’s worth noting that the version that the MyDNS backend was introduced is in the 3.x branch of PowerDNS. The version most people use is from the splendid EPEL repositories which backports patches and concerns itself with stability over “bleeding edge”. This can be seen as the version of PowerDNS in EPEL is currently part of the 2.9.x branch. Luckily Mr. Monshouwer maintains RPM’s and repositories for the latest 3.1.x branch and libraries needed (Lua, boost). He also provides the SRPMS which will allow us to quickly and easily make our own RPM with the additional MyDNS backend.

Step 1 – Grabbing the dependencies and SRPM

The first thing you need to do is get the required dependencies installed. The easiest way to do this is to install Mr. Monshouwer’s repository as it provides everything that’s needed which isn’t included in the BASE CentOS repositories.

I’m assuming you have all of the proper dependencies for building (IE: “Development Tools”).

wget http://www.monshouwer.eu/download/3rd_party/pdns-server/el5/pdns-server.el5.repo -O /etc/yum.repos.d/pdns-server.el5.repo
yum clean all && yum -y install buildsys-macros lua-devel boost-devel zlib-devel sqlite-devel mysql-devel postgresql-devel openldap-devel
wget http://www.monshouwer.eu/download/3rd_party/pdns-server/el5/SRPMS/pdns-server-3.1-1.el5.MIND.src.rpm -O /tmp/pdns-server-3.1-1.el5.MIND.src.rpm
rpm -ivh /tmp/pdns-server-3.1-1.el5.MIND.src.rpm && rm -f /tmp/pdns-server-3.1-1.el5.MIND.src.rpm

All we’re doing here is installing the repository and the required libraries to build the RPM. The specfile provided creates several RPM’s with optional backends (hence OpenLDAP, SQLite, etc).

Step 2 – Modifying the specfile for PowerDNS

Now we must modify the specfile to include the MyDNS backend. Initially we were going to create additional RPM’s to be able to dynamically install it however due to a bug we discovered we realized we have to statically build it in currently.

Here is a patch you can run against the specfile accordingly. What it does is simply adds the MyDNS backend to the configure line and moves the gMySQL backend to be built as a static module as well (due to the aforementioned bug).

You can apply the patch by running the following in the same directory as the specfile (/usr/src/redhat/SPECS):

patch -p0 --ignore-whitespace < hawkhost-pdns.patch

Alternatively you can just copy and paste this for some automagic patching:

wget http://blog.hawkhost.com/wp-content/uploads/2012/08/hawkhost-pdns.patch_.txt -O /usr/src/redhat/SPECS/hawkhost-pdns.patch && cd /usr/src/redhat/SPECS && patch -p0 --ignore-whitespace < hawkhost-pdns.patch

Step 3 – Building and installing your PowerDNS RPM

You’re now ready to build your RPM. This is remarkably simple. All you have to do is run the following:

rpmbuild -ba pdns-server.spec

Keep in mind it’s highly recommended you setup a non-privileged user or use a staging server to build RPM’s for security reasons. A specfile can run shell commands and so forth.

You should now be able to install the recently created RPM and creating the powerdns user:

useradd -s /sbin/nologin powerdns
rpm -ivh /usr/src/redhat/RPMS/`uname -i`/pdns-server-3.1*.rpm

Step 4 – Converting your DNS to use MyDNS

This is also very easy thanks to cPanel making a splendid migration script. You can do this manually by following these instructions or use the WHM interface which involves a few clicks.

What this will do is convert your existing BIND zones to use the MyDNS format and insert them into a MySQL database.

Step 5 – Configuring PowerDNS and MySQL access

PowerDNS has plenty of options you may want to tweak however to get started you only have to modify a few things.

The first thing you’ll want to do is grant a user for PowerDNS to access the database. You can do this with the following MySQL query (mydns_XX being the MyDNS database name which is dependent on your hostname):

GRANT SELECT ON mydns_XX.* TO 'powerdns'@'127.0.0.1' IDENTIFIED BY 'my_password';

There are two things you’re going to want to do. The first is disable AXFR’s so your zones can’t be transferred. The other is enabling and configuring the MyDNS backend.

You’ll want to append the following to your /etc/powerdns/pdns.conf:

disable-axfr=yes
launch=mydns
mydns-host=127.0.0.1
mydns-user=powerdns
mydns-password=my_password
mydns-dbname=mydns_XX

Adjust the database and password accordingly. You may also want to bind to a specific address. To do that simply add the following:

local-address=1.2.3.4

Step 6 – Disable MyDNS, Start PowerDNS, test test test

You should now be able to turn off and disable MyDNS then startup PowerDNS. Running the following will make sure MyDNS doesn’t start on boot, turns it off while making sure PowerDNS is starting at boot and is turned on.

We’ll also be disabling chkservd from auto-restarting MyDNS and adding PowerDNS to be monitored by chkservd. In this example I’m assuming you’re running the PowerDNS daemon under the user “powerdns”.

It may be worth noting that PowerDNS spawns a parent process under root and forks another with the process of pdns_server-instance. I’ll be only monitoring the parent process however it may be wise to setup monitoring on both.

chkconfig mydns off
service mydns stop
sed -i '/named:1/d' /etc/chkserv.d/chkservd.conf
echo 'service[powerdns]=x,x,x,/etc/init.d/pdns-server restart,pdns_server,root' > /etc/chkserv.d/powerdns
echo 'powerdns:1' >> /etc/chkserv.d/chkservd.conf
echo '+' > /var/run/chkservd/powerdns
/scripts/restartsrv_chkservd
chkconfig pdns-server on
service pdns-server start

You’ll now want to verify that everything is working. Check your /var/log/messages to see if there is any startup errors. Utilize dig to query the server to ensure everything is resolving properly.

You should now be using PowerDNS!

Problems you may experience

Now there are several issues you may run into depending on your experience with Linux and or where you build your stuff. I’ll try to create a laundry list of issues and solutions as people bring them up.

• Installing the SRPM produces an error!
  • This is usually because you don’t have the rpmbuild packages and such installed so /usr/src/redhat and its children directories aren’t created. The best bet is to install the ‘Development Tools’ group using yum groupinstall ‘Development Tools’.

   

• Yum says I can’t install ‘Development Tools’!
  • Usually this is because you’re building on a cPanel based machine which adds several excludes to the yum.conf. You’ll need to remove perl* from the exclude= line temporarily and try again. Once finished installing re-add it.

   

• When building the RPM it complains about missing libmysqlclient.so!
  • This is because the MySQL-shared package cPanel uses has it placed in a non-standard directory. The cleanest solution to fix this is to symlink it accordingly: ln -s /usr/lib/libmysqlclient.so /usr/lib/mysql

In Conclusion

Ultimately this isn’t the most ideal setup however we’ve found it to be a great tradeoff so far. We run a slightly different setup however this is the general concept.

We hope in the future cPanel will support PowerDNS directly and utilize the more sane schema suggested in the gMySQL backend.

If you have any tips, questions or suggestions please leave a comment.