The most popular name server is ISCs BIND. It’s been the market leader in name servers since the very beginning and has proven to be a reliable workhorse with robust features and reliable updates. It also has some issues that are affecting Hawk Host.
We initially started using cPanel with a DNS cluster for our primary name servers. As we grew and we started hosting a large number of websites off this cluster we began to notice a trend – when reloading BIND (which includes updates, a rndc reload, transfers, etc) or anything of that nature it will use a large amount of CPU – to the point that it crippled most processors. Another fun gotcha is when reloading zones it does not respond to requests. The result being one of our name servers would be unresponsive until a full reload took place.
This was clearly a large issue especially if there was a security release related with BIND or if a server crashed / needed to be rebooted for any reason. We weren’t the only ones to notice this. So we attempted to mitigate this issue…
Mitigating issues with BIND. Part 1.
The only graceful solution we came up with was to start deploying additional clusters and start using those while letting our initial cluster “relax” a bit. While this let us grow for another 1-2 years it was more of a kludge fix that wasn’t nearly as ideal. In addition to this as the additional clusters grew we noticed this wasn’t going to scale nicely.
We evaluated throwing more hardware at the issue which indeed helped but it still wasn’t ideal. We looked into an anycast sort of solution however it wasn’t economical nor easy with our current network.
Enter cPanel release 11.32 with MyDNS support.
Mitigating issues with BIND. Part 2.
While we loved the idea of using MyDNS as our primary DNS server we still weren’t terribly comfortable. It’s had a futile existence for sometime going into a long hiatus and not being maintained to being forked (MyDNS-NG, which cPanel is supporting). The development however is still a bit spotty for our liking.
This however did provide us with an easy, seamless way to get our DNS zones into MySQL thanks to cPanels conversion script. Now we just needed to find a way to use it with software we were comfortable with.
While looking into customizing PowerDNS queries using their gMySQL backend a colleague of ours noticed that PowerDNS actually had a MyDNS backend which allowed PowerDNS to query and understand the MyDNS database schema! Unfortunately this isn’t the “best” solution as PowerDNS has to do numerous queries per request however we felt much better with PowerDNS ability to handle a high stress environment as it’s been tested in large distributed deployments for sometime.
Using PowerDNS with cPanel and MyDNS
Now we get to the part of setting it up so PowerDNS can serve your DNS cluster. This will be a “barebones” approach however you’ll quickly realize the flexibility this offers if you’re familiar with PowerDNS.
Compiling and Installing PowerDNS
Here at Hawk Host we like to take a managed approach to our software and servers. This means instead of simply compiling software and being happy with it we build RPM’s for our servers. This ensures consistency across the board, easy management of the software and makes automation much easier.
It’s worth noting that the version that the MyDNS backend was introduced is in the 3.x branch of PowerDNS. The version most people use is from the splendid EPEL repositories which backports patches and concerns itself with stability over “bleeding edge”. This can be seen as the version of PowerDNS in EPEL is currently part of the 2.9.x branch. Luckily Mr. Monshouwer maintains RPM’s and repositories for the latest 3.1.x branch and libraries needed (Lua, boost). He also provides the SRPMS which will allow us to quickly and easily make our own RPM with the additional MyDNS backend.
Step 1 – Grabbing the dependencies and SRPM
The first thing you need to do is get the required dependencies installed. The easiest way to do this is to install Mr. Monshouwer’s repository as it provides everything that’s needed which isn’t included in the BASE CentOS repositories.
I’m assuming you have all of the proper dependencies for building (IE: “Development Tools”).
wget http://www.monshouwer.eu/download/3rd_party/pdns-server/el5/pdns-server.el5.repo -O /etc/yum.repos.d/pdns-server.el5.repo yum clean all && yum -y install buildsys-macros lua-devel boost-devel zlib-devel sqlite-devel mysql-devel postgresql-devel openldap-devel wget http://www.monshouwer.eu/download/3rd_party/pdns-server/el5/SRPMS/pdns-server-3.1-1.el5.MIND.src.rpm -O /tmp/pdns-server-3.1-1.el5.MIND.src.rpm rpm -ivh /tmp/pdns-server-3.1-1.el5.MIND.src.rpm && rm -f /tmp/pdns-server-3.1-1.el5.MIND.src.rpm
All we’re doing here is installing the repository and the required libraries to build the RPM. The specfile provided creates several RPM’s with optional backends (hence OpenLDAP, SQLite, etc).
Step 2 – Modifying the specfile for PowerDNS
Now we must modify the specfile to include the MyDNS backend. Initially we were going to create additional RPM’s to be able to dynamically install it however due to a bug we discovered we realized we have to statically build it in currently.
Here is a patch you can run against the specfile accordingly. What it does is simply adds the MyDNS backend to the configure line and moves the gMySQL backend to be built as a static module as well (due to the aforementioned bug).
You can apply the patch by running the following in the same directory as the specfile (/usr/src/redhat/SPECS):
patch -p0 --ignore-whitespace < hawkhost-pdns.patch
Alternatively you can just copy and paste this for some automagic patching:
wget http://blog.hawkhost.com/wp-content/uploads/2012/08/hawkhost-pdns.patch_.txt -O /usr/src/redhat/SPECS/hawkhost-pdns.patch && cd /usr/src/redhat/SPECS && patch -p0 --ignore-whitespace < hawkhost-pdns.patch
Step 3 – Building and installing your PowerDNS RPM
You’re now ready to build your RPM. This is remarkably simple. All you have to do is run the following:
rpmbuild -ba pdns-server.spec
Keep in mind it’s highly recommended you setup a non-privileged user or use a staging server to build RPM’s for security reasons. A specfile can run shell commands and so forth.
You should now be able to install the recently created RPM and creating the powerdns user:
useradd -s /sbin/nologin powerdns rpm -ivh /usr/src/redhat/RPMS/`uname -i`/pdns-server-3.1*.rpm
Step 4 – Converting your DNS to use MyDNS
This is also very easy thanks to cPanel making a splendid migration script. You can do this manually by following these instructions or use the WHM interface which involves a few clicks.
What this will do is convert your existing BIND zones to use the MyDNS format and insert them into a MySQL database.
Step 5 – Configuring PowerDNS and MySQL access
PowerDNS has plenty of options you may want to tweak however to get started you only have to modify a few things.
The first thing you’ll want to do is grant a user for PowerDNS to access the database. You can do this with the following MySQL query (mydns_XX being the MyDNS database name which is dependent on your hostname):
GRANT SELECT ON mydns_XX.* TO 'powerdns'@'127.0.0.1' IDENTIFIED BY 'my_password';
There are two things you’re going to want to do. The first is disable AXFR’s so your zones can’t be transferred. The other is enabling and configuring the MyDNS backend.
You’ll want to append the following to your /etc/powerdns/pdns.conf:
disable-axfr=yes launch=mydns mydns-host=127.0.0.1 mydns-user=powerdns mydns-password=my_password mydns-dbname=mydns_XX
Adjust the database and password accordingly. You may also want to bind to a specific address. To do that simply add the following:
Step 6 – Disable MyDNS, Start PowerDNS, test test test
You should now be able to turn off and disable MyDNS then startup PowerDNS. Running the following will make sure MyDNS doesn’t start on boot, turns it off while making sure PowerDNS is starting at boot and is turned on.
We’ll also be disabling chkservd from auto-restarting MyDNS and adding PowerDNS to be monitored by chkservd. In this example I’m assuming you’re running the PowerDNS daemon under the user “powerdns”.
It may be worth noting that PowerDNS spawns a parent process under root and forks another with the process of pdns_server-instance. I’ll be only monitoring the parent process however it may be wise to setup monitoring on both.
chkconfig mydns off service mydns stop sed -i '/named:1/d' /etc/chkserv.d/chkservd.conf echo 'service[powerdns]=x,x,x,/etc/init.d/pdns-server restart,pdns_server,root' > /etc/chkserv.d/powerdns echo 'powerdns:1' >> /etc/chkserv.d/chkservd.conf echo '+' > /var/run/chkservd/powerdns /scripts/restartsrv_chkservd chkconfig pdns-server on service pdns-server start
You’ll now want to verify that everything is working. Check your /var/log/messages to see if there is any startup errors. Utilize dig to query the server to ensure everything is resolving properly.
You should now be using PowerDNS!
Problems you may experience
Now there are several issues you may run into depending on your experience with Linux and or where you build your stuff. I’ll try to create a laundry list of issues and solutions as people bring them up.
- Installing the SRPM produces an error!
- This is usually because you don’t have the rpmbuild packages and such installed so /usr/src/redhat and its children directories aren’t created. The best bet is to install the ‘Development Tools’ group using yum groupinstall ‘Development Tools’.
- Yum says I can’t install ‘Development Tools’!
- Usually this is because you’re building on a cPanel based machine which adds several excludes to the yum.conf. You’ll need to remove perl* from the exclude= line temporarily and try again. Once finished installing re-add it.
- When building the RPM it complains about missing libmysqlclient.so!
- This is because the MySQL-shared package cPanel uses has it placed in a non-standard directory. The cleanest solution to fix this is to symlink it accordingly: ln -s /usr/lib/libmysqlclient.so /usr/lib/mysql
Ultimately this isn’t the most ideal setup however we’ve found it to be a great tradeoff so far. We run a slightly different setup however this is the general concept.
We hope in the future cPanel will support PowerDNS directly and utilize the more sane schema suggested in the gMySQL backend.
If you have any tips, questions or suggestions please leave a comment.
Here’s a weird idea: use the gmysql schema but add a bunch of views/triggers to emulate the MyDNS schema towards writers. That way the overhead is on the writing side and not on the reading (PowerDNS) side.
I took the liberty of adding a link to this post to the bugtracker item, by the way. The issue is that we try to reuse code between modules and core, but there is no MySQL driver in core (and there really should not be, of course). The MySQL driver therefore lives with the gmysqlbackend. However, I’m sure we can figure out a way to make this more convenient for you. I’ll try to report back here if work happens on the ticket 🙂
That’s a very interesting and good idea – I’ll need to play with that a bit. Unfortunately that’s not my forte so I’ll likely try to consult it out.
Logically speaking that makes sense however it is sort of confusing when building – especially when you have the option of creating dynamic modules and having one fail simply because it’s not “self sufficient”.
It’s not a big hurdle currently – it just caused confusion initially :).
Thanks for the feedback – I’ll make sure to make another blog post to see if we can make the views/triggers work.
Pingback: A Hawk Host Summer of Fun | Hawk Host Blog
When I had the same problem with BIND I looked about and tried all of the usual suspects. NSD3 won all of my tests and now I have a huge chunk of spare RAM on my name servers.
Almost a year later, is this still running pretty well for you? How long have you guys been using it in production as a replacement for bind on cpanel servers?
We’ve been using PowerDNS on our cPanel servers for over a year now. As mentioned in the post our implementation is different than the tutorial but no issues. We’ve had a big increase in stability compared to bind.
Is there a reason why you are only giving a general idea versus exactly how you have yours setup? If there are any additional tweaks or gotchas, I would really like to know them before I take the plunge. Thanks for your time!
The reason we don’t outline our setup is because what we changed is geared towards serving 100,000+ dns zones and serving millions of dns queries daily. I don’t think there would be much demand for this portion of our setup. Anyone who does have this need will know how to do slave PowerDNS servers already. The portion outlined in the guide tackles the difficult problem which was having a way to use PowerDNS on a cPanel server.
I have had my cPanel + PowerDNS system running for over a year now, updated cPanel on the server by accident and it’s now broken :/ running into problems when adding a zone file via the cPanel API.
It’s somehow just broken everything, anything you can see that may have caused something like this?