WordPress Theme Performance

Posted on September 24, 2009 by Tony

When people are looking for a wordpress theme they very rarely are looking at the performance of it.  The person is looking at how pretty the design is and if it fits their site.  There is however a lot more to it then simple the design of the wordpress.  A wordpress theme can add a lot of extra CPU and memory usage depending on how it’s designed.

I’m seeing more and more where wordpress themes are not using style.css or at least .css for their style sheets.  More and more they are using style.php which in turn simply outputs the style sheet.  This sort of setup suddenly requires an extra PHP process every page load thus increased memory as well to compensate.  The fact it’s just going to output CSS still does not matter it’s really bad to be doing.  This is especially bad when a wordpress blog is running say wp-super-cache they’re probably trying to serve everything via html files.  Well that purpose is defeated when every page load PHP is loading up to serve their style sheet.  Now as to why more themes are doing this I’m not sure maybe it’s make them look more professional in the eyes of the users I have no idea.

Of course if you see style sheets being displayed via php you also get the same thing with java script.  All the js files become js.php with simply the javascript being output in the php file.  There is no actual using of PHP logic besides the print “js stuff”

There is one other problem I see quite a bit and that’s extra features included in the theme.  For example we’ve had several users use themes with phpthumb included.  The theme was using it in several places to make smaller images of portions.  Along with that they also did not configure it properly and some bots found it and started using the install to generate thumbnails for other web sites.
That’s just a few examples but I would advise anyone when installing a wordpress theme take a look at what it offers and what it’s doing.  If it’s using php files everywhere you might want to see if it’s actually necessary.  If they include extra features or include third party scripts you might want to check if they even set it up properly.  I would not rely on the theme developers on these sort of things.  I’ve seen some free themes not even doing relative paths properly resulting in serving images and such via their own demo of their script.  So I really would not trust the theme maker has done a great job with the other aspects of the theme.

Posted in Tips | 3 Comments

Emergency maintenance and why it’s necessary

Posted on September 17, 2009 by Cody

This is a short rant regarding emergency maintenance. Typically the only time there is unscheduled downtime (or very abruptly scheduled downtime) is because the issue is urgent for one reason or another. For the most part there are only two to three reasons why we may pull a machine offline abruptly:

  • Network maintenance (usually out of our control)
  • Security updates (a recently released exploit, fix, etc)
  • Hardware issue

Now for the most part the last one tends to be our achilles heel. As most of you know we run a RAID-10 setup on all of our servers for redundancy purposes in case of a drive failure – normally this isn’t an issue as all of our drives are hot swappable so we can simply replace the drive on-the-fly and have the array rebuild. When it becomes an issue is when the RAID card itself is reporting degraded drives across the board or the recently replaced drive is still showing as degraded (or in some cases completely dead). Now this puts us in a pickle – typically if this happens we have the RAID card firmware upgraded, or replace the RAID card as a whole (both in which require about 10-15 minutes of downtime). We can either run with no redundancy and post proper maintenance schedules or simply provide a notice of 30-60 minutes and pull the machine offline for a very short period. We typically opt for the latter instance since any data loss will simply balloon the issue into something catastrophic – and when playing with faulty RAID cards we much rather not play a game of russian roulette with our customers data.

Unfortunately as a consequence of our beliefs we occasionally have to pull a machine offline for a short period to ensure everything is running smooth. Data loss is serious and we believe the age old adage “An ounce of prevention is worth a pound of cure”. We’ve had a huge amount of drive issues here at HawkHost – up to several a month on various servers (the majority cause little or no downtime) and have had no data loss.

So if you find that a machine is pulled offline abruptly please take a moment and investigate why that may be the case. We’re very verbose and willing to give you an explanation! Unless there is an issue at the data center we never pull servers offline unless it’s absolutely necessary.

Posted in General | 4 Comments

cPanel x86_64 Reseller unable to Add Packages

Posted on September 14, 2009 by Tony

Using 64bit is really common now in fact most hosts are using this on their newer systems due to using more than 4GB of memory.  On cPanel there is a common problem that crops up with errors showing up after a while in WHM for resellers after there is quite a few accounts on the server.  cPanel sets the memory limit relatively low and on 64bit systems int’s and such are using more memory due to the 64bit system.  cPanel added a feature which allows you to set a maximum amount of memory used by a cPanel process before it is killed.  As a result in 64bit systems they’re more frequently hitting the modest 256MB limit while 32bit systems do not run into this issue as often.  The reseller user will receive the following error when adding a package:

[an error occurred while processing this directive]

The user will see this error over and over again when adding the package.  There is also an error in the cPanel error_log which I cannot remember at this point as we had a machine give this error a while back.  The thing is we know what causes it now so I did not bother checking the error_log of cPanel.  So all you need to do is go into tweak settings and find the following setting:

The maximum memory a cPanel process can use before it is killed off (in megabytes). Values less than 256 megabytes can not be specified. A value of “0″ will disable the memory limits.

We typically set this value at 768-1024 MB on our higher end systems (12GB ram total).

Do not feel bad if you did not know as I honestly did not until I made a ticket with cPanel support.  Even then at the time they needed to login to figure out what was causing it.  It’s really not a well documented problem but it seems to be coming up more often now.  You probably see the same thing with other applications and even the languages themselves.  For example PHP the memory limit of 32MB is really problematic on 64bit systems which is why we use 128MB.

Posted in Tips | Leave a comment

Why You Should Use WP-Super-Cache

Posted on September 13, 2009 by Tony

At Hawk Host we deal with CPU usage issues with accounts like every other host out there.  I’ve noticed an ever growing pattern of the problems always coming from WordPress blogs and almost always they have no caching what so ever.  When a web page is served without any caching in wordpress it will load up PHP then grab the data via MySQL.  These are both the slowest and most intensive portions of a web page.  The images, css files ect. are all served nearly instantly by the web server and it can serve a lot more of these static files per second then PHP pages.

This is where WP-Super-Cache comes into play what it does is when pages are served the first time it checks the caches and if they do not exist it then creates the cached file and serves it.  It has two options using rewrite rules to send users to .html versions if they’re available or to have PHP still handle it but load the PHP files instead.  How the HTML version works is the rewrite rules it inserts into the .htaccess file checks for file existence in the cache folder and if it’s there that’s where the file is served out of.  This is the ideal situation as no PHP being loaded at all means a server can handle a lot of traffic meaning even being on the frontpage of digg may very well be handled fairly easily.  There are some cases where users forget this portion so we always make sure to remind users to get the rewrite rules in.  In a lot of cases their blog goes from using a lot of resources under traffic to being a user we have no idea we’re even hosting.

So how do you know if the cache is working?  Well on each page if you view the HTML source you’ll see something like this:

<!– Dynamic page generated in 0.277 seconds. –>
<!– Cached page generated by WP-Super-Cache on 2009-09-13 14:22:03 –>

You should also make sure the cache page generated time has not changed on a refresh of the page seconds later (tells you it’s broken).  So if I refresh the page again and I have made no changes to the blog I should see this again:

<!– Cached page generated by WP-Super-Cache on 2009-09-13 14:22:03 –>

This tells me the cache is working and the page was served via a html file.

I hope this convinces everyone who is not running WP-Super-Cache to install it now and configure it!  Especially if you’re hosting with us because eventually if your site becomes popular I’m sure you’ll hear from us about installing it to reduce your usage so that your site can scale up much better.

I plan on posting some more WordPress related posts in the next few weeks regarding things I’ve noticed on blogs that decrease performance and increase resource usage.  Hopefully the tips help users who think they might not belong in a shared hosting environment to think otherwise after correcting these issues.

Posted in Tips | 3 Comments

Why We Upgrade Servers

Posted on September 10, 2009 by Tony

I’m sure lots of our customers wonder why do we do hardware upgrades once a year or at least every two years.  With the migration of our Skyline server to new hardware I figured it was a good time to explain why we do it and also how we do it with minimal service impact.

History Lesson

We’ve done several upgrades over the years I’d like to quickly run through just how many specifications we’ve had:

  • Dual Xeon 2.8ghz (2 CPU cores total) 2GB RAM using two 80GB PATA drives (one backup)
  • Dual Opteron 246 (2 CPU cores total) 2GB RAM using two 250GB SATA drives (one backup)
  • Xeon 3220 (4 CPU cores total) 4GB RAM using four 250GB SATA drives raid-10
  • Xeon 5430 (4 CPU cores total) 4GB RAM using four 250GB SATA drives raid-10
  • Dual Xeon 5430 (8 CPU cores total) 6GB RAM using four 300GB Raptor drives raid-10
  • Dual Xeon 5450 (8 CPU cores total) 12GB RAM using four 300GB 15K SAS drives raid-10
  • Dual Xeon 5520 (8 CPU cores total) 12GB RAM using four 300GB 15K SAS drives raid-10

For the most part specifications jumped quite a bit each time.  The only one that did not was the Xeon 3220 to Xeon 5430.  The newer CPU’s were a newer Xeon model but overall it was not a huge jump.  A big reason for it at the time was to get rid of most our 32bit machines and add the capability later on for extra CPU’s and expanded room for memory.  In the end though we ended up just using newer machines instead.  Some of the older moves were also when we used different datacenters so the migrations were not as seamless as they are these days.

I’d like to point our some key moves in this progress.  We used to run backups on the drives of the servers.  This was pretty much how hosting worked with any company as raid was really expensive to do even if you owned the equipment it was not common.  Now raid is standard in a lot of cases raid-10 for reliability and performance.  With that use of raid meant we added a dedicated backup server as well which in itself was an upgrade.  Eventually the upgrade machine became a R1Soft system rather than rsync backups but that’s been told before.

As for the Dual 5450 and Dual 5520 they are mostly the same.  We’ll be using both depending on the VLAN the server is on.  We ran into an issue with Nehelem’s on our main Dallas VLAN which houses all our web servers.  They require an extra network port to run the IPMI system we use and unfortunately the VLAN we’re on all it’s racks available do not have this extra port available.  So for some of our upgrades we’ll see 5450 and others we’ll see 5520 and most likely all new machines we’d use 5520′s.  This is probably the first time we’ve ever had a mix and match of different CPU’s.  Which for web hosting is not a huge deal to begin with.

Why do we do it?

We’re obviously making profit on each server so why do we do it exactly?  Well the obvious one is the requirements of web sites grow.  Even the small sites use more PHP and MySQL than ever before.  The more features added to wordpress or whatever script is does take it’s toll over time depending on what it adds.  The other thing is simply a newer machine gives us greater capacity as well as improved performance.  Less servers means less work for us so the old Dual Opterons hold 1/8 of what we can put on a newer machine we have now.  That might even be an understatement.  With the increased capacity it does not mean we cram them full by any means.  It just means we had room left over before and we will again just more general use capacity available as well as more burst capacity as well.  So in the end the users on the server do win they get a better machine and it’s not just more users on that machine.

We also do it just because as we grow our buying power increases.  We do not own our servers we rent them thus as we rent more machines our cost goes down per machine.  The pricing we can negotiate is much better than it was even a year ago.  We can say to our provider look we have x amount of machines we’d like to get a good deal and we’ll use this configuration for a while.  This is a big reason we do it we can leverage our buying power to get higher end machines.

How Do we do it?

If you’ve been on a server migrated in oh the past two years you’ve noticed probably that we can do it exceptionally well.  When we moved to SoftLayer a big advantage was we no longer had to deal with routing of IP’s to each server.  We now route them to our VLAN which means as long as we have space on it we can route IP’s as we please.  We do not just do this for migration purposes either two machines could share the same IP block.  So in quite a few cases now the IP beside the one your site is on may not be on the same server.  New IP ranges obviously that may not be the case but as older servers have customers leave and if they’re on a dedicated IP the IP is thrown back into the pool for the entire VLAN.

Using the routing advantage we have we can find out which sites are on a specific IP and transfer their data to the new server.  Once all the sites on the IP are switched over to the new machine we use arp to tell the network side instantly the IP has switched locations.  This means no DNS resolution down time at all.  We have our users in various IP’s on each server to reduce other issues so this helps with making a seamless migration as well.  The fact is the majority of users do not know that we’ve even switched their site to a new server.  We inform people but most do not read it or care as long as it works.  That’s how effective it is a user does not even realize they’ve been switched over.

It’s not a perfect system though we do run into issues.  The big one is the fact that while data is moving across a site could have changes done to it.  This is mostly a problem with a discussion forum which is writing data to mysql.  Once switched over it’s not really an option to migrate the data again.  The same problem could come up data being lost as we re-sync the mysql data again.  The other thing that has come up the odd time is we cause a routing issue for an IP.  It’s basically a human error where we re-route the IP and forget to check that it worked fine.  It’s a rare occurrence for us to happen but it has.  We’ve learned a lot since we started doing it though that it’s no longer the huge worry it once was due to experience of doing it and that the issue could come up so we double check it now.

Conclusion

Well that’s summarizes why we do what we do and how we do it so well.  It helps us by adding capacity while not having to manage more servers.  It puts users on newer hardware and being put on a server with a lot more burst room than previously.  We do this all with the majority of users not having an issue what so ever.

Posted in General | 7 Comments

Catch All Email

Posted on August 28, 2009 by Tony

The catch all email was a great thing back in the day it was neat you could retrieve mail for sales@domain.com billing@domain.com ect. ect. all through one simple email address.  A lot of users used it without issue and it solved having to check various emails they could just tell users to email random addresses and it would arrive at it’s spot.  Fast forward to now and the large amounts of mail received by pretty much any domain on the internet and this once cool feature is now a major headache for us as well as any other hosting provider.

With the rise of spam and users no longer displaying their emails on sites the spam bots have become much smarter.  It’s no longer use email addresses found on web sites and only use them.  The spam bots now send mail to jimmy@domain.com, johnny@domain.com, susan@domain.com ect. ect.  As a result of this a simple spam email could become a mail bomb of thousands of emails hitting a single email account thus putting extreme strain on the server as well as quickly filling a users mail box.  So a lot of users stopped using this feature in favor of forwarders for the specific email accounts so it’s been less of a problem.  Of course some users still use this feature until they get hit by mail bomb and they end up switching out as it’s impossible to manage their own mail anymore.

Even with this no longer being a preferred feature of users we still run into problems with people having the default/catch all set to their accounts username.  We by default have this set to fail rather than catch the mail unfortunately by cPanel last I checked the default was catch-all.  The majority of web hosts do not do a great job with managing their servers so they have it set to catch all.  When we have to restore a users account from another server we inherit the ugly catch all that was setup.  So we get to deal with the spam and mail the user receives to any possible email address to their domain.

Now you might be saying well doesn’t spam help prevent all this mail?  It does but it cannot stop it all and it also slows down a server.  The mail portion on a server is one of the most intensive portions due to just the volume of mail coming in every day.  This is why a lot of providers have suggested to their users use services like google apps or have outside mail servers.  They are attempting to off load one of the most expensive things on their servers which is mail.  Now add in the fact the mail includes spam from every possible email address for a domain and it becomes much more of a problem.  So us using spam blocking techniques do not solve the issues they just help the user while we still need to process the mail going through our spam checker system then if the spam count is high enough getting thrown away.  Also spam blocking systems do not come near effective at blocking everything without blocking a lot of legitimate mail in the process.

All the CPU usage from the mail does not factor in the pure amount of mail you will receive when you have a default mail address set to say the username of your account.  We’ve had users move from another host and it’s enabled and they do not check the default email address.  A few years later and they’ve hit their quota as their default mail address is 2GB of mail or more.  So it can fill a users quota up pretty easily if left unchecked for a long time which is usually the case since the user is not aware it’s even enabled.

So the only solution here is to say no to the catch all email feature.  If you’ve never checked it out in cPanel I suggest visiting the “Default Address” feature in cPanel and making sure it is set to “Discard with error to sender (at SMTP time)”.  A lot of hosts do not have this set so if you came from another host it’s probably not set to fail on a non existent email address.

Posted in Tips | Leave a comment

The Maintenance Window

Posted on August 22, 2009 by Tony

The day started out like every other one with me waking up and starting my walk to the computer while half asleep.  I used to go shower and such like I was going to an office but after a while I figured it be best if I walk to the home office in my boxers to see what happened while I was gone.  I check out our support system just a few tickets but nothing that required my attention.  I checked my email and not a whole lot of mail just a bunch of orders in and the usual log of all the credit cards charges from our batch run.  It was looking like a great day not to busy some orders to check out but other than that great.

I always make sure to SSH into the machines a sort of habit of mine from back when we had a few.  Thankfully with SecureCRT it’s just select a few folders and I’m logged into all the machines.  After doing my usual checks of everything else I heard a beep come from SecureCRT and the first thing that came to mind is oh no something must have broke.  I start checking the servers and I get to Pluto the machine that has never had anything really happen to it.  It’s has hard drives fail and be replaced without any down time and it’s never had any issues with any of it’s hardware.  About the only issue it ever had was when a UPS blew in the server room it was in knocking it off line as well as some of our other machines.  The error was about the memory and it claimed to be a fatal error of some kind.  I did not like the sound of it yet the machine was still functioning as it should so that meant I needed to google and use my resources available to me.

The first thing I do is ask Cody what the heck does the message mean?  While I’m waiting for him to respond back I load up google and search and I find out that usually it suggests the memory is on it’s way out.  Cody finally messages me back with make a ticket with SoftLayer they’ll know what the heck that means.  Just as google suggested SoftLayer quickly determined this almost always means the memory on the machine needs replaced to avoid a major headache later on.  This meant a maintenance window needed to be scheduled for the server.  As I mentioned early I was in my half asleep state so I paste Cody our options and he says 1-4 is good we should do it then.  Being the half asleep person that I am I updated our ticket with that time.  In the past we’d do it ASAP or schedule something out of the suggested window times as they tend to be bad for us.  A few hours later after I had already posted notice on our forums I had an oh crap moment of did I just really schedule a window for 1:00AM CDT to 4:00AM CDT window?  At this point I was awake so that’s how I noticed and I realized that there was no way out of this we had informed everyone necessary already.  Since I was the fool to schedule it I’d be the sucker to have a Friday night / Saturday morning maintenance window to deal with since someone had to be there in case something bad happened.

My day went on as normal doing some work then calling it a day.  I had pizza for dinner and watched some family guy then two and half men so the usual suspects while I eat then decide what I’m going to do tonight.  I realized there wasn’t a whole lot I could do but the maintenance window was hours away so I decided to watch a movie to at least pass some time.  The movie finished and I realized I was still a long ways away from the maintenance window starting and I thought to myself what in the world am I going to do.  I went back to watching tv and ended up watching that’s 70s show and south park two shows I never really watch except they were the only decent shows on.  The window was still a ways away and I pulled out the xbox 360 and figured the window starts soon I better not play any game I won’t want to give up on in a bit so I can be on the computer.  I played a few games of Geometry wars to pass the time and finally it was 1:00 AM CDT which meant the window may start.

Unfortunately for me the window is 3 hours long so it ended up where it had not actually started yet as the machine was not off line.  Cody a few days earlier linked me to a the daily wtf post that he thought was funny.  I only really read posts on the site while I was working in an office and got bored from time to time.  I spent about an hour and a half reading all the posts on the site in the past three months as I got that bored.  I also made sure to post classics like The Source Control Shingle on our twitter while I was bored.  I also made sure to send Cody an email about how I hate maintenance windows especially ones this late and of course linked him to a post on the daily wtf he might like NPR Is Reading My Email, Just Fix It!, & More Support Stories mentioning the just fix it one that reminded me of some of the support tickets we get from time to time.  So after the hour and a half I got notice the Pluto server was finally going off line and I thought to myself finally!  I jumped on live chat because I was still bored and figured I’ll talk to some customers who come on complaining about the server being off line as everyone always does.  Usually when I try to man the live chat there is always someone who comes on who will sit there for several hours asking questions or frankly being an annoyance to me asking about if their web site looks like or what am I up to or something totally out of the scope of anything we do.  To my surprise this was not the case maybe it had something to do with the fact it was now 2:30 AM CDT and everyone was sleeping.  About the only customers we had who would be complaining or coming on would be from Asia.  Sure enough there were a few with their broken English and what not they asked what’s going on I linked them to the post and they posted some more rambled english about how Hawk Host rocks or at least I hope that’s what they meant.

As the window went on I resorted back to reading and doing basically nothing as I must have talked to anyone who cared about the server being down at 2:30AM CDT already.  I watched Jon Stewart Grills ‘Death Panel’ Originator (Canadian link) which provided me a few minutes of amusement.  The machine finally came back up a few tickets later and chats and everyone was happy the server was back online.  The best part there were no longer memory error messages being printed to console so it was a victory.  So in the end I waited around for nothing bad to happen but we always have someone to be there in case something bad does happen.

What I’ve learned about maintenance windows is don’t be a fool doing a late night one when you have nothing to do at such a late time.  If I was a midnight shift guy I might have found something to do but I was not.  So I basically had nothing planned to do and waited around then waited some more for the window to complete.  In the future I’ll make sure to have someone who actually is around during the late hours to be the guy who sits around waiting for nothing bad to happen.  Or at least schedule a more reasonable time that is late but not so late that it’s agony waiting for it to happen.  Also in the case of any window have something to do because we’re not the ones messing with the hardware so it becomes pretty boring pretty fast.

Well that’s the post I hope everyone enjoys the story and looking at it now I realized it seemed much better when I was half asleep thinking maybe it was a good idea to talk about my maintenance window adventure.  In hindsight though it’s not as exciting now that I’m awake and not going insane over sitting around waiting.

Posted in Random | 1 Comment

Foreign Transaction Fee

Posted on August 12, 2009 by Tony

The foreign transaction fee has come up a bit in the past year and I figured I’d rant about how ridiculous this fee is.  We’re a Canadian corporation and as such we have a Canadian merchant account.  For the most part we never run into issues with this and no extra fee’s for our users or anything of that nature.  There is however a few select banks in the United States who have now decided to charge their customers a 3% fee and call it a foreign transaction fee.  The best part about this fee is it shows up as if we charged it in a lot of cases when it has nothing to do with us.  This makes us look pretty bad to the few customers we have who’s banks have decided to add fee’s other banks are not adding.  Some users have suggested to us why not get a merchant account in the United States.  There are several reasons the first being we’re not incorporated in the Unites States making such a task a very difficult and if we did it would become a very costly one for us.  The second part is it’s not just the United States who get these fee’s we’ve heard users in the UK get charged a foreign transaction fee when purchasing items from Canada and the United States.  So we’d probably need a merchant account in every single country for our customers to avoid such a fee being charged by a select few banks in each country.

So a pretty big annoyance but it does not surprise me a whole lot.  We have a US credit card with our Canadian bank and we’re almost exclusively using online banking which is great to pay off our credit cards and check the balance of our accounts.  Now the strangest thing we cannot pay off our US credit card online even when we have a US currency bank account.  We have to call or go to the bank to pay as our online banking will not pay it online.  It’s very strange considering we can move funds between currencies and it takes a teller seconds to pay off our card.  So it feels like it’s some legacy policy before online banking was possible and every time we call to pay it or go to a teller it feels like we’re being an annoyance and we should be doing it online or through a ATM.

So banks are just strange in too many ways and I guess no way to avoid them.  If you’re one of the unlucky few getting charged a foreign transaction fee best to just use Paypal to pay us.  We have no way of controlling that foreign transaction fee at least with Paypal it may not if your card is charged there just yet…

Posted in General | 10 Comments

New WordPress Release

Posted on August 12, 2009 by Cody

There is a new release of WordPress that fixes a issue that allows an attacker to reset the first account in the database (usually the admin account). It doesn’t allow remote access, though can be fairly obnoxious.

More Information:

http://wordpress.org/development/2009/08/2-8-4-security-release/
http://core.trac.wordpress.org/changeset/11798

Posted in General | Leave a comment

Zend Framework 1.9 Released

Posted on August 3, 2009 by Tony

If you missed it on Friday Zend Framework 1.9 was released which as always contains new features and bug fixes.  I’d say the biggest thing would be the support for PHP 5.3 which was released last month.  We have no plans on supporting it any time soon on our servers but it’s nice to see support for it in Zend Framework which is a popular framework used on our servers.  It also of course maintains compatibility with PHP 5.2.4 and up so you can use it at Hawk Host.  We’ve already been working on upgrade all our PHP scripts that use ZF to 1.9 so we get all the fixes and new features.

Here’s a list of new features and changes

Continue reading →

Posted in General | 1 Comment