Subscribe

Archive | Articles

Googlebot Cannot Access CSS and JS on your WordPress Site?

In the past couple of days, many client sites have gotten notices from Google Webmaster Tools about being unable to access CSS and JS files.

The most widespread issue was blocking of wp-content, wp-includes and wp-admin in the robots.txt file. Googlebot is now smart enough to not crawl these items by default, so they can be removed.

The second issue we found was related to the Wordfence plugin. As of version 6.0.12 Wordfence includes a call to a dynamic script located in wp-admin, which is a default “Disallow” entry in the WordPress robots.txt file. Upgrading to Wordfence 6.0.14 or later will fix this issue.

Users of WPCmdCtrl are already covered and have the update ūüôā

Fixing Issues With Large WordPress Menus Not Saving

If you have a custom menu on your WordPress site, and you’ve found that after adding so many items, saving the menu will remove the last few items from the list. This will generally happen if you have many sub menu items.

The main culprit is the “max_input_vars” PHP setting. By default this is set to 1000, which limits your menu to ~75 items. I’ve seen some hosts set this to 500, which gives you ~35 items before things start getting chopped off.

If you have the ability to edit your site’s php.ini file just add or modify the following line:

max_input_vars = 5000

Enabling WordPress Rewrite Rules by Default With Virtualmin / Nginx

Virtualmin is a fantastic (and free!) alternative to cPanel. It gives you a nice web interface to easily add hosting for many sites on a single VPS or dedicated server.

By default, Virtualmin will setup Apache for your webserver, which is Ok, but not great. Thankfully, Nginx is available as a supported add-on. However, in the default setup, WordPress permalinks do not work correctly.

There are quite a few tutorials around on how to fix this, and they all amount to:

  1. Setup the Virtual host through the wizard
  2. Jump into Webmin
  3. Browse to the Nginx config screen
  4. Bring up the config file for the Virtual host you just created
  5. Manually paste in some lines of code and hope that you didn’t mess anything up.

I’d like to eliminate the list down to a single item.

This does involve some file editing on the command line, but it only happens once. After that, all of the hosts you add will be automatically setup for pretty permalinks.

Step 1:

Login as root, or sudo and edit the following file in your favorite editor:

/usr/share/webmin/virtualmin-nginx/virtual_feature.pl

Jump down to line 247, and you’ll see the following:

my $ploc = { 'name' => 'location',
                'words' => [ '~', '\.php$' ],
                'type' => 1,
                'members' => [
                { 'name' => 'try_files',
                    'words' => [ '$uri', '=404' ],
                },
                { 'name' => 'fastcgi_pass',
                    'words' => [ $port =~ /^\d+$/ ?
                                'localhost:'.$port :
                                'unix:'.$port ],
                },
                ],
            };
&save_directive($server, [ ], [ $ploc ]);

Right below that (before the “&flush_config_file_lines();” line), paste in the following:

my $wordpress = { 'name' => 'location',
                'words' => [ '/' ],
                'type' => 1,
                'members' => [
                { 'name' => 'try_files',
                    'words' => [ '$uri', '$uri/', '/index.php?$args' ],
                },
                ],
            };
&save_directive($server, [ ], [ $wordpress ]);

Save and exit.

Now try to create a new virtual host. If you’re not presented with any error messages congradulations! Every site you add now will automatically be setup for pretty WordPress permalinks!

Fixing Large Background Images in Mobile Safari on iPads

If you have a webpage with a very tall background image applied to your body element, Mobile Safari will try to show the entire image on the screen. This is different than desktop Safari and not what is expected to happen.

Thankfully, there is a one line change to fix this:

body{ -webkit-background-size: 2000px 1400px;}

Change the values to match the pixel dimensions of your image and viola! The background is displayed correctly.

Pinboard Backups in 1 Line

Alex Chan wrote a 1000 word blog post about a 35 line python script to archive his Pinboard bookmarks.

Here’s what I’ve been using since I’ve started using Pinboard, which is close to Alex’s solution but only one line:

curl https://ME:[email protected]/v1/posts/all?format=json -o ~/Dropbox/pinboard-backups/pinboard-`date +%y%m%d`.json

This does not create¬†separate¬†yearly folders for the backups, and the output is in JSON, but those are easy changes if that’s your preference.

I have this set to run daily with a “@daily” crontab schedule.

How To Optimize Your VPS for WordPress In 10 Minutes

WordPress get a lot of, somewhat deserved, bad press in regards to being able to stand up to large amounts of traffic. Especially on your run of the mill, everyday shared hosting.

Even on some VPS or dedicated hardware, people have had trouble keeping WordPress from falling over. There are some very simple fixes you can do to your server to make it almost bulletproof.

The below benchmarks were all run on a base 256mb Hudson Valley Host unmanaged VPS running Ubuntu 11.10 which is currently only $9/mo. This VPS has “burstable” memory to 512mb and 8 CPU cores.

I have used the following setup for all tests except where noted:

  • Base install of apache2 / MySQL (apt-get update && apt-get upgrade && apt-get install phpmyadmin mysql-server)
  • Apache and MySQL both running on the same server
  • Apache mod-rewrite enabled for fancy permalinks
  • Apache configured (by default) in /etc/apache2/apache2.conf:
    <IfModule mpm_prefork_module>
    StartServers 1
    MinSpareServers 1
    MaxSpareServers 5
    MaxClients 10
    MaxRequestsPerChild 0
    </IfModule>
  • /etc/apache2/sites-available/default edited to allow .htaccess overrides in /var/www
  • WordPress 3.4.1 installed into /var/www with the default Twenty Eleven theme and no plugins enabled, except when noted
  • Running siege on second server to benchmark (X is concurrency level of requests): siege -c X -t 1M http://wordpress.test/

The above command will send X number of simultaneous requests to the test URL for 1 minute.

Let’s get a simple baseline with 50 concurrent requests.

Transactions:        1778 hits

Availability:      100.00 %
Elapsed time:       59.25 secs
Data transferred:        3.91 MB
Response time:        1.16 secs
Transaction rate:       30.01 trans/sec
Throughput:        0.07 MB/sec
Concurrency:       34.77
Successful transactions:        1778
Failed transactions:           0
Longest transaction:        6.72
Shortest transaction:        0.26

As you can see from the “Transaction Rate” line, we were able to serve up 30.01 pages per second [1], which by itself is pretty nice. But we can probably crank up the concurrency more. Let’s try 100 requests per second.

Transactions:        1835 hits
Availability:      100.00 %
Elapsed time:       60.10 secs
Data transferred:        4.04 MB
Response time:        2.66 secs
Transaction rate:       30.53 trans/sec
Throughput:        0.07 MB/sec
Concurrency:       81.10
Successful transactions:        1835
Failed transactions:           0
Longest transaction:        7.46
Shortest transaction:        0.27

Ok, this is statistically¬†the same as the previous result. We’re going to say that 30 req/sec is our “all motor” limit.

Apache Pie Factory

This is where most sites that fall over are stuck. Basically Apache can only output 30 WordPress pages per second, if the server starts getting more than that, things get backed up.

And what happens when Apache gets backed up? It starts spinning up more processes to deal with the load. If you take a look at the snippet from the /etc/apache2/apache2.conf above, that is not the Ubuntu default. Hudson Valley Host was nice enough to put some sane defaults in there. This is what comes as default on most Ubuntu installs:

<IfModule mpm_prefork_module>
StartServers 5
MinSpareServers 5
MaxSpareServers 10
MaxClients 150
MaxRequestsPerChild 0
</IfModule>

The big issue is the MaxClients value. MaxClients is basically the number of processes Apache will spin up to deal with the incoming traffic. Each one of those individual processes uses memory. On my default install, each process was using about 16mb of memory, or a maximum total of 160mb. The above configuration will use up to 2400mb or over 2gb of RAM. As you can see, that’s a whole lot larger than the 256mb we have available on this server. It’s bigger than pretty much any affordable VPS and that’s just for Apache processes! If left like this, any decent amount of traffic will come along and kill your server. This is usually the case when people blame WordPress for not being able to handle traffic[2].

How can we fix this?

Apart from changing your¬†mpm_prefork_module settings so your server doesn’t kill itself (and you should do that as step 1), there are a few quick changes you can make to your server. We’ll start with the easiest (and least effective).

Install APC

Basically when Apache gets a request for a page on your blog, it fires up a PHP process, PHP takes the human-readable WordPress code and compiles it to something the server can then run and then that talks to MySQL, etc etc. This happens every time a page request is made, and this is horribly inefficient. What APC does is save the compiled version of the WordPress PHP code, removing that step from the process. Thankfully, that’s one of the slower parts of the whole chain. It’s enabled just by running this command (either via sudo or as root):

apt-get install php-apc && service apache2 restart

That’s it! The results from just this tweak, with the same 100 requests per second?

Transactions:        4461 hits
Availability:      100.00 %
Elapsed time:       59.23 secs
Data transferred:        9.81 MB
Response time:        0.81 secs
Transaction rate:       75.32 trans/sec
Throughput:        0.17 MB/sec
Concurrency:       61.19
Successful transactions:        4461
Failed transactions:           0
Longest transaction:        1.39
Shortest transaction:        0.18

Holy crap! That’s more than twice as fast! It’s like getting a second server for free! There is more speed available in there though…

W3 Total Cache

This is a free plugin available from the WordPress codex. Basically this plugin acts similar to APC except instead of saving the compiled version of the PHP script, it will save the HTML output of the page, bypassing the PHP to MySQL to PHP exchange. With Disk caching Apache will barely even talk to PHP and just serve up the cached HTML by itself.

There’s tons of options in here, but we’re only going to worry about the first page. I tend to disable the preview mode just so I can make my changes “live” right away for testing.

Note: Not all VPSs are created the same. Some have horrible disk I/O, some have over subscribed RAM that’s partially host swap, etc. Test the various settings to see what works best for you.

First off we’re going to look at caching everything to disk (Page cache: Disk Enhanced, DB: Disk, Object: Disk).

Transactions:        6825 hits
Availability:      100.00 %
Elapsed time:       59.83 secs
Data transferred:       15.97 MB
Response time:        0.36 secs
Transaction rate:      114.07 trans/sec
Throughput:        0.27 MB/sec
Concurrency:       41.03
Successful transactions:        6825
Failed transactions:           0
Longest transaction:        9.71

No bad! We can also stuff the cached files directly into APC (Page cache: APC, DB: APC, Object: APC). This is a bit more CPU intensive, but if your host has horrible disk I/O, this maybe your faster choice.

Transactions:        7174 hits
Availability:      100.00 %
Elapsed time:       59.43 secs
Data transferred:       16.69 MB
Response time:        0.31 secs
Transaction rate:      120.71 trans/sec
Throughput:        0.28 MB/sec
Concurrency:       37.64
Successful transactions:        7174
Failed transactions:           0
Longest transaction:        1.66
Shortest transaction:        0.14

Wow! 120.71 requests per second is about 10.5 million requests per day. That’s a lot of visitors.

Ok, what if we crank our concurrency up more? Let’s say 250.

Disk:

Transactions:        6332 hits
Availability:       99.95 %
Elapsed time:       59.78 secs
Data transferred:       14.79 MB
Response time:        1.78 secs
Transaction rate:      105.92 trans/sec
Throughput:        0.25 MB/sec
Concurrency:      188.71
Successful transactions:        6332
Failed transactions:           3
Longest transaction:       24.75
Shortest transaction:        0.30

APC:

Transactions:        5844 hits
Availability:       99.98 %
Elapsed time:       59.96 secs
Data transferred:       13.60 MB
Response time:        1.97 secs
Transaction rate:       97.46 trans/sec
Throughput:        0.23 MB/sec
Concurrency:      192.04
Successful transactions:        5844
Failed transactions:           1
Longest transaction:       28.02
Shortest transaction:        0.14

What the what? They’re LESS! Yeah, well Apache is starting to fall over here, and you can see that there’s some failed requests as well. That’s not good. Can we help poor Apache?

Nginx

Nginx is pretty awesome. It’s a web server just like Apache, but very very fast. WordPress.com uses it, so it should be good enough for us. Now, if you’re just starting out with a fresh VPS, you’re probably better off setting up Nginx + PHP-FPM. We’re not going to do that here, instead we’re going to use Nginx as a caching proxy for Apache. There’s a bunch of tutorials about setting Nginx + PHP-FPM, we’re just looking to get some more performance from our Apache install with the least amount of work with the ability to roll back easily if necessary.

Basically Nginx will sit in front of Apache. If it can serve up a file it will, if not, it will hand the request to Apache and then save the result for future requests.

Warning: We’re going to be editing some important files, make backups just in case!

/etc/apache2/ports.conf: Make these changes:

NameVirtualHost *:8080
Listen 8080

/etc/apache2/apache.conf:

KeepAlive Off

/etc/apache2/sites-available/default: Make these changes:

<VirtualHost *:8080>

Restart Apache:

service apache2 restart

Install Nginx and Apache mod_rpaf

apt-get install nginx libapache2-mod-rpaf && a2enmod rpaf

Replace /etc/nginx.conf with the following, changing your worker_processes to no more than the number of CPU cores you have:

user www-data;
worker_processes 8;

error_log /var/log/nginx/error.log;
pid /var/run/nginx.pid;

events {
worker_connections 1024;
}

http {
include /etc/nginx/mime.types;
default_type application/octet-stream;

access_log /var/log/nginx/access.log;
client_body_temp_path /var/lib/nginx/body 1 2;
gzip_buffers 32 8k;
sendfile on;
#tcp_nopush on;

#keepalive_timeout 0;
keepalive_timeout 65;
tcp_nodelay on;

gzip on;

gzip_comp_level 6;
gzip_http_version 1.0;
gzip_min_length 0;
gzip_types text/html text/css image/x-icon application/x-javascript application/javascript text/javascript application/atom+xml application/xml ;

include /etc/nginx/conf.d/*.conf;
include /etc/nginx/sites-enabled/*;
}

/etc/nginx/sites-available/default (replace yourserver.com with your actual server name)

server {
listen 80;
server_name yourserver.com;
access_log /var/log/nginx.access.log;
error_log /var/log/nginx_error.log debug;

location / {
root /var/www;
proxy_redirect off;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
if ($http_cookie ~* “comment_author_|wordpress_(?!test_cookie)|wp-postpass_”) {
set $do_not_cache 1;
}
proxy_cache_key “$scheme://$host$request_uri $do_not_cache”;
proxy_cache cache;
proxy_cache_valid 200 302 60m;
proxy_cache_valid 404 1m;
proxy_pass http://127.0.0.1:8080;
}

error_page 500 502 503 504 /50x.html;
location = /50x.html {
root /var/www/nginx-default;
}
}

Create /etc/nginx/conf.d/proxy.conf with the following:

proxy_redirect off;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
client_max_body_size 10m;
client_body_buffer_size 128k;
client_header_buffer_size 64k;
proxy_connect_timeout 90;
proxy_send_timeout 90;
proxy_read_timeout 90;
proxy_buffer_size 16k;
proxy_buffers 32 16k;
proxy_busy_buffers_size 64k;
proxy_cache_path /var/lib/nginx/cache levels=1:2 keys_zone=cache:8m max_size=1000m inactive=600m;

proxy_temp_path /var/lib/nginx/temp;

Create the following directories if they don’t exist and chown them for www-data:www-data:

/var/lib/nginx/cache
/var/lib/nginx/temp

mkdir /var/lib/nginx/cache
chown -R www-data:www-data /var/lib/nginx/cache
mkdir /var/lib/nginx/temp
chown -R www-data:www-data /var/lib/nginx/temp

Restart Apache and Nginx

service apache2 restart && service nginx restart

Check your site, you should see your WordPress homepage!

So what did all this work get us? The following is at the same 250 concurrency setting that choked Apache above, but with W3 Total Cache not enabled yet:

Transactions:       18447 hits
Availability:      100.00 %
Elapsed time:       59.37 secs
Data transferred:       40.60 MB
Response time:        0.24 secs
Transaction rate:      310.71 trans/sec
Throughput:        0.68 MB/sec
Concurrency:       75.54
Successful transactions:       18447
Failed transactions:           0
Longest transaction:       13.74
Shortest transaction:        0.14

Wow! 310.71 req/sec! That’s more than 2.5x our best Apache only result (@100 concurrency)! And it’s at almost 0 CPU usage as well.

How about 500 concurrency?

Transactions:       33119 hits
Availability:      100.00 %
Elapsed time:       60.23 secs
Data transferred:       72.87 MB
Response time:        0.35 secs
Transaction rate:      549.88 trans/sec
Throughput:        1.21 MB/sec
Concurrency:      193.97
Successful transactions:       33119
Failed transactions:           0
Longest transaction:       21.53
Shortest transaction:        0.14

Just shy of 550 requests per second! That works out to 1,425,288,960 requests per 30 day month.¬†Again at almost 0 CPU. Granted, we’ll run out of our allocated bandwidth long before the month is over, but we’ve tackled the issue of WordPress falling over.

  1. [1]These are just therotical WordPress pages, and each WordPress page, of course, has references to images, CSS and JS files. While this number does not exactly correlate to active visitors your site can serve, by offloading as much of your static content as possible by using CloudFlare, Amazon S3, or other CDNs, you can get closer to these numbers in practice.
  2. [2]Also, be aware that each plugin you use adds memory overhead. The theme you’re using may have additional memory overhead as well. Then there’s MySQL which will want to start using more memory as you add more posts and get more comments. But we’re not going to worry about that right now, you’ll see why.

Using Flurry, Tapjoy, Apsalar, etc with PhoneGap on iOS

PhoneGap is a great framework for building native iOS, Android, etc apps with just HTML, CSS & JS. However, there is a slight misconfiguration in the way PhoneGap sets up new Xcode projects, and this will not allow 3rd party analytic packages to work correctly.

Services such as Flurry and Tapjoy require you to add a bit of code into the “applicationDidFinishLaunching” section of your AppDelegate file (located in your Xcode project -> “Classes” folder -> “APPNAMEAppDelegate.m”). However, this code never actually gets executed. Below is how to setup your AppDelegate to work correctly.

First, you’ll need to include the framework’s files into your Xcode project. Follow the specific instructions from the provider as these may change from library to library and from version to version.

Second, open up the AppDelegate.m file in the “Classes” folder for your project. It’s easier to do this while in Xcode.

Add the import reference for the supplied .h file for your provider underneath the existing import references. I’ll be using Flurry Analytics as an example:

Old:

#import "APPNAMEAppDelegate.h"
#import "PhoneGapViewController.h"

New:

#import "APPNAMEAppDelegate.h"
#import "PhoneGapViewController.h"
#import "FlurryAPI.h"

Next you’ll be removing a block of code and replacing it with a new one.

Remove:

/**
* This is main kick off after the app inits, the views and Settings are setup here.
*/
- (void)applicationDidFinishLaunching:(UIApplication *)application {
[ super applicationDidFinishLaunching:application ];
}

Replace with:

/**
* This is main kick off after the app inits, the views and Settings are setup here.
*/
- (BOOL)application:(UIApplication *)application didFinishLaunchingWithOptions:(NSDictionary *)launchOptions {
//Custom Code Here
[FlurryAPI startSession:@"FLURRYSECRET"];
return [ super application:application didFinishLaunchingWithOptions:launchOptions ];
}

And that’s it! Command-S to save your file and you should be fully¬†integrated¬†with your 3rd party analytics package.

WordPress Caching Comparision – W3 Total Cache vs. WP Super Cache vs. WP Cache

Updated June 14. Added results for W3 Total Cache using Disk Enhanced caching only at the request of Frederick Townes, author of W3 Total Cache. I ran tests with both “gzip”¬†and “gzip and deflate” settings.

I’ve seen a lot of discussion lately about the various WordPress caching plugins. Here’s a quick comparison using Apache Bench. First some background on the test platform.

This is a 512mb VPS from prgmr.com

Fresh install of Ubuntu 10.04

NginX, PHP, MySQL installed via APT (PHP 5.3.2-1ubuntu4.2ppa3~lucid1 (FPM/FastCGI), NginX 0.7.65, MySQL (5.1.41-3ubuntu12.3))

WordPress 2.9.2, standard Kubrick theme, nginx Compatibility plug-in installed.

Plugin Versions:

WP-Cache 2.12
WP Super Cache 0.9.9
W3 Total Cache 0.8.5.2

The following command line was used. The -c and -n flags were modified during different runs as illustrated below.

ab -k -n XXX -c XXX http://testingserver/wordpress/2010/06/hello-world/
-n flag is the total number of requests (this is 1000 unless noted)
-c flag is for the number of concurrent requests
The Results

Click for full version
There was no -n 10000 -c 250 run for WordPress with no cache, as it would’ve taken very long and no further information would be gained.. Updated
Raw Data
50 Concurrent Requests

cache method time taken total transferred html transferred req/sec transfer rate

none 42.631 7025688 6786210 23.46 160.94

wp-cache 9.567 7157000 6872000 104.53 730.58

wp super cache (half on) 7.636 7184084 6905065 130.96 918.8

wp super cache (full) 7.411 7180000 6901000 134.93 946.07

w3 total cache (quick) 8.228 6427618 6039371 121.53 762.84

w3 total cache (APC for all) 8.384 6554037 6087793 119.28 763.44

w3 total cache (DE only) 7.787 7419504 7032408 128.41 930.43

w3 total cache (DE only gzip+deflate) 7.518 7393440 7008630 133.01 960.33

100 Concurrent Requests

cache method time taken total transferred html transferred req/sec transfer rate

none 41.277 7025688 6786210 24.23 166.22

wp-cache 7.942 7229988 6940603 125.91 888.98

wp super cache (half on) 4.711 7353500 7063600 212.28 1524.42

wp super cache (full) 4.598 7316282 7013904 217.49 1553.89

w3 total cache (quick) 4.619 6483320 6090890 216.5 1370.77

w3 total cache (APC for all) 5.326 6571413 6103337 187.76 1204.91

w3 total cache (DE only) 5.213 7467288 7076001 191.83 1398.88

w3 total cache (DE only gzip+deflate) 4.503 7458600 7068075 222.07 1617.52

250 Concurrent Requests

cache method time taken total transferred html transferred req/sec transfer rate

none 45.112 6694982 6459715 22.17 144.93

wp-cache 5.001 7370444 7067664 199.95 1439.19

wp super cache (half on) 4.782 7479736 7181764 209.11 1527.41

wp super cache (full) 3.436 7513840 7197120 291.02 2135.43

w3 total cache (quick) 5.341 6384178 5999741 187.24 1167.33

w3 total cache (APC for all) 7.17 6579256 6110264 139.46 896.05

w3 total cache (DE only) 4.648 7532448 7135446 215.14 1582.52

w3 total cache (DE only gzip+deflate) 5.265 7462944 7072038 189.93 1384.21

250 Concurrent Requests – 10000 Total Requests

cache method time taken total transferred html transferred req/sec transfer rate

none 384.545 64628053 62311067 26 164.12

wp-cache 34.552 71599548 68748013 289.42 2023.66

wp super cache (half on) 29.075 72047348 69241705 343.94 2419.91

wp super cache (full) 28.534 72337968 69365832 350.46 2475.77

w3 total cache (quick) 42.432 63564752 59751704 235.67 1462.91

w3 total cache (APC for all) 43.688 64968473 60366489 228.9 1452.25

w3 total cache (DE only gzip) 42.674 73695480 69868335 234.34 1686.48

w3 total cache (DE only gzip+deflate) 42.391 73612944 69793038 235.9 1695.81