Archive by Author

Google Calendar v3 API Gotchas

We use Google Calendar for quite a few of our clients. They get a nice interface for managing their events, and we can leverage the API to display the content in various places. Both by using our own developed tools and by using plugins like Google Calendar Events.

Just to cover all of the bases here, on November 17th, 2014, Google turned off parts of the v1 and v2 Calendar API and required access through v3. Many plugins were not updated in time and calendars went away only showing cryptic error messages. Even after the updates of some of the plugins and creation of special API keys, some sites were still unable to show their calendars.

This is the solution we had for our particular problem.

When you create a new “Server” Google API key for public access, you need to supply a list of IP address that are allowed to access the API with your key. Here’s the gotcha, if your hosting provider is IPv6 enabled, you must include your server’s IPv6 address as well as the IPv4.

That’s all well and good if you’re running your own dedicated server or VPS, but what if you’re on a shared host and they don’t list the IPv6 address of your server? We did a silly hack and popped the following code into a .php file and uploaded it to the sites that were still having problems:

<?php echo file_get_contents('http://whatismyipv6.com/'); ?>

That will give you the IPv6 address of your web server. Just take that and paste it into your Google API console.

Delete the file when you’re done of course.

Announcing WPCmdCtrl

WPCmdCtrl (WP Command and Control) is a new service from Soju Web Development designed to help you easily manage multiple WordPress installs.

It was initially developed as an in-house tool to manage all of our client sites, but as word of it leaked out more and more contacts started asking about being able to use it for their own businesses. Today we are proud to announce that WPCmdCtrl is ready for open signups.

We realized that applying WordPress core and plugin updates while extremely important, was a huge time sink. So we built a way to automatically perform them.

We’ve had clients with FTP access accidentally, and in some cases not so accidentally, delete their uploads folder along with all of the “backups” that were stored there. Automated off-site backups immediately became a major priority. So we added those in.

Nothing is worse than having a client call and ask why their site is just a white screen, and has been for the past week! The next worse thing is to track down that their hosting company silently updated PHP and broke some plugin. We needed to know about these thing ASAP. So we built a way to track that, automatically.

Site response time is a major factor in both SERP rankings and reducing bounce rate. What happens when two random plugins cause a conflict to add 2 seconds to a site’s load time? Are a group of sites on a hosting node with a bunch of noisy neighbors that bring your sites to a crawl during prime time? We needed a way to track these things, so we built it.

What if one of your client’s sites gets hacked and is redirecting off only traffic from Google? You look at the site and everything is fine but the traffic is gone. Or even worse, all of the visitors are getting a “This site is unsafe” warning screen. How can you keep up with 15, 30, 250+ sites, every day? We covered that as well.

WPCmdCtrl is also designed to be completely hands-free. Once you setup your account, you just install a single plugin one time to each of your sites. The plugin will automatically add the site to your account and start doing it’s magic. If anything goes wrong, you’ll get an email (or text message or phone call if you’d like) letting you know exactly what site has the issue. And in the event of  a catastrophe, we’ll have a complete backup of your site and database stored in multiple data centers around the world so you can bring the site up again right away.

You can signup for a risk-free 14 day free trial at wpcmdctrl.com

Fixing Issues With Large WordPress Menus Not Saving

If you have a custom menu on your WordPress site, and you’ve found that after adding so many items, saving the menu will remove the last few items from the list. This will generally happen if you have many sub menu items.

The main culprit is the “max_input_vars” PHP setting. By default this is set to 1000, which limits your menu to ~75 items. I’ve seen some hosts set this to 500, which gives you ~35 items before things start getting chopped off.

If you have the ability to edit your site’s php.ini file just add or modify the following line:

max_input_vars = 5000

Enabling WordPress Rewrite Rules by Default With Virtualmin / Nginx

Virtualmin is a fantastic (and free!) alternative to cPanel. It gives you a nice web interface to easily add hosting for many sites on a single VPS or dedicated server.

By default, Virtualmin will setup Apache for your webserver, which is Ok, but not great. Thankfully, Nginx is available as a supported add-on. However, in the default setup, WordPress permalinks do not work correctly.

There are quite a few tutorials around on how to fix this, and they all amount to:

  1. Setup the Virtual host through the wizard
  2. Jump into Webmin
  3. Browse to the Nginx config screen
  4. Bring up the config file for the Virtual host you just created
  5. Manually paste in some lines of code and hope that you didn’t mess anything up.

I’d like to eliminate the list down to a single item.

This does involve some file editing on the command line, but it only happens once. After that, all of the hosts you add will be automatically setup for pretty permalinks.

Step 1:

Login as root, or sudo and edit the following file in your favorite editor:

/usr/share/webmin/virtualmin-nginx/virtual_feature.pl

Jump down to line 247, and you’ll see the following:

my $ploc = { 'name' => 'location',
                'words' => [ '~', '\.php$' ],
                'type' => 1,
                'members' => [
                { 'name' => 'try_files',
                    'words' => [ '$uri', '=404' ],
                },
                { 'name' => 'fastcgi_pass',
                    'words' => [ $port =~ /^\d+$/ ?
                                'localhost:'.$port :
                                'unix:'.$port ],
                },
                ],
            };
&save_directive($server, [ ], [ $ploc ]);

Right below that (before the “&flush_config_file_lines();” line), paste in the following:

my $wordpress = { 'name' => 'location',
                'words' => [ '/' ],
                'type' => 1,
                'members' => [
                { 'name' => 'try_files',
                    'words' => [ '$uri', '$uri/', '/index.php?$args' ],
                },
                ],
            };
&save_directive($server, [ ], [ $wordpress ]);

Save and exit.

Now try to create a new virtual host. If you’re not presented with any error messages congradulations! Every site you add now will automatically be setup for pretty WordPress permalinks!

Fixing Large Background Images in Mobile Safari on iPads

If you have a webpage with a very tall background image applied to your body element, Mobile Safari will try to show the entire image on the screen. This is different than desktop Safari and not what is expected to happen.

Thankfully, there is a one line change to fix this:

body{ -webkit-background-size: 2000px 1400px;}

Change the values to match the pixel dimensions of your image and viola! The background is displayed correctly.

Pinboard Backups in 1 Line

Alex Chan wrote a 1000 word blog post about a 35 line python script to archive his Pinboard bookmarks.

Here’s what I’ve been using since I’ve started using Pinboard, which is close to Alex’s solution but only one line:

curl https://ME:[email protected]/v1/posts/all?format=json -o ~/Dropbox/pinboard-backups/pinboard-`date +%y%m%d`.json

This does not create separate yearly folders for the backups, and the output is in JSON, but those are easy changes if that’s your preference.

I have this set to run daily with a “@daily” crontab schedule.

HackVT 2012 – Local Ingredient Search

We participated in the HackVT 24-Hour Hackathon in Winooski this weekend.

Although we did not get to complete the event due to a combination of prior commitments, injury (!!!) and a long wait for approval for a supplied API, we did get a nice, we think anyways, demo up.

The app as it stands uses HTML5 geo-location to find the closest Vermont suppliers of a particular ingredient to your current location. If you are not currently in Vermont, this will be somewhat useless, so please feel free to use any Vermont city in the search field to update the map and the listings.

If work had not stopped at 5am, there would be integration with the Yummly API along with a shopping list and a “traveling salesman” solution to track down your ingredients.

Technologies used:

Datasets / APIs:

How To Optimize Your VPS for WordPress In 10 Minutes

WordPress get a lot of, somewhat deserved, bad press in regards to being able to stand up to large amounts of traffic. Especially on your run of the mill, everyday shared hosting.

Even on some VPS or dedicated hardware, people have had trouble keeping WordPress from falling over. There are some very simple fixes you can do to your server to make it almost bulletproof.

The below benchmarks were all run on a base 256mb Hudson Valley Host unmanaged VPS running Ubuntu 11.10 which is currently only $9/mo. This VPS has “burstable” memory to 512mb and 8 CPU cores.

I have used the following setup for all tests except where noted:

  • Base install of apache2 / MySQL (apt-get update && apt-get upgrade && apt-get install phpmyadmin mysql-server)
  • Apache and MySQL both running on the same server
  • Apache mod-rewrite enabled for fancy permalinks
  • Apache configured (by default) in /etc/apache2/apache2.conf:
    <IfModule mpm_prefork_module>
    StartServers 1
    MinSpareServers 1
    MaxSpareServers 5
    MaxClients 10
    MaxRequestsPerChild 0
    </IfModule>
  • /etc/apache2/sites-available/default edited to allow .htaccess overrides in /var/www
  • WordPress 3.4.1 installed into /var/www with the default Twenty Eleven theme and no plugins enabled, except when noted
  • Running siege on second server to benchmark (X is concurrency level of requests): siege -c X -t 1M http://wordpress.test/

The above command will send X number of simultaneous requests to the test URL for 1 minute.

Let’s get a simple baseline with 50 concurrent requests.

Transactions:        1778 hits

Availability:      100.00 %
Elapsed time:       59.25 secs
Data transferred:        3.91 MB
Response time:        1.16 secs
Transaction rate:       30.01 trans/sec
Throughput:        0.07 MB/sec
Concurrency:       34.77
Successful transactions:        1778
Failed transactions:           0
Longest transaction:        6.72
Shortest transaction:        0.26

As you can see from the “Transaction Rate” line, we were able to serve up 30.01 pages per second [1], which by itself is pretty nice. But we can probably crank up the concurrency more. Let’s try 100 requests per second.

Transactions:        1835 hits
Availability:      100.00 %
Elapsed time:       60.10 secs
Data transferred:        4.04 MB
Response time:        2.66 secs
Transaction rate:       30.53 trans/sec
Throughput:        0.07 MB/sec
Concurrency:       81.10
Successful transactions:        1835
Failed transactions:           0
Longest transaction:        7.46
Shortest transaction:        0.27

Ok, this is statistically the same as the previous result. We’re going to say that 30 req/sec is our “all motor” limit.

Apache Pie Factory

This is where most sites that fall over are stuck. Basically Apache can only output 30 WordPress pages per second, if the server starts getting more than that, things get backed up.

And what happens when Apache gets backed up? It starts spinning up more processes to deal with the load. If you take a look at the snippet from the /etc/apache2/apache2.conf above, that is not the Ubuntu default. Hudson Valley Host was nice enough to put some sane defaults in there. This is what comes as default on most Ubuntu installs:

<IfModule mpm_prefork_module>
StartServers 5
MinSpareServers 5
MaxSpareServers 10
MaxClients 150
MaxRequestsPerChild 0
</IfModule>

The big issue is the MaxClients value. MaxClients is basically the number of processes Apache will spin up to deal with the incoming traffic. Each one of those individual processes uses memory. On my default install, each process was using about 16mb of memory, or a maximum total of 160mb. The above configuration will use up to 2400mb or over 2gb of RAM. As you can see, that’s a whole lot larger than the 256mb we have available on this server. It’s bigger than pretty much any affordable VPS and that’s just for Apache processes! If left like this, any decent amount of traffic will come along and kill your server. This is usually the case when people blame WordPress for not being able to handle traffic[2].

How can we fix this?

Apart from changing your mpm_prefork_module settings so your server doesn’t kill itself (and you should do that as step 1), there are a few quick changes you can make to your server. We’ll start with the easiest (and least effective).

Install APC

Basically when Apache gets a request for a page on your blog, it fires up a PHP process, PHP takes the human-readable WordPress code and compiles it to something the server can then run and then that talks to MySQL, etc etc. This happens every time a page request is made, and this is horribly inefficient. What APC does is save the compiled version of the WordPress PHP code, removing that step from the process. Thankfully, that’s one of the slower parts of the whole chain. It’s enabled just by running this command (either via sudo or as root):

apt-get install php-apc && service apache2 restart

That’s it! The results from just this tweak, with the same 100 requests per second?

Transactions:        4461 hits
Availability:      100.00 %
Elapsed time:       59.23 secs
Data transferred:        9.81 MB
Response time:        0.81 secs
Transaction rate:       75.32 trans/sec
Throughput:        0.17 MB/sec
Concurrency:       61.19
Successful transactions:        4461
Failed transactions:           0
Longest transaction:        1.39
Shortest transaction:        0.18

Holy crap! That’s more than twice as fast! It’s like getting a second server for free! There is more speed available in there though…

W3 Total Cache

This is a free plugin available from the WordPress codex. Basically this plugin acts similar to APC except instead of saving the compiled version of the PHP script, it will save the HTML output of the page, bypassing the PHP to MySQL to PHP exchange. With Disk caching Apache will barely even talk to PHP and just serve up the cached HTML by itself.

There’s tons of options in here, but we’re only going to worry about the first page. I tend to disable the preview mode just so I can make my changes “live” right away for testing.

Note: Not all VPSs are created the same. Some have horrible disk I/O, some have over subscribed RAM that’s partially host swap, etc. Test the various settings to see what works best for you.

First off we’re going to look at caching everything to disk (Page cache: Disk Enhanced, DB: Disk, Object: Disk).

Transactions:        6825 hits
Availability:      100.00 %
Elapsed time:       59.83 secs
Data transferred:       15.97 MB
Response time:        0.36 secs
Transaction rate:      114.07 trans/sec
Throughput:        0.27 MB/sec
Concurrency:       41.03
Successful transactions:        6825
Failed transactions:           0
Longest transaction:        9.71

No bad! We can also stuff the cached files directly into APC (Page cache: APC, DB: APC, Object: APC). This is a bit more CPU intensive, but if your host has horrible disk I/O, this maybe your faster choice.

Transactions:        7174 hits
Availability:      100.00 %
Elapsed time:       59.43 secs
Data transferred:       16.69 MB
Response time:        0.31 secs
Transaction rate:      120.71 trans/sec
Throughput:        0.28 MB/sec
Concurrency:       37.64
Successful transactions:        7174
Failed transactions:           0
Longest transaction:        1.66
Shortest transaction:        0.14

Wow! 120.71 requests per second is about 10.5 million requests per day. That’s a lot of visitors.

Ok, what if we crank our concurrency up more? Let’s say 250.

Disk:

Transactions:        6332 hits
Availability:       99.95 %
Elapsed time:       59.78 secs
Data transferred:       14.79 MB
Response time:        1.78 secs
Transaction rate:      105.92 trans/sec
Throughput:        0.25 MB/sec
Concurrency:      188.71
Successful transactions:        6332
Failed transactions:           3
Longest transaction:       24.75
Shortest transaction:        0.30

APC:

Transactions:        5844 hits
Availability:       99.98 %
Elapsed time:       59.96 secs
Data transferred:       13.60 MB
Response time:        1.97 secs
Transaction rate:       97.46 trans/sec
Throughput:        0.23 MB/sec
Concurrency:      192.04
Successful transactions:        5844
Failed transactions:           1
Longest transaction:       28.02
Shortest transaction:        0.14

What the what? They’re LESS! Yeah, well Apache is starting to fall over here, and you can see that there’s some failed requests as well. That’s not good. Can we help poor Apache?

Nginx

Nginx is pretty awesome. It’s a web server just like Apache, but very very fast. WordPress.com uses it, so it should be good enough for us. Now, if you’re just starting out with a fresh VPS, you’re probably better off setting up Nginx + PHP-FPM. We’re not going to do that here, instead we’re going to use Nginx as a caching proxy for Apache. There’s a bunch of tutorials about setting Nginx + PHP-FPM, we’re just looking to get some more performance from our Apache install with the least amount of work with the ability to roll back easily if necessary.

Basically Nginx will sit in front of Apache. If it can serve up a file it will, if not, it will hand the request to Apache and then save the result for future requests.

Warning: We’re going to be editing some important files, make backups just in case!

/etc/apache2/ports.conf: Make these changes:

NameVirtualHost *:8080
Listen 8080

/etc/apache2/apache.conf:

KeepAlive Off

/etc/apache2/sites-available/default: Make these changes:

<VirtualHost *:8080>

Restart Apache:

service apache2 restart

Install Nginx and Apache mod_rpaf

apt-get install nginx libapache2-mod-rpaf && a2enmod rpaf

Replace /etc/nginx.conf with the following, changing your worker_processes to no more than the number of CPU cores you have:

user www-data;
worker_processes 8;

error_log /var/log/nginx/error.log;
pid /var/run/nginx.pid;

events {
worker_connections 1024;
}

http {
include /etc/nginx/mime.types;
default_type application/octet-stream;

access_log /var/log/nginx/access.log;
client_body_temp_path /var/lib/nginx/body 1 2;
gzip_buffers 32 8k;
sendfile on;
#tcp_nopush on;

#keepalive_timeout 0;
keepalive_timeout 65;
tcp_nodelay on;

gzip on;

gzip_comp_level 6;
gzip_http_version 1.0;
gzip_min_length 0;
gzip_types text/html text/css image/x-icon application/x-javascript application/javascript text/javascript application/atom+xml application/xml ;

include /etc/nginx/conf.d/*.conf;
include /etc/nginx/sites-enabled/*;
}

/etc/nginx/sites-available/default (replace yourserver.com with your actual server name)

server {
listen 80;
server_name yourserver.com;
access_log /var/log/nginx.access.log;
error_log /var/log/nginx_error.log debug;

location / {
root /var/www;
proxy_redirect off;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
if ($http_cookie ~* “comment_author_|wordpress_(?!test_cookie)|wp-postpass_”) {
set $do_not_cache 1;
}
proxy_cache_key “$scheme://$host$request_uri $do_not_cache”;
proxy_cache cache;
proxy_cache_valid 200 302 60m;
proxy_cache_valid 404 1m;
proxy_pass http://127.0.0.1:8080;
}

error_page 500 502 503 504 /50x.html;
location = /50x.html {
root /var/www/nginx-default;
}
}

Create /etc/nginx/conf.d/proxy.conf with the following:

proxy_redirect off;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
client_max_body_size 10m;
client_body_buffer_size 128k;
client_header_buffer_size 64k;
proxy_connect_timeout 90;
proxy_send_timeout 90;
proxy_read_timeout 90;
proxy_buffer_size 16k;
proxy_buffers 32 16k;
proxy_busy_buffers_size 64k;
proxy_cache_path /var/lib/nginx/cache levels=1:2 keys_zone=cache:8m max_size=1000m inactive=600m;

proxy_temp_path /var/lib/nginx/temp;

Create the following directories if they don’t exist and chown them for www-data:www-data:

/var/lib/nginx/cache
/var/lib/nginx/temp

mkdir /var/lib/nginx/cache
chown -R www-data:www-data /var/lib/nginx/cache
mkdir /var/lib/nginx/temp
chown -R www-data:www-data /var/lib/nginx/temp

Restart Apache and Nginx

service apache2 restart && service nginx restart

Check your site, you should see your WordPress homepage!

So what did all this work get us? The following is at the same 250 concurrency setting that choked Apache above, but with W3 Total Cache not enabled yet:

Transactions:       18447 hits
Availability:      100.00 %
Elapsed time:       59.37 secs
Data transferred:       40.60 MB
Response time:        0.24 secs
Transaction rate:      310.71 trans/sec
Throughput:        0.68 MB/sec
Concurrency:       75.54
Successful transactions:       18447
Failed transactions:           0
Longest transaction:       13.74
Shortest transaction:        0.14

Wow! 310.71 req/sec! That’s more than 2.5x our best Apache only result (@100 concurrency)! And it’s at almost 0 CPU usage as well.

How about 500 concurrency?

Transactions:       33119 hits
Availability:      100.00 %
Elapsed time:       60.23 secs
Data transferred:       72.87 MB
Response time:        0.35 secs
Transaction rate:      549.88 trans/sec
Throughput:        1.21 MB/sec
Concurrency:      193.97
Successful transactions:       33119
Failed transactions:           0
Longest transaction:       21.53
Shortest transaction:        0.14

Just shy of 550 requests per second! That works out to 1,425,288,960 requests per 30 day month. Again at almost 0 CPU. Granted, we’ll run out of our allocated bandwidth long before the month is over, but we’ve tackled the issue of WordPress falling over.

  1. [1]These are just therotical WordPress pages, and each WordPress page, of course, has references to images, CSS and JS files. While this number does not exactly correlate to active visitors your site can serve, by offloading as much of your static content as possible by using CloudFlare, Amazon S3, or other CDNs, you can get closer to these numbers in practice.
  2. [2]Also, be aware that each plugin you use adds memory overhead. The theme you’re using may have additional memory overhead as well. Then there’s MySQL which will want to start using more memory as you add more posts and get more comments. But we’re not going to worry about that right now, you’ll see why.

Using Flurry, Tapjoy, Apsalar, etc with PhoneGap on iOS

PhoneGap is a great framework for building native iOS, Android, etc apps with just HTML, CSS & JS. However, there is a slight misconfiguration in the way PhoneGap sets up new Xcode projects, and this will not allow 3rd party analytic packages to work correctly.

Services such as Flurry and Tapjoy require you to add a bit of code into the “applicationDidFinishLaunching” section of your AppDelegate file (located in your Xcode project -> “Classes” folder -> “APPNAMEAppDelegate.m”). However, this code never actually gets executed. Below is how to setup your AppDelegate to work correctly.

First, you’ll need to include the framework’s files into your Xcode project. Follow the specific instructions from the provider as these may change from library to library and from version to version.

Second, open up the AppDelegate.m file in the “Classes” folder for your project. It’s easier to do this while in Xcode.

Add the import reference for the supplied .h file for your provider underneath the existing import references. I’ll be using Flurry Analytics as an example:

Old:

#import "APPNAMEAppDelegate.h"
#import "PhoneGapViewController.h"

New:

#import "APPNAMEAppDelegate.h"
#import "PhoneGapViewController.h"
#import "FlurryAPI.h"

Next you’ll be removing a block of code and replacing it with a new one.

Remove:

/**
* This is main kick off after the app inits, the views and Settings are setup here.
*/
- (void)applicationDidFinishLaunching:(UIApplication *)application {
[ super applicationDidFinishLaunching:application ];
}

Replace with:

/**
* This is main kick off after the app inits, the views and Settings are setup here.
*/
- (BOOL)application:(UIApplication *)application didFinishLaunchingWithOptions:(NSDictionary *)launchOptions {
//Custom Code Here
[FlurryAPI startSession:@"FLURRYSECRET"];
return [ super application:application didFinishLaunchingWithOptions:launchOptions ];
}

And that’s it! Command-S to save your file and you should be fully integrated with your 3rd party analytics package.