Archive for the 'Web Development' Category

WordPress Performance Testing, part 2

Monday, February 21st, 2011

Evan and I followed up our last round of performance testing with another set of tests to see how apache compares with lighttpd on a more powerful server.  The previous round of performance tests were done on a server with only 4GB RAM and we found that apache filled up all the RAM when hit with 100 or more concurrent connections.  We were curious to see if when we increased the RAM, apache would perform better.

We ran these tests on a 20gb RAM, 40gb hd, ubuntu 10.4 server, shared 16 xeon 2.4GHz machine.  We did find apache coping much better with 100 concurrent connections.  When we scaled up the load to 500 concurrent connections, however, apache ran into problems and crashed while lighttpd really started to fly.  Even 1000 concurrent connections didn’t post any problems for lighttpd.  Here are the test results.


It is okay to use apache if you are expecting small server loads, but if you expect that you might get dug or have other sources of high traffic spikes, you’d be best spending the time to configure a high-performance web server like lighttpd or nginx.

Disqus Comments on GothamSchools

Friday, February 18th, 2011

Today we launched the Disqus commenting system on GothamSchools.  During the last several weeks, the tone of comments on the blog have deteriorated to the point where the editors had to address it directly in a blog post.  Elsewhere, there have been heated and inaccurate complaints about people being banned from GothamSchools commenting.  As a last resort, the editors considered turning on moderation for every comment before it would appear on the site.  Because of the amount of work this would take to keep discussions flowing, this is not a good long-term solution.

Maintaining civility of the community on an active site is something every site manager struggles with, if their site is truly hopping.  One site with a particularly interesting commenting system, Gawker, has all users “audition” before they are trusted to post comments publicly.  Other sites employ “Abuse” and “Like” buttons to get users to self-moderate comment threads.

Disqus combines some of these technologies in one system and today we have launched it on GothamSchools.  Along with the ability to “Like” or “Flag” comments, commenter identities can be linked to the comments from their Facebook or Twitter accounts.  This give those commenters more credibility, perhaps, than people who post anonymously.  Disqus also allows for threaded comments and aggregates “Reactions” from other sources on the web, like Twitter, to add to the discussion.

The GothamSchools editors have introduced this new system along with a new official policy on comments in the form of three questions: “Is it spam? Is it libelous? Is it off topic?”  I guess time will tell on how this latest experiment goes.  Keep an eye on that comment thread.

WordPress, Web Servers, and Cacheing Plugin Testing

Monday, February 7th, 2011

We run several high-traffic WordPress blogs on a lighttpd web server.  Combined, they get about 150,000 unique visitors per month.  We originally moved from an apache web server to lighttpd, because lighttpd can handle a much larger traffic load with the same server resources.

The problem with using lighttpd is that it is not very common for people to run WordPress on this web server.  Certain cacheing and mobile plugins need to alter the configuration of the web server.  Setup and debugging information almost always assumes an apache web server.  So in each case, we had to figure out a way to configure them for lighttpd.  When plugins get upgraded or go obsolete, we again have to find ways to get new plugins to work with lighttpd.  Over time, when we run into problems with our sites, we can’t rely on help being available on the web but have to figure it out ourselves.  Right now, for instance, we are noticing we have a memory leak somewhere which causes lighttpd to eventually fill up all available RAM and crash.

For this reason, we decided to do some performance testing on lighttpd vs. apache, as we were considering possibly going back to using apache.  While we were at it, we also decided to compare our current cacheing plugin, wp-super-cache and another leading plugin, w3-total-cache. (more…)

Civic Media Tech 2011

Thursday, January 20th, 2011

We have several high traffic blogs in the Civic Media department of OpenPlans:

Over the past while, the technology and design of the blogs have become somewhat outdated, we have tolerated some technical performance problems, yet still their readership continues to grow.  During 2011, I plan on improving these things in the hopes that doing so will foster continued community growth in both quantitative and qualitative ways.

The Plan for 2011

Here are the things I would like to do, in this order: (more…)

Browser Cacheing

Friday, November 27th, 2009

The Problem

If you go to a post on, type in a beautifully written comment, fill in the rest of the fields including 5+7 is, um, 13 and hit “Post”, you get “Error: You have entered the wrong sum in the spam protection field…”  Damn!  So you hit Back and go to correct your error only to find that your comment is gone and you’ll have to type it all over again.  Sucks! (more…)

Block Party NYC Site Launched

Wednesday, June 17th, 2009

Who knew there are over 3000 block parties in New York City each summer?  Well, we’ve just launched to be the go-to site for everything block party.  Nicholas Bergson-Shilcock and myself have been working on this site over the last few months in partnership with Transportation Alternatives.  It contains information to assist you in throwing a block party, and allows you to promote your block party on our interactive map.  You can also sign up for weekly notifications of block parties in your neighborhood.block-party-nyc

The tricky part now is to draw enough interest to get the bulk of the 3000 block parties registered on the site without which our weekly notifications would be less useful.  Transportation Alternatives is in contact with the Upper West Side Community Board to encourage them to register each block party they approve.  We hope to also work out similar partnerships with other Community Boards.  As a return service, we have developed a widget for inclusion on Community Board sites that lists their upcoming parties.

Behind the scenes is a PostGIS database we use for its spatial queries.  The site is built in pylons and WordPress, and the mapping is done using OpenLayers.  The site was built to facilitate re-skinning for other purposes.  If you are interested in using or developing with the open source software we wrote, the Block Party project page has the details.

So check it out!  Find out when parties are coming up in your neighborhood.  If you know anyone throwing a party, get them to register it on the site, and, don’t be shy, go and throw a party yourself!

Comment Geo Maps v0.2 Released

Friday, June 5th, 2009

I have just released version 0.2 of the WordPress Comment Geo Maps plugin.  I’d like to also announce that it is now being used in production on  Version 0.2 has a bunch of general improvements and bug fixes.  I have moved the project page for the plugin to its TOPP Labs trac instance but the general discussion about the plugin is still ongoing, for now, on this page.  For more information about this release, please go to the project page.


Comment Geo Maps v0.1 Released

Friday, May 8th, 2009

Over the last couple of weeks I developed a plugin that allows WordPress comments to be geo-located on a map to enable geo-based conversations.  Today I have released version 0.1.  The idea for this came from Nick Grossman.  See the Comment Geo Maps plugin page for comments and suggestions.

BlockPartyNYC 2009

Thursday, February 5th, 2009

Last year a few of us here at TOPP built BlockPartyNYC.  This project was done for Transportation Alternatives, mainly to promote their party sponsorship program.  A sponsored block party received consulting on solutions to community problems like traffic, speeding, and noise and air pollution.

For this year of block parties in NYC, we have plans to make BlockPartyNYC the main hub for all 3000 NYC parties.  We plan on receiving data feeds from Community Boards as they approve block parties and, in turn, hosting relevant parties on Community Board sites.  We plan on having more advanced mapping, facebook integration, and other features.

See the full specification.  If anyone has comments on the proposed ways of tackling the new site, please get in touch.