Into The Forrst

Caching counts with PHP and Redis

Forrst displays a lot of counts throughout the UI — comment and like counts on posts, follower and post counts on users, and so forth. Here’s an example post sidebar showing comment and like counts:

Initially, we could simply do queries like SELECT COUNT() FROM comments WHERE post_id = 1234 whenever necessary to populate the associated comment_count attribute in our Post model. As the site grew and traffic increased (and as a result, the cardinality of our main tables increased as well), doing multiple COUNT() statements per pageview started to become a performance hit when composing certain pages. For example, composing an activity page involves doing 2N COUNT(*) queries. For activity pages, we render 7 or 10 posts per page (depending on the page), so we’re looking at 14 - 20 of those counts per pageview. Not terrible by any means, but certainly not ideal. It would be great if we could avoid hitting the database altogether.

It just so happens we can: Redis^*. Anytime we do a count query now, we also write the result to a Redis key. For example:

SELECT COUNT(*) FROM comments WHERE post_id = 1234

is also written to Redis:

$redis->set("post:1234:count:comments", $count);

When pulling post records from the database, we test for the existence of relevant Redis keys for whatever counts we need; if they exist, great, we’re done. If not, they’re looked up in mysql and pushed into Redis, ready for next time. Cache misses are rare, though, since we also update the cache anytime an associated record is created or destroyed. If someone posts a comment, we update the count; likewise, if someone deletes a comment, the count is updated also. (There’s no need to use increment or decrement since the count initially comes from mysql anyhow, we’re not simply using Redis as a counter).

Since building Redis-backed count caches, the number of round trips to the database has dropped immensely across all pageviews. On a popular post or recently visited user profile, we’re seeing cache hit rates of 90%+ and composing the page in PHP takes less than 100ms in many cases. (User profiles see an average time of around 40ms.)

I hope you’ve enjoyed a glimpse into what goes into building Forrst. I’m going to try to make these kinds of posts a regular event.

— Kyle

* Actually, we could also add a bunch of *_count fields to the relevant tables in Mysql and cache them that way, but by using Redis we can easily invalidate caches without needing to hit the database; if for some reason the memory cache diverges from the true count, we just wipe out the memory cache and the app takes care of regenerating it automatically. If we were caching directly in mysql tables, we’d have to re-sync the cached counts another way. And, to be fair, this could easily be done with Memcached as well; Redis’ persistence is what led us to choose it.