Hungry Machine » Rails

ESI & Mongrel-ESI.. Request for Feedback

aaron — Wed, 04 Jun 2008 15:59:00 +0000

The Railsconf08 talk on ESI & Rails has sparked some interest in the community, and Todd, the core mongrel-esi maintainer, is asking for feedback on the mongrel-esi mailing list.

The latest rumor is he is working on a nginx port of mongrel-esi, which I have to admit sounds very interesting…

How are people planning to use ESI? Can anyone provide Todd with some feedback?

  I'm wondering what is the main set of features preventing folks
today from using mongrel-esi in production?  Is it :

* performance
* documentation
* stability

If it's performance, can someone do some load testing maybe provide
some numbers and even some target numbers?  I am working on an
improved concurrency model, that should help improve page performance
for pages with lots of esi:include tags

if it's documentation, what's the missing bits, and can any of you
help out by filling in the gaps?

if it's stability, can you provide some samples that fail?

-Todd

Advanced Rails Recipes

aaron — Tue, 13 May 2008 16:32:00 +0000

Congratulations to Mike Clark and the Pragmatic Programmers team for shipping Advanced Rails Recipes. I highly recommend this book of 70+ recipes on topics ranging from deployment to UI to security and performance, just to list a few.

Val, Warren and I are honored to have our recipes included.

Recipe 34: Play Nice with Facebook

Recipe 62: Profile in the Browser

Recipe 67: Encrypt Sensitive Data

It was a pleasure to work with Mike Clark on this project. See his screencast below.

Advanced Rails Recipes Screencast (33Mb, Quicktime)

ActsAsInsertOrUpdate

aaron — Wed, 23 Apr 2008 03:40:00 +0000

Problem

With high volume Rails applications, entities with unique constraints are expensive and error prone to create/update. ActsAsInsertOrUpdate helps solve that problem (if you’re using MySQL), by leveraging the “INSERT … ON DUPLICATE KEY UPDATE” functionality.

Scenario

Lets say you have a Person, and Entity, and a Rating. Each user can rate each entity only once, and if they re-rate the entity, it should update the value.

class Entity < ActiveRecord::Base
  has_many :ratings
end

class Person < ActiveRecord::Base
 has_many :ratings
end

class Rating < ActiveRecord::Base
 belongs_to   erson
 belongs_to :Entity
end

Here is the table that back’s Rating. Notice the Unique Key constraint on (entity_id, person_id).

CREATE TABLE `ratings` (
  `id` int(11) NOT NULL auto_increment,
  `rating` tinyint(4) default '0',
  `person_id` int(11) default NULL,
  `entity_id` int(11) default NULL,
  `created_at` datetime default NULL,
  `updated_at` datetime default NULL,
  PRIMARY KEY  (`id`),
  UNIQUE KEY `index_ratings_on_entity_id_and_person_id` (`entity_id`,`person_id`),
)

Previously, the logic would be something like:

1) Check if a rating exists for the User + Entity
2) If so, update
3) If not, insert
4) rescue the insert in case there is a unqiue constraint error
5) retrieve the record (and/or update with the new rating)

If the table is MyISAM, Steps 1-5 aren’t transactionally safe. If you’re using InnoDB, and experience heavy volumes of traffic, you’re prone to Deadlock’s. This is even more of a concern is the unique entity is shared across multiple users, as seen with a recent client of ours.

Solution:

class Rating < ActiveRecord::Base
 belongs_to   erson
 belongs_to :Entity
 acts_as_insert_or_update :field_to_update => "rating"
end

Now Steps 1-5 above become, just one. Rating.create(..)

In the background, ActsAsInsertOrUpdate overwrites the implementation of ActionRecord:Base#create, to leverage an often unsed feature of MySQL called INSERT … ON DUPLICATE KEY UPDATE. As configured above, if a duplicate record is found for the unique constraint, the rating field will be updated with the new value.

Caution

This is a brute force hack on ActiveRecord::Base#create. Use at your own risk.

Code

Waiting for a rubyforge account. Will post more info soon.

Agressive Timeouts On External API Calls

val — Mon, 31 Mar 2008 03:48:00 +0000

One of the challenges with writing a Facebook or Bebo application is staying within a limit it gives you to respond with data before it shows the Application Did Not Respond page to a user. Having a content reach application calling external APIs, like Amazon or YouTube, with response times beyond your control, forces you to keep such calls short to allow extra time for processing. We usually wrap them in aggressive timeouts with a retry. As an example is this code from the Ruby Amazon E-Commerce REST Service API gem rewritten to limit a single call attempt to two seconds with one more retry.

Original Code

module Amazon
  class Ecs

    def self.send_request(opts)
      request_url = prepare_url(opts)

      res = Net::HTTP.get_response(URI::parse(request_url))
      unless res.kind_of? Net::HTTPSuccess
        raise Amazon::RequestError, "HTTP Response: #{res.code} #{res.message}"
      end
      Response.new(res.body)
    end

  end
end

Modified Code

module Amazon
  class Ecs

    class EmptyResponse
      def items; []; end
      def total_pages; 0; end
    end

    def self.send_request(opts)

      res = timed_try(request_url, 2) do |url|

        uri = URI::parse(url)
        req = Net::HTTP.new(uri.host, uri.port)

        # Agressive timeouts
        req.open_timeout = 1
        req.read_timeout = 2

        req.start { |http| http.request_get(url) }

      end

      res.kind_of?(Net::HTTPSuccess) ? Response.new(res.body) : EmptyResponse.new

    end

private

     def timed_try(url, attempts, &block)

       attempt = 1
       begin
         block.call(url)
       rescue Timeout::Error
         if attempt >= attempts
           RAILS_DEFAULT_LOGGER.warn "[amazon_api] gave up after attempt ##{ attempt } to get data from #{ url }"
           nil
         else
           RAILS_DEFAULT_LOGGER.warn "[amazon_api] attempt ##{ attempt } timed out on getting data from #{ url }"
           attempt += 1
           retry
         end
       end

     end

  end
end

Leverage Rails Resource Routes on Facebook

eddie — Mon, 07 Jan 2008 21:57:00 +0000

Since all canvas page views are proxied through POSTs, resource routes were hopelessly broken. The Facebook platform team was kind enough to add a new feature just for us rails folks: a new signed parameter that indicates the original request type (i.e. POST v. GET) against canvas pages.

Here’s a small patch you can stick at the end of environment.rb to restore REST-ful routes.

class ActionController::Routing::RouteSet
  def extract_request_environment(request)
    contents = request.body.read
    facebook_method_override = nil
    if contents =~ /fb_sig_request_method=([A-Z]+)/
      facebook_method_override = $1
    end
    request.body.string = contents
    { :method => facebook_method_override ? facebook_method_override.downcase.intern : request.method}
  end
end

UPDATE: Here’s a slightly better implementation that allows the .get? and .post? helpers as well as a few other things to work (thanks Chris Nolan for reminding us to update this blog entry):

ActionController::AbstractRequest.class_eval do
  def request_method_with_facebook_overrides
    @request_method ||= begin
      case
        when parameters[:_method]
          parameters[:_method].downcase.to_sym
        when parameters[:fb_sig_request_method]
          parameters[:fb_sig_request_method].downcase.to_sym
        else
          request_method_without_facebook_overrides
      end
    end
  end
  alias_method_chain :request_method, :facebook_overrides
end

Super fast IP to lat/lng in Rails

aaron — Mon, 22 Oct 2007 15:33:00 +0000

In building the eye candy demo for the Graphing Social conference, I needed a quick way to geo-locate users by IP address. For Rails, GeoKit is an awesome plugin. It supports a list of providers, and overlays distance calculations, before_filter helpers, all sorts of good stuff.

It uses hostip.info to do IP to lat/lng, using their RESTful interface:

http://api.hostip.info/get_html.php?ip=12.215.42.19&position=true
  Country: UNITED STATES (US)
  City: Sugar Grove, IL
  Latitude: 41.7696
  Longitude: -88.4588

This is great, but its just too darn slow to geolocate 12 users a second. All I needed was a FAST ip to lat/long lookup mechanism. Luckily, HostInfo provides the raw data. Here’s how I built super fast GeoLoc by IP method.

Download, create a DB, and import the data.

wget http://hostip.ww.com/hostip_current.sql.gz
gunzip hostip_current.sql.gz
mysqladmin -uroot create hostip
mysql -uroot hostip < hostip_current.sql

Create a simple HostIp class. Note: I need to dynamically build this query b/c the data is sharded across tables by the A class of the IP.

class Hostip < ActiveRecord::Base
  def self.geocode(ip)
    a,b,c,d = ip.split(".")
    self.set_table_name "hostip.ip4_#{a}"
    ip = find(:first, :select => "lat,lng",
                :joins => %Q{INNER JOIN hostip.cityByCountry
                               ON hostip.ip4_#{a}.city = hostip.cityByCountry.city
                               AND hostip.ip4_#{a}.country = hostip.cityByCountry.country},
                :conditions => ["b = ? and c = ?", b,c])
    if ip
      [ip.lat, ip.lng]
    else
      ["",""]
    end
  end
end

Usage

>> Hostip.geocode("4.2.2.2")
=> ["39.944", "-105.062"]

Thats it! Now you can geolocate by IP in your own datacenter.

Thanks again to the guys at HostIp for sharing this data!

Reconfiguring the whole rails stack via a central YAML file

val — Mon, 20 Aug 2007 02:07:00 +0000

The challenge with hosting of multiple Rails-based Facebook applications is that the amount of users grow quickly. To address this problem we are using EC2 nodes that we can expand/shrink as the demand grows. The price/performance ratio isn’t quite what we first expected, so we are moving toward having a few dedicated boxes instead. Another problem that we add at least a couple of applications a week. On each box that hosts them, we need to reconfigure monit, haproxy, nginx, logrotate and nagios.

To mitigate both issues on dedicated boxes, we resolved to have a central configuration definition in svn with individual box configurations keyed on localhost name. A ruby script regenerates all those aforementioned configuration files from ERB-processed templates when it is run on a box and bounces the services. A sample config looks like:

dedicated-1:

    description: "The dedicated box #1"
    ip: 64.233.167.99
    failover: dedicated-2

    apps:

        bookshelf:
            port: 5000
            instances: 20
            response: Book

        ljconnect:
            port: 6000
            instances: 7
            virtual: ljconnect.hungrymachine.com
            response: Journal

That definition would generate a monit config with 20 instances of the bookshelf application and 7 instances of the ljconnect application plus all other configurations (including nagios health checks expecting the response value) . It is all possible because we adopt a fixed application deployment file structure and port numbering conventions (via offsets) for all servers.

Using mocks at the early stage of FB app development

val — Wed, 15 Aug 2007 12:50:00 +0000

Developing applications for Facebook is a pain. The tunnel approach helps a lot to ease that pain but even then I prefer to start a FB app as a regular application, polish the logic, and then convert it to the Facebook one by adding FBML and such. At the early stages of the development I have the mocked parameter in config/facebook.yml set to true and keep this code in config/initializers/facebook.rb:

PERSON_PROFILE_URL = "http://www.facebook.com/profile.php"

FACEBOOK_CONFIG = YAML.load_file("#{RAILS_ROOT}/config/facebook.yml")[RAILS_ENV] || {}

if FACEBOOK_CONFIG['mocked']

  class Facebook::FBMLController

    require 'ostruct'
    FB_SESSION = OpenStruct.new(:session_user_id => 1, :session_key => "12345", :is_valid? => true)

    def fbsession; FB_SESSION; end

    def require_facebook_install; true; end

    def redirect_to(url); super; end

    def url_for(*params); super; end

  end

  module Facebook::Acts::FbUser
    module InstanceMethods
      def friends
        (self.class.find(:all) - [ self ]).collect(&:uid)
      end
    end
  end

end

It mocks out just enough of Facebook on Rails functionality to use FBMLController and acts_as_fb_user from the beginning without Facebook backend.

Routing to the initial action after facebook application install

val — Tue, 14 Aug 2007 23:17:00 +0000

Some facebook applications might have multiple entries. For example, a user might be adding an application (action – new) or replying to an invitation (action – reply, param – id). Since the UI for Facebook application configuration allows to provide only static Post-Add URL it might seem like there is no way to route users back to the original action if they tried to reach when the application has not been installed for them. Luckily, we have full control on the destination via the next paramater of the post install URL. All we need is to build a URL using the incoming call parameters with the exclusion of Facebook-specific ones.

This is an example for Facebook on Rails based code that might go to the application controller:

class ApplicationController < Facebook::FBMLController

protected

  before_filter :require_facebook_install

  def require_facebook_install
    if in_canvas? && !fbsession.is_valid?
      redirect_to fbsession.get_install_url(:next => url_for(post_install_params))
      false
    end
  end

  def post_install_params
    params.merge(:init => true).delete_if { |k, v| k.starts_with?('fb_sig') }
  end

end

Notice that the code sets the init parameter so it can be used to identify a post install call

Indentifying users who just installed your facebook apps

val — Tue, 14 Aug 2007 23:13:00 +0000

Sometimes it is useful to do some action on a Facebook user right after your application has been installed by the user. For example, you might want to push some default FBML to user’s profile in case he does not complete the action you expect him to do after installation. Facebook application configuration allows to provide Post-Add URL to route users to the destination url after the application install. It could be a dedicated post_add action or, in case of a default action where you have some code in the controller and since Facebook limits amount of redirects you can use, it could be a parameter to the url, like &init=true, used to identify that it was a post-install action and execute on it.