Hungry Machine » tips

Agressive Timeouts On External API Calls

val — Mon, 31 Mar 2008 03:48:00 +0000

One of the challenges with writing a Facebook or Bebo application is staying within a limit it gives you to respond with data before it shows the Application Did Not Respond page to a user. Having a content reach application calling external APIs, like Amazon or YouTube, with response times beyond your control, forces you to keep such calls short to allow extra time for processing. We usually wrap them in aggressive timeouts with a retry. As an example is this code from the Ruby Amazon E-Commerce REST Service API gem rewritten to limit a single call attempt to two seconds with one more retry.

Original Code

module Amazon
  class Ecs

    def self.send_request(opts)
      request_url = prepare_url(opts)

      res = Net::HTTP.get_response(URI::parse(request_url))
      unless res.kind_of? Net::HTTPSuccess
        raise Amazon::RequestError, "HTTP Response: #{res.code} #{res.message}"
      end
      Response.new(res.body)
    end

  end
end

Modified Code

module Amazon
  class Ecs

    class EmptyResponse
      def items; []; end
      def total_pages; 0; end
    end

    def self.send_request(opts)

      res = timed_try(request_url, 2) do |url|

        uri = URI::parse(url)
        req = Net::HTTP.new(uri.host, uri.port)

        # Agressive timeouts
        req.open_timeout = 1
        req.read_timeout = 2

        req.start { |http| http.request_get(url) }

      end

      res.kind_of?(Net::HTTPSuccess) ? Response.new(res.body) : EmptyResponse.new

    end

private

     def timed_try(url, attempts, &block)

       attempt = 1
       begin
         block.call(url)
       rescue Timeout::Error
         if attempt >= attempts
           RAILS_DEFAULT_LOGGER.warn "[amazon_api] gave up after attempt ##{ attempt } to get data from #{ url }"
           nil
         else
           RAILS_DEFAULT_LOGGER.warn "[amazon_api] attempt ##{ attempt } timed out on getting data from #{ url }"
           attempt += 1
           retry
         end
       end

     end

  end
end

Reconfiguring the whole rails stack via a central YAML file

val — Mon, 20 Aug 2007 02:07:00 +0000

The challenge with hosting of multiple Rails-based Facebook applications is that the amount of users grow quickly. To address this problem we are using EC2 nodes that we can expand/shrink as the demand grows. The price/performance ratio isn’t quite what we first expected, so we are moving toward having a few dedicated boxes instead. Another problem that we add at least a couple of applications a week. On each box that hosts them, we need to reconfigure monit, haproxy, nginx, logrotate and nagios.

To mitigate both issues on dedicated boxes, we resolved to have a central configuration definition in svn with individual box configurations keyed on localhost name. A ruby script regenerates all those aforementioned configuration files from ERB-processed templates when it is run on a box and bounces the services. A sample config looks like:

dedicated-1:

    description: "The dedicated box #1"
    ip: 64.233.167.99
    failover: dedicated-2

    apps:

        bookshelf:
            port: 5000
            instances: 20
            response: Book

        ljconnect:
            port: 6000
            instances: 7
            virtual: ljconnect.hungrymachine.com
            response: Journal

That definition would generate a monit config with 20 instances of the bookshelf application and 7 instances of the ljconnect application plus all other configurations (including nagios health checks expecting the response value) . It is all possible because we adopt a fixed application deployment file structure and port numbering conventions (via offsets) for all servers.

Killing sneaky mongrels

val — Thu, 16 Aug 2007 18:16:00 +0000

We found that sometimes monit fails to restart all mongrel instances after deployment and some of them end up running with the pid file gone. Since there is no pid, monit believes the instance is not running so it tries to start a new one on the same port and, of course, fails. Which leads to stale mongrel instances with old code. We’re investigating a long term solution but in the meantime have wrapped the mongrel_rails start script with a replacement which finds and kills the stale mongrel instances before starting a new one.

#!/usr/bin/env ruby

class MongrelController

  def self.run_mongrel(args)
    pid = extract_pid(args)
    kill_stale_process(pid) if pid
    system "/bin/mongrel_rails #{ args.join(' ') }"
  end

  def self.extract_pid(args)
     (args[0] == 'start') && (i = args.index('-P')) && args[i + 1]
  end

  def self.kill_stale_process(pid)
    mongrel_processes(pid).each { |p| process_running?(p) && Process.kill(9, p)  }
  end

  def self.mongrel_processes(pid)
    `ps axww -o 'pid command'`.split(/\n/).inject([]) do |mongrels, process|
      mongrels << process[/^\s*(\d+)/][$1].to_i if process.match(%r{/bin/mongrel_rails\s.*\s-P\s#{ pid }\b})
      mongrels
    end
  end

  def self.process_running?(pid)
    pid && (`ps -p #{ pid }`.split(/\n/).size == 2)
  end

end

MongrelController.run_mongrel(ARGV)

Using mocks at the early stage of FB app development

val — Wed, 15 Aug 2007 12:50:00 +0000

Developing applications for Facebook is a pain. The tunnel approach helps a lot to ease that pain but even then I prefer to start a FB app as a regular application, polish the logic, and then convert it to the Facebook one by adding FBML and such. At the early stages of the development I have the mocked parameter in config/facebook.yml set to true and keep this code in config/initializers/facebook.rb:

PERSON_PROFILE_URL = "http://www.facebook.com/profile.php"

FACEBOOK_CONFIG = YAML.load_file("#{RAILS_ROOT}/config/facebook.yml")[RAILS_ENV] || {}

if FACEBOOK_CONFIG['mocked']

  class Facebook::FBMLController

    require 'ostruct'
    FB_SESSION = OpenStruct.new(:session_user_id => 1, :session_key => "12345", :is_valid? => true)

    def fbsession; FB_SESSION; end

    def require_facebook_install; true; end

    def redirect_to(url); super; end

    def url_for(*params); super; end

  end

  module Facebook::Acts::FbUser
    module InstanceMethods
      def friends
        (self.class.find(:all) - [ self ]).collect(&:uid)
      end
    end
  end

end

It mocks out just enough of Facebook on Rails functionality to use FBMLController and acts_as_fb_user from the beginning without Facebook backend.

Routing to the initial action after facebook application install

val — Tue, 14 Aug 2007 23:17:00 +0000

Some facebook applications might have multiple entries. For example, a user might be adding an application (action – new) or replying to an invitation (action – reply, param – id). Since the UI for Facebook application configuration allows to provide only static Post-Add URL it might seem like there is no way to route users back to the original action if they tried to reach when the application has not been installed for them. Luckily, we have full control on the destination via the next paramater of the post install URL. All we need is to build a URL using the incoming call parameters with the exclusion of Facebook-specific ones.

This is an example for Facebook on Rails based code that might go to the application controller:

class ApplicationController < Facebook::FBMLController

protected

  before_filter :require_facebook_install

  def require_facebook_install
    if in_canvas? && !fbsession.is_valid?
      redirect_to fbsession.get_install_url(:next => url_for(post_install_params))
      false
    end
  end

  def post_install_params
    params.merge(:init => true).delete_if { |k, v| k.starts_with?('fb_sig') }
  end

end

Notice that the code sets the init parameter so it can be used to identify a post install call

Indentifying users who just installed your facebook apps

val — Tue, 14 Aug 2007 23:13:00 +0000

Sometimes it is useful to do some action on a Facebook user right after your application has been installed by the user. For example, you might want to push some default FBML to user’s profile in case he does not complete the action you expect him to do after installation. Facebook application configuration allows to provide Post-Add URL to route users to the destination url after the application install. It could be a dedicated post_add action or, in case of a default action where you have some code in the controller and since Facebook limits amount of redirects you can use, it could be a parameter to the url, like &init=true, used to identify that it was a post-install action and execute on it.

Using a ruby-based AIM notifier in nagios

val — Tue, 14 Aug 2007 22:36:00 +0000

If you use nagios for monitoring of your rails instances, you might want to get notification not only via email or SMS-messages but to your AIM when you are online. The script (libexec/aim_notifier.rb) utilizes the Net::TOC gem for sending out notifications:

#!/usr/bin/env ruby

 require 'rubygems'
 require 'net/toc'

 user = 'your_bot_name'
 password = 'bot_password'

 msg = ARGV[0].to_s.gsub('\n', "\n")

 client = Net::TOC.new(user, password)

 client.connect

 sleep 3

 buddies = []

 client.buddy_list.each_group { |g, b| buddies = b if g == 'Friends' }

 buddies.each do |b|
   b.send_im(msg) if b.available?
 end

 sleep 3

 client.disconnect

You need to add any account you want to be notified to bot’s friends (either by logging to AIM using the bot account or using Net::TOC’s ability to add friends).

The last piece is to add a new notifier in etc/objects/commands.cfg as:

 define command{
         command_name    notify-service-by-aim
         command_line    $USER1$/aim_notifier.rb $ARG1$ $ARG2$ "***** Nagios *****\n\nNotification Ty
 pe: $NOTIFICATIONTYPE$\n\nService: $SERVICEDESC$\nHost: $HOSTALIAS$\nAddress: $HOSTADDRESS$\nState:
 $SERVICESTATE$\n\nDate/Time: $LONGDATETIME$\n\nAdditional Info:\n\n$SERVICEOUTPUT$"
         }

and to append it to the list of notifiers defined for a contact template in etc/objects/commands.cfg:

 service_notification_commands   notify-service-by-email,notify-service-by-aim

Repeat the configuration if you want to use the AIM notification for hosts as well.