With the announcement that Merb and Rails are merging, we figured that it’d be a great time to share a little bit of what we learned.

Background

• We’re running Rails 2.2.2.
• When we started messing with routes, we had 2500 generated routes (i.e. “rake routes | wc -l”).  After Aaron’s formatted routes patch, we had 1250.  We currently have over 1500.
• Routing accounted for around 4 seconds out of 6 for ./script/console and mongrel_rails to start up.
• Each mongrel_rails process was running at a minimum of 250mb in production.

Step 1: compiling

Compiling is the first part of routing and involves loading an application’s routes.rb or router.rb and storing it in some way that allows for generation and recognition.

The merb_routing plugin contains a subset of merb-core… just the classes related to routes.  It shouldn’t be a surprise that these classes assume they are running in a Merb environment and not a Rails environment.  To successfully load these merb routing classes without drastically modifying the source, we wrote two small compatibility layers (mini-merb and mini-extlib) that provide some basic functionality that Merb and ExtLib provide (i.e. Merb.logger, Merb.root, etc).  This enabled us to successfully load the Merb::Router classes in our environment without any errors.

We then rewrote our routes.rb by hand using router.rb syntax.  We had to modify Merb routing slightly because we were getting “memory exhausted” compile errors.  It turned out that Merb routing uses a single if/elsif structure to recognize routes and due to our large number of routes, we hit a ruby quirk where you can’t have more than 2498 branches of logic in a single if-elsif statement.

irb(main):001:0> eval("if 1; #{"elsif 1;" * 2498} end")
=> nil
irb(main):002:0> eval("if 1; #{"elsif 1;" * 2499} end")
SyntaxError: (eval):1:in `irb_binding': compile error
(eval):1: memory exhausted

The quick fix was using many single if statement with a return inside rather than one giant if-elsif.   After a few hooks into ActionController::Routing we had our environment using Merb/router.rb instead of Rails/routes.rb.

We had to make a few other minor changes to Merb routing to be compatible with rails:

• added support for { :method => :any } in routing conditions
• added support for BLAH_index as a named route for singular resources (Rails creates a named route called “blog_index” instead of just “blog” for singular resources)

Lastly, we wrote a rake task that overrides the default “rake routes” to pull from Merb routes instead of Rails routes.  At this point, we had the routes loaded and could see the objects in script/console.

Step 2: recognizing

Recognition is the second part of routing and happens at the beginning of every request.  It translates a URL into various parameters and figures out which controller/action to invoke for that request.

Once we had router.rb working correctly, route recognition and parameter loading was surprisingly easy to wire in.

Step 3: generating

Generation is the third part of routing and translates structured options into a URL string.  This happens every time you use url_for, a named route helper or redirect_to.

Generation was the trickiest part of this project and had the most edge cases.  Philosophically speaking, Merb and Rails route generation are very different.  Rails gives you named routes as helpers (person_path, person_url, etc) that you can use as well as a url_for() method which will search through all of the routes and find the best route given the options you provide.  Merb on the other hand provides a single method, url(), which all generation goes through.  If you don’t explicitly provide a named route, url() will use the default route (i.e. :controller/:action/:id) rather than looking for the “best route”.

Getting the named routes to work was pretty easy and only required a single method_missing catch-all for ActionController::Base and ActionView::Base.  Using a simple regexp, if the requested method ends in _url or _path, it uses Merb::Router#url and passes in the named route.

There was a bit of trickery involved in getting the path vs. url as well as the  :only_path stuff working correctly, but overall not too hard.

The last tricky piece was returning the best route instead of the default route when no named route was provided.  This is accomplished by looping through all of the available routes and determining which of the routes satisfies the most options passed in.  With 1500 routes, this turned out to be a bit slow, so some optimization ideas were borrowed from Rails and a cache is maintained of available routes given a controller/action.

Conclusion

We’ve been using this plugin successfully in production for the last month.  Our environment startup time as well as our memory overhead were both reduced drastically as soon as we put it in to production. We started this development when Rails 2.1 was the latest stable release and benchmarks against Rails 2.1 put Merb routing way ahead in just about every metric we tested.

The routing in Rails 2.2 was sped up substantially and is now comparable to Merb (Rails wins in some benchmarks, Merb in others).  BUT…. Merb still blows away rails in startup time, by 2-3x. We thought we could take out our merb-routing hacks and reduce code complexity, but after watching production restarts, we decided to put it back.

This plugin can be found on github: merb_routing

UPDATE: after talking to Carl Lerche, it sounds like the new router refactoring he is working on will support both syntaxes on the same codebase. That’ll be very cool.

ActionController::Routing.routes_for_acts_as(:commentable) do |map|
  map.resources :comments
  map.best_comment '/best-comment', :controller => 'comments', :action => 'best'
end

If you added these :acts_as to your config/routes.rb:

ActionController::Routing::Routes.draw do |map|
  map.resources :people, :acts_as => [:commentable]
  map.resources :posts, :acts_as => [:commentable]
end

You could then use these routes throughout your application:

  <%=person_comments_path(Person.first)%>
  <%=post_comments_path(Post.first)%>
  <%=person_best_comment_path(Person.first)%>

This is equivalent to doing:

ActionController::Routing::Routes.draw do |map|
  map.resources :people do |people_map|
    people_map.resources :comments
    people_map.best_comment '/best-comment', :controller => 'comments', :action => 'best'
  end
  map.resources :posts do |posts_map|
    posts_map.resources :comments
    posts_map.best_comment '/best-comment', :controller => 'comments', :action => 'best'
  end
end

The plugin is available on github: http://github.com/hungrymachine/acts_as_routing/tree/master

P.S. At some point, I’ll submit patches to Rails and Merb so this functionality is native rather than provided via a monkeypatching plugin.

Mark Imbriaco, of 37Signals, asked how it compared to MRI. I was curious too.

I can’t promise a “clean” comparison, your mileage may vary, but… MRI was:

Completed in 552ms (View: 104, DB: 14) | 200 OK [http://someurl.com/people/3]
Completed in 345ms (View: 60, DB:  | 200 OK [http://someurl.com/people/3]
Completed in 346ms (View: 60, DB: 4) | 200 OK [http://someurl.com/people/3]
Completed in 347ms (View: 56, DB: 4) | 200 OK [http://someurl.com/people/3]
Completed in 349ms (View: 58, DB: 5) | 200 OK [http://someurl.com/people/3]

Some stats on my runtime

$ java -version
java version "1.5.0_16"
Java(TM) 2 Runtime Environment, Standard Edition (build 1.5.0_16-b06-284)
Java HotSpot(TM) Client VM (build 1.5.0_16-133, mixed mode, sharing)
$ ruby -v
ruby 1.8.6 (2008-08-11 patchlevel 287) [i686-darwin9.5.0]

Both ran on a MacBook 2.53GHz. Likely the perf is better on Java6?

Installed Java6 and re-ran. Similar but better results.

Completed in 8815ms (View: 1632, DB: 429) | 200 OK [http://someurl.com/people/3]
Completed in 1017ms (View: 656, DB: 25) | 200 OK [http://someurl.com/people/3]
Completed in 998ms (View: 618, DB: 18) | 200 OK [http://someurl.com/people/3]
Completed in 1300ms (View: 437, DB: 35) | 200 OK [http://someurl.com/people/3]
Completed in 315ms (View: 119, DB: 21) | 200 OK [http://someurl.com/people/3]
Completed in 361ms (View: 140, DB: 20) | 200 OK [http://someurl.com/people/3]

Wow! Thats damn close to MRI…eventually… JRuby warmup time sucks though. Maybe warm it up during deploy before putting it back into the load balancer? Not exactly sure how to warm it correctly though.

If you just hit a URL on your app, you’ll only warm up THAT execution path… and you cant hit all actions, so…. thoughts?

That said…. one interesting, entirely un-scientific, test case. After requesting the same URL 6 times, the VM got a lot better. 3.4s to 480ms

Completed in 3380ms (View: 2739, DB: 65) | 200 OK [http://someurl.com/people/3]
Completed in 1299ms (View: 1211, DB: 18) | 200 OK [http://someurl.com/people/3]
Completed in 1020ms (View: 949, DB: 20) | 200 OK [http://someurl.com/people/3]
Completed in 852ms (View: 732, DB: 18) | 200 OK [http://someurl.com/people/3]
Completed in 743ms (View: 628, DB: 31) | 200 OK [http://someurl.com/people/3]
Completed in 485ms (View: 374, DB: 14) | 200 OK [http://someurl.com/people/3]

It doesnt appear to be the DB caching the data. That time stayed consistent.

I didnt use any crazy runtime flags.. Just jetty, a war file, and -server….

If it was something we were memcaching, then after the 2nd request, it would have been consistent.

The amount of time spent in the view improved dramatically.

I didnt think JRuby leveraged JIT’ing yet. Anyone care to guess/explain why it was significantly better per request?

Note: It did trail off in the 300ms range with further testing…

Beforehand, monit would restart instances a handful of times a day. After Rails 2.2, monit restarted instances THOUSANDS of times a day.

This is a graph of one of our haproxy instances a couple days ago.

We looked at everything, including time spent rewriting Routes, thinking that was the culprit.

This morning, we all sat around and fought the issue old school style. binary debugging… and found it: AssetTagHelper. See the patch here.

The new thread-safe asset tag code keeps a static AssetTag::Cache = {} of all asset_tags created (css,jss, and all images).

Internally, each AssetTag object keeps a reference to the controller and template objects, and in turn all instance variables you created in your request.

What does that mean? Say you have a people controller, that loads a person and their stuff, and you show images of their stuff via image_tag().

 class PeopleController < ApplicationController
   def show
     @person = Person.find(params[:id])
     @stuff = @people.stuff.find(:all, :limit => 30)
   end
 end

When image_tag() is called, it does rails magic to append file extensions, asset_ids, and the like. To be smart, it caches those objects so it doesnt hit the file system
to figure all that out on every request. The problem is it puts it in a static cache, AssetTag::Cache.

So each PeopleController instance has a reference to 1 person and 30 Stuffs. After 1000 people look at their pages, or better yet google crawls your site, you have 1k @controllers with a total of 1000 People Objects, and 1000*30 Stuff objects. This would normally be fine. The objects leave scope and get GC’ed. But, if you generate an image tag to an unique asset, AssetTag puts that into a cache, AssetTag::Cache, with a reference to the @controller of the request. So All People and their Stuff are kept around forever, unable to be GC’ed…. every time a unique image is rendered via AssetTag. Eventually monit has to kill the process.

The patch we just submitted does 2 things.

1) It now keeps a cache of just the modified path strings, caching the file access stuff. If you have tons of local images, reference them by fully qualified host. Thats better for lots of reasons. Cookie-less asset hosts with multiple subdomains FTW!

2) It stops caching absolute URL paths. You cant do anything on the filesystem to verify them, and keeping a cache of those would also grow unbounded. We have millions of items in our system, each with a reference to an image.

Here is a graph of that haproxy today… Sleeping………

In order to do some testing of your own, here’s a simplistic after_filter you can add to application.rb (or is it now application_controller.rb?). Make sure you run this in production mode or with cache_classes = true. As you click around your site, you should see that the cache retains references to controller instances, just to name a few. After you apply the patch, you’ll see the cache is just strings.

 def assettag_cache
    puts "-"*80
    puts "[AssetTag::Cache] Now #{ActionView::Helpers::AssetTagHelper::AssetTag::Cache.size} items"
    ActionView::Helpers::AssetTagHelper::AssetTag::Cache.values.each do |asset_tag|
      if asset_tag === ActionView::Helpers::AssetTagHelper::AssetTag
        puts "   [Asset] #{asset_tag.instance_variable_get("@source")}  #{asset_tag.instance_variable_get("@controller").class.to_s}"
      else
        puts "   [Asset] #{asset_tag}"
      end
    end
  end

Ohh… and we havent given up on routes… Warren is working on some very interesting enhancements to rails routing. Looking forward to blogging about that soon.

In a previous article, I called out the massive memory usage of the default rails resource behavior, and it seems others have as well. In an attempt to decrease the number of routes, I commented out the “formatted_*” routes, and manually entered them back by hand.

But after some internal discussion/testing with Warren, we realized that was sloppy and error prone. Instead, I hacked Routing segments to allow for an optional format segment, so that formatted routes and normal routes are shared. The one downside, from what I can tell so far, is you lose “formatted_*” named routes + url helpers for those routes, but passing format to a url_for still works

   person_path(:id => 1, :format => "json") =>  /people/1.json

In the below gist, you’ll see a OptionalFormatSegment, which sneakily gets around the ‘.’ regex separator, removes the formatted_* named routes added by default, and should be the same solution as the previous post, but without the need to manually put all the routes back in.

I’m still testing this approach, but am interested in what other folks think.

Note: This monkey-patch only works on Rails 2.2

Note: This monkey-patch only works on Rails 2.2

We recently noticed our mongrels, upon startup, were 244M. Eek.

PID USER            PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
53749 some_user     15   0  337m 244m 2912 S    0  2.0   0:17.49 mongrel_rails

Running Bleakhouse on our rails app, i was amazed at the number of objects created. over 2M!

$ BLEAK_HOUSE=1 ruby-bleak-house ./script/runner 'puts 1'
** Bleakhouse: installed
1
** BleakHouse: working...
** BleakHouse: complete
** Bleakhouse: Run 'bleak /tmp/bleak.53749.0.dump' to analyze.

$ bleak /tmp/bleak.53749.0.dump
Displaying top 20 most common line/class pairs
2,047,025 total objects
2,047,025 filled heap slots
2,403,764 free heap slots
1779063 __null__:__null__:__node__
197423 __null__:__null__:String
16519 __null__:__null__:Array
12728 __null__:__null__:ActionController::Routing::DividerSegment
10495 __null__:__null__:Hash
5676 __null__:__null__:ActionController::Routing::StaticSegment
5436 __null__:__null__:Class
5132 __null__:__null__:Regexp
4525 __null__:__null__:ActionController::Routing::DynamicSegment
2524 __null__:__null__:ActionController::Routing::Route
1307 __null__:__null__:Gem::Version

Whats up with all those routing objects! So i tried a blank routes.rb file….

519,307 total objects
519,307 filled heap slots
233,503 free heap slots

Arghh…. 1.5M objects just because of routes.rb. I know our application is large, lots of resources and nested resources, but thats crazy.

$ rake routes | wc -l
2516

Ok, so taking a closer look, I realized something. Each resource route has a corresponding formatted_* route, lots of which we dont use. So now for some nasty monkey-patching.

ActionController::Routing::RouteSet::Mapper.class_eval do
protected

  def map_unnamed_routes(map, path_without_format, options)
    map.connect(path_without_format, options)
    #map.connect("#{path_without_format}.:format", options)
  end

  def map_named_routes(map, name, path_without_format, options)
    map.named_route(name, path_without_format, options)
    #map.named_route("formatted_#{name}", "#{path_without_format}.:format", options)
  end
end

I then manually added back the formatted routes we needed, careful to use named routes as “map.formatted_foo_bar”

$ rake routes | wc -l
1490

$ bleak /tmp/bleak.56554.0.dump
1,242,796 total objects
1,242,796 filled heap slots
1,224,309 free heap slots

Awesome. I no longer create almost 800k objects.

PID USER            PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
56554 someuser     15   0  249m 156m 2912 S    0  1.3   0:09.92 mongrel_rails

244m –> 156m

This is not a very elegant solution… but I saved almost 100m per mongrel. Ohh.. and it reduced startup time by 50%.

How big are your mongrels?

How does Merb’s router differ from Rails routing? Worth investigating?

Depending on your data model and your application, sharding data into tables by some natural key is great if any given request uses only one shard. FiveRun’s DataFabric seems to help with that. Its obviously best to shard the data in the most used way, but occasionally you’ll need to write broadcast queries across shards. It’d be even better if those broadcast queries were executed concurrently. Well, apparently, its not that simple in MRI Ruby.

The mysql (2.7) gem stop’s the world while executing a query. A sloppy but reproducible test is here. This is not a mutex lock on the re-use of a single connection object. Eddie also reached out to the Sequel maintainer, who agrees it is likely due “to the fact that the C drivers don’t release the interpreter lock while they wait for a response from the server.” JRuby, or more accurately JDBC, acts as expected. We even tested DataMapper’s DataObjects::MySQL, as it appears they’ve re-implemented the mysql gem. Unfortunately it suffers from the same stop-the-world issue.

$ ruby mysql_locking_test.rb
Loading MRI MySQL
Serial: 4.00969815254211s
Multi-threaded: 4.00785183906555s

$ data_mapper=true ruby mysql_locking_test.rb
Loading DataMapper
Serial: 4.01330804824829s
Multi-threaded: 4.01132893562317s

$ jruby mysql_locking_test.rb
Loading JDBC-MySQL
Serial: 4.2802369594573975s
Multi-threaded: 1.0499329566955566s

Only in JRuby was the multi-threaded != sequential time, as expected.

Potentially unrelated, no one has touched the mysql gem in 3 years?!?!

$ gem spec -v 2.7 mysql | grep date
date: 2005-10-09 00:00:00 +00:00

So even Ruby ORM frameworks (Sequel, DataMapper) that say they’re thread-safe, are not concurrent on MRI… at least for mysql. For folks not using Rails, which already has a mutex lock higher in the stack, this must be a performance issue. For example, Merb + DataMapper + MySQL. If there is a 2s SQL query, all threads in that process stop for 2s.

Can others verify? “select sleep(2) from dual;” is a great way to test for this.

UPDATE: Multiple have asked, so to clarify. The sample code here creates a new connection PER thread. The mysql docs states: “Two threads can’t send a query to the MySQL server at the same time on the same connection”, but the test is creating a new connection object per thread, so that should not apply.

UPDATE 2: A couple folks have mentioned asynch drivers. One for mysql and one for postgres, but I believe they’re based on NeverBlock, which is Ruby 1.9 only. That sounds like awesome progress. What’s the realistic ETA for folks running 1.9 in a production environment? At least until Christmas for a 1.9 official release?

UPDATE 3: Looks like lots is happening here. There now is a mysql driver that supports async/threaded operations on 1.8! See the NeverBlock MySQL project. Looking forward to testing this in a production environment.

Animoto is a great idea. They take your photos and create a production quality video to the music of your choice. Its the end of those boring slide shows, for good.

We had the pleasure to work with the Animoto guys to launch their Facebook application, “Animoto Videos”, which leveraged all of the existing photos on Facebook. The growth was amazing.

(From a recent AllFacebook article here)

Scaling an application from a few hundred users to over a million in just a few days isnt easy, but we had a great team. Their backend rendering farm lived in Amazon’s Cloud, and the growth was so impressive, Jeff Bezos even spoke about them at Y Combinator’s Startup School just a few weeks ago. From 50 EC2 instances to over 4k in only a few days. See the video below.

It was a pleasure working with the entire team from Animoto, RightScale, and Amazon. See their blog posts about the application here, here, and here.

I’m sure I’ll cross paths with many of you at RailsConf. First round of beers is on me.