After porting our production application to Rails 2.2, we noticed a major memory leak.
Beforehand, monit would restart instances a handful of times a day. After Rails 2.2, monit restarted instances THOUSANDS of times a day.
This is a graph of one of our haproxy instances a couple days ago.
We looked at everything, including time spent rewriting Routes, thinking that was the culprit.
This morning, we all sat around and fought the issue old school style. binary debugging… and found it: AssetTagHelper. See the patch here.
The new thread-safe asset tag code keeps a static AssetTag::Cache = {} of all asset_tags created (css,jss, and all images).
Internally, each AssetTag object keeps a reference to the controller and template objects, and in turn all instance variables you created in your request.
What does that mean? Say you have a people controller, that loads a person and their stuff, and you show images of their stuff via image_tag().
class PeopleController < ApplicationController
def show
@person = Person.find(params[:id])
@stuff = @people.stuff.find(:all, :limit => 30)
end
end
When image_tag() is called, it does rails magic to append file extensions, asset_ids, and the like. To be smart, it caches those objects so it doesnt hit the file system
to figure all that out on every request. The problem is it puts it in a static cache, AssetTag::Cache.
So each PeopleController instance has a reference to 1 person and 30 Stuffs. After 1000 people look at their pages, or better yet google crawls your site, you have 1k @controllers with a total of 1000 People Objects, and 1000*30 Stuff objects. This would normally be fine. The objects leave scope and get GC’ed. But, if you generate an image tag to an unique asset, AssetTag puts that into a cache, AssetTag::Cache, with a reference to the @controller of the request. So All People and their Stuff are kept around forever, unable to be GC’ed…. every time a unique image is rendered via AssetTag. Eventually monit has to kill the process.
The patch we just submitted does 2 things.
1) It now keeps a cache of just the modified path strings, caching the file access stuff. If you have tons of local images, reference them by fully qualified host. Thats better for lots of reasons. Cookie-less asset hosts with multiple subdomains FTW!
2) It stops caching absolute URL paths. You cant do anything on the filesystem to verify them, and keeping a cache of those would also grow unbounded. We have millions of items in our system, each with a reference to an image.
Here is a graph of that haproxy today… Sleeping………
In order to do some testing of your own, here’s a simplistic after_filter you can add to application.rb (or is it now application_controller.rb?). Make sure you run this in production mode or with cache_classes = true. As you click around your site, you should see that the cache retains references to controller instances, just to name a few. After you apply the patch, you’ll see the cache is just strings.
def assettag_cache
puts "-"*80
puts "[AssetTag::Cache] Now #{ActionView::Helpers::AssetTagHelper::AssetTag::Cache.size} items"
ActionView::Helpers::AssetTagHelper::AssetTag::Cache.values.each do |asset_tag|
if asset_tag === ActionView::Helpers::AssetTagHelper::AssetTag
puts " [Asset] #{asset_tag.instance_variable_get("@source")} #{asset_tag.instance_variable_get("@controller").class.to_s}"
else
puts " [Asset] #{asset_tag}"
end
end
end
Ohh… and we havent given up on routes… Warren is working on some very interesting enhancements to rails routing. Looking forward to blogging about that soon.
Subscribe