SubscribeMonitoring Rails Apps: Pulse + More
Paul Gross from Thoughtworks recently created a pulse gem. The gem adds a simple action to your rails app, “/pulse”, which acts as a heartbeat.
The pulse gem currently defines the method as:
def pulse render :text => "OK" end
Then you can configure haproxy to monitor your application by hitting http://server/pulse and verifying the response is “OK”.
We’ve had a similar action in our applications for quite some time, although I really like the idea of externalizing it to a gem. Our implementation is slightly different though. Since so many rails applications depend on a database, I instead added:
def pulse rows = ActiveRecord::Base.connection.execute("select 1 from dual").num_rows rescue 0 render :text => rows == 1 ? "OK" : "Error!" end
IIRC, “select 1 from dual” is the fastest query you can run against a database, in MySQL, Oracle, and Postgres. And if you have mutliple databases, you can add union in the one pulse request, or have multiple actions.
Now you are not only testing your application is live, but that it can connect to the database.
So, what does haproxy do if gets back an error? Does it restart your db, or email you?
I’m not sure about the DB check ay.
What happens if all mongrels report having lost access to the DB? Haproxy will send back a ‘no response from server’ which to me means ‘no server here’. But getting an error page isn’t nice either…
If you rely on errors being emailed/sms’ed from your app as the only way to know about errors (I know lots of sites like this) then the rescue in the pulse action could prevent you from finding out about this at all.
So my question is, what do people do when haproxy can’t find a server that works?
Also, all filters should be skipped for the pulse action to make it as light weight as possible. If anyone knows how to stop an action from appearing in the log I’d love to know.