Executive Summary
If you are running an application using a version of Ruby on Rails prior to 2.3 (including 2.2.2), and serving it with mongrel, you are at risk of serving up incorrect internal server pages. Patch one file in rails in an uncontrovercial way and be happy again. Skip to the end to see the patch.
The long and boring details
I had a problem with some infrastructure that was causing exceptions to (occasionally) be raised accessing the session store. When this happened, it would result in rails dumping a message in the production.log with a distinctive keyword: “/!\ FAILSAFE /!\“.
Rails would then try to return a 500 status to the client, but in fact would return a 200 status with malformatted content:
Status: 500 Internal Server Error
Content-Type: text/html
<html><body><h1>500 Internal Server Error</h1></body></html>
Those two lines at the top that look like headers aren’t … they’re the beginning of the body of the response. See the actual headers with ‘curl -o /dev/null –verbose’:
> GET / HTTP/1.1
> User-Agent: curl/7.16.3 (powerpc-apple-darwin9.0) libcurl/7.16.3 OpenSSL/0.9.7l zlib/1.2.3
> Host: localhost:3000
> Accept: */*
>
< HTTP/1.1 200 OK
< Connection: close
< Date: Tue, 03 Mar 2009 19:03:49 GMT
< Content-Type: text/html
< Content-Length: 122
Infrastructure upstream treated that response as cacheable, so the intermittent error got amplified by the lack of cache error. Lots of clients wound up successfully fetching a document that said “500 Internal Server Error”.
We don’t like that. We fixed the problem that caused the original “/!\ FAILSAFE /!\”, but a little digging revealed that any failsafe produced by rails served by mongrel for any reason would have this problem.
More digging revealed that the problem has to do with the way mongrel and rails interact. In fact it has to do with the fact that mongrel subverts the CGI interface it flanges to rails with, and does so in a way that’s incompatible with the one rails code path that produces that FAILSAFE. Rails tries to do the right thing, but fails because of mongrel’s cleverness.
All of this has been tested with Rails 2.0, 2.1, and 2.2.3 with mongrel 1.1.4. Rails 2.3 doesn’t have this problem.
Mongrel is the application server that accepts http connections and calls down into rails to generate responses. It does this by constructing a Mongrel::CGIWrapper object (acts like and is-a Ruby standard libary CGI object) and an IOString (has a ‘write’ method) and passing it down to rails.
cgi = Mongrel::CGIWrapper.new(request, response)
…
Dispatcher.dispatch(cgi, ActionController::CgiRequest::DEFAULT_SESSION_OPTIONS, response.body)
Mongrel has specific expectations about how that CGI will be used, and the usual code path through rails winds up in CgiResponse#out, where the output.write method gets called twice, once to write out the headers, and again to write out the body.
class CgiResponse < AbstractResponse #:nodoc:
…
output.write(@cgi.header(@headers))
…
output.write(@body)
This might lead one to assume, as the author of the FAILSAFE handler did, that one could just write out headers and body as a whole document:
fallback_output.write “Status: #{status}\r\nContent-Type: text/html\r\n\r\n#{body}”
Too bad, because mongrel is really depending on rails calling the *header* method on the CGI it passes down and using the return from that as the header portion that gets written to the output stream.
Mongrel::CGIWrapper remembers the headers passed into CGI#header, and returns an empty string! Then when mongrel goes to do the actual transmission it uses the stored headers.
Now, the FAILSAFE handler didn’t bother using the header method to format the two trivial headers, it just wrote the whole message to the output stream. Mongrel assumed that since it didn’t know what headers it should send (or even what the status was) that everything was OK, and it should send its default 200 status headers and the body.
FAIL
Once you see the problem (or if you took the advice at the top of this post simply skipped to the end) the solution is quite straightforward. This approach treats the CGI and IOString passed down from mongrel in the way they’re expected, and mongrel does the right thing with it. Since it causes the failsafe_response to behave the same as the rest of rails, it should work in any application server. For all I know others have the same problem.
--- a/vendor/rails/actionpack/lib/action_controller/dispatcher.rb
+++ b/vendor/rails/actionpack/lib/action_controller/dispatcher.rb
@@ -54,13 +54,19 @@ module ActionController
end
# If the block raises, send status code as a last-ditch response.
- def failsafe_response(fallback_output, status, originating_exception = nil)
+ def failsafe_response(fallback_output, status, originating_exception = nil, response = nil)
yield
rescue Exception => exception
begin
log_failsafe_exception(status, originating_exception || exception)
body = failsafe_response_body(status)
- fallback_output.write "Status: #{status}\r\nContent-Type: text/html\r\n\r\n#{body}"
+ if response
+ response.headers['Status'] = status
+ response.body = body
+ response.out(fallback_output)
+ else
+ fallback_output.write "Status: #{status}\r\nContent-Type: text/html\r\n\r\n#{body}"
+ end
nil
rescue Exception => failsafe_error # Logger or IO errors
$stderr.puts "Error during failsafe response: #{failsafe_error}"
@@ -184,7 +190,7 @@ module ActionController
end
def failsafe_rescue(exception)
- self.class.failsafe_response(@output, '500 Internal Server Error', exception) do
+ self.class.failsafe_response(@output, '500 Internal Server Error', exception, @response) do
if @controller ||= defined?(::ApplicationController) ? ::ApplicationController : Base
@controller.process_with_exception(@request, @response, exception).out(@output)
else
What was the original issue with your session handling? We too are having this problem, but haven’t been able to determine the root cause.
Our session data was kept in memcached, using the memcache-client gem. Somehow (not sure how) we got a session in there that raised an exception trying to Marshal.load it. And the plugin didn’t rescue that exception.
So, another fix we put in was to rescue that exception
Bill,
Great stuff. One thing to pass along. If you are using cucumber, it redefines failsafe_response, so you will get ArgumentErrors (4 for 3)
It’s easy enough to patch cucumber:
cucumber/lib/world.rb:76 just add the 4th parameter defaulting to nil.
Thanks,
Andrew
Do I have to have rails in my vendor folder to use this patch? Is there a way to use the rails gem and apply this patch to it by putting something in my initializers folder?
Thanks so much for this tip. I completely answered my question & helped me write a patch for Rails 1.2.6. If anyone else is using Rails 1.2.6 & wants the patch, you can find it here: http://gist.github.com/373866
I’m having trouble understanding where to post the patch. Where does that code go?