Vital Logs

I got the error below recently. Good luck debugging the error message below.

The bug was a bad require path. This can also happen with missing gems in your deployed application.

=> Booting Mongrel (use ’script/server webrick’ to force WEBrick)
=> Rails application starting on http://0.0.0.0:3000
=> Call with -d to detach
=> Ctrl-C to shutdown server
** Starting Mongrel listening at 0.0.0.0:3000
** Starting Rails with development environment…
Exiting
c:/medhelp/InstantRails/ruby/lib/ruby/gems/1.8/gems/rails-2.0.1/lib/commands/ser
vers/mongrel.rb:16: warning: already initialized constant OPTIONS
c:/medhelp/InstantRails/ruby/lib/ruby/gems/1.8/gems/rails-2.0.1/lib/commands/ser
vers/mongrel.rb:19: undefined method `options’ for []:Array (NoMethodError)
from c:/medhelp/InstantRails/ruby/lib/ruby/site_ruby/1.8/rubygems/custom
_require.rb:32:in `gem_original_require’
from c:/medhelp/InstantRails/ruby/lib/ruby/site_ruby/1.8/rubygems/custom
_require.rb:32:in `require’
from c:/medhelp/InstantRails/ruby/lib/ruby/gems/1.8/gems/activesupport-2
.0.1/lib/active_support/dependencies.rb:496:in `require’
from c:/medhelp/InstantRails/ruby/lib/ruby/gems/1.8/gems/activesupport-2
.0.1/lib/active_support/dependencies.rb:342:in `new_constants_in’
from c:/medhelp/InstantRails/ruby/lib/ruby/gems/1.8/gems/activesupport-2
.0.1/lib/active_support/dependencies.rb:496:in `require’
from c:/medhelp/InstantRails/ruby/lib/ruby/gems/1.8/gems/rails-2.0.1/lib
/commands/server.rb:39
from c:/medhelp/InstantRails/ruby/lib/ruby/site_ruby/1.8/rubygems/custom
_require.rb:27:in `gem_original_require’
from c:/medhelp/InstantRails/ruby/lib/ruby/site_ruby/1.8/rubygems/custom
_require.rb:27:in `require’
from c:/medhelp/InstantRails/ruby/lib/ruby/gems/1.8/gems/activesupport-2
.0.1/lib/active_support/dependencies.rb:496:in `require’
from c:/medhelp/InstantRails/ruby/lib/ruby/gems/1.8/gems/activesupport-2
.0.1/lib/active_support/dependencies.rb:342:in `new_constants_in’
from c:/medhelp/InstantRails/ruby/lib/ruby/gems/1.8/gems/activesupport-2
.0.1/lib/active_support/dependencies.rb:496:in `require’
from ./script/server:3

I noticed something while looking at our server logs; a large number of exceptions are thrown as a result of the googlebot visiting the pages where they occur.. So it is allowing us to find obscure bugs that are data-specific that will be a nightmare to find otherwise. While the impact is low, I thought it was amusing to see google doing some of our QA for us.

A couple of days ago, I gave Ajay really bad advice. He was changing a piece of code to order the results by created_at instead of ids because ids and my code review boldly stated:

Ids will be chronologically ordered 99.9999% of the time the ids will be chronologically ordered.

While this might be true right now, it is really suboptimal. I realized that when I was thinking about the best way to setup our new database server. It occurred to me that we might want to setup a master-master replication scheme for MySQL server. Once we do that, each MySQL master will have its own range of ids to use when assigning a new automatically generated id to a record being inserted. For example, server A will be able to use ids within the range 1 - 99999 and server B 100000 to 199999

Once that happens, the chronological order of ids will be less continuous and might look like: 324, 100231, 325, 326, 100232,et.

So the lesson I admit to learning is: never sort by IDs when you mean to sort by date. In fact, I can’t think of any reason why you would ever want to sort by ID.

Cache is money to MedHelp as it helps us scale by reducing cpu, I/O, roundtrip times, etc. This is critical as we continue to grow from millions of users to tens of millions of users and beyond.

There are five popular ways to cache:

1. page caching
2. action caching
3. fragment caching
4. model caching
5. in-memory computational cache

Page Cache

Page caching caches static pages so the request does not need to hit your rails server. I can imagine, every page on wikipedia is page cached and invalidated on update. This has huge performance benefits as we can configure our web server in front of rails to return the cached html page. We could use page caching for our medical dictionary pages. Almost all our pages have or will soon have dynamic content so the value of page caching appears low.

However, an interesting use of page caching is to leverage the fact that all pages viewed by a non-logged-in users looks the same. Since, the majority of our traffic are new users from google this will have a large impact. The trick here is that we have to identify that a request is from a non-logged in user from inspecting the cookie in our front-end web server.

Methods: caches_page and expire_page or sweepers

Action Caching

In action caching, the request hits the rails server and passes through all the filters. This is useful when you require auth to differentiate logged in and out and have an authorization filter. I cannot think of a case where we would use this form of caching since we use dynamic content on almost every page.

Methods: caches_action, expire_action

Fragment Caching

Fragment caching saves rendering of portions of your view. This method is interesting as we could cache the middle div (user journals) in the people page and invalidate using a time to live.

Another interesting technique, is to use identifiers on the cache fragments so that in the controller you do not execute code related to that fragment (ie, save db calls and computations). This does poke a hole in the MVC model as it creates an additional coupling between view and controller. Here is a code example:

View:

<% cache( :controller => :post, :action => :show, :subject_id => @post.subject_id ) dp %>

  <% # beautifully written medhelp code %>

<% end %>

Controller:

def show

  unless read_fragment(
:controller => :post, :action -> :show, :subject_id => @post.subject_id  )

    # lots of db calls

  end

  # code you need for other fragments on the page

end

def edit

  ...

  expire_fragment(:controller => :post, :action -> :show, :subject_id => @post.subject_id)

end

Methods: <% cache do %> in the view expire_fragment

Model Caching

Model caching is when we store ActiveRecord and db results to save db calls which is typically the bottleneck of our web appplication. We currently use this everywhere and we are very strict in reviewing this in our bottoms up design reviews and code reviews prior to check-in. For example, we cache all users. Soon, we will be caching all posts, subjects, and forums. All hail memcache.

Methods: CACHE get/set, act_as_cachable

In-memory computational cache

This is where we save our computation in a data structure. I almost forgot this since it is second nature. An example of this is caching the key words -> links data structure that is used to link-ify user generated text. See our medical terms highlighting in user journals and forum posts.

Methods: CACHE get/set, session set/get, class variables

We have been battling with a dilemma of missing white spaces in IE6 for a while now. usernames that have been hoverized (your profile appears when you hover over them) start to stick to the text after them no matter how many white spaces you add and where. The only solution is to add an explicit non breaking space (nbsp) but then the interface looks clunky, due to the disproportional spacing.

It turns out that the reason is that IE6 doesn’t like a div inside inline text even if its display is set to none. Once I changed the div to a span, the white spaces were honored and the username link stopped sticking to the surrounding text. It also doesn’t matter if the span with display:none has divs inside it, as long as the outer tag of the non displayed html is not a block type element.

Example:

Link sticks to text after it: <a href=’somewhere’>link text</a><div style=’display:none;’>hover box</div> sticky text

link doesn’t stick to text after it: <a href=’somewhere’>link text</a><span style=’display:none;’>hover box<div>some internal div that doens’t make a difference</div></span> non-sticky text

So I was trying to use the memcache increment method during my implementation of the view counter. Whenever I issue a CACHE.incr, CACHE.get starts to fail on that key. I found out what the problem is.

When get is called without specifying the raw argument, it defaults to false, which means that the memcache-client lib will marshal the value passed before sending it to memcache. Then when we issue an incr to the memcached server, it tries to increment the marshaled version which is a binary sequence representing a ruby object (Fixnum in this case) and messes it up.

The solution is to pass the raw parameter to MemCache.set as true, which will prevent the marshaling (later we should call set with a true raw as well) which will give memcached an integer (although treated as string on the ruby side) that it can increment safely.

I created a raw_get, raw_put and raw_incr that can be used together. They are nicely contained in a Cache module (yes, I’m using the same name as the memcache_util.rb module) under our lib folder.