Pennysmalls

I’ve spent a considerable amount of time with various tools attempting to figure out why it is that our thin processes (and mongrels before them) grow so egregiously. Typically they reach about 450Mb in a day, after which we restart them via monit.

What makes them grow? Well, we are fetching a lot of stuff from the DB all the time - meaning that thousands of small strings are being instantiated - so perhaps we can attribute some growth to heap fragmentation. But we tried changing to ptmalloc3 - it didn’t help; in fact I think in our case this is rather a red herring.

In an effort to get the problem under control, I wrote a plugin to reduce the number of strings that are made, changing the implementation of the mysql library so that all_hashes actually returns fake hashes that are implemented as arrays - to prevent all those column names being saved as frozen strings (for the hash keys) for every row that is fetched from the DB. But that didn’t help much, if at all, either.

But whilst I was playing with ruby with valgrind, I noticed some memory going missing. At first I thought it was probably me. But with further investigation I found a simple expression that makes ruby leak.

a = eval "b=0"

It’s actually the eval that leaks - the a = is not really needed, but it makes the leak show as a definite leak as opposed to a possible one in this simple one liner. If you want to leak a lot of memory, this is the way:

def grow
  for i in 1..100
    eval "b#{i}=1"
  end
end
15000.times {grow}

You can fiddle with the numbers to make it grow as much as you like.

Valgrind reports the leak like this (this one made by running the loop 5000.times):

==18706== 217,988,864 bytes in 499,985 blocks are
definitely lost in loss record 6 of 6
==18706==    at 0x4A05AF7: realloc (vg_replace_malloc.c:306)
==18706==    by 0x432398: ruby_xrealloc (gc.c:151)
==18706==    by 0x465E9C: local_append (parse.y:5649)
==18706==    by 0x465F64: local_cnt (parse.y:5667)
==18706==    by 0x4646AC: assignable (parse.y:4902)
==18706==    by 0x458E80: ruby_yyparse (parse.y:844)
==18706==    by 0x45E5F4: yycompile (parse.y:2606)
==18706==    by 0x45E8F4: rb_compile_string (parse.y:2676)
==18706==    by 0x41DDF3: compile (eval.c:6412)
==18706==    by 0x41E289: eval (eval.c:6493)
==18706==    by 0x41E817: rb_f_eval (eval.c:6611)
==18706==    by 0x41C765: call_cfunc (eval.c:5700)
==18706==    by 0x41BB04: rb_call0 (eval.c:5856)
==18706==    by 0x41D291: rb_call (eval.c:6103)
==18706==    by 0x415182: rb_eval (eval.c:3494)

The memory is allocated when ruby is expanding its local variable table in the parser. But what I don’t know yet is exactly where to add a call to free to release that memory. I’m hoping that someone over at ruby-core can help. Interestingly, it appears that Rubinius leaks too, which is surprising given that it is a completely new implementation.

I’m not the only one to have found a leak in Ruby lately - I wonder if the issues with god are related to this?

Fixing this leak may not completely cure our Rails memory growth problem (probably won’t), but at least it will help.