Avoid memory leaks in your ruby/rails code and protect you against denial of service

We heard a lot about that Ruby is cool cause we do not have to care about memory, the garbage collector does it for us. Well, that’s kind of true, but this does not mean we can write code without keeping in mind on what’s is going on under our ruby code.

Ruby symbol memory leak

We all know that using symbols instead of strings is a good practice to have, it’s faster and it saves your memory. Yes but at what price ? Symbols are faster in part cause they are created just one time in memory, that’s great ! But then ? they will stay forever in memory

That means, do not convert everything in symbol ! Be sure to well know what you are converting in symbol.

Example: Somewhere in your app, you apply a to_sym on an user’s name like :

hash[current_user.name.to_sym] = something

When you have hundreds of users, that’s could be ok, but what is happening if you have one million of users ? Here are the numbers :

kwi:~$ irb
ruby-1.9.2-head >
# Current memory usage : 6608K
# Now, add one million randomly generated short symbols
ruby-1.9.2-head > 1000000.times { (Time.now.to_f.to_s).to_sym }

# Current memory usage : 153M, even after a Garbage collector run.
# Surprisingly, on Ruby 1.8.7-p249,
# the VM only grow up to 33M, but that's still a lot !

# Now, imagine if symbols are just 20x longer than that ?
ruby-1.9.2-head > 1000000.times { (Time.now.to_f.to_s * 20).to_sym }
# Current memory usage : 501M

Furthermore, NEVER convert non controlled arguments in symbol or check arguments before, this can easily lead to a denial of service.

Example: You have a website with a locale parameter in order to localize your content and you have something like this in your application controller:

before_filter :set_locale

def set_locale
  I18n.locale = params[:locale].to_sym
end

It’s really simple to call thousand of times your website with a long params[:locale] and make your application bloat !

By the way, it looks like the I18n gem converts automatically the locale in symbol, so be sure to check if the locale is valid before assigning it !
Here is the link: http://github.com/svenfuchs/i18n/blob/master/lib/i18n/config.rb#LID8

If you need to control your number of allocted symbols in your app, you can use Symbol.all_symbols.size. Add this to your log to see if you are leaking symbols over time ! (This can be a good measure to add in Newrelic; Newrelic guys, are you reading ? :)

Reference to objects leak

This leak is a fake one but can grow rapidly in your app.
It happens when you keep a variable in your code referering objects, and these objects are also referencing objects, and etc…

This often happen when using $variable or @@variable as they stay forever in memory.
Here is a little example :

# Memory usage at irb launch: 6320K

class HelloIamLeaking
  @@an_array = []

  def initialize()
    # Put something big in the array
    @@an_array << "hello world" * (4**10)
  end
end

x = HelloIamLeaking.new
x = nil # So no more HelloIamLeaking instance in our code
GC.start # Run the garbage collector to be sure this is real !
# Memory usage after : 17M

ruby-1.9.2-head > ObjectSpace.each_object(HelloIamLeaking) {|x| p x }
 => 0
# So we have no more instance of HelloIamLeaking
# but the class variable remains in memory.

Ok, this is a completely logical and dumb example but this show you the principle.

And this can grow exponentially if you have objects linking to huge array or datasets, they will never be garbage collected if just one object in your code is still referencing the source object.

This will consume your memory, but not only, this will also consume your cpu time as when the garbage collector runs, it looks on every single object, and the more objects you have, the more it spent time looking at them …

To resume reference leak : with time, it’s grow in memory and slow down dramatically the garbage collector running time.

If you want to read more about reference leaks, read the awsome post descent into darkness on the blog of Joe Damato.

Update: Find this leaks easily using the memprof gem and by using memprof.com (awsome stuff again by Joe Damato)

My app is still bloating !

After that, if you have still ruby/rails process bloating, be sure to use the latest version of gem that are using C code, they can be an easy source of memory leak.

And, this is obvious, but be sure to not load huge dataset in memory at one time ! (use find_in_batch instead for example)

Then, If you want more control over the memory allocation, here is a good link for tune up the heap easily and control your ruby process growth.



Thanks for taking the time to read and I hope this article will help you to reduce your memory consumption !

  • Share/Bookmark

Comments (27)

Thanks for the tips !

I’ll check my code ASAP and let you know if my application feels better !

Nice article. Just FYI, GC.start doesn’t FORCE a GC run, it just “suggests” it.

Thanks for the precision Seydar, and glad you read my article !

Very interesting, thanks!

So it is really bad practice to convert everything to symbols?

Not really, it’s just depend on what you are converting to symbols, you just be sure to control that !

[...] Скажу честно, после прочтения статьи нашел несколько промахов в своем коде, о которых даже не задумывался раньше. И так, статья Avoid memory leaks in your ruby/rails code and protect you against denial of service [...]

I find it almost impossible to completely control all possible leaks in ruby code. I18n.locale is just one examples, but there are many plugins and gems that leak symbols. So I prefer to simply limit the number of requests any given application server instance can provess. For example, when using Fusion Passenger I set PassengerMaxRequests to something like 200-500, so that I’m sure it can’t leak too much during its lifetime.

Hi Vitaly,

Yes you are right, it’s very hard to control all these possible leaks, but limiting to 200-500 requests looks like a litlle bit too small ! In an heavy loaded environment, this can be achieve in a few seconds easily, this will reload your rails app just too often which will slow down your server.

[...] גיליתי השבוע כי ברובי יצירה של Symbol (שם של מחלקה) היא סוג …. עכשיו חשוב להבין שזה feature מסויים, אבל השימושים שאנחנו [...]

It can be pretty painful to find the cause. We followed a trivial approach.

http://siliconsenthil.in/blog/2013/01/19/how-we-debugged-rails-memory-leak/

Great info. Lucky me I found your site by accident (stumbleupon).
I have saved it for later!

What’s up, I would like to subscribe for this web site to obtain newest updates, so where can i do it please help.

Here is my webpage … Estes Park Colorado Winter Activities (nataliepayne.Bravesites.com)

チャコ z2

excellent issues altogether, you simply won a new reader. What would you recommend about your post that you made some days ago?
Any positive?

What’s Taking place i’m new to this, I stumbled upon
this I have found It absolutely helpful and it has aided me out
loads. I’m hoping to give a contribution & help other users like its aided me.

Good job.

Stop by my weblog Brain Boost

I have read so many articles about the blogger lovers but this paragraph is genuinely a nice
piece of writing, keep it up.

This is not a suit only worn when something unlikely is happening.
Another of the aquaponics questions that is related to the ‘organic’ factor is pesticides.
It will be diamond robbery, which is to take place London.

Big business is for those who don’t know how to have a HUGE business.
Mulch mowers cut grass clippings twice into fine particles
that fall through the turf and decompose
quickly. It’s really easy to do, and you will not only accomplish a lot more,
you will also find greater fulfillment in your life.

These two forms of using the word’roll form’ will be used indistinguishably throughout this manuscript.
This article will take you through the steps to safely remove
the hinge and roller assembly and replace the roller.

That’s the same fisheye effect that you get with this
lens.

constantly i used to read smaller content that as well clear their motive,
and that is also happening with this post which I am reading at this time.

my blog post; Oceans Bounty review (arisavin.com)

Great article! That is the type of info that are
meant to be shared across the internet. Disgrace
on Google for now not positioning this submit higher!
Come on over and visit my site . Thanks =)

whoah this blog is excellent i like studying your posts.
Stay up the good work! You know, lots of people
are hunting around for this information, you could aid them greatly.

For hill work it is only useful if it is being
used horizontal, used vertically, with only one hand, a hill stick is really of no use, yet this is how most people seem to use one.

You can make fishing a great time with your closed ones or a quality time
for yourself but make sure you carry all the things like fishing jackets,
trousers, vests or anything else that you may need for your fishing session. During
a fishing trip in the ocean, the fact remains that you just don’t
have an idea of the things you could face.

This has encouraged the manufacturers to find a reliable solution to help consumers make the best possible
use of the product they acquire. Before we get into the remedial measures,
let us first understand what exactly a mold is and how do
they become our uninvited guests. When mold is found, a homeowner must contact certified mold remediation companies to
find a competent contractor to handle the situation.

Here is my page :: prefab Red Oak Staircases

This paragraph gives clear idea in support of the new users of
blogging, that really how to do blogging.

Great article, totally what I waas looking for.

My page; onlpine (Benny)

Once you create your Expert Page, you can also begin networking with other Experts on. My OS
is Windows, so I employ the Windows adaptation. Google offer a number of templates for your Adsense including
a sponsored mini search engine – ‘Adsense For Search’ and
‘You – Tube Video Units’.

Write a comment