Another further thought would be:
imagine that a db row will only have one column accessed for it [quite common for web driven stuff--you want the content of the blog entry, or the name of the school, and that's all you want].

Could be optimized maybe.
ex: have a flag per row which says “the cache has ever been hit” then check that each time–if not you can avoid the cache check [or does it already do this?]
Maybe you could cache the index of the most recently requested column.
Anyway just thinking out loud.
Thanks for your help. slim_attributes is WAY faster than even a C based all_hashes.
-=R

Just throwing these out:
instead of using malloc you may want to use ruby_xmalloc [so ruby can keep track of the memory growth]
Also, for a small speedup you may want to cache the ids for your “@variables” and then can call rb_ivar_set

Looks good though. I might roll it into an upcoming version of mysqlplus :)
-=R

MysqlPlus is a fork of 2.8pre4, so also incorporates it.

Unfortunately the current gem is an earlier version and thus still creates more garbage than necessary. But anyway if people do want it they can upgrade.

I’m hoping that the rails team will help with the slow rendering time. One can hope.

Also does this plugin work with 2.1?
Thanks!
-=R

I agree there may be some performance value in checking updated_at in C before anything is set up as ruby objects so an in-process cache can be used, but I fear it won’t just drop-in like slim-attributes does now.

Anyway, we find that only part of the performance issue is solved by doing things at the AR level. Much time is also spent processing and rendering. So we also try to avoid this by using cached fragments whose expiry is tied to particular DB models or DB rows (whatever you specify).

If an object changes or any object in a model changes then we automatically invalidate a bunch of cached items as appropriate (done via after_save hooks). It’s all automatic, and we know whether we need to run controller code also based on whether the view cache is valid or not.

For this we employ a home grown system that is strikingly similar to this, although they were developed independently:
http://blog.evanweaver.com/files/doc/fauna/interlock/files/README.html

Other options would be: tracking somehow the time of the last write to each table [memcache? mysql?] to know when to invalidate local caches–which would be annoying because of the setup involved.

Possibly use an updated_at column or compute a hash or checksum of each row to make sure it matches our cached version [in C, or use mysql's md5()].

In terms of fitting it into RAM, it appears that with our app, rails by default uses 50MB, and with all of the “common” objects in memory it uses 74MB. So a little bit of a hit but probably not too bad. The mongrels tend to grow to about 100MB total, over time, anyway, so it’s not out of scope. Also we could use an LRU cache to limit memory used.

With regard to a time expiry, that’s also a possibility [1].

An interesting thought is that if you only had one mongrel and only updated your DB through AR, you could just LRU cache everything you ever use. And never worry about having to refresh them. Which would be scary but imagine the speedup :)

Just some ideas. For all I know somebody has written this plugin I just don’t know about it :)

Take care.
-R
[1] http://www.nongnu.org/pupa/ruby-cache-README.html

In our app we do, for instance, cache in-memory the results to queries to our Countries table. For various reasons the available countries we wish to show do in fact change, but not that often. We use a 5 minute timeout for this cache, which means that if we change something it will show up reasonably quickly. I’ll post the code if you are interested.

My latest thought with this style speedup is I wonder if you can’t do something like object cacheing…
background:
if I remember correctly,
def action_name
@@my_setting ||= Setting.find(:some_thing)
end
is actually quite fast for every run [except the first :P ]

Furthermore, at least where I work, we have a few small’ish tables that we constantly query for ’similar’ data. Say one table with 100, and another with 3000 are basically queried over and over, but typically with differing queries, and different elements returned.

Also note that, as mentioned with hash_extension, lots of time you just want to view an AR object, not edit it.

So I wonder if there’s an object cache around somewhere that does something like the following.
a = Program.find(:some_conditions, :cache_the_individual_objects => true)
b = Program.find(:some_conditions_which_will_make_it_share_some_objects_with_a, :cache_the_individual_objects => true)

When it runs b, it gets back the results, then for each entry in the results, instead of immediately creating a new AR object, it first checks to see if that exact object is already existing somewhere [pre-cached]. If it is then just use it [avoid creating an AR object--just re-use the existing one].

Now throw this all into an LRU cache to not cache too much and, for some queries from a common small set, I could see this resulting in a speedup.

I suppose there could even be further optimizations, like querying ‘only the id’ from the DB [then figuring out which ones aren't cached, creating objects for them] and there could be other optimizations like a C based ‘heap based’ cache, freezing the AR instance attributes, etc.

Thoughts?
-R

The meat of it is like this (there are 100 items in the DB):

2000.times do |n|
    product = Product.find(:all)[n % 100]
    x = product.name
    y = product.comment
    z = product.created_at
 end

Just running it now gives these figures:

Without slim-attributes: 11.133781s
Without slim-attributes with hash_extension: 9.673203s
With slim-attributes: 6.658067s
With slim-attributes with hash_extension: 5.422445s

Without slim-attributes – hash_extension improvement: 13.12%
With slim-attributes – hash_extension improvement: 18.56%
Without hash_extension – slim-attributes improvement: 40.20%
With hash_extension – slim-attributes improvement: 43.94%
With hash_extension & slim-attributes improvement over plain AR: 51.30%

The table used has over 40 columns – this is realistic for our live application. If you use a smaller table then hash_extension begins to make a bigger difference than slim-attributes.

I will look at writing a postgresql implementation, it shouldn’t be too hard to port.

Roger: hash_extension naturally uses slim-attributes when slim-attributes is installed because it uses connection.select_all. This is what eventually calls all_hashes.

The benefit in hash_extension is avoiding the active record object creation. The benefit of slim-attributes is avoiding the creation of a hash, and that the attributes are lazily created as ruby objects on demand. And you get ultimate performance by combining the two – assuming you don’t need active record objects.

Also is there a benefit of using hash_extension in that ‘later’ accesses are faster? [like calling instance.name would be faster using hashes]. Thoughts?