general performance question

[Brian Le Roy]

I'm running top and when I run my app - I see the user CPU utilitization
hit ~70%, but the ruby process only shows about 3%. Is this observation
indicative of inefficient code? And to be complete, the user CPU
utilization drops to 0% when I stop the app - so it's me.

Thanks Ruby gurus!

brian

 

[Robert Klemme]

I'm running top and when I run my app - I see the user CPU
utilitization hit ~70%, but the ruby process only shows about 3%. Is
this observation indicative of inefficient code?

Hm, might be that the rest of the 70% are spent in the kernel (doing IO
probably). Do you do a lot IO? Or do you start another process (or
multiple processes) from your Ruby script? Did you have an eye on your
disk LED? I'm not too familiar with top but I think I remember that it
sometimes spits out odd values - might be another explanation.

And to be complete,
the user CPU utilization drops to 0% when I stop the app - so it's me.

Guilty!!!

Kind regards

robert

 

[Brian Le Roy]

I am doing some database queries using the MySQL-Ruby library, not DBI.
I disabled logging, so it's not IO. Are there any differences between
MySQL-Ruby and DBI?

 

[Brian Le Roy]

no, but the table has 140K records. the disk isn't crunching, so the
index should be okay. i was thinking it would be passing around the
database connection object causing overhead. i took the result set
object and created a straight ol' array for passing. no improvement -
so maybe there's nothing to do. thanks for all the responses - i'll
keep messing with it and if i find something that improves it, i'll let
you know!

 

[Reid Thompson]

no, but the table has 140K records. the disk isn't crunching, so the
index should be okay. i was thinking it would be passing around the
database connection object causing overhead. i took the result set
object and created a straight ol' array for passing. no improvement -
so maybe there's nothing to do. thanks for all the responses - i'll
keep messing with it and if i find something that improves it, i'll
let you know!

It may may no difference, but you could try the same with postgresql, or
maybe even sqlite, to see if it is mysql thats the issue.

 

[Robert Klemme]

It may may no difference, but you could try the same with postgresql,
or maybe even sqlite, to see if it is mysql thats the issue.

Before he does that I'd try a different query frontend to see whether
performance characteristics stay the same (i.e. time is spent in the DB,
which would be my guess). The overhead is much smaller compared to
installing another DBMS, transferring DDL and data to that.

Kind regards

robert

 

[Brian Le Roy]

Actually guys, I took out the queries - except for one that is run once.
Same behaviour in top. I'm going to start marking out the app with
the benchmark module and hopefully that will shed some light on it!

 

[Robert Klemme]

Actually guys, I took out the queries - except for one that is run
once. Same behaviour in top. I'm going to start marking out the
app with
the benchmark module and hopefully that will shed some light on it!

Note also that there is ruby -r profile. That way you don't have to
manually change your code.

robert

 

[Brian Le Roy]

Thanks Robert!

I think I found the culprit.

With 20 records in the result:
total ms/call
12300.00 Mysql::Result#each

With 1 record in the result:
total ms/call
600.00 Mysql::Result#each

The query behind this result set is indexed properly. Not sure why this
takes so long - shouldn't it be as fast as a regular array.each call?

 

[Joe Van Dyk]

That's odd that it would take so long with only 20 records in the
result set. With 140k in the result set, that would be expected, I
suppose. But not 20.

Are you using the C mysql bindings, or the pure Ruby bindings?

 

[Robert Klemme]

Thanks Robert!

I think I found the culprit.

With 20 records in the result:
total ms/call
12300.00 Mysql::Result#each

With 1 record in the result:
total ms/call
600.00 Mysql::Result#each

The query behind this result set is indexed properly. Not sure why
this takes so long - shouldn't it be as fast as a regular array.each
call?

Unlikely, as the data has to be fetched over the network (even if it's
local). Even with prefetching etc. this is slower than iterating an array
in mem. Also, the accumulated time of each depends on the complexity of
the operation in the block - that's sometimes overlooked. Note that #each
is called only once per iteration but the block is invoked multiple times.

Kind regards

robert

 

[Brian Le Roy]

I compiled the C bindings. I'll try building arrays and pass those.
Thanks for all the help!