no clue

[Ara.T.Howard]

Ah, ok. In my application, there's a bunch more than 3 possible keys and
they are of differing length. I am in control of the format of the incoming
strings though, and so could modify their format to make them easier/faster
to parse. Any ideas on what would be a more efficient format for
transporting the data?
binary.

(for reference, the original string format was "id: 3, x_position: 39,
y_position: 209, z_position: 39" and in my real application, there's about
twenty different attributes that are in the string.)

Perhaps it would be more efficient to not convert the string into a hash?

All I really need to be able to do is access/display a player's data via
some mechanism, and a player's data should be updated once a second, and
there's up to 400 players. The above was the best way I could come up with
transporting and accessing the data, but perhaps there's a better way of
doing it.

something similar in spirit to this would support many players:

harp:~ > cat a.rb
class PlayerData
ATTRIBUTES = %w(
id
x_position
y_position
z_position
foobar
).each{|a| attr a}

FORMAT = 'ifffi'

class << self
def create(*a); new(a.flatten.pack(FORMAT)); end
end

attr :buffer
attr :to_s

def update buffer
ATTRIBUTES.zip((@buffer = buffer).unpack(FORMAT)) do |a,v|
instance_variable_set "@#{ a }", v
end
@to_s = ATTRIBUTES.inject(''){|s,a| s << "#{ a } : #{ send a }, " }.chop.chop
end
alias initialize update
end

id, x_position, y_position, z_position, foobar =
400, 1.0, 2.0, 3.0, 0b101010

pd = PlayerData::create id, x_position, y_position, z_position, foobar

p pd
p pd.id
p pd.x_position
p pd.foobar
puts pd


begin
require 'timeout'
n = 0
Timeout::timeout(10) do
loop do
PlayerData::create id, x_position, y_position, z_position, foobar
n += 1
end
end
rescue Timeout::Error
puts "creations per second : #{ n / 10 }"
end


harp:~ > ruby a.rb
#<PlayerData:0xb75cdf04 @x_position=1.0, @foobar=42, @id=400, @buffer="\220\001\000\000\000\000\200?\000\000\000@\000\000@@*\000\000\000", @z_position=3.0, @to_s="id : 400, x_position : 1.0, y_position : 2.0, z_position : 3.0, foobar : 42", @y_position=2.0>
400
1.0
42
id : 400, x_position : 1.0, y_position : 2.0, z_position : 3.0, foobar : 42
creations per second : 14045

not to mention the message (player data strings) will be an order of
magintitude smaller to pass around - like over a network.

cheers.

-a
--
===============================================================================
| email :: ara [dot] t [dot] howard [at] noaa [dot] gov
| phone :: 303.497.6469
| Your life dwells amoung the causes of death
| Like a lamp standing in a strong breeze. --Nagarjuna
===============================================================================

 

[David A. Black]

--8323328-731294882-1123991692=:26601
Content-Type: MULTIPART/MIXED; BOUNDARY="8323328-731294882-1123991692=:26601"

This message is in MIME format. The first part should be readable text,
while the remaining parts are likely unreadable without MIME-aware tools.

--8323328-731294882-1123991692=:26601
Content-Type: TEXT/PLAIN; charset=X-UNKNOWN; format=flowed
Content-Transfer-Encoding: QUOTED-PRINTABLE

Hi --

data.each{ |j, line|
k, v =3D -2, 0
while (v =3D line.index(58, k))
h5[j][line[(k+2)...v].intern] =3D
line[(v+2)...(k =3D line.index(44, v) || line.length)]
end
}

One is loosing readability of code if optimizing for speed
has top priority - even in ruby.

Would you lose much speed if you did ?: and ?, ?

It's interesting though -- I remember the long thread back in 2000 or
2001 about finding anagrams. The fastest one, I seem to recall, used
pack/unpack (I can't remember the details, though I believe I wrote
it), and raised similar issues.


David

--=20
David A. Black
(e-mail address removed)
--8323328-731294882-1123991692=:26601--
--8323328-731294882-1123991692=:26601--

 

[Joe Van Dyk]

On Sun, 14 Aug 2005, Joe Van Dyk wrote:
=20
=20
something similar in spirit to this would support many players:
=20
harp:~ > cat a.rb
class PlayerData
ATTRIBUTES =3D %w(
id
x_position
y_position
z_position
foobar
).each{|a| attr a}
=20
FORMAT =3D 'ifffi'
=20
class << self
def create(*a); new(a.flatten.pack(FORMAT)); end
end
=20
attr :buffer
attr :to_s
=20
def update buffer
ATTRIBUTES.zip((@buffer =3D buffer).unpack(FORMAT)) do |a,v|
instance_variable_set "@#{ a }", v
end
@to_s =3D ATTRIBUTES.inject(''){|s,a| s << "#{ a } : #{ send a }= , " }.chop.chop
end
alias initialize update
end
=20
id, x_position, y_position, z_position, foobar =3D
400, 1.0, 2.0, 3.0, 0b101010
=20
pd =3D PlayerData::create id, x_position, y_position, z_position, fo= obar
=20
p pd
p pd.id
p pd.x_position
p pd.foobar
puts pd
=20
=20
begin
require 'timeout'
n =3D 0
Timeout::timeout(10) do
loop do
PlayerData::create id, x_position, y_position, z_position, foo= bar
n +=3D 1
end
end
rescue Timeout::Error
puts "creations per second : #{ n / 10 }"
end
=20
=20
harp:~ > ruby a.rb
#<PlayerData:0xb75cdf04 @x_position=3D1.0, @foobar=3D42, @id=3D400, =

@buffer=3D"\220\001\000\000\000\000\200?\000\000\000@\000\000@@*\000\000\00=
0", @z_position=3D3.0, @to_s=3D"id : 400, x_position : 1.0, y_position : 2.=
0, z_position : 3.0, foobar : 42", @y_position=3D2.0>

400
1.0
42
id : 400, x_position : 1.0, y_position : 2.0, z_position : 3.0, foob= ar : 42
creations per second : 14045
=20
not to mention the message (player data strings) will be an order of
magintitude smaller to pass around - like over a network.

Thanks! I shall study this code (there's a few new idioms that I
haven't used before) and report back early next week on the
performance improvements.

 

[Simon Kröger]

Ah, ok. In my application, there's a bunch more than 3 possible keys
and they are of differing length. I am in control of the format of
the incoming strings though, and so could modify their format to make
them easier/faster to parse. Any ideas on what would be a more
efficient format for transporting the data?

More different keys aren't a problem to this algorithm, but still there
might be a lot better ways to transport the data. One is perhaps a
string that can be 'executed' via 'eval' directly, but that is kind
of hacky.

Using binary data is perhaps the way to go, but of course much harder
to debug.

If you find a fast implementation of xml-rpc (a ruby extension,
written in c) this might combine speed with readability.

(for reference, the original string format was "id: 3, x_position: 39,
y_position: 209, z_position: 39" and in my real application, there's
about twenty different attributes that are in the string.)

Again, this will work with the algorithm, but perhaps not fast enough..

cheers

Simon

 

[Simon Kröger]

Would you lose much speed if you did ?: and ?, ?

*g*

no, probably not. That's one of the myriad ruby goodies i didn't come
across yet. (i was thinking about ':'[0])

Simon

 

[Ara.T.Howard]

Thanks! I shall study this code (there's a few new idioms that I
haven't used before) and report back early next week on the
performance improvements.

try this one - it's another order of magnitude faster:


harp:~ > ruby a.rb
#<PlayerData:0xb75cc0b4 @to_s=nil, @data=[400, 1.0, 2.0, 3.0, 42], @to_bin=nil>
400
1.0
42
pid : 400, x_position : 1.0, y_position : 2.0, z_position : 3.0, foobar : 42
creations per second : 114797


harp:~ > cat a.rb
class PlayerData
class << self
def create(*a)
new(a.pack(FORMAT))
end
end

SPEC = [
%w( pid i ),
%w( x_position f ),
%w( y_position f ),
%w( z_position f ),
%w( foobar i ),
]

ATTRIBUTES = SPEC.map{|s| s.first}

FORMAT = SPEC.map{|s| s.last}.join

SPEC.each_with_index do |spec, ix|
at, format = spec
eval <<-src
def #{ at }
@#{ at } ||= @data[#{ ix }]
end
def #{ at }= value
raise TypeError unless self.#{ at }.class == value.class
uncache
@#{ at } = @data[#{ ix }] = value
end
src
end

def update buffer
uncache
@data = buffer.unpack FORMAT
end
alias initialize update
def uncache
@to_s = @to_bin = nil
end
def to_s
@to_s ||= ATTRIBUTES.inject(''){|s,a| s << "#{ a } : #{ send a }, " }.chop.chop
end
def to_bin
@to_bin ||= @data.pack(FORMAT)
end
end

id, x_position, y_position, z_position, foobar =
400, 1.0, 2.0, 3.0, 0b101010

pd = PlayerData::create id, x_position, y_position, z_position, foobar

p pd
p pd.pid
p pd.x_position
p pd.foobar
puts pd

begin
require 'timeout'
n = 0
sec = 10
Timeout::timeout(sec) do
loop do
PlayerData::create id, x_position, y_position, z_position, foobar
n += 1
end
end
rescue Timeout::Error
puts "creations per second : #{ n / sec }"
end


basically it just unpacks everything and then uses lazy evaluation to get the
values when asked. it's hard to imagine getting faster than this, but someone
will surely prove me wrong ;-)

hth.

-a
--
===============================================================================
| email :: ara [dot] t [dot] howard [at] noaa [dot] gov
| phone :: 303.497.6469
| Your life dwells amoung the causes of death
| Like a lamp standing in a strong breeze. --Nagarjuna
===============================================================================

 

[Joel VanderWerf]

If this is going across a network, you might want 'NgggN' instead, so
everybody sees the same byte order.

 

[Ara.T.Howard]

If this is going across a network, you might want 'NgggN' instead, so
everybody sees the same byte order.

joel is absolutely correct here - the would be a very anoying thing to debug
too....

-a
--
===============================================================================
| email :: ara [dot] t [dot] howard [at] noaa [dot] gov
| phone :: 303.497.6469
| Your life dwells amoung the causes of death
| Like a lamp standing in a strong breeze. --Nagarjuna
===============================================================================

 

[Joe Van Dyk]

On Sun, 14 Aug 2005, Joe Van Dyk wrote:
=20

Thanks! I shall study this code (there's a few new idioms that I
haven't used before) and report back early next week on the
performance improvements.

=20
try this one - it's another order of magnitude faster:
=20
=20
harp:~ > ruby a.rb
#<PlayerData:0xb75cc0b4 @to_s=3Dnil, @data=3D[400, 1.0, 2.0, 3.0, 42= ], @to_bin=3Dnil>
400
1.0
42
pid : 400, x_position : 1.0, y_position : 2.0, z_position : 3.0, foo= bar : 42
creations per second : 114797
=20
=20
harp:~ > cat a.rb
class PlayerData
class << self
def create(*a)
new(a.pack(FORMAT))
end
end

Why do you use 'class << self' here?

Thanks,
Joe

 

[Julian Leviston]

Why do you use 'class << self' here?

Thanks,
Joe

Hey Joe,

That's another way to define a set of a class methods.
Another is PlayerData.create(*a)...end I think you can write
self.create(*a)..end too.
Anything within the class << self...end block will be a class method.

Julian.

 

[Ara.T.Howard]

Thanks! I shall study this code (there's a few new idioms that I
haven't used before) and report back early next week on the
performance improvements.

try this one - it's another order of magnitude faster:


harp:~ > ruby a.rb
#<PlayerData:0xb75cc0b4 @to_s=nil, @data=[400, 1.0, 2.0, 3.0, 42], @to_bin=nil>
400
1.0
42
pid : 400, x_position : 1.0, y_position : 2.0, z_position : 3.0, foobar : 42
creations per second : 114797


harp:~ > cat a.rb
class PlayerData
class << self
def create(*a)
new(a.pack(FORMAT))
end
end

Why do you use 'class << self' here?

to define class methods. in the context of a class think of it like

class C

class << self

# now i'm in class def scope

end

# now i'm in instance def scope

end

therefore you can

class C
class << self
attr :foobar
alias barfoo foobar
end
attr :foobar
alias barfoo foobar
end

and then

c = C::new

p C::foobar
p C::barfoo

p c.foobar
p c.barfoo

make sense? the scoping makes things like 'attr' and 'alias' possible in the
class scope.

so, for that method i'm defining a 'create' class method (instance factory)
which is like 'new' but accepts normal arguments and packs them into a buffer.
note that the new/initialize pair take a packed binary string as an argument -
but that is terribly inconvenient for testing. therefore i made the create
method - i makes sense for it to be a class method (vs. a global method)
because only the class knows the encoding of the string : encapsulation.

cheers.


-a
--
===============================================================================
| email :: ara [dot] t [dot] howard [at] noaa [dot] gov
| phone :: 303.497.6469
| Your life dwells amoung the causes of death
| Like a lamp standing in a strong breeze. --Nagarjuna
===============================================================================

 

[Joe Van Dyk]

On Mon, 15 Aug 2005, Joe Van Dyk wrote:
=20

On Sun, 14 Aug 2005, Joe Van Dyk wrote:

On Sun, 14 Aug 2005, Joe Van Dyk wrote:
<snip>
Thanks! I shall study this code (there's a few new idioms that I
haven't used before) and report back early next week on the
performance improvements.

try this one - it's another order of magnitude faster:


harp:~ > ruby a.rb
#<PlayerData:0xb75cc0b4 @to_s=3Dnil, @data=3D[400, 1.0, 2.0, 3.0,= 42], @to_bin=3Dnil>
400
1.0
42
pid : 400, x_position : 1.0, y_position : 2.0, z_position : 3.0, = foobar : 42
creations per second : 114797


harp:~ > cat a.rb
class PlayerData
class << self
def create(*a)
new(a.pack(FORMAT))
end
end

Why do you use 'class << self' here?

=20
to define class methods. in the context of a class think of it like
=20
class C
=20
class << self
=20
# now i'm in class def scope
=20
end
=20
# now i'm in instance def scope
=20
end
=20
therefore you can
=20
class C
class << self
attr :foobar
alias barfoo foobar
end
attr :foobar
alias barfoo foobar
end
=20
and then
=20
c =3D C::new
=20
p C::foobar
p C::barfoo
=20
p c.foobar
p c.barfoo
=20
make sense? the scoping makes things like 'attr' and 'alias' possible in= the
class scope.
=20
so, for that method i'm defining a 'create' class method (instance factor= y)
which is like 'new' but accepts normal arguments and packs them into a bu= ffer.
note that the new/initialize pair take a packed binary string as an argum= ent -
but that is terribly inconvenient for testing. therefore i made the crea= te
method - i makes sense for it to be a class method (vs. a global method)
because only the class knows the encoding of the string : encapsulation.

Isn't
def self.create(*a)
...

clearer than

class << self
def create(*a)
...=20

? I mean, they both create a class method, right?

 

[Joe Van Dyk]

On Sun, 14 Aug 2005, Joe Van Dyk wrote:
=20

Thanks! I shall study this code (there's a few new idioms that I
haven't used before) and report back early next week on the
performance improvements.

=20
try this one - it's another order of magnitude faster:
=20
=20
harp:~ > ruby a.rb
#<PlayerData:0xb75cc0b4 @to_s=3Dnil, @data=3D[400, 1.0, 2.0, 3.0, 42= ], @to_bin=3Dnil>
400
1.0
42
pid : 400, x_position : 1.0, y_position : 2.0, z_position : 3.0, foo= bar : 42
creations per second : 114797
=20
=20
harp:~ > cat a.rb
class PlayerData
class << self
def create(*a)
new(a.pack(FORMAT))
end
end
=20
SPEC =3D [
%w( pid i ),
%w( x_position f ),
%w( y_position f ),
%w( z_position f ),
%w( foobar i ),
]
=20
ATTRIBUTES =3D SPEC.map{|s| s.first}
=20
FORMAT =3D SPEC.map{|s| s.last}.join
=20
SPEC.each_with_index do |spec, ix|
at, format =3D spec
eval <<-src
def #{ at }
@#{ at } ||=3D @data[#{ ix }]
end
def #{ at }=3D value
raise TypeError unless self.#{ at }.class =3D=3D value.class
uncache
@#{ at } =3D @data[#{ ix }] =3D value
end
src
end
=20
def update buffer
uncache
@data =3D buffer.unpack FORMAT
end
alias initialize update
def uncache
@to_s =3D @to_bin =3D nil
end
def to_s
@to_s ||=3D ATTRIBUTES.inject(''){|s,a| s << "#{ a } : #{ send a= }, " }.chop.chop
end
def to_bin
@to_bin ||=3D @data.pack(FORMAT)
end
end
=20
id, x_position, y_position, z_position, foobar =3D
400, 1.0, 2.0, 3.0, 0b101010
=20
pd =3D PlayerData::create id, x_position, y_position, z_position, fo= obar
=20
p pd
p pd.pid
p pd.x_position
p pd.foobar
puts pd
=20
begin
require 'timeout'
n =3D 0
sec =3D 10
Timeout::timeout(sec) do
loop do
PlayerData::create id, x_position, y_position, z_position, foo= bar
n +=3D 1
end
end
rescue Timeout::Error
puts "creations per second : #{ n / sec }"
end
=20
=20
basically it just unpacks everything and then uses lazy evaluation to get= the
values when asked. it's hard to imagine getting faster than this, but so= meone
will surely prove me wrong ;-)

So, say I want to use a class similar to this called
'ApplicationData'. The only changes would be the SPEC.

Would the 'ruby way' be to have a base class (say, 'Data'), and then
have PlayerData and ApplicationData inherit from Data and have them
override the SPEC?

Thanks,
Joe

 

[Ara.T.Howard]

Isn't
def self.create(*a)
...

clearer than

class << self
def create(*a)

not when you end up using 'attr' type meta-programming, having class methods
which refer to class methods, aliases, and other things that complicate the
matter:


harp:~ > cat b.rb
class C
class << self
attr 'a'
attr 'b'
alias class_a a
alias class_b b
def class_initialize
@a, @b = 4, 2
end
def class_method
puts [class_a, class_b].join('')
end
end

self.class_initialize

attr 'a'
attr 'b'
alias inst_a a
alias inst_b b
def initialize
@a, @b = 'forty', 'two'
end
def instance_method
puts [inst_a, inst_b].join('-')
end
end

c = C::new
C::class_method
c.instance_method


harp:~ > ruby b.rb
42
forty-two


note how 'attr', 'alias', calls to methods that do not specify a receiver, and
instance variables are all consistently in the class scope here. to me this is
clearer than

class C
class << self
attr 'a'
attr 'b'
end
def self::c
a + b
end
end

i always (o.k. mostly) setup my classes like

class
# all constants
class << self
# all class stuff
end
# all instance stuff
end

if you use the the 'def self::method' approach you cannot be consistent since
some things don't work that way - like 'attr' and 'alias'. for me it's just
an idiom that makes my code clearer, more consistent, and avoids typing.

? I mean, they both create a class method, right?

yes.

hth.

-a
--
===============================================================================
| email :: ara [dot] t [dot] howard [at] noaa [dot] gov
| phone :: 303.497.6469
| Your life dwells amoung the causes of death
| Like a lamp standing in a strong breeze. --Nagarjuna
===============================================================================

 

[Ara.T.Howard]

So, say I want to use a class similar to this called
'ApplicationData'. The only changes would be the SPEC.

exactly right.

Would the 'ruby way' be to have a base class (say, 'Data'), and then have
PlayerData and ApplicationData inherit from Data and have them override the
SPEC?

i'd personally use a mixin here:

harp:~ > cat a.rb
module Packable
module ClassMethods
def lazy_attr at, ix
module_eval <<-src
def #{ at }
@#{ at } ||= @data[#{ ix }]
end
def #{ at }= value
raise TypeError unless self.#{ at }.class == value.class
uncache
@#{ at } = @data[#{ ix }] = value
end
src
end
def spec(list=[])
if list.empty?
@spec
else
@attributes = []
@format = ''
(@spec = list).each_with_index do |pair, ix|
at, f = pair
@attributes << at
@format << f
lazy_attr at, ix
end
end
end
attr 'attributes'
attr 'format'
def create(*a)
new(a.flatten.pack(format))
end
end
module InstanceMethods
attr 'data'
def klass
self.class
end
def update buffer
uncache
@data = buffer.unpack klass.format
end
alias initialize update
def uncache
@to_s = @to_bin = nil
end
def to_s
@to_s ||= klass.attributes.inject(''){|s,a| s << "#{ a } : #{ send a }, " }.chop.chop
end
def to_bin
@to_bin ||= @data.pack(klass.format)
end
end
def self::included other
other.extend ClassMethods
other.module_eval{ include InstanceMethods }
end
end


class PlayerData
include Packable

spec [
%w( pid i ),
%w( x_position f ),
%w( y_position f ),
%w( z_position f ),
%w( foobar i ),
]
end

class ApplicationData
include Packable

spec [
%w( aid i ),
%w( foobar i ),
%w( barfoo i ),
]
end


pid, x_position, y_position, z_position, foobar = 400, 1.0, 2.0, 3.0, 0b101010
pd = PlayerData::create pid, x_position, y_position, z_position, foobar
puts pd

aid, foobar, barfoo = 4, 2, 42
ad = ApplicationData::create aid, foobar, barfoo
puts ad



harp:~ > ruby a.rb
pid : 400, x_position : 1.0, y_position : 2.0, z_position : 3.0, foobar : 42
aid : 4, foobar : 2, barfoo : 42


in general class methods and mixins can be combined in an elegant way to
parameterize classes in a way far more flexible than inheritence.

cheers.

-a
--
===============================================================================
| email :: ara [dot] t [dot] howard [at] noaa [dot] gov
| phone :: 303.497.6469
| Your life dwells amoung the causes of death
| Like a lamp standing in a strong breeze. --Nagarjuna
===============================================================================

 

[Joe Van Dyk]

On Tue, 16 Aug 2005, Joe Van Dyk wrote:
=20

So, say I want to use a class similar to this called
'ApplicationData'. The only changes would be the SPEC.

=20
exactly right.
=20

Would the 'ruby way' be to have a base class (say, 'Data'), and then ha= ve
PlayerData and ApplicationData inherit from Data and have them override= the
SPEC?

=20
i'd personally use a mixin here:
=20
harp:~ > cat a.rb
module Packable
module ClassMethods
def lazy_attr at, ix
module_eval <<-src
def #{ at }
@#{ at } ||=3D @data[#{ ix }]
end
def #{ at }=3D value
raise TypeError unless self.#{ at }.class =3D=3D value.cla= ss
uncache
@#{ at } =3D @data[#{ ix }] =3D value
end
src
end
def spec(list=3D[])
if list.empty?
@spec
else
@attributes =3D []
@format =3D ''
(@spec =3D list).each_with_index do |pair, ix|
at, f =3D pair
@attributes << at
@format << f
lazy_attr at, ix
end
end
end
attr 'attributes'
attr 'format'
def create(*a)
new(a.flatten.pack(format))
end
end
module InstanceMethods
attr 'data'
def klass
self.class
end
def update buffer
uncache
@data =3D buffer.unpack klass.format
end
alias initialize update
def uncache
@to_s =3D @to_bin =3D nil
end
def to_s
@to_s ||=3D klass.attributes.inject(''){|s,a| s << "#{ a } : #= { send a }, " }.chop.chop
end
def to_bin
@to_bin ||=3D @data.pack(klass.format)
end
end
def self::included other
other.extend ClassMethods
other.module_eval{ include InstanceMethods }
end
end
=20
=20
class PlayerData
include Packable
=20
spec [
%w( pid i ),
%w( x_position f ),
%w( y_position f ),
%w( z_position f ),
%w( foobar i ),
]
end
=20
class ApplicationData
include Packable
=20
spec [
%w( aid i ),
%w( foobar i ),
%w( barfoo i ),
]
end
=20
=20
pid, x_position, y_position, z_position, foobar =3D 400, 1.0, 2.0, 3= 0, 0b101010
pd =3D PlayerData::create pid, x_position, y_position, z_position, f= oobar
puts pd
=20
aid, foobar, barfoo =3D 4, 2, 42
ad =3D ApplicationData::create aid, foobar, barfoo
puts ad
=20
=20
=20
harp:~ > ruby a.rb
pid : 400, x_position : 1.0, y_position : 2.0, z_position : 3.0, foo= bar : 42
aid : 4, foobar : 2, barfoo : 42
=20
=20
in general class methods and mixins can be combined in an elegant way to
parameterize classes in a way far more flexible than inheritence.

Wow... that's really impressive. Thanks again, you've been extremely helpf=
ul.