Distributed Cache Server?

[ken.faulkner]

Does anyone know if a "distributed caching system" has been developed
for use with Python? I've seen mention of memcached, but was really
after something natively python.

Yes, "distributed caching system" is a bit of a general term, but am
really just talking about something as simple as key + value (arbitrary
class) which can be split over a number of machines in an efficient
manner.

I've started development of such a system, but am wondering if
something already exists out there.

KenF

 

[Roger Binns]

Does anyone know if a "distributed caching system" has been developed
for use with Python?

BitTorrent

Yes, "distributed caching system" is a bit of a general term, but am
really just talking about something as simple as key + value (arbitrary
class) which can be split over a number of machines in an efficient
manner.

You'll need to define what the sweet spot is that you are aiming for.
Are we talking tens of thousands of keys or billions? How big is the
data (megabytes, gigabytes, terabytes?) Do you need transactional
integrity (eg when are updates seen by other readers)? Do you want
redundancy (data duplicated on multiple machines)? How many machines
are we talking about? Should failure be automatically detected? Is
there a need for security or treating the machines as untrusted?

Roger

 

[=?ISO-8859-1?Q?Gerhard_H=E4ring?=]

Does anyone know if a "distributed caching system" has been developed
for use with Python? I've seen mention of memcached, but was really
after something natively python. [...]

Too bad, because memcached implements all the difficult parts already
(like failover if one node dies), has Python bindings:
ftp://ftp.tummy.com/pub/python-memcached/ and just plain works.

[...] I've started development of such a system, but am wondering if
something already exists out there.

I don't know of any pure-Python equivalent.

-- Gerhard

 

[ken.faulkner]

har.... nah, think I'll give BT a miss for this purpose.

Yeah, know it was a loaded question (no specifics), but was really only
thinking out aloud.

The plans are:

- Multiple copies of data on different machines (just thinking 2 copies
of the data)
- Load spread over multiple machines... anything between 2->lots
(again, not being specific).
- Data itself I'm expecting to be low.... a few Gb at tops (but
ofcourse, as this extends I might want to increase the size)
- Haven't thought a LOT about automatic handling of errors...... yet.
- and its assumed to be on a trusted network.

Apart from memcached, I haven't seen anything similar to what I'm after
(plus to be perfectly honest..... I wouldn't mind coding it up anyway)


Was just "feeling about" to see if anything else is out there.

KenF

ps. btw, only part of it is theoretical, a lot of this I've already
coded, but not targetted towards large scale (100's machines etc)... So
am thinking what I'd need to consider to increase this to a larger
scale.