[Fwd: Re: managed lists?]

[Wildemar Wildenburger]

I answered off list (because besically I'm an idiot). So if you like,
read up the current news.

-------- Original Message --------
Subject: Re: managed lists?
Date: Mon, 21 May 2007 21:30:37 +0200
From: Jorgen Bodde <[email protected]>
To: Wildemar Wildenburger <[email protected]>
References:
<[email protected]>
<[email protected]>



Hi Wildemar,

Thanks for the answer. I did make something myself after i could not
find anything, just because it was fun to do I did saw array but
it was not for object, only for small types like int, char, etc to be
used for quick but efficient buffers of raw data (at least it looks
like that).

The list I created uses a strong type check to make sure all objects
in the list are of the same type, like

ol = objlist.ObjList(class_name = SomeClass)

Now only classes that are an instance of SomeClass are allowed in the
array, ObjList mimics a normal list, it can index, iterate, find,
delete, append items so it was basically a drop-in replacement for my
_list = [] solution

If you are interested i can send you the source. I created a py.test
module as well to test every method intensively.

Regards,
- Jorgen

Right now i have a list in a class that I export as a member variable
to the outside world, it is a standard list (e.g. [] ) but I wish to
have a stronger type checking when adding objects, that only some
objects are allowed and others are not. It was quite easy to create
it, but I wonder if there is already a standard solution for lists
that carry only objects of a single type?

Don't have much time on me right now, so I'll just give you a hint: I
*think* python has an array-datatype somewhere in its STD-lib (that is:
in a module you have to import). Not sure, could have been only for
NumPy or so. A quick look into the docs should clarify that quickly.

bye
W

 

[Bruno Desthuilliers]

(snip - I suppose I'm answering to Jorgen, but being another idiot
myself I may be wrong... anyway

Thanks for the answer. I did make something myself after i could not
find anything, just because it was fun to do I did saw array but
it was not for object, only for small types like int, char,

Everything in Python is an object. Including integers. And there's no
'char' type in Python.

The list I created uses a strong type check to make sure all objects
in the list are of the same type, like

ol = objlist.ObjList(class_name = SomeClass)

Now only classes that are an instance of SomeClass are allowed in the
array. ObjList mimics a normal list, it can index, iterate, find,
delete, append items so it was basically a drop-in replacement for my
_list = [] solution

The problem with this kind of "solutions" is that it creates more
problems than anything else.

What you really want is a list objects that share a common interface,
not a list of objects that are instances of a same class. As a simple
example, let's say you need a list of file-like objects. Or to be more
accurate, a list of objects that have a 'write' method that's compatible
with file.write(). So StringIO instances should be allowed, but your
code will reject them. You may have a perfectly valid reason to use
StringIO instances instead of files in some situations, but with your
(ill-named) 'ObjectList' stuff it won't be possible. For no good reason.

 

[Jorgen Bodde]

Hi Bruno,

Thanks for your answer.

Well what I am after is a list of relations to some class type. And in
that list I do not wish to have accidentally put ints, strings, only
one type of object, or interface. Now I can make the list interface
safe, but it is only meant for relational purposes only. So to
illustrate:

Song <>---->> Tab
Song <>--->> Tuning

I want a "tabs" collection only consisting of Tab objects. They are
very specific so "mimicing" a tab interface is not something that will
be done anytime soon.

I'm a traditional C++ programmer, and maybe I go through some
transitional phase I don't know but exposing my list as read /
(over)write property to the "outside world" being the rest of my
object model, is just asking for problems. So this "strong" typed list
will ensure me at "add" time the exception occurs, rather then
somewhere in my applciation who knows much later that something blows
up because the "object" from that list is retrieved and something
unpredictable goes wrong. Is that so weird to do? As I said earlier I
am a python newbie, I love the language, but the fact it can blow up
at unpredictable times, gives me shivers.

Everything in Python is an object. Including integers. And there's no
'char' type in Python.

The array type by the way says in the API that it can be constructed
with a simple type like a char as in a "c" type, an int as in a "i"
type etc..

See here:

http://www.python.org/doc/1.5.2p2/lib/module-array.html

So what I understand it's purpose is creating a buffer of some kind of
a fixed type to maybe communicate with other DLL's or objects that
require such a thing.

As I said, I might be going through a transitional phase, but exposing
my relation list as simply a list class where all kinds of objects can
be dumped in, seems too much for me at this point ;-)

Thanks,
- Jorgen

 

[Stargaming]

Hi Bruno,

Thanks for your answer.

Well what I am after is a list of relations to some class type. And in
that list I do not wish to have accidentally put ints, strings, only
one type of object, or interface. Now I can make the list interface
safe, but it is only meant for relational purposes only. So to
illustrate:

Song <>---->> Tab
Song <>--->> Tuning

I want a "tabs" collection only consisting of Tab objects. They are
very specific so "mimicing" a tab interface is not something that will
be done anytime soon.

I'm a traditional C++ programmer, and maybe I go through some
transitional phase I don't know but exposing my list as read /
(over)write property to the "outside world" being the rest of my
object model, is just asking for problems. So this "strong" typed list
will ensure me at "add" time the exception occurs, rather then
somewhere in my applciation who knows much later that something blows
up because the "object" from that list is retrieved and something
unpredictable goes wrong. Is that so weird to do? As I said earlier I
am a python newbie, I love the language, but the fact it can blow up
at unpredictable times, gives me shivers.



The array type by the way says in the API that it can be constructed
with a simple type like a char as in a "c" type, an int as in a "i"
type etc..

See here:

http://www.python.org/doc/1.5.2p2/lib/module-array.html

So what I understand it's purpose is creating a buffer of some kind of
a fixed type to maybe communicate with other DLL's or objects that
require such a thing.

As I said, I might be going through a transitional phase, but exposing
my relation list as simply a list class where all kinds of objects can
be dumped in, seems too much for me at this point ;-)

Thanks,
- Jorgen

Consider this: Who should add wrong-typed objects (see Bruno's post for
reasons why you shouldn't care about types, anyways) to your list, when
not you yourself? And if the programmer makes something wrong, he should
fix it immediately. What happens if your application throws some error
like "invalid type added to ObjectList"? Users won't unterstand it, so
the intended audience is you again.

Good night,
Stargaming

 

[Jorgen Bodde]

Hi,

Thanks. I agree that it is only 'me' that is the one doing it wrong.
But consider this scenario:

- Somewhere in my app I add a wrong type to an open regular list
- The app continues like it should
- After a (long) while I need to perform some search on the list, or whatever
- Exception occurs

It makes it hard to test this way. For example I can do something to
that list and not know about it much later .. If it blows up in my
face, I can fix it, but when that error hinders people who are using
my application, it is much worse.

With a somewhat managed list (forget about the types then, it was only
meant for my own satisfaction I guess), the access violation always
occurs on adding elements, which is always the path that a user
creates something 'new'. And if something new gets not added
correctly, I know where to look..

Anyway, I was looking for such a list class but i do understand that
most of the resposobility is for the programmer anyways and people
interacting with it. I did however need this solution so that I can
also add a better support for my database objects, add generic find
routines to that list class etc. So I basically added the interface
methods needed to let it be treated like a general list, like the
iterator, get method and what else. I was surprised it was so easy to
be done.

Python is cool when it comes to the "interface specification" v.s.
"class specification" it opens a lot of doors to paradigms otherwise
hard to implement, and I like the fact that every object can be
"adapted" to mimic another object needed by an internal routine in
Python, very-very dynamic indeed.

But being a strong typed C++ guy once, it takes some getting used to ;-)

Regards,
- Jorgen

 

[Gabriel Genellina]

En Tue, 22 May 2007 04:13:38 -0300, Jorgen Bodde

Thanks. I agree that it is only 'me' that is the one doing it wrong.
But consider this scenario:

- Somewhere in my app I add a wrong type to an open regular list
- The app continues like it should
- After a (long) while I need to perform some search on the list, or
whatever
- Exception occurs

It makes it hard to test this way. For example I can do something to
that list and not know about it much later .. If it blows up in my
face, I can fix it, but when that error hinders people who are using
my application, it is much worse.

You could use something like this to make a "restricted list":

class RestrictedList(list):
def __init__(self, items, itemclass_=None):
self.itemclass_ = itemclass_
for item in items:
self.append(item)

def check(self, item):
if not isinstance(item, self.itemclass_):
raise ValueError("This list can only contain %r instances" %
self.itemclass_)

def __setitem__(self, index, item):
if not isinstance(index,int):
raise NotImplementedError
self.check(item)
list.__setitem__(self, index, item)

def __setslice__(self, *args):
raise NotImplementedError

def append(self, item):
self.check(item)
list.append(self, item)

def insert(self, index, item):
self.check(item)
list.insert(self, index, item)

def extend(self, items):
for item in items:
self.append(item)

I think I'm catching all the ways someone could add a new item to this
list.
You may change the check() method to test for another type of conditions
instead of isinstance; by example, to ensure that all items have a "write"
method (and could be used as file-like objects to output data):

def check(self, item):
try: item.write
except AttributeError:
raise ValueError("blah blah...")

 

[Jorgen Bodde]

Hi Gabriel,

Yep that basically covered my implementation as well. It was rather
trivial to make it, and even for a python newbie it was simple which
says enough about the language itself. ;-)

Although I understand the opinions that you should not care about
types, I do believe putting a constraint on the list by either class
type or interface spec, is no bad thing. The interface specification
is hard to realise as this list now functions as some kind of template
class (hence the parameter that specifies which type is allowed).

I also added callback arguments that are called opon when there are
items added to the list, deleted or when it is cleared so that the
main object gets notified when another object changes it.

Here is my implementation (I bet it can be done better, but I am only
playing with python since about 3 months now):

--------------------------

class ObjListException(Exception):
pass

class ObjListIterator(object):
def __init__(self, objlist):
self.__objlist = objlist
self.__idx = 0

#---------------------------------------------------------------------------
def __iter__(self):
return self

#---------------------------------------------------------------------------
def next(self):
result = None
if self.__idx >= len(self.__objlist):
raise StopIteration()
else:
result = self.__objlist[self.__idx]
self.__idx += 1
return result

#===============================================================================

""" ObjList - A managed somewhat strong typed list for Python.
Expects {object}._id to be a property """

class ObjList(object):
def __init__(self, class_name, add_callback = None,
remove_callback = None, clear_callback = None):
self.__list = []
self.__cname = class_name
self.__add_cb = add_callback
self.__remove_cb = remove_callback
self.__clear_cb = clear_callback

#---------------------------------------------------------------------------
def __iter__(self):
return ObjListIterator(self.unmanaged_list())

#---------------------------------------------------------------------------
def __getitem__( self, key):
if key < len(self.__list) and key >= 0:
return self.__list[key]
return None

#---------------------------------------------------------------------------
def clear(self):
self.__list = []

#---------------------------------------------------------------------------
def append(self, obj):
if isinstance(obj, self.__cname):
if obj not in self.__list:
self.__list.append(obj)
if self.__add_cb:
self.__add_cb(self, obj)
else:
raise ObjListException()

#---------------------------------------------------------------------------
def remove(self, obj):
if obj in self.__list:
self.__list.remove(obj)
if self.__remove_cb:
self.__remove_cb(self, obj)

#---------------------------------------------------------------------------
def count(self):
return len(self.__list)

#---------------------------------------------------------------------------
def unmanaged_list(self):
return self.__list[:]

#---------------------------------------------------------------------------
def find_id(self, id):
for i in self.__list:
if i._id == id:
return i
return None

#---------------------------------------------------------------------------
def has_item(self, obj):
if obj in self.__list:
return True
return False

#---------------------------------------------------------------------------
def append_many(self, lst):
for l in lst:
self.append(l)

Regards,
- Jorgen

 

[Larry Bates]

Hi Gabriel,

Yep that basically covered my implementation as well. It was rather
trivial to make it, and even for a python newbie it was simple which
says enough about the language itself. ;-)

Although I understand the opinions that you should not care about
types, I do believe putting a constraint on the list by either class
type or interface spec, is no bad thing. The interface specification
is hard to realise as this list now functions as some kind of template
class (hence the parameter that specifies which type is allowed).

I also added callback arguments that are called opon when there are
items added to the list, deleted or when it is cleared so that the
main object gets notified when another object changes it.

Here is my implementation (I bet it can be done better, but I am only
playing with python since about 3 months now):

--------------------------

class ObjListException(Exception):
pass

class ObjListIterator(object):
def __init__(self, objlist):
self.__objlist = objlist
self.__idx = 0


#---------------------------------------------------------------------------

def __iter__(self):
return self


#---------------------------------------------------------------------------

def next(self):
result = None
if self.__idx >= len(self.__objlist):
raise StopIteration()
else:
result = self.__objlist[self.__idx]
self.__idx += 1
return result

#===============================================================================


""" ObjList - A managed somewhat strong typed list for Python.
Expects {object}._id to be a property """

class ObjList(object):
def __init__(self, class_name, add_callback = None,
remove_callback = None, clear_callback = None):
self.__list = []
self.__cname = class_name
self.__add_cb = add_callback
self.__remove_cb = remove_callback
self.__clear_cb = clear_callback


#---------------------------------------------------------------------------

def __iter__(self):
return ObjListIterator(self.unmanaged_list())


#---------------------------------------------------------------------------

def __getitem__( self, key):
if key < len(self.__list) and key >= 0:
return self.__list[key]
return None


#---------------------------------------------------------------------------

def clear(self):
self.__list = []


#---------------------------------------------------------------------------

def append(self, obj):
if isinstance(obj, self.__cname):
if obj not in self.__list:
self.__list.append(obj)
if self.__add_cb:
self.__add_cb(self, obj)
else:
raise ObjListException()


#---------------------------------------------------------------------------

def remove(self, obj):
if obj in self.__list:
self.__list.remove(obj)
if self.__remove_cb:
self.__remove_cb(self, obj)


#---------------------------------------------------------------------------

def count(self):
return len(self.__list)


#---------------------------------------------------------------------------

def unmanaged_list(self):
return self.__list[:]


#---------------------------------------------------------------------------

def find_id(self, id):
for i in self.__list:
if i._id == id:
return i
return None


#---------------------------------------------------------------------------

def has_item(self, obj):
if obj in self.__list:
return True
return False


#---------------------------------------------------------------------------

def append_many(self, lst):
for l in lst:
self.append(l)

Regards,
- Jorgen

IMHO you are trying to write C in Python. In another 3 months
I'll bet you will look back on this and say "Why did I bother?".
I would suggest that the time to do such "restrictions" would
better be spent writing good unit tests for your application.
They are what really will assist you in finding those problems way
down deep in your code and will be useful for the entire life of
the project.

You may also want to consider checking to see if the object that
is passed in implements the interface that you want instead of
requiring that it be a a specific class instance. Something like
(not tested):

class ObjListIterator(object)
#
# class variable
#
__required_methods=['methodname1','methodname2','methodname3']

def chkObjInterface(self, obj):
for method in self.__required_methods:
if not hasattr(obj, 'methodname1':
raise ObjListException("obj must implement '%s' method")

else:
raise ObjListException("obj must implement 'methodname'" \

"method")
return True



def append(self, obj):
if chkObjInterface(obj):
if obj not in self.__list:
self.__list.append(obj)
if self.__add_cb:
self.__add_cb(self, obj)
else:
raise ObjListException("append.obj must implement 'methodname'" \

method")

I would also write append_many as:

def extend(self, lst):
map(self.append, lst)
return

I would find it easier to remember extend (since that is very pythonic).

If your collection of objects is large, I would also consider
putting them in a dictionary instead of a list and indexing them
on ID.

Hope this helps in some way.

-Larry

 

[Bruno Desthuilliers]

Jorgen Bodde a écrit :

Hi Bruno,

Thanks for your answer.

Well what I am after is a list of relations to some class type. And in
that list I do not wish to have accidentally put ints, strings,

So don't do it !-)

only
one type of object, or interface. Now I can make the list interface
safe, but it is only meant for relational purposes only. So to
illustrate:

Song <>---->> Tab
Song <>--->> Tuning

I want a "tabs" collection only consisting of Tab objects. They are
very specific so "mimicing" a tab interface is not something that will
be done anytime soon.

Yes, probably. As usual, there's the general rule and the reality. But
still, while I understand both your use case and your fears, I think
you're not solving the problem the right way - at least the pythonic way !-)

I'm a traditional C++ programmer, and maybe I go through some
transitional phase
s/maybe/certainly/

I don't know but exposing my list as read /
(over)write property to the "outside world" being the rest of my
object model, is just asking for problems.

Not necessarily. I came to Python from a C/Java (and bits of C++ FWIW)
background, with the firm belief that coding with no static typing and
no access restriction was "just asking for problems". It turned out that
it usually just works - even if it sometimes requires to think about the
problem in a different way.

So this "strong" typed list
will ensure me at "add" time the exception occurs, rather then
somewhere in my applciation who knows much later that something blows
up because the "object" from that list is retrieved and something
unpredictable goes wrong. Is that so weird to do?

Nope, not at all. The goal is perfectly legitimate : ease debugging.
It's the solution that is wrong IMHO - even in your particular case.

Let's restate your problem : you want to simplify debugging by knowing
ASAP when some incompatible object is added to a Song's tunes or tabs
list. The right solution IMHO is to *not* expose these lists as part of
the interface - after all, the fact that tabs and tunes are stored in
lists is an implementation detail -, and to add the necessary API to the
Song object - ie : Song.add_tune, Song.add_tab, Song.iter_tunes,
Song.iter_tabs, etc. Then you just have to add some assert statements in
the add_XXX methods so you'll be warned at the right time when some part
of the client code pass something wrong. Then it's just a matter of
re-compiling this code with the -o flag to turn off assertion when
deploying the finished product - by this time, you should be confident
enough in the fact that the client code is doing right. As an added
bonus, you gain clean encapsulation and respect the law of Demeter.

As I said earlier I
am a python newbie, I love the language, but the fact it can blow up
at unpredictable times, gives me shivers.

Any program in any language can crash at unpredictable time. Better
learn to live with it. At least Python saves you from a lot of common
C/C++ problems related to memory management...

The array type by the way says in the API that it can be constructed
with a simple type like a char as in a "c" type, an int as in a "i"
type etc..

Yes, I know. As you noticed, this is a somewhat peculiar module - not
intended to be use as a replacement of C++ template containers. I just
wanted to emphasis the fact that in Python itself everything is an
object (at least anything you can bind to a name).

As I said, I might be going through a transitional phase, but exposing
my relation list as simply a list class where all kinds of objects can
be dumped in, seems too much for me at this point ;-)

As I said, the problem is (IMHO) more about exposing the list *as part
of the interface* than about the fact that a list can contain any kind
of object.

Oh, yes, while we're at it, and in case you don't know yet: the Python
idiom for implementation attributes (including methods - remember,
everything is an object) is to prefix them with a single underscore.

HTH

 

[Bruno Desthuilliers]

Jorgen Bodde a écrit :

(snip)

class ObjListException(Exception):
pass

class ObjListIterator(object):
def __init__(self, objlist):
self.__objlist = objlist
self.__idx = 0

You should use a single underscore here.

 

[Jorgen Bodde]

@ Larry,

As much as I like to follow the idiom that Python has, I do not see
the usefulness of making an add function that checks the interface
that other objects might need. Besides the fact it is a bit overhead
and not to mention performance draining, in an application that I
develop myself, without plugin architecture (not yet) etc, it is a bit
much to do that.

I agree with the writing C in Python, it is all I know, because that
is the language I started with. This is my first big project and most
certainly when evaluating this when it is finished I will come to the
conclusion it was all for nothing ;-)

I will change to extend() and maybe drop the type safety, but I do
wish to keep the list functionality as I find it more logical to write

song.categories.append()

Then creating a method in song called AddCategory() and kind of
duplicating the list functionality there.

Thank you for the guide lines and the nice comments! They are really helpful.

@ Bruno,

list. The right solution IMHO is to *not* expose these lists as part of
the interface - after all, the fact that tabs and tunes are stored in
lists is an implementation detail -, and to add the necessary API to the
Song object - ie : Song.add_tune, Song.add_tab, Song.iter_tunes,
Song.iter_tabs, etc. Then you just have to add some assert statements in
the add_XXX methods so you'll be warned at the right time when some part

This is how I did it before, I created Song.AddTab, Song.DeleteTab,
then there was a category class which also needed adding, deleting,
clearing, and soon my whole Song object was filled with relational
methods which were basically a copy of the list, with a check on the
presence of the class in the list.

Then I realized, a managed list with self-check capability solves this
because I do not like to duplicate code, but now that I think of it,
the type check might be a bit too much. Like said, the list solution
you described and the solution I use, are basically the same, I
encapsulated a list [] in a class, with some extra methods to warn the
'owner' of the list that something is changed.

Oh, yes, while we're at it, and in case you don't know yet: the Python
idiom for implementation attributes (including methods - remember,
everything is an object) is to prefix them with a single underscore.

Ok, but why is that? I do know semi-private symbols can be realised
using two underscores, and if I wish to hide a list / index / etc so
it cannot accidentally changed, would I not use double underscores?

Thanks for your extensive help!
- Jorgen