[Help] [Newbie] Require help migrating from Perl to Python 2.7 (namespaces)

[prilisauer]

Hello, to all,

I hope I can describe me problem correctly.

I have written a Project split up to one Main.py and different modules which are loaded using import and here is also my problem:

1. Main.py executes:
2. Import modules
3. One of the Modules is a SqliteDB datastore.
4. A second module creates an IPC socket.
5. Here is now my problem :
The IPC Socket should run a sub which is stored ad SqliteDB and returns all Rows.

Is there a elegant way to solve it? except a Queue. Is it possible to import modules multiple?! I'm unsure because the open DB file at another module.

How is this solved in bigger projects?

Thanks a lot for all answers.
BR

 

[Peter Otten]

Hello, to all,

I hope I can describe me problem correctly.

I have written a Project split up to one Main.py and different modules
which are loaded using import and here is also my problem:

1. Main.py executes:
2. Import modules
3. One of the Modules is a SqliteDB datastore.
4. A second module creates an IPC socket.
5. Here is now my problem :
The IPC Socket should run a sub which is stored ad SqliteDB and
returns all Rows.

Is there a elegant way to solve it? except a Queue. Is it possible to
import modules multiple?!

If you import a module more than once code on the module level will be
executed the first time only. Subsequent imports will find the ready-to-use
module object in a cache (sys.modules).

I'm unsure because the open DB file at another
module.

How is this solved in bigger projects?

If I'm understanding you correctly you have code on the module level that
creates a socket or opens a database. Don't do that!
Put the code into functions instead. That will give the flexibility you need
for all sizes of projects. For instance

socket_stuff.py

def send_to_socket(rows):
socket = ... # open socket
for row in rows:
# do whatever it takes to serialize the row
socket.close()

database_stuff.py

def read_table(dbname, tablename):
if table not in allowed_table_names:
raise ValueError
db = sqlite3.connect(dbname)
cursor = db.cursor()
for row in cursor.execute("select * from %s" % tablename):
yield row
db.close()


main.py

import socket_stuff
import database_stuff

if __name__ == "__main__":
socket_stuff.send_to_socket(
database_stuff.read_table("some_db", "some_table"))

 

[prilisauer]

Am Samstag, 22. Dezember 2012 12:43:54 UTC+1 schrieb Peter Otten:

wrote:














If you import a module more than once code on the module level will be

executed the first time only. Subsequent imports will find the ready-to-use

module object in a cache (sys.modules).






If I'm understanding you correctly you have code on the module level that

creates a socket or opens a database. Don't do that!

Put the code into functions instead. That will give the flexibility you need

for all sizes of projects. For instance



socket_stuff.py



def send_to_socket(rows):

socket = ... # open socket

for row in rows:

# do whatever it takes to serialize the row

socket.close()



database_stuff.py



def read_table(dbname, tablename):

if table not in allowed_table_names:

raise ValueError

db = sqlite3.connect(dbname)

cursor = db.cursor()

for row in cursor.execute("select * from %s" % tablename):

yield row

db.close()





main.py



import socket_stuff

import database_stuff



if __name__ == "__main__":

socket_stuff.send_to_socket(

database_stuff.read_table("some_db", "some_table"))

Hello Thanks for that answer:
I think I have to write it a bit larger,

This isn't a real code, but it for better overview:
main.py
/ / \ \
-------------------------------------------------------------------
Datastore | ModbusClient | DaliBusClient | Advanced Scheduler
--------------------------------------------------------------------


Main.py:
import Datastore
Datastore.startup()
.....
(Everything should run in threads)

Datastore.py
# Opens a SQLite DB
# Contains Queries to serve data for other mods
# Central point for updating / Quering etc.

ModbusClient.py
# Opens TCP connection to Modbus Device
!!! Here it's interesting !!!
This module contains predefined Queries and functions
IF I START A FUNCTION a SQL should be executed in Datastore.py


DaliBusClient.py
#Lamps, etc. Functions.
# Stored Informations should be kept in Datastore
# Advanced scheduler


I don't know, Python allways looks for me like a one script "File". But there are big projects. like the the "Model of an SQL Server", using coordinators no problems running threads and exchange Data through a Backbone. I have searched a lot, but I havent find anything to write the "core" splited upinto modules and geting over the scopes etc.

DO I have anything forgotten?! everything unclear??

 

[prilisauer]

Am Samstag, 22. Dezember 2012 12:43:54 UTC+1 schrieb Peter Otten:

wrote:














If you import a module more than once code on the module level will be

executed the first time only. Subsequent imports will find the ready-to-use

module object in a cache (sys.modules).






If I'm understanding you correctly you have code on the module level that

creates a socket or opens a database. Don't do that!

Put the code into functions instead. That will give the flexibility you need

for all sizes of projects. For instance



socket_stuff.py



def send_to_socket(rows):

socket = ... # open socket

for row in rows:

# do whatever it takes to serialize the row

socket.close()



database_stuff.py



def read_table(dbname, tablename):

if table not in allowed_table_names:

raise ValueError

db = sqlite3.connect(dbname)

cursor = db.cursor()

for row in cursor.execute("select * from %s" % tablename):

yield row

db.close()





main.py



import socket_stuff

import database_stuff



if __name__ == "__main__":

socket_stuff.send_to_socket(

database_stuff.read_table("some_db", "some_table"))

Hello Thanks for that answer:
I think I have to write it a bit larger,

This isn't a real code, but it for better overview:
main.py
/ / \ \
-------------------------------------------------------------------
Datastore | ModbusClient | DaliBusClient | Advanced Scheduler
--------------------------------------------------------------------


Main.py:
import Datastore
Datastore.startup()
.....
(Everything should run in threads)

Datastore.py
# Opens a SQLite DB
# Contains Queries to serve data for other mods
# Central point for updating / Quering etc.

ModbusClient.py
# Opens TCP connection to Modbus Device
!!! Here it's interesting !!!
This module contains predefined Queries and functions
IF I START A FUNCTION a SQL should be executed in Datastore.py


DaliBusClient.py
#Lamps, etc. Functions.
# Stored Informations should be kept in Datastore
# Advanced scheduler


I don't know, Python allways looks for me like a one script "File". But there are big projects. like the the "Model of an SQL Server", using coordinators no problems running threads and exchange Data through a Backbone. I have searched a lot, but I havent find anything to write the "core" splited upinto modules and geting over the scopes etc.

DO I have anything forgotten?! everything unclear??

 

[prilisauer]

Am Samstag, 22. Dezember 2012 13:38:11 UTC+1 schrieb (e-mail address removed):

Am Samstag, 22. Dezember 2012 12:43:54 UTC+1 schrieb Peter Otten:




Hello Thanks for that answer:

I think I have to write it a bit larger,



This isn't a real code, but it for better overview:

main.py

/ / \ \

-------------------------------------------------------------------

Datastore | ModbusClient | DaliBusClient | Advanced Scheduler

--------------------------------------------------------------------





Main.py:

import Datastore

Datastore.startup()

....

(Everything should run in threads)



Datastore.py

# Opens a SQLite DB

# Contains Queries to serve data for other mods

# Central point for updating / Quering etc.



ModbusClient.py

# Opens TCP connection to Modbus Device

!!! Here it's interesting !!!

This module contains predefined Queries and functions

IF I START A FUNCTION a SQL should be executed in Datastore.py





DaliBusClient.py

#Lamps, etc. Functions.

# Stored Informations should be kept in Datastore

# Advanced scheduler





I don't know, Python allways looks for me like a one script "File". But there are big projects. like the the "Model of an SQL Server", using coordinators no problems running threads and exchange Data through a Backbone. I have searched a lot, but I havent find anything to write the "core" splited up into modules and geting over the scopes etc.



DO I have anything forgotten?! everything unclear??

Ps.: The Socket, the DB has to be kept allways open, because of it's Serverfunctionality, A lot of Sensors, Timers, User interaction, must recived , Calculated, etc so a reaction must be send in about 16~100 ms, different modules opens and closes Sockets or files, could result in a dead situation.


Or do i didn't see any Tree's in the Wood?

Br

 

[Peter Otten]

I don't know, Python allways looks for me like a one script "File". But
there are big projects. like the the "Model of an SQL Server", using
coordinators no problems running threads and exchange Data through a
Backbone. I have searched a lot, but I havent find anything to write the
"core" splited up into modules and geting over the scopes etc.

DO I have anything forgotten?! everything unclear??

I think you have to step back and acquire some experience with a smaller,
less complex project... investigating libraries on which you can base your
efforts may also be a good idea.

If the Perl program is well-structured there is no reason not to copy its
architecture -- but I suspect you wouldn't want to migrate then.

 

[prilisauer]

Am Samstag, 22. Dezember 2012 14:54:27 UTC+1 schrieb Peter Otten:

wrote:










I think you have to step back and acquire some experience with a smaller,

less complex project... investigating libraries on which you can base your

efforts may also be a good idea.



If the Perl program is well-structured there is no reason not to copy its

architecture -- but I suspect you wouldn't want to migrate then.

Yes, but my project couldn't be smaller, my core problem is how to exchange data between modules?!...

 

[prilisauer]

Am Samstag, 22. Dezember 2012 14:54:27 UTC+1 schrieb Peter Otten:

wrote:










I think you have to step back and acquire some experience with a smaller,

less complex project... investigating libraries on which you can base your

efforts may also be a good idea.



If the Perl program is well-structured there is no reason not to copy its

architecture -- but I suspect you wouldn't want to migrate then.

Yes, but my project couldn't be smaller, my core problem is how to exchange data between modules?!...

 

[Alexander Blinne]

Am 22.12.2012 13:45, schrieb (e-mail address removed):

Ps.: The Socket, the DB has to be kept allways open, because of it's Server functionality, A lot of Sensors, Timers, User interaction, must recived , Calculated, etc so a reaction must be send in about 16~100 ms, different modules opens and closes Sockets or files, could result in a dead situation.


Or do i didn't see any Tree's in the Wood?

I would strongly recommend an object oriented view:

Suppose Datastore.py contains a Class Datastore. You can instantiate
that class once in the beginning of your main file and the resulting
object has methods to store and retrieve data in/from the store.

ModbusClient.py contains a Class Modbus. This also can be instantiated
just once which opens a TCP connection to be used many times and you can
hand over a reference to the Instance of Datastore you created earlier,
so it can speak with the Datastore. The object has methods to do the
things you want it to do.

The Same for DaliBusClient.

Now your main.py could look something linke

from Datastore import Datastore
from ModbusClient import Modbus
from DaliBusClient import DaliBus

def main():
datastore = Datastore(...)
modbus = Modbus(..., datastore)
dalibus = DaliBus(..., datastore)

modbus.read_data_and_save_to_store()
dalibus.read_data_and_save_to_store()

if __name__=="__main__":
main()

 

[prilisauer]

Am Samstag, 22. Dezember 2012 18:26:43 UTC+1 schrieb Alexander Blinne:

I would strongly recommend an object oriented view:



Suppose Datastore.py contains a Class Datastore. You can instantiate

that class once in the beginning of your main file and the resulting

object has methods to store and retrieve data in/from the store.



ModbusClient.py contains a Class Modbus. This also can be instantiated

just once which opens a TCP connection to be used many times and you can

hand over a reference to the Instance of Datastore you created earlier,

so it can speak with the Datastore. The object has methods to do the

things you want it to do.



The Same for DaliBusClient.



Now your main.py could look something linke



from Datastore import Datastore

from ModbusClient import Modbus

from DaliBusClient import DaliBus



def main():

datastore = Datastore(...)

modbus = Modbus(..., datastore)

dalibus = DaliBus(..., datastore)



modbus.read_data_and_save_to_store()

dalibus.read_data_and_save_to_store()



if __name__=="__main__":

main()

Yes,
My Project is allready > 1000 lines and even more,..
I have started writing it, each module after another ...

I've got allready a lot of experience in other programming languages ( alsoI
have worked as a programmer)

But the question is, could communicate a DaliModul, Modbus,.. Etc with the backend db.

It's for me a view of top side down, but how could the midlevel comunicate to each oter... "not hirachical"

 

[Alexander Blinne]

Am 22.12.2012 19:10, schrieb (e-mail address removed):

It's for me a view of top side down, but how could the midlevel comunicate to each oter... "not hirachical"

You could use something like the singleton pattern in order to get a
reference to the same datastore-object every time Datastore.Datastore()
is called. But you still need to close the connection properly at some
point, propably using a classmethod Datastore.close().

e.g.:

main.py:

from Datastore import Datastore
from ModbusClient import Modbus
from DaliBusClient import DaliBus

def main():
modbus = Modbus(...)
dalibus = DaliBus(...)

modbus.read_data_and_save_to_store()
dalibus.read_data_and_save_to_store()
Datastore.close()

if __name__=="__main__":
main()


ModbusClient.py:

import Datastore

class Modbus(object):
def read_data_and_save_to_store(self):
datastore = Datastore.Datastore()
#do something with datastore

 

[prilisauer]

Am Samstag, 22. Dezember 2012 20:29:49 UTC+1 schrieb Alexander Blinne:

You could use something like the singleton pattern in order to get a

reference to the same datastore-object every time Datastore.Datastore()

is called. But you still need to close the connection properly at some

point, propably using a classmethod Datastore.close().



e.g.:



main.py:



from Datastore import Datastore

from ModbusClient import Modbus

from DaliBusClient import DaliBus



def main():

modbus = Modbus(...)

dalibus = DaliBus(...)



modbus.read_data_and_save_to_store()

dalibus.read_data_and_save_to_store()

Datastore.close()



if __name__=="__main__":

main()





ModbusClient.py:



import Datastore



class Modbus(object):

def read_data_and_save_to_store(self):

datastore = Datastore.Datastore()

#do something with datastore

I Think I describe my Situation wrong, the written Project is a
Server, that should store sensor data, perfoms makros on lamps according
a sequence stored in the DB and Rule systems schould regulate home devices and plan scheduler jobs so on.

The System Runs in a threated environment. It looks for me, like the limits are at the end of file. my core problem also the only one I have is: I don't know how to get over that limits and enable dataexchange like a backbone...

 

[Terry Reedy]

Am Samstag, 22. Dezember 2012 13:38:11 UTC+1 schrieb (e-mail address removed):

And my mail reader text window is filled up without any content. If you
are going to post from google groups, learn how to do so without
doubling the spaces each time. Also learn to snip what is not needed for
your reply.

 

[Alexander Blinne]

Am 22.12.2012 21:43, schrieb (e-mail address removed):

I Think I describe my Situation wrong, the written Project is a
Server, that should store sensor data, perfoms makros on lamps according
a sequence stored in the DB and Rule systems schould regulate home devices and plan scheduler jobs so on.

I really don't understand your problem and I don't think anyone else
does. A python programm written like I have explained could do all those
things with no problem whatsoever, if only the classes contain all the
relevant methods.

The System Runs in a threated environment. It looks for me, like the limits are at the end of file. my core problem also the only one I have is: I don't know how to get over that limits and enable dataexchange like a backbone...

There is no limit at the end of a file. A module is only a namespace and
you can access the names of another module simply by importing it first.
You also can freely pass around objects as parameters in function/method
calls or even just store them to module-level global names. Threads
don't change anything about that.

Greetings.

 

[Dave Angel]

<snip>
I Think I describe my Situation wrong, the written Project is a
Server, that should store sensor data, perfoms makros on lamps
according a sequence stored in the DB and Rule systems schould
regulate home devices and plan scheduler jobs so on. The System Runs
in a threated environment. It looks for me, like the limits are at the
end of file. my core problem also the only one I have is: I don't know
how to get over that limits and enable dataexchange like a backbone...

Please post in English.

 

[prilisauer]

By the way, I think I have found the correct "wording".
for my understood, the "handover" of objects to imported modules doesn't work because, e.g. trying to hand-over an SQLite connection into a imported module, can't work because the "attributes" are not transfered.

I'm sorry for my bad english, it's fascinating. 2 years ago I've written very large english technical documents for my company. As you can see, the last two years I've forgotten a lot and it tooks me some time to get back into.

 

[prilisauer]

secondly, it is absolutely not bad meaned, but, why does people post, their personal meaning, but nothing about the "Posters" Problem?

Everybody is free to read or not, but correcting the WWW could became a very very big task, (maybe it's easier to climb the 7 summits)

Best Regards.

 

[Cameron Simpson]

| I Think I describe my Situation wrong, the written Project is a
| Server, that should store sensor data, perfoms makros on lamps according
| a sequence stored in the DB and Rule systems schould regulate home devices and plan scheduler jobs so on.
|
| The System Runs in a threated environment. It looks for me, like the
| limits are at the end of file. my core problem also the only one I have
| is: I don't know how to get over that limits and enable dataexchange
| like a backbone...

Maybe you should post some of the Perl code, in small pieces. Then we
can suggest ways those poarticular things might be done in Python.

Python threads really easily (with some limitations, but for many purposes
those limitations are irrelevant).

When I have a situation like yours seems to be, I tend to write a few
different items, connected together with a main program. Write modules
that define a class for the things you need to talk to: the database,
the sensors, etc. From the main program, create an instance of the
relevant classes, then dispatch threads doing what needs to be done.

The main program might be shaped like this:

import db_module # use a better name
# defines a class called "DB" to talk to a
# database
import sensors_module # use a better name
# defines a class called "Sensors" to report
# sensor values

def thread_function(db, sensors):
... do something that should happen in a thread ...

# get some objects to connect to db and sensors
db = db_module.DB(connection-info-here...)
sensors = sensors_module.Sensors(sensor-connection-info-here...)

# set up a Thread and start it
T = Thread(target=thread_function, args=(db, sensors))
T.start()
... create other threads as needed ...

You see here that:
- the modules do not know _specifics_ about the db or sensors;
they are told connection info
- instantiating a class instance:
db = db_module.DB(connection-info-here...)
passes the specifics
- you get back a class instance
- you pass those instances (db, sensors) to the thread_function;
it uses them to access database and sensors

So you see that the modules do not directly share information with each
other. The main program gets objects from each module and hands them to
whoever needs to work with them.

Does this clarify your namespace issues?

Cheers,

 

[prilisauer]

Thanks to all your answers, I have read a lot about namespaces, but still there's something I do not understood. I have tried your example but as I expected:

line 13, in HandoverSQLCursor
curs.execute("SELECT * FROM lager")
AttributeError: 'builtin_function_or_method' object has no attribute 'execute'

I will try my best to write tomorrow a sample as detailed as possible.

Good evening

 

[prilisauer]

Thanks to all your answers, I have read a lot about namespaces, but still there's something I do not understood. I have tried your example but as I expected:

line 13, in HandoverSQLCursor
curs.execute("SELECT * FROM lager")
AttributeError: 'builtin_function_or_method' object has no attribute 'execute'

I will try my best to write tomorrow a sample as detailed as possible.

Good evening