SQLite (with APSW) and transaction separate

[Christian Stooker]

Part one:
======

Hi !

I want to use SQLite database like the FireBird database: with big
isolation level.
What's that meaning ?

I have an application that periodically check some input directory,
process the elements in it, and copy them into a global database.
It is like a daemon, working in the background.

The end-user uses an another GUI application, that can create reports
from this database.

Ok. In FireBird I can do it fine that the user's app use another
transaction with READ REPEATABLE style, and the daemon use another
transaction too, with same style.

This style is correctly separate two transactions, the user see full
consistent state in database.
If daemon has been finished the work, commit and close it's transaction
- but that is not confused the user's reporter program.

Example:
I open the database for create reports from a master-detail table.
Example bills.
The code get every bills.
If daemon see a new data, it is get it, and put to the database.
Now it is see a new data. It is make insert/update sqls.

The reporter tool read the bill 002 head.
The daemon write this head, and append a new subitem to bill's subitems
table. Commit.
If ISOLATION LEVEL is low, the report can see the new subitems of the
bill, but the bill's head is not same - inconsistency we get.
Example: Bills head have 3 items, the total charge is 600$. In low Isol.
level when the daemon is add new records to this bill, the head total is
remaining 600$, but calculated subtotals are bigger than this.

In the big isolation level (repeatable read) the reporter not see the
any changes in the database only what it makes.

I can do same effect if I create "transaction aliasing code". I need to
extend my tables with recversion field that containing a timestamp value.
And every SQL I need to extend with where timestamp is...
But it is very ugly solution...

Please help me: have the SQLite same tr. isol. mechanism like FireBird ?
http://www.dotnetfirebird.org/transaction-isolation-levels
http://www.ibphoenix.com/main.nfs?a=ibphoenix&l=;KNOWLEDGEBASE;ID='377'
http://www.ibphoenix.com/main.nfs?a=ibphoenix&l=;KNOWLEDGEBASE;ID='128'
http://www.ibphoenix.com/main.nfs?a=ibphoenix&l=;KNOWLEDGEBASE;ID='129'

Thanx for your help:
dd


Part two:
=========

(e-mail address removed) írta:

SQLite transactions are SERIALIZABLE.

It is possible to force SQLite transactions to be
READ UNCOMMITTED under some special circumstances, but
you have to work very hard to make this happen. By
default, transactions are always SERIALIZABLE.

SERIALIZABLE gives you everything that REPEATABLE READ
gives you in terms of isolation.

Hi !

I see you don't understand what I saw about.
In FireBird every transactions have a tr. number, an integer.
If isolation level is high, the transaction with lesser number does not
see the modifications of the later opened transaction, because this
transaction have greater number (ID), and every transaction see only
modifications created by transactions are have lesser or equal ID.
So:
2. transaction is see 1. and 2., but does not see the 3., 4. etc,
because every records are versioned ("stamped" with it's creator's number).
Why it is good ?
When you are a reader, and some other people are writer, you don't see
any changes in database, because you see only your version of the database.
That's caused consistency. The later opened transactions can write
anything to the database, if you don't reopen your transaction, you
don't see any changes in it.
That is what I wroted about: the bill and it's items are reserved - if
anyone change the subtotals, your total and subtotals are not changed.

Only this way makes good result in the reporter applications, because if
anyone commit changes in databases, you don't see it, it is not confused
the database, and reports. If you have 6. items in a bill, and anyone
delete one of them, you don't see it. I you can see it, your bill's
total is not equal with sum of subelements !

I maked a try with Python APSW and SQLite. This is a double threaded
app, because it need to simulate the isolation between reader and writer.

It is use queue to send signals, and for control main/subthread.

################################
import threading,apsw,Queue

class TestThr(threading.Thread):
def __init__(self):
threading.Thread.__init__(self)
self.IQ=Queue.Queue()
self.OQ=Queue.Queue()

def run(self):
try:
print "*THREAD: Thread started"
while self.IQ.empty(): pass
self.IQ.get()
print "*THREAD: <<< Prepare database"
con=apsw.Connection('test.db')
c=con.cursor()
try:
c.execute('create table a(a integer)')
c.execute('end')
except:
pass
c.execute('begin')
c.execute('delete from a')
c.execute('end')
print "*THREAD: >>> Prepare database"
self.OQ.put(1)
while self.IQ.empty(): pass
self.IQ.get()
print "*THREAD: <<< Fillup 1000 values"
c.execute('begin')
print "*THREAD: Trans. started"
for i in range(1000):
c.execute('insert into a values(%d)'%i)
print "*THREAD: >>> Fillup 1000 values"
self.OQ.put(1)
while self.IQ.empty(): pass
self.IQ.get()
c.execute('end')
print "*THREAD: Trans. finished"
self.OQ.put(1)
while self.IQ.empty(): pass
self.IQ.get()
print "*THREAD: <<< Fillup 1000 values"
c.execute('begin')
print "Trans. started"
for i in range(1000,2000):
c.execute('insert into a values(%d)'%i)
print "*THREAD: >>> Fillup 1000 values"
c.execute('end')
print "*THREAD: Trans. finished"
self.OQ.put(1)
while self.IQ.empty(): pass
self.IQ.get()
print "*THREAD: Thread end"
self.OQ.put(1)
except:
print sys.exc_info()
sys.exit()

con=apsw.Connection('test.db')
c=con.cursor()

t=TestThr()
t.IQ.put(1)
t.start()
while t.OQ.empty(): pass
t.OQ.get()
#c.execute('begin')
def ReadLastRec():
rec=None
for rec in c.execute('select * from a'): pass
print "- MAIN: Read last record",rec
ReadLastRec()
t.IQ.put(1)
while t.OQ.empty(): pass
t.OQ.get()
ReadLastRec()
t.IQ.put(1)
while t.OQ.empty(): pass
t.OQ.get()
ReadLastRec()
t.IQ.put(1)
while t.OQ.empty(): pass
t.OQ.get()
ReadLastRec()
t.IQ.put(1)
while t.OQ.empty(): pass
#c.execute('end')

print "\n- MAIN: Finished"

################################
Commandline: C:\Python24\python.exe G:\dev\INFOGU~1\TESTPY~1\TESTRE~1.PY
Workingdirectory: G:\dev\InfoGuard\testpysqlite
Timeout: 0 ms

*THREAD: Thread started
*THREAD: <<< Prepare database
*THREAD: >>> Prepare database
- MAIN: Read last record None
*THREAD: <<< Fillup 1000 values
*THREAD: Trans. started
*THREAD: >>> Fillup 1000 values
- MAIN: Read last record None
*THREAD: Trans. finished
- MAIN: Read last record (999,)
*THREAD: <<< Fillup 1000 values
Trans. started
*THREAD: >>> Fillup 1000 values
*THREAD: Trans. finished
- MAIN: Read last record (1999,)
*THREAD: Thread end

- MAIN: Finished

Process "Pyhton Interpeter" terminated, ExitCode: 00000000
################################

If you can see, I simulate the reading (main thread) in one table when
the writer demon is working in the background (subthread).
I repeats the readings, and while I do this, the writer thr makes
changes to database.
As you see, I can read the another process'(thread's) writings - the
transactions are not separated like in FireBird.
It is caused that when I need to read from table I CAN GET ANOTHER
RECORD COUNTS, AND VALUES - AND THAT IS WHAT I WANT TO AVOIDING !

In FireBird the writer thread's changes not visible in the reader
thread, so I can see consistent data structure - a snapshot that not
changing before I reopen the reader transaction.

Please answer me: it is wrong I write about, or that is seems to be not
working in SQLite ?

Thanx for your help:
dd

 

[Dennis Lee Bieber]

Please answer me: it is wrong I write about, or that is seems to be not
working in SQLite ?

I don't think the feature you want to use is available -- heck, it
may ONLY be a firebird feature... (I've not seen anything similar in any
of the MySQL back-ends, or if there, I've not encountered it; and there
aren't enough easily read documents for MaxDB [aka SAP-DB] in print to
see if it has such a feature. My MSDE books don't mention it either, so
I suspect M$ SQL Server may not support such).

SQLite is a "file server" style database; Firebird is a
client/server model.

In Firebird, the server can track independent connections and
maintain state for each. SQLite runs "locally"; each connection
considers itself to be the only user of the database file (and locking
for updates is on a file basis, not table or record). Once you commit a
transaction in SQLite, the file itself is fully modified, and any other
connections see that modified file -- there is no temporary session
journal for a connection that holds a snapshot of that connection's
data.
--

 

[DurumDara]

Hi !

Dennis Lee Bieber írta:

Please answer me: it is wrong I write about, or that is seems to be not
working in SQLite ?

I don't think the feature you want to use is available -- heck, it
may ONLY be a firebird feature... (I've not seen anything similar in any
of the MySQL back-ends, or if there, I've not encountered it; and there
aren't enough easily read documents for MaxDB [aka SAP-DB] in print to
see if it has such a feature. My MSDE books don't mention it either, so
I suspect M$ SQL Server may not support such).

Sorry, but I don't think that this feature is exists in only Firebird.
Good RDMBS systems ***must have*** many transaction isolation levels.
The higher isolation level guaranteed a safely view for actual user.
READ COMMITTED, and REPEATABLE READ isolation level makes me sure, that
if I have a snapshot from the database, I get it same datas *everytime*
while my transaction opened. Until trs. is closed and reopened, I can
see the new modifications created by another user(s).
But when I keep alive my transaction, never I see any modifications,
only what I created. In this mode I don't see any user's updates -
vainly he or she do many commits (As I see in SQLite, the another user's
committed records visible for me - this is corrupting my view !)
That is the only way to I get good reports from an invoice manager
application, because I see current state of the database - the
sum(subtotals) everytime equal with total. In lower isolation level I
can see the new committed records - and that is bad thing, because in
this time the sum(subtotals) sometimes not equal with total.

That problem is just like thread synch. problem, where I need to protect
my subresources with locks/semaphores to avoid corrupting of data.
In Delphi I must protect string vars, because when every thread want to
write/read from this string, they easily make corrupted string from the
source...

SQLite is a "file server" style database; Firebird is a
client/server model.

In Firebird, the server can track independent connections and
maintain state for each. SQLite runs "locally"; each connection
considers itself to be the only user of the database file (and locking
for updates is on a file basis, not table or record). Once you commit a
transaction in SQLite, the file itself is fully modified, and any other
connections see that modified file -- there is no temporary session
journal for a connection that holds a snapshot of that connection's
data.

So I cannot use a transaction number that identify transactions, and
make same isolation effect what I said about before ?

Thanx for help:
dd

 

[Dennis Lee Bieber]

Hi !

Ah -- apparently this response was dual routed and the email version
I received didn't have a header that indicated it also went to the
group...

I'm not going to repeat the email response here other than to admit
that I did oversimplify somewhat (the description of "transaction IDs"
is what threw me -- I've not seen that in any documentation... Isolation
levels I've seen, but not in the RDBMs I tend to have most experience
with (and for those I have documentation of, "The Firebird Book" is the
only one that seems to be explicit about multiple generations of updates
based on transactions; others seem to imply that updates block all other
transactions until committed -- at which point other transactions would
see them).

{crawled through MSDN for MSDE/SQL Server; the MaxDB help file, Firebird
book, and a MySQL 5 book [which does show isolation levels for InnoDB
but is not clear on how they take effect]}
--

 

[Paul Boddie]

Isolation levels I've seen, but not in the RDBMs I tend to have most experience
with (and for those I have documentation of, "The Firebird Book" is the
only one that seems to be explicit about multiple generations of updates
based on transactions; others seem to imply that updates block all other
transactions until committed -- at which point other transactions would
see them).

Take a look here for some PostgreSQL documentation on transaction
isolation levels:

http://www.postgresql.org/docs/7.4/static/transaction-iso.html

Paul