Table Of Contents

Previous topic

TV Calendar Howto

Next topic

Math

This Page

Using the ZODB

Introduction

The Zope Object Database (ZODB) is a way of storing persistent data, just as with a relational database such as MySQL or PostgreSQL. The difference is that the data is not stored in rows or columns in a table, but instead in Python dictionaries, lists, and other data structures. Data objects are serialized and stored in a database file using pickle, a standard Python module. To retrieve data, the database file is opened, and the objects are accessed just as regular dictionaries, lists, and objects are.

The ZODB can be used by directly opening and closing a database file, which is what this document covers. For more enterprise or collaborative uses, the ZODB can be accessed through a server that manages transactions called ZEO.

Installing

First, you will need Python. Preferably at least version 2.4, version 2.5 if possible. Be sure to get easy_install. All you need to get started is Python and easy_install.

Once Python and easy_install are installed, open a command prompt (start, run, ‘cmd’) and run:

easy_install ZODB3

If you are on Windows, you may notice that a dependency, zope.interface, might not be able to install, giving a message about Visual Studio 2003. You can get around this problem by installing a specific earlier version. For example:

easy_install zope.interface==3.3.0
easy_install ZODB3

Imports

The imports are pretty straightforward.

# Imports needed to open the DB and interact with it:
from ZODB import DB
from ZODB.FileStorage import FileStorage
from ZODB.PersistentMapping import PersistentMapping
import transaction
# Imports needed for Persistent classes:
# PersistentDict and PersistentList are ready to use as is.
# Classes you create that need to be Persistent (ZODB aware)
#     should inherit from Persistent.
from persistent import Persistent
from persistent.dict import PersistentDict
from persistent.list import PersistentList

The following imports are not strictly needed to interact with the ZODB, but are extremely useful.

# Import needed for the writing function below
# see 'def writing_to_zdb()' for more info
import time
# Imports useful for datetime stuff
from datetime import datetime
from datetime import timedelta

Creating and accessing a ZODB

If a class inherits from Persistent, it will automatically become database aware. The only difference between regular dicts/lists and persistent dicts/lists is that you use PersistentDict() and PersistentList() instead of the typical {} and [] to create them. Regular dicts and lists don’t register changes, while the PersistentDict and PersistentList do. That makes them cooler.

# To open or create a DB:
storage = FileStorage("/location/zodb_file.fs")
db = DB(storage)
connection = db.open()
root = connection.root()
# To put things in the DB (overly simplistic
# see below about the function that writes to the ZODB):

# NOTE - the root of the database is a Python dictionary!

root['key'] = value # generates a transaction
transaction.get().commit() # commits the transaction, like flushing a buffer
# use standard dictionary (and list) methods to remove things:
del root['key'] # completely removes key and value, generates a transaction
transaction.get().commit() # commits the transaction, like flushing a buffer
# To close the DB:
# we definitely want a clean close, so abandon any pending writes
transaction.get().abort()
connection.close()
db.close()
storage.close()

Handy functions and their uses

Because opening and closing are done so much, I made functions to make a dict with the storage, db, connection, and root. The dict’s name is zdb:

def open_db():
    zdb = {}
    zdb['storage'] = FileStorage("/location/zodb_file.fs")
    zdb['db'] = DB(zdb['storage'])
    zdb['connection'] = zdb['db'].open()
    zdb['root'] = zdb['connection'].root()
    return zdb

def close_db(zdb):
    transaction.get().abort()
    zdb['connection'].close()
    zdb['db'].close()
    zdb['storage'].close()

Now a function that reads from the ZODB looks like this:

def only_reading_the_zdb():
    zdb = open_db() # this is always done

    # Read stuff here
    result_value = zdb['root']['key']

    close_db(zdb) # this is always done

    return result

A function that writes needs to take into account that other processes and threads may be accessing the ZODB. So we do a loop, attempting to write a certain number of pre-determined times and returning an error if we fail enough times.

# defined somewhere - top of the file, for example
max_retry = 10
sleep_delay = 1

def writing_to_zdb():
    zdb = open_db()

    retry = 0
    while retry < max_retry:
        try:
            # this is where you write to, delete from, & change the DB
            # for example, this creates a new {} that is persistent
            zdb['root']['new_dict'] = PersistentDict()

            transaction.get().commit()
        except ConflictError:
            retry += 1
            time.sleep(sleep_delay)
            pass
        else:
            break
    else:
        # retried too many times
        close_db(zdb)
        return "Error - transaction could not complete"

    close_db(zdb)
    return "Data written successfully"

An example application

Here is the structure of the database that we initially came up with for a time clock application. The idea is that a person can clock in and clock out, and we will keep track of when they did so. There is a need to keep track of employees, their attributes, and their in/out times.

When I work with the ZODB, I map out the data structures in Python notation. The whole ZODB is a Python dictionary, so I ask myself “what would it look like if I declared a variable with example data?” In this case, it looks like the following:

{
"Employees" :  {
      "employee_id_1" :   {
            "first_name" : "John",
            "last_name"  : "Doe",
            "pay_amount" : 800 # pay grade as ints to avoid rounding errors?
          },
      "employee_id_2"  :  {
            "first_name" : "Jane",
            "last_name"  : "Dowe",
            "pay_amount" : 950
          }
    },
"Pending" : {
      "employee_id_1" : datetime() # start time, user is clocked in
    },
"Records" : {
      "employee_id_1" :  [            # note that these are lists, not dicts
            [datetime(), datetime()], # start time, end time
            [datetime(), datetime()],
            [datetime(), datetime()]
          ],
      "employee_id_2"  :  [
            [datetime(), datetime()],
            [datetime(), datetime()],
            [datetime(), datetime()]
          ]
    }
}

Here’s a function that will write all of our above sample data into a database.

def write_sample_data():
    zdb = open_db()

    retry = 0
    while retry < max_retry:
        try:
            zdb['root']['Employees'] = PersistentDict()
            zdb['root']['Pending'] = PersistentDict()
            zdb['root']['Records'] = PersistentDict()

            zdb['root']['Employees']['employee_id_1'] = PersistentDict()
            zdb['root']['Employees']['employee_id_1']['first_name'] = "John"
            zdb['root']['Employees']['employee_id_1']['last_name'] = "Doe"
            zdb['root']['Employees']['employee_id_1']['pay_amount'] = 800

            zdb['root']['Employees']['employee_id_2'] = PersistentDict()
            zdb['root']['Employees']['employee_id_2']['first_name'] = "Jane"
            zdb['root']['Employees']['employee_id_2']['last_name'] = "Dowe"
            zdb['root']['Employees']['employee_id_2']['pay_amount'] = 950

            zdb['root']['Pending']['employee_id_1'] = datetime.strptime("2008-3-2-11-30", "%Y-%m-%d-%H-%M")

            zdb['root']['Records']['employee_id_1'] = []
            zdb['root']['Records']['employee_id_1'].append( [datetime.strptime("2008-3-2-11-30", "%Y-%m-%d-%H-%M"),
                                                          datetime.strptime("2008-3-2-13-30", "%Y-%m-%d-%H-%M")] )
            zdb['root']['Records']['employee_id_1'].append( [datetime.strptime("2008-3-3-11-30", "%Y-%m-%d-%H-%M"),
                                                          datetime.strptime("2008-3-3-13-30", "%Y-%m-%d-%H-%M")] )
            zdb['root']['Records']['employee_id_1'].append( [datetime.strptime("2008-3-4-11-30", "%Y-%m-%d-%H-%M"),
                                                          datetime.strptime("2008-3-4-13-30", "%Y-%m-%d-%H-%M")] )

            zdb['root']['Records']['employee_id_2'] = []
            zdb['root']['Records']['employee_id_2'].append( [datetime.strptime("2008-3-2-11-30", "%Y-%m-%d-%H-%M"),
                                                          datetime.strptime("2008-3-2-13-30", "%Y-%m-%d-%H-%M")] )
            zdb['root']['Records']['employee_id_2'].append( [datetime.strptime("2008-3-3-11-30", "%Y-%m-%d-%H-%M"),
                                                          datetime.strptime("2008-3-3-13-30", "%Y-%m-%d-%H-%M")] )
            zdb['root']['Records']['employee_id_2'].append( [datetime.strptime("2008-3-4-11-30", "%Y-%m-%d-%H-%M"),
                                                          datetime.strptime("2008-3-4-13-30", "%Y-%m-%d-%H-%M")] )

            transaction.get().commit()
        except ConflictError:
            retry += 1
            time.sleep(sleep_delay)
            pass
        else:
            break
    else:
        close_db(zdb)
        return "Error - transaction could not complete" # we retried too many times

    close_db(zdb)
    return "Data written successfully"

Complete example

Here is a complete example. If you have Python and ZODB3 installed correctly, it should be possible to run the code below. Be sure to run it from the terminal or command prompt. This demonstrates creating or opening a ZODB, clearing out any data that it might contain, then filling it up with example data and iterating over the entries.

from ZODB import DB
from ZODB.FileStorage import FileStorage
from ZODB.PersistentMapping import PersistentMapping
import transaction
from ZODB.POSException import ConflictError
from persistent import Persistent
from persistent.dict import PersistentDict
from persistent.list import PersistentList
import time
from datetime import datetime
from datetime import timedelta

max_retry = 10
sleep_delay = 1

def open_db():
    zdb = {}
    zdb['storage'] = FileStorage("timeclock.fs")
    zdb['db'] = DB(zdb['storage'])
    zdb['connection'] = zdb['db'].open()
    zdb['root'] = zdb['connection'].root()
    return zdb

def close_db(zdb):
    transaction.get().abort()
    zdb['connection'].close()
    zdb['db'].close()
    zdb['storage'].close()

def clear_data():
    zdb = open_db()

    if len(zdb['root'].keys()) > 0:
        retry = 0
        for k in zdb['root'].keys():
            while retry < max_retry:
                try:
                    del zdb['root'][k]
                    transaction.get().commit()
                except ConflictError:
                    retry += 1
                    time.sleep(sleep_delay)
                    pass
                else:
                    break
            else:
                close_db(zdb)
                return "Error - transaction could not complete" # we retried too many times

    close_db(zdb)
    return "Database cleared"

def get_data():
    zdb = open_db()

    result = "Employees\n---------"
    for employee_id in zdb['root']['Employees'].keys():
        result += "\nUser ID: " + employee_id + "\n"
        result += "Name: " + zdb['root']['Employees'][employee_id]['first_name'] + " " + zdb['root']['Employees'][employee_id]['last_name'] + "\n"
        result += "Pay: " + repr(zdb['root']['Employees'][employee_id]['pay_amount']) + "\n"
        if zdb['root']['Pending'].has_key(employee_id):
            result += "Currently Clocked In: " + repr(zdb['root']['Pending'][employee_id]) + "\n"
        else:
            result += "Currently Clocked In: " + "No\n"
        result += "Records\n-------"
        for record in zdb['root']['Records'][employee_id]:
            result += "\nClocked In: " + repr(record[0])
            result += "\nClocked Out: " + repr(record[1])
        result += "\n"

    close_db(zdb)
    return result

def write_sample_data():
    zdb = open_db()

    retry = 0
    while retry < max_retry:
        try:
            zdb['root']['Employees'] = PersistentDict()
            zdb['root']['Pending'] = PersistentDict()
            zdb['root']['Records'] = PersistentDict()

            zdb['root']['Employees']['employee_id_1'] = PersistentDict()
            zdb['root']['Employees']['employee_id_1']['first_name'] = "John"
            zdb['root']['Employees']['employee_id_1']['last_name'] = "Doe"
            zdb['root']['Employees']['employee_id_1']['pay_amount'] = 800

            zdb['root']['Employees']['employee_id_2'] = PersistentDict()
            zdb['root']['Employees']['employee_id_2']['first_name'] = "Jane"
            zdb['root']['Employees']['employee_id_2']['last_name'] = "Dowe"
            zdb['root']['Employees']['employee_id_2']['pay_amount'] = 950

            zdb['root']['Pending']['employee_id_1'] = datetime.strptime("2008-3-2-11-30", "%Y-%m-%d-%H-%M")

            zdb['root']['Records']['employee_id_1'] = []
            zdb['root']['Records']['employee_id_1'].append( [datetime.strptime("2008-3-2-11-30", "%Y-%m-%d-%H-%M"),
                                                          datetime.strptime("2008-3-2-13-30", "%Y-%m-%d-%H-%M")] )
            zdb['root']['Records']['employee_id_1'].append( [datetime.strptime("2008-3-3-11-30", "%Y-%m-%d-%H-%M"),
                                                          datetime.strptime("2008-3-3-13-30", "%Y-%m-%d-%H-%M")] )
            zdb['root']['Records']['employee_id_1'].append( [datetime.strptime("2008-3-4-11-30", "%Y-%m-%d-%H-%M"),
                                                          datetime.strptime("2008-3-4-13-30", "%Y-%m-%d-%H-%M")] )

            zdb['root']['Records']['employee_id_2'] = []
            zdb['root']['Records']['employee_id_2'].append( [datetime.strptime("2008-3-2-11-30", "%Y-%m-%d-%H-%M"),
                                                          datetime.strptime("2008-3-2-13-30", "%Y-%m-%d-%H-%M")] )
            zdb['root']['Records']['employee_id_2'].append( [datetime.strptime("2008-3-3-11-30", "%Y-%m-%d-%H-%M"),
                                                          datetime.strptime("2008-3-3-13-30", "%Y-%m-%d-%H-%M")] )
            zdb['root']['Records']['employee_id_2'].append( [datetime.strptime("2008-3-4-11-30", "%Y-%m-%d-%H-%M"),
                                                          datetime.strptime("2008-3-4-13-30", "%Y-%m-%d-%H-%M")] )

            transaction.get().commit()
        except ConflictError:
            retry += 1
            time.sleep(sleep_delay)
            pass
        else:
            break
    else:
        close_db(zdb)
        return "Error - transaction could not complete" # we retried too many times

    close_db(zdb)
    return "Data written successfully"

print clear_data()
print
print write_sample_data()
print
print get_data()