40. More Complex Behaviors

In this part you will:

  • Write an annotation

Topics covered:

  • Annotation Marker Interfaces

We are working in the newly created add-on starzel.votable_behavior we just created using mr.bob.

40.1. Using Annotations

We are going to store the information in an annotation. Not because it is needed but because you will find code that uses annotations and need to understand the implications.

Annotations in Zope/Plone mean that data won’t be stored directly on an object but in an indirect way with namespaces so that multiple packages can store information under the same attribute, without colliding.

So using annotations avoids namespace conflicts. The cost is an indirection. The dictionary is persistent so it has to be stored separately. Also, one could give attributes a name containing a namespace prefix to avoid naming collisions.

40.2. Using Schema

The attribute where we store our data will be declared as a schema field. We mark the field as an omitted field (using schema directive similar to read_permission or widget), because we are not going to create z3c.form widgets for entering or displaying them. We do provide a schema, because many other packages use the schema information to get knowledge of the relevant fields.

For example, when files were migrated to blobs, new objects had to be created and every schema field was copied. The code can not know about our field, except if we provide schema information.

40.3. Writing Code

To start, we create a directory behavior with an empty behavior/__init__.py file.

Next we must, as always, register our ZCML.

First, add the information that there will be another ZCML file in configure.zcml

1<configure xmlns="...">
2
3  ...
4  <include package=".behavior" />
5  ...
6
7</configure>

Next, create behavior/configure.zcml

 1<configure
 2    xmlns="http://namespaces.zope.org/zope"
 3    xmlns:plone="http://namespaces.plone.org/plone">
 4
 5  <plone:behavior
 6      title="Voting"
 7      name="starzel.voting"
 8      description="Allow voting for an item"
 9      provides="starzel.votable_behavior.interfaces.IVoting"
10      factory=".voting.Vote"
11      marker="starzel.votable_behavior.interfaces.IVotable"
12      />
13
14</tfigure>

There are some important differences to our first behavior:

  • There is a marker interface

  • There is a factory

The factory is a class that provides the behavior logic and gives access to the attributes we provide. Factories in Plone/Zope land are retrieved by adapting an object to an interface and are following the adapter pattern. If you want your behavior, you would write voting = IVoting(object).

But in order for this to work, your object may not be implementing the IVoting interface, because if it did, IVoting(object) would return the object itself! If I need a marker interface for objects providing my behavior, I must provide one, for this we use the marker attribute. My object implements IVotable. Because of this, we can write views and viewlets just for this content type.

The interfaces need to be written, in our case into a file interfaces.py:

 1# encoding=utf-8
 2from plone import api
 3from plone.autoform import directives
 4from plone.autoform.interfaces import IFormFieldProvider
 5from plone.supermodel import model
 6from plone.supermodel.directives import fieldset
 7from zope import schema
 8from zope.interface import Interface
 9from zope.interface import provider
10
11class IVotableLayer(Interface):
12    """Marker interface for the Browserlayer
13    """
14
15# Ivotable is the marker interface for contenttypes who support this behavior
16class IVotable(Interface):
17    pass
18
19# This is the behaviors interface. When doing IVoting(object), you receive an
20# adapter
21@provider(IFormFieldProvider)
22class IVoting(model.Schema):
23    if not api.env.debug_mode():
24        directives.omitted("votes")
25        directives.omitted("voted")
26
27    fieldset(
28        'debug',
29        label=u'debug',
30        fields=('votes', 'voted'),
31    )
32
33    votes = schema.Dict(title=u"Vote info",
34                        key_type=schema.TextLine(title=u"Voted number"),
35                        value_type=schema.Int(title=u"Voted so often"),
36                        required=False)
37    voted = schema.List(title=u"Vote hashes",
38                        value_type=schema.TextLine(),
39                        required=False)
40
41    def vote(request):
42        """
43        Store the vote information, store the request hash to ensure
44        that the user does not vote twice
45        """
46
47    def average_vote():
48        """
49        Return the average voting for an item
50        """
51
52    def has_votes():
53        """
54        Return whether anybody ever voted for this item
55        """
56
57    def already_voted(request):
58        """
59        Return the information wether a person already voted.
60        This is not very high level and can be tricked out easily
61        """
62
63    def clear():
64        """
65        Clear the votes. Should only be called by admins
66        """

This is a lot of code. The IVotableLayer we will need later for viewlets and browser views. Let’s add it right here. The IVotable interface is the simple marker interface. It will only be used to bind browser views and viewlets to contenttypes that provide our behavior, so no code needed.

The IVoting class is more complex, as you can see.

The @provider decorator above the class ensures that the schema fields are known to other packages. Whenever some code wants all schemas from an object, it receives the schema defined directly on the object and the additional schemata. Additional schemata are compiled by looking for behaviors and whether they provide the IFormFieldProvider functionality. Only then the fields are used as form fields.

While IVoting is just an interface, we use plone.supermodel.model.Schema for advanced dexterity features. zope.schema provides no means for hiding fields.

The directives form.omitted from plone.autoform allow us to annotate this additional information so that the autoform renderers for forms can use the additional information. We make this omit conditional. If we run Plone in debug mode, we will be able to see the internal data in the edit form.

We create minimal schema fields for our internal data structures. For a small test, I removed the form omitted directives and opened the edit view of a talk that uses the behavior. After seeing the ugliness, I decided that I should provide at least minimum of information. title and required are purely optional, but very helpful if the fields won’t be omitted, something that can be helpful when debugging the behavior. Later, when we implement the behavior, the votes and voted attributes are implemented in such a way that you can’t just modify these fields, they are read only.

Then we define the API that we are going to use in browser views and viewlets.

Now the only thing that is missing is the behavior implementation, which we must put into behavior/voting.py

 1# encoding=utf-8
 2from .interfaces import IVoting
 3from hashlib import md5
 4from persistent.dict import PersistentDict
 5from persistent.list import PersistentList
 6from zope.annotation.interfaces import IAnnotations
 7from zope.interface import implementer
 8
 9KEY = "starzel.votable_behavior.behavior.voting.Vote"
10
11
12@implementer(IVoting)
13class Vote(object):
14    def __init__(self, context):
15        self.context = context
16        annotations = IAnnotations(context)
17        if KEY not in annotations.keys():
18            annotations[KEY] = PersistentDict({
19                "voted": PersistentList(),
20                'votes': PersistentDict()
21                })
22        self.annotations = annotations[KEY]
23
24    @property
25    def votes(self):
26        return self.annotations['votes']
27
28    @property
29    def voted(self):
30        return self.annotations['voted']

In our __init__ method we get annotations from the object. We look for data with a specific key.

The key in this example is the same as what I would get with __name__+Vote.__name__. But we won’t create a dynamic name, this would be very clever and clever is bad.

By declaring a static name, we won’t run into problems if we restructure the code.

You can see that we initialize the data if it doesn’t exist. We work with PersistentDict and PersistentList. To understand why we do this, it is important to understand how the ZODB works.

See also

The ZODB can store objects. It has a special root object that you will never touch. Whatever you store there, will be part of the root object, except if it is an object subclassing persistent.Persistent. Then it will be stored independently.

Zope/ZODB persistent objects note when you change an attribute on it and mark itself as changed. Changed objects will be saved to the database. This happens automatically. Each request begins a transaction and after our code runs and the Zope Server is preparing to send back the response we generated, the transaction will be committed and everything we changed will be saved.

Now, if have a normal dictionary on a persistent object, and you will only change the dictionary, the persistent object has no way to know if the dictionary has been changed. This happens from time to time.

So one solution is to change the special attribute _p_changed to True (or any other value!) on the persistent object, or to use a PersistentDict. Latter is what we are doing here.

An important thing to note about PersistentDict and PersistentList is that they cannot handle write conflicts. What happens if two users rate the same content independently at the same time? In this case, a database conflict will occur because there is no way for Plone to know how to handle the concurrent write access. Although this is rather unlikely during this training, it is a very common problem on high traffic websites.

You can find more information in the documentation of the ZODB, in particular Rules for Persistent Classes

Next we provide the internal fields via properties. Using this form of property makes them read only properties, as we did not define write handlers. We don’t need them so we won’t add them.

As you have seen in the Schema declaration, if you run your site in debug mode, you will see an edit field for these fields. But trying to change these fields will throw an exception.

Let’s continue with the behavior/voting.py file, inside the Vote class:

 1    def _hash(self, request):
 2        """
 3        This hash can be tricked out by changing IP addresses and might allow
 4        only a single person of a big company to vote
 5        """
 6        hash_ = md5()
 7        hash_.update(request.getClientAddr())
 8        for key in ["User-Agent", "Accept-Language", "Accept-Encoding"]:
 9            hash_.update(request.getHeader(key))
10        return hash_.hexdigest()
11
12    def vote(self, vote, request):
13        if self.already_voted(request):
14            raise KeyError("You may not vote twice")
15        vote = int(vote)
16        self.annotations['voted'].append(self._hash(request))
17        votes = self.annotations['votes']
18        if vote not in votes:
19            votes[vote] = 1
20        else:
21            votes[vote] += 1
22
23    def average_vote(self):
24        if not has_votes(self):
25            return 0
26        total_votes = sum(self.annotations['votes'].values())
27        total_points = sum(
28            [vote * count for (vote, count) in self.annotations['votes'].items()])
29        return float(total_points) / total_votes
30
31    def has_votes(self):
32        return len(self.annotations.get('votes', [])) != 0
33
34    def already_voted(self, request):
35        return self._hash(request) in self.annotations['voted']
36
37    def clear(self):
38        annotations = IAnnotations(self.context)
39        annotations[KEY] = PersistentDict(
40            {'voted': PersistentList(), 'votes': PersistentDict()}
41        )
42        self.annotations = annotations[KEY]

We start with a little helper method which is not exposed via the interface. We don’t want people to vote twice. There are many ways to ensure this and each one has flaws.

We chose this way to show you how to access information from the request the browser of the user sent to us. First, we get the IP address of the user, then we access a small set of headers from the user’s browser and generate an md5 checksum of this.

The vote method wants a vote and a request. We check the preconditions, then we convert the vote to an integer, store the request to voted and the votes into the votes dictionary. We just count there how often any vote has been given.

Everything else is just python.

40.3.1. Exercises

40.3.1.1. Exercise 1

Refactor the voting behavior so that it uses BTrees instead of PersistentDict and PersistentList. Use OOBTree to replace PersistentDict and OIBTree to replace PersistentList.

Solution

change behavior/voting.py

# encoding=utf-8
from .interfaces import IVoting
from BTrees.OIBTree import OIBTree
from BTrees.OOBTree import OOBTree
from hashlib import md5
from zope.annotation.interfaces import IAnnotations
from zope.interface import implementer

KEY = "starzel.votable_behavior.behavior.voting.Vote"

@implementer(IVoting)
class Vote(object):
    def __init__(self, context):
        self.context = context
        annotations = IAnnotations(context)
        if KEY not in annotations.keys():
            self.clear()
        else:
            self.annotations = annotations[KEY]

    ...

    def vote(self, vote, request):
        if self.already_voted(request):
            raise KeyError("You may not vote twice")
        vote = int(vote)
        self.annotations['voted'].insert(
            self._hash(request),
            len(self.annotations['voted']))
        votes = self.annotations['votes']
        if vote not in votes:
            votes[vote] = 1
        else:
            votes[vote] += 1

    ...

    def clear(self):
        annotations = IAnnotations(self.context)
        annotations[KEY] = OOBTree()
        annotations[KEY]['voted'] = OIBTree()
        annotations[KEY]['votes'] = OOBTree()
        self.annotations = annotations[KEY]

40.3.1.2. Exercise 2

Write a unit test that simulates concurrent voting. The test should raise a ConflictError on the original voting behavior implementation. The solution from the first exercise should pass. Look at the file ZODB/ConflictResolution.txt in the ZODB3 egg for how to create a suitable test fixture for conflict testing. Look at the test code in zope.annotation for how to create annotatable dummy content. You will also have to write a ‘request’ dummy that mocks the getClientAddr and getHeader methods of Zope’s HTTP request object to make the _hash method of the voting behavior work.

Solution

There are no tests for starzel.votablebehavior at all at the moment. But you can refer to chapter 23 (Testing in Plone) for how to setup unit testing for a package. Put the particular test for this exercise into a file named starzel.votable_behavior/starzel/votable_behavior/tests/test_voting. Remember you need an empty __init__.py file in the tests directory to make testing work. You also need to add starzel.votable_behavior to test-eggs in buildout.cfg and re-run buildout.

 1from persistent import Persistent
 2from zope.annotation.attribute import AttributeAnnotations
 3from zope.annotation.interfaces import IAttributeAnnotatable
 4from zope.interface import implementer
 5
 6import tempfile
 7import transaction
 8import unittest
 9import ZODB
10
11@implementer(IAttributeAnnotatable)
12class Dummy(Persistent):
13    pass
14
15
16
17class RequestDummy(object):
18
19    def __init__(self, ip, headers=None):
20        self.ip = ip
21        if headers is not None:
22            self.headers = headers
23        else:
24            self.headers = {
25                'User-Agent': 'foo',
26                'Accept-Language': 'bar',
27                'Accept-Encoding': 'baz'
28                }
29
30    def getClientAddr(self):
31        return self.ip
32
33    def getHeader(self, key):
34        return self.headers[key]
35
36
37class VotingTests(unittest.TestCase):
38
39    def test_voting_conflict(self):
40        from starzel.votable_behavior.behavior.voting import Vote
41        dbname = tempfile.mktemp()
42        db = ZODB.DB(dbname)
43        tm_A = transaction.TransactionManager()
44        conn_A = db.open(transaction_manager=tm_A)
45        p_A = conn_A.root()['voting'] = Vote(AttributeAnnotations(Dummy()))
46        tm_A.commit()
47        # Now get another copy of 'p' so we can make a conflict.
48        # Think of `conn_A` (connection A) as one thread, and
49        # `conn_B` (connection B) as a concurrent thread.  `p_A`
50        # is a view on the object in the first connection, and `p_B`
51        # is a view on *the same persistent object* in the second connection.
52        tm_B = transaction.TransactionManager()
53        conn_B = db.open(transaction_manager=tm_B)
54        p_B = conn_B.root()['voting']
55        assert p_A.context.obj._p_oid == p_B.context.obj._p_oid
56        # Now we can make a conflict, and see it resolved (or not)
57        request_A = RequestDummy('192.168.0.1')
58        p_A.vote(1, request_A)
59        request_B = RequestDummy('192.168.0.5')
60        p_B.vote(2, request_B)
61        tm_B.commit()
62        tm_A.commit()