45. More Complex Behaviors

We want a group of reviewers to vote on the talks that are accepted for the conference.

In this part you will:

  • Write a behavior that enables voting on content

  • Use annotations to store the votes on a object

Topics covered:

  • Behaviors with a factory class

  • Marker Interfaces

  • Using Annotations as storage-layer

45.1. Using Annotations

We are going to store the information in an annotation. Not because it is needed but because you will find code that uses annotations and need to understand the implications.

Annotations in Zope/Plone mean that data won’t be stored directly on an object but in an indirect way with namespaces so that multiple packages can store information under the same attribute, without colliding.

So using annotations avoids namespace conflicts. The cost is an indirection. The dictionary is persistent so it has to be stored separately. Alternatively, one could give attributes a name containing a namespace prefix to avoid naming collisions.

45.2. Using Schema

The attribute where we store our data will be declared as a schema field.

We mark the field as an omitted field (using a schema directive similar to read_permission or widget), because we are not going to create z3c.form widgets for entering or displaying them. We do provide a schema, because many other packages use the schema information to gain knowledge about the relevant fields.

For example, when files were migrated to blobs, new objects had to be created and every schema field was copied. The code can not know about our field, except if we provide schema information.

45.3. Writing Code

To start, we create a directory behavior with an empty behavior/__init__.py file.

Next we must, as always, register our ZCML.

First, add the information that there will be another ZCML file in configure.zcml

1<configure xmlns="...">
2
3  ...
4  <include package=".behavior" />
5  ...
6
7</configure>

Next, create behavior/configure.zcml

 1<configure
 2    xmlns="http://namespaces.zope.org/zope"
 3    xmlns:plone="http://namespaces.plone.org/plone">
 4
 5  <plone:behavior
 6      title="Voting"
 7      name="starzel.voting"
 8      description="Allow voting for an item"
 9      provides="starzel.votable_behavior.interfaces.IVoting"
10      factory=".voting.Vote"
11      marker="starzel.votable_behavior.interfaces.IVotable"
12      />
13
14</configure>

There are some important differences to the first behavior:

  • There is a marker interface

  • There is a factory

The first behavior (discussed in Behaviors) was registered using only provides:

<plone:behavior
    title="Featured"
    name="ploneconf.featured"
    description="Control if a item is shown on the frontpage"
    provides=".featured.IFeatured"
    />

The factory is a class that provides the behavior logic and gives access to the attributes we provide. Factories in Plone/Zope land are retrieved by adapting an object to an interface and are following the adapter pattern. If you want your behavior, you would write voting = IVoting(object).

But in order for this to work, your object may not be implementing the IVoting interface, because if it did, IVoting(object) would return the object itself! If you need a marker interface for objects providing the new behavior, you must provide one, for this you use marker. This object now implements the marker-interface IVotable. Because of this, we can write views and viewlets just for content that use this behavior.

The interfaces need to be written, in our case into a file interfaces.py:

 1# encoding=utf-8
 2from plone import api
 3from plone.autoform import directives
 4from plone.autoform.interfaces import IFormFieldProvider
 5from plone.supermodel import model
 6from plone.supermodel.directives import fieldset
 7from zope import schema
 8from zope.interface import Interface
 9from zope.interface import provider
10
11class IVotableLayer(Interface):
12    """Marker interface for the Browserlayer of this addon
13    """
14
15# IVotable is the marker interface for contenttypes who support this behavior
16class IVotable(Interface):
17    pass
18
19# This is the behavior interface.
20# When doing IVoting(object), you receive an adapter
21@provider(IFormFieldProvider)
22class IVoting(model.Schema):
23    if not api.env.debug_mode():
24        directives.omitted("votes")
25        directives.omitted("voted")
26
27    fieldset(
28        'debug',
29        label=u'debug',
30        fields=('votes', 'voted'),
31    )
32
33    votes = schema.Dict(title=u"Vote info",
34                        key_type=schema.TextLine(title=u"Voted number"),
35                        value_type=schema.Int(title=u"Voted so often"),
36                        required=False,
37                        default={},
38                        missing_value={},
39                        )
40    voted = schema.List(title=u"Vote hashes",
41                        value_type=schema.TextLine(),
42                        required=False,
43                        default=[],
44                        missing_value=[],
45                    )
46
47    def vote(request):
48        """
49        Store the vote information, store the request hash to ensure
50        that the user does not vote twice
51        """
52
53    def average_vote():
54        """
55        Return the average voting for an item
56        """
57
58    def has_votes():
59        """
60        Return whether anybody ever voted for this item
61        """
62
63    def already_voted(request):
64        """
65        Return the information wether a person already voted.
66        This is not very high level and can be tricked out easily
67        """
68
69    def clear():
70        """
71        Clear the votes. Should only be called by admins
72        """

This is a lot of code.

The IVotableLayer will be needed later for viewlets and browser views to only be available when this addon is installed.

The IVotable interface is just the simple marker interface. It will only be used to bind browser views and viewlets to contenttypes that provide our behavior, so no code is needed.

The IVoting class is more complex, as you can see.

The @provider decorator above the class ensures that the schema fields are known to other packages. Whenever some code wants all schemas from an object, it receives the schema defined directly on the object and the additional schemata. Additional schemata are compiled by looking for behaviors and whether they provide the IFormFieldProvider functionality. Only then the fields are used as form fields.

While IVoting is just an interface, we use plone.supermodel.model.Schema for advanced dexterity features. zope.schema provides no means for hiding fields.

The directives form.omitted from plone.autoform allow us to annotate this additional information so that the autoform renderers for forms can use the additional information. We make this omit conditional. If we run Plone in debug mode, we will be able to see the internal data in the edit form.

We create minimal schema fields for our internal data structures. For a small test, I removed the form omitted directives and opened the edit view of a talk that uses the behavior. After seeing the ugliness, I decided that I should provide at least minimum of information. title and required are purely optional, but very helpful if the fields won’t be omitted, something that can be helpful when debugging the behavior. Later, when we implement the behavior, the votes and voted attributes are implemented in such a way that you can’t just modify these fields, they are read only.

Then we define the API that we are going to use in the frontend.

Now the only thing that is missing is the behavior implementation, which we must put into behavior/voting.py.

 1# encoding=utf-8
 2from starzel.votable_behavior.interfaces import IVotable, IVoting
 3from hashlib import md5
 4from persistent.dict import PersistentDict
 5from persistent.list import PersistentList
 6from zope.annotation.interfaces import IAnnotations
 7from zope.component import adapter
 8from zope.interface import implementer
 9
10KEY = "starzel.votable_behavior.behavior.voting.Vote"
11
12
13@implementer(IVoting)
14@adapter(IVotable)
15class Vote(object):
16    def __init__(self, context):
17        self.context = context
18        annotations = IAnnotations(context)
19        if KEY not in annotations.keys():
20            annotations[KEY] = PersistentDict({
21                "voted": PersistentList(),
22                'votes': PersistentDict()
23                })
24        self.annotations = annotations[KEY]
25
26    @property
27    def votes(self):
28        return self.annotations['votes']
29
30    @property
31    def voted(self):
32        return self.annotations['voted']

In our __init__ method we get annotations from the object. We look for data with a specific key.

The key in this example is the same as what I would get with __name__+Vote.__name__. But we won’t create a dynamic name, this would be very clever and clever is bad.

By declaring a static name, we won’t run into problems if we restructure the code.

You can see that we initialize the data if it doesn’t exist. We work with PersistentDict and PersistentList. To understand why we do this, it is important to understand how the ZODB works.

See also

The ZODB can store objects. It has a special root object that you will never touch. Whatever you store there, will be part of the root object, except if it is an object subclassing persistent.Persistent. Then it will be stored independently.

Zope/ZODB persistent objects note when you change an attribute on it and mark itself as changed. Changed objects will be saved to the database. This happens automatically. Each request begins a transaction and after our code runs and the Zope Server is preparing to send back the response we generated, the transaction will be committed and everything we changed will be saved.

Now, if you have a normal dictionary on a persistent object, and you will only change the dictionary, the persistent object has no way to know if the dictionary has been changed. This happens from time to time.

So one solution is to change the special attribute _p_changed to True (or any other value!) on the persistent object, or to use a PersistentDict. The latter is what we are doing here.

An important thing to note about PersistentDict and PersistentList is that they cannot handle write conflicts. What happens if two users rate the same content independently at the same time? In this case, a database conflict will occur because there is no way for Plone to know how to handle the concurrent write access. Although this is rather unlikely during this training, it is a very common problem on high traffic websites.

You can find more information in the documentation of the ZODB, in particular Rules for Persistent Classes

Next we provide the internal fields via properties. Using this form of property makes them read-only properties, as we did not define write handlers. We don’t need them so we won’t add them.

As you have seen in the Schema declaration, if you run your site in debug mode, you will see an edit field for these fields. But trying to change these fields will throw an exception.

Let’s continue with this file:

 1    def _hash(self, request):
 2        """
 3        This hash can be tricked out by changing IP addresses and might allow
 4        only a single person of a big company to vote
 5        """
 6        hash_ = md5()
 7        hash_.update(request.getClientAddr())
 8        for key in ["User-Agent", "Accept-Language", "Accept-Encoding"]:
 9            hash_.update(request.getHeader(key))
10        return hash_.hexdigest()
11
12    def vote(self, vote, request):
13        if self.already_voted(request):
14            raise KeyError("You may not vote twice")
15        vote = int(vote)
16        self.annotations['voted'].append(self._hash(request))
17        votes = self.annotations['votes']
18        if vote not in votes:
19            votes[vote] = 1
20        else:
21            votes[vote] += 1
22
23    def average_vote(self):
24        if not has_votes(self):
25            return 0
26        total_votes = sum(self.annotations['votes'].values())
27        total_points = sum(
28            [vote * count for (vote, count) in self.annotations['votes'].items()])
29        return float(total_points) / total_votes
30
31    def has_votes(self):
32        return len(self.annotations.get('votes', {})) != 0
33
34    def already_voted(self, request):
35        return self._hash(request) in self.annotations['voted']
36
37    def clear(self):
38        annotations = IAnnotations(self.context)
39        annotations[KEY] = PersistentDict(
40            {'voted': PersistentList(), 'votes': PersistentDict()}
41        )
42        self.annotations = annotations[KEY]

We start with a little helper method which is not exposed via the interface. We don’t want people to vote twice. There are many ways to ensure this and each one has flaws.

We chose this way to show you how to access information from the request that the browser of the user sent to us.

First, we get the IP address of the user, then we access a small set of headers from the user’s browser and generate an md5 checksum from this data.

The vote method requires a vote and a request. We check the preconditions, then we convert the vote to an integer, store the request to voted and the votes into the votes dictionary. We just count there how often any vote has been given.

Everything else is just python.

45.3.1. Exercises

45.3.1.1. Exercise 1

Refactor the voting behavior so that it uses BTrees instead of PersistentDict and PersistentList. Use OOBTree to replace PersistentDict and OIBTree to replace PersistentList.

Solution

change behavior/voting.py

# encoding=utf-8
from .interfaces import IVoting
from BTrees.OIBTree import OIBTree
from BTrees.OOBTree import OOBTree
from hashlib import md5
from zope.annotation.interfaces import IAnnotations
from zope.interface import implementer

KEY = "starzel.votable_behavior.behavior.voting.Vote"

@implementer(IVoting)
class Vote(object):
    def __init__(self, context):
        self.context = context
        annotations = IAnnotations(context)
        if KEY not in annotations.keys():
            self.clear()
        else:
            self.annotations = annotations[KEY]

    ...

    def vote(self, vote, request):
        if self.already_voted(request):
            raise KeyError("You may not vote twice")
        vote = int(vote)
        self.annotations['voted'].insert(
            self._hash(request),
            len(self.annotations['voted']))
        votes = self.annotations['votes']
        if vote not in votes:
            votes[vote] = 1
        else:
            votes[vote] += 1

    ...

    def clear(self):
        annotations = IAnnotations(self.context)
        annotations[KEY] = OOBTree()
        annotations[KEY]['voted'] = OIBTree()
        annotations[KEY]['votes'] = OOBTree()
        self.annotations = annotations[KEY]

45.3.1.2. Exercise 2

Write a unit test that simulates concurrent voting. The test should raise a ConflictError on the original voting behavior implementation. The solution from the first exercise should pass. Look at https://zodb.org/en/latest/ConflictResolution.html for how to create a suitable test fixture for conflict testing. Look at the test code in zope.annotation for how to create annotatable dummy content. You will also have to write a ‘request’ dummy that mocks the getClientAddr and getHeader methods of Zope’s HTTP request object to make the _hash method of the voting behavior work.

Solution

There are no tests for starzel.votablebehavior at all at the moment. But you can refer to https://training.plone.org/5/testing/testing_setup.html for how to setup unit testing for a package. Put the particular test for this exercise into a file named starzel.votable_behavior/starzel/votable_behavior/tests/test_voting.py. Remember you need an empty __init__.py file in the tests directory to make testing work. You also need to add starzel.votable_behavior to test-eggs in buildout.cfg and re-run buildout.

 1from persistent import Persistent
 2from zope.annotation.attribute import AttributeAnnotations
 3from zope.annotation.interfaces import IAttributeAnnotatable
 4from zope.interface import implementer
 5
 6import tempfile
 7import transaction
 8import unittest
 9import ZODB
10
11@implementer(IAttributeAnnotatable)
12class Dummy(Persistent):
13    pass
14
15
16
17class RequestDummy(object):
18
19    def __init__(self, ip, headers=None):
20        self.ip = ip
21        if headers is not None:
22            self.headers = headers
23        else:
24            self.headers = {
25                'User-Agent': 'foo',
26                'Accept-Language': 'bar',
27                'Accept-Encoding': 'baz'
28                }
29
30    def getClientAddr(self):
31        return self.ip
32
33    def getHeader(self, key):
34        return self.headers[key]
35
36
37class VotingTests(unittest.TestCase):
38
39    def test_voting_conflict(self):
40        from starzel.votable_behavior.behavior.voting import Vote
41        dbname = tempfile.mktemp()
42        db = ZODB.DB(dbname)
43        tm_A = transaction.TransactionManager()
44        conn_A = db.open(transaction_manager=tm_A)
45        p_A = conn_A.root()['voting'] = Vote(AttributeAnnotations(Dummy()))
46        tm_A.commit()
47        # Now get another copy of 'p' so we can make a conflict.
48        # Think of `conn_A` (connection A) as one thread, and
49        # `conn_B` (connection B) as a concurrent thread.  `p_A`
50        # is a view on the object in the first connection, and `p_B`
51        # is a view on *the same persistent object* in the second connection.
52        tm_B = transaction.TransactionManager()
53        conn_B = db.open(transaction_manager=tm_B)
54        p_B = conn_B.root()['voting']
55        assert p_A.context.obj._p_oid == p_B.context.obj._p_oid
56        # Now we can make a conflict, and see it resolved (or not)
57        request_A = RequestDummy('192.168.0.1')
58        p_A.vote(1, request_A)
59        request_B = RequestDummy('192.168.0.5')
60        p_B.vote(2, request_B)
61        tm_B.commit()
62        tm_A.commit()