EuroDjangoCon 2009

EuroDjangoCon 2009 is still going strong, but I wanted to share the materials from my talk as quickly as possible. My slides are on Slide Share:

Forms, Getting Your Money's Worth

View more presentations from kingkilr.

And the first examples code follows:

from django.forms.util import ErrorList
from django.utils.datastructures import SortedDict

def multiple_form_factory(form_classes, form_order=None):
   if form_order:
       form_classes = SortedDict([(prefix, form_classes[prefix]) for prefix in
           form_order])
   else:
       form_classes = SortedDict(form_classes)
   return type('MultipleForm', (MultipleFormBase,), {'form_classes': form_classes})

class MultipleFormBase(object):
   def __init__(self, data=None, files=None, auto_id='id_%s', prefix=None,
       initial=None, error_class=ErrorList, label_suffix=':',
       empty_permitted=False):
       if prefix is None:
           prefix = ''
       self.forms = [form_class(data, files, auto_id, prefix + form_prefix,
           initial[i], error_class, label_suffix, empty_permitted) for
           i, (form_prefix, form_class) in enumerate(self.form_classes.iteritems())]

   def __unicode__(self):
       return self.as_table()

   def __iter__(self):
       for form in self.forms:
           for field in form:
               yield field

   def is_valid(self):
       return all(form.is_valid() for form in self.forms)

   def as_table(self):
       return '\n'.join(form.as_table() for form in self.forms)

   def as_ul(self):
       return '\n'.join(form.as_ul() for form in self.forms)

   def as_p(self):
       return '\n'.join(form.as_p() for form in self.forms)

   def is_multipart(self):
       return any(form.is_multipart() for form in self.forms)

   def save(self, commit=True):
       return tuple(form.save(commit) for form in self.forms)
   save.alters_data = True

EuroDjangoCon has been a blast thus far and after the conference I'll do a blogpost that does it justice.

Ajax Validation Aministrivia

Small administrative note, now that Github has it's own issue tracker users wishing to report issues with django-ajax-validation can now do so on Github, instead of the old Google Code page. Django-filter users can also report issues on its Github page, in place of the former system of "message or email me", which, though functional, wasn't very convenient.

ORM Panel Recap

Having now completed what I thought was a quite successful panel I thought it would be nice to do a review of some of the decisions I made, that some people had been asking about. For those who missed it you can find a live blog of the event by James Bennett at his blog, and a video should hopefully be going up sometime soon.

Why Google App Engine

As Guido pointed out App Engine does not have an ORM, as App Engine doesn't have a relational datastore. However, it does have something that looks and acts quite a lot like other ORMs, and it does fundamentally try to serve the same purpose, offering a persistence layer. Therefore I decided it was at least in the same class of items I wanted to add. Further, with the rise of non-relational DBs that all fundamentally deal with the same issues as App Engine, and the relationship between ORMs and these new persistence layers I thought it would be advantageous to have one of these, Guido is a knowledgeable and interesting person, and that's how the cookie crumbled.

Why Not ZODB/Storm/A Talking Pony

Time. I would have loved to have as many different ORMs/things like them as exist in the Python eco-sphere, but there just wasn't time. We had 55 minutes to present and as it is that wasn't enough. I ultimately had time to ask 3 questions(one of which was just background), plus 5 shorter audience questions. I was forced to cut out several questions I wanted to ask, but didn't have time to, for those who are interested the major questions I would have liked to ask
were:

• What most often requested feature won't you add to your ORM?
• What is the connection between an ORM and a schema migration tool. Should they both be part of the same project, should they be tied together, or are they totally orthogonal?
• What's your support for geographic data? Is this(or other complex data types like it) in scope for the core of an ORM?

Despite these difficulties I thought the panel turned out very well. If there are any other questions about why things were the way they were just ask in the comments and I'll try to post a response.

PyCon Wrapup

With PyCon now over for me(and the sprints just begining for those still there) I figured I'd recap it for those who couldn't be there, and compare notes for those who were. I'll be doing a separate post to cover my panel, since I have quite a brain dump there.

Day One

We start off the morning with lightning talks, much better than last year due to the removal of so many sponsor talks(thanks Jacob), by now I've already met up with quite a few Django people. Next I move on to the "Python for CS1" talk, this was particularly interesting for me since they moved away from C++, which is exactly what my school uses. Next, on to "Introduction to CherryPy" with a few other Djangonauts. CherryPy looks interesting as a minimalist framework, however a lot of the callback/plugin architecture was confusing me, so I feel like I'd be ignoring it and just being very close to the metal, which isn't always a bad thing, something to explore in the future. For there I'm off to "How Python is Developed", it's very clear that Django's development model is based of Python's, and that seems to work for both parties.

Now we head to lunch, which was fantastic, thanks to whoever put it together. From there I go to "Panel: Python VMs", this was very informative, though I didn't have a chance to ask my question, "Eveyone on the panel has mentioned that performance is likely to improve greatly in the future as a potential reason to use their alternate VM, should Unladen Swallow succeed in their goal, how does that affect you?". After that I went to the "Twisted, AMQP, and Thrift" talk, it was fairly good, though this feels like a space I have to look at more, or have a real world use case for, to totally understand.

A quick break, and then we're on to Jesse Noller's mutliprocessing talk. Jesse really is the expert in this, and having used the multiprocessing library before it's clear to me there's still tons of cool stuff to explore. Lastly we had the "Behind the scenes of Everyblock.com" from Adrian, it's very cool to see their architecture and it's getting me excited for their going open source in June. I also got a bit out a shoutout here when Adrian was asked about future scalability and he mentioned sharding across multiple databases. We had lightning talks and that was the end of the official conference for the day.

In the evening a big group of us(officially a Pinax meetup, but I think we had more than that) went to a restaurant, as the evening progressed our group shrunk, until they finally kicked us out at closing time. A good time was had by all I think.

Day two

After staying out exceptionally late the night before I ended up sleeping on the floor with friends rather than heading home(thanks!) so, retrospectively, day two should have been tiring, but it was as high energy as could be. I began the morning with lightning talks followed by Guido's keynote, which was a little odd and jumped around a lot, but I found I really enjoyed it. After that I got to sit in one room for 3 talks in a row, "The State of Django", "Pinax", and "State of TurboGears", which I can say, without qualifications, were all great(I received another shoutout during the Django talk, I guess I really need to deliver, assuming the GSOC project is accepted). From there it was off to lunch.

After lunch it was time to take the stage for my "ORM Panel". I'm saving all the details for another blog post, but I think it went well. After this we had a Django Birds of a Feather session, which was very fun. We got to see Honza Kraal show off the cool features of the Ella admin. The highlight had to be having Jacob Kalpan-Moss rotate us around the room to get Djangonauts to meet each other.

After this I heard Jesse Noller's second concurrency talk, and then Ian Bicking's talk on ... stuff, both talks were great, but Ian's is probably in a "you had to be there" category.

After that it was time for lightning talks, followed by what was supposed to be dinner for people interested in Oebfare(Django blog application) to make some design decisions. In the end quite a few more people were there and I got to meet some really cool people like Mark Pilgrim. After dinner it was back to the Hyatt for about an hour of hacking on Django before I called it a night.

Day Three

The final day of the conference. Once again I began my day with some lightning talks. After this was the keynote, by the reddit.com creators. They gave a short, but interesting keynote and answered a lot of questions. I have to say that some of their technical answers were less than satisfying. After this I went to an open space on podcasting where I learned that I'm probably the only person who doesn't mind podcasts that are over an hour long.

After this I went to the Eve Online talk, but I ended up leaving for the testing tools panel. This panel was interesting, but I can't help but feel that the Python community would be best served by getting the testing tools that everyone agrees about into the unittest module in the Python standard library. For my last talk of the conference I saw Jacob Kaplan-Moss's talk on "Django's design decisions", thankfully this didn't step on the toes of the ORM design decision panel, and it was very interesting. From there it was off to lunch, and a final round of lightning talks, capped off my Guido running away with the Django Pony(I hope someone caught this on video), and my conference was done.

This was a really great conference for me, hopefully everyone else enjoyed it as much as I did. I'll be doing a follow up post where I answer some of the questions I've seen about the ORM panel, as well as put the rest of my thoughts on paper(hard disk). And now if you'll excuse me, I have a virtual sprint to attend.

Google Moderator for PyCon ORM Panel

I'm going to be moderating a panel this year at PyCon between 5 of the Python ORMs(Django, SQLAlchemy, SQLObject, Google App Engine, and web2py). To make my job easier, and to make sure the most interesting questions are asked I've setup a Google Moderator page for the panel here. Go ahead and submit your questions, and moderate others to try to ensure we get the best questions possible, even if you can't make it to PyCon(there will be a recording made I believe). I'll be adding my own questions shortly to make sure they are as interesting as I think they are.

Also, if you aren't already, do try to make it out to PyCon, there's still time and the talks look to be really exceptional.

Announcing django-filter

Django-filter is a reusable Django application I have been working on for a few weeks, individuals who follow me on Github may have noticed me working on it. Django-filter is basically an application that creates a generic interface for creating pages like the Django admin changelist page. It has support for defining vairous filters(including custom ones), filtering a queryset based on the user selections, and displaying an HTML form for the filter options. It has a design based around Django's ModelForm.

You can get the code, as well as tests and docs over at Github. For all bug reports, comments, and other feedback you can send me a message on Github(or message me on IRC) until I figure out a proper bug tracking system.

Enjoy.

A Second Look at Inheritance and Polymorphism with Django

Previously I wrote about ways to handle polymorphism with inheritance in Django's ORM in a way that didn't require any changes to your model at all(besides adding in a mixin), today we're going to look at a way to do this that is a little more invasive and involved, but also can provide much better performance. As we saw previously with no other information we could get the correct subclass for a given object in O(k) queries, where k is the number of subclasses. This means for a queryset with n items, we would need to do O(nk) queries, not great performance, for a queryset with 10 items, and 3 subclasses we'd need to do 30 queries, which isn't really acceptable for most websites. The major problem here is that for each object we simply guess as to which subclass a given object is. However, that's a piece of information we could know concretely if we cached it for later usage, so let's start off there, we're going to be building a mixin class just like we did last time:

from django.db import models

class InheritanceMixIn(models.Model):
    _class = models.CharField(max_length=100)
    
    class Meta:
        abstract = True

So now we have a simple abstract model that the base of our inheritance trees can subclass that has a field for caching which subclass we are. Now let's add a method to actually cache it and retrieve the subclass:

from django.db import models
from django.db.models.fields import FieldDoesNotExist
from django.db.models.related import RelatedObject

class InheritanceMixIn(models.Model):
    ...
    def save(self, *args, **kwargs):
        if not self.id:
            parent = self._meta.parents.keys()[0]
            subclasses = parent._meta.get_all_related_objects()
            for klass in subclasses:
                if isinstance(klass, RelatedObject) and klass.field.primary_key \
                    and klass.opts == self._meta:
                    self._class = klass.get_accessor_name()
                    break
        return super(InheritanceMixIn, self).save(*args, **kwargs)
    
    def get_object(self):
        try:
            if self._class and self._meta.get_field_by_name(self._class)[0].opts != self._meta:
                return getattr(self, self._class)
        except FieldDoesNotExist:
            pass
        return self

Our save method is where all the magic really happens. First, we make sure we're only doing this caching if it's the first time a model is being saved. Then we get the first parent class we have (this means this probably won't play nicely with multiple inheritance, that's unfortunate, but not as common a usecase), then we get all the related objects this class has(this includes the reverse relationship the subclasses have). Then for each of the subclasses, if it is a RelatedObject, and it is a primary key on it's model, and the class it points to is the same as us then we cache the accessor name on the model, break out, and do the normal save procedure.

Our get_object function is pretty simple, if we have our class cached, and the model we are cached as isn't of the same type as ourselves we get the attribute with the subclass and return it, otherwise we are the last descendent and just return ourselves. There is one(possible quite large) caveat here, if our inheritance chain is more than one level deep(that is to say our subclasses have subclasses) then this won't return those objects correctly. The class is actually cached correctly, but since the top level object doesn't have an attribute by the name of the 2nd level subclass it doesn't return anything. I believe this can be worked around, but I haven't found a way yet. One idea would be to actually store the full ancestor chain in the CharField, comma separated, and then just traverse it.

There is one thing we can do to make this even easier, which is to have instances automatically become the correct subclass when they are pulled in from the DB. This does have an overhead, pulling in a queryset with n items guarantees O(n) queries. This can be improved(just as it was for the previous solution) by ticket #7270 which allows select_related to traverse reverse relationships. In any event, we can write a metaclass to handle this for us automatically:

from django.db import models
from django.db.models.base import ModelBase
from django.db.models.fields import FieldDoesNotExist
from django.db.models.related import RelatedObject

class InheritanceMetaclass(ModelBase):
    def __call__(cls, *args, **kwargs):
        obj = super(InheritanceMetaclass, cls).__call__(*args, **kwargs)
        return obj.get_object()

class InheritanceMixIn(models.Model):
    __metaclass__ = InheritanceMetaclass
    ...

Here we've created a fairly trivial metaclass that subclasses the default one Django uses for it's models. The only method we've written is __call__, on a metalcass what __call__ does is handle the instantiation of an object, so it would call __init__. What we do is do whatever the default __call__ does, so that we get an instances as normal, and then we call the get_object() method we wrote earlier and return it, and that's all.

We've now looked at 2 ways to handle polymorphism, with this way being more efficient in all cases(ignoring the overhead of having the extra charfield). However, it still isn't totally efficient, and it fails in several edge cases. Whether automating the handling of something like this is a good idea is something that needs to be considered on a project by project basis, as the extra queries can be a large overhead, however, they may not be avoidable in which case automating it is probably advantages.