API Reference¶
models¶
-
class
computedfields.models.
ComputedFieldsModelType
[source]¶ Bases:
django.db.models.base.ModelBase
Metaclass for computed field models.
Handles the creation of the db fields. Also holds the needed data for graph calculations and dependency resolving.
After startup the method
_resolve_dependencies
gets called byapp.ready
to build the dependency resolving map. To avoid the expensive calculations in production mode the map can be pickled into a map file by settingCOMPUTEDFIELDS_MAP
in settings.py to a writable file path and calling the management commandcreatemap
.Note
The map file will not be updated automatically and therefore must be recreated by calling the management command
createmap
after model changes.-
classmethod
cf_mro
(cls, update_fields=None)[source]¶ Return mro for local computed field methods for a given set of
update_fields
. This method returns computed fields as self dependent to simplify field calculation insave
.
-
classmethod
preupdate_dependent
(instance, model=None, update_fields=None)[source]¶ Create a mapping of currently associated computed fields records, that would turn dirty by a follow-up bulk action.
Feed the mapping back to
update_dependent
asold
argument after your bulk action to update deassociated computed field records as well.
-
classmethod
preupdate_dependent_multi
(instances)[source]¶ Same as
preupdate_dependent
, but for multiple bulk actions at once.After done with the bulk actions, feed the mapping back to
update_dependent_multi
asold
argument to update deassociated computed field records as well.
-
classmethod
update_dependent
(instance, model=None, update_fields=None, old=None)[source]¶ Updates all dependent computed fields model objects.
This is needed if you have computed fields that depend on a model changed by bulk actions. Simply call this function after the update with the queryset containing the changed objects. The queryset may not be finalized by
distinct
or any other means.>>> Entry.objects.filter(pub_date__year=2010).update(comments_on=False) >>> update_dependent(Entry.objects.filter(pub_date__year=2010))
This can also be used with
bulk_create
. Sincebulk_create
returns the objects in a python container, you have to create the queryset yourself, e.g. with pks:>>> objs = Entry.objects.bulk_create([ ... Entry(headline='This is a test'), ... Entry(headline='This is only a test'), ... ]) >>> pks = set(obj.pk for obj in objs) >>> update_dependent(Entry.objects.filter(pk__in=pks))
Note
This function cannot be used to update computed fields on a computed fields model itself. For computed fields models always use
save
on the model objects. You still can useupdate
orbulk_create
but have to callsave
afterwards:>>> objs = SomeComputedFieldsModel.objects.bulk_create([ ... SomeComputedFieldsModel(headline='This is a test'), ... SomeComputedFieldsModel(headline='This is only a test'), ... ]) >>> for obj in objs: ... obj.save()
(This behavior might change with future versions.)
Special care is needed, if a bulk action contains foreign key changes, that are part of a computed field dependency chain. To correctly handle that case, provide the result of
preupdate_dependent
asold
argument like this:>>> # given: some computed fields model depends somehow on Entry.fk_field >>> old_relations = preupdate_dependent(Entry.objects.filter(pub_date__year=2010)) >>> Entry.objects.filter(pub_date__year=2010).update(fk_field=new_related_obj) >>> update_dependent(Entry.objects.filter(pub_date__year=2010), old=old_relations)
For completeness -
instance
can also be a single model instance. Since callingsave
on a model instance will trigger this function by thepost_save
signal it should not be invoked for single model instances, if they get saved anyway.
-
classmethod
update_dependent_multi
(instances, old=None)[source]¶ Updates all dependent computed fields model objects for multiple instances.
This function avoids redundant updates if consecutive
update_dependent
have intersections, example:>>> update_dependent(Foo.objects.filter(i='x')) # updates A, B, C >>> update_dependent(Bar.objects.filter(j='y')) # updates B, C, D >>> update_dependent(Baz.objects.filter(k='z')) # updates C, D, E
In the example the models
B
andD
would be queried twice,C
even three times. It gets even worse if the queries contain record intersections, those items would be queried and saved several times.The updates above can be rewritten as:
>>> update_dependent_multi([ ... Foo.objects.filter(i='x'), ... Bar.objects.filter(j='y'), ... Baz.objects.filter(k='z')])
where all dependent model objects get queried and saved only once. The underlying querysets are expanded accordingly.
Note
instances
can also contain model instances. Don’t use this function for model instances of the same type, instead aggregate those to querysets and useupdate_dependent
(as shown forbulk_create
above).Again special care is needed, if the bulk actions involve foreign key changes, that are part of computed field dependency chains. Use
preupdate_dependent_multi
to create a record mapping of the current state and after your bulk changes feed it back asold
argument to this function.
-
classmethod
-
computedfields.models.
get_contributing_fks
()[source]¶ Get a mapping of models and their local fk fields, that are part of a computed fields dependency chain.
Whenever a bulk action changes one of the fields listed here, you have to create a listing of the currently associated records with
preupdate_dependent
and, after doing the bulk change, feed the listing back toupdate_dependent
.This mapping can also be inspected as admin view, if
COMPUTEDFIELDS_ADMIN
is set toTrue
.
-
class
computedfields.models.
ComputedFieldsModel
(*args, **kwargs)[source]¶ Bases:
django.db.models.base.Model
Base class for a computed fields model.
To use computed fields derive your model from this class and use the
@computed
decorator:from django.db import models from computedfields.models import ComputedFieldsModel, computed class Person(ComputedFieldsModel): forename = models.CharField(max_length=32) surname = models.CharField(max_length=32) @computed(models.CharField(max_length=32), depends=[['self', ['surname', 'forename']]]) def combined(self): return u'%s, %s' % (self.surname, self.forename)
combined
will be turned into a real database field and can be accessed and searched like any other field. During saving the value gets calculated and written to the database. With the methodcompute('fieldname')
you can inspect the value that will be written, which is useful if you have pending changes:>>> person = Person(forename='Leeroy', surname='Jenkins') >>> person.combined # empty since not saved yet >>> person.compute('combined') # outputs 'Jenkins, Leeroy' >>> person.save() >>> person.combined # outputs 'Jenkins, Leeroy' >>> Person.objects.filter(combined__<some condition>) # used in a queryset
-
computedfields.models.
computed
(field, **kwargs)[source]¶ Decorator to create computed fields.
field
should be a model field instance suitable to hold the result of the decorated method. The decorator expects a keyword argumentdepends
to indicate dependencies to model fields (local or related). Listed dependencies will automatically update the computed field.Examples:
create a char field with no further dependencies (not very useful)
@computed(models.CharField(max_length=32), depends=[]) def ...
create a char field with one dependency to the field
name
of a foreign key relationfk
@computed(models.CharField(max_length=32), depends=[['fk', ['name']]]) def ...
Dependencies should be listed as
['relation_name', fieldnames_on_that_model]
. The relation can span serveral models, simply name the relation in python style with a dot (e.g.'a.b.c'
). A relation can be of any of foreign key, m2m, o2o and their back relations. The fieldnames should be a list of strings of concrete fields on the foreign model.Caution
With the dependency auto resolver you can easily create recursive dependencies by accident. Imagine the following:
class A(ComputedFieldsModel): @computed(models.CharField(max_length=32), depends=[['b_set', ['comp']]]) def comp(self): return ''.join(b.comp for b in self.b_set.all()) class B(ComputedFieldsModel): a = models.ForeignKey(A) @computed(models.CharField(max_length=32), depends=[['a', ['comp']]]) def comp(self): return a.comp
Neither an object of
A
orB
can be saved, since thecomp
fields depend on each other. While it is quite easy to spot for this simple case it might get tricky for more complicated dependencies. Therefore the dependency resolver tries to detect cyclic dependencies and raises aCycleNodeException
in case a cycle was found.If you experience this in your project try to get in-depth cycle information, either by using the
rendergraph
management command or by directly accessing the graph objects:- intermodel dependency graph:
your_model._graph
- mode local dependency graphs:
your_model._graph.modelgraphs[your_model]
- union graph:
your_model._graph.get_uniongraph()
Also see the graph documentation here.
-
class
computedfields.models.
ComputedFieldsAdminModel
(*args, **kwargs)[source]¶ Bases:
django.contrib.contenttypes.models.ContentType
Proxy model to list all
ComputedFieldsModel
models with their field dependencies in admin. This might be useful during development. To enable it, setCOMPUTEDFIELDS_ADMIN
in settings.py toTrue
.-
exception
DoesNotExist
¶ Bases:
django.contrib.contenttypes.models.DoesNotExist
-
exception
MultipleObjectsReturned
¶ Bases:
django.contrib.contenttypes.models.MultipleObjectsReturned
-
exception
-
class
computedfields.models.
ContributingModelsModel
(*args, **kwargs)[source]¶ Bases:
django.contrib.contenttypes.models.ContentType
Proxy model to list all models in admin, that contain fk fields contributing to computed fields. This might be useful during development. To enable it, set
COMPUTEDFIELDS_ADMIN
in settings.py toTrue
. An fk field is considered contributing, if it is part of a computed field dependency, thus a change to it would impact a computed field.-
exception
DoesNotExist
¶ Bases:
django.contrib.contenttypes.models.DoesNotExist
-
exception
MultipleObjectsReturned
¶ Bases:
django.contrib.contenttypes.models.MultipleObjectsReturned
-
exception
admin¶
graph¶
Module containing the graph logic for the dependency resolver.
Upon application initialization a dependency graph of all project wide
computed fields is created. The graph does a basic cycle check and
removes redundant dependencies. Finally the dependencies are translated
to the resolver map to be used later by update_dependent
and in
the signal handlers.
-
exception
computedfields.graph.
ComputedFieldsException
[source]¶ Bases:
Exception
Base exception raised from computed fields.
-
exception
computedfields.graph.
CycleException
[source]¶ Bases:
computedfields.graph.ComputedFieldsException
Exception raised during path linearization, if a cycle was found. Contains the found cycle either as list of edges or nodes in
args
.
-
exception
computedfields.graph.
CycleEdgeException
[source]¶ Bases:
computedfields.graph.CycleException
Exception raised during path linearization, if a cycle was found. Contains the found cycle as list of edges in
args
.
-
exception
computedfields.graph.
CycleNodeException
[source]¶ Bases:
computedfields.graph.CycleException
Exception raised during path linearization, if a cycle was found. Contains the found cycle as list of nodes in
args
.
-
class
computedfields.graph.
Edge
(left, right, data=None)[source]¶ Bases:
object
Class for representing an edge in
Graph
. The instances are created as singletons, callingEdge('A', 'B')
multiple times will always point to the same object.
-
class
computedfields.graph.
Node
(data)[source]¶ Bases:
object
Class for representing a node in
Graph
. The instances are created as singletons, callingNode('A')
multiple times will always point to the same object.
-
class
computedfields.graph.
Graph
[source]¶ Bases:
object
Simple directed graph implementation.
-
remove_node
(node)[source]¶ Remove a node from the graph.
Warning
Removing edges containing the removed node is not implemented.
-
remove_edge
(edge)[source]¶ Removes an edge from the graph.
Warning
Does not remove leftover contained nodes.
-
get_dot
(format='pdf', mark_edges=None, mark_nodes=None)[source]¶ Returns the graphviz object of the graph (needs the
graphviz
package).
-
render
(filename=None, format='pdf', mark_edges=None, mark_nodes=None)[source]¶ Renders the graph to file (needs the
graphviz
package).
-
view
(format='pdf', mark_edges=None, mark_nodes=None)[source]¶ Directly opens the graph in the associated desktop viewer (needs the
graphviz
package).
-
get_edgepaths
()[source]¶ Returns a list of all edge paths. An edge path is represented as list of edges.
Might raise a
CycleEdgeException
. For in-depth cycle detection useedge_cycles
, node_cycles` orget_cycles()
.
-
get_nodepaths
()[source]¶ Returns a list of all node paths. A node path is represented as list of nodes.
Might raise a
CycleNodeException
. For in-depth cycle detection useedge_cycles
,node_cycles
orget_cycles()
.
-
get_cycles
()[source]¶ Gets all cycles in graph.
This is not optimised by any means, it simply walks the whole graph recursively and aborts as soon a seen edge gets entered again. Therefore use this and all dependent properties (
edge_cycles
andnode_cycles
) for in-depth cycle inspection only.As a start node any node on the left side of an edge will be tested.
Returns a mapping of
{frozenset(<cycle edges>): { 'entries': set(edges leading to the cycle), 'path': list(cycle edges in last seen order) }}
An edge in
entries
is not necessarily part of the cycle itself, but once entered it will lead to the cycle.
-
edge_cycles
¶ Returns all cycles as list of edge lists. Use this only for in-depth cycle inspection.
-
node_cycles
¶ Returns all cycles as list of node lists. Use this only for in-depth cycle inspection.
-
is_cyclefree
¶ True if the graph contains no cycles.
For faster calculation this property relies on path linearization instead of the more expensive full cycle detection. For in-depth cycle inspection use
edge_cycles
ornode_cycles
instead.
-
remove_redundant
()[source]¶ Find and remove redundant edges. An edge is redundant if there there are multiple possibilities to reach an end node from a start node. Since the longer path triggers more needed database updates the shorter path gets discarded. Might raise a
CycleNodeException
.Returns the removed edges.
-
-
class
computedfields.graph.
ComputedModelsGraph
(computed_models)[source]¶ Bases:
computedfields.graph.Graph
Class to convert the computed fields dependency strings into a graph and generate the final resolver functions.
Steps taken:
resolve_dependencies
resolves the depends field strings to real model fields.- The dependencies are rearranged to adjacency lists for the underlying graph.
- The graph does a cycle check and removes redundant edges to lower the database penalty.
- In
generate_lookup_map
the path segments of remaining edges are collected into the final lookup map.
-
resolve_dependencies
(computed_models)[source]¶ Converts field dependencies into lookups and checks the fields’ existance. Also expands the old depends notation into the new format:
'relA.relB'
–>['relA.relB', local_fieldnames_on_B]
'relA#xy'
–>['relA', ['xy']]
- plus model local non computed fields, e.g.
['self', ['fieldA', 'fieldB']]
Warning
Dont use the old depends notation anymore, as it is underdetermined leading to ambiguity and will be removed by a later version.
-
generate_lookup_map
()[source]¶ Generates the final lookup map for queryset generation.
Structure of the map is:
{model: { '#' : dependencies 'fieldA' : dependencies } }
model
denotes the source model of a given instance.'fieldA'
points to a field that was changed. The right side contains the dependencies in the form{dependent_model: (fields, filter_strings)}
In
update_dependent
the information will be used to create a queryset and save their elements (roughly):queryset = dependent_model.objects.filter(string1=instance) queryset |= dependent_model.objects.filter(string2=instance) for obj in queryset: obj.save(update_fields=fields)
The
'#'
is a special placeholder to indicate, that a model object was saved normally. It contains the plain and non computed field dependencies.The separation of dependencies to computed fields and to other fields makes it possible to create complex computed field dependencies, even multiple times between the same objects without running into circular dependencies:
class A(ComputedFieldsModel): @computed(..., depends=['b_set#comp_b']) def comp_a(self): ... class B(ComputedFieldsModel): a = ForeignKey(B) @computed(..., depends=['a']) def comp_b(self): ...
Here
A.comp_a
depends onb.com_b
which itself somehow depends onA
. If an instance ofA
gets saved, the corresponding objects inB
will be updated, which triggers a final update ofcomp_a
fields on associatedA
objects.Caution
If there are only computed fields in
update_fields
always use those dependencies, never'#'
. This is important to ensure cycle free database updates. For computed fields the corresponding dependencies should always be used to get properly updated.Note
The created map is also used for the pickle file to circumvent the computationally expensive graph and map creation in production mode.
-
generate_local_mro_map
()[source]¶ Generate model local computed fields mro maps. Returns a mapping of models with local computed fields dependencies and their mro, example:
{ modelX: { 'base': ['c1', 'c2', 'c3'], 'fields': { 'name': ['c2', 'c3'], 'c2': ['c2', 'c3'] } } }
In the example modelX would have 3 computed fields, where c2 somehow depends on the field name. c3 itself depends on changes to c2, thus a change to name should run c2 and c3 in that specific order.
base lists all computed fields in topological execution order (mro). It is also used at runtime to cover a full update of an instance (
update_fields=None
).Note
Note that the actual values in fields are bitarrays to index positions of base, which allows quick field update merges at runtime by doing binary OR on the bitarrays.
-
get_uniongraph
()[source]¶ Build a union graph of intermodel dependencies and model local dependencies. This graph represents the final update cascades triggered by certain field updates. The union graph is needed to spot cycles introduced by model local dependencies, that otherwise might went unnoticed, example:
- global dep graph (acyclic):
A.comp --> B.comp, B.comp2 --> A.comp
- modelgraph of B (acyclic):
B.comp --> B.comp2
Here the resulting union graph is not a DAG anymore, since both subgraphs short-circuit to a cycle of
A.comp --> B.comp --> B.comp2 --> A.comp
.- global dep graph (acyclic):
-
class
computedfields.graph.
ModelGraph
(model, local_dependencies)[source]¶ Bases:
computedfields.graph.Graph
Graph to resolve model local computed field dependencies in right calculation order.
-
transitive_reduction
()[source]¶ Remove redundant single edges. Also checks for cycles. Note: Other than intermodel dependencies a model local dependencies always must be cyclefree.
-
get_topological_paths
()[source]¶ Creates a map of all possible entry nodes and their topological update path (computed fields mro).
-
generate_field_paths
(tpaths)[source]¶ Convert topological path node mapping into a mapping containing the fieldnames.
-
generate_local_mapping
(field_paths)[source]¶ Generates the final model local update table to be used during
ComputedFieldsModel.save
. Output is a mapping of local fields, that also update local computed fields, to a bitarray containing the computed fields mro, and the base topologcial order for a full update.
-
handlers¶
Module containing the database signal handlers.
The handlers are registered during application startup
in apps.ready
.
Note
The handlers are not registered in the managment
commands makemigrations
, migrate
and help
.
-
computedfields.handlers.
get_old_handler
(sender, instance, **kwargs)[source]¶ get_old_handler
handler.pre_save
signal handler to spot incoming fk relation changes. This is needed to correctly update old relations after fk changes, that would contain dirty computed field values after a save. The actual updates on old relations are done duringpost_save
.
-
computedfields.handlers.
postsave_handler
(sender, instance, **kwargs)[source]¶ post_save
handler.Directly updates dependent objects. Does nothing during fixtures.
-
computedfields.handlers.
predelete_handler
(sender, instance, **kwargs)[source]¶ pre_delete
handler.Gets all dependent objects as pk lists and saves them in thread local storage.
-
computedfields.handlers.
postdelete_handler
(sender, instance, **kwargs)[source]¶ post_delete
handler.Loads the dependent objects from the previously saved pk lists and updates them.
-
computedfields.handlers.
m2m_handler
(sender, instance, **kwargs)[source]¶ m2m_change
handler.Works like the other handlers but on the corresponding m2m actions.
Note
The handler triggers updates for both ends of the m2m relation, which might lead to massive updates and thus heavy time consuming database interaction.