django-parallelized_querysets Spread Django QuerySets on multiple cores with low memory usage
Download

django-parallelized_querysets Ranking & Summary

django-parallelized_querysets Tags

Django querysets

django-parallelized_querysets Description

django-parallelized_querysets is a Django app to handle large Django QuerySets by spreading their execution on multiple cores and keeping the memory usage low.Installationpip install django-parallelized_querysetsUsageparallelized_queryset(queryset, processes=None, function=None)Process the given queryset and return the result as a list.procesesNumber of processes to create. Defaults to the number returned by multiprocessing.cpu_count().functionApply a function the each result. Does not apply any function by default. The first argument is the Process which is calling it, and the second is the row.You can also pass two hooks (function that will be executed by the process at defined times):init_hookGive it a function taking the Process as argument and it will be executed at soon as it's created.end_hookGive it a function taking the Process as argument and it will be execute right before the Process exits. If it returns a non-None value, it will be appended to the results queue. Note Each time your function returns None, the value won't be in the resulting list. Note The order in the QuerySet won't be respected!ExampleReturn all the Article objects:>>> from parallelized_querysets import parallelized_queryset>>> qs = Article.objects.all()>>> parallelized_queryset(qs)Add all Article objects to a Redis index (assuming Article has a append_to_redis method):>>> from parallelized_querysets import parallelized_queryset>>> qs = Article.objects.all()>>> parallelized_queryset(qs, function=lambda p, x: x.append_to_redis())Do the same but on 6 processes:>>> from parallelized_querysets import parallelized_queryset>>> qs = Article.objects.all()>>> parallelized_queryset(qs, processes=6, function=lambda p, x: x.append_to_redis())parallelized_multiple_querysets(querysets, processes=None, function=None)Same as parallelized_queryset but querysets is a list of QuerySets.Product's homepage

django-parallelized_querysets Related Software