Torque A web hook task queue based on tornado and redis
Download

Torque Ranking & Summary

Torque Tags

database Python library REDIS Tornado redis database

Torque Description

A web hook task queue based on tornado and redis Torque is a web hook task queue based on tornado and redis. It's intended to provide a similar pattern to Google App Engine's taskqueue.To use Torque, you need to run a redis database, a console script that exposes a Tornado web application and one process per task queue. You can then add tasks to one or more queues, either using the python client api that torque provides or via an HTTP api (or indeed by adding them directly to the database).Tasks consist of a url and some params. When a task is executed, torque will post the params to the url. If the task errors, it backs off steeply until it errors too many times, at which point it's deleted.Tasks are stored in a redis SortedSet. Tornado is used to execute tasks asyncronously, without blocking.Install:Install the redis and Tornado dependencies. (n.b.: see ./etc/redis.tiger.patch if, like me, you're still using OSX Tiger). Then install the torque egg:python setup.py installRun:Run redis:./redis-serverStart the Tornado application:./bin/torque-serveStart the task queue:./bin/torque-processSee --help against either of the torque console scripts for a list of configuration options. For example, to run a second queue called foobar, you might use:./bin/torque-process --queue_name=foobarUse:To add a task to the queue, post to /add_task with two params: * url which is the url to the webhook you want the task to request * params which is a json encoded dictionary of the params you want to post to the webhook you're requestingAn example in python (with the Tornado application available on localhost, running on port 8889) would be:import jsonimport urllibmytask = { 'url': 'http://mywebservice.com/hooks/do/foo', 'params': json.dumps({'foo', 'somevalue', 'baz': 99})}target_url = 'http://localhost:8889/hooks/add'urllib.urlopen(target_url, urllib.urlencode(mytask))This queued a POST request to http://mywebservice.com/hooks/do/foo with the params foo=somevalue and baz=99 to be made as soon as possible.You can do something similar using any programming language that can make url requests. However, if you are using python, you can use the client api that torque provides:from torque.client import add_taskadd_task(url='http://mywebservice.com/hooks/do/foo', params={'a': 1})Note that this doesn't require json encoding the params. You can specify a delay for the task, so that it's executed after (but not necessarily at) a number of seconds:add_task(url='...', params={...}, delay=20) # will execute after 20 secondsIndividual tasks backoff exponentially if they error, upto a maximum backoff delay that's configurable as --max_task_delay, until they error --max_task_errors times (at which point they get deleted).See the source code for more info and options, or just run it and use it. Requirements: · Python

Torque Related Software