qless-py Redis-based Queue Management
Download

qless-py Ranking & Summary

qless-py Tags

queue management Python library Python queue management

qless-py Description

qless-py is a powerful Redis-based job queueing system inspired by resque, but built on a collection of Lua scripts, maintained in the qless-core repo.Philosophy and NomenclatureA job is a unit of work identified by a job id or jid. A queue can contain several jobs that are scheduled to be run at a certain time, several jobs that are waiting to run, and jobs that are currently running. A worker is a process on a host, identified uniquely, that asks for jobs from the queue, performs some process associated with that job, and then marks it as complete. When it's completed, it can be put into another queue.Jobs can only be in one queue at a time. That queue is whatever queue they were last put in. So if a worker is working on a job, and you move it, the worker's request to complete the job will be ignored.A job can be canceled, which means it disappears into the ether, and we'll never pay it any mind every again. A job can be dropped, which is when a worker fails to heartbeat or complete the job in a timely fashion, or a job can be failed, which is when a host recognizes some systematically problematic state about the job. A worker should only fail a job if the error is likely not a transient one; otherwise, that worker should just drop it and let the system reclaim it.InstallationInstall from pip:pip install qless-pyAlternatively, install qless-py from source by checking it out from github, and checking out the qless-core submodule:git clone git://github.com/seomoz/qless-py.gitcd qless-py# qless-core is a submodulegit submodule initgit submodule updatesudo python setup.py installProduct's homepage

qless-py Related Software