Taskito
An embedded task queue for Python powered by a Rust core, eliminating the need for external message brokers like Redis or RabbitMQ. Supports SQLite for local development and PostgreSQL for distributed deployments. Features include priority-based job processing, retry logic with exponential backoff, cron-based periodic scheduling, task chaining and parallel execution, progress tracking, and a built-in monitoring dashboard. Combines Rust's performance via Tokio and Diesel with Python's ease of use.
The Problem
Python task queues like Celery require a separate message broker (Redis, RabbitMQ), adding operational complexity. For many applications, a simpler embedded solution with no external dependencies would be faster to set up and easier to maintain.
The Approach
Built a Rust-based task engine using Tokio for async scheduling and Diesel ORM for database abstraction. Exposed a clean Python API via PyO3. SQLite serves as the default zero-config backend, while PostgreSQL enables distributed worker setups. Implemented priority queues, exponential backoff retries, and cron-based periodic task scheduling.
Results
- Zero-broker architecture — just pip install and go
- SQLite backend for local dev, PostgreSQL for production-scale distributed workers
- Built-in retry logic with exponential backoff and configurable max attempts
- Cron-based periodic scheduling and task workflow orchestration
Lessons Learned
- SQLite's WAL mode is essential for concurrent read/write access from multiple workers
- Task serialization format matters — JSON is debuggable but slower than MessagePack for large payloads
- A built-in dashboard dramatically reduces debugging time compared to CLI-only monitoring