russellromney

honker

Database🌱 NEW PROJECT BOOST#SQLite#PubSub#Task Queue#Rust#Backend

// summary

Honker is a SQLite extension and language binding that provides Postgres-style NOTIFY/LISTEN semantics, durable pub/sub, and task queues directly within your database. It eliminates the need for external brokers like Redis by allowing business writes and queue operations to commit atomically within the same transaction. The system achieves single-digit millisecond reaction times by monitoring SQLite's data version rather than relying on traditional client polling.

// technical analysis

Honker is a SQLite extension and multi-language library that implements Postgres-style NOTIFY/LISTEN semantics, durable pub/sub, and task queues directly within the SQLite database file. By leveraging the monotonic PRAGMA data_version counter for cross-process communication, it eliminates the need for external brokers like Redis or Celery, allowing business logic and queue operations to commit atomically within a single transaction. This design prioritizes data integrity and operational simplicity, trading off the overhead of 'over-triggering'—where subscribers wake on every database commit—to achieve sub-millisecond reaction times without complex polling.

// key highlights

Provides atomic transactional coupling, ensuring that business writes and queue operations commit or roll back together.

Achieves single-digit millisecond cross-process notification latency by monitoring PRAGMA data_version instead of using expensive polling.

Includes a robust work queue system featuring priority, delayed jobs, exponential backoff retries, and a dead-letter table.

Supports durable event streams with per-consumer offsets, allowing for reliable message replay and pub/sub functionality.

Functions as a loadable SQLite extension, enabling seamless integration with any SQLite client or ORM, including SQLAlchemy, Django, and Drizzle.

Offers broad language support with native bindings for Python, Node.js, Rust, Go, Ruby, Bun, and Elixir.

// use cases

Atomic task enqueuing alongside business logic transactions

Durable pub/sub streams with per-consumer offset tracking

Cross-process notifications and periodic crontab-style task scheduling

// getting started

To begin, install the honker package for your language (e.g., 'pip install honker') and open your SQLite database file using the library's provided open method. You can then define queues or streams and use the library's transaction context to enqueue tasks or publish events atomically with your database writes. For worker processes, use the claim iterator or the provided CLI worker command to process jobs as they are committed to the database.