Database (async/sync)¶
AsyncDatabase and Database share one API, generated from a single async source via
unasync (see docs/testing.md) — every method below exists identically on both, async/
await aside.
The .raw escape hatch — you now own error handling¶
Every connection/transaction scope (returned by db.connection()/db.transaction()) exposes a
.raw property: the underlying SQLAlchemy Connection/AsyncConnection. It exists for the two
things dbkit doesn't implement itself — tx.pipeline() and db.copy_records()'s raw driver
access both use it internally — and for genuine one-off raw-driver needs.
Using .raw directly opts you out of dbkit's entire value proposition for that statement:
no SQLSTATE classification (you get raw psycopg/asyncpg/SQLAlchemy exceptions, not
DatabaseError subclasses — an except DatabaseError handler will silently stop catching
them), no metrics, no tracing, no retry/circuit-breaker/commit-unknown handling. This is by
design (dbkit can't classify or retry a statement it never saw), but it's easy to reach for as a
shortcut and not notice you've silently disabled error handling for that code path. Prefer the
normal db.execute/fetch_*/stream/bulk methods for anything that isn't one of the two
documented raw-driver escape hatches.
Advisory locks (transaction scope)¶
The scope from db.transaction(...) also exposes transaction-scoped PostgreSQL advisory locks —
cooperative locks on a logical key (an order id, an engagement id) rather than on rows:
async with db.transaction(target=t, lock_timeout=3.0) as tx:
await tx.advisory_xact_lock("engagement:42") # blocks until granted; auto-released at commit
... # read-modify-write, serialized per key
await tx.advisory_xact_lock(key) blocks until granted and holds until the transaction ends
(there is no manual unlock — it can't leak); a wait beyond the transaction's lock_timeout raises
DatabaseLockTimeoutError. await tx.try_advisory_xact_lock(key) -> bool is the non-blocking
variant (False if another transaction holds it). key is an int (used directly) or a str
(hashed server-side); it is always bound, never interpolated. PostgreSQL only.
AsyncDatabase ¶
AsyncDatabase(
config: DbkitConfig,
*,
metrics: MetricsSink | None = None,
tracer: Tracer | None = None,
shard_resolver: ShardResolver | None = None,
replica_selector: ReplicaSelector | None = None,
)
Async, SQL-first database facade over SQLAlchemy Core (§8.1).
Build a facade from an already-validated :class:DbkitConfig.
Prefer :meth:from_config unless you need to pass a config object you built/validated
yourself. Does not connect to any database — call :meth:start before use.
engine_count
property
¶
Number of live SQLAlchemy engines currently held (one per target).
from_config
classmethod
¶
from_config(
config: DbkitConfig | Mapping[str, Any],
*,
metrics: MetricsSink | None = None,
tracer: Tracer | None = None,
shard_resolver: ShardResolver | None = None,
replica_selector: ReplicaSelector | None = None,
) -> AsyncDatabase
Build a facade from a :class:DbkitConfig or a plain dict (validated via
:meth:DbkitConfig.from_dict). The usual entry point.
start
async
¶
Create required engines and (optionally) warm connections (§27.1).
require_ready
async
¶
Raise unless every required target is reachable (§27.1).
close
async
¶
Dispose all engines (§27.2). grace_period reserved for in-flight draining.
register_database
async
¶
register_database(
name: str,
config: DatabaseConfig | Mapping[str, Any],
*,
connect: bool | None = None,
) -> bool
Register (or replace) a named database at runtime.
For topologies discovered after :meth:start — e.g. shard DSNs resolved per
tenant from a service registry — so callers do not need to maintain their own
engine registries. config is a :class:DatabaseConfig or the same dict
shape as one entry under the top-level databases mapping.
The database map is swapped copy-on-write, so concurrent readers always see a
complete mapping. Replacing an existing name first disposes its engines and
resets its limiter/breakers, so the new settings fully apply. Returns True
when an existing entry was replaced.
connect forces (or suppresses) eager engine creation; the default follows
primary.required when the facade is already started. Engine creation is
lazy in SQLAlchemy, so this never blocks on the network unless the pool is
warmed explicitly.
ensure_database
async
¶
ensure_database(
name: str,
config: DatabaseConfig | Mapping[str, Any],
*,
connect: bool | None = None,
) -> bool
Idempotent :meth:register_database: act only when name is missing or its
config changed. Returns True when a (re)registration actually happened.
The unchanged path is lock-free and cheap (build + compare one frozen config, a few µs), so callers routing by runtime topology can call this in front of every query::
await db.ensure_database(shard_id, {"primary": {"url": dsn}})
rows = await db.fetch_all(query, params,
target=DatabaseTarget(database=shard_id, role="read"))
A changed config (rotated host, credentials, pool settings) re-registers in place: old engines are disposed and the next query connects with the new settings — no restart needed, including password-only rotations.
unregister_database
async
¶
Remove a dynamically (or statically) registered database and dispose its
engines, limiter, and breakers. Returns False if name is unknown.
In-flight operations against the database finish on their checked-out
connections (idle-only disposal); new calls raise DatabaseRoutingError.
database_scope
async
¶
database_scope(
name: str,
config: DatabaseConfig | Mapping[str, Any],
*,
connect: bool | None = None,
) -> AsyncIterator[DatabaseTarget]
Register name for the duration of the block, then unregister it.
Yields a primary-routed :class:DatabaseTarget for convenience::
async with db.database_scope("tenant-42", {"primary": {"url": dsn}}) as target:
rows = await db.fetch_all(query, params, target=target)
Scope this to a unit of work, never to a request. Engines/pools are
created on first use inside the block and disposed on exit — wrapping every
HTTP request in a scope recreates connections each time and defeats pooling
entirely. Intended uses: tests, migrations, one-off tenant batch jobs, or an
ad-hoc query against a shard the process does not normally serve. Long-lived
services should call :meth:register_database once per discovered shard and
keep it registered.
fetch_all
async
¶
fetch_all(
query: object,
params: Mapping[str, Any] | None = None,
*,
target: DatabaseTarget,
map_to: Any = None,
timeout: float | None = None,
deadline: float | None = None,
) -> list[Any]
Run a one-shot read, acquiring and releasing its own connection (§11.1).
Returns every row, mapped to map_to, with no cardinality constraint. Retries and
the circuit breaker apply per the resolved target's policy.
fetch_one
async
¶
fetch_one(
query: object,
params: Mapping[str, Any] | None = None,
*,
target: DatabaseTarget,
map_to: Any = None,
timeout: float | None = None,
deadline: float | None = None,
) -> Any
Run a one-shot read expecting exactly one row.
Raises :class:~dbkit.errors.DatabaseResultError if zero or more than one row comes
back (via SQLAlchemy's own Result.one()).
fetch_optional
async
¶
fetch_optional(
query: object,
params: Mapping[str, Any] | None = None,
*,
target: DatabaseTarget,
map_to: Any = None,
timeout: float | None = None,
deadline: float | None = None,
) -> Any | None
Run a one-shot read expecting zero or one row; None if empty.
fetch_value
async
¶
fetch_value(
query: object,
params: Mapping[str, Any] | None = None,
*,
target: DatabaseTarget,
timeout: float | None = None,
deadline: float | None = None,
) -> Any
Run a one-shot read expecting exactly one row and return its first column.
fetch_values
async
¶
fetch_values(
query: object,
params: Mapping[str, Any] | None = None,
*,
target: DatabaseTarget,
timeout: float | None = None,
deadline: float | None = None,
) -> list[Any]
Run a one-shot read and return the first column of every row.
execute
async
¶
execute(
query: object,
params: Mapping[str, Any] | None = None,
*,
target: DatabaseTarget,
timeout: float | None = None,
deadline: float | None = None,
) -> ExecutionResult
Run a one-shot write/DDL statement, auto-committed on success (§11.1).
execute_many
async
¶
execute_many(
query: object,
params_seq: Sequence[Mapping[str, Any]],
*,
target: DatabaseTarget,
timeout: float | None = None,
deadline: float | None = None,
) -> ExecutionResult
Run one statement against many parameter sets (driver executemany).
connection
async
¶
connection(
*, target: DatabaseTarget, timeout: float | None = None
) -> AsyncIterator[AsyncConnectionScope]
A held connection with commit-on-success / rollback-on-error semantics (§11.2).
transaction
async
¶
transaction(
*,
target: DatabaseTarget,
isolation: str | None = None,
read_only: bool = False,
deferrable: bool = False,
timeout: float | None = None,
lock_timeout: float | None = None,
) -> AsyncIterator[Any]
An explicit transaction with isolation/timeout options (§11.3).
deferrable (PostgreSQL only) is only meaningful together with
isolation="serializable" and read_only=True: it lets a read-only serializable
transaction wait for a safe snapshot instead of risking a serialization failure.
Usage::
async with db.transaction(target=write_target) as tx:
await tx.execute(INSERT, params)
consistency_scope
async
¶
consistency_scope(
*,
mode: Literal["read_your_writes"] = "read_your_writes",
) -> AsyncIterator[None]
Force reads within this scope to the primary so they observe writes made earlier in the same scope — read-your-writes over replica routing (§23)::
async with db.consistency_scope(mode="read_your_writes"):
await db.execute(write_query, target=write_target)
row = await db.fetch_one(read_query, target=read_target) # sees the write
The override is a contextvars.ContextVar, so it never leaks across concurrent
operations in other tasks — but it also does not cross a thread boundary (a plain
threading.Thread, loop.run_in_executor, or the sync Database facade driven
from a worker thread does not inherit it). See "Read-your-writes across threads" in
docs/troubleshooting.md if you need the override to apply there too.
health
async
¶
Readiness check: SELECT 1 against every required target (§26).
pool_status ¶
A point-in-time snapshot of every currently live engine's connection pool.
drain_engine
async
¶
Force-dispose one engine's idle pooled connections by its snapshot key (as printed
by :meth:pool_status, e.g. "prod:app:default:primary:psycopg"), so the next call
routed to it rebuilds a fresh engine with fresh connections. Returns False if no live
engine currently has that key.
Useful right before a planned failover/topology change: rather than waiting for
connections to naturally recycle (PoolConfig.recycle_seconds), force them closed so
subsequent traffic reaches the new backend immediately. Only idle pooled connections are
closed — one already checked out by an in-flight call keeps working until released,
exactly like LRU eviction (AsyncEngineRegistry(evict_lru=True)).
This must be called from within the running application process — there is no
dbkit CLI command for this, deliberately: each CLI invocation is a fresh, separate
process with its own empty engine registry, so it has no way to reach into an
already-running application's live engines. Wire this to a signal handler or an admin
HTTP route in your own application instead (see docs/troubleshooting.md).
stream
async
¶
stream(
query: object,
params: Mapping[str, Any] | None = None,
*,
target: DatabaseTarget,
batch_size: int = 1000,
map_to: Any = None,
max_duration: float | None = None,
) -> AsyncResultStream
Stream a large result set with a server-side cursor, bounded memory (§20).
Usage::
async with await db.stream(EXPORT, params, target=t, batch_size=1000) as rows:
async for row in rows:
...
The stream owns its connection until the context exits, so it bypasses auto-retry.
insert_many
async
¶
insert_many(
table: Table,
rows: Sequence[Mapping[str, Any]],
*,
target: DatabaseTarget,
mode: FailureMode = "atomic",
strategy: InsertStrategy = "execute_many",
batch_size: int | None = None,
timeout: float | None = None,
) -> ExecutionResult
Insert many rows in adaptively-sized batches (§19). rows are param dicts.
strategy="unnest" (PostgreSQL only) binds one array parameter per column instead
of one parameter per column per row, so batch size isn't limited by the bind-parameter
ceiling — a mid-tier option between execute_many and COPY.
upsert_many
async
¶
upsert_many(
table: Table,
rows: Sequence[Mapping[str, Any]],
*,
target: DatabaseTarget,
conflict_index_elements: Sequence[str],
update_columns: Sequence[str] | None = None,
mode: FailureMode = "atomic",
strategy: InsertStrategy = "execute_many",
batch_size: int | None = None,
timeout: float | None = None,
) -> ExecutionResult
Upsert many rows via PostgreSQL ON CONFLICT (§19). update_columns=None means
DO NOTHING; otherwise those columns are updated from the proposed row.
copy_records
async
¶
copy_records(
table: str,
columns: Sequence[str],
records: Any,
*,
target: DatabaseTarget,
timeout: float | None = None,
) -> ExecutionResult
Bulk-ingest records into table via PostgreSQL COPY (§19.2).
records is an iterable (async or sync) of row sequences matching columns.
Fastest path for large ingests; PostgreSQL + psycopg only.
Database ¶
Database(
config: DbkitConfig,
*,
metrics: MetricsSink | None = None,
tracer: Tracer | None = None,
shard_resolver: ShardResolver | None = None,
replica_selector: ReplicaSelector | None = None,
)
Async, SQL-first database facade over SQLAlchemy Core (§8.1).
Build a facade from an already-validated :class:DbkitConfig.
Prefer :meth:from_config unless you need to pass a config object you built/validated
yourself. Does not connect to any database — call :meth:start before use.
engine_count
property
¶
Number of live SQLAlchemy engines currently held (one per target).
from_config
classmethod
¶
from_config(
config: DbkitConfig | Mapping[str, Any],
*,
metrics: MetricsSink | None = None,
tracer: Tracer | None = None,
shard_resolver: ShardResolver | None = None,
replica_selector: ReplicaSelector | None = None,
) -> Database
Build a facade from a :class:DbkitConfig or a plain dict (validated via
:meth:DbkitConfig.from_dict). The usual entry point.
start ¶
Create required engines and (optionally) warm connections (§27.1).
close ¶
Dispose all engines (§27.2). grace_period reserved for in-flight draining.
register_database ¶
register_database(
name: str,
config: DatabaseConfig | Mapping[str, Any],
*,
connect: bool | None = None,
) -> bool
Register (or replace) a named database at runtime.
For topologies discovered after :meth:start — e.g. shard DSNs resolved per
tenant from a service registry — so callers do not need to maintain their own
engine registries. config is a :class:DatabaseConfig or the same dict
shape as one entry under the top-level databases mapping.
The database map is swapped copy-on-write, so concurrent readers always see a
complete mapping. Replacing an existing name first disposes its engines and
resets its limiter/breakers, so the new settings fully apply. Returns True
when an existing entry was replaced.
connect forces (or suppresses) eager engine creation; the default follows
primary.required when the facade is already started. Engine creation is
lazy in SQLAlchemy, so this never blocks on the network unless the pool is
warmed explicitly.
ensure_database ¶
ensure_database(
name: str,
config: DatabaseConfig | Mapping[str, Any],
*,
connect: bool | None = None,
) -> bool
Idempotent :meth:register_database: act only when name is missing or its
config changed. Returns True when a (re)registration actually happened.
The unchanged path is lock-free and cheap (build + compare one frozen config, a few µs), so callers routing by runtime topology can call this in front of every query::
db.ensure_database(shard_id, {"primary": {"url": dsn}})
rows = db.fetch_all(query, params,
target=DatabaseTarget(database=shard_id, role="read"))
A changed config (rotated host, credentials, pool settings) re-registers in place: old engines are disposed and the next query connects with the new settings — no restart needed, including password-only rotations.
unregister_database ¶
Remove a dynamically (or statically) registered database and dispose its
engines, limiter, and breakers. Returns False if name is unknown.
In-flight operations against the database finish on their checked-out
connections (idle-only disposal); new calls raise DatabaseRoutingError.
database_scope ¶
database_scope(
name: str,
config: DatabaseConfig | Mapping[str, Any],
*,
connect: bool | None = None,
) -> Iterator[DatabaseTarget]
Register name for the duration of the block, then unregister it.
Yields a primary-routed :class:DatabaseTarget for convenience::
with db.database_scope("tenant-42", {"primary": {"url": dsn}}) as target:
rows = db.fetch_all(query, params, target=target)
Scope this to a unit of work, never to a request. Engines/pools are
created on first use inside the block and disposed on exit — wrapping every
HTTP request in a scope recreates connections each time and defeats pooling
entirely. Intended uses: tests, migrations, one-off tenant batch jobs, or an
ad-hoc query against a shard the process does not normally serve. Long-lived
services should call :meth:register_database once per discovered shard and
keep it registered.
fetch_all ¶
fetch_all(
query: object,
params: Mapping[str, Any] | None = None,
*,
target: DatabaseTarget,
map_to: Any = None,
timeout: float | None = None,
deadline: float | None = None,
) -> list[Any]
Run a one-shot read, acquiring and releasing its own connection (§11.1).
Returns every row, mapped to map_to, with no cardinality constraint. Retries and
the circuit breaker apply per the resolved target's policy.
fetch_one ¶
fetch_one(
query: object,
params: Mapping[str, Any] | None = None,
*,
target: DatabaseTarget,
map_to: Any = None,
timeout: float | None = None,
deadline: float | None = None,
) -> Any
Run a one-shot read expecting exactly one row.
Raises :class:~dbkit.errors.DatabaseResultError if zero or more than one row comes
back (via SQLAlchemy's own Result.one()).
fetch_optional ¶
fetch_optional(
query: object,
params: Mapping[str, Any] | None = None,
*,
target: DatabaseTarget,
map_to: Any = None,
timeout: float | None = None,
deadline: float | None = None,
) -> Any | None
Run a one-shot read expecting zero or one row; None if empty.
fetch_value ¶
fetch_value(
query: object,
params: Mapping[str, Any] | None = None,
*,
target: DatabaseTarget,
timeout: float | None = None,
deadline: float | None = None,
) -> Any
Run a one-shot read expecting exactly one row and return its first column.
fetch_values ¶
fetch_values(
query: object,
params: Mapping[str, Any] | None = None,
*,
target: DatabaseTarget,
timeout: float | None = None,
deadline: float | None = None,
) -> list[Any]
Run a one-shot read and return the first column of every row.
execute ¶
execute(
query: object,
params: Mapping[str, Any] | None = None,
*,
target: DatabaseTarget,
timeout: float | None = None,
deadline: float | None = None,
) -> ExecutionResult
Run a one-shot write/DDL statement, auto-committed on success (§11.1).
execute_many ¶
execute_many(
query: object,
params_seq: Sequence[Mapping[str, Any]],
*,
target: DatabaseTarget,
timeout: float | None = None,
deadline: float | None = None,
) -> ExecutionResult
Run one statement against many parameter sets (driver executemany).
connection ¶
A held connection with commit-on-success / rollback-on-error semantics (§11.2).
transaction ¶
transaction(
*,
target: DatabaseTarget,
isolation: str | None = None,
read_only: bool = False,
deferrable: bool = False,
timeout: float | None = None,
lock_timeout: float | None = None,
) -> Iterator[Any]
An explicit transaction with isolation/timeout options (§11.3).
deferrable (PostgreSQL only) is only meaningful together with
isolation="serializable" and read_only=True: it lets a read-only serializable
transaction wait for a safe snapshot instead of risking a serialization failure.
Usage::
with db.transaction(target=write_target) as tx:
tx.execute(INSERT, params)
consistency_scope ¶
Force reads within this scope to the primary so they observe writes made earlier in the same scope — read-your-writes over replica routing (§23)::
with db.consistency_scope(mode="read_your_writes"):
db.execute(write_query, target=write_target)
row = db.fetch_one(read_query, target=read_target) # sees the write
The override is a contextvars.ContextVar, so it never leaks across concurrent
operations in other tasks — but it also does not cross a thread boundary (a plain
threading.Thread, loop.run_in_executor, or the sync Database facade driven
from a worker thread does not inherit it). See "Read-your-writes across threads" in
docs/troubleshooting.md if you need the override to apply there too.
pool_status ¶
A point-in-time snapshot of every currently live engine's connection pool.
drain_engine ¶
Force-dispose one engine's idle pooled connections by its snapshot key (as printed
by :meth:pool_status, e.g. "prod:app:default:primary:psycopg"), so the next call
routed to it rebuilds a fresh engine with fresh connections. Returns False if no live
engine currently has that key.
Useful right before a planned failover/topology change: rather than waiting for
connections to naturally recycle (PoolConfig.recycle_seconds), force them closed so
subsequent traffic reaches the new backend immediately. Only idle pooled connections are
closed — one already checked out by an in-flight call keeps working until released,
exactly like LRU eviction (EngineRegistry(evict_lru=True)).
This must be called from within the running application process — there is no
dbkit CLI command for this, deliberately: each CLI invocation is a fresh, separate
process with its own empty engine registry, so it has no way to reach into an
already-running application's live engines. Wire this to a signal handler or an admin
HTTP route in your own application instead (see docs/troubleshooting.md).
stream ¶
stream(
query: object,
params: Mapping[str, Any] | None = None,
*,
target: DatabaseTarget,
batch_size: int = 1000,
map_to: Any = None,
max_duration: float | None = None,
) -> ResultStream
Stream a large result set with a server-side cursor, bounded memory (§20).
Usage::
with db.stream(EXPORT, params, target=t, batch_size=1000) as rows:
for row in rows:
...
The stream owns its connection until the context exits, so it bypasses auto-retry.
insert_many ¶
insert_many(
table: Table,
rows: Sequence[Mapping[str, Any]],
*,
target: DatabaseTarget,
mode: FailureMode = "atomic",
strategy: InsertStrategy = "execute_many",
batch_size: int | None = None,
timeout: float | None = None,
) -> ExecutionResult
Insert many rows in adaptively-sized batches (§19). rows are param dicts.
strategy="unnest" (PostgreSQL only) binds one array parameter per column instead
of one parameter per column per row, so batch size isn't limited by the bind-parameter
ceiling — a mid-tier option between execute_many and COPY.
upsert_many ¶
upsert_many(
table: Table,
rows: Sequence[Mapping[str, Any]],
*,
target: DatabaseTarget,
conflict_index_elements: Sequence[str],
update_columns: Sequence[str] | None = None,
mode: FailureMode = "atomic",
strategy: InsertStrategy = "execute_many",
batch_size: int | None = None,
timeout: float | None = None,
) -> ExecutionResult
Upsert many rows via PostgreSQL ON CONFLICT (§19). update_columns=None means
DO NOTHING; otherwise those columns are updated from the proposed row.
copy_records ¶
copy_records(
table: str,
columns: Sequence[str],
records: Any,
*,
target: DatabaseTarget,
timeout: float | None = None,
) -> ExecutionResult
Bulk-ingest records into table via PostgreSQL COPY (§19.2).
records is an iterable (async or sync) of row sequences matching columns.
Fastest path for large ingests; PostgreSQL + psycopg only.