sade.dev — PostgreSQL in Production

Multiple DB Users with pgBouncer auth_query

info@muhammetsafak.com.tr (Muhammet Şafak) — Sun, 19 Apr 2026 00:00:00 GMT

pgBouncer is excellent as a connection pooler, but its default configuration comes with one hostility: you have to write your user list into userlist.txt by hand. Every time you add a new DB user, you update the file, compute the hash, reload pgBouncer. An operational headache.

auth_query solves this: pgBouncer queries PostgreSQL itself for authentication.

Setting up auth_user

On the PostgreSQL side we create a low-privilege user and a pg_shadow lookup function:

CREATE ROLE pgbouncer LOGIN PASSWORD 's3cr€t_p@ssw0rd';

CREATE SCHEMA pgbouncer;
GRANT USAGE ON SCHEMA pgbouncer TO pgbouncer;

CREATE OR REPLACE FUNCTION pgbouncer.user_lookup(in i_username text,
                                                 out uname text,
                                                 out phash text)
RETURNS record AS $$
BEGIN
    SELECT usename, passwd FROM pg_catalog.pg_shadow
    WHERE usename = i_username INTO uname, phash;
    RETURN;
END;
$$ LANGUAGE plpgsql SECURITY DEFINER;

REVOKE ALL ON FUNCTION pgbouncer.user_lookup(text) FROM PUBLIC;
GRANT EXECUTE ON FUNCTION pgbouncer.user_lookup(text) TO pgbouncer;

SECURITY DEFINER matters — the function uses the privileges of the user who defined it to access pg_shadow. Otherwise the pgbouncer role can’t read pg_shadow.

auth_query runs inside the target database, so the function has to be installed into every database pgBouncer serves.

pgBouncer config

pgbouncer.ini:

[databases]
* = host=127.0.0.1 port=5432

[pgbouncer]
listen_addr = 127.0.0.1
listen_port = 6432

auth_type = scram-sha-256
auth_user = pgbouncer
auth_query = SELECT uname, phash FROM pgbouncer.user_lookup($1)

pool_mode = transaction
max_client_conn = 1000
default_pool_size = 25

No more userlist.txt. Create a new role in PostgreSQL and authentication works without requiring a pgBouncer restart.

md5 or scram-sha-256?

PostgreSQL 14+ uses scram-sha-256 by default. pgBouncer has supported it since 1.11, and SCRAM pass-through with auth_query since 1.14. If you’re forced to fall back to md5 on older clusters:

auth_type = md5

But if you’re standing up a new install, use scram-sha-256 — md5 is weak now.

Which pool mode should you choose?

There are three modes; for most Laravel/Django apps the choices are:

session — the connection stays the same from start to finish. Prepared statements work, but the pooling benefit is weak.
transaction — the connection is released at the end of each transaction. The most common choice. Protocol-level prepared statements need 1.21+; since 1.24 they are enabled by default (max_prepared_statements = 200).
statement — released at the end of each query. Problematic with most ORMs.

Practical: start with transaction; switch to session if you run into trouble.

Common mistake: the read-only role has no password

auth_query reads pg_shadow.passwd. If the user comes in via SSO or uses peer authentication, the passwd field is NULL — pgBouncer rejects the login. In that case you’d need to set up trust or peer through pg_hba.conf, but you don’t want that in production.

Monitoring

pgBouncer provides its own admin DB:

psql -p 6432 pgbouncer -U pgbouncer
> SHOW POOLS;
> SHOW STATS;
> SHOW CLIENTS;

If you see cl_waiting > 0, you may need to raise default_pool_size. If sv_idle is very high, it’s the opposite — the pool is overprovisioned.

A PostgreSQL Backup Strategy with pgBackRest

info@muhammetsafak.com.tr (Muhammet Şafak) — Fri, 08 May 2026 00:00:00 GMT

The day I stopped wrestling with pg_dump is the day I think I got more serious as a systems engineer. pg_dump is an investigation tool — not a production backup tool.

A production backup has to deliver three things:

Point-in-time recovery (PITR). Being able to say “it was deleted at 02:13 last night.”
Integrity verification. Testing that the backup file actually works when you open it.
Retention governed by policy. 6-hourly/daily/weekly rotation.

pg_dump covers none of the three.

pgBackRest setup

I prefer — and recommend — taking both a local and an S3 backup. Local backups are fast; S3 backups are fire-resistant.

Off-site copy

If the repository lives on a single disk, one VPS fire wipes out everything. pgBackRest can write to an S3 or S3-compatible (R2, MinIO) bucket:

Preferably, protect the bucket with a policy that locks versions and forbids deletion. Critical in a ransomware scenario.

To keep costs under control over the long term, set the bucket lifecycle:

30 Days → Glacier Instant Retrieval → 120 Days → Delete

/etc/pgbackrest.conf:

[global]

###################################
# Local Repository
###################################
repo1-path=/var/lib/pgbackrest
repo1-retention-full=4
repo1-retention-diff=14
repo1-bundle=y
repo1-block=y
repo1-cipher-pass=...
repo1-cipher-type=aes-256-cbc

###################################
# AWS S3 Repository
###################################
repo2-type=s3
repo2-path=/main
repo2-s3-bucket=...
repo2-s3-endpoint=s3.{region_name}.amazonaws.com
repo2-s3-region={region_name}
repo2-s3-key=...
repo2-s3-key-secret=...
repo2-bundle=y
repo2-block=y
repo2-cipher-type=aes-256-cbc
repo2-cipher-pass=...

###################################
compress-type=zst
compress-level=6
process-max=4
start-fast=y
log-level-console=info
log-level-file=detail
log-path=/var/log/pgbackrest

[main]
pg1-path=/var/lib/postgresql/16/main
pg1-port=5432

Tighten the permissions:

sudo chown postgres:postgres /etc/pgbackrest.conf
sudo chmod 640 /etc/pgbackrest.conf

Create the stanza:

sudo -u postgres pgbackrest --stanza=main stanza-create

On the postgresql.conf side, we route WAL archiving to pgBackRest:

archive_mode = on
archive_command = 'pgbackrest --stanza=main archive-push %p'
archive_timeout = 60
wal_level = replica

archive_timeout = 60 — the last 60 seconds of transactions can be lost; enough for most applications, not for financial transactions.

Backup plan

On the cron side:

# Local repository (repo1)
0 2 * * 0          pgbackrest --stanza=main --repo=1 --type=full backup
0 2 * * 1-6        pgbackrest --stanza=main --repo=1 --type=diff backup
0 0,6,12,18 * * *  pgbackrest --stanza=main --repo=1 --type=incr backup

# S3 repository (repo2)
0 4 * * 0          pgbackrest --stanza=main --repo=2 --type=full backup
0 4 * * 1-6        pgbackrest --stanza=main --repo=2 --type=diff backup
0 3,9,15,21 * * *  pgbackrest --stanza=main --repo=2 --type=incr backup

With two repositories configured, --repo is mandatory: without it pgBackRest backs up only to the highest-priority repository (repo1), so S3 would hold nothing but archived WAL — a copy that cannot be restored on its own.

Weekly full, daily diff, 6-hourly incremental on each repository. Storage consumption is surprisingly low — the repoN-bundle/repoN-block lines in the config enable block incremental backup, so pgBackRest copies only the changed blocks of a file instead of the whole file; both options are off by default.

Integrity: if there’s a backup, can it be restored?

Taking the backup is not enough. A weekly restore drill:

pgbackrest --stanza=main --delta restore --pg1-path=/tmp/pg_restore_test

Then I start that instance on a temporary port and run a few sanity queries. Skip this step and the backup becomes false confidence.

PITR rehearsal

Rolling back to a specific point in time:

pgbackrest --stanza=main \
  --type=time \
  --target='2026-05-07 02:13:00+03' \
  restore

Running this command for the first time during a real incident is a disaster. Rehearse it beforehand — write the steps into the runbook.

Retention

Cleaning up unused backups in the repository with the expire command:

pgbackrest --stanza=main expire

With repo1-retention-full=4 the last 4 full backups are kept; older ones (and the diff/incr backups depending on them) are cleaned up automatically.

Conclusion

For copying a schema or moving a small table, pg_dump is still the right tool. But on any system that needs PITR, the backup software has to be pgBackRest (or barman, or your database provider’s native tool). The moment a backup is untested, saying “I have a backup” is a sophisticated way of lying.

Read/Write Splitting: Separating Read and Write Load

info@muhammetsafak.com.tr (Muhammet Şafak) — Sat, 22 Aug 2026 00:00:00 GMT

The primary database’s CPU was constantly maxed out. The interesting part: the write rate was low. Almost all of the load was reads — report pages, listing endpoints, search. A single primary was trying to carry a pile of reads that never needed it in the first place.

Separating read and write load — read/write splitting — is the known fix for this picture. But applied at the wrong time, or without awareness of the right traps, it brings more problems than it solves.

Most load is read-heavy

How read-heavy the traffic is depends on the workload: in measurements of the standard OLTP benchmarks, TPC-E runs 90.69% reads while TPC-C stays at 65.71% — so measure your own ratio instead of assuming it. Every order is written once but read dozens of times: in the list, in the detail view, in a report, on a dashboard. This asymmetry is what makes read/write splitting appealing — because the side you need to scale is obvious.

First: is this really a capacity problem?

Stop before adding a replica. A full primary doesn’t always mean “out of capacity.” Often a single missing index makes the primary look many times busier than it is.

Adding a replica — a new server, replication setup, lag monitoring — can amount to expensively buying your way out of a problem a single CREATE INDEX would have solved. The order in the breaking points of data-intensive systems is clear: index discipline first, then replicas. Don’t skip that order.

Measure your queries with EXPLAIN ANALYZE. If the primary is genuinely saturating under correctly indexed queries — that’s when you reach for a replica.

A replica scales reads, not writes

Let’s be clear: a read replica adds nothing to your write capacity. The same writes are replayed on every replica. A replica solves a read-load problem; if you have a write-load problem, a replica is the wrong tool.

Setting it up in Laravel

Laravel supports read/write connection splitting natively:

// config/database.php
'pgsql' => [
    'driver' => 'pgsql',
    'read'   => ['host' => ['10.0.0.2']],   // replica
    'write'  => ['host' => ['10.0.0.1']],   // primary
    'sticky' => true,
    // ...shared settings
],

SELECTs go to the replica, INSERT/UPDATE/DELETEs go to the primary. The replica gets its own connection pool — separate from the primary’s; if you use pgBouncer, they are two distinct pools.

Replication lag: the real bill

The replica trails the primary by a few milliseconds — a few seconds under load. This delay is the real cost of read/write splitting, and its name is the read-after-write problem.

sticky => true partly addresses it: if you wrote within a request, the subsequent reads in that same request go to the primary. But sticky only works within the boundary of a single request.

Outside that boundary it’s still open: a user updates their profile (request 1, written to the primary), moves to the next page (request 2, read from the replica), and sees their old profile, not yet replayed on the replica. The user sees their own data as stale. This looks like a bug, but it’s actually a tradeoff the architecture accepts — one that has to be accepted deliberately.

The query classification discipline

Setting up read/write splitting requires every read to answer one question: can this query read stale data?

Can read from a replica: lists, reports, search results, dashboards. A few seconds of delay is irrelevant.
Must read from the primary: account balance, stock count, authorization checks, any read a write decision depends on.

This classification is now part of the architecture, and it needs to be documented. A new developer must have somewhere to look for “where should this query read from” — otherwise the classification quietly rots.

When do you actually need it?

Read/write splitting is the right move when these three conditions hold together:

Index discipline is complete; queries are correctly indexed and the primary is still saturating.
The load is measurably read-heavy.
There’s the discipline to do and document the stale-read classification.

If any one of these is missing, deferring the replica is cheaper.

Read/write splitting is a cheaper scaling move than growing vertically — but it isn’t free. The price is replication lag, and the cost of ignoring it is paid by showing users their own data as stale.

Before splitting reads, know which reads can tolerate staleness.

Changing Schema in Production With Zero Downtime

info@muhammetsafak.com.tr (Muhammet Şafak) — Sat, 29 Aug 2026 00:00:00 GMT

A deploy ran a single ALTER TABLE on a table with 50 million rows and locked it for four minutes. For four minutes, every request that touched that table waited; the site was effectively down. The migration itself was correct — the problem was doing it in one step.

The dangerous thing is not the schema change. The dangerous thing is making a schema change all at once, without thinking about backward compatibility.

Who takes the lock?

Not every schema change costs the same. In modern PostgreSQL, adding a column with a constant DEFAULT is a metadata operation — it’s fast. The real danger is in operations that lock the table for a long time:

CREATE INDEX — without CONCURRENTLY, it closes the table to writes.
ALTER COLUMN ... TYPE changes that rewrite the table.
NOT NULL, CHECK, or foreign key additions that scan the whole table.

These operations take a strong lock; the ones that take it in ACCESS EXCLUSIVE mode — a table rewrite, or adding NOT NULL/CHECK — queue up every query touching the table, reads included. The rest block writes only. If the table is large, the queue grows.

The dangerous part is the single step

The solution is not to avoid migrations; it’s to break every dangerous migration into small steps, each of which is safe and backward-compatible on its own. This is called the expand-contract pattern.

Suppose you want to rename a column. A single-step RENAME COLUMN instantly breaks running code that reads the old column. Instead, three deploys:

Expand. Add the new column. Have the code write to both old and new, still reading from old.
Migrate. Move the old data into the new column in batches. Now have the code read from new.
Contract. Drop the old column.

Each step works with both the code from the previous release and the new code. At no moment is the running code incompatible with the schema.

Safe recipes

The zero-downtime versions of common changes:

A new NOT NULL column. Adding NOT NULL in one step scans the table. Split it:

-- 1. Add it as nullable first
ALTER TABLE orders ADD COLUMN status text;

-- 2. Backfill existing rows in batches (on the application side)

-- 3. Add the constraint NOT VALID first, then validate in a separate step
ALTER TABLE orders ADD CONSTRAINT orders_status_not_null
    CHECK (status IS NOT NULL) NOT VALID;
ALTER TABLE orders VALIDATE CONSTRAINT orders_status_not_null;

A constraint added with NOT VALID applies immediately to new rows but does not scan existing ones; VALIDATE then scans the table with only a SHARE UPDATE EXCLUSIVE lock — it doesn’t block writes.

An index. Always CONCURRENTLY:

CREATE INDEX CONCURRENTLY idx_orders_status ON orders (status);

A foreign key. The same two-step pattern: first ADD CONSTRAINT ... NOT VALID, then VALIDATE CONSTRAINT.

lock_timeout. While a migration waits for a lock behind a long-running query, it blocks everything behind it too. To prevent this, give the migration session a short lock_timeout — if the lock can’t be taken immediately, let the migration fail instead of waiting, and retry it yourself.

Code and schema must be compatible together

The essence of expand-contract is one rule: at every intermediate step, both the old code still running and the new code must be able to work with the current schema.

The database schema is what I called a “one-way door” in the cost of just-in-case code — rolling it back is expensive. So design the change not as one big, irreversible step, but as small steps that can each be rolled back individually.

Zero-downtime migration is not a tool but a discipline: breaking every schema change into steps small enough that the running code never notices.

The dangerous thing is not the change itself, but doing it in one breath.

Indexes: Too Few Slow Reads, Too Many Kill Writes

info@muhammetsafak.com.tr (Muhammet Şafak) — Sat, 05 Sep 2026 00:00:00 GMT

I’ve seen two opposite teams. One had put no indexes on the table at all — every query a sequential scan, every list page taking seconds. The other had done the exact opposite: “just in case,” it had created an index on every column, and now every INSERT crawled along.

Both are two ends of the same fallacy: thinking an index is a free source of speed.

An index is not free speed

An index speeds up reads, because instead of scanning the whole table the database looks at an ordered structure. But that ordered structure doesn’t stay current on its own: an INSERT writes an entry into every index on the table, and an UPDATE does the same as soon as it touches an indexed column. There are exceptions — a HOT update that changes no indexed column leaves the indexes alone, a partial index is skipped for rows outside its WHERE clause, and a DELETE leaves its index entries behind for VACUUM to clean up later — but they are exceptions, not the default.

So every index is a trade: you buy read speed with write cost. Inserting a single row into a table with five indexes means updating six structures at once. An index isn’t “free read speed,” it’s “read speed paid for with writes.”

A missing index: measure, don’t guess

Find a missing index from the query plan, not from a hunch:

EXPLAIN ANALYZE
SELECT * FROM orders WHERE status = 'pending';

If you see a Seq Scan on a large table in the output and the query returns a small fraction of the rows, an index is probably missing. Scan the pg_stat_user_tables table for large tables with a high seq_scan count — those are your candidates.

Add the index for a real, slow query. An index added “in case we need it later” is the database-layer version of speculative generality.

Too many indexes: find the unused ones

In the other direction, indexes that are never used make every write more expensive for nothing. Find those by measurement too:

SELECT indexrelname, idx_scan
FROM pg_stat_user_indexes
WHERE idx_scan = 0;

An index with idx_scan = 0 means it’s speeding up no reads at all but adding cost to every write. (Keep the indexes for primary keys and unique constraints, of course.) This is the first cleanup listed at the write breaking point of data-intensive systems.

How do you choose the right index?

An index isn’t just “present or absent”; choosing the right type and shape matters:

Column order in a composite index. An (a, b) index serves equality on a + a range query on b; a lookup on b alone can still use the index, but it doesn’t narrow the portion that gets scanned — before PostgreSQL 18 the whole index is scanned, and from 18 on the skip scan optimization narrows it down. Put the column filtered by equality first, the one filtered by range last.
Partial index. If the query always looks at the same subset, limit the index to that subset too: CREATE INDEX ... WHERE status = 'active'. A smaller index, cheaper maintenance.
Covering index. With INCLUDE you can add frequently read columns to the index and keep the database from going to the table at all.
Non-B-tree types. B-tree for equality/ordering; GIN for jsonb and full text; small, cheap BRIN for purely ordered, append-heavy data.

Duplicate and overlapping indexes

An index on (a) is redundant if you already have an (a, b) index — the composite index serves queries starting with a too. Clean up overlaps like these periodically; each one is a silent write tax.

Balance is set with the query plan

One rule: don’t add an index by guesswork, and don’t drop one by guesswork either. An index is added because a real query plan asks for it; an index is dropped because the statistics show no one is using it. The balance between too few and too many is struck with EXPLAIN ANALYZE and pg_stat_user_indexes, not with gut feeling.

Index management starts with shedding the belief that “more indexes is better.” Every index is a read gain and a write cost; good management is keeping the two in balance by measuring them.

A missing index slows the query; too many slow the whole table. Measurement guards against both.