-- Emergency vacuum freeze (run immediately): VACUUM FREEZE VERBOSE <schema>.<table>; -- Increase autovacuum freeze threshold to act earlier: ALTER TABLE <schema>.<table> SET ( autovacuum_freeze_max_age = 500000000 );
-- Check deadlock count trend: SELECT deadlocks, stats_reset FROM pg_stat_database WHERE datname = current_database(); -- Lower deadlock_timeout to detect sooner (default 1s): ALTER SYSTEM SET deadlock_timeout = '500ms'; SELECT pg_reload_conf();
-- Cancel the query (soft — application can retry): SELECT pg_cancel_backend(<pid>); -- Terminate the connection entirely: SELECT pg_terminate_backend(<pid>); -- Prevent future runaway queries: ALTER DATABASE <db> SET statement_timeout = '300s';
-- Cancel the query (soft — application can retry): SELECT pg_cancel_backend(<pid>); -- Terminate the connection entirely: SELECT pg_terminate_backend(<pid>); -- Prevent future runaway queries: ALTER DATABASE <db> SET statement_timeout = '300s';
-- Cancel the query (soft — application can retry): SELECT pg_cancel_backend(<pid>); -- Terminate the connection entirely: SELECT pg_terminate_backend(<pid>); -- Prevent future runaway queries: ALTER DATABASE <db> SET statement_timeout = '300s';
-- Check replica WAL apply position:
SELECT application_name, client_addr, state,
pg_size_pretty(pg_wal_lsn_diff(pg_current_wal_lsn(), replay_lsn)) AS lag
FROM pg_stat_replication;
-- Check wal_keep_size is sufficient:
SHOW wal_keep_size;Transaction ID (XID) wraparound is a catastrophic failure mode. PostgreSQL will shut down all writes when XIDs reach 2 billion to protect data integrity. Run VACUUM FREEZE on affected tables immediately and monitor XID age daily.
-- Emergency vacuum freeze (run immediately): VACUUM FREEZE VERBOSE <schema>.<table>; -- Increase autovacuum freeze threshold to act earlier: ALTER TABLE <schema>.<table> SET ( autovacuum_freeze_max_age = 500000000 );
Deadlocks occur when two transactions each hold locks the other needs. PostgreSQL auto-resolves by cancelling one transaction, but repeated deadlocks indicate a locking order bug. Review application transaction sequences and ensure all code acquires locks in a consistent order.
-- Check deadlock count trend: SELECT deadlocks, stats_reset FROM pg_stat_database WHERE datname = current_database(); -- Lower deadlock_timeout to detect sooner (default 1s): ALTER SYSTEM SET deadlock_timeout = '500ms'; SELECT pg_reload_conf();
A query running over the threshold consumes CPU and shared buffer resources. Cancel or terminate the session to restore capacity. Set statement_timeout to cap query duration and prevent recurrence.
-- Cancel the query (soft — application can retry): SELECT pg_cancel_backend(<pid>); -- Terminate the connection entirely: SELECT pg_terminate_backend(<pid>); -- Prevent future runaway queries: ALTER DATABASE <db> SET statement_timeout = '300s';
A query running over the threshold consumes CPU and shared buffer resources. Cancel or terminate the session to restore capacity. Set statement_timeout to cap query duration and prevent recurrence.
-- Cancel the query (soft — application can retry): SELECT pg_cancel_backend(<pid>); -- Terminate the connection entirely: SELECT pg_terminate_backend(<pid>); -- Prevent future runaway queries: ALTER DATABASE <db> SET statement_timeout = '300s';
A query running over the threshold consumes CPU and shared buffer resources. Cancel or terminate the session to restore capacity. Set statement_timeout to cap query duration and prevent recurrence.
-- Cancel the query (soft — application can retry): SELECT pg_cancel_backend(<pid>); -- Terminate the connection entirely: SELECT pg_terminate_backend(<pid>); -- Prevent future runaway queries: ALTER DATABASE <db> SET statement_timeout = '300s';
Replication lag means the standby is behind the primary. During failover, recently committed transactions may be lost. Investigate network throughput, replica instance class, replica read load, and wal_keep_size.
-- Check replica WAL apply position:
SELECT application_name, client_addr, state,
pg_size_pretty(pg_wal_lsn_diff(pg_current_wal_lsn(), replay_lsn)) AS lag
FROM pg_stat_replication;
-- Check wal_keep_size is sufficient:
SHOW wal_keep_size;| Severity | Category | Object / PID | Detail | Context | Action |
|---|---|---|---|---|---|
| CRITICAL | Lock Root Blocker | PID 23960 | 1s running | /* /app/modules/app/services/sale_manager.py:151 */ SELECT DISTINCT ON ("example_data_storeproductsale"."product_id") "example_data_store | SELECT pg_terminate_backend(23960); |
| CRITICAL | Lock Root Blocker | PID 24044 | -0s running | /* /app/modules/client/action_modifiers/discontinued.py:104 */ SELECT "example_products_example_productcomplianceproductreport"."updated_at", | SELECT pg_terminate_backend(24044); |
| CRITICAL | Lock Chain | 2 sessions blocked | up to 3s wait time | — | see Lock Chains section below |
| CRITICAL | Long Query | PID 83335 | 206.7h | Client/WalSenderWaitForWAL | START_REPLICATION SLOT "replica-02" LOGICAL 0/0 (proto_version '4', origin 'any', publication_names '"pub4dwh"') | SELECT pg_terminate_backend(83335); |
| CRITICAL | Long Query | PID 83338 | 206.7h | Client/WalSenderWriteData | START_REPLICATION SLOT "replica-01" LOGICAL 0/0 (proto_version '4', origin 'any', publication_names '"replica-01"') | SELECT pg_terminate_backend(83338); |
| CRITICAL | Long Query | PID 3441 | 2.8m | IO/DataFileRead | VACUUM (FREEZE, INDEX_CLEANUP auto, ANALYZE) public.example_data_impl_sale; | SELECT pg_terminate_backend(3441); |
| CRITICAL | XID Wraparound | audit.ddl_audit | 8,448,263 transactions left | — | VACUUM FREEZE ANALYZE audit.ddl_audit; |
| CRITICAL | Replication Lag | replica-01 (10.x.x.x) | 43413 MB behind primary | — | Investigate network or replica performance |
| HIGH | Long Query | PID 22316 | 17s | IO/DataFileRead | /* /app/modules/api/views/stores.py:171 */ SELECT DISTINCT ON ("example_products_implementation_example_productstore"."example_product_id") "example_product | SELECT pg_terminate_backend(22316); |
| HIGH | Long Query | PID 23126 | 16s | CPU / running | /* /app/modules/app/views.py:1816 */ SELECT DISTINCT ON ("example_product_name", "section_name", "example_products_implementation_example_productsectionproduct | SELECT pg_terminate_backend(23126); |
| Sev | Blocked PID | Blocked For | Blocked Query | Blocker PID | Blocker Running | Blocker Query | Lock Type | Kill |
|---|---|---|---|---|---|---|---|---|
| OK | 23988 | 3s | /* /app/modules/shared/models/timestamp.py:100 */ UPDATE "example_records_implementation_example_record" SET "locked_by_id" = NULL, "locked_at" = NULL WHERE "rea | 23960 (ROOT) | 1s | /* /app/modules/app/services/sale_manager.py:151 */ SELECT DISTINCT ON ("example_data_storeproductsale"."product_id" | transactionid | pg_terminate_backend(23960) |
| OK | 24068 | 2s | /* /app/modules/shared/models/timestamp.py:100 */ UPDATE "example_records_implementation_example_record" SET "locked_by_id" = NULL, "locked_at" = NULL WHERE "rea | 24044 (ROOT) | -0s | /* /app/modules/client/action_modifiers/discontinued.py:104 */ SELECT "example_products_example_productcomplianceproductre | transactionid | pg_terminate_backend(24044) |
| Sev | PID | App | Running | Wait | Query |
|---|---|---|---|---|---|
| CRITICAL | 83335 | replica-02 | 206.7h | Client/WalSenderWaitForWAL | START_REPLICATION SLOT "replica-02" LOGICAL 0/0 (proto_version '4', origin 'any', publication_names '"pub4dwh"') |
| CRITICAL | 83338 | replica-01 | 206.7h | Client/WalSenderWriteData | START_REPLICATION SLOT "replica-01" LOGICAL 0/0 (proto_version '4', origin 'any', publication_names '"replica-01"') |
| CRITICAL | 3441 | client_user | 2.8m | IO/DataFileRead | VACUUM (FREEZE, INDEX_CLEANUP auto, ANALYZE) public.example_data_impl_sale; |
| HIGH | 22316 | client_user | 17s | IO/DataFileRead | /* /app/modules/api/views/stores.py:171 */ SELECT DISTINCT ON ("example_products_implementation_example_productstore"."example_product_id") "example_products_implementation_example_productstore"."example_product_id" FROM "planog |
| HIGH | 23126 | client_user | 16s | CPU / running | /* /app/modules/app/views.py:1816 */ SELECT DISTINCT ON ("example_product_name", "section_name", "example_products_implementation_example_productsectionproduct"."product_id") "example_products_implementation_example_productsectionp |
| Type | Event | Sessions | Distribution |
|---|---|---|---|
| IO | DataFileRead | 3 | |
| Client | WalSenderWriteData | 1 | |
| Client | WalSenderWaitForWAL | 1 | |
| Client | ClientRead | 1 |
| Sev | Table | XID Age | Transactions Left | Action |
|---|---|---|---|---|
| CRITICAL | audit.ddl_audit | 191,551,737 | 8,448,263 | VACUUM FREEZE ANALYZE audit.ddl_audit; |
| OK | public.reports_snowflakereportrunpart | 49,937,760 | 150,062,240 | — |
| OK | public.example_data_impl_storeproductprice | 48,012,912 | 151,987,088 | — |
| OK | public.example_data_impl_supplier | 46,612,169 | 153,387,831 | — |
| OK | public.example_records_implementation_pricetagrecognizedtext | 46,049,649 | 153,950,351 | — |
| OK | public.example_products_example_productcomplianceproductreport | 42,740,954 | 157,259,046 | — |
| OK | public.tasks_group | 39,699,989 | 160,300,011 | — |
| OK | public.example_records_implementation_example_recordpricetag | 36,721,710 | 163,278,290 | — |
| OK | public.example_records_implementation_pricetagtext | 36,721,710 | 163,278,290 | — |
| OK | public.example_records_implementation_pricetagproductcandidate | 36,721,710 | 163,278,290 | — |
| Sev | Table | Dead Tuples | Dead % | Last Autovacuum |
|---|---|---|---|---|
| OK | public.example_records_implementation_example_recordproductcandidate | 11,558,463 | 0.5% | 2026-05-12 06:54:19 |
| OK | public.example_data_impl_file | 10,862,366 | 0.3% | 2026-05-10 15:16:33 |
| OK | public.example_products_implementation_example_productsectionproduct | 10,237,035 | 0.1% | 2026-05-12 01:45:24 |
| OK | public.example_records_example_recordproductcoreid | 9,596,973 | 0.4% | 2026-05-10 15:15:15 |
| OK | public.example_records_implementation_example_recordproduct | 7,820,609 | 0.3% | 2026-05-12 06:36:05 |
| OK | public.example_products_example_productcomplianceexample_recordproduct | 5,533,275 | 0.3% | 2026-05-10 03:20:52 |
| OK | public.example_data_impl_storeproductprice | 4,320,732 | 0.5% | 2026-05-03 09:51:19 |
| OK | public.example_records_implementation_pricetagrecognizedtext | 4,038,455 | 0.4% | 2026-05-03 01:32:13 |
| OK | public.example_records_implementation_productpricetagmap | 2,506,953 | 0.2% | 2026-05-12 01:22:10 |
| OK | public.example_products_example_productcomplianceexample_productproduct | 2,283,113 | 0.2% | 2026-05-10 15:15:07 |
| OK | public.example_records_implementation_pricetagproductcandidate | 2,239,548 | 0.3% | 2026-05-03 10:09:39 |
| OK | public.example_products_example_productcomplianceaction | 1,331,509 | 0.4% | 2026-05-07 00:15:51 |
| OK | public.example_records_implementation_example_recordpricetag | 1,250,591 | 0.1% | 2026-05-12 08:08:51 |
| OK | public.example_products_implementation_example_productstore | 1,056,363 | 0.4% | 2026-04-26 01:49:27 |
| OK | public.example_data_storeproductsale | 1,048,711 | 0.0% | never |
| Sev | Table | Dead % | Live Rows | Total Size |
|---|---|---|---|---|
| OK | public.supervised_learning_scanlearningtransaction | 3.4% | 82,194 | 61 MB |
| OK | public.retechlabs_import_export_dataimport | 3.4% | 1,016,739 | 2865 MB |
| OK | public.supervised_learning_pricetaglearningtransaction | 3.3% | 440,672 | 107 MB |
| OK | public.example_records_implementation_pricetagtext | 1.9% | 926,934 | 115 MB |
| OK | public.supervised_learning_lastapprovedtransactionforexample_recordproduct | 1.8% | 690,861 | 112 MB |
| OK | public.supervised_learning_rejectedscan | 1.7% | 14,272 | 17 MB |
| OK | public.retechlabs_import_export_dataimportgroup | 1.5% | 10,521 | 1552 kB |
| OK | public.example_records_scancompletiontimelog | 1.2% | 542,699 | 172 MB |
| OK | public.example_flags | 1.0% | 26,981,530 | 12 GB |
| OK | public.tasks_taskdefproduct | 0.8% | 69,334 | 1223 MB |
| OK | public.supervised_learning_rejectedscanrejectedscanreason | 0.8% | 19,757 | 8920 kB |
| OK | public.example_data_impl_producttoproductgroup | 0.8% | 690,124 | 990 MB |
| OK | public.example_data_impl_aislesale | 0.8% | 2,785,944 | 426 MB |
| OK | public.tasks_trackedbannerlog | 0.7% | 101,248 | 29 MB |
| OK | public.auth_user | 0.7% | 9,216 | 3096 kB |
| Sev | Type | Hit Rate | Reads / Hits |
|---|---|---|---|
| OK | Heap (table) | 98.28% | 72,645,417,342 reads / 4,140,443,413,224 hits |
| OK | Index | 99.01% | 50,223,259,161 reads / 5,021,726,840,485 hits |
| Sev | Replica | Address | State | Lag (MB) | Sync Mode |
|---|---|---|---|---|---|
| CRITICAL | replica-01 | 10.x.x.x | streaming | 43413.4 MB | async |
| OK | replica-02 | 10.x.x.x | streaming | 2.7 MB | async |
| Table | Seq Scans | Seq % | Index Scans | Size |
|---|---|---|---|---|
| public.example_data_impl_bannerlogo | 351,109,947 | 99.2% | 2,766,047 | 80 kB |
| Sev | PID | App | Idle For | Last Query |
|---|---|---|---|---|
| OK | 110986 | client_user | 0s | /* /app/modules/app/compliance.py:2589 */ SELECT "example_products_example_productcomplianceexample_productproduct"."id", "example_products_planogr |
| OK | 19640 | client_user | 0s | /* /app/modules/app/compliance.py:2589 */ SELECT "example_products_example_productcomplianceexample_productproduct"."id", "example_products_planogr |
| OK | 94827 | client_user | -0s | /* /app/modules/app/filtering_services/size_filtering.py:98 */ SELECT DISTINCT ON ("example_data_impl_storeproductprice"."store_id", "mas |
| OK | 113891 | client_user | -0s | /* /app/modules/app/compliance.py:2589 */ SELECT "example_products_example_productcomplianceexample_productproduct"."id", "example_products_planogr |
| OK | 91051 | client_user | -0s | /* /app/modules/app/compliance.py:2589 */ SELECT "example_products_example_productcomplianceexample_productproduct"."id", "example_products_planogr |
| OK | 23895 | client_user | -0s | /* /app/modules/app/services/price_tag_mapper.py:38 */ SELECT "example_records_implementation_productpricetagmap"."updated_at", "example_records_implementation_produc |
| OK | 23669 | client_user | -0s | /* /app/modules/app/tasks.py:45 */ DELETE FROM "example_data_impl_file" WHERE "example_data_impl_file"."id" IN (2567131457) |
| OK | 23858 | client_user | -0s | /* /app/modules/app/tasks.py:45 */ DELETE FROM "example_data_impl_file" WHERE "example_data_impl_file"."id" IN (2567166384) |
| OK | 24044 | client_user | -0s | /* /app/modules/client/action_modifiers/discontinued.py:104 */ SELECT "example_products_example_productcomplianceproductreport"."updated_at", "example_products_complian |
| OK | 23803 | client_user | -0s | /* /app/modules/app/tasks.py:45 */ DELETE FROM "example_data_impl_file" WHERE "example_data_impl_file"."id" IN (2567131632) |
| OK | 24141 | client_user | -0s | /* /app/modules/app/services/spacial_manager.py:567 */ SELECT "example_records_implementation_example_recordproductcandidate"."id", "example_records_implementation_realogr |
| Event Type | Severity | Count |
|---|---|---|
| Slow queries (log ≥ 500 ms) | 112,292 | 112,292 events (30 shown below) |
| Deadlocks | 667 | 667 |
| Lock timeouts | 0 | 0 |
| Autovac-cancel queries | 0 | 0 |
| Checkpoint warnings | 327 | 327 |
| Temp file spills | 0 | 0 |
| OOM events | 0 | 0 |
| FATAL / PANIC | 183 | 183 |
| Sev | Duration (ms) | Query |
|---|---|---|
| CRITICAL | 2,480,124.6 | /* /app/apps/example_products/utils.py:743 */ SELECT "example_products_implementation_example_productstore"."id" FROM "example_products_implementation_example_productstore" WHERE "example_products_implementation_example_productstore"."id" = 318212952 ORDER BY "example_products_implementation_example_productstore" |
| CRITICAL | 2,479,954.8 | /* /app/apps/example_products/utils.py:743 */ SELECT "example_products_implementation_example_productstore"."id" FROM "example_products_implementation_example_productstore" WHERE "example_products_implementation_example_productstore"."id" = 318212952 ORDER BY "example_products_implementation_example_productstore" |
| CRITICAL | 2,436,072.5 | /* /app/apps/example_products/utils.py:743 */ SELECT "example_products_implementation_example_productstore"."id" FROM "example_products_implementation_example_productstore" WHERE "example_products_implementation_example_productstore"."id" = 318212952 ORDER BY "example_products_implementation_example_productstore" |
| CRITICAL | 2,268,821.0 | /* /app/apps/example_products/utils.py:743 */ SELECT "example_products_implementation_example_productstore"."id" FROM "example_products_implementation_example_productstore" WHERE "example_products_implementation_example_productstore"."id" = 318212952 ORDER BY "example_products_implementation_example_productstore" |
| CRITICAL | 2,266,325.9 | /* /app/apps/example_products/utils.py:743 */ SELECT "example_products_implementation_example_productstore"."id" FROM "example_products_implementation_example_productstore" WHERE "example_products_implementation_example_productstore"."id" = 318212952 ORDER BY "example_products_implementation_example_productstore" |
| CRITICAL | 2,266,263.7 | /* /app/apps/example_products/utils.py:743 */ SELECT "example_products_implementation_example_productstore"."id" FROM "example_products_implementation_example_productstore" WHERE "example_products_implementation_example_productstore"."id" = 318212952 ORDER BY "example_products_implementation_example_productstore" |
| CRITICAL | 2,253,768.9 | /* /app/apps/example_products/utils.py:743 */ SELECT "example_products_implementation_example_productstore"."id" FROM "example_products_implementation_example_productstore" WHERE "example_products_implementation_example_productstore"."id" = 318212952 ORDER BY "example_products_implementation_example_productstore" |
| CRITICAL | 2,247,731.3 | /* /app/apps/example_products/utils.py:743 */ SELECT "example_products_implementation_example_productstore"."id" FROM "example_products_implementation_example_productstore" WHERE "example_products_implementation_example_productstore"."id" = 318212952 ORDER BY "example_products_implementation_example_productstore" |
| CRITICAL | 2,224,452.0 | /* /app/apps/example_products/utils.py:743 */ SELECT "example_products_implementation_example_productstore"."id" FROM "example_products_implementation_example_productstore" WHERE "example_products_implementation_example_productstore"."id" = 318212952 ORDER BY "example_products_implementation_example_productstore" |
| CRITICAL | 2,156,732.9 | /* /app/modules/shared/middleware.py:48 */ /* /app/modules/shared/middleware.py:48 */ /* /app/apps/example_products/utils.py:743 */ SELECT "example_products_implementation_example_productstore"."id" FROM "example_products_implementation_example_productstore" WHERE "planog |
| CRITICAL | 2,135,994.1 | /* /app/apps/example_products/utils.py:743 */ SELECT "example_products_implementation_example_productstore"."id" FROM "example_products_implementation_example_productstore" WHERE "example_products_implementation_example_productstore"."id" = 318212952 ORDER BY "example_products_implementation_example_productstore" |
| CRITICAL | 2,135,765.3 | /* /app/apps/example_products/utils.py:743 */ SELECT "example_products_implementation_example_productstore"."id" FROM "example_products_implementation_example_productstore" WHERE "example_products_implementation_example_productstore"."id" = 318212952 ORDER BY "example_products_implementation_example_productstore" |
| CRITICAL | 2,063,684.8 | /* /app/apps/example_products/utils.py:743 */ SELECT "example_products_implementation_example_productstore"."id" FROM "example_products_implementation_example_productstore" WHERE "example_products_implementation_example_productstore"."id" = 318212952 ORDER BY "example_products_implementation_example_productstore" |
| CRITICAL | 1,974,615.0 | /* /app/apps/example_products/utils.py:743 */ SELECT "example_products_implementation_example_productstore"."id" FROM "example_products_implementation_example_productstore" WHERE "example_products_implementation_example_productstore"."id" = 318212952 ORDER BY "example_products_implementation_example_productstore" |
| CRITICAL | 1,966,857.5 | /* /app/apps/example_products/utils.py:743 */ SELECT "example_products_implementation_example_productstore"."id" FROM "example_products_implementation_example_productstore" WHERE "example_products_implementation_example_productstore"."id" = 318212952 ORDER BY "example_products_implementation_example_productstore" |
| CRITICAL | 1,948,463.5 | /* /app/apps/example_products/utils.py:743 */ SELECT "example_products_implementation_example_productstore"."id" FROM "example_products_implementation_example_productstore" WHERE "example_products_implementation_example_productstore"."id" = 318212952 ORDER BY "example_products_implementation_example_productstore" |
| CRITICAL | 1,948,333.7 | /* /app/apps/example_products/utils.py:743 */ SELECT "example_products_implementation_example_productstore"."id" FROM "example_products_implementation_example_productstore" WHERE "example_products_implementation_example_productstore"."id" = 318212952 ORDER BY "example_products_implementation_example_productstore" |
| CRITICAL | 1,873,230.7 | /* /app/apps/example_products/utils.py:743 */ SELECT "example_products_implementation_example_productstore"."id" FROM "example_products_implementation_example_productstore" WHERE "example_products_implementation_example_productstore"."id" = 318212952 ORDER BY "example_products_implementation_example_productstore" |
| CRITICAL | 1,866,694.4 | /* /app/apps/example_products/utils.py:743 */ SELECT "example_products_implementation_example_productstore"."id" FROM "example_products_implementation_example_productstore" WHERE "example_products_implementation_example_productstore"."id" = 318212952 ORDER BY "example_products_implementation_example_productstore" |
| CRITICAL | 1,810,103.6 | /* /app/apps/example_products/utils.py:743 */ SELECT "example_products_implementation_example_productstore"."id" FROM "example_products_implementation_example_productstore" WHERE "example_products_implementation_example_productstore"."id" = 318212952 ORDER BY "example_products_implementation_example_productstore" |