Mansoor Faizi — Engineering Journal

Django 6.1 is out: fetch modes, DB-level deletes, and MAILERS

Thu, 06 Aug 2026 09:00:00 GMT

Django 6.1 landed on 5 August 2026. It is not a rewrite release. The three headlines in the official notes are model field fetch modes, database-level ForeignKey.on_delete options, and dictionary-based email settings via MAILERS. Around those, there is a long list of admin, forms, JSONField, CSP, and deprecation work that matters the moment you open a PR to bump the pin. I read the release notes against two production codebases — a DRF inventory API and a server-rendered portal — and ordered what follows by how much it changes day-to-day Django work, not by how it appears in the docs table of contents. :::note Support window With 6.1 out, Django 6.0 is at end of mainstream support. 6.0 still gets security and data-loss fixes until April 2027. Python support for 6.1 is 3.12, 3.13, and 3.14. Postgres 14, MySQL below 8.4, and MariaDB below 10.11 are dropped. ::: ## Fetch modes: the N+1 fix that does not need a field list This is the feature I will reach for first. Accessing a deferred or related field that was not loaded with the original query used to always mean one extra query per instance — Django's classic N+1. 6.1 makes that behavior configurable with QuerySet.fetch_mode(). There are three modes in django.db.models: - FETCH_ONE — the default, and today's behavior: fetch the missing field for this instance only. - FETCH_PEERS — on first access, batch-fetch that field for every instance that came from the same QuerySet. Most N+1 loops collapse to two queries. - RAISE — any lazy field access raises FieldFetchBlocked. Use this in hot paths where a surprise query is a bug. ```python catalog/views.py from django.db import models from .models import Book def book_list(request): books = Book.objects.filter(in_stock=True).fetch_mode(models.FETCH_PEERS) # Despite touching author on every row, this is two queries: # 1) books, 2) authors for the peer set return render( request, "catalog/list.html", {"rows": [(b.title, b.author.name) for b in books]}, ) ``` FETCH_PEERS behaves like an on-demand prefetch_related(). You do not have to maintain a brittle list of every relation a template might touch. The mode also copies onto related objects reached from the same queryset, so the policy applies deeper than the top-level model. ```python catalog/managers.py from django.db import models class BookManager(models.Manager): def get_queryset(self): # Default the whole model to peer fetching. return super().get_queryset().fetch_mode(models.FETCH_PEERS) class Book(models.Model): title = models.CharField(max_length=200) author = models.ForeignKey("Author", on_delete=models.CASCADE) objects = BookManager() ``` ### When I still write select_related / prefetch_related - You already know the exact fields — spelling them is clearer and often cheaper than a reactive batch. - Filtered or ordered Prefetch() objects — FETCH_PEERS does not replace those. - Performance-critical endpoints where RAISE is better than any lazy fetch. :::note Deprecation to know Calling select_related() with no arguments (select everything) is deprecated in 6.1 and removed in 7.0. Specify fields, or use FETCH_PEERS. Same idea for ModelAdmin.list_select_related = True. ::: ## Database-level ForeignKey.on_delete ForeignKey.on_delete now supports options that compile to SQL ON DELETE instead of loading related rows in Python: DB_CASCADE, DB_SET_NULL, and DB_SET_DEFAULT (and related DB_* helpers in the docs). Deletion work stays in the database, which is faster and avoids pulling large object graphs into memory. ```python billing/models.py from django.db import models class InvoiceLine(models.Model): invoice = models.ForeignKey( "Invoice", on_delete=models.DB_CASCADE, # SQL ON DELETE CASCADE related_name="lines", ) account = models.ForeignKey( "Account", on_delete=models.DB_SET_NULL, null=True, related_name="lines", ) ``` The trade-off is intentional: DB_CASCADE does not fire pre_delete or post_delete signals, because Django never loads the doomed rows. If your audit trail or soft-delete logic lives in those signals, keep the Python-level CASCADE. If you just need referential cleanup at scale, move to DB_*. ## MAILERS: email backends as a dict Email configuration finally looks like DATABASES, CACHES, STORAGES, and TASKS. The new MAILERS setting names multiple backends; send helpers take using="alias". EMAIL_BACKEND and the flat EMAIL_* settings still work but emit deprecation warnings — they go away in Django 7.0. ```python settings.py MAILERS = { "default": { "BACKEND": "django.core.mail.backends.smtp.EmailBackend", "OPTIONS": { "host": "smtp.example.com", "port": 587, "use_tls": True, "username": env("SMTP_USER"), "password": env("SMTP_PASSWORD"), }, }, "marketing": { "BACKEND": "anymail.backends.mailgun.EmailBackend", "OPTIONS": {"api_key": env("MAILGUN_API_KEY")}, }, } ``` ```python accounts/emails.py from django.core.mail import send_mail def send_welcome(to: str) -> None: send_mail( subject="Welcome", message="…", from_email="hello@example.com", recipient_list=[to], using="default", ) def send_campaign(to: str, body: str) -> None: send_mail( subject="This month", message=body, from_email="news@example.com", recipient_list=[to], using="marketing", ) ``` mail.mailers["default"] works whether you define MAILERS or still use EMAIL_BACKEND, which makes a gradual cutover painless. get_connection() and the connection= argument on send helpers are deprecated — migrate to using= and mail.mailers. ## Worth knowing from the rest of the release ### Admin - Authenticated users hitting the admin login are redirected to next when present, not always to the index. - When list_select_related is False, the changelist only select_relateds foreign keys that appear in list_display — fewer accidental joins. - Accessibility: labels above fields, help text and errors before the input; boolean icons for related boolean columns in list_display. - action() gains location (changelist and/or change form) and description_plural. ### Models and database - JSONNull expression for an explicit JSON scalar null (prefer this over None for top-level JSONField queries — None is deprecated there). - UUID4 and UUID7 database functions; BitAnd / BitOr / BitXor aggregates move out of contrib.postgres into django.db.models. - in_bulk() chains after values() / values_list(); QuerySet.totally_ordered reports whether ordering is deterministic. - DecimalField.max_digits / decimal_places no longer required on Oracle, PostgreSQL, and SQLite. - GeneratedField stored columns (db_persist=True) on Oracle 23ai/26ai (23.7+). ```python catalog/queries.py from django.db.models import JSONNull from django.db.models.functions import UUID7 # Query top-level JSON null explicitly Product.objects.filter(metadata=JSONNull) # Generate UUIDv7 in the database Product.objects.annotate(public_id=UUID7()) ``` ### CSP, forms, redirects - csp_nonce_attr template tag for script/style nonces when the csp() context processor is on; system check W027 if CSP.NONCE is set without that processor. - BLANK_CHOICE_LABEL replaces BLANK_CHOICE_DASH for a clearer blank option label. - FilePathField.set_choices() to rescan directories per request. - RedirectView.preserve_request uses 307/308 to keep method and body across redirects. ### Auth, GIS, sessions, tasks - PBKDF2 default iterations rise from 1,200,000 to 1,500,000. - Permission names/codenames rename with model renames in migrations; Permission.user_perm_str for has_perm() strings. - Sessions support bool(session); GIS gains isempty / num_dimensions helpers; OpenLayersWidget moves to OpenLayers 10.9.0. - Tasks: @task(**kwargs) forwarded to the backend; Task / TaskResult are pickleable. ## Backwards incompatible changes that will bite 1. PostgreSQL 14, MySQL < 8.4, MariaDB < 10.11, PostGIS 3.1, older GEOS/GDAL — upgrade the database side before or with Django. 2. ArrayField of JSONField: top-level None elements save as SQL NULL, matching standalone JSONField. 3. JSONField key transform iexact=None now matches JSON null (aligned with exact=None). 4. first() / last() no longer fall back to PK ordering after an empty order_by(). 5. Annotate / JOIN aliases are systematically quoted — raw SQL that mixed case on aliases may need quoting fixes. 6. Strict Base64 validation on BinaryField, multipart parsing, and DatabaseCache — corrupt cache rows can start raising. 7. ASGI RemoteUserMiddleware no longer auto-prefixes HTTP_ on custom META lookups (restores pre-5.2 behavior). ## Deprecations to fix before 7.0 - mail.get_connection(), connection= on send helpers, and constructing smtp.EmailBackend directly — move to MAILERS + using=. - select_related() with no args; list_select_related = True without an explicit field list. - values_list(flat=True) with no field name — pass the field explicitly. - None as top-level JSON null in queries — use JSONNull. - BLANK_CHOICE_DASH / USE_BLANK_CHOICE_DASH — use BLANK_CHOICE_LABEL. - transaction.savepoint() — use savepoint_create(). - Default algorithm for salted_hmac / base64_hmac changing from sha1 to sha256 in 7.0 — pass algorithm= explicitly now. :::note How I sequence the upgrade Branch, bump Django to 6.1, run the suite with -W error::DeprecationWarning, fix warnings only, merge. Then adopt FETCH_PEERS, DB_* deletes, and MAILERS in separate PRs so a rollback never mixes 'version bump' with 'behavior change'. ::: ## Is it worth upgrading this week? If you are still on 6.0 and only care about security patches, you can wait — but mainstream support for 6.0 has already ended, so plan the bump before April 2027. If your bug tracker is full of N+1 tickets, fetch modes alone justify the afternoon. If you run transactional deletes over large child tables, DB_CASCADE is the quiet win. If you juggle SMTP for product mail and a third-party ESP for marketing, MAILERS is the cleanup you have been putting off. > 6.1 does not ask you to rewrite your stack. It asks you to stop paying for N+1s, Python-side cascades, and a single global EMAIL_BACKEND you outgrew two years ago. Full detail lives in the official Django 6.1 release notes and the new Fetch modes topic. Pin django>=6.1,<6.2, read the deprecation list once with warnings-as-errors, then ship fetch_mode where your serializers and templates still surprise you.

I Put Django on ASGI for a Lab Results Feed. It Helped One View.

Tue, 28 Jul 2026 09:00:00 GMT

A clinic chain we support runs a patient portal on Django 4.2. Most of it is boring CRUD — appointments, prescriptions, billing. One screen is not boring: the lab results dashboard pulls from our Postgres database, two internal microservices, and a third-party HL7 bridge that sometimes takes six seconds to answer. Under gunicorn with sync workers, that page's p99 was 11.2 seconds in staging. My manager asked if ASGI would fix it. It did, for exactly that screen, and almost nowhere else. ## The one view that was worth it The lab dashboard does four independent HTTP calls after authentication. Three are fast (80–200 ms). The HL7 bridge is the outlier. On sync workers, gunicorn held a worker hostage for the full chain. With eight workers and twenty concurrent clinicians refreshing during morning rounds, requests queued behind each other even though three of the four calls could have overlapped. ```python labs/views.py import asyncio import httpx from django.contrib.auth.decorators import login_required from django.http import JsonResponse from labs.selectors import patient_context_for_user from labs.services import build_results_payload HL7_TIMEOUT = httpx.Timeout(8.0, connect=2.0) @login_required async def lab_results_dashboard(request): ctx = await sync_patient_context(request.user.id) async with httpx.AsyncClient(timeout=HL7_TIMEOUT) as client: internal, imaging, hl7 = await asyncio.gather( fetch_internal_labs(client, ctx["patient_id"]), fetch_imaging_summary(client, ctx["patient_id"]), fetch_hl7_bridge(client, ctx["mrn"]), return_exceptions=True, ) payload = build_results_payload( internal=internal if not isinstance(internal, Exception) else None, imaging=imaging if not isinstance(imaging, Exception) else None, hl7=hl7 if not isinstance(hl7, Exception) else None, ) return JsonResponse(payload) ``` After deploy, that view's p99 went from 11.2 s to 2.4 s — basically the slowest downstream plus overhead. Median dropped from 3.1 s to 780 ms. The rest of the portal? Unchanged. I should have scoped ASGI to a separate deployment from the start instead of flipping the whole app. ## What broke when we went async everywhere We ran Daphne behind the same ALB path rules as before. Three things failed in the first 48 hours. - Custom audit middleware called time.sleep(0) in a loop waiting for a thread-local request ID that never got set on async views — 502s on 4% of requests until we rewrote it. - django.contrib.sessions still worked, but our Redis cache wrapper used a sync client inside async views without sync_to_async. Intermittent session loss under load. - A post_save signal on LabOrder called async_to_sync(notify_patient) from code paths that sometimes ran inside the async view's event loop. Deadlocks in staging only, which is the worst kind of bug. :::note We split deployments after week one Sync gunicorn serves 94% of routes. uvicorn serves /labs/async/* only. Same codebase, two process types, two systemd units. Ops hated the extra unit until the p99 graph made the trade obvious. ::: ## The ORM lie I kept falling for I wrote the first version using aget and acount because it felt modern. Profiling showed each await still blocked a thread from Django's default sync_to_async pool — max 40 threads on our box. At 120 concurrent dashboard loads, ORM calls queued for 400–900 ms before the HTTP fanout even started. I reverted to one bulk sync query wrapped in sync_to_async(thread_sensitive=False) and shaved 600 ms off p95. ```python labs/selectors.py from asgiref.sync import sync_to_async from django.db.models import Prefetch from labs.models import LabOrder, LabResult @sync_to_async(thread_sensitive=False) def sync_patient_context(user_id: int) -> dict: profile = ( UserProfile.objects .select_related("patient") .get(user_id=user_id) ) recent_orders = list( LabOrder.objects .filter(patient_id=profile.patient_id) .prefetch_related(Prefetch("results", queryset=LabResult.objects.order_by("-collected_at"))) .order_by("-ordered_at")[:20] ) return { "patient_id": profile.patient_id, "mrn": profile.patient.mrn, "recent_orders": recent_orders, } ``` ### Database connections and thread_sensitive=False We only use thread_sensitive=False on read-only selector functions that open their own connection via Django's pool. I tried it on a service that updated last_seen_at on the profile and got duplicate key errors twice before I understood why. That mistake cost me a Friday. ## Middleware and third-party packages django-cors-headers, whitenoise, and our JWT middleware were all sync. Django wraps sync middleware automatically in ASGI mode, but each layer adds thread-pool hops. We counted seven middleware classes. Stripping two unused ones (legacy mobile detection and a debug header injector) saved ~40 ms per request on the async routes — not huge, but measurable. ```python config/asgi.py import os from django.core.asgi import get_asgi_application from django.urls import path from channels.routing import ProtocolTypeRouter, URLRouter os.environ.setdefault("DJANGO_SETTINGS_MODULE", "config.settings.production") django_asgi = get_asgi_application() application = ProtocolTypeRouter({ "http": URLRouter([ path("api/labs/dashboard/", django_asgi), path("api/labs/stream/", django_asgi), ]), }) ``` We never adopted Channels for websockets on this project — the ProtocolTypeRouter is just a cheap way to mount only lab routes on uvicorn. Everything else never touches this file. > ASGI didn't make Django fast. It let one I/O-bound view stop blocking a worker per upstream wait. That's a narrower win than the blog posts suggest. ## What I'd do again I'd still pick ASGI for that dashboard. I wouldn't migrate the admin, the PDF export views, or anything that runs ReportLab on the request thread. I'd add a lint rule blocking async_to_sync outside Celery and management commands on day one. And I'd load-test with 150 concurrent users before telling the clinic we'd fixed their morning-rounds slowness — we didn't, the first time.

The DRF mistakes that bit us after go-live

Sat, 18 Jul 2026 09:00:00 GMT

Three days after we put a hospital appointments API into production, the night nurse desk started timing out around 8pm. The dashboard looked fine in staging — twenty seed patients, one doctor. Real data was 18,000 appointments and a serializer that touched related objects like it was free. I have a short list I run through now before any DRF service sees real traffic. It is not theoretical. ## Prefetch in the ViewSet, not in the serializer If a field on the serializer touches a related model, that relationship has to be declared on the queryset. I treat SerializerMethodField that does `.filter()` or `.all()` as a code smell in review — it almost always means we will rediscover N+1 under load. ```python appointments/views.py from rest_framework import viewsets from rest_framework.permissions import IsAuthenticated from .models import Appointment from .serializers import AppointmentSerializer class AppointmentViewSet(viewsets.ReadOnlyModelViewSet): serializer_class = AppointmentSerializer permission_classes = [IsAuthenticated] def get_queryset(self): return ( Appointment.objects.filter(clinic_id=self.request.user.clinic_id) .select_related("patient", "doctor", "room") .prefetch_related("vitals", "attachments") .order_by("-starts_at") ) ``` :::note Prove it in CI We assert query counts with assertNumQueries on the three hottest list endpoints. A jump from 4 queries to 40 fails the build. That single habit caught more regressions than any APM alert. ::: ## Stop computing totals in Python We had an order total field that summed line items in a SerializerMethodField. Fine at 10 rows. Painful at 200. Annotate once in the database and expose a read-only field. ```python orders/serializers.py from django.db.models import F, Sum from rest_framework import serializers from .models import Order class OrderSerializer(serializers.ModelSerializer): total = serializers.DecimalField(max_digits=12, decimal_places=2, read_only=True) class Meta: model = Order fields = ("id", "reference", "status", "total", "created_at") # get_queryset(): # .annotate(total=Sum(F("items__quantity") * F("items__unit_price"))) ``` ## Hard limits before someone else finds them - Default page size 25, max page size 100 — never unbounded list endpoints. - Throttle anon and authenticated users separately. Our first public endpoint had neither. - DATA_UPLOAD_MAX_MEMORY_SIZE on anything that accepts files. - Timeouts on every outbound HTTP call. Naked requests.get() has no place in production code. ```python config/settings.py REST_FRAMEWORK = { "DEFAULT_AUTHENTICATION_CLASSES": ( "rest_framework_simplejwt.authentication.JWTAuthentication", ), "DEFAULT_PAGINATION_CLASS": "config.pagination.StandardPagination", "PAGE_SIZE": 25, "DEFAULT_THROTTLE_RATES": {"anon": "60/min", "user": "600/min"}, "DEFAULT_RENDERER_CLASSES": ("rest_framework.renderers.JSONRenderer",), } ``` ## Index the filters you actually expose Every filter backend field and ordering parameter needs a matching index. On the appointments list, `clinic_id + starts_at DESC` was the plan that mattered — not a lonely index on starts_at. ```sql migrations/0042_appointments_clinic_starts.sql CREATE INDEX CONCURRENTLY appointments_clinic_starts_idx ON appointments (clinic_id, starts_at DESC); EXPLAIN (ANALYZE, BUFFERS) SELECT id, starts_at, doctor_id FROM appointments WHERE clinic_id = 12 ORDER BY starts_at DESC LIMIT 25; ``` > Staging with toy data will lie to you. Query plans under production volume will not. ## What I do the week before launch Run the punch list, capture query counts in CI, and load-test the three endpoints that carry most traffic with a realistic fixture — not 20 rows. That hour of work has saved every DRF service I have shipped from a miserable first week.

Our Django Monolith Had 47 Apps. Renaming Three Fixed More Than Refactoring.

Mon, 06 Jul 2026 09:00:00 GMT

When I joined the inventory-and-procurement module, the repo had 47 Django apps. Twelve were named common, core, shared, utils, or api_v2. A bug in purchase order approval touched seven files across five apps and nobody could tell me which team owned the approval rules. We didn't rewrite the monolith. We renamed three apps, added import-linter, and split fat models into selectors and services over six months. Query count on the PO list page went from 847 to 12. That wasn't architecture magic — it was ownership becoming visible. ## Layer apps were the original sin The project started in 2021 with api/, models/, and tasks/ as separate apps because a consultant said separation of concerns. By 2024 every feature needed a serializer in api, a model change in models, and a Celery task in tasks. Code review couldn't answer "who owns procurement?" because procurement didn't exist as a noun in the tree. ```text erp/ (after restructure) erp/ procurement/ # POs, vendors, approvals models.py selectors.py services.py api/ views.py serializers.py urls.py tasks.py tests/ inventory/ # stock, warehouses, transfers ... finance/ # invoices, GL hooks ... platform/ # auth, orgs, audit — not "core" permissions.py middleware.py pagination.py ``` Renaming core to platform was cosmetic on disk but changed how new engineers talked about the code. "Put it in core" was a junk drawer. "Does this belong in platform or procurement?" forced a decision. ## Selectors and services, not 400-line models PurchaseOrder had 38 methods when I opened models.py. approve(), reject(), recalculate_tax(), send_vendor_email(), and four variants of get_line_items_for_display() all lived on the model. Unit tests mocked the model class itself. We moved reads to selectors and writes to services. The model kept clean(), save() overrides for invariants, and nothing else. ```python procurement/selectors.py from django.db.models import Count, Prefetch, Q, QuerySet from django.utils import timezone from procurement.models import PurchaseOrder, POLineItem def pending_approval_orders(*, org_id: int) -> QuerySet[PurchaseOrder]: return ( PurchaseOrder.objects .filter( organization_id=org_id, status=PurchaseOrder.Status.PENDING_APPROVAL, deleted_at__isnull=True, ) .select_related("vendor", "requested_by") .prefetch_related( Prefetch( "line_items", queryset=POLineItem.objects.select_related("sku"), ) ) .annotate(line_count=Count("line_items")) .order_by("-submitted_at") ) def orders_stuck_over_sla(*, org_id: int, sla_hours: int = 48) -> QuerySet[PurchaseOrder]: cutoff = timezone.now() - timezone.timedelta(hours=sla_hours) return pending_approval_orders(org_id=org_id).filter(submitted_at__lt=cutoff) ``` ```python procurement/services.py from django.db import transaction from django.utils import timezone from audit.platform import log_action from procurement.models import PurchaseOrder from procurement.selectors import pending_approval_orders from procurement.tasks import notify_vendor_po_approved class ApprovalError(Exception): pass @transaction.atomic def approve_purchase_order(*, po: PurchaseOrder, approver_id: int) -> PurchaseOrder: po = PurchaseOrder.objects.select_for_update().get(pk=po.pk) if po.status != PurchaseOrder.Status.PENDING_APPROVAL: raise ApprovalError(f"PO {po.reference} is {po.status}, not pending") if po.total_amount_cents > 5_000_000 and not po.has_budget_allocation: raise ApprovalError("POs over 50,000 AFN require budget allocation") po.status = PurchaseOrder.Status.APPROVED po.approved_by_id = approver_id po.approved_at = timezone.now() po.save(update_fields=["status", "approved_by_id", "approved_at"]) log_action( actor_id=approver_id, verb="approve", target_type="purchase_order", target_id=po.id, ) transaction.on_commit(lambda: notify_vendor_po_approved.delay(po.id)) return po ``` approve_purchase_order is 25 lines and testable with two factory calls. No DRF, no request object. The view became a thin wrapper that loads the PO, calls the service, returns 204. :::note Keyword-only args saved us from a production bug A junior dev swapped approver_id and po in a service call during a refactor. Python happily passed an integer where a model was expected. We added * to every service and selector signature. Two weeks later mypy caught a similar mistake in finance/services.py before merge. ::: ## Cross-app imports were the real dependency graph inventory.models.StockLevel imported procurement.models.PurchaseOrder directly. finance.tasks.sync_invoice imported inventory internals. Circular imports got "fixed" with lazy imports inside functions — 23 of them when I grepped. import-linter in CI blocked cross-app model imports except through selectors and services. ```ini pyproject.toml [tool.importlinter] root_package = "erp" [[tool.importlinter.contracts]] name = "Domain isolation" type = "forbidden" source_modules = ["erp.procurement", "erp.inventory", "erp.finance"] forbidden_modules = [ "erp.procurement.models", "erp.inventory.models", "erp.finance.models", ] ignore_imports = [ "erp.procurement.selectors -> erp.procurement.models", "erp.procurement.services -> erp.procurement.models", "erp.inventory.selectors -> erp.inventory.models", "erp.inventory.services -> erp.inventory.models", ] ``` First CI run failed with 41 violations. We fixed 38 in a sprint and grandfathered three with tickets. Six months later, zero grandfathered. New engineers don't argue about whether they can import a foreign model — the build says no. ## Migrations at 14 engineers We ship 20–30 migrations per week across three squads. Collision on 0047_procurement_alter_whatever happened twice in one month before we enforced makemigrations --check in CI and a Slack rule: if your migration number conflicts, you rebase and regenerate, never edit someone else's merged file. - Additive schema only in production releases — nullable columns first, backfill job, NOT NULL constraint next release. - Squash per app every quarter; our procurement app went from 112 migration files to 4 squashed + delta. - migrate --plan runs against an empty Postgres in CI; caught a circular dependency once. - Renames are two deploys minimum. We learned that renaming vendor_code to supplier_code in one shot caused a 14-minute lock on a 40M-row table. ## Fat serializers were hiding business logic PurchaseOrderSerializer.validate() was 90 lines including approval threshold checks that duplicated the service. We moved validation into the service layer and left the serializer doing shape validation only. API responses got slower briefly because we removed a cached SerializerMethodField hack — then selectors with annotate fixed the N+1 properly. > Your folder structure is a map. Your import graph is the territory. We spent years editing the map while the territory was on fire. ## What stuck after a year Domain apps with selectors/services, import-linter, and keyword-only service signatures. What didn't stick: a shared BaseService class hierarchy nobody used after month two. What I'd do on a greenfield ERP today: three domain apps on day one even if two are empty, import-linter before engineer number two, and a written rule that models over 150 lines get reviewed like schema changes.

mypy on 58k Lines of Django: We Shipped With 1,400 Errors Left

Fri, 19 Jun 2026 09:00:00 GMT

Our publishing platform at Afghan Cosmos — multi-tenant CMS, subscription billing, print queue integration — hit 58,000 lines of Python last year. Zero type hints when I started adding them in March. I ran mypy strict once, got 9,800 errors, and my tech lead asked me to stop blocking the sprint. The version that survived had 1,400 errors in legacy modules we never touch and zero tolerance for new untyped public functions in modules we'd already cleaned. ## django-stubs or don't bother Plain mypy thinks ForeignKey resolves to int | None forever. It doesn't understand that filter(some_field__gte=...) is valid. django-stubs fixes enough of the ORM that errors become actionable. Without the plugin, we saw 400 false positives on models alone. ```ini mypy.ini [mypy] python_version = 3.11 plugins = mypy_django_plugin.main ignore_missing_imports = True warn_unused_ignores = True warn_redundant_casts = True show_error_codes = True [mypy.plugins.django-stubs] django_settings_module = "config.settings" [mypy-*.migrations.*] ignore_errors = True [mypy-tests.*] disallow_untyped_defs = False ``` ## Module allowlist instead of big-bang strict We tagged modules as strict only after someone annotated them and a second reviewer signed off. billing.services, subscriptions.selectors, and print_queue.tasks went strict in month one — about 4,200 lines. Everything else stayed on check_untyped_defs only. ```ini mypy.ini (strict modules) [mypy-subscriptions.services] disallow_untyped_defs = True disallow_incomplete_defs = True warn_return_any = True [mypy-subscriptions.selectors] disallow_untyped_defs = True disallow_incomplete_defs = True [mypy-print_queue.tasks] disallow_untyped_defs = True [mypy-*] disallow_untyped_defs = False check_untyped_defs = True ``` Strict surface grew from 7% to 41% of production code in nine months without a dedicated typing sprint. Modules nobody edits stayed loose. That was intentional. ## CI gate on changed lines only A bash script runs mypy with --pretty and filters output to lines touched in the PR diff. If you modify a function in a loose module, that function needs full annotations. If you add a new file under subscriptions/, it must pass the strict config for that package from day one. ```python scripts/mypy_changed.py #!/usr/bin/env python3 import re import subprocess import sys def changed_lines(base: str = "origin/main") -> dict[str, set[int]]: out = subprocess.check_output( ["git", "diff", f"{base}...HEAD", "-U0", "--", "*.py"], text=True, ) files: dict[str, set[int]] = {} current: str | None = None for line in out.splitlines(): if line.startswith("+++"): current = line[6:].split("\t", 1)[0] files.setdefault(current, set()) elif line.startswith("@@") and current: m = re.search(r"\+(\d+)(?:,(\d+))?", line) if not m: continue start = int(m.group(1)) count = int(m.group(2) or "1") files[current].update(range(start, start + count)) return files def main() -> int: diff = changed_lines() proc = subprocess.run(["mypy", "."], capture_output=True, text=True) failures = [] for raw in proc.stdout.splitlines(): m = re.match(r"^([^:]+):(\d+):", raw) if not m: continue path, line = m.group(1), int(m.group(2)) if path in diff and line in diff[path]: failures.append(raw) if failures: print("\n".join(failures)) return 1 return 0 if __name__ == "__main__": raise SystemExit(main()) ``` First month, engineers complained the script was annoying. Second month, it caught a Optional[str] passed where int was required in a Stripe webhook handler. That bug had been dropping subscription renewals silently for eleven days. ### TypedDict for webhook payloads ```python subscriptions/types.py from typing import Literal, TypedDict class StripeInvoicePaidEvent(TypedDict): id: str type: Literal["invoice.paid"] data: "StripeInvoicePaidData" class StripeInvoicePaidData(TypedDict): object: "StripeInvoiceObject" class StripeInvoiceObject(TypedDict): id: str customer: str subscription: str | None amount_paid: int currency: str def handle_invoice_paid(event: StripeInvoicePaidEvent) -> None: obj = event["data"]["object"] if obj["subscription"] is None: return renew_subscription( stripe_subscription_id=obj["subscription"], amount_cents=obj["amount_paid"], ) ``` We had been using event["data"]["object"]["subscription_id"] in one handler and "subscription" everywhere else. .get() masked it. TypedDict didn't fix runtime — Stripe still sends JSON — but mypy flagged the typo at commit time. ## QuerySet generics and the pain they’re worth ```python subscriptions/selectors.py from django.db.models import QuerySet from subscriptions.models import Subscription def active_subscriptions(*, tenant_id: int) -> QuerySet[Subscription]: return ( Subscription.objects .filter(tenant_id=tenant_id, status=Subscription.Status.ACTIVE) .select_related("plan") .order_by("current_period_end") ) def subscription_ids_due_for_renewal(*, within_hours: int = 24) -> list[int]: cutoff = timezone.now() + timezone.timedelta(hours=within_hours) return list( Subscription.objects .filter(status=Subscription.Status.ACTIVE, current_period_end__lte=cutoff) .values_list("id", flat=True) ) ``` QuerySet[Subscription] caught a refactor where someone returned Subscription.objects.filter(...) | OtherModel.objects.filter(...) — union of incompatible querysets. Rare bug, but the kind that only shows up in production when a tenant has odd data. :::note pyright in VS Code, mypy in CI Pyright gives feedback in 200–400 ms on save. mypy with django-stubs takes 45 s on the full tree. Engineers keep pyright enabled locally; CI runs mypy because plugin support for Django is better and we trust it for merge gates. ::: ## Mistakes I made - Typed 200 test fixtures before production code — wasted two weeks, reverted most of it. - Used cast() everywhere instead of fixing model definitions. warn_redundant_casts exposed 60 useless casts. - Blocked merges on global error count. Team bypassed CI with # noqa spam. Switched to changed-lines gate. - Ignored pyright's reportGeneralTypeIssues on third-party libs and missed a wrong kwarg to celery.signals. > Gradual typing without a ratchet is just typing cosplay. The ratchet was changed-lines in CI, not a roadmap slide. ## Current state 1,400 errors remain, mostly in a legacy content import package from 2020. subscriptions and print_queue are strict. New modules start strict. I won't pretend we'll hit zero errors — I will pretend we'll never add an untyped public function in a strict module again. That part's held for four months.

A Celery Task Ran Twice and Double-Charged a Clinic

Wed, 03 Jun 2026 09:00:00 GMT

Last spring a worker OOM-killed mid-task during a deploy. Celery redelivered charge_invoice. Stripe got two POSTs with different idempotency keys because we generated the key inside the task with uuid4(). A clinic in Herat got billed twice for the same monthly subscription — 1,200 AFN, small amount, huge trust problem. Refund took ten minutes. Explaining why took two days. Every background job design decision I make now starts with "what happens if this runs twice?" ## Celery vs RQ vs Postgres — what we actually run Three codebases, three choices. The publishing platform uses Celery 5.3 with Redis broker, 6 worker processes, 4 queues by priority. A small internal tools app uses RQ with one Redis instance and 12 task types — fine for 200 jobs/day. The healthcare billing service uses Celery for fanout but enqueues payment-critical jobs through a Postgres outbox table in the same transaction as the Invoice row. ```python billing/outbox.py from django.db import transaction from django.utils import timezone from billing.models import Invoice, OutboxJob @transaction.atomic def finalize_invoice_and_enqueue_charge(*, invoice_id: int) -> OutboxJob: invoice = Invoice.objects.select_for_update().get(pk=invoice_id) if invoice.status != Invoice.Status.FINALIZED: raise ValueError(f"invoice {invoice_id} is {invoice.status}") idempotency_key = f"charge:{invoice.id}:v{invoice.revision}" job, created = OutboxJob.objects.get_or_create( idempotency_key=idempotency_key, defaults={ "task_name": "billing.tasks.charge_invoice", "payload": {"invoice_id": invoice.id}, "status": OutboxJob.Status.PENDING, }, ) if not created and job.status == OutboxJob.Status.COMPLETED: return job invoice.status = Invoice.Status.PENDING_CHARGE invoice.save(update_fields=["status"]) return job ``` If the transaction rolls back, neither the status change nor the outbox row survives. If Redis is down, the outbox row still exists — a separate poller process claims it with SELECT FOR UPDATE SKIP LOCKED and pushes to Celery. We added that after a Redis failover lost 23 enqueued jobs in 2024. ```python billing/outbox_worker.py from django.db import transaction from django.utils import timezone from datetime import timedelta from billing.models import OutboxJob from billing.tasks import charge_invoice def claim_and_dispatch(limit: int = 50) -> int: dispatched = 0 for _ in range(limit): with transaction.atomic(): job = ( OutboxJob.objects .select_for_update(skip_locked=True) .filter(status=OutboxJob.Status.PENDING) .order_by("created_at") .first() ) if job is None: break job.status = OutboxJob.Status.DISPATCHED job.dispatched_at = timezone.now() job.save(update_fields=["status", "dispatched_at"]) charge_invoice.apply_async( kwargs=job.payload, task_id=job.idempotency_key, ) dispatched += 1 return dispatched ``` ## Idempotency inside the task Outbox dedup stops duplicate enqueue. It doesn't stop redelivery after the worker crashes post-Stripe-call but pre-save. ChargeAttempt table with a unique idempotency_key handles that. ```python billing/tasks.py import stripe from celery import shared_task from django.db import transaction from billing.models import ChargeAttempt, Invoice from billing.stripe_client import get_stripe from jobs.backoff import backoff_with_jitter @shared_task( bind=True, max_retries=6, acks_late=True, reject_on_worker_lost=True, ) def charge_invoice(self, invoice_id: int) -> None: invoice = Invoice.objects.get(pk=invoice_id) idempotency_key = f"charge:{invoice.id}:v{invoice.revision}" with transaction.atomic(): attempt, created = ChargeAttempt.objects.get_or_create( idempotency_key=idempotency_key, defaults={ "invoice_id": invoice.id, "amount_cents": invoice.total_cents, "status": ChargeAttempt.Status.PENDING, }, ) if attempt.status == ChargeAttempt.Status.SUCCEEDED: return stripe_client = get_stripe() try: charge = stripe_client.charges.create( amount=attempt.amount_cents, currency="usd", customer=invoice.stripe_customer_id, idempotency_key=idempotency_key, ) except stripe.error.CardError as exc: attempt.status = ChargeAttempt.Status.FAILED attempt.last_error = str(exc) attempt.save(update_fields=["status", "last_error"]) return except stripe.error.StripeError as exc: raise self.retry(exc=exc, countdown=backoff_with_jitter(self.request.retries)) attempt.status = ChargeAttempt.Status.SUCCEEDED attempt.stripe_charge_id = charge.id attempt.save(update_fields=["status", "stripe_charge_id"]) invoice.status = Invoice.Status.PAID invoice.paid_at = timezone.now() invoice.save(update_fields=["status", "paid_at"]) ``` :::note Derive the key upstream, never inside the task charge:{invoice_id}:v{revision} survives retries, redeliveries, and deploys. uuid4() inside the task is how we double-charged a clinic. I still see this pattern in PRs monthly. ::: ## RQ when Celery is overkill Internal report generator: 8 tasks, no chains, no priorities. RQ on Redis, django-rq for admin integration. Worker startup is one command. We outgrew it when we needed separate queues so thumbnail generation couldn't starve password-reset emails — migrated to Celery in a week, kept the same Redis instance with different db numbers. ```python config/celery.py from celery import Celery app = Celery("erp") app.config_from_object("django.conf:settings", namespace="CELERY") app.autodiscover_tasks() # settings.py excerpt CELERY_TASK_ROUTES = { "billing.tasks.*": {"queue": "billing"}, "notifications.tasks.*": {"queue": "notifications"}, "reports.tasks.generate_pdf": {"queue": "heavy"}, } CELERY_TASK_ACKS_LATE = True CELERY_WORKER_PREFETCH_MULTIPLIER = 1 CELERY_TASK_DEFAULT_QUEUE = "default" ``` CELERY_WORKER_PREFETCH_MULTIPLIER = 1 cost us throughput on bulk email sends — workers fetched one task at a time. For billing and payment tasks that's correct. For newsletter fanout we override prefetch to 4 on dedicated workers. ## Retries, jitter, and dead letters ```python jobs/backoff.py import random def backoff_with_jitter(retry_count: int, base: float = 2.0, cap: float = 600.0) -> float: """Seconds to wait before retry. retry_count is Celery self.request.retries.""" ceiling = min(cap, base * (2 ** retry_count)) return random.uniform(base, ceiling) ``` During a Stripe outage in February, 3,400 tasks failed within four minutes. Fixed-delay retry would have hammered Stripe again at T+60s. Jitter spread retries over 90–600 seconds. Still ugly, but recovery didn't extend the outage. ```python billing/tasks.py (dead letter) @shared_task(bind=True, max_retries=6) def charge_invoice(self, invoice_id: int) -> None: try: _charge_invoice_impl(invoice_id) except NonRetryableBillingError as exc: DeadLetter.objects.create( task_name="charge_invoice", payload={"invoice_id": invoice_id}, error=str(exc)[:2000], ) return except stripe.error.StripeError as exc: if self.request.retries >= self.max_retries: DeadLetter.objects.create( task_name="charge_invoice", payload={"invoice_id": invoice_id}, error=str(exc)[:2000], ) return raise self.retry(exc=exc, countdown=backoff_with_jitter(self.request.retries)) ``` DeadLetter rows get a Slack alert at >5 per hour. A human reviews before bulk retry. We learned not to auto-retry dead letters after a bad deploy replayed 200 malformed payloads and filled logs for a day. ## Metrics that caught real problems - task_age_seconds — time from enqueue to worker start. Grew to 45 min during a thumbnail backlog while success rate stayed 99%. - attempts histogram per task name. charge_invoice with attempts>1 is normal; attempts>4 gets investigated. - Trace ID in task kwargs, propagated to Stripe metadata and structlog context. - Queue depth per queue name, not aggregate — billing queue at 200 is an incident; default at 200 might be fine. > Brokers give you at-least-once delivery. Your code has to give you at-most-once side effects. Everything else is configuration theater. ## What I require before a task ships Written idempotency key formula. Explicit behavior on duplicate run (no-op vs error). Classification of failures into retry vs dead-letter. For money movement, provider-level idempotency too. charge_invoice wouldn't have double-charged if we'd had half of that checklist in place. It does now.

How our React app stopped refetching the same patient six times

Tue, 02 Jun 2026 09:00:00 GMT

The patient chart screen opened five child widgets. Each one called `/patients/:id`. Five network tabs, five loading states, five chances to disagree about the response shape. We did not need a new state library. We needed one place that owned the network. ## Query keys are a public API String keys scattered through components made invalidation a treasure hunt. We put keys in one module and treated them like routes — rename carefully, never invent a new string in a component. ```ts src/api/keys.ts export const queryKeys = { patients: { all: ["patients"] as const, list: (filters: PatientFilters) => [...queryKeys.patients.all, "list", filters] as const, detail: (id: string) => [...queryKeys.patients.all, "detail", id] as const, }, appointments: { day: (clinicId: string, day: string) => ["appointments", clinicId, day] as const, }, } as const; ``` ## One hook per resource Components import hooks, never the HTTP client. If a screen imports `http`, the boundary has leaked and we will be debugging duplicate fetches again within a month. ```tsx src/api/patients.ts import { useMutation, useQuery, useQueryClient } from "@tanstack/react-query"; import { http } from "./http"; import { queryKeys } from "./keys"; import { parsePatient } from "./schemas"; export function usePatient(id: string) { return useQuery({ queryKey: queryKeys.patients.detail(id), queryFn: async ({ signal }) => { const raw = await http.get(`/patients/${id}`, { signal }); return parsePatient(raw); }, staleTime: 30_000, }); } export function useUpdatePatient() { const qc = useQueryClient(); return useMutation({ mutationFn: (input: UpdatePatientInput) => http.patch(`/patients/${input.id}`, input), onSuccess: (_data, input) => { qc.invalidateQueries({ queryKey: queryKeys.patients.detail(input.id) }); qc.invalidateQueries({ queryKey: queryKeys.patients.all }); }, }); } ``` :::note Validate at the edge Zod (or similar) parses the response once in the queryFn. Screens trust the type. When DRF renames a field, one schema fails loudly instead of six widgets silently showing blanks. ::: ## What stayed out of Redux 1. Server data lives in TanStack Query — patients, appointments, invoices. 2. UI-only state stays local or in a small Zustand store — sidebar open, selected tab. 3. Error and loading UI sit at the route boundary, not inside every card. 4. Optimistic updates only where the user can see the lag (archive, status toggle). That split carried the clinic MIS from a handful of screens to well over sixty without a data-layer rewrite. The rule is boring on purpose: one cache, one key module, no fetch in the leaf components.

I stopped putting API data in Redux

Mon, 18 May 2026 09:00:00 GMT

The bug report said the patient header showed one blood type and the lab panel showed another. Same visit, same patient ID, two screens mounted at once. Redux had the header data from a fetch on login. The lab panel had its own useEffect that hit /api/visits/{id}/ and wrote into component state. Someone had PATCHed the record in a third tab, and neither cache knew about it. We spent a week adding manual refresh buttons before I admitted the problem wasn't stale data — it was three separate caches pretending to be one source of truth. ## Server state and client state are different animals Client state is yours: which drawer is open, the current step in a wizard, a textarea before submit. Server state is a copy of something that lives in Postgres and can change while your user is mid-form. Redux and Zustand are fine for the first. They are a bad fit for the second because you end up hand-rolling loading flags, deduplication, background refetch, and invalidation that TanStack Query ships on day one. :::note How I tell them apart If another tab, another user, or a Celery task can change the value without your React tree doing anything, it is server state. Put it in TanStack Query. If only local interaction changes it, useState or a small Zustand slice is enough. ::: ## Query keys are a schema, not string soup The cache bug that took longest to find was a typo: one file used ['patient', id] and another used ['patients', id]. Invalidation after a mutation cleared the list but left the detail view stale for hours until someone hard-refreshed. I now treat query keys like URL paths — a factory per resource, typed with as const, so invalidation targets are obvious and grep-able. ```typescript src/features/patients/queries.ts export const patientKeys = { all: ["patients"] as const, lists: () => [...patientKeys.all, "list"] as const, list: (wardId: string, filters: PatientFilters) => [...patientKeys.lists(), wardId, filters] as const, details: () => [...patientKeys.all, "detail"] as const, detail: (id: string) => [...patientKeys.details(), id] as const, }; export function usePatient(id: string) { return useQuery({ queryKey: patientKeys.detail(id), queryFn: () => api.get(`/api/patients/${id}/`), enabled: Boolean(id), staleTime: 60_000, }); } ``` Ward-scoped lists matter in a hospital MIS. A nurse on Ward 3 invalidating patientKeys.lists() should not blow away Ward 7's cache. The factory makes that explicit: list(wardId, filters) is part of the key, not a hidden query param someone forgets to include. ### Invalidation with a blast radius The lazy fix after any mutation is queryClient.invalidateQueries() with no key. On our ERP dashboard that refetched twelve widgets because everything was still mounted. I use three tiers now depending on how wide the change spreads. - Narrow: setQueryData on the detail key when the PATCH response already returns the full record — no round trip. - Medium: invalidateQueries on patientKeys.lists() plus the one detail key when a single row changed. - Wide: invalidate everything only for bulk imports, role changes, or anything that could touch unrelated queries. ```typescript src/features/patients/mutations.ts export function useUpdatePatient() { const queryClient = useQueryClient(); return useMutation({ mutationFn: (payload: UpdatePatientPayload) => api.patch(`/api/patients/${payload.id}/`, payload), onSuccess: (updated) => { queryClient.setQueryData(patientKeys.detail(updated.id), updated); queryClient.invalidateQueries({ queryKey: patientKeys.lists() }); }, }); } ``` ## Optimistic updates and the race I lost a day to We added optimistic archive on invoice rows in an e-commerce admin. Click archive, row greys out instantly, feels great. Except sometimes it un-archived half a second later with no error. The table had refetchInterval: 5000. An in-flight background refetch landed after our optimistic setQueryData and overwrote it with stale data. cancelQueries in onMutate fixed it. I had never seen that race in dev because local latency is too fast to overlap. ```typescript src/features/invoices/useArchiveInvoice.ts export function useArchiveInvoice() { const queryClient = useQueryClient(); return useMutation({ mutationFn: (id: string) => api.post(`/api/invoices/${id}/archive/`), onMutate: async (id) => { await queryClient.cancelQueries({ queryKey: invoiceKeys.detail(id) }); const previous = queryClient.getQueryData(invoiceKeys.detail(id)); queryClient.setQueryData(invoiceKeys.detail(id), (old) => old ? { ...old, archived: true } : old ); return { previous, id }; }, onError: (_err, _id, ctx) => { if (ctx?.previous) { queryClient.setQueryData(invoiceKeys.detail(ctx.id), ctx.previous); } toast.error("Archive failed — reverted."); }, onSettled: (_data, _err, id) => { queryClient.invalidateQueries({ queryKey: invoiceKeys.detail(id) }); }, }); } ``` > An optimistic UI that rolls back silently is worse than a spinner. The user already acted on the lie. ## Where I still use a global store Theme, sidebar collapse, and a multi-step purchase-order draft before the final POST still live in Zustand. Auth bootstrap flags resolved once at app load sit in context. The rule is simple: if it came from an API and can go stale, it does not enter the store. Mixing both in Redux is how you get two blood types on one screen. ## What I actually do on new projects - TanStack Query for every remote read and write; Redux only if the team already has it for genuine client-only state. - Query key factories per domain module, checked in code review like API routes. - setQueryData when the mutation response is authoritative; invalidate when it is not. - cancelQueries before every optimistic write when background refetch or polling is enabled. - Document staleTime per resource — vitals panels can tolerate 60s; open encounter notes cannot.

Our clinic intake form had 31 props

Wed, 22 Apr 2026 09:00:00 GMT

I opened PatientIntakeSection.tsx and counted the props interface: thirty-one fields, half of them optional booleans. showGuardianFields, guardianRequired, showInsuranceBlock, insuranceReadOnly, collapseVitals, vitalsDefaultOpen. Every new ward request added another flag. The render function was four hundred lines of nested ternaries. Call sites passed seven props they did not understand just to turn off a block they did not need. That is props explosion — one component trying to be every layout instead of pieces you assemble. ## Composition beats configuration The refactor did not start with context or a state library. We split the section into IntakeSection, IntakeSection.Header, IntakeSection.Body, and small domain blocks — DemographicsBlock, GuardianBlock, InsuranceBlock — that the ward-specific page composed explicitly. ```tsx src/features/intake/PediatricWardIntake.tsx export function PediatricWardIntake({ visitId }: { visitId: string }) { return ( ); } export function EmergencyIntake({ visitId }: { visitId: string }) { return ( ); } ``` Emergency intake dropped guardian and insurance at the JSX level, not via showGuardianFields={false}. New ward layouts stop asking the shared component to grow another boolean. ## Compound components for shared coordination The intake sections still needed shared state: which block had validation errors, whether the section was submitting, visitId for child blocks. That belongs in a provider scoped to IntakeSection, not in props drilled through six levels. ```tsx src/features/intake/IntakeSection.tsx const IntakeContext = createContext<{ visitId: string; isSubmitting: boolean; registerError: (blockId: string) => void; } | null>(null); function useIntakeSection() { const ctx = useContext(IntakeContext); if (!ctx) throw new Error("IntakeSection.* must render inside "); return ctx; } export function IntakeSection({ visitId, children }: Props) { const [isSubmitting, setIsSubmitting] = useState(false); const value = useMemo( () => ({ visitId, isSubmitting, registerError: () => {} }), [visitId, isSubmitting] ); return {children}; } IntakeSection.Header = function Header({ title }: { title: string }) { const { isSubmitting } = useIntakeSection(); return (

{title}

{isSubmitting && Saving…}

); }; ``` Throwing when useIntakeSection runs outside the provider is intentional. Mis-wired compound components fail at the call site in dev, not as undefined three files deep. ## Headless hooks when markup diverges Ward UIs looked nothing alike but shared behavior: expand/collapse, dirty tracking, keyboard focus order. We pulled that into hooks with no JSX. useCollapsible, useBlockDirtyState. Presentation stayed in each ward's TSX; behavior stayed testable without rendering a four-hundred-line god component. ```tsx src/hooks/useCollapsible.ts export function useCollapsible(initial = false) { const [open, setOpen] = useState(initial); const toggle = useCallback(() => setOpen((v) => !v), []); const panelProps = { hidden: !open, id: useId(), }; const triggerProps = { "aria-expanded": open, onClick: toggle, }; return { open, toggle, panelProps, triggerProps }; } ``` ## Context is not the default escape hatch After the intake refactor someone suggested a single AppFormContext for all forms in the MIS. I pushed back. Context re-renders every consumer when the value reference changes. A search string in context that fifty table rows read repaints the whole list every keystroke. We kept intake context scoped to IntakeSection's subtree. Auth user and theme stay in separate providers split by change frequency. :::note Before I add context How often does this value change, and how many components read it? Theme toggled once per session across the whole app — fine. Live filter text consumed by every row — use a store with selectors or pass props two levels, not app-wide context. ::: - Prop drilling three levels is cheaper than tracing mystery context. - Split bloated providers: AuthContext, ThemeContext, ConnectionContext — not one AppContext that updates websocket status and repaints the tree. - Compound component context is scoped to one feature subtree, not exported as a global pattern. - If only one call site needs a layout, inline the JSX there instead of adding another boolean to a shared component. > A healthy component API grows in parts you can combine, not in flags you have to look up in a spreadsheet. ## What changed after the split - New ward layouts are new page files composing existing blocks, not PRs to a shared boolean matrix. - IntakeSection.tsx went from 400 lines to about 80; domain blocks own their validation display. - Code review catches showX props on shared components — that is a signal to extract a block instead. - Scoped context replaced twelve props that existed only to pass visitId and submit state downward.

One compose file that matches what we run in production

Tue, 21 Apr 2026 09:00:00 GMT

A new hire should clone the repo, copy `.env.example`, run one command, and hit a working stack. If that is not true, we are not containerized — we are collecting Dockerfiles. The goal is identical behaviour from laptop to CI to the VPS, not novelty. ## Backend image that does not ship the build toolchain Install deps in a builder stage, copy site-packages into a slim runtime, run as non-root. A one-line view change must not reinstall every wheel. ```dockerfile backend/Dockerfile FROM python:3.12-slim AS base ENV PYTHONDONTWRITEBYTECODE=1 PYTHONUNBUFFERED=1 FROM base AS deps WORKDIR /app COPY requirements.txt . RUN pip install --no-cache-dir -r requirements.txt FROM base AS runtime WORKDIR /app RUN useradd --create-home --uid 1000 app COPY --from=deps /usr/local/lib/python3.12/site-packages /usr/local/lib/python3.12/site-packages COPY --from=deps /usr/local/bin /usr/local/bin COPY . . RUN python manage.py collectstatic --noinput \ && chown -R app:app /app USER app CMD ["gunicorn", "config.wsgi:application", "--bind", "0.0.0.0:8000", "--workers", "3"] ``` ## Compose for the whole product ```yaml docker-compose.yml services: db: image: postgres:16-alpine environment: POSTGRES_DB: app POSTGRES_PASSWORD: ${POSTGRES_PASSWORD} volumes: ["pgdata:/var/lib/postgresql/data"] healthcheck: test: ["CMD-SHELL", "pg_isready -U postgres"] interval: 5s retries: 10 api: build: ./backend env_file: .env depends_on: db: { condition: service_healthy } ports: ["8000:8000"] web: build: ./frontend depends_on: [api] ports: ["5173:80"] volumes: pgdata: ``` :::note Layer caching Copy requirements.txt / package-lock.json before the rest of the source. Otherwise every commit busts the dependency layer and CI times go from 3 minutes to 12. ::: ## CI runs the same images ```bash .github/workflows/ci.sh docker compose -f docker-compose.yml -f docker-compose.ci.yml build docker compose run --rm api python manage.py migrate --check docker compose run --rm api pytest -q --maxfail=1 docker compose run --rm web npm run build ``` When local, CI, and production share the same Dockerfiles, the class of bug that starts with "but it worked on my laptop" mostly disappears. That is the whole point.

DRF said billing_address, React expected billingAddress

Fri, 03 Apr 2026 09:00:00 GMT

Support ticket: user submits clinic registration, gets a generic toast, form looks clean. Network tab shows 400 with { "billing_address": ["This field is required."] }. Our React field is billingAddress.line1. mapServerErrors existed but only handled top-level keys — nested DRF errors never reached setError. The user clicked Submit four times because nothing pointed at the empty address line. That bug cost us more support time than the whole form took to build. ## Zod schema first, serializer second I read the DRF serializer before writing the Zod schema. The serializer is what Postgres and business rules enforce. Client validation is a courtesy for typing speed, not a security boundary. The pattern is standard: zodResolver, z.infer for types, one schema object per form. ```tsx src/features/clients/ClientRegistrationForm.tsx const clientSchema = z.object({ name: z.string().min(2).max(100), email: z.string().email(), phone: z .string() .regex(/^\+?[0-9]{7,15}$/) .optional() .or(z.literal("")), billingAddress: z.object({ line1: z.string().min(1, "Address is required"), city: z.string().min(1, "City is required"), postalCode: z.string().min(3, "Postal code is too short"), }), }); type ClientFormValues = z.infer; const form = useForm({ resolver: zodResolver(clientSchema), mode: "onBlur", defaultValues: { name: "", email: "", phone: "", billingAddress: { line1: "", city: "", postalCode: "" }, }, }); ``` mode: onBlur, not onChange. Validating every keystroke flashes errors while someone is still typing an email address. Screen readers announce each premature error. onBlur waits until they leave the field. ```python clients/serializers.py class ClientSerializer(serializers.ModelSerializer): phone = serializers.RegexField( regex=r"^\+?[0-9]{7,15}$", required=False, allow_blank=True ) class Meta: model = Client fields = ["id", "name", "email", "phone", "billing_address"] extra_kwargs = { "name": {"min_length": 2, "max_length": 100}, } ``` ## Catching drift before production Backend added emergency_contact as required. Frontend schema did not. Users passed client validation and hit a 400 with no field mapping. I do not generate Zod from DRF — that tooling never paid off on my teams. A blunt CI script comparing field names, required flags, and max_length from a dev schema endpoint catches the incidents that actually hurt. ```typescript scripts/check-client-schema-drift.ts type BackendField = { required: boolean; max_length?: number }; async function main() { const backend: Record = await fetch( "http://localhost:8000/api/clients/schema/" ).then((r) => r.json()); const mismatches: string[] = []; for (const [field, meta] of Object.entries(backend)) { if (field === "id") continue; const camel = field.replace(/_([a-z])/g, (_, c) => c.toUpperCase()); if (!(camel in clientSchema.shape)) { mismatches.push(`Backend field "${field}" missing from Zod schema`); } } if (mismatches.length) { console.error(mismatches.join("\n")); process.exit(1); } } ``` :::note What the drift check does not do It does not prove regexes match or that nested object shapes align. It catches missing required fields and renamed columns — the ones that become support tickets. ::: ## Mapping DRF errors onto RHF fields DRF returns snake_case. Our form names are camelCase. Nested keys arrive as billing_address or billing_address.postal_code depending on how the serializer reports errors. The mapper has to walk both shapes and call setError with dot paths React Hook Form understands. ```typescript src/lib/mapDrfErrors.ts type DrfErrors = Record; function toCamelPath(key: string): string { return key.replace(/_([a-z])/g, (_, c) => c.toUpperCase()); } export function mapDrfErrors( errors: DrfErrors, setError: UseFormSetError, prefix = "" ) { for (const [key, value] of Object.entries(errors)) { if (key === "non_field_errors" && Array.isArray(value)) { setError("root.serverError" as Path, { message: value[0] }); continue; } const segment = toCamelPath(key); const path = prefix ? `${prefix}.${segment}` : segment; if (Array.isArray(value)) { setError(path as Path, { type: "server", message: value[0] }); continue; } mapDrfErrors(value as DrfErrors, setError, path); } } ``` Wire it once in the submit handler. On 400, parse response.data, call mapDrfErrors, then setFocus on the first key in formState.errors so keyboard users land on the problem field. ```tsx src/features/clients/ClientRegistrationForm.tsx async function handleSubmit(values: ClientFormValues) { try { await api.post("/api/clients/", toSnakePayload(values)); } catch (err) { if (isAxiosError(err) && err.response?.status === 400) { mapDrfErrors(err.response.data, form.setError); const first = Object.keys(form.formState.errors)[0]; if (first) form.setFocus(first as Path); return; } throw err; } } ``` ## Accessibility wiring I audit on every form PR - aria-invalid tied per field, not a global form-has-errors flag. - aria-describedby pointing at a stable error element id. - role="alert" on error text so announcements fire without refocus. - noValidate on the form element — we validate deliberately, not with browser defaults that fight Zod messages. - Focus first invalid field on submit failure via setFocus in the onInvalid callback. > Client validation saves round trips. Server validation is the contract. They should describe the same contract, not two guesses that matched last sprint. ## Checklist I reuse across MIS and e-commerce forms - Write Zod after reading the serializer, not from memory. - Run a drift script in CI for field names and required flags. - mapDrfErrors handles nested snake_case and non_field_errors. - toSnakePayload on submit, toCamel on errors — same bridge both directions. - onBlur validation, field-level ARIA, setFocus on failure.

The useEffect that doubled every keystroke

Sat, 14 Mar 2026 09:00:00 GMT

Operations complained the order search box felt sticky. React DevTools Profiler showed two commits per keystroke on a table with 3,200 rows. The filter lived in useEffect: setFiltered when orders or search changed. First render still had the old filter result; the effect ran; second render painted the right rows. Removing the effect and deriving filtered during render cut input-to-paint from about 68ms to 31ms on the same laptop. No virtualisation change, no library swap — just stopped fighting React's render model. ## useEffect is for synchronizing outside React If you can compute it from props and state you already have, compute it during render. Effects are for subscriptions, DOM APIs, and network calls whose result React cannot derive. Filtering a list is not synchronization — it is a pure function of orders and search. ```tsx OrderList.before.tsx function OrderList({ orders, search }: { orders: Order[]; search: string }) { const [filtered, setFiltered] = useState(orders); useEffect(() => { setFiltered( orders.filter((o) => o.customerName.toLowerCase().includes(search.toLowerCase()) ) ); }, [orders, search]); return ; } ``` ```tsx OrderList.after.tsx function OrderList({ orders, search }: { orders: Order[]; search: string }) { const filtered = useMemo( () => orders.filter((o) => o.customerName.toLowerCase().includes(search.toLowerCase()) ), [orders, search] ); return ; } ``` :::note When useMemo is worth it For twenty rows, plain const filtered = orders.filter(...) is fine. At thousands of rows on a hot path, memoize the filter so unrelated parent re-renders do not re-scan the array. ::: ## Derived state stored in useState Same ERP module: selectedOrder lived in useState, synced from selectedId and orders via useEffect. When a background poll replaced orders, selectedOrder sometimes pointed at a row that no longer existed — the UI showed stale line items until you clicked another row. Fix: store selectedId only; derive selectedOrder = orders.find(...) during render. ```tsx OrderTable.fixed.tsx function OrderTable({ orders }: { orders: Order[] }) { const [selectedId, setSelectedId] = useState(null); const selectedOrder = orders.find((o) => o.id === selectedId) ?? null; return ( <> {selectedOrder && } ); } ``` Every useState that mirrors something computable is a future desync bug. Store the minimum fact — the id, the search string — and derive the rest. ## React.memo defeated by inline props We wrapped OrderRow in React.memo and saw zero improvement on sidebar toggles. The parent passed style={{ padding: 8 }} and onClick={() => onSelect(order.id)} inline. New object and function every render; memo comparison always fails. Profiling a parent re-render went from repainting all 3,200 rows in about 190ms to repainting zero once callbacks and styles were stable. ```tsx OrderRow.stable.tsx const rowPadding = { padding: 8 }; const OrderRow = React.memo(function OrderRow({ order, onSelect, }: { order: Order; onSelect: (id: string) => void; }) { const handleClick = useCallback(() => onSelect(order.id), [onSelect, order.id]); return ( {order.customerName} {order.total} ); }); function OrderTableRows({ orders, onSelect }: Props) { return ( {orders.map((order) => ( ))} ); } ``` I do not memo everything by default. I profile first, then stabilise props at the boundary that actually re-renders expensive children. Blanket useCallback often adds noise with no measurable win. ## Suspense boundaries that blanked the whole dashboard Our clinic analytics page wrapped RevenueSummary, RecentAdmissions, and a slow chart in one Suspense with a full-page spinner. Changing the date range on the chart suspended everything — including widgets that did not depend on the range. Users lost scroll position and saw a flash of empty layout on every filter change. ```tsx Dashboard.suspense.tsx function ClinicDashboard() { return ( <> }> }> }> ); } ``` Each widget suspends independently. Fast metrics stay visible while the chart loads. For range changes that trigger a refetch, wrap the state update in startTransition so isPending dims the chart but keeps the previous series on screen instead of unmounting to the fallback. ```tsx WardOccupancyChart.tsx function WardOccupancyChart() { const [range, setRange] = useState(defaultRange); const [isPending, startTransition] = useTransition(); const onRangeChange = (next: DateRange) => { startTransition(() => setRange(next)); }; return (

); } ``` > Every rendering fix that shipped in production started in the Profiler, not in a blog post about memoization. ## What I check when a screen feels slow - Derived values in useEffect causing double commits — move to render or useMemo. - useState mirroring props or lookup results — store the id, derive the object. - React.memo on children receiving inline objects or arrow functions — stabilise or remove the memo. - One Suspense boundary around unrelated async trees — split per widget. - Refetch-triggered state updates without useTransition — stale UI beats a full fallback flash.

Stop Guessing Where the Milliseconds Go

Wed, 11 Mar 2026 09:00:00 GMT

We spent most of 2024 arguing about whether the app was slow. Backend said the API was fine — p95 around 180ms on /api/dashboard/summary/. Frontend said the dashboard felt sticky on real devices. Both sides had Grafana screenshots. Both sides were technically right, which is the worst kind of argument to lose. What broke the stalemate wasn't another optimization sprint. It was agreeing on a latency budget in actual milliseconds, then tracing one user session end to end until the numbers stopped contradicting each other. Ours ended up being: 250ms p95 server-side for the hydration API, 2.2s p75 LCP on the dashboard route, 200ms p75 INP on the filter controls. Vague goals like 'make it faster' disappeared once those three numbers existed. ## The server was never the problem I added OpenTelemetry spans around the Django view, each ORM query, Redis calls, and the JSON serialization step. The view itself averaged 94ms. Redis permission lookup: 11ms. Postgres: 38ms across four queries. Total server time well inside budget. Then I looked at the browser waterfall for the same session and felt stupid. TTFB was 210ms — fine. Then 680ms of render-blocking CSS from a third-party component library we imported globally. Then a 140KB analytics bundle executing synchronously before React even mounted. By the time our 'fast' API response arrived, the main thread had already been busy for over a second. No amount of query tuning was going to fix that. ```python core/middleware/timing.py import time from opentelemetry import trace tracer = trace.get_tracer("app.http") class RequestTimingMiddleware: BUDGET_MS = 250 def __init__(self, get_response): self.get_response = get_response def __call__(self, request): t0 = time.perf_counter() with tracer.start_as_current_span(f"{request.method} {request.path}") as span: response = self.get_response(request) elapsed = (time.perf_counter() - t0) * 1000 span.set_attribute("duration_ms", round(elapsed, 1)) span.set_attribute("query_count", len(getattr(request, "_query_count", []))) if elapsed > self.BUDGET_MS: span.add_event("budget_exceeded", {"budget_ms": self.BUDGET_MS}) response["Server-Timing"] = f"django;dur={elapsed:.1f}" return response ``` Server-Timing is boring and it works. Drop it on every response, open DevTools on any slow page, and you immediately know whether to walk toward the database or toward the bundle. We stopped filing cross-team tickets with screenshots of unrelated metrics. ## RUM vs Lighthouse — you need both, hate both equally Lighthouse CI on staging gave us a stable 1.4s LCP on the dashboard. Reproducible, good for catching regressions in PRs. Completely useless for understanding why production users on a Galaxy A14 in Jakarta saw 4.1s. We ship web-vitals to our own collector — not because I love running another service, but because CrUX data is too coarse for debugging a specific route. ```typescript src/lib/rum.ts import { onINP, onLCP, onTTFB, type Metric } from "web-vitals"; function report(metric: Metric) { const payload = { name: metric.name, value: Math.round(metric.name === "CLS" ? metric.value * 1000 : metric.value), rating: metric.rating, path: location.pathname, effectiveType: (navigator as Navigator & { connection?: { effectiveType?: string } }) .connection?.effectiveType ?? "unknown", deviceMemory: (navigator as Navigator & { deviceMemory?: number }).deviceMemory ?? null, }; navigator.sendBeacon("/api/rum/", JSON.stringify(payload)); } onLCP(report); onINP(report); onTTFB(report); ``` Segmenting by effectiveType was the ah-ha moment. 31% of dashboard sessions reported '3g' or 'slow-2g'. Their LCP p75 was 3.8s. Datacenter Lighthouse runs on gigabit fiber will never show you that split. Synthetic catches your regressions. RUM tells you who you're regressing against. :::note What actually changed prioritization I joined RUM sessions to trial signup events by session ID. Sessions with poor LCP (>4s) converted at 2.9%. Good LCP (<2.5s) converted at 6.2%. That gap got us two sprints of perf work that three months of 'it feels slow' never did. ::: ## Enforcing the budget so it outlives the postmortem Budgets written in Notion die in Notion. Ours live in CI and alerting. Lighthouse CI fails if perf score drops more than 4 points on /dashboard, /projects, and /billing. Datadog fires if API p95 on the hydration endpoints crosses 250ms for ten minutes. Not glamorous. Effective. ```yaml .github/workflows/lighthouse.yml name: lighthouse-budget on: pull_request: paths: - "frontend/**" - "static/**" jobs: audit: runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 - uses: treosh/lighthouse-ci-action@v11 with: urls: | https://staging.internal/dashboard https://staging.internal/billing budgetPath: ./lighthouse-budget.json runs: 3 ``` ### Eight months later Dashboard LCP p75 went from 2.9s to 1.8s — mostly deferring non-critical CSS and killing the sync analytics load. API p95 stayed around 165ms because we stopped wasting time on it. INP p75 on the project filter dropped from 380ms to 140ms after we fixed a separate frontend issue (different post). None of it came from one heroic rewrite. It came from knowing which layer was lying. - Pick one server budget and one client budget in milliseconds. Write them down where PRs can fail against them. - Trace browser → CDN → app → DB. Server-only APM will convince you backend work is done when users still suffer. - Run Lighthouse in CI for regressions, RUM in prod for reality. They answer different questions. - Tie vitals to a business number before asking for sprint capacity. Conversion beats opinion. > We optimized Postgres for six weeks while a 140KB script blocked the main thread. Measure the whole path or you'll optimize the wrong thing with confidence.

Shared tables, tenant_id, and the RLS mistake that almost shipped

Mon, 09 Mar 2026 09:00:00 GMT

Multi-tenancy is a decision you make once and pay for every year after. On a clinic product with dozens of facilities sharing one database, we chose shared tables plus `tenant_id`. Schema-per-tenant looked cleaner on a whiteboard and fell apart the first time we had to run 80 identical migrations. ## The three options, without the marketing - Shared tables + tenant_id — cheapest to operate; isolation is only as strong as your filters (and RLS). - Schema per tenant — nicer isolation; migration and connection overhead grow with every customer. - Database per tenant — strongest walls; ops cost most teams cannot afford until they are huge. Under a few thousand tenants, shared tables with row-level security is the default I recommend. Everything else needs a written reason. ## RLS without the owner bypass We enabled RLS and congratulated ourselves. The app still connected as the table owner, so policies did not apply. FORCE ROW LEVEL SECURITY is not optional if your Django role owns the tables. ```sql migrations/0004_rls.sql ALTER TABLE invoices ENABLE ROW LEVEL SECURITY; ALTER TABLE invoices FORCE ROW LEVEL SECURITY; CREATE POLICY tenant_isolation ON invoices USING (tenant_id = current_setting('app.tenant_id')::uuid) WITH CHECK (tenant_id = current_setting('app.tenant_id')::uuid); -- Set once per request, inside the transaction SET LOCAL app.tenant_id = '3f7c9e2a-1d54-4f9b-9a11-6f2c0e4b7d55'; ``` :::note FORCE or it is theatre Without FORCE, the table owner bypasses policies. That is usually the role your application uses. We caught this in a security review, not in staging. ::: ## Set the tenant on every request ```python core/middleware.py from django.db import connection class TenantMiddleware: def __init__(self, get_response): self.get_response = get_response def __call__(self, request): tenant_id = getattr(request.user, "tenant_id", None) if tenant_id: with connection.cursor() as cursor: cursor.execute( "SET LOCAL app.tenant_id = %s", [str(tenant_id)], ) return self.get_response(request) ``` > Isolation the database enforces beats isolation every query is supposed to remember. Lead composite indexes with `tenant_id`. When one tenant's rows start dominating a hot index, partition that table — do not wait until the support ticket writes itself.

We Had Three Caching Layers and None of Them Worked Right

Tue, 24 Feb 2026 09:00:00 GMT

Our caching story before last year was embarrassing. Public product listings went through Redis with a 60s TTL. The CDN also cached them. Postgres got hit on every cache miss anyway because invalidation was scattered across views, Celery tasks, and one admin action nobody remembered writing. Then launch day: 4,000 warmed keys expired at the same second, Redis lock contention spiked, and our read replica hit 100% CPU for two minutes. Postmortem word was 'stampede.' I had to look that up. ## Pick the layer that matches the data HTTP caching (browser + CDN) is for responses identical across users that can be slightly stale — public catalogs, static JSON, marketing pages. Redis is for expensive per-user or per-tenant computation — permission matrices, dashboard aggregates, rate limit counters. In-process (lru_cache, a TTL dict) is for tiny hot data where even Redis RTT hurts — feature flags, FX rates we read on every request. We were using Redis for all three jobs. The public /api/v2/catalog/ endpoint didn't need it. Same JSON for every anonymous visitor. We ripped out the Redis wrapper, added Cache-Control and a weak ETag, and origin traffic on that route dropped 6x in a week. ## ETags without the footguns Weak ETag based on max(updated_at) plus page number. Cheap to compute, lets the CDN return 304 when nothing changed. The stale-while-revalidate header did more for perceived speed than anything else in this section — users get an instant stale response while the CDN revalidates in the background. ```python catalog/views.py import hashlib from django.http import JsonResponse from django.utils.cache import patch_response_headers from django.views.decorators.http import etag def catalog_etag(request, *args, **kwargs): latest = ( CatalogItem.objects.filter(published=True) .order_by("-updated_at") .values_list("updated_at", flat=True) .first() ) page = request.GET.get("page", "1") seed = f"catalog:{latest}:{page}" return hashlib.md5(seed.encode()).hexdigest() @etag(catalog_etag) def catalog_list(request): page = int(request.GET.get("page", 1)) qs = CatalogItem.objects.filter(published=True).order_by("id") items = qs[(page - 1) * 50 : page * 50] resp = JsonResponse({"results": [i.to_dict() for i in items]}) resp["Cache-Control"] = "public, max-age=60, stale-while-revalidate=180" patch_response_headers(resp, cache_timeout=120) return resp ``` CDN hit ratio on that endpoint went from 68% to 93%. Not because we got smarter about Redis — because we stopped using Redis for a problem HTTP caching solves for free. ## Redis: stampede guards and TTL jitter Application cache stays on Redis for things that actually vary per user. Dashboard summary for a tenant: 11 Postgres queries, ~120ms to compute, requested on every page load. Cache-aside with a lock so one worker recomputes while the others wait (briefly) instead of all hammering the database. ```python cache/helpers.py import json import random import time import redis r = redis.Redis.from_url(settings.REDIS_URL, decode_responses=True) def get_or_set(key: str, compute, ttl: int = 300): hit = r.get(key) if hit is not None: return json.loads(hit) lock = f"lock:{key}" acquired = r.set(lock, "1", nx=True, ex=10) if not acquired: for _ in range(25): time.sleep(0.04 + random.random() * 0.04) hit = r.get(key) if hit is not None: return json.loads(hit) # worst case: compute anyway. better than hanging forever. try: value = compute() jitter = ttl + random.randint(-45, 45) r.setex(key, jitter, json.dumps(value, default=str)) return value finally: if acquired: r.delete(lock) ``` The jitter matters. We had a nightly job warming ~4,000 tenant dashboard keys with identical 3600s TTL. They all expired together the next night. Postgres p95 spiked to 890ms. Adding ±45 seconds of jitter spread the recomputation window enough that the alert never fired again. Boring fix. Saved us twice since. :::note Stampede checklist (learned the hard way) Jitter TTLs. Single-flight or distributed lock on hot keys. Prefer stale-while-revalidate over hard expiry when stale-by-a-few-seconds is acceptable. Never warm thousands of keys with the same TTL in a loop without jitter. ::: ## Invalidation belongs with the write Every caching bug I've shipped was an invalidation bug. Stale prices. Deleted products still in list cache. The rule we finally enforced: the code that mutates the row invalidates the keys. Not the view that reads them. Not a separate 'cache service' nobody owns. ```python catalog/models.py from django.db.models.signals import post_save, post_delete from django.dispatch import receiver @receiver(post_save, sender=CatalogItem) @receiver(post_delete, sender=CatalogItem) def bust_catalog_cache(sender, instance, **kwargs): redis_client.delete(f"catalog:item:{instance.id}") # version bump beats SCAN on large keyspaces redis_client.incr("catalog:list:version") def catalog_list_key(tenant_id: int, page: int) -> str: ver = redis_client.get("catalog:list:version") or "0" return f"catalog:list:{tenant_id}:v{ver}:p{page}" ``` Versioned keys instead of SCAN for list invalidation. Bumping catalog:list:version instantly orphans old list keys without blocking Redis on a pattern delete. Old keys expire on their own. I know SCAN works for small keyspaces — we tried it first and got a 40ms pause during a flash sale. Not again. - HTTP/CDN for shared, slightly-stale responses. Redis for expensive per-tenant work. In-process for tiny hot flags. - stale-while-revalidate on public endpoints — users stop waiting on revalidation. - Jitter TTLs. Lock on recompute. Assume stampedes will happen at the worst time. - Invalidate in the model save path, not in random views. - Track hit ratio per key prefix. Under 40% hit rate usually means stop caching it. > There are only two hard things in computer science: cache invalidation, naming things, and not running KEYS in production. Six weeks after the rework: origin requests on catalog API down 71%, Redis memory down 38% (we stopped double-caching), zero stampede incidents vs three the prior quarter. The launch that caused the postmortem would've been fine with jitter and a CDN-first public catalog. Live and learn.

780KB Gzipped on Every Page Load (Including Login)

Thu, 05 Feb 2026 09:00:00 GMT

Our React SPA had one main chunk. 780KB gzipped. Shipped to every route — login, password reset, a public status page — because we'd been lazy-importing nothing for two years. Dev machines and office WiFi hid it. RUM on mid-range Android didn't. LCP p75 on /app/projects was 3.4s. INP p75 on the project search was 520ms. Different problems. Same root cause: nobody was looking at the bundle report. ## source-map-explorer first, opinions second I ran source-map-explorer before changing a line of code. Arguing about which dependency is heavy is pointless when the HTML report shows you the rectangles. ```bash terminal npm run build npx source-map-explorer dist/assets/index-*.js --html reports/bundle.html # what we found (gzipped, approx): # moment + all locales 138 KB (used in 2 date pickers) # recharts 112 KB (one analytics route) # lodash (default import) 68 KB (_.get in 4 files) # admin module tree 94 KB (pulled into main via barrel export) ``` Three easy wins: date-fns with explicit imports (9KB for what we actually use), lodash-es per-function imports, fix the barrel export that dragged admin routes into the main chunk. Recharts stayed — but only on the analytics route via lazy(). ## Split routes, prefetch the likely next one React.lazy alone just moves cost from first load to first navigation. We prefetch the chunk users statistically hit next. Analytics data showed 38% of /app/home sessions navigate to /app/projects within 30 seconds. ```typescript src/routes/index.tsx import { lazy, Suspense } from "react"; import { createBrowserRouter } from "react-router-dom"; import { RouteShell } from "./RouteShell"; const Home = lazy(() => import("./Home")); const Projects = lazy(() => import(/* webpackPrefetch: true */ "./Projects")); const Analytics = lazy(() => import("./Analytics")); const Admin = lazy(() => import("./Admin")); export const router = createBrowserRouter([ { path: "/app", element: , children: [ { index: true, element: }, { path: "projects", element: }, { path: "analytics", element: }, { path: "admin/*", element: }, ], }, ]); function Lazy({ children }: { children: React.ReactNode }) { return }>{children}; } ``` Main chunk: 780KB → 187KB gzipped. Analytics chunk: 112KB, loaded on demand. Admin: 91KB, never prefetched — internal users only, under 2% of sessions. LCP p75 on /app/home dropped to 1.9s without touching the API. ## Images were half the bytes Marketing screens were serving 2400px PNGs into 400px containers. Fixed with AVIF/WebP sources, explicit width/height on every img (CLS was 0.24 — brutal), and fetchpriority='high' on the hero image that's actually the LCP element. ```tsx src/components/Picture.tsx type Props = { base: string; // without extension alt: string; width: number; height: number; priority?: boolean; }; export function Picture({ base, alt, width, height, priority }: Props) { return ( ); } ``` Width and height aren't optional decoration. Without them the browser can't reserve space — our CLS went from 0.24 to 0.04 before we changed a single image file. Just attributes. ## INP: the search box from hell Bundle size fixes load time. INP measures what happens after. Our project list search filtered 8,200 rows synchronously on every keystroke. Chrome reported 520ms INP — the full cost from keydown to next paint, not just input delay. FID would've looked fine. INP didn't. ```typescript src/features/projects/ProjectSearch.tsx import { useDeferredValue, useMemo, useState, useTransition } from "react"; export function ProjectSearch({ projects }: { projects: Project[] }) { const [query, setQuery] = useState(""); const deferred = useDeferredValue(query); const [isPending, startTransition] = useTransition(); const filtered = useMemo(() => { const q = deferred.trim().toLowerCase(); if (!q) return projects; return projects.filter((p) => p.name.toLowerCase().includes(q)); }, [projects, deferred]); return ( <> { const v = e.target.value; startTransition(() => setQuery(v)); }} aria-busy={isPending} /> ); } ``` useDeferredValue plus startTransition: input updates immediately, filter runs at lower priority and gets interrupted by the next keystroke. INP on that control dropped to 95ms p75. Still not instant with 8K rows — we'd need virtualization next — but the metric we were failing finally went green. :::note INP is not FID with a new label FID only measured delay before your handler ran. INP measures the whole interaction through paint — including your handler and every re-render it triggers. Long synchronous work in onChange handlers shows up here. ::: ## Polling was re-rendering the whole tree Separate issue, same dashboard. React Query refetch every 5s. Profiler showed 47 components re-rendering per tick because our queryFn parsed ISO date strings into new Date objects every time, breaking structural sharing even when the payload was identical. ```typescript src/features/dashboard/queries.ts export function useDashboardSummary() { return useQuery({ queryKey: ["dashboard", "summary"], queryFn: async () => { const raw = await api.get("/dashboard/summary/"); // stable parsing — reuse date strings as keys, parse once return { ...raw, updatedAt: raw.updated_at, // keep string; parse in display layer widgets: raw.widgets.map((w) => ({ ...w, value: Number(w.value) })), }; }, refetchInterval: 5_000, structuralSharing: true, }); } ``` Fixing deserialization dropped re-renders per poll from 47 to 4. Main thread time during idle polling fell enough that unrelated interactions stopped spiking. Profiler beats guessing. - Run source-map-explorer before optimizing. The top three rectangles are usually embarrassing. - Lazy-load by route; prefetch the chunk analytics says users hit next. - Set width/height on images. Cheap CLS win. - Defer expensive filter/sort work — INP cares about the whole interaction. - Keep referential stability in queryFns or polling will cascade re-renders. > A 1.8s LCP and a 520ms keystroke is still a slow app. Load time and interaction latency are different tickets. Where we landed: 187KB main chunk (from 780KB), LCP p75 1.9s, CLS 0.04, INP p75 118ms on interaction-heavy routes. No framework migration. Just bundle analysis, splitting, and actually reading the profiler.

214 Queries to Show 50 Orders

Wed, 21 Jan 2026 09:00:00 GMT

Ticket: 'Orders page slow for enterprise customers.' /api/v1/orders/ p95 was 2.3s. First suggestion in Slack: add an index on created_at. I checked — we already had a composite on (account_id, created_at DESC). The query planner was fine. The ORM was issuing 214 queries per request. Nobody had counted because django-silk wasn't enabled in staging and production logging only showed wall time. ## Count queries before touching indexes I added middleware that logs path, query count, and total query time when either exceeds a threshold. Ugly, effective. First slow request in staging: 214 queries, 1.8s in Postgres alone for 50 orders. ```python core/middleware/query_log.py import logging from django.db import connection, reset_queries logger = logging.getLogger("slow_queries") class QueryCountMiddleware: QUERY_BUDGET = 15 TIME_BUDGET = 0.15 # seconds def __init__(self, get_response): self.get_response = get_response def __call__(self, request): reset_queries() response = self.get_response(request) count = len(connection.queries) total = sum(float(q["time"]) for q in connection.queries) if count > self.QUERY_BUDGET or total > self.TIME_BUDGET: logger.warning( "slow_db path=%s queries=%d db_time=%.3fs", request.path, count, total, ) return response ``` That log line is what killed the 'just add an index' theory. 214 round trips to Postgres. Each query fast individually. Together: disaster. ## N+1 in the wild Classic pattern. Loop orders, touch customer.name (FK query), lineitem_set.all() (reverse FK query), shipping_address.city (another FK). Fifty orders × four extra queries plus the base query. Textbook. ```python orders/views_before.py # 1 + 3N queries — do not ship this def order_list(request): orders = Order.objects.filter(account=request.account).order_by("-created_at")[:50] return JsonResponse({ "results": [ { "id": o.id, "total": str(o.total), "customer": o.customer.name, "items": [li.sku for li in o.lineitem_set.all()], "city": o.shipping_address.city, } for o in orders ] }) ``` ```python orders/views_after.py from django.db.models import Prefetch # 3 queries flat def order_list(request): orders = ( Order.objects.filter(account=request.account) .select_related("customer", "shipping_address") .prefetch_related( Prefetch("lineitem_set", queryset=LineItem.objects.only("order_id", "sku")) ) .order_by("-created_at")[:50] ) return JsonResponse({"results": [serialize(o) for o in orders]}) ``` select_related for forward FKs — one join. prefetch_related for reverse FKs — one extra query for all line items across the page. Query count: 214 → 3. p95: 2.3s → 340ms before we touched pagination. I should've run django-debug-toolbar on this view two years ago. :::note CI guard we added after the third regression Silk against fixture data in CI. Fail if any endpoint's query count grows more than 20% vs baseline. Caught six N+1 regressions in six months. Cheaper than another postmortem. ::: ## Offset pagination lies to you 340ms at page 1. Page 40 — offset 2000 — back to 940ms p95. OFFSET doesn't skip rows. Postgres walks and discards everything before your LIMIT. Enterprise customers with years of order history hit this constantly. ```sql explain_offset_vs_keyset.sql -- OFFSET: cost grows with page depth EXPLAIN ANALYZE SELECT o.id, o.created_at, o.total FROM orders o WHERE o.account_id = 9912 ORDER BY o.created_at DESC LIMIT 50 OFFSET 2000; -- Seq scan on discarded rows adds up -- Keyset: stable cost at any depth EXPLAIN ANALYZE SELECT o.id, o.created_at, o.total FROM orders o WHERE o.account_id = 9912 AND o.created_at < TIMESTAMPTZ '2025-10-14 09:31:22+00' ORDER BY o.created_at DESC LIMIT 50; -- Index scan on (account_id, created_at DESC), no discard loop ``` We switched to cursor pagination. Signed cursor in the query string so clients can't inject arbitrary filters. Fetch 51 rows to detect has_next without a COUNT(*). Tradeoff: no 'jump to page 37.' Nobody was doing that — they clicked 'load more.' ```python orders/views_cursor.py from django.core.signing import Signer, BadSignature signer = Signer(salt="order-cursor") def order_list_cursor(request): qs = ( Order.objects.filter(account=request.account) .select_related("customer", "shipping_address") .prefetch_related("lineitem_set") .order_by("-created_at") ) cursor = request.GET.get("cursor") if cursor: try: ts = signer.unsign(cursor) qs = qs.filter(created_at__lt=ts) except BadSignature: return JsonResponse({"error": "invalid cursor"}, status=400) rows = list(qs[:51]) has_next = len(rows) > 50 page = rows[:50] next_cursor = None if has_next and page: next_cursor = signer.sign(page[-1].created_at.isoformat()) return JsonResponse({ "results": [serialize(o) for o in page], "next_cursor": next_cursor, }) ``` Deep pagination p95 flattened to ~90ms. Same 3 queries whether you're on page 1 or page 200. ## Streaming the export nobody should wait on Separate endpoint: CSV export of all orders. Original version loaded everything into memory, held a gunicorn worker for 18+ seconds, timed out for large accounts. Two paths — stream when the client waits inline, background job when aggregation is genuinely multi-minute. ```python orders/export.py from django.http import StreamingHttpResponse import csv class Echo: """csv.writer expects a file-like; this yields strings.""" def write(self, value): return value def export_orders_csv(request): def rows(): pseudo_buffer = Echo() writer = csv.writer(pseudo_buffer) yield writer.writerow(["id", "created_at", "total", "status"]) qs = ( Order.objects.filter(account=request.account) .order_by("id") .iterator(chunk_size=2000) ) for order in qs: yield writer.writerow([order.id, order.created_at, order.total, order.status]) resp = StreamingHttpResponse(rows(), content_type="text/csv") resp["Content-Disposition"] = 'attachment; filename="orders.csv"' return resp ``` iterator(chunk_size=2000) is the part people skip. Without it Django caches the whole queryset in memory and streaming is theater. First bytes hit the client in ~180ms now. The monthly reconciliation report — multi-join, minutes of CPU — went to Celery with a poll endpoint. Different shape, same principle: don't hold a worker on work that outlasts patience. ```python orders/tasks.py @shared_task(bind=True, acks_late=True) def build_reconciliation_export(self, account_id: int, job_id: str): job = ExportJob.objects.get(id=job_id) job.status = "running" job.save(update_fields=["status"]) path = _write_reconciliation_csv(account_id) # the slow part job.status = "complete" job.file_path = path job.save(update_fields=["status", "file_path"]) ``` - Log query count per request before assuming indexes will help. - select_related for FK forward, prefetch_related for reverse/M2M. Fix N+1 first. - Keyset pagination anywhere users page deep into large ordered tables. - iterator() with StreamingHttpResponse for large inline downloads. - Background task + status poll for work that takes minutes, not seconds. > The fastest query is the one you don't run. The second fastest is the one you run once instead of two hundred times. Final numbers on /api/v1/orders/: p95 2.3s → 88ms, query count 214 → 3, no new hardware. CSV export: 18s blocking → 180ms time-to-first-byte streaming. The index we'd have added would've changed nothing. Profiling would've saved six weeks if we'd done it first.

I Cut Our Django Image From 1.2GB to 187MB — Here's the Dockerfile

Fri, 09 Jan 2026 09:00:00 GMT

The first time I checked docker images on our staging host I thought the numbers were wrong. Our Django API was 1.21GB. Same codebase, same requirements.txt, sitting next to a Go service at 28MB. Security scanner flagged it for running as root and shipping a full gcc toolchain. Registry storage was climbing every sprint. I spent a weekend rewriting the Dockerfile and we shipped 187MB. This is the exact path I took, with the sizes at each step. ## What the naive image actually costs Cold pulls on Kubernetes matter more than vanity. When HPA adds nodes during a traffic spike, each new node pulls the image before the pod can start. At 1.2GB that was 45–90 seconds on our registry path. At 187MB it's under 10. Attack surface shrinks too — you don't need make, g++, and python3-dev in a runtime container that only ever runs gunicorn. ```dockerfile Dockerfile.naive FROM python:3.11 WORKDIR /app RUN apt-get update && apt-get install -y \ build-essential \ libpq-dev \ curl \ git COPY requirements.txt . RUN pip install --no-cache-dir -r requirements.txt COPY . . CMD ["gunicorn", "config.wsgi", "--bind", "0.0.0.0:8000"] ``` docker history showed the apt-get layer alone at ~280MB. python:3.11 (full Debian, not slim) is already ~900MB before you add anything. Everything needed to compile psycopg2 stayed in the final image forever. We ran as root. No healthcheck. Classic. ## Split build from runtime Builder stage gets the compilers. Runtime stage gets only the wheels and the shared libraries the process links against. I switched pip to uv for installs — lockfile resolve that used to take ~40s in CI now finishes in under 4s on a warm cache. ```dockerfile Dockerfile # ---- build ---- FROM python:3.11-slim AS builder RUN apt-get update && apt-get install -y --no-install-recommends \ build-essential \ libpq-dev \ && rm -rf /var/lib/apt/lists/* RUN pip install --no-cache-dir uv WORKDIR /app COPY pyproject.toml uv.lock ./ RUN uv export --frozen --no-dev --format requirements-txt > requirements.txt \ && uv pip install --system --no-cache --target=/deps -r requirements.txt # ---- runtime ---- FROM python:3.11-slim AS runtime RUN apt-get update && apt-get install -y --no-install-recommends \ libpq5 \ curl \ && rm -rf /var/lib/apt/lists/* \ && groupadd --gid 1000 app \ && useradd --uid 1000 --gid app --shell /bin/bash --create-home app WORKDIR /app COPY --from=builder /deps /usr/local/lib/python3.11/site-packages COPY --chown=app:app . . USER app ENV PYTHONUNBUFFERED=1 \ PYTHONDONTWRITEBYTECODE=1 HEALTHCHECK --interval=30s --timeout=3s --start-period=10s --retries=3 \ CMD curl -f http://localhost:8000/healthz/ || exit 1 EXPOSE 8000 CMD ["gunicorn", "config.wsgi", "--bind", "0.0.0.0:8000", "--workers", "3"] ``` libpq-dev in the builder, libpq5 in runtime. Same story for any -dev package: headers and static libs compile the extension; the shared library is what the binary needs at night. Miss that swap and you drag ~50–80MB of build junk into every deploy. :::note Measure every layer After each build I run `docker history --format '{{.Size}}\t{{.CreatedBy}}'`. Any RUN that adds more than ~50MB without an obvious reason gets cut or merged. Guessing is how we got to 1.2GB in the first place. ::: ## Non-root is not optional USER app drops privileges before gunicorn starts. If someone gets RCE through a bad deserialization path, they don't own the container as UID 0. I've watched a team mount /var/run/docker.sock into an app container 'just for CI helpers' while running as root — that's effectively handing the host to whoever breaks the app. Don't mount the socket. Don't run as root. Gotcha: if the process writes uploads or local logs, chown those paths before USER or you'll only see PermissionError in the container, never on your laptop. ## Layer order and BuildKit caches Copy lockfiles before source. Dependency install only invalidates when pyproject.toml or uv.lock change. On our monorepo that cut code-only CI builds from ~4 minutes to ~25 seconds because the /deps layer stayed cached. - Least-changed instructions first, COPY . . last - Lockfiles before application source - BuildKit --mount=type=cache for uv/pip so CI runners don't redownload every cold start - Pin base images by digest in production, not just :3.11-slim - Collapse apt-get update + install + rm into one RUN so you don't leave package indexes in a layer ### Cache mounts in CI ```dockerfile Dockerfile.cache-mount # syntax=docker/dockerfile:1.7 FROM python:3.11-slim AS builder RUN --mount=type=cache,target=/root/.cache/uv \ pip install --no-cache-dir uv \ && uv pip install --system --no-cache -r requirements.txt ``` Needs BuildKit (default on recent Docker). In GitHub Actions I set cache-from/cache-to type=gha on docker/build-push-action. Without that, every fresh runner redownloads wheels and half the win disappears. ## Numbers from the same app 1. Naive python:3.11 + pip: 1.21GB, cold CI build 6m40s 2. python:3.11-slim single stage: 640MB, cold build 4m10s 3. Multi-stage + uv + slim runtime: 187MB, cold 48s, cached code change 6s > An image is a liability the second you push it. Ship the smallest one you can defend. 187MB holds because the runtime never sees a compiler, never keeps pip's cache, and only carries libpq5. On spot nodes that difference is a five-second cold start versus thirty. ## Healthchecks that exercise real deps TCP 'port is open' is not healthy. Our /healthz/ pings Postgres and Redis. During a DB failover we want the orchestrator to stop routing before users hit 500s. ```python healthz.py from django.db import connections from django.http import JsonResponse from django_redis import get_redis_connection def healthz(request): try: connections["default"].cursor().execute("SELECT 1") get_redis_connection("default").ping() except Exception as exc: return JsonResponse({"status": "error", "detail": str(exc)}, status=503) return JsonResponse({"status": "ok"}) ``` ## What I keep on the checklist - Multi-stage: compile in builder, copy artifacts into slim runtime - Runtime libs only — libpq5 not libpq-dev, same for every native dep - Layer by change frequency; BuildKit cache mounts in CI - USER non-root; chown writable dirs first - Healthcheck hits DB/cache, not just process liveness - docker history after every size claim — no vibes

docker compose up Is Not a Deploy Strategy

Tue, 16 Dec 2025 09:00:00 GMT

We ran production on docker compose up -d for about a year. It worked until a Friday deploy killed the site for four minutes: new containers accepted traffic before migrations finished, old ones died before in-flight requests drained. Compose never forced us to think about ordering. This is the checklist I use to make a compose-based deploy reliable without jumping straight to Kubernetes. ## Secrets leave the compose file DATABASE_URL in docker-compose.yml, or a committed .env, is still the most common footgun I inherit. Compose can point at an env_file that your deploy script writes from Vault (or whatever you use) and shreds when the script exits. Values never live in git. ```yaml docker-compose.prod.yml services: web: image: registry.example.com/app:${IMAGE_TAG} env_file: - ./secrets/app.env deploy: replicas: 3 update_config: parallelism: 1 delay: 10s order: start-first failure_action: rollback rollback_config: parallelism: 1 order: stop-first healthcheck: test: ["CMD", "curl", "-f", "http://localhost:8000/healthz/"] interval: 10s timeout: 3s retries: 3 start_period: 15s ``` ```bash deploy.sh #!/usr/bin/env bash set -euo pipefail IMAGE_TAG="$1" SECRETS_PATH="secrets/app.env" vault kv get -format=json secret/app/prod \ | jq -r '.data.data | to_entries[] | "\(.key)=\(.value)"' > "$SECRETS_PATH" chmod 600 "$SECRETS_PATH" trap 'shred -u "$SECRETS_PATH"' EXIT export IMAGE_TAG docker compose -f docker-compose.prod.yml pull docker compose -f docker-compose.prod.yml up -d --no-deps --scale web=3 web ``` :::note Secrets hygiene Generate the env file at deploy time, chmod 600, shred on EXIT. If a secret sits on disk longer than the deploy, treat it as leaked and rotate. ::: ## Migrate once, before traffic moves Running manage.py migrate in the same entrypoint as gunicorn is how you get three replicas racing migrations on boot. For anything heavier than a trivial ADD COLUMN that can deadlock or leave half-applied state depending on your lock behavior. ```bash deploy.sh docker compose -f docker-compose.prod.yml run --rm \ --no-deps web python manage.py migrate --noinput if [ $? -ne 0 ]; then echo "migration failed, aborting deploy" >&2 exit 1 fi docker compose -f docker-compose.prod.yml up -d --no-deps web ``` Failed migrate exits before we touch running web containers. Old version keeps serving. That's the whole point of a separate job. ## start-first and graceful drain order: start-first starts the new container and waits for healthcheck before stopping the old one. Default stop-first kills the old container immediately — fine for local, wrong for live connections. Pair it with a healthcheck that hits Postgres and Redis, not just TCP. ```python gunicorn.conf.py bind = "0.0.0.0:8000" workers = 3 worker_class = "gthread" threads = 4 graceful_timeout = 30 timeout = 60 def worker_exit(server, worker): server.log.info("worker exiting, draining connections") ``` graceful_timeout = 30 lets gunicorn finish in-flight work after SIGTERM. Set compose stop_grace_period (or k8s terminationGracePeriodSeconds) higher than that, or the runtime SIGKILLs you mid-drain. ## Logs and memory ceilings json-file with no max-size will fill the disk. We had a host go read-only at 2am because a retry loop wrote stack traces until /var/lib/docker/containers ate everything. ```yaml docker-compose.prod.yml services: web: logging: driver: json-file options: max-size: "10m" max-file: "5" deploy: resources: limits: cpus: "1.5" memory: 768M reservations: cpus: "0.5" memory: 256M ``` Without a memory limit, a Celery leak can OOM-kill Postgres on the same host. I've watched that exact sequence. Caps are reliability, not just cost control. ## Rollback you can run at 3am failure_action: rollback covers healthcheck failures during the roll. You still need a manual path when the new version is 'healthy' but wrong under real traffic. - Tag images with git SHA — never :latest in prod - Keep at least the previous two tags in the registry - One-line rollback script for on-call - Know before you migrate whether the schema change is backward-compatible with the previous app version - Rehearse rollback in staging quarterly — first time should not be an incident ```bash rollback.sh #!/usr/bin/env bash set -euo pipefail PREVIOUS_TAG="$1" export IMAGE_TAG="$PREVIOUS_TAG" docker compose -f docker-compose.prod.yml pull docker compose -f docker-compose.prod.yml up -d --no-deps web echo "rolled back to $PREVIOUS_TAG" ``` > The deploy script isn't done when it ships forward. It's done when someone half-awake can undo it. ## Expand/contract or rollback is a trap Perfect compose mechanics won't save you if a migration dropped a column the old code still reads. Expand the schema, deploy code that tolerates both shapes, contract later once you're sure you won't roll back. That's a schema design problem, not a Docker problem. ## Checklist - Secrets generated at deploy time, shredded after - Migrations as a one-shot job; abort deploy on failure - start-first + real healthcheck for rolling updates - gunicorn graceful_timeout below orchestrator grace period - Log rotation and CPU/memory limits on every service - Rehearsed rollback; backward-compatible migrations so rollback is safe

EXPLAIN (ANALYZE, BUFFERS) Before You CREATE INDEX

Thu, 27 Nov 2025 09:00:00 GMT

I don't add indexes from intuition anymore. I run EXPLAIN (ANALYZE, BUFFERS), read what the planner did, pick the index type that matches the access pattern, then run the same EXPLAIN again. Below are real plans from an orders table with ~14 million rows — the customer order-history path that was burning 800ms+ on every load. ## Read the plan, not the guess EXPLAIN alone is the planner's estimate. ANALYZE executes the query and reports real timings and row counts. BUFFERS shows shared-buffer hits vs disk reads. Optimizing from EXPLAIN without ANALYZE is optimizing fiction. ```sql before.sql EXPLAIN (ANALYZE, BUFFERS) SELECT id, status, total_cents, created_at FROM orders WHERE customer_id = 48213 AND status = 'pending' ORDER BY created_at DESC LIMIT 20; ``` ```text before_plan.txt Limit (cost=0.00..14832.10 rows=20 width=32) (actual time=812.441..812.449 rows=14 loops=1) -> Gather Merge (cost=0.00..... rows=..) (actual time=812.439..812.446 rows=14 loops=1) -> Parallel Seq Scan on orders (cost=0.00..894213.00 rows=48901 width=32) (actual time=0.031..798.112 rows=16340 loops=3) Filter: ((customer_id = 48213) AND (status = 'pending'::text)) Rows Removed by Filter: 4,647,321 Buffers: shared hit=2104 read=189302 Planning Time: 0.312 ms Execution Time: 812.601 ms ``` Parallel seq scan, 189,302 pages from disk, 4.6M rows filtered away to return 14. 812ms on a hot page. That's the smoking gun. ## Composite B-tree matching the filters Equality on customer_id and status, then sort by created_at. Index column order follows that: equalities first, sort key last. ```sql index.sql CREATE INDEX CONCURRENTLY idx_orders_customer_status_created ON orders (customer_id, status, created_at DESC); ``` CONCURRENTLY on 14M rows is non-negotiable. Plain CREATE INDEX takes a SHARE lock and blocks writes for the whole build — minutes of stuck inserts on a table this size. CONCURRENTLY takes longer and may need a second pass, but writes keep flowing. ```text after_plan.txt Limit (cost=0.43..8.91 rows=20 width=32) (actual time=0.052..0.089 rows=14 loops=1) -> Index Scan using idx_orders_customer_status_created on orders (cost=0.43..8.91 rows=20 width=32) (actual time=0.051..0.086 rows=14 loops=1) Index Cond: ((customer_id = 48213) AND (status = 'pending'::text)) Buffers: shared hit=6 Planning Time: 0.198 ms Execution Time: 0.112 ms ``` 812ms → 0.112ms. Buffer reads 189,302 → 6. That index paid for itself on the next deploy. :::note Column order (customer_id, status, created_at DESC) supports this query. (created_at, customer_id, status) forces a filter step instead of a seek. Equality columns first, sort column last. ::: ## GIN when B-tree can't help We store gateway payloads in metadata jsonb and support filters with metadata @> '{"flagged": true}'. B-tree doesn't accelerate JSONB containment. GIN does. ```sql gin_index.sql CREATE INDEX CONCURRENTLY idx_orders_metadata_gin ON orders USING gin (metadata jsonb_path_ops); ``` jsonb_path_ops is smaller and faster for @> than default jsonb_ops, but it doesn't support ? key-existence. Our queries were containment-only, so path_ops won: ~340MB vs ~510MB on this table. ## Partial indexes for skewed status 94% of rows are completed or cancelled. Hot paths only touch pending/processing. Indexing every status wastes disk on rows nobody filters that way. ```sql partial_index.sql CREATE INDEX CONCURRENTLY idx_orders_active_pending ON orders (created_at DESC) WHERE status IN ('pending', 'processing'); ``` Projected full index ~210MB; partial landed at ~14MB (~6% of rows). The planner only uses it when the query's WHERE matches or implies that predicate — the app has to filter the same way. ## Covering indexes and index-only scans Dashboard query only needs id, status, total_cents. INCLUDE those columns and Postgres can answer from the index without heap fetches — when the visibility map cooperates. ```sql covering_index.sql CREATE INDEX CONCURRENTLY idx_orders_customer_covering ON orders (customer_id, status) INCLUDE (total_cents, id) WHERE status <> 'cancelled'; ``` Heavy update churn leaves Heap Fetches in the plan until autovacuum refreshes the visibility map. If you expected index-only and still see heap access, check pg_stat_user_tables.n_dead_tup and VACUUM. ## Find dead weight ```sql bloat_check.sql SELECT schemaname, indexrelname, pg_size_pretty(pg_relation_size(indexrelid)) AS index_size, idx_scan FROM pg_stat_user_indexes JOIN pg_index USING (indexrelid) WHERE idx_scan = 0 AND indisunique IS FALSE ORDER BY pg_relation_size(indexrelid) DESC LIMIT 20; ``` idx_scan = 0 for weeks means pure write overhead. DROP INDEX CONCURRENTLY. For bloat on indexes you still need, REINDEX CONCURRENTLY (Postgres 12+) rebuilds without blocking reads or writes. - B-tree: equality, ranges, sorting — default for a reason - GIN: JSONB/array containment, full-text vectors - Partial: skewed data where queries only touch a predictable slice - Covering (INCLUDE): hot reads where skipping the heap matters - BRIN: huge naturally ordered tables (time-series) when you want a tiny index > Every index is a bet that this read pattern is worth the write cost forever. Don't place it without a plan on screen. ## Takeaways - Always EXPLAIN (ANALYZE, BUFFERS) before and after - Composite order: equality filters first, sort column last - GIN + jsonb_path_ops for containment-only JSONB - Partial indexes when status (or similar) is heavily skewed - INCLUDE for index-only scans; watch Heap Fetches vs visibility map - CREATE / REINDEX / DROP with CONCURRENTLY on live tables - Audit idx_scan = 0 periodically