문제: 3차 수정의 확인 리뷰가 새 Important 없이 Minor 19건을 남겼다. 동작에 닿는 것이 셋이다. JDK read-idle watchdog이 공용 ForkJoinPool에서 돌아 그 풀이 바쁘면 300ms timeout이 6초 가까이 늦었다. 붙여 쓴 -x<task>를 verifyStrictLaneStages가 읽지 않아 lane을 뺀 workflow가 통과했다. 사용자 전송이 중간 클래스를 거쳐 @SystemCaller bean을 부르면 ArchUnit 규칙을 지났다. 잠금 대기 검사기는 FK·trigger·REINDEX와 lock_timeout = 0을 못 봤고, 쿼터 보정 SQL은 코드와 다른 모양의 행을 넣었다. 방법: 영역별 4묶음이 맡은 Minor만 test-driven-development 절차로 고쳤다(재리뷰 README 12절). - watchdog은 전용 데몬 스레드 하나에서 돌고 스트림을 닫으면 작업을 취소한다. read당 비용은 그대로다(21–24ns). - 사용자 전송 → @SystemCaller는 직접·간접 의존 모두 거부한다. 리뷰가 권한 직접 의존 검사는 중간 클래스 하나로 다시 열려서 버렸다. - 검사기를 보강하면서 새로 걸린 이미 배포된 마이그레이션 5개는 내용을 바꾸지 않고 동결 목록에 더했다(main과 blob이 같다). - 비밀 목록 bean은 규칙보다 먼저 만들어 실제 클래스로 검사한다. 위험·한계: 회복 필요 예외 메시지 끝에 scope digest가 붙고 기동 실패 문구가 바뀌어, 문구 전체에 맞추던 로그 매처는 고쳐야 한다. 코드가 든 Redis 계정 참조의 fixture- 검사는 문서로만 막았다(코드로 막으려면 새 의존 edge가 필요하다). failure-retention ≥ replay-ttl 기동 검사는 넣지 않았다. -mx 같은 짧은 플래그 묶음과 graphql 전송 fixture는 남았다. spotbugsTestFixtures(게이트 밖)의 정규식 경고가 15건으로 늘었다. 검증: ci, 두 빌드의 verifyStrictLaneStages --dry-run, mainCheck, :integrationCheck, -p optional-platforms ci integrationCheck performanceCheck, :performanceCheck, :releaseCheck(-PgitRevision), verify-compose-profile-contracts(18 lane) 첫 실행 통과. watchdog은 공용 풀을 채운 상태에서 300ms 설정이 300ms에 끝나는 것을 쟀다. Refs: docs/reviews/2026-11-03-rereview/README.md 11·12절, round3/
200 lines
8.7 KiB
Markdown
200 lines
8.7 KiB
Markdown
# Fileserver runbooks
|
|
|
|
Each runbook names the exact metric that fires it and the exact command that resolves it. A runbook
|
|
whose trigger is "someone noticed" is not actionable, so every one below starts from a signal.
|
|
|
|
## Storage full
|
|
|
|
**Signal** — `fileserver.quota{result="rejected"}` rising, or `507` responses appearing.
|
|
|
|
Storage capacity is exhausted or the high-water guard tripped. Uploads are rejected before any bytes
|
|
are written, so nothing is corrupt; the system is refusing work it cannot complete.
|
|
|
|
```bash
|
|
curl -s $ADMIN/internal/fileserver/storage-health | jq '.usedFraction, .usableBytes'
|
|
curl -s -X POST "$ADMIN/internal/fileserver/uploads:cleanup?maxItems=500&maxBytes=10737418240"
|
|
curl -s "$ADMIN/internal/fileserver/orphans?limit=200" | jq '[.[].sizeBytes] | add'
|
|
```
|
|
|
|
Drain the cleanup backlog first — it reclaims space the system already knows is dead. Only then
|
|
consider an orphan reconcile, and start with a dry run.
|
|
|
|
## Orphan growth
|
|
|
|
**Signal** — `fileserver.cleanup{result="skipped"}` climbing, or the orphan scan returning more
|
|
objects each run.
|
|
|
|
Physical objects exist with no metadata record pointing at them. This is not immediately dangerous —
|
|
nothing serves them — but it consumes capacity indefinitely.
|
|
|
|
```bash
|
|
# Always look first. A reconcile without dryRun=false is a plan, not an action.
|
|
curl -s -X POST "$ADMIN/internal/fileserver/orphans:reconcile" \
|
|
-H 'content-type: application/json' -d '{"limit":100}' | jq '.candidates'
|
|
|
|
# Apply only the fingerprints you were just shown.
|
|
curl -s -X POST "$ADMIN/internal/fileserver/orphans:reconcile" \
|
|
-H 'content-type: application/json' \
|
|
-d '{"dryRun":false,"limit":100,"maxBytes":1073741824,
|
|
"expectedFingerprints":["<from the dry run>"],"reasonCode":"ORPHAN_GROWTH_RUNBOOK"}'
|
|
```
|
|
|
|
Echoing the fingerprints is the safety property: an object that changed between the scan and the
|
|
apply is skipped rather than deleted.
|
|
|
|
## Verification backlog
|
|
|
|
**Signal** — `fileserver.verification.queue{age_bucket="old"}` non-zero, or files sitting in
|
|
VERIFYING.
|
|
|
|
A verifier is slow or unavailable. Files stay non-public, which is the correct failure direction: a
|
|
`RETRY` verdict never becomes an `ACCEPT`.
|
|
|
|
```bash
|
|
curl -s $ADMIN/internal/fileserver/capabilities | jq '.storageType'
|
|
# Once the verifier is healthy, quarantined files can be re-examined individually.
|
|
curl -s -X POST "$ADMIN/internal/fileserver/files/$FILE_ID:reverify"
|
|
```
|
|
|
|
Do not clear the backlog by disabling verification. A file that reached READY without an accepting
|
|
verdict cannot be distinguished later from one that was verified.
|
|
|
|
## NFS ambiguity
|
|
|
|
**Signal** — problem documents carrying `"ambiguous": true`, or
|
|
`fileserver.transfer.interruption{reason="stale_handle"}`.
|
|
|
|
An operation's outcome could not be determined: the response was lost after the write or rename may
|
|
have landed. These are never retried automatically.
|
|
|
|
```bash
|
|
# The recovery queue holds the files awaiting a decision.
|
|
curl -s "$ADMIN/internal/fileserver/uploads/incomplete?limit=100" | jq
|
|
```
|
|
|
|
Reconciliation compares the physical size and digest against the record and only confirms READY when
|
|
all four of key, size, digest, and version agree. Anything short of that is reported, never guessed.
|
|
|
|
## PVC remount
|
|
|
|
**Signal** — startup failure naming "atomic move", "same file store", or "not writable".
|
|
|
|
The volume was remounted somewhere the probe can no longer prove a required capability. The
|
|
application refuses traffic rather than serving from storage it cannot publish to atomically.
|
|
|
|
```bash
|
|
kubectl apply -f infra/fileserver/kubernetes/pvc-certification-job.yaml
|
|
kubectl logs job/fileserver-pvc-certification
|
|
```
|
|
|
|
Compare the printed tuple with the certified one in `docs/fileserver/storage-certification.md`. A
|
|
mismatch in CSI driver, StorageClass, access mode, or mount options is the cause; the certification
|
|
does not carry across it.
|
|
|
|
## Nginx delegation failure
|
|
|
|
**Signal** — `fileserver.download.delegation{delegated="true"}` with client-visible `404`s.
|
|
|
|
The internal location is misconfigured, so the proxy cannot resolve the redirect it was handed.
|
|
|
|
```bash
|
|
# The internal prefix must resolve to the content root and must be marked `internal`.
|
|
grep -A5 '__files' infra/fileserver/nginx/nginx.conf
|
|
curl -s $ADMIN/internal/fileserver/capabilities | jq '.capabilities.delegatedDownload'
|
|
```
|
|
|
|
Turning delegation off is a safe immediate mitigation: the application serves the transfer itself,
|
|
slower but correct.
|
|
|
|
```bash
|
|
app.fileserver-platform.nginx.enabled=false
|
|
```
|
|
|
|
## Cleanup backlog
|
|
|
|
**Signal** — `fileserver.cleanup{result="deferred"}` rising, or reclaimed bytes flat while deletes
|
|
continue.
|
|
|
|
Items are being deferred faster than they drain. The usual cause is an active writer lease still
|
|
holding staging objects, which is correct behaviour, not a fault.
|
|
|
|
```bash
|
|
curl -s "$ADMIN/internal/fileserver/uploads/incomplete?limit=100" \
|
|
| jq '[.[] | select(.leaseUntil != null)] | length'
|
|
curl -s -X POST "$ADMIN/internal/fileserver/uploads:cleanup?maxItems=500&maxBytes=10737418240"
|
|
```
|
|
|
|
If the deferrals are all `ACTIVE_WRITER_LEASE`, the backlog resolves itself as those uploads expire.
|
|
Never delete staging content to clear a backlog: an upload that is mid-flight will corrupt.
|
|
|
|
## Namespace quota drift
|
|
|
|
**Signal** — a namespace's usage (`fileserver.quota` rejections, or the drift query below) disagrees
|
|
with the bytes its files actually hold. The known cause is history: before the 2026-11 maturity
|
|
review, `move` relocated a file without moving its quota. The bytes of every file moved before that
|
|
fix are still booked on the namespace it left, and deleting such a file reclaims from the namespace
|
|
it now lives in: the reclaim draws down the bookings of that namespace's other files, newest first,
|
|
or stops at zero, and the old booking stays behind either way. A derived
|
|
service that moved files across namespaces before taking the fix has this drift; one that never did
|
|
should see none.
|
|
|
|
What the ledger must equal: quota is committed when a file is published and reclaimed when cleanup
|
|
deletes its content, so a namespace's committed bytes are the sizes of its published files that are
|
|
not yet `DELETED`. The query compares the two and changes nothing.
|
|
|
|
```sql
|
|
WITH booked AS (
|
|
SELECT scope_value AS namespace, sum(committed_bytes) AS booked_bytes
|
|
FROM fs_quota_reservation
|
|
WHERE scope_type = 'namespace' AND status = 'COMMITTED'
|
|
GROUP BY scope_value),
|
|
held AS (
|
|
SELECT namespace, sum(actual_size) AS held_bytes
|
|
FROM fs_file
|
|
WHERE published_at IS NOT NULL AND state <> 'DELETED'
|
|
GROUP BY namespace)
|
|
SELECT coalesce(b.namespace, h.namespace) AS namespace,
|
|
coalesce(b.booked_bytes, 0) AS booked_bytes,
|
|
coalesce(h.held_bytes, 0) AS held_bytes,
|
|
coalesce(b.booked_bytes, 0) - coalesce(h.held_bytes, 0) AS drift_bytes
|
|
FROM booked b
|
|
FULL JOIN held h ON h.namespace = b.namespace
|
|
WHERE coalesce(b.booked_bytes, 0) <> coalesce(h.held_bytes, 0);
|
|
```
|
|
|
|
A positive `drift_bytes` is booked capacity no file holds (the namespace a file left); a negative one
|
|
is held content nothing booked (the namespace it arrived in). While uploads, copies, moves or
|
|
cleanup run, a row can show a drift that only means work in flight, so judge drift with the
|
|
fileserver's writes stopped.
|
|
|
|
To correct a namespace, stop every fileserver write to it first — uploads, copy, move, delete, and
|
|
the cleanup worker (`APP_FILESERVER_PLATFORM_CLEANUP_ENABLED=false`) — because the statement below
|
|
recomputes from `fs_file` and a publish or reclaim that lands between its two steps would be lost.
|
|
It zeroes the namespace's committed rows and books the recomputed total as one row, shaped the way a
|
|
publish leaves one (`COMMITTED`, `reserved_bytes` 0, the bytes in `committed_bytes`); the per-row
|
|
split has no meaning once reclaims have drawn rows down, and enforcement only ever reads the sum.
|
|
`RESERVED` rows (uploads in progress) are left alone.
|
|
|
|
```sql
|
|
BEGIN;
|
|
SET LOCAL lock_timeout = '5s';
|
|
UPDATE fs_quota_reservation
|
|
SET committed_bytes = 0, version = version + 1, updated_at = now()
|
|
WHERE scope_type = 'namespace' AND scope_value = :'namespace'
|
|
AND status = 'COMMITTED' AND committed_bytes > 0;
|
|
INSERT INTO fs_quota_reservation (
|
|
reservation_id, scope_type, scope_value, reserved_bytes, committed_bytes,
|
|
expires_at, status, version, created_at, updated_at)
|
|
SELECT gen_random_uuid(), 'namespace', :'namespace', 0, held,
|
|
now(), 'COMMITTED', 0, now(), now()
|
|
FROM (SELECT coalesce(sum(actual_size), 0) AS held
|
|
FROM fs_file
|
|
WHERE namespace = :'namespace'
|
|
AND published_at IS NOT NULL AND state <> 'DELETED') AS recomputed
|
|
WHERE held > 0;
|
|
COMMIT;
|
|
```
|
|
|
|
Run it with `psql -v namespace=<name>`, then run the drift query again: the namespace must be gone
|
|
from its result before writes resume.
|