Files
clean-architecture-backend-…/docs/fileserver/operations.md
T
DongHyeonka f40c03c6d3 fix: 3차 확인 리뷰의 Minor 19건을 고친다
문제: 3차 수정의 확인 리뷰가 새 Important 없이 Minor 19건을 남겼다.
동작에 닿는 것이 셋이다. JDK read-idle watchdog이 공용 ForkJoinPool에서
돌아 그 풀이 바쁘면 300ms timeout이 6초 가까이 늦었다. 붙여 쓴
-x<task>를 verifyStrictLaneStages가 읽지 않아 lane을 뺀 workflow가
통과했다. 사용자 전송이 중간 클래스를 거쳐 @SystemCaller bean을 부르면
ArchUnit 규칙을 지났다. 잠금 대기 검사기는 FK·trigger·REINDEX와
lock_timeout = 0을 못 봤고, 쿼터 보정 SQL은 코드와 다른 모양의 행을
넣었다.

방법: 영역별 4묶음이 맡은 Minor만 test-driven-development 절차로
고쳤다(재리뷰 README 12절).
- watchdog은 전용 데몬 스레드 하나에서 돌고 스트림을 닫으면 작업을
  취소한다. read당 비용은 그대로다(21–24ns).
- 사용자 전송 → @SystemCaller는 직접·간접 의존 모두 거부한다. 리뷰가
  권한 직접 의존 검사는 중간 클래스 하나로 다시 열려서 버렸다.
- 검사기를 보강하면서 새로 걸린 이미 배포된 마이그레이션 5개는 내용을
  바꾸지 않고 동결 목록에 더했다(main과 blob이 같다).
- 비밀 목록 bean은 규칙보다 먼저 만들어 실제 클래스로 검사한다.

위험·한계: 회복 필요 예외 메시지 끝에 scope digest가 붙고 기동 실패
문구가 바뀌어, 문구 전체에 맞추던 로그 매처는 고쳐야 한다. 코드가 든
Redis 계정 참조의 fixture- 검사는 문서로만 막았다(코드로 막으려면 새
의존 edge가 필요하다). failure-retention ≥ replay-ttl 기동 검사는 넣지
않았다. -mx 같은 짧은 플래그 묶음과 graphql 전송 fixture는 남았다.
spotbugsTestFixtures(게이트 밖)의 정규식 경고가 15건으로 늘었다.

검증: ci, 두 빌드의 verifyStrictLaneStages --dry-run, mainCheck,
:integrationCheck, -p optional-platforms ci integrationCheck
performanceCheck, :performanceCheck, :releaseCheck(-PgitRevision),
verify-compose-profile-contracts(18 lane) 첫 실행 통과. watchdog은
공용 풀을 채운 상태에서 300ms 설정이 300ms에 끝나는 것을 쟀다.

Refs: docs/reviews/2026-11-03-rereview/README.md 11·12절, round3/
2026-11-04 03:00:21 +09:00

200 lines
8.7 KiB
Markdown

# Fileserver runbooks
Each runbook names the exact metric that fires it and the exact command that resolves it. A runbook
whose trigger is "someone noticed" is not actionable, so every one below starts from a signal.
## Storage full
**Signal** — `fileserver.quota{result="rejected"}` rising, or `507` responses appearing.
Storage capacity is exhausted or the high-water guard tripped. Uploads are rejected before any bytes
are written, so nothing is corrupt; the system is refusing work it cannot complete.
```bash
curl -s $ADMIN/internal/fileserver/storage-health | jq '.usedFraction, .usableBytes'
curl -s -X POST "$ADMIN/internal/fileserver/uploads:cleanup?maxItems=500&maxBytes=10737418240"
curl -s "$ADMIN/internal/fileserver/orphans?limit=200" | jq '[.[].sizeBytes] | add'
```
Drain the cleanup backlog first — it reclaims space the system already knows is dead. Only then
consider an orphan reconcile, and start with a dry run.
## Orphan growth
**Signal** — `fileserver.cleanup{result="skipped"}` climbing, or the orphan scan returning more
objects each run.
Physical objects exist with no metadata record pointing at them. This is not immediately dangerous —
nothing serves them — but it consumes capacity indefinitely.
```bash
# Always look first. A reconcile without dryRun=false is a plan, not an action.
curl -s -X POST "$ADMIN/internal/fileserver/orphans:reconcile" \
-H 'content-type: application/json' -d '{"limit":100}' | jq '.candidates'
# Apply only the fingerprints you were just shown.
curl -s -X POST "$ADMIN/internal/fileserver/orphans:reconcile" \
-H 'content-type: application/json' \
-d '{"dryRun":false,"limit":100,"maxBytes":1073741824,
"expectedFingerprints":["<from the dry run>"],"reasonCode":"ORPHAN_GROWTH_RUNBOOK"}'
```
Echoing the fingerprints is the safety property: an object that changed between the scan and the
apply is skipped rather than deleted.
## Verification backlog
**Signal** — `fileserver.verification.queue{age_bucket="old"}` non-zero, or files sitting in
VERIFYING.
A verifier is slow or unavailable. Files stay non-public, which is the correct failure direction: a
`RETRY` verdict never becomes an `ACCEPT`.
```bash
curl -s $ADMIN/internal/fileserver/capabilities | jq '.storageType'
# Once the verifier is healthy, quarantined files can be re-examined individually.
curl -s -X POST "$ADMIN/internal/fileserver/files/$FILE_ID:reverify"
```
Do not clear the backlog by disabling verification. A file that reached READY without an accepting
verdict cannot be distinguished later from one that was verified.
## NFS ambiguity
**Signal** — problem documents carrying `"ambiguous": true`, or
`fileserver.transfer.interruption{reason="stale_handle"}`.
An operation's outcome could not be determined: the response was lost after the write or rename may
have landed. These are never retried automatically.
```bash
# The recovery queue holds the files awaiting a decision.
curl -s "$ADMIN/internal/fileserver/uploads/incomplete?limit=100" | jq
```
Reconciliation compares the physical size and digest against the record and only confirms READY when
all four of key, size, digest, and version agree. Anything short of that is reported, never guessed.
## PVC remount
**Signal** — startup failure naming "atomic move", "same file store", or "not writable".
The volume was remounted somewhere the probe can no longer prove a required capability. The
application refuses traffic rather than serving from storage it cannot publish to atomically.
```bash
kubectl apply -f infra/fileserver/kubernetes/pvc-certification-job.yaml
kubectl logs job/fileserver-pvc-certification
```
Compare the printed tuple with the certified one in `docs/fileserver/storage-certification.md`. A
mismatch in CSI driver, StorageClass, access mode, or mount options is the cause; the certification
does not carry across it.
## Nginx delegation failure
**Signal** — `fileserver.download.delegation{delegated="true"}` with client-visible `404`s.
The internal location is misconfigured, so the proxy cannot resolve the redirect it was handed.
```bash
# The internal prefix must resolve to the content root and must be marked `internal`.
grep -A5 '__files' infra/fileserver/nginx/nginx.conf
curl -s $ADMIN/internal/fileserver/capabilities | jq '.capabilities.delegatedDownload'
```
Turning delegation off is a safe immediate mitigation: the application serves the transfer itself,
slower but correct.
```bash
app.fileserver-platform.nginx.enabled=false
```
## Cleanup backlog
**Signal** — `fileserver.cleanup{result="deferred"}` rising, or reclaimed bytes flat while deletes
continue.
Items are being deferred faster than they drain. The usual cause is an active writer lease still
holding staging objects, which is correct behaviour, not a fault.
```bash
curl -s "$ADMIN/internal/fileserver/uploads/incomplete?limit=100" \
| jq '[.[] | select(.leaseUntil != null)] | length'
curl -s -X POST "$ADMIN/internal/fileserver/uploads:cleanup?maxItems=500&maxBytes=10737418240"
```
If the deferrals are all `ACTIVE_WRITER_LEASE`, the backlog resolves itself as those uploads expire.
Never delete staging content to clear a backlog: an upload that is mid-flight will corrupt.
## Namespace quota drift
**Signal** — a namespace's usage (`fileserver.quota` rejections, or the drift query below) disagrees
with the bytes its files actually hold. The known cause is history: before the 2026-11 maturity
review, `move` relocated a file without moving its quota. The bytes of every file moved before that
fix are still booked on the namespace it left, and deleting such a file reclaims from the namespace
it now lives in: the reclaim draws down the bookings of that namespace's other files, newest first,
or stops at zero, and the old booking stays behind either way. A derived
service that moved files across namespaces before taking the fix has this drift; one that never did
should see none.
What the ledger must equal: quota is committed when a file is published and reclaimed when cleanup
deletes its content, so a namespace's committed bytes are the sizes of its published files that are
not yet `DELETED`. The query compares the two and changes nothing.
```sql
WITH booked AS (
SELECT scope_value AS namespace, sum(committed_bytes) AS booked_bytes
FROM fs_quota_reservation
WHERE scope_type = 'namespace' AND status = 'COMMITTED'
GROUP BY scope_value),
held AS (
SELECT namespace, sum(actual_size) AS held_bytes
FROM fs_file
WHERE published_at IS NOT NULL AND state <> 'DELETED'
GROUP BY namespace)
SELECT coalesce(b.namespace, h.namespace) AS namespace,
coalesce(b.booked_bytes, 0) AS booked_bytes,
coalesce(h.held_bytes, 0) AS held_bytes,
coalesce(b.booked_bytes, 0) - coalesce(h.held_bytes, 0) AS drift_bytes
FROM booked b
FULL JOIN held h ON h.namespace = b.namespace
WHERE coalesce(b.booked_bytes, 0) <> coalesce(h.held_bytes, 0);
```
A positive `drift_bytes` is booked capacity no file holds (the namespace a file left); a negative one
is held content nothing booked (the namespace it arrived in). While uploads, copies, moves or
cleanup run, a row can show a drift that only means work in flight, so judge drift with the
fileserver's writes stopped.
To correct a namespace, stop every fileserver write to it first — uploads, copy, move, delete, and
the cleanup worker (`APP_FILESERVER_PLATFORM_CLEANUP_ENABLED=false`) — because the statement below
recomputes from `fs_file` and a publish or reclaim that lands between its two steps would be lost.
It zeroes the namespace's committed rows and books the recomputed total as one row, shaped the way a
publish leaves one (`COMMITTED`, `reserved_bytes` 0, the bytes in `committed_bytes`); the per-row
split has no meaning once reclaims have drawn rows down, and enforcement only ever reads the sum.
`RESERVED` rows (uploads in progress) are left alone.
```sql
BEGIN;
SET LOCAL lock_timeout = '5s';
UPDATE fs_quota_reservation
SET committed_bytes = 0, version = version + 1, updated_at = now()
WHERE scope_type = 'namespace' AND scope_value = :'namespace'
AND status = 'COMMITTED' AND committed_bytes > 0;
INSERT INTO fs_quota_reservation (
reservation_id, scope_type, scope_value, reserved_bytes, committed_bytes,
expires_at, status, version, created_at, updated_at)
SELECT gen_random_uuid(), 'namespace', :'namespace', 0, held,
now(), 'COMMITTED', 0, now(), now()
FROM (SELECT coalesce(sum(actual_size), 0) AS held
FROM fs_file
WHERE namespace = :'namespace'
AND published_at IS NOT NULL AND state <> 'DELETED') AS recomputed
WHERE held > 0;
COMMIT;
```
Run it with `psql -v namespace=<name>`, then run the drift query again: the namespace must be gone
from its result before writes resume.