문제: 재리뷰 수정 커밋(bbe944bd)만 다시 본 2차 재리뷰가 Important 7건,
Minor 22건을 찾았다. 결과 불명 503 뒤 같은 키를 다시 보내면 Redis
store가 lease 지난 실행을 넘겨받아 작업을 다시 돌렸다. replay TTL이
lease보다 짧으면 lease 도중 key가 사라져 같은 키가 두 번 실행됐다.
@SystemInvoked 규칙이 매뉴얼 09장의 이벤트 소비 경로까지 막았다. V22가
outbox_event에 가장 강한 잠금을 lock_timeout 없이 걸었다. prod의
fixture- 거부가 HOSTNAME 같은 비밀 아닌 값까지 막았다. 데모 허용 목록의
test·demo 프로필로 sample이 뜨지 않았다. 기동 rule bean의 @Value 필드가
조용히 null이 됐다.
방법: 영역별 4묶음이 test-driven-development 절차로 실패 테스트를
먼저 쓰고 고쳤다. 결정은 재리뷰 README 9.1절이다.
- 두 멱등성 store가 같은 계약을 따른다. lease가 지난 실행 기록은 다시
돌리지 않고 포기로 표시해 500 RECOVERY_REQUIRED로 답한다. 기록 수명은
lease + replay TTL이고, replay-ttl < processing-lease면 기동이 실패한다.
- 사용자 transport만 @SystemInvoked use case에 닿지 못한다. 메시지
handler·scheduler는 새 표시 @SystemCaller(reason)를 달고 use case를
생성자에서 new로 만든다. 매뉴얼 경로를 테스트로 고정했다.
- 아직 어디에도 적용되지 않은 마이그레이션(V20·V22·알림 V11~V13·sample
V8_1·grpc V002·V003)에 lock_timeout 5s를 넣고, 텍스트 검사기와 실 PG
lane을 더했다. 이미 배포된 24개는 동결 목록으로 둔다.
- fixture- 검사는 비밀로 분류된 값만 본다. sample은 local·test·demo에서
object storage를 끈다. 기동 rule bean에 주입 멤버가 있으면 모든
프로필에서 기동이 실패한다. 리뷰어가 제안한 "static @Bean 선언 강제"는
문제없는 등록을 막고 @Value 필드는 통과시켜서 버렸다.
- 이 수정 전에 move된 파일의 쿼터 차이를 찾는 쿼리와 재계산 SQL을
docs/fileserver/operations.md에 적었다.
위험·한계: lease 뒤 늦게 끝난 첫 시도의 결과는 기록되지 않는다(그 요청도
500). @SystemCaller가 없는 메시지 handler, 주입 멤버가 있는 기동 rule,
working-directory 없이 ./gradlew를 부르는 workflow 명령은 파생 서비스에서
새로 실패하거나 세지지 않는다(업그레이드 안내 B-14~16·B-36·B-39).
spotbugsTestFixtures는 게이트 밖이고 기준 커밋에서도 실패하며, 새 검사기의
정규식 경고 8건이 더해졌다. 실제 CI runner 실행은 확인하지 못했다(Gitea
repo의 Actions가 꺼져 있다).
검증: ci, 두 빌드의 verifyStrictLaneStages --dry-run, mainCheck,
:integrationCheck, -p optional-platforms ci integrationCheck
performanceCheck, :performanceCheck, :releaseCheck(-PgitRevision),
verify-compose-profile-contracts(18 lane) 첫 실행 통과. 쿼터 재계산
SQL은 실 PostgreSQL 16에서 차이 2행 → 0행을 확인했다.
Refs: docs/reviews/2026-11-03-rereview/README.md 9·9.1·10절
BREAKING CHANGE: replay-ttl < processing-lease 설정은 기동 실패, ledger
release가 claim token을 받는다, HttpBodyReadException(IOException 하위)이
본문 실패를 싣는다, 기동 rule·비밀 목록 bean의 주입 멤버 금지,
verifyStrictLaneStages가 workflow 명령의 working-directory로 빌드를 정한다.
198 lines
8.6 KiB
Markdown
198 lines
8.6 KiB
Markdown
# Fileserver runbooks
|
|
|
|
Each runbook names the exact metric that fires it and the exact command that resolves it. A runbook
|
|
whose trigger is "someone noticed" is not actionable, so every one below starts from a signal.
|
|
|
|
## Storage full
|
|
|
|
**Signal** — `fileserver.quota{result="rejected"}` rising, or `507` responses appearing.
|
|
|
|
Storage capacity is exhausted or the high-water guard tripped. Uploads are rejected before any bytes
|
|
are written, so nothing is corrupt; the system is refusing work it cannot complete.
|
|
|
|
```bash
|
|
curl -s $ADMIN/internal/fileserver/storage-health | jq '.usedFraction, .usableBytes'
|
|
curl -s -X POST "$ADMIN/internal/fileserver/uploads:cleanup?maxItems=500&maxBytes=10737418240"
|
|
curl -s "$ADMIN/internal/fileserver/orphans?limit=200" | jq '[.[].sizeBytes] | add'
|
|
```
|
|
|
|
Drain the cleanup backlog first — it reclaims space the system already knows is dead. Only then
|
|
consider an orphan reconcile, and start with a dry run.
|
|
|
|
## Orphan growth
|
|
|
|
**Signal** — `fileserver.cleanup{result="skipped"}` climbing, or the orphan scan returning more
|
|
objects each run.
|
|
|
|
Physical objects exist with no metadata record pointing at them. This is not immediately dangerous —
|
|
nothing serves them — but it consumes capacity indefinitely.
|
|
|
|
```bash
|
|
# Always look first. A reconcile without dryRun=false is a plan, not an action.
|
|
curl -s -X POST "$ADMIN/internal/fileserver/orphans:reconcile" \
|
|
-H 'content-type: application/json' -d '{"limit":100}' | jq '.candidates'
|
|
|
|
# Apply only the fingerprints you were just shown.
|
|
curl -s -X POST "$ADMIN/internal/fileserver/orphans:reconcile" \
|
|
-H 'content-type: application/json' \
|
|
-d '{"dryRun":false,"limit":100,"maxBytes":1073741824,
|
|
"expectedFingerprints":["<from the dry run>"],"reasonCode":"ORPHAN_GROWTH_RUNBOOK"}'
|
|
```
|
|
|
|
Echoing the fingerprints is the safety property: an object that changed between the scan and the
|
|
apply is skipped rather than deleted.
|
|
|
|
## Verification backlog
|
|
|
|
**Signal** — `fileserver.verification.queue{age_bucket="old"}` non-zero, or files sitting in
|
|
VERIFYING.
|
|
|
|
A verifier is slow or unavailable. Files stay non-public, which is the correct failure direction: a
|
|
`RETRY` verdict never becomes an `ACCEPT`.
|
|
|
|
```bash
|
|
curl -s $ADMIN/internal/fileserver/capabilities | jq '.storageType'
|
|
# Once the verifier is healthy, quarantined files can be re-examined individually.
|
|
curl -s -X POST "$ADMIN/internal/fileserver/files/$FILE_ID:reverify"
|
|
```
|
|
|
|
Do not clear the backlog by disabling verification. A file that reached READY without an accepting
|
|
verdict cannot be distinguished later from one that was verified.
|
|
|
|
## NFS ambiguity
|
|
|
|
**Signal** — problem documents carrying `"ambiguous": true`, or
|
|
`fileserver.transfer.interruption{reason="stale_handle"}`.
|
|
|
|
An operation's outcome could not be determined: the response was lost after the write or rename may
|
|
have landed. These are never retried automatically.
|
|
|
|
```bash
|
|
# The recovery queue holds the files awaiting a decision.
|
|
curl -s "$ADMIN/internal/fileserver/uploads/incomplete?limit=100" | jq
|
|
```
|
|
|
|
Reconciliation compares the physical size and digest against the record and only confirms READY when
|
|
all four of key, size, digest, and version agree. Anything short of that is reported, never guessed.
|
|
|
|
## PVC remount
|
|
|
|
**Signal** — startup failure naming "atomic move", "same file store", or "not writable".
|
|
|
|
The volume was remounted somewhere the probe can no longer prove a required capability. The
|
|
application refuses traffic rather than serving from storage it cannot publish to atomically.
|
|
|
|
```bash
|
|
kubectl apply -f infra/fileserver/kubernetes/pvc-certification-job.yaml
|
|
kubectl logs job/fileserver-pvc-certification
|
|
```
|
|
|
|
Compare the printed tuple with the certified one in `docs/fileserver/storage-certification.md`. A
|
|
mismatch in CSI driver, StorageClass, access mode, or mount options is the cause; the certification
|
|
does not carry across it.
|
|
|
|
## Nginx delegation failure
|
|
|
|
**Signal** — `fileserver.download.delegation{delegated="true"}` with client-visible `404`s.
|
|
|
|
The internal location is misconfigured, so the proxy cannot resolve the redirect it was handed.
|
|
|
|
```bash
|
|
# The internal prefix must resolve to the content root and must be marked `internal`.
|
|
grep -A5 '__files' infra/fileserver/nginx/nginx.conf
|
|
curl -s $ADMIN/internal/fileserver/capabilities | jq '.capabilities.delegatedDownload'
|
|
```
|
|
|
|
Turning delegation off is a safe immediate mitigation: the application serves the transfer itself,
|
|
slower but correct.
|
|
|
|
```bash
|
|
app.fileserver-platform.nginx.enabled=false
|
|
```
|
|
|
|
## Cleanup backlog
|
|
|
|
**Signal** — `fileserver.cleanup{result="deferred"}` rising, or reclaimed bytes flat while deletes
|
|
continue.
|
|
|
|
Items are being deferred faster than they drain. The usual cause is an active writer lease still
|
|
holding staging objects, which is correct behaviour, not a fault.
|
|
|
|
```bash
|
|
curl -s "$ADMIN/internal/fileserver/uploads/incomplete?limit=100" \
|
|
| jq '[.[] | select(.leaseUntil != null)] | length'
|
|
curl -s -X POST "$ADMIN/internal/fileserver/uploads:cleanup?maxItems=500&maxBytes=10737418240"
|
|
```
|
|
|
|
If the deferrals are all `ACTIVE_WRITER_LEASE`, the backlog resolves itself as those uploads expire.
|
|
Never delete staging content to clear a backlog: an upload that is mid-flight will corrupt.
|
|
|
|
## Namespace quota drift
|
|
|
|
**Signal** — a namespace's usage (`fileserver.quota` rejections, or the drift query below) disagrees
|
|
with the bytes its files actually hold. The known cause is history: before the 2026-11 maturity
|
|
review, `move` relocated a file without moving its quota. The bytes of every file moved before that
|
|
fix are still booked on the namespace it left, and deleting such a file reclaims from the namespace
|
|
it now lives in, where the reclaim stops at zero and leaves the old booking behind. A derived
|
|
service that moved files across namespaces before taking the fix has this drift; one that never did
|
|
should see none.
|
|
|
|
What the ledger must equal: quota is committed when a file is published and reclaimed when cleanup
|
|
deletes its content, so a namespace's committed bytes are the sizes of its published files that are
|
|
not yet `DELETED`. The query compares the two and changes nothing.
|
|
|
|
```sql
|
|
WITH booked AS (
|
|
SELECT scope_value AS namespace, sum(committed_bytes) AS booked_bytes
|
|
FROM fs_quota_reservation
|
|
WHERE scope_type = 'namespace' AND status = 'COMMITTED'
|
|
GROUP BY scope_value),
|
|
held AS (
|
|
SELECT namespace, sum(actual_size) AS held_bytes
|
|
FROM fs_file
|
|
WHERE published_at IS NOT NULL AND state <> 'DELETED'
|
|
GROUP BY namespace)
|
|
SELECT coalesce(b.namespace, h.namespace) AS namespace,
|
|
coalesce(b.booked_bytes, 0) AS booked_bytes,
|
|
coalesce(h.held_bytes, 0) AS held_bytes,
|
|
coalesce(b.booked_bytes, 0) - coalesce(h.held_bytes, 0) AS drift_bytes
|
|
FROM booked b
|
|
FULL JOIN held h ON h.namespace = b.namespace
|
|
WHERE coalesce(b.booked_bytes, 0) <> coalesce(h.held_bytes, 0);
|
|
```
|
|
|
|
A positive `drift_bytes` is booked capacity no file holds (the namespace a file left); a negative one
|
|
is held content nothing booked (the namespace it arrived in). While uploads, copies, moves or
|
|
cleanup run, a row can show a drift that only means work in flight, so judge drift with the
|
|
fileserver's writes stopped.
|
|
|
|
To correct a namespace, stop every fileserver write to it first — uploads, copy, move, delete, and
|
|
the cleanup worker (`APP_FILESERVER_PLATFORM_CLEANUP_ENABLED=false`) — because the statement below
|
|
recomputes from `fs_file` and a publish or reclaim that lands between its two steps would be lost.
|
|
It zeroes the namespace's committed rows and books the recomputed total as one row; the per-row
|
|
split has no meaning once reclaims have drawn rows down, and enforcement only ever reads the sum.
|
|
`RESERVED` rows (uploads in progress) are left alone.
|
|
|
|
```sql
|
|
BEGIN;
|
|
SET LOCAL lock_timeout = '5s';
|
|
UPDATE fs_quota_reservation
|
|
SET committed_bytes = 0, version = version + 1, updated_at = now()
|
|
WHERE scope_type = 'namespace' AND scope_value = :'namespace'
|
|
AND status = 'COMMITTED' AND committed_bytes > 0;
|
|
INSERT INTO fs_quota_reservation (
|
|
reservation_id, scope_type, scope_value, reserved_bytes, committed_bytes,
|
|
expires_at, status, version, created_at, updated_at)
|
|
SELECT gen_random_uuid(), 'namespace', :'namespace', held, held,
|
|
now(), 'COMMITTED', 0, now(), now()
|
|
FROM (SELECT coalesce(sum(actual_size), 0) AS held
|
|
FROM fs_file
|
|
WHERE namespace = :'namespace'
|
|
AND published_at IS NOT NULL AND state <> 'DELETED') AS recomputed
|
|
WHERE held > 0;
|
|
COMMIT;
|
|
```
|
|
|
|
Run it with `psql -v namespace=<name>`, then run the drift query again: the namespace must be gone
|
|
from its result before writes resume.
|