Files
clean-architecture-backend-…/docs/fileserver/operations.md
T
DongHyeonka 1885bf8da7 fix!: 2차 재리뷰가 찾은 결함 29건을 고친다
문제: 재리뷰 수정 커밋(bbe944bd)만 다시 본 2차 재리뷰가 Important 7건,
Minor 22건을 찾았다. 결과 불명 503 뒤 같은 키를 다시 보내면 Redis
store가 lease 지난 실행을 넘겨받아 작업을 다시 돌렸다. replay TTL이
lease보다 짧으면 lease 도중 key가 사라져 같은 키가 두 번 실행됐다.
@SystemInvoked 규칙이 매뉴얼 09장의 이벤트 소비 경로까지 막았다. V22가
outbox_event에 가장 강한 잠금을 lock_timeout 없이 걸었다. prod의
fixture- 거부가 HOSTNAME 같은 비밀 아닌 값까지 막았다. 데모 허용 목록의
test·demo 프로필로 sample이 뜨지 않았다. 기동 rule bean의 @Value 필드가
조용히 null이 됐다.

방법: 영역별 4묶음이 test-driven-development 절차로 실패 테스트를
먼저 쓰고 고쳤다. 결정은 재리뷰 README 9.1절이다.
- 두 멱등성 store가 같은 계약을 따른다. lease가 지난 실행 기록은 다시
  돌리지 않고 포기로 표시해 500 RECOVERY_REQUIRED로 답한다. 기록 수명은
  lease + replay TTL이고, replay-ttl < processing-lease면 기동이 실패한다.
- 사용자 transport만 @SystemInvoked use case에 닿지 못한다. 메시지
  handler·scheduler는 새 표시 @SystemCaller(reason)를 달고 use case를
  생성자에서 new로 만든다. 매뉴얼 경로를 테스트로 고정했다.
- 아직 어디에도 적용되지 않은 마이그레이션(V20·V22·알림 V11~V13·sample
  V8_1·grpc V002·V003)에 lock_timeout 5s를 넣고, 텍스트 검사기와 실 PG
  lane을 더했다. 이미 배포된 24개는 동결 목록으로 둔다.
- fixture- 검사는 비밀로 분류된 값만 본다. sample은 local·test·demo에서
  object storage를 끈다. 기동 rule bean에 주입 멤버가 있으면 모든
  프로필에서 기동이 실패한다. 리뷰어가 제안한 "static @Bean 선언 강제"는
  문제없는 등록을 막고 @Value 필드는 통과시켜서 버렸다.
- 이 수정 전에 move된 파일의 쿼터 차이를 찾는 쿼리와 재계산 SQL을
  docs/fileserver/operations.md에 적었다.

위험·한계: lease 뒤 늦게 끝난 첫 시도의 결과는 기록되지 않는다(그 요청도
500). @SystemCaller가 없는 메시지 handler, 주입 멤버가 있는 기동 rule,
working-directory 없이 ./gradlew를 부르는 workflow 명령은 파생 서비스에서
새로 실패하거나 세지지 않는다(업그레이드 안내 B-14~16·B-36·B-39).
spotbugsTestFixtures는 게이트 밖이고 기준 커밋에서도 실패하며, 새 검사기의
정규식 경고 8건이 더해졌다. 실제 CI runner 실행은 확인하지 못했다(Gitea
repo의 Actions가 꺼져 있다).

검증: ci, 두 빌드의 verifyStrictLaneStages --dry-run, mainCheck,
:integrationCheck, -p optional-platforms ci integrationCheck
performanceCheck, :performanceCheck, :releaseCheck(-PgitRevision),
verify-compose-profile-contracts(18 lane) 첫 실행 통과. 쿼터 재계산
SQL은 실 PostgreSQL 16에서 차이 2행 → 0행을 확인했다.

Refs: docs/reviews/2026-11-03-rereview/README.md 9·9.1·10절
BREAKING CHANGE: replay-ttl < processing-lease 설정은 기동 실패, ledger
release가 claim token을 받는다, HttpBodyReadException(IOException 하위)이
본문 실패를 싣는다, 기동 rule·비밀 목록 bean의 주입 멤버 금지,
verifyStrictLaneStages가 workflow 명령의 working-directory로 빌드를 정한다.
2026-11-04 01:47:50 +09:00

8.6 KiB

Fileserver runbooks

Each runbook names the exact metric that fires it and the exact command that resolves it. A runbook whose trigger is "someone noticed" is not actionable, so every one below starts from a signal.

Storage full

Signal — fileserver.quota{result="rejected"} rising, or 507 responses appearing.

Storage capacity is exhausted or the high-water guard tripped. Uploads are rejected before any bytes are written, so nothing is corrupt; the system is refusing work it cannot complete.

curl -s $ADMIN/internal/fileserver/storage-health | jq '.usedFraction, .usableBytes'
curl -s -X POST "$ADMIN/internal/fileserver/uploads:cleanup?maxItems=500&maxBytes=10737418240"
curl -s "$ADMIN/internal/fileserver/orphans?limit=200" | jq '[.[].sizeBytes] | add'

Drain the cleanup backlog first — it reclaims space the system already knows is dead. Only then consider an orphan reconcile, and start with a dry run.

Orphan growth

Signal — fileserver.cleanup{result="skipped"} climbing, or the orphan scan returning more objects each run.

Physical objects exist with no metadata record pointing at them. This is not immediately dangerous — nothing serves them — but it consumes capacity indefinitely.

# Always look first. A reconcile without dryRun=false is a plan, not an action.
curl -s -X POST "$ADMIN/internal/fileserver/orphans:reconcile" \
  -H 'content-type: application/json' -d '{"limit":100}' | jq '.candidates'

# Apply only the fingerprints you were just shown.
curl -s -X POST "$ADMIN/internal/fileserver/orphans:reconcile" \
  -H 'content-type: application/json' \
  -d '{"dryRun":false,"limit":100,"maxBytes":1073741824,
       "expectedFingerprints":["<from the dry run>"],"reasonCode":"ORPHAN_GROWTH_RUNBOOK"}'

Echoing the fingerprints is the safety property: an object that changed between the scan and the apply is skipped rather than deleted.

Verification backlog

Signal — fileserver.verification.queue{age_bucket="old"} non-zero, or files sitting in VERIFYING.

A verifier is slow or unavailable. Files stay non-public, which is the correct failure direction: a RETRY verdict never becomes an ACCEPT.

curl -s $ADMIN/internal/fileserver/capabilities | jq '.storageType'
# Once the verifier is healthy, quarantined files can be re-examined individually.
curl -s -X POST "$ADMIN/internal/fileserver/files/$FILE_ID:reverify"

Do not clear the backlog by disabling verification. A file that reached READY without an accepting verdict cannot be distinguished later from one that was verified.

NFS ambiguity

Signal — problem documents carrying "ambiguous": true, or fileserver.transfer.interruption{reason="stale_handle"}.

An operation's outcome could not be determined: the response was lost after the write or rename may have landed. These are never retried automatically.

# The recovery queue holds the files awaiting a decision.
curl -s "$ADMIN/internal/fileserver/uploads/incomplete?limit=100" | jq

Reconciliation compares the physical size and digest against the record and only confirms READY when all four of key, size, digest, and version agree. Anything short of that is reported, never guessed.

PVC remount

Signal — startup failure naming "atomic move", "same file store", or "not writable".

The volume was remounted somewhere the probe can no longer prove a required capability. The application refuses traffic rather than serving from storage it cannot publish to atomically.

kubectl apply -f infra/fileserver/kubernetes/pvc-certification-job.yaml
kubectl logs job/fileserver-pvc-certification

Compare the printed tuple with the certified one in docs/fileserver/storage-certification.md. A mismatch in CSI driver, StorageClass, access mode, or mount options is the cause; the certification does not carry across it.

Nginx delegation failure

Signal — fileserver.download.delegation{delegated="true"} with client-visible 404s.

The internal location is misconfigured, so the proxy cannot resolve the redirect it was handed.

# The internal prefix must resolve to the content root and must be marked `internal`.
grep -A5 '__files' infra/fileserver/nginx/nginx.conf
curl -s $ADMIN/internal/fileserver/capabilities | jq '.capabilities.delegatedDownload'

Turning delegation off is a safe immediate mitigation: the application serves the transfer itself, slower but correct.

app.fileserver-platform.nginx.enabled=false

Cleanup backlog

Signal — fileserver.cleanup{result="deferred"} rising, or reclaimed bytes flat while deletes continue.

Items are being deferred faster than they drain. The usual cause is an active writer lease still holding staging objects, which is correct behaviour, not a fault.

curl -s "$ADMIN/internal/fileserver/uploads/incomplete?limit=100" \
  | jq '[.[] | select(.leaseUntil != null)] | length'
curl -s -X POST "$ADMIN/internal/fileserver/uploads:cleanup?maxItems=500&maxBytes=10737418240"

If the deferrals are all ACTIVE_WRITER_LEASE, the backlog resolves itself as those uploads expire. Never delete staging content to clear a backlog: an upload that is mid-flight will corrupt.

Namespace quota drift

Signal — a namespace's usage (fileserver.quota rejections, or the drift query below) disagrees with the bytes its files actually hold. The known cause is history: before the 2026-11 maturity review, move relocated a file without moving its quota. The bytes of every file moved before that fix are still booked on the namespace it left, and deleting such a file reclaims from the namespace it now lives in, where the reclaim stops at zero and leaves the old booking behind. A derived service that moved files across namespaces before taking the fix has this drift; one that never did should see none.

What the ledger must equal: quota is committed when a file is published and reclaimed when cleanup deletes its content, so a namespace's committed bytes are the sizes of its published files that are not yet DELETED. The query compares the two and changes nothing.

WITH booked AS (
    SELECT scope_value AS namespace, sum(committed_bytes) AS booked_bytes
      FROM fs_quota_reservation
     WHERE scope_type = 'namespace' AND status = 'COMMITTED'
     GROUP BY scope_value),
held AS (
    SELECT namespace, sum(actual_size) AS held_bytes
      FROM fs_file
     WHERE published_at IS NOT NULL AND state <> 'DELETED'
     GROUP BY namespace)
SELECT coalesce(b.namespace, h.namespace)                         AS namespace,
       coalesce(b.booked_bytes, 0)                                AS booked_bytes,
       coalesce(h.held_bytes, 0)                                  AS held_bytes,
       coalesce(b.booked_bytes, 0) - coalesce(h.held_bytes, 0)    AS drift_bytes
  FROM booked b
  FULL JOIN held h ON h.namespace = b.namespace
 WHERE coalesce(b.booked_bytes, 0) <> coalesce(h.held_bytes, 0);

A positive drift_bytes is booked capacity no file holds (the namespace a file left); a negative one is held content nothing booked (the namespace it arrived in). While uploads, copies, moves or cleanup run, a row can show a drift that only means work in flight, so judge drift with the fileserver's writes stopped.

To correct a namespace, stop every fileserver write to it first — uploads, copy, move, delete, and the cleanup worker (APP_FILESERVER_PLATFORM_CLEANUP_ENABLED=false) — because the statement below recomputes from fs_file and a publish or reclaim that lands between its two steps would be lost. It zeroes the namespace's committed rows and books the recomputed total as one row; the per-row split has no meaning once reclaims have drawn rows down, and enforcement only ever reads the sum. RESERVED rows (uploads in progress) are left alone.

BEGIN;
SET LOCAL lock_timeout = '5s';
UPDATE fs_quota_reservation
   SET committed_bytes = 0, version = version + 1, updated_at = now()
 WHERE scope_type = 'namespace' AND scope_value = :'namespace'
   AND status = 'COMMITTED' AND committed_bytes > 0;
INSERT INTO fs_quota_reservation (
    reservation_id, scope_type, scope_value, reserved_bytes, committed_bytes,
    expires_at, status, version, created_at, updated_at)
SELECT gen_random_uuid(), 'namespace', :'namespace', held, held,
       now(), 'COMMITTED', 0, now(), now()
  FROM (SELECT coalesce(sum(actual_size), 0) AS held
          FROM fs_file
         WHERE namespace = :'namespace'
           AND published_at IS NOT NULL AND state <> 'DELETED') AS recomputed
 WHERE held > 0;
COMMIT;

Run it with psql -v namespace=<name>, then run the drift query again: the namespace must be gone from its result before writes resume.