Files
clean-architecture-backend-…/docs/fileserver/operations.md
T
DongHyeonka a025f244c7 refactor(repo)!: 쓰지 않는 파일과 운영 인프라 파일을 지운다
문제: infra/ 43개 중 12개는 코드·테스트·workflow·스크립트 어디서도
읽지 않는다(httpclient·jpa·messaging·notification의 toxiproxy compose와
README). nginx 설정, Kubernetes PVC 인증 job, DB 역할 SQL 셋은 운영
인프라라 개발 repo에 둘 것이 아니다(소유자 결정 2026-11-07). release.yml
의 fileserver-pvc-certification job은 그 매니페스트 문자열만 grep했다.

고른 방법:
- 15개를 지운다.
- 운영 조건만 그 기능 문서에 한국어로 남긴다.
  docs/fileserver/operations.md 「운영 환경이 지켜야 할 조건」(nginx
  internal location·alias·X-Accel-Redirect 헤더 제거·스트리밍, PVC
  ReadWriteOnce·fsGroup), storage-certification.md(인증 절차),
  docs/jpa/security.md 「역할과 권한」·「서버 설정」(역할 SQL과
  jpa/postgres README 내용), docs/messaging/security.md(인증서 SAN).
  httpclient TLS README 내용은 docs/httpclient/security.md에 이미 있다.
- release.yml job과 needs 줄, fileserver-pr.yml·pr-adapters.yml의 지운
  경로 필터를 지운다.
- fileserver 지원 표의 "Kubernetes ReadWriteOnce PVC — Limited" 행은
  근거 job이 없어져 「주장하지 않음」으로 옮긴다.
버린 대안: 인프라 파일을 project-infra로 옮기기(설계 Q6). 그 repo는
특정 프로젝트용이라 범용 템플릿 파일을 받을 자리가 없다.

위험·한계: 지원 표에서 PVC 주장이 빠진다. 파생 서비스가 그 파일을
배포에 썼다면 병합이 지운다(B-50). 클러스터 인증은 이제 배포하는 쪽이
문서 절차로 한다.

검증(desktop, 트리 749a3328, ~/t1-layout-logs/):
- :composition:fileserver-platform:check :app-bootstrap:check
  :app-bootstrap:documentContractTest rc=0 (repositoryContractTest,
  FileserverDocumentationCoverageTest 실행)
- :verifyStrictLaneStages --dry-run rc=0
- scripts/verify-compose-profile-contracts.sh rc=0 (18 lane)
- check-work-docs.py, 상대 링크 검사 rc=0

Refs: design-docs repo-structure/01-backend-layout.md 4.4, 6.3
BREAKING CHANGE: infra/fileserver/nginx, infra/fileserver/kubernetes, infra/jpa, infra/httpclient, infra/messaging, infra/notification/toxiproxy와 release.yml fileserver-pvc-certification job이 없어진다. docs/upgrade/2026-11-maturity-review.md B-50.
2026-11-07 12:22:47 +09:00

226 lines
11 KiB
Markdown

# Fileserver runbooks
Each runbook names the exact metric that fires it and the exact command that resolves it. A runbook
whose trigger is "someone noticed" is not actionable, so every one below starts from a signal.
## Storage full
**Signal** — `fileserver.quota{result="rejected"}` rising, or `507` responses appearing.
Storage capacity is exhausted or the high-water guard tripped. Uploads are rejected before any bytes
are written, so nothing is corrupt; the system is refusing work it cannot complete.
```bash
curl -s $ADMIN/internal/fileserver/storage-health | jq '.usedFraction, .usableBytes'
curl -s -X POST "$ADMIN/internal/fileserver/uploads:cleanup?maxItems=500&maxBytes=10737418240"
curl -s "$ADMIN/internal/fileserver/orphans?limit=200" | jq '[.[].sizeBytes] | add'
```
Drain the cleanup backlog first — it reclaims space the system already knows is dead. Only then
consider an orphan reconcile, and start with a dry run.
## Orphan growth
**Signal** — `fileserver.cleanup{result="skipped"}` climbing, or the orphan scan returning more
objects each run.
Physical objects exist with no metadata record pointing at them. This is not immediately dangerous —
nothing serves them — but it consumes capacity indefinitely.
```bash
# Always look first. A reconcile without dryRun=false is a plan, not an action.
curl -s -X POST "$ADMIN/internal/fileserver/orphans:reconcile" \
-H 'content-type: application/json' -d '{"limit":100}' | jq '.candidates'
# Apply only the fingerprints you were just shown.
curl -s -X POST "$ADMIN/internal/fileserver/orphans:reconcile" \
-H 'content-type: application/json' \
-d '{"dryRun":false,"limit":100,"maxBytes":1073741824,
"expectedFingerprints":["<from the dry run>"],"reasonCode":"ORPHAN_GROWTH_RUNBOOK"}'
```
Echoing the fingerprints is the safety property: an object that changed between the scan and the
apply is skipped rather than deleted.
## Verification backlog
**Signal** — `fileserver.verification.queue{age_bucket="old"}` non-zero, or files sitting in
VERIFYING.
A verifier is slow or unavailable. Files stay non-public, which is the correct failure direction: a
`RETRY` verdict never becomes an `ACCEPT`.
```bash
curl -s $ADMIN/internal/fileserver/capabilities | jq '.storageType'
# Once the verifier is healthy, quarantined files can be re-examined individually.
curl -s -X POST "$ADMIN/internal/fileserver/files/$FILE_ID:reverify"
```
Do not clear the backlog by disabling verification. A file that reached READY without an accepting
verdict cannot be distinguished later from one that was verified.
## NFS ambiguity
**Signal** — problem documents carrying `"ambiguous": true`, or
`fileserver.transfer.interruption{reason="stale_handle"}`.
An operation's outcome could not be determined: the response was lost after the write or rename may
have landed. These are never retried automatically.
```bash
# The recovery queue holds the files awaiting a decision.
curl -s "$ADMIN/internal/fileserver/uploads/incomplete?limit=100" | jq
```
Reconciliation compares the physical size and digest against the record and only confirms READY when
all four of key, size, digest, and version agree. Anything short of that is reported, never guessed.
## PVC remount
**Signal** — startup failure naming "atomic move", "same file store", or "not writable".
The volume was remounted somewhere the probe can no longer prove a required capability. The
application refuses traffic rather than serving from storage it cannot publish to atomically.
```bash
# staging과 content가 같은 파일 저장소(같은 장치 번호)인지 pod 안에서 본다
kubectl exec <pod> -- stat -c %d /var/lib/backend/files/staging /var/lib/backend/files/content
```
`docs/fileserver/storage-certification.md`의 인증 절차로 지금 volume의 묶음(CSI driver, StorageClass,
접근 모드, backend, mount 옵션)을 다시 적고, 인증한 묶음과 비교한다. 하나라도 다르면 그것이 원인이다.
인증은 그 차이를 넘어 옮겨지지 않는다.
## Nginx delegation failure
**Signal** — `fileserver.download.delegation{delegated="true"}` with client-visible `404`s.
The internal location is misconfigured, so the proxy cannot resolve the redirect it was handed.
프록시 설정의 내부 location(기본 `/__files/`)이 아래 「운영 환경이 지켜야 할 조건」을 지키는지 본다.
```bash
curl -s $ADMIN/internal/fileserver/capabilities | jq '.capabilities.delegatedDownload'
```
Turning delegation off is a safe immediate mitigation: the application serves the transfer itself,
slower but correct.
```bash
app.fileserver-platform.nginx.enabled=false
```
## 운영 환경이 지켜야 할 조건
이 repo는 프록시 설정 파일과 Kubernetes 매니페스트를 두지 않는다(2026-11-07 소유자 결정). 그 파일은 서비스를
배포하는 쪽의 인프라 repo가 갖는다. 그 파일이 지켜야 할 조건은 아래다.
**프런트 프록시(nginx `X-Accel-Redirect` 위임)** — `app.fileserver-platform.nginx.enabled=true`일 때
1. 내부 location(`app.fileserver-platform.nginx.internal-prefix`, 기본 `/__files/`)은 `internal`이다. 클라이언트가
그 경로를 직접 부르면 404이고 content 폴더에 닿지 않는다. 애플리케이션이 준 내부 redirect로만 닿는다.
2. 그 location의 `alias`는 저장 루트의 content 폴더다(기본 `/var/lib/backend/files/content/`). 애플리케이션은
절대 경로를 내보내지 않고 내부 접두사 아래 상대 URI만 내보낸다. 경로가 실제 위치가 되는 곳은 이 `alias`뿐이다.
3. 앱으로 넘기는 요청에서 클라이언트가 보낸 `X-Accel-Redirect` 헤더를 지운다. 남기면 호출자가 임의의 내부
객체를 가리킬 수 있다.
4. 업로드는 스트리밍으로 넘긴다(`proxy_request_buffering off`, 본문 크기 제한은 애플리케이션이 정하므로 프록시는
막지 않는다). 프록시가 큰 본문을 디스크에 버퍼링하면 쓰기가 두 번 일어난다.
5. 내부 location 응답에 `X-Content-Type-Options: nosniff`를 붙이고, 앱이 정한 `Content-Disposition`·
`Cache-Control`·`ETag`를 그대로 넘긴다. 올라온 내용이 자기 형식을 스스로 정하게 두지 않는다.
**Kubernetes 저장 volume**
1. PersistentVolumeClaim을 저장 루트(`/var/lib/backend/files`)에 마운트하고, 접근 모드는 `ReadWriteOnce`다.
`ReadWriteMany`(여러 노드의 동시 쓰기)는 인증하지 않았다(`support-matrix.md`).
2. pod `securityContext.fsGroup`을 이미지의 앱 그룹으로 둔다. 그래야 앱 계정이 volume에 쓸 수 있다.
3. 그 volume이 `storage-certification.md`의 인증 항목(같은 파일 저장소, 원자적 rename, `O_EXCL` 생성)을
통과해야 한다. 기동 때 `FileserverStartupValidator`도 같은 probe를 다시 돌린다.
## Cleanup backlog
**Signal** — `fileserver.cleanup{result="deferred"}` rising, or reclaimed bytes flat while deletes
continue.
Items are being deferred faster than they drain. The usual cause is an active writer lease still
holding staging objects, which is correct behaviour, not a fault.
```bash
curl -s "$ADMIN/internal/fileserver/uploads/incomplete?limit=100" \
| jq '[.[] | select(.leaseUntil != null)] | length'
curl -s -X POST "$ADMIN/internal/fileserver/uploads:cleanup?maxItems=500&maxBytes=10737418240"
```
If the deferrals are all `ACTIVE_WRITER_LEASE`, the backlog resolves itself as those uploads expire.
Never delete staging content to clear a backlog: an upload that is mid-flight will corrupt.
## Namespace quota drift
**Signal** — a namespace's usage (`fileserver.quota` rejections, or the drift query below) disagrees
with the bytes its files actually hold. The known cause is history: before the 2026-11 maturity
review, `move` relocated a file without moving its quota. The bytes of every file moved before that
fix are still booked on the namespace it left, and deleting such a file reclaims from the namespace
it now lives in: the reclaim draws down the bookings of that namespace's other files, newest first,
or stops at zero, and the old booking stays behind either way. A derived
service that moved files across namespaces before taking the fix has this drift; one that never did
should see none.
What the ledger must equal: quota is committed when a file is published and reclaimed when cleanup
deletes its content, so a namespace's committed bytes are the sizes of its published files that are
not yet `DELETED`. The query compares the two and changes nothing.
```sql
WITH booked AS (
SELECT scope_value AS namespace, sum(committed_bytes) AS booked_bytes
FROM fs_quota_reservation
WHERE scope_type = 'namespace' AND status = 'COMMITTED'
GROUP BY scope_value),
held AS (
SELECT namespace, sum(actual_size) AS held_bytes
FROM fs_file
WHERE published_at IS NOT NULL AND state <> 'DELETED'
GROUP BY namespace)
SELECT coalesce(b.namespace, h.namespace) AS namespace,
coalesce(b.booked_bytes, 0) AS booked_bytes,
coalesce(h.held_bytes, 0) AS held_bytes,
coalesce(b.booked_bytes, 0) - coalesce(h.held_bytes, 0) AS drift_bytes
FROM booked b
FULL JOIN held h ON h.namespace = b.namespace
WHERE coalesce(b.booked_bytes, 0) <> coalesce(h.held_bytes, 0);
```
A positive `drift_bytes` is booked capacity no file holds (the namespace a file left); a negative one
is held content nothing booked (the namespace it arrived in). While uploads, copies, moves or
cleanup run, a row can show a drift that only means work in flight, so judge drift with the
fileserver's writes stopped.
To correct a namespace, stop every fileserver write to it first — uploads, copy, move, delete, and
the cleanup worker (`APP_FILESERVER_PLATFORM_CLEANUP_ENABLED=false`) — because the statement below
recomputes from `fs_file` and a publish or reclaim that lands between its two steps would be lost.
It zeroes the namespace's committed rows and books the recomputed total as one row, shaped the way a
publish leaves one (`COMMITTED`, `reserved_bytes` 0, the bytes in `committed_bytes`); the per-row
split has no meaning once reclaims have drawn rows down, and enforcement only ever reads the sum.
`RESERVED` rows (uploads in progress) are left alone.
```sql
BEGIN;
SET LOCAL lock_timeout = '5s';
UPDATE fs_quota_reservation
SET committed_bytes = 0, version = version + 1, updated_at = now()
WHERE scope_type = 'namespace' AND scope_value = :'namespace'
AND status = 'COMMITTED' AND committed_bytes > 0;
INSERT INTO fs_quota_reservation (
reservation_id, scope_type, scope_value, reserved_bytes, committed_bytes,
expires_at, status, version, created_at, updated_at)
SELECT gen_random_uuid(), 'namespace', :'namespace', 0, held,
now(), 'COMMITTED', 0, now(), now()
FROM (SELECT coalesce(sum(actual_size), 0) AS held
FROM fs_file
WHERE namespace = :'namespace'
AND published_at IS NOT NULL AND state <> 'DELETED') AS recomputed
WHERE held > 0;
COMMIT;
```
Run it with `psql -v namespace=<name>`, then run the drift query again: the namespace must be gone
from its result before writes resume.