문제: infra/ 43개 중 12개는 코드·테스트·workflow·스크립트 어디서도
읽지 않는다(httpclient·jpa·messaging·notification의 toxiproxy compose와
README). nginx 설정, Kubernetes PVC 인증 job, DB 역할 SQL 셋은 운영
인프라라 개발 repo에 둘 것이 아니다(소유자 결정 2026-11-07). release.yml
의 fileserver-pvc-certification job은 그 매니페스트 문자열만 grep했다.
고른 방법:
- 15개를 지운다.
- 운영 조건만 그 기능 문서에 한국어로 남긴다.
docs/fileserver/operations.md 「운영 환경이 지켜야 할 조건」(nginx
internal location·alias·X-Accel-Redirect 헤더 제거·스트리밍, PVC
ReadWriteOnce·fsGroup), storage-certification.md(인증 절차),
docs/jpa/security.md 「역할과 권한」·「서버 설정」(역할 SQL과
jpa/postgres README 내용), docs/messaging/security.md(인증서 SAN).
httpclient TLS README 내용은 docs/httpclient/security.md에 이미 있다.
- release.yml job과 needs 줄, fileserver-pr.yml·pr-adapters.yml의 지운
경로 필터를 지운다.
- fileserver 지원 표의 "Kubernetes ReadWriteOnce PVC — Limited" 행은
근거 job이 없어져 「주장하지 않음」으로 옮긴다.
버린 대안: 인프라 파일을 project-infra로 옮기기(설계 Q6). 그 repo는
특정 프로젝트용이라 범용 템플릿 파일을 받을 자리가 없다.
위험·한계: 지원 표에서 PVC 주장이 빠진다. 파생 서비스가 그 파일을
배포에 썼다면 병합이 지운다(B-50). 클러스터 인증은 이제 배포하는 쪽이
문서 절차로 한다.
검증(desktop, 트리 749a3328, ~/t1-layout-logs/):
- :composition:fileserver-platform:check :app-bootstrap:check
:app-bootstrap:documentContractTest rc=0 (repositoryContractTest,
FileserverDocumentationCoverageTest 실행)
- :verifyStrictLaneStages --dry-run rc=0
- scripts/verify-compose-profile-contracts.sh rc=0 (18 lane)
- check-work-docs.py, 상대 링크 검사 rc=0
Refs: design-docs repo-structure/01-backend-layout.md 4.4, 6.3
BREAKING CHANGE: infra/fileserver/nginx, infra/fileserver/kubernetes, infra/jpa, infra/httpclient, infra/messaging, infra/notification/toxiproxy와 release.yml fileserver-pvc-certification job이 없어진다. docs/upgrade/2026-11-maturity-review.md B-50.
226 lines
11 KiB
Markdown
226 lines
11 KiB
Markdown
# Fileserver runbooks
|
|
|
|
Each runbook names the exact metric that fires it and the exact command that resolves it. A runbook
|
|
whose trigger is "someone noticed" is not actionable, so every one below starts from a signal.
|
|
|
|
## Storage full
|
|
|
|
**Signal** — `fileserver.quota{result="rejected"}` rising, or `507` responses appearing.
|
|
|
|
Storage capacity is exhausted or the high-water guard tripped. Uploads are rejected before any bytes
|
|
are written, so nothing is corrupt; the system is refusing work it cannot complete.
|
|
|
|
```bash
|
|
curl -s $ADMIN/internal/fileserver/storage-health | jq '.usedFraction, .usableBytes'
|
|
curl -s -X POST "$ADMIN/internal/fileserver/uploads:cleanup?maxItems=500&maxBytes=10737418240"
|
|
curl -s "$ADMIN/internal/fileserver/orphans?limit=200" | jq '[.[].sizeBytes] | add'
|
|
```
|
|
|
|
Drain the cleanup backlog first — it reclaims space the system already knows is dead. Only then
|
|
consider an orphan reconcile, and start with a dry run.
|
|
|
|
## Orphan growth
|
|
|
|
**Signal** — `fileserver.cleanup{result="skipped"}` climbing, or the orphan scan returning more
|
|
objects each run.
|
|
|
|
Physical objects exist with no metadata record pointing at them. This is not immediately dangerous —
|
|
nothing serves them — but it consumes capacity indefinitely.
|
|
|
|
```bash
|
|
# Always look first. A reconcile without dryRun=false is a plan, not an action.
|
|
curl -s -X POST "$ADMIN/internal/fileserver/orphans:reconcile" \
|
|
-H 'content-type: application/json' -d '{"limit":100}' | jq '.candidates'
|
|
|
|
# Apply only the fingerprints you were just shown.
|
|
curl -s -X POST "$ADMIN/internal/fileserver/orphans:reconcile" \
|
|
-H 'content-type: application/json' \
|
|
-d '{"dryRun":false,"limit":100,"maxBytes":1073741824,
|
|
"expectedFingerprints":["<from the dry run>"],"reasonCode":"ORPHAN_GROWTH_RUNBOOK"}'
|
|
```
|
|
|
|
Echoing the fingerprints is the safety property: an object that changed between the scan and the
|
|
apply is skipped rather than deleted.
|
|
|
|
## Verification backlog
|
|
|
|
**Signal** — `fileserver.verification.queue{age_bucket="old"}` non-zero, or files sitting in
|
|
VERIFYING.
|
|
|
|
A verifier is slow or unavailable. Files stay non-public, which is the correct failure direction: a
|
|
`RETRY` verdict never becomes an `ACCEPT`.
|
|
|
|
```bash
|
|
curl -s $ADMIN/internal/fileserver/capabilities | jq '.storageType'
|
|
# Once the verifier is healthy, quarantined files can be re-examined individually.
|
|
curl -s -X POST "$ADMIN/internal/fileserver/files/$FILE_ID:reverify"
|
|
```
|
|
|
|
Do not clear the backlog by disabling verification. A file that reached READY without an accepting
|
|
verdict cannot be distinguished later from one that was verified.
|
|
|
|
## NFS ambiguity
|
|
|
|
**Signal** — problem documents carrying `"ambiguous": true`, or
|
|
`fileserver.transfer.interruption{reason="stale_handle"}`.
|
|
|
|
An operation's outcome could not be determined: the response was lost after the write or rename may
|
|
have landed. These are never retried automatically.
|
|
|
|
```bash
|
|
# The recovery queue holds the files awaiting a decision.
|
|
curl -s "$ADMIN/internal/fileserver/uploads/incomplete?limit=100" | jq
|
|
```
|
|
|
|
Reconciliation compares the physical size and digest against the record and only confirms READY when
|
|
all four of key, size, digest, and version agree. Anything short of that is reported, never guessed.
|
|
|
|
## PVC remount
|
|
|
|
**Signal** — startup failure naming "atomic move", "same file store", or "not writable".
|
|
|
|
The volume was remounted somewhere the probe can no longer prove a required capability. The
|
|
application refuses traffic rather than serving from storage it cannot publish to atomically.
|
|
|
|
```bash
|
|
# staging과 content가 같은 파일 저장소(같은 장치 번호)인지 pod 안에서 본다
|
|
kubectl exec <pod> -- stat -c %d /var/lib/backend/files/staging /var/lib/backend/files/content
|
|
```
|
|
|
|
`docs/fileserver/storage-certification.md`의 인증 절차로 지금 volume의 묶음(CSI driver, StorageClass,
|
|
접근 모드, backend, mount 옵션)을 다시 적고, 인증한 묶음과 비교한다. 하나라도 다르면 그것이 원인이다.
|
|
인증은 그 차이를 넘어 옮겨지지 않는다.
|
|
|
|
## Nginx delegation failure
|
|
|
|
**Signal** — `fileserver.download.delegation{delegated="true"}` with client-visible `404`s.
|
|
|
|
The internal location is misconfigured, so the proxy cannot resolve the redirect it was handed.
|
|
|
|
프록시 설정의 내부 location(기본 `/__files/`)이 아래 「운영 환경이 지켜야 할 조건」을 지키는지 본다.
|
|
|
|
```bash
|
|
curl -s $ADMIN/internal/fileserver/capabilities | jq '.capabilities.delegatedDownload'
|
|
```
|
|
|
|
Turning delegation off is a safe immediate mitigation: the application serves the transfer itself,
|
|
slower but correct.
|
|
|
|
```bash
|
|
app.fileserver-platform.nginx.enabled=false
|
|
```
|
|
|
|
## 운영 환경이 지켜야 할 조건
|
|
|
|
이 repo는 프록시 설정 파일과 Kubernetes 매니페스트를 두지 않는다(2026-11-07 소유자 결정). 그 파일은 서비스를
|
|
배포하는 쪽의 인프라 repo가 갖는다. 그 파일이 지켜야 할 조건은 아래다.
|
|
|
|
**프런트 프록시(nginx `X-Accel-Redirect` 위임)** — `app.fileserver-platform.nginx.enabled=true`일 때
|
|
|
|
1. 내부 location(`app.fileserver-platform.nginx.internal-prefix`, 기본 `/__files/`)은 `internal`이다. 클라이언트가
|
|
그 경로를 직접 부르면 404이고 content 폴더에 닿지 않는다. 애플리케이션이 준 내부 redirect로만 닿는다.
|
|
2. 그 location의 `alias`는 저장 루트의 content 폴더다(기본 `/var/lib/backend/files/content/`). 애플리케이션은
|
|
절대 경로를 내보내지 않고 내부 접두사 아래 상대 URI만 내보낸다. 경로가 실제 위치가 되는 곳은 이 `alias`뿐이다.
|
|
3. 앱으로 넘기는 요청에서 클라이언트가 보낸 `X-Accel-Redirect` 헤더를 지운다. 남기면 호출자가 임의의 내부
|
|
객체를 가리킬 수 있다.
|
|
4. 업로드는 스트리밍으로 넘긴다(`proxy_request_buffering off`, 본문 크기 제한은 애플리케이션이 정하므로 프록시는
|
|
막지 않는다). 프록시가 큰 본문을 디스크에 버퍼링하면 쓰기가 두 번 일어난다.
|
|
5. 내부 location 응답에 `X-Content-Type-Options: nosniff`를 붙이고, 앱이 정한 `Content-Disposition`·
|
|
`Cache-Control`·`ETag`를 그대로 넘긴다. 올라온 내용이 자기 형식을 스스로 정하게 두지 않는다.
|
|
|
|
**Kubernetes 저장 volume**
|
|
|
|
1. PersistentVolumeClaim을 저장 루트(`/var/lib/backend/files`)에 마운트하고, 접근 모드는 `ReadWriteOnce`다.
|
|
`ReadWriteMany`(여러 노드의 동시 쓰기)는 인증하지 않았다(`support-matrix.md`).
|
|
2. pod `securityContext.fsGroup`을 이미지의 앱 그룹으로 둔다. 그래야 앱 계정이 volume에 쓸 수 있다.
|
|
3. 그 volume이 `storage-certification.md`의 인증 항목(같은 파일 저장소, 원자적 rename, `O_EXCL` 생성)을
|
|
통과해야 한다. 기동 때 `FileserverStartupValidator`도 같은 probe를 다시 돌린다.
|
|
|
|
## Cleanup backlog
|
|
|
|
**Signal** — `fileserver.cleanup{result="deferred"}` rising, or reclaimed bytes flat while deletes
|
|
continue.
|
|
|
|
Items are being deferred faster than they drain. The usual cause is an active writer lease still
|
|
holding staging objects, which is correct behaviour, not a fault.
|
|
|
|
```bash
|
|
curl -s "$ADMIN/internal/fileserver/uploads/incomplete?limit=100" \
|
|
| jq '[.[] | select(.leaseUntil != null)] | length'
|
|
curl -s -X POST "$ADMIN/internal/fileserver/uploads:cleanup?maxItems=500&maxBytes=10737418240"
|
|
```
|
|
|
|
If the deferrals are all `ACTIVE_WRITER_LEASE`, the backlog resolves itself as those uploads expire.
|
|
Never delete staging content to clear a backlog: an upload that is mid-flight will corrupt.
|
|
|
|
## Namespace quota drift
|
|
|
|
**Signal** — a namespace's usage (`fileserver.quota` rejections, or the drift query below) disagrees
|
|
with the bytes its files actually hold. The known cause is history: before the 2026-11 maturity
|
|
review, `move` relocated a file without moving its quota. The bytes of every file moved before that
|
|
fix are still booked on the namespace it left, and deleting such a file reclaims from the namespace
|
|
it now lives in: the reclaim draws down the bookings of that namespace's other files, newest first,
|
|
or stops at zero, and the old booking stays behind either way. A derived
|
|
service that moved files across namespaces before taking the fix has this drift; one that never did
|
|
should see none.
|
|
|
|
What the ledger must equal: quota is committed when a file is published and reclaimed when cleanup
|
|
deletes its content, so a namespace's committed bytes are the sizes of its published files that are
|
|
not yet `DELETED`. The query compares the two and changes nothing.
|
|
|
|
```sql
|
|
WITH booked AS (
|
|
SELECT scope_value AS namespace, sum(committed_bytes) AS booked_bytes
|
|
FROM fs_quota_reservation
|
|
WHERE scope_type = 'namespace' AND status = 'COMMITTED'
|
|
GROUP BY scope_value),
|
|
held AS (
|
|
SELECT namespace, sum(actual_size) AS held_bytes
|
|
FROM fs_file
|
|
WHERE published_at IS NOT NULL AND state <> 'DELETED'
|
|
GROUP BY namespace)
|
|
SELECT coalesce(b.namespace, h.namespace) AS namespace,
|
|
coalesce(b.booked_bytes, 0) AS booked_bytes,
|
|
coalesce(h.held_bytes, 0) AS held_bytes,
|
|
coalesce(b.booked_bytes, 0) - coalesce(h.held_bytes, 0) AS drift_bytes
|
|
FROM booked b
|
|
FULL JOIN held h ON h.namespace = b.namespace
|
|
WHERE coalesce(b.booked_bytes, 0) <> coalesce(h.held_bytes, 0);
|
|
```
|
|
|
|
A positive `drift_bytes` is booked capacity no file holds (the namespace a file left); a negative one
|
|
is held content nothing booked (the namespace it arrived in). While uploads, copies, moves or
|
|
cleanup run, a row can show a drift that only means work in flight, so judge drift with the
|
|
fileserver's writes stopped.
|
|
|
|
To correct a namespace, stop every fileserver write to it first — uploads, copy, move, delete, and
|
|
the cleanup worker (`APP_FILESERVER_PLATFORM_CLEANUP_ENABLED=false`) — because the statement below
|
|
recomputes from `fs_file` and a publish or reclaim that lands between its two steps would be lost.
|
|
It zeroes the namespace's committed rows and books the recomputed total as one row, shaped the way a
|
|
publish leaves one (`COMMITTED`, `reserved_bytes` 0, the bytes in `committed_bytes`); the per-row
|
|
split has no meaning once reclaims have drawn rows down, and enforcement only ever reads the sum.
|
|
`RESERVED` rows (uploads in progress) are left alone.
|
|
|
|
```sql
|
|
BEGIN;
|
|
SET LOCAL lock_timeout = '5s';
|
|
UPDATE fs_quota_reservation
|
|
SET committed_bytes = 0, version = version + 1, updated_at = now()
|
|
WHERE scope_type = 'namespace' AND scope_value = :'namespace'
|
|
AND status = 'COMMITTED' AND committed_bytes > 0;
|
|
INSERT INTO fs_quota_reservation (
|
|
reservation_id, scope_type, scope_value, reserved_bytes, committed_bytes,
|
|
expires_at, status, version, created_at, updated_at)
|
|
SELECT gen_random_uuid(), 'namespace', :'namespace', 0, held,
|
|
now(), 'COMMITTED', 0, now(), now()
|
|
FROM (SELECT coalesce(sum(actual_size), 0) AS held
|
|
FROM fs_file
|
|
WHERE namespace = :'namespace'
|
|
AND published_at IS NOT NULL AND state <> 'DELETED') AS recomputed
|
|
WHERE held > 0;
|
|
COMMIT;
|
|
```
|
|
|
|
Run it with `psql -v namespace=<name>`, then run the drift query again: the namespace must be gone
|
|
from its result before writes resume.
|