Compare commits

...
Author SHA1 Message Date
DongHyeonka 9465582b5d chore: 실행 환경 구성 문서 추가 및 수정 2026-09-10 15:55:36 +09:00
DongHyeonkaandClaude Opus 5 6f6ab86345 docs(guides): reproduction guides for all 26 experiments
Written by subagents running under the writing-practitioner-guides skill,
one guide per experiment, 22,566 lines. Each walks a reader from baseline
capture through injection, injection verification, observation and recovery.

Section 3 carries the weight in most of them. Injection failed silently nine
times in this lab, and a failed injection looks exactly like no effect — so
the guides verify the target is actually in the intended state before
reading any result. A-4 makes virsh list the only proof because the node
reads Ready for 40 seconds after the machine is off; A-5 makes the packet
counter the sole go/no-go because a rule on the wrong node produces an empty
result that reads like a finding; A-6 quotes the run where 적용완료 was
printed between four Cannot find device "eth0" lines.

The traps the guides are built around are ones that invert a conclusion
rather than merely annoy:

  A-0   emptying the session table without a restart leaves cache entries
        that get counted as replication arriving
  A-2   dropping -o /dev/null fuses body and status into one string
  A-3   presence of "ready to accept connections" instead of its timestamp
  B-2   row count alone reads an UPDATE as nothing having happened
  B-4   tr ',' '\n' splits ["admin","editor"] so only admin is seen
  B-7   no login screen means the cookie died and SSO re-authenticated
  C-1   counting sessions without joining realm counts your own kcadm one
  D-1   kubectl exec without -i restores nothing and still exits 0
  D-4a  "ran with error output" is what success looks like

Every quoted block is copied from docs/evidence/ and marked 실측; reshaped
commands are marked 미검증 rather than passed off as measured. Where a source
document carries a ★ correction the guides follow the corrected claim — A-7's
REVOKED_TOKEN hypothesis, C-1's session count, B-2's schema attribution.

Two hazards are stated rather than smoothed over: B-6 deletes a key that
cannot be recreated, and D-1/D-4 need host sudo, which asks for a password,
so those steps say a person must type them.

Audit over all 26: 672 interpretation pairs, 486 evidence citations, 117
undo sections, and zero occurrences of the patterns the skill forbids —
no python data processing, no deprecated kubectl get endpoints, no
placeholders, no bare kcadm.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-07 18:29:00 +09:00
DongHyeonkaandClaude Opus 5 8062cc9a19 docs(guides): bring the seven setup guides in line with the practitioner skill
The guides were written before the skill existed and a scored audit found
three gaps. Fixed by a subagent running under the skill, with measured
values, versions, IPs and quoted output declared off limits.

The big one: every step showed a command and its output, and almost none
said which line to look at or what it meant. 64 interpretation pairs added
across the seven files, weighted where the reading is hardest — 20 in the
Keycloak stage, where a Secret existing and a pod having received it are
different facts.

Only the extracting form of curl appeared. Where the reader meets a response
for the first time the guides now open with curl -I or curl -v and name the
lines worth reading; -w '%{http_code}' survives only where the code is a
value being compared — two upstream nodes against each other, or the 900-run
control loop.

Listing Secret keys went from a three-stage pipe to kubectl describe secret,
which prints the key names and their byte counts in one native command
without exposing a value.

And a tool assumption: jq and yamllint are installed on neither the lab host
nor the guests. The guides now say so where JSON is read by eye, rather than
sending the reader to install something mid-diagnosis. cloud-init schema is
on the guests and is now the guest-side check.

Also removes a stray Playwright screenshot committed at the repository root
in 919547a; the evidence copy under docs/evidence/b7a-orphan-session/ is the
one the document references.

Four things the audit left standing are recorded in the agent's report rather
than papered over — notably that 04's reload measurements are stated without
a reproduction procedure, and that 05 and 06 reference each other as
prerequisites.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-07 17:33:31 +09:00
DongHyeonkaandClaude Opus 5 88b7bd4bf0 docs(guides): hands-on guides for the seven setup stages
The experiment documents record what was found. These record what to type to
reproduce it, in folders per stage.

Two kinds of command are kept apart. 하기/확인 is what somebody actually types
at a terminal — short, one at a time. 근거를 재려면 is the long measuring form
this lab used to put evidence in a document, marked as not needed day to day.
The same split applies to curl: -I to look once, -w '%{http_code}' only when
comparing across repetitions.

No placeholders. Where a value is needed the command that produces it is
given, and secrets are checked by length rather than printed:

  TOKEN=$(ssh kc-lab-1 'sudo cat /var/lib/rancher/k3s/server/node-token')
  echo "${#TOKEN} 자"

Stage 05 verifies resources in layers, because a Secret existing and a pod
having received it are different facts: keys, then length, then the value
inside the container, then which env var came from which Secret. Same for
workloads — Deployment to ReplicaSet to Pod, with the seven ReplicaSets this
cluster actually carries as the worked example.

Two commands were wrong and re-running them caught it. kubectl get endpoints
prints a deprecation warning on v1.33+, so the guide uses describe svc and
EndpointSlice. And the Keycloak image has no curl, so reading metrics from
inside the container fails with exit 127 — the guide asks Prometheus instead,
or runs a throwaway curl pod.

Read-only checks were executed against the running lab and their output is
quoted verbatim. Creating commands could not be re-run without destroying the
lab, so they are the ones used at build time; the README says which is which.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-07 16:48:10 +09:00
DongHyeonkaandClaude Opus 5 cdac9b8178 docs: give the twelve experiments that had no architecture diagram one
An audit against the standard the series set — concepts, procedure,
commands, architecture diagram, evidence table, terminal output — found the
three new experiments met it while twelve of the original ones had no
diagram at all: A-0, A-1, A-3, A-4, A-5, A-6, A-8, B-0, B-2, B-7, C-2, D-2.

Each now has one drawn from what that experiment actually found, not filler:
A-0 shows sharing going through PostgreSQL rather than between the caches;
A-3 the gap between the 200 and the WAL flush, with both failed injections;
A-5 the three silent injection failures; A-6 the two places latency is
multiplied; B-0 the repository keyed by principal with no session id; B-2
the primary key that causes the overwrite; D-2 why the rolling update
stopped the accident halfway.

Also corrected the index's stale claim of 11 experiments without a
screenshot — it is 14, and the reason is recorded: those experiments were
measured from terminals, the database and logs, and the observability stack
does not scrape Redis, the BFF or PostgreSQL, so there is no console to
photograph.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-04 21:44:36 +09:00
DongHyeonkaandClaude Opus 5 a2f749292a docs: add architecture diagrams for the three follow-up experiments
Each of A-7a, B-7a and D-4a had the concepts, procedure, commands and
terminal evidence but no diagram, which the rest of the series carries.

  a7a-cache-temperature.svg  the three cache states and the SQL each issues
  b7a-orphan-lifecycle.svg   ticket structure, accumulation across two
                             rotations, and why TTL is the only signal
  d4a-hook-timeline.svg      the 38m25s gap against the 1-2s one, on one axis

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-04 21:37:00 +09:00
DongHyeonkaandClaude Opus 5 e9baac21d5 docs(d4a): the deploy hook works — issuance to serving drops from 38m25s to 1-2s
D-4 named the fix and never ran it. Running it:

  /etc/letsencrypt/renewal-hooks/deploy/reload-nginx.sh
    #!/bin/sh
    nginx -t && nginx -s reload

certbot reports "Hook 'deploy-hook' ran", the master stays at PID 585 and
the worker is replaced 28829 -> 37252, and the served certificate changes.

One trap worth naming: certbot prefixes the hook output with "ran with error
output" whenever the hook writes anything to stderr, and nginx's routine
types_hash warning goes to stderr. Everything inside is success — "test is
successful", "signal process started". A monitor that greps for "error"
would read a working hook as a failure. The worker PID is what to check.

Timing needed a clock correction. test-server has NTP off and runs 106
seconds fast; the dev machine matches Google and the Let's Encrypt ACME
endpoint exactly. Corrected, the hook's nginx -t lands at 12:27:50 UTC
against the new certificate's SCT at 12:27:49.054 — one second. The
correction validates itself: uncorrected, the hook would have run 104
seconds before the certificate existed.

That same skew is why D-4's gap was reported 106 seconds short; corrected to
2305s in the previous commit.

Remaining and left alone: whether certbot-renew.timer performs a real
renewal, which cannot be tested for about 89 days.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-04 21:35:00 +09:00
DongHyeonkaandClaude Opus 5 3d7778bd3c docs(d4): correct the gap to 38m25s — the two timestamps came from different clocks
The 2199 seconds reported for D-4 subtracted a test-server timestamp
(archive/cert2.pem mtime) from a dev-machine timestamp (the serial change
observed by the poll), without noting they are different clocks.

Checked against external references: the dev machine matches Google and the
Let's Encrypt ACME endpoint to the second, while test-server is 105 seconds
fast and reports NTPSynchronized=no. Three round-trip measurements put the
offset at +106.1s every time.

Corrected:

  new certificate written to disk  08:20:27 UTC   (mtime 17:22:13 KST - 106s)
  actually served                  08:58:52 UTC   (dev observation, no correction)
  gap                              2305s = 38m25s

The correction validates itself in D-4a, where the new certificate's SCT —
signed by CT logs on their own accurate clock at 12:27:49.054 GMT — lands
one second before the skew-corrected hook time. Without the correction the
hook would appear to have run 104 seconds before the certificate existed.

Updated across the experiment doc, the index, the follow-up doc, the verdict
evidence file and the SVG.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-04 21:33:01 +09:00
DongHyeonkaandClaude Opus 5 919547a025 docs(b7a): the orphan sessions can be deleted — oauth2-proxy just cannot do it
B-7 stopped at "could not delete the server-side session". The reason it
gave was right: the ticket carries the session id, the ticket is encrypted
with the cookie secret, so after a rotation the proxy cannot work out which
Redis key to remove. But that is a limitation of the proxy, not of Redis.

Measured across two rotations:

- The orphan does expire. TTL falls one second per second and is not
  refreshed by requests (the startup log says refresh:disabled, and
  --cookie-refresh is unset), so it dies exactly one hour after creation.
- An operator can delete it. `redis-cli del` returned 1, dbsize went 2 to 1,
  and the live session answered /oauth2/userinfo with 200 immediately after.
- But nothing in Redis says which key is the orphan. Same name prefix, same
  type, the same 3510 bytes, and the values are encrypted.
- TTL is the only signal, and because it is never refreshed it is an exact
  function of creation time. Anything created before the rotation is an
  orphan. Derived creation time 11:30:26 against the AuthSuccess log line at
  11:30:27 — one second out. The rule was then run and removed the orphan
  while leaving the live session.
- They accumulate: the session that survived the first rotation became the
  orphan of the second.

The caveat is recorded too: turning on --cookie-refresh breaks the
derivation, and at that point flushing and forcing everyone to
re-authenticate is the more honest option.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-04 20:37:11 +09:00
DongHyeonkaandClaude Opus 5 b9f4ef7bc2 docs(a7a): the volatile refresh failure is a client-scope lookup, not REVOKED_TOKEN
A-7 left the cause as a hypothesis and said to confirm it with the statement
logging A-3 used. Doing that shows the hypothesis was wrong.

Login issues no SQL at all in volatile mode. Refresh issues exactly one
statement, and it is not the one A-7 guessed:

  select cscme1_0.SCOPE_ID from CLIENT_SCOPE_CLIENT cscme1_0
   where cscme1_0.CLIENT_ID=$1 and cscme1_0.DEFAULT_SCOPE=$2
   parameters: $1 = '131a9912-...', $2 = 'f'

REVOKED_TOKEN never appears. DEFAULT_SCOPE='f' means this is the optional
client scope list, which refresh needs because it recomputes which scopes go
into the new access token.

The larger finding is that A-7's table was conditional and did not say so.
The single statement is issued once and then cached, so the same
configuration produces three different outcomes depending only on cache
warmth — all three reproduced:

  fully cold      login 400  (select ce1_0.ID from CLIENT ...)
  CLIENT warm     login 200, refresh 500  <- what A-7 measured
  fully warm      login 200, refresh 200

So "volatile means you can log in without a database" is also conditional:
on a cold start the client lookup itself fails. In each case the Keycloak
log names the failing SQL directly, so this is determined rather than
inferred.

A-7 keeps its original wording with a correction banner above it, so what
was mis-guessed and why stays visible.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-04 20:27:23 +09:00
DongHyeonkaandClaude Opus 5 9dbee18a42 docs(b4): item 3 is no longer unmeasured — link it to the follow-up result
B-4 left role propagation open because oauth2-proxy was not deployed yet.
B-7 deployed it and the follow-up measured it: the value does not change
with request count, only when a new session is created.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-04 18:10:43 +09:00
DongHyeonkaandClaude Opus 5 905392947d docs: close the three untested items — all three are now measured
D-2 forward upgrade and B-4 role propagation were filled earlier. D-4 forced
renewal was the third, and it needed a person at the keyboard because the
host asks for a sudo password.

It turned out to hold the largest finding of the three: the renewal
succeeded and went unserved for 36 minutes 39 seconds, because nothing in
the setup reloads nginx — no ExecStartPost, three empty hook directories,
no nginx plugin. And the half of the plan's question that was still an
expectation is now a measurement: the reload is graceful, 8856 connections
with zero failures and an in-flight request surviving intact.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-04 18:10:04 +09:00
DongHyeonkaandClaude Opus 5 faf55dc1be docs(d4): the renewal succeeded and was not served for 36 minutes 39 seconds
The forced renewal did work. What did not work was anything making nginx
notice.

  disk     cert2.pem written 2026-09-04 17:22:13 KST
  network  old serial through 08:58:47 UTC, new serial from 08:58:52 UTC
  gap      2199 seconds, and the old certificate was observed 428 times in it

The gap closed only because a person ran `nginx -s reload`. Nothing else
would have: certbot-renew.service is just `ExecStart=/usr/bin/certbot -q
renew` with no ExecStartPost, all three of renewal-hooks/{deploy,post,pre}
are empty, and certbot 5.7.0 here has no nginx plugin (dns-cloudflare,
manual, null, standalone, webroot). Three paths, all empty. Any one of them
would have been enough.

nginx holds the certificate in memory from startup, and certbot swaps the
live/ symlink rather than the path — so the config still looks right while
the served certificate is stale. Master 585 and worker 586 sharing a start
time 22.4 hours old is what proved no reload had happened.

This is invisible for 88 days. The timer ran twice today and exited
0/SUCCESS both times, because a renewal is not due yet. The day it becomes
due, the symptom is an expired certificate and the log still says SUCCESS.

The other half of the plan's question now has a measurement rather than an
expectation. The reload is genuinely graceful: 8856 polled connections, zero
non-200, p95 205.7ms before against 204.3ms after. And an in-flight request
— 845KB pulled at 20k/s, so still transferring 12 seconds in when the reload
landed — completed with all 845361 bytes over a single connection.

The earlier note that nginx reload "is graceful by design but must not be
written that way without checking" was right to hold back. The neighbouring
belief, that renewing gets the new certificate served, was the one that was
false, and there was no way to tell in advance which it would be.

Adds diagrams/d4-renewal-gap.svg and evidence 09-13.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-04 18:04:44 +09:00
DongHyeonkaandClaude Opus 5 0d84921a56 docs(d4): the renewal unit has no reload, and nginx has not reloaded in 22 hours
The forced renewal did not change the certificate being served — 161 serial
samples over 13 minutes, all identical, notBefore still Sep 3.

Two things were readable without sudo and both matter.

nginx has never been reloaded: master 585 and worker 586 share a start time
and both are 80529 seconds old. A reload keeps the master and replaces the
workers, so the original fork still being there means no reload has happened
since Sep 3 19:00. nginx reads the certificate at startup and holds it in
memory; without a reload it serves the old one no matter what is on disk.

And the unit that is supposed to renew has nothing that would reload it:

  [Service]
  Type=oneshot
  ExecStart=/usr/bin/certbot -q renew
  PrivateTmp=true

No ExecStartPost, no --deploy-hook. The timer runs twice a day and exited
0/SUCCESS both times today, which is precisely why this stays invisible —
the renewal is not due for 88 days, so nothing has failed yet.

What remains undetermined is whether the user's --force-renewal failed, or
succeeded and left new files that nginx never picked up. Telling those apart
needs /etc/letsencrypt, which is root-only. If it is the latter, that is the
most valuable thing this lab has produced: "renewal succeeded" and "the new
certificate is being served" are different events.

CT logs looked like a way around the permission problem and were not: the
served certificate carries two valid SCTs, yet crt.sh returns zero rows for
auth.hyeonworks.com. Same shape as A-2's `up` metric — the observing tool
sees a subset of the truth.

Also recorded: 76 in-flight failures at 08:15:04 are a local artifact, not a
server event. The concurrent 0.2s poll was clean across the same second,
num_connects was 0, and time_total was 50 microseconds. Not reproducible in
100 retries. The monitor now records curl's exit code so a recurrence is
diagnosable.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-04 17:28:01 +09:00
DongHyeonkaandClaude Opus 5 716e62524a docs(d4): measure the controls before the injection that needs a password
The forced renewal needs sudo on the host, and the host asks for a password.
That blocks the injection, not the experiment — the part worth doing first
was the control anyway.

Control 1 — new connections, 0.2s x 900 over 180s: 900/900 = 200, zero
failures. The noise floor is 0, so a single non-200 during the renewal can
be attributed to the renewal. Without this the observation would have been
uninterpretable, which is the mistake A-6 made calling a -41% control
"no effect" and A-8 made claiming zero-downtime from 9 samples.

Control 2 — the poll cannot answer the question the plan actually asked.
TLS handshakes were 900/900, meaning every request is a fresh connection,
so it measures "are new connections accepted", not "what happens to a
request already in flight". A separate device: the 845KB admin console
bundle pulled at --limit-rate 20k, holding one request open for 42 seconds.
Baseline 200 / 845361 bytes / 1 connection.

Monitors are running under setsid, and stop three minutes after the
certificate serial changes.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-04 17:12:31 +09:00
DongHyeonkaandClaude Opus 5 74c9b3cea7 docs: replace prose placeholders in reproduction steps with executable commands
The audit found ~80 placeholders, and the damaging ones were where the
measuring apparatus itself was prose rather than a command:

  a6  "( curl ... ) & 를 20개 띄우고 wait"  — the 22.2s headline came from this
  a3  "<로그인 반복, sid 를 /tmp/sids 에>"  — the whole RPO measurement
  a3  "<sid 목록>"                          — the control it is compared against
  a5  "<수신 파드IP>"                       — the injection
  a8  writes /tmp/tok, reads /tmp/rt        — self-inconsistent, sent an empty token
  b3  $KC / $RT / $NEW never assigned
  c2  bare kcadm.sh with no kubectl exec
  a1  conntrack tuples written by hand, though the direction flips per restart

Each is now a shell-expandable form: pod IPs from jsonpath, the admin password
from the secret, ids from kcadm --format csv, conntrack tuples derived from
"conntrack -L" with awk rather than transcribed.

Then the rewritten commands were executed against the live cluster, and one
of them failed — the 20-way load generator, written as "kubectl run --rm -i",
lost its output stream twice in a row. That is a trap this series already hit
once, and the rewrite reintroduced it. A-6 now uses a resident probe pod that
collects into a file and is cat-ed once; verified 20/20 lines.

Evidence: docs/evidence/followup/05-command-reproducibility.txt

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-04 17:01:54 +09:00
DongHyeonkaandClaude Opus 5 f3f3a8da46 docs: re-measure B-4 role propagation and record it as evidence
The twelve-request result and the post-reauth value were printed in the document but never written to a file; the audit flagged it as a conclusion standing without evidence. Measured again with timestamps into 03-b4-role-propagation.txt, which also surfaced a 107-second clock skew between the browser host and the lab host that the file now records so the two timelines can be compared.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-04 16:54:05 +09:00
DongHyeonkaandClaude Opus 5 e0d27d47ce docs: correct the places where documents contradicted their own evidence
An independent audit found ten documents printing values their evidence files do not contain. C-1 printed a session count of 0 where the evidence says 4, C-2 printed a success readback for a command that exited 1, and A-1 credited the conntrack flush with a split that the timestamps attribute to a pod restart four seconds earlier.

Also measured wal_writer_delay, which A-3 had asserted as matching without ever querying it, relabelled the A-6 control that moved 41 percent, noted A-8's nine-sample resolution, corrected D-1's RTO to the 41 seconds its own timeline shows, and added a correction banner to D-2. Every experiment document now links its evidence files with their real collection times, and the duplicate screenshots are documented as duplicates.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-04 16:35:49 +09:00
DongHyeonkaandClaude Opus 5 78b270559c docs: add SVG diagrams, explicit concept sections and the diagram convention
Twelve SVG architecture diagrams cover the experiments whose documents had little or no structure drawing, embedded under a 구조 heading with a shared convention file. Seven documents carried their concepts under narrative headings and now have an explicit 개념 section so they can be found.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-04 16:20:05 +09:00
DongHyeonkaandClaude Opus 5 98a74e90a5 docs: fill the untested items and record why the B layer has no graphs
The forward upgrade to 26.7.3 was zero downtime across 87 samples, and since databasechangelog stayed at 210 the rollback to 26.7.0 also succeeded, which narrows D-2's conclusion: rolling back fails when the schema moved, not because of the version number. The row count is the check.

Role changes never reach the upstream through request repetition; the session is a snapshot taken at login and only a new session picks up the new claim. Auditing the docs also surfaced that Prometheus scrapes only keycloak, kubelet, node-exporter and itself, so the B-layer experiments have no metrics to screenshot rather than missing screenshots.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-04 16:18:33 +09:00
DongHyeonkaandClaude Opus 5 b5528fae87 docs: index all 23 experiments with what each measured
One table per experiment with its branch and result, plus the nine injections that silently did nothing and the five predictions that turned out wrong.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-04 15:11:32 +09:00
DongHyeonkaandClaude Opus 5 4864d837f1 docs: D-4 — the certificate is fine and the renewal itself went untested
Three SAN entries and no wildcard is the constraint that cost something real in B-7, where oauth2-proxy had to borrow Grafana's app2 hostname because a fourth name was not available. The served chain is four deep and verifies, so fullchain.pem is configured rather than the cert.pem mistake that only breaks clients without a cached intermediate.

The forced renewal and the reload behaviour could not be measured because sudo on the host asks for a password, the same silent failure first noticed in B-7. nginx reload is graceful by design, but this lab has repeatedly shown that by design is not the same as measured, so it is recorded as untested rather than assumed.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-04 15:09:36 +09:00
DongHyeonkaandClaude Opus 5 027c24ee27 docs: D-3 — only RBAC actually hides anything
Every secret in the lab prints in four commands, while kubectl describe shows just a byte count and creates the impression that something is hidden. k3s reports encryption at rest disabled and the plaintext password is present in state.db, so one node disk carries the whole cluster's secrets, and inside the pod they are ordinary environment variables visible to exec, /proc and crash dumps.

The default service account cannot read secrets, which makes RBAC the one control doing real work here and the thing worth tightening.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-04 15:07:35 +09:00
169 changed files with 33336 additions and 89 deletions
@@ -0,0 +1,2 @@
[ 19340ms] [ERROR] Failed to load resource: the server responded with a status of 403 () @ https://app2.hyeonworks.com/oauth2/callback?state=rKipZCUv8W5a-xgYheJbjBsInoD5Il1AaF1RlM_RB2s%3A%2Fapi%2Fecho&session_state=Mw52KcQijFB9Bq4rN-C4SF5Y&iss=https%3A%2F%2Fauth.hyeonworks.com%2Frealms%2Fkeycloak-patterns&code=f9a4835a-2af3-b886-bd04-10b5347ee8d2.Mw52KcQijFB9Bq4rN-C4SF5Y.80431dbc-af81-4673-9790-ad06d1570b2e:0
[ 20374ms] [ERROR] Failed to load resource: the server responded with a status of 401 () @ https://app2.hyeonworks.com/favicon.ico:0
@@ -0,0 +1 @@
[ 210ms] [ERROR] Failed to load resource: the server responded with a status of 401 () @ https://app2.hyeonworks.com/favicon.ico:0
@@ -0,0 +1 @@
[ 423ms] [ERROR] Failed to load resource: the server responded with a status of 401 () @ https://app2.hyeonworks.com/favicon.ico:0
@@ -0,0 +1,8 @@
[ 1127ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 2377ms] [WARNING] <meta name="apple-mobile-web-app-capable" content="yes"> is deprecated. Please include <meta name="mobile-web-app-capable" content="yes"> @ https://app2.hyeonworks.com/explore?schemaVersion=1&panes=%7B%22h4a%22%3A%7B%22datasource%22%3A%22PBFA97CFB590B2093%22%2C%22queries%22%3A%5B%7B%22refId%22%3A%22A%22%2C%22expr%22%3A%22vendor_cluster_size%22%2C%22range%22%3Atrue%2C%22instant%22%3Afalse%2C%22editorMode%22%3A%22code%22%2C%22legendFormat%22%3A%22cluster_size+%7B%7Bpod%7D%7D%22%2C%22datasource%22%3A%7B%22type%22%3A%22prometheus%22%2C%22uid%22%3A%22PBFA97CFB590B2093%22%7D%7D%2C%7B%22refId%22%3A%22B%22%2C%22expr%22%3A%22up%7Bjob%3D%5C%22keycloak%5C%22%7D%22%2C%22range%22%3Atrue%2C%22instant%22%3Afalse%2C%22editorMode%22%3A%22code%22%2C%22legendFormat%22%3A%22up+%7B%7Bpod%7D%7D%22%2C%22datasource%22%3A%7B%22type%22%3A%22prometheus%22%2C%22uid%22%3A%22PBFA97CFB590B2093%22%7D%7D%5D%2C%22range%22%3A%7B%22from%22%3A%22now-30m%22%2C%22to%22%3A%22now%22%7D%7D%7D&orgId=1:0
[ 2472ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 3501ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 5119ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 8511ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 14956ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 28065ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
@@ -0,0 +1,10 @@
[ 1362ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 1874ms] [WARNING] <meta name="apple-mobile-web-app-capable" content="yes"> is deprecated. Please include <meta name="mobile-web-app-capable" content="yes"> @ https://app2.hyeonworks.com/explore?schemaVersion=1&orgId=1&panes=%7B%22a%22%3A%7B%22datasource%22%3A%22PBFA97CFB590B2093%22%2C%22queries%22%3A%5B%7B%22refId%22%3A%22A%22%2C%22expr%22%3A%22vendor_cluster_size%22%2C%22range%22%3Atrue%2C%22instant%22%3Afalse%2C%22editorMode%22%3A%22code%22%2C%22legendFormat%22%3A%22cluster_size%20%7B%7Bpod%7D%7D%22%2C%22datasource%22%3A%7B%22type%22%3A%22prometheus%22%2C%22uid%22%3A%22PBFA97CFB590B2093%22%7D%7D%2C%7B%22refId%22%3A%22B%22%2C%22expr%22%3A%22up%7Bjob%3D%5C%22keycloak%5C%22%7D%22%2C%22range%22%3Atrue%2C%22instant%22%3Afalse%2C%22editorMode%22%3A%22code%22%2C%22legendFormat%22%3A%22up%20%7B%7Bpod%7D%7D%22%2C%22datasource%22%3A%7B%22type%22%3A%22prometheus%22%2C%22uid%22%3A%22PBFA97CFB590B2093%22%7D%7D%5D%2C%22range%22%3A%7B%22from%22%3A%221788502680000%22%2C%22to%22%3A%221788503520000%22%7D%7D%7D:0
[ 2907ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 3998ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 6253ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 9426ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 12495ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 24486ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 31338ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 46196ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
@@ -0,0 +1,198 @@
[ 1319ms] [WARNING] <meta name="apple-mobile-web-app-capable" content="yes"> is deprecated. Please include <meta name="mobile-web-app-capable" content="yes"> @ https://app2.hyeonworks.com/explore?schemaVersion=1&orgId=1&panes=%7B%22a%22%3A%7B%22datasource%22%3A%22PBFA97CFB590B2093%22%2C%22queries%22%3A%5B%7B%22refId%22%3A%22A%22%2C%22expr%22%3A%22vendor_cluster_size%22%2C%22range%22%3Atrue%2C%22instant%22%3Afalse%2C%22editorMode%22%3A%22code%22%2C%22legendFormat%22%3A%22cluster_size%20%7B%7Bpod%7D%7D%22%2C%22datasource%22%3A%7B%22type%22%3A%22prometheus%22%2C%22uid%22%3A%22PBFA97CFB590B2093%22%7D%7D%2C%7B%22refId%22%3A%22B%22%2C%22expr%22%3A%22vendor_statistics_approximate_entries_unique%7Bcache%3D%5C%22sessions%5C%22%7D%22%2C%22range%22%3Atrue%2C%22instant%22%3Afalse%2C%22editorMode%22%3A%22code%22%2C%22legendFormat%22%3A%22sessions%20%5Cuce90%5Cuc2dc%20%7B%7Bpod%7D%7D%22%2C%22datasource%22%3A%7B%22type%22%3A%22prometheus%22%2C%22uid%22%3A%22PBFA97CFB590B2093%22%7D%7D%5D%2C%22range%22%3A%7B%22from%22%3A%221788497040000%22%2C%22to%22%3A%221788499080000%22%7D%7D%7D:0
[ 5941ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 11107ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 14234ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 17005ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 23049ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 30565ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 44135ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 54992ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 63653ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 70658ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 90768ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 105475ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 114995ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 125443ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 137321ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 147252ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 167227ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 179100ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 187598ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 207362ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 213255ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 230544ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 248833ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 257549ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 262324ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 274443ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 294124ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 310689ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 312049ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 327025ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 339933ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 342391ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 362305ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 370693ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 378685ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 397930ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 414007ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 419130ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 420355ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 428557ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 442088ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 445041ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 446677ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 460087ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 462505ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 463673ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 482762ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 495123ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 513442ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 516414ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 536581ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 543441ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 563624ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 580510ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 588408ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 594209ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 604166ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 605295ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 621987ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 639705ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 653320ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 660911ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 678105ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 693662ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 703089ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 704712ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 719727ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 725514ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 730429ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 735135ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 736677ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 751626ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 767197ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 782052ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 797775ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 803343ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 822168ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 828683ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 845938ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 865903ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 883116ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 894375ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 898471ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 914886ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 933184ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 953353ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 963902ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 982846ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 998975ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 1000721ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 1003229ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 1009879ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 1016334ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 1023805ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 1043684ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 1049467ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 1059107ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 1075856ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 1076956ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 1094159ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 1102825ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 1116077ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 1134809ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 1136553ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 1141191ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 1158221ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 1173507ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 1190834ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 1198506ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 1217761ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 1231342ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 1233528ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 1243017ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 1253381ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 1256056ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 1258513ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 1271317ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 1284530ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 1301830ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 1307562ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 1310330ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 1317970ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 1328668ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 1336338ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 1340155ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 1346989ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 1348836ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 1352113ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 1367871ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 1371158ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 1390720ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 1405259ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 1420920ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 1433518ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 1450513ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 1453685ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 1460654ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 1470589ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 1489330ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 1503666ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 1521587ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 1537970ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 1540426ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 1548105ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 1549441ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 1567671ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 1585493ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 1597054ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 1602669ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 1607866ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 1623290ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 1631460ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 1640086ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 1645979ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 1650742ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 1652863ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 1663908ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 1666587ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 1668427ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 1676825ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 1687882ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 1691062ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 1698532ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 1709597ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 1725669ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 1739294ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 1744412ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 1753319ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 1761792ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 1765298ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 1784381ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 1786396ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 1798982ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 1807180ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 1823569ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 1837292ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 1838720ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 1848387ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 1856847ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 1866742ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 1879249ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 1884700ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 1900774ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 1912256ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 1931088ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 1937844ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 1948191ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 1951360ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 1957512ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 1958836ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 1979011ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 1993551ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 2011369ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 2024782ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 2040763ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 2044650ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 2061337ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 302 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
@@ -0,0 +1 @@
[ 12202ms] [ERROR] Failed to load resource: the server responded with a status of 401 () @ https://app2.hyeonworks.com/favicon.ico:0
@@ -0,0 +1 @@
[ 451ms] [ERROR] Failed to load resource: the server responded with a status of 401 () @ https://app2.hyeonworks.com/favicon.ico:0
@@ -0,0 +1 @@
[ 14441ms] [ERROR] Failed to load resource: the server responded with a status of 401 () @ chrome-error://chromewebdata/:0
@@ -0,0 +1 @@
[ 296ms] [ERROR] Failed to load resource: the server responded with a status of 401 () @ chrome-error://chromewebdata/:0
@@ -0,0 +1 @@
[ 158ms] [ERROR] Failed to load resource: the server responded with a status of 401 () @ https://app2.hyeonworks.com/favicon.ico:0
@@ -0,0 +1 @@
[ 67ms] [ERROR] Failed to load resource: the server responded with a status of 401 () @ https://app2.hyeonworks.com/favicon.ico:0
@@ -0,0 +1,2 @@
[ 40ms] [ERROR] Failed to load resource: the server responded with a status of 401 () @ https://app2.hyeonworks.com/oauth2/userinfo:0
[ 162ms] [ERROR] Failed to load resource: the server responded with a status of 401 () @ https://app2.hyeonworks.com/favicon.ico:0
@@ -0,0 +1 @@
[ 117ms] [ERROR] Failed to load resource: the server responded with a status of 401 () @ chrome-error://chromewebdata/:0
@@ -0,0 +1,16 @@
- generic [ref=f55e3]:
- banner [ref=f55e4]:
- generic [ref=f55e5]: keycloak-patterns
- main [ref=f55e6]:
- heading "Sign in to your account" [level=1] [ref=f55e8]
- generic [ref=f55e12]:
- generic [ref=f55e13]:
- generic [ref=f55e14]: Username or email
- textbox "Username or email" [ref=f55e17]
- generic [ref=f55e18]:
- generic [ref=f55e19]: Password
- generic [ref=f55e21]:
- textbox "Password" [ref=f55e24]
- button "Show password" [ref=f55e26] [cursor=pointer]:
- generic [aria-hidden] [ref=f55e27]:
- button "Sign In" [ref=f55e30] [cursor=pointer]
@@ -0,0 +1,16 @@
- generic [ref=f55e3]:
- banner [ref=f55e4]:
- generic [ref=f55e5]: keycloak-patterns
- main [ref=f55e6]:
- heading "Sign in to your account" [level=1] [ref=f55e8]
- generic [ref=f55e12]:
- generic [ref=f55e13]:
- generic [ref=f55e14]: Username or email
- textbox "Username or email" [ref=f55e17]: labuser
- generic [ref=f55e18]:
- generic [ref=f55e19]: Password
- generic [ref=f55e21]:
- textbox "Password" [ref=f55e24]: labpass
- button "Show password" [ref=f55e26] [cursor=pointer]:
- generic [aria-hidden] [ref=f55e27]:
- button "Sign In" [ref=f55e30] [cursor=pointer]
@@ -0,0 +1,17 @@
- generic [ref=f56e1]:
- generic [ref=f56e3]:
- generic [ref=f56e4]: "403"
- heading "Forbidden" [level=1] [ref=f56e6]
- generic [ref=f56e8]:
- paragraph [ref=f56e9]: More Info
- generic [ref=f56e10] [cursor=pointer]:
- separator [ref=f56e12]
- generic [ref=f56e13]:
- button "Go back" [ref=f56e16] [cursor=pointer]
- button "Sign in" [ref=f56e19] [cursor=pointer]
- contentinfo [ref=f56e20]:
- paragraph [ref=f56e22]:
- text: Secured with
- link "OAuth2 Proxy" [ref=f56e23] [cursor=pointer]:
- /url: https://github.com/oauth2-proxy/oauth2-proxy#oauth2_proxy
- text: version v7.7.1
@@ -0,0 +1 @@
- generic [ref=f57e1]: "{ \"headers\" : { \"host\" : [ \"app2.hyeonworks.com\" ], \"user-agent\" : [ \"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/152.0.0.0 Safari/537.36\" ], \"accept\" : [ \"text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7\" ], \"accept-encoding\" : [ \"gzip, deflate, br, zstd\" ], \"accept-language\" : [ \"en-US,en;q=0.9\" ], \"cookie\" : [ \"grafana_session=60c3e7ae41ffc00665f4a2c377def399; grafana_session_expiry=1788497124; _oauth2_proxy=djIuWDI5aGRYUm9NbDl3Y205NGVTMDNZMkZrTUdZM01tRmlZekkwWldFell6a3lOREJpTW1KaE5UZGpOVEJoWWcuZVVmckp5VHRqY1VsXzdiV1hiX3hwdw==|1788503135|yDSo7VdgRSlvoVfj9raHPClKTlQiWDg7FauLfoUayw4=\" ], \"priority\" : [ \"u=0, i\" ], \"sec-ch-ua\" : [ \"\\\"Chromium\\\";v=\\\"152\\\", \\\"Not?A_Brand\\\";v=\\\"24\\\", \\\"Google Chrome\\\";v=\\\"152\\\"\" ], \"sec-ch-ua-mobile\" : [ \"?0\" ], \"sec-ch-ua-platform\" : [ \"\\\"Linux\\\"\" ], \"sec-fetch-dest\" : [ \"document\" ], \"sec-fetch-mode\" : [ \"navigate\" ], \"sec-fetch-site\" : [ \"none\" ], \"sec-fetch-user\" : [ \"?1\" ], \"upgrade-insecure-requests\" : [ \"1\" ], \"x-forwarded-email\" : [ \"labuser@example.com\" ], \"x-forwarded-host\" : [ \"app2.hyeonworks.com\" ], \"x-forwarded-port\" : [ \"443\" ], \"x-forwarded-preferred-username\" : [ \"labuser\" ], \"x-forwarded-proto\" : [ \"https\" ], \"x-forwarded-server\" : [ \"traefik-5d6fcf895-wpfhr\" ], \"x-forwarded-user\" : [ \"27df5ea9-8703-4ec5-badd-d972c583e1ff\" ], \"x-real-ip\" : [ \"100.123.124.30\" ] }, \"remoteAddr\" : \"100.123.124.30\", \"localAddr\" : \"10.42.0.53\", \"scheme\" : \"https\", \"secure\" : true, \"serverName\" : \"app2.hyeonworks.com\", \"serverPort\" : 443, \"requestUrl\" : \"https://app2.hyeonworks.com/api/echo\" }"
@@ -0,0 +1 @@
- generic [ref=f58e1]: "{ \"headers\" : { \"host\" : [ \"app2.hyeonworks.com\" ], \"user-agent\" : [ \"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/152.0.0.0 Safari/537.36\" ], \"accept\" : [ \"text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7\" ], \"accept-encoding\" : [ \"gzip, deflate, br, zstd\" ], \"accept-language\" : [ \"en-US,en;q=0.9\" ], \"cookie\" : [ \"grafana_session=60c3e7ae41ffc00665f4a2c377def399; grafana_session_expiry=1788497124; _oauth2_proxy=djIuWDI5aGRYUm9NbDl3Y205NGVTMHpabUV5TVdKaVpEa3lOemRrTlRFMU9USTBaV00wWWpGaE16bGhNak0zT1EuN2tTa3dnWUdISDkwMGFSSTVOSUFFUQ==|1788503202|snWKU5IRfRLoD9-bXEodGjEgfHeAw8PQaoHnecpFH90=\" ], \"priority\" : [ \"u=0, i\" ], \"sec-ch-ua\" : [ \"\\\"Chromium\\\";v=\\\"152\\\", \\\"Not?A_Brand\\\";v=\\\"24\\\", \\\"Google Chrome\\\";v=\\\"152\\\"\" ], \"sec-ch-ua-mobile\" : [ \"?0\" ], \"sec-ch-ua-platform\" : [ \"\\\"Linux\\\"\" ], \"sec-fetch-dest\" : [ \"document\" ], \"sec-fetch-mode\" : [ \"navigate\" ], \"sec-fetch-site\" : [ \"none\" ], \"sec-fetch-user\" : [ \"?1\" ], \"upgrade-insecure-requests\" : [ \"1\" ], \"x-forwarded-email\" : [ \"changed-labuser@example.com\" ], \"x-forwarded-host\" : [ \"app2.hyeonworks.com\" ], \"x-forwarded-port\" : [ \"443\" ], \"x-forwarded-preferred-username\" : [ \"labuser\" ], \"x-forwarded-proto\" : [ \"https\" ], \"x-forwarded-server\" : [ \"traefik-5d6fcf895-wpfhr\" ], \"x-forwarded-user\" : [ \"27df5ea9-8703-4ec5-badd-d972c583e1ff\" ], \"x-real-ip\" : [ \"100.123.124.30\" ] }, \"remoteAddr\" : \"100.123.124.30\", \"localAddr\" : \"10.42.1.132\", \"scheme\" : \"https\", \"secure\" : true, \"serverName\" : \"app2.hyeonworks.com\", \"serverPort\" : 443, \"requestUrl\" : \"https://app2.hyeonworks.com/api/echo\" }"
@@ -0,0 +1,175 @@
- generic [ref=f59e1]:
- generic [ref=f59e4]:
- link "Skip to main content" [ref=f59e5] [cursor=pointer]:
- /url: "#pageContent"
- banner [ref=f59e7]:
- generic [ref=f59e8]:
- link [ref=f59e10] [cursor=pointer]:
- /url: /
- img "Grafana" [ref=f59e11]
- generic [ref=f59e14]:
- button "Search or jump to..." [ref=f59e18] [cursor=pointer]
- generic [ref=f59e19]: ctrl+k
- generic [ref=f59e23]:
- button "New" [ref=f59e24] [cursor=pointer]
- button "Help" [ref=f59e30] [cursor=pointer]
- button "News" [ref=f59e33] [cursor=pointer]
- button "Profile" [ref=f59e36] [cursor=pointer]:
- img "User avatar" [ref=f59e37]
- generic [ref=f59e38]:
- button "Open menu" [ref=f59e40] [cursor=pointer]
- navigation "Breadcrumbs" [ref=f59e43]:
- list [ref=f59e44]:
- listitem [ref=f59e45]:
- link "Home" [ref=f59e46] [cursor=pointer]:
- /url: /
- listitem [ref=f59e50]:
- link "Explore" [ref=f59e51] [cursor=pointer]:
- /url: /explore
- listitem [ref=f59e55]:
- generic "Prometheus" [ref=f59e56]
- generic [ref=f59e57]:
- generic [ref=f59e60]:
- button "Copy shortened URL" [ref=f59e61] [cursor=pointer]
- button "Open copy link options" [ref=f59e64] [cursor=pointer]
- button "Toggle top search bar" [ref=f59e68] [cursor=pointer]
- main [ref=f59e74]:
- generic [ref=f59e76]:
- heading "Explore" [level=1] [ref=f59e77]
- generic [ref=f59e82]:
- navigation "Explore toolbar" [ref=f59e84]:
- navigation "Search links" [ref=f59e86]:
- generic [ref=f59e87]:
- button "Content outline" [expanded] [ref=f59e89] [cursor=pointer]:
- generic [ref=f59e92]: Outline
- generic [ref=f59e97] [cursor=pointer]:
- img "Prometheus logo" [ref=f59e99]
- textbox "Select a data source" [ref=f59e100]:
- /placeholder: Prometheus
- generic [ref=f59e104]:
- button "Split the pane" [ref=f59e106] [cursor=pointer]:
- generic [ref=f59e109]: Split
- button "Add" [ref=f59e111] [cursor=pointer]
- generic [ref=f59e116]:
- 'button "Time range selected: Last 30 minutes" [ref=f59e117] [cursor=pointer]'
- button "Zoom out time range" [ref=f59e122] [cursor=pointer]
- generic [ref=f59e126]:
- button "Cancel" [ref=f59e127] [cursor=pointer]
- button "Auto refresh turned off. Choose refresh time interval" [ref=f59e129] [cursor=pointer]
- generic [ref=f59e133]:
- generic [ref=f59e137]:
- button "Collapse outline" [expanded] [ref=f59e139] [cursor=pointer]:
- img "arrow-from-right" [ref=f59e140]
- generic [ref=f59e142]:
- button "Content outline item collapse button" [ref=f59e143] [cursor=pointer]:
- img "angle-right" [ref=f59e144]
- button "Queries" [ref=f59e146] [cursor=pointer]:
- img "arrow" [ref=f59e147]
- generic [ref=f59e154]:
- generic [ref=f59e156]:
- generic [ref=f59e157]:
- generic "Query editor row" [ref=f59e159]:
- generic [ref=f59e160]:
- generic [ref=f59e162]:
- generic [ref=f59e163]:
- button "Collapse query row" [expanded] [ref=f59e164] [cursor=pointer]
- generic [ref=f59e167]:
- button "Query editor row title A" [ref=f59e168] [cursor=pointer]:
- generic [ref=f59e169]: A
- emphasis [ref=f59e170]: (Prometheus)
- generic [ref=f59e171]:
- button "Show data source help" [ref=f59e173] [cursor=pointer]
- button "Duplicate query" [ref=f59e177] [cursor=pointer]
- button "Hide response" [ref=f59e181] [cursor=pointer]
- button "Remove query" [ref=f59e185] [cursor=pointer]
- button "Drag and drop to reorder" [ref=f59e188]:
- img "Drag and drop to reorder" [ref=f59e189]
- generic [ref=f59e192]:
- generic [ref=f59e193]:
- button "Kick start your query" [ref=f59e194] [cursor=pointer]
- generic [ref=f59e197]:
- generic [ref=f59e198] [cursor=pointer]: Explain
- generic [ref=f59e199]:
- checkbox "Explain Toggle switch" [ref=f59e200]
- generic "Toggle switch" [ref=f59e201] [cursor=pointer]
- radiogroup [ref=f59e206]:
- generic [ref=f59e207]:
- radio "Builder" [ref=f59e208] [cursor=pointer]
- generic [ref=f59e209] [cursor=pointer]: Builder
- generic [ref=f59e210]:
- radio "Code" [checked] [ref=f59e211] [cursor=pointer]
- generic [ref=f59e212] [cursor=pointer]: Code
- generic [ref=f59e214]:
- generic [ref=f59e216]:
- button "Loading metrics..." [disabled] [ref=f59e217] [cursor=pointer]
- code [ref=f59e224]:
- generic [ref=f59e225]:
- generic [ref=f59e230]: vendor_cluster_size
- textbox "Editor content;Press Alt+F1 for Accessibility Options." [ref=f59e235]: vendor_cluster_size
- 'button "Options Legend: cluster_size {{pod}} Format: Time series Step: auto Type: Range Exemplars: false" [ref=f59e241] [cursor=pointer]':
- generic [ref=f59e245]:
- heading "Options" [level=6] [ref=f59e246]
- generic [ref=f59e247]:
- generic [ref=f59e248]: "Legend: cluster_size {{pod}}"
- generic [ref=f59e249]: "Format: Time series"
- generic [ref=f59e250]: "Step: auto"
- generic [ref=f59e251]: "Type: Range"
- generic [ref=f59e252]: "Exemplars: false"
- generic "Query editor row" [ref=f59e254]:
- generic [ref=f59e255]:
- generic [ref=f59e257]:
- generic [ref=f59e258]:
- button "Collapse query row" [expanded] [ref=f59e259] [cursor=pointer]
- generic [ref=f59e262]:
- button "Query editor row title B" [ref=f59e263] [cursor=pointer]:
- generic [ref=f59e264]: B
- emphasis [ref=f59e265]: (Prometheus)
- generic [ref=f59e266]:
- button "Show data source help" [ref=f59e268] [cursor=pointer]
- button "Duplicate query" [ref=f59e272] [cursor=pointer]
- button "Hide response" [ref=f59e276] [cursor=pointer]
- button "Remove query" [ref=f59e280] [cursor=pointer]
- button "Drag and drop to reorder" [ref=f59e283]:
- img "Drag and drop to reorder" [ref=f59e284]
- generic [ref=f59e287]:
- generic [ref=f59e288]:
- button "Kick start your query" [ref=f59e289] [cursor=pointer]
- generic [ref=f59e292]:
- generic [ref=f59e293] [cursor=pointer]: Explain
- generic [ref=f59e294]:
- checkbox "Explain Toggle switch" [ref=f59e295]
- generic "Toggle switch" [ref=f59e296] [cursor=pointer]
- radiogroup [ref=f59e301]:
- generic [ref=f59e302]:
- radio "Builder" [ref=f59e303] [cursor=pointer]
- generic [ref=f59e304] [cursor=pointer]: Builder
- generic [ref=f59e305]:
- radio "Code" [checked] [ref=f59e306] [cursor=pointer]
- generic [ref=f59e307] [cursor=pointer]: Code
- generic [ref=f59e309]:
- generic [ref=f59e311]:
- button "Loading metrics..." [disabled] [ref=f59e312] [cursor=pointer]
- code [ref=f59e319]:
- generic [ref=f59e320]:
- generic [ref=f59e325]: "up{job=\"keycloak\"}"
- textbox "Editor content;Press Alt+F1 for Accessibility Options." [ref=f59e330]: "up{job=\"keycloak\"}"
- 'button "Options Legend: up {{pod}} Format: Time series Step: auto Type: Range Exemplars: false" [ref=f59e336] [cursor=pointer]':
- generic [ref=f59e340]:
- heading "Options" [level=6] [ref=f59e341]
- generic [ref=f59e342]:
- generic [ref=f59e343]: "Legend: up {{pod}}"
- generic [ref=f59e344]: "Format: Time series"
- generic [ref=f59e345]: "Step: auto"
- generic [ref=f59e346]: "Type: Range"
- generic [ref=f59e347]: "Exemplars: false"
- generic [ref=f59e348]:
- button "Add query" [ref=f59e349] [cursor=pointer]
- button "Query history" [ref=f59e353] [cursor=pointer]
- button "Query inspector" [ref=f59e357] [cursor=pointer]
- generic:
- main
- generic [ref=f59e364]:
- alert
- alert
- complementary
- complementary
@@ -0,0 +1,56 @@
- generic [ref=f62e4]:
- link "Skip to main content" [ref=f62e5] [cursor=pointer]:
- /url: "#pageContent"
- banner [ref=f62e7]:
- generic [ref=f62e8]:
- link [ref=f62e10] [cursor=pointer]:
- /url: /
- img "Grafana" [ref=f62e11]
- generic [ref=f62e14]:
- button "Search or jump to..." [ref=f62e18] [cursor=pointer]
- generic [ref=f62e19]: ctrl+k
- generic [ref=f62e23]:
- button "New" [ref=f62e24] [cursor=pointer]
- button "Help" [ref=f62e30] [cursor=pointer]
- button "News" [ref=f62e33] [cursor=pointer]
- button "Profile" [ref=f62e36] [cursor=pointer]:
- img "User avatar" [ref=f62e37]
- generic [ref=f62e38]:
- button "Open menu" [ref=f62e40] [cursor=pointer]
- navigation "Breadcrumbs" [ref=f62e43]:
- list [ref=f62e44]:
- listitem [ref=f62e45]:
- link "Home" [ref=f62e46] [cursor=pointer]:
- /url: /
- listitem [ref=f62e50]:
- link "Explore" [ref=f62e51] [cursor=pointer]:
- /url: /explore
- listitem [ref=f62e55]:
- generic "Prometheus" [ref=f62e56]
- generic [ref=f62e57]:
- button "Show more items" [ref=f62e60] [cursor=pointer]
- button "Toggle top search bar" [ref=f62e64] [cursor=pointer]
- main [ref=f62e70]:
- generic [ref=f62e72]:
- heading "Explore" [level=1] [ref=f62e73]
- generic [ref=f62e78]:
- navigation "Explore toolbar" [ref=f62e80]:
- navigation "Search links" [ref=f62e82]:
- generic [ref=f62e83]:
- button "Content outline" [expanded] [ref=f62e85] [cursor=pointer]:
- generic [ref=f62e88]: Outline
- generic [ref=f62e93] [cursor=pointer]:
- img "Prometheus logo" [ref=f62e95]
- textbox "Select a data source" [ref=f62e96]:
- /placeholder: Prometheus
- button "Show more items" [ref=f62e102] [cursor=pointer]
- generic [ref=f62e106]:
- button "Collapse outline" [expanded] [ref=f62e112] [cursor=pointer]:
- img "arrow-from-right" [ref=f62e113]
- generic [ref=f62e119]:
- generic [ref=f62e122]:
- button "Add query" [ref=f62e123] [cursor=pointer]
- button "Query history" [ref=f62e127] [cursor=pointer]
- button "Query inspector" [ref=f62e131] [cursor=pointer]
- generic:
- main
@@ -0,0 +1,29 @@
- generic [ref=f65e4]:
- link "Skip to main content" [ref=f65e5] [cursor=pointer]:
- /url: "#pageContent"
- banner [ref=f65e7]:
- generic [ref=f65e8]:
- link [ref=f65e10] [cursor=pointer]:
- /url: /
- img "Grafana" [ref=f65e11]
- generic [ref=f65e14]:
- button "Search or jump to..." [ref=f65e18] [cursor=pointer]
- generic [ref=f65e19]: ctrl+k
- generic [ref=f65e23]:
- button "New" [ref=f65e24] [cursor=pointer]
- button "Help" [ref=f65e30] [cursor=pointer]
- button "News" [ref=f65e33] [cursor=pointer]
- button "Profile" [ref=f65e36] [cursor=pointer]:
- img "User avatar" [ref=f65e37]
- generic [ref=f65e38]:
- button "Open menu" [ref=f65e40] [cursor=pointer]
- navigation "Breadcrumbs" [ref=f65e43]:
- list [ref=f65e44]:
- listitem [ref=f65e45]:
- link "Home" [ref=f65e46] [cursor=pointer]:
- /url: /
- listitem [ref=f65e50]:
- generic "Explore" [ref=f65e51]
- button "Toggle top search bar" [ref=f65e53] [cursor=pointer]
- main [ref=f65e59]:
- heading "Explore" [level=1] [ref=f65e62]
@@ -0,0 +1,16 @@
- generic [ref=f68e3]:
- banner [ref=f68e4]:
- generic [ref=f68e5]: keycloak-patterns
- main [ref=f68e6]:
- heading "Sign in to your account" [level=1] [ref=f68e8]
- generic [ref=f68e12]:
- generic [ref=f68e13]:
- generic [ref=f68e14]: Username or email
- textbox "Username or email" [ref=f68e17]
- generic [ref=f68e18]:
- generic [ref=f68e19]: Password
- generic [ref=f68e21]:
- textbox "Password" [ref=f68e24]
- button "Show password" [ref=f68e26] [cursor=pointer]:
- generic [aria-hidden] [ref=f68e27]:
- button "Sign In" [ref=f68e30] [cursor=pointer]
@@ -0,0 +1 @@
- generic [active] [ref=f69e1]: "{ \"headers\" : { \"host\" : [ \"app2.hyeonworks.com\" ], \"user-agent\" : [ \"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/152.0.0.0 Safari/537.36\" ], \"accept\" : [ \"text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7\" ], \"accept-encoding\" : [ \"gzip, deflate, br, zstd\" ], \"accept-language\" : [ \"en-US,en;q=0.9\" ], \"cache-control\" : [ \"max-age=0\" ], \"cookie\" : [ \"grafana_session=bfb5f2c2406b3f163ea01063ac1b8b83; grafana_session_expiry=1788508427; _oauth2_proxy=djIuWDI5aGRYUm9NbDl3Y205NGVTMHhNamhoWXprNFpHSmxORGRoT0RObU16UmlNemMxTTJJd016ZGlZakpqTWcuQVVUYldxYkFlejYxNXozUUZBcGh1dw==|1788508274|biz4o9E4vZXX07LnF32LU7tIEMrvfwxwCIBsZkxpwbY=\" ], \"priority\" : [ \"u=0, i\" ], \"sec-ch-ua\" : [ \"\\\"Chromium\\\";v=\\\"152\\\", \\\"Not?A_Brand\\\";v=\\\"24\\\", \\\"Google Chrome\\\";v=\\\"152\\\"\" ], \"sec-ch-ua-mobile\" : [ \"?0\" ], \"sec-ch-ua-platform\" : [ \"\\\"Linux\\\"\" ], \"sec-fetch-dest\" : [ \"document\" ], \"sec-fetch-mode\" : [ \"navigate\" ], \"sec-fetch-site\" : [ \"same-site\" ], \"sec-fetch-user\" : [ \"?1\" ], \"upgrade-insecure-requests\" : [ \"1\" ], \"x-forwarded-email\" : [ \"labuser@example.com\" ], \"x-forwarded-host\" : [ \"app2.hyeonworks.com\" ], \"x-forwarded-port\" : [ \"443\" ], \"x-forwarded-preferred-username\" : [ \"labuser\" ], \"x-forwarded-proto\" : [ \"https\" ], \"x-forwarded-server\" : [ \"traefik-5d6fcf895-wpfhr\" ], \"x-forwarded-user\" : [ \"27df5ea9-8703-4ec5-badd-d972c583e1ff\" ], \"x-real-ip\" : [ \"100.123.124.30\" ] }, \"remoteAddr\" : \"100.123.124.30\", \"localAddr\" : \"10.42.1.132\", \"scheme\" : \"https\", \"secure\" : true, \"serverName\" : \"app2.hyeonworks.com\", \"serverPort\" : 443, \"requestUrl\" : \"https://app2.hyeonworks.com/api/echo\" }"
@@ -0,0 +1 @@
- generic [ref=f70e1]: "{ \"headers\" : { \"host\" : [ \"app2.hyeonworks.com\" ], \"user-agent\" : [ \"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/152.0.0.0 Safari/537.36\" ], \"accept\" : [ \"text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7\" ], \"accept-encoding\" : [ \"gzip, deflate, br, zstd\" ], \"accept-language\" : [ \"en-US,en;q=0.9\" ], \"cookie\" : [ \"grafana_session=bfb5f2c2406b3f163ea01063ac1b8b83; grafana_session_expiry=1788508427; _oauth2_proxy=djIuWDI5aGRYUm9NbDl3Y205NGVTMWhOakZqTlRjM1ptSTVORFF5TVdWbU1qQmpaak5pWXpka1pXRTFPR1F6WWcuOXF3TmRyQzJIOFlrSDh6T1pTMl91QQ==|1788508374|Ehcd4hjNk0noEy2MSPI-C5jg84JZmilWcKhuj8OH3uw=\" ], \"priority\" : [ \"u=0, i\" ], \"sec-ch-ua\" : [ \"\\\"Chromium\\\";v=\\\"152\\\", \\\"Not?A_Brand\\\";v=\\\"24\\\", \\\"Google Chrome\\\";v=\\\"152\\\"\" ], \"sec-ch-ua-mobile\" : [ \"?0\" ], \"sec-ch-ua-platform\" : [ \"\\\"Linux\\\"\" ], \"sec-fetch-dest\" : [ \"document\" ], \"sec-fetch-mode\" : [ \"navigate\" ], \"sec-fetch-site\" : [ \"none\" ], \"sec-fetch-user\" : [ \"?1\" ], \"upgrade-insecure-requests\" : [ \"1\" ], \"x-forwarded-email\" : [ \"changed-labuser@example.com\" ], \"x-forwarded-host\" : [ \"app2.hyeonworks.com\" ], \"x-forwarded-port\" : [ \"443\" ], \"x-forwarded-preferred-username\" : [ \"labuser\" ], \"x-forwarded-proto\" : [ \"https\" ], \"x-forwarded-server\" : [ \"traefik-5d6fcf895-wpfhr\" ], \"x-forwarded-user\" : [ \"27df5ea9-8703-4ec5-badd-d972c583e1ff\" ], \"x-real-ip\" : [ \"100.123.124.30\" ] }, \"remoteAddr\" : \"100.123.124.30\", \"localAddr\" : \"10.42.0.53\", \"scheme\" : \"https\", \"secure\" : true, \"serverName\" : \"app2.hyeonworks.com\", \"serverPort\" : 443, \"requestUrl\" : \"https://app2.hyeonworks.com/api/echo\" }"
@@ -0,0 +1,16 @@
- generic [ref=e3]:
- banner [ref=e4]:
- generic [ref=e5]: keycloak-patterns
- main [ref=e6]:
- heading "Sign in to your account" [level=1] [ref=e8]
- generic [ref=e12]:
- generic [ref=e13]:
- generic [ref=e14]: Username or email
- textbox "Username or email" [active] [ref=e17]
- generic [ref=e18]:
- generic [ref=e19]: Password
- generic [ref=e21]:
- textbox "Password" [ref=e24]
- button "Show password" [ref=e26] [cursor=pointer]:
- generic [aria-hidden] [ref=e27]:
- button "Sign In" [ref=e30] [cursor=pointer]
@@ -0,0 +1,6 @@
- generic [ref=f1e3]:
- generic [ref=f1e6]:
- heading "This page isnt working" [level=1] [ref=f1e7]
- paragraph [ref=f1e8]: If the problem continues, contact the site owner.
- generic [ref=f1e9]: HTTP ERROR 401
- button "Reload" [ref=f1e12] [cursor=pointer]
@@ -0,0 +1 @@
- generic [active] [ref=f3e1]: "{\"user\":\"27df5ea9-8703-4ec5-badd-d972c583e1ff\",\"email\":\"labuser@example.com\",\"preferredUsername\":\"labuser\"}"
@@ -0,0 +1 @@
- generic [active] [ref=f4e1]: "{\"user\":\"27df5ea9-8703-4ec5-badd-d972c583e1ff\",\"email\":\"labuser@example.com\",\"preferredUsername\":\"labuser\"}"
@@ -0,0 +1 @@
- generic [active] [ref=f5e1]: Unauthorized
+18 -6
View File
@@ -18,15 +18,23 @@
브라우저 / SSH (tailnet) 브라우저 / SSH (tailnet)
│ https://{auth,app1,app2}.hyeonworks.com → 100.83.212.4 │ https://{auth,app1,app2}.hyeonworks.com → 100.83.212.4
lab host ── nginx :443 TLS 종료 · X-Forwarded-* 주입 lab host ── nftables DNAT :80,:443 → 192.168.122.10
nginx :80 301 → https (물리 호스트가 실험대를 위해 하는 일의 전부)
│ virbr0 192.168.122.0/24 (libvirt NAT) │ virbr0 192.168.122.0/24 (libvirt NAT)
├──▶ kc-lab-1 .11 k3s server Traefik :80 ├──▶ kc-lab-edge .10 nginx :443 TLS 종료 · X-Forwarded-* 주입
└──▶ kc-lab-2 .12 k3s agent Traefik :80 │ │ nginx :80 301 → https
└──▶ Pod │ │ certbot · 갱신 타이머 · deploy 훅
│ ├──▶ kc-lab-1 .11 Traefik :80 ──▶ Pod
│ └──▶ kc-lab-2 .12 Traefik :80 ──▶ Pod
├──▶ kc-lab-1 .11 k3s server
└──▶ kc-lab-2 .12 k3s agent
``` ```
**L7 홉은 두 겹 그대로다**(엣지 nginx → Traefik). 앞에 늘어난 것은 커널이
하는 L4 전달 한 번뿐이고, 그 대가로 **인증서·nginx 설정·certbot 이 전부
일회용 게스트 안**으로 들어갔다.
`nginx → Traefik` **2홉**이 운영 구조와 같다는 점이 이 배치의 핵심이다. `nginx → Traefik` **2홉**이 운영 구조와 같다는 점이 이 배치의 핵심이다.
L7 프록시가 두 겹인 이유는 역할이 다르기 때문이다 — nginx는 바깥세상과의 L7 프록시가 두 겹인 이유는 역할이 다르기 때문이다 — nginx는 바깥세상과의
접점(TLS·인증서·헤더)을, Traefik은 클러스터 내부의 동적 라우팅을 맡는다. 접점(TLS·인증서·헤더)을, Traefik은 클러스터 내부의 동적 라우팅을 맡는다.
@@ -36,7 +44,11 @@ L7 프록시가 두 겹인 이유는 역할이 다르기 때문이다 — nginx
| 경로 | 역할 | | 경로 | 역할 |
|---|---| |---|---|
| `cloud-init/kc-lab.yaml.example` | 게스트 부트스트랩 템플릿 | | `cloud-init/kc-lab.yaml.example` | 게스트 부트스트랩 템플릿 |
| `host/nginx-keycloak-lab.conf` | lab host`sites-available/keycloak-lab` | | `edge/nginx-keycloak-lab.conf` | `kc-lab-edge` `sites-available/keycloak-lab` |
| `edge/reload-nginx.sh` | certbot deploy 훅. 없으면 갱신이 서빙에 반영되지 않는다 (D-4) |
| `edge/lab-edge-dnat.nft` | 물리 호스트의 유일한 트래픽 규칙 |
| `edge/lab-edge-dnat.service` | 위 규칙을 부팅 때 적용 |
| `scripts/migrate-to-edge.sh` | 엣지 계층을 호스트에서 게스트로 옮긴다 |
| `k8s/echo.yaml` | 2홉 헤더 계약 측정용 워크로드 | | `k8s/echo.yaml` | 2홉 헤더 계약 측정용 워크로드 |
| `scripts/rebuild-seed.sh` | cloud-init 시드 ISO 재생성 + 풀 업로드 | | `scripts/rebuild-seed.sh` | cloud-init 시드 ISO 재생성 + 풀 업로드 |
| `scripts/build-and-import.sh` | 이미지 빌드 → 각 노드 containerd 반입 | | `scripts/build-and-import.sh` | 이미지 빌드 → 각 노드 containerd 반입 |
+12 -1
View File
@@ -17,7 +17,12 @@ users:
shell: /bin/bash shell: /bin/bash
# NOPASSWD is required: the k3s installer and the fault-injection scripts # NOPASSWD is required: the k3s installer and the fault-injection scripts
# run non-interactively and would block on a password prompt. # run non-interactively and would block on a password prompt.
sudo: ['ALL=(ALL) NOPASSWD:ALL'] #
# A string, not a list. The list form still boots, but `cloud-init schema -c`
# (22.4.2 on the guests) rejects it and prints the whole users.0 block with
# "is not valid under any of the given schemas" — naming no key. That makes
# the guide's own validation step look broken when it is not.
sudo: "ALL=(ALL) NOPASSWD:ALL"
# Console-only escape hatch. Without it, a cloud-init failure leaves a guest # Console-only escape hatch. Without it, a cloud-init failure leaves a guest
# that cannot be logged into at all, so its own failure log is unreadable. # that cannot be logged into at all, so its own failure log is unreadable.
# ssh_pwauth stays false, so this never widens SSH exposure. # ssh_pwauth stays false, so this never widens SSH exposure.
@@ -35,3 +40,9 @@ package_update: true
packages: packages:
- curl - curl
- nftables - nftables
# kc-lab-edge only. The k3s nodes do not need these, and the edge does not need
# anything else — nginx terminates TLS and certbot renews the certificate, both
# inside this disposable guest.
# - nginx
# - certbot
# - python3-certbot-dns-cloudflare
+33
View File
@@ -0,0 +1,33 @@
#!/usr/sbin/nft -f
# Forward the tailnet entry point to the edge guest.
#
# This is the ONLY lab traffic rule the physical host carries. Everything else
# that used to live here — nginx config, certificates, certbot, the deploy hook
# — now lives on kc-lab-edge and is destroyed with it.
#
# DNAT only, never SNAT. The guests' default route is the host, so replies come
# back through here and conntrack reverses the translation on its own. Adding a
# masquerade would rewrite the source and the edge would see 192.168.122.1 for
# every client — which would silently invalidate the X-Forwarded-For contract
# that this lab measures.
#
# PREROUTING nat runs before the routing decision, so this wins over any local
# socket on :80/:443. That makes the cutover atomic and the rollback a single
# `nft delete table ip lab_edge`.
table ip lab_edge
delete table ip lab_edge
table ip lab_edge {
chain prerouting {
type nat hook prerouting priority dstnat; policy accept;
iifname "tailscale0" tcp dport { 80, 443 } dnat to 192.168.122.10
}
# libvirt's own forward rules accept RELATED,ESTABLISHED into the guest
# subnet but not a NEW inbound connection. This runs ahead of them.
chain forward {
type filter hook forward priority filter - 10; policy accept;
ip daddr 192.168.122.10 tcp dport { 80, 443 } ct state new accept
}
}
+13
View File
@@ -0,0 +1,13 @@
[Unit]
Description=Lab edge DNAT (tailnet :80/:443 -> kc-lab-edge)
After=network-online.target libvirtd.service
Wants=network-online.target
[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=/usr/sbin/nft -f /etc/nftables.d/lab-edge-dnat.nft
ExecStop=/usr/sbin/nft delete table ip lab_edge
[Install]
WantedBy=multi-user.target
@@ -25,8 +25,10 @@ server {
} }
server { server {
listen 443 ssl default_server; # The http2 parameter of listen, not the separate `http2 on;` directive:
http2 on; # that directive needs nginx >= 1.25.1 and the edge guest is Debian 12
# (nginx 1.22). This form works on both and is what the lab actually runs.
listen 443 ssl http2 default_server;
server_name _; server_name _;
# fullchain.pem, never cert.pem: omitting the intermediates passes on # fullchain.pem, never cert.pem: omitting the intermediates passes on
+12
View File
@@ -0,0 +1,12 @@
#!/bin/sh
# certbot deploy hook. Install as
# /etc/letsencrypt/renewal-hooks/deploy/reload-nginx.sh (chmod +x)
#
# deploy/ runs only when a certificate was actually renewed (RENEWED_LINEAGE is
# set). post/ would run twice a day whether or not anything changed, reloading
# nginx for nothing.
#
# Without this, D-4 measured the failure exactly: the renewal succeeds, the
# timer reports SUCCESS, and the old certificate keeps being served for 38m25s
# — with no error anywhere.
nginx -t && nginx -s reload
+68
View File
@@ -0,0 +1,68 @@
#!/usr/bin/env bash
# Remove the lab's host layer from test-server. Packages stay.
#
# sudo bash deploy/lab/host/teardown-host.sh
#
# The host's sudo asks for a password, so run this in a terminal — not over a
# non-interactive ssh, where sudo fails silently into an empty result.
#
# ★ Certificates are BACKED UP, NOT DELETED. Let's Encrypt allows 5 duplicate
# certificates per week for the same name set, and this lab's names resolve to
# a tailnet address (100.64.0.0/10, not routable from the public internet), so
# an HTTP-01 reissue cannot be validated from here. Deleting the files turns a
# free restore into a problem that has to be solved first. Restoring is:
#
# sudo tar xzf ~/letsencrypt-backup-<stamp>.tgz -C /etc
set -u
STAMP="$(date +%Y%m%d-%H%M%S)"
HOME_DIR="${SUDO_USER:+/home/$SUDO_USER}"
HOME_DIR="${HOME_DIR:-$HOME}"
echo "===== 1) 인증서 백업 (지우지 않는다) ====="
if [ -d /etc/letsencrypt ]; then
out="$HOME_DIR/letsencrypt-backup-$STAMP.tgz"
tar czf "$out" -C /etc letsencrypt
chown "${SUDO_USER:-root}" "$out"
echo "백업: $out ($(du -h "$out" | cut -f1))"
echo "현재 인증서:"
certbot certificates 2>/dev/null | grep -E "Certificate Name|Domains|Expiry Date" || true
echo "검증 방식 (재발급이 되는지의 답):"
grep -H authenticator /etc/letsencrypt/renewal/*.conf 2>/dev/null || echo " (renewal 설정 없음)"
else
echo "/etc/letsencrypt 없음 — 건너뜀"
fi
echo
echo "===== 2) nginx 실험대 설정 제거 ====="
if [ -f /etc/nginx/sites-available/keycloak-lab ]; then
cp /etc/nginx/sites-available/keycloak-lab "$HOME_DIR/keycloak-lab.nginx.$STAMP.bak"
echo "백업: $HOME_DIR/keycloak-lab.nginx.$STAMP.bak"
fi
rm -fv /etc/nginx/sites-enabled/keycloak-lab
rm -fv /etc/nginx/sites-available/keycloak-lab
systemctl disable --now nginx
echo
echo "===== 3) certbot 갱신 타이머 정지 ====="
# 인증서 파일은 남기지만, 갱신 시도는 멈춘다. 지금 DNS 로는 HTTP-01 검증이
# 실패하고, 실패가 로그에만 쌓이면서 「왜 안 되지」의 원인이 된다.
systemctl disable --now certbot-renew.timer 2>/dev/null || true
echo
echo "===== 4) 엣지 DNAT (있으면) ====="
systemctl disable --now lab-edge-dnat.service 2>/dev/null || true
rm -fv /etc/systemd/system/lab-edge-dnat.service /etc/nftables.d/lab-edge-dnat.nft
systemctl daemon-reload
nft delete table ip lab_edge 2>/dev/null || true
echo
echo "===== 5) 확인 ====="
echo "-- nginx: $(systemctl is-active nginx) / $(systemctl is-enabled nginx 2>&1)"
echo "-- certbot timer: $(systemctl is-active certbot-renew.timer 2>&1) / $(systemctl is-enabled certbot-renew.timer 2>&1)"
echo "-- 80/443 리스너:"; ss -tlnp | grep -E ':(80|443) ' || echo " (없음 — 정상)"
echo "-- sites-enabled:"; ls -A /etc/nginx/sites-enabled 2>/dev/null || echo " (비었음 — 정상)"
echo "-- letsencrypt:"; ls /etc/letsencrypt/live 2>/dev/null || echo " (없음)"
echo "-- libvirt 도메인:"; virsh list --all 2>/dev/null | tail -n +3 | grep -v '^$' || echo " (없음 — 정상)"
echo
echo "완료. 패키지(nginx · libvirt · qemu · certbot · kubectl)와 base.qcow2 는 남아 있다."
+12
View File
@@ -0,0 +1,12 @@
# 다이어그램 규약
| 표현 | 뜻 |
|---|---|
| 실선 상자 | 살아 있는 구성 요소 |
| 붉은 점선 상자 | 이 실험에서 죽이거나 막은 것 |
| ✂ 붉은 X | 주입 지점 |
| 실선 화살표 | 정상 경로 |
| 붉은 점선 화살표 | 실험에서 깨진 경로 |
| 회색 글씨 | 측정값 |
SVG 는 GitHub 에서 그대로 렌더링되며 외부 폰트를 쓰지 않는다.
+48
View File
@@ -0,0 +1,48 @@
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 860 400" font-family="-apple-system,BlinkMacSystemFont,Segoe UI,Helvetica,sans-serif">
<style>
.box{fill:#fff;stroke:#8c959f;stroke-width:1.4;rx:6}
.dead{fill:#fff5f5;stroke:#cf222e;stroke-width:1.6;stroke-dasharray:5 3;rx:6}
.ok{fill:#f6fdf6;stroke:#1a7f37;stroke-width:1.6;rx:6}
.warn{fill:#fff8f0;stroke:#bf8700;stroke-width:1.5;rx:6}
.t{font-size:12px;fill:#1f2328}
.tb{font-size:12.5px;fill:#1f2328;font-weight:600}
.s{font-size:10.5px;fill:#656d76}
.ttl{font-size:14px;fill:#1f2328;font-weight:600}
.r{stroke:#cf222e;stroke-width:1.6;fill:none}
.g{stroke:#1a7f37;stroke-width:1.6;fill:none}
.k{stroke:#8c959f;stroke-width:1.3;fill:none}
</style>
<defs>
<marker id="a" markerWidth="8" markerHeight="8" refX="7" refY="3" orient="auto"><path d="M0,0 L7,3 L0,6 z" fill="#8c959f"/></marker>
<marker id="ar" markerWidth="8" markerHeight="8" refX="7" refY="3" orient="auto"><path d="M0,0 L7,3 L0,6 z" fill="#cf222e"/></marker>
<marker id="ag" markerWidth="8" markerHeight="8" refX="7" refY="3" orient="auto"><path d="M0,0 L7,3 L0,6 z" fill="#1a7f37"/></marker>
</defs>
<text x="20" y="26" class="ttl">A-0 — 세션을 공유하는 것은 Infinispan 이 아니라 PostgreSQL 이다</text>
<text x="20" y="44" class="s">클러스터가 형성됐다는 것과 세션이 복제된다는 것은 다른 얘기였다</text>
<rect x="30" y="76" width="330" height="120" class="box"/>
<text x="46" y="98" class="tb">keycloak-0 (kc-lab-2)</text>
<rect x="46" y="112" width="298" height="34" class="box"/>
<text x="58" y="134" class="t">Infinispan sessions 캐시</text>
<text x="46" y="166" class="s">자기가 처리한 로그인만 들어 있다</text>
<text x="46" y="184" class="s">entries_unique 는 노드마다 다르다</text>
<rect x="500" y="76" width="330" height="120" class="box"/>
<text x="516" y="98" class="tb">keycloak-1 (kc-lab-1)</text>
<rect x="516" y="112" width="298" height="34" class="box"/>
<text x="528" y="134" class="t">Infinispan sessions 캐시</text>
<text x="516" y="166" class="s">여기에도 자기 것만 있다</text>
<text x="516" y="184" class="s">상대 세션은 들어오지 않는다</text>
<path d="M362 130 L498 130" class="r" stroke-dasharray="5 3" marker-end="url(#ar)"/>
<path d="M498 146 L362 146" class="r" stroke-dasharray="5 3" marker-end="url(#ar)"/>
<text x="430" y="122" class="tb" text-anchor="middle" fill="#cf222e">복제 없음</text>
<text x="430" y="170" class="s" text-anchor="middle" fill="#cf222e">세션 엔트리는 노드 사이를 건너가지 않는다</text>
<rect x="300" y="250" width="260" height="76" class="ok"/>
<text x="316" y="272" class="tb">PostgreSQL</text>
<text x="316" y="292" class="s">OFFLINE_USER_SESSION (offline_flag='0')</text>
<text x="316" y="310" class="s">두 노드가 같은 행을 본다</text>
<path d="M150 198 L330 246" class="g" marker-end="url(#ag)"/>
<path d="M710 198 L530 246" class="g" marker-end="url(#ag)"/>
<text x="180" y="232" class="s">SELECT / INSERT</text>
<text x="600" y="232" class="s">SELECT / INSERT</text>
<rect x="30" y="346" width="800" height="40" class="box"/>
<text x="46" y="370" class="s">근거 — 노드 A 로 로그인하고 노드 B 로 refresh 했을 때, 반대편 노드가 날린 SQL 을 문장 로깅으로 직접 잡았다</text>
</svg>

After

Width:  |  Height:  |  Size: 3.2 KiB

+47
View File
@@ -0,0 +1,47 @@
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 860 430" font-family="-apple-system,BlinkMacSystemFont,Segoe UI,Helvetica,sans-serif">
<style>
.box{fill:#fff;stroke:#8c959f;stroke-width:1.4;rx:6}
.dead{fill:#fff5f5;stroke:#cf222e;stroke-width:1.6;stroke-dasharray:5 3;rx:6}
.ok{fill:#f6fdf6;stroke:#1a7f37;stroke-width:1.6;rx:6}
.warn{fill:#fff8f0;stroke:#bf8700;stroke-width:1.5;rx:6}
.t{font-size:12px;fill:#1f2328}
.tb{font-size:12.5px;fill:#1f2328;font-weight:600}
.s{font-size:10.5px;fill:#656d76}
.ttl{font-size:14px;fill:#1f2328;font-weight:600}
.r{stroke:#cf222e;stroke-width:1.6;fill:none}
.g{stroke:#1a7f37;stroke-width:1.6;fill:none}
.k{stroke:#8c959f;stroke-width:1.3;fill:none}
</style>
<defs>
<marker id="a" markerWidth="8" markerHeight="8" refX="7" refY="3" orient="auto"><path d="M0,0 L7,3 L0,6 z" fill="#8c959f"/></marker>
<marker id="ar" markerWidth="8" markerHeight="8" refX="7" refY="3" orient="auto"><path d="M0,0 L7,3 L0,6 z" fill="#cf222e"/></marker>
<marker id="ag" markerWidth="8" markerHeight="8" refX="7" refY="3" orient="auto"><path d="M0,0 L7,3 L0,6 z" fill="#1a7f37"/></marker>
</defs>
<text x="20" y="26" class="ttl">A-1 — 7800 을 막아도 세션 공유는 안 깨진다. 깨지는 것은 로그아웃 전파다</text>
<text x="20" y="44" class="s">예측 하나가 빗나갔고, 예상 못 한 것이 둘 나왔다</text>
<rect x="30" y="72" width="300" height="104" class="box"/>
<text x="46" y="94" class="tb">keycloak-0</text>
<text x="46" y="116" class="s">로그인 처리 · 세션을 DB 에 쓴다</text>
<rect x="530" y="72" width="300" height="104" class="box"/>
<text x="546" y="94" class="tb">keycloak-1</text>
<text x="546" y="116" class="s">DB 를 읽어 같은 세션을 안다</text>
<rect x="352" y="86" width="156" height="34" class="dead"/>
<text x="430" y="107" class="t" text-anchor="middle" fill="#cf222e">TCP 7800 차단</text>
<text x="430" y="140" class="s" text-anchor="middle">NetworkPolicy 는 허용목록이다</text>
<text x="430" y="156" class="s" text-anchor="middle">8080·9000 만 열고 7800 은 누락시킨다</text>
<rect x="300" y="206" width="260" height="64" class="ok"/>
<text x="316" y="228" class="tb">PostgreSQL</text>
<text x="316" y="248" class="s">세션은 여기 있다 → 교차 노드 refresh 200</text>
<path d="M150 178 L320 202" class="g" marker-end="url(#ag)"/>
<path d="M710 178 L540 202" class="g" marker-end="url(#ag)"/>
<rect x="30" y="290" width="390" height="76" class="dead"/>
<text x="46" y="312" class="tb">깨진 것 — 로그아웃 전파</text>
<text x="46" y="332" class="s">무효화 통지가 7800 을 탄다</text>
<text x="46" y="350" class="s">400 이어야 할 refresh 가 200 을 반환했다</text>
<rect x="440" y="290" width="390" height="76" class="warn"/>
<text x="456" y="312" class="tb">★ 주입이 먹지 않았다 — conntrack</text>
<text x="456" y="332" class="s">ESTABLISHED 연결은 규칙 평가를 건너뛴다</text>
<text x="456" y="350" class="s">cluster_size 가 25분간 2 로 남았다 → 삭제 후에야 갈렸다</text>
<text x="30" y="392" class="s">그리고 예상 못 한 둘째 — 분단된 노드가 readiness 실패로 스스로 로드밸런서에서 빠진다</text>
<text x="30" y="410" class="s">덕분에 외부에서는 장애가 보이지 않는다. 관측 지점을 밖에만 두면 이 실험은 「아무 일도 없음」으로 보인다</text>
</svg>

After

Width:  |  Height:  |  Size: 3.3 KiB

+26
View File
@@ -0,0 +1,26 @@
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 700 300" width="700" height="300" font-family="system-ui,-apple-system,Segoe UI,Roboto,sans-serif">
<style>
.t{font-size:13px;fill:#1f2328}.s{font-size:11px;fill:#59636e}.h{font-size:14px;font-weight:600;fill:#1f2328}
.r{font-size:12px;fill:#cf222e;font-weight:600}
.box{fill:#fff;stroke:#8c959f;stroke-width:1.4;rx:6}
.dead{fill:#fff5f5;stroke:#cf222e;stroke-width:1.6;stroke-dasharray:5 3;rx:6}
.ok{fill:#f6fdf6;stroke:#1a7f37;stroke-width:1.6;rx:6}
.ln{stroke:#59636e;stroke-width:1.4;fill:none;marker-end:url(#a)}
.bad{stroke:#cf222e;stroke-width:1.6;stroke-dasharray:5 3;fill:none;marker-end:url(#b)}
</style>
<defs>
<marker id="a" markerWidth="9" markerHeight="9" refX="8" refY="4.5" orient="auto"><path d="M0,0 L9,4.5 L0,9 z" fill="#59636e"/></marker>
<marker id="b" markerWidth="9" markerHeight="9" refX="8" refY="4.5" orient="auto"><path d="M0,0 L9,4.5 L0,9 z" fill="#cf222e"/></marker>
</defs>
<text class="h" x="16" y="24">A-2 · PostgreSQL 정지 — 살아남는 노드가 없다</text>
<rect class="box" x="30" y="46" width="180" height="52"/><text class="t" x="120" y="68" text-anchor="middle">keycloak-1</text><text class="s" x="120" y="86" text-anchor="middle">캐시: 세션 N개</text>
<rect class="box" x="480" y="46" width="180" height="52"/><text class="t" x="570" y="68" text-anchor="middle">keycloak-0</text><text class="s" x="570" y="86" text-anchor="middle">캐시: 세션 M개</text>
<path class="ln" d="M210,72 L480,72"/><text class="s" x="345" y="66" text-anchor="middle">7800 · 살아 있다</text>
<path class="bad" d="M120,100 L300,150"/><path class="bad" d="M570,100 L400,150"/>
<rect class="dead" x="270" y="156" width="160" height="52"/>
<text class="r" x="350" y="178" text-anchor="middle">postgres ✗</text><text class="s" x="350" y="196" text-anchor="middle">replicas=0</text>
<rect class="dead" x="30" y="228" width="290" height="52"/>
<text class="r" x="175" y="250" text-anchor="middle">양쪽 모두 NotReady</text><text class="s" x="175" y="268" text-anchor="middle">ready 주소 = [] · 외부 503</text>
<rect class="box" x="370" y="228" width="290" height="52"/>
<text class="t" x="515" y="250" text-anchor="middle">up{job="keycloak"} = 1</text><text class="s" x="515" y="268" text-anchor="middle">프로세스는 살아 있다 — up 은 못 잡는다</text>
</svg>

After

Width:  |  Height:  |  Size: 2.4 KiB

+55
View File
@@ -0,0 +1,55 @@
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 860 400" font-family="-apple-system,BlinkMacSystemFont,Segoe UI,Helvetica,sans-serif">
<style>
.box{fill:#fff;stroke:#8c959f;stroke-width:1.4;rx:6}
.dead{fill:#fff5f5;stroke:#cf222e;stroke-width:1.6;stroke-dasharray:5 3;rx:6}
.ok{fill:#f6fdf6;stroke:#1a7f37;stroke-width:1.6;rx:6}
.warn{fill:#fff8f0;stroke:#bf8700;stroke-width:1.5;rx:6}
.t{font-size:12px;fill:#1f2328}
.tb{font-size:12.5px;fill:#1f2328;font-weight:600}
.s{font-size:10.5px;fill:#656d76}
.ttl{font-size:14px;fill:#1f2328;font-weight:600}
.r{stroke:#cf222e;stroke-width:1.6;fill:none}
.g{stroke:#1a7f37;stroke-width:1.6;fill:none}
.k{stroke:#8c959f;stroke-width:1.3;fill:none}
</style>
<defs>
<marker id="a" markerWidth="8" markerHeight="8" refX="7" refY="3" orient="auto"><path d="M0,0 L7,3 L0,6 z" fill="#8c959f"/></marker>
<marker id="ar" markerWidth="8" markerHeight="8" refX="7" refY="3" orient="auto"><path d="M0,0 L7,3 L0,6 z" fill="#cf222e"/></marker>
<marker id="ag" markerWidth="8" markerHeight="8" refX="7" refY="3" orient="auto"><path d="M0,0 L7,3 L0,6 z" fill="#1a7f37"/></marker>
</defs>
<text x="20" y="26" class="ttl">A-3 — 200 을 받은 로그인 153건 중 4건이 DB 에 없다</text>
<text x="20" y="44" class="s">버그가 아니라 synchronous_commit OFF 의 대가를 실측한 것이다</text>
<rect x="30" y="74" width="180" height="70" class="box"/>
<text x="46" y="96" class="tb">클라이언트</text>
<text x="46" y="116" class="s">로그인 요청</text>
<text x="46" y="134" class="s">200 + 토큰 수신</text>
<rect x="250" y="74" width="200" height="70" class="box"/>
<text x="266" y="96" class="tb">Keycloak</text>
<text x="266" y="116" class="s">트랜잭션마다</text>
<text x="266" y="134" class="s">SET LOCAL synchronous_commit OFF</text>
<rect x="490" y="74" width="180" height="70" class="warn"/>
<text x="506" y="96" class="tb">PostgreSQL</text>
<text x="506" y="116" class="s">COMMIT 을 즉시 반환</text>
<text x="506" y="134" class="s">WAL 은 아직 메모리에</text>
<rect x="710" y="74" width="120" height="70" class="box"/>
<text x="726" y="96" class="tb">디스크</text>
<text x="726" y="116" class="s">wal_writer_delay</text>
<text x="726" y="134" class="s">200ms 뒤 기록</text>
<path d="M212 108 L246 108" class="k" marker-end="url(#a)"/>
<path d="M452 108 L486 108" class="k" marker-end="url(#a)"/>
<path d="M672 108 L706 108" class="r" stroke-dasharray="4 3" marker-end="url(#ar)"/>
<text x="676" y="164" class="s" fill="#cf222e">이 구간이 비어 있다</text>
<rect x="620" y="180" width="210" height="40" class="dead"/>
<text x="636" y="205" class="t" fill="#cf222e">✂ 여기서 강제 종료</text>
<rect x="30" y="250" width="380" height="96" class="dead"/>
<text x="46" y="272" class="tb">결과 — RPO 는 0 이 아니다</text>
<text x="46" y="296" class="t">클라이언트가 200 받은 로그인 153 건</text>
<text x="46" y="316" class="t">그중 DB 에 실제로 존재 149 건</text>
<text x="46" y="336" class="t" fill="#cf222e">★ 유실 4 건</text>
<rect x="430" y="250" width="400" height="96" class="box"/>
<text x="446" y="272" class="tb">주입도 두 번 실패했다</text>
<text x="446" y="294" class="s">kubectl delete --force 는 크래시가 아니다 — 런타임이 SIGTERM 을 보내</text>
<text x="446" y="310" class="s">PostgreSQL 이 정상 플러시했고 유실이 0 이었다</text>
<text x="446" y="332" class="s">kill -9 1 도 무시된다 — PID 1 은 자기 네임스페이스의 SIGKILL 을 받지 않는다</text>
<text x="30" y="372" class="s">백엔드 프로세스를 죽여 postmaster 가 reinitialize 하게 만들자 비로소 "not properly shut down / redo starts" 가 찍혔다</text>
</svg>

After

Width:  |  Height:  |  Size: 3.7 KiB

+52
View File
@@ -0,0 +1,52 @@
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 860 420" font-family="-apple-system,BlinkMacSystemFont,Segoe UI,Helvetica,sans-serif">
<style>
.box{fill:#fff;stroke:#8c959f;stroke-width:1.4;rx:6}
.dead{fill:#fff5f5;stroke:#cf222e;stroke-width:1.6;stroke-dasharray:5 3;rx:6}
.ok{fill:#f6fdf6;stroke:#1a7f37;stroke-width:1.6;rx:6}
.warn{fill:#fff8f0;stroke:#bf8700;stroke-width:1.5;rx:6}
.t{font-size:12px;fill:#1f2328}
.tb{font-size:12.5px;fill:#1f2328;font-weight:600}
.s{font-size:10.5px;fill:#656d76}
.ttl{font-size:14px;fill:#1f2328;font-weight:600}
.r{stroke:#cf222e;stroke-width:1.6;fill:none}
.g{stroke:#1a7f37;stroke-width:1.6;fill:none}
.k{stroke:#8c959f;stroke-width:1.3;fill:none}
</style>
<defs>
<marker id="a" markerWidth="8" markerHeight="8" refX="7" refY="3" orient="auto"><path d="M0,0 L7,3 L0,6 z" fill="#8c959f"/></marker>
<marker id="ar" markerWidth="8" markerHeight="8" refX="7" refY="3" orient="auto"><path d="M0,0 L7,3 L0,6 z" fill="#cf222e"/></marker>
<marker id="ag" markerWidth="8" markerHeight="8" refX="7" refY="3" orient="auto"><path d="M0,0 L7,3 L0,6 z" fill="#1a7f37"/></marker>
</defs>
<text x="20" y="26" class="ttl">A-4 — 둘 다 전면 장애지만 이유가 다르다</text>
<text x="20" y="44" class="s">그리고 죽은 파드가 산 파드보다 건강해 보인다</text>
<rect x="30" y="70" width="390" height="150" class="box"/>
<text x="46" y="92" class="tb">4a — 워커(kc-lab-2) 상실</text>
<rect x="46" y="104" width="170" height="46" class="dead"/>
<text x="58" y="124" class="t">keycloak-0</text>
<text x="58" y="142" class="s">PostgreSQL 도 여기 있었다</text>
<rect x="236" y="104" width="170" height="46" class="warn"/>
<text x="248" y="124" class="t">keycloak-1 살아있음</text>
<text x="248" y="142" class="s">그러나 DB 가 없다</text>
<text x="46" y="172" class="t">외부 응답 <tspan fill="#cf222e" font-weight="600">503</tspan> · kubectl 정상</text>
<text x="46" y="194" class="s">DB 가 같이 죽어서 장애다. 남은 파드는 돌지만 아무것도 못 한다</text>
<text x="46" y="212" class="s">PVC 가 local-path 라 다른 노드로 재배치되지 않는다</text>
<rect x="440" y="70" width="390" height="150" class="box"/>
<text x="456" y="92" class="tb">4b — 컨트롤 플레인(kc-lab-1) 상실</text>
<rect x="456" y="104" width="170" height="46" class="ok"/>
<text x="468" y="124" class="t">keycloak-0</text>
<text x="468" y="142" class="s">★ 계속 돌고 있다</text>
<rect x="646" y="104" width="170" height="46" class="dead"/>
<text x="658" y="124" class="t">API 서버 · traefik</text>
<text x="658" y="142" class="s">들어갈 길이 없다</text>
<text x="456" y="172" class="t">외부 응답 <tspan fill="#cf222e" font-weight="600">000</tspan> · kubectl <tspan fill="#cf222e">불통</tspan></text>
<text x="456" y="194" class="s">워크로드는 멀쩡한데 도달할 수 없어 장애다</text>
<text x="456" y="212" class="s">진입점이 단일 노드에 있으면 워크로드 이중화는 의미가 없다</text>
<rect x="30" y="242" width="800" height="86" class="warn"/>
<text x="46" y="264" class="tb">★ 예상하지 못한 것 셋</text>
<text x="46" y="286" class="s">1. 죽은 파드가 산 파드보다 건강해 보인다 — kubelet 이 사라져 상태가 갱신되지 않으니 Running 으로 남는다</text>
<text x="46" y="304" class="s">2. StatefulSet 은 Terminating 파드의 대체를 만들지 않는다 — 이름이 같아야 하므로 지워지기를 기다린다</text>
<text x="46" y="322" class="s">3. node-monitor-grace-period 40초 + tolerationSeconds 300초 = 축출까지 5분 40초</text>
<rect x="30" y="348" width="800" height="46" class="ok"/>
<text x="46" y="370" class="tb">복구는 둘 다 virsh start 이후 60초</text>
<text x="46" y="388" class="s">장애 시간의 대부분은 복구가 아니라 "누가 죽은 것을 알아채는 데" 걸린 시간이다</text>
</svg>

After

Width:  |  Height:  |  Size: 3.9 KiB

+49
View File
@@ -0,0 +1,49 @@
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 860 420" font-family="-apple-system,BlinkMacSystemFont,Segoe UI,Helvetica,sans-serif">
<style>
.box{fill:#fff;stroke:#8c959f;stroke-width:1.4;rx:6}
.dead{fill:#fff5f5;stroke:#cf222e;stroke-width:1.6;stroke-dasharray:5 3;rx:6}
.ok{fill:#f6fdf6;stroke:#1a7f37;stroke-width:1.6;rx:6}
.warn{fill:#fff8f0;stroke:#bf8700;stroke-width:1.5;rx:6}
.t{font-size:12px;fill:#1f2328}
.tb{font-size:12.5px;fill:#1f2328;font-weight:600}
.s{font-size:10.5px;fill:#656d76}
.ttl{font-size:14px;fill:#1f2328;font-weight:600}
.r{stroke:#cf222e;stroke-width:1.6;fill:none}
.g{stroke:#1a7f37;stroke-width:1.6;fill:none}
.k{stroke:#8c959f;stroke-width:1.3;fill:none}
</style>
<defs>
<marker id="a" markerWidth="8" markerHeight="8" refX="7" refY="3" orient="auto"><path d="M0,0 L7,3 L0,6 z" fill="#8c959f"/></marker>
<marker id="ar" markerWidth="8" markerHeight="8" refX="7" refY="3" orient="auto"><path d="M0,0 L7,3 L0,6 z" fill="#cf222e"/></marker>
<marker id="ag" markerWidth="8" markerHeight="8" refX="7" refY="3" orient="auto"><path d="M0,0 L7,3 L0,6 z" fill="#1a7f37"/></marker>
</defs>
<text x="20" y="26" class="ttl">A-5 — 비대칭 차단은 클러스터를 가르지 못한다. 그리고 갈라도 서비스는 계속된다</text>
<text x="20" y="44" class="s">주입을 세 번 실패했고, 세 번 모두 「아무 일도 없었다」로 보였다</text>
<rect x="30" y="72" width="390" height="112" class="box"/>
<text x="46" y="94" class="tb">① 한 방향만 차단</text>
<rect x="46" y="106" width="130" height="34" class="box"/>
<text x="58" y="128" class="t">keycloak-0</text>
<rect x="290" y="106" width="116" height="34" class="box"/>
<text x="302" y="128" class="t">keycloak-1</text>
<path d="M178 116 L286 116" class="r" stroke-dasharray="4 3" marker-end="url(#ar)"/>
<path d="M286 132 L178 132" class="g" marker-end="url(#ag)"/>
<text x="46" y="164" class="s" fill="#1a7f37">열린 방향으로 재연결한다 → cluster_size 2 유지. 가르지 못한다</text>
<rect x="440" y="72" width="390" height="112" class="box"/>
<text x="456" y="94" class="tb">② 양방향 완전 차단</text>
<rect x="456" y="106" width="130" height="34" class="dead"/>
<text x="468" y="128" class="t">keycloak-0</text>
<rect x="700" y="106" width="116" height="34" class="ok"/>
<text x="712" y="128" class="t">keycloak-1</text>
<path d="M588 116 L696 116" class="r" stroke-dasharray="4 3" marker-end="url(#ar)"/>
<path d="M696 132 L588 132" class="r" stroke-dasharray="4 3" marker-end="url(#ar)"/>
<text x="456" y="164" class="s">양쪽 모두 멤버 1개. 그런데 <tspan fill="#cf222e" font-weight="600">한쪽만 DOWN</tspan> 이 된다 — 코디네이터 쪽이 살아남는다</text>
<rect x="30" y="204" width="800" height="48" class="ok"/>
<text x="46" y="226" class="tb">그래서 전면 장애 경로가 없다 — 외부 응답 200 유지</text>
<text x="46" y="244" class="s">분단된 쪽이 스스로 readiness 를 떨어뜨려 로드밸런서에서 빠지고, 남은 쪽이 계속 응답한다</text>
<rect x="30" y="272" width="800" height="130" class="warn"/>
<text x="46" y="294" class="tb">★ 세 번의 주입 실패 — 전부 「조용히」 실패했다</text>
<text x="46" y="318" class="s">1. iptables -I FORWARD 1 이 무시됐다 — kube-router 가 자기 체인을 FORWARD 맨 위에 다시 끼워 넣는다 (패킷 0)</text>
<text x="46" y="338" class="s">2. 방향이 뒤집혀 있었다 — JGroups 의 client/server 역할은 재시작마다 바뀐다. raw 규칙이 엉뚱한 노드에 걸렸다</text>
<text x="46" y="358" class="s">3. dev eth0 이 없다 — Debian 은 enp1s0 이고, 게다가 flannel VXLAN 이 이미 캡슐화해 파드 IP 가 안 보인다</text>
<text x="46" y="382" class="s" fill="#cf222e">해결: raw 테이블 PREROUTING(conntrack 보다 먼저) + flannel.1 인터페이스에서 필터</text>
</svg>

After

Width:  |  Height:  |  Size: 3.8 KiB

+51
View File
@@ -0,0 +1,51 @@
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 860 400" font-family="-apple-system,BlinkMacSystemFont,Segoe UI,Helvetica,sans-serif">
<style>
.box{fill:#fff;stroke:#8c959f;stroke-width:1.4;rx:6}
.dead{fill:#fff5f5;stroke:#cf222e;stroke-width:1.6;stroke-dasharray:5 3;rx:6}
.ok{fill:#f6fdf6;stroke:#1a7f37;stroke-width:1.6;rx:6}
.warn{fill:#fff8f0;stroke:#bf8700;stroke-width:1.5;rx:6}
.t{font-size:12px;fill:#1f2328}
.tb{font-size:12.5px;fill:#1f2328;font-weight:600}
.s{font-size:10.5px;fill:#656d76}
.ttl{font-size:14px;fill:#1f2328;font-weight:600}
.r{stroke:#cf222e;stroke-width:1.6;fill:none}
.g{stroke:#1a7f37;stroke-width:1.6;fill:none}
.k{stroke:#8c959f;stroke-width:1.3;fill:none}
</style>
<defs>
<marker id="a" markerWidth="8" markerHeight="8" refX="7" refY="3" orient="auto"><path d="M0,0 L7,3 L0,6 z" fill="#8c959f"/></marker>
<marker id="ar" markerWidth="8" markerHeight="8" refX="7" refY="3" orient="auto"><path d="M0,0 L7,3 L0,6 z" fill="#cf222e"/></marker>
<marker id="ag" markerWidth="8" markerHeight="8" refX="7" refY="3" orient="auto"><path d="M0,0 L7,3 L0,6 z" fill="#1a7f37"/></marker>
</defs>
<text x="20" y="26" class="ttl">A-6 — 200밀리초가 22초가 된다</text>
<text x="20" y="44" class="s">지연은 왕복 횟수만큼 곱해지고, 커넥션 풀에서 한 번 더 곱해진다</text>
<rect x="30" y="76" width="180" height="60" class="ok"/>
<text x="46" y="98" class="tb">대조군 keycloak-0</text>
<text x="46" y="120" class="s">DB 와 같은 노드 · 66ms</text>
<rect x="250" y="76" width="180" height="60" class="dead"/>
<text x="266" y="98" class="tb">시험군 keycloak-1</text>
<text x="266" y="120" class="s">tc netem delay 200ms</text>
<text x="230" y="112" class="tb" fill="#cf222e"></text>
<rect x="470" y="76" width="180" height="60" class="warn"/>
<text x="486" y="98" class="tb">단일 요청</text>
<text x="486" y="120" class="t">1,872 ms <tspan class="s">(28배)</tspan></text>
<rect x="690" y="76" width="140" height="60" class="dead"/>
<text x="706" y="98" class="tb">동시 20건</text>
<text x="706" y="120" class="t" fill="#cf222e">22.2 초</text>
<path d="M432 106 L466 106" class="k" marker-end="url(#a)"/>
<path d="M652 106 L686 106" class="r" marker-end="url(#ar)"/>
<text x="20" y="176" class="tb">왜 200ms 가 1,872ms 가 되는가 — 왕복 횟수</text>
<rect x="30" y="190" width="800" height="46" class="box"/>
<text x="46" y="210" class="s">로그인 한 번은 DB 왕복을 여러 번 한다: 클라이언트 조회 · 사용자 조회 · 세션 INSERT · 커밋 …</text>
<text x="46" y="228" class="s">왕복마다 200ms 가 더해진다. 애플리케이션은 아무것도 잘못하지 않았다</text>
<text x="20" y="266" class="tb">왜 1,872ms 가 22.2초가 되는가 — 커넥션 풀</text>
<rect x="30" y="280" width="390" height="76" class="dead"/>
<text x="46" y="302" class="s">요청이 커넥션을 오래 붙들고 있으면</text>
<text x="46" y="320" class="s">뒤의 요청은 풀에서 대기한다</text>
<text x="46" y="342" class="t" fill="#cf222e">agroal 커넥션 획득 대기 최대 20,000 ms</text>
<rect x="440" y="280" width="390" height="76" class="warn"/>
<text x="456" y="302" class="tb">그리고 파드가 죽는다</text>
<text x="456" y="322" class="s">readiness 프로브가 타임아웃으로 실패 →</text>
<text x="456" y="342" class="s">느린 노드가 로드밸런서에서 빠진다. 느림이 장애로 승격된다</text>
<text x="30" y="384" class="s">측정 장치 주의 — 동시 20건을 kubectl run --rm -i 로 돌리면 출력 스트림이 유실된다. 상주 탐침에 파일로 모아야 한다</text>
</svg>

After

Width:  |  Height:  |  Size: 3.6 KiB

+36
View File
@@ -0,0 +1,36 @@
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 720 330" width="720" height="330" font-family="system-ui,-apple-system,Segoe UI,Roboto,sans-serif">
<style>
.t{font-size:13px;fill:#1f2328}.s{font-size:11px;fill:#59636e}.h{font-size:14px;font-weight:600;fill:#1f2328}
.r{font-size:12px;fill:#cf222e;font-weight:600}.g{font-size:12px;fill:#1a7f37;font-weight:600}
.box{fill:#fff;stroke:#8c959f;stroke-width:1.4;rx:6}
.ok{fill:#f6fdf6;stroke:#1a7f37;stroke-width:1.6;rx:6}
.dead{fill:#fff5f5;stroke:#cf222e;stroke-width:1.6;stroke-dasharray:5 3;rx:6}
.ln{stroke:#59636e;stroke-width:1.4;fill:none;marker-end:url(#a)}
</style>
<defs><marker id="a" markerWidth="9" markerHeight="9" refX="8" refY="4.5" orient="auto"><path d="M0,0 L9,4.5 L0,9 z" fill="#59636e"/></marker></defs>
<text class="h" x="16" y="24">A-7 · 같은 주입, 같은 관측, 정반대 결과</text>
<text class="h" x="180" y="52" text-anchor="middle">persistent (KC 26 기본)</text>
<rect class="box" x="30" y="62" width="300" height="54"/>
<text class="t" x="180" y="84" text-anchor="middle">keycloak ×2 — 로컬 캐시</text>
<path class="ln" d="M180,116 L180,140"/>
<rect class="ok" x="70" y="142" width="220" height="40"/>
<text class="g" x="180" y="167" text-anchor="middle">PostgreSQL — 진실의 원천</text>
<text class="h" x="540" y="52" text-anchor="middle">volatile (KC 24 이전 방식)</text>
<rect class="box" x="390" y="62" width="300" height="54"/>
<text class="t" x="540" y="84" text-anchor="middle">keycloak ×2 — 캐시가 곧 진실</text>
<path class="ln" d="M470,116 L470,140"/><path class="ln" d="M610,140 L610,116"/>
<rect class="ok" x="430" y="142" width="220" height="40"/>
<text class="g" x="540" y="167" text-anchor="middle">클러스터 복제 (7800)</text>
<rect class="box" x="30" y="202" width="660" height="112"/>
<text class="t" x="360" y="224" text-anchor="middle">뒤집힌 세 결과</text>
<text class="s" x="200" y="248" text-anchor="middle">A-1 7800 차단 후 교차 refresh</text>
<text class="g" x="430" y="248" text-anchor="middle">200</text><text class="r" x="560" y="248" text-anchor="middle">400 Session not active</text>
<text class="s" x="200" y="272" text-anchor="middle">A-8 롤링 재시작 후 refresh</text>
<text class="g" x="430" y="272" text-anchor="middle">200</text><text class="r" x="560" y="272" text-anchor="middle">400 Session not active</text>
<text class="s" x="200" y="296" text-anchor="middle">A-2 DB 정지 중 새 로그인</text>
<text class="r" x="430" y="296" text-anchor="middle">500</text><text class="g" x="560" y="296" text-anchor="middle">200</text>
<text class="s" x="430" y="230" text-anchor="middle">persistent</text><text class="s" x="560" y="230" text-anchor="middle">volatile</text>
</svg>

After

Width:  |  Height:  |  Size: 2.8 KiB

+70
View File
@@ -0,0 +1,70 @@
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 860 470" font-family="-apple-system,BlinkMacSystemFont,'Segoe UI',Helvetica,sans-serif">
<style>
.box{fill:#fff;stroke:#8c959f;stroke-width:1.4;rx:6}
.dead{fill:#fff5f5;stroke:#cf222e;stroke-width:1.6;stroke-dasharray:5 3;rx:6}
.ok{fill:#f6fdf6;stroke:#1a7f37;stroke-width:1.6;rx:6}
.warn{fill:#fff8f0;stroke:#bf8700;stroke-width:1.5;rx:6}
.t{font-size:12px;fill:#1f2328}
.tb{font-size:12.5px;fill:#1f2328;font-weight:600}
.s{font-size:10.5px;fill:#656d76}
.ttl{font-size:14px;fill:#1f2328;font-weight:600}
.r{stroke:#cf222e;stroke-width:1.6;fill:none}
.g{stroke:#1a7f37;stroke-width:1.6;fill:none}
.k{stroke:#8c959f;stroke-width:1.3;fill:none}
</style>
<defs>
<marker id="a" markerWidth="8" markerHeight="8" refX="7" refY="3" orient="auto"><path d="M0,0 L7,3 L0,6 z" fill="#8c959f"/></marker>
<marker id="ar" markerWidth="8" markerHeight="8" refX="7" refY="3" orient="auto"><path d="M0,0 L7,3 L0,6 z" fill="#cf222e"/></marker>
<marker id="ag" markerWidth="8" markerHeight="8" refX="7" refY="3" orient="auto"><path d="M0,0 L7,3 L0,6 z" fill="#1a7f37"/></marker>
</defs>
<text x="20" y="26" class="ttl">A-7a — volatile + DB 정지의 결과는 「무엇을 하느냐」가 아니라 「캐시가 더운가」로 갈린다</text>
<text x="20" y="44" class="s">문장 로깅(log_statement=all)으로 측정 · 세 상태 모두 재현</text>
<rect x="20" y="66" width="380" height="120" class="box"/>
<text x="34" y="88" class="tb">로그인이 쏘는 SQL</text>
<rect x="36" y="100" width="348" height="34" class="ok"/>
<text x="48" y="122" class="t">0개 — realm·사용자·클라이언트가 전부 캐시</text>
<text x="36" y="152" class="s">MARK_LOGIN_START / MARK_LOGIN_END 사이가 비어 있다</text>
<text x="36" y="170" class="s">volatile 이라 세션 쓰기도 없다</text>
<rect x="440" y="66" width="400" height="120" class="box"/>
<text x="454" y="88" class="tb">refresh 가 쏘는 SQL</text>
<rect x="456" y="100" width="368" height="34" class="warn"/>
<text x="466" y="115" class="t">select cscme1_0.SCOPE_ID from CLIENT_SCOPE_CLIENT</text>
<text x="466" y="129" class="s">where CLIENT_ID=$1 and DEFAULT_SCOPE=$2 ('f' = optional)</text>
<text x="454" y="152" class="s">★ REVOKED_TOKEN 은 한 번도 나오지 않는다 — A-7 의 가설은 틀렸다</text>
<text x="454" y="170" class="s">그리고 첫 refresh 만 쏜다. 이후 3회는 SQL 0건 = 캐시됨</text>
<text x="20" y="222" class="tb">그래서 같은 설정이 세 가지 답을 낸다 — PostgreSQL 정지 시</text>
<rect x="20" y="240" width="266" height="120" class="dead"/>
<text x="34" y="262" class="tb">① 완전 냉시동</text>
<text x="34" y="282" class="s">재시작 직후, 아무것도 안 함</text>
<text x="34" y="306" class="t">로그인 <tspan fill="#cf222e" font-weight="600">400</tspan> refresh 400</text>
<text x="34" y="328" class="s">select ce1_0.ID from CLIENT</text>
<text x="34" y="344" class="s">클라이언트 조회조차 캐시에 없다</text>
<rect x="298" y="240" width="266" height="120" class="warn"/>
<text x="312" y="262" class="tb">② CLIENT 만 더움</text>
<text x="312" y="282" class="s">로그인 1회 뒤 · ← A-7 이 본 상태</text>
<text x="312" y="306" class="t">로그인 200 refresh <tspan fill="#bf8700" font-weight="600">500</tspan></text>
<text x="312" y="328" class="s">CLIENT_SCOPE_CLIENT 조회 실패</text>
<text x="312" y="344" class="s">Keycloak 로그가 이 SQL 을 직접 지목</text>
<rect x="576" y="240" width="264" height="120" class="ok"/>
<text x="590" y="262" class="tb">③ 완전히 더움</text>
<text x="590" y="282" class="s">refresh 3회로 캐시를 채운 뒤</text>
<text x="590" y="306" class="t">로그인 200 refresh <tspan fill="#1a7f37" font-weight="600">200</tspan></text>
<text x="590" y="328" class="s">SQL 0건 — DB 를 아예 안 본다</text>
<text x="590" y="344" class="s">A-7 의 표와 정반대</text>
<path d="M286 300 L294 300" class="k" marker-end="url(#a)"/>
<path d="M564 300 L572 300" class="k" marker-end="url(#a)"/>
<text x="292" y="290" class="s">로그인 1회</text>
<text x="570" y="290" class="s">refresh 3회</text>
<rect x="20" y="384" width="820" height="66" class="box"/>
<text x="34" y="406" class="tb">남기는 것</text>
<text x="34" y="426" class="s">· A-7 의 표에는 조건이 빠져 있었다 — 「volatile 이면 DB 없이 로그인된다」도 냉시동에서는 거짓이다</text>
<text x="34" y="442" class="s">· 상태가 결과를 바꾸는데 그 상태가 안 보인다. A-1 의 conntrack 과 같은 계열의 함정이다</text>
</svg>

After

Width:  |  Height:  |  Size: 4.5 KiB

+47
View File
@@ -0,0 +1,47 @@
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 860 380" font-family="-apple-system,BlinkMacSystemFont,Segoe UI,Helvetica,sans-serif">
<style>
.box{fill:#fff;stroke:#8c959f;stroke-width:1.4;rx:6}
.dead{fill:#fff5f5;stroke:#cf222e;stroke-width:1.6;stroke-dasharray:5 3;rx:6}
.ok{fill:#f6fdf6;stroke:#1a7f37;stroke-width:1.6;rx:6}
.warn{fill:#fff8f0;stroke:#bf8700;stroke-width:1.5;rx:6}
.t{font-size:12px;fill:#1f2328}
.tb{font-size:12.5px;fill:#1f2328;font-weight:600}
.s{font-size:10.5px;fill:#656d76}
.ttl{font-size:14px;fill:#1f2328;font-weight:600}
.r{stroke:#cf222e;stroke-width:1.6;fill:none}
.g{stroke:#1a7f37;stroke-width:1.6;fill:none}
.k{stroke:#8c959f;stroke-width:1.3;fill:none}
</style>
<defs>
<marker id="a" markerWidth="8" markerHeight="8" refX="7" refY="3" orient="auto"><path d="M0,0 L7,3 L0,6 z" fill="#8c959f"/></marker>
<marker id="ar" markerWidth="8" markerHeight="8" refX="7" refY="3" orient="auto"><path d="M0,0 L7,3 L0,6 z" fill="#cf222e"/></marker>
<marker id="ag" markerWidth="8" markerHeight="8" refX="7" refY="3" orient="auto"><path d="M0,0 L7,3 L0,6 z" fill="#1a7f37"/></marker>
</defs>
<text x="20" y="26" class="ttl">A-8 — 세션은 살아남고 캐시만 사라진다</text>
<text x="20" y="44" class="s">이것이 persistent-user-sessions 를 켜는 진짜 이유다</text>
<rect x="30" y="74" width="800" height="30" class="box"/>
<text x="46" y="94" class="tb">kubectl rollout restart statefulset/keycloak — 한 번에 한 파드씩</text>
<rect x="30" y="124" width="250" height="110" class="dead"/>
<text x="46" y="146" class="tb">사라지는 것 — 메모리</text>
<text x="46" y="168" class="s">Infinispan sessions 캐시</text>
<text x="46" y="188" class="t">entries_unique → <tspan fill="#cf222e" font-weight="600">0</tspan></text>
<text x="46" y="212" class="s">파드와 함께 없어진다</text>
<rect x="305" y="124" width="250" height="110" class="ok"/>
<text x="321" y="146" class="tb">남는 것 — PostgreSQL</text>
<text x="321" y="168" class="s">OFFLINE_USER_SESSION</text>
<text x="321" y="188" class="t">151 → <tspan fill="#1a7f37" font-weight="600">151</tspan> 그대로</text>
<text x="321" y="212" class="s">재시작과 무관하다</text>
<rect x="580" y="124" width="250" height="110" class="ok"/>
<text x="596" y="146" class="tb">그래서 사용자는</text>
<text x="596" y="168" class="s">재시작 전 발급한 refresh token</text>
<text x="596" y="188" class="t">여전히 <tspan fill="#1a7f37" font-weight="600">200</tspan></text>
<text x="596" y="212" class="s">전 구간 중단 없음</text>
<path d="M282 178 L301 178" class="k" marker-end="url(#a)"/>
<path d="M557 178 L576 178" class="k" marker-end="url(#a)"/>
<rect x="30" y="254" width="800" height="52" class="box"/>
<text x="46" y="276" class="tb">클러스터는 자동 재형성된다</text>
<text x="46" y="296" class="s">JGROUPS_PING 테이블에 다시 등록되고 cluster_size 가 2 로 돌아온다 — 사람이 할 일이 없다</text>
<rect x="30" y="322" width="800" height="44" class="warn"/>
<text x="46" y="344" class="s">★ A-7 과 대조 — volatile 이면 같은 재시작에서 refresh 가 400 Session not active 가 된다.</text>
<text x="46" y="360" class="s">세션이 메모리에만 있었으므로 캐시가 사라지는 순간 세션도 사라진다. 이 실험의 결론은 버전 조건부다</text>
</svg>

After

Width:  |  Height:  |  Size: 3.3 KiB

+45
View File
@@ -0,0 +1,45 @@
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 860 400" font-family="-apple-system,BlinkMacSystemFont,Segoe UI,Helvetica,sans-serif">
<style>
.box{fill:#fff;stroke:#8c959f;stroke-width:1.4;rx:6}
.dead{fill:#fff5f5;stroke:#cf222e;stroke-width:1.6;stroke-dasharray:5 3;rx:6}
.ok{fill:#f6fdf6;stroke:#1a7f37;stroke-width:1.6;rx:6}
.warn{fill:#fff8f0;stroke:#bf8700;stroke-width:1.5;rx:6}
.t{font-size:12px;fill:#1f2328}
.tb{font-size:12.5px;fill:#1f2328;font-weight:600}
.s{font-size:10.5px;fill:#656d76}
.ttl{font-size:14px;fill:#1f2328;font-weight:600}
.r{stroke:#cf222e;stroke-width:1.6;fill:none}
.g{stroke:#1a7f37;stroke-width:1.6;fill:none}
.k{stroke:#8c959f;stroke-width:1.3;fill:none}
</style>
<defs>
<marker id="a" markerWidth="8" markerHeight="8" refX="7" refY="3" orient="auto"><path d="M0,0 L7,3 L0,6 z" fill="#8c959f"/></marker>
<marker id="ar" markerWidth="8" markerHeight="8" refX="7" refY="3" orient="auto"><path d="M0,0 L7,3 L0,6 z" fill="#cf222e"/></marker>
<marker id="ag" markerWidth="8" markerHeight="8" refX="7" refY="3" orient="auto"><path d="M0,0 L7,3 L0,6 z" fill="#1a7f37"/></marker>
</defs>
<text x="20" y="26" class="ttl">B-0 — 아무것도 설정하지 않으면 Spring 이 무엇을 고르는가</text>
<text x="20" y="44" class="s">추측이 맞았지만, 추측으로 두면 안 되는 이유가 둘째 줄에 있다</text>
<rect x="30" y="72" width="800" height="118" class="box"/>
<text x="46" y="94" class="tb">/actuator/beans 가 실제로 답한 것</text>
<text x="46" y="118" class="t">authorizedClientService → InMemoryOAuth2AuthorizedClientService</text>
<text x="46" y="138" class="t">authorizedClientRepository → AuthenticatedPrincipalOAuth2AuthorizedClientRepository</text>
<text x="46" y="158" class="t">SessionRepository → 없음 (서블릿 컨테이너 in-memory)</text>
<text x="46" y="178" class="t" fill="#cf222e">Redis / Spring Session → 없음</text>
<rect x="30" y="212" width="390" height="150" class="warn"/>
<text x="46" y="234" class="tb">★ 이름이 곧 설명이다</text>
<text x="46" y="256" class="s">AuthenticatedPrincipal…Repository 는</text>
<text x="46" y="276" class="t">principal 이름으로 찾는다</text>
<rect x="46" y="288" width="358" height="30" class="dead"/>
<text x="58" y="308" class="s">조회 키에 session id 가 없다</text>
<text x="46" y="336" class="s">그래서 세션 저장소를 Redis 로 옮겨도</text>
<text x="46" y="354" class="s">토큰은 따라오지 않는다 — B-1 이 겪는 문제의 뿌리</text>
<rect x="440" y="212" width="390" height="150" class="box"/>
<text x="456" y="234" class="tb">두 개가 서로 다른 것을 저장한다</text>
<rect x="456" y="248" width="358" height="40" class="box"/>
<text x="468" y="266" class="t">Application Session</text>
<text x="468" y="282" class="s">누가 로그인했는지 · 세션 id 로 찾는다</text>
<rect x="456" y="298" width="358" height="40" class="box"/>
<text x="468" y="316" class="t">OAuth2AuthorizedClient</text>
<text x="468" y="332" class="s">access / refresh token · principal 이름으로 찾는다</text>
<text x="456" y="356" class="s">이 둘을 하나로 생각하면 B-1·B-2 의 결과를 해석할 수 없다</text>
</svg>

After

Width:  |  Height:  |  Size: 3.2 KiB

+33
View File
@@ -0,0 +1,33 @@
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 700 330" width="700" height="330" font-family="system-ui,-apple-system,Segoe UI,Roboto,sans-serif">
<style>
.t{font-size:13px;fill:#1f2328}.s{font-size:11px;fill:#59636e}.h{font-size:14px;font-weight:600;fill:#1f2328}
.r{font-size:12px;fill:#cf222e;font-weight:600}
.box{fill:#fff;stroke:#8c959f;stroke-width:1.4;rx:6}
.dead{fill:#fff5f5;stroke:#cf222e;stroke-width:1.6;stroke-dasharray:5 3;rx:6}
.ok{fill:#f6fdf6;stroke:#1a7f37;stroke-width:1.6;rx:6}
.ln{stroke:#59636e;stroke-width:1.4;fill:none;marker-end:url(#a)}
.bad{stroke:#cf222e;stroke-width:1.6;stroke-dasharray:5 3;fill:none;marker-end:url(#b)}
</style>
<defs>
<marker id="a" markerWidth="9" markerHeight="9" refX="8" refY="4.5" orient="auto"><path d="M0,0 L9,4.5 L0,9 z" fill="#59636e"/></marker>
<marker id="b" markerWidth="9" markerHeight="9" refX="8" refY="4.5" orient="auto"><path d="M0,0 L9,4.5 L0,9 z" fill="#cf222e"/></marker>
</defs>
<text class="h" x="16" y="24">B-1 · Redis 는 세션만 옮기고 토큰은 두고 간다</text>
<rect class="box" x="30" y="46" width="140" height="46"/><text class="t" x="100" y="66" text-anchor="middle">bff-0</text><text class="s" x="100" y="82" text-anchor="middle">kc-lab-1</text>
<rect class="box" x="30" y="104" width="140" height="46"/><text class="t" x="100" y="124" text-anchor="middle">bff-1</text><text class="s" x="100" y="140" text-anchor="middle">kc-lab-2</text>
<rect class="ok" x="330" y="46" width="330" height="60"/>
<text class="t" x="495" y="68" text-anchor="middle">Redis — Application Session</text>
<text class="s" x="495" y="86" text-anchor="middle">sessionRepository → RedisSessionRepository ✔ 옮겨졌다</text>
<text class="s" x="495" y="100" text-anchor="middle">필드: SPRING_SECURITY_CONTEXT · TTL 1772초</text>
<rect class="dead" x="330" y="122" width="330" height="60"/>
<text class="r" x="495" y="144" text-anchor="middle">프로세스 메모리 — OAuth2AuthorizedClient</text>
<text class="s" x="495" y="162" text-anchor="middle">InMemoryOAuth2AuthorizedClientService ✗ 그대로</text>
<text class="s" x="495" y="176" text-anchor="middle">access token · refresh token 이 여기 있다</text>
<path class="ln" d="M170,69 L330,69"/><path class="ln" d="M170,127 L330,80"/>
<path class="bad" d="M170,140 L330,150"/>
<rect class="box" x="30" y="210" width="630" height="90"/>
<text class="t" x="345" y="234" text-anchor="middle">그 결과 사용자에게 보이는 것</text>
<text class="s" x="345" y="256" text-anchor="middle">principal: labuser ← 로그인은 되어 있다</text>
<text class="s" x="345" y="272" text-anchor="middle">accessTokenStoredOnServer: false ← 토큰이 없다</text>
<text class="r" x="345" y="292" text-anchor="middle">완전히 로그아웃되는 편이 차라리 낫다</text>
</svg>

After

Width:  |  Height:  |  Size: 2.8 KiB

@@ -0,0 +1,55 @@
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 860 440" font-family="-apple-system,BlinkMacSystemFont,Segoe UI,Helvetica,sans-serif">
<style>
.box{fill:#fff;stroke:#8c959f;stroke-width:1.4;rx:6}
.dead{fill:#fff5f5;stroke:#cf222e;stroke-width:1.6;stroke-dasharray:5 3;rx:6}
.ok{fill:#f6fdf6;stroke:#1a7f37;stroke-width:1.6;rx:6}
.warn{fill:#fff8f0;stroke:#bf8700;stroke-width:1.5;rx:6}
.t{font-size:12px;fill:#1f2328}
.tb{font-size:12.5px;fill:#1f2328;font-weight:600}
.s{font-size:10.5px;fill:#656d76}
.ttl{font-size:14px;fill:#1f2328;font-weight:600}
.r{stroke:#cf222e;stroke-width:1.6;fill:none}
.g{stroke:#1a7f37;stroke-width:1.6;fill:none}
.k{stroke:#8c959f;stroke-width:1.3;fill:none}
</style>
<defs>
<marker id="a" markerWidth="8" markerHeight="8" refX="7" refY="3" orient="auto"><path d="M0,0 L7,3 L0,6 z" fill="#8c959f"/></marker>
<marker id="ar" markerWidth="8" markerHeight="8" refX="7" refY="3" orient="auto"><path d="M0,0 L7,3 L0,6 z" fill="#cf222e"/></marker>
<marker id="ag" markerWidth="8" markerHeight="8" refX="7" refY="3" orient="auto"><path d="M0,0 L7,3 L0,6 z" fill="#1a7f37"/></marker>
</defs>
<text x="20" y="26" class="ttl">B-2 — 저장소를 나눠 문제를 풀자 다른 두 문제가 남았다</text>
<text x="20" y="44" class="s">Q1 의 네 항목 중 둘이 통과, 둘이 실패</text>
<rect x="30" y="72" width="240" height="90" class="box"/>
<text x="46" y="94" class="tb">bff 인스턴스 2개</text>
<text x="46" y="116" class="s">어느 쪽으로 요청이 가도 된다</text>
<text x="46" y="136" class="s">enableServiceLinks: false</text>
<text x="46" y="154" class="s">(REDIS_PORT 충돌 회피)</text>
<rect x="330" y="72" width="230" height="90" class="ok"/>
<text x="346" y="94" class="tb">Redis</text>
<text x="346" y="116" class="s">Application Session</text>
<text x="346" y="136" class="s">세션 id 로 조회</text>
<text x="346" y="154" class="s">로그아웃 시 정리됨</text>
<rect x="600" y="72" width="230" height="90" class="warn"/>
<text x="616" y="94" class="tb">PostgreSQL</text>
<text x="616" y="116" class="s">OAuth2AuthorizedClient</text>
<text x="616" y="136" class="s">principal 이름으로 조회</text>
<text x="616" y="154" class="s">로그아웃해도 남는다</text>
<path d="M272 108 L326 108" class="g" marker-end="url(#ag)"/>
<path d="M562 108 L596 108" class="g" marker-end="url(#ag)"/>
<rect x="30" y="188" width="390" height="100" class="ok"/>
<text x="46" y="210" class="tb">통과</text>
<text x="46" y="234" class="t">① 다른 인스턴스로 요청해도 된다</text>
<text x="46" y="256" class="t">② 재시작 후에도 로그인 유지</text>
<text x="46" y="278" class="s">저장소를 밖으로 뺐으니 당연한 결과다</text>
<rect x="440" y="188" width="390" height="100" class="dead"/>
<text x="456" y="210" class="tb">실패</text>
<text x="456" y="234" class="t" fill="#cf222e">③ 같은 사용자의 다른 브라우저가 덮어쓴다</text>
<text x="456" y="256" class="t" fill="#cf222e">④ 로그아웃해도 한쪽만 정리된다</text>
<text x="456" y="278" class="s">둘 다 저장소 선택의 문제가 아니라 스키마의 문제다</text>
<rect x="30" y="308" width="800" height="70" class="warn"/>
<text x="46" y="330" class="tb">★ 뿌리는 DDL 한 줄이다</text>
<text x="46" y="352" class="t">PRIMARY KEY (client_registration_id, principal_name)</text>
<text x="46" y="370" class="s">세션 id 가 키에 없다 → 같은 사용자의 두 세션이 같은 행을 쓴다 → 나중 로그인이 앞의 토큰을 덮어쓴다</text>
<rect x="30" y="392" width="800" height="40" class="dead"/>
<text x="46" y="416" class="s">로그아웃 후: Redis 세션 0 키 (정리됨) · PostgreSQL 토큰 1 행 — 평문 refresh token 이 그대로 남는다</text>
</svg>

After

Width:  |  Height:  |  Size: 3.7 KiB

+29
View File
@@ -0,0 +1,29 @@
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 700 300" width="700" height="300" font-family="system-ui,-apple-system,Segoe UI,Roboto,sans-serif">
<style>
.t{font-size:13px;fill:#1f2328}.s{font-size:11px;fill:#59636e}.h{font-size:14px;font-weight:600;fill:#1f2328}
.r{font-size:12px;fill:#cf222e;font-weight:600}
.box{fill:#fff;stroke:#8c959f;stroke-width:1.4;rx:6}
.dead{fill:#fff5f5;stroke:#cf222e;stroke-width:1.6;stroke-dasharray:5 3;rx:6}
.ok{fill:#f6fdf6;stroke:#1a7f37;stroke-width:1.6;rx:6}
.ln{stroke:#59636e;stroke-width:1.4;fill:none;marker-end:url(#a)}
.bad{stroke:#cf222e;stroke-width:1.6;stroke-dasharray:5 3;fill:none;marker-end:url(#b)}
</style>
<defs>
<marker id="a" markerWidth="9" markerHeight="9" refX="8" refY="4.5" orient="auto"><path d="M0,0 L9,4.5 L0,9 z" fill="#59636e"/></marker>
<marker id="b" markerWidth="9" markerHeight="9" refX="8" refY="4.5" orient="auto"><path d="M0,0 L9,4.5 L0,9 z" fill="#cf222e"/></marker>
</defs>
<text class="h" x="16" y="24">B-3 · 동시 refresh — 경쟁이 아니라 세션 파괴</text>
<rect class="box" x="30" y="46" width="150" height="40"/><text class="t" x="105" y="71" text-anchor="middle">같은 refresh token ×5</text>
<path class="ln" d="M180,66 L280,66"/>
<rect class="box" x="280" y="42" width="180" height="48"/><text class="t" x="370" y="62" text-anchor="middle">Keycloak</text>
<text class="s" x="370" y="80" text-anchor="middle">rotation ON · maxReuse=0</text>
<path class="ln" d="M460,58 L560,58"/><rect class="box" x="560" y="42" width="110" height="24"/><text class="s" x="615" y="58" text-anchor="middle">1× HTTP 200</text>
<path class="bad" d="M460,78 L560,78"/><rect class="dead" x="560" y="66" width="110" height="24"/><text class="s" x="615" y="82" text-anchor="middle">4× HTTP 400</text>
<rect class="dead" x="120" y="120" width="460" height="70"/>
<text class="r" x="350" y="144" text-anchor="middle">재사용 탐지가 client session 을 제거한다</text>
<text class="s" x="350" y="164" text-anchor="middle">user_session 은 남고 client_session = 0 (정상 세션은 1)</text>
<text class="s" x="350" y="180" text-anchor="middle">그래서 오류가 "Session doesn't have required client"</text>
<rect class="dead" x="120" y="210" width="460" height="60"/>
<text class="r" x="350" y="234" text-anchor="middle">★ 이긴 요청의 새 토큰도 곧바로 400</text>
<text class="s" x="350" y="254" text-anchor="middle">재시도로 회복 불가 → Q2 의 판정은 lock</text>
</svg>

After

Width:  |  Height:  |  Size: 2.5 KiB

+37
View File
@@ -0,0 +1,37 @@
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 700 290" width="700" height="290" font-family="system-ui,-apple-system,Segoe UI,Roboto,sans-serif">
<style>
.t{font-size:13px;fill:#1f2328}.s{font-size:11px;fill:#59636e}.h{font-size:14px;font-weight:600;fill:#1f2328}
.r{font-size:12px;fill:#cf222e;font-weight:600}.g{font-size:12px;fill:#1a7f37;font-weight:600}
.box{fill:#fff;stroke:#8c959f;stroke-width:1.4;rx:6}
.dead{fill:#fff5f5;stroke:#cf222e;stroke-width:1.6;stroke-dasharray:5 3;rx:6}
.ok{fill:#f6fdf6;stroke:#1a7f37;stroke-width:1.6;rx:6}
.ln{stroke:#59636e;stroke-width:1.4;fill:none;marker-end:url(#a)}
.bad{stroke:#cf222e;stroke-width:1.6;stroke-dasharray:5 3;fill:none;marker-end:url(#b)}
</style>
<defs>
<marker id="a" markerWidth="9" markerHeight="9" refX="8" refY="4.5" orient="auto"><path d="M0,0 L9,4.5 L0,9 z" fill="#59636e"/></marker>
<marker id="b" markerWidth="9" markerHeight="9" refX="8" refY="4.5" orient="auto"><path d="M0,0 L9,4.5 L0,9 z" fill="#cf222e"/></marker>
</defs>
<text class="h" x="16" y="24">B-4 · edge 가 설정하지 않은 헤더는 그대로 통과한다</text>
<rect class="box" x="30" y="46" width="130" height="60"/><text class="t" x="95" y="68" text-anchor="middle">공격자</text>
<text class="s" x="95" y="86" text-anchor="middle">X-Auth-Request-</text><text class="s" x="95" y="100" text-anchor="middle">Roles: admin</text>
<path class="ln" d="M160,76 L250,76"/>
<rect class="box" x="250" y="40" width="180" height="72"/><text class="t" x="340" y="62" text-anchor="middle">nginx</text>
<text class="s" x="340" y="82" text-anchor="middle">proxy_set_header 한 것만 덮어쓴다</text>
<text class="r" x="340" y="100" text-anchor="middle">X-Auth-Request-* 는 설정이 없다</text>
<path class="bad" d="M430,76 L520,76"/>
<rect class="dead" x="520" y="46" width="150" height="60"/>
<text class="r" x="595" y="68" text-anchor="middle">upstream</text>
<text class="s" x="595" y="86" text-anchor="middle">['viewer','admin']</text><text class="s" x="595" y="100" text-anchor="middle">둘 다 도착 · 검증 없음</text>
<rect class="box" x="30" y="134" width="310" height="66"/>
<text class="t" x="185" y="156" text-anchor="middle">구분자 문제</text>
<text class="s" x="185" y="176" text-anchor="middle">"admin,editor" 와 "role-with,comma" 가</text>
<text class="s" x="185" y="192" text-anchor="middle">도착 시점에 구별되지 않는다</text>
<rect class="box" x="360" y="134" width="310" height="66"/>
<text class="t" x="515" y="156" text-anchor="middle">크기는 절벽이다</text>
<text class="s" x="515" y="176" text-anchor="middle">4KB 통과 · 8KB → Tomcat 400</text>
<text class="s" x="515" y="192" text-anchor="middle">16KB → 연결 끊김 (nginx)</text>
<rect class="dead" x="30" y="222" width="640" height="50"/>
<text class="r" x="350" y="244" text-anchor="middle">헤더가 인가 근거가 되면 위조 가능성이 곧 권한 상승이다</text>
<text class="s" x="350" y="262" text-anchor="middle">Q4 의 5문항 중 2·4번 해당 → Q4 자신의 기준으로 BFF 구조</text>
</svg>

After

Width:  |  Height:  |  Size: 3.1 KiB

+35
View File
@@ -0,0 +1,35 @@
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 700 300" width="700" height="300" font-family="system-ui,-apple-system,Segoe UI,Roboto,sans-serif">
<style>
.t{font-size:13px;fill:#1f2328}.s{font-size:11px;fill:#59636e}.h{font-size:14px;font-weight:600;fill:#1f2328}
.r{font-size:12px;fill:#cf222e;font-weight:600}.g{font-size:12px;fill:#1a7f37;font-weight:600}
.box{fill:#fff;stroke:#8c959f;stroke-width:1.4;rx:6}
.dead{fill:#fff5f5;stroke:#cf222e;stroke-width:1.6;stroke-dasharray:5 3;rx:6}
.ok{fill:#f6fdf6;stroke:#1a7f37;stroke-width:1.6;rx:6}
.ln{stroke:#59636e;stroke-width:1.4;fill:none;marker-end:url(#a)}
.bad{stroke:#cf222e;stroke-width:1.6;stroke-dasharray:5 3;fill:none;marker-end:url(#b)}
</style>
<defs>
<marker id="a" markerWidth="9" markerHeight="9" refX="8" refY="4.5" orient="auto"><path d="M0,0 L9,4.5 L0,9 z" fill="#59636e"/></marker>
<marker id="b" markerWidth="9" markerHeight="9" refX="8" refY="4.5" orient="auto"><path d="M0,0 L9,4.5 L0,9 z" fill="#cf222e"/></marker>
</defs>
<text class="h" x="16" y="24">B-5 · 파드가 Ready 인 채로 계속 실패한다</text>
<rect class="box" x="30" y="46" width="180" height="76"/>
<text class="t" x="120" y="68" text-anchor="middle">bff ×2</text>
<text class="g" x="120" y="90" text-anchor="middle">Ready = true</text>
<text class="s" x="120" y="110" text-anchor="middle">Service 에 그대로 남는다</text>
<path class="bad" d="M210,84 L330,84"/>
<rect class="dead" x="330" y="58" width="160" height="52"/>
<text class="r" x="410" y="80" text-anchor="middle">redis ✗</text><text class="s" x="410" y="98" text-anchor="middle">replicas=0</text>
<rect class="box" x="30" y="146" width="310" height="110"/>
<text class="t" x="185" y="168" text-anchor="middle">health group 이 갈랐다</text>
<text class="r" x="185" y="192" text-anchor="middle">/actuator/health → 503</text>
<text class="g" x="185" y="214" text-anchor="middle">/actuator/health/readiness → 200 UP</text>
<text class="s" x="185" y="238" text-anchor="middle">redis 지표가 readiness 그룹에 없다</text>
<rect class="box" x="360" y="146" width="310" height="110"/>
<text class="t" x="515" y="168" text-anchor="middle">A-2 와 정반대</text>
<text class="s" x="515" y="192" text-anchor="middle">A-2 Keycloak: DB 검사가 readiness 에</text>
<text class="s" x="515" y="208" text-anchor="middle">→ NotReady → 503 (명확)</text>
<text class="s" x="515" y="230" text-anchor="middle">B-5 BFF: 없음 → Ready 유지</text>
<text class="r" x="515" y="248" text-anchor="middle">→ HTTP 000 (멈춤)</text>
<text class="s" x="16" y="284">영속화: 볼륨 없이 AOF 만 켜면 appendonlydir 은 생기지만 파드 삭제로 전부 사라진다 — 볼륨이 먼저다</text>
</svg>

After

Width:  |  Height:  |  Size: 2.8 KiB

+34
View File
@@ -0,0 +1,34 @@
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 700 260" width="700" height="260" font-family="system-ui,-apple-system,Segoe UI,Roboto,sans-serif">
<style>
.t{font-size:13px;fill:#1f2328}.s{font-size:11px;fill:#59636e}.h{font-size:14px;font-weight:600;fill:#1f2328}
.r{font-size:12px;fill:#cf222e;font-weight:600}.g{font-size:12px;fill:#1a7f37;font-weight:600}
.box{fill:#fff;stroke:#8c959f;stroke-width:1.4;rx:6}
.dead{fill:#fff5f5;stroke:#cf222e;stroke-width:1.6;stroke-dasharray:5 3;rx:6}
.ok{fill:#f6fdf6;stroke:#1a7f37;stroke-width:1.6;rx:6}
.ln{stroke:#59636e;stroke-width:1.4;fill:none;marker-end:url(#a)}
.bad{stroke:#cf222e;stroke-width:1.6;stroke-dasharray:5 3;fill:none;marker-end:url(#b)}
</style>
<defs>
<marker id="a" markerWidth="9" markerHeight="9" refX="8" refY="4.5" orient="auto"><path d="M0,0 L9,4.5 L0,9 z" fill="#59636e"/></marker>
<marker id="b" markerWidth="9" markerHeight="9" refX="8" refY="4.5" orient="auto"><path d="M0,0 L9,4.5 L0,9 z" fill="#cf222e"/></marker>
</defs>
<text class="h" x="16" y="24">B-6 · 회전은 안전하고 옛 키를 버리는 순간이 위험하다</text>
<rect class="ok" x="30" y="46" width="190" height="76"/>
<text class="t" x="125" y="68" text-anchor="middle">t0 — 키 A 만</text>
<text class="s" x="125" y="88" text-anchor="middle">발급 A · 검증 A</text>
<text class="s" x="125" y="108" text-anchor="middle">JWKS RS256 1개</text>
<path class="ln" d="M220,84 L255,84"/>
<rect class="ok" x="255" y="46" width="190" height="76"/>
<text class="t" x="350" y="68" text-anchor="middle">t1 — B 추가 (priority 200)</text>
<text class="g" x="350" y="88" text-anchor="middle">발급 B · 검증 A+B</text>
<text class="s" x="350" y="108" text-anchor="middle">옛 토큰 200 · 새 토큰 200</text>
<path class="bad" d="M445,84 L480,84"/>
<rect class="dead" x="480" y="46" width="190" height="76"/>
<text class="t" x="575" y="68" text-anchor="middle">t2 — A 제거</text>
<text class="r" x="575" y="88" text-anchor="middle">옛 토큰 즉시 401</text>
<text class="s" x="575" y="108" text-anchor="middle">캐시가 유예를 주지 않는다</text>
<rect class="box" x="30" y="146" width="640" height="60"/>
<text class="t" x="350" y="168" text-anchor="middle">겹침 구간(t1~t2)의 최소 길이 = 옛 키로 서명된 것 중 가장 오래 사는 것의 수명</text>
<text class="s" x="350" y="190" text-anchor="middle">access token 60초 · refresh token 1800초 → 최소 30분</text>
<text class="s" x="16" y="234">모르는 kid 를 만나면 JWKS 를 다시 받으므로 제거가 즉시 반영된다. 유예는 옛 키를 남겨두는 기간으로 만든다.</text>
</svg>

After

Width:  |  Height:  |  Size: 2.7 KiB

+52
View File
@@ -0,0 +1,52 @@
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 860 420" font-family="-apple-system,BlinkMacSystemFont,Segoe UI,Helvetica,sans-serif">
<style>
.box{fill:#fff;stroke:#8c959f;stroke-width:1.4;rx:6}
.dead{fill:#fff5f5;stroke:#cf222e;stroke-width:1.6;stroke-dasharray:5 3;rx:6}
.ok{fill:#f6fdf6;stroke:#1a7f37;stroke-width:1.6;rx:6}
.warn{fill:#fff8f0;stroke:#bf8700;stroke-width:1.5;rx:6}
.t{font-size:12px;fill:#1f2328}
.tb{font-size:12.5px;fill:#1f2328;font-weight:600}
.s{font-size:10.5px;fill:#656d76}
.ttl{font-size:14px;fill:#1f2328;font-weight:600}
.r{stroke:#cf222e;stroke-width:1.6;fill:none}
.g{stroke:#1a7f37;stroke-width:1.6;fill:none}
.k{stroke:#8c959f;stroke-width:1.3;fill:none}
</style>
<defs>
<marker id="a" markerWidth="8" markerHeight="8" refX="7" refY="3" orient="auto"><path d="M0,0 L7,3 L0,6 z" fill="#8c959f"/></marker>
<marker id="ar" markerWidth="8" markerHeight="8" refX="7" refY="3" orient="auto"><path d="M0,0 L7,3 L0,6 z" fill="#cf222e"/></marker>
<marker id="ag" markerWidth="8" markerHeight="8" refX="7" refY="3" orient="auto"><path d="M0,0 L7,3 L0,6 z" fill="#1a7f37"/></marker>
</defs>
<text x="20" y="26" class="ttl">B-7 — BFF 와 정반대의 성질: 공유할 상태가 없고, 대신 겹침 구간도 없다</text>
<text x="20" y="44" class="s">oauth2-proxy 는 세션을 쿠키에 담는다</text>
<rect x="30" y="72" width="380" height="120" class="box"/>
<text x="46" y="94" class="tb">replica 는 무엇을 공유하나</text>
<rect x="46" y="106" width="160" height="34" class="box"/>
<text x="58" y="128" class="t">oauth2-proxy A</text>
<rect x="250" y="106" width="146" height="34" class="box"/>
<text x="262" y="128" class="t">oauth2-proxy B</text>
<rect x="106" y="152" width="230" height="30" class="ok"/>
<text x="118" y="172" class="s">같은 k8s Secret 을 읽는다</text>
<path d="M126 142 L180 150" class="g"/>
<path d="M316 142 L262 150" class="g"/>
<text x="46" y="196" class="s"> </text>
<rect x="440" y="72" width="390" height="120" class="ok"/>
<text x="456" y="94" class="tb">그래서 얻는 것</text>
<text x="456" y="118" class="s">공유할 서버 상태가 없다 → 콜백이 다른 replica 로 가도 된다</text>
<text x="456" y="140" class="s">replica 를 늘려도 세션 저장소 고민이 없다</text>
<text x="456" y="164" class="s">BFF 가 Redis·PostgreSQL 로 풀던 문제가 아예 생기지 않는다</text>
<rect x="30" y="212" width="380" height="106" class="dead"/>
<text x="46" y="234" class="tb">★ 대신 겹침 구간을 만들 수 없다</text>
<text x="46" y="258" class="t">--cookie-secret 은 단수다</text>
<text x="46" y="280" class="s">「옛 secret 도 당분간 받아준다」가 불가능하다</text>
<text x="46" y="300" class="s">교체하는 순간 모든 쿠키가 한꺼번에 무효가 된다</text>
<rect x="440" y="212" width="390" height="106" class="warn"/>
<text x="456" y="234" class="tb">교체하면 벌어지는 일</text>
<text x="456" y="258" class="s">옛 쿠키 → session ticket cookie failed validation</text>
<text x="456" y="278" class="s">Keycloak SSO 가 살아 있으면 로그인 화면 없이 조용히 재인증</text>
<text x="456" y="300" class="s" fill="#cf222e">서버 쪽 세션은 고아로 남는다 — 티켓을 못 풀어 지우지도 못한다</text>
<rect x="30" y="338" width="800" height="66" class="ok"/>
<text x="46" y="360" class="tb">→ B-7a 가 이어받았다</text>
<text x="46" y="382" class="s">「지울 수 없다」는 oauth2-proxy 의 한계일 뿐이었다. 운영자는 지울 수 있고, TTL 역산으로 고아만 골라낼 수 있다</text>
<text x="46" y="398" class="s">그리고 고아는 생성 후 정확히 1시간에 사라진다 — TTL 이 요청으로 갱신되지 않기 때문이다</text>
</svg>

After

Width:  |  Height:  |  Size: 3.7 KiB

+74
View File
@@ -0,0 +1,74 @@
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 860 500" font-family="-apple-system,BlinkMacSystemFont,'Segoe UI',Helvetica,sans-serif">
<style>
.box{fill:#fff;stroke:#8c959f;stroke-width:1.4;rx:6}
.dead{fill:#fff5f5;stroke:#cf222e;stroke-width:1.6;stroke-dasharray:5 3;rx:6}
.ok{fill:#f6fdf6;stroke:#1a7f37;stroke-width:1.6;rx:6}
.warn{fill:#fff8f0;stroke:#bf8700;stroke-width:1.5;rx:6}
.t{font-size:12px;fill:#1f2328}
.tb{font-size:12.5px;fill:#1f2328;font-weight:600}
.s{font-size:10.5px;fill:#656d76}
.ttl{font-size:14px;fill:#1f2328;font-weight:600}
.r{stroke:#cf222e;stroke-width:1.6;fill:none}
.g{stroke:#1a7f37;stroke-width:1.6;fill:none}
.k{stroke:#8c959f;stroke-width:1.3;fill:none}
</style>
<defs>
<marker id="a" markerWidth="8" markerHeight="8" refX="7" refY="3" orient="auto"><path d="M0,0 L7,3 L0,6 z" fill="#8c959f"/></marker>
<marker id="ar" markerWidth="8" markerHeight="8" refX="7" refY="3" orient="auto"><path d="M0,0 L7,3 L0,6 z" fill="#cf222e"/></marker>
<marker id="ag" markerWidth="8" markerHeight="8" refX="7" refY="3" orient="auto"><path d="M0,0 L7,3 L0,6 z" fill="#1a7f37"/></marker>
</defs>
<text x="20" y="26" class="ttl">B-7a — 고아 세션은 지울 수 있다. oauth2-proxy 가 못 할 뿐이다</text>
<text x="20" y="44" class="s">회전 2회 측정 · Redis + oauth2-proxy 로그 + 브라우저</text>
<text x="20" y="76" class="tb">왜 프록시는 못 지우는가 — 티켓의 구조</text>
<rect x="20" y="88" width="400" height="86" class="box"/>
<text x="34" y="110" class="t">쿠키에 담기는 것 = 티켓</text>
<rect x="36" y="120" width="180" height="30" class="box"/>
<text x="48" y="140" class="s">세션 ID → Redis 키 이름</text>
<rect x="224" y="120" width="180" height="30" class="box"/>
<text x="236" y="140" class="s">암호화 키 → 값 복호화</text>
<text x="34" y="166" class="s">티켓 전체가 cookie secret 으로 암호화되어 있다</text>
<rect x="444" y="88" width="396" height="86" class="dead"/>
<text x="458" y="110" class="tb">secret 을 바꾸면</text>
<text x="458" y="130" class="s">티켓을 못 연다 → 세션 ID 조차 못 읽는다</text>
<text x="458" y="148" class="s">→ 어느 Redis 키를 지울지 모른다</text>
<text x="458" y="166" class="s">Error removing session: error decoding ticket to clear session</text>
<path d="M424 131 L440 131" class="r" marker-end="url(#ar)"/>
<text x="20" y="208" class="tb">회전할 때마다 누적한다</text>
<line x1="70" y1="256" x2="800" y2="256" class="k"/>
<line x1="130" y1="244" x2="130" y2="268" stroke="#cf222e" stroke-width="2"/>
<line x1="430" y1="244" x2="430" y2="268" stroke="#cf222e" stroke-width="2"/>
<text x="92" y="288" class="s">1차 회전 11:29:56</text>
<text x="392" y="288" class="s">2차 회전 11:33:27</text>
<rect x="70" y="224" width="60" height="16" class="ok"/>
<text x="76" y="236" class="s">세션 A</text>
<rect x="130" y="224" width="300" height="16" class="dead"/>
<text x="150" y="236" class="s" fill="#cf222e">A 는 고아 · TTL 만료까지 남는다</text>
<rect x="130" y="224" width="0" height="0"/>
<rect x="150" y="304" width="280" height="16" class="ok"/>
<text x="156" y="316" class="s">세션 B (1차 회전 후 생성)</text>
<rect x="430" y="304" width="370" height="16" class="dead"/>
<text x="450" y="316" class="s" fill="#cf222e">B 도 고아가 된다</text>
<rect x="450" y="224" width="350" height="16" class="ok"/>
<text x="456" y="236" class="s">세션 C (2차 회전 후 생성)</text>
<text x="70" y="338" class="s">회전 1회 = 그 시점 로그인 사용자 수만큼의 고아</text>
<rect x="20" y="356" width="400" height="126" class="dead"/>
<text x="34" y="378" class="tb">Redis 만 보고는 구분할 수 없다</text>
<text x="34" y="400" class="s">이름 _oauth2_proxy-&lt;불투명한 32자 hex&gt; (동일 형식)</text>
<text x="34" y="418" class="s">type string (동일)</text>
<text x="34" y="436" class="s">크기 3510 바이트 (바이트까지 동일)</text>
<text x="34" y="454" class="s">값 암호화됨 (읽을 수 없다)</text>
<text x="34" y="474" class="s" fill="#cf222e">→ 다른 것은 TTL 하나뿐이다</text>
<rect x="444" y="356" width="396" height="126" class="ok"/>
<text x="458" y="378" class="tb">그래서 TTL 이 정리 규칙이 된다</text>
<text x="458" y="398" class="s">--cookie-refresh 가 없어 TTL 이 갱신되지 않는다 (refresh:disabled)</text>
<text x="458" y="416" class="s">→ TTL 은 생성 시각의 정확한 함수다</text>
<rect x="460" y="424" width="364" height="26" class="box"/>
<text x="470" y="442" class="t">생성시각 = 지금 (cookie-expire TTL)</text>
<text x="458" y="466" class="s" fill="#1a7f37">이 값이 회전 시각보다 이르면 고아. 검증: 역산 11:30:26 vs 로그 11:30:27 — 1초 오차</text>
</svg>

After

Width:  |  Height:  |  Size: 4.8 KiB

+31
View File
@@ -0,0 +1,31 @@
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 700 300" width="700" height="300" font-family="system-ui,-apple-system,Segoe UI,Roboto,sans-serif">
<style>
.t{font-size:13px;fill:#1f2328}.s{font-size:11px;fill:#59636e}.h{font-size:14px;font-weight:600;fill:#1f2328}
.r{font-size:12px;fill:#cf222e;font-weight:600}.g{font-size:12px;fill:#1a7f37;font-weight:600}
.box{fill:#fff;stroke:#8c959f;stroke-width:1.4;rx:6}
.dead{fill:#fff5f5;stroke:#cf222e;stroke-width:1.6;stroke-dasharray:5 3;rx:6}
.ok{fill:#f6fdf6;stroke:#1a7f37;stroke-width:1.6;rx:6}
.ln{stroke:#59636e;stroke-width:1.4;fill:none;marker-end:url(#a)}
.bad{stroke:#cf222e;stroke-width:1.6;stroke-dasharray:5 3;fill:none;marker-end:url(#b)}
</style>
<defs>
<marker id="a" markerWidth="9" markerHeight="9" refX="8" refY="4.5" orient="auto"><path d="M0,0 L9,4.5 L0,9 z" fill="#59636e"/></marker>
<marker id="b" markerWidth="9" markerHeight="9" refX="8" refY="4.5" orient="auto"><path d="M0,0 L9,4.5 L0,9 z" fill="#cf222e"/></marker>
</defs>
<text class="h" x="16" y="24">C-1 · SSO 의 구조와 IdP 로그아웃의 한계</text>
<rect class="box" x="220" y="42" width="260" height="52"/>
<text class="t" x="350" y="64" text-anchor="middle">Keycloak user session ×1</text>
<text class="s" x="350" y="82" text-anchor="middle">oqOjHekin4JU-BZjgQLjUByW</text>
<path class="ln" d="M300,96 L180,130"/><path class="ln" d="M400,96 L520,130"/>
<rect class="box" x="60" y="134" width="240" height="52"/>
<text class="t" x="180" y="156" text-anchor="middle">client session — bff-confidential</text>
<text class="s" x="180" y="174" text-anchor="middle">app1 · Redis 세션 + PostgreSQL 토큰</text>
<rect class="box" x="400" y="134" width="240" height="52"/>
<text class="t" x="520" y="156" text-anchor="middle">client session — oauth2-proxy</text>
<text class="s" x="520" y="174" text-anchor="middle">app2 · 쿠키 티켓 + Redis 세션</text>
<rect class="dead" x="220" y="206" width="260" height="40"/>
<text class="r" x="350" y="231" text-anchor="middle">IdP 세션 삭제 ✗</text>
<path class="bad" d="M300,246 L200,262"/><path class="bad" d="M400,246 L500,262"/>
<rect class="ok" x="60" y="256" width="240" height="34"/><text class="g" x="180" y="278" text-anchor="middle">app1 그대로 동작</text>
<rect class="ok" x="400" y="256" width="240" height="34"/><text class="g" x="520" y="278" text-anchor="middle">app2 그대로 동작</text>
</svg>

After

Width:  |  Height:  |  Size: 2.4 KiB

+44
View File
@@ -0,0 +1,44 @@
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 860 400" font-family="-apple-system,BlinkMacSystemFont,Segoe UI,Helvetica,sans-serif">
<style>
.box{fill:#fff;stroke:#8c959f;stroke-width:1.4;rx:6}
.dead{fill:#fff5f5;stroke:#cf222e;stroke-width:1.6;stroke-dasharray:5 3;rx:6}
.ok{fill:#f6fdf6;stroke:#1a7f37;stroke-width:1.6;rx:6}
.warn{fill:#fff8f0;stroke:#bf8700;stroke-width:1.5;rx:6}
.t{font-size:12px;fill:#1f2328}
.tb{font-size:12.5px;fill:#1f2328;font-weight:600}
.s{font-size:10.5px;fill:#656d76}
.ttl{font-size:14px;fill:#1f2328;font-weight:600}
.r{stroke:#cf222e;stroke-width:1.6;fill:none}
.g{stroke:#1a7f37;stroke-width:1.6;fill:none}
.k{stroke:#8c959f;stroke-width:1.3;fill:none}
</style>
<defs>
<marker id="a" markerWidth="8" markerHeight="8" refX="7" refY="3" orient="auto"><path d="M0,0 L7,3 L0,6 z" fill="#8c959f"/></marker>
<marker id="ar" markerWidth="8" markerHeight="8" refX="7" refY="3" orient="auto"><path d="M0,0 L7,3 L0,6 z" fill="#cf222e"/></marker>
<marker id="ag" markerWidth="8" markerHeight="8" refX="7" refY="3" orient="auto"><path d="M0,0 L7,3 L0,6 z" fill="#1a7f37"/></marker>
</defs>
<text x="20" y="26" class="ttl">C-2 — 로그아웃이 안 퍼지는 이유는 단순했다. 아무도 구현하지 않았다</text>
<text x="20" y="44" class="s">C-1 이 관측한 현상의 원인</text>
<text x="20" y="76" class="tb">백채널 로그아웃이 동작하려면 양쪽이 다 있어야 한다</text>
<rect x="30" y="94" width="380" height="110" class="dead"/>
<text x="46" y="116" class="tb">① IdP 쪽 — Keycloak 클라이언트 설정</text>
<text x="46" y="140" class="t">backchannelLogoutUrl</text>
<text x="46" y="162" class="t" fill="#cf222e">두 클라이언트 모두 비어 있다</text>
<text x="46" y="186" class="s">Keycloak 은 부를 주소를 모른다</text>
<rect x="440" y="94" width="390" height="110" class="dead"/>
<text x="456" y="116" class="tb">② 앱 쪽 — 수신 엔드포인트</text>
<text x="456" y="140" class="t">/logout/connect/back-channel/{registrationId}</text>
<text x="456" y="162" class="t" fill="#cf222e">BFF 소스에 oidcLogout 설정이 없다</text>
<text x="456" y="186" class="s">주소를 알려줘도 받을 곳이 없다</text>
<rect x="30" y="228" width="800" height="60" class="warn"/>
<text x="46" y="250" class="tb">★ 그래서 IdP 쪽만 설정하면 되는 줄 알면 틀린다</text>
<text x="46" y="272" class="s">backchannelLogoutUrl 을 넣어 봤지만 앱 세션은 그대로 남았다. 받는 쪽이 없기 때문이다</text>
<rect x="30" y="308" width="380" height="72" class="ok"/>
<text x="46" y="330" class="tb">네트워크 문제가 아니라는 확인</text>
<text x="46" y="352" class="s">Keycloak 파드에서 앱 URL 로 요청 → HTTP 200</text>
<text x="46" y="372" class="s">닿기는 한다. 닿아도 처리할 코드가 없을 뿐이다</text>
<rect x="440" y="308" width="390" height="72" class="box"/>
<text x="456" y="330" class="tb">남기는 것</text>
<text x="456" y="352" class="s">「설정이 빠졌다」와 「기능이 없다」는 다르게 고쳐야 한다</text>
<text x="456" y="372" class="s">여기는 둘 다였고, 확인 순서를 바꿨다면 한쪽만 고치고 끝냈을 것이다</text>
</svg>

After

Width:  |  Height:  |  Size: 3.2 KiB

+30
View File
@@ -0,0 +1,30 @@
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 700 270" width="700" height="270" font-family="system-ui,-apple-system,Segoe UI,Roboto,sans-serif">
<style>
.t{font-size:13px;fill:#1f2328}.s{font-size:11px;fill:#59636e}.h{font-size:14px;font-weight:600;fill:#1f2328}
.r{font-size:12px;fill:#cf222e;font-weight:600}.g{font-size:12px;fill:#1a7f37;font-weight:600}
.box{fill:#fff;stroke:#8c959f;stroke-width:1.4;rx:6}
.dead{fill:#fff5f5;stroke:#cf222e;stroke-width:1.6;stroke-dasharray:5 3;rx:6}
.ok{fill:#f6fdf6;stroke:#1a7f37;stroke-width:1.6;rx:6}
.ln{stroke:#59636e;stroke-width:1.4;fill:none;marker-end:url(#a)}
.bad{stroke:#cf222e;stroke-width:1.6;stroke-dasharray:5 3;fill:none;marker-end:url(#b)}
</style>
<defs>
<marker id="a" markerWidth="9" markerHeight="9" refX="8" refY="4.5" orient="auto"><path d="M0,0 L9,4.5 L0,9 z" fill="#59636e"/></marker>
<marker id="b" markerWidth="9" markerHeight="9" refX="8" refY="4.5" orient="auto"><path d="M0,0 L9,4.5 L0,9 z" fill="#cf222e"/></marker>
</defs>
<text class="h" x="16" y="24">D-1 · 빈 데이터베이스가 200 을 냈다</text>
<rect class="box" x="30" y="46" width="180" height="46"/><text class="t" x="120" y="66" text-anchor="middle">pg_dump</text><text class="s" x="120" y="82" text-anchor="middle">395KB · 101 테이블 · 세션 포함</text>
<path class="ln" d="M210,69 L270,69"/>
<rect class="dead" x="270" y="42" width="180" height="54"/>
<text class="r" x="360" y="64" text-anchor="middle">DROP SCHEMA CASCADE</text><text class="s" x="360" y="84" text-anchor="middle">남은 테이블 0</text>
<path class="ln" d="M450,69 L510,69"/>
<rect class="ok" x="510" y="42" width="160" height="54"/>
<text class="g" x="590" y="64" text-anchor="middle">복구 1초</text><text class="s" x="590" y="84" text-anchor="middle">오류 0건 · 재시작 0회</text>
<rect class="box" x="30" y="118" width="640" height="76"/>
<text class="t" x="350" y="140" text-anchor="middle">테이블이 0개일 때 무엇이 깨졌는가 — 전부가 아니다</text>
<text class="g" x="350" y="162" text-anchor="middle">/protocol/openid-connect/certs → 200 (realm 키가 캐시에 있다)</text>
<text class="r" x="350" y="182" text-anchor="middle">/.well-known → 500 토큰 발급 → 400</text>
<rect class="dead" x="30" y="210" width="640" height="46"/>
<text class="r" x="350" y="232" text-anchor="middle">헬스체크는 "DB 가 살아 있다"만 보고 "데이터가 있다"는 안 본다</text>
<text class="s" x="350" y="250" text-anchor="middle">RPO = 백업 주기 + A-3 의 synchronous_commit 손실 · 덤프는 같은 호스트 /tmp 에 있었다</text>
</svg>

After

Width:  |  Height:  |  Size: 2.6 KiB

+48
View File
@@ -0,0 +1,48 @@
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 860 420" font-family="-apple-system,BlinkMacSystemFont,Segoe UI,Helvetica,sans-serif">
<style>
.box{fill:#fff;stroke:#8c959f;stroke-width:1.4;rx:6}
.dead{fill:#fff5f5;stroke:#cf222e;stroke-width:1.6;stroke-dasharray:5 3;rx:6}
.ok{fill:#f6fdf6;stroke:#1a7f37;stroke-width:1.6;rx:6}
.warn{fill:#fff8f0;stroke:#bf8700;stroke-width:1.5;rx:6}
.t{font-size:12px;fill:#1f2328}
.tb{font-size:12.5px;fill:#1f2328;font-weight:600}
.s{font-size:10.5px;fill:#656d76}
.ttl{font-size:14px;fill:#1f2328;font-weight:600}
.r{stroke:#cf222e;stroke-width:1.6;fill:none}
.g{stroke:#1a7f37;stroke-width:1.6;fill:none}
.k{stroke:#8c959f;stroke-width:1.3;fill:none}
</style>
<defs>
<marker id="a" markerWidth="8" markerHeight="8" refX="7" refY="3" orient="auto"><path d="M0,0 L7,3 L0,6 z" fill="#8c959f"/></marker>
<marker id="ar" markerWidth="8" markerHeight="8" refX="7" refY="3" orient="auto"><path d="M0,0 L7,3 L0,6 z" fill="#cf222e"/></marker>
<marker id="ag" markerWidth="8" markerHeight="8" refX="7" refY="3" orient="auto"><path d="M0,0 L7,3 L0,6 z" fill="#1a7f37"/></marker>
</defs>
<text x="20" y="26" class="ttl">D-2 — 롤백은 안 된다. 다만 롤링 업데이트가 사고를 절반에서 멈춰줬다</text>
<text x="20" y="44" class="s">그리고 이 결론은 조건부였다 — 스키마가 안 바뀌면 롤백된다(후속에서 정밀화)</text>
<rect x="30" y="72" width="380" height="126" class="box"/>
<text x="46" y="94" class="tb">앞으로 (26.7.0 → 26.7.3)</text>
<text x="46" y="118" class="t" fill="#1a7f37">된다. 무중단</text>
<text x="46" y="140" class="s">87회 요청 전부 200</text>
<text x="46" y="160" class="s">databasechangelog 210 → 210 (스키마 변경 없음)</text>
<text x="46" y="180" class="s">세션 유지 · Infinispan 16.0.12 → 16.0.14</text>
<rect x="440" y="72" width="390" height="126" class="dead"/>
<text x="456" y="94" class="tb">뒤로 (스키마가 이미 움직였을 때)</text>
<text x="456" y="118" class="t" fill="#cf222e">안 된다</text>
<text x="456" y="140" class="s">liquibase ValidationFailedException:</text>
<text x="456" y="158" class="s">1 changesets check sum</text>
<text x="456" y="180" class="s">새 버전이 남긴 체크섬을 옛 버전이 거부한다</text>
<rect x="30" y="222" width="800" height="72" class="ok"/>
<text x="46" y="244" class="tb">★ 그런데 서비스는 살아 있었다 — StatefulSet 롤링 업데이트 덕분이다</text>
<text x="46" y="266" class="s">한 번에 한 파드씩 바꾼다. 첫 파드가 기동에 실패하면 거기서 멈추고 나머지는 건드리지 않는다</text>
<text x="46" y="286" class="s">그래서 남은 파드가 외부 200 을 계속 냈다. 「롤백 계획」이 없어도 사고가 전면화되지 않았다</text>
<rect x="30" y="314" width="380" height="90" class="warn"/>
<text x="46" y="336" class="tb">그러면 무엇으로 판단하나</text>
<text x="46" y="358" class="t">select count(*) from databasechangelog</text>
<text x="46" y="380" class="s">업그레이드 전후 이 수가 같으면 롤백 가능</text>
<text x="46" y="398" class="s">늘었으면 스키마가 움직였다 = 롤백 불가</text>
<rect x="440" y="314" width="390" height="90" class="box"/>
<text x="456" y="336" class="tb">전제 — 백업이 먼저다 (D-1)</text>
<text x="456" y="358" class="s">롤백이 막히는 상황에서 되돌릴 방법은 덤프 복원뿐이다</text>
<text x="456" y="380" class="s">D-1 이 그 절차를 재고, D-2 가 그 절차가 필요한 이유를 만든다</text>
<text x="456" y="398" class="s">순서를 바꾸면 되돌릴 수 없는 상태에서 백업을 배우게 된다</text>
</svg>

After

Width:  |  Height:  |  Size: 3.6 KiB

+31
View File
@@ -0,0 +1,31 @@
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 700 280" width="700" height="280" font-family="system-ui,-apple-system,Segoe UI,Roboto,sans-serif">
<style>
.t{font-size:13px;fill:#1f2328}.s{font-size:11px;fill:#59636e}.h{font-size:14px;font-weight:600;fill:#1f2328}
.r{font-size:12px;fill:#cf222e;font-weight:600}.g{font-size:12px;fill:#1a7f37;font-weight:600}
.box{fill:#fff;stroke:#8c959f;stroke-width:1.4;rx:6}
.dead{fill:#fff5f5;stroke:#cf222e;stroke-width:1.6;stroke-dasharray:5 3;rx:6}
.ok{fill:#f6fdf6;stroke:#1a7f37;stroke-width:1.6;rx:6}
.ln{stroke:#59636e;stroke-width:1.4;fill:none;marker-end:url(#a)}
.bad{stroke:#cf222e;stroke-width:1.6;stroke-dasharray:5 3;fill:none;marker-end:url(#b)}
</style>
<defs>
<marker id="a" markerWidth="9" markerHeight="9" refX="8" refY="4.5" orient="auto"><path d="M0,0 L9,4.5 L0,9 z" fill="#59636e"/></marker>
<marker id="b" markerWidth="9" markerHeight="9" refX="8" refY="4.5" orient="auto"><path d="M0,0 L9,4.5 L0,9 z" fill="#cf222e"/></marker>
</defs>
<text class="h" x="16" y="24">D-3 · 네 경로 중 RBAC 만 실제로 감춘다</text>
<rect class="dead" x="30" y="46" width="310" height="60"/>
<text class="r" x="185" y="68" text-anchor="middle">kubectl get -o jsonpath | base64 -d</text>
<text class="s" x="185" y="88" text-anchor="middle">POSTGRES_PASSWORD = lab-postgres-change-me</text>
<rect class="dead" x="360" y="46" width="310" height="60"/>
<text class="r" x="515" y="68" text-anchor="middle">저장소 (at rest)</text>
<text class="s" x="515" y="88" text-anchor="middle">Encryption Disabled · state.db 에 평문</text>
<rect class="dead" x="30" y="120" width="310" height="60"/>
<text class="r" x="185" y="142" text-anchor="middle">파드 안</text>
<text class="s" x="185" y="162" text-anchor="middle">KEYCLOAK_CLIENT_SECRET=... 환경변수</text>
<rect class="ok" x="360" y="120" width="310" height="60"/>
<text class="g" x="515" y="142" text-anchor="middle">RBAC</text>
<text class="s" x="515" y="162" text-anchor="middle">default SA 는 get secrets 불가</text>
<rect class="box" x="30" y="200" width="640" height="60"/>
<text class="t" x="350" y="222" text-anchor="middle">describe 는 "14 bytes" 만 보여줘 감춰졌다는 착각을 준다</text>
<text class="s" x="350" y="244" text-anchor="middle">base64 는 감추기 위한 것이 아니라 YAML 에 임의 바이트를 담기 위한 인코딩이다</text>
</svg>

After

Width:  |  Height:  |  Size: 2.4 KiB

+30
View File
@@ -0,0 +1,30 @@
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 700 250" width="700" height="250" font-family="system-ui,-apple-system,Segoe UI,Roboto,sans-serif">
<style>
.t{font-size:13px;fill:#1f2328}.s{font-size:11px;fill:#59636e}.h{font-size:14px;font-weight:600;fill:#1f2328}
.r{font-size:12px;fill:#cf222e;font-weight:600}.g{font-size:12px;fill:#1a7f37;font-weight:600}
.box{fill:#fff;stroke:#8c959f;stroke-width:1.4;rx:6}
.dead{fill:#fff5f5;stroke:#cf222e;stroke-width:1.6;stroke-dasharray:5 3;rx:6}
.ok{fill:#f6fdf6;stroke:#1a7f37;stroke-width:1.6;rx:6}
.ln{stroke:#59636e;stroke-width:1.4;fill:none;marker-end:url(#a)}
.bad{stroke:#cf222e;stroke-width:1.6;stroke-dasharray:5 3;fill:none;marker-end:url(#b)}
</style>
<defs>
<marker id="a" markerWidth="9" markerHeight="9" refX="8" refY="4.5" orient="auto"><path d="M0,0 L9,4.5 L0,9 z" fill="#59636e"/></marker>
<marker id="b" markerWidth="9" markerHeight="9" refX="8" refY="4.5" orient="auto"><path d="M0,0 L9,4.5 L0,9 z" fill="#cf222e"/></marker>
</defs>
<text class="h" x="16" y="24">D-4 · 인증서 체인과 SAN 제약</text>
<rect class="ok" x="30" y="46" width="300" height="104"/>
<text class="t" x="180" y="68" text-anchor="middle">체인 4단계 · Verify return code: 0</text>
<text class="s" x="180" y="90" text-anchor="middle">0 CN=auth.hyeonworks.com</text>
<text class="s" x="180" y="106" text-anchor="middle">1 Let's Encrypt YE2</text>
<text class="s" x="180" y="122" text-anchor="middle">2 ISRG Root YE</text>
<text class="s" x="180" y="138" text-anchor="middle">3 ISRG Root X2</text>
<rect class="box" x="360" y="46" width="310" height="104"/>
<text class="t" x="515" y="68" text-anchor="middle">SAN 3개 · 와일드카드 아님</text>
<text class="s" x="515" y="90" text-anchor="middle">auth · app1 · app2</text>
<text class="r" x="515" y="114" text-anchor="middle">네 번째 이름이 없다</text>
<text class="s" x="515" y="134" text-anchor="middle">B-7 에서 Grafana 의 app2 를 빌려야 했다</text>
<rect class="box" x="30" y="168" width="640" height="60"/>
<text class="t" x="350" y="190" text-anchor="middle">단계가 1개면 cert.pem, 2개 이상이면 fullchain.pem 이다</text>
<text class="s" x="350" y="212" text-anchor="middle">브라우저는 중간 인증서를 캐시하므로 cert.pem 실수는 캐시 없는 클라이언트에서만 드러난다</text>
</svg>

After

Width:  |  Height:  |  Size: 2.4 KiB

+94
View File
@@ -0,0 +1,94 @@
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 860 520" font-family="-apple-system,BlinkMacSystemFont,'Segoe UI',Helvetica,sans-serif">
<style>
.box{fill:#fff;stroke:#8c959f;stroke-width:1.4;rx:6}
.dead{fill:#fff5f5;stroke:#cf222e;stroke-width:1.6;stroke-dasharray:5 3;rx:6}
.ok{fill:#f6fdf6;stroke:#1a7f37;stroke-width:1.6;rx:6}
.t{font-size:12px;fill:#1f2328}
.tb{font-size:12.5px;fill:#1f2328;font-weight:600}
.s{font-size:10.5px;fill:#656d76}
.ttl{font-size:14px;fill:#1f2328;font-weight:600}
.r{stroke:#cf222e;stroke-width:1.6;fill:none}
.g{stroke:#1a7f37;stroke-width:1.6;fill:none}
.k{stroke:#8c959f;stroke-width:1.3;fill:none}
</style>
<defs>
<marker id="a" markerWidth="8" markerHeight="8" refX="7" refY="3" orient="auto"><path d="M0,0 L7,3 L0,6 z" fill="#8c959f"/></marker>
<marker id="ar" markerWidth="8" markerHeight="8" refX="7" refY="3" orient="auto"><path d="M0,0 L7,3 L0,6 z" fill="#cf222e"/></marker>
<marker id="ag" markerWidth="8" markerHeight="8" refX="7" refY="3" orient="auto"><path d="M0,0 L7,3 L0,6 z" fill="#1a7f37"/></marker>
</defs>
<text x="20" y="26" class="ttl">D-4 — 「갱신 성공」과 「새 인증서 서빙」 사이의 38분 25초</text>
<text x="20" y="44" class="s">측정: 2026-09-04 · 일련번호 5초 간격 564표본 · 새 연결 0.2초 간격 8856건</text>
<!-- 좌: 디스크 -->
<rect x="20" y="70" width="250" height="196" class="box"/>
<text x="34" y="92" class="tb">디스크 — certbot</text>
<rect x="36" y="104" width="218" height="46" class="box"/>
<text x="48" y="122" class="t">cert1.pem</text>
<text x="48" y="138" class="s">09-03 10:47:40 · notAfter Dec 2</text>
<rect x="36" y="158" width="218" height="46" class="ok"/>
<text x="48" y="176" class="t">cert2.pem</text>
<text x="48" y="192" class="s">09-04 17:22:13 · notAfter Dec 3</text>
<text x="36" y="224" class="s">certbot renew --force-renewal</text>
<text x="36" y="240" class="s">→ 성공. live/ 링크가 2번을 가리킨다</text>
<text x="36" y="256" class="s">경로는 그대로, 내용만 바뀐다</text>
<!-- 중: 끊긴 고리 -->
<rect x="300" y="96" width="260" height="144" class="dead"/>
<text x="316" y="118" class="tb">★ 잇는 것이 없다</text>
<text x="316" y="140" class="t">1. ExecStartPost</text>
<text x="470" y="140" class="t" fill="#cf222e">없음</text>
<text x="316" y="160" class="t">2. renewal-hooks/deploy/</text>
<text x="470" y="160" class="t" fill="#cf222e">비었음</text>
<text x="316" y="180" class="t">3. certbot nginx 플러그인</text>
<text x="470" y="180" class="t" fill="#cf222e">없음</text>
<text x="316" y="206" class="s">post/ · pre/ 도 비어 있다 (total 8)</text>
<text x="316" y="222" class="s">셋 다 비면 아무도 reload 를 부르지 않는다</text>
<path d="M270 168 L296 168" class="r" marker-end="url(#ar)"/>
<path d="M300 230 L560 108" stroke="#cf222e" stroke-width="1.2" stroke-dasharray="4 3"/>
<path d="M300 108 L560 230" stroke="#cf222e" stroke-width="1.2" stroke-dasharray="4 3"/>
<!-- 우: nginx -->
<rect x="590" y="70" width="250" height="196" class="box"/>
<text x="604" y="92" class="tb">nginx — 메모리</text>
<rect x="606" y="104" width="218" height="60" class="dead"/>
<text x="618" y="124" class="t">기동 시 읽은 cert1</text>
<text x="618" y="140" class="s">master 585 / worker 586</text>
<text x="618" y="156" class="s">둘 다 09-03 19:00:39, 22.4시간째</text>
<text x="606" y="184" class="s">reload 없이는 디스크를 다시 안 본다.</text>
<text x="606" y="200" class="s">reload 하면 마스터는 남고</text>
<text x="606" y="216" class="s">워커만 새로 뜬다 → 워커 PID 로 확인</text>
<text x="606" y="240" class="s">ssl_certificate .../fullchain.pem</text>
<path d="M564 168 L586 168" class="k" marker-end="url(#a)"/>
<!-- 타임라인 -->
<text x="20" y="306" class="tb">밖에서 본 것 — 일련번호 (5초 간격)</text>
<line x1="60" y1="356" x2="800" y2="356" class="k"/>
<line x1="60" y1="350" x2="60" y2="362" class="k"/>
<line x1="300" y1="350" x2="300" y2="362" class="k"/>
<line x1="660" y1="344" x2="660" y2="368" stroke="#1a7f37" stroke-width="2"/>
<line x1="800" y1="350" x2="800" y2="362" class="k"/>
<text x="46" y="382" class="s">08:10:51</text>
<text x="278" y="382" class="s">08:22:13</text>
<text x="620" y="382" class="s">08:58:52</text>
<text x="770" y="382" class="s">09:02</text>
<text x="278" y="398" class="s">디스크 기록</text>
<text x="618" y="398" class="s">nginx -s reload</text>
<rect x="60" y="330" width="600" height="16" class="dead"/>
<text x="300" y="342" class="s" text-anchor="middle" fill="#cf222e">0520BB…B853 (옛 인증서) — 이 구간에서 428회 관측</text>
<rect x="660" y="330" width="140" height="16" class="ok"/>
<text x="730" y="342" class="s" text-anchor="middle" fill="#1a7f37">06C7CB…EA1D</text>
<path d="M300 416 L660 416" class="r" marker-end="url(#ar)"/>
<path d="M660 416 L300 416" class="r" marker-end="url(#ar)"/>
<text x="480" y="410" class="tb" text-anchor="middle" fill="#cf222e">38분 25초</text>
<text x="480" y="434" class="s" text-anchor="middle">사람이 reload 했기에 36분이다. 아무도 안 했다면 다음 재시작까지 — 무기한</text>
<!-- reload 판정 -->
<rect x="20" y="452" width="400" height="52" class="ok"/>
<text x="34" y="472" class="tb">reload 는 무중단이었다</text>
<text x="34" y="490" class="s">새 연결 8856건 전부 200 · p95 205.7 → 204.3ms (변화 없음)</text>
<rect x="440" y="452" width="400" height="52" class="ok"/>
<text x="454" y="472" class="tb">진행 중이던 요청도 살아남았다</text>
<text x="454" y="490" class="s">전송 12초째에 reload · 845361바이트 전량 · 연결수 1</text>
</svg>

After

Width:  |  Height:  |  Size: 5.6 KiB

+57
View File
@@ -0,0 +1,57 @@
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 860 430" font-family="-apple-system,BlinkMacSystemFont,'Segoe UI',Helvetica,sans-serif">
<style>
.box{fill:#fff;stroke:#8c959f;stroke-width:1.4;rx:6}
.dead{fill:#fff5f5;stroke:#cf222e;stroke-width:1.6;stroke-dasharray:5 3;rx:6}
.ok{fill:#f6fdf6;stroke:#1a7f37;stroke-width:1.6;rx:6}
.warn{fill:#fff8f0;stroke:#bf8700;stroke-width:1.5;rx:6}
.t{font-size:12px;fill:#1f2328}
.tb{font-size:12.5px;fill:#1f2328;font-weight:600}
.s{font-size:10.5px;fill:#656d76}
.ttl{font-size:14px;fill:#1f2328;font-weight:600}
.r{stroke:#cf222e;stroke-width:1.6;fill:none}
.g{stroke:#1a7f37;stroke-width:1.6;fill:none}
.k{stroke:#8c959f;stroke-width:1.3;fill:none}
</style>
<defs>
<marker id="a" markerWidth="8" markerHeight="8" refX="7" refY="3" orient="auto"><path d="M0,0 L7,3 L0,6 z" fill="#8c959f"/></marker>
<marker id="ar" markerWidth="8" markerHeight="8" refX="7" refY="3" orient="auto"><path d="M0,0 L7,3 L0,6 z" fill="#cf222e"/></marker>
<marker id="ag" markerWidth="8" markerHeight="8" refX="7" refY="3" orient="auto"><path d="M0,0 L7,3 L0,6 z" fill="#1a7f37"/></marker>
</defs>
<text x="20" y="26" class="ttl">D-4a — deploy 훅 하나로 38분 25초가 1~2초가 된다</text>
<text x="20" y="44" class="s">시각은 전부 실제 UTC (test-server 시계에서 106초 보정)</text>
<rect x="20" y="68" width="820" height="112" class="dead"/>
<text x="34" y="90" class="tb">D-4 — 훅이 없을 때</text>
<line x1="60" y1="130" x2="800" y2="130" class="k"/>
<line x1="90" y1="120" x2="90" y2="140" stroke="#cf222e" stroke-width="2"/>
<line x1="740" y1="120" x2="740" y2="140" stroke="#1a7f37" stroke-width="2"/>
<rect x="90" y="122" width="650" height="16" class="dead"/>
<text x="415" y="134" class="s" text-anchor="middle" fill="#cf222e">옛 인증서를 계속 서빙 — 428회 관측</text>
<text x="62" y="158" class="s">08:20:27 디스크 기록</text>
<text x="660" y="158" class="s">08:58:52 사람이 nginx -s reload</text>
<text x="415" y="112" class="tb" text-anchor="middle" fill="#cf222e">2305초 = 38분 25초</text>
<text x="34" y="174" class="s">아무도 reload 하지 않았다면 다음 nginx 재시작까지 = 사실상 무기한</text>
<rect x="20" y="196" width="820" height="112" class="ok"/>
<text x="34" y="218" class="tb">D-4a — deploy 훅을 넣었을 때</text>
<line x1="60" y1="258" x2="800" y2="258" class="k"/>
<line x1="90" y1="248" x2="90" y2="268" stroke="#1a7f37" stroke-width="2"/>
<line x1="104" y1="248" x2="104" y2="268" stroke="#1a7f37" stroke-width="2"/>
<rect x="90" y="250" width="14" height="16" class="ok"/>
<text x="62" y="286" class="s">12:27:49.05 발급 (SCT — CT 로그의 독립 시계)</text>
<text x="360" y="286" class="s">12:27:50 nginx -t · 새 워커 37252 · 12:27:51 reload</text>
<text x="150" y="244" class="tb" fill="#1a7f37">1~2초, 자동</text>
<text x="34" y="302" class="s">certbot 이 파일을 쓴 직후 같은 실행 안에서 deploy 훅을 부른다 — 사람이 개입할 자리가 없다</text>
<rect x="20" y="324" width="400" height="92" class="box"/>
<text x="34" y="346" class="tb">판정은 문구가 아니라 워커 PID 로</text>
<text x="34" y="366" class="s">master 585 → 585 (유지)</text>
<text x="34" y="384" class="s">worker 28829 → 37252 (교체 = reload 됨)</text>
<text x="34" y="404" class="s">nginx 는 인증서를 기동 시 읽어 메모리에 든다</text>
<rect x="444" y="324" width="396" height="92" class="warn"/>
<text x="458" y="346" class="tb">★ 함정 — 성공을 실패로 오독한다</text>
<text x="458" y="366" class="s">Hook 'deploy-hook' ran with error output:</text>
<text x="458" y="384" class="s"> [warn] could not build optimal types_hash …</text>
<text x="458" y="402" class="s">nginx 경고가 stderr 로 나갔을 뿐. 내용은 test is successful</text>
</svg>

After

Width:  |  Height:  |  Size: 3.8 KiB

+57
View File
@@ -0,0 +1,57 @@
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 760 470" width="760" height="470" font-family="system-ui,-apple-system,Segoe UI,Roboto,sans-serif">
<style>
.t{font-size:13px;fill:#1f2328}.s{font-size:11px;fill:#59636e}.h{font-size:14px;font-weight:600;fill:#1f2328}
.box{fill:#fff;stroke:#8c959f;stroke-width:1.4;rx:6}
.host{fill:#f6f8fa;stroke:#59636e;stroke-width:1.6;rx:8}
.node{fill:#fff;stroke:#0969da;stroke-width:1.6;rx:8}
.ln{stroke:#59636e;stroke-width:1.4;fill:none;marker-end:url(#a)}
</style>
<defs><marker id="a" markerWidth="9" markerHeight="9" refX="8" refY="4.5" orient="auto"><path d="M0,0 L9,4.5 L0,9 z" fill="#59636e"/></marker></defs>
<rect class="box" x="290" y="12" width="180" height="38"/>
<text class="t" x="380" y="30" text-anchor="middle">개발 노트북</text>
<text class="s" x="380" y="44" text-anchor="middle">브라우저 · kubectl · Playwright</text>
<path class="ln" d="M380,52 L380,80"/>
<text class="s" x="392" y="70">https · tailnet 100.x · split DNS</text>
<rect class="host" x="40" y="84" width="680" height="66"/>
<text class="h" x="56" y="106">test-server</text>
<text class="s" x="56" y="122">Arch Linux · 12GB · WiFi only · sudo 는 비밀번호 필요</text>
<rect class="box" x="430" y="94" width="270" height="46"/>
<text class="t" x="565" y="112" text-anchor="middle">nginx :443 — TLS 종료</text>
<text class="s" x="565" y="128" text-anchor="middle">auth / app1 / app2 (SAN 3개, 와일드카드 아님)</text>
<path class="ln" d="M240,152 L200,186"/><path class="ln" d="M520,152 L560,186"/>
<text class="s" x="330" y="172" text-anchor="middle">http · libvirt NAT (virbr0)</text>
<rect class="node" x="40" y="190" width="320" height="250"/>
<text class="h" x="56" y="212">kc-lab-1 · 5120MB</text>
<text class="s" x="56" y="228">k3s server · 10.42.0.0/24</text>
<rect class="box" x="56" y="238" width="130" height="26"/><text class="t" x="121" y="255" text-anchor="middle">traefik ×1</text>
<rect class="box" x="196" y="238" width="148" height="26"/><text class="t" x="270" y="255" text-anchor="middle">coredns</text>
<rect class="box" x="56" y="272" width="130" height="26"/><text class="t" x="121" y="289" text-anchor="middle">keycloak-1</text>
<rect class="box" x="196" y="272" width="148" height="26"/><text class="t" x="270" y="289" text-anchor="middle">bff (1/2)</text>
<rect class="box" x="56" y="306" width="288" height="26"/><text class="t" x="200" y="323" text-anchor="middle">oauth2-proxy (1/2)</text>
<rect class="box" x="56" y="340" width="288" height="46"/>
<text class="t" x="200" y="358" text-anchor="middle">prometheus (PVC) · grafana</text>
<text class="s" x="200" y="374" text-anchor="middle">관측 스택은 여기 고정 — 죽이지 않는다</text>
<rect class="node" x="400" y="190" width="320" height="250"/>
<text class="h" x="416" y="212">kc-lab-2 · 4096MB</text>
<text class="s" x="416" y="228">k3s agent · 10.42.1.0/24</text>
<rect class="box" x="416" y="238" width="288" height="26"/><text class="t" x="560" y="255" text-anchor="middle">keycloak-0</text>
<rect class="box" x="416" y="272" width="140" height="26"/><text class="t" x="486" y="289" text-anchor="middle">bff (2/2)</text>
<rect class="box" x="566" y="272" width="138" height="26"/><text class="t" x="635" y="289" text-anchor="middle">oauth2-proxy</text>
<rect class="box" x="416" y="306" width="140" height="46"/>
<text class="t" x="486" y="324" text-anchor="middle">postgres</text><text class="s" x="486" y="340" text-anchor="middle">PVC (노드 고정)</text>
<rect class="box" x="566" y="306" width="138" height="46"/>
<text class="t" x="635" y="324" text-anchor="middle">redis</text><text class="s" x="635" y="340" text-anchor="middle">PVC + AOF</text>
<text class="s" x="560" y="374" text-anchor="middle">장애 주입은 여기</text>
<path d="M360,290 L400,290" stroke="#0969da" stroke-width="1.6" fill="none" marker-end="url(#a)"/>
<path d="M400,300 L360,300" stroke="#0969da" stroke-width="1.6" fill="none" marker-end="url(#a)"/>
<text class="s" x="380" y="284" text-anchor="middle">7800</text>
<text class="s" x="380" y="318" text-anchor="middle">JGroups</text>
<text class="s" x="40" y="460">A-0 에서 확인: 세션은 이 7800 이 아니라 postgres 를 통해 공유된다</text>
</svg>

After

Width:  |  Height:  |  Size: 4.3 KiB

@@ -0,0 +1,9 @@
=== A-3 이 가정만 하고 재지 않은 값 ===
name | setting | unit | source
------------------------+---------+------+---------
commit_delay | 0 | | default
synchronous_commit | on | | default
wal_writer_delay | 200 | ms | default
wal_writer_flush_after | 128 | 8kB | default
(4 rows)
@@ -0,0 +1,19 @@
# 주의 — 이 파일은 원 실험 시점에 0바이트로 저장됐다.
# 리다이렉션이 stdout 만 받았는데 출력이 stderr 로 갔거나 tee 앞 파이프가
# 비어 있었던 것으로 보인다. README 는 그 사이 파일 내용을 서술하고 있었는데,
# 그것은 화면에서 본 것을 적은 것이지 이 파일에서 온 것이 아니었다.
#
# 아래는 사후에 다시 수집한 것이며, 원 시점의 DROP 규칙(0 패킷)은 이미
# 제거되어 재현되지 않는다. 구조적 사실(kube-router 가 자기 체인을 FORWARD
# 최상단에 유지한다)만 확인할 수 있다.
# 원 실험의 결정적 증거는 04-correct-direction.txt 의 패킷 카운터 19/21 이다.
=== A-5 재수집 — filter 테이블 규칙이 CNI 체인에 밀리는 것 ===
Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
num pkts bytes target prot opt in out source destination
1 1690 3386K KUBE-ROUTER-FORWARD 0 -- * * 0.0.0.0/0 0.0.0.0/0 /* kube-router netpol - TEMCG2JMHZYE7H7T */
2 7 612 KUBE-PROXY-FIREWALL 0 -- * * 0.0.0.0/0 0.0.0.0/0 ctstate NEW /* kubernetes load balancer firewall */
3 40 13196 KUBE-FORWARD 0 -- * * 0.0.0.0/0 0.0.0.0/0 /* kubernetes forwarding rules */
(원 실험 시점의 규칙은 이미 제거됐다. 아래는 kube-router 가 자기 체인을
FORWARD 최상단에 유지한다는 구조적 사실만 보여준다 — 그것이 실패 원인이었다.)
@@ -0,0 +1,117 @@
A-7a — volatile 모드에서 refresh 가 500 인 진짜 이유
=======================================================
수집: 2026-09-04 11:18 ~ 11:24 UTC · kc-lab-1 에서 sudo kubectl
A-7 이 남긴 가설
----------------
> **측정은 확실하지만 원인은 확정하지 못했다.** 유력한 후보는
> `REVOKED_TOKEN` 테이블이다 — refresh token 회전에서 이미 쓴 토큰인지
> 확인하려면 그 테이블을 봐야 하고, 그 경로는 캐시되지 않는다.
★ 이 가설은 틀렸다.
방법 — PostgreSQL 문장 로깅 (A-3 기법)
---------------------------------------
ALTER SYSTEM SET log_statement = 'all'; SELECT pg_reload_conf();
표식을 넣어 로그인과 refresh 가 각각 어떤 SQL 을 쏘는지 구분했다.
select 'MARK_LOGIN_START' ... 로그인 ... select 'MARK_LOGIN_END'
select 'MARK_REFRESH_START' ... refresh ... select 'MARK_REFRESH_END'
volatile 전환 확인
args: ["start","--features-disabled=persistent-user-sessions"]
로그인 200 · offline_user_session 행수 = 0 ← volatile 맞다
[측정 1] 로그인은 SQL 을 0개 쏜다
----------------------------------
11:18:49.461 statement: select 'MARK_LOGIN_START'
11:18:49.743 statement: select 'MARK_LOGIN_END'
↑ 사이에 아무것도 없다
realm·사용자·클라이언트가 전부 Infinispan 캐시에 있어 DB 를 안 본다.
[측정 2] ★ refresh 는 CLIENT_SCOPE_CLIENT 를 본다
--------------------------------------------------
11:18:52.009 statement: select 'MARK_REFRESH_START'
11:18:52.137 statement: BEGIN
11:18:52.137 execute <unnamed>/C_107:
select cscme1_0.SCOPE_ID from CLIENT_SCOPE_CLIENT cscme1_0
where cscme1_0.CLIENT_ID=$1 and cscme1_0.DEFAULT_SCOPE=$2
parameters: $1 = '131a9912-b578-4b9c-b16a-97518704077e', $2 = 'f'
11:18:52.148 execute S_2: COMMIT
11:18:52.253 statement: select 'MARK_REFRESH_END'
REVOKED_TOKEN 은 **한 번도 나오지 않는다.**
`DEFAULT_SCOPE='f'` 이므로 **선택적(optional) 클라이언트 스코프** 조회다.
[측정 3] 그 조회는 한 번뿐이다 — 캐시에 들어간다
-------------------------------------------------
연속 refresh 3회, 전부 200. 표식 사이 SQL:
statement: select 'MARK_R1'
statement: select 'MARK_R2'
statement: select 'MARK_R3'
statement: select 'MARK_R_END'
↑ SQL 0건
첫 refresh 가 캐시를 채우고, 이후로는 DB 를 보지 않는다.
════ 그래서 A-7 의 표 자체가 조건부였다 ════
같은 설정에서 **캐시 온도만으로 결과가 셋으로 갈린다.** 전부 측정했다.
┌──────────────────────┬────────┬─────────┬──────────────────────────────┐
│ 캐시 상태 │ 로그인 │ refresh │ 실패한 SQL │
├──────────────────────┼────────┼─────────┼──────────────────────────────┤
│ 완전 냉시동 │ 400 │ 400 │ select ce1_0.ID from CLIENT │
│ (재시작 직후) │ │ │ where CLIENT_ID=? REALM_ID=?│
├──────────────────────┼────────┼─────────┼──────────────────────────────┤
│ CLIENT 만 더움 │ 200 │ ★ 500 │ select cscme1_0.SCOPE_ID │
│ ← A-7 이 본 상태 │ │ │ from CLIENT_SCOPE_CLIENT │
├──────────────────────┼────────┼─────────┼──────────────────────────────┤
│ 완전히 더움 │ 200 │ 200 │ 없음 (SQL 0건) │
└──────────────────────┴────────┴─────────┴──────────────────────────────┘
[재현 A] 완전 냉시동 — 로그인부터 400
keycloak 재시작 → postgres 정지 → 로그인
로그인 400 {"error":"unauthorized_client",
"error_description":"Unexpected error when authenticating client"}
ERROR [org.keycloak.services] KC-SERVICES0015: Unexpected error when
authenticating client: org.hibernate.exception.GenericJDBCException:
JDBC exception executing SQL [FATAL: terminating connection due to
administrator command]
[select ce1_0.ID from CLIENT ce1_0 where ce1_0.CLIENT_ID=? and ce1_0.REALM_ID=?]
→ A-7 은 "volatile 이면 DB 없이 로그인된다"고 적었지만,
**냉시동에서는 그것도 안 된다.**
[재현 B] ★ A-7 이 본 그 조건 — 로그인 200, refresh 500
keycloak 재시작 → (DB 살아있을 때) 로그인 1회 → postgres 정지 → refresh
로그인 200
refresh 500 {"error":"unknown_error"}
JDBC exception executing SQL [FATAL: terminating connection due to
administrator command]
[select cscme1_0.SCOPE_ID from CLIENT_SCOPE_CLIENT cscme1_0
where cscme1_0.CLIENT_ID=? and cscme1_0.DEFAULT_SCOPE=?]
★ 로그가 실패한 SQL 을 직접 지목한다. CLIENT_SCOPE_CLIENT 다.
[재현 C] 완전히 더움 — 둘 다 200
refresh 를 3회 미리 돌려 캐시를 채운 뒤 postgres 정지
로그인 200
refresh 200 ← A-7 의 표와 정반대다
════ 결론 ════
1. **원인은 REVOKED_TOKEN 이 아니라 선택적 클라이언트 스코프 조회다.**
`CLIENT_SCOPE_CLIENT` 를 `DEFAULT_SCOPE='f'` 로 조회하는 그 한 문장이다.
2. **A-7 의 "volatile → refresh 500" 은 캐시가 차가울 때만 참이다.**
더워지면 200 이 된다. A-7 은 그 조건을 몰랐고, 그래서 표에 적지 않았다.
3. **"volatile 이면 DB 없이 로그인된다" 도 조건부다.**
냉시동에서는 400 이다. 클라이언트 조회조차 캐시에 없기 때문이다.
4. **일반화** — volatile 모드에서 DB 정지 시의 동작은 "무엇을 하느냐"가
아니라 **"그 경로가 이미 캐시를 채웠느냐"** 로 결정된다. 그래서 같은
명령이 재시작 직후와 얼마 쓴 뒤에 다른 답을 낸다. 이런 종류의 결과는
**한 번 재보고 표로 적으면 안 된다** — A-7 이 그렇게 했다.
@@ -0,0 +1,15 @@
# A-7a — volatile refresh 500 의 원인 확정 증거
2026-09-04 11:18 11:24 UTC
해설: [`docs/experiment-a7a-volatile-cause.md`](../../experiment-a7a-volatile-cause.md)
| 파일 | 무엇을 보여주는가 |
|---|---|
| `01-cause-determined.txt` | **가설이 틀렸다.** `REVOKED_TOKEN` 이 아니라 `CLIENT_SCOPE_CLIENT` 조회다. 문장 로깅으로 잡고, 실패 로그가 그 SQL 을 직접 지목한다. 그리고 **같은 설정에서 캐시 온도만으로 400/500/200 셋이 나온다** |
## 핵심 네 줄
1. **원인은 선택적 클라이언트 스코프 조회다.** `select cscme1_0.SCOPE_ID from CLIENT_SCOPE_CLIENT where CLIENT_ID=? and DEFAULT_SCOPE=?` — A-7 이 지목한 `REVOKED_TOKEN` 은 로그에 한 번도 안 나온다.
2. **로그인은 SQL 을 0개 쏜다.** refresh 만 이 한 문장을 쏘고, 그것도 **첫 번째만** 쏜다. 이후로는 캐시된다.
3. **A-7 의 표가 조건부였다.** 완전 냉시동이면 로그인부터 400, 캐시가 완전히 더우면 refresh 도 200. A-7 이 본 「로그인 200 · refresh 500」은 그 사이의 한 상태다 — 셋 다 재현했다.
4. **교훈** — volatile + DB 정지의 동작은 "무엇을 하느냐"가 아니라 **"그 경로가 캐시를 채웠느냐"** 로 갈린다. 한 번 재보고 표로 적으면 안 되는 종류의 측정이다.
@@ -16,3 +16,10 @@
1. **세션은 옮겨졌고 토큰은 안 옮겨졌다.** 빈 81개가 늘었는데 authorized client 관련은 하나도 안 바뀌었다. 1. **세션은 옮겨졌고 토큰은 안 옮겨졌다.** 빈 81개가 늘었는데 authorized client 관련은 하나도 안 바뀌었다.
2. **refresh token 은 Redis 에 평문으로 있는 게 아니라 아예 없다.** 암호화를 고민하기 전에 이걸 알아야 한다. 2. **refresh token 은 Redis 에 평문으로 있는 게 아니라 아예 없다.** 암호화를 고민하기 전에 이걸 알아야 한다.
3. **"로그인은 되어 있는데 아무것도 못 하는" 상태가 만들어진다** — 완전 로그아웃보다 나쁘다. 3. **"로그인은 되어 있는데 아무것도 못 하는" 상태가 만들어진다** — 완전 로그아웃보다 나쁘다.
## 스크린샷 주의
`b1-login-works-two-replicas.png``b1-token-boundary-after-redis.png`
**동일 파일**이며 `b2-before-relogin.png` 와도 같다 (md5 `6de826a7…`).
세 시점 모두 `accessTokenStoredOnServer: false` 인 같은 화면이었다.
**시점 구별은 터미널 출력과 Redis/DB 조회가 한다.**
@@ -19,3 +19,14 @@
2. **refresh token 은 평문이다.** DB 읽기 권한이면 작동하는 토큰을 얻는다. 2. **refresh token 은 평문이다.** DB 읽기 권한이면 작동하는 토큰을 얻는다.
3. **같은 사용자의 두 번째 로그인이 첫 번째를 덮어쓴다.** 기본키에 session id 가 없어 구조적으로 그렇다. 3. **같은 사용자의 두 번째 로그인이 첫 번째를 덮어쓴다.** 기본키에 session id 가 없어 구조적으로 그렇다.
4. **로그아웃은 셋 중 하나만 지운다.** 평문 토큰과 Keycloak SSO 세션이 남는다. 4. **로그아웃은 셋 중 하나만 지운다.** 평문 토큰과 Keycloak SSO 세션이 남는다.
## 스크린샷 주의
`b2-tokens-shared-across-instances.png` 는 **B-0 의
`b0-bff-token-boundary.png` 와 동일 파일**이다 (md5 `9ed00537…`).
두 시점 모두 `accessTokenStoredOnServer: true` 인 같은 화면이라 바이트가 같다.
**그래서 이 png 는 "JDBC 전환으로 토큰이 공유된다" 를 단독으로 증명하지
못한다.** 그 증명은 `01-jdbc-store-deploy.txt`(테이블 생성)과
`03-plaintext-tokens.txt`(행에 토큰이 들어 있음)가 한다.
`b2-before-relogin.png` 는 B-1 의 캡처와 동일 파일이다.
@@ -0,0 +1,147 @@
B-7a — cookie secret 회전이 남기는 고아 세션의 수명과 정리
=============================================================
수집: 2026-09-04 11:29 ~ 11:34 UTC · Redis + oauth2-proxy 로그 + Playwright
B-7 이 남긴 것
--------------
| 관찰 | |
|---|---|
| 옛 쿠키 | 검증 실패 — session ticket cookie failed validation |
| 사용자 경험 | Keycloak SSO 가 살아 있어 조용히 재로그인 |
| **서버 쪽 세션** | **★ 지우지 못했다** |
`Error removing session: error decoding ticket to clear session`
→ **티켓을 못 푸니 Redis 키를 계산할 수 없고, 그래서 지울 수도 없다.**
B-7 은 여기서 멈췄다. 남은 물음 셋을 잰다.
(1) 고아의 TTL 은 실제로 줄어드는가 — 정말 사라지긴 하는가
(2) 운영자가 직접 지울 수 있는가 · 지우면 산 세션이 다치는가
(3) ★ 어느 키가 고아인지 구분할 수 있는가
[기준선] 회전 전 — 11:29:42 UTC
------------------------------------
secret = COOKIE_SECRET_A
_oauth2_proxy-f6a9201fd534a047998278452001ccbf
type=string ttl=3568초 크기=3510바이트
dbsize=1
설정: --cookie-expire=1h --session-store-type=redis
기동 로그: Cookie settings: name:_oauth2_proxy secure(https):true
httponly:true expiry:1h0m0s ... refresh:disabled
[주입] 1차 회전 A → B — 11:29:56 UTC
--------------------------------------
kubectl patch deployment oauth2-proxy ... COOKIE_SECRET_B
회전 직후 Redis: 키 그대로 1개 (회전만으로는 아무 일도 안 일어난다)
브라우저가 접근한 순간(11:30:27) 로그:
[stored_session.go:94] Error loading cookied session:
session ticket cookie failed validation: <nil>, removing session
[stored_session.go:97] Error removing session:
error decoding ticket to clear session: session ticket cookie failed validation
[oauthproxy.go:1024] No valid authentication in request. Initiating login.
[AuthSuccess] Authenticated via OAuth2: Session{email:labuser@example.com ...}
→ B-7 의 관찰 그대로 재현. Keycloak SSO 가 살아 있어 로그인 화면 없이 통과했다.
Redis:
_oauth2_proxy-87faa1c94db3bd72c11c4e100c3ca593 ttl=3588 ← 새 세션
_oauth2_proxy-f6a9201fd534a047998278452001ccbf ttl=3511 ← ★ 고아
dbsize=2
[측정 1] ★ Redis 만 보고는 구분할 수 없다
-------------------------------------------
키 type strlen ttl
_oauth2_proxy-87faa1c9…(새) string 3510 3558
_oauth2_proxy-f6a9201f…(고아) string 3510 3480
· 이름 접두사가 같다 (_oauth2_proxy-)
· 뒤는 불투명한 32자 hex — 사용자·시각·상태 어느 것도 안 담긴다
· 타입이 같다, 크기가 **바이트 단위로 같다** (3510)
· 값은 암호화되어 있다
새 "\xcb\xb3h\xfa\x98\xedc\xe4@<\x9b\x83\xce\xc1\x18<…"
고아 "N\xf5\x0e=\xe1N\xfc|\xa2qE\xde\x1b\x82k\x88\x05…"
md5 f9ad43cc6bbb2db4 / 9b31f7c4138e6472 (다르지만 뜻을 읽을 수 없다)
→ **다른 것은 TTL 뿐이다.**
[측정 2] TTL 은 정직하게 줄어든다 — 그리고 갱신되지 않는다
-----------------------------------------------------------
30초 간격 3회:
t+00초 새=3557 고아=3479
t+30초 새=3526 고아=3448
t+60초 새=3494 고아=3417
1초에 1초씩. 고아는 **생성 후 정확히 1시간에 사라진다.**
요청을 보내도 늘지 않는다 (11:32:26, 11:32:49 두 번 요청 후):
살아있는 세션 ttl=3464 ← 계속 줄어든다
기동 로그의 `refresh:disabled` 와 일치한다. `--cookie-refresh` 가 없기 때문이다.
★ 이것이 다음 측정의 열쇠가 된다 — TTL 이 갱신되지 않으므로
TTL 은 **생성 시각의 정확한 함수**다.
[측정 3] 운영자는 지울 수 있다 — 산 세션은 다치지 않는다
----------------------------------------------------------
redis-cli del _oauth2_proxy-f6a9201f… → 반환 1
dbsize 2 → 1
남은 키: _oauth2_proxy-87faa1c9…
삭제 직후 브라우저 요청 (11:32:49):
app2.hyeonworks.com GET - "/oauth2/userinfo" ... labuser@example.com 200 108
→ **200. 산 세션은 영향이 없다.**
oauth2-proxy 는 못 지우지만 **운영자는 지울 수 있다.**
[측정 4] ★ 누적한다 — 회전할 때마다
-------------------------------------
2차 회전 B → A — 11:33:27 UTC. 브라우저 재접근 후:
키 TTL 생성시각(추정) 판정
_oauth2_proxy-dad9c9fb… 3581 11:33:54 살아있음
_oauth2_proxy-87faa1c9… 3373 11:30:26 ★ 고아
dbsize=2
**1차 회전에서 살아남았던 세션이 2차 회전에서 고아가 됐다.**
회전 1회 = 그 시점 로그인 사용자 수만큼의 고아.
[측정 5] ★ 그래서 정리 규칙을 유도할 수 있다
----------------------------------------------
TTL 이 갱신되지 않으므로(측정 2):
생성시각 = 지금 - (cookie-expire - TTL)
이 값이 **회전 시각보다 이르면 그 키는 고아다.** 회전 이후에 만들어진
세션은 새 secret 으로 만들어졌으므로 반드시 유효하기 때문이다.
검증 — 추정 생성시각 11:30:26 vs 로그의 AuthSuccess 11:30:27.
**1초 오차.** 추정이 아니라 사실상 정확하다.
실행:
NOW=$(date -u +%s); ROT=<회전 시각 epoch>
redis-cli --scan --pattern '_oauth2_proxy-*' | while read K; do
T=$(redis-cli ttl "$K")
C=$(( NOW - (3600 - T) ))
[ $C -lt $ROT ] && redis-cli del "$K"
done
실제 실행 결과: `삭제: _oauth2_proxy-87faa1c9…` · 남은 dbsize=1
산 세션은 남고 고아만 사라졌다.
════ 결론 ════
1. **"지울 수 없다"는 oauth2-proxy 의 한계이지 Redis 의 한계가 아니다.**
프록시는 티켓을 못 풀어 키를 계산할 수 없다. 운영자는 키를 직접 안다.
2. **고아는 반드시 사라진다 — 생성 후 1시간.** TTL 이 갱신되지 않기 때문에
"쓰고 있으면 안 지워진다" 같은 일이 없다. 다만 그 1시간 동안은 남는다.
3. **어느 것이 고아인지는 Redis 값으로 알 수 없다.** 이름·타입·크기가
같고 값은 암호화되어 있다. **TTL 만이 신호다.**
4. **그 TTL 로 정리 규칙이 유도된다.** 회전 시각 이전에 생성된 키는 전부
고아다. 1초 오차로 정확히 골라낼 수 있고, 실제로 골라내 지웠다.
5. **전제가 하나 있다 — `--cookie-refresh` 를 켜면 이 규칙이 깨진다.**
TTL 이 갱신되면 생성 시각을 역산할 수 없기 때문이다. 그때는 회전 후
`FLUSHDB` 로 전부 지우고 모두 재인증시키는 편이 오히려 정직하다.
@@ -0,0 +1,17 @@
# B-7a — 고아 세션의 수명과 정리 증거
2026-09-04 11:29 11:34 UTC
해설: [`docs/experiment-b7a-orphan-session.md`](../../experiment-b7a-orphan-session.md)
| 파일 | 무엇을 보여주는가 |
|---|---|
| `01-orphan-lifecycle.txt` | 회전 2회로 고아가 **누적**하는 것 · TTL 이 1초/초로 줄고 **요청으로 갱신되지 않는 것** · Redis 만으로는 **구분 불가**(이름·타입·크기 동일, 값 암호화) · `redis-cli del` 로 지워도 산 세션은 **200** · TTL 역산 정리 규칙이 **1초 오차**로 맞는 것 |
| `b7a-live-session-after-orphan-delete.png` | 고아를 지운 직후 살아있는 세션이 `/oauth2/userinfo` 를 정상 응답하는 브라우저 화면 |
## 핵심 다섯 줄
1. **「지울 수 없다」는 oauth2-proxy 의 한계이지 Redis 의 한계가 아니다.** 프록시는 티켓을 못 풀어 키를 계산 못 한다. 운영자는 키를 직접 안다 — `del` 반환 1, dbsize 2→1, 산 세션은 그대로 200.
2. **고아는 반드시 사라진다 — 생성 후 정확히 1시간.** TTL 이 요청으로 갱신되지 않기 때문이다(`refresh:disabled`). 다만 그 1시간은 남는다.
3. **회전할 때마다 누적한다.** 1차 회전을 살아남은 세션이 2차 회전에서 고아가 됐다. 회전 1회 = 그 시점 로그인 사용자 수만큼.
4. **Redis 값으로는 고아를 못 고른다.** 이름 접두사·타입·크기(3510바이트)가 같고 값은 암호화되어 있다. **TTL 만이 신호다.**
5. **그 TTL 로 정리 규칙이 유도된다.** `생성시각 = 지금 (cookie-expire TTL)` 이 회전 시각보다 이르면 고아다. 추정 11:30:26 대 로그 11:30:27 — **1초 오차**. 실제로 골라 지웠고 산 세션만 남았다.
Binary file not shown.

After

Width:  |  Height:  |  Size: 18 KiB

+8
View File
@@ -17,3 +17,11 @@
1. **SSO 는 user session 1개에 client session N개** 구조다 — A-3(전체 소실)과 B-3(client 만 제거)의 차이가 여기서 의미를 갖는다. 1. **SSO 는 user session 1개에 client session N개** 구조다 — A-3(전체 소실)과 B-3(client 만 제거)의 차이가 여기서 의미를 갖는다.
2. **IdP 세션을 죽여도 두 앱은 계속 동작한다.** 세 층(IdP·앱·토큰)의 수명이 각자이기 때문이다. 2. **IdP 세션을 죽여도 두 앱은 계속 동작한다.** 세 층(IdP·앱·토큰)의 수명이 각자이기 때문이다.
3. **IdP 는 "로그인 경로"의 단일 장애점이지 "이미 로그인한 사용자"의 단일 장애점이 아니다.** 장애는 앱 세션 수명만큼 지연되어 몰려온다. 3. **IdP 는 "로그인 경로"의 단일 장애점이지 "이미 로그인한 사용자"의 단일 장애점이 아니다.** 장애는 앱 세션 수명만큼 지연되어 몰려온다.
## 스크린샷 주의
`c1-sso-app2-no-login-screen.png``c1-apps-alive-after-idp-logout.png`
**바이트 단위로 동일한 파일**이다 (md5 `2c703176…`). 두 시점의 화면이 실제로
같은 내용이었기 때문이며, 조작이 아니다. **다만 그래서 두 시점을 구별하는
증거가 되지 못한다** — 구별은 `03-``04-` 의 터미널 출력(client_sessions
1→2, 그리고 IdP 세션 삭제 후 Redis 키 잔존)이 한다.
+2 -1
View File
@@ -7,10 +7,11 @@
|---|---| |---|---|
| `01-pre-upgrade.txt` | 백업 396KB · 이미지 26.7.0 · **마이그레이션 210건** · 세션 4 | | `01-pre-upgrade.txt` | 백업 396KB · 이미지 26.7.0 · **마이그레이션 210건** · 세션 4 |
| `02-rollback-attempt.txt` | 26.0 으로 내리자 `Running(0/1) → Error → CrashLoopBackOff`. **`liquibase.exception.ValidationFailedException`** | | `02-rollback-attempt.txt` | 26.0 으로 내리자 `Running(0/1) → Error → CrashLoopBackOff`. **`liquibase.exception.ValidationFailedException`** |
| `d2-upgrade-window.png` | Grafana — 26.7.3 업그레이드 구간의 `cluster_size` 2→1→2 두 번과 파드별 `up` 시계열 교체 (후속 작업에서 촬영) |
| `03-roll-forward.txt` | **서비스는 `HTTP 200` 유지**(ready 주소 1개) · 오류 원인 `1 changesets check sum` · 26.7.0 복귀 후 마이그레이션 210·세션 4 그대로 | | `03-roll-forward.txt` | **서비스는 `HTTP 200` 유지**(ready 주소 1개) · 오류 원인 `1 changesets check sum` · 26.7.0 복귀 후 마이그레이션 210·세션 4 그대로 |
## 핵심 세 줄 ## 핵심 세 줄
1. **롤백은 안 된다.** 체크섬이 안 맞아 Liquibase 가 기동 자체를 거부한다 — "모르는 변경"이 아니라 "아는 변경인데 정의가 다르다". 1. **스키마가 바뀌었으면 롤백은 안 된다.** (26.7.0↔26.7.3 처럼 안 바뀌면 된다 — [`followup`](../followup/) 참조.) 체크섬이 안 맞아 Liquibase 가 기동 자체를 거부한다 — "모르는 변경"이 아니라 "아는 변경인데 정의가 다르다".
2. **StatefulSet 이 사고를 절반에서 멈춰줬다.** 한 파드가 남아 외부 200 을 유지했다. replica 1 이었다면 전면 장애다. 2. **StatefulSet 이 사고를 절반에서 멈춰줬다.** 한 파드가 남아 외부 200 을 유지했다. replica 1 이었다면 전면 장애다.
3. **실패한 기동은 스키마를 안 건드렸다.** 그래서 이미지만 되돌려도 복구됐다 — 이미 적용된 뒤였다면 DB 복구(D-1)가 필요하다. 3. **실패한 기동은 스키마를 안 건드렸다.** 그래서 이미지만 되돌려도 복구됐다 — 이미 적용된 뒤였다면 DB 복구(D-1)가 필요하다.
Binary file not shown.

After

Width:  |  Height:  |  Size: 105 KiB

@@ -0,0 +1,17 @@
=== 실험대의 Secret 목록 ===
bff-secrets Opaque keys=1
keycloak-lab-secrets Opaque keys=2
oauth2-proxy-secrets Opaque keys=3
=== ★ base64 는 암호화가 아니다 — 한 줄로 읽힌다 ===
keycloak-lab-secrets/POSTGRES_PASSWORD = lab-postgres-change-me
keycloak-lab-secrets/KC_BOOTSTRAP_ADMIN_PASSWORD = lab-admin-change-me
bff-secrets/KEYCLOAK_CLIENT_SECRET = bff-lab-secret
oauth2-proxy-secrets/COOKIE_SECRET_A = lab-cookie-secret-aaaaaaaaaaaaaa
=== describe 는 값을 감춘다 (그래서 안전하다고 착각한다) ===
Type: Opaque
Data
====
KEYCLOAK_CLIENT_SECRET: 14 bytes
@@ -0,0 +1,23 @@
=== k3s 의 데이터 저장소 ===
Encryption Status: Disabled, no configuration file found
=== 저장 파일 ===
total 23336
drwx------ 2 root root 4096 Sep 2 09:12 .
drwx------ 8 root root 4096 Sep 4 03:23 ..
-rw-r--r-- 1 root root 13078528 Sep 4 06:05 state.db
-rw-r--r-- 1 root root 32768 Sep 4 06:06 state.db-shm
-rw-r--r-- 1 root root 10769712 Sep 4 06:06 state.db-wal
=== ★ 저장 파일에서 비밀번호가 그대로 보이는가 ===
state.db 안의 평문 일치: 2
=== 평문이 저장 파일에 있다는 것을 눈으로 ===
client secret 평문 등장 횟수: 0
=== 누가 Secret 을 읽을 수 있는가 ===
default SA: no
(Role 이 없으면 네임스페이스에 별도 제한이 없다는 뜻)
=== 파드 안에서는 어떻게 보이는가 ===
KEYCLOAK_CLIENT_SECRET=bff-lab-secret
BFF_DB_PASSWORD=lab-postgres-change-me
@@ -0,0 +1,15 @@
# D-3 — 비밀 관리 증거
2026-09-04 17:1517:25 KST
해설: [`docs/experiment-d3-secret-management.md`](../../experiment-d3-secret-management.md)
| 파일 | 무엇을 보여주는가 |
|---|---|
| `01-base64-not-encryption.txt` | 실험대의 **모든 비밀이 명령 네 줄로** 평문 출력. `describe``14 bytes` 만 보여줘 착각을 준다 |
| `02-at-rest.txt` | **`Encryption Status: Disabled`** · `state.db` 안에 비밀번호 평문 **2회 일치** · 파드 안에서는 `KEYCLOAK_CLIENT_SECRET=bff-lab-secret` 환경변수 · `default` SA 는 **읽을 수 없음** |
## 핵심 세 줄
1. **base64 는 감추려는 것이 아니라 YAML 에 바이트를 담기 위한 것이다.** `describe` 가 값을 가려 안전하다는 착각을 준다.
2. **저장소 암호화가 꺼져 있고 노드 디스크에 평문이 있다.** 노드 디스크 하나가 전 클러스터의 비밀이다.
3. **네 경로 중 RBAC 만 제 역할을 한다.** 그것이 실질적 방어선이며, 관리자에게는 아무 방어가 없다.
@@ -0,0 +1,38 @@
=== 현재 인증서 (외부 관측, sudo 불필요) ===
subject=CN = auth.hyeonworks.com
issuer=C = US, O = Let's Encrypt, CN = YE2
notBefore=Sep 3 00:47:23 2026 GMT
notAfter=Dec 2 00:47:22 2026 GMT
X509v3 Subject Alternative Name:
DNS:app1.hyeonworks.com, DNS:app2.hyeonworks.com, DNS:auth.hyeonworks.com
→ 세 호스트가 같은 인증서를 쓴다 (SAN 3개, 와일드카드 아님)
=== 체인 완결성 (fullchain vs cert 실수 확인) ===
0 s:CN = auth.hyeonworks.com
1 s:C = US, O = Let's Encrypt, CN = YE2
2 s:C = US, O = ISRG, CN = Root YE
3 s:C = US, O = Internet Security Research Group, CN = ISRG Root X2
Verify return code: 0 (ok)
→ 중간 인증서가 함께 제공된다. fullchain.pem 이 올바로 설정되어 있다.
=== 갱신 자동화 ===
NEXT LEFT LAST PASSED UNIT
Fri 2026-09-04 17:03:46 KST 1h 54min Fri 2026-09-04 03:19:39 KST 11h ago certbot-renew.timer
타이머 enabled: enabled
타이머 active: active
=== 남은 기간 ===
만료: Dec 2 00:47:22 2026 GMT
남은 일수: 88일
Let's Encrypt 90일 발급 · 30일 남으면 갱신 → 실제 갱신까지 약 58일
=== 강제 갱신은 하지 못했다 ===
$ sudo -n -l
sudo: a password is required
$ sudo -n systemctl reload nginx
sudo: a password is required
→ test-server 의 sudo 는 비밀번호를 요구한다 (게스트 kc-lab-1/2 는 무암호).
certbot renew --force-renewal 도 nginx reload 도 실행할 수 없다.
@@ -0,0 +1,19 @@
D-4 대조군 — 주입 없는 상태의 가용성 잡음 바닥
수집 시각(dev): 2026-09-04T08:03:07Z UTC
대상: https://auth.hyeonworks.com/realms/master · 0.2초 간격 900회 = 180초
형식: http_code time_total time_appconnect(TLS 핸드셰이크까지)
왜: 갱신 중 000 이 한 번 나와도, 평시 오류율을 모르면 그게 갱신 탓인지 알 수 없다.
----------------------------------------------------------------
표본 900 개
[상태코드 분포]
900 200
[응답시간 ms]
최소 67 중앙 98 p95 195 최대 1121 평균 106.9
[TLS 핸드셰이크 ms — 0 이면 연결 재사용, >0 이면 새 핸드셰이크]
핸드셰이크 발생 900회 / 900 평균 83 ms 최대 1100 ms
[비정상 응답 원문 — 있으면 아래에 전부]
비200 총 0
@@ -0,0 +1,22 @@
D-4 대조군 2 — '진행 중이던 요청' 측정 장치의 무주입 동작
수집 시각(dev): 2026-09-04T08:08:45Z UTC
왜 이 장치가 따로 필요한가
05-control 의 0.2초 폴링은 매 요청이 새 TCP 연결이다(핸드셰이크 900/900).
그래서 '새 연결을 받아주는가'는 재지만, D-4 가 묻는 '이미 진행 중이던
요청이 어떻게 되는가'는 재지 못한다. 재하려면 reload 순간에 실제로
전송 중인 요청이 있어야 한다.
장치
845KB 짜리 관리 콘솔 번들을 --limit-rate 20k 로 내려받는다.
응답을 일부러 느리게 읽어 요청을 ~42초 동안 살아 있게 만든다.
대상: https://auth.hyeonworks.com/resources/55yjq/admin/keycloak.v2/assets/main-BbID33M6.js
----------------------------------------------------------------
[대조군: 주입 없이 1회]
코드=200 받은바이트=845361 총시간=41.392198s 연결수=1 실효속도=20423B/s
기대 크기 845361 / 실제 845361 bytes
판정 기준 (주입 시 이 값들과 비교한다)
· 코드 200 + 크기 845361 = 진행 중이던 요청이 끝까지 살아남았다(graceful)
· 코드 000 또는 크기 부족 = reload 가 진행 중이던 연결을 끊었다
· 연결수 2 이상 = 중간에 끊겨 curl 이 다시 붙었다
@@ -0,0 +1,112 @@
D-4 — 갱신은 되는데 nginx 가 집지 않는다: 배포 훅 부재
=========================================================
수집: 2026-09-04 17:2x KST · test-server (sudo 없이 읽을 수 있는 범위)
발단
----
사용자가 `sudo certbot renew --force-renewal` 을 실행했다고 알려왔다.
그런데 밖에서 본 인증서가 바뀌지 않았다.
serial=0520BB6416D569E26697B1691440F523B853
notBefore=Sep 3 00:47:23 2026 GMT ← 어제 것 그대로
notAfter=Dec 2 00:47:22 2026 GMT
일련번호 감시 161표본(약 13분) 동안 단 한 번도 바뀌지 않았다.
[증거 1] nginx 는 reload 된 적이 없다
--------------------------------------
$ ps -eo pid,ppid,etimes,lstart,args | grep nginx
585 1 80529 Thu Sep 3 19:00:39 2026 nginx: master process /usr/bin/nginx
586 585 80529 Thu Sep 3 19:00:39 2026 nginx: worker process
읽는 법 — nginx 의 reload 는 마스터를 유지한 채 **워커만 새로 띄운다.**
· 마스터 585, 워커 586 = 마스터 기동 직후의 첫 fork
· 둘의 lstart 가 같고 etimes 도 같다(80529초 = 22.4시간)
→ reload 가 한 번도 없었다. 22.4시간 전 기동 그대로다.
인증서 파일이 새로 써졌더라도, nginx 는 기동 시점에 읽어 메모리에 들고 있다.
reload 가 없으면 **옛 인증서를 계속 서빙한다.**
[증거 2] ★ 갱신 유닛에 reload 가 없다
--------------------------------------
$ systemctl cat certbot-renew.service
# /usr/lib/systemd/system/certbot-renew.service
[Unit]
Description=Renew certificates acquired via Certbot
[Service]
Type=oneshot
ExecStart=/usr/bin/certbot -q renew
PrivateTmp=true
`ExecStart` 가 전부다. **ExecStartPost 도, --deploy-hook 도 없다.**
배포판(Arch)이 넣어준 기본 유닛이 그렇다.
$ systemctl cat certbot-renew.timer
OnCalendar=*-*-* 00/12:00:00
RandomizedDelaySec=12h
Persistent=true
[증거 3] 타이머는 정상 동작한다 — 그래서 더 위험하다
------------------------------------------------------
$ systemctl status certbot-renew.service
Active: inactive (dead) since Fri 2026-09-04 17:04:11 KST
Process: 28452 ExecStart=/usr/bin/certbot -q renew (code=exited, status=0/SUCCESS)
$ journalctl -u certbot-renew.service --since today
Sep 04 03:19:39 Starting Renew certificates acquired via Certbot...
Sep 04 03:19:41 Finished Renew certificates acquired via Certbot.
Sep 04 17:04:09 Starting Renew certificates acquired via Certbot...
Sep 04 17:04:11 Finished Renew certificates acquired via Certbot.
오늘 두 번 돌았고 두 번 다 status=0/SUCCESS 다. 만료까지 88일 남아 실제
갱신은 하지 않았으므로 **아직은** 아무 문제가 없다.
[증거 4] nginx 가 무엇을 물고 있나
-----------------------------------
$ grep -rn ssl_certificate /etc/nginx/
/etc/nginx/sites-available/keycloak-lab:18: ssl_certificate /etc/letsencrypt/live/auth.hyeonworks.com/fullchain.pem;
/etc/nginx/sites-available/keycloak-lab:19: ssl_certificate_key /etc/letsencrypt/live/auth.hyeonworks.com/privkey.pem;
`live/` 는 심볼릭 링크다. certbot 이 갱신하면 링크가 새 `archive/` 파일을
가리키도록 바뀐다 — **파일 경로는 그대로인데 내용이 바뀐다.** 그래서 nginx
설정은 고칠 필요가 없고, 바로 그 때문에 "설정이 그대로니 괜찮다"고 착각하기 쉽다.
필요한 것은 설정 변경이 아니라 **reload** 다.
무엇을 확인하지 못했나
----------------------
`/etc/letsencrypt/renewal-hooks/deploy/` 에 스크립트가 있으면 유닛 파일과
무관하게 실행된다. 이 디렉터리는 root 전용이라 읽지 못했다.
$ ls -laR /etc/letsencrypt/renewal-hooks/
ls: cannot access '/etc/letsencrypt/renewal-hooks/': Permission denied
따라서 두 갈래가 남는다.
(a) 강제 갱신이 실패했다 → 파일도 안 바뀌었고 reload 도 없었다
(b) 강제 갱신이 성공했다 → 파일은 바뀌었는데 훅이 없어 reload 가 안 됐다
(b) 라면 이 실험대에서 가장 값진 발견이다. **"갱신 성공"과 "새 인증서 서빙"이
다른 사건**이라는 것을, 88일 뒤가 아니라 지금 잡은 것이기 때문이다.
가리는 방법 — root 권한 한 번:
sudo certbot certificates # notAfter 가 오늘+90일이면 (b)
sudo ls -la /etc/letsencrypt/archive/auth.hyeonworks.com/
sudo tail -60 /var/log/letsencrypt/letsencrypt.log
곁다리로 확인한 것 — CT 로그로는 가릴 수 없다
-----------------------------------------------
발급 사실은 Certificate Transparency 에 남으므로 sudo 없이 확인할 수 있을 것
같았다. 실제로 서빙 중인 인증서에는 SCT 가 2개 박혀 있다.
CT Precertificate SCTs:
Log ID: C2:31:7E:57:...:52:CD Timestamp: Sep 3 01:45:53.183 2026 GMT
Log ID: 46:AF:86:3D:...:50:5F Timestamp: Sep 3 01:45:53.352 2026 GMT
그런데 crt.sh 는 이 이름을 모른다.
$ curl -s 'https://crt.sh/?q=auth.hyeonworks.com&output=json'
[] ← 0건
$ curl -s 'https://crt.sh/?q=hyeonworks.com&output=json'
13건, 최신 not_before=2026-08-11 ← auth 는 없다
**인증서에 SCT 가 박혀 있다는 것과 crt.sh 가 그것을 색인했다는 것은 다르다.**
관측 도구가 진실의 부분집합만 본다는, A-2 의 `up` 지표와 같은 종류의 함정이다.
@@ -0,0 +1,66 @@
D-4 — in-flight 감시에서 나온 76건 실패: 서버 탓이 아니다
============================================================
수집: 2026-09-04 · dev 머신에서 실행한 감시 로그
무엇이 보였나
-------------
42초짜리 in-flight 요청을 연달아 돌리는 감시에서, 91건 중 76건이 실패했다.
08:13:40 코드=200 바이트=845361 시간=41.041830 연결수=1
08:14:22 코드=200 바이트=845361 시간=42.338482 연결수=1
08:15:04 코드=000 바이트=0 시간=0.001148 연결수=0 ← 여기부터
08:15:04 코드=000 바이트=0 시간=0.001140 연결수=0
08:15:04 코드=000 바이트=0 시간=0.000055 연결수=0
... (76건, 전부 08:15:04)
08:15:04 코드=200 바이트=845361 시간=42.236496 연결수=1 ← 곧바로 복귀
08:15:47 코드=200 바이트=845361 시간=42.327129 연결수=1
이걸 "갱신 중 진행 중이던 요청이 끊겼다"로 읽으면 안 되는 이유
----------------------------------------------------------------
1. **같은 순간 폴링은 멀쩡했다.** 0.2초 간격 폴링의 08:15:0008:15:20 구간
표본 49건이 전부 200이다. 같은 호스트, 같은 TLS, 같은 머신이다.
08:15:00.24 200 0.092491
08:15:01.15 200 0.099177
08:15:02.38 200 0.078729
... 49건 / 비200 0건
2. **연결수=0 이다.** curl 이 TCP 연결을 시도조차 못 했다는 뜻이다.
서버가 연결을 끊었다면 연결수는 1이고 바이트는 0보다 크다.
3. **시간이 0.00005초다.** 50마이크로초. DNS 조회조차 이보다 오래 걸린다.
요청이 나가기 전에 로컬에서 실패했다.
4. **76건이 1초 안에 몰렸다.** 실패가 즉시 반환되니 while 루프가 폭주한 것이다.
시간축에 퍼진 장애가 아니라 순간의 스파이크다.
5. **재현되지 않는다.**
같은 URL 100회 연속 → 실패 0/100
DNS 해석 200회 → 실패 0/200
URL 유효성 → 코드=200 크기=845361, 콘솔이 참조하는 경로와 동일
6. **nginx 는 그 시각에 아무 일도 하지 않았다.** 마스터·워커가 22.4시간째
같은 프로세스다(07 참조). reload 도 재시작도 없었다.
결론
----
**로컬(dev 머신) 쪽 일시적 원인이며, 서버 가용성 사건이 아니다.**
정확한 원인은 특정하지 못했다 — curl 을 `-s` 로 돌려 오류 메시지를 버렸고,
종료 코드도 기록하지 않았기 때문이다.
고친 것
-------
감시를 종료 코드까지 남기는 형태로 교체했다. 다음에 같은 일이 생기면
curl 의 종료 코드(6=DNS, 7=connect, 35=TLS ...)로 원인이 바로 나온다.
실패 시 1초 쉬게 해 루프 폭주도 막았다.
R=$(curl -s --limit-rate 20k -o /dev/null -w "..." "$URL" 2>/dev/null); E=$?
echo "$T $R curl종료=$E"
[ $E -ne 0 ] && sleep 1
남기는 교훈
-----------
측정 장치가 실패했을 때 **왜 실패했는지 남기지 않으면, 그 실패를 대상 탓으로
돌릴지 장치 탓으로 돌릴지 판단할 근거가 없다.** 여기서는 대조 폴링이 같은
시각에 멀쩡했다는 사실 하나로 겨우 갈랐다. 대조군이 없었다면 이 76건은
"갱신 중 대규모 요청 실패"라는 그럴듯한 오보가 됐을 것이다.
@@ -0,0 +1,564 @@
08:10:51 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:10:56 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:11:01 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:11:06 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:11:12 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:11:17 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:11:22 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:11:27 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:11:32 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:11:37 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:11:42 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:11:47 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:11:52 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:11:58 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:12:03 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:12:08 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:12:13 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:12:18 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:12:23 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:12:28 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:12:33 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:12:38 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:12:44 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:12:49 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:12:54 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:12:59 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:13:04 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:13:09 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:13:14 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:13:19 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:13:25 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:13:30 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:13:35 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:13:40 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:13:45 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:13:50 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:13:55 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:14:00 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:14:05 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:14:10 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:14:15 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:14:21 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:14:26 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:14:31 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:14:36 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:14:41 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:14:46 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:14:51 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:14:56 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:15:01 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:15:07 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:15:12 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:15:17 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:15:22 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:15:27 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:15:32 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:15:37 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:15:42 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:15:48 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:15:53 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:15:58 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:16:03 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:16:08 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:16:13 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:16:19 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:16:24 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:16:29 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:16:34 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:16:39 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:16:44 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:16:50 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:16:55 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:17:00 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:17:05 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:17:10 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:17:15 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:17:20 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:17:25 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:17:31 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:17:36 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:17:41 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:17:46 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:17:51 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:17:56 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:18:01 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:18:06 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:18:12 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:18:17 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:18:22 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:18:27 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:18:32 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:18:37 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:18:42 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:18:48 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:18:53 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:18:58 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:19:03 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:19:08 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:19:13 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:19:18 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:19:23 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:19:29 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:19:34 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:19:39 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:19:44 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:19:49 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:19:54 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:19:59 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:20:05 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:20:10 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:20:15 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:20:20 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:20:25 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:20:30 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:20:35 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:20:40 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:20:45 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:20:51 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:20:56 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:21:01 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:21:06 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:21:11 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:21:16 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:21:21 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:21:26 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:21:31 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:21:37 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:21:42 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:21:47 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:21:52 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:21:57 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:22:02 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:22:07 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:22:12 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:22:17 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:22:22 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:22:28 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:22:33 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:22:38 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:22:43 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:22:48 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:22:53 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:22:58 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:23:03 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:23:08 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:23:13 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:23:19 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:23:24 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:23:29 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:23:34 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:23:39 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:23:44 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:23:49 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:23:54 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:24:00 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:24:05 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:24:10 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:24:15 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:24:20 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:24:25 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:24:30 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:24:35 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:24:40 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:24:46 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:24:51 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:24:56 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:25:01 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:25:06 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:25:11 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:25:16 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:25:21 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:25:26 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:25:31 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:25:37 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:25:42 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:25:47 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:25:52 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:25:57 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:26:02 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:26:07 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:26:12 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:26:17 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:26:23 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:26:28 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:26:33 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:26:38 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:26:43 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:26:48 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:26:53 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:26:58 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:27:03 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:27:08 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:27:14 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:27:19 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:27:24 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:27:29 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:27:34 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:27:39 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:27:44 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:27:49 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:27:54 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:28:00 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:28:05 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:28:10 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:28:15 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:28:20 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:28:25 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:28:30 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:28:35 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:28:40 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:28:46 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:28:51 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:28:56 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:29:01 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:29:06 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:29:11 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:29:16 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:29:21 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:29:26 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:29:31 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:29:36 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:29:42 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:29:47 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:29:52 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:29:57 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:30:02 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:30:07 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:30:12 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:30:17 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:30:22 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:30:28 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:30:33 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:30:38 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:30:43 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:30:48 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:30:53 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:30:58 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:31:03 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:31:08 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:31:13 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:31:19 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:31:24 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:31:29 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:31:34 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:31:39 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:31:44 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:31:49 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:31:54 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:31:59 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:32:05 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:32:10 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:32:15 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:32:20 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:32:25 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:32:30 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:32:35 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:32:40 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:32:46 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:32:51 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:32:56 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:33:01 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:33:06 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:33:11 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:33:16 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:33:21 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:33:26 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:33:32 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:33:37 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:33:42 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:33:47 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:33:52 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:33:57 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:34:02 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:34:07 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:34:13 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:34:18 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:34:23 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:34:28 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:34:33 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:34:38 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:34:43 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:34:48 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:34:54 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:34:59 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:35:04 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:35:09 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:35:14 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:35:19 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:35:24 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:35:29 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:35:35 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:35:40 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:35:45 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:35:50 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:35:56 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:36:01 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:36:06 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:36:11 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:36:16 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:36:22 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:36:27 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:36:32 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:36:37 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:36:42 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:36:47 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:36:52 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:36:57 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:37:03 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:37:08 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:37:13 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:37:18 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:37:23 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:37:28 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:37:33 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:37:38 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:37:43 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:37:48 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:37:54 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:37:59 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:38:04 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:38:09 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:38:14 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:38:19 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:38:24 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:38:29 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:38:35 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:38:40 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:38:45 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:38:50 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:38:55 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:39:00 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:39:05 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:39:10 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:39:15 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:39:21 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:39:26 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:39:31 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:39:36 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:39:41 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:39:46 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:39:51 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:39:56 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:40:01 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:40:07 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:40:12 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:40:17 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:40:22 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:40:27 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:40:32 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:40:38 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:40:43 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:40:48 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:40:53 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:40:58 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:41:03 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:41:08 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:41:14 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:41:19 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:41:24 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:41:29 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:41:35 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:41:40 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:41:45 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:41:50 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:41:55 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:42:00 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:42:05 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:42:10 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:42:16 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:42:21 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:42:26 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:42:31 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:42:36 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:42:41 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:42:46 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:42:51 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:42:56 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:43:01 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:43:07 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:43:12 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:43:17 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:43:22 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:43:27 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:43:32 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:43:37 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:43:42 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:43:48 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:43:53 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:43:58 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:44:03 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:44:08 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:44:14 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:44:19 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:44:24 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:44:29 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:44:34 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:44:39 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:44:45 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:44:50 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:44:55 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:45:00 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:45:06 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:45:11 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:45:16 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:45:21 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:45:26 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:45:31 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:45:36 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:45:41 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:45:47 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:45:52 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:45:57 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:46:02 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:46:07 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:46:12 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:46:17 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:46:23 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:46:28 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:46:33 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:46:38 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:46:43 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:46:49 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:46:54 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:46:59 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:47:04 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:47:09 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:47:15 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:47:20 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:47:25 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:47:30 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:47:35 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:47:40 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:47:45 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:47:50 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:47:56 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:48:01 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:48:06 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:48:11 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:48:16 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:48:21 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:48:26 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:48:31 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:48:37 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:48:42 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:48:47 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:48:52 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:48:57 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:49:03 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:49:08 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:49:13 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:49:18 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:49:23 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:49:28 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:49:33 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:49:39 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:49:44 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:49:49 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:49:54 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:49:59 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:50:04 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:50:10 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:50:15 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:50:20 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:50:25 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:50:30 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:50:35 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:50:40 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:50:46 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:50:51 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:50:56 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:51:01 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:51:06 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:51:11 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:51:16 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:51:21 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:51:27 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:51:32 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:51:37 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:51:42 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:51:47 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:51:52 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:51:57 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:52:02 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:52:08 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:52:13 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:52:18 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:52:23 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:52:28 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:52:33 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:52:38 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:52:43 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:52:48 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:52:54 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:52:59 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:53:04 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:53:09 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:53:14 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:53:19 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:53:24 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:53:29 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:53:35 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:53:40 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:53:45 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:53:50 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:53:55 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:54:00 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:54:05 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:54:10 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:54:16 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:54:21 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:54:26 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:54:31 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:54:36 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:54:41 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:54:46 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:54:51 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:54:57 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:55:02 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:55:07 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:55:12 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:55:17 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:55:22 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:55:27 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:55:32 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:55:37 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:55:43 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:55:48 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:55:53 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:55:58 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:56:03 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:56:08 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:56:13 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:56:19 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:56:24 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:56:29 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:56:34 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:56:39 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:56:44 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:56:49 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:56:54 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:56:59 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:57:05 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:57:10 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:57:15 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:57:20 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:57:25 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:57:30 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:57:35 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:57:40 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:57:46 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:57:51 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:57:56 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:58:01 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:58:06 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:58:11 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:58:16 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:58:22 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:58:27 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:58:32 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:58:37 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:58:42 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:58:47 serial=0520BB6416D569E26697B1691440F523B853 notAfter=Dec 2 00:47:22 2026 GMT
08:58:52 serial=06C7CB6DF1DA8A6D7995D93C264BB9ECEA1D notAfter=Dec 3 07:21:52 2026 GMT
08:58:52 ★ 일련번호 변경 감지 — 3분 더 재고 종료
@@ -0,0 +1,177 @@
08:10:51.60 200 0.104996
08:10:51.91 200 0.070124
08:10:52.19 200 0.131785
... (중략)
08:58:30.03 200 0.077607
08:58:30.32 200 0.109888
08:58:30.64 200 0.195213
08:58:31.06 200 0.103103
08:58:31.38 200 0.079529
08:58:31.67 200 0.187888
08:58:32.07 200 0.194786
08:58:32.48 200 0.192114
08:58:32.89 200 0.108582
08:58:33.22 200 0.079331
08:58:33.51 200 0.189853
08:58:33.92 200 0.189635
08:58:34.33 200 0.195752
08:58:34.74 200 0.195164
08:58:35.15 200 0.078307
08:58:35.45 200 0.102306
08:58:35.76 200 0.193694
08:58:36.18 200 0.193912
08:58:36.58 200 0.079270
08:58:36.88 200 0.100873
08:58:37.20 200 0.077209
08:58:37.49 200 0.102600
08:58:37.81 200 0.195354
08:58:38.22 200 0.194041
08:58:38.63 200 0.079332
08:58:38.93 200 0.077710
08:58:39.22 200 0.073483
08:58:39.51 200 0.126482
08:58:39.86 200 0.196844
08:58:40.27 200 0.189230
08:58:40.68 200 0.089921
08:58:40.98 200 0.080912
08:58:41.28 200 0.093971
08:58:41.60 200 0.191146
08:58:42.00 200 0.079236
08:58:42.30 200 0.105411
08:58:42.62 200 0.195868
08:58:43.03 200 0.147224
08:58:43.40 200 0.136670
08:58:43.75 200 0.191708
08:58:44.16 200 0.084091
08:58:44.46 200 0.079146
08:58:44.75 200 0.114239
08:58:45.08 200 0.078791
08:58:45.38 200 0.074542
08:58:45.67 200 0.078430
08:58:45.97 200 0.073271
08:58:46.26 200 0.081292
08:58:46.55 200 0.086169
08:58:46.86 200 0.078664
08:58:47.16 200 0.077433
08:58:47.45 200 0.076830
08:58:47.74 200 0.078261
08:58:48.04 200 0.082249
08:58:48.34 200 0.077844
08:58:48.64 200 0.084788
08:58:48.94 200 0.080198
08:58:49.24 200 0.078426
08:58:49.53 200 0.078641
08:58:49.82 200 0.080504
08:58:50.12 200 0.079685
08:58:50.42 200 0.079559
08:58:50.71 200 0.078883
08:58:51.01 200 0.077754
08:58:51.30 200 0.078970
08:58:51.60 200 0.084448
08:58:51.90 200 0.131753
08:58:52.25 200 0.197813
08:58:52.66 200 0.091437
08:58:52.97 200 0.199400
08:58:53.39 200 0.085250
08:58:53.69 200 0.084521
08:58:53.99 200 0.072331
08:58:54.28 200 0.081943
08:58:54.58 200 0.113594
08:58:54.92 200 0.185589
08:58:55.32 200 0.200308
08:58:55.74 200 0.098051
08:58:56.05 200 0.086943
08:58:56.36 200 0.192842
08:58:56.77 200 0.178977
08:58:57.17 200 0.083091
08:58:57.47 200 0.194195
08:58:57.88 200 0.147293
08:58:58.24 200 0.149510
08:58:58.61 200 0.088007
08:58:58.91 200 0.190298
08:58:59.32 200 0.199024
08:58:59.73 200 0.183439
08:59:00.13 200 0.206752
08:59:00.55 200 0.179475
08:59:00.95 200 0.089502
08:59:01.25 200 0.096302
08:59:01.57 200 0.203361
08:59:01.99 200 0.175288
08:59:02.38 200 0.200164
08:59:02.80 200 0.157565
08:59:03.17 200 0.081179
08:59:03.47 200 0.133562
08:59:03.82 200 0.207667
08:59:04.24 200 0.156695
08:59:04.61 200 0.126136
08:59:04.95 200 0.185645
08:59:05.36 200 0.197049
08:59:05.77 200 0.090621
08:59:06.08 200 0.182576
08:59:06.48 200 0.107587
08:59:06.80 200 0.176183
08:59:07.19 200 0.076604
08:59:07.48 200 0.112886
08:59:07.81 200 0.197961
08:59:08.23 200 0.197695
08:59:08.64 200 0.192117
08:59:09.05 200 0.086968
08:59:09.36 200 0.073667
08:59:09.64 200 0.107286
08:59:09.97 200 0.187141
08:59:10.37 200 0.196110
08:59:10.78 200 0.080634
08:59:11.08 200 0.101278
08:59:11.40 200 0.190454
08:59:11.81 200 0.195550
08:59:12.22 200 0.084865
08:59:12.52 200 0.097978
08:59:12.84 200 0.188273
08:59:13.24 200 0.078268
08:59:13.54 200 0.098529
08:59:13.85 200 0.196250
08:59:14.26 200 0.198583
08:59:14.67 200 0.193794
08:59:15.08 200 0.142313
08:59:15.44 200 0.142774
08:59:15.80 200 0.203481
08:59:16.22 200 0.190197
08:59:16.62 200 0.193118
08:59:17.03 200 0.191270
08:59:17.44 200 0.190572
08:59:17.85 200 0.159825
08:59:18.23 200 0.115453
08:59:18.56 200 0.212529
08:59:18.99 200 0.178715
08:59:19.38 200 0.373160
08:59:19.98 200 0.114451
08:59:20.31 200 0.198672
08:59:20.72 200 0.186258
08:59:21.12 200 0.103469
08:59:21.44 200 0.197932
08:59:21.86 200 0.077019
08:59:22.15 200 0.180685
08:59:22.55 200 0.203363
08:59:22.97 200 0.197513
08:59:23.38 200 0.196509
08:59:23.80 200 0.094951
08:59:24.11 200 0.175434
08:59:24.50 200 0.372545
08:59:25.10 200 0.077048
08:59:25.39 200 0.077420
08:59:25.69 200 0.138478
08:59:26.04 200 0.190167
08:59:26.45 200 0.192613
08:59:26.86 200 0.161352
08:59:27.24 200 0.081419
08:59:27.54 200 0.131994
08:59:27.88 200 0.202003
08:59:28.30 200 0.186137
08:59:28.71 200 0.091572
08:59:29.02 200 0.077290
08:59:29.31 200 0.090913
08:59:29.62 200 0.198752
... (중략)
09:01:10.29 200 0.105354
09:01:10.62 200 0.173389
09:01:11.00 200 0.191564
@@ -0,0 +1,146 @@
08:10:51 코드=200 바이트=845361 시간=42.335582 연결수=1
08:11:33 코드=200 바이트=845361 시간=42.328216 연결수=1
08:12:16 코드=200 바이트=845361 시간=42.332256 연결수=1
08:12:58 코드=200 바이트=845361 시간=42.327378 연결수=1
08:13:40 코드=200 바이트=845361 시간=41.041830 연결수=1
08:14:22 코드=200 바이트=845361 시간=42.338482 연결수=1
08:15:04 코드=000 바이트=0 시간=0.001148 연결수=0
08:15:04 코드=000 바이트=0 시간=0.001140 연결수=0
08:15:04 코드=000 바이트=0 시간=0.001124 연결수=0
08:15:04 코드=000 바이트=0 시간=0.000055 연결수=0
08:15:04 코드=000 바이트=0 시간=0.000080 연결수=0
08:15:04 코드=000 바이트=0 시간=0.000077 연결수=0
08:15:04 코드=000 바이트=0 시간=0.001167 연결수=0
08:15:04 코드=000 바이트=0 시간=0.000048 연결수=0
08:15:04 코드=000 바이트=0 시간=0.000079 연결수=0
08:15:04 코드=000 바이트=0 시간=0.000096 연결수=0
08:15:04 코드=000 바이트=0 시간=0.000039 연결수=0
08:15:04 코드=000 바이트=0 시간=0.000083 연결수=0
08:15:04 코드=000 바이트=0 시간=0.000075 연결수=0
08:15:04 코드=000 바이트=0 시간=0.000041 연결수=0
08:15:04 코드=000 바이트=0 시간=0.000061 연결수=0
08:15:04 코드=000 바이트=0 시간=0.001141 연결수=0
08:15:04 코드=000 바이트=0 시간=0.000078 연결수=0
08:15:04 코드=000 바이트=0 시간=0.000053 연결수=0
08:15:04 코드=000 바이트=0 시간=0.000079 연결수=0
08:15:04 코드=000 바이트=0 시간=0.000083 연결수=0
08:15:04 코드=000 바이트=0 시간=0.000079 연결수=0
08:15:04 코드=000 바이트=0 시간=0.000099 연결수=0
08:15:04 코드=000 바이트=0 시간=0.000083 연결수=0
08:15:04 코드=000 바이트=0 시간=0.001164 연결수=0
08:15:04 코드=000 바이트=0 시간=0.001106 연결수=0
08:15:04 코드=000 바이트=0 시간=0.000074 연결수=0
08:15:04 코드=000 바이트=0 시간=0.000087 연결수=0
08:15:04 코드=000 바이트=0 시간=0.000079 연결수=0
08:15:04 코드=000 바이트=0 시간=0.000072 연결수=0
08:15:04 코드=000 바이트=0 시간=0.000085 연결수=0
08:15:04 코드=000 바이트=0 시간=0.000085 연결수=0
08:15:04 코드=000 바이트=0 시간=0.000048 연결수=0
08:15:04 코드=000 바이트=0 시간=0.000046 연결수=0
08:15:04 코드=000 바이트=0 시간=0.001147 연결수=0
08:15:04 코드=000 바이트=0 시간=0.000044 연결수=0
08:15:04 코드=000 바이트=0 시간=0.000053 연결수=0
08:15:04 코드=000 바이트=0 시간=0.000076 연결수=0
08:15:04 코드=000 바이트=0 시간=0.000043 연결수=0
08:15:04 코드=000 바이트=0 시간=0.000091 연결수=0
08:15:04 코드=000 바이트=0 시간=0.000074 연결수=0
08:15:04 코드=000 바이트=0 시간=0.000049 연결수=0
08:15:04 코드=000 바이트=0 시간=0.000051 연결수=0
08:15:04 코드=000 바이트=0 시간=0.000087 연결수=0
08:15:04 코드=000 바이트=0 시간=0.000091 연결수=0
08:15:04 코드=000 바이트=0 시간=0.000079 연결수=0
08:15:04 코드=000 바이트=0 시간=0.000095 연결수=0
08:15:04 코드=000 바이트=0 시간=0.000055 연결수=0
08:15:04 코드=000 바이트=0 시간=0.000076 연결수=0
08:15:04 코드=000 바이트=0 시간=0.000077 연결수=0
08:15:04 코드=000 바이트=0 시간=0.000088 연결수=0
08:15:04 코드=000 바이트=0 시간=0.000085 연결수=0
08:15:04 코드=000 바이트=0 시간=0.000092 연결수=0
08:15:04 코드=000 바이트=0 시간=0.000080 연결수=0
08:15:04 코드=000 바이트=0 시간=0.000085 연결수=0
08:15:04 코드=000 바이트=0 시간=0.001157 연결수=0
08:15:04 코드=000 바이트=0 시간=0.000048 연결수=0
08:15:04 코드=000 바이트=0 시간=0.001097 연결수=0
08:15:04 코드=000 바이트=0 시간=0.000060 연결수=0
08:15:04 코드=000 바이트=0 시간=0.000080 연결수=0
08:15:04 코드=000 바이트=0 시간=0.000087 연결수=0
08:15:04 코드=000 바이트=0 시간=0.000048 연결수=0
08:15:04 코드=000 바이트=0 시간=0.000081 연결수=0
08:15:04 코드=000 바이트=0 시간=0.000084 연결수=0
08:15:04 코드=000 바이트=0 시간=0.000087 연결수=0
08:15:04 코드=000 바이트=0 시간=0.000079 연결수=0
08:15:04 코드=000 바이트=0 시간=0.001153 연결수=0
08:15:04 코드=000 바이트=0 시간=0.000053 연결수=0
08:15:04 코드=000 바이트=0 시간=0.000088 연결수=0
08:15:04 코드=000 바이트=0 시간=0.000052 연결수=0
08:15:04 코드=000 바이트=0 시간=0.000085 연결수=0
08:15:04 코드=000 바이트=0 시간=0.000084 연결수=0
08:15:04 코드=000 바이트=0 시간=0.000080 연결수=0
08:15:04 코드=000 바이트=0 시간=0.000079 연결수=0
08:15:04 코드=000 바이트=0 시간=0.001106 연결수=0
08:15:04 코드=000 바이트=0 시간=0.000051 연결수=0
08:15:04 코드=000 바이트=0 시간=0.000056 연결수=0
08:15:04 코드=200 바이트=845361 시간=42.236496 연결수=1
08:15:47 코드=200 바이트=845361 시간=42.327129 연결수=1
08:16:29 코드=200 바이트=845361 시간=41.442808 연결수=1
08:17:10 코드=200 바이트=845361 시간=42.332025 연결수=1
08:17:53 코드=200 바이트=845361 시간=40.434517 연결수=1
08:18:33 코드=200 바이트=845361 시간=41.438410 연결수=1
08:19:15 코드=200 바이트=845361 시간=42.334617 연결수=1
08:19:57 코드=200 바이트=845361 시간=42.333846 연결수=1
08:20:39 코드=200 바이트=845361 시간=42.232748 연결수=1
08:21:22 코드=200 바이트=845361 시간=42.324095 연결수=1
08:22:04 코드=200 바이트=845361 시간=42.331593 연결수=1
08:22:46 코드=200 바이트=845361 시간=41.626941 연결수=1
08:23:28 코드=200 바이트=845361 시간=42.330280 연결수=1
08:24:59 코드=200 바이트=845361 시간=42.331694 연결수=1 curl종료=0
08:25:42 코드=200 바이트=845361 시간=42.333930 연결수=1 curl종료=0
08:26:24 코드=200 바이트=845361 시간=41.839874 연결수=1 curl종료=0
08:27:06 코드=200 바이트=845361 시간=41.834401 연결수=1 curl종료=0
08:27:48 코드=200 바이트=845361 시간=41.840212 연결수=1 curl종료=0
08:28:30 코드=200 바이트=845361 시간=41.032736 연결수=1 curl종료=0
08:29:11 코드=200 바이트=845361 시간=41.433394 연결수=1 curl종료=0
08:29:52 코드=200 바이트=845361 시간=42.325976 연결수=1 curl종료=0
08:30:34 코드=200 바이트=845361 시간=42.037070 연결수=1 curl종료=0
08:31:16 코드=200 바이트=845361 시간=42.343058 연결수=1 curl종료=0
08:31:59 코드=200 바이트=845361 시간=41.837942 연결수=1 curl종료=0
08:32:41 코드=200 바이트=845361 시간=42.318577 연결수=1 curl종료=0
08:33:23 코드=200 바이트=845361 시간=42.336079 연결수=1 curl종료=0
08:34:05 코드=200 바이트=845361 시간=42.328392 연결수=1 curl종료=0
08:34:48 코드=200 바이트=845361 시간=41.839243 연결수=1 curl종료=0
08:35:30 코드=200 바이트=845361 시간=42.339658 연결수=1 curl종료=0
08:36:12 코드=200 바이트=845361 시간=42.330835 연결수=1 curl종료=0
08:36:54 코드=200 바이트=845361 시간=41.844755 연결수=1 curl종료=0
08:37:36 코드=200 바이트=845361 시간=41.442783 연결수=1 curl종료=0
08:38:18 코드=200 바이트=845361 시간=42.334462 연결수=1 curl종료=0
08:39:00 코드=200 바이트=845361 시간=42.331275 연결수=1 curl종료=0
08:39:42 코드=200 바이트=845361 시간=41.243768 연결수=1 curl종료=0
08:40:23 코드=200 바이트=845361 시간=41.837874 연결수=1 curl종료=0
08:41:05 코드=200 바이트=845361 시간=42.336582 연결수=1 curl종료=0
08:41:48 코드=200 바이트=845361 시간=42.327934 연결수=1 curl종료=0
08:42:30 코드=200 바이트=845361 시간=42.333559 연결수=1 curl종료=0
08:43:12 코드=200 바이트=845361 시간=41.841294 연결수=1 curl종료=0
08:43:54 코드=200 바이트=845361 시간=42.329699 연결수=1 curl종료=0
08:44:37 코드=200 바이트=845361 시간=41.845127 연결수=1 curl종료=0
08:45:18 코드=200 바이트=845361 시간=42.040360 연결수=1 curl종료=0
08:46:00 코드=200 바이트=845361 시간=41.845192 연결수=1 curl종료=0
08:46:42 코드=200 바이트=845361 시간=42.327749 연결수=1 curl종료=0
08:47:25 코드=200 바이트=845361 시간=41.837216 연결수=1 curl종료=0
08:48:06 코드=200 바이트=845361 시간=42.329675 연결수=1 curl종료=0
08:48:49 코드=200 바이트=845361 시간=42.036368 연결수=1 curl종료=0
08:49:31 코드=200 바이트=845361 시간=42.341275 연결수=1 curl종료=0
08:50:13 코드=200 바이트=845361 시간=42.332664 연결수=1 curl종료=0
08:50:56 코드=200 바이트=845361 시간=42.334345 연결수=1 curl종료=0
08:51:38 코드=200 바이트=845361 시간=41.837146 연결수=1 curl종료=0
08:52:20 코드=200 바이트=845361 시간=42.328092 연결수=1 curl종료=0
08:53:02 코드=200 바이트=845361 시간=42.344341 연결수=1 curl종료=0
08:53:44 코드=200 바이트=845361 시간=42.332296 연결수=1 curl종료=0
08:54:27 코드=200 바이트=845361 시간=41.839230 연결수=1 curl종료=0
08:55:09 코드=200 바이트=845361 시간=42.336469 연결수=1 curl종료=0
08:55:51 코드=200 바이트=845361 시간=42.336864 연결수=1 curl종료=0
08:56:33 코드=200 바이트=845361 시간=42.043518 연결수=1 curl종료=0
08:57:15 코드=200 바이트=845361 시간=42.340137 연결수=1 curl종료=0
08:57:58 코드=200 바이트=845361 시간=42.335696 연결수=1 curl종료=0
08:58:40 코드=200 바이트=845361 시간=40.840349 연결수=1 curl종료=0
08:59:21 코드=200 바이트=845361 시간=42.340130 연결수=1 curl종료=0
09:00:03 코드=200 바이트=845361 시간=41.841100 연결수=1 curl종료=0
@@ -0,0 +1,41 @@
Saving debug log to /var/log/letsencrypt/letsencrypt.log
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Found the following certs:
Certificate Name: auth.hyeonworks.com
Serial Number: 6c7cb6df1da8a6d7995d93c264bb9ecea1d
Key Type: ECDSA
Identifiers: auth.hyeonworks.com app1.hyeonworks.com app2.hyeonworks.com
Expiry Date: 2026-12-03 07:21:52+00:00 (VALID: 89 days)
Certificate Path: /etc/letsencrypt/live/auth.hyeonworks.com/fullchain.pem
Private Key Path: /etc/letsencrypt/live/auth.hyeonworks.com/privkey.pem
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
=== renewal-hooks ===
/etc/letsencrypt/archive/auth.hyeonworks.com/:
total 48
drwxr-xr-x 2 root root 4096 2026-09-04 17:22:13.508658811 +0900 .
drwx------ 3 root root 4096 2026-09-03 10:47:40.915270972 +0900 ..
-rw-r--r-- 1 root root 1359 2026-09-03 10:47:40.915923507 +0900 cert1.pem
-rw-r--r-- 1 root root 1359 2026-09-04 17:22:13.508494637 +0900 cert2.pem
-rw-r--r-- 1 root root 3523 2026-09-03 10:47:40.916215769 +0900 chain1.pem
-rw-r--r-- 1 root root 3523 2026-09-04 17:22:13.508658811 +0900 chain2.pem
-rw-r--r-- 1 root root 4882 2026-09-03 10:47:40.916339551 +0900 fullchain1.pem
-rw-r--r-- 1 root root 4882 2026-09-04 17:22:13.508821612 +0900 fullchain2.pem
-rw------- 1 root root 241 2026-09-03 10:47:40.916079294 +0900 privkey1.pem
-rw------- 1 root root 241 2026-09-04 17:22:13.507717972 +0900 privkey2.pem
/etc/letsencrypt/renewal-hooks/deploy/:
total 8
drwxr-xr-x 2 root root 4096 2026-09-03 10:46:54.658474560 +0900 .
drwxr-xr-x 5 root root 4096 2026-09-03 10:46:54.658520760 +0900 ..
/etc/letsencrypt/renewal-hooks/post/:
total 8
drwxr-xr-x 2 root root 4096 2026-09-03 10:46:54.658505772 +0900 .
drwxr-xr-x 5 root root 4096 2026-09-03 10:46:54.658520760 +0900 ..
/etc/letsencrypt/renewal-hooks/pre/:
total 8
drwxr-xr-x 2 root root 4096 2026-09-03 10:46:54.658474560 +0900 .
drwxr-xr-x 5 root root 4096 2026-09-03 10:46:54.658520760 +0900 ..
@@ -0,0 +1,101 @@
D-4 판정 — 갱신은 성공했고, 38분 25초 동안 반영되지 않았다
(시계 보정 후. 처음 적은 36분 39초는 서로 다른 시계를 뺀 값이었다)
=============================================================
수집: 2026-09-04 · 감시 08:10:51 ~ 09:02 UTC (KST-9)
════ 답 1 — "갱신 실패"가 아니라 "갱신됐는데 안 집었다" ════
07-renewal-hook-missing.txt 에서 두 갈래로 남겼던 것이 갈렸다.
(a) 강제 갱신이 실패했다
(b) 성공했는데 훅이 없어 reload 가 안 됐다
→ ★ (b) 였다.
디스크 (12-certbot-state.txt)
/etc/letsencrypt/archive/auth.hyeonworks.com/
cert1.pem 2026-09-03 10:47:40 KST ← 처음 발급
cert2.pem 2026-09-04 17:22:13 KST ← ★ 강제 갱신, 새로 써졌다
fullchain2.pem 2026-09-04 17:22:13 KST
privkey2.pem 2026-09-04 17:22:13 KST
Certificate Name: auth.hyeonworks.com
Serial Number: 6c7cb6df1da8a6d7995d93c264bb9ecea1d
Expiry Date: 2026-12-03 07:21:52+00:00 (VALID: 89 days)
밖에서 본 것 (09-serial-timeline.txt, 5초 간격 564표본)
08:10:51 ~ 08:58:47 serial=0520BB...B853 notAfter=Dec 2 ← 옛 것
08:58:52 serial=06C7CB...EA1D notAfter=Dec 3 ← 바뀐 순간
08:22:13 ~ 08:58:52 구간에서 옛 인증서로 관측된 횟수: 428회
시간 계산
새 인증서 디스크 기록 : 08:22:13 UTC (= 17:22:13 KST, archive mtime)
실제 서빙 시작 : 08:58:52 UTC (= 17:58:52 KST, nginx -s reload)
────────────────────────────────────────────
공백 : 2305초 = 38분 25초 ← test-server 가 106초 빠른 것을 보정한 값
(보정 전 2199초로 적었던 것을 정정. D-4a 참조)
그리고 이 36분은 **우연히 짧았을 뿐이다.** reload 를 시킨 것은 사람이지
자동화가 아니다. 아무도 안 했다면 다음 nginx 재시작까지 — 즉 무기한 —
옛 인증서를 계속 서빙했을 것이다.
왜 아무도 reload 하지 않았나 — 원인 세 가지가 겹쳤다
1. certbot-renew.service 에 ExecStartPost 가 없다 (07 참조)
2. /etc/letsencrypt/renewal-hooks/{deploy,post,pre}/ 가 전부 비어 있다
deploy/ total 8 (. 과 .. 뿐)
post/ total 8
pre/ total 8
3. certbot 에 nginx 플러그인이 없다 — 그래서 certbot 이 스스로 못 고친다
Discovered plugins: dns-cloudflare, manual, null, standalone, webroot
(certbot 5.7.0)
★ 세 경로 전부가 비어 있다. 하나라도 있었으면 자동으로 반영됐다.
왜 이게 88일 동안 안 보이나
타이머는 정상이고 오늘도 두 번 status=0/SUCCESS 로 끝났다.
만료 30일 전(= 약 58일 뒤)까지는 certbot 이 갱신 자체를 하지 않으므로
"반영되지 않는다"는 결함이 발현할 기회가 없다. 발현하는 날 증상은
**인증서 만료** 이고, 그때 로그는 여전히 SUCCESS 라고 적혀 있을 것이다.
════ 답 2 — reload 는 무중단이다 (측정됨) ════
계획서의 질문: "nginx reload 중 진행 중이던 요청은 어떻게 되는가"
[새 연결] 0.2초 폴링, 08:10:51 ~ 09:02
전체 표본 8856건 / 비200 **0건**
응답시간 n 중앙 p95 최대
─────────────────────────────────────────────────────────
장기 평시 08:20~08:50 5398 98.0ms 205.7ms 1942.9ms
reload 직전 2분56초 489 116.0ms 200.8ms 387.7ms
reload 직후 2분08초 342 132.5ms 204.3ms 475.0ms
p95 가 205.7 → 204.3 으로 사실상 동일하고, 최대값은 오히려 낮다.
10초 구간 중앙값은 reload 전후 모두 80~190ms 사이를 오간다(WiFi 잡음).
→ **reload 로 인한 지연 증가는 관측되지 않았다.**
[진행 중이던 요청] 845KB 를 20k/s 로 받아 42초간 살려 둔 요청
08:58:40 시작 · 소요 40.84초 → 08:59:21 종료
reload 는 08:58:52, 즉 **요청 시작 12초 뒤 전송 한가운데**였다
코드=200 바이트=845361(전량) 연결수=1 curl종료=0
· 바이트가 전량이다 → 잘리지 않았다
· 연결수가 1이다 → 중간에 끊겨 재연결한 것이 아니다
· 옛 인증서로 시작한 연결이 새 워커 전환을 관통해 끝까지 갔다
in-flight 전체 50건 중 종료코드≠0 은 0건.
(앞서 76건 실패는 로컬 아티팩트였다 — 08-inflight-artifact.txt)
════ 세 줄 요약 ════
1. **"갱신 성공"과 "새 인증서 서빙"은 다른 사건이다.** 38분 25초 동안
디스크에는 새 인증서가, 네트워크에는 옛 인증서가 있었다. 428번 관측했다.
2. **reload 는 정말 무중단이었다.** 8856건 전부 200, p95 변화 없음,
전송 한가운데였던 42초짜리 요청이 845361바이트를 온전히 받았다.
D-4 문서가 "확인하지 않았으므로 그렇게 쓰면 안 된다"고 유보했던 명제가
이제 측정으로 뒷받침된다.
3. **위험한 쪽은 reload 가 아니라 reload 를 부르는 사람이 없다는 것이다.**
훅 디렉터리 3개가 전부 비었고, 유닛에 ExecStartPost 가 없고,
nginx 플러그인도 없다. 그리고 그 상태로 타이머는 SUCCESS 를 찍는다.
@@ -0,0 +1,28 @@
# D-4 — 인증서 갱신 증거
2026-09-04 17:25 18:02 KST
해설: [`docs/experiment-d4-certificate-renewal.md`](../../experiment-d4-certificate-renewal.md)
| 파일 | 무엇을 보여주는가 |
|---|---|
| `01-certificate-state.txt` | SAN 3개(와일드카드 아님) · **체인 4단계, `Verify return code: 0`** · `certbot-renew.timer` enabled·active, 11시간 전 실행 · 88일 남음 · **`sudo: a password is required` 로 강제 갱신 불가** |
| `05-control-no-injection.txt` | **대조군 1 — 잡음 바닥.** 0.2초 × 900회 / 180초 동안 **900전부 200, 오류 0**. 중앙 98ms · p95 195ms. **TLS 핸드셰이크 900/900** = 매 요청이 새 연결이다 |
| `06-inflight-control.txt` | **대조군 2 — '진행 중이던 요청' 측정 장치.** 845KB 번들을 `--limit-rate 20k` 로 받아 요청을 **42초간 살려 둔다.** 무주입 시 코드 200 · 845361바이트 · 연결수 1 |
| `05-control-no-injection.txt` | **대조군 1 — 잡음 바닥.** 0.2초 × 900회 / 180초 동안 **900 전부 200, 오류 0**. 중앙 98ms · p95 195ms. **TLS 핸드셰이크 900/900** = 매 요청이 새 연결 |
| `06-inflight-control.txt` | **대조군 2 — '진행 중이던 요청' 장치.** 845KB 를 `--limit-rate 20k` 로 받아 요청을 **42초간 살려 둔다.** 무주입 시 200 · 845361바이트 · 연결수 1 |
| `07-renewal-hook-missing.txt` | **nginx 는 reload 된 적이 없다** — 마스터 585·워커 586 이 같은 시각 기동, 22.4시간째. 유닛은 `ExecStart=certbot -q renew` 가 전부. crt.sh 는 SCT 가 박힌 인증서를 0건으로 답한다 |
| `08-inflight-artifact.txt` | **76건 실패는 서버 탓이 아니다** — 같은 순간 폴링 49건 전부 200, `연결수=0`, 50µs, 재현 0/100. 대조군이 오보를 막았다 |
| `09-serial-timeline.txt` | **일련번호 564표본.** `08:10:51 ~ 08:58:47` 옛 것 → **`08:58:52` 새 것** |
| `10-reload-poll-window.txt` | reload 전후 60초 새 연결 원문 — **비200 0건**, 최대 373ms |
| `11-inflight-full.txt` | in-flight 전체 50건. **08:58:40 시작 요청이 08:58:52 reload 를 관통**해 845361바이트 전량 수신 |
| `12-certbot-state.txt` | `cert2.pem` **09-04 17:22:13 기록됨** · `renewal-hooks/{deploy,post,pre}/` **셋 다 비었음** · 플러그인 목록에 nginx 없음 |
| `13-verdict.txt` | **판정 전문** — 38분 25초 공백(428회 관측) + reload 무중단(8856건 0실패) |
## 핵심 다섯 줄
1. **「갱신 성공」과 「새 인증서 서빙」은 다른 사건이다.** 새 인증서가 디스크에 있는 채로 **38분 25초** 동안 옛 인증서를 서빙했고, 그 구간에서 **428번** 관측했다.
2. **그 36분은 우연히 짧았다.** reload 를 시킨 것은 사람이다. 아무도 안 했다면 다음 nginx 재시작까지 무기한이었다.
3. **원인이 셋 겹쳤다.** 유닛에 `ExecStartPost` 없음 · 훅 디렉터리 3개 전부 비었음 · certbot 에 nginx 플러그인 없음. 하나라도 있었으면 자동 반영됐다.
4. **reload 자체는 무중단이었다.** 새 연결 8856건 전부 200, p95 205.7 → 204.3ms, 그리고 **전송 12초째에 reload 를 맞은 42초 요청이 845361바이트를 온전히** 받았다(연결수 1).
5. **이 결함은 88일 동안 안 보인다.** 타이머는 오늘도 두 번 `SUCCESS` 로 끝났다. 만료 30일 전까지는 갱신 자체를 하지 않으므로 발현할 기회가 없고, 발현하는 날의 증상은 **인증서 만료**다 — 그날에도 로그는 `SUCCESS` 라고 적혀 있다.
@@ -0,0 +1,125 @@
D-4a — deploy 훅이 실제로 nginx 를 reload 하는가
==================================================
수집: 2026-09-04 12:27 UTC (실제) · test-server
D-4 가 남긴 것
--------------
| 항목 | 상태 |
|---|---|
| deploy 훅을 넣으면 자동 반영되는가 | **미측정. 훅은 아직 넣지 않았다** |
D-4 는 처방을 적어두고 **검증하지 않았다.** 처방이 듣는지 모르는 채로
"이렇게 고치면 된다"고 쓰는 것은 이 실험대가 계속 경계해 온 바로 그 실수다.
[주입] 훅 하나를 넣고 강제 갱신
--------------------------------
/etc/letsencrypt/renewal-hooks/deploy/reload-nginx.sh
#!/bin/sh
# D-4a: certbot deploy hook. 갱신이 실제로 일어났을 때만 실행된다.
nginx -t && nginx -s reload
install -m755 /tmp/reload-nginx.sh /etc/letsencrypt/renewal-hooks/deploy/
certbot renew --force-renewal
[결과] certbot 출력 — 훅이 실행됐다
------------------------------------
Processing /etc/letsencrypt/renewal/auth.hyeonworks.com.conf
Renewing an existing certificate for auth.hyeonworks.com and 2 more
Hook 'deploy-hook' ran with error output:
2026/09/04 21:29:36 [warn] 37250#37250: could not build optimal types_hash, …
nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful
2026/09/04 21:29:37 [warn] 37251#37251: could not build optimal types_hash, …
2026/09/04 21:29:37 [notice] 37251#37251: signal process started
Congratulations, all renewals succeeded:
/etc/letsencrypt/live/auth.hyeonworks.com/fullchain.pem (success)
★ 함정 — "Hook 'deploy-hook' ran with error output" 이라고 적혀 있지만
실패가 아니다. nginx 가 types_hash 경고를 stderr 로 내보내서 certbot 이
그것을 "error output" 으로 표시한 것뿐이다. 내용은 전부 성공이다:
· nginx -t → "test is successful"
· nginx -s reload → "signal process started"
로그에서 'error' 를 grep 하면 성공한 훅을 실패로 오독하게 된다.
[검증 1] nginx 워커가 교체됐다
-------------------------------
전 (D-4a 기준선)
585 1 ... Thu Sep 3 19:00:39 nginx: master process
28829 585 ... Fri Sep 4 18:00:35 nginx: worker process ← D-4 에서 사람이 reload 한 것
585 1 95412 Thu Sep 3 19:00:39 nginx: master process ← 마스터는 그대로
37252 585 74 Fri Sep 4 21:29:36 nginx: worker process ← ★ 자동 교체됨
마스터 PID 는 유지되고 워커만 바뀌었다 = reload 의 서명. D-4 에서 세운
진단 방법이 그대로 작동한다.
[검증 2] 서빙 인증서가 바뀌었다
--------------------------------
serial=06F3E0EF4D1BB03DE58130EAAD1176101373
notBefore=Sep 4 11:29:18 2026 GMT
notAfter=Dec 3 11:29:17 2026 GMT
SAN: app1 / app2 / auth .hyeonworks.com
D-4 의 인증서(06C7CB…EA1D)에서 바뀌었다.
[검증 3] ★ 얼마나 빨랐나 — 시계 보정이 필요하다
--------------------------------------------------
훅 로그와 워커 lstart 는 **test-server 시계**다. 그런데 test-server 는
NTP 동기가 꺼져 있다(NTPSynchronized=no).
외부 기준으로 두 시계를 검증했다.
dev → Google 차이 +0초
dev → Let's Encrypt ACME 차이 +0초
test-server → Google 차이 -105초 (즉 test-server 가 105초 빠르다)
ssh 왕복 왜곡 3회 측정: +106.1 / +106.1 / +106.1초 (안정적)
→ **dev 가 정확하고 test-server 가 106초 빠르다.**
독립 검증 — 새 인증서의 SCT (CT 로그가 서명한 시각, 자체 정확 시계):
Signed Certificate Timestamp: Sep 4 12:27:49.054 2026 GMT
Signed Certificate Timestamp: Sep 4 12:27:49.048 2026 GMT
보정한 타임라인 (전부 실제 UTC):
12:27:49.05 인증서 발급 ← SCT. 외부 권위 기준
12:27:50 훅 nginx -t ← 로그 21:29:36 KST 106초
12:27:50 새 워커 37252 기동 ← lstart 21:29:36 KST 106초
12:27:51 훅 nginx -s reload ← 로그 21:29:37 KST 106초
★ 발급 → 서빙까지 **1~2초.**
★ 그리고 보정이 자기 검증된다 — 독립 시계인 SCT 가 보정한 훅 시각의
1초 앞에 정확히 놓인다. 보정하지 않으면 훅이 발급보다 104초 먼저
실행된 것이 되어 물리적으로 불가능해진다.
════ D-4 와의 대조 ════
훅 없음 (D-4) 훅 있음 (D-4a)
갱신 → 서빙 2305초 = 38분 25초 1~2초
무엇이 reload 했나 사람 (`nginx -s reload`) certbot deploy 훅
안 했다면 다음 nginx 재시작까지 해당 없음
= 사실상 무기한
차이 약 1150배
════ 부수 정정 — D-4 의 2199초는 틀렸다 ════
D-4 에서 적은 **2199초(36분 39초)** 는 `archive/cert2.pem` 의 mtime
(test-server 시계)과 일련번호 관측 시각(dev 시계)을 **그대로 뺀** 값이었다.
106초 보정을 적용하면:
새 인증서 디스크 기록 08:20:27 UTC (실제) ← mtime 17:22:13 KST 106초
실제 서빙 시작 08:58:52 UTC (실제) ← dev 관측, 보정 불필요
공백 2305초 = 38분 25초
관련 문서를 전부 정정했다. **두 시계에서 온 값을 빼면서 그 사실을 적지
않으면, 자릿수가 아니라 방향까지 틀릴 수 있다** — 여기서는 그 오차가
100초 단위였고, 1~2초를 재는 D-4a 에서는 결과를 완전히 뒤집었다.
════ 결론 ════
1. **D-4 의 처방은 듣는다.** 훅 파일 하나로 발급 → 서빙이 38분에서 1~2초가 됐다.
2. **certbot 출력의 "ran with error output" 을 실패로 읽으면 안 된다.**
nginx 의 경고가 stderr 로 나갔을 뿐이며, 내용은 전부 성공이다.
3. **진단 방법이 재확인됐다** — 마스터 PID 유지 + 워커 PID 교체 = reload.
4. **시계를 섞어 빼면 안 된다.** test-server 는 NTP 가 꺼져 106초 빠르고,
그 사실을 적지 않은 채 계산한 D-4 의 공백은 106초 짧았다.
@@ -0,0 +1,17 @@
Saving debug log to /var/log/letsencrypt/letsencrypt.log
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Processing /etc/letsencrypt/renewal/auth.hyeonworks.com.conf
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Renewing an existing certificate for auth.hyeonworks.com and 2 more
Hook 'deploy-hook' ran with error output:
2026/09/04 21:29:36 [warn] 37250#37250: could not build optimal types_hash, you should increase either types_hash_max_size: 1024 or types_hash_bucket_size: 64; ignoring types_hash_bucket_size
nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful
2026/09/04 21:29:37 [warn] 37251#37251: could not build optimal types_hash, you should increase either types_hash_max_size: 1024 or types_hash_bucket_size: 64; ignoring types_hash_bucket_size
2026/09/04 21:29:37 [notice] 37251#37251: signal process started
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Congratulations, all renewals succeeded:
/etc/letsencrypt/live/auth.hyeonworks.com/fullchain.pem (success)
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
@@ -0,0 +1,13 @@
=== 실행 후 상태 (dev 머신에서 관측) ===
수집 2026-09-04T12:29:05Z UTC
-- nginx 프로세스
585 1 95412 Thu Sep 3 19:00:39 2026 nginx: master process /usr/bin/nginx
37252 585 74 Fri Sep 4 21:29:36 2026 nginx: worker process
-- 서빙 인증서
serial=06F3E0EF4D1BB03DE58130EAAD1176101373
notBefore=Sep 4 11:29:18 2026 GMT
notAfter=Dec 3 11:29:17 2026 GMT
X509v3 Subject Alternative Name:
DNS:app1.hyeonworks.com, DNS:app2.hyeonworks.com, DNS:auth.hyeonworks.com
+17
View File
@@ -0,0 +1,17 @@
# D-4a — deploy 훅 검증 증거
2026-09-04 12:27 UTC (실제 시각)
해설: [`docs/experiment-d4a-deploy-hook.md`](../../experiment-d4a-deploy-hook.md)
| 파일 | 무엇을 보여주는가 |
|---|---|
| `01-hook-verified.txt` | **판정 전문.** 훅 실행 로그 · 워커 PID 교체 · 시계 보정과 SCT 교차검증 · D-4 와의 대조 |
| `02-certbot-with-hook.txt` | `certbot renew --force-renewal` 원문. `Hook 'deploy-hook' ran``all renewals succeeded` |
| `03-after-state.txt` | 실행 후 nginx 프로세스와 서빙 인증서 |
## 핵심 네 줄
1. **D-4 의 처방은 듣는다.** 훅 파일 하나로 발급 → 서빙이 **38분 25초에서 1~2초**가 됐다. 약 1150배.
2. **`Hook 'deploy-hook' ran with error output` 을 실패로 읽으면 안 된다.** nginx 의 `types_hash` 경고가 stderr 로 나갔을 뿐이고 내용은 전부 성공이다 — `test is successful`, `signal process started`. 로그에서 `error` 를 grep 하면 성공한 훅을 실패로 오독한다.
3. **진단 방법이 재확인됐다.** 마스터 585 는 그대로, 워커만 28829 → **37252** 로 교체. D-4 에서 세운 「마스터 유지 + 워커 교체 = reload」가 그대로 작동한다.
4. **★ 시계를 섞어 빼면 안 된다.** test-server 는 NTP 가 꺼져 **106초 빠르다**(외부 기준 3회 측정). 이 사실을 적지 않고 계산한 D-4 의 공백은 106초 짧았고, 1~2초를 재는 D-4a 에서는 보정 없이는 **훅이 발급보다 104초 먼저 실행된 것**이 되어 물리적으로 불가능해진다. 보정은 새 인증서 SCT(`12:27:49.054 GMT`, CT 로그의 독립 시계)로 교차검증된다.
@@ -0,0 +1,32 @@
=== D-1 절차대로 먼저 백업 ===
백업: 395375 bytes
마이그레이션 전: 210
세션 전: 3
=== ★ 정방향 업그레이드 + 1초 간격 가용성 측정 ===
시작: 15:22:59
partitioned roll out complete: 2 new pods have been updated...
완료: 15:24:26
=== 업그레이드 중 외부 응답 시계열 ===
200 200 200 200 200 200 200 200 200 200 200 200 200 200 200 200 200 200 200 200
200 200 200 200 200 200 200 200 200 200 200 200 200 200 200 200 200 200 200 200
200 200 200 200 200 200 200 200 200 200 200 200 200 200 200 200 200 200 200 200
200 200 200 200 200 200 200 200 200 200 200 200 200 200 200 200 200 200 200 200
200 200 200 200 200 200 200
200 응답: 87 회
비200 : 0
0 회
=== 업그레이드 후 ===
quay.io/keycloak/keycloak:26.7.3
Keycloak 26.7.3
마이그레이션 후: 210 (전: 210)
세션 후: 3 (전: 3)
=== 스키마 마이그레이션이 실제로 있었는가 ===
(없으면 26.7.0→26.7.3 에 스키마 변경이 없다는 뜻)
=== 파드 상태와 클러스터 ===
keycloak-0 1/1 Running restarts=0
keycloak-1 1/1 Running restarts=0
cluster: [keycloak-1-11418(v=16.0.14)|47] (2) [keycloak-1-11418(v=16.0.14), keycloak-0-58996(v=16.0.14)]
@@ -0,0 +1,20 @@
=== ★ 가설: 스키마 변경이 없으면 롤백이 된다 (26.7.3 → 26.7.0) ===
시작: 15:25:08
partitioned roll out complete: 2 new pods have been updated...
완료: 15:25:53
200 응답: 43 회 / 비200: 1
keycloak-0 1/1 Running restarts=0
keycloak-1 1/1 Running restarts=0
Keycloak 26.7.0
마이그레이션: 210
세션: 3
=== 롤백 중 응답 시계열 (비200 위치) ===
200 200 200 200 200 200 200 200 200 200 200 200 200 200 200 200 200 200 200 200
200 200 200 200 000 200 200 200 200 200 200 200 200 200 200 200 200 200 200 200
200 200 200 200
비200 값: 000
=== 대조: 정방향 업그레이드 때는 ===
200: 87 / 비200: 0
@@ -0,0 +1,66 @@
# B-4 ③ — IdP 의 클레임 변경은 언제 upstream 에 반영되는가
#
# 이 파일은 재수집본이다. 첫 수집본은 "이제 요청을 반복해 본다" 에서 끝났고,
# 그 뒤의 12회 결과는 브라우저 콘솔에서만 보고 파일로 남기지 않았다.
# 서브에이전트 감사가 "증거 없이 서 있는 결론" 으로 지적했고, 다시 측정했다.
#
# 수집 방법:
# - 요청부: Playwright 로 연 브라우저에서 fetch('/api/echo') 를 반복.
# oauth2-proxy 세션 쿠키가 HttpOnly 라 curl 로는 재현할 수 없어 브라우저를 썼다.
# 아래 출력은 브라우저 콘솔이 돌려준 문자열 그대로다.
# - 변경부·저장소 확인부: test-server 의 kubectl 출력 그대로.
=== [1] 기준선 — 변경 전 (브라우저 fetch) ===
2026-09-04T07:51:23.862Z req#1 HTTP 200 x-forwarded-email=labuser@example.com x-forwarded-preferred-username=labuser
2026-09-04T07:51:24.304Z req#2 HTTP 200 x-forwarded-email=labuser@example.com x-forwarded-preferred-username=labuser
2026-09-04T07:51:24.722Z req#3 HTTP 200 x-forwarded-email=labuser@example.com x-forwarded-preferred-username=labuser
=== [2] IdP 에서 email 을 바꾼다 (kubectl 출력) ===
변경 시각(UTC): 2026-09-04T07:53:32.000Z
IdP 의 값:
[ {
"email" : "changed-labuser@example.com"
} ]
oauth2-proxy 세션: 1 개 (그대로 살아 있다)
=== [시계 보정] 두 시계가 다르다 — 해석에 필요하다 ===
개발 머신(브라우저 fetch 의 타임스탬프): 2026-09-04T07:52:20Z
test-server (kubectl 출력의 타임스탬프): 2026-09-04T07:54:07Z
→ test-server 가 약 107초 앞선다.
브라우저 07:51:56 = 서버 07:53:43 이므로, 아래 12회는 변경(07:53:32) 11초 뒤다.
=== [3] 변경 후 12회 반복 (브라우저 fetch) ===
2026-09-04T07:51:56.300Z req#1 HTTP 200 x-forwarded-email=labuser@example.com
2026-09-04T07:51:56.864Z req#2 HTTP 200 x-forwarded-email=labuser@example.com
2026-09-04T07:51:57.489Z req#3 HTTP 200 x-forwarded-email=labuser@example.com
2026-09-04T07:51:58.018Z req#4 HTTP 200 x-forwarded-email=labuser@example.com
2026-09-04T07:51:58.602Z req#5 HTTP 200 x-forwarded-email=labuser@example.com
2026-09-04T07:51:59.217Z req#6 HTTP 200 x-forwarded-email=labuser@example.com
2026-09-04T07:51:59.743Z req#7 HTTP 200 x-forwarded-email=labuser@example.com
2026-09-04T07:52:00.342Z req#8 HTTP 200 x-forwarded-email=labuser@example.com
2026-09-04T07:52:00.964Z req#9 HTTP 200 x-forwarded-email=labuser@example.com
2026-09-04T07:52:01.574Z req#10 HTTP 200 x-forwarded-email=labuser@example.com
2026-09-04T07:52:02.187Z req#11 HTTP 200 x-forwarded-email=labuser@example.com
2026-09-04T07:52:02.719Z req#12 HTTP 200 x-forwarded-email=labuser@example.com
→ 12회 · 약 6.4초 동안 전부 옛 값. 요청 횟수로는 반영되지 않는다.
=== [4] IdP 값은 이미 바뀌어 있었다 (대조) ===
IdP: "email" : "changed-labuser@example.com"
=== [5] 세션을 지워 재인증을 강제한다 ===
남은 oauth2-proxy 세션: 0 개
(Keycloak SSO 는 살아 있으므로 조용히 재인증된다)
=== [6] 재인증 후 (브라우저 fetch) ===
2026-09-04T07:53:01.121Z req#1 HTTP 200 x-forwarded-email=changed-labuser@example.com
2026-09-04T07:53:01.456Z req#2 HTTP 200 x-forwarded-email=changed-labuser@example.com
2026-09-04T07:53:01.785Z req#3 HTTP 200 x-forwarded-email=changed-labuser@example.com
=== 결론 ===
변경 후 12회 요청(6.4초) → labuser@example.com (옛 값)
세션 삭제 후 재인증 → changed-labuser@example.com (새 값)
→ 반영 시점은 요청 횟수가 아니라 세션이 새로 만들어지는가로 정해진다.
--cookie-refresh 가 설정되어 있지 않으므로, 실제 운영에서는
쿠키 만료(1시간) 또는 재인증까지 옛 값이 간다.
@@ -0,0 +1,17 @@
=== Prometheus 가 실제로 긁는 대상 (2026-09-04 18:10 KST) ===
keycloak 2개
kubelet 2개
node-exporter 2개
prometheus 1개
=== B층 구성 요소의 지표가 있는가 ===
redis_up 시계열 0개
redis_connected_clients 시계열 0개
redis_memory_used_bytes 시계열 0개
pg_up 시계열 0개
pg_stat_database_numbackends 시계열 0개
→ B-1·B-2·B-3·B-5 는 Grafana 증거를 만들 수 없다.
스크린샷을 안 찍은 것이 아니라 긁는 대상에 없다.
보완하려면 redis_exporter · postgres_exporter · BFF 의 /actuator/prometheus 를
scrape 대상에 추가해야 한다.

Some files were not shown because too many files have changed in this diff Show More