refactor: 백엔드 리펙토링 진행 중

This commit is contained in:
DongHyeonka
2026-09-25 17:57:07 +09:00
parent 5e44be9ab4
commit 4d4cc35e6c
19 changed files with 1205 additions and 139 deletions
+31 -6
View File
@@ -177,6 +177,8 @@ app:
consumer: { type: SASL_SCRAM, credential-id: kafka-consumer }
# admin은 application runtime에 설정하지 않는다
hostname-verification: true
# 사설 CA로 서명된 TLS 브로커면(tls-enabled: true일 때만):
# trust-store: { location: /etc/kafka/ca.p12, type: PKCS12, password-credential-id: kafka-truststore }
access:
publishable: [order-events]
consumable: []
@@ -191,13 +193,36 @@ app:
연결 시점에 해석하므로, 설정 덤프나 힙 덤프에서 나오는 것은 이름뿐이다. producer와 consumer는
서로 다른 `credential-id`를 써야 하며, 같으면 부팅에 실패한다.
`SASL_SCRAM`·`USERNAME_PASSWORD`의 사용자 이름은 `CredentialProvider.username(credentialId)`가
준다. credential id를 사용자 이름으로 보내지 않는다. 비밀번호와 사용자 이름은 JAAS 문자열에 들어가지
않고, 연결이 인증할 때마다 provider에서 읽힌다. 그래서 provider가 만료 시각을 주는 credential은
재기동 없이 rotation된다(`security.md`).
`MUTUAL_TLS`는 클라이언트 인증서를 key store로 제출한다. `credential-id`는 key store 비밀번호를
가리킨다.
```text
producer:
type: MUTUAL_TLS
credential-id: kafka-producer-tls # key store 비밀번호
key-store: { location: /etc/kafka/producer.p12, type: PKCS12 }
```
`trust-store`를 생략하면 JVM 기본 trust store를 쓴다. 사설 CA 브로커는 `trust-store.location`
(PKCS12·JKS·PEM)과, 잠긴 store면 `password-credential-id`를 준다.
Kafka producer는 이 항목을 실제 클라이언트 설정(`security.protocol`, `sasl.mechanism`,
`sasl.jaas.config`, `ssl.enabled.protocols=TLSv1.3,TLSv1.2`, hostname 검증)으로 옮긴다. 평문으로
조용히 넘어가지 않도록 다음 경우 부팅에 실패한다: 브로커가 `tls-enabled`나
`authentication-enabled`를 켰는데 `app.messaging.security.<broker>`가 없을 때, security 항목은
있는데 `CredentialProvider` bean이 없을 때, 같은 브로커에 맞는 항목이 둘 이상일 때.
한계: truststore/keystore 설정 키가 없어 TLS는 JVM 기본 truststore를 쓰고, `MUTUAL_TLS`는 클라이언트
인증서를 제출할 수 없다(미지원).
`sasl.jaas.config`, `sasl.client.callback.handler.class`, `ssl.truststore.*`, `ssl.keystore.*`,
`ssl.enabled.protocols=TLSv1.3,TLSv1.2`, hostname 검증)으로 옮긴다. 평문으로 조용히 넘어가지
않도록 다음 경우 부팅에 실패한다: 브로커가 `tls-enabled`나 `authentication-enabled`를 켰는데
`app.messaging.security.<broker>`가 없을 때, security 항목은 있는데 `CredentialProvider` bean이 없을
때, 같은 브로커에 맞는 항목이 둘 이상일 때, provider가 SASL 사용자 이름을 주지 않을 때,
`MUTUAL_TLS`인데 `key-store`가 없거나 `tls-enabled`가 꺼져 있을 때, `trust-store`를 줬는데
`tls-enabled`가 꺼져 있을 때, store 파일을 열 수 없을 때.
`access.publishable`은 강제된다. security 항목이 있는 브로커의 destination은 이 목록에 있어야
publish된다. 목록 밖 publish는 `REJECTED`(`PUBLISH_FORBIDDEN`)이고, 빈 목록이면 아무것도 publish되지
않는다.
## Backpressure
+22 -6
View File
@@ -36,17 +36,30 @@ heap dump나 설정 출력에서 사용 가능한 credential이 나오지 않는
`CredentialIds`는 `bearer `, `sk-`, `-----begin`, `eyJ` 같은 접두사를 거부한다.
참조가 들어갈 자리에 secret 자체를 붙여넣는 가장 흔한 사고를 막는다.
SASL(PLAIN·SCRAM) 사용자 이름도 `CredentialProvider.username(credentialId)`가 준다. credential id는
저장소 안의 참조이지 broker가 아는 사용자 이름이 아니다. provider가 이름을 주지 않으면 기동에
실패한다.
key store·trust store는 경로만 설정에 둔다. 비밀번호는 credential id로 provider가 해석한다
(`configuration-reference.md` 보안 절).
## Rotation
`CredentialRotationPlan.isDue()`는 만료 **전에** 참이 된다.
broker가 연결을 거부하기 시작한 시점에는 이미 publish가 실패하고 consumer가 멈춰 있다.
rotation은 세대 교체다. `DefaultMessagingRuntimeRegistry.install()`이 새 세대를 원자적으로
게시하고, 이전 세대는 마지막 lease가 닫힐 때까지 열려 있다가 닫힌다.
진행 중인 publish는 시작한 연결에서 confirm을 받는다.
Kafka SASL credential은 producer를 다시 만들지 않고 바뀐다. JAAS 설정에는 login module 이름만
있고, `KafkaCredentialCallbackHandler`가 연결이 인증할 때마다 `CredentialRuntimeRegistry`에서
사용자 이름과 비밀번호를 읽는다. registry는 provider가 준 만료 시각의 rotation lead(기본 30분)
안으로 들어오면 새로 가져오므로, 새 연결과 재인증(broker `connections.max.reauth.ms`)이 새 값을
쓴다. 만료 시각을 주지 않는 provider의 credential은 캐시된 값을 계속 쓴다.
drain deadline이 이 대기를 제한한다. 없으면 lease 하나가 새면 폐기된 credential이
무기한 열려 있고, rotation이 보안상 무의미해진다.
key store·trust store 비밀번호는 예외다. Kafka가 SSL context를 만들 때 한 번 읽으므로 바꾸려면
producer를 다시 만들어야 한다(재기동).
transport 세대 교체(`DefaultMessagingRuntimeRegistry.install()`)는 transport 자체를 바꿀 때 쓴다.
새 세대를 원자적으로 게시하고, 이전 세대는 마지막 lease가 닫힐 때까지 열려 있다가 닫힌다.
drain deadline이 이 대기를 제한한다.
## Header
@@ -66,10 +79,13 @@ logical identity가 호출자 제어가 된다.
## ACL
`DestinationAccessValidator`가 broker ACL **이전에** 검사한다.
publisher가 broker ACL **이전에** 검사한다(`PUBLISH_FORBIDDEN`).
broker ACL 거부는 애플리케이션 컨텍스트가 없는 연결 수준 오류로 도착하므로
"어느 모듈이 어디에 publish하려 했는가"가 조사 대상이 된다.
publish 가능한 destination은 선언된 destination이다. broker에 `app.messaging.security.<broker>`
항목이 있으면 그중 `access.publishable`에 적힌 것만 남는다(빈 목록이면 publish 불가).
## 관측성 누출
`MessagingRedactor`는 denylist다.
@@ -0,0 +1,84 @@
package dev.caskeleton.messaging.kafka;
import dev.caskeleton.messaging.api.error.MessagingConfigurationException;
import dev.caskeleton.messaging.security.CredentialRuntime;
import dev.caskeleton.messaging.security.CredentialRuntimeRegistry;
import java.time.Instant;
import java.util.Arrays;
import java.util.List;
import java.util.Map;
import javax.security.auth.callback.Callback;
import javax.security.auth.callback.NameCallback;
import javax.security.auth.callback.PasswordCallback;
import javax.security.auth.callback.UnsupportedCallbackException;
import javax.security.auth.login.AppConfigurationEntry;
import org.apache.kafka.common.security.auth.AuthenticateCallbackHandler;
/**
* Answers the SASL client's user name and password from {@link CredentialRuntimeRegistry}, at each
* authentication.
*
* <p>The JAAS entry used to carry both values, fixed when the producer was built: a rotated secret
* never reached a connection until the process restarted, and the user name was the credential id.
* Kafka asks this handler every time a connection authenticates — a new connection, or a
* re-authentication when the broker sets {@code connections.max.reauth.ms} — so each one reads the
* registry's current generation.
*
* <p>Kafka instantiates this class itself, so the registry and the credential id arrive through the
* client configuration under {@link #REGISTRY} and {@link #CREDENTIAL_ID}.
*/
public final class KafkaCredentialCallbackHandler implements AuthenticateCallbackHandler {
/** Client configuration key carrying the {@link CredentialRuntimeRegistry}. */
public static final String REGISTRY = "caskeleton.messaging.credential.registry";
/** Client configuration key carrying the credential id this client authenticates with. */
public static final String CREDENTIAL_ID = "caskeleton.messaging.credential.id";
private CredentialRuntimeRegistry registry;
private String credentialId;
/** Created by Kafka through reflection. */
public KafkaCredentialCallbackHandler() {}
@Override
public void configure(
Map<String, ?> configs, String saslMechanism, List<AppConfigurationEntry> jaasConfigEntries) {
if (!(configs.get(REGISTRY) instanceof CredentialRuntimeRegistry configured)
|| !(configs.get(CREDENTIAL_ID) instanceof String id)) {
throw new MessagingConfigurationException(
"SASL_CALLBACK_NOT_CONFIGURED",
"the Kafka client names this callback handler but does not carry "
+ REGISTRY
+ " and "
+ CREDENTIAL_ID);
}
this.registry = configured;
this.credentialId = id;
}
@Override
public void handle(Callback[] callbacks) throws UnsupportedCallbackException {
CredentialRuntime credential = registry.resolve(credentialId, Instant.now());
for (Callback callback : callbacks) {
if (callback instanceof NameCallback name) {
name.setName(
credential
.username()
.orElseThrow(() -> KafkaSecurityConfigurer.usernameMissing(credentialId)));
} else if (callback instanceof PasswordCallback password) {
char[] material = credential.material();
// PasswordCallback keeps its own copy, so this one is wiped as soon as it is handed over.
password.setPassword(material);
Arrays.fill(material, '\0');
} else {
// SCRAM asks for extensions and carries on without them when the handler declines.
throw new UnsupportedCallbackException(callback);
}
}
}
@Override
public void close() {}
}
@@ -1,5 +1,6 @@
package dev.caskeleton.messaging.kafka;
import dev.caskeleton.messaging.api.error.MessagingConfigurationException;
import dev.caskeleton.messaging.security.BrokerCredentialProfile;
import dev.caskeleton.messaging.security.BrokerSecurityProfile;
import dev.caskeleton.messaging.security.BrokerTlsPolicy;
@@ -14,17 +15,18 @@ import java.util.Objects;
/**
* Builds the Kafka client security properties from a validated security profile.
*
* <p>Nothing here reads an environment variable or a property file. The credential arrives already
* resolved from {@link CredentialRuntimeRegistry}, which is what makes rotation possible: a client
* configured from a value read once at startup holds that value until the process restarts, so the
* rotation the credential store performs never reaches the broker connection.
* <p>Nothing here reads an environment variable or a property file. Credentials resolve through
* {@link CredentialRuntimeRegistry}.
*
* <p>The JAAS config is assembled into a {@code String} because that is the shape Kafka's client
* configuration accepts. The configurer keeps no field reference after returning the map, but the
* returned immutable String contains the credential and cannot be zeroized; its lifetime is the
* Kafka/client configuration lifetime. The source {@link CredentialRuntime} material is still
* zeroizable, but converting it to JAAS necessarily weakens that property and must not be described
* as if the String never existed.
* <p>SASL user names and passwords are not written into the configuration at all. The JAAS entry
* names the login module and nothing else, and {@link KafkaCredentialCallbackHandler} answers the
* user name and password from the registry each time a connection authenticates — so a credential
* the store rotates (one whose provider reports an expiry) reaches the next connection without a
* restart, and no immutable {@code String} holds the password for the client's lifetime.
*
* <p>Key and trust store passwords are the exception: Kafka reads them once, when the client builds
* its SSL context, and accepts them only as configuration values. They are resolved here and live
* as long as the client does; rotating one means rebuilding the client.
*/
public final class KafkaSecurityConfigurer {
@@ -37,6 +39,9 @@ public final class KafkaSecurityConfigurer {
/** Kafka's key for the JAAS configuration. */
public static final String SASL_JAAS_CONFIG = "sasl.jaas.config";
/** Kafka's key for the SASL client callback handler class. */
public static final String SASL_CLIENT_CALLBACK_HANDLER = "sasl.client.callback.handler.class";
/** Kafka's key for the endpoint identification algorithm. */
public static final String ENDPOINT_IDENTIFICATION = "ssl.endpoint.identification.algorithm";
@@ -78,6 +83,9 @@ public final class KafkaSecurityConfigurer {
// Validate before resolving. A profile that will be refused should not cause a credential to be
// fetched and briefly held in memory for nothing.
tlsPolicy.validate(profile, enabledProtocols);
if (!profile.tlsEnabled()) {
requireNoTlsMaterial(profile, credential);
}
Map<String, Object> properties = new LinkedHashMap<>();
properties.put(SECURITY_PROTOCOL, securityProtocol(profile, credential));
@@ -87,31 +95,51 @@ public final class KafkaSecurityConfigurer {
// Kafka disables hostname verification by setting this to the empty string. The policy has
// already refused that combination, so the value here is always the verifying one.
properties.put(ENDPOINT_IDENTIFICATION, "https");
profile
.trustStore()
.ifPresent(
store -> {
properties.put("ssl.truststore.location", store.location());
properties.put("ssl.truststore.type", store.type());
store
.passwordCredentialId()
.ifPresent(id -> properties.put("ssl.truststore.password", secret(id, now)));
});
}
switch (credential) {
case BrokerCredentialProfile.SaslScram scram -> {
CredentialRuntime resolved = credentials.resolve(scram.credentialId(), now);
properties.put(SASL_MECHANISM, "SCRAM-SHA-512");
properties.put(SASL_JAAS_CONFIG, scramJaas(scram.credentialId(), resolved));
}
case BrokerCredentialProfile.SaslScram scram ->
properties.putAll(
sasl(
"SCRAM-SHA-512",
"org.apache.kafka.common.security.scram.ScramLoginModule",
scram.credentialId(),
now));
case BrokerCredentialProfile.OAuth2 oauth -> {
// Refused rather than half-configured. Setting the mechanism name without a callback
// handler produces a client that authenticates with nothing and fails at connect time,
// which is later and harder to attribute than failing here.
throw new dev.caskeleton.messaging.api.error.MessagingConfigurationException(
throw new MessagingConfigurationException(
"SASL_OAUTH_NOT_CONFIGURED",
"OAuth bearer needs a login callback handler and a token endpoint; this platform"
+ " configures the mechanism name only, so the profile is refused rather than"
+ " producing a client that cannot authenticate: "
+ oauth.credentialId());
}
case BrokerCredentialProfile.MutualTls ignored -> properties.put(SASL_MECHANISM, "NONE");
case BrokerCredentialProfile.UsernamePassword plain -> {
CredentialRuntime resolved = credentials.resolve(plain.credentialId(), now);
properties.put(SASL_MECHANISM, "PLAIN");
properties.put(SASL_JAAS_CONFIG, plainJaas(plain.credentialId(), resolved));
case BrokerCredentialProfile.MutualTls tls -> {
String password = secret(tls.credentialId(), now);
properties.put("ssl.keystore.location", tls.keyStoreLocation());
properties.put("ssl.keystore.type", tls.keyStoreType());
properties.put("ssl.keystore.password", password);
properties.put("ssl.key.password", password);
}
case BrokerCredentialProfile.UsernamePassword plain ->
properties.putAll(
sasl(
"PLAIN",
"org.apache.kafka.common.security.plain.PlainLoginModule",
plain.credentialId(),
now));
case BrokerCredentialProfile.Nkey ignored ->
throw new IllegalArgumentException(
"NKey credentials are a NATS concept, not a Kafka one");
@@ -119,6 +147,62 @@ public final class KafkaSecurityConfigurer {
return Map.copyOf(properties);
}
/**
* The SASL properties: a JAAS entry naming only the login module, and the handler that answers.
*
* <p>The credential is resolved once here so that a provider with no user name, or no secret,
* fails startup rather than the first connection.
*/
private Map<String, Object> sasl(
String mechanism, String loginModule, String credentialId, Instant now) {
if (credentials.resolve(credentialId, now).username().isEmpty()) {
throw usernameMissing(credentialId);
}
return Map.of(
SASL_MECHANISM,
mechanism,
SASL_JAAS_CONFIG,
loginModule + " required;",
SASL_CLIENT_CALLBACK_HANDLER,
KafkaCredentialCallbackHandler.class,
KafkaCredentialCallbackHandler.REGISTRY,
credentials,
KafkaCredentialCallbackHandler.CREDENTIAL_ID,
credentialId);
}
private String secret(String credentialId, Instant now) {
CredentialRuntime resolved = credentials.resolve(credentialId, now);
return new String(resolved.material());
}
/** TLS material on a plaintext connection is a configuration that cannot mean what it says. */
private static void requireNoTlsMaterial(
BrokerSecurityProfile profile, BrokerCredentialProfile credential) {
if (profile.trustStore().isPresent()) {
throw new MessagingConfigurationException(
"TRUST_STORE_REQUIRES_TLS",
"broker %s configures a trust store but not TLS; nothing would ever read it"
.formatted(profile.broker()));
}
if (credential instanceof BrokerCredentialProfile.MutualTls) {
throw new MessagingConfigurationException(
"MUTUAL_TLS_REQUIRES_TLS",
"broker %s authenticates with a client certificate but does not enable TLS, and a"
.formatted(profile.broker())
+ " certificate is only presented inside a TLS handshake");
}
}
static MessagingConfigurationException usernameMissing(String credentialId) {
return new MessagingConfigurationException(
"SASL_USERNAME_MISSING",
"credential "
+ credentialId
+ " authenticates with a user name and password, and CredentialProvider.username"
+ " returned none; the credential id is a reference, not the broker's user name");
}
private static String securityProtocol(
BrokerSecurityProfile profile, BrokerCredentialProfile credential) {
boolean sasl = !(credential instanceof BrokerCredentialProfile.MutualTls);
@@ -127,50 +211,4 @@ public final class KafkaSecurityConfigurer {
}
return sasl ? "SASL_PLAINTEXT" : "PLAINTEXT";
}
private static String scramJaas(String credentialId, CredentialRuntime resolved) {
return jaas("org.apache.kafka.common.security.scram.ScramLoginModule", credentialId, resolved);
}
private static String plainJaas(String credentialId, CredentialRuntime resolved) {
return jaas("org.apache.kafka.common.security.plain.PlainLoginModule", credentialId, resolved);
}
/**
* Builds a JAAS entry with both values escaped.
*
* <p>The password was interpolated into a quoted string unescaped. A perfectly ordinary generated
* secret containing a quote or a backslash then produced a JAAS config that either fails to parse
* — an authentication error that names nothing useful — or, with a quote and a semicolon,
* terminates the option early and injects another one. A secret is attacker-influenced in exactly
* one sense that matters here: nobody reads it before it is used.
*/
private static String jaas(String loginModule, String credentialId, CredentialRuntime resolved) {
return loginModule
+ " required username=\""
+ escapeJaas(credentialId)
+ "\" password=\""
+ escapeJaas(new String(resolved.material()))
+ "\";";
}
/**
* Escapes a JAAS quoted-string value.
*
* <p>Backslash first, then quote — the other order double-escapes the backslash it just added.
* Control characters are refused rather than escaped: JAAS has no escape for them, and a secret
* containing one cannot be expressed in this format at all.
*/
private static String escapeJaas(String value) {
for (int index = 0; index < value.length(); index++) {
char character = value.charAt(index);
if (character < 0x20 || character == 0x7F) {
throw new dev.caskeleton.messaging.api.error.MessagingConfigurationException(
"JAAS_VALUE_UNREPRESENTABLE",
"a credential containing a control character cannot be expressed in a JAAS entry;"
+ " configure the client with a callback handler instead");
}
}
return value.replace("\\", "\\\\").replace("\"", "\\\"");
}
}
@@ -2,80 +2,106 @@ package dev.caskeleton.messaging.kafka;
import static org.assertj.core.api.Assertions.assertThat;
import static org.assertj.core.api.Assertions.assertThatThrownBy;
import static org.assertj.core.api.Assertions.catchThrowable;
import dev.caskeleton.messaging.api.error.MessagingConfigurationException;
import dev.caskeleton.messaging.security.BrokerCredentialProfile;
import dev.caskeleton.messaging.security.BrokerSecurityProfile;
import dev.caskeleton.messaging.security.BrokerTlsPolicy;
import dev.caskeleton.messaging.security.BrokerTrustStore;
import dev.caskeleton.messaging.security.CredentialProvider;
import dev.caskeleton.messaging.security.CredentialRuntimeRegistry;
import dev.caskeleton.messaging.security.DestinationAccessPolicy;
import java.time.Duration;
import java.time.Instant;
import java.util.List;
import java.util.Map;
import java.util.Optional;
import org.junit.jupiter.api.DisplayName;
import org.junit.jupiter.api.Test;
/**
* What reaches a JAAS entry, and what is refused outright.
* What reaches the Kafka client configuration, and what is refused outright.
*
* <p>The password was interpolated into a quoted string with no escaping. A generated secret
* containing a quote or a backslash produced a config that fails to parse — an authentication error
* naming nothing useful — and one containing a quote and a semicolon terminated the option early
* and injected another.
* <p>The JAAS entry used to carry the credential id as the user name and the resolved secret as an
* immutable string, fixed for the life of the producer. It now names the login module and nothing
* else: the user name and password are answered by {@link KafkaCredentialCallbackHandler} at each
* authentication.
*/
class KafkaSecurityConfigurerTest {
private static final Instant NOW = Instant.parse("2026-08-14T00:00:00Z");
@Test
@DisplayName("a secret containing a quote and a backslash is escaped, not injected")
void aSecretWithQuotesIsEscaped() {
Map<String, Object> properties = configure("pa\"ss\\word;");
private static final BrokerCredentialProfile SCRAM =
new BrokerCredentialProfile.SaslScram("kafka-app");
String jaas = String.valueOf(properties.get("sasl.jaas.config"));
assertThat(jaas)
.as("an unescaped quote closes the option and everything after it becomes new options")
.contains("password=\"pa\\\"ss\\\\word;\";");
assertThat(jaas)
.as("the raw form would close the password option and make the rest new options")
.doesNotContain("password=\"pa\"");
@Test
void theJaasEntryCarriesNeitherTheUserNameNorTheSecret() {
Map<String, Object> properties = configure(registry(Optional.of("svc-orders")), tls(SCRAM));
assertThat(properties.get("sasl.jaas.config"))
.isEqualTo("org.apache.kafka.common.security.scram.ScramLoginModule required;");
}
@Test
@DisplayName("a secret containing a control character is refused")
void aSecretWithAControlCharacterIsRefused() {
assertThatThrownBy(() -> configure("pass\nword"))
void aSaslCredentialWhoseProviderNamesNoUserIsRefused() {
// given
CredentialRuntimeRegistry anonymous = registry(Optional.empty());
// when
Throwable thrown = catchThrowable(() -> configure(anonymous, tls(SCRAM)));
// then
assertThat(thrown)
.isInstanceOf(MessagingConfigurationException.class)
.hasMessageContaining("control character");
.extracting(failure -> ((MessagingConfigurationException) failure).failure().code())
.isEqualTo("SASL_USERNAME_MISSING");
}
@Test
void aTrustStoreOnABrokerWithoutTlsIsRefused() {
// given
BrokerSecurityProfile plaintext =
new BrokerSecurityProfile(
"kafka-primary",
false,
false,
true,
SCRAM,
SCRAM,
Optional.empty(),
DestinationAccessPolicy.denyAll(),
Optional.of(new BrokerTrustStore("/etc/kafka/ca.p12", "PKCS12", Optional.empty())));
// when
Throwable thrown =
catchThrowable(() -> configure(registry(Optional.of("svc-orders")), plaintext));
// then
assertThat(thrown)
.isInstanceOf(MessagingConfigurationException.class)
.extracting(failure -> ((MessagingConfigurationException) failure).failure().code())
.isEqualTo("TRUST_STORE_REQUIRES_TLS");
}
@Test
@DisplayName("an OAuth profile is refused rather than half-configured")
void anOAuthProfileIsRefused() {
CredentialRuntimeRegistry credentials = registry("secret");
CredentialRuntimeRegistry credentials = registry(Optional.of("svc-orders"));
BrokerCredentialProfile oauth = new BrokerCredentialProfile.OAuth2("kafka-oauth");
assertThatThrownBy(
() ->
new KafkaSecurityConfigurer(
credentials, new dev.caskeleton.messaging.security.BrokerTlsPolicy())
.configure(profile(oauth), oauth, java.util.List.of("TLSv1.3"), NOW))
// when & then
assertThatThrownBy(() -> configure(credentials, tls(oauth)))
.as("naming the mechanism without a callback handler fails at connect time instead")
.isInstanceOf(MessagingConfigurationException.class)
.hasMessageContaining("callback handler");
}
private static Map<String, Object> configure(String secret) {
return new KafkaSecurityConfigurer(
registry(secret), new dev.caskeleton.messaging.security.BrokerTlsPolicy())
.configure(
profile(new BrokerCredentialProfile.SaslScram("kafka-app")),
new BrokerCredentialProfile.SaslScram("kafka-app"),
java.util.List.of("TLSv1.3"),
NOW);
private static Map<String, Object> configure(
CredentialRuntimeRegistry credentials, BrokerSecurityProfile profile) {
return new KafkaSecurityConfigurer(credentials, new BrokerTlsPolicy())
.configure(profile, profile.producer(), List.of("TLSv1.3"), NOW);
}
private static BrokerSecurityProfile profile(BrokerCredentialProfile credential) {
private static BrokerSecurityProfile tls(BrokerCredentialProfile credential) {
return new BrokerSecurityProfile(
"kafka-primary",
true,
@@ -84,21 +110,26 @@ class KafkaSecurityConfigurerTest {
credential,
credential,
Optional.empty(),
dev.caskeleton.messaging.security.DestinationAccessPolicy.denyAll());
DestinationAccessPolicy.denyAll());
}
private static CredentialRuntimeRegistry registry(String secret) {
private static CredentialRuntimeRegistry registry(Optional<String> username) {
return new CredentialRuntimeRegistry(
new CredentialProvider() {
@Override
public char[] resolve(String credentialId) {
return secret.toCharArray();
return "secret".toCharArray();
}
@Override
public Optional<Instant> expiresAt(String credentialId) {
return Optional.of(NOW.plus(Duration.ofHours(1)));
}
@Override
public Optional<String> username(String credentialId) {
return username;
}
},
Duration.ofMinutes(5));
}
@@ -44,13 +44,28 @@ public sealed interface BrokerCredentialProfile
}
/**
* Mutual TLS authentication.
* Mutual TLS authentication: the client certificate in a key store, unlocked by the credential.
*
* @param credentialId the credential id
* <p>The key store was not part of the profile, so a {@code MUTUAL_TLS} credential configured a
* TLS connection with no client certificate — one every broker that asks for a certificate
* refuses. The path is not a secret and lives here; the password is, and resolves through the
* credential id like every other secret.
*
* @param credentialId the credential whose material unlocks the key store
* @param keyStoreLocation the key store holding the client certificate and private key
* @param keyStoreType the key store format, such as {@code PKCS12}
*/
record MutualTls(String credentialId) implements BrokerCredentialProfile {
record MutualTls(String credentialId, String keyStoreLocation, String keyStoreType)
implements BrokerCredentialProfile {
public MutualTls {
CredentialIds.require(credentialId);
if (keyStoreLocation == null || keyStoreLocation.isBlank()) {
throw new IllegalArgumentException(
"mutual TLS presents a client certificate, and without a key store there is none");
}
if (keyStoreType == null || keyStoreType.isBlank()) {
throw new IllegalArgumentException("keyStoreType must not be blank");
}
}
}
@@ -18,6 +18,7 @@ import java.util.Optional;
* @param consumer the consumer credential
* @param admin the admin credential, absent in an application runtime
* @param access the destination access policy
* @param trustStore the certificates the connection trusts; empty means the JVM default trust store
*/
public record BrokerSecurityProfile(
String broker,
@@ -27,15 +28,39 @@ public record BrokerSecurityProfile(
BrokerCredentialProfile producer,
BrokerCredentialProfile consumer,
Optional<BrokerCredentialProfile> admin,
DestinationAccessPolicy access) {
DestinationAccessPolicy access,
Optional<BrokerTrustStore> trustStore) {
public BrokerSecurityProfile {
Objects.requireNonNull(producer, "producer credential must not be null");
Objects.requireNonNull(consumer, "consumer credential must not be null");
Objects.requireNonNull(admin, "admin credential must not be null");
Objects.requireNonNull(access, "access policy must not be null");
Objects.requireNonNull(trustStore, "trust store must not be null");
if (broker == null || broker.isBlank()) {
throw new IllegalArgumentException("broker must not be blank");
}
}
/** A profile that trusts what the JVM's default trust store trusts. */
public BrokerSecurityProfile(
String broker,
boolean production,
boolean tlsEnabled,
boolean hostnameVerification,
BrokerCredentialProfile producer,
BrokerCredentialProfile consumer,
Optional<BrokerCredentialProfile> admin,
DestinationAccessPolicy access) {
this(
broker,
production,
tlsEnabled,
hostnameVerification,
producer,
consumer,
admin,
access,
Optional.empty());
}
}
@@ -0,0 +1,30 @@
package dev.caskeleton.messaging.security;
import java.util.Objects;
import java.util.Optional;
/**
* The certificates a broker connection trusts, when the JVM's default trust store is not enough.
*
* <p>A broker signed by a private CA could not be reached at all: there was no setting for a trust
* store, so the client verified the broker against the JVM defaults and refused it. The path is not
* a secret; the store password, when the format has one, resolves through a credential id.
*
* @param location the trust store file
* @param type the trust store format, such as {@code PKCS12} or {@code PEM}
* @param passwordCredentialId the credential whose material unlocks the store, when it is locked
*/
public record BrokerTrustStore(
String location, String type, Optional<String> passwordCredentialId) {
public BrokerTrustStore {
Objects.requireNonNull(passwordCredentialId, "passwordCredentialId must not be null");
if (location == null || location.isBlank()) {
throw new IllegalArgumentException("a trust store needs a location");
}
if (type == null || type.isBlank()) {
throw new IllegalArgumentException("a trust store needs a type");
}
passwordCredentialId.ifPresent(CredentialIds::require);
}
}
@@ -26,4 +26,21 @@ public interface CredentialProvider {
* @return the expiry instant when known
*/
Optional<Instant> expiresAt(String credentialId);
/**
* Returns the user name the secret authenticates, for mechanisms that send one.
*
* <p>SASL PLAIN and SCRAM send a user name and a password. The credential id is a reference into
* the store, not the principal the broker knows, and sending it as the user name authenticated as
* whoever happened to share the reference's spelling. A store that rotates by alternating between
* two users rotates the name together with the secret, so the name is resolved alongside it.
*
* <p>Empty by default. A SASL credential whose provider names no user is refused at startup.
*
* @param credentialId the identifier
* @return the user name, when this credential has one
*/
default Optional<String> username(String credentialId) {
return Optional.empty();
}
}
@@ -26,12 +26,14 @@ public final class CredentialRuntime {
private final Instant resolvedAt;
private final Optional<Instant> expiresAt;
private final Duration rotationLead;
private final Optional<String> username;
private char[] material;
/**
* Creates a runtime entry.
*
* @param credentialId the credential reference
* @param username the user name the secret authenticates, when the mechanism sends one
* @param material the resolved secret, taken by reference and cleared by {@link #clear()}
* @param resolvedAt when the material was resolved
* @param expiresAt when it stops being valid, when known
@@ -39,11 +41,13 @@ public final class CredentialRuntime {
*/
public CredentialRuntime(
String credentialId,
Optional<String> username,
char[] material,
Instant resolvedAt,
Optional<Instant> expiresAt,
Duration rotationLead) {
CredentialIds.require(credentialId);
Objects.requireNonNull(username, "username must not be null");
Objects.requireNonNull(material, "material must not be null");
Objects.requireNonNull(resolvedAt, "resolvedAt must not be null");
Objects.requireNonNull(expiresAt, "expiresAt must not be null");
@@ -55,6 +59,7 @@ public final class CredentialRuntime {
throw new IllegalArgumentException("credential material must not be empty");
}
this.credentialId = credentialId;
this.username = username;
this.material = material;
this.resolvedAt = resolvedAt;
this.expiresAt = expiresAt;
@@ -70,6 +75,15 @@ public final class CredentialRuntime {
return credentialId;
}
/**
* Returns the user name this secret authenticates, when the mechanism sends one.
*
* @return the user name
*/
public Optional<String> username() {
return username;
}
/**
* Returns a copy of the material.
*
@@ -134,6 +134,7 @@ public final class CredentialRuntimeRegistry {
private CredentialRuntime fetch(String credentialId, Instant now) {
char[] material = provider.resolve(credentialId);
Optional<Instant> expiresAt = provider.expiresAt(credentialId);
return new CredentialRuntime(credentialId, material, now, expiresAt, rotationLead);
return new CredentialRuntime(
credentialId, provider.username(credentialId), material, now, expiresAt, rotationLead);
}
}
@@ -245,8 +245,9 @@ public class KafkaMessagingAutoConfiguration {
@ConditionalOnMissingBean(dev.caskeleton.messaging.transport.MessagingTransport.class)
public dev.caskeleton.messaging.kafka.KafkaMessagingTransport messagingKafkaTransport(
org.apache.kafka.clients.producer.Producer<byte[], byte[]> producer) {
// Generation 1: the credential generation a rotation increments. Starting elsewhere would make
// the first rotation look like it had already happened.
// Generation 1 for the life of the producer. A SASL credential rotation does not replace the
// producer: KafkaCredentialCallbackHandler reads the rotated secret at the next authentication,
// so there is no second transport generation to install.
return new dev.caskeleton.messaging.kafka.KafkaMessagingTransport("kafka", 1L, producer);
}
@@ -169,7 +169,7 @@ class MessagingLiveRoundTripQualificationTest {
* same constant: a round trip that read a different topic from the one the transport wrote would
* pass or fail for reasons unrelated to the platform.
*/
private static dev.caskeleton.messaging.policy.DestinationProfile profile(
static dev.caskeleton.messaging.policy.DestinationProfile profile(
DestinationName name, DestinationName deadLetter) {
return new dev.caskeleton.messaging.policy.DestinationProfile(
name,
@@ -198,7 +198,7 @@ class MessagingLiveRoundTripQualificationTest {
false);
}
private static MessageEnvelope<String> envelope() {
static MessageEnvelope<String> envelope() {
return new MessageEnvelope<>(
MessageId.newId(),
new MessageType("order.created"),
@@ -0,0 +1,396 @@
package dev.caskeleton.messaging.autoconfigure;
import static org.assertj.core.api.Assertions.assertThat;
import dev.caskeleton.messaging.api.MessageType;
import dev.caskeleton.messaging.api.SchemaVersion;
import dev.caskeleton.messaging.api.destination.DestinationName;
import dev.caskeleton.messaging.api.destination.MessageDestination;
import dev.caskeleton.messaging.api.publish.MessagePublisher;
import dev.caskeleton.messaging.api.publish.PublishCompletion;
import dev.caskeleton.messaging.api.publish.PublishOptions;
import dev.caskeleton.messaging.api.publish.PublishResult;
import dev.caskeleton.messaging.security.CredentialProvider;
import java.io.IOException;
import java.nio.charset.StandardCharsets;
import java.nio.file.Path;
import java.time.Instant;
import java.util.ArrayList;
import java.util.List;
import java.util.Map;
import java.util.Optional;
import java.util.UUID;
import java.util.concurrent.TimeUnit;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.condition.EnabledIf;
import org.junit.jupiter.api.io.TempDir;
import org.springframework.boot.autoconfigure.AutoConfigurations;
import org.springframework.boot.test.context.runner.ApplicationContextRunner;
import org.testcontainers.DockerClientFactory;
import org.testcontainers.kafka.KafkaContainer;
import org.testcontainers.utility.MountableFile;
/**
* A publish reaches a broker that trusts nobody by default (Review Focus 5, M-2e/NEW-4b, M-2d).
*
* <p>Both brokers present a certificate signed by a CA generated for this test, which no JVM trusts
* — so a publish is confirmed only if the configured trust store reached the client. The first
* broker then requires SASL PLAIN, and accepts only the user name the credential provider returns;
* the second requires a client certificate, which only the configured key store can present.
*
* <p>No secret is written in this file: the store and broker passwords are generated per run.
*/
@EnabledIf(value = "dockerAvailable", disabledReason = "Docker not available for the live broker")
class MessagingSecuredBrokerQualificationTest {
private static final String IMAGE = "apache/kafka:4.1.0";
private static final String BROKER_USER = "svc-orders";
private static final String BROKER_STORE = "/etc/kafka/secrets/broker.p12";
private static final String BROKER_TRUST = "/etc/kafka/secrets/truststore.p12";
static boolean dockerAvailable() {
return DockerClientFactory.instance().isDockerAvailable();
}
@Test
void aPublishIsConfirmedByAPrivateCaBrokerThatRequiresSaslAsTheProviderUser(
@TempDir Path directory) {
// given
PrivateCa ca = new PrivateCa(directory);
String userPassword = UUID.randomUUID().toString();
try (KafkaContainer kafka =
broker(ca, "SASL_SSL")
.withEnv("KAFKA_SASL_ENABLED_MECHANISMS", "PLAIN")
.withEnv(
"KAFKA_LISTENER_NAME_PLAINTEXT_PLAIN_SASL_JAAS_CONFIG",
"org.apache.kafka.common.security.plain.PlainLoginModule required user_"
+ BROKER_USER
+ "=\""
+ userPassword
+ "\";")) {
kafka.start();
ApplicationContextRunner runner =
platform(kafka, ca, Map.of("orders-producer", userPassword, "orders-consumer", "unused"))
.withPropertyValues(
"app.messaging.security.kafka.producer.type=USERNAME_PASSWORD",
"app.messaging.security.kafka.producer.credential-id=orders-producer",
"app.messaging.security.kafka.consumer.type=USERNAME_PASSWORD",
"app.messaging.security.kafka.consumer.credential-id=orders-consumer");
// when
List<PublishResult> results = new ArrayList<>();
runner.run(context -> results.add(publish(context.getBean(MessagePublisher.class))));
// then
assertThat(results)
.extracting(PublishResult::completion)
.containsExactly(PublishCompletion.CONFIRMED);
}
}
@Test
void aPublishIsConfirmedByAPrivateCaBrokerThatRequiresAClientCertificate(
@TempDir Path directory) {
// given
PrivateCa ca = new PrivateCa(directory);
Path clientStore = ca.issue("orders-producer", "dns:orders-producer");
try (KafkaContainer kafka =
broker(ca, "SSL")
.withEnv("KAFKA_SSL_CLIENT_AUTH", "required")
.withEnv("KAFKA_SSL_TRUSTSTORE_LOCATION", BROKER_TRUST)
.withEnv("KAFKA_SSL_TRUSTSTORE_TYPE", "PKCS12")
.withEnv("KAFKA_SSL_TRUSTSTORE_PASSWORD", ca.password)
.withCopyToContainer(MountableFile.forHostPath(ca.trustStore, 0644), BROKER_TRUST)) {
kafka.start();
ApplicationContextRunner runner =
platform(
kafka,
ca,
Map.of("orders-producer-tls", ca.password, "orders-consumer-tls", ca.password))
.withPropertyValues(
"app.messaging.security.kafka.producer.type=MUTUAL_TLS",
"app.messaging.security.kafka.producer.credential-id=orders-producer-tls",
"app.messaging.security.kafka.producer.key-store.location=" + clientStore,
"app.messaging.security.kafka.consumer.type=MUTUAL_TLS",
"app.messaging.security.kafka.consumer.credential-id=orders-consumer-tls",
"app.messaging.security.kafka.consumer.key-store.location=" + clientStore);
// when
List<PublishResult> results = new ArrayList<>();
runner.run(context -> results.add(publish(context.getBean(MessagePublisher.class))));
// then
assertThat(results)
.extracting(PublishResult::completion)
.containsExactly(PublishCompletion.CONFIRMED);
}
}
/**
* The Testcontainers broker with its external listener switched to a secured protocol.
*
* <p>The container names its host-facing listener {@code PLAINTEXT} and advertises it by that
* name; only the protocol that name maps to changes here.
*/
@SuppressWarnings("resource")
private static KafkaContainer broker(PrivateCa ca, String protocol) {
Path brokerStore = ca.issue("broker", ca.brokerSubjectAlternativeNames());
return new KafkaContainer(IMAGE)
.withEnv(
"KAFKA_LISTENER_SECURITY_PROTOCOL_MAP",
"BROKER:PLAINTEXT,PLAINTEXT:" + protocol + ",CONTROLLER:PLAINTEXT")
.withEnv("KAFKA_SSL_KEYSTORE_LOCATION", BROKER_STORE)
.withEnv("KAFKA_SSL_KEYSTORE_TYPE", "PKCS12")
.withEnv("KAFKA_SSL_KEYSTORE_PASSWORD", ca.password)
.withEnv("KAFKA_SSL_KEY_PASSWORD", ca.password)
.withCopyToContainer(MountableFile.forHostPath(brokerStore, 0644), BROKER_STORE);
}
/** The assembled platform, pointed at the broker through the private CA's trust store. */
private static ApplicationContextRunner platform(
KafkaContainer kafka, PrivateCa ca, Map<String, String> secrets) {
Map<String, String> all = new java.util.HashMap<>(secrets);
all.put("kafka-trust", ca.password);
return new ApplicationContextRunner()
.withConfiguration(AutoConfigurations.of(MessagingPlatformRootAutoConfiguration.class))
.withPropertyValues(
"app.messaging.enabled=true",
"app.messaging.broker=kafka",
"spring.kafka.bootstrap-servers=" + kafka.getBootstrapServers(),
"app.messaging.brokers.kafka.type=kafka",
"app.messaging.brokers.kafka.bootstrap-servers[0]=" + kafka.getBootstrapServers(),
"app.messaging.brokers.kafka.tls-enabled=true",
"app.messaging.brokers.kafka.authentication-enabled=true",
"app.messaging.security.kafka.trust-store.location=" + ca.trustStore,
"app.messaging.security.kafka.trust-store.password-credential-id=kafka-trust",
"app.messaging.security.kafka.access.publishable[0]=order-events")
.withBean(CredentialProvider.class, () -> provider(all))
.withBean(
MessageContracts.class,
() ->
new MessageContracts(
Map.of(
new dev.caskeleton.messaging.schema.MessageContractKey(
new MessageType("order.created"), new SchemaVersion(1)),
String.class)))
.withBean(
"ordersProfile",
dev.caskeleton.messaging.policy.DestinationProfile.class,
() ->
MessagingLiveRoundTripQualificationTest.profile(
new DestinationName("order-events"), new DestinationName("order-events-dlq")))
.withBean(
"ordersDlqProfile",
dev.caskeleton.messaging.policy.DestinationProfile.class,
() ->
MessagingLiveRoundTripQualificationTest.profile(
new DestinationName("order-events-dlq"), null));
}
private static PublishResult publish(MessagePublisher publisher) {
return publisher
.publish(
new MessageDestination<>(
new DestinationName("order-events"),
new MessageType("order.created"),
String.class),
MessagingLiveRoundTripQualificationTest.envelope(),
PublishOptions.defaults())
.toCompletableFuture()
.join();
}
private static CredentialProvider provider(Map<String, String> secrets) {
return new CredentialProvider() {
@Override
public char[] resolve(String credentialId) {
return secrets.get(credentialId).toCharArray();
}
@Override
public Optional<Instant> expiresAt(String credentialId) {
return Optional.empty();
}
@Override
public Optional<String> username(String credentialId) {
return Optional.of(BROKER_USER);
}
};
}
/**
* A certificate authority made for one test run with the JDK's {@code keytool}.
*
* <p>Every store shares one generated password, which is also what the credential provider hands
* the client for the trust store and the client key store.
*/
private static final class PrivateCa {
private final Path directory;
private final String password = UUID.randomUUID().toString();
private final Path authority;
private final Path certificate;
private final Path trustStore;
private PrivateCa(Path directory) {
this.directory = directory;
this.authority = directory.resolve("ca.p12");
this.certificate = directory.resolve("ca.pem");
this.trustStore = directory.resolve("truststore.p12");
keytool(
"-genkeypair",
"-alias",
"ca",
"-keyalg",
"RSA",
"-keysize",
"2048",
"-validity",
"2",
"-dname",
"CN=messaging-test-ca",
"-ext",
"bc:c",
"-keystore",
authority.toString(),
"-storetype",
"PKCS12",
"-storepass",
password);
keytool(
"-exportcert",
"-rfc",
"-alias",
"ca",
"-keystore",
authority.toString(),
"-storepass",
password,
"-file",
certificate.toString());
keytool(
"-importcert",
"-noprompt",
"-alias",
"ca",
"-file",
certificate.toString(),
"-keystore",
trustStore.toString(),
"-storetype",
"PKCS12",
"-storepass",
password);
}
/** The names the client may reach the broker by, so hostname verification can pass. */
private String brokerSubjectAlternativeNames() {
String host = DockerClientFactory.instance().dockerHostIpAddress();
String dockerHost =
host.chars().allMatch(c -> Character.isDigit(c) || c == '.')
? "ip:" + host
: "dns:" + host;
return "dns:localhost,ip:127.0.0.1," + dockerHost;
}
/** Issues a key store holding a CA-signed certificate for {@code name}. */
private Path issue(String name, String subjectAlternativeNames) {
Path store = directory.resolve(name + ".p12");
Path request = directory.resolve(name + ".csr");
Path signed = directory.resolve(name + ".pem");
keytool(
"-genkeypair",
"-alias",
name,
"-keyalg",
"RSA",
"-keysize",
"2048",
"-validity",
"2",
"-dname",
"CN=" + name,
"-keystore",
store.toString(),
"-storetype",
"PKCS12",
"-storepass",
password);
keytool(
"-certreq",
"-alias",
name,
"-keystore",
store.toString(),
"-storepass",
password,
"-file",
request.toString());
keytool(
"-gencert",
"-rfc",
"-alias",
"ca",
"-keystore",
authority.toString(),
"-storepass",
password,
"-infile",
request.toString(),
"-outfile",
signed.toString(),
"-validity",
"2",
"-ext",
"SAN=" + subjectAlternativeNames);
keytool(
"-importcert",
"-noprompt",
"-alias",
"ca",
"-file",
certificate.toString(),
"-keystore",
store.toString(),
"-storepass",
password);
keytool(
"-importcert",
"-alias",
name,
"-file",
signed.toString(),
"-keystore",
store.toString(),
"-storepass",
password);
return store;
}
private static void keytool(String... arguments) {
List<String> command = new ArrayList<>();
command.add(Path.of(System.getProperty("java.home"), "bin", "keytool").toString());
command.addAll(List.of(arguments));
try {
Process process = new ProcessBuilder(command).redirectErrorStream(true).start();
String output = new String(process.getInputStream().readAllBytes(), StandardCharsets.UTF_8);
if (!process.waitFor(60, TimeUnit.SECONDS) || process.exitValue() != 0) {
throw new IllegalStateException("keytool " + arguments[0] + " failed: " + output);
}
} catch (IOException failure) {
throw new IllegalStateException("keytool could not run", failure);
} catch (InterruptedException interrupted) {
Thread.currentThread().interrupt();
throw new IllegalStateException("keytool was interrupted", interrupted);
}
}
}
}
@@ -8,7 +8,9 @@ import org.springframework.boot.context.properties.bind.DefaultValue;
*
* <p>No field here holds a secret, and that is a structural rule rather than a convention: the
* platform stores a credential id and resolves the material through a {@code CredentialProvider} at
* connect time, so a configuration dump or a heap dump yields a name and nothing usable.
* connect time, so a configuration dump or a heap dump yields a name and nothing usable. Key and
* trust store passwords follow the same rule — the stores are named by path, their passwords by
* credential id.
*
* <p>The admin credential is bindable and its presence on a production broker is a startup failure.
* Leaving it unbindable would have been the safer-looking choice and the wrong one: an operator who
@@ -19,6 +21,7 @@ import org.springframework.boot.context.properties.bind.DefaultValue;
* @param consumer the consumer credential
* @param admin the admin credential, which an application runtime must not hold in production
* @param hostnameVerification whether the server certificate hostname is verified
* @param trustStore the certificates the connection trusts; absent means the JVM default store
* @param access which destinations each role may touch
*/
public record BrokerSecuritySettings(
@@ -26,6 +29,7 @@ public record BrokerSecuritySettings(
Credential consumer,
Credential admin,
@DefaultValue("true") boolean hostnameVerification,
TrustStore trustStore,
@DefaultValue Access access) {
/**
@@ -33,12 +37,35 @@ public record BrokerSecuritySettings(
*
* @param type the authentication mechanism
* @param credentialId the identifier the credential provider resolves
* @param keyStore the client certificate, for {@code MUTUAL_TLS} and nothing else
*/
public record Credential(String type, String credentialId) {}
public record Credential(String type, String credentialId, KeyStore keyStore) {}
/**
* A client certificate store, unlocked by the credential it belongs to.
*
* @param location the key store file
* @param type the key store format
*/
public record KeyStore(String location, @DefaultValue("PKCS12") String type) {}
/**
* The certificates the connection trusts.
*
* @param location the trust store file
* @param type the trust store format
* @param passwordCredentialId the credential that unlocks the store, when it is locked
*/
public record TrustStore(
String location, @DefaultValue("PKCS12") String type, String passwordCredentialId) {}
/**
* Which destinations each role may touch.
*
* <p>{@code publishable} is enforced: once a broker has a security entry, the platform publishes
* only to the declared destinations of that broker that this list names. An empty list therefore
* publishes nothing.
*
* @param publishable destinations the producer credential may publish to
* @param consumable destinations the consumer credential may consume
* @param administrable destinations the admin credential may operate on
@@ -14,6 +14,7 @@ import dev.caskeleton.messaging.policy.RetryPolicy;
import dev.caskeleton.messaging.policy.SchemaPolicy;
import dev.caskeleton.messaging.security.BrokerCredentialProfile;
import dev.caskeleton.messaging.security.BrokerSecurityProfile;
import dev.caskeleton.messaging.security.BrokerTrustStore;
import dev.caskeleton.messaging.security.DestinationAccessPolicy;
import java.util.ArrayList;
import java.util.LinkedHashSet;
@@ -235,7 +236,22 @@ public final class MessagingConfigurationCompiler {
new DestinationAccessPolicy(
destinationNames(security.access().publishable()),
destinationNames(security.access().consumable()),
destinationNames(security.access().administrable())));
destinationNames(security.access().administrable())),
Optional.ofNullable(security.trustStore()).map(store -> trustStore(name, store)));
}
private static BrokerTrustStore trustStore(
String broker, BrokerSecuritySettings.TrustStore store) {
if (blankToNull(store.location()) == null) {
throw configurationError(
key("security", broker, "trust-store.location"),
"a trust store entry needs the file it names; without one the JVM default store is used,"
+ " which is what leaving the entry out already says");
}
return new BrokerTrustStore(
store.location(),
store.type(),
Optional.ofNullable(blankToNull(store.passwordCredentialId())));
}
private static BrokerCredentialProfile credential(
@@ -252,10 +268,25 @@ public final class MessagingConfigurationCompiler {
}
String type = credential.type() == null ? "" : credential.type().toUpperCase(Locale.ROOT);
String id = credential.credentialId();
BrokerSecuritySettings.KeyStore keyStore = credential.keyStore();
boolean hasKeyStore = keyStore != null && blankToNull(keyStore.location()) != null;
if ("MUTUAL_TLS".equals(type) && !hasKeyStore) {
throw configurationError(
key("security", broker, role + ".key-store"),
"MUTUAL_TLS presents a client certificate, and without a key store there is none to"
+ " present; the broker refuses the handshake");
}
if (!"MUTUAL_TLS".equals(type) && keyStore != null) {
throw configurationError(
key("security", broker, role + ".key-store"),
"only a MUTUAL_TLS credential presents a key store; a %s credential would ignore it"
.formatted(credential.type()));
}
return switch (type) {
case "SASL_SCRAM" -> new BrokerCredentialProfile.SaslScram(id);
case "OAUTH2" -> new BrokerCredentialProfile.OAuth2(id);
case "MUTUAL_TLS" -> new BrokerCredentialProfile.MutualTls(id);
case "MUTUAL_TLS" ->
new BrokerCredentialProfile.MutualTls(id, keyStore.location(), keyStore.type());
case "USERNAME_PASSWORD" -> new BrokerCredentialProfile.UsernamePassword(id);
case "NKEY" -> new BrokerCredentialProfile.Nkey(id);
default ->
@@ -408,14 +408,48 @@ public class MessagingCoreAutoConfiguration {
* Declaring a destination profile is already an act of configuration, so a message to one nobody
* declared is a typo or a module reaching past its contract — which is what this check is for.
*
* <p>A broker with a security entry narrows that further to its {@code access.publishable} list.
* The list was compiled and then read by nothing, so a destination the security entry left out
* was published to anyway.
*
* @param destinations the validated registry
* @param compiled the security profiles compiled from configuration
* @param declared the security profiles the application declared as beans
* @return the access policy
*/
@Bean
@ConditionalOnMissingBean
public dev.caskeleton.messaging.security.DestinationAccessPolicy messagingDestinationAccess(
ValidatedDestinationRegistry destinations) {
return dev.caskeleton.messaging.runtime.DeclaredDestinationAccess.of(destinations.all());
ValidatedDestinationRegistry destinations,
CompiledMessagingConfiguration compiled,
ObjectProvider<dev.caskeleton.messaging.security.BrokerSecurityProfile> declared) {
java.util.Map<String, dev.caskeleton.messaging.security.DestinationAccessPolicy> byBroker =
new java.util.HashMap<>();
java.util.stream.Stream.concat(compiled.security().stream(), declared.orderedStream())
.forEach(
profile -> {
if (byBroker.putIfAbsent(profile.broker(), profile.access()) != null) {
throw new IllegalStateException(
"broker '"
+ profile.broker()
+ "' has more than one security profile; its access list is ambiguous");
}
});
java.util.Map<
dev.caskeleton.messaging.api.destination.DestinationName,
dev.caskeleton.messaging.policy.DestinationProfile>
permitted = new java.util.HashMap<>();
destinations
.all()
.forEach(
(name, profile) -> {
dev.caskeleton.messaging.security.DestinationAccessPolicy acl =
byBroker.get(profile.broker());
if (acl == null || acl.mayPublish(name)) {
permitted.put(name, profile);
}
});
return dev.caskeleton.messaging.runtime.DeclaredDestinationAccess.of(permitted);
}
/**
@@ -0,0 +1,103 @@
package dev.caskeleton.messaging.autoconfigure;
import static org.assertj.core.api.Assertions.assertThat;
import dev.caskeleton.messaging.api.ContentType;
import dev.caskeleton.messaging.api.MessageEnvelope;
import dev.caskeleton.messaging.api.MessageId;
import dev.caskeleton.messaging.api.MessageType;
import dev.caskeleton.messaging.api.ProducerId;
import dev.caskeleton.messaging.api.SchemaVersion;
import dev.caskeleton.messaging.api.TraceContext;
import dev.caskeleton.messaging.api.destination.DestinationName;
import dev.caskeleton.messaging.api.destination.MessageDestination;
import dev.caskeleton.messaging.api.error.FailureDescriptor;
import dev.caskeleton.messaging.api.header.MessageHeaders;
import dev.caskeleton.messaging.api.publish.MessagePublisher;
import dev.caskeleton.messaging.api.publish.PublishOptions;
import dev.caskeleton.messaging.api.publish.PublishResult;
import java.time.Instant;
import java.util.Optional;
import org.junit.jupiter.api.Test;
import org.springframework.boot.autoconfigure.AutoConfigurations;
import org.springframework.boot.test.context.runner.ApplicationContextRunner;
/**
* The broker's {@code access.publishable} list is enforced by the publish path, not only compiled.
*
* <p>The list was bound, validated and turned into a {@code DestinationAccessPolicy} that nothing
* read: the publisher's access check was built from the declared destinations alone, so a
* destination the security entry left out was published to anyway.
*/
class BrokerAccessListTest {
private final ApplicationContextRunner runner =
new ApplicationContextRunner()
.withConfiguration(AutoConfigurations.of(MessagingPlatformRootAutoConfiguration.class))
.withUserConfiguration(KafkaProducerSecurityTest.Credentials.class)
.withPropertyValues(
"app.messaging.enabled=true",
"app.messaging.broker=kafka",
"spring.kafka.bootstrap-servers=localhost:9092",
"app.messaging.brokers.kafka-primary.type=kafka",
"app.messaging.brokers.kafka-primary.bootstrap-servers[0]=localhost:9092",
"app.messaging.security.kafka-primary.producer.type=SASL_SCRAM",
"app.messaging.security.kafka-primary.producer.credential-id=kafka-producer",
"app.messaging.security.kafka-primary.consumer.type=SASL_SCRAM",
"app.messaging.security.kafka-primary.consumer.credential-id=kafka-consumer",
"app.messaging.security.kafka-primary.access.publishable[0]=order-events")
.withPropertyValues(destination("order-events"))
.withPropertyValues(destination("audit-events"));
@Test
void aDeclaredDestinationTheSecurityEntryDoesNotListIsRejectedAsForbidden() {
runner.run(
context -> {
// given
MessagePublisher publisher = context.getBean(MessagePublisher.class);
// when
PublishResult result = publish(publisher, "audit-events");
// then
assertThat(result.failure().map(FailureDescriptor::code)).contains("PUBLISH_FORBIDDEN");
});
}
private static PublishResult publish(MessagePublisher publisher, String destination) {
return publisher
.publish(
new MessageDestination<>(
new DestinationName(destination), new MessageType("order.created"), String.class),
new MessageEnvelope<>(
MessageId.newId(),
new MessageType("order.created"),
new SchemaVersion(1),
Instant.parse("2026-09-25T00:00:00Z"),
Optional.empty(),
new ProducerId("orders"),
Optional.empty(),
Optional.empty(),
ContentType.JSON,
Optional.empty(),
Optional.empty(),
Optional.empty(),
TraceContext.none(),
MessageHeaders.empty(),
"{}"),
PublishOptions.defaults())
.toCompletableFuture()
.join();
}
private static String[] destination(String name) {
String prefix = "app.messaging.destinations." + name + ".";
return new String[] {
prefix + "broker=kafka-primary",
prefix + "kind=EVENT_STREAM",
prefix + "physical.topic=" + name,
prefix + "schema.message-types[0]=order.created",
prefix + "guarantees.delivery=AT_MOST_ONCE",
};
}
}
@@ -6,8 +6,14 @@ import dev.caskeleton.messaging.kafka.KafkaSecurityConfigurer;
import dev.caskeleton.messaging.security.BrokerSecurityProfile;
import dev.caskeleton.messaging.security.CredentialProvider;
import java.time.Instant;
import java.util.List;
import java.util.Map;
import java.util.Optional;
import java.util.concurrent.atomic.AtomicReference;
import javax.security.auth.callback.Callback;
import javax.security.auth.callback.NameCallback;
import javax.security.auth.callback.PasswordCallback;
import org.apache.kafka.common.security.auth.AuthenticateCallbackHandler;
import org.junit.jupiter.api.Test;
import org.springframework.boot.autoconfigure.AutoConfigurations;
import org.springframework.boot.test.context.assertj.AssertableApplicationContext;
@@ -44,6 +50,16 @@ class KafkaProducerSecurityTest {
"app.messaging.security.kafka-primary.consumer.credential-id=kafka-consumer",
};
private static final String[] MUTUAL_TLS_CREDENTIALS = {
"app.messaging.brokers.kafka-primary.tls-enabled=true",
"app.messaging.security.kafka-primary.producer.type=MUTUAL_TLS",
"app.messaging.security.kafka-primary.producer.credential-id=kafka-producer-tls",
"app.messaging.security.kafka-primary.producer.key-store.location=/etc/kafka/producer.p12",
"app.messaging.security.kafka-primary.consumer.type=MUTUAL_TLS",
"app.messaging.security.kafka-primary.consumer.credential-id=kafka-consumer-tls",
"app.messaging.security.kafka-primary.consumer.key-store.location=/etc/kafka/consumer.p12",
};
@Test
void aTlsBrokerWithScramCredentialsConfiguresTheProducerForSaslSsl() {
runner
@@ -81,16 +97,107 @@ class KafkaProducerSecurityTest {
}
@Test
void theProducerLogsInWithTheProducerCredentialResolvedFromTheProvider() {
void theProducerAuthenticatesAsTheUsernameTheProviderReturnsNotAsTheCredentialId() {
runner
.withUserConfiguration(Credentials.class)
.withPropertyValues(SCRAM_CREDENTIALS)
.withPropertyValues("app.messaging.brokers.kafka-primary.tls-enabled=true")
.run(
context -> {
// given
Map<String, Object> config = producerConfig(context);
// when
Answer answer = authenticate(config);
// then
assertThat(answer).isEqualTo(new Answer(Credentials.USERNAME, Credentials.SECRET));
});
}
@Test
void aRotatedPasswordReachesTheNextAuthenticationWithoutRebuildingTheProducer() {
runner
.withUserConfiguration(RotatingCredentials.class)
.withPropertyValues(SCRAM_CREDENTIALS)
.withPropertyValues("app.messaging.brokers.kafka-primary.tls-enabled=true")
.run(
context -> {
// given
Map<String, Object> config = producerConfig(context);
RotatingCredentials.CURRENT.set("rotated-secret");
// when
Answer answer = authenticate(config);
// then
assertThat(answer.password()).isEqualTo("rotated-secret");
});
}
@Test
void aPrivateCaTrustStoreReachesTheProducer() {
runner
.withUserConfiguration(Credentials.class, StoreFilesAbsent.class)
.withPropertyValues(SCRAM_CREDENTIALS)
.withPropertyValues(
"app.messaging.brokers.kafka-primary.tls-enabled=true",
"app.messaging.security.kafka-primary.trust-store.location=/etc/kafka/private-ca.p12",
"app.messaging.security.kafka-primary.trust-store.password-credential-id=kafka-trust")
.run(
context ->
assertThat(String.valueOf(producerConfig(context).get("sasl.jaas.config")))
.startsWith("org.apache.kafka.common.security.scram.ScramLoginModule required")
.contains("username=\"kafka-producer\"", "password=\"" + Credentials.SECRET));
assertThat(producerConfig(context))
.extractingByKeys(
"ssl.truststore.location", "ssl.truststore.type", "ssl.truststore.password")
.containsExactly("/etc/kafka/private-ca.p12", "PKCS12", Credentials.SECRET));
}
@Test
void mutualTlsPresentsTheProducerKeyStoreUnlockedByItsCredential() {
runner
.withUserConfiguration(Credentials.class, StoreFilesAbsent.class)
.withPropertyValues(MUTUAL_TLS_CREDENTIALS)
.run(
context ->
assertThat(producerConfig(context))
.extractingByKeys(
"security.protocol",
"ssl.keystore.location",
"ssl.keystore.type",
"ssl.keystore.password")
.containsExactly(
"SSL", "/etc/kafka/producer.p12", "PKCS12", Credentials.SECRET));
}
@Test
void mutualTlsWithoutAKeyStoreFailsStartupNamingTheKey() {
runner
.withUserConfiguration(Credentials.class)
.withPropertyValues(MUTUAL_TLS_CREDENTIALS)
.withPropertyValues("app.messaging.security.kafka-primary.producer.key-store.location=")
.run(
context ->
assertThat(context)
.hasFailed()
.getFailure()
.as("a client certificate nobody configured cannot be presented")
.hasMessageContaining(
"app.messaging.security.kafka-primary.producer.key-store: MUTUAL_TLS"));
}
@Test
void mutualTlsOnABrokerWithoutTlsFailsStartup() {
runner
.withUserConfiguration(Credentials.class)
.withPropertyValues(MUTUAL_TLS_CREDENTIALS)
.withPropertyValues("app.messaging.brokers.kafka-primary.tls-enabled=false")
.run(
context ->
assertThat(context)
.hasFailed()
.getFailure()
.as("a certificate only authenticates inside a TLS handshake")
.hasMessageContaining("authenticates with a client certificate but does not"));
}
@Test
@@ -127,12 +234,34 @@ class KafkaProducerSecurityTest {
context.getBeanProvider(KafkaSecurityConfigurer.class));
}
/** A credential source that answers every id with the same test secret. */
/** What a SASL client is told when it asks for a name and a password. */
private record Answer(String name, String password) {}
/**
* Authenticates the way Kafka's SASL client does: instantiate the configured handler class,
* configure it with the client's own configuration, then ask it for a name and a password.
*/
private static Answer authenticate(Map<String, Object> config) throws Exception {
assertThat(config).containsKey("sasl.client.callback.handler.class");
Class<?> type = (Class<?>) config.get("sasl.client.callback.handler.class");
AuthenticateCallbackHandler handler =
(AuthenticateCallbackHandler) type.getDeclaredConstructor().newInstance();
handler.configure(config, String.valueOf(config.get("sasl.mechanism")), List.of());
NameCallback name = new NameCallback("name");
PasswordCallback password = new PasswordCallback("password", false);
handler.handle(new Callback[] {name, password});
handler.close();
return new Answer(name.getName(), new String(password.getPassword()));
}
/** A credential source that answers every id with the same test principal and secret. */
@Configuration(proxyBeanMethods = false)
static class Credentials {
static final String SECRET = "test-only-secret";
static final String USERNAME = "svc-orders";
@Bean
CredentialProvider credentialProvider() {
return new CredentialProvider() {
@@ -145,6 +274,55 @@ class KafkaProducerSecurityTest {
public Optional<Instant> expiresAt(String credentialId) {
return Optional.empty();
}
@Override
public Optional<String> username(String credentialId) {
return Optional.of(USERNAME);
}
};
}
}
/**
* Stands in for the producer when the store paths in a test name no real file.
*
* <p>A real producer opens its key and trust stores while it is constructed, so a path that does
* not exist fails startup — which is the behaviour a deployment wants, and not what these tests
* are about. They read the configuration the producer would have been built from.
*/
@Configuration(proxyBeanMethods = false)
static class StoreFilesAbsent {
@Bean
org.apache.kafka.clients.producer.Producer<byte[], byte[]> messagingKafkaProducer() {
return new org.apache.kafka.clients.producer.MockProducer<>();
}
}
/** A credential store whose secret has already expired, so every resolution reads it again. */
@Configuration(proxyBeanMethods = false)
static class RotatingCredentials {
static final AtomicReference<String> CURRENT = new AtomicReference<>("first-secret");
@Bean
CredentialProvider credentialProvider() {
CURRENT.set("first-secret");
return new CredentialProvider() {
@Override
public char[] resolve(String credentialId) {
return CURRENT.get().toCharArray();
}
@Override
public Optional<Instant> expiresAt(String credentialId) {
return Optional.of(Instant.EPOCH);
}
@Override
public Optional<String> username(String credentialId) {
return Optional.of(Credentials.USERNAME);
}
};
}
}