Skip to content

Commit 51cd333

Browse files
github-actions[bot]upload-charms-docs-bot
github-actions[bot]
and
upload-charms-docs-bot
authored
[DPE-5555;DPE-5262;DPE-5220] Security, Backup/Restore and K8s expose documentation (#138)
Co-authored-by: upload-charms-docs-bot <[email protected]>
1 parent f4d9336 commit 51cd333

9 files changed

+938
-5
lines changed

‎docs/explanation/e-cluster-configuration.md

+19
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,19 @@
1+
# Overview of a cluster configuration content
2+
3+
[Apache Kafka](https://kafka.apache.org) is an open-source distributed event streaming platform that requires an external solution to coordinate and sync metadata between all active brokers.
4+
One of such solutions is [ZooKeeper](https://zookeeper.apache.org).
5+
6+
Here are some of the responsibilities of ZooKeeper in a Kafka cluster:
7+
8+
- **Cluster membership**: through regular heartbeats, it keeps tracks of the brokers entering and leaving the cluster, providing an up-to-date list of brokers.
9+
- **Controller election**: one of the Kafka brokers is responsible for managing the leader/follower status for all the partitions. ZooKeeper is used to elect a controller and to make sure there is only one of it.
10+
- **Topic configuration**: each topic can be replicated on multiple partitions. ZooKeeper keeps track of the locations of the partitions and replicas, so that high-availability is still attained when a broker shuts down. Topic-specific configuration overrides (e.g. message retention and size) are also stored in ZooKeeper.
11+
- **Access control and authentication**: ZooKeeper stores access control lists (ACL) for Kafka resources, to ensure only the proper, authorized, users or groups can read or write on each topic.
12+
13+
The values for the configuration parameters mentioned above are stored in znodes, the hierarchical unit data structure in ZooKeeper.
14+
A znode is represented by its path and can both have data associated with it and children nodes.
15+
ZooKeeper clients interact with its data structure similarly to a remote file system that would be sync-ed between the ZooKeeper units for high availability.
16+
For a Charmed Kafka K8s related to a Charmed ZooKeeper K8s:
17+
- the list of the broker ids of the cluster can be found in `/kafka-k8s/brokers/ids`
18+
- the endpoint used to access the broker with id `0` can be found in `/kafka-k8s/brokers/ids/0`
19+
- the credentials for the Charmed Kafka K8s users can be found in `/kafka-k8s/config/users`

‎docs/explanation/e-hardening.md

+134
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,134 @@
1+
# Security hardening guide
2+
3+
This document provides guidance and instructions to achieve
4+
a secure deployment of [Charmed Kafka K8s](https://github.com/canonical/kafka-k8s-bundle), including setting up and managing a secure environment.
5+
The document is divided into the following sections:
6+
7+
1. Environment, outlining the recommendation for deploying a secure environment
8+
2. Applications, outlining the product features that enable a secure deployment of a Kafka cluster
9+
3. Additional resources, providing any further information about security and compliance
10+
11+
## Environment
12+
13+
The environment where applications operate can be divided in two components:
14+
15+
1. Kubernetes
16+
2. Juju
17+
18+
### Kubernetes
19+
20+
Charmed Spark can be deployed on top of several Kubernetes distributions.
21+
The following table provides references for the security documentation for the
22+
main supported cloud platforms.
23+
24+
| Cloud | Security guide |
25+
|--------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
26+
| Charmed Kubernetes | [Security in Charmed Kubernetes](https://ubuntu.com/kubernetes/docs/security) |
27+
| AWS EKS | [Best Practices for Security, Identity and Compliance](https://aws.amazon.com/architecture/security-identity-compliance), [AWS security credentials](https://docs.aws.amazon.com/IAM/latest/UserGuide/security-creds.html#access-keys-and-secret-access-keys), [Security in EKS](https://docs.aws.amazon.com/eks/latest/userguide/security.html) |
28+
| Azure | [Azure security best practices and patterns](https://learn.microsoft.com/en-us/azure/security/fundamentals/best-practices-and-patterns), [Managed identities for Azure resource](https://learn.microsoft.com/en-us/entra/identity/managed-identities-azure-resources/), [Security in AKS](https://learn.microsoft.com/en-us/azure/aks/concepts-security) |
29+
30+
### Juju
31+
32+
Juju is the component responsible for orchestrating the entire lifecycle, from deployment to Day 2 operations, of
33+
all applications. Therefore, Juju must be set up securely. For more information, see the
34+
[Juju security page](/t/juju-security/15684) and the [How to harden your deployment guide](https://juju.is/docs/juju/harden-your-deployment).
35+
36+
#### Cloud credentials
37+
38+
When configuring the cloud credentials to be used with Juju, ensure that the users have correct permissions to operate at the required level on the Kubernetes cluster.
39+
Juju superusers responsible for bootstrapping and managing controllers require elevated permissions to manage several kind of resources. For this reason, the
40+
K8s user used for bootstrapping and managing the deployments should have full permissions, such as:
41+
42+
* create, delete, patch, and list:
43+
* namespaces
44+
* services
45+
* deployments
46+
* stateful sets
47+
* pods
48+
* PVCs
49+
50+
In general, it is common practice to run Juju using the admin role of K8s, to have full permissions on the Kubernetes cluster.
51+
52+
#### Juju users
53+
54+
It is very important that juju users are set up with minimal permissions depending on the scope of their operations.
55+
Please refer to the [User access levels](https://juju.is/docs/juju/user-permissions) documentation for more information on the access level and corresponding abilities
56+
that the different users can be granted.
57+
58+
Juju user credentials must be stored securely and rotated regularly to limit the chances of unauthorized access due to credentials leakage.
59+
60+
## Applications
61+
62+
In the following we provide guidance on how to harden your deployment using:
63+
64+
1. Base images
65+
2. Charmed operator security upgrades
66+
3. Encryption
67+
4. Authentication
68+
5. Monitoring and auditing
69+
70+
### Base images
71+
72+
Charmed Kafka K8s and Charmed ZooKeeper K8s run on top of rockcraft-based image shipping the Apache Kafka and Apache ZooKeeper
73+
distribution binaries built by Canonical, and available in the [Kafka release page](https://launchpad.net/kafka-releases) and
74+
[ZooKeeper release page](https://launchpad.net/zookeeper-releases), respectively. Both images are based on Ubuntu 22.04.
75+
The images that can be found in the [Charmed Kafka rock](https://github.com/canonical/charmed-spark-rock) and
76+
[Charmed ZooKeeper rock](https://github.com/canonical/charmed-zookeeper-rock) Github repositories are used as the base
77+
images for the different pods providing Kafka and ZooKeeper services.
78+
The following table summarise the relation between the component and its underlying base image.
79+
80+
| Component | Image |
81+
|-------------------|-------------------------------------------------------------------------------------------------------|
82+
| Charmed Kafka | [`charmed-kafka`](https://github.com/orgs/canonical/packages/container/package/charmed-kafka) |
83+
| Charmed ZooKeeper | [`charmed-zookeeper`](https://github.com/orgs/canonical/packages/container/package/charmed-zookeeper) |
84+
85+
New versions of Charmed Kafka and Charmed ZooKeeper images may be released to provide patching of vulnerabilities (CVEs).
86+
87+
### Charmed operator security upgrades
88+
89+
Charmed Kafka K8s and Charmed ZooKeeper K8s operators install a pinned revision of the images outlined in the previous table
90+
in order to provide reproducible and secure environments.
91+
New versions of Charmed Kafka K8s and Charmed ZooKeeper K8s operators may therefore be released to provide patching of vulnerabilities (CVEs).
92+
It is important to refresh the charm regularly to make sure the workload is as secure as possible.
93+
For more information on how to refresh the charm, see the [how-to refresh](/t/charmed-kafka-k8s-documentation-how-to-upgrade/13267) user guide.
94+
95+
### Wire Encryption
96+
97+
In order to deploy a hardened solution, Charmed Kafka K8s must be deployed with wire encryption enabled.
98+
To do that, you need to relate Kafka and ZooKeeper charms to one of the TLS certificate operator charms.
99+
Please refer to the [Charming Security page](https://charmhub.io/topics/security-with-x-509-certificates) for more information on how to select the right certificate
100+
provider for your use-case.
101+
102+
For more information on encryption setup, see the [How to enable encryption](/t/charmed-kafka-k8s-how-to-enable-encryption/10289) guide.
103+
104+
### Authentication
105+
106+
Charmed Kafka supports the following authentication layers:
107+
108+
1. [SCRAM-based SASL Authentication](/t/charmed-kafka-k8s-how-to-manage-app/10293)
109+
2. [certificate-base Authentication (mTLS)](/t/create-mtls-client-credentials/11079)
110+
3. OAuth Authentication using [Hydra](/t/how-to-connect-to-kafka-using-hydra-as-oidc-provider/14610) or [Google](/t/how-to-connect-to-kafka-using-google-as-oidc-provider/14611)
111+
112+
Each combination of authentication scheme and encryption is associated to the dedicated listener and it maps to a well-defined port.
113+
Please refer to the [listener reference documentation](/t/charmed-kafka-k8s-documentation-reference-listeners/13270) for more information.
114+
115+
### Monitoring and auditing
116+
117+
Charmed Kafka provides native integration with the [Canonical Observability Stack (COS)](https://charmhub.io/topics/canonical-observability-stack).
118+
To reduce the blast radius of infrastructure disruptions, the general recommendation is to deploy COS and the observed application into
119+
separate environments, isolated one another. Refer to the [COS production deployments best practices](https://charmhub.io/topics/canonical-observability-stack/reference/best-practices)
120+
for more information.
121+
122+
Refer to How-To user guide for more information on:
123+
* [how to integrate the Charmed Kafka deployment with COS](/t/charmed-kafka-k8s-how-to-enable-monitoring/10291)
124+
* [how to customise the alerting rules and dashboards](/t/charmed-kafka-k8s-documentation-how-to-integrate-custom-alerting-rules-and-dashboards/13528)
125+
126+
External user access to Kafka is logged to the `kafka-authorizer.log` that is pushes to [Loki endpoint](https://charmhub.io/loki-k8s) and exposed via [Grafana](https://charmhub.io/grafana), both components being part of the COS stack.
127+
Access denials are logged at INFO level, whereas allowed accesses are logged at DEBUG level. Depending on the auditing needs,
128+
customize the logging level either for all logs via the [`log_level`](https://charmhub.io/kafka-k8s/configurations?channel=3/stable#log_level) config option or
129+
only tune the logging level of the `authorizerAppender` in the `log4j.properties` file. Refer to the Reference documentation, for more information about
130+
the [file system paths](/t/charmed-kafka-k8s-documentation-reference-file-system-paths/13269).
131+
132+
## Additional Resources
133+
134+
For more information and details on the security and cryptography used by Charmed Kafka, see the [Security Explanation page](/t/charmed-kafka-k8s-documentation-explanation-security/15715).

‎docs/explanation/e-security.md

+108
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,108 @@
1+
# Cryptography
2+
3+
This document describes cryptography used by Charmed Kafka K8s.
4+
5+
## Resource checksums
6+
7+
Every version of the Charmed Kafka K8s and Charmed ZooKeeper K8s operators employs a pinned revision of the Charmed Kafka rock
8+
and Charmed ZooKeeper rock, respectively, to
9+
provide reproducible and secure environments.
10+
The [Charmed Kafka rock](https://github.com/canonical/charmed-kafka-rock/pkgs/container/charmed-kafka) and
11+
[Charmed ZooKeeper rock](https://github.com/canonical/charmed-zookeeper-rock/pkgs/container/charmed-zookeeper) are OCI images
12+
derived by the Charmed Kafka snap and Charmed ZooKeeper snap, that package the Kafka and ZooKeeper workload together with
13+
a set of dependencies and utilities required by the lifecycle of the operators (see [Charmed Kafka snap contents](https://github.com/canonical/charmed-kafka-snap/blob/3/edge/snap/snapcraft.yaml) and [Charmed ZooKeeper snap contents](https://github.com/canonical/charmed-zookeeper-snap/blob/3/edge/snap/snapcraft.yaml)).
14+
15+
Every artifact bundled into the Charmed Kafka snap and Charmed ZooKeeper snap is verified against their MD5, SHA256 or SHA512 checksum after download.
16+
The installation of certified snaps into the rock is ensured by snap primitives that verifies their
17+
squashfs filesystems images GPG signature. For more information on the snap verification process, refer to the [snapcraft.io documentation](https://snapcraft.io/docs/assertions).
18+
19+
## Sources verification
20+
21+
Charmed Kafka sources are stored in:
22+
23+
* GitHub repositories for snaps, rocks and charms
24+
* LaunchPad repositories for the Kafka and ZooKeeper upstream fork used for building their respective distributions
25+
26+
### LaunchPad
27+
28+
Distributions are built using private repositories only, hosted as part of the [SOSS namespace](https://launchpad.net/soss) to eventually
29+
integrate with Canonical's standard process for fixing CVEs.
30+
Branches associated with releases are mirrored to a public repository, hosted in the [Data Platform namespace](https://launchpad.net/~data-platform)
31+
to also provide the community with the patched source code.
32+
33+
### GitHub
34+
35+
All Charmed Kafka and Charmed ZooKeeper artifacts are published and released
36+
programmatically using release pipelines implemented via GitHub Actions.
37+
Distributions are published as both GitHub and LaunchPad releases via the [central-uploader repository](https://github.com/canonical/central-uploader), while
38+
charms, snaps and rocks are published using the workflows of their respective repositories.
39+
40+
All repositories in GitHub are set up with branch protection rules, requiring:
41+
42+
* new commits to be merged to main branches via Pull-Request with at least 2 approvals from repository maintainers
43+
* new commits to be signed (e.g. using GPG keys)
44+
* developers to sign the [Canonical Contributor License Agreement (CLA)](https://ubuntu.com/legal/contributors)
45+
46+
## Encryption
47+
48+
The Charmed Kafka operator can be used to deploy a secure Kafka cluster on K8s that provides encryption-in-transit capabilities out of the box
49+
for:
50+
51+
* Interbroker communications
52+
* ZooKeeper connection
53+
* External client connection
54+
55+
To set up a secure connection Kafka and ZooKeeper applications need to be integrated with TLS Certificate Provider charms, e.g.
56+
`self-signed-certificates` operator. CSRs are generated for every unit using `tls_certificates_interface` library that uses `cryptography`
57+
python library to create X.509 compatible certificates. The CSR is signed by the TLS Certificate Provider and returned to the units, and
58+
stored in a password-protected Keystore file. The password of the Keystore is stored in Juju secrets starting from revision 168 on Kafka
59+
and revision 130 on ZooKeeper. The relation provides also the certificate for the CA to be loaded in a password-protected Truststore file.
60+
61+
When encryption is enabled, hostname verification is turned on for client connections, including inter-broker communication. Cipher suite can
62+
be customized by providing a list of allowed cipher suite to be used for external clients and zookeeper connections, using the charm config options
63+
`ssl_cipher_suites` and `zookeeper_ssl_cipher_suites` config options respectively. Please refer to the [reference documentation](https://charmhub.io/kafka-k8s/configurations)
64+
for more information.
65+
66+
Encryption at rest is currently not supported, although it can be provided by the substrate (cloud or on-premises).
67+
68+
## Authentication
69+
70+
In the Charmed Kafka solution, authentication layers can be enabled for
71+
72+
1. ZooKeeper connections
73+
2. Kafka inter-broker communication
74+
3. Kafka Clients
75+
76+
### Kafka authentication to ZooKeeper
77+
78+
Authentication to ZooKeeper is based on Simple Authentication and Security Layer (SASL) using digested MD5 hashes of
79+
username and password, and implemented both for client-server (with Kafka) and server-server communication.
80+
Username and passwords are exchanged using peer relations among ZooKeeper units and using normal relations between Kafka and ZooKeeper.
81+
Juju secrets are used for exchanging credentials starting from revision 168 on Kafka and revision 130 on ZooKeeper.
82+
83+
Username and password for the different users are stored in ZooKeeper servers in a JAAS configuration file in plain format.
84+
Permission on the file is restricted to the root user.
85+
86+
### Kafka Inter-broker authentication
87+
88+
Authentication among brokers is based on SCRAM-SHA-512 protocol. Username and passwords are exchanged
89+
via peer relations, using Juju secrets from revision 168 on Charmed Kafka.
90+
91+
Kafka username and password used by brokers to authenticate one another are stored
92+
both in a ZooKeeper zNode and in a JAAS configuration file in the Kafka server in plain format.
93+
The file needs to be readable and
94+
writable by root (as it is created by the charm), and be readable by the `snap_daemon` user running the Kafka server snap commands.
95+
96+
### Client authentication to Kafka
97+
98+
Clients can authenticate to Kafka using:
99+
100+
1. username and password exchanged using SCRAM-SHA-512 protocols
101+
2. client certificates or CAs (mTLS)
102+
103+
When using SCRAM, username and passwords are stored in ZooKeeper to be used by the Kafka processes,
104+
in peer-relation data to be used by the Kafka charm and in external relation to be shared with client applications.
105+
Starting from revision 168 on Kafka, Juju secrets are used for storing the credentials in place of plain unencrypted text.
106+
107+
When using mTLS, client certificates are loaded into a `tls-certificates` operator and provided to the Kafka charm via the plain-text unencrypted
108+
relation. Certificates are stored in the password-protected Truststore file.

0 commit comments

Comments
 (0)