feat: kafka-gitops work and major refactor

Author: devshawn

.github/workflows/gradle.yml

@@ -23,5 +23,5 @@ jobs:
     - name: Upload build artifact
       uses: actions/upload-artifact@v1
       with:
-        name: kafka-dsf.jar
-        path: build/libs/kafka-dsf-all.jar
+        name: kafka-gitops.jar
+        path: build/libs/kafka-gitops-all.jar

.github/workflows/release.yml

@@ -26,7 +26,7 @@ jobs:
       - name: Build & Push Docker Image
         uses: mr-smithers-excellent/docker-build-push@v2
         with:
-          image: devshawn/kafka-dsf
+          image: devshawn/kafka-gitops
           tag: ${{ steps.get_version.outputs.VERSION }}
           registry: docker.io
           username: ${{ secrets.DOCKER_USERNAME }}
@@ -50,6 +50,6 @@ jobs:
           GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
         with:
           upload_url: ${{ steps.create_release.outputs.upload_url }} 
-          asset_path: ./build/distributions/kafka-dsf.zip
-          asset_name: kafka-dsf.zip
+          asset_path: ./build/distributions/kafka-gitops.zip
+          asset_name: kafka-gitops.zip
           asset_content_type: application/zip

CONTRIBUTING.md

@@ -1,6 +1,6 @@
 # Contributing
 
-Contributions are very welcome! Any existing issues labeled ["help wanted"](https://github.com/devshawn/kafka-dsf/labels/help%20wanted) or ["good first issue"](https://github.com/devshawn/kafka-dsf/labels/good%20first%20issue) are free to be pursued.
+Contributions are very welcome! Any existing issues labeled ["help wanted"](https://github.com/devshawn/kafka-gitops/labels/help%20wanted) or ["good first issue"](https://github.com/devshawn/kafka-gitops/labels/good%20first%20issue) are free to be pursued.
 
 ## Feature Requests & Bug Reports
 For feature requests and bug reports, [submit an issue][issues].
@@ -17,5 +17,5 @@ The preferred way to contribute is to fork the [main repository][repository] on
 4. Once changes are completed, open a pull request for review against the master branch.
 
 
-[repository]: https://github.com/devshawn/kafka-dsf
-[issues]: https://github.com/devshawn/kafka-dsf/issues
+[repository]: https://github.com/devshawn/kafka-gitops
+[issues]: https://github.com/devshawn/kafka-gitops/issues

Dockerfile

@@ -2,6 +2,6 @@ FROM openjdk:8-slim
 
 RUN apt-get update && apt-get install -y python3 python3-pip curl
 
-COPY ./build/libs/kafka-dsf-all.jar /usr/local/bin/kafka-dsf-all.jar
-COPY ./kafka-dsf /usr/local/bin/kafka-dsf
+COPY ./build/libs/kafka-gitops-all.jar /usr/local/bin/kafka-gitops-all.jar
+COPY ./kafka-gitops /usr/local/bin/kafka-gitops
 

README.md

@@ -1,4 +1,4 @@
-# kafka-dsf
+# kafka-gitops
 
 Manage Apache Kafka topics and ACLs through a desired state file.
 
@@ -8,22 +8,42 @@ Manage Apache Kafka topics and ACLs through a desired state file.
 
 ## Overview
 
-This project allows you to manage Kafka topics and ACLs through use of a desired state file, much like Terraform and similar infrastructure-as-code projects. 
+Kafka GitOps is an Apache Kafka resources-as-code tool which allows you to automate the management of your Apache Kafka topics and ACLs from version controlled code. It allows you to define topics and services through the use of a desired state file, much like Terraform and other infrastructure-as-code tools.
 
-Topics and ACLs get defined in a YAML file. When run, `kafka-dsf` compares your desired state to the actual state of the cluster and generates a plan to execute against the cluster. This will make your topics and ACLs match your desired state.
+Topics and services get defined in a YAML file. When run, `kafka-gitops` compares your desired state to the actual state of the cluster and generates a plan to execute against the cluster. This will make your topics and ACLs match your desired state.
+
+This tool also generates the needed ACLs for each type of application. There is no need to manually create a bunch of ACLs for Kafka Connect, Kafka Streams, etc. By defining your services, `kafka-gitops` will build the necessary ACLs.
+
+This tool supports self-hosted Kafka, managed Kafka, and Confluent Cloud clusters.
+
+## Features
+
+- 🚀  **Built For CI/CD**: Made for CI/CD pipelines to automate the management of topics & ACLs.
+- 🔥  **Configuration as code**: Describe your desired state and manage it from a version-controlled declarative file.
+- 👍  **Easy to use**: Deep knowledge of Kafka administration or ACL management is **NOT** required. 
+- ⚡️️  **Plan & Apply**: Generate and view a plan with or without executing it against your cluster.
+- 💻  **Portable**: Works across self-hosted clusters, managed clusters, and even Confluent Cloud clusters.
+- 🦄  **Idempotency**: Executing the same desired state file on an up-to-date cluster will yield the same result.
+- ☀️  **Continue from failures**: If a specific step fails during an apply, you can fix your desired state and re-run the command. You can execute `kafka-gitops` again without needing to rollback any partial successes.
+
+## Getting Started
+
+Documentation on how to install and use this tool can be found on our [documentation site][documentation].
 
 ## Usage
 
-Run `kafka-dsf` to view the help output.
+Run `kafka-gitops` to view the help output.
 
 ```bash
-Usage: kafka-dsf [-hvV] [-f=<file>] [COMMAND]
+Usage: kafka-gitops [-hvV] [--no-delete] [-f=<file>] [COMMAND]
 Manage Kafka resources with a desired state file.
   -f, --file=<file>   Specify the desired state file.
   -h, --help          Display this help message.
+      --no-delete     Disable the ability to delete resources.
   -v, --verbose       Show more detail during execution.
   -V, --version       Print the current version of this tool.
 Commands:
+  account   Create Confluent Cloud service accounts.
   apply     Apply changes to Kafka resources.
   plan      Generate an execution plan of changes to Kafka resources.
   validate  Validates the desired state file.
@@ -51,7 +71,7 @@ The following configuration is generated:
 
 ## State File
 
-By default, `kafka-dsf` looks for `state.yaml` in the current directory. You can also use `kafka-dsf -f` to pass a file.
+By default, `kafka-gitops` looks for `state.yaml` in the current directory. You can also use `kafka-gitops -f` to pass a file.
 
 An example desired state file:
 
@@ -63,15 +83,13 @@ topics:
     configs:
       cleanup.policy: compact
 
-acls:
-  example-topic-read-acl:
-    name: example-topic
-    type: TOPIC
-    pattern: LITERAL
-    principal: User:super.admin
-    host: "*"
-    operation: WRITE
-    permission: ALLOW
+services:
+  example-service:
+    type: application
+    produces:
+      - example-topic
+    consumes:
+      - example-topic
 ```
 
 ## Contributing
@@ -84,5 +102,6 @@ Copyright (c) 2020 Shawn Seymour.
 
 Licensed under the [Apache 2.0 license][license].
 
+[documentation]: https://kafkagitops.devshawn.com
 [contributing]: ./CONTRIBUTING.md
 [license]: ./LICENSE

build.gradle

@@ -1,5 +1,6 @@
 plugins {
     id 'java'
+    id 'groovy'
     id 'application'
     id 'idea'
     id 'org.inferred.processors' version '1.2.10'
@@ -11,7 +12,7 @@ apply plugin: 'net.ltgt.apt-idea'
 
 group 'com.devshawn'
 
-mainClassName = 'com.devshawn.kafka.dsf.MainCommand'
+mainClassName = 'com.devshawn.kafka.gitops.MainCommand'
 
 sourceCompatibility = 1.8
 
@@ -33,20 +34,24 @@ dependencies {
     processor 'org.inferred:freebuilder:2.5.0'
 
     testCompile group: 'junit', name: 'junit', version: '4.12'
+    testCompile group: 'org.codehaus.groovy', name: 'groovy-all', version: '2.4.4'
+    testCompile group: 'org.spockframework', name: 'spock-core', version: '1.0-groovy-2.4'
+    testCompile group: 'cglib', name: 'cglib-nodep', version: '2.2'
+
 }
 
 jar {
     manifest {
         attributes(
                 'Class-Path': configurations.compile.collect { it.getName() }.join(' '),
-                'Main-Class': 'com.devshawn.kafka.dsf.MainCommand'
+                'Main-Class': 'com.devshawn.kafka.gitops.MainCommand'
         )
     }
 }
 
 task buildRelease(type: Zip, group: "build") {
     from("$buildDir/libs")
-    from("./kafka-dsf")
+    from("./kafka-gitops")
 }
 
 buildRelease.dependsOn shadowJar

docs/.nojekyll

@@ -0,0 +1,9 @@
+
+# kafka-gitops
+
+> GitOps for Apache Kafka
+
+- Automated Topic & ACL Management
+- Manage topics, services, and ACLs with desired state files
+
+[Documentation](/documentation.md)

docs/_coverpage.md

@@ -0,0 +1,10 @@
+- **Getting Started**
+- [Introduction](/documentation.md)
+- [Installation](/installation.md)
+- [Quick Start](/quick-start.md)
+- [Services](/services.md)
+- [Confluent Cloud](/confluent-cloud.md)
+- [Specification](/specification.md)
+- **Links**
+- [Contributing](https://github.com/devshawn/kafka-gitops/blob/master/CONTRIBUTING.md)
+- [GitHub](https://github.com/devshawn/kafka-gitops)

docs/_sidebar.md

@@ -0,0 +1,115 @@
+# Confluent Cloud
+
+This tool was designed to work with Confluent Cloud. It can manage service accounts, topics, and ACLs for Confluent Cloud clusters.
+
+## Getting Started
+
+Ensure you have installed `kafka-gitops` or are using the `kafka-gitops` docker image as described in the [installation][installation] instructions.
+
+You must have the `ccloud` command line tools installed if you wish to auto-populate the `principal` fields on services.
+
+## Desired State File
+
+Create a basic desired state file, `state.yaml`, such as:
+
+```yaml
+settings:
+  ccloud:
+    enabled: true
+
+topics:
+  test-topic:
+    partitions: 6
+    replication: 3
+
+services:
+  test-service:
+    type: application
+    produces:
+      - test-topic
+```
+
+To give an overview, throughout this guide, this will create:
+
+- A topic named `test-topic`
+- A service account named `test-service`
+- An `WRITE` ACL for topic `test-topic` tied to the service account `test-service`
+
+## Configuration
+
+To use `kafka-gitops` with Confluent Cloud, you'll need to set a few environment variables.
+
+* `KAFKA_BOOTSTRAP_SERVERS`: Your Confluent Cloud cluster URL
+* `KAFKA_SASL_JAAS_USERNAME`: Your Confluent Cloud API key
+* `KAFKA_SASL_JAAS_USERNAME`: Your Confluent Cloud API secret
+* `KAFKA_SECURITY_PROTOCOL`: `SASL_SSL`
+* `KAFKA_SASL_MECHANISM`: `PLAIN`
+* `KAFKA_SSL_ENDPOINT_IDENTIFICATION_ALGORITHM`: `HTTPS`
+
+Additionally, you'll need to login to the `ccloud` tool. You can automate this by setting the following environment variables:
+
+* `XX_CCLOUD_EMAIL`: Your Confluent Cloud administrator email
+* `XX_CCLOUD_PASSWORD`: Your Confluent Cloud administrator password
+
+Then, you can run `ccloud login` and it will run without a prompt. This is great for CI builds.
+
+## Validate
+
+First, validate your state file is correct by running:
+
+```bash
+kafka-gitops -f state.yaml validate
+```
+
+An example success message would look like:
+
+```text
+[VALID] Successfully validated the desired state file.
+```
+
+## Accounts
+
+Before generating an execution plan, you will need to create the service accounts. This can be done by running:
+
+```bash
+kafka-gitops -f state.yaml accounts
+```
+
+This currently only creates service accounts; it will not delete any.
+
+## Plan
+
+We're now ready to generate a plan to execute against the cluster. By using the plan command, we are **NOT** changing the cluster.
+
+Generate a plan by running:
+
+```bash
+kafka-gitops -f state.yaml plan -o plan.json
+```
+
+This will generate an execution plan to run against the cluster and save the output to `plan.json`. 
+
+The command will also pretty-print what changes it wants to make to the cluster. 
+
+## Apply
+
+To execute a plan against the cluster, we use the apply command.
+
+!> **WARNING**: This will apply changes to the cluster. This can be potentially destructive if you do not have all topics and ACLs defined.
+
+By default, the apply command will generate a plan and then apply it. For most situations, **you should output a plan file from the plan command and pass it to the apply command**.
+
+Example:
+
+```bash
+kafka-gitops -f state.yaml apply -p plan.json
+```
+
+Congrats! You're now using `kafka-gitops` to manage your Confluent Cloud cluster. Once you've practiced locally, you should commit your state file to a repository and create CI/CD pipelines to create plans and execute them against the cluster. 
+
+Welcome to GitOps! 
+
+
+
+
+[installation]: /installation.md

docs/confluent-cloud.md

@@ -0,0 +1,50 @@
+<div align="center">
+    <h1>GitOps for Apache Kafka</h1>
+    <img src="https://i.imgur.com/eAIAv0w.png"/>
+    <span>Manage Apache Kafka topics, services, and ACLs through a desired state file.</span>
+</div>
+
+## Overview
+
+Kafka GitOps is an Apache Kafka resources-as-code tool which allows you to automate the management of your Apache Kafka topics and ACLs from version controlled code. It allows you to define topics and services through the use of a desired state file, much like Terraform and other infrastructure-as-code tools.
+
+Topics and services get defined in a YAML file. When run, `kafka-gitops` compares your desired state to the actual state of the cluster and generates a plan to execute against the cluster. This will make your topics and ACLs match your desired state.
+
+This tool also generates the needed ACLs for each type of application. There is no need to manually create a bunch of ACLs for Kafka Connect, Kafka Streams, etc. By defining your services, `kafka-gitops` will build the necessary ACLs.
+
+## Features
+
+- 🚀  **Built For CI/CD**: Made for CI/CD pipelines to automate the management of topics & ACLs.
+- 🔥  **Configuration as code**: Describe your desired state and manage it from a version-controlled declarative file.
+- 👍  **Easy to use**: Deep knowledge of Kafka administration or ACL management is **NOT** required. 
+- ⚡️️  **Plan & Apply**: Generate and view a plan with or without executing it against your cluster.
+- 💻  **Portable**: Works across self-hosted clusters, managed clusters, and even Confluent Cloud clusters.
+- 🦄  **Idempotency**: Executing the same desired state file on an up-to-date cluster will yield the same result.
+- ☀️  **Continue from failures**: If a specific step fails during an apply, you can fix your desired state and re-run the command. You can execute `kafka-gitops` again without needing to rollback any partial successes.
+
+## Getting Started
+
+Check out the **[Quick Start](/quick-start.md)** documentation to install `kafka-gitops` and get started.
+
+## Ideology
+
+The idea behind this project is to manage Kafka topics and ACLs through desired state files. These files define what your cluster should look like and `kafka-gitops` modifies your cluster's actual state to match the desired state file. Typically, this looks like:
+
+- State files are stored in version control, such as a git repository.
+- Developers add, change, and remove topics & services from files and open a PR.
+- Operations teams valdiate and merge the PR.
+- A CI/CD pipeline generates a plan using `kafka-gitops`. 
+- A human then validates the plan and applies the changes to the cluster.
+
+## Contributing
+
+Contributions are very welcome. See [CONTRIBUTING.md][contributing] for details.
+
+## License
+
+Copyright (c) 2020 Shawn Seymour.
+
+Licensed under the [Apache 2.0 license][license].
+
+[contributing]: https://github.com/devshawn/kafka-gitops/blob/master/CONTRIBUTING.md
+[license]: https://github.com/devshawn/kafka-gitops/blob/master/LICENSE