Skip to content

Commit d058349

Browse files
morfien101morfien101
committed
Adding in the README files.
1 parent 4367381 commit d058349

File tree

5 files changed

+338
-3
lines changed

5 files changed

+338
-3
lines changed

‎.gitignore

-3
Original file line numberDiff line numberDiff line change
@@ -3,6 +3,3 @@
33
# Available publically and likely to fall out of date.
44
# https://help.papertrailapp.com/kb/configuration/encrypting-remote-syslog-with-tls-ssl/
55
papertrail_cert.crt
6-
7-
# while I sort these out
8-
READMEs/

‎READMEs/ConfigrationFile.md

+208
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,208 @@
1+
# Configuration File
2+
3+
Configuration is used to tell Launch what to do.
4+
5+
It is used to determine where logs go, what processes run and what data needs to be collected.
6+
7+
An example of the configuration file can be obtained by running:
8+
9+
```bash
10+
# Get example configuration file
11+
./launch -example-config
12+
```
13+
14+
Below is each section of the configuration with the relevant values and details. If you would like more information regarding the configuration all data is layed out in the [configfile folder](../configfile).
15+
16+
## Understanding double rendering
17+
18+
Configuration files have templating built in, see later templating section. This allows for environment variables to be used in the configuration file.
19+
However many of those environment variables will be collected during the secrets and parameters collection phase.
20+
21+
The configuration file is rendered twice when the Launch is started.
22+
The first render will be done on start up, it will produce the configuration with the available environment variables as the container starts.
23+
The Launch will then proceed to collect secrets and parameters. Once complete the configuration is rendered a second time.
24+
This allows the configuration to make use of parameters and secrets as part of the configuration. Previously blanked environment settings will now be filled in.
25+
26+
An example use case is using an override for hostname in the syslog logging configuration, or setting a environment variable.
27+
28+
Normal templating rules will be followed when using both renders. This means that you can still use the environment variables for secret collection but, they __MUST__ be available when the container starts. Eg. making use of docker -e|--env flags.
29+
This also means that any templating that makes use of the `required` and `env` functions together __MUST__ be available on the first render or the configuration will fail to render and cause the Launch to stop.
30+
31+
Secrets are only collected once at startup.
32+
33+
## process_manager
34+
35+
`process_manager` configures the Launch process itself. It needs to know where to send it's logs and also if it needs to enable debug logging.
36+
37+
Example:
38+
39+
```yaml
40+
# Process manager is Launch itself
41+
process_manager:
42+
# logging_config is the logging the Launch process needs to use.
43+
logging_config:
44+
# This section contains a Logging config _see below_
45+
# debug_logging will toggle on and off the debug logger inside
46+
# the Launch
47+
debug_logging: (true|false)
48+
# Debug options should be off by default. Use them only if you need to.
49+
debug_options:
50+
# Prints the configuration that will be used for running processes. This happens after secrets are collected
51+
# and the second config rendering has taken place.
52+
show_generated_config: (true|false)
53+
```
54+
55+
## processes
56+
57+
`processes` tells the Launch what start and where to send the logs. There is 3 sections here: secret_processes, init_processes and main_processes
58+
59+
Example:
60+
61+
```yaml
62+
# processes tells Launch what to start and how.
63+
processes:
64+
# Secret processes start before logging so use the console logger.
65+
# Secrets can export to environment variables and are the only process
66+
# that can share environment variables.
67+
secret_processes:
68+
- name: Process1
69+
# Path to executable
70+
command: /example/bin1
71+
# List of arguments
72+
arguments:
73+
- --arg1
74+
- two
75+
# How long to allow the process to run. This helps with run away processes.
76+
termination_timeout_seconds: 60
77+
# skip stops this execution if required.
78+
# Use the template functions to set this value.
79+
skip: false
80+
# init_processes start after secrets and run sequentially
81+
# This is a list and can have as many items as required.
82+
init_processes:
83+
# name is a descriptive name for the process.
84+
- name: first init processes
85+
# command is the full path to executable
86+
command: /binary/to/execute
87+
# List of arguments to pass onto the executable
88+
arguments:
89+
# Strings, strings with spaces, and numbers are accepted
90+
# Add as many as you need
91+
- -c
92+
- /tmp/text.txt
93+
- 10
94+
- words and more
95+
# termination_timeout_seconds how long the grace period is for processes to terminate.
96+
# The default is 1 second.
97+
termination_timeout_seconds: 3
98+
# logging_config is used to forward on the logs from this process.
99+
logging_config:
100+
# This section contains a Logging config _see below_
101+
# main_processes looks exactly the same as init_processes.
102+
main_processes:
103+
- name: first main
104+
command: /binary/to/execute
105+
arguments:
106+
- -a
107+
- 1
108+
- -b
109+
- this and that
110+
logging_config:
111+
# This section contains a Logging config _see below_
112+
- name: second main
113+
command: /another/binary/to/execute
114+
arguments:
115+
- -a
116+
- 1
117+
- -b
118+
- foo and bar
119+
logging_config:
120+
# This section contains a Logging config _see below_
121+
```
122+
123+
## default_logger_config
124+
125+
`default_logger_config` is a section that allows you to put in any defaults that you don't want to repeat.
126+
It contains a `logging_config` but is intended to allow for all configuration for all engines to be defined here.
127+
128+
Example:
129+
130+
```yaml
131+
default_logger_config:
132+
logging_config:
133+
# This section contains a Logging config _see below_
134+
```
135+
## Logging
136+
137+
Logging is used to tell the Launch to send logs to a logging engine eg. syslog, console, etc...
138+
139+
Below is the logging configuration. However this is a slightly different one to the rest.
140+
141+
In this configuration you put the details for the logging engine for the process and then select which logging engine you want to make use of. These will appear in all processes.
142+
143+
valid engine names:
144+
145+
- console
146+
- devnull
147+
- syslog
148+
- logfile
149+
150+
Example:
151+
152+
```yaml
153+
logging_config:
154+
# Engine is where the logs should go
155+
engine: any valid engine name from above
156+
# descriptive name for your binary. Shipped to logging engine if possible.
157+
process_name: amazing_project
158+
# Only one of the below is required when used on a process.
159+
# Normally the one that is related the engine selected.
160+
# Syslog and file logger both require extra config as below.
161+
syslog:
162+
# Overrides the process name with this value. Can be defaulted so all processes that
163+
# don't have a value set will get this one.
164+
# If your process does have this or a name you will get the hostname which is ugly.
165+
# So have at least one of them set.
166+
program_name: syslog program name
167+
# Tries to read the log level if specific condition are met
168+
extract_log_level: (true|false)
169+
# Override the hostname sent to papertrail
170+
override_hostname: hostname_to_use
171+
# These both add the containers hostname to the end of the value they control.
172+
# remember that you need to add your own separator.
173+
append_container_name_to_tag: (true|false)
174+
append_container_name_to_hostname: (true|false)
175+
file_config:
176+
# filepath is where to store these logs
177+
filepath: /var/logs/process_name.log
178+
# size_limit is how large the file can be before rotation happens.
179+
size_limit: 100mb
180+
# historical_files_limit is how many files are to be kept.
181+
historical_files_limit: 3
182+
```
183+
184+
## Template Functions
185+
186+
Template functions are used to make the configuration files somewhat dynamic. They allow values to be put in place at boot time of the process. Reading and rendering the configuration file is the very first thing Launch will do. Therefore if the configuration does not render correctly then it will not start.
187+
188+
Below is a list of the functions available to you and an example of the syntax used.
189+
190+
`hint: it's just golang's template syntax`
191+
192+
Function | Description | Example
193+
---|---|---
194+
env | Sets the value to an available Environment Variable | {{ env "ENVIRONMENT" }}
195+
default | Use this default value if function fails | {{ default .NonExisting "default value" }}
196+
required | This value must be satisfied or the launch will fail | {{ required (env "ALWAYS_THERE") }}
197+
zerolen | Allows you to check if a value is zero length, returns the value for true or false | {{ zerolen (env "SOMETHING") "true value" "false value" }}
198+
199+
Example in configuration file.
200+
201+
```yaml
202+
secrets:
203+
parameter_store:
204+
- key_path: {{ env "SECRET_PATH" }}
205+
recursive_lookup: true
206+
with_decryption: true
207+
skip: {{ zerolen (env "SECRET_PATH") "true" "false" }}
208+
```

‎READMEs/Logging.md

+75
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,75 @@
1+
# Logging Engines Documentation
2+
3+
A more detailed description of the loggers that are available is documented below.
4+
5+
## Startup logging
6+
7+
The Launch needs to start logging before it is actually capable of logging fully. This is because some loggers require authentication that is expected to be collected by the secrets.
8+
To overcome this the Launch will log to the console until after the secrets have been collected.
9+
At which point it will read the configuration file and use the specified loggers.
10+
11+
## Default logging
12+
13+
Default values for logging can be set. These values will be used if there is not value present for a process or they will be merged. This allows most the configuration to be setup here, and only specifics to be set at the process level.
14+
15+
Processes will still need to select a logging engine.
16+
17+
## DevNull
18+
19+
DevNull is basically the same as /dev/null. Its a black hole for logs to go and never return.
20+
21+
Use this if you have an application that you simply don't care about the logging.
22+
23+
## Console
24+
25+
Console logging will forward out the logs it receives to the Launch STDOUT and STDERR. This is more useful for development environments where you don't want to forward your logs to a central logging platform.
26+
27+
## File Logging
28+
29+
File logging consumes the messages from your application via STDOUT and STDERR and forwards them to a file. The file is rotated on a regular basis to keep the container footprint small. The configuration values for rotation can be set with the configuration files.
30+
31+
File logging is only really useful in development environments. In most production environments the disks of the containers will be removed once the container is terminated. If you want to use this in production it is recommend that you link the volumes where the files are to be written.
32+
33+
## Syslog
34+
35+
Syslog is a pretty standard linux way of sending logs. These logs are sent as lines and multiline logs are unfortunetly split.
36+
37+
The logger allows you to override the name that you see in syslog. By default it will use the process name. If you set the `program_name` key under the syslog logging configuration it will use that in its place.
38+
39+
Due to the logs being presented to Launch via stdout we are not able to know if the log is critical, warning or informational. This information might be available in the text, however the Launch will simply forward on the message with out inspecting its contents by default.
40+
41+
If you set `extract_log_level: true` the logger will attempt to detect the level from your message. There are limitations here and your messages need to be structured correctly.
42+
43+
1. The logs `MUST` be in JSON format.
44+
1. The logs `MUST` have have `level` key.
45+
1. The `level` key `MUST` have one of the following values. [syslog wikipedia](https://en.wikipedia.org/wiki/Syslog#Severity_level)
46+
47+
* emerg
48+
* alert
49+
* crit
50+
* err
51+
* warning
52+
* notice
53+
* info
54+
* debug
55+
56+
Example:
57+
58+
```json
59+
{
60+
"name": "my_test",
61+
"msg": "This is a test log",
62+
"level": "crit"
63+
}
64+
```
65+
66+
The last point to think about is that large logs (excess of 1000 chars) may have a small impact on performance. This is compounded with rapid logging. Currently it takes about 0.003 Milliseconds to read a 500 char log. If you have hundreds of logs per second, Expect problems with detection.
67+
68+
Syslog logging reports the hostname of the host that sent the message. This is by default the hostname of the container. However for searching the hostname of the container can actually be useless and rather you would want the applications to send under a common hostname. This can be done by making use the `override_hostname` configuration. This can be set as a default logging variable and process level. The process level will win if both are set.
69+
You can combine this with the configuration file templating to make names that reflect your environment such as `override_hostname: awesome_app_{{env NODE_NAME }}_{{env NODE_REGION}}`.
70+
71+
The normal rules for host names will apply. No special chars, no space, etc...
72+
73+
For easier tracking of the instances themselves you can append the containers hostname to either the process or to the hostname. If you do this you need to include any _ or - as it will simple tack on the hostname.
74+
The can be set at the default configuration OR the process configuration.
75+
Use `append_container_name_to_tag` and `append_container_name_to_hostname` to control these features.

‎READMEs/README.md

+18
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,18 @@
1+
# README files
2+
3+
The README files for all the features are contained here.
4+
Take a look around.
5+
6+
## Contents
7+
8+
### Configuration files
9+
10+
[Configuration Documentation](ConfigurationFile.md)
11+
12+
### Logging
13+
14+
[Logging Documentation](Logging.md)
15+
16+
### Secrets
17+
18+
[Secrets Documentation](Secrets.md)

‎READMEs/Secrets.md

+37
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,37 @@
1+
# Secret Collection Engines
2+
3+
Secrets are collected before anything else has a chance to start.
4+
This is because the loggers often need credentials to send the logs onwards.
5+
Due to this logging is done to the console of the container itself, because we can not be sure we would be able to actually send the log messages on.
6+
7+
Launch allows you to run your own binary/command to collect secrets within the containers. Doing this means that you can use any secret management system you can write code for.
8+
9+
## Secret Processes
10+
11+
A secret process is used to go collect secrets. These could be in AWS Secret Manager, Hashicorp Vault or some inhouse secret manager.
12+
The idea is that you can create your own binary to use to collect secrets and have Launch inject them into the environment.
13+
14+
```text
15+
Because Launch is the parent process, child processes CAN NOT update the environment of the parent.
16+
Therefore your process can not expose environment variables for later processes to see.
17+
```
18+
To over come this, Launch will read a key value pair in JSON as the stdout of your process and expose those as environment variables for you.
19+
20+
Output expected:
21+
22+
```json
23+
{"key":"value", "key2":"value2"}
24+
```
25+
26+
It is also possible that you process writes to files that other processes can collect. It is expected that no output to STDOUT is given in this case.
27+
28+
With this being said here are the requirements for Secret Processes:
29+
30+
1. They must complete successfully to continue execution.
31+
1. They are executed sequentially as shown in the configuration file.
32+
1. output must be valid JSON and the only output on STDOUT.
33+
34+
Secrets might not always need to be collected. Consider if you are using in a Dev environment.
35+
Make use of the `skip` field to stop a process from running.
36+
You can determine the value by using one of the templating functions.
37+

0 commit comments

Comments
 (0)