Skip to content

Commit 8a96ccc

Browse files
averikitschknative-prow-robot
averikitsch
authored and
knative-prow-robot
committed
Update Telemetry readmes (knative#248)
* telemetry draft * style updates to logging pages * moved elasticsearch instructions * telemetry draft * style updates to logging pages * moved elasticsearch instructions * move elastic search instructions back to installation page * Add more details to specific steps * fix markdown
1 parent fb82eeb commit 8a96ccc

File tree

5 files changed

+302
-192
lines changed

5 files changed

+302
-192
lines changed

‎serving/accessing-logs.md

+77-71
Original file line numberDiff line numberDiff line change
@@ -1,110 +1,116 @@
11
# Accessing logs
22

3-
If you have not yet installed the logging and monitoring components, go through the
4-
[installation instructions](./installing-logging-metrics-traces.md) to set up the
3+
If you have not yet installed the logging and monitoring components, go through the
4+
[installation instructions](./installing-logging-metrics-traces.md) to set up the
55
necessary components first.
66

77
## Kibana and Elasticsearch
88

9-
To open the Kibana UI (the visualization tool for [Elasticsearch](https://info.elastic.co),
10-
enter the following command:
9+
* To open the Kibana UI (the visualization tool for [Elasticsearch](https://info.elastic.co)),
10+
start a local proxy with the following command:
11+
```shell
12+
kubectl proxy
13+
```
1114

12-
```shell
13-
kubectl proxy
14-
```
15-
16-
This command starts a local proxy of Kibana on port 8001. For security reasons, the
17-
Kibana UI is exposed only within the cluster.
15+
This command starts a local proxy of Kibana on port 8001. For security reasons,
16+
the Kibana UI is exposed only within the cluster.
1817

19-
Navigate to the
20-
[Kibana UI](http://localhost:8001/api/v1/namespaces/monitoring/services/kibana-logging/proxy/app/kibana)
21-
(*It might take a couple of minutes for the proxy to work*).
18+
* Navigate to the
19+
[Kibana UI](http://localhost:8001/api/v1/namespaces/monitoring/services/kibana-logging/proxy/app/kibana).
20+
*It might take a couple of minutes for the proxy to work*.
2221

23-
The Discover tab of the Kibana UI looks like this:
22+
The Discover tab of the Kibana UI looks like this:
2423

25-
![Kibana UI Discover tab](./images/kibana-discover-tab-annotated.png)
24+
![Kibana UI Discover tab](./images/kibana-discover-tab-annotated.png)
2625

27-
You can change the time frame of logs Kibana displays in the upper right corner
28-
of the screen. The main search bar is across the top of the Discover page.
26+
You can change the time frame of logs Kibana displays in the upper right corner
27+
of the screen. The main search bar is across the top of the Discover page.
2928

30-
As more logs are ingested, new fields will be discovered. To have them indexed,
31-
go to Management > Index Patterns > Refresh button (on top right) > Refresh
32-
fields.
29+
* As more logs are ingested, new fields will be discovered. To have them indexed,
30+
go to "Management" > "Index Patterns" > Refresh button (on top right) > "Refresh
31+
fields".
3332

3433
<!-- TODO: create a video walkthrough of the Kibana UI -->
3534

3635
### Accessing configuration and revision logs
3736

38-
To access the logs for a configuration, enter the following search query in Kibana:
37+
To access the logs for a configuration:
3938

40-
```text
41-
kubernetes.labels.serving_knative_dev\/configuration: "configuration-example"
39+
* Find the configuration's name with the following command:
4240
```
43-
44-
Replace `configuration-example` with your configuration's name. Enter the following
45-
command to get your configuration's name:
46-
47-
```shell
4841
kubectl get configurations
4942
```
5043

51-
To access logs for a revision, enter the following search query in Kibana:
44+
* Replace `<CONFIGURATION_NAME>` and enter the following search query in Kibana:
45+
```
46+
kubernetes.labels.serving_knative_dev\/configuration: <CONFIGURATION_NAME>
47+
```
5248

53-
```text
54-
kubernetes.labels.serving_knative_dev\/revision: "configuration-example-00001"
49+
To access logs for a revision:
50+
* Find the revision's name with the following command:
51+
```
52+
kubectl get revisions
5553
```
5654

57-
Replace `configuration-example-00001` with your revision's name.
55+
* Replace `<REVISION_NAME>` and enter the following search query in Kibana:
56+
```
57+
kubernetes.labels.serving_knative_dev\/revision: <REVISION_NAME>
58+
```
5859

5960
### Accessing build logs
6061

61-
To access the logs for a build, enter the following search query in Kibana:
62-
63-
```text
64-
kubernetes.labels.build\-name: "test-build"
62+
To access logs for a [Knative Build](../build/README.md):
63+
64+
* Find the build's name in the specified in the `.yaml` file:
65+
```yaml
66+
apiVersion: build.knative.dev/v1alpha1
67+
kind: Build
68+
metadata:
69+
name: <BUILD_NAME>
70+
```
71+
Or find build names with the following command:
72+
```
73+
kubectl get builds
74+
```
75+
76+
* Replace `<BUILD_NAME>` and enter the following search query in Kibana:
6577
```
66-
67-
Replace `test-build` with your build's name. The build name is specified in the `.yaml` file as follows:
68-
69-
```yaml
70-
apiVersion: build.knative.dev/v1alpha1
71-
kind: Build
72-
metadata:
73-
name: test-build
78+
kubernetes.labels.build\-name: <BUILD_NAME>
7479
```
7580

7681
### Accessing request logs
7782

78-
To access to request logs, enter the following search in Kibana:
79-
80-
```text
83+
To access to request logs:
84+
* Enter the following search in Kibana:
85+
```
8186
tag: "requestlog.logentry.istio-system"
8287
```
8388

84-
Request logs contain details about requests served by the revision. Below is a sample request log:
85-
86-
```text
87-
@timestamp July 10th 2018, 10:09:28.000
88-
destinationConfiguration configuration-example
89-
destinationNamespace default
90-
destinationRevision configuration-example-00001
91-
destinationService configuration-example-00001-service.default.svc.cluster.local
92-
latency 1.232902ms
93-
method GET
94-
protocol http
95-
referer unknown
96-
requestHost route-example.default.example.com
97-
requestSize 0
98-
responseCode 200
99-
responseSize 36
100-
severity Info
101-
sourceNamespace istio-system
102-
sourceService unknown
103-
tag requestlog.logentry.istio-system
104-
traceId 986d6faa02d49533
105-
url /
106-
userAgent curl/7.60.0
107-
```
89+
Request logs contain details about requests served by the revision. Below is
90+
a sample request log:
91+
92+
```text
93+
@timestamp July 10th 2018, 10:09:28.000
94+
destinationConfiguration configuration-example
95+
destinationNamespace default
96+
destinationRevision configuration-example-00001
97+
destinationService configuration-example-00001-service.default.svc.cluster.local
98+
latency 1.232902ms
99+
method GET
100+
protocol http
101+
referer unknown
102+
requestHost route-example.default.example.com
103+
requestSize 0
104+
responseCode 200
105+
responseSize 36
106+
severity Info
107+
sourceNamespace istio-system
108+
sourceService unknown
109+
tag requestlog.logentry.istio-system
110+
traceId 986d6faa02d49533
111+
url /
112+
userAgent curl/7.60.0
113+
```
108114

109115
### Accessing end to end request traces
110116

‎serving/accessing-metrics.md

+17-16
Original file line numberDiff line numberDiff line change
@@ -1,30 +1,31 @@
11
# Accessing metrics
22

3-
You access metrics through the [Grafana](https://grafana.com/) UI. Grafana is the visualization tool
4-
for [Prometheus](https://prometheus.io/). To open Grafana, enter the following command:
3+
You access metrics through the [Grafana](https://grafana.com/) UI. Grafana is
4+
the visualization tool for [Prometheus](https://prometheus.io/).
55

6-
```shell
6+
1. To open Grafana, enter the following command:
7+
```
78
kubectl port-forward -n monitoring $(kubectl get pods -n monitoring --selector=app=grafana --output=jsonpath="{.items..metadata.name}") 3000
89
```
910

10-
This starts a local proxy of Grafana on port 3000. For security reasons, the Grafana UI is exposed only within
11-
the cluster.
11+
* This starts a local proxy of Grafana on port 3000. For security reasons, the Grafana UI is exposed only within the cluster.
12+
13+
2. Navigate to the Grafana UI at [http://localhost:3000](http://localhost:3000).
1214

13-
Navigate to the Grafana UI at [http://localhost:3000](http://localhost:3000).
14-
Select the **Home** button on the top of the page to see the list of pre-installed dashboards (screenshot below):
15+
3. Select the **Home** button on the top of the page to see the list of pre-installed dashboards (screenshot below):
1516
![Knative Dashboards](./images/grafana1.png)
1617

17-
The following dashboards are pre-installed with Knative Serving:
18+
The following dashboards are pre-installed with Knative Serving:
1819

19-
* **Revision HTTP Requests:** HTTP request count, latency, and size metrics per revision and per configuration
20-
* **Nodes:** CPU, memory, network, and disk metrics at node level
21-
* **Pods:** CPU, memory, and network metrics at pod level
22-
* **Deployment:** CPU, memory, and network metrics aggregated at deployment level
23-
* **Istio, Mixer and Pilot:** Detailed Istio mesh, Mixer, and Pilot metrics
24-
* **Kubernetes:** Dashboards giving insights into cluster health, deployments, and capacity usage
20+
* **Revision HTTP Requests:** HTTP request count, latency, and size metrics per revision and per configuration
21+
* **Nodes:** CPU, memory, network, and disk metrics at node level
22+
* **Pods:** CPU, memory, and network metrics at pod level
23+
* **Deployment:** CPU, memory, and network metrics aggregated at deployment level
24+
* **Istio, Mixer and Pilot:** Detailed Istio mesh, Mixer, and Pilot metrics
25+
* **Kubernetes:** Dashboards giving insights into cluster health, deployments, and capacity usage
2526

26-
To sign in as an administrator and modify or add dashboards, sign in with username `admin` and password `admin`.
27-
Before you expose the Grafana UI outside the cluster, make sure to change the password.
27+
4. Set up an administrator account to modify or add dashboards by signing in with username: `admin` and password: `admin`.
28+
* Before you expose the Grafana UI outside the cluster, make sure to change the password.
2829

2930
---
3031

‎serving/installing-logging-metrics-traces.md

+88-48
Original file line numberDiff line numberDiff line change
@@ -1,72 +1,112 @@
11
# Monitoring, Logging and Tracing Installation
22

3-
Knative Serving offers two different monitoring setups:
4-
One that uses Elasticsearch, Kibana, Prometheus and Grafana and
5-
another that uses Stackdriver, Prometheus and Grafana. See below
6-
for installation instructions for these two setups. You can install
7-
only one of these two setups and side-by-side installation of these two are not supported.
3+
Knative Serving offers two different monitoring setups:
4+
[Elasticsearch, Kibana, Prometheus and Grafana](#Elasticsearch,-Kibana,-Prometheus-&-Grafana-Setup)
5+
or
6+
[Stackdriver, Prometheus and Grafana](#Stackdriver,-Prometheus-&-Grafana-Setup).
7+
You can install only one of these two setups and side-by-side installation of
8+
these two are not supported.
89

910
## Elasticsearch, Kibana, Prometheus & Grafana Setup
1011

11-
*If you installed Knative Serving using [Easy Install](../install/README.md#Installing-Knative) guide,
12-
skip this step and continue to [Create Elasticsearch Indices](#Create-Elasticsearch-Indices)*
12+
If you installed the
13+
[full Knative release](../install/README.md#Installing-Knative),
14+
skip this step and continue to
15+
[Create Elasticsearch Indices](#Create-Elasticsearch-Indices)
16+
17+
- Install Knative monitoring components:
18+
19+
```shell
20+
kubectl apply -R -f config/monitoring/100-common \
21+
-f config/monitoring/150-elasticsearch \
22+
-f third_party/config/monitoring/common \
23+
-f third_party/config/monitoring/elasticsearch \
24+
-f config/monitoring/200-common \
25+
-f config/monitoring/200-common/100-istio.yaml
26+
```
27+
28+
- The installation is complete when logging & monitoring components are all
29+
reported `Running` or `Completed`:
30+
31+
```shell
32+
kubectl get pods -n monitoring --watch
33+
```
34+
35+
```
36+
NAME READY STATUS RESTARTS AGE
37+
elasticsearch-logging-0 1/1 Running 0 2d
38+
elasticsearch-logging-1 1/1 Running 0 2d
39+
fluentd-ds-5kc85 1/1 Running 0 2d
40+
fluentd-ds-vhrcq 1/1 Running 0 2d
41+
fluentd-ds-xghk9 1/1 Running 0 2d
42+
grafana-798cf569ff-v4q74 1/1 Running 0 2d
43+
kibana-logging-7d474fbb45-6qb8x 1/1 Running 0 2d
44+
kube-state-metrics-75bd4f5b8b-8t2h2 4/4 Running 0 2d
45+
node-exporter-cr6bh 2/2 Running 0 2d
46+
node-exporter-mf6k7 2/2 Running 0 2d
47+
node-exporter-rhzr7 2/2 Running 0 2d
48+
prometheus-system-0 1/1 Running 0 2d
49+
prometheus-system-1 1/1 Running 0 2d
50+
```
51+
52+
CTRL+C to exit watch.
1353

54+
### Create Elasticsearch Indices
1455

15-
Run:
16-
17-
```shell
18-
kubectl apply -R -f config/monitoring/100-common \
19-
-f config/monitoring/150-elasticsearch \
20-
-f third_party/config/monitoring/common \
21-
-f third_party/config/monitoring/elasticsearch \
22-
-f config/monitoring/200-common \
23-
-f config/monitoring/200-common/100-istio.yaml
24-
```
56+
To visualize logs with Kibana, you need to set which Elasticsearch indices to explore. We will create two indices in Elasticsearch using `Logstash` for application logs and `Zipkin`
57+
for request traces.
2558

26-
Monitor logging & monitoring components, until all of the components report Running or Completed:
59+
- To open the Kibana UI (the visualization tool for
60+
[Elasticsearch](https://info.elastic.co)), start a local proxy with the
61+
following command:
2762

28-
```shell
29-
kubectl get pods -n monitoring --watch
30-
```
63+
```shell
64+
kubectl proxy
65+
```
3166

32-
CTRL+C when it's done.
67+
This command starts a local proxy of Kibana on port 8001. For security
68+
reasons, the Kibana UI is exposed only within the cluster.
3369

34-
### Create Elasticsearch Indices
35-
We will create two indexes in ElasticSearch - one for application logs and one for request traces.
36-
To create the indexes, open Kibana Index Management UI at this [link](http://localhost:8001/api/v1/namespaces/monitoring/services/kibana-logging/proxy/app/kibana#/management/kibana/index)
37-
(*it might take a couple of minutes for the proxy to work the first time after the installation*).
70+
- Navigate to the
71+
[Kibana UI](http://localhost:8001/api/v1/namespaces/monitoring/services/kibana-logging/proxy/app/kibana).
72+
_It might take a couple of minutes for the proxy to work_.
3873

39-
Within the "Configure an index pattern" page, enter `logstash-*` to `Index pattern` and select `@timestamp`
40-
from `Time Filter field name` and click on `Create` button. See below for a screenshot:
74+
- Within the "Configure an index pattern" page, enter `logstash-*` to
75+
`Index pattern` and select `@timestamp` from `Time Filter field name` and
76+
click on `Create` button.
4177

4278
![Create logstash-* index](images/kibana-landing-page-configure-index.png)
4379

44-
To create the second index, select `Create Index Pattern` button on top left of the page.
45-
Enter `zipkin*` to `Index pattern` and select `timestamp_millis` from `Time Filter field name`
46-
and click on `Create` button.
80+
- To create the second index, select `Create Index Pattern` button on top left
81+
of the page. Enter `zipkin*` to `Index pattern` and select `timestamp_millis`
82+
from `Time Filter field name` and click on `Create` button.
83+
4784

48-
Next, visit instructions below to access to logs, metrics and traces:
85+
## Stackdriver, Prometheus & Grafana Setup
4986

50-
* [Accessing Logs](./accessing-logs.md)
51-
* [Accessing Metrics](./accessing-metrics.md)
52-
* [Accessing Traces](./accessing-traces.md)
87+
If your Knative Serving is not built on a Google Cloud Platform (GCP) based
88+
cluster or you want to send logs to another GCP project, you need to build your
89+
own Fluentd image and modify the configuration first. See
5390

54-
## Stackdriver(logs), Prometheus & Grafana Setup
91+
1. Install
92+
[Fluentd image on Knative Serving](https://github.com/knative/serving/blob/master/image/fluentd/README.md).
93+
2. [Set up a logging plugin](setting-up-a-logging-plugin.md).
94+
3. Install Knative monitoring components:
5595

56-
If your Knative Serving is not built on a GCP based cluster or you want to send logs to
57-
another GCP project, you need to build your own Fluentd image and modify the
58-
configuration first. See
96+
```shell
97+
kubectl apply -R -f config/monitoring/100-common \
98+
-f config/monitoring/150-stackdriver-prod \
99+
-f third_party/config/monitoring/common \
100+
-f config/monitoring/200-common \
101+
-f config/monitoring/200-common/100-istio.yaml
102+
```
59103

60-
1. [Fluentd image on Knative Serving](/image/fluentd/README.md)
61-
2. [Setting up a logging plugin](setting-up-a-logging-plugin.md)
104+
## Learn More
62105

63-
```shell
64-
kubectl apply -R -f config/monitoring/100-common \
65-
-f config/monitoring/150-stackdriver \
66-
-f third_party/config/monitoring/common \
67-
-f config/monitoring/200-common \
68-
-f config/monitoring/200-common/100-istio.yaml
69-
```
106+
- Learn more about accessing logs, metrics, and traces:
107+
- [Accessing Logs](./accessing-logs.md)
108+
- [Accessing Metrics](./accessing-metrics.md)
109+
- [Accessing Traces](./accessing-traces.md)
70110

71111
---
72112

0 commit comments

Comments
 (0)