Skip to content

Commit c8e8ca9

Browse files
tiffany76svrnmchalin
authored
Blog post: Insights from the OpenTelemetry Docs Usability Survey (#5797)
Co-authored-by: Severin Neumann <[email protected]> Co-authored-by: Patrice Chalin <[email protected]>
1 parent b580ea5 commit c8e8ca9

File tree

7 files changed

+199
-0
lines changed

7 files changed

+199
-0
lines changed

‎content/en/blog/2024/otel-docs-survey/component-explanations.png

18 KB
Loading

‎content/en/blog/2024/otel-docs-survey/feature-request.png

32.3 KB
Loading

‎content/en/blog/2024/otel-docs-survey/index.md

+191
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,191 @@
1+
---
2+
title: Insights from the OpenTelemetry Docs Usability Survey
3+
linkTitle: OTel-docs Survey
4+
date: 2024-12-18
5+
author: '[Tiffany Hrabusa](https://github.com/tiffany76) (Grafana Labs)'
6+
issue: https://github.com/open-telemetry/opentelemetry.io/issues/5793
7+
sig: Communications, End-User
8+
cSpell:ignore: Hrabusa
9+
---
10+
11+
[The OpenTelemetry End-User SIG](/community/end-user/) recently surveyed the
12+
community to find out how user-friendly [OpenTelemetry's documentation](/docs/)
13+
is. In an earlier survey, two-thirds of respondents named comprehensive
14+
documentation as a top resource they wished they'd had when getting started with
15+
OpenTelemetry. So we decided to dig a little deeper.
16+
17+
The Docs Usability Survey asked users where they go for OTel documentation, what
18+
they'd like to see more of in the docs, and how they rate the current state of
19+
the docs. We received 48 responses, which we'll use to focus our documentation
20+
efforts and help us improve in key areas.
21+
22+
A big thank you to everyone who participated in the survey! Let's review the
23+
results.
24+
25+
## Key takeaways
26+
27+
- Respondents expressed a desire for **more visual aids**, such as diagrams and
28+
screenshots.
29+
- Of the three types of documentation we asked about (component concepts,
30+
installation instructions, and troubleshooting), the **troubleshooting docs**
31+
were identified as needing the most improvement.
32+
- When asked about the information they’d most like to see added to OTel's docs,
33+
the top responses were **more examples** and **expanded coverage**, both in
34+
depth and breadth.
35+
- The [Collector](/docs/collector/) docs emerged as the most frequently
36+
consulted resource, a finding that aligns with the page view analysis in the
37+
SIG Communications'
38+
[year-end review](../year-in-review/#which-pages-were-the-most-popular).
39+
- After normalization and weighting, the [Java](/docs/languages/java/)
40+
documentation received the **highest overall rating**, reflecting the positive
41+
impact of recent improvements to its
42+
[organization](../year-in-review/#ia-improvements). Conversely, the
43+
[Swift](/docs/languages/swift/) docs received the lowest overall rating.
44+
- Among the six most popular documentation sets, the
45+
[JavaScript](/docs/languages/js/) docs received the **lowest rating**.
46+
47+
## Detailed insights
48+
49+
### About the respondents
50+
51+
- 79% are using OTel in production.
52+
- 21% work for an observability or APM vendor.
53+
- 98% have previous knowledge of observability: intermediate (60%) or expert
54+
(38%).
55+
56+
#### Q: What source do you primarily rely on when you're looking for information about OpenTelemetry?
57+
58+
- Overall, the majority of respondents (52%) rely on the [opentelemetry.io]
59+
documentation.
60+
- Respondents early in their observability practice (beginner and intermediate)
61+
are more likely to use the [opentelemetry.io] documentation.
62+
- Expert observability practitioners prefer the code repository documentation.
63+
64+
[opentelemetry.io]: /docs/
65+
66+
> **Respondents who use [opentelemetry.io] as their primary information**
67+
> source<br>_By level of observability knowledge_
68+
>
69+
> | Beginners | Intermediates | Experts |
70+
> | :-------: | :-----------: | :-----: |
71+
> | 100% | 62% | 44% |
72+
73+
### Documentation wish list
74+
75+
#### Q: What features or information would you like to see added to opentelemetry.io that aren't currently available?
76+
77+
We asked respondents to describe in their own words what they'd like to see
78+
added to the [opentelemetry.io] documentation. We loosely grouped their
79+
responses into six categories. Some answers fell in more than one category. For
80+
full responses, see [Docs Usability Survey Responses].
81+
82+
[Docs Usability Survey Responses]:
83+
https://docs.google.com/spreadsheets/d/1kpJQYiEGtpZorICbl-QfYL3mKfeoRLiUywvKcV8fcNA
84+
85+
- More examples: 17 (35%)
86+
- Deeper or broader coverage: 13 (27%)
87+
- Better structure: 8 (17%)
88+
- Add code repository docs: 5 (10%)
89+
- Other: 2 (4%)
90+
- No response: 7 (15%)
91+
92+
![Horizontal bar graph showing the number of feature requests by category](feature-request.png)
93+
94+
#### Q: Would more visual aids (e.g., diagrams and screenshots) explaining OpenTelemetry concepts be helpful?
95+
96+
An overwhelming 81% said yes: they want more visual aids.
97+
98+
### Current state of the docs
99+
100+
#### Q: How well do the current docs at opentelemetry.io explain the different components of OpenTelemetry?
101+
102+
Most respondents felt the component conceptual documentation was average, with a
103+
top score of 3.
104+
105+
![Stacked row chart showing the percentage of users who rated the component conceptual documentation from 1 (low) to 5 (high)](component-explanations.png)
106+
107+
#### Q: How straightforward and user-friendly are the installation instructions for OpenTelemetry?
108+
109+
Most respondents found OTel instructions better than average, with a top score
110+
of 4. Respondents with intermediate-level observability knowledge rated them
111+
higher than experts: 55% of intermediates rated the installation instructions 4
112+
or 5, compared with only 17% of experts.
113+
114+
![Stacked row chart showing the percentage of users who rated the installation instructions from 1 (low) to 5 (high)](installation-instructions.png)
115+
116+
#### Q: How comprehensive are the troubleshooting sections?
117+
118+
Most respondents believe that this section of docs needs work. Only 15% rated
119+
the troubleshooting docs 4 or 5, and they were all intermediate-level
120+
respondents. None of the expert-level respondents rated the troubleshooting docs
121+
above a 3.
122+
123+
![Stacked row chart showing the percentage of users who rated the troubleshooting documentation from 1 (low) to 5 (high)](troubleshooting.png)
124+
125+
#### Q: How would you rate your experience using the current OTel documentation for the following languages and components?
126+
127+
Respondents were asked to rate only the documentation that applied to them, so
128+
we can infer based on their responses which docs sets are the most used.
129+
130+
- The Collector documentation is the most used: 77% of respondents rated it.
131+
- The next five documentation sets are close in popularity, with 50 to 67% of
132+
respondents rating them.
133+
134+
![Horizontal bar graph showing the number of ratings per OTel component](top-six-bar.png)
135+
136+
Here are the tabulated ratings for all languages and components. When the
137+
results are normalized and weighted, we can see additional insights:
138+
139+
- The Java documentation has the highest overall rating.
140+
- The Swift documentation has the lowest overall rating.
141+
142+
> **How would you rate your experience using the current OTel documentation for
143+
> the following languages and components?**
144+
>
145+
> | Language or component | Poor | Okay | Great | Total responses | Normalized & weighted |
146+
> | :-------------------- | :--: | :--: | :---: | :-------------: | :-------------------: |
147+
> | Java | 3 | 16 | 8 | 27 | 7.3333 |
148+
> | PHP | 1 | 4 | 2 | 7 | 7.1429 |
149+
> | GO | 6 | 12 | 9 | 27 | 7.1111 |
150+
> | Collector | 9 | 17 | 11 | 37 | 6.8108 |
151+
> | Python | 6 | 17 | 8 | 31 | 6.7742 |
152+
> | Kubernetes | 6 | 20 | 6 | 32 | 6.3750 |
153+
> | C++ | 0 | 7 | 0 | 7 | 6.0000 |
154+
> | JavaScript | 3 | 19 | 2 | 24 | 6.0000 |
155+
> | Ruby | 1 | 5 | 1 | 7 | 6.2857 |
156+
> | Rust | 4 | 4 | 2 | 10 | 5.6000 |
157+
> | .NET | 4 | 8 | 2 | 14 | 5.7143 |
158+
> | Erlang | 1 | 6 | 0 | 7 | 5.4286 |
159+
> | FaaS | 5 | 7 | 0 | 12 | 4.3333 |
160+
> | Swift | 3 | 3 | 0 | 6 | 4.0000 |
161+
> | **Total** | 52 | 145 | 51 | | |
162+
163+
If we combine these insights, we can see that the documentation used by the most
164+
people that needs the most work is the JavaScript documentation.
165+
166+
> **Where should we focus our improvement efforts?**<br> _JavaScript is one of
167+
> the six most-used docs sets, but its rating is the lowest._
168+
>
169+
> | Language or component | Poor | Okay | Great | Total responses | Normalized & weighted |
170+
> | :-------------------- | :--: | :--: | :---: | :-------------: | :-------------------: |
171+
> | GO | 6 | 12 | 9 | 27 | 7.1111 |
172+
> | Java | 3 | 16 | 8 | 27 | 7.3333 |
173+
> | JavaScript | 3 | 19 | 2 | 24 | 6.0000 |
174+
> | Python | 6 | 17 | 8 | 31 | 6.7742 |
175+
> | Collector | 9 | 17 | 11 | 37 | 6.8108 |
176+
> | Kubernetes | 6 | 20 | 6 | 32 | 6.3750 |
177+
178+
## Learn more
179+
180+
For detailed survey results, see [Docs Usability Survey Responses].
181+
182+
## Your feedback is essential
183+
184+
Thanks again to everyone who participated in the survey! Your feedback is
185+
crucial for guiding the future development of OpenTelemetry and ensuring it
186+
continues to meet your evolving needs. Stay connected and learn about upcoming
187+
surveys through the following channels:
188+
189+
- [#otel-sig-end-user Slack channel](https://cloud-native.slack.com/archives/C01RT3MSWGZ)
190+
- [#otel-comms Slack channel](https://cloud-native.slack.com/archives/C02UN96HZH6)
191+
- [End user resources page](/community/end-user/)

‎content/en/blog/2024/otel-docs-survey/installation-instructions.png

17.2 KB
Loading

‎content/en/blog/2024/otel-docs-survey/top-six-bar.png

38 KB
Loading

‎content/en/blog/2024/otel-docs-survey/troubleshooting.png

16.2 KB
Loading

‎static/refcache.json

+8
Original file line numberDiff line numberDiff line change
@@ -2783,6 +2783,14 @@
27832783
"StatusCode": 200,
27842784
"LastSeen": "2024-01-30T06:05:57.197702-05:00"
27852785
},
2786+
"https://docs.google.com/spreadsheets/d/1kpJQYiEGtpZorICbl-QfYL3mKfeoRLiUywvKcV8fcNA": {
2787+
"StatusCode": 200,
2788+
"LastSeen": "2024-12-17T17:45:19.478549783-08:00"
2789+
},
2790+
"https://docs.google.com/spreadsheets/d/1kpJQYiEGtpZorICbl-QfYL3mKfeoRLiUywvKcV8fcNA/edit": {
2791+
"StatusCode": 200,
2792+
"LastSeen": "2024-12-13T20:24:13.356675448-08:00"
2793+
},
27862794
"https://docs.gradle.org/current/userguide/dependency_resolution.html#sec:version-conflict": {
27872795
"StatusCode": 206,
27882796
"LastSeen": "2024-07-23T10:18:05.857983751+02:00"

0 commit comments

Comments
 (0)