Skip to content

Commit 09bb64b

Browse files
committed
Adds 'Intro to Streams' talk
1 parent 5d8e48d commit 09bb64b

File tree

1 file changed

+234
-0
lines changed

1 file changed

+234
-0
lines changed

intro_to_streams_by_tyler.markdown

+234
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,234 @@
1+
#Intro To Using Streams
2+
3+
* What is a stream?
4+
5+
* Why use streams?
6+
7+
* How do I use streams?
8+
---
9+
#What is a stream?
10+
11+
* The idea that programs that modify data should have a standard interface for communicating with each other.
12+
13+
* Data is passed as it is retrieved.
14+
---
15+
#Why use streams?
16+
17+
* Your program can use and be used by other programs, with little extra work.
18+
19+
* Encourages design of small programs that do one thing well, which has proven itself to be a dependable way to compose large systems.
20+
21+
* Seperation of concerns which helps reduce overall complexity of the code.
22+
23+
* Streams are used in many different languages and situations, and familiarity with the concept will allow you to flow easily between the details of the situation at hand.
24+
---
25+
#What is a stream?
26+
27+
Visualizing streams
28+
29+
* The garden hose:
30+
31+
> We should have some ways of connecting programs like garden hose--screw in another segment when it becomes necessary to massage data in another way.
32+
- Doug Mcilroy, Oct. 11, 1964 (thanks substack!)
33+
34+
* Water analogies ("back pressure", "overflow", etc.)
35+
---
36+
#How do I use streams? Experiment 1
37+
38+
You already know how! Streams have long been implemented in your operating system. Pull up a terminal and try this experiment.
39+
40+
```bash
41+
ls | grep x
42+
```
43+
where x is a part of a file name in your current directory.
44+
---
45+
#How do I use streams? Experiment 1
46+
47+
What just happened? First, some definitions:
48+
---
49+
#How do I use streams? Experiment 1
50+
51+
What just happened? First, some definitions:
52+
53+
The utility `ls` is a program that lists files in the current directory.
54+
---
55+
#How do I use streams? Experiment 1
56+
57+
What just happened? First, some definitions:
58+
59+
The utility `ls` is a program that lists files in the current directory.
60+
61+
The utility `grep` finds files that match the `pattern` of the text to the right of the word `grep`.
62+
---
63+
#How do I use streams? Experiment 1
64+
65+
What just happened? First, some definitions:
66+
67+
The utility `ls` is a program that lists files in the current directory.
68+
69+
The utility `grep` finds files that match the `pattern` of the text to the right of the word `grep`.
70+
71+
`|`, pronounced "pipe", will take the *output* of the program on the left and use it as the *input* of the program on the right.
72+
---
73+
#How do I use streams? Experiment 1
74+
75+
What just happened?
76+
77+
`ls` outputs ->
78+
which `|` takes and puts into the input of ->
79+
`grep` outputs ->
80+
text to your screen.
81+
---
82+
#How do I use streams? Experiment 1
83+
84+
`ls` and `grep` were solely designed to manipulate data.
85+
86+
`|` is designed to communicate that data between programs that know how to deal with `streams`. How does it do this? Here be dragons.
87+
---
88+
#How do I use streams? Experiment 2
89+
90+
To demonstrate the idea of `blocking` and how streams actually communicate to each other, lets try another experiment.
91+
---
92+
#How do I use streams? Experiment 2
93+
94+
First, create a "named pipe" using the utility `mkfifo`.
95+
96+
```bash
97+
mkfifo mypipe
98+
```
99+
100+
`ls` to see that there is a file called "mypipe" in your current directory.
101+
102+
`ls -l` gives you a bit more detail about the type of file "mypipe" is. Notice the "p", which notates that the file is a "pipe", in front of the typical file permissions.
103+
---
104+
#How do I use streams? Experiment 2
105+
106+
Now instead of using `|` to magically communicate between programs, we now have a sort of file representation of a pipe that we can "look at".
107+
108+
```bash
109+
ls -l > mypipe
110+
```
111+
in one terminal window and
112+
113+
```bash
114+
cat < mypipe
115+
```
116+
in another terminal window.
117+
---
118+
#How do I use streams? Experiment 2
119+
120+
The output of the command run on the first terminal shows up on the second, after you run the second command. It seems we have "slowed down" the process of `|`.
121+
122+
The order in which the run the commands doesn't matter. What matters is that one program is "writing" to the pipe, and the other is "reading." Both the writing and reading needs to happen in order for the pipe to finish `blocking` the control being returned to the user.
123+
---
124+
#What is a stream?
125+
126+
* An abstract definition with specific implementations that helps programs talk to each other!
127+
128+
And, in Node,
129+
130+
* An asynchronous, non-blocking implementation of unix-style `|`
131+
---
132+
#How do I use streams? Node
133+
134+
Node has its own `|` command, called pipe().
135+
136+
Lets see how pipe() works.
137+
---
138+
#How do I use streams? Node experiment 1
139+
140+
Borrowing from substacks `stream-adventure` module on npm, lets write some JavaScript that creates a very simple stream that takes input from the user and outputs it to the console, similar to our first pipe experiment.
141+
---
142+
#How do I use streams? Node experiment 1
143+
144+
First, create a JavaScript file:
145+
146+
```bash
147+
touch in-out.js
148+
```
149+
---
150+
#How do I use streams? Node experiment 1
151+
152+
Now, lets edit it to create a simple stream using Node's pipe().
153+
154+
Add this to "in-out.js":
155+
156+
```JavaScript
157+
process.stdin.pipe(process.stdout);
158+
```
159+
---
160+
#How do I use streams? Node experiment 1
161+
162+
Run "in-out.js" with Node:
163+
164+
```bash
165+
node in-out.js
166+
```
167+
168+
Notice that everytime you type something and press enter, the program responds by typing that same thing back to you!
169+
---
170+
#How do I use streams? Node experiment 1
171+
172+
In our very first experiment, this happened:
173+
174+
`ls` outputs ->
175+
which `|` takes and puts into the input of ->
176+
`grep` outputs ->
177+
text to your screen.
178+
179+
And in this experiment, this is happening:
180+
181+
stdin outputs ->
182+
which .pipe() takes and puts into the input of ->
183+
stdout outputs ->
184+
text to your screen.
185+
186+
(stdin and stdout are defined abstractly, but stand for "standard input / output" which, in this example, means keyboard input and terminal screen text output)
187+
---
188+
#How do I use streams? Node experiment 1
189+
190+
Notice how similar these two examples were!
191+
192+
Node has some differences then the unix `|`, most of which are sweet bonuses that you will come to appreciate.
193+
---
194+
#How do I use streams? Node
195+
196+
Oh, so I can just pipe anything and it will magically be "picked up"?
197+
---
198+
#How do I use streams? Node
199+
200+
Oh, so I can just pipe anything and it will magically be "picked up"?
201+
202+
No, but almost.
203+
---
204+
#How do I use streams? Node
205+
206+
pipe() is just a function that Node provides that takes a "readable stream" as its input and channels the output to a "writable stream."
207+
208+
There are five types of Node `stream`s: readable, writeable, transform, duplex and "classic." Each has its own purpose, though all of them can be pipe()'d.
209+
---
210+
#How do I use streams? Node
211+
212+
Notice that in our previous example:
213+
214+
```JavaScript
215+
process.stdin.pipe(process.stdout);
216+
```
217+
218+
`process.stdin` is a readable stream and process.stdout is a writeable stream, so pipe() knows what to do with them.
219+
220+
When you want your data to be piped, you will have to "streamify" your data.
221+
---
222+
#How do I use streams? Node
223+
224+
From a high level, "all you will need to do" is create the correct type of stream you want your data to be represented by and follow its conventions for using it.
225+
226+
Of course, the details can be a bit messier then that. Details for a future talk!
227+
---
228+
# Continued learning and citations.
229+
230+
For an excellent beginner to pro Node-style streams manual, see substack's [stream-handbook](https://github.com/substack/stream-handbook)
231+
232+
For more on the first two experiments, see this article on the [Linux Journal](http://www.linuxjournal.com/article/2156) which goes into much greater detail named pipes in unix.
233+
234+
For some great practice with creating and using pipes in Node (and where I got the idea for the first Node experiment), checkout the [stream-adventures](https://www.npmjs.com/package/stream-adventure) module by substack.

0 commit comments

Comments
 (0)