|
1 | | -00:00 Let me introduce you to another program |
2 | | -00:02 that we're going to use here. |
3 | | -00:03 This is the web scraping example that I just spoke about |
4 | | -00:05 and it's going to use a couple of libraries requests |
5 | | -00:07 and Beautiful Soup, and so I've added |
6 | | -00:09 a requirements file for it. |
7 | | -00:10 Just start by Pip installing those requirements |
8 | | -00:13 which we can do by copying the path like that. |
9 | | -00:19 Okay, so we have our code, and this is just |
10 | | -00:20 where we're going to start from. |
11 | | -00:22 We don't have to write anything |
12 | | -00:23 but I want to walk you through it |
13 | | -00:24 so you see what it is we're doing |
14 | | -00:27 when we get to the async stuff. |
15 | | -00:28 So there's a couple methods up here, get_html and get_title. |
16 | | -00:32 Now the get_title is purely in memory |
17 | | -00:35 CPU bound sort of thing. |
18 | | -00:37 It's going to use Beautiful Soup which is a library |
19 | | -00:39 that understands the HTML DOM in memory |
20 | | -00:42 on the client side like here, and it's going to let you |
21 | | -00:45 do queries against it that are basically like CSS queries. |
22 | | -00:49 So we're going to say give me the header |
23 | | -00:51 and either we're going to get missing |
24 | | -00:52 or it'll give us the text cleaned up out of the header. |
25 | | -00:56 This get_html is more interesting. |
26 | | -00:58 We're going to give it an episode number |
27 | | -01:00 so we're going to go to my podcast talkpython.fm |
28 | | -01:03 and it's going to use this like short, little URL here |
29 | | -01:06 this shortcut, talkpython.fm/<some episode number> |
30 | | -01:09 and it will go and redirect us to the right place |
31 | | -01:12 follow that redirect, and then get that data. |
32 | | -01:15 So we're just going to use requests, do a get on the URL. |
33 | | -01:18 We're going to verify that that worked |
34 | | -01:20 and then we're going to return just the text |
35 | | -01:23 and if we look down here, we're going to say get_title. |
36 | | -01:26 So what it's going to do is go from episode 150 to 160. |
37 | | -01:31 It's going to first get the HTML, and then it's going to |
38 | | -01:34 get the title, and then it's going to print. |
39 | | -01:36 Now of course this is all serial code, right? |
40 | | -01:39 This is in order, so when we run this |
41 | | -01:42 don't expect any speedups, but you should see it working. |
42 | | -01:48 So here it is, it's getting the HTML for episode 150 |
43 | | -01:52 then it's getting the title, and it found that |
44 | | -01:54 it's Technical Lessons Learned from Pythonic Refactoring |
45 | | -01:57 then 151, and then Gradual Typing |
46 | | -02:02 for Production Applications, and so on |
47 | | -02:03 and we're finally eventually done. |
48 | | -02:06 Let me just run it one more time |
49 | | -02:07 without me around and just watch how long it takes |
50 | | -02:09 get a sense for the speed here. It's doing one. |
51 | | -02:12 It's waiting on the server to respond. |
52 | | -02:14 It's getting a response back. It's not terribly slow. |
53 | | -02:16 My website's pretty fast, but it's not that fast |
54 | | -02:20 and of course, I'm on the West Coast of the United States |
55 | | -02:23 the server is on the East Coast. |
56 | | -02:24 There's a long ping time just to go from |
57 | | -02:29 west to east back, about a 100 milliseconds, I'm guessing. |
58 | | -02:34 There's a lot of places for improvement here. |
59 | | -02:37 That's what we're going to work with |
60 | | -02:38 and our job is going to be to take this program |
61 | | -02:41 apply asyncio to the places where we're waiting |
62 | | -02:44 namely that line right there |
63 | | -02:47 you're doing lots of waiting there |
64 | | -02:48 and make this go much, much, faster. |
65 | | -02:50 And it turns out, the algorithm that we have here |
66 | | -02:53 won't actually make it go faster, at least for |
67 | | -02:56 this particular application. |
68 | | -02:58 If we had a bunch of different attempts running parallel |
69 | | -03:01 it's a really straightforward transition |
70 | | -03:02 but there's one more thing to be learned |
71 | | -03:04 to make this actually go faster in pure performance. |
| 1 | +00:00 Let me introduce you to another program |
| 2 | +00:02 that we're going to use here. |
| 3 | +00:03 This is the web scraping example that I just spoke about |
| 4 | +00:05 and it's going to use a couple of libraries requests |
| 5 | +00:07 and Beautiful Soup, and so I've added |
| 6 | +00:09 a requirements file for it. |
| 7 | +00:10 Just start by pip installing those requirements |
| 8 | +00:13 which we can do by copying the path like that. |
| 9 | +00:19 Okay, so we have our code, and this is just |
| 10 | +00:20 where we're going to start from. |
| 11 | +00:22 We don't have to write anything |
| 12 | +00:23 but I want to walk you through it |
| 13 | +00:24 so you see what it is we're doing |
| 14 | +00:27 when we get to the async stuff. |
| 15 | +00:28 So there's a couple methods up here, get_html and get_title. |
| 16 | +00:32 Now the get_title is purely in memory |
| 17 | +00:35 CPU bound sort of thing. |
| 18 | +00:37 It's going to use Beautiful Soup which is a library |
| 19 | +00:39 that understands the HTML DOM in memory |
| 20 | +00:42 on the client side like here, and it's going to let you |
| 21 | +00:45 do queries against it that are basically like CSS queries. |
| 22 | +00:49 So we're going to say give me the header |
| 23 | +00:51 and either we're going to get missing |
| 24 | +00:52 or it'll give us the text cleaned up out of the header. |
| 25 | +00:56 This get_html is more interesting. |
| 26 | +00:58 We're going to give it an episode number |
| 27 | +01:00 so we're going to go to my podcast talkpython.fm |
| 28 | +01:03 and it's going to use this like short, little URL here |
| 29 | +01:06 this shortcut, talkpython.fm/<some episode number> |
| 30 | +01:09 and it will go and redirect us to the right place |
| 31 | +01:12 follow that redirect, and then get that data. |
| 32 | +01:15 So we're just going to use requests, do a get on the URL. |
| 33 | +01:18 We're going to verify that that worked |
| 34 | +01:20 and then we're going to return just the text |
| 35 | +01:23 and if we look down here, we're going to say get_title. |
| 36 | +01:26 So what it's going to do is go from episode 150 to 160. |
| 37 | +01:31 It's going to first get the HTML, and then it's going to |
| 38 | +01:34 get the title, and then it's going to print. |
| 39 | +01:36 Now of course this is all serial code, right? |
| 40 | +01:39 This is in order, so when we run this |
| 41 | +01:42 don't expect any speedups, but you should see it working. |
| 42 | +01:48 So here it is, it's getting the HTML for episode 150 |
| 43 | +01:52 then it's getting the title, and it found that |
| 44 | +01:54 it's Technical Lessons Learned from Pythonic Refactoring |
| 45 | +01:57 then 151, and then Gradual Typing |
| 46 | +02:02 for Production Applications, and so on |
| 47 | +02:03 and we're finally eventually done. |
| 48 | +02:06 Let me just run it one more time |
| 49 | +02:07 without me around and just watch how long it takes |
| 50 | +02:09 get a sense for the speed here. It's doing one. |
| 51 | +02:12 It's waiting on the server to respond. |
| 52 | +02:14 It's getting a response back. It's not terribly slow. |
| 53 | +02:16 My website's pretty fast, but it's not that fast |
| 54 | +02:20 and of course, I'm on the West Coast of the United States |
| 55 | +02:23 the server is on the East Coast. |
| 56 | +02:24 There's a long ping time just to go from |
| 57 | +02:29 west to east back, about a 100 milliseconds, I'm guessing. |
| 58 | +02:34 There's a lot of places for improvement here. |
| 59 | +02:37 That's what we're going to work with |
| 60 | +02:38 and our job is going to be to take this program |
| 61 | +02:41 apply asyncio to the places where we're waiting |
| 62 | +02:44 namely that line right there |
| 63 | +02:47 you're doing lots of waiting there |
| 64 | +02:48 and make this go much, much, faster. |
| 65 | +02:50 And it turns out, the algorithm that we have here |
| 66 | +02:53 won't actually make it go faster, at least for |
| 67 | +02:56 this particular application. |
| 68 | +02:58 If we had a bunch of different attempts running parallel |
| 69 | +03:01 it's a really straightforward transition |
| 70 | +03:02 but there's one more thing to be learned |
| 71 | +03:04 to make this actually go faster in pure performance. |
0 commit comments