-
Notifications
You must be signed in to change notification settings - Fork 0
/
atom.xml
543 lines (332 loc) · 423 KB
/
atom.xml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
<?xml version="1.0" encoding="utf-8"?>
<feed xmlns="http://www.w3.org/2005/Atom">
<title>ZJ_BLOG</title>
<subtitle>Silence的博客</subtitle>
<link href="/atom.xml" rel="self"/>
<link href="http://zj2626.github.io/"/>
<updated>2021-06-03T13:33:22.031Z</updated>
<id>http://zj2626.github.io/</id>
<author>
<name>一起沉默</name>
</author>
<generator uri="http://hexo.io/">Hexo</generator>
<entry>
<title>4.OutOfMemoryError</title>
<link href="http://zj2626.github.io/2021/06/01/20210601001_OutOfMemoryError/"/>
<id>http://zj2626.github.io/2021/06/01/20210601001_OutOfMemoryError/</id>
<published>2021-06-01T14:21:10.642Z</published>
<updated>2021-06-03T13:33:22.031Z</updated>
<content type="html"><![CDATA[<h2 id="1-溢出"><a href="#1-溢出" class="headerlink" title="1. 溢出"></a>1. 溢出</h2><p>虚拟机规范中, 除了程序计数器, 其他部分都可能发生OOM异常</p><h2 id="2-溢出种类"><a href="#2-溢出种类" class="headerlink" title="2.溢出种类"></a>2.溢出种类</h2><blockquote><p>java堆存储对象实例, 只需不断创建对象且GC Roots到对象可达避免垃圾回收, 达到最大堆限制即溢出</p></blockquote><figure class="highlight java"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line"></span><br><span class="line"></span><br></pre></td></tr></table></figure><a id="more"></a><h3 id="1-Java堆溢出"><a href="#1-Java堆溢出" class="headerlink" title="1.Java堆溢出"></a>1.Java堆溢出</h3><p><img src="20210601001_OutOfMemoryError/image-20210603213230669.png" alt="image-20210603213230669"></p><!--more--><h3 id="2-虚拟机栈和本地方法栈溢出"><a href="#2-虚拟机栈和本地方法栈溢出" class="headerlink" title="2.虚拟机栈和本地方法栈溢出"></a>2.虚拟机栈和本地方法栈溢出</h3><h3 id="3-方法去和运行时常量池溢出"><a href="#3-方法去和运行时常量池溢出" class="headerlink" title="3.方法去和运行时常量池溢出"></a>3.方法去和运行时常量池溢出</h3><h3 id="4-本机直接内存溢出"><a href="#4-本机直接内存溢出" class="headerlink" title="4.本机直接内存溢出"></a>4.本机直接内存溢出</h3><!--more-->]]></content>
<summary type="html">
<h2 id="1-溢出"><a href="#1-溢出" class="headerlink" title="1. 溢出"></a>1. 溢出</h2><p>虚拟机规范中, 除了程序计数器, 其他部分都可能发生OOM异常</p>
<h2 id="2-溢出种类"><a href="#2-溢出种类" class="headerlink" title="2.溢出种类"></a>2.溢出种类</h2><blockquote>
<p>java堆存储对象实例, 只需不断创建对象且GC Roots到对象可达避免垃圾回收, 达到最大堆限制即溢出</p>
</blockquote>
<figure class="highlight java"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line"></span><br><span class="line"></span><br></pre></td></tr></table></figure>
</summary>
<category term="java虚拟机" scheme="http://zj2626.github.io/categories/java%E8%99%9A%E6%8B%9F%E6%9C%BA/"/>
<category term="深入了解java虚拟机" scheme="http://zj2626.github.io/tags/%E6%B7%B1%E5%85%A5%E4%BA%86%E8%A7%A3java%E8%99%9A%E6%8B%9F%E6%9C%BA/"/>
<category term="java" scheme="http://zj2626.github.io/tags/java/"/>
</entry>
<entry>
<title>使用多种算法对泰坦尼克号乘客获救原因进行分析</title>
<link href="http://zj2626.github.io/2021/06/01/2018031301/"/>
<id>http://zj2626.github.io/2021/06/01/2018031301/</id>
<published>2021-06-01T14:21:10.641Z</published>
<updated>2021-06-01T01:18:34.851Z</updated>
<content type="html"><![CDATA[<figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">import</span> pandas</span><br><span class="line">titanic = pandas.read_csv(<span class="string">'titanic_train.csv'</span>)</span><br><span class="line">titanic.head(<span class="number">20</span>)</span><br></pre></td></tr></table></figure><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment"># 数据简单统计</span></span><br><span class="line">titanic.describe()</span><br></pre></td></tr></table></figure><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment"># 由表Age列有部分缺失，影响最终模型效果</span></span><br><span class="line"><span class="comment"># 解决方法： 填充中位数 (使用fillna()方法填充缺失值NaN， 使用median()获得中位数/中值)</span></span><br><span class="line">titanic[<span class="string">'Age'</span>] = titanic[<span class="string">'Age'</span>].fillna(titanic[<span class="string">'Age'</span>].median())</span><br><span class="line">titanic.describe()</span><br></pre></td></tr></table></figure><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment"># 处理原始数据，其中有的是字符量需要映射为数值</span></span><br><span class="line"><span class="keyword">print</span> (titanic[<span class="string">'Sex'</span>].unique())</span><br><span class="line">titanic.loc[titanic[<span class="string">'Sex'</span>] == <span class="string">'male'</span>, <span class="string">'Sex'</span>] = <span class="number">0</span></span><br><span class="line">titanic.loc[titanic[<span class="string">'Sex'</span>] == <span class="string">'female'</span>, <span class="string">'Sex'</span>] = <span class="number">1</span></span><br><span class="line"></span><br><span class="line"><span class="keyword">print</span> (titanic[<span class="string">'Embarked'</span>].unique())</span><br><span class="line">titanic[<span class="string">'Embarked'</span>] = titanic[<span class="string">'Embarked'</span>].fillna(<span class="string">'S'</span>)<span class="comment"># 缺失值填充，填充数量最多的类别</span></span><br><span class="line">titanic.loc[titanic[<span class="string">'Embarked'</span>] == <span class="string">'S'</span>, <span class="string">'Embarked'</span>] = <span class="number">0</span></span><br><span class="line">titanic.loc[titanic[<span class="string">'Embarked'</span>] == <span class="string">'C'</span>, <span class="string">'Embarked'</span>] = <span class="number">1</span></span><br><span class="line">titanic.loc[titanic[<span class="string">'Embarked'</span>] == <span class="string">'Q'</span>, <span class="string">'Embarked'</span>] = <span class="number">2</span></span><br><span class="line"></span><br><span class="line">titanic.head()</span><br></pre></td></tr></table></figure><h3 id="线性回归-步骤"><a href="#线性回归-步骤" class="headerlink" title="线性回归 步骤"></a>线性回归 步骤</h3><ol><li>观察数据，填充缺失值，改变数据形式（数据映射）</li><li>使用交叉验证减少过拟合风险</li><li>使用线性回归算法计算预测值</li><li>确定阈值，计算精度</li></ol><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment"># 线性回归</span></span><br><span class="line"><span class="keyword">from</span> sklearn.linear_model <span class="keyword">import</span> LinearRegression</span><br><span class="line"><span class="keyword">from</span> sklearn.cross_validation <span class="keyword">import</span> KFold</span><br><span class="line"></span><br><span class="line"><span class="comment"># 要使用的特征 船仓等级、性别、年龄、兄弟姐妹人数、老人孩子人数、船票价格、上船位置</span></span><br><span class="line">predictors = [<span class="string">'Pclass'</span>, <span class="string">'Sex'</span>, <span class="string">'Age'</span>, <span class="string">'SibSp'</span>, <span class="string">'Parch'</span>, <span class="string">'Fare'</span>, <span class="string">'Embarked'</span>]</span><br><span class="line"></span><br><span class="line"><span class="comment"># 构建线性回归</span></span><br><span class="line">alg = LinearRegression()</span><br><span class="line">kf = KFold(titanic.shape[<span class="number">0</span>], n_folds=<span class="number">3</span>, random_state=<span class="number">0</span>)</span><br><span class="line"></span><br><span class="line">predictions0 = []</span><br><span class="line"><span class="keyword">for</span> train, test <span class="keyword">in</span> kf:</span><br><span class="line"> <span class="comment"># print (train.shape, test.shape)</span></span><br><span class="line"> train_predictors = titanic.loc[train, predictors]</span><br><span class="line"> train_targets = titanic.loc[train, [<span class="string">'Survived'</span>]]</span><br><span class="line"> <span class="keyword">print</span> (train_predictors.shape, train_targets.shape)</span><br><span class="line"> test_features = titanic.loc[test, predictors]</span><br><span class="line"> </span><br><span class="line"> <span class="comment"># 训练 拟合数据 使用交叉验证拆分出来的训练集</span></span><br><span class="line"> alg.fit(train_predictors, train_targets)</span><br><span class="line"> <span class="comment"># 验证 使用交叉验证拆分出来的验证集</span></span><br><span class="line"> test_predictions = alg.predict(test_features)</span><br><span class="line"> </span><br><span class="line"> predictions0.append(test_predictions)</span><br><span class="line"></span><br><span class="line"><span class="comment"># 三组预测值</span></span><br><span class="line"><span class="keyword">print</span> (len(predictions0), <span class="string">'\n\n'</span>)</span><br><span class="line"></span><br><span class="line"><span class="keyword">import</span> numpy <span class="keyword">as</span> np</span><br><span class="line"><span class="comment"># 设定一个阈值 大于阈值表示获救 小于阈值表示未获救</span></span><br><span class="line">predictions = np.concatenate(predictions0, axis=<span class="number">0</span>)</span><br><span class="line"></span><br><span class="line">predictions[predictions > <span class="number">0.5</span>] = <span class="number">1</span></span><br><span class="line">predictions[predictions <= <span class="number">0.5</span>] = <span class="number">0</span></span><br><span class="line"></span><br><span class="line"><span class="keyword">print</span> (type(predictions), type(titanic[<span class="string">'Survived'</span>].values))</span><br><span class="line"><span class="keyword">print</span> (predictions.shape,titanic[<span class="string">'Survived'</span>].values.reshape(<span class="number">-1</span>, <span class="number">1</span>).shape)</span><br><span class="line"></span><br><span class="line">accuracy = len(predictions[predictions == titanic[<span class="string">'Survived'</span>].values.reshape(<span class="number">-1</span>, <span class="number">1</span>)]) / len(predictions)</span><br><span class="line"><span class="keyword">print</span> (accuracy)</span><br></pre></td></tr></table></figure><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment"># 随机森林</span></span><br><span class="line"><span class="keyword">from</span> sklearn <span class="keyword">import</span> cross_validation</span><br><span class="line"><span class="keyword">from</span> sklearn.ensemble <span class="keyword">import</span> RandomForestClassifier</span><br><span class="line"></span><br><span class="line">predictors = [<span class="string">'Pclass'</span>, <span class="string">'Sex'</span>, <span class="string">'Age'</span>, <span class="string">'SibSp'</span>, <span class="string">'Parch'</span>, <span class="string">'Fare'</span>, <span class="string">'Embarked'</span>]</span><br></pre></td></tr></table></figure><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment"># 构造随机森林</span></span><br><span class="line"><span class="comment"># n_estimators：构造的树个数 min_samples_split：最小切分点 min_samples_leaf：叶子节点最小个数</span></span><br><span class="line">alg = RandomForestClassifier(random_state=<span class="number">1</span>, n_estimators=<span class="number">10</span>, min_samples_split=<span class="number">2</span>, min_samples_leaf=<span class="number">1</span>)</span><br><span class="line"><span class="comment"># 交叉验证</span></span><br><span class="line">kf = cross_validation.KFold(titanic.shape[<span class="number">0</span>], n_folds=<span class="number">3</span>, random_state=<span class="number">1</span>)</span><br><span class="line"><span class="comment"># 准确率</span></span><br><span class="line">scores = cross_validation.cross_val_score(alg, titanic[predictors], titanic[<span class="string">'Survived'</span>], cv=kf)</span><br><span class="line"><span class="comment"># 平均准确率</span></span><br><span class="line"><span class="keyword">print</span> (scores.mean())</span><br></pre></td></tr></table></figure><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment"># 随机森林调优： 修改参数 </span></span><br><span class="line"><span class="string">'''</span></span><br><span class="line"><span class="string">> RandomForestClassifier参数：</span></span><br><span class="line"><span class="string">1. max_features:随机森林允许单个决策树使用特征的最大数量, 增加max_features一般能提高模型的性能,同时降低算法的速度</span></span><br><span class="line"><span class="string">2. n_estimators:在利用最大投票数或平均值来预测之前，想要建立子树的数量。较多的子树可以让模型有更好的性能，但同时让你的代码变慢</span></span><br><span class="line"><span class="string">3. min_sample_leaf:最小样本叶片大小</span></span><br><span class="line"><span class="string">'''</span></span><br><span class="line"><span class="comment">#1 构建随机森林模型(分类器)</span></span><br><span class="line">alg = RandomForestClassifier(random_state=<span class="number">1</span>, n_estimators=<span class="number">50</span>, min_samples_split=<span class="number">4</span>, min_samples_leaf=<span class="number">10</span>)</span><br><span class="line"><span class="comment">#2 构建交叉验证</span></span><br><span class="line">kf = cross_validation.KFold(titanic.shape[<span class="number">0</span>], n_folds=<span class="number">3</span>, random_state=<span class="number">1</span>)</span><br><span class="line"></span><br><span class="line"><span class="string">'''</span></span><br><span class="line"><span class="string">> cross_validation.cross_val_score参数：</span></span><br><span class="line"><span class="string">1. alg:分类器，可以是任何的分类器，比如支持向量机分类器。alg = svm.SVC(kernel='linear', C=1)</span></span><br><span class="line"><span class="string">2. cv：交叉验证（cross validation）方法，如果cv是一个int数字的话，并且如果提供了raw target参数，那么就代表使用StratifiedKFold分类方式，如果没有提供raw target参数，那么就代表使用KFold分类方式。</span></span><br><span class="line"><span class="string">3. raw data,raw target：验证使用的数据（feature以及label）</span></span><br><span class="line"><span class="string"></span></span><br><span class="line"><span class="string">4. 返回值：对于每次不同的的划分raw data时，在test data（验证集）上得到的分类的准确率</span></span><br><span class="line"><span class="string">'''</span></span><br><span class="line"><span class="comment">#3 进行交叉验证</span></span><br><span class="line">scores = cross_validation.cross_val_score(alg, titanic[predictors], titanic[<span class="string">'Survived'</span>], cv=kf)</span><br><span class="line"><span class="keyword">print</span> (len(scores), <span class="string">'-----'</span>, scores.mean(), <span class="string">'-----'</span>, scores)</span><br></pre></td></tr></table></figure><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment"># 随机森林调优： 添加特征</span></span><br><span class="line">titanic[<span class="string">'FamilySize'</span>] = titanic[<span class="string">'SibSp'</span>] + titanic[<span class="string">'Parch'</span>]</span><br><span class="line">titanic[<span class="string">'NameLength'</span>] = titanic[<span class="string">'Name'</span>].apply(<span class="keyword">lambda</span> x: len(x))</span><br></pre></td></tr></table></figure><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment"># 特征重要程度</span></span><br><span class="line"><span class="keyword">import</span> numpy <span class="keyword">as</span> np</span><br><span class="line"><span class="keyword">from</span> sklearn.feature_selection <span class="keyword">import</span> SelectKBest,f_classif</span><br><span class="line"><span class="keyword">import</span> matplotlib.pyplot <span class="keyword">as</span> plt</span><br><span class="line">predictors = [<span class="string">'Pclass'</span>, <span class="string">'Sex'</span>, <span class="string">'Age'</span>, <span class="string">'SibSp'</span>, <span class="string">'Parch'</span>, <span class="string">'Fare'</span>, <span class="string">'Embarked'</span>, <span class="string">'FamilySize'</span>, <span class="string">'NameLength'</span>]</span><br></pre></td></tr></table></figure><h3 id="Univariate-feature-selection：单变量的特征选择"><a href="#Univariate-feature-selection：单变量的特征选择" class="headerlink" title="Univariate feature selection：单变量的特征选择"></a>Univariate feature selection：单变量的特征选择</h3><p>单变量特征选择的原理是分别单独的计算每个变量的某个统计指标，根据该指标来判断哪些指标重要。剔除那些不重要的指标。</p><p>sklearn.feature_selection模块中主要有以下几个方法：</p><blockquote><p>SelectKBest和SelectPercentile比较相似，前者选择排名排在前n个的变量，后者选择排名排在前n%的变量。而他们通过什么指标来给变量排名呢？这需要二外的指定。</p></blockquote><blockquote><p>对于regression问题，可以使用f_regression指标。对于classification问题，可以使用chi2或者f_classif变量。</p></blockquote><p><em>使用的例子：</em><br><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">from</span> sklearn.feature_selectionimport SelectPercentile, f_classif</span><br><span class="line">selector =SelectPercentile(f_classif, percentile=<span class="number">10</span>)</span><br></pre></td></tr></table></figure></p><h3 id="Recursive-feature-elimination：循环特征选择"><a href="#Recursive-feature-elimination：循环特征选择" class="headerlink" title="Recursive feature elimination：循环特征选择"></a>Recursive feature elimination：循环特征选择</h3><p>不单独的检验某个变量的价值，而是将其聚集在一起检验。它的基本思想是，对于一个数量为d的feature的集合，他的所有的子集的个数是2的d次方减1（包含空集）。指定一个外部的学习算法，比如SVM之类的。通过该算法计算所有子集的validationerror。选择error最小的那个子集作为所挑选的特征。</p><p>由以下两个方法实现：sklearn.feature_selection.RFE，sklearn.feature_selection.RFECV</p><h3 id="L1-based-featureselection："><a href="#L1-based-featureselection：" class="headerlink" title="L1-based featureselection："></a>L1-based featureselection：</h3><p>该思路的原理是：在linearregression模型中，有的时候会得到sparsesolution。意思是说很多变量前面的系数都等于0或者接近于0。这说明这些变量不重要，那么可以将这些变量去除。</p><h3 id="Tree-based-featureselection：决策树特征选择"><a href="#Tree-based-featureselection：决策树特征选择" class="headerlink" title="Tree-based featureselection：决策树特征选择"></a>Tree-based featureselection：决策树特征选择</h3><p>基于决策树算法做出特征选择</p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br></pre></td><td class="code"><pre><span class="line">selector = SelectKBest(f_classif, k=<span class="number">5</span>)</span><br><span class="line">selector.fit(titanic[predictors], titanic[<span class="string">'Survived'</span>])</span><br><span class="line"></span><br><span class="line">scores = -np.log10(selector.pvalues_)</span><br><span class="line"><span class="keyword">print</span> (scores)</span><br><span class="line"></span><br><span class="line">plt.bar(range(len(predictors)), scores)</span><br><span class="line">plt.xticks(range(len(predictors)), predictors, rotation=<span class="string">'vertical'</span>)</span><br><span class="line">plt.show()</span><br></pre></td></tr></table></figure><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment"># 多种算法集成预测</span></span><br><span class="line"><span class="keyword">from</span> sklearn.ensemble <span class="keyword">import</span> GradientBoostingClassifier</span><br><span class="line"><span class="keyword">from</span> sklearn.linear_model <span class="keyword">import</span> LogisticRegression</span><br><span class="line"><span class="keyword">import</span> numpy <span class="keyword">as</span> np</span><br><span class="line"></span><br><span class="line"><span class="comment"># 两个算法集成到一起使用</span></span><br><span class="line">algorithms = [</span><br><span class="line"> [GradientBoostingClassifier(random_state=<span class="number">1</span>, n_estimators=<span class="number">25</span>, max_depth=<span class="number">3</span>), [<span class="string">'Pclass'</span>, <span class="string">'Sex'</span>, <span class="string">'Age'</span>, <span class="string">'Fare'</span>, <span class="string">'Embarked'</span>, <span class="string">'FamilySize'</span>]],</span><br><span class="line"> [LogisticRegression(random_state=<span class="number">1</span>), [<span class="string">'Pclass'</span>, <span class="string">'Sex'</span>, <span class="string">'Age'</span>, <span class="string">'Fare'</span>, <span class="string">'Embarked'</span>]]</span><br><span class="line">]</span><br><span class="line"></span><br><span class="line">kf = KFold(titanic.shape[<span class="number">0</span>], n_folds=<span class="number">3</span>, random_state=<span class="number">1</span>)</span><br><span class="line"></span><br><span class="line">predictions = []</span><br><span class="line"></span><br><span class="line"><span class="comment"># 两个算法分别预测</span></span><br><span class="line"><span class="keyword">for</span> train, test <span class="keyword">in</span> kf:</span><br><span class="line"> train_target = titanic.loc[train, [<span class="string">'Survived'</span>]]</span><br><span class="line"> full_test_predictions = []</span><br><span class="line"> </span><br><span class="line"> <span class="comment"># 循环分类算法，获得不同算法求得的概率</span></span><br><span class="line"> <span class="keyword">for</span> alg, predictors <span class="keyword">in</span> algorithms:</span><br><span class="line"> <span class="comment"># alg：分类算法 predictors：特征</span></span><br><span class="line"> </span><br><span class="line"> <span class="comment"># 使用训练集 拟合数据</span></span><br><span class="line"> alg.fit(titanic.loc[train, predictors], train_target)</span><br><span class="line"> <span class="comment"># 使用验证集 计算 (这里 直接取预测的概率的第二个值-->[:, 1])</span></span><br><span class="line"> test_predictions = alg.predict_proba(titanic.loc[test, predictors].astype(float))[:, <span class="number">1</span>]</span><br><span class="line"> full_test_predictions.append(test_predictions)</span><br><span class="line"> </span><br><span class="line"> <span class="comment"># 使用不同的分类算法之后计算平均概率</span></span><br><span class="line"> test_predictions = (full_test_predictions[<span class="number">0</span>] + full_test_predictions[<span class="number">1</span>]) / <span class="number">2</span></span><br><span class="line"> test_predictions[test_predictions <= <span class="number">0.5</span>] = <span class="number">0</span></span><br><span class="line"> test_predictions[test_predictions > <span class="number">0.5</span>] = <span class="number">1</span></span><br><span class="line"> </span><br><span class="line"> predictions.append(test_predictions)</span><br><span class="line"></span><br><span class="line"><span class="comment"># 获得的分类结果</span></span><br><span class="line">predictions = np.concatenate(predictions, axis=<span class="number">0</span>)</span><br><span class="line"></span><br><span class="line"><span class="comment"># 计算精确度</span></span><br><span class="line">accuracy = len(predictions[predictions == titanic[<span class="string">'Survived'</span>]]) / len(predictions)</span><br><span class="line"><span class="keyword">print</span> (accuracy)</span><br></pre></td></tr></table></figure><blockquote><p>个人博客 欢迎来访： <a href="http://zj2626.github.io">http://zj2626.github.io</a></p></blockquote>]]></content>
<summary type="html">
使用多种算法对泰坦尼克号乘客获救原因进行分析
</summary>
<category term="机器学习" scheme="http://zj2626.github.io/categories/%E6%9C%BA%E5%99%A8%E5%AD%A6%E4%B9%A0/"/>
<category term="python" scheme="http://zj2626.github.io/tags/python/"/>
<category term="机器学习" scheme="http://zj2626.github.io/tags/%E6%9C%BA%E5%99%A8%E5%AD%A6%E4%B9%A0/"/>
</entry>
<entry>
<title>关于使用sklearn进行数据预处理-归一化/标准化/正则化(转)</title>
<link href="http://zj2626.github.io/2021/06/01/2018031202/"/>
<id>http://zj2626.github.io/2021/06/01/2018031202/</id>
<published>2021-06-01T14:21:10.638Z</published>
<updated>2021-06-01T01:18:34.847Z</updated>
<content type="html"><![CDATA[<h2 id="一、标准化（Z-Score），或者去除均值和方差缩放"><a href="#一、标准化（Z-Score），或者去除均值和方差缩放" class="headerlink" title="一、标准化（Z-Score），或者去除均值和方差缩放"></a>一、标准化（Z-Score），或者去除均值和方差缩放</h2><p>公式为：(X-mean)/std 计算时对每个属性/每列分别进行。</p><p>将数据按期属性（按列进行）减去其均值，并处以其方差。得到的结果是，对于每个属性/每列来说所有数据都聚集在0附近，方差为1。</p><p>实现时，有两种不同的方式：</p><ul><li>使用sklearn.preprocessing.scale()函数，可以直接将给定数据进行标准化。</li></ul><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br></pre></td><td class="code"><pre><span class="line"><span class="meta">>>> </span><span class="keyword">from</span> sklearn <span class="keyword">import</span> preprocessing</span><br><span class="line"><span class="meta">>>> </span><span class="keyword">import</span> numpy <span class="keyword">as</span> np</span><br><span class="line"><span class="meta">>>> </span>X = np.array([[ <span class="number">1.</span>, <span class="number">-1.</span>, <span class="number">2.</span>],</span><br><span class="line"><span class="meta">... </span> [ <span class="number">2.</span>, <span class="number">0.</span>, <span class="number">0.</span>],</span><br><span class="line"><span class="meta">... </span> [ <span class="number">0.</span>, <span class="number">1.</span>, <span class="number">-1.</span>]])</span><br><span class="line"><span class="meta">>>> </span>X_scaled = preprocessing.scale(X)</span><br><span class="line"> </span><br><span class="line"><span class="meta">>>> </span>X_scaled </span><br><span class="line">array([[ <span class="number">0.</span> ..., <span class="number">-1.22</span>..., <span class="number">1.33</span>...],</span><br><span class="line"> [ <span class="number">1.22</span>..., <span class="number">0.</span> ..., <span class="number">-0.26</span>...],</span><br><span class="line"> [<span class="number">-1.22</span>..., <span class="number">1.22</span>..., <span class="number">-1.06</span>...]])</span><br><span class="line"> </span><br><span class="line">>>><span class="comment">#处理后数据的均值和方差</span></span><br><span class="line"><span class="meta">>>> </span>X_scaled.mean(axis=<span class="number">0</span>)</span><br><span class="line">array([ <span class="number">0.</span>, <span class="number">0.</span>, <span class="number">0.</span>])</span><br><span class="line"> </span><br><span class="line"><span class="meta">>>> </span>X_scaled.std(axis=<span class="number">0</span>)</span><br><span class="line">array([ <span class="number">1.</span>, <span class="number">1.</span>, <span class="number">1.</span>])</span><br></pre></td></tr></table></figure><ul><li>使用sklearn.preprocessing.StandardScaler类，使用该类的好处在于可以保存训练集中的参数（均值、方差）直接使用其对象转换测试集数据。<figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br></pre></td><td class="code"><pre><span class="line"><span class="meta">>>> </span>scaler = preprocessing.StandardScaler().fit(X)</span><br><span class="line"><span class="meta">>>> </span>scaler</span><br><span class="line">StandardScaler(copy=<span class="literal">True</span>, with_mean=<span class="literal">True</span>, with_std=<span class="literal">True</span>)</span><br><span class="line"> </span><br><span class="line"><span class="meta">>>> </span>scaler.mean_ </span><br><span class="line">array([ <span class="number">1.</span> ..., <span class="number">0.</span> ..., <span class="number">0.33</span>...])</span><br><span class="line"> </span><br><span class="line"><span class="meta">>>> </span>scaler.std_ </span><br><span class="line">array([ <span class="number">0.81</span>..., <span class="number">0.81</span>..., <span class="number">1.24</span>...])</span><br><span class="line"> </span><br><span class="line"><span class="meta">>>> </span>scaler.transform(X) </span><br><span class="line">array([[ <span class="number">0.</span> ..., <span class="number">-1.22</span>..., <span class="number">1.33</span>...],</span><br><span class="line"> [ <span class="number">1.22</span>..., <span class="number">0.</span> ..., <span class="number">-0.26</span>...],</span><br><span class="line"> [<span class="number">-1.22</span>..., <span class="number">1.22</span>..., <span class="number">-1.06</span>...]])</span><br><span class="line"> </span><br><span class="line"> </span><br><span class="line">>>><span class="comment">#可以直接使用训练集对测试集数据进行转换</span></span><br><span class="line"><span class="meta">>>> </span>scaler.transform([[<span class="number">-1.</span>, <span class="number">1.</span>, <span class="number">0.</span>]]) </span><br><span class="line">array([[<span class="number">-2.44</span>..., <span class="number">1.22</span>..., <span class="number">-0.26</span>...]])</span><br></pre></td></tr></table></figure></li></ul><h2 id="二、将属性缩放到一个指定范围"><a href="#二、将属性缩放到一个指定范围" class="headerlink" title="二、将属性缩放到一个指定范围"></a>二、将属性缩放到一个指定范围</h2><p>除了上述介绍的方法之外，另一种常用的方法是将属性缩放到一个指定的最大和最小值（通常是1-0）之间，这可以通过preprocessing.MinMaxScaler类实现。</p><p>使用这种方法的目的包括：</p><ol><li><p>对于方差非常小的属性可以增强其稳定性。</p></li><li><p>维持稀疏矩阵中为0的条目</p></li></ol><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br></pre></td><td class="code"><pre><span class="line"><span class="meta">>>> </span>X_train = np.array([[ <span class="number">1.</span>, <span class="number">-1.</span>, <span class="number">2.</span>],</span><br><span class="line"><span class="meta">... </span> [ <span class="number">2.</span>, <span class="number">0.</span>, <span class="number">0.</span>],</span><br><span class="line"><span class="meta">... </span> [ <span class="number">0.</span>, <span class="number">1.</span>, <span class="number">-1.</span>]])</span><br><span class="line">...</span><br><span class="line"><span class="meta">>>> </span>min_max_scaler = preprocessing.MinMaxScaler()</span><br><span class="line"><span class="meta">>>> </span>X_train_minmax = min_max_scaler.fit_transform(X_train)</span><br><span class="line"><span class="meta">>>> </span>X_train_minmax</span><br><span class="line">array([[ <span class="number">0.5</span> , <span class="number">0.</span> , <span class="number">1.</span> ],</span><br><span class="line"> [ <span class="number">1.</span> , <span class="number">0.5</span> , <span class="number">0.33333333</span>],</span><br><span class="line"> [ <span class="number">0.</span> , <span class="number">1.</span> , <span class="number">0.</span> ]])</span><br><span class="line"> </span><br><span class="line"><span class="meta">>>> </span><span class="comment">#将相同的缩放应用到测试集数据中</span></span><br><span class="line"><span class="meta">>>> </span>X_test = np.array([[ <span class="number">-3.</span>, <span class="number">-1.</span>, <span class="number">4.</span>]])</span><br><span class="line"><span class="meta">>>> </span>X_test_minmax = min_max_scaler.transform(X_test)</span><br><span class="line"><span class="meta">>>> </span>X_test_minmax</span><br><span class="line">array([[<span class="number">-1.5</span> , <span class="number">0.</span> , <span class="number">1.66666667</span>]])</span><br><span class="line"> </span><br><span class="line"> </span><br><span class="line"><span class="meta">>>> </span><span class="comment">#缩放因子等属性</span></span><br><span class="line"><span class="meta">>>> </span>min_max_scaler.scale_ </span><br><span class="line">array([ <span class="number">0.5</span> , <span class="number">0.5</span> , <span class="number">0.33</span>...])</span><br><span class="line"> </span><br><span class="line"><span class="meta">>>> </span>min_max_scaler.min_ </span><br><span class="line">array([ <span class="number">0.</span> , <span class="number">0.5</span> , <span class="number">0.33</span>...])</span><br></pre></td></tr></table></figure><p>当然，在构造类对象的时候也可以直接指定最大最小值的范围：feature_range=(min, max)，此时应用的公式变为：</p><p>X_std=(X-X.min(axis=0))/(X.max(axis=0)-X.min(axis=0))</p><p>X_scaled=X_std/(max-min)+min</p><h2 id="三、正则化（Normalization）"><a href="#三、正则化（Normalization）" class="headerlink" title="三、正则化（Normalization）"></a>三、正则化（Normalization）</h2><p>正则化的过程是将每个样本缩放到单位范数（每个样本的范数为1），如果后面要使用如二次型（点积）或者其它核方法计算两个样本之间的相似性这个方法会很有用。</p><p>Normalization主要思想是对每个样本计算其p-范数，然后对该样本中每个元素除以该范数，这样处理的结果是使得每个处理后样本的p-范数（l1-norm,l2-norm）等于1。</p><pre><code>p-范数的计算公式：||X||p=(|x1|^p+|x2|^p+...+|xn|^p)^1/p</code></pre><p>该方法主要应用于文本分类和聚类中。例如，对于两个TF-IDF向量的l2-norm进行点积，就可以得到这两个向量的余弦相似性。</p><ol><li>可以使用preprocessing.normalize()函数对指定数据进行转换：</li></ol><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br></pre></td><td class="code"><pre><span class="line"><span class="meta">>>> </span>X = [[ <span class="number">1.</span>, <span class="number">-1.</span>, <span class="number">2.</span>],</span><br><span class="line"><span class="meta">... </span> [ <span class="number">2.</span>, <span class="number">0.</span>, <span class="number">0.</span>],</span><br><span class="line"><span class="meta">... </span> [ <span class="number">0.</span>, <span class="number">1.</span>, <span class="number">-1.</span>]]</span><br><span class="line"><span class="meta">>>> </span>X_normalized = preprocessing.normalize(X, norm=<span class="string">'l2'</span>)</span><br><span class="line"> </span><br><span class="line"><span class="meta">>>> </span>X_normalized </span><br><span class="line">array([[ <span class="number">0.40</span>..., <span class="number">-0.40</span>..., <span class="number">0.81</span>...],</span><br><span class="line"> [ <span class="number">1.</span> ..., <span class="number">0.</span> ..., <span class="number">0.</span> ...],</span><br><span class="line"> [ <span class="number">0.</span> ..., <span class="number">0.70</span>..., <span class="number">-0.70</span>...]])</span><br></pre></td></tr></table></figure><ol start="2"><li>可以使用processing.Normalizer()类实现对训练集和测试集的拟合和转换：</li></ol><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br></pre></td><td class="code"><pre><span class="line"><span class="meta">>>> </span>normalizer = preprocessing.Normalizer().fit(X) <span class="comment"># fit does nothing</span></span><br><span class="line"><span class="meta">>>> </span>normalizer</span><br><span class="line">Normalizer(copy=<span class="literal">True</span>, norm=<span class="string">'l2'</span>)</span><br><span class="line"> </span><br><span class="line">>>></span><br><span class="line"><span class="meta">>>> </span>normalizer.transform(X) </span><br><span class="line">array([[ <span class="number">0.40</span>..., <span class="number">-0.40</span>..., <span class="number">0.81</span>...],</span><br><span class="line"> [ <span class="number">1.</span> ..., <span class="number">0.</span> ..., <span class="number">0.</span> ...],</span><br><span class="line"> [ <span class="number">0.</span> ..., <span class="number">0.70</span>..., <span class="number">-0.70</span>...]])</span><br><span class="line"> </span><br><span class="line"><span class="meta">>>> </span>normalizer.transform([[<span class="number">-1.</span>, <span class="number">1.</span>, <span class="number">0.</span>]]) </span><br><span class="line">array([[<span class="number">-0.70</span>..., <span class="number">0.70</span>..., <span class="number">0.</span> ...]])</span><br></pre></td></tr></table></figure><p><em>补充：</em></p><img src="/2021/06/01/2018031202/091414004623860.png" title="如图 alt:图片说明 extend:?imageView2/2/w/800"><p>转载来自<a href="https://www.cnblogs.com/chaosimple/p/4153167.html" target="_blank" rel="noopener">这里</a></p><blockquote><p>个人博客 欢迎来访： <a href="http://zj2626.github.io">http://zj2626.github.io</a></p></blockquote>]]></content>
<summary type="html">
<h2 id="一、标准化（Z-Score），或者去除均值和方差缩放"><a href="#一、标准化（Z-Score），或者去除均值和方差缩放" class="headerlink" title="一、标准化（Z-Score），或者去除均值和方差缩放"></a>一、标准化（Z-
</summary>
<category term="python" scheme="http://zj2626.github.io/categories/python/"/>
<category term="python" scheme="http://zj2626.github.io/tags/python/"/>
<category term="机器学习" scheme="http://zj2626.github.io/tags/%E6%9C%BA%E5%99%A8%E5%AD%A6%E4%B9%A0/"/>
</entry>
<entry>
<title>scikit-learn数据预处理fit_transform()与transform()的区别(转)</title>
<link href="http://zj2626.github.io/2021/06/01/2018031201/"/>
<id>http://zj2626.github.io/2021/06/01/2018031201/</id>
<published>2021-06-01T14:21:10.637Z</published>
<updated>2021-06-01T01:18:34.845Z</updated>
<content type="html"><![CDATA[<h3 id="问题："><a href="#问题：" class="headerlink" title="问题："></a>问题：</h3><p>scikit-learn中fit_transform()与transform()到底有什么区别，能不能混用？</p><a id="more"></a><ul><li>二者的功能都是对数据进行某种统一处理（比如标准化~N(0,1)，将数据缩放(映射)到某个固定区间，归一化，正则化等）</li><li>fit_transform(partData)对部分数据先拟合fit，找到该part的整体指标，如均值、方差、最大值最小值等等（根据具体转换的目的），然后对该partData进行转换transform，从而实现数据的标准化、归一化等等。。</li><li>根据对之前部分fit的整体指标，对剩余的数据（restData）使用同样的均值、方差、最大最小值等指标进行转换transform(restData)，从而保证part、rest处理方式相同。</li><li>必须先用fit_transform(partData)，之后再transform(restData)</li><li>如果直接transform(partData)，程序会报错</li><li>如果fit_transfrom(partData)后，使用fit_transform(restData)而不用transform(restData)，虽然也能归一化，但是两个结果不是在同一个“标准”下的，具有明显差异。</li></ul><h3 id="实验："><a href="#实验：" class="headerlink" title="实验："></a>实验：</h3><p>使用preprocessing.MinMaxScaler()对象对数据进行归一化。原理是：(x-xMin)/(xMax - xMin)，从而将所有数据映射到【0,1】区间。</p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br><span class="line">48</span><br><span class="line">49</span><br><span class="line">50</span><br><span class="line">51</span><br><span class="line">52</span><br><span class="line">53</span><br><span class="line">54</span><br><span class="line">55</span><br><span class="line">56</span><br><span class="line">57</span><br><span class="line">58</span><br><span class="line">59</span><br><span class="line">60</span><br><span class="line">61</span><br><span class="line">62</span><br><span class="line">63</span><br><span class="line">64</span><br><span class="line">65</span><br><span class="line">66</span><br><span class="line">67</span><br><span class="line">68</span><br><span class="line">69</span><br><span class="line">70</span><br><span class="line">71</span><br><span class="line">72</span><br><span class="line">73</span><br><span class="line">74</span><br><span class="line">75</span><br><span class="line">76</span><br><span class="line">77</span><br><span class="line">78</span><br><span class="line">79</span><br><span class="line">80</span><br><span class="line">81</span><br><span class="line">82</span><br><span class="line">83</span><br><span class="line">84</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">import</span> numpy <span class="keyword">as</span> np </span><br><span class="line"></span><br><span class="line"><span class="keyword">from</span> sklearn.preprocessing <span class="keyword">import</span> MinMaxScaler</span><br><span class="line"></span><br><span class="line">data = np.array(np.random.randint(<span class="number">-100</span>,<span class="number">100</span>,<span class="number">24</span>).reshape(<span class="number">6</span>,<span class="number">4</span>))</span><br><span class="line"></span><br><span class="line">data</span><br><span class="line">Out[<span class="number">55</span>]: </span><br><span class="line">array([[ <span class="number">68</span>, <span class="number">-63</span>, <span class="number">-31</span>, <span class="number">-10</span>],</span><br><span class="line"> [ <span class="number">49</span>, <span class="number">-49</span>, <span class="number">73</span>, <span class="number">18</span>],</span><br><span class="line"> [ <span class="number">46</span>, <span class="number">65</span>, <span class="number">75</span>, <span class="number">-78</span>],</span><br><span class="line"> [<span class="number">-72</span>, <span class="number">30</span>, <span class="number">90</span>, <span class="number">-80</span>],</span><br><span class="line"> [ <span class="number">95</span>, <span class="number">-88</span>, <span class="number">79</span>, <span class="number">-49</span>],</span><br><span class="line"> [ <span class="number">34</span>, <span class="number">-81</span>, <span class="number">57</span>, <span class="number">83</span>]])</span><br><span class="line"></span><br><span class="line">train = data[:<span class="number">4</span>]</span><br><span class="line"></span><br><span class="line">test = data[<span class="number">4</span>:]</span><br><span class="line"></span><br><span class="line">train</span><br><span class="line">Out[<span class="number">58</span>]: </span><br><span class="line">array([[ <span class="number">68</span>, <span class="number">-63</span>, <span class="number">-31</span>, <span class="number">-10</span>],</span><br><span class="line"> [ <span class="number">49</span>, <span class="number">-49</span>, <span class="number">73</span>, <span class="number">18</span>],</span><br><span class="line"> [ <span class="number">46</span>, <span class="number">65</span>, <span class="number">75</span>, <span class="number">-78</span>],</span><br><span class="line"> [<span class="number">-72</span>, <span class="number">30</span>, <span class="number">90</span>, <span class="number">-80</span>]])</span><br><span class="line"></span><br><span class="line">test</span><br><span class="line">Out[<span class="number">59</span>]: </span><br><span class="line">array([[ <span class="number">95</span>, <span class="number">-88</span>, <span class="number">79</span>, <span class="number">-49</span>],</span><br><span class="line"> [ <span class="number">34</span>, <span class="number">-81</span>, <span class="number">57</span>, <span class="number">83</span>]])</span><br><span class="line"></span><br><span class="line">minmaxTransformer = MinMaxScaler(feature_range=(<span class="number">0</span>,<span class="number">1</span>))</span><br><span class="line"></span><br><span class="line"><span class="comment">#先对train用fit_transformer(),包括拟合fit找到xMin,xMax,再transform归一化</span></span><br><span class="line">train_transformer = minmaxTransformer.fit_transform(train)</span><br><span class="line"></span><br><span class="line"><span class="comment">#根据train集合的xMin，xMax,对test集合进行归一化transform.</span></span><br><span class="line"><span class="comment">#(如果test中的某个值比之前的xMin还要小，依然用原来的xMin；同理如果test中的某个值比之前的xMax还要大，依然用原来的xMax.</span></span><br><span class="line"><span class="comment">#所以，对test集合用同样的xMin和xMax，**有可能不再映射到【0,1】**)</span></span><br><span class="line">test_transformer = minmaxTransformer.transform(test)</span><br><span class="line"></span><br><span class="line">train_transformer</span><br><span class="line">Out[<span class="number">64</span>]: </span><br><span class="line">array([[ <span class="number">1.</span> , <span class="number">0.</span> , <span class="number">0.</span> , <span class="number">0.71428571</span>],</span><br><span class="line"> [ <span class="number">0.86428571</span>, <span class="number">0.109375</span> , <span class="number">0.85950413</span>, <span class="number">1.</span> ],</span><br><span class="line"> [ <span class="number">0.84285714</span>, <span class="number">1.</span> , <span class="number">0.87603306</span>, <span class="number">0.02040816</span>],</span><br><span class="line"> [ <span class="number">0.</span> , <span class="number">0.7265625</span> , <span class="number">1.</span> , <span class="number">0.</span> ]])</span><br><span class="line"></span><br><span class="line">test_transformer</span><br><span class="line">Out[<span class="number">65</span>]: </span><br><span class="line">array([[ <span class="number">1.19285714</span>, <span class="number">-0.1953125</span> , <span class="number">0.90909091</span>, <span class="number">0.31632653</span>],</span><br><span class="line"> [ <span class="number">0.75714286</span>, <span class="number">-0.140625</span> , <span class="number">0.72727273</span>, <span class="number">1.66326531</span>]])</span><br><span class="line"></span><br><span class="line"><span class="comment">#如果少了fit环节，直接transform(partData),则会报错</span></span><br><span class="line"></span><br><span class="line">minmaxTransformer = MinMaxScaler(feature_range=(<span class="number">0</span>,<span class="number">1</span>))</span><br><span class="line"></span><br><span class="line">train_transformer2 = minmaxTransformer.transform(train)</span><br><span class="line">Traceback (most recent call last):</span><br><span class="line"></span><br><span class="line"> File <span class="string">"<ipython-input-68-a2aeaf2132be>"</span>, line <span class="number">1</span>, <span class="keyword">in</span> <module></span><br><span class="line"> train_transformer2 = minmaxTransformer.transform(train)</span><br><span class="line"></span><br><span class="line"> File <span class="string">"D:\Program Files\Anaconda3\lib\site-packages\sklearn\preprocessing\data.py"</span>, line <span class="number">352</span>, <span class="keyword">in</span> transform</span><br><span class="line"> check_is_fitted(self, <span class="string">'scale_'</span>)</span><br><span class="line"></span><br><span class="line"> File <span class="string">"D:\Program Files\Anaconda3\lib\site-packages\sklearn\utils\validation.py"</span>, line <span class="number">690</span>, <span class="keyword">in</span> check_is_fitted</span><br><span class="line"> <span class="keyword">raise</span> _NotFittedError(msg % {<span class="string">'name'</span>: type(estimator).__name__})</span><br><span class="line"></span><br><span class="line">NotFittedError: This MinMaxScaler instance <span class="keyword">is</span> <span class="keyword">not</span> fitted yet. Call <span class="string">'fit'</span> <span class="keyword">with</span> appropriate arguments before using this method.</span><br><span class="line"></span><br><span class="line"><span class="comment">#如果对test也用fit_transform(),则结果跟之前不一样。对于许多机器学习算法来说，对于train和test的处理应该统一。</span></span><br><span class="line"></span><br><span class="line">test_transformer2 = minmaxTransformer.fit_transform(test)</span><br><span class="line"></span><br><span class="line">test_transformer2</span><br><span class="line">Out[<span class="number">71</span>]: </span><br><span class="line">array([[ <span class="number">1.</span>, <span class="number">0.</span>, <span class="number">1.</span>, <span class="number">0.</span>],</span><br><span class="line"> [ <span class="number">0.</span>, <span class="number">1.</span>, <span class="number">0.</span>, <span class="number">1.</span>]])</span><br><span class="line"></span><br><span class="line">test_transformer</span><br><span class="line">Out[<span class="number">72</span>]: </span><br><span class="line">array([[ <span class="number">1.19285714</span>, <span class="number">-0.1953125</span> , <span class="number">0.90909091</span>, <span class="number">0.31632653</span>],</span><br><span class="line"> [ <span class="number">0.75714286</span>, <span class="number">-0.140625</span> , <span class="number">0.72727273</span>, <span class="number">1.66326531</span>]])</span><br></pre></td></tr></table></figure><p>转载来自<a href="http://blog.csdn.net/anecdotegyb/article/details/74857055" target="_blank" rel="noopener">这里</a></p><blockquote><p>个人博客 欢迎来访： <a href="http://zj2626.github.io">http://zj2626.github.io</a></p></blockquote>]]></content>
<summary type="html">
<h3 id="问题："><a href="#问题：" class="headerlink" title="问题："></a>问题：</h3><p>scikit-learn中fit_transform()与transform()到底有什么区别，能不能混用？</p>
</summary>
<category term="python" scheme="http://zj2626.github.io/categories/python/"/>
<category term="python" scheme="http://zj2626.github.io/tags/python/"/>
<category term="机器学习" scheme="http://zj2626.github.io/tags/%E6%9C%BA%E5%99%A8%E5%AD%A6%E4%B9%A0/"/>
</entry>
<entry>
<title>构建逻辑回归模型实例</title>
<link href="http://zj2626.github.io/2021/06/01/2018030701/"/>
<id>http://zj2626.github.io/2021/06/01/2018030701/</id>
<published>2021-06-01T14:21:10.609Z</published>
<updated>2021-06-01T01:18:34.826Z</updated>
<content type="html"><![CDATA[<h1 id="逻辑回归"><a href="#逻辑回归" class="headerlink" title="逻辑回归"></a>逻辑回归</h1><blockquote><p>逻辑回归是应用非常广泛的一个分类机器学习算法，它将数据拟合到一个logit函数(或者叫做logistic函数)中，从而能够完成对事件发生的概率进行预测。</p></blockquote><h1 id="构建逻辑回归模型步骤："><a href="#构建逻辑回归模型步骤：" class="headerlink" title="构建逻辑回归模型步骤："></a>构建逻辑回归模型步骤：</h1><ul><li>导入数据</li><li>预处理数据</li><li>对不平衡的数据进行下采样（或者过采样）处理</li><li>把处理之后的数据进行切分，切分为训训练集和测试集</li><li>对训练集进行交叉验证，同时寻找最佳的正则化参数以减少过拟合</li><li>使用最佳的正则化参数对处理之后的数据进行训练并预测，观察召回率和精确率</li><li>使用最佳的正则化参数对处理之后的数据进行训练并预测，观察召回率和精确率</li><li>修改阈值以获得更好的召回率和精确率</li></ul><h2 id="1-数据与任务"><a href="#1-数据与任务" class="headerlink" title="1. 数据与任务"></a>1. 数据与任务</h2><h3 id="信用卡欺诈数据"><a href="#信用卡欺诈数据" class="headerlink" title="信用卡欺诈数据"></a>信用卡欺诈数据</h3><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">import</span> pandas <span class="keyword">as</span> pd</span><br><span class="line"><span class="keyword">import</span> matplotlib.pyplot <span class="keyword">as</span> plt</span><br><span class="line"><span class="keyword">import</span> numpy <span class="keyword">as</span> np</span><br><span class="line"></span><br><span class="line">%matplotlib inline</span><br><span class="line"></span><br><span class="line">data = pd.read_csv(<span class="string">"creditcard.csv"</span>)</span><br><span class="line">data.head()</span><br></pre></td></tr></table></figure><img src="/2021/06/01/2018030701/20180307115819.png" title="如图"><h3 id="要使用逻辑回归对数据进行建模-任务：二分类，-把数据分为有欺诈和无欺诈的两种数据"><a href="#要使用逻辑回归对数据进行建模-任务：二分类，-把数据分为有欺诈和无欺诈的两种数据" class="headerlink" title="要使用逻辑回归对数据进行建模 任务：二分类， 把数据分为有欺诈和无欺诈的两种数据"></a>要使用逻辑回归对数据进行建模 任务：二分类， 把数据分为有欺诈和无欺诈的两种数据</h3><h2 id="2-使用sklearn进行数据预处理"><a href="#2-使用sklearn进行数据预处理" class="headerlink" title="2. 使用sklearn进行数据预处理"></a>2. 使用sklearn进行数据预处理</h2><blockquote><p>公式为：(X-mean)/std 计算时对每个属性/每列分别进行。</p></blockquote><p>Standardization标准化:将特征数据的分布调整成标准正太分布，也叫高斯分布，也就是使得数据的均值维0，方差为1</p><p>标准化的原因在于如果有些特征的方差过大，则会主导目标函数从而使参数估计器无法正确地去学习其他特征。</p><p>标准化的过程为两步：去均值的中心化（均值变为0）；方差的规模化（方差变为1）。</p><p>在sklearn.preprocessing中提供了一个scale的方法，可以实现以上功能。如下面所示:</p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br></pre></td><td class="code"><pre><span class="line">x = np.array([[<span class="number">1.</span>, <span class="number">-1.</span>, <span class="number">2.</span>],</span><br><span class="line"> [<span class="number">2.</span>, <span class="number">0.</span>, <span class="number">0.</span>],</span><br><span class="line"> [<span class="number">0.</span>, <span class="number">1.</span>, <span class="number">-1.</span>]])</span><br><span class="line"><span class="comment"># 将每一列特征标准化为标准正太分布，注意，标准化是针对每一列而言的</span></span><br><span class="line">x_scale = preprocessing.scale(x)</span><br><span class="line">x_scale</span><br></pre></td></tr></table></figure><p>preprocessing这个模块还提供了一个实用类StandarScaler，它可以在训练数据集上做了标准转换操作之后，把相同的转换应用到测试训练集中。<br>可以对训练数据，测试数据应用相同的转换，以后有新的数据进来也可以直接调用，不用再重新把数据放在一起再计算一次了。</p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment"># 调用fit方法，根据已有的训练数据创建一个标准化的转换器</span></span><br><span class="line">scaler = preprocessing.StandardScaler().fit(x)</span><br><span class="line"></span><br><span class="line">scaler</span><br><span class="line"></span><br><span class="line">StandardScaler(copy=<span class="literal">True</span>, with_mean=<span class="literal">True</span>, with_std=<span class="literal">True</span>)</span><br><span class="line"></span><br><span class="line"><span class="comment"># 使用上面这个转换器去转换训练数据x,调用transform方法</span></span><br><span class="line">scaler.transform(x)</span><br></pre></td></tr></table></figure><p><em>StandardScaler()中可以传入两个参数：with_mean,with_std.这两个都是布尔型的参数，默认情况下都是true,但也可以自定义成false.即不要均值中心化或者不要方差规模化为1.</em></p><h3 id="1-处理数据-数据下采样"><a href="#1-处理数据-数据下采样" class="headerlink" title="1. 处理数据 数据下采样"></a>1. 处理数据 数据下采样</h3><h4 id="1-1-预处理数据-修改列”Amount”数据分布"><a href="#1-1-预处理数据-修改列”Amount”数据分布" class="headerlink" title="1.1 预处理数据,修改列”Amount”数据分布"></a>1.1 预处理数据,修改列”Amount”数据分布</h4><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment"># 导入预处理sklearn中预处理模块的标准化模块</span></span><br><span class="line"><span class="keyword">from</span> sklearn.preprocessing <span class="keyword">import</span> StandardScaler</span><br><span class="line"></span><br><span class="line"><span class="keyword">if</span> <span class="string">'Amount'</span> <span class="keyword">in</span> data.columns:</span><br><span class="line"> <span class="comment"># 转化特征为新的特征</span></span><br><span class="line"> data[<span class="string">'normAount'</span>] = StandardScaler().fit_transform(data[<span class="string">'Amount'</span>].reshape(<span class="number">-1</span>, <span class="number">1</span>)) <span class="comment"># reshape:改变数组的形状,参数为改变后的行列数 </span></span><br><span class="line"><span class="comment"># fit_transform：对数据进行变换 矩阵旋转：-1表示自动识别 根据另一个矩阵列（行）数确定本行（列）数</span></span><br></pre></td></tr></table></figure><h3 id="1-2-数据处理-去除不需要的特征"><a href="#1-2-数据处理-去除不需要的特征" class="headerlink" title="1.2 数据处理,去除不需要的特征"></a>1.2 数据处理,去除不需要的特征</h3><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment"># 去掉两个没用的特征（列） axis=1表示对每一行去做这个操作，axis=0表示对每一列做相同的这个操作</span></span><br><span class="line"><span class="keyword">if</span> (<span class="string">'Time'</span>) <span class="keyword">in</span> data.columns:</span><br><span class="line"> data = data.drop([<span class="string">'Time'</span>], axis=<span class="number">1</span>) </span><br><span class="line"><span class="keyword">if</span> (<span class="string">'Amount'</span>) <span class="keyword">in</span> data.columns:</span><br><span class="line"> data = data.drop([<span class="string">'Amount'</span>], axis=<span class="number">1</span>) </span><br><span class="line">print(data.columns, len(data.columns))</span><br><span class="line"></span><br><span class="line"><span class="comment"># 1.2.1 数据图形化展示(1的数据太少索引看上去没有)</span></span><br><span class="line">count_classes = pd.value_counts(data[<span class="string">'Class'</span>], sort=<span class="literal">True</span>).sort_index() <span class="comment"># 画图显示按某列分类之后的数据数量比例</span></span><br><span class="line">count_classes.plot(kind = <span class="string">'bar'</span>) <span class="comment"># bar：条形图</span></span><br><span class="line">plt.xlabel(<span class="string">"Class"</span>)</span><br><span class="line">plt.ylabel(<span class="string">"Frequency"</span>)</span><br><span class="line"></span><br><span class="line"><span class="comment"># 1.2.2 原数据特征和分类</span></span><br><span class="line">X = data.loc[:, data.columns != <span class="string">"Class"</span>]</span><br><span class="line">y = data.loc[:, data.columns == <span class="string">'Class'</span>]</span><br><span class="line"><span class="keyword">print</span> (<span class="string">"SHAPE"</span>, X.shape, y.shape)</span><br><span class="line"></span><br><span class="line"><span class="comment"># Class为0的数量远远大于1的数据，需要使数据个数相近 解决方案： 1.下采样（多的数据抽取部分） 2.过采样（少的数据生成更多）</span></span><br></pre></td></tr></table></figure><img src="/2021/06/01/2018030701/20180307120121.png" title="如图"><h2 id="3-下采样"><a href="#3-下采样" class="headerlink" title="3.下采样"></a>3.下采样</h2><blockquote><p>把数据相对多的减少,可减少为和数据少的数量相同的数量</p></blockquote><h3 id="1-3-区分正常数据和异常数据-通过特征’Class’区分"><a href="#1-3-区分正常数据和异常数据-通过特征’Class’区分" class="headerlink" title="1.3 区分正常数据和异常数据: 通过特征’Class’区分"></a>1.3 区分正常数据和异常数据: 通过特征’Class’区分</h3><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment"># 1.3.1 异常数据-信息</span></span><br><span class="line">number_records = data[data.Class == <span class="number">1</span>]</span><br><span class="line"><span class="comment"># 1.3.2 异常数据个数</span></span><br><span class="line">number_records_fraud = len(number_records)</span><br><span class="line"><span class="comment"># 1.3.3 异常数据索引</span></span><br><span class="line">frand_indices = np.array(number_records.index)</span><br><span class="line"><span class="keyword">print</span> (<span class="string">"异常样本索引 有{}个"</span>.format(number_records_fraud), frand_indices[:<span class="number">10</span>])</span><br><span class="line"></span><br><span class="line"><span class="comment"># 1.3.4 正常数据-索引</span></span><br><span class="line">normal_indices = data[data.Class == <span class="number">0</span>].index</span><br><span class="line"><span class="keyword">print</span> (<span class="string">"正常样本索引 有{}个"</span>.format(len(normal_indices)), normal_indices[<span class="number">-10</span>:])</span><br><span class="line"></span><br><span class="line"><span class="keyword">print</span> (<span class="string">">>>>>>>>>>>>>>>>>>>>所有数据 正常异常比 "</span>, len(normal_indices), <span class="string">'\t'</span>, number_records_fraud)</span><br><span class="line">print(<span class="string">"**************"</span>)</span><br></pre></td></tr></table></figure><h3 id="1-4-下采样处理数据-把多的一方数据进行随机减少到与少的一方相同"><a href="#1-4-下采样处理数据-把多的一方数据进行随机减少到与少的一方相同" class="headerlink" title="1.4 下采样处理数据 把多的一方数据进行随机减少到与少的一方相同"></a>1.4 下采样处理数据 把多的一方数据进行随机减少到与少的一方相同</h3><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment"># 在所有的正常样本索引normal_indices中随机获取，随机选取number_records_fraud个</span></span><br><span class="line"><span class="comment"># np.random.choice: 可以从一个int数字或1维array里随机选取内容，并将选取结果放入n维array中返回。</span></span><br><span class="line">random_normal_indices = np.random.choice(normal_indices, number_records_fraud, replace=<span class="literal">False</span>)</span><br><span class="line">random_normal_indices = np.array(random_normal_indices)</span><br><span class="line"><span class="keyword">print</span> (<span class="string">"'下采样'后有正常样本个数："</span>, len(random_normal_indices))</span><br></pre></td></tr></table></figure><h3 id="1-5-数据索引合并-意思就是把新的正常数据和原来的异常数据进行拼接"><a href="#1-5-数据索引合并-意思就是把新的正常数据和原来的异常数据进行拼接" class="headerlink" title="1.5 数据索引合并 (意思就是把新的正常数据和原来的异常数据进行拼接)"></a>1.5 数据索引合并 (意思就是把新的正常数据和原来的异常数据进行拼接)</h3><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br></pre></td><td class="code"><pre><span class="line">under_sample_indices = np.concatenate([frand_indices, random_normal_indices])</span><br><span class="line"><span class="keyword">print</span> (<span class="string">"合并后有样本个数："</span>, len(under_sample_indices))</span><br><span class="line">under_sample_data = data.iloc[under_sample_indices, :]</span><br><span class="line"><span class="keyword">print</span> (<span class="string">"合并后样本："</span>, under_sample_data[:<span class="number">1</span>])</span><br><span class="line"></span><br><span class="line"><span class="keyword">print</span> (<span class="string">">>>>>>>>>>>>>>>>>>>>下采样数据 正常异常比 "</span>, len(under_sample_data[under_sample_data == <span class="number">0</span>]), <span class="string">'\t'</span>, len(under_sample_data[under_sample_data == <span class="number">1</span>]))</span><br></pre></td></tr></table></figure><h3 id="1-6-获取合并数据中的feature-特征-和label-分类"><a href="#1-6-获取合并数据中的feature-特征-和label-分类" class="headerlink" title="1.6 获取合并数据中的feature(特征)和label(分类)"></a>1.6 获取合并数据中的feature(特征)和label(分类)</h3><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><span class="line">X_undersample = under_sample_data.loc[:, under_sample_data.columns != <span class="string">'Class'</span>]</span><br><span class="line">y_undersample = under_sample_data.loc[:, under_sample_data.columns == <span class="string">'Class'</span>]</span><br><span class="line"></span><br><span class="line"><span class="keyword">print</span> (X_undersample.shape, y_undersample.shape)</span><br><span class="line"><span class="keyword">print</span> (len(under_sample_data[under_sample_data[<span class="string">"Class"</span>] == <span class="number">1</span>]), len(under_sample_data[under_sample_data[<span class="string">"Class"</span>] == <span class="number">0</span>]))</span><br></pre></td></tr></table></figure><img src="/2021/06/01/2018030701/20180307120308.png" title="如图"><h3 id="2-切分数据为训练和测试"><a href="#2-切分数据为训练和测试" class="headerlink" title="2. 切分数据为训练和测试"></a>2. 切分数据为训练和测试</h3><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment"># 切分原始数据 取数据集中80%的数据作为训练集（建立model） 其他20%的为测试集(测试model)</span></span><br><span class="line"><span class="keyword">from</span> sklearn.cross_validation <span class="keyword">import</span> train_test_split</span><br><span class="line"></span><br><span class="line"><span class="keyword">print</span> (X.shape, y.shape)</span><br><span class="line"><span class="comment">## 2.1.对原始数据进行切分 （最终需要使用原数据集中的测试数据进行测试）</span></span><br><span class="line">X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=<span class="number">0.3</span>, random_state=<span class="number">0</span>) <span class="comment"># test_size测试集所占比例 random_state切分之前进行乱序</span></span><br><span class="line"><span class="keyword">print</span> (<span class="string">"1.训练集数据大小"</span>, X_train.shape)</span><br><span class="line"><span class="keyword">print</span> (<span class="string">"2.测试集数据大小"</span>, X_test.shape)</span><br><span class="line"><span class="keyword">print</span> (len(X_train) + len(X_test), len(y_train), len(y_test), <span class="string">"\n"</span>)</span><br><span class="line"></span><br><span class="line"><span class="comment">## 2.2.对下采样数据进行切分</span></span><br><span class="line">X_train_undersample, X_test_undersample, y_train_undersample, y_test_undersample = train_test_split(X_undersample, y_undersample, test_size=<span class="number">0.2</span>, random_state=<span class="number">0</span>) <span class="comment"># test_size测试集大小 random_state切分之前进行乱序</span></span><br><span class="line"><span class="keyword">print</span> (<span class="string">"3.训练集数据大小"</span>, X_train_undersample.shape)</span><br><span class="line"><span class="keyword">print</span> (<span class="string">"4.测试集数据大小"</span>, X_test_undersample.shape)</span><br><span class="line"><span class="keyword">print</span> (len(X_train_undersample) + len(X_test_undersample), len(y_train_undersample), len(y_test_undersample), <span class="string">"\n"</span>)</span><br><span class="line"></span><br><span class="line"><span class="comment"># 切分训练集 把训练集平均切分为三分然后进行交叉验证 （三组数据分别进行建模和验证）</span></span><br></pre></td></tr></table></figure><img src="/2021/06/01/2018030701/20180307120401.png" title="如图"><h2 id="评估标准：召回率（recall）"><a href="#评估标准：召回率（recall）" class="headerlink" title="评估标准：召回率（recall）"></a>评估标准：召回率（recall）</h2><p><em>不适用准确率，因为准确率不能正确的得到所求的，是没用的</em></p><h3 id="模型评估表："><a href="#模型评估表：" class="headerlink" title="模型评估表："></a>模型评估表：</h3><table><thead><tr><th></th><th style="text-align:center">相关（Relevant），正类</th><th style="text-align:right">不相关（NonRelevant），负类</th></tr></thead><tbody><tr><td>被检测到（Retrieved）</td><td style="text-align:center">true positives （TP）</td><td style="text-align:right">false positives （FP）</td></tr><tr><td>未被检测到（Retrieved）</td><td style="text-align:center">false negatives （FN）</td><td style="text-align:right">true negatives （TN）</td></tr></tbody></table><h3 id="一些术语："><a href="#一些术语：" class="headerlink" title="一些术语："></a>一些术语：</h3><ul><li>TP：True Positive，即正确预测出的正样本个数</li><li>FP：False Positive，即错误预测出的正样本个数（本来是负样本，被我们预测成了正样本）</li><li>TN：True Negative，即正确预测出的负样本个数</li><li>FN：False Negative，即错误预测出的负样本个数（本来是正样本，被我们预测成了负样本）</li></ul><h3 id="分类器性能评价指标"><a href="#分类器性能评价指标" class="headerlink" title="分类器性能评价指标"></a>分类器性能评价指标</h3><p>由以上四个指标，可以进一步衍生出其他三个常用的评价分类器性能的指标</p><ul><li>Precision(精确率)：TP÷(TP+FP)TP÷(TP+FP)，分类器预测出的正样本中，真实正样本的比例</li><li>Recall(召回率)：TP÷(TP+FN)TP÷(TP+FN)，在所有真实正样本中，分类器中能找到多少</li><li>Accuracy(准确率)：(TP+TN)÷(TP+NP+TN+FN)(TP+TN)÷(TP+NP+TN+FN)，分类器对整体的判断能力，即正确预测的比例</li></ul><blockquote><p>过拟合： 数据在训练集表现很好 在测试集表现很差</p></blockquote><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">from</span> sklearn.linear_model <span class="keyword">import</span> LogisticRegression <span class="comment"># 逻辑回归</span></span><br><span class="line"><span class="comment"># 注意这里导入的 不是from sklearn.model_selection import KFold</span></span><br><span class="line"><span class="keyword">from</span> sklearn.cross_validation <span class="keyword">import</span> KFold <span class="comment"># 交叉验证 # cross_val_score</span></span><br><span class="line"><span class="keyword">from</span> sklearn.metrics <span class="keyword">import</span> confusion_matrix, recall_score, classification_report <span class="comment"># 混淆矩阵</span></span><br></pre></td></tr></table></figure><h3 id="3-通过多次循环交叉验证-确定正则化参数-random-state：随机种子数"><a href="#3-通过多次循环交叉验证-确定正则化参数-random-state：随机种子数" class="headerlink" title="3. 通过多次循环交叉验证 确定正则化参数 random_state：随机种子数"></a>3. 通过多次循环交叉验证 确定正则化参数 random_state：随机种子数</h3><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br><span class="line">48</span><br><span class="line">49</span><br><span class="line">50</span><br><span class="line">51</span><br><span class="line">52</span><br><span class="line">53</span><br><span class="line">54</span><br><span class="line">55</span><br><span class="line">56</span><br><span class="line">57</span><br><span class="line">58</span><br><span class="line">59</span><br><span class="line">60</span><br></pre></td><td class="code"><pre><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">printing_Kfold_scores</span><span class="params">(x_train_data, y_train_data)</span>:</span></span><br><span class="line"> <span class="comment"># KFold：切分数据集 （这里切分为5部分） shuffle:是否每次都"洗牌"(Falses时，其效果等同于random_state等于整数，每次划分的结果相同)</span></span><br><span class="line"> fold = KFold(len(y_train_data), <span class="number">5</span>, shuffle=<span class="literal">False</span>) </span><br><span class="line"> <span class="keyword">print</span> (type(fold), len(y_train_data), len(fold)) <span class="comment"># 长度是5</span></span><br><span class="line"> </span><br><span class="line"> <span class="comment"># 正则化惩罚项(正则化参数) 预设了多个惩罚值，具体使用哪个需要尝试 列举了5个</span></span><br><span class="line"> c_param_range = [<span class="number">0.01</span>, <span class="number">0.1</span>, <span class="number">1</span>, <span class="number">10</span>, <span class="number">100</span>]</span><br><span class="line"> </span><br><span class="line"> <span class="comment"># 新建DataFrame类型的数据用来存放不同正则化之后的结果</span></span><br><span class="line"> results_table = pd.DataFrame(index = range(len(c_param_range)), columns = [<span class="string">'C_parameter'</span>, <span class="string">'Mean recall score'</span>])</span><br><span class="line"> results_table[<span class="string">'C_parameter'</span>] = c_param_range</span><br><span class="line"></span><br><span class="line"> <span class="comment"># 先按照正则化参数进行循环以确定最好的参数 然后对每个逻辑回归进行交叉验证以获得最好的逻辑回归函数</span></span><br><span class="line"> <span class="comment"># 循环正则化参数 获取最好的c参数</span></span><br><span class="line"> <span class="keyword">for</span> index, c_param <span class="keyword">in</span> enumerate(c_param_range):</span><br><span class="line"> <span class="keyword">print</span> (<span class="string">">>>>>>>>>>>>>>>>>>>>>>>>>>"</span>)</span><br><span class="line"> <span class="keyword">print</span> (<span class="string">"C_parameter "</span>, c_param)</span><br><span class="line"> </span><br><span class="line"> recall_accs = []</span><br><span class="line"> <span class="comment"># 循环进行交叉验证 </span></span><br><span class="line"> <span class="comment"># 每次循环次数为数据切分的大小,切分为n块就交叉验证n次,每次都是区其中n-1块为训练集1块为验证集</span></span><br><span class="line"> <span class="comment"># start=1:开始索引为1</span></span><br><span class="line"> <span class="comment"># iteration为索引 indices为划分好的数据:其中有n-1数据大小的训练集以及1数据代销的验证集</span></span><br><span class="line"> <span class="comment"># 循环中集合每次都不一样,所有的数据都会当一次验证集:例如 三个数据[1,2,3],循环使得数据分别为训练和验证每次为:[[1],[2, 3]], [[2],[1, 3]], [[3],[1, 2]]</span></span><br><span class="line"> <span class="keyword">for</span> iteration, indices <span class="keyword">in</span> enumerate(fold, start=<span class="number">1</span>):</span><br><span class="line"> <span class="comment"># 这里并不是用fold直接划分训练集数据, 而是把索引进行1:5的划分, 然后按照索引获取数据中的对应的数据</span></span><br><span class="line"> <span class="keyword">print</span> (iteration, len(indices[<span class="number">0</span>]), len(indices[<span class="number">1</span>]))</span><br><span class="line"> </span><br><span class="line"> <span class="comment"># 建立逻辑回归模型</span></span><br><span class="line"> lr = LogisticRegression(C = c_param, penalty = <span class="string">'l1'</span>) <span class="comment"># C:正则化参数; penalty:惩罚项:使用L1正则化(惩罚) ‘l1’ or ‘l2’(默认: ‘l2’)</span></span><br><span class="line"> <span class="comment"># 在调参时如果我们主要的目的只是为了解决过拟合，一般penalty选择L2正则化就够了。</span></span><br><span class="line"> <span class="comment"># 但是如果选择L2正则化发现还是过拟合，即预测效果差的时候，就可以考虑L1正则化。</span></span><br><span class="line"> <span class="comment"># 另外，如果模型的特征非常多，我们希望一些不重要的特征系数归零，从而让模型系数稀疏化的话，也可以使用L1正则化。</span></span><br><span class="line"> <span class="comment"># print ("LR-逻辑回归表达式---", lr)</span></span><br><span class="line"> </span><br><span class="line"> <span class="comment"># 训练 参数一:训练数据特征(feature) 参数二:训练数据分类(label)</span></span><br><span class="line"> lr.fit(x_train_data.iloc[indices[<span class="number">0</span>],:], y_train_data.iloc[indices[<span class="number">0</span>],:].values.ravel())</span><br><span class="line"> </span><br><span class="line"> <span class="comment"># 预测</span></span><br><span class="line"> y_pred_undersample = lr.predict(x_train_data.iloc[indices[<span class="number">1</span>], :].values)</span><br><span class="line"> </span><br><span class="line"> <span class="comment"># 计算召回率 召回率 =提取出的正确信息条数 /样本中的信息条数。通俗地说，就是所有准确的条目有多少被检索出来了。</span></span><br><span class="line"> <span class="comment"># 参数: 1.真实数据集 2.预测数据集</span></span><br><span class="line"> recall_acc = recall_score(y_train_data.iloc[indices[<span class="number">1</span>],:].values, y_pred_undersample)</span><br><span class="line"> recall_accs.append(recall_acc)</span><br><span class="line"> <span class="keyword">print</span> (len(indices), <span class="string">"Iteration "</span>, iteration, <span class="string">": recall score = "</span>, recall_acc)</span><br><span class="line"> </span><br><span class="line"> <span class="comment"># 求每个惩罚值经过交叉验证之后平均召回率</span></span><br><span class="line"> results_table.loc[index, <span class="string">'Mean recall score'</span>] = np.mean(recall_accs)</span><br><span class="line"></span><br><span class="line"> <span class="keyword">print</span> (<span class="string">'\nMean recall score '</span>, np.mean(recall_accs), <span class="string">'\n'</span>)</span><br><span class="line"> </span><br><span class="line"> <span class="keyword">print</span> (results_table)</span><br><span class="line"> </span><br><span class="line"> best_c = results_table.loc[results_table[<span class="string">'Mean recall score'</span>].idxmax()][<span class="string">'C_parameter'</span>]</span><br><span class="line"> <span class="keyword">print</span> (<span class="string">"finally-------best is--------> "</span>, best_c)</span><br><span class="line"> </span><br><span class="line"> <span class="keyword">return</span> best_c</span><br><span class="line"></span><br><span class="line">best_c = printing_Kfold_scores(X_train_undersample, y_train_undersample)</span><br></pre></td></tr></table></figure><img src="/2021/06/01/2018030701/20180307120616.png" title="如图"><img src="/2021/06/01/2018030701/20180307120702.png" title="如图"><img src="/2021/06/01/2018030701/20180307120717.png" title="如图"><h4 id="ndarray数据格式化-set-printoptions"><a href="#ndarray数据格式化-set-printoptions" class="headerlink" title="ndarray数据格式化: set_printoptions"></a>ndarray数据格式化: set_printoptions</h4><blockquote><p>set_printoptions(precision=None,<br> threshold=None,<br> edgeitems=None,<br> linewidth=None,<br> suppress=None,<br> nanstr=None,<br> infstr=None,<br> formatter=None)</p></blockquote><ul><li>precision:输出结果保留精度的位数 (num)</li><li>threshold:array数量的个数在小于threshold的时候不会被折叠 (num)</li><li>edgeitems:在array已经被折叠后，开头和结尾都会显示edgeitems个数 (num)</li><li>formatter:这个很有意思，像python3里面str.format(),就是可以对你的输出进行自定义的格式化 其他的暂时没用到</li></ul><h3 id="4-使用最好的正则化参数-构建逻辑回归模型并进行测试"><a href="#4-使用最好的正则化参数-构建逻辑回归模型并进行测试" class="headerlink" title="4. 使用最好的正则化参数 构建逻辑回归模型并进行测试"></a>4. 使用最好的正则化参数 构建逻辑回归模型并进行测试</h3><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br><span class="line">48</span><br><span class="line">49</span><br><span class="line">50</span><br><span class="line">51</span><br><span class="line">52</span><br><span class="line">53</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment"># 构建逻辑回归模型</span></span><br><span class="line">lr = LogisticRegression(C = best_c, penalty=<span class="string">'l1'</span>)</span><br><span class="line"><span class="comment"># 训练回归模型</span></span><br><span class="line">lr.fit(X_train_undersample, y_train_undersample.values.ravel())</span><br><span class="line"><span class="comment"># 使用模型进行测试</span></span><br><span class="line">y_pred_undersample = lr.predict(X_test_undersample.values)</span><br><span class="line"><span class="comment"># y_pred_undersample为预测(分类)值, y_test_undersample为真实测试集的(分类)值</span></span><br><span class="line"></span><br><span class="line"><span class="keyword">print</span> (type(y_pred_undersample), len(y_pred_undersample), <span class="string">"\n"</span>)</span><br><span class="line"></span><br><span class="line"><span class="comment"># 打印和绘制混淆矩阵</span></span><br><span class="line"><span class="keyword">import</span> itertools</span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">plot_confusion_matrix</span><span class="params">(cm, classes, title=<span class="string">'Confussion matrix'</span>, cmap=plt.cm.Blues)</span>:</span></span><br><span class="line"> <span class="comment">#设置显示混淆矩阵</span></span><br><span class="line"> plt.imshow(cm, interpolation=<span class="string">'nearest'</span>, cmap=cmap)</span><br><span class="line"> plt.title(title)</span><br><span class="line"> plt.colorbar()</span><br><span class="line"> </span><br><span class="line"> <span class="comment"># 设置坐标数</span></span><br><span class="line"> tick_marks = np.arange(len(classes))</span><br><span class="line"> plt.xticks(tick_marks, classes, rotation=<span class="number">0</span>)</span><br><span class="line"> plt.yticks(tick_marks, classes)</span><br><span class="line"> </span><br><span class="line"> thresh = cm.max() / <span class="number">2</span></span><br><span class="line"> </span><br><span class="line"> <span class="comment"># itertools.product可进行多层次循环 传入参数个数(n)和索引个数相同 可循环n^2次</span></span><br><span class="line"> <span class="comment"># 设置每个方块中的文字 </span></span><br><span class="line"> <span class="keyword">for</span> i, j <span class="keyword">in</span> itertools.product(range(cm.shape[<span class="number">0</span>]), range(cm.shape[<span class="number">1</span>])):</span><br><span class="line"> <span class="comment"># print (j, i, cm[i, j])</span></span><br><span class="line"> <span class="comment"># 因为i表示横坐标的位置, j表示纵坐标的位置 所以需要把i和j交换位置</span></span><br><span class="line"> plt.text(j, i, cm[i, j], horizontalalignment=<span class="string">"center"</span>, color=<span class="string">"white"</span> <span class="keyword">if</span> cm[i, j] > thresh <span class="keyword">else</span> <span class="string">"black"</span>)</span><br><span class="line"> </span><br><span class="line"> plt.tight_layout()</span><br><span class="line"> <span class="comment"># 设置坐标文字</span></span><br><span class="line"> plt.ylabel(<span class="string">"True label"</span>) <span class="comment"># 真实数据 </span></span><br><span class="line"> plt.xlabel(<span class="string">"Predicted label"</span>) <span class="comment"># 预测数据 1表示正例 0表示负例</span></span><br><span class="line"></span><br><span class="line"><span class="comment"># 画混淆矩阵图 参数: 1.y_true, 2.y_pred</span></span><br><span class="line">cnf_matrix = confusion_matrix(y_test_undersample, y_pred_undersample)</span><br><span class="line">np.set_printoptions(precision=<span class="number">2</span>)</span><br><span class="line"></span><br><span class="line"><span class="keyword">print</span> (<span class="string">"Recall metric in the testing dataset: "</span>, cnf_matrix[<span class="number">1</span>, <span class="number">1</span>]/(cnf_matrix[<span class="number">1</span>, <span class="number">0</span>] + cnf_matrix[<span class="number">1</span>, <span class="number">1</span>]))</span><br><span class="line"></span><br><span class="line">class_names = [<span class="number">0</span>, <span class="number">1</span>]</span><br><span class="line">plt.figure()</span><br><span class="line">plot_confusion_matrix(cnf_matrix, classes=class_names, title=<span class="string">'Confusion matrix'</span>)</span><br><span class="line">plt.show()</span><br><span class="line"></span><br><span class="line"><span class="comment"># 由图可见, 召回率为 85 / (85 + 6) = 93.41%</span></span><br><span class="line"><span class="comment"># 精确率为 (85) / (85 + 9) = 90.43%</span></span><br><span class="line"><span class="comment"># 准确率为 (85 + 97) / (85 + 9 + 6 + 97) = 92.39%</span></span><br><span class="line"></span><br><span class="line"><span class="comment"># 以上计算都是基于下采样数据集的,还需要在原数据的测试集上进行测试操作 (与上面同理)</span></span><br></pre></td></tr></table></figure><img src="/2021/06/01/2018030701/20180307120847.png" title="如图"><h3 id="4-使用最好的正则化参数-构建逻辑回归模型并进行测试-使用原始数据的测试集和训练集"><a href="#4-使用最好的正则化参数-构建逻辑回归模型并进行测试-使用原始数据的测试集和训练集" class="headerlink" title="4. 使用最好的正则化参数 构建逻辑回归模型并进行测试 (使用原始数据的测试集和训练集)"></a>4. 使用最好的正则化参数 构建逻辑回归模型并进行测试 (使用原始数据的测试集和训练集)</h3><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment"># 在原数据的测试集上进行测试操作</span></span><br><span class="line">lr = LogisticRegression(C = best_c, penalty=<span class="string">'l1'</span>)</span><br><span class="line">lr.fit(X_train, y_train.values.ravel())</span><br><span class="line">y_pred = lr.predict(X_test.values)</span><br><span class="line"><span class="comment"># y_pred为预测(分类)值, y_test为真实测试集的(分类)值</span></span><br><span class="line"></span><br><span class="line"></span><br><span class="line">cnf_matrix = confusion_matrix(y_test, y_pred)</span><br><span class="line">np.set_printoptions(precision=<span class="number">2</span>)</span><br><span class="line"></span><br><span class="line"><span class="keyword">print</span> (<span class="string">"Recall metric in the testing dataset: "</span>, cnf_matrix[<span class="number">1</span>, <span class="number">1</span>]/(cnf_matrix[<span class="number">1</span>, <span class="number">0</span>] + cnf_matrix[<span class="number">1</span>, <span class="number">1</span>]))</span><br><span class="line"></span><br><span class="line">class_names = [<span class="number">0</span>, <span class="number">1</span>]</span><br><span class="line">plt.figure()</span><br><span class="line">plot_confusion_matrix(cnf_matrix, classes=class_names, title=<span class="string">'Confusion matrix'</span>)</span><br><span class="line">plt.show()</span><br></pre></td></tr></table></figure><h3 id="5-修改阈值以获取最好的逻辑回归模型"><a href="#5-修改阈值以获取最好的逻辑回归模型" class="headerlink" title="5. 修改阈值以获取最好的逻辑回归模型"></a>5. 修改阈值以获取最好的逻辑回归模型</h3><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment"># 阈值: 默认使用sigma函数默认值:0.5, 意思是当预测概率大于0.5表示True,概率小鱼0.5表示False</span></span><br><span class="line"></span><br><span class="line">lr = LogisticRegression(C = best_c, penalty=<span class="string">'l1'</span>)</span><br><span class="line"><span class="comment"># 训练</span></span><br><span class="line">lr.fit(X_train, y_train.values.ravel())</span><br><span class="line"><span class="comment"># 预测 这里是预测概率值 每个数据的预测包含两个值，对于二分类问题，也就是被判断为0的概率和被判断为1的概率</span></span><br><span class="line">y_pred_undersample_proba = lr.predict_proba(X_test_undersample.values) <span class="comment"># 预测概率值而不是类别值</span></span><br><span class="line"></span><br><span class="line"><span class="comment"># 可能的阈值</span></span><br><span class="line">thresholds = [<span class="number">0.1</span>, <span class="number">0.2</span>, <span class="number">0.3</span>, <span class="number">0.4</span>, <span class="number">0.5</span>, <span class="number">0.6</span>, <span class="number">0.7</span>, <span class="number">0.8</span>, <span class="number">0.9</span>]</span><br><span class="line"></span><br><span class="line">plt.figure(figsize=(<span class="number">10</span>, <span class="number">10</span>)) <span class="comment"># 画图域</span></span><br><span class="line"></span><br><span class="line"><span class="keyword">for</span> index, i <span class="keyword">in</span> enumerate(thresholds):</span><br><span class="line"> <span class="comment"># 预测概率</span></span><br><span class="line"> y_test_predictions_high_recall = y_pred_undersample_proba[:, <span class="number">1</span>] > i</span><br><span class="line"> </span><br><span class="line"> plt.subplot(<span class="number">3</span>, <span class="number">3</span>, index + <span class="number">1</span>)</span><br><span class="line"> </span><br><span class="line"> cnf_matrix = confusion_matrix(y_test_undersample, y_test_predictions_high_recall)</span><br><span class="line"> np.set_printoptions(precision=<span class="number">2</span>)</span><br><span class="line"> </span><br><span class="line"> <span class="keyword">print</span> (i, <span class="string">"Recall metric in the testing dataset: "</span>, cnf_matrix[<span class="number">1</span>, <span class="number">1</span>] / (cnf_matrix[<span class="number">1</span>, <span class="number">0</span>] + cnf_matrix[<span class="number">1</span>, <span class="number">1</span>]))</span><br><span class="line"> </span><br><span class="line"> class_names = [<span class="number">0</span>, <span class="number">1</span>]</span><br><span class="line"> plot_confusion_matrix(cnf_matrix, classes=class_names, title=<span class="string">"Threshold >= %s"</span> %i)</span><br><span class="line"> </span><br><span class="line"><span class="comment">## 随着阈值上升 召回率不断变化 其中本来是1的被误检测为0的越来越多 可见 要选取最合适的阈值以达到召回率最高</span></span><br></pre></td></tr></table></figure><img src="/2021/06/01/2018030701/20180307120937.png" title="如图"><img src="/2021/06/01/2018030701/20180307121001.png" title="如图"><img src="/2021/06/01/2018030701/20180307121022.png" title="如图"><h2 id="过采样"><a href="#过采样" class="headerlink" title="过采样"></a>过采样</h2><blockquote><p>把数据相对少的增加,可增加为和数据多的数量相同的数量 (生成)</p></blockquote><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">import</span> pandas <span class="keyword">as</span> pd</span><br><span class="line"><span class="keyword">from</span> imblearn.over_sampling <span class="keyword">import</span> SMOTE</span><br><span class="line"><span class="keyword">from</span> sklearn.ensemble <span class="keyword">import</span> RandomForestClassifier</span><br><span class="line"><span class="keyword">from</span> sklearn.metrics <span class="keyword">import</span> confusion_matrix</span><br><span class="line"><span class="keyword">from</span> sklearn.model_selection <span class="keyword">import</span> train_test_split</span><br><span class="line"></span><br><span class="line">credit_cards = pd.read_csv(<span class="string">"creditcard.csv"</span>)</span><br><span class="line">columns = credit_cards.columns</span><br><span class="line"></span><br><span class="line">features_columns = columns.delete(len(columns) - <span class="number">1</span>) <span class="comment">#删除最后一列数据</span></span><br><span class="line"><span class="keyword">print</span> (features_columns)</span><br><span class="line"></span><br><span class="line">features = credit_cards[features_columns]</span><br><span class="line">labels = credit_cards[<span class="string">'Class'</span>]</span><br><span class="line"></span><br><span class="line"><span class="keyword">print</span> (<span class="string">"原始的数据个数"</span>, (credit_cards[credit_cards[<span class="string">'Class'</span>] == <span class="number">0</span>]).shape, (credit_cards[credit_cards[<span class="string">'Class'</span>] == <span class="number">1</span>]).shape)</span><br></pre></td></tr></table></figure><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">features_train, features_test, labels_train, labels_test = train_test_split(features, labels, test_size=<span class="number">0.2</span>, random_state=<span class="number">0</span>)</span><br><span class="line"><span class="keyword">print</span> (features_train.shape, features_test.shape, labels_train.shape, labels_test.shape)</span><br></pre></td></tr></table></figure><img src="/2021/06/01/2018030701/20180307121128.png" title="如图"><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br></pre></td><td class="code"><pre><span class="line">oversampler = SMOTE(random_state = <span class="number">0</span>) <span class="comment"># SMOTE随机生成数据 生成只能是训练集生成数据, 而测试集不生成</span></span><br><span class="line"><span class="comment"># 只生成训练集数据 使得Class为1和为0的数量相同 返回训练集的特征和分类</span></span><br><span class="line">os_features, os_labels = oversampler.fit_sample(features_train, labels_train)</span><br><span class="line"></span><br><span class="line"><span class="keyword">print</span> (<span class="string">"可见 的确生成了新的数据,补充了异常的数据 "</span>, len(os_labels[os_labels[:] == <span class="number">1</span>]), len(os_labels[os_labels[:] == <span class="number">0</span>]))</span><br><span class="line"></span><br><span class="line"><span class="keyword">print</span> ((os_features).shape, len(os_features[os_features == <span class="number">1</span>]), len(os_features[os_features == <span class="number">0</span>]), </span><br><span class="line"> (os_labels).shape, len(os_labels[os_labels == <span class="number">1</span>]), len(os_labels[os_labels == <span class="number">0</span>]))</span><br></pre></td></tr></table></figure><img src="/2021/06/01/2018030701/20180307121154.png" title="如图"><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br></pre></td><td class="code"><pre><span class="line">os_features = pd.DataFrame(os_features)</span><br><span class="line">os_labels = pd.DataFrame(os_labels)</span><br><span class="line"><span class="comment"># 获取最佳参数</span></span><br><span class="line">best_c = printing_Kfold_scores(os_features, os_labels)</span><br><span class="line"></span><br><span class="line"><span class="comment"># plot_confusion_matrix</span></span><br></pre></td></tr></table></figure><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br></pre></td><td class="code"><pre><span class="line">lr = LogisticRegression(C = best_c, penalty=<span class="string">'l1'</span>)</span><br><span class="line"><span class="comment"># 训练 使用生成的数据</span></span><br><span class="line">lr.fit(os_features, os_labels.values.ravel())</span><br><span class="line"><span class="comment"># 使用真实数据测试</span></span><br><span class="line">y_pred = lr.predict(features_test.values)</span><br><span class="line"></span><br><span class="line"><span class="comment"># 打印和绘制混淆矩阵</span></span><br><span class="line">cnf_matrix = confusion_matrix(labels_test, y_pred)</span><br><span class="line">np.set_printoptions(precision=<span class="number">2</span>)</span><br><span class="line"></span><br><span class="line"><span class="keyword">print</span> (<span class="string">"Recall metric in the testing dataset: "</span>, cnf_matrix[<span class="number">1</span>, <span class="number">1</span>] / (cnf_matrix[<span class="number">1</span>, <span class="number">0</span>] + cnf_matrix[<span class="number">1</span>, <span class="number">1</span>]))</span><br><span class="line"></span><br><span class="line">class_names = [<span class="number">0</span>, <span class="number">1</span>]</span><br><span class="line">plt.figure()</span><br><span class="line">plot_confusion_matrix(cnf_matrix, classes=class_names, title=<span class="string">'Confusion matrix'</span>)</span><br><span class="line"></span><br><span class="line">plt.show()</span><br></pre></td></tr></table></figure><blockquote><p>个人博客 欢迎来访： <a href="http://zj2626.github.io">http://zj2626.github.io</a></p></blockquote>]]></content>
<summary type="html">
使用逻辑回归对信用卡欺诈数据进行训练分析预测
</summary>
<category term="机器学习" scheme="http://zj2626.github.io/categories/%E6%9C%BA%E5%99%A8%E5%AD%A6%E4%B9%A0/"/>
<category term="python" scheme="http://zj2626.github.io/tags/python/"/>
<category term="逻辑回归" scheme="http://zj2626.github.io/tags/%E9%80%BB%E8%BE%91%E5%9B%9E%E5%BD%92/"/>
<category term="机器学习" scheme="http://zj2626.github.io/tags/%E6%9C%BA%E5%99%A8%E5%AD%A6%E4%B9%A0/"/>
</entry>
<entry>
<title>正则表达式</title>
<link href="http://zj2626.github.io/2021/06/01/20171225_regular/"/>
<id>http://zj2626.github.io/2021/06/01/20171225_regular/</id>
<published>2021-06-01T14:21:10.567Z</published>
<updated>2021-06-01T01:18:34.791Z</updated>
<content type="html"><![CDATA[<h2 id="正则表达式：-waiting。。。"><a href="#正则表达式：-waiting。。。" class="headerlink" title="正则表达式： waiting。。。"></a>正则表达式： waiting。。。</h2><blockquote><p>个人博客 欢迎来访： <a href="http://zj2626.github.io">http://zj2626.github.io</a></p></blockquote>]]></content>
<summary type="html">
<h2 id="正则表达式：-waiting。。。"><a href="#正则表达式：-waiting。。。" class="headerlink" title="正则表达式： waiting。。。"></a>正则表达式： waiting。。。</h2><blockquote>
</summary>
<category term="正则" scheme="http://zj2626.github.io/categories/%E6%AD%A3%E5%88%99/"/>
<category term="java" scheme="http://zj2626.github.io/tags/java/"/>
<category term="python" scheme="http://zj2626.github.io/tags/python/"/>
<category term="正则表达式" scheme="http://zj2626.github.io/tags/%E6%AD%A3%E5%88%99%E8%A1%A8%E8%BE%BE%E5%BC%8F/"/>
</entry>
<entry>
<title>Python 爬虫实战（4）</title>
<link href="http://zj2626.github.io/2021/06/01/20171222_crawler4/"/>
<id>http://zj2626.github.io/2021/06/01/20171222_crawler4/</id>
<published>2021-06-01T14:21:10.566Z</published>
<updated>2021-06-01T01:18:34.790Z</updated>
<content type="html"><![CDATA[<blockquote><p>多线程</p></blockquote><p><em>待添加 条件变量 condition</em></p><a id="more"></a><h2 id="个人代码："><a href="#个人代码：" class="headerlink" title="个人代码："></a>个人代码：</h2><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br></pre></td><td class="code"><pre><span class="line"></span><br><span class="line"><span class="comment">#生产者消费者</span></span><br><span class="line"><span class="keyword">import</span> threading <span class="comment"># , _thread ()python2x是thread</span></span><br><span class="line"><span class="keyword">import</span> time</span><br><span class="line"><span class="keyword">import</span> queue <span class="comment">#python2x是Queue</span></span><br><span class="line"><span class="keyword">import</span> random</span><br><span class="line"></span><br><span class="line">mutex = threading.Lock() <span class="comment">#等价于_thread.allocate_lock()，也等价于_thread.allocate()</span></span><br><span class="line">myq = queue.Queue(<span class="number">10</span>)</span><br><span class="line">num = range(<span class="number">10</span>)</span><br><span class="line"></span><br><span class="line"><span class="class"><span class="keyword">class</span> <span class="title">Producer</span><span class="params">(threading.Thread)</span>:</span></span><br><span class="line"> <span class="function"><span class="keyword">def</span> <span class="title">run</span><span class="params">(self)</span>:</span></span><br><span class="line"> <span class="keyword">global</span> myq, num</span><br><span class="line"> <span class="keyword">for</span> i <span class="keyword">in</span> range(<span class="number">30</span>):</span><br><span class="line"> time.sleep(<span class="number">0.3</span>)</span><br><span class="line"> <span class="keyword">if</span> <span class="keyword">not</span> myq.full() <span class="keyword">and</span> mutex.acquire():</span><br><span class="line"> info = random.choice(num)</span><br><span class="line"> myq.put(info)</span><br><span class="line"> print(<span class="string">"put in data: "</span>, info, <span class="string">"; queue size"</span>, myq.qsize())</span><br><span class="line"> mutex.release()</span><br><span class="line"></span><br><span class="line"><span class="class"><span class="keyword">class</span> <span class="title">Consumer</span><span class="params">(threading.Thread)</span>:</span></span><br><span class="line"> <span class="function"><span class="keyword">def</span> <span class="title">run</span><span class="params">(self)</span>:</span></span><br><span class="line"> <span class="keyword">global</span> myq, num</span><br><span class="line"> <span class="keyword">for</span> i <span class="keyword">in</span> range(<span class="number">30</span>):</span><br><span class="line"> time.sleep(<span class="number">1</span>)</span><br><span class="line"> <span class="keyword">if</span> <span class="keyword">not</span> myq.empty() <span class="keyword">and</span> mutex.acquire():</span><br><span class="line"> info = myq.get()</span><br><span class="line"> print(<span class="string">"get out data: "</span>, info, <span class="string">"; queue size"</span>, myq.qsize())</span><br><span class="line"> mutex.release()</span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">main</span><span class="params">()</span>:</span></span><br><span class="line"> <span class="keyword">print</span> (<span class="string">'main'</span>)</span><br><span class="line"> p = Producer()</span><br><span class="line"> c = Consumer()</span><br><span class="line"> p.start()</span><br><span class="line"> c.start()</span><br><span class="line"></span><br><span class="line"><span class="keyword">if</span> __name__ == <span class="string">'__main__'</span>:</span><br><span class="line"> main()</span><br></pre></td></tr></table></figure><h2 id="别人家的代码【滑稽】："><a href="#别人家的代码【滑稽】：" class="headerlink" title="别人家的代码【滑稽】："></a>别人家的代码【滑稽】：</h2><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment">#!/usr/bin/env python</span></span><br><span class="line"><span class="comment"># -*- coding:utf-8 -*-</span></span><br><span class="line"></span><br><span class="line"><span class="keyword">import</span> thread, threading</span><br><span class="line"><span class="keyword">import</span> urllib2</span><br><span class="line"><span class="keyword">import</span> time, random</span><br><span class="line"><span class="keyword">import</span> Queue</span><br><span class="line"></span><br><span class="line">share_queue = Queue.Queue() <span class="comment">#共享队列</span></span><br><span class="line">my_lock = thread.allocate_lock()</span><br><span class="line"><span class="class"><span class="keyword">class</span> <span class="title">Producer</span><span class="params">(threading.Thread)</span> :</span></span><br><span class="line"></span><br><span class="line"> <span class="function"><span class="keyword">def</span> <span class="title">run</span><span class="params">(self)</span> :</span></span><br><span class="line"> products = range(<span class="number">5</span>)</span><br><span class="line"> <span class="keyword">global</span> share_queue</span><br><span class="line"> <span class="keyword">while</span> <span class="literal">True</span> :</span><br><span class="line"> num = random.choice(products)</span><br><span class="line"> my_lock.acquire()</span><br><span class="line"> share_queue.put(num)</span><br><span class="line"> <span class="keyword">print</span> <span class="string">"Produce : "</span>, num</span><br><span class="line"> my_lock.release()</span><br><span class="line"> time.sleep(random.random())</span><br><span class="line"></span><br><span class="line"><span class="class"><span class="keyword">class</span> <span class="title">Consumer</span><span class="params">(threading.Thread)</span> :</span></span><br><span class="line"></span><br><span class="line"> <span class="function"><span class="keyword">def</span> <span class="title">run</span><span class="params">(self)</span> :</span></span><br><span class="line"> <span class="keyword">global</span> share_queue</span><br><span class="line"> <span class="keyword">while</span> <span class="literal">True</span>:</span><br><span class="line"> my_lock.acquire()</span><br><span class="line"> <span class="keyword">if</span> share_queue.empty() : <span class="comment">#这里没有使用信号量机制进行阻塞等待, </span></span><br><span class="line"> <span class="keyword">print</span> <span class="string">"Queue is Empty..."</span> </span><br><span class="line"> my_lock.release()</span><br><span class="line"> time.sleep(random.random())</span><br><span class="line"> <span class="keyword">continue</span></span><br><span class="line"> num = share_queue.get()</span><br><span class="line"> <span class="keyword">print</span> <span class="string">"Consumer : "</span>, num</span><br><span class="line"> my_lock.release()</span><br><span class="line"> time.sleep(random.random())</span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">main</span><span class="params">()</span> :</span></span><br><span class="line"> producer = Producer()</span><br><span class="line"> consumer = Consumer()</span><br><span class="line"> producer.start()</span><br><span class="line"> consumer.start()</span><br><span class="line"></span><br><span class="line"><span class="keyword">if</span> __name__ == <span class="string">'__main__'</span>:</span><br><span class="line"> main()</span><br></pre></td></tr></table></figure><blockquote><p>转载自 链接地址: <a href="http://www.jianshu.com/p/86b8e78c418a" target="_blank" rel="noopener">http://www.jianshu.com/p/86b8e78c418a</a></p></blockquote><blockquote><p>个人博客 欢迎来访： <a href="http://zj2626.github.io">http://zj2626.github.io</a></p></blockquote>]]></content>
<summary type="html">
<blockquote>
<p>多线程</p>
</blockquote>
<p><em>待添加 条件变量 condition</em></p>
</summary>
<category term="爬虫" scheme="http://zj2626.github.io/categories/%E7%88%AC%E8%99%AB/"/>
<category term="python" scheme="http://zj2626.github.io/tags/python/"/>
<category term="爬虫" scheme="http://zj2626.github.io/tags/%E7%88%AC%E8%99%AB/"/>
</entry>
<entry>
<title>TypeError, a bytes-like object is required, not 'str'</title>
<link href="http://zj2626.github.io/2021/06/01/20171219_python_error/"/>
<id>http://zj2626.github.io/2021/06/01/20171219_python_error/</id>
<published>2021-06-01T14:21:10.558Z</published>
<updated>2021-06-01T01:18:34.784Z</updated>
<content type="html"><![CDATA[<blockquote><p>问题分析: 该问题主要是由于当前操作的字符串是bytes类型的字符串对象，并对该bytes类型的字符串对象进行按照str类型的操作。</p></blockquote><img src="/2021/06/01/20171219_python_error/20171226111345.png" title="问题 alt:问题 extend:?imageView2/2/w/600"><ul><li>解决办法，将s转码成为str类型</li></ul><img src="/2021/06/01/20171219_python_error/20171226111804.png" title="问题 alt:问题 extend:?imageView2/2/w/600"><a id="more"></a><h2 id="str和bytes类型之间的常用转码方式"><a href="#str和bytes类型之间的常用转码方式" class="headerlink" title="str和bytes类型之间的常用转码方式"></a>str和bytes类型之间的常用转码方式</h2><blockquote><p>str to bytes</p></blockquote><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br></pre></td><td class="code"><pre><span class="line">s = <span class="string">'ab bbb'</span></span><br><span class="line"><span class="keyword">print</span> (type(s))</span><br><span class="line"></span><br><span class="line">b = bytes(s, encoding = <span class="string">'utf-8'</span>)</span><br><span class="line"><span class="keyword">print</span> (type(b))</span><br><span class="line"></span><br><span class="line">b2 = s.encode(<span class="string">'utf-8'</span>)</span><br><span class="line"><span class="keyword">print</span> (type(b2))</span><br><span class="line"></span><br><span class="line">b3 = str.encode(s)</span><br><span class="line"><span class="keyword">print</span> (type(b3))</span><br></pre></td></tr></table></figure><img src="/2021/06/01/20171219_python_error/20171226115104.png" title="问题 alt:问题 extend:?imageView2/2/w/600"><blockquote><p>bytes to str</p></blockquote><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br></pre></td><td class="code"><pre><span class="line">b = <span class="string">b'abbbb'</span></span><br><span class="line"><span class="keyword">print</span> (type(b))</span><br><span class="line"></span><br><span class="line">s = str(b, encoding = <span class="string">'utf-8'</span>)</span><br><span class="line"><span class="keyword">print</span> (type(s))</span><br><span class="line"></span><br><span class="line"></span><br><span class="line">s2 = b.decode()</span><br><span class="line"><span class="keyword">print</span> (type(s2))</span><br><span class="line"></span><br><span class="line">s3 = b.decode(<span class="string">'utf-8'</span>)</span><br><span class="line"><span class="keyword">print</span> (type(s3))</span><br><span class="line"></span><br><span class="line"></span><br><span class="line">s4 = bytes.decode(b)</span><br><span class="line"><span class="keyword">print</span> (type(s4))</span><br></pre></td></tr></table></figure><img src="/2021/06/01/20171219_python_error/20171226115528.png" title="问题 alt:问题 extend:?imageView2/2/w/600"><blockquote><p>转载自 链接地址: <a href="http://blog.csdn.net/bible_reader/article/details/53047550" target="_blank" rel="noopener">http://blog.csdn.net/bible_reader/article/details/53047550</a></p></blockquote><blockquote><p>个人博客 欢迎来访： <a href="http://zj2626.github.io">http://zj2626.github.io</a></p></blockquote>]]></content>
<summary type="html">
str和bytes类型之间的常用转码方式
</summary>
<category term="BUG解决" scheme="http://zj2626.github.io/categories/BUG%E8%A7%A3%E5%86%B3/"/>
<category term="python" scheme="http://zj2626.github.io/tags/python/"/>
<category term="转码" scheme="http://zj2626.github.io/tags/%E8%BD%AC%E7%A0%81/"/>
</entry>
<entry>
<title>Python 爬虫实战（3）</title>
<link href="http://zj2626.github.io/2021/06/01/20171219_crawler3/"/>
<id>http://zj2626.github.io/2021/06/01/20171219_crawler3/</id>
<published>2021-06-01T14:21:10.556Z</published>
<updated>2021-06-01T01:18:34.783Z</updated>
<content type="html"><![CDATA[<blockquote><p>Socket 网络编程</p></blockquote><p>Socket(套接字),是操作系统内核中的一个数据结构，它是网络中的节点进行相互通信的门户。它是网络进程的ID。网络通信，归根到底还是进程间的通信（不同计算机上的进程间通信, 又称进程间通信, IP协议进行的主要是端到端通信）。在网络中，每一个节点（计算机或路由）都有一个网络地址，也就是IP地址。两个进程通信时，首先要确定各自所在的网络节点的网络地址。但是，网络地址只能确定进程所在的计算机，而一台计算机上很可能同时运行着多个进程，所以仅凭网络地址还不能确定到底是和网络中的哪一个进程进行通信，因此套接口中还需要包括其他的信息，也就是端口号（PORT）。在一台计算机中，一个端口号一次只能分配给一个进程，也就是说，在一台计算机中，端口号和进程之间是一一对应关系。</p><p>所以，使用端口号和网络地址的组合可以唯一的确定整个网络中的一个网络进程</p><p>端口号的范围从0~65535，一类是由互联网指派名字和号码公司ICANN负责分配给一些常用的应用程序固定使用的“周知的端口”，其值一般为0~1023, 用户自定义端口号一般大于等于1024</p><p>每一个socket都用一个半相关描述{协议、本地地址、本地端口}来表示；一个完整的套接字则用一个相关描述{协议、本地地址、本地端口、远程地址、远程端口}来表示。socket也有一个类似于打开文件的函数调用，该函数返回一个整型的socket描述符，随后的连接建立、数据传输等操作都是通过socket来实现的</p><a id="more"></a><blockquote><p>原文见： <a href="http://python.jobbole.com/88396/" target="_blank" rel="noopener">http://python.jobbole.com/88396/</a></p></blockquote><h2 id="个人代码："><a href="#个人代码：" class="headerlink" title="个人代码："></a>个人代码：</h2><h3 id="TCP"><a href="#TCP" class="headerlink" title="TCP"></a>TCP</h3><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br><span class="line">48</span><br><span class="line">49</span><br><span class="line">50</span><br><span class="line">51</span><br><span class="line">52</span><br><span class="line">53</span><br><span class="line">54</span><br><span class="line">55</span><br><span class="line">56</span><br><span class="line">57</span><br><span class="line">58</span><br><span class="line">59</span><br><span class="line">60</span><br><span class="line">61</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment">#服务器端</span></span><br><span class="line"><span class="comment">#!/usr/bin/env python</span></span><br><span class="line"><span class="comment"># -*- coding:utf-8 -*-</span></span><br><span class="line"> </span><br><span class="line"><span class="keyword">import</span> sys</span><br><span class="line"><span class="keyword">import</span> socket <span class="comment">#socket模块</span></span><br><span class="line"> </span><br><span class="line">BUF_SIZE = <span class="number">1024</span> <span class="comment">#设置缓冲区大小</span></span><br><span class="line">server_addr = (<span class="string">'127.0.0.1'</span>, <span class="number">51230</span>) <span class="comment">#IP和端口构成表示地址</span></span><br><span class="line"><span class="keyword">try</span> :</span><br><span class="line"> server = socket.socket(socket.AF_INET, socket.SOCK_STREAM) <span class="comment">#生成一个新的socket对象</span></span><br><span class="line"><span class="keyword">except</span> socket.error <span class="keyword">as</span> msg :</span><br><span class="line"> <span class="keyword">print</span> (<span class="string">"Creating Socket Failure. Error Code : "</span> + str(msg[<span class="number">0</span>]) + <span class="string">" Message : "</span> + msg[<span class="number">1</span>])</span><br><span class="line"> sys.exit()</span><br><span class="line"><span class="keyword">print</span> (<span class="string">"Socket Created!"</span>)</span><br><span class="line">server.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, <span class="number">1</span>) <span class="comment">#设置地址复用</span></span><br><span class="line"><span class="keyword">try</span> : </span><br><span class="line"> server.bind(server_addr) <span class="comment">#绑定地址</span></span><br><span class="line"><span class="keyword">except</span> socket.error <span class="keyword">as</span> msg :</span><br><span class="line"> <span class="keyword">print</span> (<span class="string">"Binding Failure. Error Code : "</span> + str(msg[<span class="number">0</span>]) + <span class="string">" Message : "</span> + msg[<span class="number">1</span>])</span><br><span class="line"> sys.exit()</span><br><span class="line"><span class="keyword">print</span> (<span class="string">"Socket Bind!"</span>)</span><br><span class="line">server.listen(<span class="number">5</span>) <span class="comment">#监听, 最大监听数为5</span></span><br><span class="line"><span class="keyword">print</span> (<span class="string">"Socket listening"</span>)</span><br><span class="line"><span class="keyword">while</span> <span class="literal">True</span>:</span><br><span class="line"> client, client_addr = server.accept() <span class="comment">#接收TCP连接, 并返回新的套接字和地址, 阻塞函数</span></span><br><span class="line"> <span class="keyword">print</span> (<span class="string">'Connected by'</span>, client_addr)</span><br><span class="line"> <span class="keyword">while</span> <span class="literal">True</span> :</span><br><span class="line"> data = client.recv(BUF_SIZE) <span class="comment">#从客户端接收数据</span></span><br><span class="line"> <span class="keyword">print</span> (str(data, encoding = <span class="string">"utf-8"</span>))</span><br><span class="line"> <span class="keyword">if</span> data == <span class="string">b'exit'</span>:</span><br><span class="line"> <span class="keyword">break</span></span><br><span class="line"> client.sendall(data) <span class="comment">#发送数据到客户端</span></span><br><span class="line">server.close()</span><br><span class="line"></span><br><span class="line"></span><br><span class="line"><span class="comment">#客户端</span></span><br><span class="line"><span class="comment">#!/usr/bin/env python</span></span><br><span class="line"><span class="comment"># -*- coding:utf-8 -*-</span></span><br><span class="line"> </span><br><span class="line"><span class="keyword">import</span> sys</span><br><span class="line"><span class="keyword">import</span> socket</span><br><span class="line"> </span><br><span class="line">BUF_SIZE = <span class="number">1024</span> <span class="comment">#设置缓冲区的大小</span></span><br><span class="line">server_addr = (<span class="string">'127.0.0.1'</span>, <span class="number">51230</span>) <span class="comment">#IP和端口构成表示地址</span></span><br><span class="line"><span class="keyword">try</span> : </span><br><span class="line"> client = socket.socket(socket.AF_INET, socket.SOCK_STREAM) <span class="comment">#返回新的socket对象</span></span><br><span class="line"><span class="keyword">except</span> socket.error <span class="keyword">as</span> msg :</span><br><span class="line"> <span class="keyword">print</span> (<span class="string">"Creating Socket Failure. Error Code : "</span> + str(msg[<span class="number">0</span>]) + <span class="string">" Message : "</span> + msg[<span class="number">1</span>])</span><br><span class="line"> sys.exit()</span><br><span class="line">client.connect(server_addr) <span class="comment">#要连接的服务器地址</span></span><br><span class="line"><span class="keyword">while</span> <span class="literal">True</span>:</span><br><span class="line"> data = input(<span class="string">"Please input some string > "</span>) </span><br><span class="line"> <span class="keyword">if</span> <span class="keyword">not</span> data :</span><br><span class="line"> <span class="keyword">print</span> (<span class="string">"input can't empty, Please input again.."</span>)</span><br><span class="line"> <span class="keyword">continue</span></span><br><span class="line"> client.sendall(bytes(data, encoding = <span class="string">'utf-8'</span>)) <span class="comment">#发送数据到服务器</span></span><br><span class="line"> <span class="comment"># client.sendall(str.encode(data)) </span></span><br><span class="line"> data = client.recv(BUF_SIZE) <span class="comment">#从服务器端接收数据</span></span><br><span class="line"> <span class="keyword">print</span> (str(data, encoding = <span class="string">"utf-8"</span>))</span><br><span class="line">client.close()</span><br></pre></td></tr></table></figure><h3 id="UDP"><a href="#UDP" class="headerlink" title="UDP"></a>UDP</h3><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment">#服务器端</span></span><br><span class="line"><span class="comment">#!/usr/bin/env python</span></span><br><span class="line"><span class="comment"># -*- coding:utf-8 -*-</span></span><br><span class="line"></span><br><span class="line"><span class="keyword">import</span> socket <span class="comment">#socket模块</span></span><br><span class="line">BUFF_SIZE = <span class="number">1024</span></span><br><span class="line">server = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)</span><br><span class="line">address = (<span class="string">'127.0.0.1'</span>, <span class="number">12346</span>)</span><br><span class="line">server.bind(address)</span><br><span class="line"></span><br><span class="line"><span class="keyword">while</span> <span class="literal">True</span>:</span><br><span class="line"> <span class="keyword">print</span> (<span class="string">"WAIT"</span>)</span><br><span class="line"> data, client_address = server.recvfrom(BUFF_SIZE)</span><br><span class="line"> <span class="keyword">print</span> (str(data, encoding=<span class="string">'utf-8'</span>), <span class="string">'from'</span> ,client_address)</span><br><span class="line"> server.sendto(data, client_address)</span><br><span class="line"> </span><br><span class="line"><span class="comment">#客户端</span></span><br><span class="line"><span class="comment">#!/usr/bin/env python</span></span><br><span class="line"><span class="comment"># -*- coding:utf-8 -*-</span></span><br><span class="line"></span><br><span class="line"><span class="keyword">import</span> socket <span class="comment">#socket模块</span></span><br><span class="line"><span class="keyword">import</span> sys</span><br><span class="line">BUFF_SIZE = <span class="number">1024</span></span><br><span class="line">client = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)</span><br><span class="line">address = (<span class="string">'127.0.0.1'</span>, <span class="number">12346</span>)</span><br><span class="line"></span><br><span class="line"><span class="keyword">while</span> <span class="literal">True</span>:</span><br><span class="line"> data = input(<span class="string">"Please input some "</span>) </span><br><span class="line"> <span class="keyword">if</span> data == <span class="string">'exit'</span>:</span><br><span class="line"> <span class="keyword">break</span></span><br><span class="line"> client.sendto(bytes(data, encoding=<span class="string">'utf-8'</span>), address)</span><br><span class="line"> data = client.recv(BUFF_SIZE)</span><br><span class="line"> <span class="keyword">print</span> (str(data, encoding=<span class="string">'utf-8'</span>))</span><br></pre></td></tr></table></figure><h2 id="别人家的代码【滑稽】："><a href="#别人家的代码【滑稽】：" class="headerlink" title="别人家的代码【滑稽】："></a>别人家的代码【滑稽】：</h2><h3 id="TCP-1"><a href="#TCP-1" class="headerlink" title="TCP"></a>TCP</h3><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br><span class="line">48</span><br><span class="line">49</span><br><span class="line">50</span><br><span class="line">51</span><br><span class="line">52</span><br><span class="line">53</span><br><span class="line">54</span><br><span class="line">55</span><br><span class="line">56</span><br><span class="line">57</span><br><span class="line">58</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment">#服务器端</span></span><br><span class="line"><span class="comment">#!/usr/bin/env python</span></span><br><span class="line"><span class="comment"># -*- coding:utf-8 -*-</span></span><br><span class="line"> </span><br><span class="line"><span class="keyword">import</span> sys</span><br><span class="line"><span class="keyword">import</span> socket <span class="comment">#socket模块</span></span><br><span class="line"> </span><br><span class="line">BUF_SIZE = <span class="number">1024</span> <span class="comment">#设置缓冲区大小</span></span><br><span class="line">server_addr = (<span class="string">'127.0.0.1'</span>, <span class="number">8888</span>) <span class="comment">#IP和端口构成表示地址</span></span><br><span class="line"><span class="keyword">try</span> :</span><br><span class="line"> server = socket.socket(socket.AF_INET, socket.SOCK_STREAM) <span class="comment">#生成一个新的socket对象</span></span><br><span class="line"><span class="keyword">except</span> socket.error, msg :</span><br><span class="line"> <span class="keyword">print</span> <span class="string">"Creating Socket Failure. Error Code : "</span> + str(msg[<span class="number">0</span>]) + <span class="string">" Message : "</span> + msg[<span class="number">1</span>]</span><br><span class="line"> sys.exit()</span><br><span class="line"><span class="keyword">print</span> <span class="string">"Socket Created!"</span></span><br><span class="line">server.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, <span class="number">1</span>) <span class="comment">#设置地址复用</span></span><br><span class="line"><span class="keyword">try</span> : </span><br><span class="line"> server.bind(server_addr) <span class="comment">#绑定地址</span></span><br><span class="line"><span class="keyword">except</span> socket.error, msg :</span><br><span class="line"> <span class="keyword">print</span> <span class="string">"Binding Failure. Error Code : "</span> + str(msg[<span class="number">0</span>]) + <span class="string">" Message : "</span> + msg[<span class="number">1</span>]</span><br><span class="line"> sys.exit()</span><br><span class="line"><span class="keyword">print</span> <span class="string">"Socket Bind!"</span></span><br><span class="line">server.listen(<span class="number">5</span>) <span class="comment">#监听, 最大监听数为5</span></span><br><span class="line"><span class="keyword">print</span> <span class="string">"Socket listening"</span></span><br><span class="line"><span class="keyword">while</span> <span class="literal">True</span>:</span><br><span class="line"> client, client_addr = server.accept() <span class="comment">#接收TCP连接, 并返回新的套接字和地址, 阻塞函数</span></span><br><span class="line"> <span class="keyword">print</span> <span class="string">'Connected by'</span>, client_addr</span><br><span class="line"> <span class="keyword">while</span> <span class="literal">True</span> :</span><br><span class="line"> data = client.recv(BUF_SIZE) <span class="comment">#从客户端接收数据</span></span><br><span class="line"> <span class="keyword">print</span> data</span><br><span class="line"> client.sendall(data) <span class="comment">#发送数据到客户端</span></span><br><span class="line">server.close()</span><br><span class="line"></span><br><span class="line"><span class="comment">#客户端</span></span><br><span class="line"><span class="comment">#!/usr/bin/env python</span></span><br><span class="line"><span class="comment"># -*- coding:utf-8 -*-</span></span><br><span class="line"> </span><br><span class="line"><span class="keyword">import</span> sys</span><br><span class="line"><span class="keyword">import</span> socket</span><br><span class="line"> </span><br><span class="line"> </span><br><span class="line">BUF_SIZE = <span class="number">1024</span> <span class="comment">#设置缓冲区的大小</span></span><br><span class="line">server_addr = (<span class="string">'127.0.0.1'</span>, <span class="number">8888</span>) <span class="comment">#IP和端口构成表示地址</span></span><br><span class="line"><span class="keyword">try</span> : </span><br><span class="line"> client = socket.socket(socket.AF_INET, socket.SOCK_STREAM) <span class="comment">#返回新的socket对象</span></span><br><span class="line"><span class="keyword">except</span> socket.error, msg :</span><br><span class="line"> <span class="keyword">print</span> <span class="string">"Creating Socket Failure. Error Code : "</span> + str(msg[<span class="number">0</span>]) + <span class="string">" Message : "</span> + msg[<span class="number">1</span>]</span><br><span class="line"> sys.exit()</span><br><span class="line">client.connect(server_addr) <span class="comment">#要连接的服务器地址</span></span><br><span class="line"><span class="keyword">while</span> <span class="literal">True</span>:</span><br><span class="line"> data = raw_input(<span class="string">"Please input some string > "</span>) </span><br><span class="line"> <span class="keyword">if</span> <span class="keyword">not</span> data :</span><br><span class="line"> <span class="keyword">print</span> <span class="string">"input can't empty, Please input again.."</span></span><br><span class="line"> <span class="keyword">continue</span></span><br><span class="line"> client.sendall(data) <span class="comment">#发送数据到服务器</span></span><br><span class="line"> data = client.recv(BUF_SIZE) <span class="comment">#从服务器端接收数据</span></span><br><span class="line"> <span class="keyword">print</span> data</span><br><span class="line">client.close()</span><br></pre></td></tr></table></figure><h3 id="UDP-1"><a href="#UDP-1" class="headerlink" title="UDP"></a>UDP</h3><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment">#服务器端</span></span><br><span class="line"><span class="comment">#!/usr/bin/env python</span></span><br><span class="line"><span class="comment"># -*- coding:utf-8 -*-</span></span><br><span class="line"> </span><br><span class="line"><span class="keyword">import</span> socket</span><br><span class="line"> </span><br><span class="line">BUF_SIZE = <span class="number">1024</span> <span class="comment">#设置缓冲区大小</span></span><br><span class="line">server_addr = (<span class="string">'127.0.0.1'</span>, <span class="number">8888</span>) <span class="comment">#IP和端口构成表示地址</span></span><br><span class="line">server = socket.socket(socket.AF_INET, socket.SOCK_DGRAM) <span class="comment">#生成新的套接字对象</span></span><br><span class="line">server.bind(server_addr) <span class="comment">#套接字绑定IP和端口</span></span><br><span class="line"><span class="keyword">while</span> <span class="literal">True</span> :</span><br><span class="line"> <span class="keyword">print</span> <span class="string">"waitting for data"</span></span><br><span class="line"> data, client_addr = server.recvfrom(BUF_SIZE) <span class="comment">#从客户端接收数据</span></span><br><span class="line"> <span class="keyword">print</span> <span class="string">'Connected by'</span>, client_addr, <span class="string">' Receive Data : '</span>, data</span><br><span class="line"> server.sendto(data, client_addr) <span class="comment">#发送数据给客户端</span></span><br><span class="line">server.close()</span><br><span class="line"></span><br><span class="line"><span class="comment">#客户端</span></span><br><span class="line"><span class="comment">#!/usr/bin/env python</span></span><br><span class="line"><span class="comment"># -*- coding:utf-8 -*-</span></span><br><span class="line"> </span><br><span class="line"><span class="keyword">import</span> socket</span><br><span class="line"><span class="keyword">import</span> struct</span><br><span class="line"> </span><br><span class="line">BUF_SIZE = <span class="number">1024</span> <span class="comment">#设置缓冲区</span></span><br><span class="line">server_addr = (<span class="string">'127.0.0.1'</span>, <span class="number">8888</span>) <span class="comment">#IP和端口构成表示地址</span></span><br><span class="line">client = socket.socket(socket.AF_INET, socket.SOCK_DGRAM) <span class="comment">#生成新的套接字对象</span></span><br><span class="line"> </span><br><span class="line"><span class="keyword">while</span> <span class="literal">True</span> :</span><br><span class="line"> data = raw_input(<span class="string">'Please Input data > '</span>)</span><br><span class="line"> client.sendto(data, server_addr) <span class="comment">#向服务器发送数据</span></span><br><span class="line"> data, addr = client.recvfrom(BUF_SIZE) <span class="comment">#从服务器接收数据</span></span><br><span class="line"> <span class="keyword">print</span> <span class="string">"Data : "</span>, data</span><br><span class="line">client.close()</span><br></pre></td></tr></table></figure><h3 id="其他"><a href="#其他" class="headerlink" title="其他"></a>其他</h3><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br></pre></td><td class="code"><pre><span class="line">s.getpeername()</span><br><span class="line"><span class="comment">#返回连接套接字的远程地址。返回值通常是元组（ipaddr,port）。</span></span><br><span class="line"> </span><br><span class="line">s.getsockname()</span><br><span class="line"><span class="comment">#返回套接字自己的地址。通常是一个元组(ipaddr,port)</span></span><br><span class="line"> </span><br><span class="line">s.setsockopt(level,optname,value)</span><br><span class="line"><span class="comment">#设置给定套接字选项的值。</span></span><br><span class="line"> </span><br><span class="line">s.getsockopt(level,optname[.buflen])</span><br><span class="line"><span class="comment">#返回套接字选项的值。</span></span><br><span class="line"> </span><br><span class="line">s.settimeout(timeout)</span><br><span class="line"><span class="comment">#设置套接字操作的超时期，timeout是一个浮点数，单位是秒。值为None表示没有超时期。一般，超时期应该在刚创建套接字时设置，因为它们可能用于连接的操作（如connect()）</span></span><br><span class="line"> </span><br><span class="line">s.gettimeout()</span><br><span class="line"><span class="comment">#返回当前超时期的值，单位是秒，如果没有设置超时期，则返回None。</span></span><br><span class="line"> </span><br><span class="line">s.fileno()</span><br><span class="line"><span class="comment">#返回套接字的文件描述符。</span></span><br><span class="line"> </span><br><span class="line">s.setblocking(flag)</span><br><span class="line"><span class="comment">#如果flag为0，则将套接字设为非阻塞模式，否则将套接字设为阻塞模式（默认值）。非阻塞模式下，如果调用recv()没有发现任何数据，或send()调用无法立即发送数据，那么将引起socket.error异常。</span></span><br><span class="line"> </span><br><span class="line">s.makefile()</span><br><span class="line"><span class="comment">#创建一个与该套接字相关连的文件</span></span><br></pre></td></tr></table></figure><blockquote><p>转载自 链接地址: <a href="http://python.jobbole.com/88396/" target="_blank" rel="noopener">http://python.jobbole.com/88396/</a></p></blockquote><blockquote><p>个人博客 欢迎来访： <a href="http://zj2626.github.io">http://zj2626.github.io</a></p></blockquote>]]></content>
<summary type="html">
<blockquote>
<p>Socket 网络编程</p>
</blockquote>
<p>Socket(套接字),是操作系统内核中的一个数据结构，它是网络中的节点进行相互通信的门户。它是网络进程的ID。网络通信，归根到底还是进程间的通信（不同计算机上的进程间通信, 又称进程间通信, IP协议进行的主要是端到端通信）。在网络中，每一个节点（计算机或路由）都有一个网络地址，也就是IP地址。两个进程通信时，首先要确定各自所在的网络节点的网络地址。但是，网络地址只能确定进程所在的计算机，而一台计算机上很可能同时运行着多个进程，所以仅凭网络地址还不能确定到底是和网络中的哪一个进程进行通信，因此套接口中还需要包括其他的信息，也就是端口号（PORT）。在一台计算机中，一个端口号一次只能分配给一个进程，也就是说，在一台计算机中，端口号和进程之间是一一对应关系。</p>
<p>所以，使用端口号和网络地址的组合可以唯一的确定整个网络中的一个网络进程</p>
<p>端口号的范围从0~65535，一类是由互联网指派名字和号码公司ICANN负责分配给一些常用的应用程序固定使用的“周知的端口”，其值一般为0~1023, 用户自定义端口号一般大于等于1024</p>
<p>每一个socket都用一个半相关描述{协议、本地地址、本地端口}来表示；一个完整的套接字则用一个相关描述{协议、本地地址、本地端口、远程地址、远程端口}来表示。socket也有一个类似于打开文件的函数调用，该函数返回一个整型的socket描述符，随后的连接建立、数据传输等操作都是通过socket来实现的</p>
</summary>
<category term="爬虫" scheme="http://zj2626.github.io/categories/%E7%88%AC%E8%99%AB/"/>
<category term="python" scheme="http://zj2626.github.io/tags/python/"/>
<category term="爬虫" scheme="http://zj2626.github.io/tags/%E7%88%AC%E8%99%AB/"/>
</entry>
<entry>
<title>Python 爬虫实战（2）</title>
<link href="http://zj2626.github.io/2021/06/01/20171219_crawler2/"/>
<id>http://zj2626.github.io/2021/06/01/20171219_crawler2/</id>
<published>2021-06-01T14:21:10.547Z</published>
<updated>2021-06-01T01:18:34.775Z</updated>
<content type="html"><![CDATA[<h2 id="目标：-获取上交所和深交所所有股票的名称和交易信息-存储到一个本地文件中"><a href="#目标：-获取上交所和深交所所有股票的名称和交易信息-存储到一个本地文件中" class="headerlink" title="目标： 获取上交所和深交所所有股票的名称和交易信息,存储到一个本地文件中"></a>目标： 获取上交所和深交所所有股票的名称和交易信息,存储到一个本地文件中</h2><blockquote><p>网站选择原则： 股票信息静态存在于html页面中，非js代码生成，没有Robbts协议限制</p></blockquote><blockquote><p>选取方法： 打开网页，查看源代码，搜索网页的股票价格数据是否存在于源代码中</p></blockquote><a id="more"></a><ul><li>下面的百度股市通中，股票的信息完全再html代码中，符合要求（并且发现网址中包含我们需要的关键字:sz代表深交所，而后面的数字就是股票代码了）</li></ul><img src="/2021/06/01/20171219_crawler2/20171226092624.png" title="百度股市通 alt:输出结果 extend:?imageView2/2/w/600"><img src="/2021/06/01/20171219_crawler2/20171226092340.png" title="百度股市通 alt:输出结果 extend:?imageView2/2/w/600"><ul><li>除了单个股票的信息，我们需要所有交所和深交所的股票，访问 <a href="http://quote.eastmoney.com/stocklist.html" target="_blank" rel="noopener">http://quote.eastmoney.com/stocklist.html</a> 查看页面</li></ul><img src="/2021/06/01/20171219_crawler2/20171226093006.png" title="可获得所有的股票代码 alt:股票代码 extend:?imageView2/2/w/600"><ul><li>所有我们只需要先获取所有的股票代码，然后循环访问百度即可获得所有的股票信息</li></ul><blockquote><p>输出结果： </p></blockquote><img src="/2021/06/01/20171219_crawler2/20171226091742.png" title="输出结果 alt:输出结果 extend:?imageView2/2/w/600"><p><strong>大部分讲解都在 Python 爬虫实战（1） 中介绍过了，需要请查看 <a href="http://zj2626.github.io/2017/12/14/20171214_crawler">http://zj2626.github.io/2017/12/14/20171214_crawler</a></strong></p><h3 id="python文件读写"><a href="#python文件读写" class="headerlink" title="python文件读写"></a>python文件读写</h3><ul><li>Python内置了读写文件的函数，用法和C是兼容的。</li></ul><p><em>在磁盘上读写文件的功能都是由操作系统提供的，现代操作系统不允许普通的程序直接操作磁盘，所以，读写文件就是请求操作系统打开一个文件对象（通常称为文件描述符），然后，通过操作系统提供的接口从这个文件对象中读取数据（读文件），或者把数据写入这个文件对象（写文件）。</em></p><blockquote><p>读文件</p></blockquote><ul><li>python内置的open()函数，返回一个文件对象;(参数中 r代表读 w代表写)</li></ul><img src="/2021/06/01/20171219_crawler2/20171226094739.png" title="读文件 alt:读文件 extend:?imageView2/2/w/600"><ul><li>得到文件对象，则可以直接调用f.read()把文件内容读取到内存中来<figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">f.read()</span><br></pre></td></tr></table></figure></li></ul><p>读取时发生的问题：</p><ol><li>如果文件不存在，open()函数就会抛出一个IOError的错误，并且给出错误码和详细的信息告诉你文件不存在</li><li>‘gbk’ codec can’t decode byte 0xaf in position …</li></ol><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment">#问题2解决方案两个：</span></span><br><span class="line"><span class="comment"># 1. 打开文件的时候就指定编码的类型</span></span><br><span class="line"> f = open(<span class="string">'E:/Data.txt'</span>, <span class="string">'r'</span>,encoding = <span class="string">'utf-8'</span>)</span><br><span class="line"> f.read()</span><br><span class="line"><span class="comment"># 2. 修改文件编码为utf-8</span></span><br></pre></td></tr></table></figure><ul><li>最后一步是调用close()方法关闭文件。文件使用完毕后必须关闭，因为文件对象会占用操作系统的资源，并且操作系统同一时间能打开的文件数量也是有限的</li></ul><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">f.close()</span><br></pre></td></tr></table></figure><ul><li>为了防止中途出现异常而无法关闭文件，使用try finally语句</li></ul><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">try</span>:</span><br><span class="line"> f = open(<span class="string">'E:/Data.txt'</span>, <span class="string">'r'</span>, encoding = <span class="string">'utf-8'</span>)</span><br><span class="line"> print(f.read())</span><br><span class="line"><span class="keyword">finally</span>:</span><br><span class="line"> <span class="keyword">if</span> f:</span><br><span class="line"> f.close()</span><br></pre></td></tr></table></figure><ul><li>为了简化代码，python提供了一个更好的更简洁的方法读取文件(和try…finally一样的)</li></ul><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">with</span> open(<span class="string">'E:/Data.txt'</span>, <span class="string">'r'</span>, encoding = <span class="string">'utf-8'</span>) <span class="keyword">as</span> f:</span><br><span class="line"> print(f.read())</span><br></pre></td></tr></table></figure><ul><li>read()方法一次把所有的文件内容读取进来，如果文件太大就不太好用，所以要反复调用read(size)来一部分一部分的读取，也可以调用readline()一次读取一行，或者调用readlines()一次读取全部并返回list<figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">with</span> open(<span class="string">'E:/Data.txt'</span>, <span class="string">'r'</span>, encoding = <span class="string">'utf-8'</span>) <span class="keyword">as</span> f:</span><br><span class="line"> print(f.readline())</span><br><span class="line"> print(f.readline())</span><br><span class="line"> print(f.readline())</span><br></pre></td></tr></table></figure></li></ul><p><em>如果文件很小，read()一次性读取最方便；如果不能确定文件大小，反复调用read(size)比较保险；如果是配置文件，调用readlines()最方便</em></p><blockquote><p>写文件</p></blockquote><ul><li>第二个参数传入标识符’w’或者’wb’表示写文本文件或写二进制文件; write()函数会把数据替换掉原文件中内容<figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">try</span>:</span><br><span class="line"> f = open(<span class="string">'E:/Data.txt'</span>, <span class="string">'w'</span>, encoding = <span class="string">'utf-8'</span>)</span><br><span class="line"> f.write(<span class="string">'ffffffffffffffffffffffff'</span>)</span><br><span class="line"><span class="keyword">finally</span>:</span><br><span class="line"> <span class="keyword">if</span> f:</span><br><span class="line"> f.close()</span><br></pre></td></tr></table></figure></li></ul><p><em>当我们写文件时，操作系统往往不会立刻把数据写入磁盘，而是放到内存缓存起来，空闲的时候再慢慢写入。只有调用close()方法时，操作系统才保证把没有写入的数据全部写入磁盘。忘记调用close()的后果是数据可能只写了一部分到磁盘，剩下的丢失了。</em></p><ul><li>同读取一样 系统提供更好的<figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">with</span> open(<span class="string">'E:/Data.txt'</span>, <span class="string">'w'</span>, encoding = <span class="string">'utf-8'</span>) <span class="keyword">as</span> f:</span><br><span class="line"> f.write(<span class="string">'kkkkkkkkkkkkkkkk'</span>)</span><br></pre></td></tr></table></figure></li></ul><blockquote><p>二进制文件</p></blockquote><ul><li>前面讲的默认都是读取文本文件，并且是UTF-8编码的文本文件。要读取二进制文件，比如图片、视频等等，用’rb’模式打开文件即可</li></ul><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">with</span> open(<span class="string">'D:/20171226101748.png'</span>, <span class="string">'rb'</span>) <span class="keyword">as</span> f:</span><br><span class="line"> f.read()</span><br></pre></td></tr></table></figure><h2 id="个人代码："><a href="#个人代码：" class="headerlink" title="个人代码："></a>个人代码：</h2><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br><span class="line">48</span><br><span class="line">49</span><br><span class="line">50</span><br><span class="line">51</span><br><span class="line">52</span><br><span class="line">53</span><br><span class="line">54</span><br><span class="line">55</span><br><span class="line">56</span><br><span class="line">57</span><br><span class="line">58</span><br><span class="line">59</span><br><span class="line">60</span><br><span class="line">61</span><br><span class="line">62</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">from</span> urllib <span class="keyword">import</span> request</span><br><span class="line"><span class="keyword">from</span> bs4 <span class="keyword">import</span> BeautifulSoup <span class="keyword">as</span> bs</span><br><span class="line"><span class="keyword">import</span> re</span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">getCodes</span><span class="params">()</span>:</span></span><br><span class="line"> url = <span class="string">'http://quote.eastmoney.com/stocklist.html'</span>;</span><br><span class="line"> resp = request.urlopen(url)</span><br><span class="line"> resp_text = resp.read().decode(<span class="string">'gbk'</span>)</span><br><span class="line"> soap = bs(resp_text, <span class="string">'html.parser'</span>)</span><br><span class="line"> list = soap.find_all(<span class="string">'div'</span>, id = <span class="string">'quotesearch'</span>)[<span class="number">0</span>].find_all(<span class="string">'ul'</span>)[<span class="number">0</span>].find_all(<span class="string">'li'</span>)</span><br><span class="line"></span><br><span class="line"> codeList = []</span><br><span class="line"> <span class="keyword">for</span> li <span class="keyword">in</span> list:</span><br><span class="line"> <span class="keyword">try</span>: </span><br><span class="line"> <span class="comment">#eg: sh603183</span></span><br><span class="line"> codeList.append(re.findall(<span class="string">r"[s][hz]\d{6}"</span>, li.find(<span class="string">'a'</span>)[<span class="string">'href'</span>]))</span><br><span class="line"> <span class="keyword">except</span>:</span><br><span class="line"> <span class="keyword">continue</span></span><br><span class="line"> </span><br><span class="line"> <span class="keyword">return</span> codeList</span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">makeDict</span><span class="params">(code)</span>:</span></span><br><span class="line"> infoDict = {}</span><br><span class="line"> url = <span class="string">'https://gupiao.baidu.com/stock/'</span>+ code +<span class="string">'.html'</span>;</span><br><span class="line"> resp = request.urlopen(url)</span><br><span class="line"> resp_text = resp.read().decode(<span class="string">'utf-8'</span>)</span><br><span class="line"> soap = bs(resp_text, <span class="string">'html.parser'</span>)</span><br><span class="line"> </span><br><span class="line"> <span class="keyword">try</span>:</span><br><span class="line"> stockInfo = soap.find_all(attrs = {<span class="string">'class'</span>,<span class="string">'stock-bets'</span>})</span><br><span class="line"> name = soap.find(attrs = {<span class="string">'class'</span>, <span class="string">'bets-name'</span>})</span><br><span class="line"> <span class="keyword">if</span> name <span class="keyword">is</span> <span class="literal">None</span>:</span><br><span class="line"> <span class="keyword">return</span> <span class="literal">None</span></span><br><span class="line"> infoDict[<span class="string">'name'</span>] = name.text.strip()</span><br><span class="line"> keys = soap.find_all(<span class="string">'dt'</span>)</span><br><span class="line"> values = soap.find_all(<span class="string">'dd'</span>)</span><br><span class="line"> <span class="keyword">for</span> i <span class="keyword">in</span> range(len(keys)):</span><br><span class="line"> infoDict[keys[i].text] = values[i].text</span><br><span class="line"></span><br><span class="line"> <span class="keyword">return</span> infoDict</span><br><span class="line"> <span class="keyword">except</span>:</span><br><span class="line"> <span class="keyword">return</span> <span class="literal">None</span></span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">writeFile</span><span class="params">(codeList)</span>:</span></span><br><span class="line"> i=<span class="number">0</span></span><br><span class="line"> <span class="keyword">for</span> code <span class="keyword">in</span> codeList:</span><br><span class="line"> i = i+<span class="number">1</span></span><br><span class="line"> <span class="comment"># 下面两个判断是因为前45个股票百度并没有信息，所以跳过了，200个以后的数据就不再取了，太多了，科科</span></span><br><span class="line"> <span class="keyword">if</span> i < <span class="number">45</span>:</span><br><span class="line"> <span class="keyword">continue</span></span><br><span class="line"> <span class="keyword">if</span> i > <span class="number">200</span>:</span><br><span class="line"> <span class="keyword">break</span></span><br><span class="line"> infoDict = makeDict(code[<span class="number">0</span>])</span><br><span class="line"> <span class="keyword">if</span> infoDict <span class="keyword">is</span> <span class="literal">None</span>:</span><br><span class="line"> <span class="keyword">continue</span></span><br><span class="line"> <span class="keyword">print</span> (infoDict)</span><br><span class="line"> <span class="keyword">with</span> open(<span class="string">'E://Data.txt'</span>, <span class="string">'a'</span>, encoding = <span class="string">'utf-8'</span>) <span class="keyword">as</span> f:</span><br><span class="line"> f.write(str(infoDict) + <span class="string">'\n'</span>)</span><br><span class="line"></span><br><span class="line">codeList = getCodes();</span><br><span class="line">print(<span class="string">"start"</span>)</span><br><span class="line">writeFile(codeList)</span><br></pre></td></tr></table></figure><h2 id="别人家的代码【滑稽】："><a href="#别人家的代码【滑稽】：" class="headerlink" title="别人家的代码【滑稽】："></a>别人家的代码【滑稽】：</h2><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br><span class="line">48</span><br><span class="line">49</span><br><span class="line">50</span><br><span class="line">51</span><br><span class="line">52</span><br><span class="line">53</span><br><span class="line">54</span><br><span class="line">55</span><br><span class="line">56</span><br><span class="line">57</span><br><span class="line">58</span><br><span class="line">59</span><br><span class="line">60</span><br><span class="line">61</span><br><span class="line">62</span><br><span class="line">63</span><br><span class="line">64</span><br><span class="line">65</span><br><span class="line">66</span><br><span class="line">67</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment"># -*- coding: utf-8 -*-</span></span><br><span class="line"> </span><br><span class="line"><span class="keyword">import</span> requests</span><br><span class="line"><span class="keyword">from</span> bs4 <span class="keyword">import</span> BeautifulSoup</span><br><span class="line"><span class="keyword">import</span> traceback</span><br><span class="line"><span class="keyword">import</span> re</span><br><span class="line"> </span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">getHTMLText</span><span class="params">(url)</span>:</span></span><br><span class="line"> <span class="keyword">try</span>:</span><br><span class="line"> r = requests.get(url)</span><br><span class="line"> r.raise_for_status()</span><br><span class="line"> r.encoding = r.apparent_encoding</span><br><span class="line"> <span class="keyword">return</span> r.text</span><br><span class="line"> <span class="keyword">except</span>:</span><br><span class="line"> <span class="keyword">return</span> <span class="string">""</span></span><br><span class="line"> </span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">getStockList</span><span class="params">(lst, stockURL)</span>:</span></span><br><span class="line"> html = getHTMLText(stockURL)</span><br><span class="line"> soup = BeautifulSoup(html, <span class="string">'html.parser'</span>) </span><br><span class="line"> a = soup.find_all(<span class="string">'a'</span>)</span><br><span class="line"> <span class="keyword">for</span> i <span class="keyword">in</span> a:</span><br><span class="line"> <span class="keyword">try</span>:</span><br><span class="line"> href = i.attrs[<span class="string">'href'</span>]</span><br><span class="line"> lst.append(re.findall(<span class="string">r"[s][hz]\d{6}"</span>, href)[<span class="number">0</span>])</span><br><span class="line"> <span class="keyword">except</span>:</span><br><span class="line"> <span class="keyword">continue</span></span><br><span class="line"> </span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">getStockInfo</span><span class="params">(lst, stockURL, fpath)</span>:</span></span><br><span class="line"> count = <span class="number">0</span></span><br><span class="line"> <span class="keyword">for</span> stock <span class="keyword">in</span> lst:</span><br><span class="line"> url = stockURL + stock + <span class="string">".html"</span></span><br><span class="line"> html = getHTMLText(url)</span><br><span class="line"> <span class="keyword">try</span>:</span><br><span class="line"> <span class="keyword">if</span> html==<span class="string">""</span>:</span><br><span class="line"> <span class="keyword">continue</span></span><br><span class="line"> infoDict = {}</span><br><span class="line"> soup = BeautifulSoup(html, <span class="string">'html.parser'</span>)</span><br><span class="line"> stockInfo = soup.find(<span class="string">'div'</span>,attrs={<span class="string">'class'</span>:<span class="string">'stock-bets'</span>})</span><br><span class="line"> </span><br><span class="line"> name = stockInfo.find_all(attrs={<span class="string">'class'</span>:<span class="string">'bets-name'</span>})[<span class="number">0</span>]</span><br><span class="line"> infoDict.update({<span class="string">'股票名称'</span>: name.text.split()[<span class="number">0</span>]})</span><br><span class="line"> </span><br><span class="line"> keyList = stockInfo.find_all(<span class="string">'dt'</span>)</span><br><span class="line"> valueList = stockInfo.find_all(<span class="string">'dd'</span>)</span><br><span class="line"> <span class="keyword">for</span> i <span class="keyword">in</span> range(len(keyList)):</span><br><span class="line"> key = keyList[i].text</span><br><span class="line"> val = valueList[i].text</span><br><span class="line"> infoDict[key] = val</span><br><span class="line"> </span><br><span class="line"> <span class="keyword">with</span> open(fpath, <span class="string">'a'</span>, encoding=<span class="string">'utf-8'</span>) <span class="keyword">as</span> f:</span><br><span class="line"> f.write( str(infoDict) + <span class="string">'\n'</span> )</span><br><span class="line"> count = count + <span class="number">1</span></span><br><span class="line"> print(<span class="string">"\r当前进度: {:.2f}%"</span>.format(count*<span class="number">100</span>/len(lst)),end=<span class="string">""</span>)</span><br><span class="line"> <span class="keyword">except</span>:</span><br><span class="line"> count = count + <span class="number">1</span></span><br><span class="line"> print(<span class="string">"\r当前进度: {:.2f}%"</span>.format(count*<span class="number">100</span>/len(lst)),end=<span class="string">""</span>)</span><br><span class="line"> <span class="keyword">continue</span></span><br><span class="line"> </span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">main</span><span class="params">()</span>:</span></span><br><span class="line"> stock_list_url = <span class="string">'http://quote.eastmoney.com/stocklist.html'</span></span><br><span class="line"> stock_info_url = <span class="string">'https://gupiao.baidu.com/stock/'</span></span><br><span class="line"> output_file = <span class="string">'D:/BaiduStockInfo.txt'</span></span><br><span class="line"> slist=[]</span><br><span class="line"> getStockList(slist, stock_list_url)</span><br><span class="line"> getStockInfo(slist, stock_info_url, output_file)</span><br><span class="line"> </span><br><span class="line">main()</span><br></pre></td></tr></table></figure><blockquote><p>转载自 链接地址: <a href="http://python.jobbole.com/88350/" target="_blank" rel="noopener">http://python.jobbole.com/88350/</a></p></blockquote><blockquote><p>个人博客 欢迎来访： <a href="http://zj2626.github.io">http://zj2626.github.io</a></p></blockquote>]]></content>
<summary type="html">
第二章相较于第一章， 加入了字典的应用，文件读写，也加深了对爬虫的了解，了解到Robbts协议（个人百度的）；另外链接中的代码有的我本地并不能运行成功，所以加入了写自己的写法。
</summary>
<category term="爬虫" scheme="http://zj2626.github.io/categories/%E7%88%AC%E8%99%AB/"/>
<category term="python" scheme="http://zj2626.github.io/tags/python/"/>
<category term="爬虫" scheme="http://zj2626.github.io/tags/%E7%88%AC%E8%99%AB/"/>
</entry>
<entry>
<title>Python 爬虫实战（1）</title>
<link href="http://zj2626.github.io/2021/06/01/20171214_crawler/"/>
<id>http://zj2626.github.io/2021/06/01/20171214_crawler/</id>
<published>2021-06-01T14:21:10.528Z</published>
<updated>2021-06-01T01:18:34.722Z</updated>
<content type="html"><![CDATA[<h2 id="前言："><a href="#前言：" class="headerlink" title="前言："></a>前言：</h2><p>在我学习完python的基础知识之后，当然想要练练手，加深一下对python以及其语法的理解，<br>所以听说爬虫特别有成就感，非常有利于学习and娱乐，以及培养学习的兴趣，so就到处百度爬虫的相关文章，网上的确有很多相关的，但我还是决定自己写写,<br><strong>只有自己写下了讲出来才能代表真的学会了这么技术</strong></p><a id="more"></a><ul><li>我也是第一次学python和抓包，是根据网上的各种讲解以及自己的摸索，慢慢学会的，有什么说的不对的，欢迎指正</li><li>刚开始的时候我的使用Anaconda管理包和环境（py3.6），然而后来我在学多线程的时候，就出现了问题：setdaemon(true)一直没效果（设置守护线程后，守护线程本来应该在所有非守护线程执行完就立马结束而不管守护线程是否结束的，but没用，网上各种查也查不到，后来我把代码写到.py文件里直接在cmd里执行该脚本就没问题了）</li><li>使用Anaconda是因为我打算入坑深度学习，所以提前熟悉熟悉这个管理工具，科科</li></ul><h3 id="本章目的：-抓取豆瓣电影网站正在上映列表的评价关键词，并使用词云表示出来"><a href="#本章目的：-抓取豆瓣电影网站正在上映列表的评价关键词，并使用词云表示出来" class="headerlink" title="本章目的： 抓取豆瓣电影网站正在上映列表的评价关键词，并使用词云表示出来"></a>本章目的： 抓取豆瓣电影网站正在上映列表的评价关键词，并使用词云表示出来</h3><ul><li>豆瓣正在上映列表如图</li></ul><img src="/2021/06/01/20171214_crawler/20171222001.png" title="正在上映"><ul><li>豆瓣电影《芳华》短评列表如图</li></ul><img src="/2021/06/01/20171214_crawler/20171222164027.png" title="芳华"><ul><li>最终获得的《芳华》短评词云如图</li></ul><img src="/2021/06/01/20171214_crawler/20171225134154.png" title="芳华"><h2 id="抓取步骤大致分为三步，具体的又分为下面几步："><a href="#抓取步骤大致分为三步，具体的又分为下面几步：" class="headerlink" title="抓取步骤大致分为三步，具体的又分为下面几步："></a>抓取步骤大致分为三步，具体的又分为下面几步：</h2><h3 id="1-访问并获取网页数据并抽取出来有用信息"><a href="#1-访问并获取网页数据并抽取出来有用信息" class="headerlink" title="1.访问并获取网页数据并抽取出来有用信息"></a>1.访问并获取网页数据并抽取出来有用信息</h3><ul><li><h4 id="首先要获取网页数据"><a href="#首先要获取网页数据" class="headerlink" title="首先要获取网页数据"></a>首先要获取网页数据</h4></li></ul><blockquote><p>获取豆瓣正在上映列表的网页数据<br>使用Python的urllib模块: 其提供了一个从指定的URL地址获取网页数据(创建一个表示远程url的类文件对象)，通过该对象可以对其进行分析处理，获取想要的数据</p></blockquote><ul><li>函数原型</li></ul><img src="/2021/06/01/20171214_crawler/20171225094820.png" title="函数原型 alt:函数原型 extend:?imageView2/2/w/600"><ol><li>url: 请求的地址（除了url，其他都可以不填）</li><li>data: 访问url时请求的参数</li><li>timeout: 超时时间</li></ol><ul><li>python2.X写法：</li></ul><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">import</span> urllib</span><br><span class="line">urltext = urllib.urlopen(url)</span><br></pre></td></tr></table></figure><ul><li>python3.X写法：</li></ul><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">import</span> urllib.request <span class="comment">#from urllib import request</span></span><br><span class="line">urltext = urllib.request.urlopen(url)</span><br></pre></td></tr></table></figure><ul><li>urlopen()方法返回值类型：</li></ul><p>如果请求的是http或者https地址，则返回http.client.HTTPResponse对象<br>如果请求的是ftp或者Data URL地址(以及requests explicitly handled by legacyURLopener and FancyURLopener classes <-原谅我没看懂)，则返回urllib.response.addinfourl对象</p><ul><li>上面的对象包含多个方法可供我们使用</li></ul><img src="/2021/06/01/20171214_crawler/20171225105358.png" title="来自源码request.py extend:?imageView2/2/w/600"><ol><li>geturl() :返回请求的网页地址</li><li>info() :返回一个httplib.HTTPMessage对象，表示远程服务器返回的头信息</li><li>getcode():返回HTTP状态码</li><li>read() , readline() , readlines() , fileno() , close() 这些方法的使用方式与文件对象完全一样</li></ol><blockquote><p>实例</p></blockquote><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">import</span> urllib</span><br><span class="line">urltext = urllib.request.urlopen(<span class="string">'https://movie.douban.com/nowplaying/beijing/'</span>)</span><br><span class="line"><span class="keyword">print</span> (urltext)</span><br><span class="line"><span class="keyword">print</span> (urltext.geturl())</span><br><span class="line"><span class="keyword">print</span> (urltext.info())</span><br><span class="line"><span class="keyword">print</span> (urltext.getcode())</span><br></pre></td></tr></table></figure><img src="/2021/06/01/20171214_crawler/20171225111200.png" title="实例 extend:?imageView2/2/w/600"><img src="/2021/06/01/20171214_crawler/20171225113314.png" title="实例 extend:?imageView2/2/w/600"><ul><li><h4 id="其次-分析网页数据，抓取想要的数据"><a href="#其次-分析网页数据，抓取想要的数据" class="headerlink" title="其次,分析网页数据，抓取想要的数据"></a>其次,分析网页数据，抓取想要的数据</h4></li></ul><blockquote><p>找到网页上你要的电影列表的位置，看看有什么标签特点</p></blockquote><img src="/2021/06/01/20171214_crawler/20171225112215.png" title="来自豆瓣 extend:?imageView2/2/w/600"><p>我们发现所有的电影列表都在id为nowplaying的div下面的一个ul下，该ul的class为lists，并且每个电影的li标签的class为list-item<br>该li标签中有许多熟悉，我们发现data-title为电影标题，data-score为电影评分，data-star为打星…,最最重要的是id，每个电影都不同，可推测应该是电影的唯一标识（编号）；<br>我们要通过某一个标识来查询该电影的短评， 通过查看电影主页的网址（<a href="https://movie.douban.com/subject/26862829/" target="_blank" rel="noopener">https://movie.douban.com/subject/26862829/</a> ）可知，这个id就是我们需要的</p><blockquote><p>使用python的BeautifulSoup库进行网页信息的抓取（网页解析库）<br>BeautifulSoup 是一个可以从 HTML 或 XML 文件中提取数据的 Python 库.它能够通过你喜欢的转换器实现惯用的文档导航,查找,修改文档的方式.Beautiful Soup 会帮你节省数小时甚至数天的工作时间.</p></blockquote><ul><li>BeautifulSoup:Beautiful Soup提供一些简单的、python式的函数用来处理导航、搜索、修改分析树等功能。<br>它是一个工具箱，通过解析文档为用户提供需要抓取的数据，因为简单，所以不需要多少代码就可以写出一个完整的应用程序。<br>Beautiful Soup自动将输入文档转换为Unicode编码，输出文档转换为utf-8编码。你不需要考虑编码方式，除非文档没有指定一个编码方式，这时，Beautiful Soup就不能自动识别编码方式了。然后，你仅仅需要说明一下原始编码方式就可以了。<br>Beautiful Soup已成为和lxml、html6lib一样出色的python解释器，为用户灵活地提供不同的解析策略或强劲的速度。(from <a href="http://beautifulsoup.readthedocs.io/zh_CN/latest/" target="_blank" rel="noopener">http://beautifulsoup.readthedocs.io/zh_CN/latest/</a> )</li></ul><ul><li>使用BeautifulSoup解析代码,能够得到一个 BeautifulSoup 的对象,并能按照标准的缩进格式的结构输出</li><li>Beautiful Soup将复杂HTML文档转换成一个复杂的树形结构,每个节点都是Python对象,所有对象可以归纳为4种: Tag, NavigableString, BeautifulSoup, Comment<figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><span class="line">Tag: <class 'bs4.element.Tag'>标签对象，两个属性：name, attribute (直接调用：tag.name，tag['class']), 如果是多值属性，则返回list，也可以赋值为多值属性（假的多值属性返回字符串，如id="aaa bbb"）</span><br><span class="line">NavigableString: <class 'bs4.element.NavigableString'> tag中的字符串对象，即tag.string; tag中包含的字符串不能编辑,但是可以被替换成其它的字符串,用 replace_with() 方法</span><br><span class="line">BeautifulSoup : <class 'bs4.BeautifulSoup'>BeautifulSoup对象表示的是一个文档的全部内容.大部分时候,可以把它当作 Tag 对象,它支持 遍历文档树 和 搜索文档树 中描述的大部分的方法; 一个方法：soup.name # [document]</span><br><span class="line">Comment : <class 'bs4.element.Comment'>文档的注释部分，Comment 对象是一个特殊类型的 NavigableString 对象</span><br></pre></td></tr></table></figure></li></ul><p>BeautifulSoup对象使用示例：</p><ol><li><p>解析时，可以传入一段字符串或一个文件句柄.</p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">from</span> bs4 <span class="keyword">import</span> BeautifulSoup <span class="comment">#导入模块</span></span><br><span class="line">soup = BeautifulSoup(open(<span class="string">"index.html"</span>))</span><br><span class="line">soup = BeautifulSoup(<span class="string">"<html>data</html>"</span>)</span><br><span class="line"></span><br><span class="line"><span class="comment">#首先,文档被转换成Unicode,并且HTML的实例都被转换成Unicode编码</span></span><br><span class="line"><span class="comment">#然后,BeautifulSoup选择最合适的解析器来解析这段文档,如果手动指定解析器那么Beautiful Soup会选择指定的解析器来解析文档</span></span><br></pre></td></tr></table></figure></li><li><p>soup.title # <title>标签对象：<title>北京 - 在线购票&影讯</title></title></p></li><li>soup.title.name # <title>标签名称：title</title></li><li>soup.title.string # <title>标签内容：北京 - 在线购票&影讯</title></li><li>soup.p # 第一个p标签对象：<p class="appintro-title">豆瓣</p></li><li><p>soup.p[‘class’] # 第一个p标签对象的类属性<br>7.</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br></pre></td><td class="code"><pre><span class="line">原型：find_all( name , attrs , recursive , string , **kwargs ) 搜索当前tag的所有tag子节点,并判断是否符合过滤器的条件</span><br><span class="line"> name: name 参数可以查找所有名字为 name 的tag,字符串对象会被自动忽略掉.</span><br><span class="line"> attrs: 通过属性选择器查询，有两种写法</span><br><span class="line"> 1. soup.find_all(class_='value', id='value2')</span><br><span class="line"> 2. soup.find_all(attrs={"class": "value", "id":"value2"})</span><br><span class="line"> limit: 限制查询结果个数</span><br><span class="line"> recursive: 调用tag的 find_all() 方法时,Beautiful Soup会检索当前tag的所有子孙节点,如果只想搜索tag的直接子节点,可以使用参数 recursive=False</span><br><span class="line"> string: 通过 string 参数可以搜搜文档中的字符串内容. soup.find_all("a", string="value") #查询标签中文字包含value的a标签</span><br></pre></td></tr></table></figure></li><li><p>soup.find(‘a’).get(‘href’) # 找到第一个a标签 并返回其href属性内容 （ find_all() 方法的返回结果是值包含一个元素的列表,而 find() 方法直接返回结果.）</p></li><li>更多用法见BeautifulSoup官网中文文档：<a href="http://beautifulsoup.readthedocs.io/zh_CN/latest/" target="_blank" rel="noopener">http://beautifulsoup.readthedocs.io/zh_CN/latest/</a></li></ol><ul><li><p>解析网页代码,并编码为utf-8</p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">import</span> urllib</span><br><span class="line">urltext = urllib.request.urlopen(<span class="string">'https://movie.douban.com/nowplaying/beijing/'</span>)</span><br><span class="line">html_data = urltext.read().decode(<span class="string">'utf-8'</span>)</span><br><span class="line"><span class="keyword">print</span> (html_data)</span><br></pre></td></tr></table></figure></li><li><p>获取正在上映列表数据 nowplaying_movie_list列表（List）</p></li></ul><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">import</span> urllib</span><br><span class="line">urltext = urllib.request.urlopen(<span class="string">'https://movie.douban.com/nowplaying/beijing/'</span>)</span><br><span class="line">html_data = urltext.read().decode(<span class="string">'utf-8'</span>)</span><br><span class="line"></span><br><span class="line"><span class="keyword">from</span> bs4 <span class="keyword">import</span> BeautifulSoup <span class="keyword">as</span> bs</span><br><span class="line">soup = bs(html_data, <span class="string">'html.parser'</span>)</span><br><span class="line">nowplaying_movie = soup.find_all(<span class="string">'div'</span>, id = <span class="string">'nowplaying'</span>) <span class="comment"># 先获取id为nowplaying的div</span></span><br><span class="line"><span class="comment"># print (nowplaying_movie) # 只有一条数据，因为id是唯一的</span></span><br><span class="line">nowplaying_movie_list = nowplaying_movie[<span class="number">0</span>].find_all(<span class="string">'li'</span>, class_ = <span class="string">'list-item'</span>)<span class="comment"># 再获取class为list-item的li</span></span><br><span class="line"><span class="keyword">print</span> (nowplaying_movie_list)</span><br></pre></td></tr></table></figure><ul><li>至此已经获得了最内部一层的电影数据， 可以直接获得每个电影的id了</li></ul><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">print</span> (nowplaying_movie_list[<span class="number">0</span>][<span class="string">'id'</span>], <span class="string">'\n'</span>) <span class="comment">#获取第一个电影的id数据</span></span><br></pre></td></tr></table></figure><ul><li><p>现在 我们需要获取其中某一个id，通过这个id获取对应电影的短评，然后就可以进行处理了</p></li><li><p>你也可以自由发挥，制作一个查询的功能，通过输入电影名称指定某一个电影进行分析</p></li></ul><h3 id="2-分析网页中有用信息并进行处理"><a href="#2-分析网页中有用信息并进行处理" class="headerlink" title="2.分析网页中有用信息并进行处理"></a>2.分析网页中有用信息并进行处理</h3><ul><li><h4 id="首先按照上面的步骤访问电影首页，抽取短评信息，存放到一个List中"><a href="#首先按照上面的步骤访问电影首页，抽取短评信息，存放到一个List中" class="headerlink" title="首先按照上面的步骤访问电影首页，抽取短评信息，存放到一个List中"></a>首先按照上面的步骤访问电影首页，抽取短评信息，存放到一个List中</h4></li><li><p>首先解析网页代码</p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br></pre></td><td class="code"><pre><span class="line">requrl = <span class="string">"https://movie.douban.com/subject/"</span> + nowplaying_movie_list[<span class="number">0</span>][<span class="string">'id'</span>] + <span class="string">"/comments?start=0&limit=20"</span></span><br><span class="line">resp = urllib.request.urlopen(requrl)</span><br><span class="line">html_data = resp.read().decode(<span class="string">'utf-8'</span>)</span><br><span class="line">soup = bs(html_data, <span class="string">'html.parser'</span>)</span><br><span class="line"></span><br><span class="line">title = soup.find(<span class="string">'title'</span>) <span class="comment"># 直接获取title标签</span></span><br><span class="line">print(title.string) <span class="comment">#获取标签中内容</span></span><br><span class="line">comment_div_list = soup.find_all(<span class="string">'div'</span>, class_ = <span class="string">'comment'</span>)</span><br><span class="line"><span class="keyword">print</span> (comment_div_list) <span class="comment">#所有的短片标签列表</span></span><br></pre></td></tr></table></figure></li><li><p>通过下面的源码可知，所有的短评文字都放在class为comment-item的div下的一个p标签中，所有我们要得到所有的p标签并组成一个List</p></li></ul><img src="/2021/06/01/20171214_crawler/20171225120114.png" title="来自豆瓣 extend:?imageView2/2/w/600"><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><span class="line">commentList = [] <span class="comment">#存放所有的短评内容数据 List</span></span><br><span class="line"><span class="keyword">for</span> cm <span class="keyword">in</span> comment_div_list:</span><br><span class="line"> <span class="keyword">if</span> cm.find_all(<span class="string">'p'</span>)[<span class="number">0</span>] <span class="keyword">is</span> <span class="keyword">not</span> <span class="literal">None</span>:</span><br><span class="line"> commentList.append(cm.find_all(<span class="string">'p'</span>)[<span class="number">0</span>].string) <span class="comment">#把短评内容存放在列表中</span></span><br><span class="line"><span class="keyword">print</span> (commentList)</span><br></pre></td></tr></table></figure><ul><li>已得短评List，但是该List中包含大量的单引号（List自带的），换行符等不需要的东西，并且由于我们要做成词云，所有的符号都不要，只要文字</li></ul><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">while</span> <span class="literal">True</span>:</span><br><span class="line"> <span class="keyword">if</span> <span class="literal">None</span> <span class="keyword">in</span> commentList:</span><br><span class="line"> commentList.remove(<span class="literal">None</span>) <span class="comment">#去除NoneType数据</span></span><br><span class="line"> <span class="keyword">else</span>:</span><br><span class="line"> <span class="keyword">break</span></span><br><span class="line">comments = <span class="string">''</span>.join(commentList) <span class="comment">#拼接字符串</span></span><br><span class="line">comments = comments.replace(<span class="string">' '</span>,<span class="string">''</span>).replace(<span class="string">"\n"</span>, <span class="string">""</span>).replace(<span class="string">"\t"</span>, <span class="string">""</span>)</span><br><span class="line"><span class="keyword">print</span> (comments)</span><br></pre></td></tr></table></figure><ul><li><p>词云展示的只是关键词，所以去除用户短评中的所有的标点符号（正则表达式）</p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">import</span> re <span class="comment">#正则表达式</span></span><br><span class="line">pattern = re.compile(<span class="string">r'[\u4e00-\u9fa5]+'</span>) <span class="comment">#去除标点符号(正则表达式)</span></span><br><span class="line">filterdata = re.findall(pattern, comments)</span><br><span class="line">cleaned_comments = <span class="string">''</span>.join(filterdata) <span class="comment"># 把filterdata按照空字符串为间隔连接起来</span></span><br><span class="line"><span class="keyword">print</span> (cleaned_comments)</span><br></pre></td></tr></table></figure></li><li><p>目前所有的评价都没有间隔的展示在这里，我们需要把其中的词语取出来得到所有的关键词</p><blockquote><p>使用jieba分词, 把字符串中的所有的词语分出来，组成一个List</p></blockquote></li></ul><blockquote><p>结巴（jieba）是国人出的一个精品插件，可以对一段中文进行分词，有三种分词模式，可以适应不同需求。</p></blockquote><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br></pre></td><td class="code"><pre><span class="line">jieba.cut 方法接受三个输入参数: 需要分词的字符串；cut_all 参数用来控制是否采用全模式；HMM 参数用来控制是否使用 HMM 模型</span><br><span class="line">jieba.cut_for_search 方法接受两个参数：需要分词的字符串；是否使用 HMM 模型。该方法适合用于搜索引擎构建倒排索引的分词，粒度比较细</span><br><span class="line">待分词的字符串可以是 unicode 或 UTF-8 字符串、GBK 字符串。注意：不建议直接输入 GBK 字符串，可能无法预料地错误解码成 UTF-8</span><br><span class="line">jieba.cut 以及 jieba.cut_for_search 返回的结构都是一个可迭代的 generator，可以使用 for 循环来获得分词后得到的每一个词语(unicode)，或者用</span><br><span class="line">jieba.lcut 以及 jieba.lcut_for_search 直接返回 list</span><br><span class="line">jieba.Tokenizer(dictionary=DEFAULT_DICT) 新建自定义分词器，可用于同时使用不同词典。jieba.dt 为默认分词器，所有全局分词相关函数都是该分词器的映射。</span><br><span class="line"></span><br><span class="line">也可以添加自定义词典 （from： http://blog.csdn.net/qq_27231343/article/details/51898940 ）</span><br></pre></td></tr></table></figure><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment">#代码示例</span></span><br><span class="line"><span class="comment"># encoding=utf-8</span></span><br><span class="line"><span class="keyword">import</span> jieba</span><br><span class="line"></span><br><span class="line">seg_list = jieba.cut(<span class="string">"我来到北京清华大学"</span>, cut_all=<span class="literal">True</span>)</span><br><span class="line">print(<span class="string">"Full Mode: "</span> + <span class="string">"/ "</span>.join(seg_list)) <span class="comment"># 全模式</span></span><br><span class="line"></span><br><span class="line">seg_list = jieba.cut(<span class="string">"我来到北京清华大学"</span>, cut_all=<span class="literal">False</span>)</span><br><span class="line">print(<span class="string">"Default Mode: "</span> + <span class="string">"/ "</span>.join(seg_list)) <span class="comment"># 精确模式</span></span><br><span class="line"></span><br><span class="line">seg_list = jieba.cut(<span class="string">"他来到了网易杭研大厦"</span>) <span class="comment"># 默认是精确模式</span></span><br><span class="line">print(<span class="string">"* "</span>.join(seg_list))</span><br><span class="line"></span><br><span class="line">seg_list = jieba.cut_for_search(<span class="string">"小明硕士毕业于中国科学院计算所，后在日本京都大学深造"</span>) <span class="comment"># 搜索引擎模式</span></span><br><span class="line">print(<span class="string">", "</span>.join(seg_list))</span><br><span class="line"></span><br><span class="line"><span class="comment">#输出结果</span></span><br><span class="line"><span class="comment"># Full Mode: 我/ 来到/ 北京/ 清华/ 清华大学/ 华大/ 大学</span></span><br><span class="line"><span class="comment"># Default Mode: 我/ 来到/ 北京/ 清华大学</span></span><br><span class="line"><span class="comment"># 他* 来到* 了* 网易* 杭研* 大厦 (此处，“杭研”并没有在词典中，但是也被Viterbi算法识别出来了)</span></span><br><span class="line"><span class="comment"># 小明, 硕士, 毕业, 于, 中国, 科学, 学院, 科学院, 中国科学院, 计算, 计算所, ，, 后, 在, 日本, 京都, 大学, 日本京都大学, 深造</span></span><br></pre></td></tr></table></figure><ul><li><p>使用jieba分割短评，获取返回的分词List</p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">import</span> jieba</span><br><span class="line">segment = jieba.lcut(cleaned_comments)</span><br><span class="line"><span class="keyword">print</span> (segment)</span><br></pre></td></tr></table></figure></li><li><p>数据中有“的”、“是”、“我”、“你”等虚词（停用词），而这些词在任何场景中都是高频时，并且没有实际的含义，所以我们要他们进行清除。</p><blockquote><p>使用pandas</p></blockquote></li></ul><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">import</span> pandas <span class="keyword">as</span> pd</span><br><span class="line">words_df = pd.DataFrame({<span class="string">'segment'</span>:segment}) <span class="comment">#格式转换 把List转化为Dict</span></span><br><span class="line"><span class="comment"># words_df.head()</span></span><br><span class="line"><span class="comment"># print(words_df)</span></span><br><span class="line"><span class="comment"># print (words_df.segment)</span></span><br><span class="line"><span class="comment">#从网上下载常用停用词文件 stopwords.txt 然后对比去除统计结果中所有的停用词</span></span><br><span class="line">stopwords=pd.read_csv(<span class="string">"E:/stopwords.txt"</span>,index_col=<span class="literal">False</span>,quoting=<span class="number">3</span>,sep=<span class="string">"\t"</span>,names=[<span class="string">'stopword'</span>], encoding=<span class="string">'utf-8'</span>)<span class="comment">#quoting=3全不引用</span></span><br><span class="line"><span class="comment"># print (stopwords.stopword)</span></span><br><span class="line"><span class="comment"># print (words_df.segment.isin(stopwords.stopword))</span></span><br><span class="line">words_df = words_df[~words_df.segment.isin(stopwords.stopword)] <span class="comment">#stopwords.txt不能有空格</span></span><br><span class="line">words_df.head()</span><br></pre></td></tr></table></figure><p><strong>我的停用词文件： <a href="http://p18j2ow6f.bkt.clouddn.com/static/file/stopwords.txt" target="_blank" rel="noopener">http://p18j2ow6f.bkt.clouddn.com/static/file/stopwords.txt</a></strong></p><ul><li>清洗了关键词以后，我们把剩下的词语进行分类统计，观察每个词语的频率<blockquote><p>使用numpy</p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">import</span> numpy <span class="comment">#numpy计算包</span></span><br><span class="line">words_stat = words_df.groupby(by=[<span class="string">'segment'</span>])[<span class="string">'segment'</span>].agg({<span class="string">"计数"</span>:numpy.size}) <span class="comment"># 按照segment分类</span></span><br><span class="line">words_stat = words_stat.reset_index().sort_values(by=[<span class="string">"计数"</span>],ascending=<span class="literal">False</span>) <span class="comment">#词频按照 计数 由大到小排列</span></span><br><span class="line">words_stat.head()</span><br></pre></td></tr></table></figure></blockquote></li></ul><h3 id="3-制作为词云"><a href="#3-制作为词云" class="headerlink" title="3.制作为词云"></a>3.制作为词云</h3><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">import</span> matplotlib</span><br><span class="line"><span class="comment"># %matplotlib inline</span></span><br><span class="line"></span><br><span class="line">matplotlib.rcParams[<span class="string">'figure.figsize'</span>] = (<span class="number">10.0</span>, <span class="number">5.0</span>)</span><br><span class="line"><span class="keyword">from</span> wordcloud <span class="keyword">import</span> WordCloud <span class="comment">#词云包</span></span><br><span class="line"></span><br><span class="line">wordcloud=WordCloud(font_path=<span class="string">"E:/simhei.ttf"</span>,background_color=<span class="string">"white"</span>,max_font_size=<span class="number">80</span>) <span class="comment">#指定字体类型、字体大小和字体颜色</span></span><br><span class="line"><span class="comment"># print (wordcloud)</span></span><br><span class="line">word_frequence = {x[<span class="number">0</span>]:x[<span class="number">1</span>] <span class="keyword">for</span> x <span class="keyword">in</span> words_stat.head(<span class="number">1000</span>).values}</span><br><span class="line"><span class="comment"># print (word_frequence)</span></span><br><span class="line"></span><br><span class="line">wordcloud=wordcloud.fit_words(word_frequence)</span><br><span class="line">matplotlib.pyplot.imshow(wordcloud)</span><br></pre></td></tr></table></figure><p><strong>我的字体文件： <a href="http://p18j2ow6f.bkt.clouddn.com/static/file/simhei.ttf" target="_blank" rel="noopener">http://p18j2ow6f.bkt.clouddn.com/static/file/simhei.ttf</a></strong></p><blockquote><p>最终效果</p></blockquote><img src="/2021/06/01/20171214_crawler/20171225134154.png" title="芳华"><h2 id="遇到403-forbidden以及503-Service-Unavailable问题的解决方法："><a href="#遇到403-forbidden以及503-Service-Unavailable问题的解决方法：" class="headerlink" title="遇到403: forbidden以及503: Service Unavailable问题的解决方法："></a>遇到403: forbidden以及503: Service Unavailable问题的解决方法：</h2><p>这是网站对自动化爬虫的禁止需要用python的模块urllib2模块(对于3.6版本使用 urllib.request)</p><p><em>User-Agent是浏览器特有的属性，通过浏览器查看源代码就可以查看到(其他的属性也可以通过浏览器点击F12中的network窗口发现)</em></p><img src="/2021/06/01/20171214_crawler/20180101164037.png" title="403问题解决"><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br></pre></td><td class="code"><pre><span class="line"></span><br><span class="line"><span class="keyword">import</span> urllib.request</span><br><span class="line"><span class="keyword">import</span> random</span><br><span class="line"><span class="keyword">from</span> bs4 <span class="keyword">import</span> BeautifulSoup <span class="keyword">as</span> bs</span><br><span class="line"><span class="keyword">import</span> re</span><br><span class="line"><span class="keyword">import</span> csv</span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">getCodes</span><span class="params">()</span>:</span></span><br><span class="line"> headers=[<span class="string">"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.84 Safari/537.36"</span>]</span><br><span class="line"> randdom_header=random.choice(headers)</span><br><span class="line"></span><br><span class="line"> url = <span class="string">'http://bj.meituan.com/meishi/c17/'</span>;</span><br><span class="line"> req=request.Request(url)</span><br><span class="line"> req.add_header(<span class="string">"User-Agent"</span>,randdom_header)</span><br><span class="line"> req.add_header(<span class="string">"Host"</span>,<span class="string">"bj.meituan.com"</span>)</span><br><span class="line"> req.add_header(<span class="string">"Referer"</span>,<span class="string">"http://bj.meituan.com/"</span>)</span><br><span class="line"> req.add_header(<span class="string">"GET"</span>,url)</span><br><span class="line"> resp_text=request.urlopen(req).read()</span><br><span class="line"></span><br><span class="line"> soap = bs(resp_text, <span class="string">'html.parser'</span>)</span><br><span class="line"> list = soap.find_all(<span class="string">'ul'</span>, class_ = <span class="string">'list-ul'</span>)[<span class="number">0</span>].find_all(<span class="string">'li'</span>)</span><br><span class="line"> <span class="keyword">print</span> (list)</span><br><span class="line"></span><br><span class="line"> codeList = []</span><br><span class="line"> <span class="keyword">for</span> div <span class="keyword">in</span> list:</span><br><span class="line"> <span class="keyword">try</span>:</span><br><span class="line"> url = div.find(<span class="string">'div'</span>, class_ = <span class="string">'img '</span>).find(<span class="string">'a'</span>)[<span class="string">'href'</span>]</span><br><span class="line"><span class="comment"># len = url.index('?')</span></span><br><span class="line"> codeList.append(url) <span class="comment"># url[: len]</span></span><br><span class="line"> <span class="keyword">except</span>:</span><br><span class="line"> <span class="keyword">continue</span></span><br><span class="line"></span><br><span class="line"> <span class="keyword">return</span> codeList</span><br></pre></td></tr></table></figure><h2 id="个人完整代码："><a href="#个人完整代码：" class="headerlink" title="个人完整代码："></a>个人完整代码：</h2><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br><span class="line">48</span><br><span class="line">49</span><br><span class="line">50</span><br><span class="line">51</span><br><span class="line">52</span><br><span class="line">53</span><br><span class="line">54</span><br><span class="line">55</span><br><span class="line">56</span><br><span class="line">57</span><br><span class="line">58</span><br><span class="line">59</span><br><span class="line">60</span><br><span class="line">61</span><br><span class="line">62</span><br><span class="line">63</span><br><span class="line">64</span><br><span class="line">65</span><br><span class="line">66</span><br><span class="line">67</span><br><span class="line">68</span><br><span class="line">69</span><br><span class="line">70</span><br><span class="line">71</span><br><span class="line">72</span><br><span class="line">73</span><br><span class="line">74</span><br><span class="line">75</span><br><span class="line">76</span><br><span class="line">77</span><br><span class="line">78</span><br><span class="line">79</span><br><span class="line">80</span><br><span class="line">81</span><br><span class="line">82</span><br><span class="line">83</span><br><span class="line">84</span><br><span class="line">85</span><br><span class="line">86</span><br><span class="line">87</span><br><span class="line">88</span><br><span class="line">89</span><br><span class="line">90</span><br><span class="line">91</span><br><span class="line">92</span><br><span class="line">93</span><br><span class="line">94</span><br><span class="line">95</span><br><span class="line">96</span><br><span class="line">97</span><br><span class="line">98</span><br><span class="line">99</span><br><span class="line">100</span><br><span class="line">101</span><br><span class="line">102</span><br><span class="line">103</span><br><span class="line">104</span><br><span class="line">105</span><br><span class="line">106</span><br><span class="line">107</span><br><span class="line">108</span><br><span class="line">109</span><br><span class="line">110</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">from</span> urllib <span class="keyword">import</span> request <span class="comment">#python3.X写法</span></span><br><span class="line"><span class="comment">#import urllib #python2.X写法</span></span><br><span class="line"><span class="keyword">from</span> bs4 <span class="keyword">import</span> BeautifulSoup <span class="keyword">as</span> bs</span><br><span class="line"><span class="keyword">import</span> re <span class="comment">#正则表达式</span></span><br><span class="line"><span class="keyword">import</span> jieba <span class="comment">#分词包 中文分词操作 结巴分词</span></span><br><span class="line"><span class="keyword">import</span> pandas <span class="keyword">as</span> pd</span><br><span class="line"><span class="keyword">import</span> numpy <span class="comment">#numpy计算包</span></span><br><span class="line"></span><br><span class="line"></span><br><span class="line"><span class="string">"""</span></span><br><span class="line"><span class="string">python2.X 关于 urllib的用法</span></span><br><span class="line"><span class="string"> import urllib</span></span><br><span class="line"><span class="string"> text = urllib.urlopen(url).read()</span></span><br><span class="line"><span class="string"></span></span><br><span class="line"><span class="string">python3.X 关于 urllib的用法</span></span><br><span class="line"><span class="string"> import urllib.request #from urllib import request</span></span><br><span class="line"><span class="string"> response = urllib.request.urlopen(url)</span></span><br><span class="line"><span class="string"> text = response.read()</span></span><br><span class="line"><span class="string">"""</span></span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">getList</span><span class="params">()</span>:</span></span><br><span class="line"> resp = request.urlopen(<span class="string">'https://movie.douban.com/nowplaying/beijing/'</span>) <span class="comment">#获取url下的影片列表;python2.x下使用urllib.urlopen()</span></span><br><span class="line"> html_data = resp.read().decode(<span class="string">'utf-8'</span>) <span class="comment"># 读取返回的数据(返回页面的html代码)</span></span><br><span class="line"> <span class="comment"># print(html_data)</span></span><br><span class="line"></span><br><span class="line"> soup = bs(html_data, <span class="string">'html.parser'</span>) <span class="comment"># 解析html代码 开始获取其中的数据</span></span><br><span class="line"> nowplaying_movie = soup.find_all(<span class="string">'div'</span>, id = <span class="string">'nowplaying'</span>) <span class="comment">#获取id为nowplaying的div标签以及内部的代码 (得到的是一个list)</span></span><br><span class="line"> <span class="comment"># print (nowplaying_movie);</span></span><br><span class="line"> nowplaying_movie_list = nowplaying_movie[<span class="number">0</span>].find_all(<span class="string">'li'</span>, class_ = <span class="string">'list-item'</span>) <span class="comment">#获取class是list-item的所有li标签</span></span><br><span class="line"> <span class="comment"># print (nowplaying_movie_list);</span></span><br><span class="line"> <span class="comment"># print (nowplaying_movie_list[0]['id'], '\n'); # 打印第一个影片的id</span></span><br><span class="line"></span><br><span class="line"> <span class="string">"""测试代码 开始"""</span></span><br><span class="line"> <span class="comment"># test = nowplaying_movie_list[0].find_all('ul')</span></span><br><span class="line"> <span class="comment"># print (test)</span></span><br><span class="line"> <span class="comment"># test = nowplaying_movie_list[0].find_all('ul')[0].find_all('li')[1]</span></span><br><span class="line"> <span class="comment"># print (test)</span></span><br><span class="line"> <span class="string">"""测试代码 结束"""</span></span><br><span class="line"></span><br><span class="line"> <span class="keyword">return</span> nowplaying_movie_list</span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">getComments</span><span class="params">(nowplaying_movie_list, num)</span>:</span></span><br><span class="line"> requrl = <span class="string">"https://movie.douban.com/subject/"</span> + nowplaying_movie_list[num][<span class="string">'id'</span>] + <span class="string">"/comments?start=0&limit=20"</span> <span class="comment">#获取url下的影片短评列表</span></span><br><span class="line"> resp = urllib.request.urlopen(requrl)</span><br><span class="line"> html_data = resp.read().decode(<span class="string">'utf-8'</span>)</span><br><span class="line"></span><br><span class="line"> soup = bs(html_data, <span class="string">'html.parser'</span>)</span><br><span class="line"></span><br><span class="line"> title = soup.find(<span class="string">'title'</span>)</span><br><span class="line"> print(title.string)</span><br><span class="line"></span><br><span class="line"> comment_div_list = soup.find_all(<span class="string">'div'</span>, class_ = <span class="string">'comment'</span>)</span><br><span class="line"> <span class="comment">#print (comment_div_list)</span></span><br><span class="line"> commentList = [] <span class="comment">#存放所有的短评内容数据</span></span><br><span class="line"> <span class="keyword">for</span> cm <span class="keyword">in</span> comment_div_list:</span><br><span class="line"> <span class="keyword">if</span> cm.find_all(<span class="string">'p'</span>)[<span class="number">0</span>] <span class="keyword">is</span> <span class="keyword">not</span> <span class="literal">None</span>:</span><br><span class="line"> commentList.append(cm.find_all(<span class="string">'p'</span>)[<span class="number">0</span>].string) <span class="comment">#把短评内容存放在列表中</span></span><br><span class="line"> <span class="comment"># print (comments)</span></span><br><span class="line"></span><br><span class="line"> comments = <span class="string">''</span></span><br><span class="line"> <span class="keyword">for</span> k <span class="keyword">in</span> range(len(commentList)):</span><br><span class="line"> comments = comments + (str(commentList[k])).strip()</span><br><span class="line"> <span class="comment">#print (comments)</span></span><br><span class="line"> pattern = re.compile(<span class="string">r'[\u4e00-\u9fa5]+'</span>) <span class="comment">#去除标点符号(正则表达式)</span></span><br><span class="line"> filterdata = re.findall(pattern, comments)</span><br><span class="line"> cleaned_comments = <span class="string">''</span>.join(filterdata) <span class="comment"># 把filterdata按照空字符串为间隔连接起来</span></span><br><span class="line"> <span class="comment"># print (cleaned_comments)</span></span><br><span class="line"></span><br><span class="line"> segment = jieba.lcut(cleaned_comments) <span class="comment">#list</span></span><br><span class="line"> <span class="comment"># print (segment)</span></span><br><span class="line"> words_df = pd.DataFrame({<span class="string">'segment'</span>:segment}) <span class="comment">#格式转换</span></span><br><span class="line"> <span class="comment"># words_df.head()</span></span><br><span class="line"> <span class="comment"># print(words_df)</span></span><br><span class="line"> <span class="comment"># print (words_df.segment)</span></span><br><span class="line"> <span class="comment"># 数据中有“的”、“是”、“我”、“你”等虚词（停用词），而这些词在任何场景中都是高频时，并且没有实际的含义，所以我们要他们进行清除。</span></span><br><span class="line"></span><br><span class="line"> <span class="comment">#从网上下载常用停用词文件 stopwords.txt 然后对比去除统计结果中所有的停用词</span></span><br><span class="line"> stopwords=pd.read_csv(<span class="string">"E:/stopwords.txt"</span>,index_col=<span class="literal">False</span>,quoting=<span class="number">3</span>,sep=<span class="string">"\t"</span>,names=[<span class="string">'stopword'</span>], encoding=<span class="string">'utf-8'</span>)<span class="comment">#quoting=3全不引用</span></span><br><span class="line"> <span class="comment"># print (stopwords.stopword)</span></span><br><span class="line"> <span class="comment"># print (words_df.segment.isin(stopwords.stopword))</span></span><br><span class="line"> words_df = words_df[~words_df.segment.isin(stopwords.stopword)] <span class="comment">#stopwords.txt不能有空格</span></span><br><span class="line"> words_df.head()</span><br><span class="line"></span><br><span class="line"> <span class="comment">#进行词频统计</span></span><br><span class="line"> words_stat = words_df.groupby(by=[<span class="string">'segment'</span>])[<span class="string">'segment'</span>].agg({<span class="string">"计数"</span>:numpy.size}) <span class="comment"># 按照segment分类</span></span><br><span class="line"> words_stat = words_stat.reset_index().sort_values(by=[<span class="string">"计数"</span>],ascending=<span class="literal">False</span>) <span class="comment">#词频按照 计数 由大到小排列</span></span><br><span class="line"> words_stat.head()</span><br><span class="line"></span><br><span class="line"> <span class="keyword">return</span> words_stat</span><br><span class="line"></span><br><span class="line"><span class="comment">#词云展示</span></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">show</span><span class="params">(words_stat)</span>:</span></span><br><span class="line"> <span class="keyword">import</span> matplotlib</span><br><span class="line"> %matplotlib inline</span><br><span class="line"></span><br><span class="line"> matplotlib.rcParams[<span class="string">'figure.figsize'</span>] = (<span class="number">10.0</span>, <span class="number">5.0</span>)</span><br><span class="line"> <span class="keyword">from</span> wordcloud <span class="keyword">import</span> WordCloud <span class="comment">#词云包</span></span><br><span class="line"></span><br><span class="line"> wordcloud=WordCloud(font_path=<span class="string">"E:/simhei.ttf"</span>,background_color=<span class="string">"white"</span>,max_font_size=<span class="number">80</span>) <span class="comment">#指定字体类型、字体大小和字体颜色</span></span><br><span class="line"> <span class="comment"># print (wordcloud)</span></span><br><span class="line"> word_frequence = {x[<span class="number">0</span>]:x[<span class="number">1</span>] <span class="keyword">for</span> x <span class="keyword">in</span> words_stat.head(<span class="number">1000</span>).values}</span><br><span class="line"> <span class="comment"># print (word_frequence)</span></span><br><span class="line"></span><br><span class="line"> wordcloud=wordcloud.fit_words(word_frequence)</span><br><span class="line"> matplotlib.pyplot.imshow(wordcloud)</span><br><span class="line"></span><br><span class="line">num = <span class="number">0</span> <span class="comment">#从0开始, 获取豆瓣最新上映电影短评关键信息</span></span><br><span class="line">movie_list = getList()</span><br><span class="line">words_stat = getComments(movie_list, num)</span><br><span class="line">show(words_stat)</span><br></pre></td></tr></table></figure><h2 id="别人家的代码【滑稽】："><a href="#别人家的代码【滑稽】：" class="headerlink" title="别人家的代码【滑稽】："></a>别人家的代码【滑稽】：</h2><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br><span class="line">48</span><br><span class="line">49</span><br><span class="line">50</span><br><span class="line">51</span><br><span class="line">52</span><br><span class="line">53</span><br><span class="line">54</span><br><span class="line">55</span><br><span class="line">56</span><br><span class="line">57</span><br><span class="line">58</span><br><span class="line">59</span><br><span class="line">60</span><br><span class="line">61</span><br><span class="line">62</span><br><span class="line">63</span><br><span class="line">64</span><br><span class="line">65</span><br><span class="line">66</span><br><span class="line">67</span><br><span class="line">68</span><br><span class="line">69</span><br><span class="line">70</span><br><span class="line">71</span><br><span class="line">72</span><br><span class="line">73</span><br><span class="line">74</span><br><span class="line">75</span><br><span class="line">76</span><br><span class="line">77</span><br><span class="line">78</span><br><span class="line">79</span><br><span class="line">80</span><br><span class="line">81</span><br><span class="line">82</span><br><span class="line">83</span><br><span class="line">84</span><br><span class="line">85</span><br><span class="line">86</span><br><span class="line">87</span><br><span class="line">88</span><br><span class="line">89</span><br><span class="line">90</span><br><span class="line">91</span><br><span class="line">92</span><br><span class="line">93</span><br><span class="line">94</span><br><span class="line">95</span><br><span class="line">96</span><br><span class="line">97</span><br><span class="line">98</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment">#coding:utf-8</span></span><br><span class="line">__author__ = <span class="string">'hang'</span></span><br><span class="line"></span><br><span class="line"><span class="keyword">import</span> warnings</span><br><span class="line">warnings.filterwarnings(<span class="string">"ignore"</span>)</span><br><span class="line"><span class="keyword">import</span> jieba <span class="comment">#分词包</span></span><br><span class="line"><span class="keyword">import</span> numpy <span class="comment">#numpy计算包</span></span><br><span class="line"><span class="keyword">import</span> codecs <span class="comment">#codecs提供的open方法来指定打开的文件的语言编码，它会在读取的时候自动转换为内部unicode</span></span><br><span class="line"><span class="keyword">import</span> re</span><br><span class="line"><span class="keyword">import</span> pandas <span class="keyword">as</span> pd</span><br><span class="line"><span class="keyword">import</span> matplotlib.pyplot <span class="keyword">as</span> plt</span><br><span class="line"><span class="keyword">from</span> urllib <span class="keyword">import</span> request</span><br><span class="line"><span class="keyword">from</span> bs4 <span class="keyword">import</span> BeautifulSoup <span class="keyword">as</span> bs</span><br><span class="line">%matplotlib inline</span><br><span class="line"></span><br><span class="line"><span class="keyword">import</span> matplotlib</span><br><span class="line">matplotlib.rcParams[<span class="string">'figure.figsize'</span>] = (<span class="number">10.0</span>, <span class="number">5.0</span>)</span><br><span class="line"><span class="keyword">from</span> wordcloud <span class="keyword">import</span> WordCloud<span class="comment">#词云包</span></span><br><span class="line"></span><br><span class="line"><span class="comment">#分析网页函数</span></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">getNowPlayingMovie_list</span><span class="params">()</span>:</span></span><br><span class="line"> resp = request.urlopen(<span class="string">'https://movie.douban.com/nowplaying/hangzhou/'</span>)</span><br><span class="line"> html_data = resp.read().decode(<span class="string">'utf-8'</span>)</span><br><span class="line"> soup = bs(html_data, <span class="string">'html.parser'</span>)</span><br><span class="line"> nowplaying_movie = soup.find_all(<span class="string">'div'</span>, id=<span class="string">'nowplaying'</span>)</span><br><span class="line"> nowplaying_movie_list = nowplaying_movie[<span class="number">0</span>].find_all(<span class="string">'li'</span>, class_=<span class="string">'list-item'</span>)</span><br><span class="line"> nowplaying_list = []</span><br><span class="line"> <span class="keyword">for</span> item <span class="keyword">in</span> nowplaying_movie_list:</span><br><span class="line"> nowplaying_dict = {}</span><br><span class="line"> nowplaying_dict[<span class="string">'id'</span>] = item[<span class="string">'data-subject'</span>]</span><br><span class="line"> <span class="keyword">for</span> tag_img_item <span class="keyword">in</span> item.find_all(<span class="string">'img'</span>):</span><br><span class="line"> nowplaying_dict[<span class="string">'name'</span>] = tag_img_item[<span class="string">'alt'</span>]</span><br><span class="line"> nowplaying_list.append(nowplaying_dict)</span><br><span class="line"> <span class="keyword">return</span> nowplaying_list</span><br><span class="line"></span><br><span class="line"><span class="comment">#爬取评论函数</span></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">getCommentsById</span><span class="params">(movieId, pageNum)</span>:</span></span><br><span class="line"> eachCommentList = [];</span><br><span class="line"> <span class="keyword">if</span> pageNum><span class="number">0</span>:</span><br><span class="line"> start = (pageNum<span class="number">-1</span>) * <span class="number">20</span></span><br><span class="line"> <span class="keyword">else</span>:</span><br><span class="line"> <span class="keyword">return</span> <span class="literal">False</span></span><br><span class="line"> requrl = <span class="string">'https://movie.douban.com/subject/'</span> + movieId + <span class="string">'/comments'</span> +<span class="string">'?'</span> +<span class="string">'start='</span> + str(start) + <span class="string">'&limit=20'</span></span><br><span class="line"> print(requrl)</span><br><span class="line"> resp = request.urlopen(requrl)</span><br><span class="line"> html_data = resp.read().decode(<span class="string">'utf-8'</span>)</span><br><span class="line"> soup = bs(html_data, <span class="string">'html.parser'</span>)</span><br><span class="line"> comment_div_lits = soup.find_all(<span class="string">'div'</span>, class_=<span class="string">'comment'</span>)</span><br><span class="line"> <span class="keyword">for</span> item <span class="keyword">in</span> comment_div_lits:</span><br><span class="line"> <span class="keyword">if</span> item.find_all(<span class="string">'p'</span>)[<span class="number">0</span>].string <span class="keyword">is</span> <span class="keyword">not</span> <span class="literal">None</span>:</span><br><span class="line"> eachCommentList.append(item.find_all(<span class="string">'p'</span>)[<span class="number">0</span>].string)</span><br><span class="line"> <span class="keyword">return</span> eachCommentList</span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">main</span><span class="params">()</span>:</span></span><br><span class="line"> <span class="comment">#循环获取第一个电影的前10页评论</span></span><br><span class="line"> commentList = []</span><br><span class="line"> NowPlayingMovie_list = getNowPlayingMovie_list()</span><br><span class="line"> <span class="keyword">for</span> i <span class="keyword">in</span> range(<span class="number">10</span>):</span><br><span class="line"> num = i + <span class="number">1</span></span><br><span class="line"> commentList_temp = getCommentsById(NowPlayingMovie_list[<span class="number">0</span>][<span class="string">'id'</span>], num)</span><br><span class="line"> commentList.append(commentList_temp)</span><br><span class="line"></span><br><span class="line"> <span class="comment">#将列表中的数据转换为字符串</span></span><br><span class="line"> comments = <span class="string">''</span></span><br><span class="line"> <span class="keyword">for</span> k <span class="keyword">in</span> range(len(commentList)):</span><br><span class="line"> comments = comments + (str(commentList[k])).strip()</span><br><span class="line"></span><br><span class="line"> <span class="comment">#使用正则表达式去除标点符号</span></span><br><span class="line"> pattern = re.compile(<span class="string">r'[\u4e00-\u9fa5]+'</span>)</span><br><span class="line"> filterdata = re.findall(pattern, comments)</span><br><span class="line"> cleaned_comments = <span class="string">''</span>.join(filterdata)</span><br><span class="line"></span><br><span class="line"> <span class="comment">#使用结巴分词进行中文分词</span></span><br><span class="line"> segment = jieba.lcut(cleaned_comments)</span><br><span class="line"> words_df=pd.DataFrame({<span class="string">'segment'</span>:segment})</span><br><span class="line"></span><br><span class="line"> <span class="comment">#去掉停用词</span></span><br><span class="line"> stopwords=pd.read_csv(<span class="string">"stopwords.txt"</span>,index_col=<span class="literal">False</span>,quoting=<span class="number">3</span>,sep=<span class="string">"\t"</span>,names=[<span class="string">'stopword'</span>], encoding=<span class="string">'utf-8'</span>)<span class="comment">#quoting=3全不引用</span></span><br><span class="line"> words_df=words_df[~words_df.segment.isin(stopwords.stopword)]</span><br><span class="line"></span><br><span class="line"> <span class="comment">#统计词频</span></span><br><span class="line"> words_stat=words_df.groupby(by=[<span class="string">'segment'</span>])[<span class="string">'segment'</span>].agg({<span class="string">"计数"</span>:numpy.size})</span><br><span class="line"> words_stat=words_stat.reset_index().sort_values(by=[<span class="string">"计数"</span>],ascending=<span class="literal">False</span>)</span><br><span class="line"></span><br><span class="line"> <span class="comment">#用词云进行显示</span></span><br><span class="line"> wordcloud=WordCloud(font_path=<span class="string">"simhei.ttf"</span>,background_color=<span class="string">"white"</span>,max_font_size=<span class="number">80</span>)</span><br><span class="line"> word_frequence = {x[<span class="number">0</span>]:x[<span class="number">1</span>] <span class="keyword">for</span> x <span class="keyword">in</span> words_stat.head(<span class="number">1000</span>).values}</span><br><span class="line"></span><br><span class="line"> word_frequence_list = []</span><br><span class="line"> <span class="keyword">for</span> key <span class="keyword">in</span> word_frequence:</span><br><span class="line"> temp = (key,word_frequence[key])</span><br><span class="line"> word_frequence_list.append(temp)</span><br><span class="line"></span><br><span class="line"> wordcloud=wordcloud.fit_words(word_frequence_list)</span><br><span class="line"> plt.imshow(wordcloud)</span><br><span class="line"></span><br><span class="line"><span class="comment">#主函数</span></span><br><span class="line">main()</span><br></pre></td></tr></table></figure><blockquote><p>转载自 链接地址: <a href="http://python.jobbole.com/88325/" target="_blank" rel="noopener">http://python.jobbole.com/88325/</a></p></blockquote><blockquote><p>个人博客 欢迎来访： <a href="http://zj2626.com" target="_blank" rel="noopener">http://zj2626.com</a></p></blockquote>]]></content>
<summary type="html">
<h2 id="前言："><a href="#前言：" class="headerlink" title="前言："></a>前言：</h2><p>在我学习完python的基础知识之后，当然想要练练手，加深一下对python以及其语法的理解，<br>所以听说爬虫特别有成就感，非常有利于学习and娱乐，以及培养学习的兴趣，so就到处百度爬虫的相关文章，网上的确有很多相关的，但我还是决定自己写写,<br><strong>只有自己写下了讲出来才能代表真的学会了这么技术</strong></p>
</summary>
<category term="爬虫" scheme="http://zj2626.github.io/categories/%E7%88%AC%E8%99%AB/"/>
<category term="python" scheme="http://zj2626.github.io/tags/python/"/>
<category term="爬虫" scheme="http://zj2626.github.io/tags/%E7%88%AC%E8%99%AB/"/>
</entry>
<entry>
<title>VM options配置</title>
<link href="http://zj2626.github.io/2021/06/01/20170924_VM-options%E9%85%8D%E7%BD%AE/"/>
<id>http://zj2626.github.io/2021/06/01/20170924_VM-options配置/</id>
<published>2021-06-01T14:21:10.527Z</published>
<updated>2021-06-01T01:18:34.720Z</updated>
<content type="html"><![CDATA[<blockquote><p>转载自 链接地址: <a href="http://www.cnblogs.com/mingforyou/archive/2012/03/03/2378143.html" target="_blank" rel="noopener">http://www.cnblogs.com/mingforyou/archive/2012/03/03/2378143.html</a><br>转载自 链接地址: <a href="http://unixboy.iteye.com/blog/174173/" target="_blank" rel="noopener">http://unixboy.iteye.com/blog/174173/</a></p></blockquote><a id="more"></a><h1 id="文章一"><a href="#文章一" class="headerlink" title="文章一"></a>文章一</h1><p>Eclipse崩溃，错误提示：<br>MyEclipse has detected that less than 5% of the 64MB of Perm<br>Gen (Non-heap memory) space remains. It is strongly recommended<br>that you exit and restart MyEclipse with new virtual machine memory<br>paramters to increase this memory. Failure to do so can result in<br>data loss. The recommended Eclipse memory parameters are:<br>eclipse.exe -vmargs -Xms128M -Xmx512M -XX:PermSize=64M -XX:MaxPermSize=128M</p><p>1.参数的含义<br>-vmargs -Xms128M -Xmx512M -XX:PermSize=64M -XX:MaxPermSize=128M<br>-vmargs 说明后面是VM的参数，所以后面的其实都是JVM的参数了<br>-Xms128m JVM初始分配的堆内存<br>-Xmx512m JVM最大允许分配的堆内存，按需分配<br>-XX:PermSize=64M JVM初始分配的非堆内存<br>-XX:MaxPermSize=128M JVM最大允许分配的非堆内存，按需分配</p><p>我们首先了解一下JVM内存管理的机制，然后再解释每个参数代表的含义。</p><h2 id="堆-Heap-和非堆-Non-heap-内存"><a href="#堆-Heap-和非堆-Non-heap-内存" class="headerlink" title="堆(Heap)和非堆(Non-heap)内存"></a>堆(Heap)和非堆(Non-heap)内存</h2><p> 按照官方的说法：“Java 虚拟机具有一个堆，堆是运行时数据区域，所有类实例和数组的内存均从此处分配。堆是在 Java 虚拟机启动时创建的。”“在JVM中堆之外的内存称为非堆内存(Non-heap memory)”。<br> 可以看出JVM主要管理两种类型的内存：堆和非堆。简单来说堆就是Java代码可及的内存，是留给开发人员使用的；非堆就是JVM留给自己用的，<br> 所以方法区、JVM内部处理或优化所需的内存(如JIT编译后的代码缓存)、每个类结构(如运行时常数池、字段和方法数据)以及方法和构造方法的代码都在非堆内存中。 </p><h3 id="堆内存分配"><a href="#堆内存分配" class="headerlink" title="堆内存分配"></a>堆内存分配</h3><p> JVM初始分配的堆内存由-Xms指定，默认是物理内存的1/64；JVM最大分配的堆内存由-Xmx指定，默认是物理内存的1/4。默认空余堆内存小于40%时，JVM就会增大堆直到-Xmx的最大限制；<br> 空余堆内存大于70%时，JVM会减少堆直到-Xms的最小限制。因此服务器一般设置-Xms、-Xmx 相等以避免在每次GC 后调整堆的大小。<br> 说明：如果-Xmx 不指定或者指定偏小，应用可能会导致java.lang.OutOfMemory错误，此错误来自JVM，不是Throwable的，无法用try…catch捕捉。 </p><h3 id="非堆内存分配"><a href="#非堆内存分配" class="headerlink" title="非堆内存分配"></a>非堆内存分配</h3><p> JVM使用-XX:PermSize设置非堆内存初始值，默认是物理内存的1/64；由XX:MaxPermSize设置最大非堆内存的大小，默认是物理内存的1/4。（还有一说：MaxPermSize缺省值和-server -client选项相关，<br> -server选项下默认MaxPermSize为64m，-client选项下默认MaxPermSize为32m。这个我没有实验。）<br> 上面错误信息中的PermGen space的全称是Permanent Generation space，是指内存的永久保存区域。还没有弄明白PermGen space是属于非堆内存，还是就是非堆内存，但至少是属于了。<br>XX:MaxPermSize设置过小会导致java.lang.OutOfMemoryError: PermGen space 就是内存益出。<br>说说为什么会内存益出：<br>（1）这一部分内存用于存放Class和Meta的信息，Class在被 Load的时候被放入PermGen space区域，它和存放Instance的Heap区域不同。<br>（2）GC(Garbage Collection)不会在主程序运行期对PermGen space进行清理，所以如果你的APP会LOAD很多CLASS 的话,就很可能出现PermGen space错误。<br> 这种错误常见在web服务器对JSP进行pre compile的时候。 </p><h2 id="JVM内存限制-最大值"><a href="#JVM内存限制-最大值" class="headerlink" title="JVM内存限制(最大值)"></a>JVM内存限制(最大值)</h2><p> 首先JVM内存限制于实际的最大物理内存，假设物理内存无限大的话，JVM内存的最大值跟操作系统有很大的关系。简单的说就32位处理器虽然可控内存空间有4GB,但是具体的操作系统会给一个限制，<br> 这个限制一般是2GB-3GB（一般来说Windows系统下为1.5G-2G，Linux系统下为2G-3G），而64bit以上的处理器就不会有限制了。</p><ol start="2"><li><p>为什么有的机器我将-Xmx和-XX:MaxPermSize都设置为512M之后Eclipse可以启动，而有些机器无法启动？<br>通过上面对JVM内存管理的介绍我们已经了解到JVM内存包含两种：堆内存和非堆内存，另外JVM最大内存首先取决于实际的物理内存和操作系统。所以说设置VM参数导致程序无法启动主要有以下几种原因：<br>1) 参数中-Xms的值大于-Xmx，或者-XX:PermSize的值大于-XX:MaxPermSize；<br>2) -Xmx的值和-XX:MaxPermSize的总和超过了JVM内存的最大限制，比如当前操作系统最大内存限制，或者实际的物理内存等等。说到实际物理内存这里需要说明一点的是，<br>如果你的内存是1024MB，但实际系统中用到的并不可能是1024MB，因为有一部分被硬件占用了。</p></li><li><p>为何将上面的参数写入到eclipse.ini文件Eclipse没有执行对应的设置？<br>那为什么同样的参数在快捷方式或者命令行中有效而在eclipse.ini文件中是无效的呢？这是因为我们没有遵守eclipse.ini文件的设置规则：<br>参数形如“项 值”这种形式，中间有空格的需要换行书写，如果值中有空格的需要用双引号包括起来。比如我们使用-vm C:/Java/jre1.6.0/bin/javaw.exe参数设置虚拟机，<br>在eclipse.ini文件中要写成这样：</p></li></ol><p>-vm<br>C:/Java/jre1.6.0/bin/javaw.exe<br>-vmargs<br>-Xms128M<br>-Xmx512M<br>-XX:PermSize=64M<br>-XX:MaxPermSize=128M </p><p>实际运行的结果可以通过Eclipse中“Help”-“About Eclipse SDK”窗口里面的“Configuration Details”按钮进行查看。<br>另外需要说明的是，Eclipse压缩包中自带的eclipse.ini文件内容是这样的：<br>-showsplash<br>org.eclipse.platform<br>–launcher.XXMaxPermSize<br>256m<br>-vmargs<br>-Xms40m<br>-Xmx256m<br>其中–launcher.XXMaxPermSize（注意最前面是两个连接线）跟-XX:MaxPermSize参数的含义基本是一样的，我觉得唯一的区别就是前者是eclipse.exe启动的时候设置的参数，<br>而后者是eclipse所使用的JVM中的参数。其实二者设置一个就可以了，所以这里可以把–launcher.XXMaxPermSize和下一行使用#注释掉。</p><ol start="4"><li>其他的启动参数。 如果你有一个双核的CPU，也许可以尝试这个参数:<br>-XX:+UseParallelGC<br>让GC可以更快的执行。（只是JDK 5里对GC新增加的参数）</li></ol><p>补充：<br> 如果你的WEB APP下都用了大量的第三方jar，其大小超过了服务器jvm默认的大小，那么就会产生内存益出问题了。<br>解决方法： 设置MaxPermSize大小<br>可以在myelipse里选中相应的服务器比如tomcat5，展开里面的JDK子项页面，来增加服务器启动的JVM参数设置：</p><p>-Xms128m<br>-Xmx256m<br>-XX:PermSize=128M<br>-XX:MaxNewSize=256m<br>-XX:MaxPermSize=256m</p><p>或者手动设置MaxPermSize大小,比如tomcat，<br>修改TOMCAT_HOME/bin/catalina.bat，在echo “Using CATALINA_BASE: $CATALINA_BASE”上面加入以下行：<br>JAVA_OPTS=”-server -XX:PermSize=64M -XX:MaxPermSize=128m</p><p>建议：将相同的第三方jar文件移置到tomcat/shared/lib目录下，这样可以减少jar 文档重复占用内存</p><h1 id="文章二"><a href="#文章二" class="headerlink" title="文章二"></a>文章二</h1><h2 id="1-堆大小设置"><a href="#1-堆大小设置" class="headerlink" title="1.堆大小设置"></a>1.堆大小设置</h2><p>JVM 中最大堆大小有三方面限制：相关操作系统的数据模型（32-bt还是64-bit）限制；系统的可用虚拟内存限制；系统的可用物理内存限制。32位系统下，一般限制在1.5G~2G；64为操作系统对内存无限制。我在Windows Server 2003 系统，3.5G物理内存，JDK5.0下测试，最大可设置为1478m。</p><h3 id="典型设置："><a href="#典型设置：" class="headerlink" title="典型设置："></a>典型设置：</h3><ul><li>java -Xmx3550m -Xms3550m -Xmn2g -Xss128k</li></ul><p>-Xmx3550m：设置JVM最大可用内存为3550M。<br>-Xms3550m：设置JVM初始内存为3550m。此值可以设置与-Xmx相同，以避免每次垃圾回收完成后JVM重新分配内存。<br>-Xmn2g：设置年轻代大小为2G。整个JVM内存大小=年轻代大小 + 年老代大小 + 持久代大小。持久代一般固定大小为64m，所以增大年轻代后，将会减小年老代大小。此值对系统性能影响较大，Sun官方推荐配置为整个堆的3/8。<br>-Xss128k：设置每个线程的堆栈大小。JDK5.0以后每个线程堆栈大小为1M，以前每个线程堆栈大小为256K。更具应用的线程所需内存大小进行调整。在相同物理内存下，减小这个值能生成更多的线程。但是操作系统对一个进程内的线程数还是有限制的，不能无限生成，经验值在3000~5000左右。</p><ul><li>java -Xmx3550m -Xms3550m -Xss128k -XX:NewRatio=4 -XX:SurvivorRatio=4 -XX:MaxPermSize=16m -XX:MaxTenuringThreshold=0</li></ul><p>-XX:NewRatio=4:设置年轻代（包括Eden和两个Survivor区）与年老代的比值（除去持久代）。设置为4，则年轻代与年老代所占比值为1：4，年轻代占整个堆栈的1/5<br>-XX:SurvivorRatio=4：设置年轻代中Eden区与Survivor区的大小比值。设置为4，则两个Survivor区与一个Eden区的比值为2:4，一个Survivor区占整个年轻代的1/6<br>-XX:MaxPermSize=16m:设置持久代大小为16m。<br>-XX:MaxTenuringThreshold=0：设置垃圾最大年龄。如果设置为0的话，则年轻代对象不经过Survivor区，直接进入年老代。对于年老代比较多的应用，可以提高效率。如果将此值设置为一个较大值，则年轻代对象会在Survivor区进行多次复制，这样可以增加对象再年轻代的存活时间，增加在年轻代即被回收的概论。</p><h2 id="2-回收器选择"><a href="#2-回收器选择" class="headerlink" title="2.回收器选择"></a>2.回收器选择</h2><p>JVM给了三种选择：串行收集器、并行收集器、并发收集器，但是串行收集器只适用于小数据量的情况，所以这里的选择主要针对并行收集器和并发收集器。默认情况下，JDK5.0以前都是使用串行收集器，如果想使用其他收集器需要在启动时加入相应参数。JDK5.0以后，JVM会根据当前系统配置进行判断。</p><h3 id="吞吐量优先的并行收集器"><a href="#吞吐量优先的并行收集器" class="headerlink" title="吞吐量优先的并行收集器"></a>吞吐量优先的并行收集器</h3><p>如上文所述，并行收集器主要以到达一定的吞吐量为目标，适用于科学技术和后台处理等。</p><blockquote><p>典型配置：</p></blockquote><ul><li><p>java -Xmx3800m -Xms3800m -Xmn2g -Xss128k -XX:+UseParallelGC -XX:ParallelGCThreads=20<br>-XX:+UseParallelGC：选择垃圾收集器为并行收集器。此配置仅对年轻代有效。即上述配置下，年轻代使用并发收集，而年老代仍旧使用串行收集。<br>-XX:ParallelGCThreads=20：配置并行收集器的线程数，即：同时多少个线程一起进行垃圾回收。此值最好配置与处理器数目相等。</p></li><li><p>java -Xmx3550m -Xms3550m -Xmn2g -Xss128k -XX:+UseParallelGC -XX:ParallelGCThreads=20 -XX:+UseParallelOldGC<br>-XX:+UseParallelOldGC：配置年老代垃圾收集方式为并行收集。JDK6.0支持对年老代并行收集。</p></li><li><p>java -Xmx3550m -Xms3550m -Xmn2g -Xss128k -XX:+UseParallelGC -XX:MaxGCPauseMillis=100<br>-XX:MaxGCPauseMillis=100:设置每次年轻代垃圾回收的最长时间，如果无法满足此时间，JVM会自动调整年轻代大小，以满足此值。</p></li><li><p>java -Xmx3550m -Xms3550m -Xmn2g -Xss128k -XX:+UseParallelGC -XX:MaxGCPauseMillis=100 -XX:+UseAdaptiveSizePolicy<br>-XX:+UseAdaptiveSizePolicy：设置此选项后，并行收集器会自动选择年轻代区大小和相应的Survivor区比例，以达到目标系统规定的最低相应时间或者收集频率等，此值建议使用并行收集器时，一直打开。</p></li></ul><h3 id="响应时间优先的并发收集器"><a href="#响应时间优先的并发收集器" class="headerlink" title="响应时间优先的并发收集器"></a>响应时间优先的并发收集器</h3><p>如上文所述，并发收集器主要是保证系统的响应时间，减少垃圾收集时的停顿时间。适用于应用服务器、电信领域等。</p><blockquote><p>典型配置：</p></blockquote><ul><li>java -Xmx3550m -Xms3550m -Xmn2g -Xss128k -XX:ParallelGCThreads=20 -XX:+UseConcMarkSweepGC -XX:+UseParNewGC<br>-XX:+UseConcMarkSweepGC：设置年老代为并发收集。测试中配置这个以后，-XX:NewRatio=4的配置失效了，原因不明。所以，此时年轻代大小最好用-Xmn设置。<br>-XX:+UseParNewGC:设置年轻代为并行收集。可与CMS收集同时使用。JDK5.0以上，JVM会根据系统配置自行设置，所以无需再设置此值。</li><li>java -Xmx3550m -Xms3550m -Xmn2g -Xss128k -XX:+UseConcMarkSweepGC -XX:CMSFullGCsBeforeCompaction=5 -XX:+UseCMSCompactAtFullCollection<br>-XX:CMSFullGCsBeforeCompaction：由于并发收集器不对内存空间进行压缩、整理，所以运行一段时间以后会产生“碎片”，使得运行效率降低。此值设置运行多少次GC以后对内存空间进行压缩、整理。<br>-XX:+UseCMSCompactAtFullCollection：打开对年老代的压缩。可能会影响性能，但是可以消除碎片</li></ul><blockquote><p>辅助信息</p></blockquote><p>JVM提供了大量命令行参数，打印信息，供调试使用。主要有以下一些：</p><ul><li>-XX:+PrintGC<br>输出形式：[GC 118250K->113543K(130112K), 0.0094143 secs]<pre><code>[Full GC 121376K->10414K(130112K), 0.0650971 secs]</code></pre></li><li>-XX:+PrintGCDetails<br>输出形式：[GC [DefNew: 8614K->781K(9088K), 0.0123035 secs] 118250K->113543K(130112K), 0.0124633 secs]<pre><code>[GC [DefNew: 8614K->8614K(9088K), 0.0000665 secs][Tenured: 112761K->10414K(121024K), 0.0433488 secs] 121376K->10414K(130112K), 0.0436268 secs]</code></pre></li><li>-XX:+PrintGCTimeStamps -XX:+PrintGC：PrintGCTimeStamps可与上面两个混合使用<br>输出形式：11.851: [GC 98328K->93620K(130112K), 0.0082960 secs]</li><li>-XX:+PrintGCApplicationConcurrentTime:打印每次垃圾回收前，程序未中断的执行时间。可与上面混合使用<br>输出形式：Application time: 0.5291524 seconds</li><li>-XX:+PrintGCApplicationStoppedTime：打印垃圾回收期间程序暂停的时间。可与上面混合使用<br>输出形式：Total time for which application threads were stopped: 0.0468229 seconds</li><li>-XX:PrintHeapAtGC:打印GC前后的详细堆栈信息</li></ul><p>输出形式：</p><pre><code>34.702: [GC {Heap before gc invocations=7: def new generation total 55296K, used 52568K [0x1ebd0000, 0x227d0000, 0x227d0000)eden space 49152K, 99% used [0x1ebd0000, 0x21bce430, 0x21bd0000)from space 6144K, 55% used [0x221d0000, 0x22527e10, 0x227d0000) to space 6144K, 0% used [0x21bd0000, 0x21bd0000, 0x221d0000) tenured generation total 69632K, used 2696K [0x227d0000, 0x26bd0000, 0x26bd0000)the space 69632K, 3% used [0x227d0000, 0x22a720f8, 0x22a72200, 0x26bd0000) compacting perm gen total 8192K, used 2898K [0x26bd0000, 0x273d0000, 0x2abd0000) the space 8192K, 35% used [0x26bd0000, 0x26ea4ba8, 0x26ea4c00, 0x273d0000) ro space 8192K, 66% used [0x2abd0000, 0x2b12bcc0, 0x2b12be00, 0x2b3d0000) rw space 12288K, 46% used [0x2b3d0000, 0x2b972060, 0x2b972200, 0x2bfd0000)34.735: [DefNew: 52568K->3433K(55296K), 0.0072126 secs] 55264K->6615K(124928K)Heap after gc invocations=8: def new generation total 55296K, used 3433K [0x1ebd0000, 0x227d0000, 0x227d0000)eden space 49152K, 0% used [0x1ebd0000, 0x1ebd0000, 0x21bd0000) from space 6144K, 55% used [0x21bd0000, 0x21f2a5e8, 0x221d0000) to space 6144K, 0% used [0x221d0000, 0x221d0000, 0x227d0000) tenured generation total 69632K, used 3182K [0x227d0000, 0x26bd0000, 0x26bd0000)the space 69632K, 4% used [0x227d0000, 0x22aeb958, 0x22aeba00, 0x26bd0000) compacting perm gen total 8192K, used 2898K [0x26bd0000, 0x273d0000, 0x2abd0000) the space 8192K, 35% used [0x26bd0000, 0x26ea4ba8, 0x26ea4c00, 0x273d0000) ro space 8192K, 66% used [0x2abd0000, 0x2b12bcc0, 0x2b12be00, 0x2b3d0000) rw space 12288K, 46% used [0x2b3d0000, 0x2b972060, 0x2b972200, 0x2bfd0000)}, 0.0757599 secs]</code></pre><ul><li>-Xloggc:filename:与上面几个配合使用，把相关日志信息记录到文件以便分析。</li></ul><blockquote><p>常见配置汇总</p></blockquote><p>1.堆设置</p><ul><li>-Xms:初始堆大小</li><li>-Xmx:最大堆大小</li><li>-XX:NewSize=n:设置年轻代大小</li><li>-XX:NewRatio=n:设置年轻代和年老代的比值。如:为3，表示年轻代与年老代比值为1：3，年轻代占整个年轻代年老代和的1/4</li><li>-XX:SurvivorRatio=n:年轻代中Eden区与两个Survivor区的比值。注意Survivor区有两个。如：3，表示Eden：Survivor=3：2，一个Survivor区占整个年轻代的1/5</li><li>-XX:MaxPermSize=n:设置持久代大小</li></ul><p>2.收集器设置</p><ul><li>-XX:+UseSerialGC:设置串行收集器</li><li>-XX:+UseParallelGC:设置并行收集器</li><li>-XX:+UseParalledlOldGC:设置并行年老代收集器</li><li>-XX:+UseConcMarkSweepGC:设置并发收集器</li></ul><p>3.垃圾回收统计信息</p><ul><li>-XX:+PrintGC</li><li>-XX:+PrintGCDetails</li><li>-XX:+PrintGCTimeStamps</li><li>-Xloggc:filename</li></ul><p>4.并行收集器设置</p><ul><li>-XX:ParallelGCThreads=n:设置并行收集器收集时使用的CPU数。并行收集线程数。</li><li>-XX:MaxGCPauseMillis=n:设置并行收集最大暂停时间</li><li>-XX:GCTimeRatio=n:设置垃圾回收时间占程序运行时间的百分比。公式为1/(1+n)</li></ul><p>5.并发收集器设置</p><ul><li>-XX:+CMSIncrementalMode:设置为增量模式。适用于单CPU情况。</li><li>-XX:ParallelGCThreads=n:设置并发收集器年轻代收集方式为并行收集时，使用的CPU数。并行收集线程数。</li></ul><h2 id="调优总结"><a href="#调优总结" class="headerlink" title="调优总结"></a>调优总结</h2><p>1.年轻代大小选择</p><ul><li>响应时间优先的应用：尽可能设大，直到接近系统的最低响应时间限制（根据实际情况选择）。在此种情况下，年轻代收集发生的频率也是最小的。同时，减少到达年老代的对象。</li><li>吞吐量优先的应用：尽可能的设置大，可能到达Gbit的程度。因为对响应时间没有要求，垃圾收集可以并行进行，一般适合8CPU以上的应用。</li></ul><p>2.年老代大小选择</p><ul><li><p>响应时间优先的应用：年老代使用并发收集器，所以其大小需要小心设置，一般要考虑并发会话率和会话持续时间等一些参数。如果堆设置小了，可以会造成内存碎片、高回收频率以及应用暂停而使用传统的标记清除方式；如果堆大了，则需要较长的收集时间。最优化的方案，一般需要参考以下数据获得：</p><pre><code>并发垃圾收集信息持久代并发收集次数传统GC信息花在年轻代和年老代回收上的时间比例</code></pre></li></ul><p>减少年轻代和年老代花费的时间，一般会提高应用的效率</p><ul><li>吞吐量优先的应用：一般吞吐量优先的应用都有一个很大的年轻代和一个较小的年老代。原因是，这样可以尽可能回收掉大部分短期对象，减少中期的对象，而年老代尽存放长期存活对象。</li></ul><p>3.较小堆引起的碎片问题<br>因为年老代的并发收集器使用标记、清除算法，所以不会对堆进行压缩。当收集器回收时，他会把相邻的空间进行合并，这样可以分配给较大的对象。但是，当堆空间较小时，运行一段时间以后，就会出现“碎片”，如果并发收集器找不到足够的空间，那么并发收集器将会停止，然后使用传统的标记、清除方式进行回收。如果出现“碎片”，可能需要进行如下配置：</p><ul><li>-XX:+UseCMSCompactAtFullCollection：使用并发收集器时，开启对年老代的压缩。</li><li>-XX:CMSFullGCsBeforeCompaction=0：上面配置开启的情况下，这里设置多少次Full GC后，对年老代进行压缩</li></ul><blockquote><p>个人博客 欢迎来访： <a href="http://zj2626.github.io">http://zj2626.github.io</a></p></blockquote>]]></content>
<summary type="html">
<blockquote>
<p>转载自 链接地址: <a href="http://www.cnblogs.com/mingforyou/archive/2012/03/03/2378143.html" target="_blank" rel="noopener">http://www.cnblogs.com/mingforyou/archive/2012/03/03/2378143.html</a><br>转载自 链接地址: <a href="http://unixboy.iteye.com/blog/174173/" target="_blank" rel="noopener">http://unixboy.iteye.com/blog/174173/</a></p>
</blockquote>
</summary>
<category term="java虚拟机" scheme="http://zj2626.github.io/categories/java%E8%99%9A%E6%8B%9F%E6%9C%BA/"/>
<category term="深入了解java虚拟机" scheme="http://zj2626.github.io/tags/%E6%B7%B1%E5%85%A5%E4%BA%86%E8%A7%A3java%E8%99%9A%E6%8B%9F%E6%9C%BA/"/>
<category term="java" scheme="http://zj2626.github.io/tags/java/"/>
</entry>
<entry>
<title>JVM运行原理</title>
<link href="http://zj2626.github.io/2021/06/01/20170924_JVM%E8%BF%90%E8%A1%8C%E5%8E%9F%E7%90%86/"/>
<id>http://zj2626.github.io/2021/06/01/20170924_JVM运行原理/</id>
<published>2021-06-01T14:21:10.519Z</published>
<updated>2021-06-01T01:18:34.711Z</updated>
<content type="html"><![CDATA[<p>1、动态编译（dynamic compilation）指的是“在运行时进行编译”；与之相对的是事前编译（ahead-of-time compilation，简称AOT），也叫静态编译（static compilation）。</p><p>2、JIT编译（just-in-time compilation）狭义来说是当某段代码即将第一次被执行时进行编译，因而叫“即时编译”。JIT编译是动态编译的一种特例。JIT编译一词后来被泛化，时常与动态编译等价；但要注意广义与狭义的JIT编译所指的区别。</p><p>3、自适应动态编译（adaptive dynamic compilation）也是一种动态编译，但它通常执行的时机比JIT编译迟，先让程序“以某种式”先运行起来，收集一些信息之后再做动态编译。这样的编译可以更加优化。</p><a id="more"></a><h2 id="概述"><a href="#概述" class="headerlink" title="概述"></a>概述</h2><h3 id="JVM运行原理"><a href="#JVM运行原理" class="headerlink" title="JVM运行原理"></a>JVM运行原理</h3><img src="/2021/06/01/20170924_JVM运行原理/20160812104144969.jpg" title="图"><p>在部分商用虚拟机中（如HotSpot），Java程序最初是通过解释器（Interpreter）进行解释执行的，当虚拟机发现某个方法或代码块的运行特别频繁时，就会把这些代码认定为“热点代码”。为了提高热点代码的执行效率，在运行时，虚拟机将会把这些代码编译成与本地平台相关的机器码，并进行各种层次的优化，完成这个任务的编译器称为即时编译器（Just In Time Compiler，下文统称JIT编译器）。<br>即时编译器并不是虚拟机必须的部分，Java虚拟机规范并没有规定Java虚拟机内必须要有即时编译器存在，更没有限定或指导即时编译器应该如何去实现。但是，即时编译器编译性能的好坏、代码优化程度的高低却是衡量一款商用虚拟机优秀与否的最关键的指标之一，它也是虚拟机中最核心且最能体现虚拟机技术水平的部分。</p><p>由于Java虚拟机规范并没有具体的约束规则去限制即使编译器应该如何实现，所以这部分功能完全是与虚拟机具体实现相关的内容，如无特殊说明，我们提到的编译器、即时编译器都是指Hotspot虚拟机内的即时编译器，虚拟机也是特指HotSpot虚拟机。</p><h3 id="为什么HotSpot虚拟机要使用解释器与编译器并存的架构？"><a href="#为什么HotSpot虚拟机要使用解释器与编译器并存的架构？" class="headerlink" title="为什么HotSpot虚拟机要使用解释器与编译器并存的架构？"></a>为什么HotSpot虚拟机要使用解释器与编译器并存的架构？</h3><p>尽管并不是所有的Java虚拟机都采用解释器与编译器并存的架构，但许多主流的商用虚拟机（如HotSpot），都同时包含解释器和编译器。解释器与编译器两者各有优势：当程序需要迅速启动和执行的时候，解释器可以首先发挥作用，省去编译的时间，立即执行。在程序运行后，随着时间的推移，编译器逐渐发挥作用，把越来越多的代码编译成本地代码之后，可以获取更高的执行效率。当程序运行环境中内存资源限制较大（如部分嵌入式系统中），可以使用解释器执行节约内存，反之可以使用编译执行来提升效率。此外，如果编译后出现“罕见陷阱”，可以通过逆优化退回到解释执行。</p><img src="/2021/06/01/20170924_JVM运行原理/20160812102841736.png" title="图"><h3 id="编译的时间开销"><a href="#编译的时间开销" class="headerlink" title="编译的时间开销"></a>编译的时间开销</h3><p>解释器的执行，抽象的看是这样的：<br>输入的代码 -> [ 解释器 解释执行 ] -> 执行结果<br>而要JIT编译然后再执行的话，抽象的看则是：<br>输入的代码 -> [ 编译器 编译 ] -> 编译后的代码 -> [ 执行 ] -> 执行结果<br>说JIT比解释快，其实说的是“执行编译后的代码”比“解释器解释执行”要快，并不是说“编译”这个动作比“解释”这个动作快。<br>JIT编译再怎么快，至少也比解释执行一次略慢一些，而要得到最后的执行结果还得再经过一个“执行编译后的代码”的过程。<br>所以，对“只执行一次”的代码而言，解释执行其实总是比JIT编译执行要快。</p><p>怎么算是“只执行一次的代码”呢？粗略说，下面两个条件同时满足时就是严格的“只执行一次”<br>1、只被调用一次，例如类的构造器（class initializer，<clinit>()）<br>2、没有循环</clinit></p><p><strong>对只执行一次的代码做JIT编译再执行，可以说是得不偿失。</strong><br><strong>对只执行少量次数的代码，JIT编译带来的执行速度的提升也未必能抵消掉最初编译带来的开销。</strong><br><strong>只有对频繁执行的代码，JIT编译才能保证有正面的收益。</strong></p><h3 id="编译的空间开销"><a href="#编译的空间开销" class="headerlink" title="编译的空间开销"></a>编译的空间开销</h3><p>对一般的Java方法而言，编译后代码的大小相对于字节码的大小，膨胀比达到10x是很正常的。同上面说的时间开销一样，这里的空间开销也是，只有对执行频繁的代码才值得编译，如果把所有代码都编译则会显著增加代码所占空间，导致“代码爆炸”。<br>这也就解释了为什么有些JVM会选择不总是做JIT编译，而是选择用解释器+JIT编译器的混合执行引擎。</p><h3 id="为何HotSpot虚拟机要实现两个不同的即时编译器？"><a href="#为何HotSpot虚拟机要实现两个不同的即时编译器？" class="headerlink" title="为何HotSpot虚拟机要实现两个不同的即时编译器？"></a>为何HotSpot虚拟机要实现两个不同的即时编译器？</h3><p>HotSpot虚拟机中内置了两个即时编译器：Client Complier和Server Complier，简称为C1、C2编译器，分别用在客户端和服务端。目前主流的HotSpot虚拟机中默认是采用解释器与其中一个编译器直接配合的方式工作。程序使用哪个编译器，取决于虚拟机运行的模式。HotSpot虚拟机会根据自身版本与宿主机器的硬件性能自动选择运行模式，用户也可以使用“-client”或“-server”参数去强制指定虚拟机运行在Client模式或Server模式。<br><strong>用Client Complier获取更高的编译速度，用Server Complier 来获取更好的编译质量</strong>;为什么提供多个即时编译器与为什么提供多个垃圾收集器类似，都是为了适应不同的应用场景。</p><h3 id="哪些程序代码会被编译为本地代码？如何编译为本地代码？"><a href="#哪些程序代码会被编译为本地代码？如何编译为本地代码？" class="headerlink" title="哪些程序代码会被编译为本地代码？如何编译为本地代码？"></a>哪些程序代码会被编译为本地代码？如何编译为本地代码？</h3><p>程序中的代码只有是热点代码时，才会编译为本地代码，那么什么是热点代码呢？<br>运行过程中会被即时编译器编译的“热点代码”有两类：<br>1、被多次调用的方法。<br>2、被多次执行的循环体。<br>两种情况，编译器都是以整个方法作为编译对象。 这种编译方法因为编译发生在方法执行过程之中，因此形象的称之为栈上替换（On Stack Replacement，OSR），即方法栈帧还在栈上，方法就被替换了</p><h3 id="如何判断方法或一段代码或是不是热点代码呢？"><a href="#如何判断方法或一段代码或是不是热点代码呢？" class="headerlink" title="如何判断方法或一段代码或是不是热点代码呢？"></a>如何判断方法或一段代码或是不是热点代码呢？</h3><p>要知道方法或一段代码是不是热点代码，是不是需要触发即时编译，需要进行Hot Spot Detection（热点探测）。<br>目前主要的热点探测方式有以下两种：</p><ul><li>基于采样的热点探测</li></ul><p>采用这种方法的虚拟机会周期性地检查各个线程的栈顶，如果发现某些方法经常出现在栈顶，那这个方法就是“热点方法”。这种探测方法的好处是实现简单高效，还可以很容易地获取方法调用关系（将调用堆栈展开即可），缺点是很难精确地确认一个方法的热度，容易因为受到线程阻塞或别的外界因素的影响而扰乱热点探测。</p><ul><li>基于计数器的热点探测</li></ul><p>采用这种方法的虚拟机会为每个方法（甚至是代码块）建立计数器，统计方法的执行次数，如果执行次数超过一定的阀值，就认为它是“热点方法”。这种统计方法实现复杂一些，需要为每个方法建立并维护计数器，而且不能直接获取到方法的调用关系，但是它的统计结果相对更加精确严谨。</p><h3 id="HotSpot虚拟机中使用的是哪钟热点检测方式呢？"><a href="#HotSpot虚拟机中使用的是哪钟热点检测方式呢？" class="headerlink" title="HotSpot虚拟机中使用的是哪钟热点检测方式呢？"></a>HotSpot虚拟机中使用的是哪钟热点检测方式呢？</h3><p>在HotSpot虚拟机中使用的是第二种——基于计数器的热点探测方法，因此它为每个方法准备了两个计数器：方法调用计数器和回边计数器。在确定虚拟机运行参数的前提下，这两个计数器都有一个确定的阈值，当计数器超过阈值溢出了，就会触发JIT编译。</p><blockquote><p>方法调用计数器</p></blockquote><img src="/2021/06/01/20170924_JVM运行原理/20160812101630575.png" title="图"><pre><code>顾名思义，这个计数器用于统计方法被调用的次数。当一个方法被调用时，会先检查该方法是否存在被JIT编译过的版本，如果存在，则优先使用编译后的本地代码来执行。如果不存在已被编译过的版本，则将此方法的调用计数器值加1，然后判断方法调用计数器与回边计数器值之和是否超过方法调用计数器的阈值。如果超过阈值，那么将会向即时编译器提交一个该方法的代码编译请求。如果不做任何设置，执行引擎并不会同步等待编译请求完成，而是继续进行解释器按照解释方式执行字节码，直到提交的请求被编译器编译完成。当编译工作完成之后，这个方法的调用入口地址就会系统自动改写成新的，下一次调用该方法时就会使用已编译的版本。</code></pre><blockquote><p>回边计数器</p></blockquote><img src="/2021/06/01/20170924_JVM运行原理/20160812102239062.png" title="图"><pre><code>它的作用就是统计一个方法中循环体代码执行的次数，在字节码中遇到控制流向后跳转的指令称为“回边”。</code></pre><h3 id="如何编译为本地代码？"><a href="#如何编译为本地代码？" class="headerlink" title="如何编译为本地代码？"></a>如何编译为本地代码？</h3><p>Server Compiler和Client Compiler两个编译器的编译过程是不一样的。<br>对Client Compiler来说，它是一个简单快速的编译器，主要关注点在于局部优化，而放弃许多耗时较长的全局优化手段。<br>而Server Compiler则是专门面向服务器端的，并为服务端的性能配置特别调整过的编译器，是一个充分优化过的高级编译器。</p><p>参考:<br>《深入理解Java虚拟机》<br><a href="http://blog.csdn.net/u010412719/article/details/47008717" target="_blank" rel="noopener">http://blog.csdn.net/u010412719/article/details/47008717</a><br><a href="https://zhuanlan.zhihu.com/p/19977592" target="_blank" rel="noopener">https://zhuanlan.zhihu.com/p/19977592</a><br><a href="http://www.zhihu.com/question/37389356/answer/73820511" target="_blank" rel="noopener">http://www.zhihu.com/question/37389356/answer/73820511</a></p><blockquote><p>个人博客 欢迎来访： <a href="http://zj2626.github.io">http://zj2626.github.io</a></p></blockquote>]]></content>
<summary type="html">
<p>1、动态编译（dynamic compilation）指的是“在运行时进行编译”；与之相对的是事前编译（ahead-of-time compilation，简称AOT），也叫静态编译（static compilation）。</p>
<p>2、JIT编译（just-in-time compilation）狭义来说是当某段代码即将第一次被执行时进行编译，因而叫“即时编译”。JIT编译是动态编译的一种特例。JIT编译一词后来被泛化，时常与动态编译等价；但要注意广义与狭义的JIT编译所指的区别。</p>
<p>3、自适应动态编译（adaptive dynamic compilation）也是一种动态编译，但它通常执行的时机比JIT编译迟，先让程序“以某种式”先运行起来，收集一些信息之后再做动态编译。这样的编译可以更加优化。</p>
</summary>
<category term="java虚拟机" scheme="http://zj2626.github.io/categories/java%E8%99%9A%E6%8B%9F%E6%9C%BA/"/>
<category term="深入了解java虚拟机" scheme="http://zj2626.github.io/tags/%E6%B7%B1%E5%85%A5%E4%BA%86%E8%A7%A3java%E8%99%9A%E6%8B%9F%E6%9C%BA/"/>
<category term="java" scheme="http://zj2626.github.io/tags/java/"/>
</entry>
<entry>
<title>JVM的年轻代以及GC回收细节</title>
<link href="http://zj2626.github.io/2021/06/01/20170924_JVM%E7%9A%84%E5%B9%B4%E8%BD%BB%E4%BB%A3%E4%BB%A5%E5%8F%8AGC%E5%9B%9E%E6%94%B6%E7%BB%86%E8%8A%82/"/>
<id>http://zj2626.github.io/2021/06/01/20170924_JVM的年轻代以及GC回收细节/</id>
<published>2021-06-01T14:21:10.515Z</published>
<updated>2021-06-01T01:18:34.649Z</updated>
<content type="html"><![CDATA[<blockquote><p>转载自并发编程网 链接地址: <a href="http://ifeve.com/jvm-yong-generation/" target="_blank" rel="noopener">http://ifeve.com/jvm-yong-generation/</a></p></blockquote><a id="more"></a><h2 id="为什么会有年轻代"><a href="#为什么会有年轻代" class="headerlink" title="为什么会有年轻代"></a>为什么会有年轻代</h2><p>我们先来屡屡，为什么需要把堆分代？不分代不能完成他所做的事情么？其实不分代完全可以，分代的唯一理由就是优化GC性能。你先想想，如果没有分代，那我们所有的对象都在一块，GC的时候我们要找到哪些对象没用，这样就会对堆的所有区域进行扫描。而我们的很多对象都是朝生夕死的，如果分代的话，我们把新创建的对象放到某一地方，当GC的时候先把这块存“朝生夕死”对象的区域进行回收，这样就会腾出很大的空间出来。</p><h2 id="年轻代中的GC"><a href="#年轻代中的GC" class="headerlink" title="年轻代中的GC"></a>年轻代中的GC</h2><p>HotSpot JVM把年轻代分为了三部分：1个Eden区和2个Survivor区（分别叫from和to）。默认比例为8：1,为啥默认会是这个比例，接下来我们会聊到。一般情况下，新创建的对象都会被分配到Eden区(一些大对象特殊处理),这些对象经过第一次Minor GC后，如果仍然存活，将会被移到Survivor区。对象在Survivor区中每熬过一次Minor GC，年龄就会增加1岁，当它的年龄增加到一定程度时，就会被移动到年老代中。</p><p>因为年轻代中的对象基本都是朝生夕死的(80%以上)，所以在年轻代的垃圾回收算法使用的是复制算法，复制算法的基本思想就是将内存分为两块，每次只用其中一块，当这一块内存用完，就将还活着的对象复制到另外一块上面。复制算法不会产生内存碎片。</p><p>在GC开始的时候，对象只会存在于Eden区和名为“From”的Survivor区，Survivor区“To”是空的。紧接着进行GC，Eden区中所有存活的对象都会被复制到“To”，而在“From”区中，仍存活的对象会根据他们的年龄值来决定去向。年龄达到一定值(年龄阈值，可以通过-XX:MaxTenuringThreshold来设置)的对象会被移动到年老代中，没有达到阈值的对象会被复制到“To”区域。经过这次GC后，Eden区和From区已经被清空。这个时候，“From”和“To”会交换他们的角色，也就是新的“To”就是上次GC前的“From”，新的“From”就是上次GC前的“To”。不管怎样，都会保证名为To的Survivor区域是空的。Minor GC会一直重复这样的过程，直到“To”区被填满，“To”区被填满之后，会将所有对象移动到年老代中。</p><img src="/2021/06/01/20170924_JVM的年轻代以及GC回收细节/young_gc.png" title="图"><h2 id="一个对象的这一辈子"><a href="#一个对象的这一辈子" class="headerlink" title="一个对象的这一辈子"></a>一个对象的这一辈子</h2><p>我是一个普通的java对象，我出生在Eden区，在Eden区我还看到和我长的很像的小兄弟，我们在Eden区中玩了挺长时间。有一天Eden区中的人实在是太多了，我就被迫去了Survivor区的“From”区，自从去了Survivor区，我就开始漂了，有时候在Survivor的“From”区，有时候在Survivor的“To”区，居无定所。直到我18岁的时候，爸爸说我成人了，该去社会上闯闯了。于是我就去了年老代那边，年老代里，人很多，并且年龄都挺大的，我在这里也认识了很多人。在年老代里，我生活了20年(每次GC加一岁)，然后被回收。</p><h2 id="有关年轻代的JVM参数"><a href="#有关年轻代的JVM参数" class="headerlink" title="有关年轻代的JVM参数"></a>有关年轻代的JVM参数</h2><p>1)-XX:NewSize和-XX:MaxNewSize</p><p>用于设置年轻代的大小，建议设为整个堆大小的1/3或者1/4,两个值设为一样大。</p><p>2)-XX:SurvivorRatio</p><p>用于设置Eden和其中一个Survivor的比值，这个值也比较重要。</p><p>3)-XX:+PrintTenuringDistribution</p><p>这个参数用于显示每次Minor GC时Survivor区中各个年龄段的对象的大小。</p><p>4).-XX:InitialTenuringThreshol和-XX:MaxTenuringThreshold</p><p>用于设置晋升到老年代的对象年龄的最小值和最大值，每个对象在坚持过一次Minor GC之后，年龄就加1。</p><blockquote><p>个人博客 欢迎来访： <a href="http://zj2626.github.io">http://zj2626.github.io</a></p></blockquote>]]></content>
<summary type="html">
<blockquote>
<p>转载自并发编程网 链接地址: <a href="http://ifeve.com/jvm-yong-generation/" target="_blank" rel="noopener">http://ifeve.com/jvm-yong-generation/</a></p>
</blockquote>
</summary>
<category term="java虚拟机" scheme="http://zj2626.github.io/categories/java%E8%99%9A%E6%8B%9F%E6%9C%BA/"/>
<category term="深入了解java虚拟机" scheme="http://zj2626.github.io/tags/%E6%B7%B1%E5%85%A5%E4%BA%86%E8%A7%A3java%E8%99%9A%E6%8B%9F%E6%9C%BA/"/>
<category term="java" scheme="http://zj2626.github.io/tags/java/"/>
</entry>
<entry>
<title>Minor GC、Major GC和Full GC之间的区别</title>
<link href="http://zj2626.github.io/2021/06/01/20170924_GC%E4%B9%8B%E9%97%B4%E7%9A%84%E5%8C%BA%E5%88%AB/"/>
<id>http://zj2626.github.io/2021/06/01/20170924_GC之间的区别/</id>
<published>2021-06-01T14:21:10.512Z</published>
<updated>2021-06-01T01:18:34.646Z</updated>
<content type="html"><![CDATA[<blockquote><p>转载链接地址: <a href="http://www.importnew.com/15820.html" target="_blank" rel="noopener">http://www.importnew.com/15820.html</a></p></blockquote><a id="more"></a><p>在 Plumbr 从事 GC 暂停检测相关功能的工作时，我被迫用自己的方式，通过大量文章、书籍和演讲来介绍我所做的工作。在整个过程中，经常对 Minor、Major、和 Full GC 事件的使用感到困惑。这也是我写这篇博客的原因，我希望能清楚地解释这其中的一些疑惑。</p><p>文章要求读者熟悉 JVM 内置的通用垃圾回收原则。堆内存划分为 Eden、Survivor 和 Tenured/Old 空间，代假设和其他不同的 GC 算法超出了本文讨论的范围。</p><img src="/2021/06/01/20170924_GC之间的区别/fd0c0db33776f042f62e5386131e487c.jpg" title="图"><h2 id="Minor-GC"><a href="#Minor-GC" class="headerlink" title="Minor GC"></a>Minor GC</h2><p>从年轻代空间（包括 Eden 和 Survivor 区域）回收内存被称为 Minor GC。这一定义既清晰又易于理解。但是，当发生Minor GC事件的时候，有一些有趣的地方需要注意到：</p><p>当 JVM 无法为一个新的对象分配空间时会触发 Minor GC，比如当 Eden 区满了。所以分配率越高，越频繁执行 Minor GC。<br>内存池被填满的时候，其中的内容全部会被复制，指针会从0开始跟踪空闲内存。Eden 和 Survivor 区进行了标记和复制操作，取代了经典的标记、扫描、压缩、清理操作。所以 Eden 和 Survivor 区不存在内存碎片。写指针总是停留在所使用内存池的顶部。<br>执行 Minor GC 操作时，不会影响到永久代。从永久代到年轻代的引用被当成 GC roots，从年轻代到永久代的引用在标记阶段被直接忽略掉。<br>质疑常规的认知，所有的 Minor GC 都会触发“全世界的暂停（stop-the-world）”，停止应用程序的线程。对于大部分应用程序，停顿导致的延迟都是可以忽略不计的。其中的真相就 是，大部分 Eden 区中的对象都能被认为是垃圾，永远也不会被复制到 Survivor 区或者老年代空间。如果正好相反，Eden 区大部分新生对象不符合 GC 条件，Minor GC 执行时暂停的时间将会长很多。<br>所以 Minor GC 的情况就相当清楚了——每次 Minor GC 会清理年轻代的内存。</p><h2 id="Major-GC-vs-Full-GC"><a href="#Major-GC-vs-Full-GC" class="headerlink" title="Major GC vs Full GC"></a>Major GC vs Full GC</h2><p>大家应该注意到，目前，这些术语无论是在 JVM 规范还是在垃圾收集研究论文中都没有正式的定义。但是我们一看就知道这些在我们已经知道的基础之上做出的定义是正确的，Minor GC 清理年轻带内存应该被设计得简单：</p><p>Major GC 是清理老年代。<br>Full GC 是清理整个堆空间—包括年轻代和老年代。<br>很不幸，实际上它还有点复杂且令人困惑。首先，许多 Major GC 是由 Minor GC 触发的，所以很多情况下将这两种 GC 分离是不太可能的。另一方面，许多现代垃圾收集机制会清理部分永久代空间，所以使用“cleaning”一词只是部分正确。</p><p>这使得我们不用去关心到底是叫 Major GC 还是 Full GC，大家应该关注当前的 GC 是否停止了所有应用程序的线程，还是能够并发的处理而不用停掉应用程序的线程。</p><p>这种混乱甚至内置到 JVM 标准工具。下面一个例子很好的解释了我的意思。让我们比较两个不同的工具 Concurrent Mark 和 Sweep collector (-XX:+UseConcMarkSweepGC)在 JVM 中运行时输出的跟踪记录。</p><hr><blockquote><p>第一次尝试通过 jstat 输出：</p></blockquote><p>my-precious: me$ jstat -gc -t 4235 1s</p><p>Time S0C S1C S0U S1U EC EU OC OU MC MU CCSC CCSU YGC YGCT FGC FGCT GCT<br> 5.7 34048.0 34048.0 0.0 34048.0 272640.0 194699.7 1756416.0 181419.9 18304.0 17865.1 2688.0 2497.6 3 0.275 0 0.000 0.275<br> 6.7 34048.0 34048.0 34048.0 0.0 272640.0 247555.4 1756416.0 263447.9 18816.0 18123.3 2688.0 2523.1 4 0.359 0 0.000 0.359<br> 7.7 34048.0 34048.0 0.0 34048.0 272640.0 257729.3 1756416.0 345109.8 19072.0 18396.6 2688.0 2550.3 5 0.451 0 0.000 0.451<br> 8.7 34048.0 34048.0 34048.0 34048.0 272640.0 272640.0 1756416.0 444982.5 19456.0 18681.3 2816.0 2575.8 7 0.550 0 0.000 0.550<br> 9.7 34048.0 34048.0 34046.7 0.0 272640.0 16777.0 1756416.0 587906.3 20096.0 19235.1 2944.0 2631.8 8 0.720 0 0.000 0.720<br>10.7 34048.0 34048.0 0.0 34046.2 272640.0 80171.6 1756416.0 664913.4 20352.0 19495.9 2944.0 2657.4 9 0.810 0 0.000 0.810<br>11.7 34048.0 34048.0 34048.0 0.0 272640.0 129480.8 1756416.0 745100.2 20608.0 19704.5 2944.0 2678.4 10 0.896 0 0.000 0.896<br>12.7 34048.0 34048.0 0.0 34046.6 272640.0 164070.7 1756416.0 822073.7 20992.0 19937.1 3072.0 2702.8 11 0.978 0 0.000 0.978<br>13.7 34048.0 34048.0 34048.0 0.0 272640.0 211949.9 1756416.0 897364.4 21248.0 20179.6 3072.0 2728.1 12 1.087 1 0.004 1.091<br>14.7 34048.0 34048.0 0.0 34047.1 272640.0 245801.5 1756416.0 597362.6 21504.0 20390.6 3072.0 2750.3 13 1.183 2 0.050 1.233<br>15.7 34048.0 34048.0 0.0 34048.0 272640.0 21474.1 1756416.0 757347.0 22012.0 20792.0 3200.0 2791.0 15 1.336 2 0.050 1.386<br>16.7 34048.0 34048.0 34047.0 0.0 272640.0 48378.0 1756416.0 838594.4 22268.0 21003.5 3200.0 2813.2 16 1.433 2 0.050 1.484</p><blockquote><p>这个片段是 JVM 启动后第17秒提取的。基于该信息，我们可以得出这样的结果，运行了12次 Minor GC、2次 Full GC，时间总跨度为50毫秒。通过 jconsole 或者 jvisualvm 这样的基于GUI的工具你能得到同样的结果。</p></blockquote><p>java -XX:+PrintGCDetails -XX:+UseConcMarkSweepGC eu.plumbr.demo.GarbageProducer</p><p>3.157: [GC (Allocation Failure) 3.157: [ParNew: 272640K->34048K(306688K), 0.0844702 secs] 272640K->69574K(2063104K), 0.0845560 secs] [Times: user=0.23 sys=0.03, real=0.09 secs]<br>4.092: [GC (Allocation Failure) 4.092: [ParNew: 306688K->34048K(306688K), 0.1013723 secs] 342214K->136584K(2063104K), 0.1014307 secs] [Times: user=0.25 sys=0.05, real=0.10 secs]<br>… cut for brevity …<br>11.292: [GC (Allocation Failure) 11.292: [ParNew: 306686K->34048K(306688K), 0.0857219 secs] 971599K->779148K(2063104K), 0.0857875 secs] [Times: user=0.26 sys=0.04, real=0.09 secs]<br>12.140: [GC (Allocation Failure) 12.140: [ParNew: 306688K->34046K(306688K), 0.0821774 secs] 1051788K->856120K(2063104K), 0.0822400 secs] [Times: user=0.25 sys=0.03, real=0.08 secs]<br>12.989: [GC (Allocation Failure) 12.989: [ParNew: 306686K->34048K(306688K), 0.1086667 secs] 1128760K->931412K(2063104K), 0.1087416 secs] [Times: user=0.24 sys=0.04, real=0.11 secs]<br>13.098: [GC (CMS Initial Mark) [1 CMS-initial-mark: 897364K(1756416K)] 936667K(2063104K), 0.0041705 secs] [Times: user=0.02 sys=0.00, real=0.00 secs]<br>13.102: [CMS-concurrent-mark-start]<br>13.341: [CMS-concurrent-mark: 0.238/0.238 secs] [Times: user=0.36 sys=0.01, real=0.24 secs]<br>13.341: [CMS-concurrent-preclean-start]<br>13.350: [CMS-concurrent-preclean: 0.009/0.009 secs] [Times: user=0.03 sys=0.00, real=0.01 secs]<br>13.350: [CMS-concurrent-abortable-preclean-start]<br>13.878: [GC (Allocation Failure) 13.878: [ParNew: 306688K->34047K(306688K), 0.0960456 secs] 1204052K->1010638K(2063104K), 0.0961542 secs] [Times: user=0.29 sys=0.04, real=0.09 secs]<br>14.366: [CMS-concurrent-abortable-preclean: 0.917/1.016 secs] [Times: user=2.22 sys=0.07, real=1.01 secs]<br>14.366: [GC (CMS Final Remark) [YG occupancy: 182593 K (306688 K)]14.366: [Rescan (parallel) , 0.0291598 secs]14.395: [weak refs processing, 0.0000232 secs]14.395: [class unloading, 0.0117661 secs]14.407: [scrub symbol table, 0.0015323 secs]14.409: [scrub string table, 0.0003221 secs][1 CMS-remark: 976591K(1756416K)] 1159184K(2063104K), 0.0462010 secs] [Times: user=0.14 sys=0.00, real=0.05 secs]<br>14.412: [CMS-concurrent-sweep-start]<br>14.633: [CMS-concurrent-sweep: 0.221/0.221 secs] [Times: user=0.37 sys=0.00, real=0.22 secs]<br>14.633: [CMS-concurrent-reset-start]<br>14.636: [CMS-concurrent-reset: 0.002/0.002 secs] [Times: user=0.00 sys=0.00, real=0.00</p><hr><blockquote><p>在点头同意这个结论之前，让我们看看来自同一个 JVM 启动收集的垃圾收集日志的输出。显然- XX ： + PrintGCDetails 告诉我们一个不同且更详细的故事：</p></blockquote><p>基于这些信息，我们可以看到12次 Minor GC 后开始有些和上面不一样了。没有运行两次 Full GC，这不同的地方在于单个 GC 在永久代中不同阶段运行了两次：</p><ul><li>最初的标记阶段，用了0.0041705秒也就是4ms左右。这个阶段会暂停“全世界（ stop-the-world）”的事件，停止所有应用程序的线程，然后开始标记。</li><li>并行执行标记和清洗阶段。这些都是和应用程序线程并行的。</li><li>最后 Remark 阶段，花费了0.0462010秒约46ms。这个阶段会再次暂停所有的事件。</li><li>并行执行清理操作。正如其名，此阶段也是并行的，不会停止其他线程。</li></ul><p>所以，正如我们从垃圾回收日志中所看到的那样，实际上只是执行了 Major GC 去清理老年代空间而已，而不是执行了两次 Full GC。</p><p>如果你是后期做决 定的话，那么由 jstat 提供的数据会引导你做出正确的决策。它正确列出的两个暂停所有事件的情况，导致所有线程停止了共计50ms。但是如果你试图优化吞吐量，你会被误导的。清 单只列出了回收初始标记和最终 Remark 阶段，jstat的输出看不到那些并发完成的工作。</p><h2 id="结论"><a href="#结论" class="headerlink" title="结论"></a>结论</h2><p>考虑到这种情况，最好避免以 Minor、Major、Full GC 这种方式来思考问题。而应该监控应用延迟或者吞吐量，然后将 GC 事件和结果联系起来。</p><p>随着这些 GC 事件的发生，你需要额外的关注某些信息，GC 事件是强制所有应用程序线程停止了还是并行的处理了部分事件。</p><blockquote><p>个人博客 欢迎来访： <a href="http://zj2626.github.io">http://zj2626.github.io</a></p></blockquote>]]></content>
<summary type="html">
<blockquote>
<p>转载链接地址: <a href="http://www.importnew.com/15820.html" target="_blank" rel="noopener">http://www.importnew.com/15820.html</a></p>
</blockquote>
</summary>
<category term="java虚拟机" scheme="http://zj2626.github.io/categories/java%E8%99%9A%E6%8B%9F%E6%9C%BA/"/>
<category term="深入了解java虚拟机" scheme="http://zj2626.github.io/tags/%E6%B7%B1%E5%85%A5%E4%BA%86%E8%A7%A3java%E8%99%9A%E6%8B%9F%E6%9C%BA/"/>
<category term="java" scheme="http://zj2626.github.io/tags/java/"/>
</entry>
<entry>
<title>图</title>
<link href="http://zj2626.github.io/2021/06/01/20170922_map/"/>
<id>http://zj2626.github.io/2021/06/01/20170922_map/</id>
<published>2021-06-01T14:21:10.506Z</published>
<updated>2021-06-01T01:18:34.638Z</updated>
<content type="html"><![CDATA[<blockquote><p>##图: 图是由顶点集（VertexSet）和边集（EdgeSet）组成，针对图G，顶点集和边集分别记为V(G)和E(G)。依据图的边集是否为有向，可把图分为有向图和无向图，根据图是否有权重，可以分为有权图和无权图。</p></blockquote><p><strong>线性表中的元素是“一对一”的关系，树中的元素是“一对多”的关系，而图结构中的元素则是“多对多”的关系</strong></p><pre><code>顶点(Vertex)、弧(Arc)、弧头(初始点)、弧尾(终结点)、边(Edge)、有向图(Directed graph)、无向图(Undigraph)、完全图(Completed grapg)、有向完全图、稀疏图(Sparse graph)、稠密图(Dense graph)、权(weigh)、网(network)、无向网、有向网、子图(Subgraph)、邻接点(Adjacent)、度(Degree)、入度(Indegree)、出度(Outdegree)、路径(path)、回路(环)、简单路径、简单回路（简单环）、连通、连通图(Connected graph)、连通分量(Connected Component)、强连通图、强连通分量(有向图中的极大强连通子图)、生成树、极小连通子图、有向树。</code></pre><a id="more"></a><blockquote><p>##图的存储: </p></blockquote><ol><li>邻接矩阵:用一个二维数组表示图中顶点和顶点,边的关系;形成的矩阵中可以自定义边的权值表示, 例如:0表示没有边, 其他大于0的数n表示两个顶点有边且权值为n</li></ol><p> <img src="/2021/06/01/20170922_map/map.png" title="图"></p><p> <img src="/2021/06/01/20170922_map/map2.png" title="图"></p><ol start="2"><li>邻接表</li></ol><pre><code>接矩阵与邻接表相比，它会造成空间的一定损失，它需要为每个顶点都分配n个边的空间，其实有很多边都是不存在边，但是邻接表的实现就不一样，它只关心存在的边，不关心不存在的边。</code></pre><p> <img src="/2021/06/01/20170922_map/map3.png" title="图"></p><blockquote><p>##图的遍历:</p></blockquote><p>1.广度优先</p><pre><code>1.从isTrav数组中选择一个未被访问的邻接点,标记为已访问2.依次访问Vi的所有未被访问的邻接点,并标记为已被访问3.从邻接点出发进行广度优先遍历,直到图中所有和Vi有路径相通的顶点都被访问4.重复1-3的步骤直到所有的顶点都被访问</code></pre><p>2.深度优先</p><pre><code>1.从isTrav数组中选择一个未被访问的邻接点Vi,标记为已访问2.从Vi邻接点出发进行深度优先遍历3.重复2步骤,直到所有的和Vi有路径相通的顶点都被访问过4.重复1-3操作,直到所有顶点都被访问过</code></pre><blockquote><p>##代码演示</p></blockquote><figure class="highlight c"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br><span class="line">48</span><br><span class="line">49</span><br><span class="line">50</span><br><span class="line">51</span><br><span class="line">52</span><br><span class="line">53</span><br><span class="line">54</span><br><span class="line">55</span><br><span class="line">56</span><br><span class="line">57</span><br><span class="line">58</span><br><span class="line">59</span><br><span class="line">60</span><br><span class="line">61</span><br><span class="line">62</span><br><span class="line">63</span><br><span class="line">64</span><br><span class="line">65</span><br><span class="line">66</span><br><span class="line">67</span><br><span class="line">68</span><br><span class="line">69</span><br><span class="line">70</span><br><span class="line">71</span><br><span class="line">72</span><br><span class="line">73</span><br><span class="line">74</span><br><span class="line">75</span><br><span class="line">76</span><br><span class="line">77</span><br><span class="line">78</span><br><span class="line">79</span><br><span class="line">80</span><br><span class="line">81</span><br><span class="line">82</span><br><span class="line">83</span><br><span class="line">84</span><br><span class="line">85</span><br><span class="line">86</span><br><span class="line">87</span><br><span class="line">88</span><br><span class="line">89</span><br><span class="line">90</span><br><span class="line">91</span><br><span class="line">92</span><br><span class="line">93</span><br><span class="line">94</span><br><span class="line">95</span><br><span class="line">96</span><br><span class="line">97</span><br><span class="line">98</span><br></pre></td><td class="code"><pre><span class="line"></span><br><span class="line"><span class="comment">//邻接矩阵</span></span><br><span class="line"> <span class="meta">#<span class="meta-keyword">include</span> <span class="meta-string"><stdio.h></span></span></span><br><span class="line"> <span class="meta">#<span class="meta-keyword">include</span> <span class="meta-string"><stdlib.h></span></span></span><br><span class="line"> </span><br><span class="line"> <span class="meta">#<span class="meta-keyword">define</span> VERTEX_MAX 26</span></span><br><span class="line"> <span class="meta">#<span class="meta-keyword">define</span> MAXVALUE 32767 <span class="comment">//表示当权值为MAXVALUE时,两个顶点没有相连</span></span></span><br><span class="line"> </span><br><span class="line"> <span class="keyword">typedef</span> <span class="class"><span class="keyword">struct</span></span></span><br><span class="line"><span class="class"> {</span></span><br><span class="line"> <span class="keyword">char</span> Vertex[VERTEX_MAX]; <span class="comment">//保存顶点信息（序号或字母）</span></span><br><span class="line"> <span class="keyword">int</span> Edges[VERTEX_MAX][VERTEX_MAX]; <span class="comment">//保存边的权</span></span><br><span class="line"> <span class="keyword">int</span> isTrav[VERTEX_MAX]; <span class="comment">//遍历标志</span></span><br><span class="line"> <span class="keyword">int</span> VertexNum; <span class="comment">//顶点数量</span></span><br><span class="line"> <span class="keyword">int</span> EdgeNum; <span class="comment">//边数量</span></span><br><span class="line"> <span class="keyword">int</span> GraphType; <span class="comment">//图的类型(0无向图 1有向图)</span></span><br><span class="line"> } MatrixGraph; <span class="comment">//定义邻接矩阵图结构</span></span><br><span class="line"> </span><br><span class="line"> <span class="function"><span class="keyword">void</span> <span class="title">CreateMatrixGraph</span><span class="params">(MatrixGraph *G)</span></span>; <span class="comment">//创建邻接矩阵图</span></span><br><span class="line"> <span class="function"><span class="keyword">void</span> <span class="title">OutMatrix</span><span class="params">(MatrixGraph *G)</span></span>; <span class="comment">//输出邻接矩阵</span></span><br><span class="line"> </span><br><span class="line"> <span class="function"><span class="keyword">void</span> <span class="title">CreateMatrixGraph</span><span class="params">(MatrixGraph *G)</span></span></span><br><span class="line"><span class="function"> </span>{</span><br><span class="line"> <span class="keyword">int</span> i, j, k, weight;</span><br><span class="line"> <span class="keyword">char</span> start, <span class="built_in">end</span>; <span class="comment">//边的起始顶点</span></span><br><span class="line"> </span><br><span class="line"> <span class="built_in">printf</span>(<span class="string">"输入各顶点的信息：\n"</span>); <span class="comment">//输入顶点</span></span><br><span class="line"> <span class="keyword">for</span>(i=<span class="number">0</span>; i<G->VertexNum; i++)</span><br><span class="line"> {</span><br><span class="line"> fflush(<span class="built_in">stdin</span>);</span><br><span class="line"> <span class="built_in">printf</span>(<span class="string">"第%d个顶点信息："</span>, i+<span class="number">1</span>);</span><br><span class="line"> <span class="built_in">scanf</span>(<span class="string">"%c"</span>, &(G->Vertex[i]));</span><br><span class="line"> }</span><br><span class="line"> </span><br><span class="line"> <span class="built_in">printf</span>(<span class="string">"输入构成各边的两个顶点以及权值：\n"</span>);</span><br><span class="line"> </span><br><span class="line"> <span class="keyword">for</span>(k=<span class="number">0</span>; k<G->EdgeNum; k++)</span><br><span class="line"> {</span><br><span class="line"> <span class="built_in">printf</span>(<span class="string">"第%d条边："</span>, k+<span class="number">1</span>);</span><br><span class="line"> fflush(<span class="built_in">stdin</span>);</span><br><span class="line"> <span class="built_in">scanf</span>(<span class="string">"%c %c %d"</span>, &start, &<span class="built_in">end</span>, &weight);</span><br><span class="line"> <span class="keyword">for</span>(i=<span class="number">0</span>; start!=G->Vertex[i]; i++); <span class="comment">//查找已有的顶点中是否包含当前的"起始顶点"</span></span><br><span class="line"> <span class="keyword">for</span>(j=<span class="number">0</span>; <span class="built_in">end</span>!=G->Vertex[j]; j++); <span class="comment">//同上</span></span><br><span class="line"> </span><br><span class="line"> <span class="comment">// printf("%d %d", i, j);</span></span><br><span class="line"> G->Edges[i][j] = weight; <span class="comment">//对应位置保存权值，表示有一条边</span></span><br><span class="line"> <span class="keyword">if</span>(G->GraphType == <span class="number">0</span>) <span class="comment">//判断是不是无向图</span></span><br><span class="line"> {</span><br><span class="line"> G->Edges[j][i] = weight; <span class="comment">//在对角位置保存权值</span></span><br><span class="line"> }</span><br><span class="line"> }</span><br><span class="line"> }</span><br><span class="line"> </span><br><span class="line"> <span class="function"><span class="keyword">void</span> <span class="title">OutMatrix</span><span class="params">(MatrixGraph *G)</span></span></span><br><span class="line"><span class="function"> </span>{</span><br><span class="line"> <span class="keyword">int</span> i, j;</span><br><span class="line"> </span><br><span class="line"> <span class="built_in">printf</span>(<span class="string">" "</span>);</span><br><span class="line"> <span class="keyword">for</span>(j=<span class="number">0</span>; j<G->VertexNum; j++)</span><br><span class="line"> <span class="built_in">printf</span>(<span class="string">"\t%c"</span>, G->Vertex[j]); <span class="comment">//输出顶点信息</span></span><br><span class="line"> <span class="built_in">printf</span>(<span class="string">"\n"</span>);</span><br><span class="line"> </span><br><span class="line"> <span class="keyword">for</span>(i=<span class="number">0</span>; i<G->VertexNum; i++)</span><br><span class="line"> {</span><br><span class="line"> <span class="built_in">printf</span>(<span class="string">"%c"</span>, G->Vertex[i]);</span><br><span class="line"> <span class="keyword">for</span>(j=<span class="number">0</span>; j<G->VertexNum; j++)</span><br><span class="line"> {</span><br><span class="line"> <span class="keyword">if</span>(G->Edges[i][j] >= MAXVALUE) <span class="comment">//如果权值为最大值</span></span><br><span class="line"> <span class="built_in">printf</span>(<span class="string">"\t #"</span>); <span class="comment">//输出无穷大符号</span></span><br><span class="line"> <span class="keyword">else</span></span><br><span class="line"> <span class="built_in">printf</span>(<span class="string">"\t%d"</span>, G->Edges[i][j]); <span class="comment">//输出边的权值</span></span><br><span class="line"> }</span><br><span class="line"> </span><br><span class="line"> <span class="built_in">printf</span>(<span class="string">"\n"</span>);</span><br><span class="line"> }</span><br><span class="line"> }</span><br><span class="line"> </span><br><span class="line"> <span class="function"><span class="keyword">int</span> <span class="title">main</span><span class="params">()</span></span></span><br><span class="line"><span class="function"> </span>{</span><br><span class="line"> MatrixGraph G;</span><br><span class="line"> <span class="keyword">int</span> i, j;</span><br><span class="line"> <span class="built_in">printf</span>(<span class="string">"输入生成图的类型 （0无向图 1有向图）"</span>);</span><br><span class="line"> <span class="built_in">scanf</span>(<span class="string">"%d"</span>, &G.GraphType);</span><br><span class="line"> <span class="built_in">printf</span>(<span class="string">"输入图的顶点数量和边数量"</span>);</span><br><span class="line"> <span class="built_in">scanf</span>(<span class="string">"%d %d"</span>, &G.VertexNum, &G.EdgeNum); <span class="comment">//输入图顶点数和边数</span></span><br><span class="line"> <span class="keyword">for</span>(i=<span class="number">0</span>; i<G.VertexNum; i++)</span><br><span class="line"> {</span><br><span class="line"> <span class="keyword">for</span>(j=<span class="number">0</span>; j<G.VertexNum; j++)</span><br><span class="line"> {</span><br><span class="line"> G.Edges[i][j] = MAXVALUE; <span class="comment">//初始化元素值为最大值</span></span><br><span class="line"> }</span><br><span class="line"> }</span><br><span class="line"> CreateMatrixGraph(&G);</span><br><span class="line"> </span><br><span class="line"> OutMatrix(&G);</span><br><span class="line"> </span><br><span class="line"> <span class="keyword">return</span> <span class="number">0</span>;</span><br><span class="line"> }</span><br></pre></td></tr></table></figure><hr><hr><figure class="highlight c"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br><span class="line">48</span><br><span class="line">49</span><br><span class="line">50</span><br><span class="line">51</span><br><span class="line">52</span><br><span class="line">53</span><br><span class="line">54</span><br><span class="line">55</span><br><span class="line">56</span><br><span class="line">57</span><br><span class="line">58</span><br><span class="line">59</span><br><span class="line">60</span><br><span class="line">61</span><br><span class="line">62</span><br><span class="line">63</span><br><span class="line">64</span><br><span class="line">65</span><br><span class="line">66</span><br><span class="line">67</span><br><span class="line">68</span><br><span class="line">69</span><br><span class="line">70</span><br><span class="line">71</span><br><span class="line">72</span><br><span class="line">73</span><br><span class="line">74</span><br><span class="line">75</span><br><span class="line">76</span><br><span class="line">77</span><br><span class="line">78</span><br><span class="line">79</span><br><span class="line">80</span><br><span class="line">81</span><br><span class="line">82</span><br><span class="line">83</span><br><span class="line">84</span><br><span class="line">85</span><br><span class="line">86</span><br></pre></td><td class="code"><pre><span class="line"></span><br><span class="line"><span class="comment">//邻接表</span></span><br><span class="line"> <span class="meta">#<span class="meta-keyword">include</span> <span class="meta-string"><stdio.h></span></span></span><br><span class="line"> <span class="meta">#<span class="meta-keyword">include</span> <span class="meta-string"><stdlib.h></span></span></span><br><span class="line"> </span><br><span class="line"> <span class="meta">#<span class="meta-keyword">define</span> VERTEX_MAX 26</span></span><br><span class="line"> </span><br><span class="line"> <span class="keyword">typedef</span> <span class="class"><span class="keyword">struct</span> <span class="title">edgeNode</span></span></span><br><span class="line"><span class="class"> {</span></span><br><span class="line"> <span class="keyword">int</span> Vertex; <span class="comment">//顶点信息</span></span><br><span class="line"> <span class="keyword">int</span> weight; <span class="comment">//权值</span></span><br><span class="line"> <span class="class"><span class="keyword">struct</span> <span class="title">edgeNode</span> *<span class="title">next</span>;</span> <span class="comment">//下一个顶点地址指针</span></span><br><span class="line"> }EdgeNode; <span class="comment">//边的结构</span></span><br><span class="line"> </span><br><span class="line"> <span class="keyword">typedef</span> <span class="class"><span class="keyword">struct</span></span></span><br><span class="line"><span class="class"> {</span></span><br><span class="line"> EdgeNode* AdjList[VERTEX_MAX]; <span class="comment">//顶点指针</span></span><br><span class="line"> <span class="keyword">int</span> VextexNum, EdgeNum; <span class="comment">//顶点数量,边数量</span></span><br><span class="line"> <span class="keyword">int</span> GraphType; <span class="comment">//图的类型(0无向图 1有向图)</span></span><br><span class="line"> }ListGraph; <span class="comment">//图的结构</span></span><br><span class="line"> </span><br><span class="line"> <span class="function"><span class="keyword">void</span> <span class="title">CreateGraph</span><span class="params">(ListGraph *G)</span></span>;</span><br><span class="line"> <span class="function"><span class="keyword">void</span> <span class="title">OutList</span><span class="params">(ListGraph *G)</span></span>;</span><br><span class="line"> </span><br><span class="line"> <span class="function"><span class="keyword">void</span> <span class="title">CreateGraph</span><span class="params">(ListGraph *G)</span></span></span><br><span class="line"><span class="function"> </span>{</span><br><span class="line"> <span class="keyword">int</span> i, weight;</span><br><span class="line"> <span class="keyword">int</span> start, <span class="built_in">end</span>;</span><br><span class="line"> EdgeNode *s;</span><br><span class="line"> <span class="keyword">for</span>(i=<span class="number">0</span>; i<=G->VextexNum; i++) <span class="comment">//初始化</span></span><br><span class="line"> G->AdjList[i] = <span class="literal">NULL</span>;</span><br><span class="line"> </span><br><span class="line"> <span class="built_in">printf</span>(<span class="string">"输入构成各边的两个顶点以及权值：\n"</span>);</span><br><span class="line"> </span><br><span class="line"> <span class="keyword">for</span>(i=<span class="number">0</span>; i<G->EdgeNum; i++)</span><br><span class="line"> {</span><br><span class="line"> <span class="built_in">printf</span>(<span class="string">"第%d条边 开始顶点 结束顶点 权值"</span>, i+<span class="number">1</span>);</span><br><span class="line"> fflush(<span class="built_in">stdin</span>);</span><br><span class="line"> <span class="built_in">scanf</span>(<span class="string">"%d %d %d"</span>, &start, &<span class="built_in">end</span>, &weight);</span><br><span class="line"> s = (EdgeNode *)<span class="built_in">malloc</span>(<span class="keyword">sizeof</span>(EdgeNode));</span><br><span class="line"> s->next = G->AdjList[start];</span><br><span class="line"> s->Vertex = <span class="built_in">end</span>;</span><br><span class="line"> s->weight = weight;</span><br><span class="line"> G->AdjList[start] = s;<span class="comment">//把生成的边信息赋值给图</span></span><br><span class="line"> </span><br><span class="line"> <span class="keyword">if</span>(G->GraphType == <span class="number">0</span>) <span class="comment">//判断是不是有向图</span></span><br><span class="line"> {</span><br><span class="line"> s = (EdgeNode *)<span class="built_in">malloc</span>(<span class="keyword">sizeof</span>(EdgeNode));</span><br><span class="line"> s->next = G->AdjList[<span class="built_in">end</span>];</span><br><span class="line"> s->Vertex = start;</span><br><span class="line"> s->weight = weight;</span><br><span class="line"> G->AdjList[<span class="built_in">end</span>] = s;<span class="comment">//把生成的边信息赋值给图</span></span><br><span class="line"> }</span><br><span class="line"> }</span><br><span class="line"> }</span><br><span class="line"> </span><br><span class="line"> <span class="function"><span class="keyword">void</span> <span class="title">OutList</span><span class="params">(ListGraph *G)</span></span></span><br><span class="line"><span class="function"> </span>{</span><br><span class="line"> <span class="keyword">int</span> i;</span><br><span class="line"> EdgeNode *s;</span><br><span class="line"> <span class="keyword">for</span>(i=<span class="number">0</span>; i<=G->VextexNum; i++)</span><br><span class="line"> {</span><br><span class="line"> <span class="built_in">printf</span>(<span class="string">"顶点%d"</span>, i);</span><br><span class="line"> s = G->AdjList[i];</span><br><span class="line"> <span class="keyword">while</span>(s)</span><br><span class="line"> {</span><br><span class="line"> <span class="built_in">printf</span>(<span class="string">"->%d(%d)"</span>, s->Vertex, s->weight);</span><br><span class="line"> s = s->next;</span><br><span class="line"> }</span><br><span class="line"> <span class="built_in">printf</span>(<span class="string">"\n"</span>);</span><br><span class="line"> }</span><br><span class="line"> }</span><br><span class="line"> </span><br><span class="line"> <span class="function"><span class="keyword">int</span> <span class="title">main</span><span class="params">()</span></span></span><br><span class="line"><span class="function"> </span>{</span><br><span class="line"> ListGraph G;</span><br><span class="line"> <span class="built_in">printf</span>(<span class="string">"输入生成图的类型 （0无向图 1有向图）"</span>);</span><br><span class="line"> <span class="built_in">scanf</span>(<span class="string">"%d"</span>, &G.GraphType);</span><br><span class="line"> <span class="built_in">printf</span>(<span class="string">"输入图的顶点数量和边数量"</span>);</span><br><span class="line"> <span class="built_in">scanf</span>(<span class="string">"%d %d"</span>, &G.VextexNum, &G.EdgeNum);</span><br><span class="line"> </span><br><span class="line"> CreateGraph(&G);</span><br><span class="line"> OutList(&G);</span><br><span class="line"> </span><br><span class="line"> <span class="keyword">return</span> <span class="number">0</span>;</span><br><span class="line"> }</span><br></pre></td></tr></table></figure><hr><hr><figure class="highlight c"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br><span class="line">48</span><br><span class="line">49</span><br><span class="line">50</span><br><span class="line">51</span><br><span class="line">52</span><br><span class="line">53</span><br><span class="line">54</span><br><span class="line">55</span><br><span class="line">56</span><br><span class="line">57</span><br><span class="line">58</span><br><span class="line">59</span><br><span class="line">60</span><br><span class="line">61</span><br><span class="line">62</span><br><span class="line">63</span><br><span class="line">64</span><br><span class="line">65</span><br><span class="line">66</span><br><span class="line">67</span><br><span class="line">68</span><br><span class="line">69</span><br><span class="line">70</span><br><span class="line">71</span><br><span class="line">72</span><br><span class="line">73</span><br><span class="line">74</span><br><span class="line">75</span><br><span class="line">76</span><br><span class="line">77</span><br><span class="line">78</span><br><span class="line">79</span><br><span class="line">80</span><br><span class="line">81</span><br><span class="line">82</span><br><span class="line">83</span><br><span class="line">84</span><br><span class="line">85</span><br><span class="line">86</span><br><span class="line">87</span><br><span class="line">88</span><br><span class="line">89</span><br><span class="line">90</span><br><span class="line">91</span><br><span class="line">92</span><br><span class="line">93</span><br><span class="line">94</span><br><span class="line">95</span><br><span class="line">96</span><br><span class="line">97</span><br><span class="line">98</span><br><span class="line">99</span><br><span class="line">100</span><br><span class="line">101</span><br><span class="line">102</span><br><span class="line">103</span><br><span class="line">104</span><br><span class="line">105</span><br><span class="line">106</span><br><span class="line">107</span><br><span class="line">108</span><br><span class="line">109</span><br><span class="line">110</span><br><span class="line">111</span><br><span class="line">112</span><br><span class="line">113</span><br><span class="line">114</span><br><span class="line">115</span><br><span class="line">116</span><br><span class="line">117</span><br><span class="line">118</span><br><span class="line">119</span><br><span class="line">120</span><br><span class="line">121</span><br><span class="line">122</span><br><span class="line">123</span><br><span class="line">124</span><br><span class="line">125</span><br><span class="line">126</span><br><span class="line">127</span><br><span class="line">128</span><br><span class="line">129</span><br><span class="line">130</span><br><span class="line">131</span><br><span class="line">132</span><br><span class="line">133</span><br><span class="line">134</span><br><span class="line">135</span><br><span class="line">136</span><br><span class="line">137</span><br><span class="line">138</span><br><span class="line">139</span><br><span class="line">140</span><br><span class="line">141</span><br><span class="line">142</span><br><span class="line">143</span><br><span class="line">144</span><br><span class="line">145</span><br><span class="line">146</span><br><span class="line">147</span><br><span class="line">148</span><br><span class="line">149</span><br><span class="line">150</span><br><span class="line">151</span><br><span class="line">152</span><br><span class="line">153</span><br><span class="line">154</span><br><span class="line">155</span><br><span class="line">156</span><br><span class="line">157</span><br><span class="line">158</span><br><span class="line">159</span><br><span class="line">160</span><br><span class="line">161</span><br><span class="line">162</span><br><span class="line">163</span><br><span class="line">164</span><br><span class="line">165</span><br><span class="line">166</span><br><span class="line">167</span><br><span class="line">168</span><br><span class="line">169</span><br><span class="line">170</span><br><span class="line">171</span><br><span class="line">172</span><br><span class="line">173</span><br><span class="line">174</span><br></pre></td><td class="code"><pre><span class="line"></span><br><span class="line"><span class="comment">//深度优先和广度优先</span></span><br><span class="line"> <span class="meta">#<span class="meta-keyword">include</span> <span class="meta-string"><stdio.h></span></span></span><br><span class="line"> <span class="meta">#<span class="meta-keyword">include</span> <span class="meta-string"><stdlib.h></span></span></span><br><span class="line"> <span class="meta">#<span class="meta-keyword">include</span> <span class="meta-string">"AdjMatrixGraph.h"</span></span></span><br><span class="line"> </span><br><span class="line"> <span class="meta">#<span class="meta-keyword">define</span> QUEUE_MAXSIZE 30 <span class="comment">//队列的最大容量</span></span></span><br><span class="line"> </span><br><span class="line"> <span class="keyword">typedef</span> <span class="class"><span class="keyword">struct</span></span></span><br><span class="line"><span class="class"> {</span></span><br><span class="line"> <span class="keyword">int</span> Data[QUEUE_MAXSIZE];<span class="comment">//数据域</span></span><br><span class="line"> <span class="keyword">int</span> head; <span class="comment">//队头指针</span></span><br><span class="line"> <span class="keyword">int</span> tail; <span class="comment">//队尾指针</span></span><br><span class="line"> }SeqQueue; <span class="comment">//队列结构</span></span><br><span class="line"> </span><br><span class="line"> <span class="comment">//队列操作函数</span></span><br><span class="line"> <span class="function"><span class="keyword">void</span> <span class="title">QueueInit</span><span class="params">(SeqQueue *q)</span></span>; <span class="comment">//初始化一个队列</span></span><br><span class="line"> <span class="function"><span class="keyword">int</span> <span class="title">QueueIsEmpty</span><span class="params">(SeqQueue q)</span></span>; <span class="comment">//判断队列是否为空</span></span><br><span class="line"> <span class="function"><span class="keyword">int</span> <span class="title">QueueIn</span><span class="params">(SeqQueue *q, <span class="keyword">int</span> n)</span></span>; <span class="comment">//入队</span></span><br><span class="line"> <span class="function"><span class="keyword">int</span> <span class="title">QueueOut</span><span class="params">(SeqQueue *q, <span class="keyword">int</span> *ch)</span></span>; <span class="comment">//出队</span></span><br><span class="line"> </span><br><span class="line"> <span class="comment">//图操作函数</span></span><br><span class="line"> <span class="function"><span class="keyword">void</span> <span class="title">DFSTraverse</span><span class="params">(MatrixGraph *G)</span></span>; <span class="comment">//深度优先遍历</span></span><br><span class="line"> <span class="function"><span class="keyword">void</span> <span class="title">BFSTraverse</span><span class="params">(MatrixGraph *G)</span></span>; <span class="comment">//广度优先遍历</span></span><br><span class="line"> <span class="function"><span class="keyword">void</span> <span class="title">DFSM</span><span class="params">(MatrixGraph *G, <span class="keyword">int</span> i)</span></span>;</span><br><span class="line"> <span class="function"><span class="keyword">void</span> <span class="title">BFSM</span><span class="params">(MatrixGraph *G, <span class="keyword">int</span> i)</span></span>;</span><br><span class="line"> </span><br><span class="line"> <span class="function"><span class="keyword">void</span> <span class="title">QueueInit</span><span class="params">(SeqQueue *q)</span></span>{</span><br><span class="line"> q->head = q->tail = <span class="number">0</span>;</span><br><span class="line"> }</span><br><span class="line"> </span><br><span class="line"> <span class="function"><span class="keyword">int</span> <span class="title">QueueIsEmpty</span><span class="params">(SeqQueue q)</span></span></span><br><span class="line"><span class="function"> </span>{</span><br><span class="line"> <span class="keyword">return</span> q.head = q.tail;</span><br><span class="line"> }</span><br><span class="line"> </span><br><span class="line"> <span class="function"><span class="keyword">int</span> <span class="title">QueueIn</span><span class="params">(SeqQueue *q, <span class="keyword">int</span> n)</span></span></span><br><span class="line"><span class="function"> </span>{</span><br><span class="line"> <span class="keyword">if</span>((q->tail + <span class="number">1</span>) % q->head != <span class="number">0</span>)</span><br><span class="line"> {</span><br><span class="line"> q->Data[q->tail] = n;</span><br><span class="line"> q->tail = (q->tail + <span class="number">1</span>) % QUEUE_MAXSIZE;</span><br><span class="line"> <span class="keyword">return</span> <span class="number">1</span>;</span><br><span class="line"> }</span><br><span class="line"> <span class="keyword">else</span></span><br><span class="line"> {</span><br><span class="line"> <span class="keyword">return</span> <span class="number">0</span>;</span><br><span class="line"> }</span><br><span class="line"> }</span><br><span class="line"> </span><br><span class="line"> <span class="function"><span class="keyword">int</span> <span class="title">QueueOut</span><span class="params">(SeqQueue *q, <span class="keyword">int</span> *ch)</span></span></span><br><span class="line"><span class="function"> </span>{</span><br><span class="line"> <span class="keyword">if</span>(q->head != q->tail)</span><br><span class="line"> {</span><br><span class="line"> *ch = q->Data[q->head];</span><br><span class="line"> q->head = (q->head + <span class="number">1</span>) % QUEUE_MAXSIZE;</span><br><span class="line"> <span class="keyword">return</span> <span class="number">1</span>;</span><br><span class="line"> }</span><br><span class="line"> <span class="keyword">else</span></span><br><span class="line"> {</span><br><span class="line"> <span class="keyword">return</span> <span class="number">0</span>;</span><br><span class="line"> }</span><br><span class="line"> }</span><br><span class="line"> </span><br><span class="line"> <span class="function"><span class="keyword">void</span> <span class="title">DFSTraverse</span><span class="params">(MatrixGraph *G)</span></span></span><br><span class="line"><span class="function"> </span>{</span><br><span class="line"> <span class="keyword">int</span> i;</span><br><span class="line"> <span class="keyword">for</span>(i=<span class="number">0</span>; i<G->VertexNum; i++)</span><br><span class="line"> {</span><br><span class="line"> G->isTrav[i] = <span class="number">0</span>;</span><br><span class="line"> }</span><br><span class="line"> </span><br><span class="line"> <span class="built_in">printf</span>(<span class="string">"深度优先遍历"</span>);</span><br><span class="line"> <span class="keyword">for</span>(i=<span class="number">0</span>; i<G->VertexNum; i++)</span><br><span class="line"> {</span><br><span class="line"> <span class="keyword">if</span>(!G->isTrav[i])</span><br><span class="line"> {</span><br><span class="line"> DFSM(G, i);</span><br><span class="line"> }</span><br><span class="line"> }</span><br><span class="line"> }</span><br><span class="line"> </span><br><span class="line"> <span class="function"><span class="keyword">void</span> <span class="title">BFSTraverse</span><span class="params">(MatrixGraph *G)</span></span></span><br><span class="line"><span class="function"> </span>{</span><br><span class="line"> <span class="keyword">int</span> i;</span><br><span class="line"> <span class="keyword">for</span>(i=<span class="number">0</span>; i<G->VertexNum; i++)</span><br><span class="line"> {</span><br><span class="line"> G->isTrav[i] = <span class="number">0</span>;</span><br><span class="line"> }</span><br><span class="line"> </span><br><span class="line"> <span class="built_in">printf</span>(<span class="string">"广度优先遍历"</span>);</span><br><span class="line"> <span class="keyword">for</span>(i=<span class="number">0</span>; i<G->VertexNum; i++)</span><br><span class="line"> {</span><br><span class="line"> <span class="keyword">if</span>(!G->isTrav[i])</span><br><span class="line"> {</span><br><span class="line"> BFSM(G, i);</span><br><span class="line"> }</span><br><span class="line"> }</span><br><span class="line"> }</span><br><span class="line"> </span><br><span class="line"> <span class="function"><span class="keyword">void</span> <span class="title">DFSM</span><span class="params">(MatrixGraph *G, <span class="keyword">int</span> i)</span></span></span><br><span class="line"><span class="function"> </span>{</span><br><span class="line"> <span class="keyword">int</span> j;</span><br><span class="line"> G->isTrav[i] = i;</span><br><span class="line"> <span class="built_in">printf</span>(<span class="string">"->%c"</span>, G->Vertex[i]);</span><br><span class="line"> </span><br><span class="line"> <span class="keyword">for</span>(j=<span class="number">0</span>; j<G->VertexNum; j++)</span><br><span class="line"> {</span><br><span class="line"> <span class="keyword">if</span>(G->Edges[i][j] != MAXVALUE && !G->isTrav[i])</span><br><span class="line"> {</span><br><span class="line"> DFSM(G, j);</span><br><span class="line"> }</span><br><span class="line"> }</span><br><span class="line"> }</span><br><span class="line"> </span><br><span class="line"> <span class="function"><span class="keyword">void</span> <span class="title">BFSM</span><span class="params">(MatrixGraph *G, <span class="keyword">int</span> k)</span></span></span><br><span class="line"><span class="function"> </span>{</span><br><span class="line"> <span class="keyword">int</span> i, j;</span><br><span class="line"> SeqQueue Q;</span><br><span class="line"> QueueInit(&Q);</span><br><span class="line"> </span><br><span class="line"> G->isTrav[k] = <span class="number">1</span>;</span><br><span class="line"> <span class="built_in">printf</span>(<span class="string">"->%c"</span>, G->Vertex[k]);</span><br><span class="line"> </span><br><span class="line"> QueueIn(&Q, k);</span><br><span class="line"> <span class="keyword">while</span>(!QueueIsEmpty(Q))</span><br><span class="line"> {</span><br><span class="line"> QueueOut(&Q, &i);</span><br><span class="line"> <span class="keyword">for</span>(j=<span class="number">0</span>; j<G->VertexNum; j++)</span><br><span class="line"> {</span><br><span class="line"> <span class="keyword">if</span>(G->Edges[i][j] != MAXVALUE && !G->isTrav[j])</span><br><span class="line"> {</span><br><span class="line"> <span class="built_in">printf</span>(<span class="string">"->%c"</span>, G->Vertex[j]);</span><br><span class="line"> G->isTrav[j] = <span class="number">1</span>;</span><br><span class="line"> QueueIn(&Q, j);</span><br><span class="line"> }</span><br><span class="line"> }</span><br><span class="line"> }</span><br><span class="line"> }</span><br><span class="line"> </span><br><span class="line"> <span class="function"><span class="keyword">int</span> <span class="title">main</span><span class="params">()</span></span></span><br><span class="line"><span class="function"> </span>{</span><br><span class="line"> MatrixGraph G;</span><br><span class="line"> <span class="keyword">int</span> i, j;</span><br><span class="line"> <span class="keyword">char</span> select;</span><br><span class="line"> <span class="keyword">do</span></span><br><span class="line"> {</span><br><span class="line"> <span class="built_in">printf</span>(<span class="string">"输入生成图的类型: (0:无向图, 1:有向图)"</span>);</span><br><span class="line"> fflush(<span class="built_in">stdin</span>);</span><br><span class="line"> <span class="built_in">scanf</span>(<span class="string">"%d"</span>, &G.GraphType);</span><br><span class="line"> <span class="built_in">printf</span>(<span class="string">"输入图的顶点数量和边数量"</span>);</span><br><span class="line"> fflush(<span class="built_in">stdin</span>);</span><br><span class="line"> <span class="built_in">scanf</span>(<span class="string">"%d %d"</span>, &G.VertexNum, &G.EdgeNum);</span><br><span class="line"> <span class="keyword">for</span>(i=<span class="number">0</span>; i<G.VertexNum; i++)</span><br><span class="line"> {</span><br><span class="line"> <span class="keyword">for</span>(j=<span class="number">0</span>; j<G.VertexNum; j++)</span><br><span class="line"> {</span><br><span class="line"> G.Edges[i][j] = MAXVALUE;</span><br><span class="line"> }</span><br><span class="line"> }</span><br><span class="line"> </span><br><span class="line"> CreateMatrixGraph(&G);</span><br><span class="line"> </span><br><span class="line"> <span class="built_in">printf</span>(<span class="string">"邻接矩阵数据:\n"</span>);</span><br><span class="line"> OutMatrix(&G);</span><br><span class="line"> DFSTraverse(&G);</span><br><span class="line"> BFSTraverse(&G);</span><br><span class="line"> </span><br><span class="line"> <span class="built_in">printf</span>(<span class="string">"两种遍历结束"</span>);</span><br><span class="line"> fflush(<span class="built_in">stdin</span>);</span><br><span class="line"> <span class="built_in">scanf</span>(<span class="string">"%c"</span>, &select);</span><br><span class="line"> }<span class="keyword">while</span>(select!= <span class="string">'N'</span> && select != <span class="string">'\n'</span>);</span><br><span class="line"> <span class="keyword">return</span> <span class="number">0</span>;</span><br><span class="line"> }</span><br></pre></td></tr></table></figure><blockquote><p>个人博客 欢迎来访： <a href="http://zj2626.github.io">http://zj2626.github.io</a></p></blockquote>]]></content>
<summary type="html">
<blockquote>
<p>##图: 图是由顶点集（VertexSet）和边集（EdgeSet）组成，针对图G，顶点集和边集分别记为V(G)和E(G)。依据图的边集是否为有向，可把图分为有向图和无向图，根据图是否有权重，可以分为有权图和无权图。</p>
</blockquote>
<p><strong>线性表中的元素是“一对一”的关系，树中的元素是“一对多”的关系，而图结构中的元素则是“多对多”的关系</strong></p>
<pre><code>顶点(Vertex)、弧(Arc)、弧头(初始点)、弧尾(终结点)、边(Edge)、有向图(Directed graph)、
无向图(Undigraph)、完全图(Completed grapg)、有向完全图、稀疏图(Sparse graph)、
稠密图(Dense graph)、权(weigh)、网(network)、无向网、有向网、子图(Subgraph)、
邻接点(Adjacent)、度(Degree)、入度(Indegree)、出度(Outdegree)、路径(path)、
回路(环)、简单路径、简单回路（简单环）、连通、连通图(Connected graph)、连通分量(Connected Component)、
强连通图、强连通分量(有向图中的极大强连通子图)、生成树、极小连通子图、有向树。
</code></pre>
</summary>
<category term="数据结构和算法" scheme="http://zj2626.github.io/categories/%E6%95%B0%E6%8D%AE%E7%BB%93%E6%9E%84%E5%92%8C%E7%AE%97%E6%B3%95/"/>
<category term="C语言" scheme="http://zj2626.github.io/tags/C%E8%AF%AD%E8%A8%80/"/>
</entry>
<entry>
<title>网络协议记录</title>
<link href="http://zj2626.github.io/2021/06/01/20170918_protocol/"/>
<id>http://zj2626.github.io/2021/06/01/20170918_protocol/</id>
<published>2021-06-01T14:21:10.501Z</published>
<updated>2021-06-01T01:18:34.631Z</updated>
<content type="html"><![CDATA[<p> <img src="/2021/06/01/20170918_protocol/11185081040.jpg" title="ISO七层模型与对应协议"></p><p> <img src="/2021/06/01/20170918_protocol/11187081041.jpg" title="ISO七层模型与对应协议"></p><a id="more"></a> <blockquote><p>ISO: (International Organization for Standards) 国际标准化组织</p></blockquote><blockquote><p>OSI(参考模型): (Open Systems Interconnection) 开放式通信系统互联网参考模型：把通信功能划分为7个分层</p></blockquote><blockquote><p>OSI协议: OSI协议以OSI参考模型为基础界定了每个阶层的协议和每个阶层之间接口相关的标准；遵循OSI协议的的产品为OSI产品；</p></blockquote><blockquote><p>MAC地址: (Media Access Control)介质访问控制；也叫物理地址、硬件地址；</p></blockquote><blockquote><p>Unicast: 单播；Broadcast: 广播；Multicast: 多播；Anycast: 任播</p></blockquote><blockquote><p>网卡: NIC(Network Interface Card)，使计算机联网的设备</p></blockquote><blockquote><p>中继器: (Repeater) ，物理层上延长网络的设备(处于第一层)</p></blockquote><pre><code>由电缆传过来的电信号或光信号经由中继器的波形调整和放大再传给另一个电缆</code></pre><blockquote><p>网桥: (Bridge) 也叫2层交换机，从数据链路层上延长网络的设备(处于第二层)，根据MAC地址进行处理</p></blockquote><pre><code>能识别数据帧，并把数据帧临时存储于内存，并重新生成信号作为新的帧转发给相连的另一个网段地址自学机制 地址过滤功能</code></pre><blockquote><p>路由器: (Router) 也叫3层交换机，通过网络层转发分组数据的设备(处于第三层)，根据IP地址处理</p></blockquote><blockquote><p>4~7层交换机：处理传输层以上各层网络传输的设备</p></blockquote><blockquote><p>网关: (Gateway) 转换协议的设备，协议转换，数据转发(从传输层到应用层的数据转换、转发)</p></blockquote><blockquote><p>FCS: (Frame Check Sequence) 帧检验序列，(网桥)校验数据是否正确的送达目的地</p></blockquote><blockquote><p>CRC: (Cyclic Redundancy Check) 循环冗余校验码，CRC循环冗余码校验FCS帧检验序列</p></blockquote><blockquote><p>ATM: (Asynchronous Transfer Mode) 异步传输</p></blockquote><blockquote><p>网络协议族: (Internet Protocol Suite) 组成网际协议的一组协议</p></blockquote><blockquote><p>IETF: (The Internet Engineering Task Force) 国际互联网工程任务组，一个公开性质的大型民间国际团体</p></blockquote><blockquote><p>RFC: (Request For Comment) 记录TCP/IP协议及其实现和运用信息</p></blockquote><blockquote><p>互联网: (The Internet) </p></blockquote><blockquote><p>NOC: (Network Operation Center) 网络操作中心，互联网中每个网络由NOC相连 </p></blockquote><blockquote><p>IX: (Internet Exchange) 网络交换中心，连接异构网络(不同运营商导致)需要IX支持</p></blockquote><blockquote><p>ISP: (Internet Service Provider) 互联网服务提供商</p></blockquote><blockquote><p>IP: (Internet Protocol)跨越网络传送数据包(分组交换)</p></blockquote><blockquote><p>ICMP: (Internet Control Message Protocol) Internet控制报文协议，</p></blockquote><pre><code>当IP数据包发送途中发生异常无法达到目的地址，需要给发送端发送一个异常的通知(可以诊断网络健康状况)</code></pre><blockquote><p>ARP: (Address Resolution Protocol)从分组数据包的IP地址中解析出物理地址(MAC地址)的一种协议</p></blockquote><blockquote><p>TCP: (Transmission Control Protocol) 传输控制协议，面向有连接，传输层协议，</p></blockquote><blockquote><p>UDP: (User Datagram Protocol) 用户数据报协议，面向无连接，传输层协议，</p></blockquote><blockquote><p>WWW: (World Wide Web) </p></blockquote><blockquote><p>HTTP: (HyperText Transfer Protocol) 超文本传输协议 </p></blockquote><blockquote><p>SMTP: (Simple Mail Transfer Protocol) </p></blockquote><blockquote><p>MIME: () (表示层协议)</p></blockquote><blockquote><p>FTP: (File Transfer Protocol) 文件传输协议</p></blockquote><blockquote><p>TELNET: 网络远程登陆协议 </p></blockquote><blockquote><p>SSH: (Secure Shell) </p></blockquote><blockquote><p>SNMP: (Simple Network Management Protocol) 简单网络管理协议 (应用层协议)</p></blockquote><pre><code>在TCP/IP中进行网络管理， 采用SNMP管理的主机、网桥、路由器称为SNMP代理(Agent)，进行管理的那一段叫做管理器(Manager)Agent和Manager用到该协议在SNMP代理端，保存网络接口的信息、通信数据量、异常数据量等信息可以及时检查网络拥堵情况</code></pre><blockquote><p>MIB: () (表示层协议)</p></blockquote><blockquote><p>CSMA: (Carrier Sense Multiple Access) 载波侦听多路访问，争夺获取数据传输的权力</p></blockquote><blockquote><p>CSMA/CD: 相对于CSMA，要求每个站提前检测冲突，发生冲突，则尽早释放信道</p></blockquote><blockquote><p>FDDI: (Fiber Distributed Data Interface) 光纤分布式数据接口</p></blockquote><blockquote><p>BPDU: (Bridge Protocol Data Unit) 生成树方式中，每个网桥必须在每1-10秒内小胡交换BPDU包，用来判断哪些端口使用哪些不使用，以便消除环路</p></blockquote><blockquote><p>RSTP: (Rapid Spanning Tree Protocol) 能将发生环路时的恢复时间缩短到几秒之内 </p></blockquote><blockquote><p>无线PAN: (Personal Area Network) 通信距离10米左右，应用：蓝牙</p></blockquote><blockquote><p>无线LAN: (Local Area Network) 通信距离100米左右，应用：Wi-Fi</p></blockquote><blockquote><p>无线MAN: (Metropolitan Area Network) 通信距离1km-100km，应用：WiMAX</p></blockquote><blockquote><p>无线RAN: (Regional Area Network) 通信距离200k-700km</p></blockquote><blockquote><p>无线WAN: (Wide Area Network) 应用3G,LTE,4G等</p></blockquote><blockquote><p>PPP: (Point to Point Protocol) 点对点协议，相当于2层的数据链路层</p></blockquote><pre><code>ppp协议的主要功能中主要包括两个协议： 1.LCP：(Link Control Protocol)，不依赖上层， 通过两次握手进行用户名密码验证，明文传输密码，不安全 主要负责建立和断开连接，设置最大接受单元（MRU，Maximum Receive Unit）， 设置验证协议[PAP(Password Authentication Protocol) 或 CHAP(Challenge Handshake Authentication Protocol)]， 设置进行通信质量监控与否 2.NCP: (Network Control Protocol)， 依赖上层，如果上层是IP，则此时NCP也叫IPCP 使用一次性密码OTP(One Time Password)， 安全，防止窃听 IPCP主要IP地址设置以及是否进行TCP/IP首部压缩等设置 ppp协议连接时，需要进行用户名密码验证</code></pre><blockquote><p>PPPoE: (PPP over Ethernet) 互联网接入服务商在以太网上提供PPP功能，可以提供计费功能</p></blockquote><blockquote><p>ADSL: (Asymmetric Digital Subscriber Line) 非对称数字用户环路</p></blockquote><blockquote><p>FTTH: (Fiber To The Home) 光纤到户</p></blockquote><blockquote><p>VPN: (Virtual Private Network) 虚拟专用网络</p></blockquote><blockquote><p>MTU: (Maximum Transmission Unit) 最大传输单元</p></blockquote><blockquote><p>CIDR: (Classless Inter-Domain Routing) 无类型域间选路</p></blockquote><blockquote><p>CIDR: (Border Gateway Protocol) 边界网关协议</p></blockquote><blockquote><p>VLSM: (Variable Length Subnet Mask) 可变长子网掩码</p></blockquote><blockquote><p>ICANN: (Internet Corporation for Assigned Names and Numbers) 互联网名称与数字地址分配机构</p></blockquote><blockquote><p>NAT: (Network Address Translation) 网络地址转换</p></blockquote><hr><hr><blockquote><p>个人博客 欢迎来访： <a href="http://zj2626.github.io">http://zj2626.github.io</a></p></blockquote>]]></content>
<summary type="html">
<p> <img src="/2021/06/01/20170918_protocol/11185081040.jpg" title="ISO七层模型与对应协议"></p>
<p> <img src="/2021/06/01/20170918_protocol/11187081041.jpg" title="ISO七层模型与对应协议"></p>
</summary>
<category term="计算机网络" scheme="http://zj2626.github.io/categories/%E8%AE%A1%E7%AE%97%E6%9C%BA%E7%BD%91%E7%BB%9C/"/>
<category term="定义" scheme="http://zj2626.github.io/tags/%E5%AE%9A%E4%B9%89/"/>
</entry>
<entry>
<title>待完成</title>
<link href="http://zj2626.github.io/2021/06/01/20170911_syc/"/>
<id>http://zj2626.github.io/2021/06/01/20170911_syc/</id>
<published>2021-06-01T14:21:10.500Z</published>
<updated>2021-06-01T01:18:34.630Z</updated>
<content type="html"><![CDATA[<p><a href="http://www.cnblogs.com/XHJT/p/3897440.html" target="_blank" rel="noopener">http://www.cnblogs.com/XHJT/p/3897440.html</a></p><p><a href="http://blog.csdn.net/suifeng3051/article/details/52611233" target="_blank" rel="noopener">http://blog.csdn.net/suifeng3051/article/details/52611233</a></p><a id="more"></a> <blockquote><p>个人博客 欢迎来访： <a href="http://zj2626.github.io">http://zj2626.github.io</a></p></blockquote>]]></content>
<summary type="html">
<p><a href="http://www.cnblogs.com/XHJT/p/3897440.html" target="_blank" rel="noopener">http://www.cnblogs.com/XHJT/p/3897440.html</a></p>
<p><a href="http://blog.csdn.net/suifeng3051/article/details/52611233" target="_blank" rel="noopener">http://blog.csdn.net/suifeng3051/article/details/52611233</a></p>
</summary>
<category term="数据结构和算法" scheme="http://zj2626.github.io/categories/%E6%95%B0%E6%8D%AE%E7%BB%93%E6%9E%84%E5%92%8C%E7%AE%97%E6%B3%95/"/>
<category term="C语言" scheme="http://zj2626.github.io/tags/C%E8%AF%AD%E8%A8%80/"/>
</entry>
<entry>
<title>待完成</title>
<link href="http://zj2626.github.io/2021/06/01/20170911_list_clone/"/>
<id>http://zj2626.github.io/2021/06/01/20170911_list_clone/</id>
<published>2021-06-01T14:21:10.498Z</published>
<updated>2021-06-01T01:18:34.629Z</updated>
<content type="html"><![CDATA[<p><a href="http://www.cnblogs.com/lyajs/p/5779021.html" target="_blank" rel="noopener">http://www.cnblogs.com/lyajs/p/5779021.html</a></p><a id="more"></a> <pre><code>package collection;import org.junit.Test;import java.util.ArrayList;import java.util.ArrayList;/** * Created by zj on 2017/9/11. */public class Link { @Test public void test() { ArrayList<Student> list = new ArrayList<>(); //添加两个元素 Student stJack = new Student("Jack", "AAAA"); Student stTom = new Student("Tom", "BBBBB"); list.add(stJack); list.add(stTom); //克隆 ArrayList<Student> listCopy = new ArrayList<>(); /*list.forEach(li -> { listCopy.add(li); });*/ listCopy = (ArrayList<Student>) list.clone(); listCopy.get(1).setId("FFFFFFFFFFF"); System.out.println(list); System.out.println(list.get(1).hashCode()); System.out.println(listCopy); System.out.println(listCopy.get(1).hashCode()); /*ArrayList<String> ArrayList = new ArrayList<>(); ArrayList.add("A"); ArrayList.add("B"); ArrayList.add("C"); ArrayList.add("D"); ArrayList.add("E"); System.out.println(ArrayList.clone()); ArrayList.forEach(li -> { System.out.println(li); });*/ }}package collection;import java.util.HashSet;import java.util.Set;/** * Created by zj on 2017/9/10. */public class Student { private String id; private String name; private Set courses; public Student() { } public Student(String id, String name) { this.id = id; this.name = name; this.courses = new HashSet(); } public Student(String id, String name, Set courses) { this.id = id; this.name = name; this.courses = courses; } @Override public String toString() { return "Student{" + "id='" + id + '\'' + ", name='" + name + '\'' + ", courses=" + courses + '}'; } @Override protected Object clone() throws CloneNotSupportedException { return new Student(this.id, this.name); } public String getId() { return id; } public void setId(String id) { this.id = id; } public String getName() { return name; } public void setName(String name) { this.name = name; } public Set getCourses() { return courses; } public void setCourses(Set courses) { this.courses = courses; }}public static void main(String[] args) { Map<String, Student> map = new HashMap<String, Student>(); map.put("a", new Student("A", "FFF")); map.put("b", new Student("B", "AAA")); map.put("d", new Student("D", "KKK")); map.put("c", new Student("C", "KKK")); List<Map.Entry<String, Student>> list = new ArrayList<>(map.entrySet()); Collections.sort(list, new Comparator<Map.Entry<String, Student>>() { @Override public int compare(Map.Entry<String, Student> o1, Map.Entry<String, Student> o2) { return o1.getValue().compareTo(o2.getValue()); } }); for (Map.Entry<String, Student> mapping : list) { System.out.println(mapping.getKey() + ":" + mapping.getValue()); } }</code></pre><blockquote><p>个人博客 欢迎来访： <a href="http://zj2626.github.io">http://zj2626.github.io</a></p></blockquote>]]></content>
<summary type="html">
<p><a href="http://www.cnblogs.com/lyajs/p/5779021.html" target="_blank" rel="noopener">http://www.cnblogs.com/lyajs/p/5779021.html</a></p>
</summary>
<category term="数据结构和算法" scheme="http://zj2626.github.io/categories/%E6%95%B0%E6%8D%AE%E7%BB%93%E6%9E%84%E5%92%8C%E7%AE%97%E6%B3%95/"/>
<category term="C语言" scheme="http://zj2626.github.io/tags/C%E8%AF%AD%E8%A8%80/"/>
</entry>
<entry>
<title>哈夫曼(霍夫曼)树以及哈夫曼编码</title>
<link href="http://zj2626.github.io/2021/06/01/20170908_hfms/"/>
<id>http://zj2626.github.io/2021/06/01/20170908_hfms/</id>
<published>2021-06-01T14:21:10.487Z</published>
<updated>2021-06-01T01:18:34.619Z</updated>
<content type="html"><![CDATA[<blockquote><p>哈夫曼树：又称最优二叉树，是一种带权路径长度最短的二叉树；<br>哈夫曼编码：哈夫曼树的一个应用，如JPEG中就应用；</p></blockquote><pre><code>所谓树的带权路径长度，就是树中所有的叶结点的权值乘上其到根结点的路径长度（若根结点为0层，叶结点到根结点的路径长度为叶结点的层数）。树的带权路径长度记为WPL = (W1*L1+W2*L2+W3*L3+...+Wn*Ln)，N个权值Wi(i=1,2,...n)构成一棵有N个叶结点的二叉树，相应的叶结点的路径长度为Li(i=1,2,...n)。可以证明哈夫曼树的WPL是最小的。一般权值用来表示频率大小，频率越大则权值越高</code></pre><a id="more"></a> <blockquote><p>参考博客1： <a href="http://www.cnblogs.com/junyuhuang/p/4127095.html" target="_blank" rel="noopener">http://www.cnblogs.com/junyuhuang/p/4127095.html</a></p></blockquote><blockquote><p>参考博客2： <a href="http://www.cnblogs.com/Jezze/archive/2011/12/23/2299884.html" target="_blank" rel="noopener">http://www.cnblogs.com/Jezze/archive/2011/12/23/2299884.html</a></p></blockquote><blockquote><p>####哈弗曼编码原理(转载)</p></blockquote><pre><code> 哈夫曼编码使用一种特别的方法为信号源中的每个符号设定二进制码。出现频率更大的符号将获得更短的比特，出现频率更小的符号将被分配更长的比特，以此来提高数据压缩率，提高传输效率。具体编码步骤主要为， 1、统计： 在开始编码时，通常都需要对信号源，也就是本文的一段文字，进行处理，计算出每个符号出现的频率，得到信号源的基本情况。接下来就是对统计信息进行处理了 2、构造优先对列： 把得到的符号添加到优先队列中，此优先队列的进出逻辑是频率低的先出，因此在设计优先队列时需要如此设计，如果不熟悉优先队列，请阅读相关书籍，在此不做过多概述。得到包含所有字符的优先队列后，就是处理优先队列中的数据了。 3、构造哈夫曼树： 哈夫曼树是带权值得二叉树，我们使用的哈夫曼树的权值自然就是符号的频率了，我们构建哈夫曼树是自底向上的，先构建叶子节点，然后逐步向上，最终完成整颗树。先把队列中的一个符号出列，也就是最小频率的符号，，然后再出列一个符号。这两个符号将作为哈夫曼树的节点，而且这两个节点将作为新节点，也就是它们父节点，的左右孩子节点。新节点的频率，即权值，为孩子节点的和。把这个新节点添加到队列中(队列会重新根据权值排序)。重复上面的步骤，两个符号出列，构造新的父节点，入列……直到队列最后只剩下一个节点，这个节点也就是哈夫曼树的根节点了。 4、为哈弗曼树编码： 哈夫曼树的来自信号源的符号都是叶子节点，需要知道下。树的根节点分配比特0，左子树分配0，右字数分配1。然后就可以得到符号的码值了。</code></pre><blockquote><p>####哈夫曼编码步骤 </p></blockquote><p>1.首先构建一个元素为哈夫曼树结点的数组用于存储哈夫曼树（线性）；</p><pre><code>哈夫曼树数据结构包括元素数据、权值、父结点位置（数组下标）、两个子结点位置typedef struct{ int weight; int parent; int lchild; int rchild; int value;} HNodeType</code></pre><p>2.初始化数组，设置树元素的默认属性；然后输入要编码的数据及其权值，存储到数组前几位，这几个即为哈夫曼树的叶子结点；</p><pre><code>哈夫曼树中有效的数据仅仅是叶子结点，而非叶子结点是为了构建哈夫曼树而加入的</code></pre><p>3.开始构建哈夫曼树：找出整个数组中有数据且权值最低的两个作为新构造的二叉树的左右子树，新二叉树的根结点的权值为其左右子树的根结点的权值之和；<br> 然后把新构建的根结点存放到数中空位置，然后继续从整个数组中寻找。。。直到有效元素全部读取。</p><p>4.编码,树的根节点分配比特0，左子树分配0，右字数分配1;</p><p>5.解码,思路就是 把要解码的字符串像学中学方程一样代入到哈夫曼树中，按照左子树分配0，右字数分配1的原则，一个一个遍历出来</p><hr><hr><pre><code>示例(转载)： 假如我有A,B,C,D,E五个字符，出现的频率（即权值）分别为5,4,3,2,1,那么我们第一步先取两个最小权值作为左右子树构造一个新树，即取1，2构成新树，其结点为1+2=3，如图：</code></pre><p> <img src="/2021/06/01/20170908_hfms/201112231832078695.png" title="哈夫曼树"></p><pre><code> 虚线为新生成的结点，第二步再把新生成的权值为3的结点放到剩下的集合中，所以集合变成{5,4,3,3}，再根据第二步，取最小的两个权值构成新树，如图：</code></pre><p> <img src="/2021/06/01/20170908_hfms/201112231832087092.png" title="哈夫曼树"></p><pre><code>再依次建立哈夫曼树，如下图：</code></pre><p> <img src="/2021/06/01/20170908_hfms/201112231832084301.jpg" title="哈夫曼树"></p><pre><code>其中各个权值替换对应的字符即为下图：</code></pre><p> <img src="/2021/06/01/20170908_hfms/201112231832086286.jpg" title="哈夫曼树"></p><pre><code>如下图也可以加深大家的理解：</code></pre><p> <img src="/2021/06/01/20170908_hfms/272122409653995.gif" title="哈夫曼树"></p><hr><hr><blockquote><p>代码实例</p></blockquote><figure class="highlight c"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br><span class="line">48</span><br><span class="line">49</span><br><span class="line">50</span><br><span class="line">51</span><br><span class="line">52</span><br><span class="line">53</span><br><span class="line">54</span><br><span class="line">55</span><br><span class="line">56</span><br><span class="line">57</span><br><span class="line">58</span><br><span class="line">59</span><br><span class="line">60</span><br><span class="line">61</span><br><span class="line">62</span><br><span class="line">63</span><br><span class="line">64</span><br><span class="line">65</span><br><span class="line">66</span><br><span class="line">67</span><br><span class="line">68</span><br><span class="line">69</span><br><span class="line">70</span><br><span class="line">71</span><br><span class="line">72</span><br><span class="line">73</span><br><span class="line">74</span><br><span class="line">75</span><br><span class="line">76</span><br><span class="line">77</span><br><span class="line">78</span><br><span class="line">79</span><br><span class="line">80</span><br><span class="line">81</span><br><span class="line">82</span><br><span class="line">83</span><br><span class="line">84</span><br><span class="line">85</span><br><span class="line">86</span><br><span class="line">87</span><br><span class="line">88</span><br><span class="line">89</span><br><span class="line">90</span><br><span class="line">91</span><br><span class="line">92</span><br><span class="line">93</span><br><span class="line">94</span><br><span class="line">95</span><br><span class="line">96</span><br><span class="line">97</span><br><span class="line">98</span><br><span class="line">99</span><br><span class="line">100</span><br><span class="line">101</span><br><span class="line">102</span><br><span class="line">103</span><br><span class="line">104</span><br><span class="line">105</span><br><span class="line">106</span><br><span class="line">107</span><br><span class="line">108</span><br><span class="line">109</span><br><span class="line">110</span><br><span class="line">111</span><br><span class="line">112</span><br><span class="line">113</span><br><span class="line">114</span><br><span class="line">115</span><br><span class="line">116</span><br><span class="line">117</span><br><span class="line">118</span><br><span class="line">119</span><br><span class="line">120</span><br><span class="line">121</span><br><span class="line">122</span><br><span class="line">123</span><br><span class="line">124</span><br><span class="line">125</span><br><span class="line">126</span><br><span class="line">127</span><br><span class="line">128</span><br><span class="line">129</span><br><span class="line">130</span><br><span class="line">131</span><br><span class="line">132</span><br><span class="line">133</span><br><span class="line">134</span><br><span class="line">135</span><br><span class="line">136</span><br><span class="line">137</span><br><span class="line">138</span><br><span class="line">139</span><br><span class="line">140</span><br><span class="line">141</span><br><span class="line">142</span><br><span class="line">143</span><br><span class="line">144</span><br><span class="line">145</span><br><span class="line">146</span><br><span class="line">147</span><br><span class="line">148</span><br><span class="line">149</span><br><span class="line">150</span><br><span class="line">151</span><br><span class="line">152</span><br><span class="line">153</span><br><span class="line">154</span><br><span class="line">155</span><br><span class="line">156</span><br><span class="line">157</span><br><span class="line">158</span><br><span class="line">159</span><br><span class="line">160</span><br><span class="line">161</span><br><span class="line">162</span><br><span class="line">163</span><br><span class="line">164</span><br><span class="line">165</span><br><span class="line">166</span><br><span class="line">167</span><br><span class="line">168</span><br><span class="line">169</span><br><span class="line">170</span><br><span class="line">171</span><br><span class="line">172</span><br><span class="line">173</span><br><span class="line">174</span><br><span class="line">175</span><br><span class="line">176</span><br><span class="line">177</span><br><span class="line">178</span><br><span class="line">179</span><br><span class="line">180</span><br><span class="line">181</span><br><span class="line">182</span><br><span class="line">183</span><br><span class="line">184</span><br><span class="line">185</span><br><span class="line">186</span><br><span class="line">187</span><br><span class="line">188</span><br><span class="line">189</span><br><span class="line">190</span><br><span class="line">191</span><br><span class="line">192</span><br><span class="line">193</span><br><span class="line">194</span><br><span class="line">195</span><br><span class="line">196</span><br><span class="line">197</span><br><span class="line">198</span><br><span class="line">199</span><br><span class="line">200</span><br><span class="line">201</span><br><span class="line">202</span><br><span class="line">203</span><br><span class="line">204</span><br><span class="line">205</span><br><span class="line">206</span><br><span class="line">207</span><br><span class="line">208</span><br><span class="line">209</span><br><span class="line">210</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment">/*-------------------------------------------------------------------------</span></span><br><span class="line"><span class="comment"> * Name: 哈夫曼编码源代码。</span></span><br><span class="line"><span class="comment"> * Date: 2011.04.16</span></span><br><span class="line"><span class="comment"> * Author: Jeffrey Hill+Jezze(解码部分)</span></span><br><span class="line"><span class="comment"> * 在 Win-TC 下测试通过</span></span><br><span class="line"><span class="comment"> * 实现过程：着先通过 HuffmanTree() 函数构造哈夫曼树，然后在主函数 main()中</span></span><br><span class="line"><span class="comment"> * 自底向上开始(也就是从数组序号为零的结点开始)向上层层判断，若在</span></span><br><span class="line"><span class="comment"> * 父结点左侧，则置码为 0,若在右侧,则置码为 1。最后输出生成的编码。</span></span><br><span class="line"><span class="comment"> *------------------------------------------------------------------------*/</span></span><br><span class="line"></span><br><span class="line"></span><br><span class="line"><span class="comment">//结点个数 = 叶子结点个数 * 2 - 1</span></span><br><span class="line"><span class="meta">#<span class="meta-keyword">include</span> <span class="meta-string"><stdio.h></span></span></span><br><span class="line"><span class="meta">#<span class="meta-keyword">include</span><span class="meta-string"><stdlib.h></span></span></span><br><span class="line"></span><br><span class="line"><span class="meta">#<span class="meta-keyword">define</span> MAXBIT 100</span></span><br><span class="line"><span class="meta">#<span class="meta-keyword">define</span> MAXVALUE 10000</span></span><br><span class="line"><span class="meta">#<span class="meta-keyword">define</span> MAXLEAF 30</span></span><br><span class="line"><span class="meta">#<span class="meta-keyword">define</span> MAXNODE MAXLEAF*2 -1</span></span><br><span class="line"></span><br><span class="line"><span class="keyword">typedef</span> <span class="class"><span class="keyword">struct</span></span></span><br><span class="line"><span class="class">{</span></span><br><span class="line"> <span class="keyword">int</span> <span class="built_in">bit</span>[MAXBIT]; <span class="comment">//结点的编码</span></span><br><span class="line"> <span class="keyword">int</span> start; <span class="comment">//结点编码的起始位，有效位置，如当start=2， 则该结点的编码从bit[2]开始</span></span><br><span class="line">} HCodeType; <span class="comment">/* 编码结构体 */</span></span><br><span class="line"></span><br><span class="line"><span class="keyword">typedef</span> <span class="class"><span class="keyword">struct</span></span></span><br><span class="line"><span class="class">{</span></span><br><span class="line"> <span class="keyword">int</span> weight;</span><br><span class="line"> <span class="keyword">int</span> parent;</span><br><span class="line"> <span class="keyword">int</span> lchild;</span><br><span class="line"> <span class="keyword">int</span> rchild;</span><br><span class="line"> <span class="keyword">int</span> value;</span><br><span class="line">} HNodeType; <span class="comment">/* 结点结构体 */</span></span><br><span class="line"></span><br><span class="line"><span class="comment">/* 构造一颗哈夫曼树 */</span></span><br><span class="line"><span class="function"><span class="keyword">void</span> <span class="title">HuffmanTree</span><span class="params">(HNodeType HuffNode[MAXNODE], <span class="keyword">int</span> n)</span></span></span><br><span class="line"><span class="function"></span>{</span><br><span class="line"> <span class="comment">/* i、j： 循环变量，</span></span><br><span class="line"><span class="comment"> m1、m2：构造哈夫曼树不同过程中两个最小权值结点的权值，</span></span><br><span class="line"><span class="comment"> x1、x2：构造哈夫曼树不同过程中两个最小权值结点在数组中的序号。*/</span></span><br><span class="line"> <span class="keyword">int</span> i, j, m1, m2, x1, x2;</span><br><span class="line"> <span class="comment">/* 初始化存放哈夫曼树数组 HuffNode[] 中的结点 */</span></span><br><span class="line"> <span class="keyword">for</span>(i = <span class="number">0</span>; i < <span class="number">2</span> * n - <span class="number">1</span>; i++)</span><br><span class="line"> {</span><br><span class="line"> HuffNode[i].weight = <span class="number">0</span>;<span class="comment">//权值</span></span><br><span class="line"> HuffNode[i].parent = <span class="number">-1</span>;<span class="comment">//父结点位置初始化为-1，表示没有父结点，到时候要通过该属性进行判断，排除掉已加入到树的元素</span></span><br><span class="line"> HuffNode[i].lchild = <span class="number">-1</span>;<span class="comment">//初始化左子树根节点位置</span></span><br><span class="line"> HuffNode[i].rchild = <span class="number">-1</span>;<span class="comment">//初始化右子树根节点位置</span></span><br><span class="line"> HuffNode[i].value = i; <span class="comment">//实际值，可根据情况替换为字母</span></span><br><span class="line"> } <span class="comment">/* end for */</span></span><br><span class="line"></span><br><span class="line"> <span class="comment">/* 用户输入 n 个叶子结点的权值 */</span></span><br><span class="line"> <span class="keyword">for</span>(i = <span class="number">0</span>; i < n; i++)</span><br><span class="line"> {</span><br><span class="line"> <span class="built_in">printf</span>(<span class="string">"请输入第 %d 个元素的权值: \n"</span>, i);</span><br><span class="line"> <span class="built_in">scanf</span>(<span class="string">"%d"</span>, &HuffNode[i].weight);</span><br><span class="line"> } <span class="comment">/* end for */</span></span><br><span class="line"></span><br><span class="line"> <span class="comment">/* 循环构造 Huffman 树 */</span></span><br><span class="line"> <span class="keyword">for</span>(i = <span class="number">0</span>; i < n - <span class="number">1</span>; i++) <span class="comment">// 循环叶子结点的个数，一次循环要形成一个“新”的二叉树</span></span><br><span class="line"> {</span><br><span class="line"> m1 = m2 = MAXVALUE; <span class="comment">//m1、m2中存放两个无父结点且结点权值最小的两个结点</span></span><br><span class="line"> x1 = x2 = <span class="number">0</span>;</span><br><span class="line"> <span class="comment">//找出所有结点中权值最小、无父结点的两个结点，并合并之为一颗二叉树</span></span><br><span class="line"> <span class="keyword">for</span>(j = <span class="number">0</span>; j < n + i; j++) <span class="comment">//循环数组中元素个数</span></span><br><span class="line"> {</span><br><span class="line"> <span class="comment">//要找到当前数组中所有的元素中最小的两位，处理之后又把结果存到数组中所以要(n+i)</span></span><br><span class="line"> <span class="keyword">if</span>(HuffNode[j].weight < m1 && HuffNode[j].parent == <span class="number">-1</span>)</span><br><span class="line"> {</span><br><span class="line"> <span class="comment">//当前的权值更小，把当前元素赋值给m1，把原来m1的赋值给m2</span></span><br><span class="line"> m2 = m1;</span><br><span class="line"> x2 = x1;</span><br><span class="line"> m1 = HuffNode[j].weight;</span><br><span class="line"> x1 = j;</span><br><span class="line"> }</span><br><span class="line"> <span class="keyword">else</span> <span class="keyword">if</span>(HuffNode[j].weight < m2 && HuffNode[j].parent == <span class="number">-1</span>)</span><br><span class="line"> {</span><br><span class="line"> <span class="comment">//当前的权值大于m1但是小于m2，把其赋值给m2</span></span><br><span class="line"> m2 = HuffNode[j].weight;</span><br><span class="line"> x2 = j;</span><br><span class="line"> }</span><br><span class="line"> } <span class="comment">/* end for */</span></span><br><span class="line"> <span class="comment">/* 最终，m1每次存放最小权值，m2存放次小的 */</span></span><br><span class="line"></span><br><span class="line"> <span class="comment">/* 设置 找到的两个子结点 x1、x2 的父结点信息 */</span></span><br><span class="line"> <span class="comment">/* 父结点存放到数组的下标为(n + i)的位置 */</span></span><br><span class="line"> HuffNode[x1].parent = n + i;</span><br><span class="line"> HuffNode[x2].parent = n + i;</span><br><span class="line"> <span class="comment">/* 设置 父结点的属性 */</span></span><br><span class="line"> HuffNode[n + i].weight = HuffNode[x1].weight + HuffNode[x2].weight;</span><br><span class="line"> HuffNode[n + i].lchild = x1;</span><br><span class="line"> HuffNode[n + i].rchild = x2;</span><br><span class="line"></span><br><span class="line"> <span class="built_in">printf</span>(<span class="string">"第 %d 次循环的两个结果的权值为: %d, %d\n"</span>, i + <span class="number">1</span>, HuffNode[x1].weight, HuffNode[x2].weight); <span class="comment">/* 用于测试 */</span></span><br><span class="line"> <span class="built_in">printf</span>(<span class="string">"\n"</span>);</span><br><span class="line"> } <span class="comment">/* end for */</span></span><br><span class="line"> <span class="comment">/* for(i=0;i<n+2;i++)</span></span><br><span class="line"><span class="comment"> {</span></span><br><span class="line"><span class="comment"> printf(" Parents:%d,lchild:%d,rchild:%d,value:%d,weight:%d\n",HuffNode[i].parent,HuffNode[i].lchild,HuffNode[i].rchild,HuffNode[i].value,HuffNode[i].weight);</span></span><br><span class="line"><span class="comment"> }*/</span><span class="comment">//测试</span></span><br><span class="line">} <span class="comment">/* end HuffmanTree */</span></span><br><span class="line"></span><br><span class="line"><span class="comment">//解码</span></span><br><span class="line"><span class="function"><span class="keyword">void</span> <span class="title">decodeing</span><span class="params">(<span class="keyword">char</span> <span class="built_in">string</span>[], HNodeType Buf[], <span class="keyword">int</span> Num)</span></span></span><br><span class="line"><span class="function"></span>{</span><br><span class="line"> <span class="comment">//tmp：循环中临时存放哈夫曼树数组的元素下标，从大到小</span></span><br><span class="line"> <span class="keyword">int</span> i, tmp = <span class="number">0</span>, code[<span class="number">1024</span>];</span><br><span class="line"> <span class="keyword">int</span> m = <span class="number">2</span> * Num - <span class="number">1</span>;</span><br><span class="line"> <span class="keyword">char</span> *nump;</span><br><span class="line"> <span class="keyword">char</span> num[<span class="number">1024</span>];</span><br><span class="line"> <span class="keyword">int</span> len = <span class="built_in">strlen</span>(<span class="built_in">string</span>);</span><br><span class="line"></span><br><span class="line"> <span class="keyword">for</span>(i = <span class="number">0</span>; i < len; i++)</span><br><span class="line"> {</span><br><span class="line"> <span class="keyword">if</span>(<span class="built_in">string</span>[i] == <span class="string">'0'</span>)</span><br><span class="line"> num[i] = <span class="number">0</span>;</span><br><span class="line"> <span class="keyword">else</span></span><br><span class="line"> num[i] = <span class="number">1</span>;</span><br><span class="line"> }</span><br><span class="line"> i = <span class="number">0</span>;</span><br><span class="line"> nump = &num[<span class="number">0</span>];<span class="comment">//nump指向要解码的字符数组的第一个元素的地址</span></span><br><span class="line"></span><br><span class="line"> <span class="built_in">printf</span>(<span class="string">" 解码结果为： "</span>);</span><br><span class="line"> <span class="keyword">while</span>(nump < (&num[len]))<span class="comment">//循环遍历要解码的字符数组（即输入的解码前字符串），直到最后一个数组元素</span></span><br><span class="line"> {</span><br><span class="line"> tmp = m - <span class="number">1</span>;<span class="comment">//每次循环设置初始元素下标，设置为 哈夫曼树数组 的最后一个有效数据元素的下标（树的根节点）</span></span><br><span class="line"> <span class="keyword">while</span>((Buf[tmp].lchild != <span class="number">-1</span>) && (Buf[tmp].rchild != <span class="number">-1</span>))<span class="comment">//循环判断该元素有没有子结点，直到没有子结点，则说明遍历到叶子结点，则说明找到一个解码的结果</span></span><br><span class="line"> {</span><br><span class="line"> <span class="keyword">if</span>(*nump == <span class="number">0</span>) <span class="comment">//判断要解码的字符数组的当前元素是否为0 0表示左结点 1表示有结点</span></span><br><span class="line"> {</span><br><span class="line"> tmp = Buf[tmp].lchild ;</span><br><span class="line"> }</span><br><span class="line"> <span class="keyword">else</span></span><br><span class="line"> {</span><br><span class="line"> tmp = Buf[tmp].rchild;</span><br><span class="line"> }</span><br><span class="line"> nump++; <span class="comment">//数组元素是字符，每个元素只占一个字节，所以++也就是地址加一，指向数组下一个元素</span></span><br><span class="line"> }</span><br><span class="line"></span><br><span class="line"> <span class="built_in">printf</span>(<span class="string">" %d"</span>, nump, &num[<span class="built_in">strlen</span>(<span class="built_in">string</span>)], Buf[tmp].value);</span><br><span class="line"> }</span><br><span class="line">}</span><br><span class="line"></span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">int</span> <span class="title">main</span><span class="params">(<span class="keyword">void</span>)</span></span></span><br><span class="line"><span class="function"></span>{</span><br><span class="line"></span><br><span class="line"> HNodeType HuffNode[MAXNODE]; <span class="comment">/* 定义一个结点结构体数组 */</span></span><br><span class="line"> HCodeType HuffCode[MAXLEAF], cd; <span class="comment">/* 定义一个编码结构体数组， 同时定义一个临时变量来存放求解编码时的信息(即cd.bit存放当前结点的编码的倒序，cd.start存放结点编码开始位置) */</span></span><br><span class="line"> <span class="comment">/*</span></span><br><span class="line"><span class="comment"> i、j： 循环变量，</span></span><br><span class="line"><span class="comment"> c 循环体中当前的结点在数组中的下标</span></span><br><span class="line"><span class="comment"> p 当前结点的父结点在数组中的下标</span></span><br><span class="line"><span class="comment"> n 有效数据的个数（叶子结点的个数）</span></span><br><span class="line"><span class="comment"> */</span></span><br><span class="line"> <span class="keyword">int</span> i, j, c, p, n;</span><br><span class="line"> <span class="keyword">char</span> pp[<span class="number">100</span>];</span><br><span class="line"> <span class="built_in">printf</span>(<span class="string">"Please input n:\n"</span>);</span><br><span class="line"> <span class="built_in">scanf</span>(<span class="string">"%d"</span>, &n);</span><br><span class="line"> HuffmanTree(HuffNode, n);<span class="comment">//生成哈夫曼树</span></span><br><span class="line"></span><br><span class="line"> <span class="comment">/*对哈夫曼树进行编码*/</span></span><br><span class="line"> <span class="keyword">for</span>(i = <span class="number">0</span>; i < n; i++)<span class="comment">//循环所有的有效数据，一个一个进行编码 (0 ~ n-1)</span></span><br><span class="line"> {</span><br><span class="line"> cd.start = n - <span class="number">1</span>; <span class="comment">//</span></span><br><span class="line"> c = i; <span class="comment">//c 当前结点在数组中的下标 整体上来说可以表示循环在“树”中走过的结点下标</span></span><br><span class="line"> p = HuffNode[c].parent; <span class="comment">//p 当前结点的父结点在数组中的下标</span></span><br><span class="line"> <span class="keyword">while</span>(p != <span class="number">-1</span>) <span class="comment">/* 判断父结点存在 */</span></span><br><span class="line"> {</span><br><span class="line"> <span class="keyword">if</span>(HuffNode[p].lchild == c) <span class="comment">//判断当前结点是不是父结点的左子树根节点 如果是的话就 赋值0</span></span><br><span class="line"> cd.<span class="built_in">bit</span>[cd.start] = <span class="number">0</span>;</span><br><span class="line"> <span class="keyword">else</span> <span class="comment">//如果不是的话就 赋值1 >>>> 因为“左子树分配0，右字数分配1”</span></span><br><span class="line"> cd.<span class="built_in">bit</span>[cd.start] = <span class="number">1</span>;</span><br><span class="line"> cd.start--; <span class="comment">/* 求编码的低一位 */</span></span><br><span class="line"></span><br><span class="line"> <span class="comment">/* 设置下一循环条件 */</span></span><br><span class="line"> c = p; <span class="comment">//设置c为父结点的下标 准备进行下次while循环，则那时候的“当前的结点”就会变成现在结点的父结点</span></span><br><span class="line"> p = HuffNode[c].parent; <span class="comment">//同理</span></span><br><span class="line"></span><br><span class="line"> }<span class="comment">/* end while */</span></span><br><span class="line"></span><br><span class="line"> <span class="comment">/* 保存 求出的每个叶结点的哈夫曼编码和编码的起始位 */</span></span><br><span class="line"> <span class="keyword">for</span>(j = cd.start + <span class="number">1</span>; j < n; j++)</span><br><span class="line"> {</span><br><span class="line"> HuffCode[i].<span class="built_in">bit</span>[j] = cd.<span class="built_in">bit</span>[j];</span><br><span class="line"> }</span><br><span class="line"> HuffCode[i].start = cd.start + <span class="number">1</span>; <span class="comment">//编码的起始位</span></span><br><span class="line"></span><br><span class="line"> } <span class="comment">/* end for */</span></span><br><span class="line"> <span class="comment">/*对哈夫曼树进行编码结束*/</span></span><br><span class="line"></span><br><span class="line"> <span class="comment">/* 打印已保存好的所有存在编码的哈夫曼编码 */</span></span><br><span class="line"> <span class="keyword">for</span>(i = <span class="number">0</span>; i < n; i++)</span><br><span class="line"> {</span><br><span class="line"> <span class="built_in">printf</span>(<span class="string">"第%d位置的树结点的编码为：: "</span>, i);</span><br><span class="line"> <span class="keyword">for</span>(j = HuffCode[i].start; j < n; j++)<span class="comment">//从有效位置开始输出该结点的编码</span></span><br><span class="line"> {</span><br><span class="line"> <span class="built_in">printf</span>(<span class="string">"%d"</span>, HuffCode[i].<span class="built_in">bit</span>[j]);</span><br><span class="line"> }</span><br><span class="line"> <span class="built_in">printf</span>(<span class="string">" 结点的属性bit数组中有效的编码开始位置为:bit[%d]\n"</span>, HuffCode[i].start);</span><br><span class="line"> }</span><br><span class="line"> <span class="comment">/* 打印结束 */</span></span><br><span class="line"></span><br><span class="line"> <span class="built_in">printf</span>(<span class="string">"Decoding?Please Enter code:\n"</span>);</span><br><span class="line"> <span class="built_in">scanf</span>(<span class="string">"%s"</span>, &pp);</span><br><span class="line"> decodeing(pp, HuffNode, n);</span><br><span class="line"> getch();</span><br><span class="line"> <span class="keyword">return</span> <span class="number">0</span>;</span><br><span class="line">}</span><br></pre></td></tr></table></figure><blockquote><p>个人博客 欢迎来访： <a href="http://zj2626.github.io">http://zj2626.github.io</a></p></blockquote>]]></content>
<summary type="html">
<blockquote>
<p>哈夫曼树：又称最优二叉树，是一种带权路径长度最短的二叉树；<br>哈夫曼编码：哈夫曼树的一个应用，如JPEG中就应用；</p>
</blockquote>
<pre><code>所谓树的带权路径长度，就是树中所有的叶结点的权值乘上其到根结点的路径长度
（若根结点为0层，叶结点到根结点的路径长度为叶结点的层数）。
树的带权路径长度记为WPL = (W1*L1+W2*L2+W3*L3+...+Wn*Ln)，
N个权值Wi(i=1,2,...n)构成一棵有N个叶结点的二叉树，相应的叶结点的路径长度为Li(i=1,2,...n)。
可以证明哈夫曼树的WPL是最小的。
一般权值用来表示频率大小，频率越大则权值越高
</code></pre>
</summary>
<category term="数据结构和算法" scheme="http://zj2626.github.io/categories/%E6%95%B0%E6%8D%AE%E7%BB%93%E6%9E%84%E5%92%8C%E7%AE%97%E6%B3%95/"/>
<category term="C语言" scheme="http://zj2626.github.io/tags/C%E8%AF%AD%E8%A8%80/"/>
</entry>
</feed>