/
feed.xml
555 lines (435 loc) · 91.5 KB
/
feed.xml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
<?xml version="1.0" encoding="utf-8"?>
<feed xmlns="http://www.w3.org/2005/Atom" xml:lang="en">
<title type="text">Luke's Blog</title>
<generator uri="https://github.com/mojombo/jekyll">Jekyll</generator>
<link rel="self" type="application/atom+xml" href="http://crooksey.github.io/feed.xml" />
<link rel="alternate" type="text/html" href="http://crooksey.github.io" />
<updated>2014-08-02T13:33:14+01:00</updated>
<id>http://crooksey.github.io/</id>
<author>
<name>Luke Crooks</name>
<uri>http://crooksey.github.io/</uri>
<email>luke@pumalo.org</email>
</author>
<entry>
<title type="html"><![CDATA[Python TSP]]></title>
<link rel="alternate" type="text/html" href="http://crooksey.github.io/python-tsp/" />
<id>http://crooksey.github.io/python-tsp</id>
<updated>2014-08-02T00:00:00-00:00</updated>
<published>2014-08-02T00:00:00+01:00</published>
<author>
<name>Luke Crooks</name>
<uri>http://crooksey.github.io</uri>
<email>luke@pumalo.org</email>
</author>
<content type="html"><h3 id="the-problem">The problem</h3>
<p>Wikipedia states, the travelling salesman problem as this: “The travelling salesman problem asks the following question: Given a list of cities and the distances between each pair of cities, what is the shortest possible route that visits each city exactly once and returns to the origin city?”</p>
<p>The actual equation behind this formula is very complex and the explanation of this is is probably best left to the <a href="http://en.wikipedia.org/wiki/Travelling_salesman_problem">Wikipedia Page</a></p>
<h3 id="my-use-case">My use case</h3>
<p>I wrote an application that takes X number of orders, and has to split them between 9 vehicles, based on their, weight dimension and delivery location. I was first going to use my own formula to do this, but the
time and power taken to run this in production, is a bit out of the question. So instead, I decided to sort
all of the orders out into what vehicle they would end up in (code available for sale).</p>
<p>Now, I have 10-15 different addresses that I need to calculate the most efficient route for. So instead of writing a system from scratch, I decided to connect to a third party API.</p>
<h3 id="it-started-with-google">It started with Google…</h3>
<p>So I first wrote the following script in python, to work with the Google Maps API. (I didn’t realise they wanted £7,500 a year ex VAT to use their API, so I moved onto an open source solution). If you are a Google maps user and want to take advantage of my code, it is shown below with comments: (it could be compressed a lot more, but I have tried to make it easy to read for those from a less technical background) </p>
<div class="highlight"><pre><code class="language-python" data-lang="python"><span class="c">################################################################### Start and finish at same place ######## go via, with route optimise ##</span>
<span class="c"># this method takes advantage of the optimisiation that Google lets us use.</span>
<span class="n">url</span><span class="o">=</span> <span class="s">&quot;https://maps.googleapis.com/maps/api/directions/json?origin=Portsmouth,UK&amp;destination=Portsmouth,UK&amp;waypoints=optimize:true|Southampton,UK|Poole,UK|Reading,UK|Bridport,Dorset&amp;sensor=false&amp;key=YOURAPIKEYHERE&quot;</span>
<span class="c"># set the waypoints in the url</span>
<span class="kn">import</span> <span class="nn">urllib2</span>
<span class="n">data</span> <span class="o">=</span> <span class="n">urllib2</span><span class="o">.</span><span class="n">urlopen</span><span class="p">(</span><span class="n">url</span><span class="p">)</span>
<span class="c"># now get the json response</span>
<span class="n">pre_json</span> <span class="o">=</span> <span class="n">data</span><span class="o">.</span><span class="n">read</span><span class="p">()</span>
<span class="kn">import</span> <span class="nn">json</span>
<span class="n">obj</span> <span class="o">=</span> <span class="n">json</span><span class="o">.</span><span class="n">loads</span><span class="p">(</span><span class="n">pre_json</span><span class="p">)</span>
<span class="n">data_new</span> <span class="o">=</span> <span class="n">json</span><span class="o">.</span><span class="n">dumps</span><span class="p">(</span><span class="n">obj</span><span class="p">,</span> <span class="n">indent</span><span class="o">=</span><span class="mi">2</span><span class="p">)</span>
<span class="c">#load into dict</span>
<span class="n">d</span> <span class="o">=</span> <span class="n">json</span><span class="o">.</span><span class="n">loads</span><span class="p">(</span><span class="n">data_new</span><span class="p">)</span>
<span class="n">routes</span> <span class="o">=</span> <span class="p">[]</span>
<span class="c"># get the starting address</span>
<span class="n">start_address</span> <span class="o">=</span> <span class="n">d</span><span class="p">[</span><span class="s">&#39;routes&#39;</span><span class="p">][</span><span class="mi">0</span><span class="p">][</span><span class="s">&#39;legs&#39;</span><span class="p">][</span><span class="mi">0</span><span class="p">][</span><span class="s">&#39;start_address&#39;</span><span class="p">]</span>
<span class="n">start_lat</span> <span class="o">=</span> <span class="n">d</span><span class="p">[</span><span class="s">&#39;routes&#39;</span><span class="p">][</span><span class="mi">0</span><span class="p">][</span><span class="s">&#39;legs&#39;</span><span class="p">][</span><span class="mi">0</span><span class="p">][</span><span class="s">&#39;start_location&#39;</span><span class="p">][</span><span class="s">u&#39;lat&#39;</span><span class="p">]</span>
<span class="n">start_long</span> <span class="o">=</span> <span class="n">d</span><span class="p">[</span><span class="s">&#39;routes&#39;</span><span class="p">][</span><span class="mi">0</span><span class="p">][</span><span class="s">&#39;legs&#39;</span><span class="p">][</span><span class="mi">0</span><span class="p">][</span><span class="s">&#39;start_location&#39;</span><span class="p">][</span><span class="s">u&#39;lng&#39;</span><span class="p">]</span>
<span class="n">routes</span><span class="o">.</span><span class="n">append</span><span class="p">({</span><span class="s">&#39;address&#39;</span><span class="p">:</span> <span class="n">start_address</span><span class="p">,</span> <span class="s">&#39;lat&#39;</span><span class="p">:</span> <span class="n">start_lat</span><span class="p">,</span> <span class="s">&#39;lng&#39;</span><span class="p">:</span> <span class="n">start_long</span><span class="p">})</span>
<span class="n">total_time</span> <span class="o">=</span> <span class="mi">0</span>
<span class="n">total_distance</span> <span class="o">=</span> <span class="mi">0</span>
<span class="k">for</span> <span class="n">x</span> <span class="ow">in</span> <span class="n">d</span><span class="p">[</span><span class="s">&#39;routes&#39;</span><span class="p">][</span><span class="mi">0</span><span class="p">][</span><span class="s">&#39;legs&#39;</span><span class="p">]:</span>
<span class="n">total_time</span> <span class="o">+=</span> <span class="n">x</span><span class="p">[</span><span class="s">&#39;duration&#39;</span><span class="p">][</span><span class="s">&#39;value&#39;</span><span class="p">]</span>
<span class="n">total_distance</span> <span class="o">+=</span> <span class="n">x</span><span class="p">[</span><span class="s">&#39;distance&#39;</span><span class="p">][</span><span class="s">&#39;value&#39;</span><span class="p">]</span>
<span class="n">address</span> <span class="o">=</span> <span class="n">x</span><span class="p">[</span><span class="s">&#39;end_address&#39;</span><span class="p">]</span>
<span class="n">lat</span> <span class="o">=</span> <span class="n">x</span><span class="p">[</span><span class="s">&#39;end_location&#39;</span><span class="p">][</span><span class="s">u&#39;lat&#39;</span><span class="p">]</span>
<span class="n">lng</span> <span class="o">=</span> <span class="n">x</span><span class="p">[</span><span class="s">&#39;end_location&#39;</span><span class="p">][</span><span class="s">u&#39;lng&#39;</span><span class="p">]</span>
<span class="n">routes</span><span class="o">.</span><span class="n">append</span><span class="p">({</span><span class="s">&#39;address&#39;</span><span class="p">:</span> <span class="n">address</span><span class="p">,</span> <span class="s">&#39;lat&#39;</span><span class="p">:</span> <span class="n">lat</span><span class="p">,</span> <span class="s">&#39;lng&#39;</span><span class="p">:</span> <span class="n">lng</span><span class="p">})</span>
<span class="k">print</span> <span class="s">&#39;##############################################################################&#39;</span>
<span class="k">print</span> <span class="s">&#39;Optimised Route&#39;</span>
<span class="k">print</span> <span class="s">&#39;##############################################################################&#39;</span>
<span class="n">tt</span> <span class="o">=</span> <span class="p">(</span><span class="n">total_time</span> <span class="o">/</span> <span class="mf">3600.00</span><span class="p">)</span>
<span class="n">td</span> <span class="o">=</span> <span class="n">total_distance</span><span class="o">*</span><span class="mf">0.000621371192</span>
<span class="k">print</span> <span class="s">&quot;Total Time </span><span class="si">%.2f</span><span class="s"> hours&quot;</span> <span class="o">%</span> <span class="nb">round</span><span class="p">(</span><span class="n">tt</span><span class="p">,</span><span class="mi">2</span><span class="p">)</span>
<span class="k">print</span> <span class="s">&quot;Total Distance </span><span class="si">%.2f</span><span class="s"> miles&quot;</span> <span class="o">%</span> <span class="nb">round</span><span class="p">(</span><span class="n">td</span><span class="p">,</span><span class="mi">2</span><span class="p">)</span>
<span class="k">for</span> <span class="n">x</span> <span class="ow">in</span> <span class="n">routes</span><span class="p">:</span>
<span class="k">print</span> <span class="n">x</span>
<span class="c">## ORIG ROUTE ##</span>
<span class="c"># Now we put the route in, without any optimisation flags so we can compare</span>
<span class="n">url2</span> <span class="o">=</span> <span class="s">&quot;https://maps.googleapis.com/maps/api/directions/json?origin=Portsmouth,UK&amp;destination=Portsmouth,UK&amp;waypoints=|Southampton,UK|Poole,UK|Reading,UK|Bridport,Dorset&amp;sensor=false&amp;key=YOURAPIKEYHERE&quot;</span>
<span class="n">data2</span> <span class="o">=</span> <span class="n">urllib2</span><span class="o">.</span><span class="n">urlopen</span><span class="p">(</span><span class="n">url2</span><span class="p">)</span>
<span class="n">pre_json2</span> <span class="o">=</span> <span class="n">data2</span><span class="o">.</span><span class="n">read</span><span class="p">()</span>
<span class="kn">import</span> <span class="nn">json</span>
<span class="n">obj2</span> <span class="o">=</span> <span class="n">json</span><span class="o">.</span><span class="n">loads</span><span class="p">(</span><span class="n">pre_json2</span><span class="p">)</span>
<span class="n">data_new2</span> <span class="o">=</span> <span class="n">json</span><span class="o">.</span><span class="n">dumps</span><span class="p">(</span><span class="n">obj2</span><span class="p">,</span> <span class="n">indent</span><span class="o">=</span><span class="mi">2</span><span class="p">)</span>
<span class="c">#load into dict</span>
<span class="n">d2</span> <span class="o">=</span> <span class="n">json</span><span class="o">.</span><span class="n">loads</span><span class="p">(</span><span class="n">data_new2</span><span class="p">)</span>
<span class="n">routes2</span> <span class="o">=</span> <span class="p">[]</span>
<span class="n">start_address2</span> <span class="o">=</span> <span class="n">d2</span><span class="p">[</span><span class="s">&#39;routes&#39;</span><span class="p">][</span><span class="mi">0</span><span class="p">][</span><span class="s">&#39;legs&#39;</span><span class="p">][</span><span class="mi">0</span><span class="p">][</span><span class="s">&#39;start_address&#39;</span><span class="p">]</span>
<span class="n">start_lat2</span> <span class="o">=</span> <span class="n">d2</span><span class="p">[</span><span class="s">&#39;routes&#39;</span><span class="p">][</span><span class="mi">0</span><span class="p">][</span><span class="s">&#39;legs&#39;</span><span class="p">][</span><span class="mi">0</span><span class="p">][</span><span class="s">&#39;start_location&#39;</span><span class="p">][</span><span class="s">u&#39;lat&#39;</span><span class="p">]</span>
<span class="n">start_long2</span> <span class="o">=</span> <span class="n">d2</span><span class="p">[</span><span class="s">&#39;routes&#39;</span><span class="p">][</span><span class="mi">0</span><span class="p">][</span><span class="s">&#39;legs&#39;</span><span class="p">][</span><span class="mi">0</span><span class="p">][</span><span class="s">&#39;start_location&#39;</span><span class="p">][</span><span class="s">u&#39;lng&#39;</span><span class="p">]</span>
<span class="n">routes2</span><span class="o">.</span><span class="n">append</span><span class="p">({</span><span class="s">&#39;address&#39;</span><span class="p">:</span> <span class="n">start_address</span><span class="p">,</span> <span class="s">&#39;lat&#39;</span><span class="p">:</span> <span class="n">start_lat</span><span class="p">,</span> <span class="s">&#39;lng&#39;</span><span class="p">:</span> <span class="n">start_long</span><span class="p">})</span>
<span class="n">total_time2</span> <span class="o">=</span> <span class="mi">0</span>
<span class="n">total_distance2</span> <span class="o">=</span> <span class="mi">0</span>
<span class="k">for</span> <span class="n">x</span> <span class="ow">in</span> <span class="n">d2</span><span class="p">[</span><span class="s">&#39;routes&#39;</span><span class="p">][</span><span class="mi">0</span><span class="p">][</span><span class="s">&#39;legs&#39;</span><span class="p">]:</span>
<span class="n">total_time2</span> <span class="o">+=</span> <span class="n">x</span><span class="p">[</span><span class="s">&#39;duration&#39;</span><span class="p">][</span><span class="s">&#39;value&#39;</span><span class="p">]</span>
<span class="n">total_distance2</span> <span class="o">+=</span> <span class="n">x</span><span class="p">[</span><span class="s">&#39;distance&#39;</span><span class="p">][</span><span class="s">&#39;value&#39;</span><span class="p">]</span>
<span class="n">address</span> <span class="o">=</span> <span class="n">x</span><span class="p">[</span><span class="s">&#39;end_address&#39;</span><span class="p">]</span>
<span class="n">lat</span> <span class="o">=</span> <span class="n">x</span><span class="p">[</span><span class="s">&#39;end_location&#39;</span><span class="p">][</span><span class="s">u&#39;lat&#39;</span><span class="p">]</span>
<span class="n">lng</span> <span class="o">=</span> <span class="n">x</span><span class="p">[</span><span class="s">&#39;end_location&#39;</span><span class="p">][</span><span class="s">u&#39;lng&#39;</span><span class="p">]</span>
<span class="n">routes2</span><span class="o">.</span><span class="n">append</span><span class="p">({</span><span class="s">&#39;address&#39;</span><span class="p">:</span> <span class="n">address</span><span class="p">,</span> <span class="s">&#39;lat&#39;</span><span class="p">:</span> <span class="n">lat</span><span class="p">,</span> <span class="s">&#39;lng&#39;</span><span class="p">:</span> <span class="n">lng</span><span class="p">})</span>
<span class="k">print</span> <span class="s">&#39;##############################################################################&#39;</span>
<span class="k">print</span> <span class="s">&#39;Original Route&#39;</span>
<span class="k">print</span> <span class="s">&#39;##############################################################################&#39;</span>
<span class="n">tt2</span> <span class="o">=</span> <span class="p">(</span><span class="n">total_time2</span> <span class="o">/</span> <span class="mf">3600.00</span><span class="p">)</span>
<span class="n">td2</span> <span class="o">=</span> <span class="n">total_distance2</span><span class="o">*</span><span class="mf">0.000621371192</span>
<span class="k">print</span> <span class="s">&quot;Total Time </span><span class="si">%.2f</span><span class="s"> hours&quot;</span> <span class="o">%</span> <span class="nb">round</span><span class="p">(</span><span class="n">tt2</span><span class="p">,</span><span class="mi">2</span><span class="p">)</span>
<span class="k">print</span> <span class="s">&quot;Total Distance </span><span class="si">%.2f</span><span class="s"> miles&quot;</span> <span class="o">%</span> <span class="nb">round</span><span class="p">(</span><span class="n">td2</span><span class="p">,</span><span class="mi">2</span><span class="p">)</span>
<span class="k">for</span> <span class="n">x</span> <span class="ow">in</span> <span class="n">routes2</span><span class="p">:</span>
<span class="k">print</span> <span class="n">x</span>
<span class="k">print</span> <span class="s">&#39;##############################################################################&#39;</span>
<span class="k">print</span> <span class="s">&#39;Summary&#39;</span>
<span class="k">print</span> <span class="s">&#39;##############################################################################&#39;</span>
<span class="n">total_saving_time</span> <span class="o">=</span> <span class="n">tt2</span> <span class="o">-</span> <span class="n">tt</span>
<span class="n">total_saving_distance</span> <span class="o">=</span> <span class="n">td2</span> <span class="o">-</span> <span class="n">td</span>
<span class="k">print</span> <span class="s">&#39;The optimised route is </span><span class="si">%.2f</span><span class="s"> miles shorter&#39;</span> <span class="o">%</span> <span class="nb">round</span><span class="p">(</span><span class="n">total_saving_distance</span><span class="p">,</span><span class="mi">2</span><span class="p">)</span>
<span class="k">print</span> <span class="s">&#39;And also saves </span><span class="si">%.2f</span><span class="s"> hours of time&#39;</span> <span class="o">%</span> <span class="nb">round</span><span class="p">(</span><span class="n">total_saving_time</span><span class="p">,</span><span class="mi">2</span><span class="p">)</span></code></pre></div>
<p>Here is an example output:</p>
<div class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="c">##############################################################################</span>
Optimised Route
<span class="c">##############################################################################</span>
Total Time 5.95 hours
Total Distance 274.23 miles
<span class="o">{</span><span class="s1">&#39;lat&#39;</span>: 50.8166745, <span class="s1">&#39;lng&#39;</span>: -1.0833259, <span class="s1">&#39;address&#39;</span>: u<span class="s1">&#39;Portsmouth, UK&#39;</span><span class="o">}</span>
<span class="o">{</span><span class="s1">&#39;lat&#39;</span>: 51.4544776, <span class="s1">&#39;lng&#39;</span>: -0.9781547, <span class="s1">&#39;address&#39;</span>: u<span class="s1">&#39;Reading, UK&#39;</span><span class="o">}</span>
<span class="o">{</span><span class="s1">&#39;lat&#39;</span>: 50.7150463, <span class="s1">&#39;lng&#39;</span>: -1.9872525, <span class="s1">&#39;address&#39;</span>: u<span class="s1">&#39;Poole, UK&#39;</span><span class="o">}</span>
<span class="o">{</span><span class="s1">&#39;lat&#39;</span>: 50.7335746, <span class="s1">&#39;lng&#39;</span>: -2.7583416, <span class="s1">&#39;address&#39;</span>: u<span class="s1">&#39;Bridport, Dorset DT6, UK&#39;</span><span class="o">}</span>
<span class="o">{</span><span class="s1">&#39;lat&#39;</span>: 50.9096948, <span class="s1">&#39;lng&#39;</span>: -1.4047779, <span class="s1">&#39;address&#39;</span>: u<span class="s1">&#39;Southampton, UK&#39;</span><span class="o">}</span>
<span class="o">{</span><span class="s1">&#39;lat&#39;</span>: 50.8166745, <span class="s1">&#39;lng&#39;</span>: -1.0833259, <span class="s1">&#39;address&#39;</span>: u<span class="s1">&#39;Portsmouth, UK&#39;</span><span class="o">}</span>
<span class="c">##############################################################################</span>
Original Route
<span class="c">##############################################################################</span>
Total Time 7.32 hours
Total Distance 334.71 miles
<span class="o">{</span><span class="s1">&#39;lat&#39;</span>: 50.8166745, <span class="s1">&#39;lng&#39;</span>: -1.0833259, <span class="s1">&#39;address&#39;</span>: u<span class="s1">&#39;Portsmouth, UK&#39;</span><span class="o">}</span>
<span class="o">{</span><span class="s1">&#39;lat&#39;</span>: 50.9096948, <span class="s1">&#39;lng&#39;</span>: -1.4047779, <span class="s1">&#39;address&#39;</span>: u<span class="s1">&#39;Southampton, UK&#39;</span><span class="o">}</span>
<span class="o">{</span><span class="s1">&#39;lat&#39;</span>: 50.7150463, <span class="s1">&#39;lng&#39;</span>: -1.9872525, <span class="s1">&#39;address&#39;</span>: u<span class="s1">&#39;Poole, UK&#39;</span><span class="o">}</span>
<span class="o">{</span><span class="s1">&#39;lat&#39;</span>: 51.4544776, <span class="s1">&#39;lng&#39;</span>: -0.9781547, <span class="s1">&#39;address&#39;</span>: u<span class="s1">&#39;Reading, UK&#39;</span><span class="o">}</span>
<span class="o">{</span><span class="s1">&#39;lat&#39;</span>: 50.7335746, <span class="s1">&#39;lng&#39;</span>: -2.7583416, <span class="s1">&#39;address&#39;</span>: u<span class="s1">&#39;Bridport, Dorset DT6, UK&#39;</span><span class="o">}</span>
<span class="o">{</span><span class="s1">&#39;lat&#39;</span>: 50.8166745, <span class="s1">&#39;lng&#39;</span>: -1.0833259, <span class="s1">&#39;address&#39;</span>: u<span class="s1">&#39;Portsmouth, UK&#39;</span><span class="o">}</span>
<span class="c">##############################################################################</span>
Summary
<span class="c">##############################################################################</span>
<span class="s1">&#39;&#39;&#39;</span>
<span class="s1">The optimised route is 60.48 miles shorter</span>
<span class="s1">And also saves 1.36 hours of time</span>
<span class="s1">&#39;&#39;&#39;</span>
<span class="c">###############################################################################</span>
<span class="nb">time </span>python tsp.py
real 0m5.843s
user 0m0.128s
sys 0m0.040s
<span class="c">###############################################################################</span></code></pre></div>
<h3 id="open-source-style">Open Source Style..</h3>
<p>Now, lets do the same thing, but with a FREE licence and using Open Source data, using the brilliant API from <a href="http://developer.mapquest.com/web/products/open">MapQuest</a></p>
<div class="highlight"><pre><code class="language-python" data-lang="python"><span class="kn">import</span> <span class="nn">requests</span>
<span class="kn">import</span> <span class="nn">json</span>
<span class="c"># set your app key</span>
<span class="n">my_app_key</span> <span class="o">=</span> <span class="s">&quot;YOUR_KEY&quot;</span>
<span class="c"># add points to the route</span>
<span class="n">route</span> <span class="o">=</span> <span class="p">{</span><span class="s">&quot;locations&quot;</span><span class="p">:[]}</span>
<span class="n">route</span><span class="p">[</span><span class="s">&#39;locations&#39;</span><span class="p">]</span><span class="o">.</span><span class="n">append</span><span class="p">(</span> <span class="p">{</span><span class="s">&quot;latLng&quot;</span><span class="p">:{</span><span class="s">&quot;lat&quot;</span><span class="p">:</span><span class="mf">51.524410144966154</span><span class="p">,</span><span class="s">&quot;lng&quot;</span><span class="p">:</span><span class="o">-</span><span class="mf">0.12989273652335526</span><span class="p">}})</span>
<span class="n">route</span><span class="p">[</span><span class="s">&#39;locations&#39;</span><span class="p">]</span><span class="o">.</span><span class="n">append</span><span class="p">(</span> <span class="p">{</span><span class="s">&quot;latLng&quot;</span><span class="p">:{</span><span class="s">&quot;lat&quot;</span><span class="p">:</span><span class="mf">51.54495915136182</span><span class="p">,</span><span class="s">&quot;lng&quot;</span><span class="p">:</span><span class="o">-</span><span class="mf">0.16518885449221493</span><span class="p">}})</span>
<span class="n">route</span><span class="p">[</span><span class="s">&#39;locations&#39;</span><span class="p">]</span><span class="o">.</span><span class="n">append</span><span class="p">(</span> <span class="p">{</span><span class="s">&quot;latLng&quot;</span><span class="p">:{</span><span class="s">&quot;lat&quot;</span><span class="p">:</span><span class="mf">51.52061842826141</span><span class="p">,</span><span class="s">&quot;lng&quot;</span><span class="p">:</span><span class="o">-</span><span class="mf">0.1495479641837033</span><span class="p">}})</span>
<span class="n">route</span><span class="p">[</span><span class="s">&#39;locations&#39;</span><span class="p">]</span><span class="o">.</span><span class="n">append</span><span class="p">(</span> <span class="p">{</span><span class="s">&quot;latLng&quot;</span><span class="p">:{</span><span class="s">&quot;lat&quot;</span><span class="p">:</span><span class="mf">51.52850609658769</span><span class="p">,</span><span class="s">&quot;lng&quot;</span><span class="p">:</span><span class="o">-</span><span class="mf">0.20170525707760403</span><span class="p">}})</span>
<span class="c"># url to post of API</span>
<span class="n">url</span> <span class="o">=</span> <span class="s">&quot;http://open.mapquestapi.com/directions/v2/optimizedroute?key=&quot;</span> <span class="o">+</span> <span class="n">my_app_key</span>
<span class="n">url_basic</span> <span class="o">=</span> <span class="s">&quot;http://open.mapquestapi.com/directions/v2/route?key=&quot;</span> <span class="o">+</span> <span class="n">my_app_key</span>
<span class="c"># Important, we need to add headers saying we are posting JSON</span>
<span class="n">headers</span> <span class="o">=</span> <span class="p">{</span><span class="s">&#39;Content-type&#39;</span><span class="p">:</span> <span class="s">&#39;application/json&#39;</span><span class="p">,</span> <span class="s">&#39;Accept&#39;</span><span class="p">:</span> <span class="s">&#39;text/plain&#39;</span><span class="p">}</span>
<span class="c"># build the request</span>
<span class="n">r</span> <span class="o">=</span> <span class="n">requests</span><span class="o">.</span><span class="n">post</span><span class="p">(</span><span class="n">url</span><span class="p">,</span> <span class="n">data</span><span class="o">=</span><span class="n">json</span><span class="o">.</span><span class="n">dumps</span><span class="p">(</span><span class="n">route</span><span class="p">),</span> <span class="n">headers</span><span class="o">=</span><span class="n">headers</span><span class="p">)</span>
<span class="n">r_basic</span> <span class="o">=</span> <span class="n">requests</span><span class="o">.</span><span class="n">post</span><span class="p">(</span><span class="n">url_basic</span><span class="p">,</span> <span class="n">data</span><span class="o">=</span><span class="n">json</span><span class="o">.</span><span class="n">dumps</span><span class="p">(</span><span class="n">route</span><span class="p">),</span> <span class="n">headers</span><span class="o">=</span><span class="n">headers</span><span class="p">)</span>
<span class="c"># load the json respone into the praser </span>
<span class="n">route_json</span> <span class="o">=</span> <span class="n">json</span><span class="o">.</span><span class="n">dumps</span><span class="p">(</span><span class="n">r</span><span class="o">.</span><span class="n">json</span><span class="p">(),</span> <span class="n">indent</span><span class="o">=</span><span class="mi">2</span><span class="p">)</span>
<span class="n">route_json_basic</span> <span class="o">=</span> <span class="n">json</span><span class="o">.</span><span class="n">dumps</span><span class="p">(</span><span class="n">r_basic</span><span class="o">.</span><span class="n">json</span><span class="p">(),</span> <span class="n">indent</span><span class="o">=</span><span class="mi">2</span><span class="p">)</span>
<span class="c"># load into a python data structure</span>
<span class="n">data</span> <span class="o">=</span> <span class="n">json</span><span class="o">.</span><span class="n">loads</span><span class="p">(</span><span class="n">route_json</span><span class="p">)</span>
<span class="n">data_basic</span> <span class="o">=</span> <span class="n">json</span><span class="o">.</span><span class="n">loads</span><span class="p">(</span><span class="n">route_json_basic</span><span class="p">)</span>
<span class="c"># Map URL to generate a map showing the routes we are taking on this trip</span>
<span class="n">map_placeholder</span> <span class="o">=</span> <span class="s">&quot;http://open.mapquestapi.com/staticmap/v4/getmap?key={my_app_key}&amp;bestfit={bestfit}&amp;shape={shape}&amp;size=600,600&amp;type=map&amp;imagetype=jpeg&quot;</span>
<span class="c"># Define some functions to return clean data</span>
<span class="k">def</span> <span class="nf">get_bounding_box</span><span class="p">(</span><span class="n">data</span><span class="p">):</span>
<span class="c"># load the bounding box data which forms the edges</span>
<span class="c"># of the static map we will be using</span>
<span class="n">bestfit</span> <span class="o">=</span> <span class="s">&quot;{lat1},{long1},{lat2},{long2}&quot;</span>
<span class="n">bestfit_clean</span> <span class="o">=</span> <span class="n">bestfit</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">lat1</span><span class="o">=</span><span class="n">data</span><span class="p">[</span><span class="s">&#39;ul&#39;</span><span class="p">][</span><span class="s">&#39;lat&#39;</span><span class="p">],</span>
<span class="n">long1</span><span class="o">=</span><span class="n">data</span><span class="p">[</span><span class="s">&#39;ul&#39;</span><span class="p">][</span><span class="s">&#39;lng&#39;</span><span class="p">],</span> <span class="n">lat2</span><span class="o">=</span><span class="n">data</span><span class="p">[</span><span class="s">&#39;lr&#39;</span><span class="p">][</span><span class="s">&#39;lat&#39;</span><span class="p">],</span>
<span class="n">long2</span><span class="o">=</span><span class="n">data</span><span class="p">[</span><span class="s">&#39;lr&#39;</span><span class="p">][</span><span class="s">&#39;lng&#39;</span><span class="p">])</span>
<span class="c"># now return the clean variable for use in the static map url</span>
<span class="k">return</span> <span class="n">bestfit_clean</span>
<span class="k">def</span> <span class="nf">get_shape</span><span class="p">(</span><span class="n">data</span><span class="p">):</span>
<span class="n">seq</span> <span class="o">=</span> <span class="s">&quot;&quot;</span>
<span class="k">for</span> <span class="n">x</span> <span class="ow">in</span> <span class="n">data</span><span class="p">[</span><span class="s">&#39;locations&#39;</span><span class="p">]:</span>
<span class="n">latLng</span> <span class="o">=</span> <span class="nb">str</span><span class="p">(</span><span class="n">x</span><span class="p">[</span><span class="s">&#39;displayLatLng&#39;</span><span class="p">][</span><span class="s">&#39;lat&#39;</span><span class="p">])</span> <span class="o">+</span> <span class="s">&#39;,&#39;</span> <span class="o">+</span> <span class="nb">str</span><span class="p">(</span><span class="n">x</span><span class="p">[</span><span class="s">&#39;displayLatLng&#39;</span><span class="p">][</span><span class="s">&#39;lng&#39;</span><span class="p">])</span> <span class="o">+</span> <span class="s">&#39;,&#39;</span>
<span class="n">seq</span> <span class="o">=</span> <span class="n">seq</span> <span class="o">+</span> <span class="n">latLng</span>
<span class="c"># remove last trailing comma</span>
<span class="n">shape_clean</span> <span class="o">=</span> <span class="n">seq</span><span class="p">[:</span><span class="o">-</span><span class="mi">1</span><span class="p">]</span>
<span class="k">return</span> <span class="n">shape_clean</span>
<span class="c"># run data through the functions</span>
<span class="n">shape_basic</span> <span class="o">=</span> <span class="n">get_shape</span><span class="p">(</span><span class="n">data_basic</span><span class="p">[</span><span class="s">&#39;route&#39;</span><span class="p">])</span>
<span class="n">shape_optimised</span> <span class="o">=</span> <span class="n">get_shape</span><span class="p">(</span><span class="n">data</span><span class="p">[</span><span class="s">&#39;route&#39;</span><span class="p">])</span>
<span class="n">bestfit_basic</span> <span class="o">=</span> <span class="n">get_bounding_box</span><span class="p">(</span><span class="n">data_basic</span><span class="p">[</span><span class="s">&#39;route&#39;</span><span class="p">][</span><span class="s">&#39;boundingBox&#39;</span><span class="p">])</span>
<span class="n">bestfit_optimised</span> <span class="o">=</span> <span class="n">get_bounding_box</span><span class="p">(</span><span class="n">data</span><span class="p">[</span><span class="s">&#39;route&#39;</span><span class="p">][</span><span class="s">&#39;boundingBox&#39;</span><span class="p">])</span>
<span class="c"># generate map URL</span>
<span class="n">map_basic</span> <span class="o">=</span> <span class="n">map_placeholder</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">my_app_key</span><span class="o">=</span><span class="n">my_app_key</span><span class="p">,</span> <span class="n">bestfit</span><span class="o">=</span><span class="n">bestfit_basic</span><span class="p">,</span>
<span class="n">shape</span><span class="o">=</span><span class="n">shape_basic</span><span class="p">)</span>
<span class="n">map_optimised</span> <span class="o">=</span> <span class="n">map_placeholder</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">my_app_key</span><span class="o">=</span><span class="n">my_app_key</span><span class="p">,</span> <span class="n">bestfit</span><span class="o">=</span><span class="n">bestfit_optimised</span><span class="p">,</span>
<span class="n">shape</span><span class="o">=</span><span class="n">shape_optimised</span><span class="p">)</span>
<span class="c"># Get a printout of the data</span>
<span class="k">print</span> <span class="s">&#39;&gt;------------&lt;&#39;</span>
<span class="k">print</span> <span class="s">&#39;Original Route&#39;</span>
<span class="k">print</span> <span class="s">&#39;&gt;------------&lt;&#39;</span>
<span class="c"># Show original order</span>
<span class="k">for</span> <span class="n">x</span> <span class="ow">in</span> <span class="n">data_basic</span><span class="p">[</span><span class="s">&#39;route&#39;</span><span class="p">][</span><span class="s">&#39;locationSequence&#39;</span><span class="p">]:</span>
<span class="k">print</span> <span class="n">route</span><span class="p">[</span><span class="s">&#39;locations&#39;</span><span class="p">][</span><span class="nb">int</span><span class="p">(</span><span class="n">x</span><span class="p">)][</span><span class="s">&#39;latLng&#39;</span><span class="p">][</span><span class="s">&#39;lat&#39;</span><span class="p">]</span>
<span class="k">print</span> <span class="s">&quot;Total Distance &quot;</span> <span class="o">+</span> <span class="nb">str</span><span class="p">(</span><span class="n">data_basic</span><span class="p">[</span><span class="s">&#39;route&#39;</span><span class="p">][</span><span class="s">&#39;distance&#39;</span><span class="p">])</span> <span class="o">+</span> <span class="s">&#39; miles&#39;</span>
<span class="k">print</span> <span class="s">&quot;Total Fuel Used &quot;</span> <span class="o">+</span> <span class="nb">str</span><span class="p">(</span><span class="n">data_basic</span><span class="p">[</span><span class="s">&#39;route&#39;</span><span class="p">][</span><span class="s">&#39;fuelUsed&#39;</span><span class="p">])</span> <span class="o">+</span> <span class="s">&#39; litres&#39;</span>
<span class="k">print</span> <span class="s">&quot;Total Time &quot;</span> <span class="o">+</span> <span class="nb">str</span><span class="p">(</span><span class="n">data_basic</span><span class="p">[</span><span class="s">&#39;route&#39;</span><span class="p">][</span><span class="s">&#39;formattedTime&#39;</span><span class="p">])</span>
<span class="k">print</span> <span class="s">&#39;Map &#39;</span> <span class="o">+</span> <span class="n">map_basic</span>
<span class="k">print</span> <span class="s">&#39;&#39;</span>
<span class="k">print</span> <span class="s">&#39;&gt;-------------&lt;&#39;</span>
<span class="k">print</span> <span class="s">&#39;Optimised Route&#39;</span>
<span class="k">print</span> <span class="s">&#39;&gt;-------------&lt;&#39;</span>
<span class="c"># Show optimised order</span>
<span class="k">for</span> <span class="n">x</span> <span class="ow">in</span> <span class="n">data</span><span class="p">[</span><span class="s">&#39;route&#39;</span><span class="p">][</span><span class="s">&#39;locationSequence&#39;</span><span class="p">]:</span>
<span class="k">print</span> <span class="n">route</span><span class="p">[</span><span class="s">&#39;locations&#39;</span><span class="p">][</span><span class="nb">int</span><span class="p">(</span><span class="n">x</span><span class="p">)]</span>
<span class="k">print</span> <span class="s">&quot;Total Distance &quot;</span> <span class="o">+</span> <span class="nb">str</span><span class="p">(</span><span class="n">data</span><span class="p">[</span><span class="s">&#39;route&#39;</span><span class="p">][</span><span class="s">&#39;distance&#39;</span><span class="p">])</span> <span class="o">+</span> <span class="s">&#39; miles&#39;</span>
<span class="k">print</span> <span class="s">&quot;Total Fuel Used &quot;</span> <span class="o">+</span> <span class="nb">str</span><span class="p">(</span><span class="n">data</span><span class="p">[</span><span class="s">&#39;route&#39;</span><span class="p">][</span><span class="s">&#39;fuelUsed&#39;</span><span class="p">])</span> <span class="o">+</span> <span class="s">&#39; litres&#39;</span>
<span class="k">print</span> <span class="s">&quot;Total Time &quot;</span> <span class="o">+</span> <span class="nb">str</span><span class="p">(</span><span class="n">data</span><span class="p">[</span><span class="s">&#39;route&#39;</span><span class="p">][</span><span class="s">&#39;formattedTime&#39;</span><span class="p">])</span>
<span class="k">print</span> <span class="s">&#39;Map &#39;</span> <span class="o">+</span> <span class="n">map_optimised</span></code></pre></div>
<p>The results shown below, you can use their free GeoCoding service to take address strings and turn them
into coordinates. </p>
<div class="highlight"><pre><code class="language-bash" data-lang="bash">luke@xbu64:~/python<span class="nv">$ </span>python open_tsp.py
&gt;------------&lt;
Original Route
&gt;------------&lt;
51.524410145
51.5449591514
51.5206184283
51.5285060966
Total Distance 8.482 miles
Total Fuel Used 0.52 litres
Total Time 00:25:13
Map http://open.mapquestapi.com/staticmap/v4/getmap?key<span class="o">=</span>YOURKEY<span class="p">&amp;</span><span class="nv">bestfit</span><span class="o">=</span>51.547508,-0.201385,51.519817,-0.127893<span class="p">&amp;</span><span class="nv">shape</span><span class="o">=</span>51.524410145,-0.129892736523,51.5449591514,-0.165188854492,51.5206184283,-0.149547964184,51.5285060966,-0.201705257078<span class="p">&amp;</span><span class="nv">size</span><span class="o">=</span>600,600<span class="p">&amp;</span><span class="nb">type</span><span class="o">=</span>map<span class="p">&amp;</span><span class="nv">imagetype</span><span class="o">=</span>jpeg
&gt;-------------&lt;
Optimised Route
&gt;-------------&lt;
<span class="o">{</span><span class="s1">&#39;latLng&#39;</span>: <span class="o">{</span><span class="s1">&#39;lat&#39;</span>: 51.524410144966154, <span class="s1">&#39;lng&#39;</span>: -0.12989273652335526<span class="o">}}</span>
<span class="o">{</span><span class="s1">&#39;latLng&#39;</span>: <span class="o">{</span><span class="s1">&#39;lat&#39;</span>: 51.52061842826141, <span class="s1">&#39;lng&#39;</span>: -0.1495479641837033<span class="o">}}</span>
<span class="o">{</span><span class="s1">&#39;latLng&#39;</span>: <span class="o">{</span><span class="s1">&#39;lat&#39;</span>: 51.54495915136182, <span class="s1">&#39;lng&#39;</span>: -0.16518885449221493<span class="o">}}</span>
<span class="o">{</span><span class="s1">&#39;latLng&#39;</span>: <span class="o">{</span><span class="s1">&#39;lat&#39;</span>: 51.52850609658769, <span class="s1">&#39;lng&#39;</span>: -0.20170525707760403<span class="o">}}</span>
Total Distance 7.116 miles
Total Fuel Used 0.43 litres
Total Time 00:21:37
Map http://open.mapquestapi.com/staticmap/v4/getmap?key<span class="o">=</span>YOURKEY<span class="p">&amp;</span><span class="nv">bestfit</span><span class="o">=</span>51.547508,-0.201385,51.519962,-0.130076<span class="p">&amp;</span><span class="nv">shape</span><span class="o">=</span>51.524410145,-0.129892736523,51.5206184283,-0.149547964184,51.5449591514,-0.165188854492,51.5285060966,-0.201705257078<span class="p">&amp;</span><span class="nv">size</span><span class="o">=</span>600,600<span class="p">&amp;</span><span class="nb">type</span><span class="o">=</span>map<span class="p">&amp;</span><span class="nv">imagetype</span><span class="o">=</span>jpeg</code></pre></div>
<p>The above code also generates a static map so you can see a visual representation of the route you
just solved :)</p>
<p>Need any help with a project that uses Maps, Geo-positional or GIS? Hit me up luke@pumalo.org</p>
<p><a href="http://crooksey.github.io/python-tsp/">Python TSP</a> was originally published by Luke Crooks at <a href="http://crooksey.github.io">Luke's Blog</a> on August 02, 2014.</p></content>
</entry>
<entry>
<title type="html"><![CDATA[How to scale a web application with one docker container]]></title>
<link rel="alternate" type="text/html" href="http://crooksey.github.io/docker-jenkins/" />
<id>http://crooksey.github.io/docker-jenkins</id>
<updated>2014-07-27T00:00:00-00:00</updated>
<published>2014-07-27T00:00:00+01:00</published>
<author>
<name>Luke Crooks</name>
<uri>http://crooksey.github.io</uri>
<email>luke@pumalo.org</email>
</author>
<content type="html"><h3 id="the-problem">The problem</h3>
<p>Anyone who has worked in/for a startup will know you aren’t just (your job role) you job is
(your job role) + everything else tech related. Back end programmers are writing javascript,
front end developers are configuring nginx, you get the jist.</p>
<p>I am part of a new startup <a href="http://getwork2day.com">GetWork2Day</a> and currently we have 10 people
working for the startup, I am the only one who is doing anything technical, which means I am responsible
for: Backend, front end, database, server architecture and everything else. Our current server configuration
is a basic Ubuntu server, with nginx and everything else needed, its behind a load balancer to handle scaling
and it does the job. However, I don’t want to manage a server full time and when we get more staff, I don’t want people have no server experience, poking around with my server, ill come back from a few days off and
can foresee the problems already. I need a service where I can just push code to github and it deploys for me.</p>
<h3 id="wait-what-about">Wait, what about….</h3>
<p><a href="http://docs.aws.amazon.com/elasticbeanstalk/latest/dg/create_deploy_Python_flask.html">Elastic Beanstalk</a>, <a href="https://www.heroku.com/">Heroku</a>, <a href="https://www.digitalocean.com/">Digital Ocean</a>.</p>
<p>Well all of thes are great, they really are, but running things outside of the normal remit causes problems, <a href="http://getwork2day.com">GetWork2Day</a> uses very unique and obscure libraries for some parts,
so I wanted more control over the deployment, also we can’t afford to run on any of the three major players.</p>
<h3 id="so-how-can-i-homebrew-this">So how can I homebrew this?</h3>
<p>Well, I first looked at <a href="http://puppetlabs.com/puppet/what-is-puppet">Puppet</a> and <a href="https://www.atlassian.com/software/bamboo">Bamboo</a>. Both of these are great bits of software, but they are overpriced (for us) and they also work on a fixed IP address (from what I was reading) to connect to your application. If you’re application is behind a load balancer, you IP is always random and so I needed something where my application reached out and connected to my git server, downloaded the source code (if there was new commits), rebuilt it and then re-launched the application. This led me to jenkins.</p>
<h3 id="why-jenkins">Why Jenkins?</h3>
<p>Jenkins is free and open source and with its great array of plugins, I can make it do whatever I want (within reason). What I did was create a basic jenkins setup with a git project, set it to check for new git commits every hour, if there was new code to download, jenkins would pull the changes, copy the code to the running location with rsync and then restart the local application server. Meaning now all I have to to is push code to the master branch (or however you configure jenkins) then, jenkins will automatically sort everything out for the re-deployment. Even if my application had scaled across 20 servers, each server would stay up-to-date with the latest code, without any further integration from me (or anyone). Jenkins can also run a custom command or script after a job is cloned/build completed successfully, mine is:</p>
<div class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="c"># After successfull build</span>
<span class="c"># rsync the build dir to the app dir</span>
rsync -av --delete --delete-excluded --exclude <span class="s2">&quot;.pyc&quot;</span> /var/lib/jenkins/jobs/g1/workspace/ /home/app/
<span class="c"># restart supervisor</span>
supervisorctl restart tangable:</code></pre></div>
<h3 id="what-does-this-have-to-do-with-docker">What does this have to do with Docker?</h3>
<p>Well, with the current setup, I still have a server to maintain and I didn’t wan’t that, I wanted this entire server bundled into one build script, and now I am going to tell you how I built this entire framework with a script in under 40 lines, with the brilliant <a href="https://docs.docker.com/reference/builder/">Dockerfile</a>.</p>
<h3 id="the-basics">The basics</h3>
<p>Ok, so lets start:</p>
<div class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="c"># Set base image</span>
FROM ubuntu
MAINTAINER Luke Crooks <span class="s2">&quot;luke@pumalo.org&quot;</span></code></pre></div>
<p>Now, we have the basics set, we can install some packages:</p>
<div class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="c">### APT SECTION ###</span>
<span class="c"># Update aptitude with new repo</span>
RUN apt-get update
<span class="c"># Install software </span>
RUN apt-get install -y git nginx supervisor wget python-virtualenv curl
RUN apt-get install -y python-dev postgresql-server-dev-9.3 rsync
<span class="c"># Install libaries</span>
RUN apt-get install -y libjpeg62 libjpeg62-dev zlib1g-dev libpng12-0
RUN apt-get install -y libtiff5 libgif4 libgeos-dev</code></pre></div>
<p>Now we have the basics we need, we can go ahead and start configuring.
In the same folder that you are building your Dockerfile, you will also need the copy of your ssh key (used at github or bitbucket to verify your credentials).</p>
<div class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="c">### SSH SECTION ###</span>
<span class="c"># Make ssh dir</span>
RUN mkdir /root/.ssh/
<span class="c"># Copy over private key, and set permissions</span>
ADD id_rsa /root/.ssh/id_rsa
RUN chmod <span class="m">0600</span> /root/.ssh/id_rsa
<span class="c"># Add bitbuckets key to known_hosts</span>
RUN ssh-keyscan bitbucket.org &gt;&gt; /root/.ssh/known_hosts</code></pre></div>
<p>Now we can clone our private repositories, one for all our docker configuration, the other for our application:</p>
<div class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="c"># Clone the conf files into the docker container</span>
RUN git clone git@bitbucket.org:Username/docker-conf.git /home/docker-conf
<span class="c"># Clone the repo locally</span>
RUN git clone git@bitbucket.org:Username/application.git /home/app</code></pre></div>
<p>Awesome, now we have all the configuration filed we need, we can remove the default files and copy
your custom configuration files to where they need to be:</p>
<div class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="c"># remove default nginx configs</span>
RUN rm /etc/nginx/sites-available/default <span class="o">&amp;&amp;</span> rm /etc/nginx/sites-enabled/default
<span class="c"># Copy new settings</span>
RUN cp /home/docker-conf/configs/nginx/nginx.conf /etc/nginx/nginx.conf
RUN cp /home/docker-conf/configs/nginx/sites-available/site.conf /etc/nginx/sites-available/
<span class="c"># Enable the new site with a symbolic link</span>
RUN ln -s /etc/nginx/sites-available/site.conf /etc/nginx/sites-enabled/site.conf</code></pre></div>
<p>Next up, we are going to install Jenkins and copy over our local config files:</p>
<div class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="c"># Install Jenkins</span>
RUN wget -q -O - http://pkg.jenkins-ci.org/debian/jenkins-ci.org.key <span class="p">|</span> sudo apt-key add -
RUN sh -c <span class="s1">&#39;echo deb http://pkg.jenkins-ci.org/debian binary/ &gt; /etc/apt/sources.list.d/jenkins.list&#39;</span>
RUN apt-get update
RUN apt-get install -y jenkins
<span class="c"># Copy over configs</span>
RUN rm /etc/default/jenkins
RUN cp /home/docker-conf/jenkins/default /etc/default/jenkins
RUN chmod +x /home/docker-conf/jenkins/after-build
RUN rm -rf /var/lib/jenkins/*
RUN cp -a /home/docker-conf/jenkins/root/* /var/lib/jenkins/
<span class="c"># Reset file permissions after moving files</span>
RUN chown jenkins:jenkins -R /var/lib/jenkins/
RUN chmod <span class="m">775</span> -R /var/lib/jenkins/
<span class="c"># Give jenkins permissions to manage the supervisor service</span>
RUN <span class="nb">echo</span> <span class="s2">&quot;jenkins ALL = NOPASSWD: ALL&quot;</span> &gt;&gt; /etc/sudoers</code></pre></div>
<p>Now this is fairly python specific, but the same principles will apply for other languages, we need to create an environment to install application dependencies, and install the dependencies from the requirements file:</p>
<div class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="c"># Install the python requirements from requirements.txt</span>
RUN virtualenv --no-site-packages <span class="s2">&quot;/home/env&quot;</span>
RUN /home/env/bin/pip install -r /home/app/requirements.txt
<span class="c"># Lastly install the app into the virtualenv</span>
RUN /home/env/bin/python2.7 /home/app/setup.py install</code></pre></div>
<h3 id="not-normal-docker-behaviour">Not normal docker behaviour</h3>
<p>Usually a docker container is meant to one run process/command, e.g just an application, or a web server or whatever you want. But, if you use a program such as <a href="http://supervisord.org/">Supervisor</a> you can configure it to run multiple programs, e.g. in old money, you can run an entire LAMP stack inside a docker container. Explaining how to configure Supervisor is outside the scope of this post, but check out their website for some guides.</p>
<div class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="c">### SUPERVISOR ###</span>
<span class="c"># Now copy the supervisor configs over</span>
RUN cp /home/docker-conf/configs/supervisor/supervisord.conf /etc/supervisor/supervisord.conf
RUN supervisord -c /etc/supervisor/supervisord.conf
<span class="c"># Web app configs</span>
RUN cp /home/docker-conf/configs/supervisor/site.conf /etc/supervisor/conf.d/app.conf
<span class="c"># Nginx configs</span>
RUN cp /home/docker-conf/configs/supervisor/nginx.conf /etc/supervisor/conf.d/nginx.conf
<span class="c"># Jenkins confgs</span>
RUN cp /home/docker-conf/configs/supervisor/jenkins.conf /etc/supervisor/conf.d/jenkins.conf</code></pre></div>
<p>Now docker is running our application, nginx and jenkins, all we need to do is expose port 80 and tell docker to run the supervisor command on start:</p>
<div class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="c"># Expose the python app to world, note jenkins will not be accessible as we</span>
<span class="c"># are only expising port 80, not 8080 which is where jenkins runs.</span>
EXPOSE 80
CMD <span class="o">[</span><span class="s2">&quot;supervisord&quot;</span>, <span class="s2">&quot;-n&quot;</span><span class="o">]</span></code></pre></div>
<h3 id="in-summary">In Summary</h3>
<p>So in summary, we have just created one script that builds an Ubuntu image, installs all dependencies, configures the build process, manages the application and web server and takes care of everything else.
It is also capable of running behind a load balancer, so if your docker container was scaled into 20 different instances, they would all stay up to date, without having to configure a management application to deploy new code to different IP addresses. </p>
<h3 id="note">Note</h3>
<p>This is not usually the norm for docker deployment, but it works and it works well. Use at your own peril! No databases configuration is shown here as we are using a remote RDS amazon database. You should never host a databased behind your own load balancer as they are destroyed and created frequently. </p>
<p><a href="http://crooksey.github.io/docker-jenkins/">How to scale a web application with one docker container</a> was originally published by Luke Crooks at <a href="http://crooksey.github.io">Luke's Blog</a> on July 27, 2014.</p></content>
</entry>
<entry>
<title type="html"><![CDATA[Why I stopped using Django]]></title>
<link rel="alternate" type="text/html" href="http://crooksey.github.io/why-i-stopped-django/" />
<id>http://crooksey.github.io/why-i-stopped-django</id>
<updated>2014-07-21T00:00:00-00:00</updated>
<published>2014-07-21T00:00:00+01:00</published>
<author>
<name>Luke Crooks</name>
<uri>http://crooksey.github.io</uri>
<email>luke@pumalo.org</email>
</author>
<content type="html"><p>This is my own personal opinion, many people love Django, I however am not one of those users.</p>
<h3 id="background">Background</h3>
<p>I was an avid Django user for at least 18 months, using and contributing to several projects. At first everything was rosy, then I experienced upgrading from 1.25 to 1.3.</p>
<p>When I first started with Django it was more due to the frustration with Ruby on Rails, I had several projects using Spree and Radiant CMS. I had been using Ruby/Rails for around a year, doing fairly small projects, it was great for what I needed, but I never really felt comfortable with it. I honestly can’t remember what versions of Ruby I was using, but I think it was 1.8. I had a spree site, linked to a Radiant CMS, then spree upgraded to Ruby 1.9 and Radiant didn’t. To get the new features (and support) of spree, I ended up having to split my site across two applications, this ran for about 6 months, until I had given up fighting, and looked at writing a new platform.</p>
<p>I eventually migrated to python (which I had used in college) and re-wrote my setup for Django. I used the brilliant Django-cms, I wrote a custom application adapted from Arkestra. This was a time when the CMS and Arkestra were both under rapid development, but I loved both projects and even helped contribute.</p>
<h3 id="the-honeymoon-period">The honeymoon period</h3>
<p>Everything was great, then django 1.3 became a final release and apps started moving, It wasn’t really a problem but I remember migrating from django admin media to static files and various other code changes, not something I really enjoyed. Then everything was fine again, until 1.4 came out. This whole process really got on my nerves and changing code every version upgrade was a pain, especially when different libraries updated at different paces. Which basically means, if you want to use code that others have written for django, expect to be updating a lot of it as you migrate and projects get left behind.</p>
<p>I ended up re-writing a lot of third party apps, just so I could migrate my CMS to the the latest version.</p>
<h3 id="and-then-i-needed-more">And then I needed more</h3>
<p>I was then approached to write a custom CRM system for a company, whilst evaluating the best options, I stumbled across Pyramid + SQLAlchemy. It was love at first sight, it was writing pure python with a few web renderers. I could see that major version upgrades wouldn’t be a problem, as I was just writing python.</p>
<p>SQLAlchemy is just a god send, words cannot describe how awesome this package is.</p>
<p>Coding with SQLAlchemy and Pyramid feels brilliant after coming from Django, there are literally no limitations, write what you want quickly and easily and knowing that I am not going to have a headache when a new major version comes out, is even better. Its been 18months with Pyramid now and the documentation and the community are brilliant.</p>
<p>If you have a rainy day to have a go with Pyramid, do it. You won’t regret it.</p>
<p><a href="http://crooksey.github.io/why-i-stopped-django/">Why I stopped using Django</a> was originally published by Luke Crooks at <a href="http://crooksey.github.io">Luke's Blog</a> on July 21, 2014.</p></content>
</entry>
<entry>
<title type="html"><![CDATA[PostgreSQL cheat sheet]]></title>
<link rel="alternate" type="text/html" href="http://crooksey.github.io/psql-cheats/" />
<id>http://crooksey.github.io/psql-cheats</id>
<updated>2014-07-20T00:00:00-00:00</updated>
<published>2014-07-20T00:00:00+01:00</published>
<author>
<name>Luke Crooks</name>
<uri>http://crooksey.github.io</uri>
<email>luke@pumalo.org</email>
</author>
<content type="html"><p>This post will be updated with any cool cheats/commands that I use on a daily basis, that may help others out.</p>
<h3 id="make-a-local-user-superuser">Make a local user, superuser</h3>
<p>Everytime I work on a new machine/environment I need to be able to create and delete databases as a local
user quickly and easily, I am by no means a PostgreSQL expert, but I need to be able to create databases,
ready for web application development. So the below code should get you started.</p>
<p>First, login as the potgres user, and access the psql shell:</p>
<div class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="nv">$ </span>sudo su - postgres
<span class="nv">$ </span>psql</code></pre></div>
<p>Now, we need to create a new postgres user, for the sake of this guide, lets assume the name
“luke” and obviously replace ‘somepassword’ with your own, secure password.</p>
<div class="highlight"><pre><code class="language-sql" data-lang="sql"><span class="n">postgres</span><span class="o">=#</span> <span class="k">create</span> <span class="k">user</span> <span class="n">luke</span> <span class="k">with</span> <span class="n">password</span> <span class="s1">&#39;somepassword&#39;</span><span class="p">;</span>
<span class="k">CREATE</span> <span class="k">ROLE</span></code></pre></div>
<p>Now we have the user setup, lets give him ‘Super User’ credentials:</p>
<div class="highlight"><pre><code class="language-sql" data-lang="sql"><span class="n">postgres</span><span class="o">=#</span> <span class="k">ALTER</span> <span class="k">USER</span> <span class="n">luke</span> <span class="k">with</span> <span class="n">SUPERUSER</span><span class="p">;</span>
<span class="k">ALTER</span> <span class="k">ROLE</span></code></pre></div>
<p>Now, open up a new terminal and you should be able to create and delete databases:</p>
<div class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="nv">$ </span>createdb staff
<span class="nv">$ </span>dropdb staff</code></pre></div>
<p><a href="http://crooksey.github.io/psql-cheats/">PostgreSQL cheat sheet</a> was originally published by Luke Crooks at <a href="http://crooksey.github.io">Luke's Blog</a> on July 20, 2014.</p></content>
</entry>
</feed>