-
Notifications
You must be signed in to change notification settings - Fork 1
/
Lets-Make-Predictions.html
646 lines (634 loc) · 44.7 KB
/
Lets-Make-Predictions.html
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
<?xml version="1.0" encoding="UTF-8" ?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
<link rel="icon" type="image/png" href="./images/favicon-32x32.png" sizes="32x32" />
<link rel="icon" type="image/png" href="./images/favicon-16x16.png" sizes="16x16" />
<title>Let's Make Predictions - SPK's Rationality Essays</title>
<link rel="stylesheet" type="text/css" href="./css/default.css" />
<link rel="stylesheet" type="text/css" href="./css/highlight.css" />
<!-- <script src="http://ajax.googleapis.com/ajax/libs/jquery/1.10.2/jquery.min.js"></script> -->
<!-- <script type="text/javascript" src="/js/header-links.js"></script> -->
<script type="text/javascript" src="http://cdn.mathjax.org/mathjax/latest/MathJax.js?config=TeX-AMS-MML_HTMLorMML"></script>
<link href="atom.xml" type="application/atom+xml" rel="alternate" title="Sitewide ATOM/RSS Feed" />
<!-- Google Analytics stuff -->
<!-- Google tag (gtag.js) -->
<script async src="https://www.googletagmanager.com/gtag/js?id=G-DEWF2J5BG8"></script>
<script>
window.dataLayer = window.dataLayer || [];
function gtag(){dataLayer.push(arguments);}
gtag('js', new Date());
gtag('config', 'G-DEWF2J5BG8');
</script>
<script type="text/javascript" src="https://fast.fonts.net/jsapi/f7f47a40-b25b-44ee-9f9c-cfdfc8bb2741.js"></script>
</head>
<body>
<div id="header">
<div id="logo">
<a href="./">SPK's Rationality Essays</a>
</div>
<div id="navigation">
<a href="./">Home</a>
<a href="./notes.html">Notes</a>
<!-- <a href="/about.html">About</a> -->
<a href="./archive.html">Archive</a>
<a href="./atom.xml" type="application/atom+xml" rel="alternate" title="Sitewide ATOM/RSS Feed">RSS</a>
</div>
</div>
<div id="content">
<h1 id="post-title">Let's Make Predictions</h1>
<!-- <center><img src="https://fbcdn-sphotos-d-a.akamaihd.net/hphotos-ak-prn1/t31.0-8/p600x600/10257116_10202295769100492_2438594605053717342_o.jpg" height="400" width="300" class="sujeet-pic" alt="Sujeet pic" /></center> -->
<p><strong>Status</strong>: In Progress</p>
<hr />
<h1 id="the-great-predictions-experiment">The Great Predictions Experiment</h1>
<p>I need to make more and more predictions.</p>
<h1 id="why">Why?</h1>
<p>Cos your Prediction Score determines your success in life completely.</p>
<h1 id="how">How?</h1>
<p>How about this? I make at least 10 predictions every day, with probability estimates.</p>
<p>This is just Version 1.0. We’ll do this for a week and take it from there.</p>
<h1 id="wait-a-minute.">Wait a minute.</h1>
<p>Aren’t Measurements supposed to Matter? Shouldn’t there be some Decision to which this Measurement matters? What’s the point of making predictions (a form of measurement) in the air?</p>
<p>Wait. Are Predictions a form of Measurement?</p>
<p>A measurement is the quantitative reduction of uncertainty about something.</p>
<p>A prediction is a probability distribution over the possible outcomes in some event, as given by some hypothesis.</p>
<p>Why do I want to record my predictions and then get feedback on them?</p>
<h1 id="i-want-to-test-my-hypotheses">I want to test my hypotheses</h1>
<p>A button affords pushing. A hypothesis affords testing.</p>
<p>One way or another, I will get to know something new about my hypotheses. I will be able to update them. Bad hypotheses get thrown out. Good ones get upgraded.</p>
<p>By default, we never make predictions. We never test our hypotheses. And so, we rarely ever get direct evidence about our hypotheses. We can rest safely knowing that our cherished (and most probably, false) hypothesis can never be falsified.</p>
<p>Also, costly and hard-to-test predictions are useless to us.</p>
<p>So: Make easily testable predictions.</p>
<h1 id="i-want-to-calibrate-my-estimates">I want to calibrate my estimates</h1>
<p>The previous case was about making predictions based on your hypotheses and, then, updating belief-levels of the hypotheses based on the results.</p>
<p>This is about <em>making</em> predictions in the first place from hypotheses.</p>
<blockquote>
<p>Yo, won’t it be clear what predictions a hypothesis makes?</p>
</blockquote>
<p>It would, ideally. But our head is hardly an ideal place. We keep hypotheses around implicitly. We don’t even know exactly which hypotheses we have, let alone what our probability estimates for them are.</p>
<p>Our overall prediction comes out from the predictions of all our hypotheses weighted by our beliefs in them.</p>
<p>But we neither know the quantitative predictions made by each hypothesis nor our belief-level for the hypothesis.</p>
<p>Then, how are we justified in pulling predictions out of our ass?</p>
<p>.</p>
<p>.</p>
<p>.</p>
<p>.</p>
<p>(Rhetorical)</p>
<h1 id="feeling-quantitatively-and-reasoning-quantitatively">Feeling quantitatively and Reasoning quantitatively</h1>
<p>Ok. So shit’s bad right now. How do we fix things?</p>
<p>We need to calibrate our probability estimates. i.e., we need to know how to translate implicit belief-levels in our minds into actual probabilities.</p>
<p>i.e., intuitive feeling (“I’m pretty sure”) -> a quantitative probability (95%).</p>
<p>We need to know the hypotheses we have, know what quantitative predictions they make, know how much we believe in each hypothesis, and aggregate all that to come up with our actual predictions. That is the ideal reasoning process.</p>
<p>i.e., make our reasoning process more explicit and quantitative.</p>
<p>Alternatively, we could skip all that quantitative reasoning thing and just go with our final gut feeling, the <em>implicit prediction</em> in our head, and translate that into a quantitative prediction.</p>
<h1 id="wrong-translation-vs-wrong-beliefs">Wrong Translation vs Wrong Beliefs</h1>
<p>I was finding it difficult to separate the two.</p>
<p>You could have the wrong belief-levels about some hypotheses - an honest mistaken belief about the world - and thus make a wrong prediction. You might believe Kohli is not playing in the match (i.e., you have low belief-level in the hypothesis that Kohli is playing) and predict that India won’t chase the total down, whereas he <em>is</em> playing and scores a century.</p>
<p>Or, you could have correct belief-levels about your hypotheses but go wrong in translating the belief-levels into probability - and thus make a wrong prediction. e.g., “not too sure whether it is A or B” into 30%-70% when it should be 50%-50%.</p>
<p>Whether it is a wrong translation or wrong belief, in both cases, the <em>predictions</em> you make will be wrong. How will you know where you went wrong? Was your hypothesis wrong or was your translation wrong?</p>
<p>Where do these two hypotheses (Wrong beliefs vs Wrong translation) differ in their predictions?</p>
<h2 id="will-they-lead-to-different-prediction-results">Will they lead to different prediction results?</h2>
<p>Right belief + Right translation: Your explicit predictions will be correct.</p>
<p>Right belief + Wrong translation: Your explicit predictions will be wrong.</p>
<p>Wrong belief + Right translation: Your explicit predictions will be wrong.</p>
<p>Wrong belief + Wrong translation: Your explicit predictions may be right or wrong.</p>
<p>So, all that you can tell from wrong predictions is that you <em>don’t</em> have both the right belief and the right translation (else, the prediction would definitely have been right). It doesn’t tell you whether it was your belief that was wrong or your translation or both.</p>
<p>Note, even if you get the right predictions, it could still mean that your wrong belief and wrong translation have cancelled out luckily. But, that is highly unlikely to happen, right? So, it’s strong evidence that you have both the right belief and the right translation.</p>
<h2 id="will-they-lead-to-different-levels-of-surprise">Will they lead to different levels of surprise?</h2>
<p>I’m assuming that your mind makes you feel surprise <em>only</em> when things go against your implicit prediction. Else, you’re not surprised. Everything feels normal.</p>
<p>Even if something is against your prediction and should surprise you, you may still not notice it. That’s fine. Humans are dumb. But, when you <em>do</em> feel surprised, it is because it countered your implicit prediction.</p>
<p><strong>Note</strong>: Your explicit predictions (the translation of your belief-levels to probabilities) have no effect on your surprise levels. The surprise is about the stuff in your brain. The numbers you wrote down have no effect there, in general.</p>
<p>Right belief: You won’t feel surprised at all, cos <em>all</em> your implicit predictions will come true. (say, when you are a technical expert in a specific domain)</p>
<p>Wrong belief: If you feel surprise at all, it must be because of wrong beliefs. Wrong belief means that your implicit prediction was different from the correct prediction.</p>
<p>It is possible to have Wrong beliefs and still not be surprised by anything you see that contradicts your beliefs (see: religious people). But, if you do get surprised, it has to come from Wrong beliefs.</p>
<p>I think that <em>Noticing Surprises</em> is a skill and can be trained. If you force yourself to look actively for surprises, if you consciously bring the outcome to your awareness and test it against your implicit prediction, I think you will know whether you are surprised or not. In most cases, we aren’t even aware that our model of the world has made a prediction and so we let surprises pass under our radar. But since this whole discussion is about <em>explicit</em> predictions, we can always do a conscious mental check and ask if we are surprised.</p>
<p>In <em>this</em> case, surprise becomes proper strong evidence. If you are <em>surprised</em> by the outcome, then your belief was wrong. End of story. If you aren’t <em>surprised</em>, then your belief was right.</p>
<p>(TODO Question: How do judge the level of surprise (especially in cases where the outcomes have a range)? Example: what will the final score be? 250 runs? 270? 300? 280-290 runs? How surprised will you be if you say 280-290 and you get 270? What if you get 324?)</p>
<h2 id="what-can-we-say-about-our-translation-accuracy">What can we say about our translation accuracy?</h2>
<p><strong>Cool</strong>! We have a differing prediction between Right beliefs and Wrong beliefs. And, since it is independent of our translation accuracy (and thus our explicit predictions), we can use this to help solve the mystery of whether a wrong prediction was caused by a Wrong belief or a Wrong translation of belief-levels.</p>
<p>Now, what does this tell us about explicit predictions (and thus our translation accuracy)?</p>
<p>Say we got a wrong explicit prediction. We don’t know if it was cos of a wrong belief or cos of wrong translation.</p>
<p>So, we check our surprise level. If we did get surprised, it means that our belief was wrong. If not, our belief was right.</p>
<p>Now, let’s try out the different possibilities:</p>
<h4 id="right-belief-confirmed">Right belief (confirmed)</h4>
<p>Right Translation: Right explicit prediction</p>
<p>Wrong Translation: Wrong explicit prediction</p>
<h4 id="wrong-belief-confirmed">Wrong belief (confirmed)</h4>
<p>Right Translation: Wrong explicit prediction</p>
<p>Wrong Translation: Most probably Wrong explicit prediction (cos it’s unlikely that things cancel out the right way)</p>
<hr />
<p>When you have a right belief (as confirmed by the surprise test), the correctness of the explicit prediction tells you <strong>perfectly</strong> whether your translation of belief-levels was right or wrong. Correct explicit prediction means correct translation. Wrong explicit prediction means wrong translation.</p>
<p>When you have a wrong belief (as confirmed by the surprise test), the explicit prediction result tells you <strong>pretty much nothing</strong>. In fact, in the unlikely case that you are surprised but your explicit prediction was <em>correct</em>, it means your <em>translation</em> was wrong. Weird, huh!</p>
<h1 id="calibration">Calibration</h1>
<p>I think they call “translation accuracy” as your level of calibration. Calibration is about correctly translating belief-levels into quantitative estimates. This includes correcting common errors that humans make in this process. The more calibrated you are, the better you will be at translating your belief-levels into quantitative estimates (explicit predictions).</p>
<p>If my above theory is true (about the surprise thing and the connection between implicit predictions and translation accuracy), then we now know some things about how to calibrate.</p>
<p>Most importantly, there is no point trying to calibrate using explicit predictions when you have Wrong beliefs about the thing. If the explicit prediction is wrong, then you can’t say anything about your translation, because your belief itself is wrong in the first place. If the explicit prediction is right, then, it means your translation was wrong - but this case will be pretty rare, IMHO.</p>
<p>So, you need to calibrate yourself (aka improve your translation accuracy) by making predictions only about things where you have <em>correct</em> beliefs.</p>
<p>And how would you know that? Well, you won’t be surprised by stuff that happens in that domain, even when you carefully take every event and consciously check whether you’re surprised or not. That is the only way to see if your implicit predictions are correct or not.</p>
<p>I’m not sure how this would work. Are there even <em>any</em> fields where I have correct beliefs?</p>
<p>Well, you don’t need perfectly correct beliefs. If you have reasonably correct beliefs, you will make mostly correct implicit predictions. So, you will have few surprises when you check consciously. That’s good enough. We can then go ahead and make explicit predictions and correct our translation processes.</p>
<p>Need to test out these hypotheses. Will do it soon.</p>
<hr />
<p><strong>Tentative solution</strong>: Make predictions about a bunch of events (could be trivia, college studies, whatever). Now, pick out those for which you were <em>not</em> surprised by the answer.</p>
<p>Those are predictions for which your implicit belief was right (or, at least, not wrong). Any error would be due to your translation. Fix it! Calibrate yourself.</p>
<h1 id="fixing-your-prediction-process">Fixing your Prediction Process</h1>
<p>We saw how we could calibrate our translation processes and get quantitative predictions that are close to our beliefs. Our explicit predictions will be wrong just as often as we are surprised. They will be closer and closer to our implicit predictions.</p>
<p>So far, it’s all about translating the final implicit prediction that our mind gives us into an explicit prediction.</p>
<p>Cool. But what if our implicit predictions are wrong in the first place?</p>
<p>How do we correct the prediction process by which our mind comes up with its implicit prediction?</p>
<p>The default process our mind uses is evidently not even close to the ideal way to do it.</p>
<p>How should we come to a prediction (ideally)?</p>
<p>Assuming you have a bunch of hypotheses in your head, and a bunch of belief-levels about those hypotheses, and a bunch of predictions that each hypothesis makes, what you need to do is take the average of all those predictions weighted by the belief-levels of the hypotheses. <em>That</em> is the final prediction, using the full force of your knowledge.</p>
<p>Which means, you need to be able to enumerate your hypotheses correctly. You need to translate their belief-levels accurately into probabilities. You need to enumerate the outcomes of an event and translate each hypothesis’ likelihood for each outcome into a probability. And, then, you need to calculate the weighted-average.</p>
<p>(Of course, all this is assuming that your hypotheses’ belief-levels and predictions are correct in the first place. God help you if you haven’t been updating properly on evidence, as per Bayes Theorem.)</p>
<p>So, these are the skills you need to improve to fix your prediction process. Converting belief-levels into probabilities calls again upon our old friend, calibration.</p>
<p>There’s lots more work to be done here. Will do it soon.</p>
<hr />
<p><strong>Note</strong>: Correct Beliefs doesn’t mean that you know what’s gonna happen with 100% certainty. It just means that when you say (implicitly) “X is gonna happen with 30% probability”, then the predicted outcome will happen around 30 times out of 100 such events.</p>
<h1 id="no-longer-afraid-of-the-outcome">No Longer Afraid of the Outcome</h1>
<p>When I make a prediction on PredictionBook, I feel kind of relieved. I no longer have the pressure of being right. If I’m right, my calibration score will increase. And if I’m wrong, it will decrease. But the damage is isolated to the calibration score. It doesn’t hurt me. When I think about the outcome of the event, I’m not worried anymore. In fact, I’m just curious to see what happens.</p>
<p>Somehow, making a prediction on PredictionBook distances me from the prediction. No longer is my self-worth tied to the correctness of the prediction. My self-worth is not at stake. If needed, I will pay a price in terms of my calibration score, but not my self-worth. In contrast, if I didn’t make a prediction on PredictionBook but just thought to myself what would happen, I would <em>desperately</em> want the prediction to come out my way, or risk feeling like a crappy thinker.</p>
<p>This might be the same force at work that turns off your feelings when you bring in another token of exchange. I remember hearing about a late pick-up fee at a child care centre leading to even later pick-ups by parents, because now parents didn’t have to pay in terms of their self-worth (“I’m such a bad parent”). They could just pay the fee and do what they really wanted to do anyway!</p>
<p>This makes those parents seem like bad people, but we can turn this to our advantage. Wherever we’re feeling like losers for no good reason, say when our predictions go wrong or when our project fails or somebody rejects us, we can avert that feeling by instead paying in some other token. Put a price on the failure and be willing to pay that price. Suddenly, you no longer have to pay an unspecified, below-the-counter sum in raw self-worth damage.</p>
<p>It’s like we expect to feel a certain amount of shame for something we did wrong. But if we pay for that failure in dollars or something else, then we no longer have to pay in shame. If your project failed, you’re already suffering the monetary cost. Why pay twice in shame? We can still walk with our heads held high, which is what we want. This means you don’t have prerequisites for worthiness - no matter how you’re doing, you feel worthy of love and belonging… even when you fail!</p>
<p>Corollary: If we avoid feedback because we fear being wrong, then using another token of payment can help lift the fear of failure. Maybe this is one reason why people don’t fear failure so much in games (according to Jane McGonigal) - you pay only in terms of your score, but not your self-worth.</p>
<p>So, make more predictions on PredictionBook. You’ll fear the outcome less and thus improve your calibration in more areas.</p>
<p>In other areas, look to pay in money instead of self-worth. For example, say you’re taking a risk career move. If you fail, you will pay in terms of wages lost and reduced earning capacity. That’s cool. But you will not be a loser. You are willing to pay the price in money, so you can keep your head unbowed.</p>
<p>Lesson: Express your doubts in predictions. Don’t keep ruminating about them. For example, I was worrying about whether I would get money back from a certain friend. I kept thinking about how things might go wrong and I would not see a rupee of it. But instead of worrying, I can just make a prediction about it - “I will get money back from friend by the end of this month.”</p>
<h1 id="flinching-away">Flinching away</h1>
<p>I found myself flinching away from making a pessimistic prediction. I wanted to record that I thought I <em>wouldn’t</em> practice for more than 3 hours tomorrow. I haven’t actually done that for a while now, so it’s completely reasonable to expect that. But putting it down on paper like that seems… almost like a betrayal. “Whose side are you on? How <em>dare</em> you suggest that I/you won’t do this very important thing?”, my mind seems to be asking me. (I’m afraid to do that even after writing this paragraph. It <em>really</em> feels like betrayal.)</p>
<p>I was asking the wrong question. I was asking “Will I do this?” and pat came the answer from my mind, like any true believer, “Of course you will!” Doubting my future prospects seems like something only an enemy would do. Hence the flinching away.</p>
<p>Instead, let’s reframe the question: “What will an external observer predict about my performance tomorrow, given my history over the last month?” This question is straightforward to answer. It forces me to put aside my own prejudices and look at the system dispassionately, exactly the way Eliezer’s <a href>“Beyond the Reach of God”</a> thought experiment forces you to put aside your beliefs about a just world and simply imagine the consequences of an action. I believe they call this the <a href>Outside View</a>.</p>
<h1 id="what-predictions-should-i-make">What Predictions Should I Make?</h1>
<p>Why do I want to make predictions? One reason is to calibrate myself. I need to know what <a href="http://rationalpoker.com/2011/04/21/this-is-what-5-feels-like/">5% probability</a> feels like.</p>
<p>Another reason is that I want myself to notice how poorly-specified my beliefs are. Most of the time, I don’t demand clear probabilities from them (“35% chance of India winning this match”). I don’t expect my beliefs to <a href="http://lesswrong.com/lw/i3/making_beliefs_pay_rent_in_anticipated_experiences/">pay rent</a> in anticipated experiences.</p>
<p>This means I don’t realize when my beliefs are wrong. For example, I may plan to write an essay by the end of the day because I believe that I can write a good essay within two hours. However, I haven’t tested that belief using my past experiences. When I sat down and actually looked at the numbers, I found that a representative 1400-word post I wrote recently took me 4.5 hours. And I consider that a small essay! Therefore, my hopes are going to be dashed when I actually sit down to write the essay within two hours. I’ll probably ruin my other plans by taking over their time and, worse, feel like a complete loser because I couldn’t do such a “reasonable” task.</p>
<p>Hypothesis: Most of our awful feelings arise from poor calibration.</p>
<p>For example, a while back, I felt guilty for stacking up a huge pile of empty cartons and trash in the corner of my kitchen. I felt like I was the only guy who was lazy enough to do something like this. Then I visited a friend’s house and found a mountain of trash that would put mine to shame. I was not alone!</p>
<p>In related news, one of shame researcher <a href="https://en.wikipedia.org/wiki/Brene_Brown">Brene Brown</a>’s books is called “I Thought It Was Just Me (but it isn’t)”. We seem to overestimate how alone we are in our shameful behaviour.</p>
<p>So, good calibration can help you set more realistic expectations. Instead of aiming to bang out a finished program within an hour, expect it to take a whole day. Realize that tasks take a finite, non-zero amount of time - brushing and flossing takes me around 10 minutes, washing the dishes takes around 10 minutes, and so on. When I make plans, I see only the big items (writing, studying, exercising) and fail to budget for the small fry.</p>
<p>Sadly, some inaccurate estimates cost me weeks and months and years instead of a few extra hours. This is nothing but the <a href="http://lesswrong.com/lw/jg/planning_fallacy/">planning fallacy</a> and it can be <a href="http://lesswrong.com/lw/jh/kahnemans_planning_anecdote/">very costly</a> indeed.</p>
<p>Lesson: Make predictions about all your planning estimates (hourly, daily, and weekly). If you plan to do three tasks, you’re implicitly estimating that you can fit them within your work hours. So, predict it on PB and see how you fare.</p>
<h1 id="making-better-predictions">Making Better Predictions</h1>
<p>Enumerate all the possibilities before you make your prediction. Right now, I feel like I’m focussing only on the outcomes I like, for example whether India loses one wicket or zero in the first ten overs. It doesn’t feel real to me that India could lose four wickets in the first ten overs. Or that it might rain! I don’t even push my mind to that extreme.</p>
<p>So, list all the outcomes and ask yourself what probabilities you put on each of them. Thinking in terms of binary predictions might be a problem. You tend to focus on one outcome (“India will lose one wicket”) and lump everything else under “other outcomes”. You thus fail to consider certain outcomes and their probabilities (a good spell by a bowler, great batting, or even just rain). Don’t make a prediction without listing all the outcomes (or at least three exclusive events).</p>
<h1 id="fallacy-of-even-odds">Fallacy of Even Odds</h1>
<p>A big related problem is that I look at the “two” outcomes (say, biotech company X will create a vaccine and it won’t) and decide that I don’t know much about biotech vaccines and so give it fair odds - 50% probability.</p>
<p>But that’s a false model! Framing it as “company X succeeds vs fails” makes me try to distribute 100% probability among these two outcomes, and thinking that I don’t know anything makes me distribute it evenly. But I <strong>do</strong> know something, which is that there are several biotech companies in the world, many of whom are trying to create that vaccine. So, to truly distribute my probability mass evenly, I must look at the number of biotech companies (say 8) and give them all equal probability (12.5%).</p>
<p>This, I think, is one of the biggest problems in becoming calibrated: we start with the wrong probability estimates! We put too much confidence in some hypotheses. It’s hard to recover from that because you don’t expect to be proven wrong (you’re confident) and so you don’t go looking at the experiments that you would (ideally) expect to surprise you.</p>
<p>I’m reminded of the 2-4-6 task in HPMOR. Hermione (and I and many others) were doomed right from the beginning simply by focussing too much on the “even plus two” rule. We didn’t ask about weird triplets like (-2432, 34, 3.34) because we fully expected them to be invalid and so didn’t want to waste a question. But we should. We’re just starting out - we haven’t got enough information to <a href="http://lesswrong.com/lw/19m/privileging_the_hypothesis/">locate the right hypothesis</a>. But to no avail. We’re doomed.</p>
<p>Prediction: If you get yourself accurate prior probabilities, then you can update your way to decent calibrated estimates.</p>
<h1 id="test-does-new-information-change-your-predictions">Test: Does new information change your predictions?</h1>
<p>Does your prediction change after you get more information? For example, I made my prediction about a cricket match without knowing who won the toss or what changes they made to the team. How can my prediction remain exactly the same after I find out? That would mean that India batting first had exactly the same odds of winning for me as India bowling first. Or that the opposition’s main bowler getting injured didn’t change their chances at all.</p>
<h1 id="features-i-would-like">Features I Would Like</h1>
<p>When I’m making a 60% prediction, it would be cool to see the last 20 predictions where I made 60% predictions. This way I can see what 60% felt like and what it should have felt like (perhaps more like 55% in some cases). Seeing the stats table for all my 60% predictions ever isn’t as helpful because many of them are from prehistoric times when I was prediction-illiterate. Also they don’t give me concrete instances to reflect on (“remember how uncertain you were about getting that refund?”) so I can’t make analogies using the past.</p>
<h1 id="current-pb-statistics">Current PB Statistics</h1>
<p>As it stands, the current PB stats don’t distinguish at all between a 70% prediction and a 79% prediction. They both fall in the 70% band and add the same amount to your accuracy if they come out right.</p>
<p>This is not good, because <a href="http://slatestarcodex.com/2016/02/04/book-review-superforecasting/">apparently</a> rounding the predictions of good forecasters to the nearest 5% or 10% removes a lot of their advantage.</p>
<h1 id="multiple-outcomes-are-okay">Multiple Outcomes are Okay</h1>
<p>I’m afraid to make a 20% prediction for one of five outcomes, even though that’s the uniform prior. This is because if I only register this one prediction, then I feel will be penalized. Is that true?</p>
<p>Remember that you’re testing your calibration, not discrimination. So, it’s ok if you just book one 20% prediction. If you really are well-calibrated, then for every ten such 20% predictions, eight will come wrong and your stats will be good. Have faith even if it goes wrong once in a while. This is the challenge - to be able to withstand temporary setbacks and trust the numbers.</p>
<h1 id="uncertainty---take-on-a-little-not-a-lot">Uncertainty - Take on a little, not a lot</h1>
<p>It’s easier to predict the winners for a series of cricket matches than to predict the exact outcome of each ball of a match. Why? Because you have lower uncertainty in the former.</p>
<p>Take 10 matches. Say there can only be two outcomes - win or loss. Your maximum uncertainty about those matches is 1 bit per match x 10 matches = 10 bits of uncertainty. This is because each match can have only two outcomes and so a maximum of 1 bit of uncertainty (when you put 50-50 on win and loss).</p>
<p>However, take just 10 balls in a single match. Now, there are several possible outcomes for each ball - the batsman could get out, hit a four, take a single, leave it alone, etc. This is excluding the finer details like how fast the ball was, where it pitched, where he hit the ball, who fielded, etc. Even if we just restrict it to 16 possible outcomes per ball, that gives us a maximum of 4 bits of uncertainty, and so 40 bits of uncertainty total about those 10 balls.</p>
<p>So, you have to do strictly more work in the second challenge than in the first (40 bits of uncertainty vs 10 bits) to get to the correct answer. And that was for just 10 balls! A full T20 match has 40 overs x 6 balls = 240 balls. With even just 16 possible outcomes per ball, you get a maximum of 960 bits of uncertainty. That is orders of magnitude more than the 10-match scenario. It’s the difference between looking for a diamond in a set of 2^10 = ~10^3 closed boxes and looking for a diamond in a set of 2^960 = ~10^288 closed boxes. Good luck trying to locate the diamond within the latter.</p>
<p>For example, if India plays Bangladesh in a cricket match, it’s hard to predict the outcome from ball to ball (like Raina getting out in the 9th over), but quite easy to predict that India is going to win the match. Don’t beat yourself up if you get unexpected outcomes in between - it’s too hard.</p>
<p>Lesson: If you want to make correct predictions, stay away from high-uncertainty situations. The deck is stacked against you there.</p>
<p>If you want to remain <em>calibrated</em> in such situations, stick to the uniform prior predictions, otherwise you’ll be surprised very often.</p>
<p>What if you want to make correct predictions? Go for places where you have low maximum uncertainty, like who will win the match (just two possible outcomes) or who will be the man of the match (only a few stand-out players) or what the run rate will be in the last five overs (there are only a few options - 5-6, 6-7, 7-8, etc.). But <em>not</em> how many runs India will score to a precision of 5 runs - there are lots of options and a lot can go wrong.</p>
<p>In short, don’t trust yourself to reduce a lot of uncertainty. Stick to small, manageable problems.</p>
<blockquote>
<p>I thought I could get away with thinking complicated thoughts myself, in the literary style of the complicated thoughts I read in science books, not realizing that correct complexity is only possible when every step is pinned down overwhelmingly. Today, one of the chief pieces of advice I give to aspiring young rationalists is “Do not attempt long chains of reasoning or complicated plans.”</p>
<p>– Eliezer Yudkowsky, <a href="http://lesswrong.com/lw/iy/my_wild_and_reckless_youth/">My Wild and Reckless Youth</a></p>
</blockquote>
<h1 id="how-to-judge-a-confidence-interval">How to Judge a Confidence Interval</h1>
<p>Let’s say you estimated that you would wake up tomorrow between 5am and 8am with probability 81%. When would you feel more confident in your prediction ability: if you woke up at 6:30am or 7:59am?</p>
<p>Somehow, you would feel that you barely got it right in the second case. Two more minutes and your whole 81% prediction would have gone wrong, whereas in the first case, you’d still be comfortably within range.</p>
<p>So, not all outcomes within a confidence interval are equal. Ideally, 81% probability mass distributed between 5am and 8am would give 81%/180 = 0.45% probability mass to each minute. Then, a wake-up time of 7:59am would count for just as much Bayesian evidence as 6:30am. But we treat them differently. It’s like we have an implicit bell curve within the confidence interval, with central points like 6:30am counting for more than peripheral points like 7:59am.</p>
<p>For a strong prediction, an outcome that barely falls within your confidence interval suggests that you were overconfident. For a weak prediction, an outcome that falls near the centre of your confidence interval suggests that you were overconfident.</p>
<p>(I think this holds even if you didn’t create an explicit confidence interval. Not sure how.)</p>
<h1 id="resources-for-making-predictions">Resources for Making Predictions</h1>
<h2 id="gwerns-advice-on-making-predictions">Gwern’s advice on Making Predictions</h2>
<p>He <a href="http://www.gwern.net/Prediction%20markets#how-i-make-predictions">talks</a> about 3 parts of making predictions: specifying the prediction, deciding the due-date, and assigning a probability to the prediction.</p>
<p>The prediction should be “a statement on an objective and easily checkable fact”, so that even an arch-enemy trying to poke holes would <em>have</em> to agree when you claim that your prediction was right (or wrong).</p>
<p>He goes into great detail on how to make calibrated probability estimates. Will delve into those after some time. I just want to get started now.</p>
<h2 id="douglas-hubbard---how-to-measure-anything">Douglas Hubbard - How to Measure Anything</h2>
<p>He gives lots of brilliant techniques for making calibrated predictions. Will go into all of them soon.</p>
<h2 id="calibration-1">Calibration</h2>
<ul>
<li><a href="http://lesswrong.com/r/lesswrong/lw/1f8/test_your_calibration/">Test your Calibration!</a></li>
</ul>
<h2 id="philip-tetlock">Philip Tetlock</h2>
<p>Main way to become better at predicting: keep score.</p>
<h1 id="ps-ideas">PS: Ideas</h1>
<ul>
<li>Wrong beliefs vs Wrong translation</li>
</ul>
<h1 id="my-pb-calibration-scores">My PB Calibration Scores</h1>
<p>February 13, 2016:</p>
<div class="statistics">
<table>
<thead>
<tr>
<th class="narrow">
Confidence
</th>
<th>
50%
</th>
<th>
60%
</th>
<th>
70%
</th>
<th>
80%
</th>
<th>
90%
</th>
<th>
100%
</th>
<th>
Total
</th>
</tr>
</thead>
<tbody>
<tr>
<th class="narrow">
Accuracy
</th>
<td>
30%
</td>
<td>
51%
</td>
<td>
67%
</td>
<td>
73%
</td>
<td>
85%
</td>
<td>
0%
</td>
<td>
</td>
</tr>
<tr>
<th class="narrow">
Sample Size
</th>
<td>
23
</td>
<td>
43
</td>
<td>
55
</td>
<td>
37
</td>
<td>
60
</td>
<td>
0
</td>
<td>
218
</td>
</tr>
</tbody>
</table>
<p><img src="http://chart.apis.google.com/chart?cht=s&chg=20,20&chds=50,100,0,100,0,60&chm=o,AAAAFF,0,-1,25,1|r,44FF44,0,0.49,0.51|D,FFCCCC,0,0,0.5,-1&chxt=x,x,y,y&chxr=0,50,100|1,0,100&chd=t:50,60,70,80,90,100|30,51,67,73,85,0|23,43,55,37,60,0&chs=355x200&chxl=1:|Confidence (%)|3:|Accuracy&chxp=1,50|3,50" alt="Chart?cht=s&chg=20,20&chds=50,100,0,100,0,60&chm=o,aaaaff,0,-1,25,1|r,44ff44,0,0.49,0.51|d,ffcccc,0,0,0"></p>
</div>
<p>Feb 23, 2016:</p>
<div class="statistics">
<h2>
Statistics
</h2>
<table>
<thead>
<tr>
<th class="narrow">
Confidence
</th>
<th>
50%
</th>
<th>
60%
</th>
<th>
70%
</th>
<th>
80%
</th>
<th>
90%
</th>
<th>
100%
</th>
<th>
Total
</th>
</tr>
</thead>
<tbody>
<tr>
<th class="narrow">
Accuracy
</th>
<td>
36%
</td>
<td>
60%
</td>
<td>
67%
</td>
<td>
77%
</td>
<td>
86%
</td>
<td>
100%
</td>
<td>
</td>
</tr>
<tr>
<th class="narrow">
Sample Size
</th>
<td>
28
</td>
<td>
57
</td>
<td>
63
</td>
<td>
53
</td>
<td>
66
</td>
<td>
1
</td>
<td>
268
</td>
</tr>
</tbody>
</table>
<img src="http://chart.apis.google.com/chart?cht=s&chg=20,20&chds=50,100,0,100,0,66&chm=o,AAAAFF,0,-1,25,1|r,44FF44,0,0.49,0.51|D,FFCCCC,0,0,0.5,-1&chxt=x,x,y,y&chxr=0,50,100|1,0,100&chd=t:50,60,70,80,90,100|36,60,67,77,86,100|28,57,63,53,66,1&chs=355x200&chxl=1:|Confidence (%)|3:|Accuracy&chxp=1,50|3,50" alt="Chart?cht=s&chg=20,20&chds=50,100,0,100,0,66&chm=o,aaaaff,0,-1,25,1|r,44ff44,0,0.49,0.51|d,ffcccc,0,0,0">
</div>
<p>July 19, 2016:</p>
<div class="statistics">
<h2>
Statistics
</h2>
<table>
<thead>
<tr>
<th class="narrow">
Confidence
</th>
<th>
50%
</th>
<th>
60%
</th>
<th>
70%
</th>
<th>
80%
</th>
<th>
90%
</th>
<th>
100%
</th>
<th>
Total
</th>
</tr>
</thead>
<tbody>
<tr>
<th class="narrow">
Accuracy
</th>
<td>
48%
</td>
<td>
66%
</td>
<td>
66%
</td>
<td>
75%
</td>
<td>
86%
</td>
<td>
100%
</td>
<td>
</td>
</tr>
<tr>
<th class="narrow">
Sample Size
</th>
<td>
44
</td>
<td>
79
</td>
<td>
77
</td>
<td>
61
</td>
<td>
79
</td>
<td>
1
</td>
<td>
341
</td>
</tr>
</tbody>
</table>
<p><img alt="Chart?cht=s&chg=20,20&chds=50,100,0,100,0,79&chm=o,aaaaff,0, 1,25,1|r,44ff44,0,0.49,0.51|d,ffcccc,0,0,0" src="http://chart.apis.google.com/chart?cht=s&chg=20,20&chds=50,100,0,100,0,79&chm=o,AAAAFF,0,-1,25,1|r,44FF44,0,0.49,0.51|D,FFCCCC,0,0,0.5,-1&chxt=x,x,y,y&chxr=0,50,100|1,0,100&chd=t:50,60,70,80,90,100|48,66,66,75,86,100|44,79,77,61,79,1&chs=355x200&chxl=1:|Confidence (%)|3:|Accuracy&chxp=1,50|3,50"></p>
</div>
<p>January 28, 2017:</p>
<div class="statistics">
<h2>
Statistics
</h2>
<table>
<thead>
<tr>
<th class="narrow">
Confidence
</th>
<th>
50%
</th>
<th>
60%
</th>
<th>
70%
</th>
<th>
80%
</th>
<th>
90%
</th>
<th>
100%
</th>
<th>
Total
</th>
</tr>
</thead>
<tbody>
<tr>
<th class="narrow">
Accuracy
</th>
<td>
51%
</td>
<td>
64%
</td>
<td>
69%
</td>
<td>
77%
</td>
<td>
86%
</td>
<td>
100%
</td>
<td>
</td>
</tr>
<tr>
<th class="narrow">
Sample Size
</th>
<td>
59
</td>
<td>
89
</td>
<td>
95
</td>
<td>
70
</td>
<td>
88
</td>
<td>
3
</td>
<td>
404
</td>
</tr>
</tbody>
</table>
<p><img src="http://chart.apis.google.com/chart?cht=s&chg=20,20&chds=50,100,0,100,0,95&chm=o,AAAAFF,0,-1,25,1|r,44FF44,0,0.49,0.51|D,FFCCCC,0,0,0.5,-1&chxt=x,x,y,y&chxr=0,50,100|1,0,100&chd=t:50,60,70,80,90,100|51,64,69,77,86,100|59,89,95,70,88,3&chs=355x200&chxl=1:|Confidence (%)|3:|Accuracy&chxp=1,50|3,50" alt="Chart?cht=s&chg=20,20&chds=50,100,0,100,0,95&chm=o,aaaaff,0, 1,25,1|r,44ff44,0,0.49,0.51|d,ffcccc,0,0,0"></p>
</div>
<h1 id="my-calibration-scores-on-the-quiz-from-software-estimation">My Calibration Scores on the quiz from Software Estimation</h1>
<p>The format is “[my low estimate - my high estimate] question - actual answer”.</p>
<blockquote>
<p>[ 1000K - 50000K ] Surface temperature of the Sun - 6000 + 273 [ 0 - 200 ] Latitude of Shanghai - 31 degs North [ 20 - 40 million square miles] Area of the Asian continent - 17 million square miles (NOT COOL!) [ 1000 BC - 1200 ] The year of Alexander the Great’s birth - 356 BC [ $100B - $7T ] Total value of U.S. currency in circulation in 2004 - $700B (too close) [ 2 x 10^8 m^3 - 12 x 10^11 m^3 ] Total volume of the Great Lakes - 6 x 10^20 m^3 (not even in the ballpark) [ $500M - $6B ] Worldwide box office receipts for the movie Titanic - $1.835 [ 1000 miles - 20000 miles ] Total length of the coastline of the Pacific Ocean - 84000 miles (JFC) [ 2500 - 500,000,000 ] Number of book titles published in the U.S. since 1776 - 22 million (huh, good call) [ 1 ton - 10^8 kg ] Heaviest blue whale ever recorded - 170,000 kg (nice call)</p>
<p>7/10 - JFC</p>
</blockquote>
<p>I came so close with the area of the Asian continent. I actually added up the areas I knew for India, China, and Russia. However, those were in square <strong>kilometer</strong>! Damn. Screwed by the unit. However, I just looked it up and even in square kilometers, the area is 44.58 million square kilometers. Which means my high estimate was just not high enough. So, I need to widen my confidence intervals. I shouldn’t have put so much faith on my uncertain beliefs about things outside my professional domain.</p>
<p>Similarly, I had too small an interval for the total coastline of the Pacific Ocean. I figured that the west coast of the Americas from north to south would be around 3000 miles (don’t know why I thought that). Turns out the Pacific coastline for the US is around 7000 miles. But the length of the US is only 1200 miles. Not sure how the coastline is calculated. But I anchored hard on the 3000 miles thing and doubled it for both sides of the Pacific and tripled it for safety. Turns out it was still too low. Not because I didn’t double and triple enough, but because I didn’t consider the range of values for my sole input number - the length of the west coast of the Americas. I should have gone for 20x my original estimate just because I was so unsure about how a “coastline” was calculated. After all, this is a 90% confidence interval.</p>
<p>Finally, I was off by 8 orders of magnitude on the volume of the Great Lakes. I thought their area would be no more than the area of India (which I thought was 3.5 million squared miles, when it was actually 3.2 million squared <strong>kilometers</strong>) and that their depth would be from 10m to 200m. Turns out their area is just 94000 squared miles and their average depth is 150m. Their total volume is 22000 km^3 or 2 x 10^13 m^3. So, his answer seems wrong. And mine is off by one order of magnitude. Fuck.</p>
<p>I was so close! Apparently less than 5% of the 600 people Steve McConnell surveyed got 70% or more (like me). Most people got 30% or less. I could have easily got to 90% by widening my intervals for the area and coastline questions and just taken the hit for the volume question.</p>
<blockquote>
<p>When I find the rare person who gets 7 or 8 answers correct, I ask “How did you get that many correct?” The typical response? “I made my ranges too wide.”</p>
<p>My response is, “No, you didn’t! You didn’t make your ranges wide enough!” If you got only 7 or 8 correct, your ranges were still too narrow to include the correct answer as often as you should have.</p>
<p>– Chapter 2, Software Estimation</p>
</blockquote>
<p>True. I could have made them wider.</p>
<div class="info">Created: November 3, 2014</div>
<div class="info">Last modified: September 28, 2019</div>
<div class="info">Status: finished</div>
<div class="info"><b>Tags</b>: prediction</div>
<br />
<div id="disqus_thread"></div>
<script type="text/javascript">
/* * * CONFIGURATION VARIABLES: EDIT BEFORE PASTING INTO YOUR WEBPAGE * * */
var disqus_shortname = 'spkrationalitytrainingground'; // required: replace example with your forum shortname
var disqus_identifier = '/Lets-Make-Predictions.html';
var disqus_title = 'Let\'s Make Predictions';
/* * * DON'T EDIT BELOW THIS LINE * * */
(function() {
var dsq = document.createElement('script'); dsq.type = 'text/javascript'; dsq.async = true;
dsq.src = '//' + disqus_shortname + '.disqus.com/embed.js';
(document.getElementsByTagName('head')[0] || document.getElementsByTagName('body')[0]).appendChild(dsq);
})();
</script>
<script type="text/javascript" src="https://fast.fonts.net/jsapi/f7f47a40-b25b-44ee-9f9c-cfdfc8bb2741.js"></script>
<noscript>Please enable JavaScript to view the <a href="http://disqus.com/?ref_noscript">comments powered by Disqus.</a></noscript>
<a href="http://disqus.com" class="dsq-brlink">comments powered by <span class="logo-disqus">Disqus</span></a>
</div>
<div id="footer">
Site proudly generated by
<a href="http://jaspervdj.be/hakyll">Hakyll</a>
</div>
</body>
</html>