Eliezer Yudkowsky - How to Actually Change Your Mind

Written by Eliezer Yudkowsky

Published in 2018 by the
Machine Intelligence Research Institute
Berkeley 94704
United States of America
intelligence.org

Cover artwork and design by Jimmy Rintjema

How to Actually Change Your Mind is licensed under Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International.

BY-NC-SA 4.0

ISBN-13: 978-1-939311-28-3

In Orthodox Judaism there is a saying: The previous generation is to the next one as angels are to men; the next generation is to the previous one as donkeys are to men. This follows from the Orthodox Jewish belief that all Judaic law was given to Moses by God at Mount Sinai. After all, its not as if you could do an experiment to gain new halachic knowledge; the only way you can know is if someone tells you (who heard it from someone else, who heard it from God). Since there is no new source of information; it can only be degraded in transmission from generation to generation.

Thus, modern rabbis are not allowed to overrule ancient rabbis. Crawly things are ordinarily unkosher, but it is permissible to eat a worm found in an applethe ancient rabbis believed the worm was spontaneously generated inside the apple, and therefore was part of the apple. A modern rabbi cannot say, Yeah, well, the ancient rabbis knew diddly-squat about biology. Overruled! A modern rabbi cannot possibly know a halachic principle the ancient rabbis did not, because how could the ancient rabbis have passed down the answer from Mount Sinai to him? Knowledge derives from authority, and therefore is only ever lost, not gained, as time passes.

When I was first exposed to the angels-and-donkeys proverb in (religious) elementary school, I was not old enough to be a full-blown atheist, but I still thought to myself: Torah loses knowledge in every generation. Science gains knowledge with every generation. No matter where they started out, sooner or later science must surpass Torah.

The most important thing is that there should be progress. So long as you keep moving forward you will reach your destination; but if you stop moving you will never reach it.

Tsuyoku naritai is Japanese. Tsuyoku is strong; naru is becoming, and the form naritai is want to become. Together it means, I want to become stronger, and it expresses a sentiment embodied more intensely in Japanese works than in any Western literature Ive read. You might say it when expressing your determination to become a professional Go playeror after you lose an important match, but you havent given upor after you win an important match, but youre not a ninth-dan player yetor after youve become the greatest Go player of all time, but you still think you can do better. That is tsuyoku naritai, the will to transcendence.

Each year on Yom Kippur, an Orthodox Jew recites a litany which begins Ashamnu, bagadnu, gazalnu, dibarnu dofi, and goes on through the entire Hebrew alphabet: We have acted shamefully, we have betrayed, we have stolen, we have slandered...

As you pronounce each word, you strike yourself over the heart in penitence. Theres no exemption whereby, if you manage to go without stealing all year long, you can skip the word gazalnu and strike yourself one less time. That would violate the community spirit of Yom Kippur, which is about confessing sinsnot avoiding sins so that you have less to confess.

By the same token, the Ashamnu does not end, But that was this year, and next year I will do better.

The Ashamnu bears a remarkable resemblance to the notion that the way of rationality is to beat your fist against your heart and say, We are all biased, we are all irrational, we are not fully informed, we are overconfident, we are poorly calibrated...

Fine. Now tell me how you plan to become less biased, less irrational, more informed, less overconfident, better calibrated.

There is an old Jewish joke: During Yom Kippur, the rabbi is seized by a sudden wave of guilt, and prostrates himself and cries, God, I am nothing before you! The cantor is likewise seized by guilt, and cries, God, I am nothing before you! Seeing this, the janitor at the back of the synagogue prostrates himself and cries, God, I am nothing before you! And the rabbi nudges the cantor and whispers, Look who thinks hes nothing.

Take no pride in your confession that you too are biased; do not glory in your self-awareness of your flaws. This is akin to the principle of not taking pride in confessing your ignorance; for if your ignorance is a source of pride to you, you may become loath to relinquish your ignorance when evidence comes knocking. Likewise with our flawswe should not gloat over how self-aware we are for confessing them; the occasion for rejoicing is when we have a little less to confess.

Otherwise, when the one comes to us with a plan for correcting the bias, we will snarl, Do you think to set yourself above us? We will shake our heads sadly and say, You must not be very self-aware.

Never confess to me that you are just as flawed as I am unless you can tell me what you plan to do about it. Afterward you will still have plenty of flaws left, but thats not the point; the important thing is to do better, to keep moving ahead, to take one more step forward. Tsuyoku naritai!

It is widely recognized that good science requires some kind of humility. What sort of humility is more controversial.

Consider the creationist who says: But who can really know whether evolution is correct? It is just a theory. You should be more humble and open-minded. Is this humility? The creationist practices a very selective underconfidence, refusing to integrate massive weights of evidence in favor of a conclusion they find uncomfortable. I would say that whether you call this humility or not, it is the wrong step in the dance.

What about the engineer who humbly designs fail-safe mechanisms into machinery, even though theyre damn sure the machinery wont fail? This seems like a good kind of humility to me. Historically, its not unheard-of for an engineer to be damn sure a new machine wont fail, and then it fails anyway.

What about the student who humbly double-checks the answers on their math test? Again Id categorize that as good humility. The student who double-checks their answers wants to become stronger; they react to a possible inner flaw by doing what they can to repair the flaw.

What about a student who says, Well, no matter how many times I check, I cant ever be certain my test answers are correct, and therefore doesnt check even once? Even if this choice stems from an emotion similar to the emotion felt by the previous student, it is less wise.

You suggest studying harder, and the student replies: No, it wouldnt work for me; Im not one of the smart kids like you; nay, one so lowly as myself can hope for no better lot. This is social modesty, not humility. It has to do with regulating status in the tribe, rather than scientific process. If you ask someone to be more humble, by default theyll associate the words to social modestywhich is an intuitive, everyday, ancestrally relevant concept. Scientific humility is a more recent and rarefied invention, and it is not inherently social. Scientific humility is something you would practice even if you were alone in a spacesuit, light years from Earth with no one watching. Or even if you received an absolute guarantee that no one would ever criticize you again, no matter what you said or thought of yourself. Youd still double-check your calculations if you were wise.