Topic: What do you think about Eliezer Yudkowsky / LessWrong?

[ST]

Eliezer Yudkowsky has been a panelist/speaker on least two skeptical events, Skepticon and NECSS. And in the video of his Skepticon talk, there was some interesting stuff about probability, and how many people get it wrong. His community site LessWrong contains some interesting stuff on biases. He is also the author of the fanfiction Harry Potter and the Methods of Rationality.

However, not all of his ideas are very sensible:

Would it be a stretch to call Yudkowsky a crank?

- He is self-educated with no academic degrees.
- He lacks peer-reviewed publications, or any publications whatsoever outside of his own websites and fanfiction.net.
- He is derisive of mainstream science, accusing it of collective bias and blindness.
- He has been pushing fringe ideas that are treated, at best, with skepticism by mainstream science, including attempts to prove a particular interpretation of quantum mechanics while, by definition, no particular interpretation can be singled out empirically as correct.
- He has formed a bit of a doomsday cult around himself, which holds the AI apocalypse to be near and treats cryonics and donating to SIAI as the path for salvation for the chosen. This bit may be unintentional, though, in a Life of Brian kind of way.

The last three are noteworthy. I find it a bit odd that he is a speaker at (a few) skeptic conventions given his views on mainstream science (one acolyte calls for a reform of science) and his push of some very dubious ideas (singularity/cryonics etc), as skeptical groups tend to take it upon themselves to defend science. That Wikipedia refers to him as a "researcher" is a bit funny. What research has he produced? RationalWiki has a less-than-flattering review, though the discussions on the talk page are mostly a circle-jerk.

Basically I want to know how "sound" the ideas of EY/LW are (as a whole), or if they will just completely mess up your brain? What to make of their heterodox views of science, for instance? Are their ideas about rationality from the field of cognitive science as they claim, or is it their own invention? What about their futurism stuff?

I think this is relevant because both the skeptical community and the LessWrong community strive to become better thinkers, and LW seems interested in attracting people from the skeptic and atheist communities.

[Cowtown Cody]

I have to offer some defense for Yudkowsky.  Not because I know him or even knew about LessWrong before just now, but because it looks to me like most of the talk page is pretty unfair about their criticism.

 - Being self-educated does not matter if he spends a lot of his time being right.  As a society we love credentials to tell us who knows things and who doesn't, but they aren't the last word and they're an imperfect proxy for what we're really interested in.  Yudkowsky being an autodidact doesn't bother me in the least.  None of us would feel any better even if he had a PhD from a not-very-good school, so what does it matter, exactly?

 - I have no peer-reviewed articles or publications.  Most of us don't.  Why does it matter that Yudkowsky doesn't?  It's one thing to say that he is not published in the scientific literature.  It's entirely another to say that he constantly submits things for publication and is constantly rejected because his experiments are bad or he's being a crackpot - and to my knowledge, that's not the case with Yudkowsky.  And anyway, we just got done saying he hasn't got a formal degree, so why would we expect him to be published in peer-reviewed literature?  Who exactly would his peers be?  What research should we expect him to have produced?  We're kind of double-counting here.

 - Some (not all) of his criticism of mainstream science as practiced is quite well-deserved, and it does in fact have biases.  This isn't anything new to skeptics either; SGU has talked a number of times about publication biases that favor new research and do not publish replication attempts.  Mainstream journals - even good ones - sometimes publish bullcrap just because it's well-written and the reviewers aren't well-versed enough in statistics to say what's wrong.  Another thread here just a short while ago had someone post a number of links to articles demonstrating just how poorly many scientists understand what a p-value is - and shit, some of us were wrong about it too.

And on a more direct level, we know for a fact that scientists are people and their brains are subject to a flood of biases, some of which they're aware of and many of which they are not aware of.  It's why we have double-blind tests, even when we have honest researchers.  The areas of cognitive science they're referencing when they're talking about bias are very well supported. 

 - In point of fact, when you cite that acolyte who wants to "reform science," reading through the article he is not making the case for opening science to crackpots.  He's actually making a pretty measured, mainstream criticism of mainstream science that isn't at all out of line with good skepticism.  Most of the arguments and suggestions will be very familiar to skeptics, because we make them too.

 - As far as the futurism stuff goes, it's easy to very confidently, very aggressively get it wrong.  Think Rocket Mail.  I wouldn't throw my hat in with anyone making predictions about the future beyond one or two years, and even then only on a very carefully constrained and rigorously modeled question.  That can work very well; spitballing about what's coming in ten or twenty or fifty usually does not.  To me that just looks like a chance for every would-be predictioneer to wind up equally wrong, because there are just too many interactions for a human brain to track.

[AngleWyrm]

Some of his behavior resembles that of a snake oil salesman.

He has developed a research institute which states it's primary objective is the safety of mankind with regard to the future development of AI that might otherwise prove hostile.

But two key points against it are: First, that he states the point at which AI becomes "smarter" than people cannot be explored by people, because we are only as smart as people. Additionally, references to AI development are in future context, not past or present context.

And second, he also has objections to emergent AI, the very technology that conquers limited understanding. These objections extend even to the point of social censorship on the forums of the use of the word emergent. Presumably this behavior is because emergent AI does not mesh well with solutions involving programatically enforced morality.

To ignore research being conducted and executed by others around the world on emergent AI voids any claims to safety.

Taken together, these points suggest a general uselessness of the institute bordering on fraud. Therefore I hold no expectation of useful productions coming out of that institution.

[ST]

BTW, they do have a message for skeptics: From Skepticism to Technical Rationality

Do you agree or disagree with it?

I have to offer some defense for Yudkowsky.  Not because I know him or even knew about LessWrong before just now, but because it looks to me like most of the talk page is pretty unfair about their criticism.

 - Being self-educated does not matter if he spends a lot of his time being right.  As a society we love credentials to tell us who knows things and who doesn't, but they aren't the last word and they're an imperfect proxy for what we're really interested in.  Yudkowsky being an autodidact doesn't bother me in the least.  None of us would feel any better even if he had a PhD from a not-very-good school, so what does it matter, exactly?

I agree somewhat, but how do we know the credibility of what he says? Especially of rather non-ordinary things like quantum mechanics and decision theory?

- I have no peer-reviewed articles or publications.  Most of us don't.  Why does it matter that Yudkowsky doesn't?  It's one thing to say that he is not published in the scientific literature.  It's entirely another to say that he constantly submits things for publication and is constantly rejected because his experiments are bad or he's being a crackpot - and to my knowledge, that's not the case with Yudkowsky.  And anyway, we just got done saying he hasn't got a formal degree, so why would we expect him to be published in peer-reviewed literature?  Who exactly would his peers be?  What research should we expect him to have produced?  We're kind of double-counting here.

Given that he is titled as an AI researcher, maybe something within the field of AI?

- Some (not all) of his criticism of mainstream science as practiced is quite well-deserved, and it does in fact have biases.  This isn't anything new to skeptics either; SGU has talked a number of times about publication biases that favor new research and do not publish replication attempts.  Mainstream journals - even good ones - sometimes publish bullcrap just because it's well-written and the reviewers aren't well-versed enough in statistics to say what's wrong.  Another thread here just a short while ago had someone post a number of links to articles demonstrating just how poorly many scientists understand what a p-value is - and shit, some of us were wrong about it too.

But if science has those flaws, how come scientists haven't thought about them?

And on a more direct level, we know for a fact that scientists are people and their brains are subject to a flood of biases, some of which they're aware of and many of which they are not aware of.  It's why we have double-blind tests, even when we have honest researchers.  The areas of cognitive science they're referencing when they're talking about bias are very well supported.

True, that's why peer-review exists.

- In point of fact, when you cite that acolyte who wants to "reform science," reading through the article he is not making the case for opening science to crackpots.  He's actually making a pretty measured, mainstream criticism of mainstream science that isn't at all out of line with good skepticism.  Most of the arguments and suggestions will be very familiar to skeptics, because we make them too.

Again, why haven't scientists thought about that?

- As far as the futurism stuff goes, it's easy to very confidently, very aggressively get it wrong.  Think Rocket Mail.  I wouldn't throw my hat in with anyone making predictions about the future beyond one or two years, and even then only on a very carefully constrained and rigorously modeled question.  That can work very well; spitballing about what's coming in ten or twenty or fifty usually does not.  To me that just looks like a chance for every would-be predictioneer to wind up equally wrong, because there are just too many interactions for a human brain to track.

I agree. It's also hillarious about people having nightmares of being tortured by an AI for not donating enough to SIAI.

Some of his behavior resembles that of a snake oil salesman.

He has developed a research institute which states it's primary objective is the safety of mankind with regard to the future development of AI that might otherwise prove hostile.

But two key points against it are: First, that he states the point at which AI becomes "smarter" than people cannot be explored by people, because we are only as smart as people. Additionally, references to AI development are in future context, not past or present context.

And second, he also has objections to emergent AI, the very technology that conquers limited understanding. These objections extend even to the point of social censorship on the forums of the use of the word emergent. Presumably this behavior is because emergent AI does not mesh well with solutions involving programatically enforced morality.

To ignore research being conducted and executed by others around the world on emergent AI voids any claims to safety.

Taken together, these points suggest a general uselessness of the institute bordering on fraud. Therefore I hold no expectation of useful productions coming out of that institution.

Well to be fair, all reading material is available online for free, even though they would very much want you to donate to SIAI.

[Zytheran]

- He is derisive of mainstream science, accusing it of collective bias and blindness.

...and deservedly so. There is plenty of research which shows this. And there is plenty of "science" that is pure BS, especially in the medical field. For example, stents. (My day job is a professional scientist who researches cognitive bias and unfortunately cognitive bias affects scientists as well as humans) The drug industry in particular... to the extent the problem was so bad researchers now have to register all their trials publicly before they do them so the bad results can't suffer from the file drawer problem if they don't show what they wanted. Obviously, some fields in science are worse than others.

I'll post up the link for a book that goes into this in detail on Monday.

We all go on about repeatability being core in science, however try to publish research that simply repeats an experiment and no-one is interested. Seriously ...WTF?

And scientists haven't dealt with the issue of their own cognitive biases because very few have the faintest clue about cognitive biases in general, let alone their own.

[Cowtown Cody]

I agree somewhat, but how do we know the credibility of what he says? Especially of rather non-ordinary things like quantum mechanics and decision theory?

You don't, and you wouldn't even if he had a PhD.  The PhD who was tasked with teaching me Poli Sci methodology knew absolutely nothing about statistics or how to design an experiment, so I ran like hell over to the Psychology department because I knew they had real statisticians and researchers there who actually knew what they were talking about.  They all had PhDs, but you absolutely can't trust them all equally. 

You'd have to know a lot about those non-ordinary things, enough to understand them yourself.  Or you could check what he says against another source that you think carries more weight.  A huge part of being a good skeptic is being willing to do your own legwork.  Any idiot can check credentials.

But if science has those flaws, how come scientists haven't thought about them?

They have, and they do.  Very often.  Listen to William Uttal on the Brain Science Podcast.  Listen to all the shows covering that horrific flap over Psi research being published in a reputable journal a while back.  Listen to the flak NEJM took for publishing papers with unsupported conclusions about the efficacy of acupuncture.  Or the group that said a Gamma Ray burst would hit earth.  Or listen to any of a dozen SGU episodes in the last year where the bias of peer-reviewed journals and occasional failure of peer review has come up.

These aren't completely unknown things to scientists or the scientific community.  They do know and think about them.  I'm not sure why you would think that they don't.

Given that he is titled as an AI researcher, maybe something within the field of AI?

Yes, as stated in the Rationalwiki comments and repeated once already.  But you're skirting the point here, which is about the nature of what we can or should expect from him.  The defining characteristic of a researcher is that they do research, not that they produce research.  That's something experimental researchers do, but not all research is experimentation with an output that looks like a lab report.  So I see no conflict at all in Yudkowsky being called "a researcher" or "research fellow" provided that he actually studies and reads about AI in an official capacity and thinks a lot about it.

And he does.  So he's a researcher.  It's not a big deal.  He doesn't have to produce anything to do research.

And if we want to insist that he cannot be a researcher until he is an experimental researcher who is putting out research product, then there's simply no getting around the fact that you cannot expect him to publish peer-reviewed experiments because he literally cannot publish experimentation results that we would call "peer-reviewed" in a way that we would accept, because he doesn't have his credentials in line. 

So it's good that he isn't producing research because we would only be more likely to call him a crank if he did.

True, that's why peer-review exists.

Peer review does not beat bias.  Peer review beats your special, unique biases, and it beats your methodological mistakes and misjudgements.  It does not beat institution-wide biases, unconscious biases shared between the reviewers and the author, or biases common to the field of research.    When all of your peers make the same mistakes and have the same unconscious biases, none of you are capable of correcting it because none of you can see it.

Peer review is a step better, but it isn't a magic bullet.  Cognitive biases are too murderously subtle to be beaten that easily.

[Citizen Skeptic]

You don't, and you wouldn't even if he had a PhD.  The PhD who was tasked with teaching me Poli Sci methodology knew absolutely nothing about statistics or how to design an experiment, so I ran like hell over to the Psychology department because I knew they had real statisticians and researchers there who actually knew what they were talking about.  They all had PhDs, but you absolutely can't trust them all equally. 

I agree. I had the unnfortunate experience of having to take a probability and statistics class (the calculus variety, not basic) and I can honestly say of all the math classes I've had, and I've had some weird ones being a CS major, p&s is one of the hardest subjects to get your head around. Most poli sci majors I've known haven't had the inclination or background to really understand the subject. If they did, they'd understand the futility of their endeavor.

[TalkingBook]

Very good responses from Cody in this thread.

Criticism of science is not anti-skeptical; blindly following anything, even the consensus of mainstream science, however, is. That some within the 'Skeptical Movement' have taken to equating skepticism with uncritical scientism is an ironic perversion of what skepticism is really all about. 'Science' is consistently and explicitly praised for its built-in self correction mechanisms, but these are actually quite limited in scope and depth if restricted only to specific theories within the broader framework, while forbidding questioning of the more basic assumptions of the scientific endeavor as a whole.

Of course criticism of science doesn't have to entail throwing out the baby with the bathwater. Science, for all its faults, is a great boon to most people - but it would be a mistake to think that any attempt to point out or correct its faults should be interpreted as an attack on science, or that anyone engaging in such criticism must necessarily be a crank.

[ST]

I agree somewhat, but how do we know the credibility of what he says? Especially of rather non-ordinary things like quantum mechanics and decision theory?

You don't, and you wouldn't even if he had a PhD.  The PhD who was tasked with teaching me Poli Sci methodology knew absolutely nothing about statistics or how to design an experiment, so I ran like hell over to the Psychology department because I knew they had real statisticians and researchers there who actually knew what they were talking about.  They all had PhDs, but you absolutely can't trust them all equally.

You'd have to know a lot about those non-ordinary things, enough to understand them yourself.  Or you could check what he says against another source that you think carries more weight.  A huge part of being a good skeptic is being willing to do your own legwork.  Any idiot can check credentials.

I know very well that academia is imperfect. But in academia there are at least some checks and balances. Here it's just a random person on the Internet (granted, who has a big fan club). And his description of quantum mechanics is not uncontroversial.

They have, and they do.  Very often.  Listen to William Uttal on the Brain Science Podcast.  Listen to all the shows covering that horrific flap over Psi research being published in a reputable journal a while back.  Listen to the flak NEJM took for publishing papers with unsupported conclusions about the efficacy of acupuncture.  Or the group that said a Gamma Ray burst would hit earth.  Or listen to any of a dozen SGU episodes in the last year where the bias of peer-reviewed journals and occasional failure of peer review has come up.

These aren't completely unknown things to scientists or the scientific community.  They do know and think about them.  I'm not sure why you would think that they don't.

Well the LW critique declared science was broken (not imperfect, but broken) and you defended it, so I got the impression that what was therein was new information.

Yes, as stated in the Rationalwiki comments and repeated once already.  But you're skirting the point here, which is about the nature of what we can or should expect from him.  The defining characteristic of a researcher is that they do research, not that they produce research.  That's something experimental researchers do, but not all research is experimentation with an output that looks like a lab report.  So I see no conflict at all in Yudkowsky being called "a researcher" or "research fellow" provided that he actually studies and reads about AI in an official capacity and thinks a lot about it.

And he does.  So he's a researcher.  It's not a big deal.  He doesn't have to produce anything to do research.

And if we want to insist that he cannot be a researcher until he is an experimental researcher who is putting out research product, then there's simply no getting around the fact that you cannot expect him to publish peer-reviewed experiments because he literally cannot publish experimentation results that we would call "peer-reviewed" in a way that we would accept, because he doesn't have his credentials in line. 

So it's good that he isn't producing research because we would only be more likely to call him a crank if he did.

Eh what, a researcher doesn't have to produce research ("do" and "produce" are just word-picking)? Am I a researcher as long as I study, read and think a lot about a subject?

Peer review does not beat bias.  Peer review beats your special, unique biases, and it beats your methodological mistakes and misjudgements.  It does not beat institution-wide biases, unconscious biases shared between the reviewers and the author, or biases common to the field of research.    When all of your peers make the same mistakes and have the same unconscious biases, none of you are capable of correcting it because none of you can see it.

Peer review is a step better, but it isn't a magic bullet.  Cognitive biases are too murderously subtle to be beaten that easily.

Ok then, what beats cognitive biases better than peer review? I'm all in that peer review isn't perfect, because no human institution is.


In regards to science in general, do you agree that Bayes is superior to science?

[Zytheran]

Yes, as stated in the Rationalwiki comments and repeated once already.  But you're skirting the point here, which is about the nature of what we can or should expect from him.  The defining characteristic of a researcher is that they do research, not that they produce research.  That's something experimental researchers do, but not all research is experimentation with an output that looks like a lab report.  So I see no conflict at all in Yudkowsky being called "a researcher" or "research fellow" provided that he actually studies and reads about AI in an official capacity and thinks a lot about it.

And he does.  So he's a researcher.  It's not a big deal.  He doesn't have to produce anything to do research.

And if we want to insist that he cannot be a researcher until he is an experimental researcher who is putting out research product, then there's simply no getting around the fact that you cannot expect him to publish peer-reviewed experiments because he literally cannot publish experimentation results that we would call "peer-reviewed" in a way that we would accept, because he doesn't have his credentials in line. 

So it's good that he isn't producing research because we would only be more likely to call him a crank if he did.

Eh what, a researcher doesn't have to produce research ("do" and "produce" are just word-picking)? Am I a researcher as long as I study, read and think a lot about a subject?

I read CC's comments as "do" meaning the actual act of research whilst "produce" being writing reports and papers.
In this regard he is totally spot on. I work in an oganisation with a few thousand scientists, half with PhD's. Although all do science (or technology or engineering) not all produce . In academia there are many who "do" research as well as teach and yet "produce" very little. Someone who teaches a science subject may well do a a lot of research and yet 'all' they produce is new up to date lessons every year and published papers less frequently.

Peer review does not beat bias.  Peer review beats your special, unique biases, and it beats your methodological mistakes and misjudgements.  It does not beat institution-wide biases, unconscious biases shared between the reviewers and the author, or biases common to the field of research.    When all of your peers make the same mistakes and have the same unconscious biases, none of you are capable of correcting it because none of you can see it.

Peer review is a step better, but it isn't a magic bullet.  Cognitive biases are too murderously subtle to be beaten that easily.

Ok then, what beats cognitive biases better than peer review? I'm all in that peer review isn't perfect, because no human institution is.

Nothing yet known to humanity beats cognitive biases much at all, in any endeavour, let alone peer review. Even if you are trained about cognitive biases, repeatedly, and understand them, the empirical evidence shows only a small reduction in their effects of say 30%. Cognitive biases work on your subconscious, not the concious part of your mind you have control over. And in my professional experience as a researcher in cognitive biases, very ,  very few people are aware of them, let alone know about them in detail , let alone try to actively do something about them. Cognitive biases are part of being human and our evolutionary baggage that comes with intelligence and creativity.

Cognitive biases are the reason the SGU podcast and this forum exist.

[Cowtown Cody]

I know very well that academia is imperfect. But in academia there are at least some checks and balances. Here it's just a random person on the Internet (granted, who has a big fan club).

Hm.  Show me where I've made the argument that says "lots of people are listening to him so we should, too."  I'm pretty sure I haven't made that one.  If I ever did, you'd be right to call me on it, because that would be crap.

It seems to me that you're looking for a quick, easy way to sort this guy, and you'd really like to use credentialism to do it.  You're free to do that if you want to, but that would be so easy that I don't think we'd even be having this conversation if you were really comfortable with it.  We both know it's a crappy tool, and if we'd hesitate to use it then let's use something else.

And, let's also not commit the fallacy fallacy, and say that because he's wrong about quantum mechanics (and he may well be, I have nothing like enough knowledge of the field to even pretend to know), he must also be wrong about Bayes' Rule and his understanding of rationality.  That doesn't work.  At the end of the day, you have to decouple the person from the arguments and evaluate the arguments that you can understand for yourself.  That's challenging and it's a labor-intensive pain in the ass sometimes, but there isn't a better tool.

Well the LW critique declared science was broken (not imperfect, but broken) and you defended it, so I got the impression that what was therein was new information.

Did you not read the post, or were you just unfamiliar with the arguments he was making?  They're pretty common.  "Broken" and "imperfect" is a matter of salesmanship on both sides, and I didn't get very hung up on it.  Just read the arguments.

But to take that a shade further, it does sometimes bother me when I hear the term "imperfect" applied to science, as if "perfect" isn't so very far away.  Compared to outright woo like ass-reading, sure.  But we shouldn't walk away with the idea that science as it exists now is so close to everything we could ever want that no adjustments are really necessary.  There's no reason to think that science is as good as it can be in practice, and there's also no reason to think that there's no more room for conceptual improvements in the process of science, which is perhaps more important.

Science is not the same thing it was during the Enlightenment, and it's a good thing it isn't.  Science changes as we get better at it and as we learn more about what it can and can't tell us.  So why shouldn't it continue to evolve, and why shouldn't we take seriously the ideas that intelligent people have about making it better?

Eh what, a researcher doesn't have to produce research ("do" and "produce" are just word-picking)? Am I a researcher as long as I study, read and think a lot about a subject?

Well first, everything Zytheran said because he's exactly right about what I meant.  And, if you are legally employed by some institution or company that's paying you to do those things, and refers to the sum of those practices as "research," then yes.  You're a researcher.  It's not a regulated title so it's not worth getting our underwear twisted over.

And, for the sake of being bouncy in argumentation, I'll add something else.  A research product can be much more than just a published experiment or a lab report.  Another common product of extensive research is a book, or an in-depth discussion of a subject, or an just an explanation of a complicated event.  Like Yudkowsky's very long blog posts on AI and rationality and decision theory.  Why don't you regard that as a research product when it clearly is exactly a product of how he spends his work day, wherein he is called a "research fellow"?

In regards to science in general, do you agree that Bayes is superior to science?

While I don't understand the first thing about quantum mechanics so I can't get my head around that post, I do understand Bayes' Rule.  And it is an incredibly powerful, almost ridiculously useful bit of math.  To the extent that someone is rational about their expectations, they are following Bayes' Rule.  But where I don't agree with Yudkowsky is where he tries to say that reality must conform to our expectations of rationality, and therefore that what Bayes' Rule says we should expect based on our priors is a superior answer to scientific experimentation.

You still have to run the experiment to see if your expectations are correct.  You can (and I think should) use Bayesian expectations to help inform you of possible methodological or setup errors you made during the experiment, but you cannot replace the experiment so long as it's possible for the experiment to truly and without error tell you something other than what you expected.

[Citizen Skeptic]

WRT Bayes - formalizing a guess doesn't make it any better. At least I've never seen any evidence to that effect.

[Zytheran]

FYI if you have never worked with scientists "research" can be:

• reading through some papers, finding common themes and/or new ideas or critiquing the papers and presenting them to your peers as part of group meeting
• reading through a whole pile of papers and other research and setting up a research program that others actually do
• representing your organisation as an SME (subject matter expert) in another forum after you have developed expertise in an area
  
• performing a technical review of a tool for it's usability
• investigating failure modes or analysis of a failed tool (physical, software or system) and reporting what went wrong and making recommendations of how to avoid future failures.
• designing and running an experiment to test a hypothesis in your field of research
• and multitude of other things

The actual results might simply be a viewgraph presentation to peers, an internal technical report, an private report at an international conference, working papers that are only looked at locally or even shock, horror an external peer reviewed paper. Or even a talk and presentation to the public.
And on top of this, depending on where you work, there will be a layer of confidentiality due to business concerns or military/national security.

[Citizen Skeptic]

They also tend to take one experiment and drag it into a career.

[jt512]

They also tend to take one experiment and drag it into a career.

Not if they want to be considered good scientists.