Gleanings from Double Crux on “The Craft is Not The Community”

Epistemic status: This is a bunch of semi-remembered rephrasings of a conversation.

At the CFAR alumni reunion, John Salvatier and I had a public double crux on my last post.

A double crux is a technique CFAR invented, which I think is much better than a debate. The goal is to simply pin down where exactly two people disagree. This can take a while. Even the best, most respectful debates are adversarial: it’s my opinion vs. yours, and we see which is stronger in an (ideally fair) contest. A double crux is collaborative: we’re just trying to find which is the exact point of contention here, so that if we go on to have an actual debate we won’t be talking past each other.

John’s motivation for disagreeing with my post was that he didn’t think I should be devaluing the intellectual side of the “rationality community”. My post divided projects into into community-building (mostly things like socializing and mutual aid) versus outward-facing (business, research, activism, etc.); John thought I was neglecting the importance of a community of people who support and take an interest in intellectual inquiry.

I agreed with him on that point — intellectual activity is important to me — but doubted that we had any intellectual community worth preserving.  I was skeptical that rationalist-led intellectual projects were making much progress, so I thought the reasonable thing to do was to start fresh.

John is actually working on an intellectual project of his own — he’s trying to explore what the building blocks of creative thinking are, and how one can improve it — and he thinks his work is productive/useful, so that seemed a good place to dig in deeper.

I mentioned that by a lot of metrics, his work doesn’t have a lot of output. He has done a lot of one-on-one conversations and informal experiments with people in the community, but there’s no writeup, and certainly no formal psychological research, papers, or collaboration with psychologists. How could an outsider possibly tell if there’s a real thing here?

John said that I might be over-valuing formality. He’s pretty confident that the “informal” phase of work — the part when you’re just playing with an idea, or planning out your strategy, before you sit down to execute — is actually the most important part, in the sense that it’s highest-leverage. After some discussion, I came to agree with him.

I’ve definitely had the experience that creative work is “bursty” — that most days you produce piles of junk, and some days you produce solid gold, whether it’s writing, math, or code. I’ve also heard this from other people, both friends and famous historical figures.  It also seems that when something’s going right about your “pre-work” cognitive processes — planning, imagining, even emotional attitudes — you do much better work at the formal, sit-down-and-produce-output stage.  Work goes hugely better when the “muse” is friendly.

John additionally believes that it’s possible to “train your muse” to help you work better, and said that learning to do this himself allowed him to contribute much better to open-source software projects (where he built a statistics library.)

He also pointed out that when it comes to dealing with the distant future, general-purpose and speculative cognitive processes will have to be more important than trained skills, because the future will contain unfamiliar situations that we haven’t trained for. People who excel at the sit-down-and-execute activities that help you succeed in your field aren’t necessarily going to be able to reason about the weirdness of a changing world.

(I agreed that the ability to “philosophize” well seems to be much rarer than the ability to execute well; I’ve seen many prominent computer scientists whose theories about general intelligence just don’t make sense.)

So the speculative, philosophical, imaginative stuff that comes before sitting down and executing is important for success, important for humanity, and maybe something we can learn to do better. John certainly thinks so, and wants the rationality community to be a sort of laboratory or nursery for these ideas.

It’s also true that formally executing on these ideas can be really hard, if you define “formally” strictly enough. Here’s Scott Alexander reflecting on the bureaucratic hell of trying to get a psychiatry study on human subjects approved by an IRB — when it only involved giving them a questionnaire!  If that’s what it takes to do academic experimental research on humans, I don’t want to claim that anybody who’s thinking about the human mind without publishing papers can be rounded down to “doing nothing.”

That still leaves us with the question of “how do I know — not an IRB, not the ‘general public’, but Sarah, your friendly acquaintance — that you’re making real progress?”  I’m still going to need to be shown some kind of results, if not peer-reviewed ones.  This is why I’m a fan of blogging and something in the neighborhood of “citizen science.”  If a programmer tests the speed of two different programs and writes up the results, code included, I believe them, and if I’m skeptical, I can try to duplicate their results. It’s in the spirit of the scientific method, even if it’s not part of the official edifice of Science(TM).

So, John and I still have an unresolved disagreement about the general status of these “how to think real good” projects in the community.  He thinks they’re moving forward; I still haven’t seen evidence that convinces me.  This is our “double crux” — both of us agree (the “double” part) that it’s the key (“crux”) to our disagreement.

But I definitely agree with John that if there were promising ways to “think real good” being developed in our community, then it would be important to support and encourage that exploration.

One interesting thing that we had in common was that we both viewed “community” from a strongly individualist standpoint. John said he would evaluate someone as a potential collaborator on a project pretty much the same way whether they were a community member or not — track records for success, recommendations from friends he respects, and so on.  The “community” is useful because it’s a social network that sometimes floats cool people to his attention.  Deeper notions of tribe or belonging didn’t seem to apply, at least concerning his intellectual aims.  He had no interest in kicking people out for not following community standards, or trying to get everybody in the community to be a certain way; if a person considered themselves “part of the community” but John couldn’t see benefit from associating with that person, he just wouldn’t associate.  This is not everybody’s point of view — in fact, some people might say that John’s idea of a community is equivalent to not having a community at all.  So a lot of the things that seem to spark a lot of debate these days — community standards, community norms, etc — just didn’t show up in this double-crux at all, because neither of us really had strong intuitions about governance or collective issues.

Mostly, I came away with a lot of food for thought about the reflection vs. execution thing.  If there’s a spectrum between musing about the thing and doing the thing, I’m pretty far towards the “musing” side relative to the general population, so I’d generally assumed that I do too much musing and not enough executing.  “Head-in-the-clouds dreamer” and “impractical intellectual” and all that.  (Introspection falls into this category too; thinking too much about your own psyche is “navel-gazing”.)  But reflecting well seems to be incredibly high-reward relative to the time and effort spent, for compounding reasons. Strategizing so that you work on the right project, or putting attention into your mental health now so that you’re systematically more productive in the future, has a much bigger impact than just spending one more marginal hour on the daily slog.  Reflecting and strategizing gave my friend Satvik much more success at work.

It’s always felt a little presumptuous to me — like “who am I to think about what I’m doing? I’m supposed to keep my head down, keep slogging, and not ask questions!  Isn’t it terribly selfish to wonder what helps me do my best, rather than just doing my duty?”  But that’s a set of norms that gets applied to children, soldiers, and laborers (and maybe it shouldn’t even then), not to people like me. My peers expect that a person who does “knowledge work” for a living and writes essays will, of course, reflect on what she’s doing.

So maybe I ought to be going back and reading what reflective people write, taking it seriously this time around. “The unexamined life is not worth living.”  What if you literally meant that?  What if thinking about stuff was not a half-forbidden luxury but the most important thing about being human?

The Craft is Not The Community

Epistemic status: argumentative. I expect this to start a discussion, not end it.

“Company culture” is not, as I’ve learned, a list of slogans on a poster.  Culture consists of the empirical patterns of what’s rewarded and punished within the company. Do people win promotions and praise by hitting sales targets? By coming up with ideas? By playing nice?  These patterns reveal what the company actually values.

And, so, with community cultures.

It seems to me that the increasingly ill-named “Rationalist Community” in Berkeley has, in practice, a core value of “unconditional tolerance of weirdos.”  It is a haven for outcasts and a paradise for bohemians. It is a social community based on warm connections of mutual support and fun between people who don’t fit in with the broader society.

I think it’s good that such a haven exists. More than that, I want to live in one.

I think institutions like sharehouses and alloparenting and homeschooling are more practical and humane than typical American living arrangements; I want to raise children with far more freedom than traditional parenting allows; I believe in community support for the disabled and mentally ill and mutual aid for the destitute.  I think runaways and sexual minorities deserve a safe and welcoming place to go.  And the Berkeley community stands a reasonable chance of achieving those goals!  We’re far from perfect, and we obviously can’t extend to include everyone (esp. since the cost of living in the Bay is nontrivial), but I like our chances. I think we may actually, in the next ten years, succeed at building an accepting and nurturing community for our members.

We’ve built, over the years, a number of sharehouses, a serious plan for a baugruppe, preliminary plans for an unschooling center, and the beginnings of mutual aid organizations and dispute resolution mechanisms.  We’re actually doing this.  It takes time, but there’s visible progress on the ground.

I live on a street with my friends as neighbors. Hardly anybody in my generation gets to say that.

What we’re not doing well at, as a community, is external-facing projects.

And I think it’s time to take a hard look at that, without blame or judgment.

The thing about external-outcome-oriented projects is that they require standards. You have to be able to reject people for incompetence, and expect results from your colleagues.  I don’t think there’s any other way to achieve goals.

That means that an external-oriented project can’t actually serve all of a person’s emotional needs.  It can’t give you unconditional love. It can’t promise you a vibrant social scene. It can’t give you a place of refuge when your life goes to hell.  It can’t replace family or community.

As Robert Frost said, “Home is where, when you go there, they have to take you in.”

But Tesla Motors and MIT don’t have to take you in. And they wouldn’t work if they did.

Internally focused groups, whose goals are about the well-being of their own members, are intrinsically different. You have to care more about inclusion, consensus, and making the process itself rewarding and enjoyable for the participants. If you’re organizing parties for each other, making the social group gel well and making everyone feel welcome is not a side issue — it’s part of the main goal.  A Berkeley community organization that didn’t serve the people who currently live in Berkeley and meet their needs would no longer be an organization for our community; you can’t fire the community and get another.  The whole point is benefiting these specific people.

An externally-focused goal, by contrast, can and should be “no respecter of persons” — you have to focus on achieving good outcomes, regardless of who’s involved.

So far, when members of our community focus on external goals, I think they’ve done much better when they haven’t tried to marry those goals with making community institutions.

Some rationalists have created successful startups and many more have successful careers in the tech industry — but these are basically never “rationalist endeavors”, staffed exclusively by community members or focused on serving this community.  And they shouldn’t be. If you want to build a company, you hire the most competent people for the job, not necessarily your friends or neighbors. A company is oriented towards an external outcome, and so has to be objective and strategic about that goal. It’s by nature outward-facing, not inward-facing to the community.

My own outward-facing goal is to make an impact on treating disease.  Mainly I’m working towards that through working in drug development — at a company which is by no means a “rationalist community project.” It shouldn’t be! What we need are good biologists and engineers and data scientists, regardless of what in-jokes they tell or who they’re friends with.

In the long run, I hope to work on things (like anti-aging or tighter bench-to-bedside feedback loops) that are somewhat more controversial. But I don’t think that changes the calculus. You still want the most competent people you can get, who are also willing to get on board with your mission. Idealism and radicalism don’t negate the need for excellence, if you’re working on an external goal.

Some other people in the community have more purely intellectual projects, that are closer to Eliezer Yudkowsky’s original goals. To research artificial intelligence; to develop tools for training Tetlock-style good judgment; to practice philosophical discourse.  But I still think these are ultimately outcome-focused, external projects.

Artificial intelligence research is science, and requires the strongest possible computer scientists and engineers. (And perhaps cognitive scientists and philosophers.) To their credit, I think most people working on AI are aware of the need for expertise and are trying to attract great talent, but I still think it needs to be said.

“Good judgment” or reducing cognitive biases is social science, and requires people with expertise in psychology, behavioral economics, decision theory, cognitive science, and the like. It might also benefit from collaboration with people who work in finance, who (according to Tetlock’s research) are more effective than average at avoiding cognitive biases, and have a long tradition of valuing strategy and quantitative thinking.

Even philosophical discourse, in my opinion, is ultimately external-outcome-focused. For all that it’s hard to measure success, the people who want to create better discourse norms do have a concern with quality, and ultimately consider this a broad issue affecting modern society, not exclusively a Berkeley-local issue.  Progress on improving discourse should produce results (in the form of writing or teaching) that can be shared with the wider world. It might be worth prioritizing good humanists, writers, teachers, and scholars who have a track record of building high-quality conversations.

None of these projects need to be community-focused!  In fact, I think it would be better if they freed themselves from the Berkeley community and from the particular quirks and prejudices of this group of people. It doesn’t benefit your ability to do AI research that you primarily draw your talent from a particular social group.  It also doesn’t straightforwardly benefit the social group that there’s a lot of overlap with AI research.  (Is your research going to make you better at babysitting? Or cooking? Or resolving roommate drama?)

Cross-pollination between the Berkeley community and outcome-oriented projects would still be good. After all, ambitious people make good company!  I don’t think that the Bay Area is going to stop being a business and academic hub any time soon, and it makes sense for there to be friendships and relationships between people who primarily focus on community and people who primarily focus on external projects.  (After all, that’s one traditional division of labor in a marriage.)

But I think it muddies the water tremendously when people conflate community-building with external-facing projects.

Does maintaining good social cohesion within the Berkeley community actually advance the art of human rationality? I’m skeptical, because rationality training empirically doesn’t improve our scores on reasoning questions.  [I seem to recall, though I can’t find the source, that community members also don’t score higher than other well-educated people on the Cognitive Reflection Test, a standard measure of cognitive bias.] [ETA: I remembered wrong! As of the 2012 LessWrong survey, LessWrongers scored significantly better on cognitive bias questions than the participants in the original papers.  So it’s still possible, though not obvious, that we’re in some sense a more-rational-than-average community.]  If we’re not actually more rational than you’d expect in the absence of a community, why should rationality-promoters necessarily focus on community-building within Berkeley? Social cohesion is good for people who live together, but it’s a stretch to say that it promotes the cause of critical thinking in general.

Does having fun discussions with friends advance the state of human discourse?  Does building interesting psychological models and trying self-help practices advance the state of psychology?  Again, it’s really easy to confuse that with highbrow forms of just-for-fun socializing. Which are good in themselves, because they are enjoyable and rewarding for us!  But it’s disingenuous to call that progress in a global and objective sense.

I consider charismatic social crazes to be essentially a form of entertainment. People enjoy getting swept up in the emotional thrill of a cult of personality or mass movement for pretty much the same reasons they enjoy falling in love, watching movies, or reading adventure stories. Thrills are personal (they only create pleasure for the recipient and don’t spill over much to the wider world) and temporary (you can’t stay thrilled or entertained by the same thing forever).  Interpersonal thrills, unlike works of art, are inherently ephemeral; they last only as long as the personal relationship does.  These factors place limits on how much value can be derived from charisma alone, if it doesn’t build more lasting outcomes.

That means personality cults and mass enthusiasms belong in the “community-building” bucket, not the “outward-facing project” bucket. Even from a community perspective, you might not think they’re are a great idea, and that’s a separate discussion. But I’m primarily pushing back against the idea that they can be world-saving projects.  Something that only affects us and the insides of our heads, without leaving any lasting products or documents that can be shared with the world, is a purely internal affair.  Essentially, it’s just a glorified personal relationship.  And so it should be evaluated on the basis of whether it’s good for the people involved and the people they have personal relationships with. You look at it wearing your “community member” hat, not your “world-changing” hat.  Even if it’s nominally a nonprofit or a corporation, or associated with some ideology, if it doesn’t produce something for the world at large, it’s a community institution.

(An analogy is fandom debates. Sometimes these pose as political activism, but they are really arguments about fiction, by fans and for fans, with barely any impact on the non-fandom world. Fandom is a leisure activity, and so fandom debates are also a leisure activity.  Real activism, as practiced by professionals, is work; it’s not always fun, has standards for competence, and has tangible external goals that matter to people other than the activists themselves.)

I think distinguishing external-facing goals from community goals sidesteps the eternal debates over “what should the rationalist community be, and who should be in it?”

I think, in practice, the people who go to the same events in Berkeley, live together, parent together, and regularly communicate with each other, form a community. That community exists and deserves the love and attention of the people who value being part of it.  Not for any external reason, but, as they say in Red Dawn, “because we live here.”  We are people, our quality of life matters, our friendships matter, and putting effort into making our lives good is valuable to us.  We won’t choose the universal best way of life for all mankind, because that doesn’t exist; we’ll have the community norms and institutions that suit us, which is what having a local community means.

But there are individual people who are dissatisfied because that particular community, as it exists today, is not well-suited to accomplishing their external-facing goals. And I think that’s also a valid concern, and the natural solution is to divorce those goals from the purely communitarian ones. If you wonder “why doesn’t anybody around here care about my goal?” the natural thing to do is to focus on finding collaborators who do care about your goal — who may not be here!

If you’re frustrated that this isn’t a community based around excellence, I think you’ll be more likely to find what you’re looking for in institutions that have external goals and standards for membership. Some of those exist already, and some are worth creating.

A local, residential community isn’t really equipped to be a team of superstars.  Certainly a multigenerational community can’t be a team of superstars — you can’t just exclude someone’s kid if they don’t make the cut.

I don’t want to overstate this — Classical Athens was a town, and it had a remarkable track record of producing human achievement. But even there, we were talking about a population of 300,000 people.  Most of them didn’t go down in history.  Most of them were the “populace” that Plato thought were not competent to rule.  90% of them weren’t even adult male citizens. I don’t know how you build a new Athens, but it’s important to remember that it’s going to contain a lot of farming and weaving along with the philosophy and poetry.

Small teams of excellent people, though, are pretty much the tried-and-true formula for getting external-facing things done, whether practical or theoretical.  And the usual evaluative tools of industry and academia are, I think, correct in outline: judge by track records, not by personal relationships; measure outcomes objectively; consider ideas that challenge your preconceptions; publish, or ship, your results.

I think more of us who have concrete external goals should be seeking these kinds of focused teams, and not relying on the residential community to provide them.