Thanks!

I can think of a few different explanations:

• Even extreme childhood abuse doesn't have a major effect on life outcomes.
  • (Including this one for completeness though I consider it obviously implausible.)
• The level of abuse that would affect life outcomes is rare enough not to be picked up on in the studies.
• The methodology of the studies creates on floor on the badness of outcomes that gets picked up; e.g. maybe adoptive parents are screened well enough to make the worst abuse not happen, and the people drawn from national twin registers and contacted to fill in surveys don't bother responding if their lives are so messed up they don't have the time or energy for that.
  • But at least studies that use national registers about e.g. incarceration should be able to control for this.
• There's something wrong about the correlation argument.

I vote for the second one—the result is usually “shared environment effects on adult outcomes are statistically indistinguishable from zero” but that doesn’t mean they’re exactly 0.00000….  :)

It gave me this cite which reports a 26% shared environment effect on antisocial behavior defined as various forms of crime (property offenses, violent offenses, and drug-related offenses), measured from childhood to early adulthood [20 years], and also cited some previous work with similar findings. 

There are definitely huge shared environment effects during the period where kids are living with their parents. No question about it!

(Also, for the record, some measurements seem to be adult outcomes, but are also partly measuring stuff that happened when kids were living with their parents—e.g. “having ever attended college”, “having ever been depressed”, “having ever gotten arrested”, etc. Those tend to have big shared environment effects too.)

[Waller et al. 2018] sounds like even identical twins may be treated differently enough by the same parents for it to have noticeable effects?

The result there is “parents are harsher and less warm towards their kids who are more callous and aggressive”, and when you phrase it that way, it seems to me that the obvious explanation is that parents behave in a way that is responsive to a kid’s personality.

Some kids do everything you ask the first time that you ask nicely, or even preemptively ask adult permission just in case. Other kids are gleefully defiant and limit-testing all the time. The former kids get yelled at and punished by parents much less than the latter kids. (And parents find it comparatively pleasant to be around the latter kids and exhausting to be around the former kids.) This all seems very obvious to me, right?

Thus, if per Will Eden “parents think they treat their kids the same… but the kids think the parents treat them differently, and outside observations would support this claim”, I’d guess that the parent would say something like: “the household rule is: I’ll watch TV at night with any child who wants to do that and who sits quietly during the show, and another household rule is: if you jump on the couch then you have to go to your room, etc. I apply these rules consistently to all my children”. And the parent is correct—they are in fact pretty consistent in applying those rules. But meanwhile, the kids and outside observers just notice that one sibling winds up getting punished all the time and never joining in the evening TV, while the other sibling is never punished and always welcome for TV.

In my post I poked fun at a study in the same genre as Waller et al. 2018. I wrote: “I propose that the authors of that paper should be banned from further research until they have had two or more children.” Of course, for all I know, they have lots of kids, and they have babysat and hung out with diverse classes of preschoolers and kids (as I have), and yet they still subscribe to this way of thinking. I find it baffling how people can look at the same world and interpret it so differently. ¯\_(ツ)_/¯

Anyway, that other study didn’t even mention the (IMO primary and obvious) causal pathway from child personality to parental treatment at all, IIRC. The Waller et al. 2018 study does a bit better: it mentions something like that pathway, albeit with an unnecessarily-exotic description: “Evocative rGE reflects situations in which the child elicits an environment consonant with his/her genes (e.g., a callous child frequently rejects parental warmth, causing his/her parents to eventually reduce their levels of warmth).”), and they claim that their study design controls for it. What they mean is actually that they (imperfectly) controlled for the “child genes → child personality → parental treatment” pathway (because the children are identical twins). But they don’t control for the “random fluctuations in such-and-such molecular signaling pathway during brain development or whatever → child personality → parental treatment” pathway. I find that pathway much more plausible than their implied preferred causal pathway of (I guess) “parents are systematically warmer towards one twin than another, just randomly, for absolutely no upstream reason at all → child personality”. Right?

I think the only way to see parental effects without getting tripped up by the child personality → parental treatment pathway is to rely on the fact that some parents are much more patient or harsh than others, which (my common sense says) is a huge source of variation. Just look around and see how differently different parents, different babysitters, different teachers interact with the very same child. That brings us to adoption studies, which find that parenting effects on adult outcomes are indistinguishable from zero. So I’m inclined to trust that finding over the studies like Waller et al. 2018.

By the way, Will Eden cites Plomin, but meanwhile Turkheimer reviews many of the same studies and says the results are basically zero (he calls this “the gloomy prospect”). (Turkheimer is Plomin’s reference 41.) It would be interesting to read them side-by-side and figure out why they disagree and who to believe—I haven’t done that myself.

I'm not sure that explaining it by "maybe parents just have a minimal effect overall" makes things any less weird and counterintuitive!

I don’t find these things counterintuitive, but rather obvious common sense. I can talk a bit about where I’m coming from.

There are many things that I did as a kid, and when I was an adult I found that I didn’t enjoy doing them or find it satisfying, so I stopped doing them. Likewise, I’ve “tried on” a lot of personalities and behaviors in my life as an independent adult—I can think of times and relationships in which I tried out being kind, mean, shy, outgoing, frank, dishonest, impulsive, cautious, you name it. The ways-of-being that felt good and right, I kept doing, the ones that felt bad and wrong, I stopped. This is the picture I suggested in Heritability, Behaviorism, and Within-Lifetime RL, and feels very intuitive to me.

Also, my personality and values are very very different from either my parents’ personalities, or the personality that my parents would have wanted to instill in me.

I guess the childhood trauma thing is important to your intuitions, which we were chatting about in the comments of my post. I can share my first-person perspective on that too: I was blessed with a childhood free of any abuse or trauma. But I’m kinda neurotic, and consequently have wound up with very very dumb memories that feel rather traumatic to me and painful to think about. There is absolutely no good reason for these memories to feel that way—I’m thinking of perfectly fine and normal teenage things that I have no objective reason to be embarrassed about, things in the same ballpark as “my parents walked on me masturbating, and promptly apologized for not knocking and politely left, and never brought it up again”. (My actual painful memories are even dumber than that!) Just as you were speculating in that comment thread, I think I’m predisposed to dwell on certain types of negative memories (I’m very big into embarrassment and guilt), and in the absence of any actual objectively terrible memories to grab onto, my brain has grabbed onto stupid random teenager stuff.

I am able to take the harsh edge off these memories by CBT-type techniques, although I haven’t really bothered to do that much because I’m lazy and busy and AFAICT those memories are not affecting my current behavior too much. (I’m somewhat introverted in part from being oversensitive to social embarrassment and guilt, but it’s not that bad, and my uninformed guess is that finding peace with my dumb teenage memories wouldn’t help much.)

things you do as a parent will have generally small or zero predictable effects on what the kid will be like as an adult

Is there any action-relevant difference between “no effect” and “no predictable effect”?

Like I always say, the context in which you’re bringing up heritability matters. It seems that the context here is something like:

Some people say shared environment effects are ≈0 in twin & adoption studies, therefore we should believe “the bio-determinist child-rearing rule-of-thumb”. But in fact, parenting often involves treating different kids differently, so ‘shared environment effects are ≈0’ is irrelevant, and therefore we should reject “the bio-determinist child-rearing rule-of-thumb” after all.

If that’s the context, then I basically disagree. Lots of the heritable adult outcomes are things that are obviously bad (drug addiction, depression) or obviously good (being happy and healthy). Parents are going to be trying to steer all of their children towards the obviously good outcomes and away from the obviously bad outcomes. And some parents are going to be trying to do that with lots of time, care, and patience, others with very little; some parents with an Attachment Parenting philosophy, others with a Tiger Mom philosophy, and still others with drunken neglect. If a parent is better-than-average at increasing the odds that one of their children has the good outcomes and avoids the bad outcomes, then common sense would suggest that this same parent can do the same for their other children too, at least better than chance. That doesn’t require an assumption that the parents are doing the exact same things for all their children. It’s just saying that a parent who can respond well to the needs of one kid would probably (i.e. more-than-chance) respond well to the needs of another kid, whatever they are, whereas the (e.g. drunk and negligent) parents who are poor at responding to the needs of one kid are probably (i.e. more-than-chance) worse-than-average at responding to the needs of another kid.

And yet, the twin and adoption studies show that shared environmental effects are ≈0 for obviously good and obviously bad adult outcomes, just like pretty much every other kind of adult outcome.

In other words, nobody is questioning that a parent can be abusive towards one child but not another. Rather, it would be awfully strange if a parent who was abusive towards one child was abusive towards another child at exactly the population average rate. There’s gonna be a correlation! And we learn something important from the fact that this correlation in child-rearing has immeasurably small impact on adult outcomes.

Likewise, adoptive siblings may have different screen time limitations, parents attending or not attending their football games, eating organic versus non-organic food, parents flying off the handle at them, being in a better or worse school district, etc. But they sure are gonna be substantially correlated, right?

So I think that the argument for “the bio-determinist child-rearing rule of thumb” goes through. (Although it has various caveats as discussed at that link.)

Suppose we had a CoT-style transcript of every thought, word, email and action by the founder of a successful startup over the course of several years of its founding, and used this for RL: then we'd get a reward signal every time they landed a funding round, sales went up significantly, a hire they made or contract they signed clearly worked out well, and so forth — not enough training data by itself for RL, but perhaps a useful contribution.

I don’t think this approach would lead to an AI that can autonomously come up with a new out-of-the-box innovative business plan, and found the company, and grow it to $1B/year revenue, over the course of years, all with literally zero human intervention.

…So I expect that future AI programmers will keep trying different approaches until they succeed via some other approach.

And such “other approaches” certainly exist—for example, Jeff Bezos’s brain was able to found Amazon without training on any such dataset, right?

(Such datasets don’t exist anyway, and can’t exist, since human founders can’t write down every one of their thoughts, there are too many of them and they are not generally formulated in English.)

I was trying to argue in favor of:

CLAIM: there are AI capabilities things that cannot be done without RL training (or something functionally equivalent to RL training).

It seems to me that, whether this claim is true or false, it has nothing to do with alignment, right?

Hmm, you’re probably right.

But I think my point would have worked if I had suggested a modified version of Go rather than chess?

Let me give you a detailed presciption…

For example, people want AIs that can autonomously come up with a new out-of-the-box innovative business plan, and found the company, and grow it to $1B/year revenue, over the course of years, all with literally zero human intervention.

People are trying to build such AIs as we speak, and I don’t expect them to quit until they succeed (or until we all die from their attempt).

And it’s possible—if human brains (or groups of human brains) can do this, so can AI algorithms. But human brains involve (model-based) RL. It’s an open question whether there exists a non-RL algorithm that can also do that. (LLMs as of today obviously cannot.)

I think the issue here is: “some aspect of the proposed input would need to not be computable/generatable for us”.

If the business is supposed to be new and out-of-the-box and innovative, then how do you generate on-distribution data? It’s gonna be something that nobody has ever tried before; “out-of-distribution” is part of the problem description, right?

Can you (or anyone) explain to me why there could be a problem that we can only solve using RL on rated examples, and could not do via SGD on labeled examples?

Not all RL is “RL on [human] rated examples” in the way that you’re thinking of it. Jeff Bezos’s brain involves (model-based) RL, but it’s not like he tried millions of times to found millions of companies, and his brain gave a reward signal for the companies that grew to $1B/year revenue, and that’s how he wound up able to found and run Amazon. In fact Amazon was the first company he ever founded.

Over the course of my lifetime I’ve had a billion or so ideas pass through my head. My own brain RL system was labeling these ideas as good or bad (motivating or demotivating), and this has led to my learning over time to have more good ideas (“good” according to certain metrics in my own brain reward function). If a future AI was built like that, having a human hand-label the AI’s billion-or-so “thoughts” as good or bad would not be viable. (Futher discussion in §1.1 here). For one thing, there’s too many things to label. For another thing, the ideas-to-be-rated are inscrutable from the outside.

I’m also still curious how you think about RLVR. Companies are using RLVR right now to make their models better at math. Do you have thoughts on how they can make their models equally good at math without using RLVR, or any kind of RL, or anything functionally equivalent to RL?

Also, here’s a challenge which IMO requires RL [Update: oops, bad example, see Zack’s response]. I have just invented a chess variant, Steve-chess. It’s just like normal chess except that the rooks and bishops can only move up to four spaces at a time. I want to make a computer play that chess variant much better than any unassisted human ever will. I only want to spend a few person-years of R&D effort to make that happen (which rules out laborious hand-coding of strategy rules).

That’s the Steve-chess challenge. I can think of one way to solve the Steve-chess challenge: the AlphaZero approach. But that involves RL. Can you name any way to solve this same challenge without RL (or something functionally equivalent to RL)?

Right, but what I'm saying is that there's at least a possibility that RL is the only way to train a frontier system that's human-level or above.

In that case, if the alignment plan is "Well just don't use RL!", then that would be synonymous with "Well just don't build AGI at all, ever!". Right?

...And yeah sure, you can say that, but it would be misleading to call it a solution to inner alignment, if indeed that's the situation we're in.

There’s a potential failure mode where RL (e.g. RLVR or otherwise) is necessary to get powerful capabilities. Right?

I for one don’t really care about whether the LLMs of May 2025 are aligned or not, because they’re not that capable. E.g. they would not be able to autonomously write a business plan and found a company and grow it to $1B/year of revenue. So something has to happen between now and then to make AI more capable. And I for one expect that “something” to involve RL, for better or worse (well, mostly worse). I’ve been saying that RL is necessary for powerful capabilities, i.e. (self)-supervised learning will only get you so far, since I think 2020, shortly after I got into AGI safety, and that prediction of mine is arguably being borne out in a small way by the rise of RLVR (and I personally expect a much bigger shift towards RL before we get superintelligence).

What’s your take on that? This post seems to only talk about RL in the context of alignment not capabilities, unless I missed it. I didn’t read the linked papers.

(Thanks for your patient engagement!)

If you believe

• it is probably true that future pure imitation learning techniques can capture the process by which humans figure out new scientific ideas over millions of seconds, AND
• it is “certainly false” that future pure imitation learning techniques can capture the process by which AlphaZero figures out new chess strategies over millions of games

then I’m curious what accounts for the difference, in your mind?

More detail, just to make sure we’re on the same page: The analogy I’m suggesting is:

(A1) AlphaZero goes from Elo 0 to Elo 2500

(A2) …via self-play RL

(A3) Future pure imitation learner extrapolates this process forward to get Elo 3500 chess skill

-versus-

(B1) Human civilization goes from “totally clueless about nanotech design principles / technical alignment / whatever” in 1900 to “somewhat confused about nanotech design principles / technical alignment / whatever” in 2025

(B2) …via whatever human brains are doing (which I claim centrally involves RL)

(B3) Future pure imitation learner extrapolates this process forward to get crystal-clear understanding of nanotech design principles / technical alignment / whatever

You think that (A3) is “certainly false” while (B3) is plausible, and I’m asking what you see as the disanalogy.

(For the record, I think both (A3) and (B3) are implausible. I think that LLM in-context learning can capture the way that humans figure out new things over seconds, but not the way that humans figure out new things over weeks and months. And I don’t think that’s a solvable problem, but rather points to a deep deficiency in imitation learning, a deficiency which is only solvable by learning algorithms with non-imitation-learning objectives.)

Hmm, I don’t particularly disagree with anything you wrote. I think you’re misunderstanding the context of this conversation.

I wasn’t bringing up tree search because I think tree search is required for AGI. (I don’t think that.)

Rather, I was making a point that there will need to be some system that updates the weights (not activations) of an AGI as it runs, just as adult humans learn and figure out new things over time as they work on a project.

What is this system that will update the weights? I have opinions, but in general, there are lots of possible approaches. Self-play-RL with tree search is one possibility. RL without tree search is another possibility. The system you described in your comment is yet a third possibility. Whatever! I don’t care, that’s not my point here.

What is my point? How did this come up? Well, Cole’s OP is relying on the fact that “[pure] imitation learning is probably existentially safe”. And I was saying that pure imitation learning imposes a horrific capability tax that destroys his whole plan, because a human has open-ended autonomous learning, whereas a model trained by pure imitation learning (on that same human) does not. So you cannot simply swap out the former for the latter.

In Cole’s most recent reply, it appears that what he has in mind is actually a system that’s initialized by being trained to imitate humans, but then it also has some system for open-ended continuous learning from that starting point.

And then I replied that this would solve the capability issue, but only by creating a new problem that “[pure] imitation learning is probably existentially safe” can no longer function as part of his safety argument, because the continuous learning may affect alignment.

For example, if you initialize a PacMan RL agent on human imitation (where the humans were all very nice to the ghosts during play), and then you set up that agent to continuously improve by RL policy optimization, using the score as the reward function, then it’s gonna rapidly stop being nice to the ghosts.

Does that help explain where I’m coming from?