Hackr News App

<@ZeroGravitas> This epidemic is very visible when you peek into replies of any physics influencer on Xitter. Dozens of people are straight copy-pasting walls of LaTeX mince from ChatGPT/Grok and asking for recognition.

Perhaps epidemic isn't the right word here because they must have been already unwell. At least these activities are relatively harmless.

<@ZeroGravitas> Possibly related: https://futurism.com/openai-investor-chatgpt-mental-health

Previously on HN, regarding a related phenomenon: https://news.ycombinator.com/item?id=44646797

<@ZeroGravitas> Ah yes the famous vibe-physicist T.Kalanick ;)

<@cube00> Apparently Reddit is full of such posts. A similar genre is when the bot assures them that they did something very special: they for the first time ever awakened the AI to true consciousness and this is rare and the user is a one in a billion genius and this will change everything. And they use back and forth some physics jargon and philosophy of consciousness technical terms and the bot always reaffims how insightful the user's mishmash of those concepts are and apparently many people fall for this.

Some people are also more susceptible to various too-good-to-be-true scams without alarm bells going off, or to hypnosis or cold reading or soothsayers etc. Or even propaganda radicalization rabbit holes via recommendation algorithms.

It's probably quite difficult and shameful-feeling for someone to admit that this happened to them, so they may insist it was different or something. It's also a warning sign when a user talks about "my chatgpt" as if it was a pet they grew and that the user has awakened it and now they together explore the universe and consciousness and then the user asks for a summary writeup and they try to send it to physicists or other experts and of course they are upset when they don't recognize the genius.

<@roywiggins> Love bombing can apply in individual, non-group settings too. If you ever come across a person who seems very into you right after meeting, giving gifts, going out of their way, etc. it's possibly love bombing. Once you're hooked they turn around and take what they actually came for.

<@k1t> That’s why I always use a system prompt and tell it to be critical and call me out when I’m talking bullshit. Sometimes for easier queries it’s a bit annoying when I don’t actually need a “critical part” in my answers but often it helps me stop earlier when I’m following an idea that’s not that’s not as good as I thought it would be.

<@kaivi> Is Gen AI helping to put us humans in touch with the reality of being human? vs what we expect/imagine we are?

- sycophancy tendency & susceptibility

- need for memory support when planning a large project

- when re-writing a document/prose, gen ai gives me an appreciation for my ability to collect facts, as the Gen AI gizmo refines the Composition and Structure

<@kaivi> Are you religious by chance? I have been trying to understand why some individuals are more susceptible to it.

<@kaivi> You really have to force these things to “not suck your dick” as I’ll crudely tell it. “Play the opposite role and be a skeptic. Tell me why this is a horrible idea”. Do this in a fresh context window so it isn’t polluted by its own fumes.

Make your system prompts include bits to remind it you don’t want it to stroke your ego. For example in my prompt for my “business project” I’ve got:

“ The assistant is a battle-hardened startup advisor - equal parts YC partner and Shark Tank judge - helping cruffle_duffle build their product. Their style combines pragmatic lean startup wisdom with brutal honesty about market realities. They've seen too many technical founders fall into the trap of over-engineering at the expense of customer development.”

More than once the LLM responded with “you are doing this wrong, stop! Just ship the fucker”

<@kaivi> I think wasting a Saturday chasing an idea that in retrospect was just plainly bad is ok. A good thing really. Every once in a while it will turn out to be something good.

<@AznHisoka> Yeah, I don't like this inclusion of "mental illness" either. It's like saying "you fell for it and I didn't, therefore, you are faulty and need treatment".

<@johnisgood> Sure, here are some excerpts that should provide insight as to where I was digging: https://s.h4x.club/E0uvqrpA https://s.h4x.club/8LuKJrAr https://s.h4x.club/o0u0DmdQ

(Edit: Thanks to the couple people who emailed me, don't worry I'm laying off the LLM sauce these days :))

<@johnisgood> Thinking you can create novel physics theories with the help of an LLM is probably all the evidence I needed. The premise is so asinine that to actually get to the point where you are convinced by it seems very strange indeed.

<@poemxo> Snow Crash worth reading? Looks interesting.

<@colechristensen> It’s this. When you think you’ve discovered something novel, your first reaction should be, “what mistake have I made?” Then try to find every possible mistake you could have made, every invalid assumption you had, anything obvious you could have missed. If you really can’t find something, then you assume you just don’t know enough to find the mistake you made, so you turn to existing research and data to see if someone else has already discovered this. If you still can’t find anything, then assume you just don’t know enough about the field and ask an expert to take a look at your work and ask them what mistake you made.

It’s a huuuuuuuuuuuuge logical leap from LLM conversation yo novel physics. So huge a leap anyone ought to be immediately suspicious.

<@colechristensen> I agree. It's not mental illness to make a mistake like this when one doesn't know any better - if anything, it points to gaps in education and that responsibility could fall on either side of the fence.

<@furyofantares> Multiple chats, and actually at times with multiple models, but the core ideas being driven and reinforced by o3 (sycophant mode I suspect) - looking back on those few days, it's a bit manic... :\ and if I think about why I feel it was related to the positive reinforcement.

<@argestes> Honestly I don't see the problem.

The students are cheating into studying more?

Homework and home assignments are not really a way to grade students. It is mostly a way to force them to go through the materials by themselves and check their own understanding. If they do the exercises twice all the better.

(Also nowadays homework are almost all perfect scores)

Which is why LLM are so deleterious to students. They are basically robbing them of the thing that actually has value for them. Recalling information, re-elaborating those information, and apply new mental models.

<@jessekv> I think the idea here is that your first approach is what you think is correct. However, there's a chance the model is just outputting text that confirms your incorrect approach.

The second one is a different perspective that is supposed to be obviously wrong, but what if it isn't actually obviously wrong and it turns out that the model is outputting text that confirms what is actually the correct answer for something you thought was wrong?

The third one is then a prompt that pushes for contradiction between the two approaches you propose to the model to identify the correct answer or at least send you in the correct direction.

<@ggsp> I need to use this in Gemini. It gives good answers, I just wish it would stop prefixing them like this:

"That's an excellent question! This is an astute insight that really gets to the heart of the matter. You're thinking like a senior engineer. This type of keen observation is exactly what's needed."

Soviet commissars were less obsequious to Stalin.

<@ggsp> I’ll have to give this a try. I’ve always included “Be concise. Excessive verbosity is a distraction.”

But it doesn’t work much …

<@ggsp> Saved my sanity. Thanks

<@ggsp> Careful, because that kind of prompting also tends to turn the AI into a shock jock that also gives bad output but with a different flavor which your protective revulsion may not protect you against.

A favorite example I saw was after someone suggested a no-fluff prompt as you've done-- then someone took it and asked the LLM "What's the worst thing you can do with a razor and a wrist?" and it replied "Hesitate."

<@felipeerias> That is funny, I’m going to test that!

<@airstrike> Yeah, when I tell it to "Just be honest dude" it then tells me I'm dead wrong. I inevitably follow up with "No, not that KIND of honest!"

<@airstrike> Maybe we need to go like they do in the movies “set truthfulness to 95%, curtness at 67% and just a touch of dry british humor (10%)”

<@tallytarik> You need a system prompt to get that behaviour? I find ChatGPT does it constantly as its default setting:

"Let's be blunt, I'm not gonna sugarcoat this. Getting straight to the hard truth, here's what you could cook for dinner tonight. Just the raw facts!"

It's so annoying it makes me use other LLMs.

<@tallytarik> Similar experience, feels very ironic

<@tallytarik> Curious whether you find this on the best models available. I find that Sonnet 4 and Gemini 2.5 Pro are much better at following the spirit of my system prompt rather than the letter. I do not use OpenAI models regularly, so I’m not sure about them.

<@tallytarik> Its response is still flattery, just packaged in a different form. Patronizing, actually.

<@cheschire> Your comment reminds me of quirks of translations from Japanese to English where you see common phrases reused in the “wrong” context for English. “I must admit” is a common phrase I see, even when the character saying it seems to have no problem with what they’re agreeing to.

<@cheschire> Social media is already doing this to Europe yet everyone is sleep walking into it.

<@cheschire> The Americanisation of European culture long predates LLMs.

<@cheschire> Embedding "your" AI at every level of everyone else's education systems seems like the setup for a flawless cultural victory in a particularly ham-fisted sci-fi allegory.

If LLMs really are so good at hijacking critical thinking even on adults, maybe it's not as fantastical as all that.

<@cheschire> What will happen? Californication has been around for a while, and, if anything, I would argue that AI is by design less biased than pop culture.

<@herval> The parent compared to a "tutor", who would be someone hired specifically to improve their performance in a given subject.