Not a unique thought here, but the religious faith that people have in AI is so very clear in posts like this. Not just people with this level of zeal either. Everyone who talks positively about AI does it in the context of the amazing things what will someday - not now but someday in the future - be possible. Even the goals of AI are all aligned with what most religions proclaim as the end goal or result. You might as well just come out and ask people to believe that someday a savior is going to come and solve all our problems and if you submit to that savior then you too can be one of the chosen ones.
What are the aspects of working on a bus that make it impractical? When I find myself in your position usually I end up realizing I'm self-conscious about people seeing what I'm doing more than I'm concerned about any practical downside or benefit.
In my case it's mostly the shaking - trains are mostly fine, but buses are just too unstable. They also tend to be more crowded, meaning I need to tuck my elbows in and adopt an even-less-stable position which compounds the problem.
Teen Vogue has been doing incredible journalism for years now. Seriously. At least 10 years (I remember they lit on fire when DT was elected the first time), possibly 20. They're good enough to be on my rotation of news sites I don't visit every day but do visit regularly.
Similarly, given its name and origin, I was not expecting BuzzFeed News to ever be worth reading but while it was running (until 2 years ago) they funded actual investigative journalism and published a number of good, well-researched articles (which I came across on this site).
It’s intriguing that certain publications often surprise us with exceptional journalism that diverges from their primary brand. Rolling Stone Magazine, for instance, has produced remarkable content unrelated to music.
I mentioned this in one of my other comments but even if AI figures this out where humans can't, I think this will involve coming to the conclusion that there really is an objective, knowable truth at the core of the universe or at least a set of tools will be developed for better discerning what that truth is. Even if you're talking about knowing who someone's spouse is, I'm not convinced we can ever teach a machine how you know what is true unless we're willing to have some sort of basic tenet of what truth is or how to discern it.
That was the last generation of AI attempts: things like Cyc https://cyc.com/ , trying to turn all of human knowledge into a really big database of predicate logic.
I might be getting overly philosophical here but I'd say it's because they truly don't know anything at all (as opposed to knowing some things but not others). To be able to say "I don't know" you have to first "know" on a deeper level that there is a fundamental true or correct answer to a question and that you are disconnected from it.
Well, yes. "AI" skips over all the difficulties and contradictions of philosophy, all the challenges of working out what it means to know something, things like "justified true belief" and so on. It 'just' (!) uses a probabilistic model to emit strings of text. It's basically a super-pundit. It can predict conventional wisdom really well.
True, no argument there. What fascinates me more is why people continue to think we can teach a chatbot how to recognize what's true and give us answers that we can't find for ourselves. At best a chatbot is going to be a tool that enables us to gain insights we didn't have before the same way a dictionary can "teach" you words you didn't know before.
I think the idea of using technology to solve life's ultimate conundrums has long since jumped the shark and veered into the area of religious belief. People are literally putting their faith in AI even if they wouldn't use religious vocabulary to label and define it as such.
I think some of this is the ultimate logical conclusion of postmodernism/deconstruction. In the early 20th century people were a lot more confident about the possibility of finding absolute truth; at the end of it this had completely dissipated into uncertainty and relativism.
The "Sokal Hoax" was a 90s experiment in which a physicist created a fake paper and submitted it to a cultural studies journal. He did not base his paper on anything he would have considered "true", rather on a desire to look as much like a valid text as possible. This is a simplified version of how the LLM training/scoring process works. Nowadays everywhere is having to deal with the same kind of thing done by LLM users. It's the perfect technology for non-rigorous academia.
I don't think it's overly philosophical to point out that these are large language models, not truth engines or AGI or knowledge directories. They're not using logic to reason their way to an answer. They're just predicting the next word that would sound like part of a human answer.
Fair enough. I think a lot of people are going to end up blindly trusting AI because its right often enough. But for those who are interested in what it really means to know something, I wonder if this will push people back towards embracing the idea that there is fundamental, objective, knowable truth at the core of the universe even if we can't ever know that truth perfectly.
They're just outputting tokens that resemble a reasoning process. The underlying tech is still the same LLM it always has been.
I can't deny that doing it that way improves results, but any model could do the same thing if you add extra prompts to encourage the reasoning process, then use that as context for the final solution. People discovered that trick before "reasoning" models became the hot thing. It's the "Work it out step by step" trick but in a dedicated fine-tune.
> They're just outputting tokens that resemble a reasoning process.
Looking at one such process of emulating reasoning (got deepseek-70B locally), I'm starting to wonder how does that differ from actual reasoning? We "think" about something, may make errors in that thinking, look for things that don't make sense and correct ourselves. That "think" step is still a blackbox.
I asked that llm a typical question of gas exchange between containers, it made some errors and noticed some calculations that didn't make sense:
> Moles left A: ~0.0021 mol
> Moles entered B: ~0.008 mol
> But 0.0021 +0.008=0.0101 mol, which doesn't make sense because that would imply a net increase of moles in the system.
Well, that's totally invalid calculation, it should be "-" in there. It also noticed that those quantities should be same in other place.
Eventually, after 102 minutes and 10141 tokens, involving checking answers from different angles multiple times, it outputted approximately correct response.
Does it matter if it doesn't know why this particular pattern is suitable? Also, do you always ask yourself why you use that particular pattern all the time, or do you just use them?
It seems like you are implying that I don't think before I speak. Maybe that is sometimes the case, but I would venture to say, "not usually, and certainly not always."
The point I'm making here is that all of these observations are made after-the-fact. We humans see five different categories of output:
1. "I do know X" where X is indeed correct information
2. "I do know X" where X is false information or nonsense
3. "I don't know" when it really doesn't
4. "I don't know" when a slightly different prompt would lead to option #1
5. Output that is not phrased as a direct answer to a question.
The article introduced #2 as "hallucinations". I introduced #4 in my previous comment (and just now #5), and propose that all five are hallucinations.
As far as the LLM is concerned, there is only one category of output: the most likely next token. Which of the five that will be is determined by the examples present in the training corpus, which are later weighed during training.
Logic is not present in the process. It is only present in the result.
> It seems like you are implying that I don't think before I speak.
I'm implying that most times you don't think before you think or after you think (you or me typically don't meta-think).
I'm saying that very often I (and looks like a lot of people around me) don't think much before I speak. I have internal monologue when I'm "thinking something out", but I typically don't think things through when I'm speaking with people in day-to-day conversations, only when I encounter a problem I didn't see yet and I'm not "trained" in solving it. Maybe some people can make fully reasoned sentences in split seconds before they start talking, but not me. IIRC those two modes of thinking are called slow and fast thinking.
> Logic is not present in the process. It is only present in the result.
I'm talking about that process. Have you seen "thinking" part of current reasoning LLM's? It does indeed look like a process of using logic. After "thinking" part, there is "output" part that makes conclusions form the process of thinking. Recently I asked local version of deepseek about a gas exchange problem and it thought a lot about this, making some small mistakes in logic, correcting them, ultimately returning approximately valid result. It even made some small errors in calculations and corrected itself by multiplying parts of numbers and adding them for correct result. I've put that example online[1] if you'd like to read it, it's pretty interesting.
I guess the crux of it is this: is it training or awareness?
What I see happening between the <think> tags of Deepseek-R1 is essentially a premade set of circular prompts. Each of these prompts is useful, because it explores a path of tokens that are likely to match a written instance of logical deduction.
When the <think> continuation rewrites part of a prompt as a truthy assertion, it reaches a sort of fork in the road: to present a story of either acceptance or rejection of that assertion. The path most likely followed depends entirely on how the assertion is phrased (both in the prompt, and in the training corpus). Remember that back in the training corpus, example assertions that look sensible are usually followed by a statement of acceptance, and example assertions that look contradictory or fallacious are usually followed by a statement of rejection.
Because the token generation process follows an implicit branching structure, and because that branching structure is very likely to match a story of logical deduction, the result is likely to be logically coherent. It's even likely to be correct!
The distinction I want to make here is that these branches are not logic. They are literary paths that align to a story, and that story is - to us - a well-formed example of written logical deduction. Whether that story leads to fact or fiction is no more and no less than an accident. We humans often tend to follow a similar process, but we can actively choose to do real critical thinking instead.
This design pattern is really useful for a few reasons:
- it keeps the subjects of the prompt in context
- it presents the subjects of the prompt from different perspectives
- it often stumbles into a result that is equivalent to real critical thinking
On the other hand,
- it may fill the context window with repetitive conversation, and lose track of important content
- it may get caught in a loop that never ends
- it may confidently present a false conclusion to itself, then expand that conclusion into a whole thread
- the false conclusions it presents will be much less obvious, because they will always be written as if they came out of a thorough process of logical deduction
I find that all of these problems are much more likely to occur when using a smaller locally hosted copy of the model than when using the full-sized one that is hosted on chat.deepseek.com. That doesn't mean these are solved by using a bigger model, only that the set of familiar examples is large enough to fit most use cases. The more unique and interesting your conversation is, the less utility these models will have.
> We humans often tend to follow a similar process, but we can actively choose to do real critical thinking instead.
> - it may confidently present a false conclusion to itself, then expand that conclusion into a whole thread
I want to know how that differs from human "real critical thinking", because I may be missing this function. How do you know what you thought of is true or false? I only know it because I think I know it. I had made a lot of mistakes in past with a lot of confidence.
> The more unique and interesting your conversation is, the less utility these models will have.
Yeah, that also happens with a lot of people I know.
> ... the result is likely to be logically coherent. It's even likely to be correct!
Yeah, a lot of training data made sure that what it outputs is as correct as possible. I still remember my training over many days and nights to be able to multiply properly, with two different versions of multiplying table and many false results until I got it right.
> I guess the crux of it is this: is it training or awareness?
I don't think LLM's are really aware (yet). But they do indeed follow logical reasoning method, even if not perfect yet.
Just a thought: when do you think about how and what you think (awareness of your thoughts)? When you actually think through a problem, or after that thinking? Maybe to be self-aware, AI's should be given some "free-thinking time". Currently it's "think about this problem and then immediately stop, do not think any more". Currently training data discourages any "out-of-context" thinking, so they don't.
We know what true and false mean. An LLM knows what true and false are likely to be surrounded with.
The problem is that expressions of logic are written many ways. Because we are talking about instances of natural language, they are often ambiguous. LLMs do not resolve ambiguity. Instead, they continue it with the most familiar patterns of writing. This works out when two things are true:
1. Everything written so far is constructed in a familiar writing pattern.
2. The familiar writing pattern that follows will not mix up the logic somehow.
The self prompting train of thought LLM pattern is good at keeping its exploration inside these two domains. It starts by attempting to phrase its prompt and context in a particular familiar structure, then continues to rephrase it with a pattern of structures that we expect to work.
Much of the logic we actually write is quite simple. The complexity is in the subjects we logically tie together. We also have some generalized preferences for how conditions, conclusions, etc. are structured around each other. This means we have imperfectly simplified the domain that the train of thought writing pattern is exploring. On top of that, the training corpus may include many instances of unfamiliar logical expressions, each followed by a restatement of that expression in a more familiar/compatible writing style. That can help trim the edge cases, but it isn't perfect.
---
What I'm trying to design is a way to actually resolve ambiguity, and do real logical deduction from there. Because ambiguity cannot be resolved to a single correct result (that's what ambiguity means), my plan is to, each time, use an arbitrary backstory for disambiguation. This way, we could be intentional about the process instead of relying on the statistical familiarity of tokens to choose for us. We would also guarantee that the process itself is logically sound, and fix it where it breaks.
They are machines designed to produce a facsimile of knowledge. Or at least an approximation. If they refused to answer, that's a failure by the terms of what they product aims to do
You are actually getting overly philosophical. The reason is that a step of chatbot training is to fine tune the base model to less frequently respond with non answers.
I have not read the article, but the answer is: Because they are specifically built not to. Its like asking why so few salesmen end a call with "yeah it seems our product is not the right solution for you"
Only person who understands this is a style choice and not a deep limitation of the robot condition.
If a politician has non-answers for difficult questions, does that mean they aren't conscious? If a student writes crap for a test question, aiming for partial marks, were they raised wrong?
Instead, we believe that those scientists did provide correct answers. Which is true about 99.9% of time, but not 100%. There is still believing involved, because we can't really check out everything for ourselves, there is no time in one person's life for this.
I'm assuming you're exaggerating for effect at least a little but with that caveat I couldn't agree more. CS Lewis has a great argument for this in his introduction to Athanasius' On the Incarnation. Paraphrasing his argument: Time naturally filters out the nonsense and what we're left with are the books that are worth reading by virtue of the fact that they have stood the test of time. Truth or at least the closest we can get to it naturally bubbles up to the surface over time.
This is horrible advice if you want to work on anything innovative. You don't have time to wait for things to bubble up. For example, physics textbooks from the early 1900s rarely use linear algebra, even if they're written well.
Point taken. I wouldn't recommend avoiding anything modern across the board and neither does CS Lewis. And innovation is great but I would guard against assuming that innovation is always positive and a step in the right direction even if not directly. It's also true that many old texts, religious or otherwise, contain timeless wisdom that can inform innovative efforts. And I'm not talking about old by many centuries either. For example, I think many of the hacker types that frequent HN and seek to build something innovative would probably benefit from reading some of Alan Turing's writings. On the other end of the spectrum, maybe Sam Altman could benefit from studying the story of the tower of Babel.
I think pretty soon we will get to the point where there’s some sort of significant boundary at all levels between online and real life because the only way to be sure you’re seeing something real is to be interacting with it in real life. The internet will not be something you visit on a web browser to get information but will become a place you go where you will simply have to acknowledge that nothing is real. Obviously that’s a concern now but I wonder if we’ll get to a point where it’s taken for granted at large that whatever you see on the internet just isn’t real. And I wonder what implications that will have.
This is exactly what’s at stake. I heard an artist say one time that he’d rather listen to Bob Dylan miss a note than listen to a song that had all the imperfections engineered out of it.
They're not connecting to the autotune, but to the artist. People have a lot of opinions about Taylor Swift's music but "not being personal enough" is definitely not a common one.
If you wanna advocate for unplugged music being more gratifying, I don't disagree, but acting like the autotune is what people are getting out of Taylor Swift songs is goofy.
I have no idea about Taylor Swift so I'll ask in general: can't we have a human showing an autotuned personality? Like, you are what you are in private, but in interviews you focus on things suggested by your AI conselor, your lyrics are fine tuned by AI, all this to show a better marketable personality? Maybe that's the autotune we should worry about. Again, nothing new (looking at you, Village People) but nowadays the potential powered by AI is many orders of magnitude higher... you could say yes only until the fans catch wind of it, true, but by that time the next figure shows up and so on. Not sure where this arms escalation can lead us. Because also acceptance levels are shifting, so what we reject today as unacceptable lies could be fine tomorrow, look already at the AI influencers doing a decent job while overtly fake.
I’m convinced it’s already being done, or at least played with. Lots of public figures only speak through a teleprompter. It would be easy to put a fine tuned LLM on the other side of that teleprompter where even unscripted questions can be met with scripted answers.
