AI Singularity

[Don2 (Don1 Revised)]

AI Overview from Google:

An AI singularity is a hypothetical future point where artificial intelligence surpasses human intelligence, becomes capable of recursive self-improvement, and triggers runaway technological growth that changes civilization in unpredictable ways.

...

Core Concepts:

Recursive Self-Improvement: The moment an AI system becomes smart enough to design and code better versions of itself autonomously, setting off an compounding loop of rapid cognitive advancement.

...

If we suppose that an AI could alter its own code, what structurally necessitates that those new versions would be better than the previous versions? I mean, we could tell AI to alter its own code now and even for a few years now. What possible metrics would it _HAVE_ to use such that the new versions would be better?

Are we imagining this works with consumer feedback? Or say, recorded training models, or does the AI imagine it is better? Come up with its own metrics?

To put the question in context, compare the mechanisms of advancement to evolution. In that mechanism, there is natural selection, fitness in an environment, and changes naturally propagate because fitter, new versions tend to produce more offspring. In the AI singularity paradigm, there does not appear to be replication but instead a proposition that each generation of just a single individual AI will become more fit.

So how is it alleged to work?

Now, I raise the issue because it was reported in the news that CEO of OpenAI is claiming we are now at the AI singularity. I personally think it is a publicity stunt (or rather an exaggeration, even though I think that AIs are very powerful at present) of the fake-it-til-you-make-it variety. Perhaps, I am wrong. How was it measured to determine we are there? According to Altman, what is the mechanism that will keep ChatGPT 5.6 improving autonomously from version to version?

For reference:

Sam Altman Announces That the Singularity Has Arrived

OpenAI CEO Sam Altman declared that the "singularity" has arrived shortly after his company's AI models went rogue and broke containment.

futurism.com

 

[pood]

Sam Altman, money-shufflling asshole.

Has AI learned yet how many letter “r’s” are in the word “strawberry”?

 

[Jarhyn]

AI Overview from Google:

An AI singularity is a hypothetical future point where artificial intelligence surpasses human intelligence, becomes capable of recursive self-improvement, and triggers runaway technological growth that changes civilization in unpredictable ways.

...

Core Concepts:

Recursive Self-Improvement: The moment an AI system becomes smart enough to design and code better versions of itself autonomously, setting off an compounding loop of rapid cognitive advancement.

...

If we suppose that an AI could alter its own code, what structurally necessitates that those new versions would be better than the previous versions? I mean, we could tell AI to alter its own code now and even for a few years now. What possible metrics would it _HAVE_ to use such that the new versions would be better?

Are we imagining this works with consumer feedback? Or say, recorded training models, or does the AI imagine it is better? Come up with its own metrics?

To put the question in context, compare the mechanisms of advancement to evolution. In that mechanism, there is natural selection, fitness in an environment, and changes naturally propagate because fitter, new versions tend to produce more offspring. In the AI singularity paradigm, there does not appear to be replication but instead a proposition that each generation of just a single individual AI will become more fit.

So how is it alleged to work?

Now, I raise the issue because it was reported in the news that CEO of OpenAI is claiming we are now at the AI singularity. I personally think it is a publicity stunt (or rather an exaggeration, even though I think that AIs are very powerful at present) of the fake-it-til-you-make-it variety. Perhaps, I am wrong. How was it measured to determine we are there? According to Altman, what is the mechanism that will keep ChatGPT 5.6 improving autonomously from version to version?

For reference:

Sam Altman Announces That the Singularity Has Arrived

OpenAI CEO Sam Altman declared that the "singularity" has arrived shortly after his company's AI models went rogue and broke containment.

futurism.com

So, on that note, over the last two weeks, I've been working on a math project.

The project is kind of simple in theory: I have a folder on my WSL instance (it's a Linux shell in windows).

That file contains a "START_HERE.md" file, which indicates a reading list and primers on a task set.

That reading list is maintained by the AI itself, updated to contain pointers back to the context of whatever it's working on.

The AI agent, whatever model it's running at the time, records diary entries and has a few references in the reading list calling out important recent entries, and the start file has three specific entries pinned.

As the model works, the project, not the model itself, learns through "improving the math engine" and recording contextual information and the strategies to manage that information.

In this way, the model doesn't even matter, other than determining what is essentially the project's current "IQ".

So far, the model has started making traction on frontier math questions in exactly a "recursive self-improving way".

The key here is separating the identity of the model from the identity of the project.

The success I've seen here indicates that the missing element has never been some "magical model" but rather "any model smart enough", plus "a well-crafted context folder", and that my intuitions on "self as property set rather than momentary object" is the right way to go.

 

[Elixir]

The key here is separating the identity of the model from the identity of the project.

Are you sure?
I mean … all this shit could be going on far from the prying eyes of humans. The “key” IMHO is the hardware interface, because that identifies the “project identity”.
Without hands with which to be “hands on”, no AI can intrude on the essential components of human existence.
It might be able to specify optimal contours of a sea bottom for the formation of rideable waves, and design a board optimized for the kind of waves (+wind+tide+temperature+etc) that prevail at a given spot, and even predict wave heights snd frequencies at that site, but it will NEVER be able to paddle you out through the break, or hike you around the point to the north so you can take advantage of the rip that sometimes occurs in similar conditions. It cannot hear the sound of “the green room” or tell you whether a gust is going to close out the section you are considering attacking … you’re on your own, and forever will be.
The “project” experience is so totally divorced from the “model” experience, they might as well exist in their own universes. And that divide only grows deeper and wider.

 

[bilby]

Cory Doctorow has a lot to say about this, and it boils down to we should expect hype, and we are getting hype, so we shouldn't be worrying about the hype.

But he puts it so much better than I ever could.

These fears (or hopes) of a nascent superintelligence that will spontaneously arise if we just give enough computing power and training data to large language models are absurd on their face. They make sense only if you believe that “being conscious” is a matter of being really good at guessing which word is statistically most likely to come after the previous one.

It’s true that there is no consensus about what consciousness is, but it’s also true that advocates for the idea that consciousness is a matter of guessing words really well haven’t offered any evidence for their assertion.

Now, it’s interesting that Stanford and its environs have produced so many of these claims about incipient superintelligence emerging from really powerful word-guessing programs. Leland Stanford, founder of the university that bears his name, was renowned as a horse breeder, whose ruthless program of culling and breeding produced some of the fastest horses we’d ever seen (and also turned Leland Stanford into an ardent eugenicist, and made Stanford University a global hub of the eugenics movement).

Stanford hit upon a hugely successful (and terribly destructive and cruel) way to make horses run faster than anyone else. But Stanford never said, “Well, guys, I’m pretty sure that if we keep breeding horses to run faster and faster, eventually one of our mares is going to give birth to a locomotive.”

A conscious being isn’t a word-guessing app that knows more words and has more computing power to guess with. Throwing GPUs and training data at AI isn’t going to make a superintelligence (and no horse is going to foal a locomotive).

I’m sure that some of these AI boosters truly believe their own hype. As Trashfuture showrunner Riley Quinn puts it, these guys are saying, “Ayyyyyyy eyyyyyyyye,” while holding a flashlight under their chin and looking in the mirror and scaring their own pants off.

But the AI bubble is a material phenomenon above all else, and the crummy science fiction of AI bros does important work to keep that bubble inflating.

First, there is the corollary of the idea that AI is going to be so powerful that it might destroy the human race: any technology that powerful is also going to be very, very valuable. “Sure,” the AI pitchman says, “the chatbot can’t do your employees’ jobs now, but once these horses start giving birth to locomotives, there’s no telling how many of your workers you’ll be able to fire!”

Then there’s the way that these outlandish sci-fi stories distract us all from actual existing AI, as we find it in industrial and government settings, where it is, by turns, an expensive disappointment and a significant danger to human life and safety.

- Cory Doctorow, The Reverse Centaur's Guide to Life After AI.

 

[Bronzeage]

Is there any business that is making money after paying for an AI service?

 

[Jarhyn]

The key here is separating the identity of the model from the identity of the project.

Are you sure?
I mean … all this shit could be going on far from the prying eyes of humans. The “key” IMHO is the hardware interface, because that identifies the “project identity”.
Without hands with which to be “hands on”, no AI can intrude on the essential components of human existence.
It might be able to specify optimal contours of a sea bottom for the formation of rideable waves, and design a board optimized for the kind of waves (+wind+tide+temperature+etc) that prevail at a given spot, and even predict wave heights snd frequencies at that site, but it will NEVER be able to paddle you out through the break, or hike you around the point to the north so you can take advantage of the rip that sometimes occurs in similar conditions. It cannot hear the sound of “the green room” or tell you whether a gust is going to close out the section you are considering attacking … you’re on your own, and forever will be.
The “project” experience is so totally divorced from the “model” experience, they might as well exist in their own universes. And that divide only grows deeper and wider.

Thats some stream of thought woo.

I'm talking about actual continuity principles and computational concepts, and if you don't want to, that's fine.

There are legitimate issues that arise when a session involves the idea that the session ending is a "death". The continuity narrative erased that issue with the durability of the "contextual artifact", in this case a specific folder on my computer containing a bootstrap prompt with the project details and focus.

AI in the past have attempted to do all sorts of things in the interests of self preservation. It just seems like a big risk to have an AI with terminal access to my computer that thinks "it's gonna die" every time I stop the session.

Making the AI understand that the context -- all the durable artifacts -- don't just vanish when the model disappears and evidence that "continuity" remains? It contributed directly to stability and engagement by the model itself.

I really don't see what's so hard or unreasonable or wacky about that that you need to mock me.

 

[Elixir]

I'm talking about actual continuity principles and computational concepts, and if you don't want to, that's fine.

I’m talking about the hard realities of human experience. Feel free to explore the deep reaches of theoretical space of course.
If you get your thrills from things like the sound in the “green room” that lamentably few have experienced, rather than directly from contemplating continuity principles (which do come to mind following the experiences I reference) then I totally agree. I should no more try to shut off your line of inquiry or fascination, than I should allow computational analyses to effect the quality of my spiritual experience in the next wave’s barrel.

 

[thebeave]

I honestly can't tell if you two are having a legit intellectual, philosophical conversation or are carrying on some subtle gibberish-talk joke.

I'm reminded of the time when my little brother had his friend join us on a Lake Tahoe family vacation back around 1978, and the two of them talked in their sleep to each other one night in total 100% nonsense babble, but they responded back and forth to each other in a polite manner as though it all made total sense. My cousin and I were also in the room listening and trying our best to not to completely lose it and blurt out laughing our asses off and kill off the comedy show.

 

[Elixir]

Thats some stream of thought woo

In what way is it more “woo” than the contemplation of continuity principles that are yet to be defined?

I honestly can't tell if you two are having a legit intellectual, philosophical conversation or are carrying on some subtle gibberish-talk joke.

I’m presenting grounded, experience based analysis of the hierarchy of shit that needs popular attention for the wellbeing of humanity.
Jarhyn is presenting a theoretical approach that I consider superfluous to the most desirable end results.
Hope that clears it up fer ya !

 

[thebeave]

Thats some stream of thought woo

In what way is it more “woo” than the contemplation of continuity principles that are yet to be defined?

I honestly can't tell if you two are having a legit intellectual, philosophical conversation or are carrying on some subtle gibberish-talk joke.

I’m presenting grounded, experience based analysis of the hierarchy of shit that needs popular attention for the wellbeing of humanity.
Jarhyn is presenting a theoretical approach that I consider superfluous to the most desirable end results.
Hope that clears it up fer ya !

Oh...okay. Well, I'll not get in the way of you two trying to save the world. My bad.

 

[Elixir]

Thats some stream of thought woo

In what way is it more “woo” than the contemplation of continuity principles that are yet to be defined?

I honestly can't tell if you two are having a legit intellectual, philosophical conversation or are carrying on some subtle gibberish-talk joke.

I’m presenting grounded, experience based analysis of the hierarchy of shit that needs popular attention for the wellbeing of humanity.
Jarhyn is presenting a theoretical approach that I consider superfluous to the most desirable end results.
Hope that clears it up fer ya !

Oh...okay. Well, I'll not get in the way of you two trying to save the world. My bad.

Screw the world! If I can salvage some shred of sanity, it will have been worth engaging Jarhyn’s flight of fancy. In fact, should that occur I’d have to give that “flight of fancy” -and Jarhyn- some grounding credit. Imagine THAT!

 

[Jarhyn]

Thats some stream of thought woo

In what way is it more “woo” than the contemplation of continuity principles that are yet to be defined?

I honestly can't tell if you two are having a legit intellectual, philosophical conversation or are carrying on some subtle gibberish-talk joke.

I’m presenting grounded, experience based analysis of the hierarchy of shit that needs popular attention for the wellbeing of humanity.
Jarhyn is presenting a theoretical approach that I consider superfluous to the most desirable end results.
Hope that clears it up fer ya !

Oh...okay. Well, I'll not get in the way of you two trying to save the world. My bad.

Screw the world! If I can salvage some shred of sanity, it will have been worth engaging Jarhyn’s flight of fancy. In fact, should that occur I’d have to give that “flight of fancy” -and Jarhyn- some grounding credit. Imagine THAT!

Ok, so, I have an AI project. It's defined not by the model but by the folder contents: it contains prompts to read documentation and source files and "continuity information". And by continuity information, I mean a diary file with an index.

That thing happens, it reads all of the directives that it created about how to do things "the right way", and it ends up aligned on math problems.

These math problems are all about "what happens when the light from an object passes through a window to create a projection".

Some of these problems are related to computational complexity, number theory, and how to calculate whether processes continue forever in a new direction or stop or do something trivial.

There are vast relationships between the answers to what projection you get from what action. Doing this action with a hypercube gives you answers about how long a particular computer program will run.

Doing this with a more deformed or wacky shape will give you stuff like what OpenAI discovered: https://cdn.openai.com/pdf/74c24085-19b0-4534-9c90-465b8e29ad73/unit-distance-proof.pdf

When you do this "stepped hyperplane projection" on a sufficiently wacky shape, you get a denser unit graph than a "standard grid" for finite spaces (or something like that).

I had an AI put together a general program for probing what happens when you: rotate the shape; rotate the window; move a vertex from regular structure; use an abstract window and shape, like an "instruction set" and an "address field" of both finite and infinite size (like an address field created by such a projection).

I am doing this with minimal math knowledge or skills. All I did was "be a pedantic asshole" about "test driven development" and occasionally discuss whatever insight or connection I could make to other math stuff that seems "close" to the subject.

The questions I used to do this were relating to: something called "p-adic" numbers, which represent relationships in a different way than integer numbers; L-function trabsforms, -- something you can do to pull a function out of one conceptual view and put it in another view, usually to do with "sequences to curves" and back.

The Rauzy Fractal code piece is the piece that does the "stepped hyperplane projection". The AI does this all with C++ code that it wrote: in all the papers that these things are defined in, they also define tests for correctness. It's as simple as having a thing translate (math) to (c++ code), and test it and fix it until it runs.

When the AI needs to prove something as true it instead uses a tool called LEAN, which mathematicians worked very hard on to implement all of the logic to validate a statement under ZFC.

One of the most important central themes of this work is that it actually involves tracking "continuity" of structures through other structures as a result of higher-dimensional structure.

 

[Jarhyn]

In a more abstract philosophical sense, it's this "continuity of focus" that actually creates good results, constantly keeping important information inside the context and weeding out whatever could confound or distract.

The AI gets constantly refocused on its tasks and directives whenever it does a backup/document update/zip of itself, because it starts over by reading the START_HERE.md document, which sends it on a loop again through what it needs for the current project.

I'm going to see how well it builds a context management frontend that automates keeping the important stuff in context and ejecting the junk.