artificial-intelligence.bigb

= Artificial intelligence
{wiki}

= AI
{c}
{synonym}
{title2}

= AI by capability
{c}
{parent=Artificial intelligence}

= Artificial general intelligence
{parent=AI by capability}
{wiki}

= AGI
{c}
{synonym}
{title2}

Given enough computational power per dollar, AGI is inevitable, but it is not sure certain ever happen given the end of <Moore's law>[end of Moore's Law].

Alternatively, it could also be achieved genetically modified biological brains + <brain in a vat>.

Imagine a brain the size of a building, perfectly engineered to solve certain engineering problems, and giving hints to human operators + taking feedback from cameras and audio attached to the operators.

This likely implies <transhumanism>, and <mind uploading>.

<Ciro Santilli> joined the silicon industry at one point to help increase our computational capacity and reach AGI.

Ciro believes that the easiest route to full AI, if any, could involve <Ciro's 2D reinforcement learning games>.

= Principles of AGI
{parent=Artificial general intelligence}

= The missing link between continuous and discrete AI
{parent=Principles of AGI}

<Ciro Santilli> has felt that perhaps what is missing in 2020's <AGI> research is:
* the interface between:
  * the continuous/noisy level (now well developed under <artificial neural network> techniques of the 2010's)
  * and <symbolic AI> level AI
  The key question is somewhat how to extract symbols out of the space-time continuous experiences.
* more specialized accelerators that somehow interface with more generic <artificial neural networks>. Notably some kind of speialized processing of spacial elements is obviously hardcoded into the brain, see e.g. <grid cell>{full}

Forcing these boundaries to be tested was one of the main design goals of <Ciro's 2D reinforcement learning games>.

In those games, for example:
* when you press a button here, a door opens somewhere far away
* when you touch certain types of objects, a chemical reaction may happen, but not other types of objects
Therefore, those continuous objects would also have "magic" effects that could not be explained by "simple" "what is touching what" ideas.

Bibliography:
* https://mitpress.mit.edu/9780262632683/the-algebraic-mind/

= Intelligence is hierarchical
{parent=Principles of AGI}

This point is beautifully argued in lots of different sources, and is clearly a pillar of <AGI>.

Perhaps one may argue that our <deep learning> layers do form some kind of hierarchy, e.g. this is very clear in certain models such as <convolutional neural network>. But many of those models cannot have arbitrarily deep hierarchies, which appears to be a fundamental aspect of intelligence.

<How to Create a Mind>:
> The lists of steps in my mind are organized in hierarchies. I follow a routine procedure before going to sleep. The first step is to brush my teeth. But this action is in turn broken into a smaller series of steps, the first of which is to put toothpaste on the toothbrush. That step in turn is made up of yet smaller steps, such as finding the toothpaste, removing the cap, and so on. The step of finding the toothpaste also has steps, the first of which is to open the bathroom cabinet. That step in turn requires steps, the first of which is to grab the outside of the cabinet door. This nesting actually continues down to a very fine grain of movements, so that there are literally thousands of little actions constituting my nighttime routine. Although I may have difficulty remembering details of a walk I took just a few hours ago, I have no difficulty recalling all of these many steps in preparing for bed - so much so that I am able to think about other things while I go through these procedures. It is important to point out that this list is not stored as one long list of thousands of steps - rather, each of our routine procedures is remembered as an elaborate hierarchy of nested activities.

<Human Compatible>: TODO get exact quote. It was something along: life goal: save world from hunger. Subgoal: apply for some grant. Sub-sub-goal: eat, sleep, take shower. Sub-sub-sub-goal: move muscles to get me to table and open a can.

= AGI architecture
{c}
{parent=Principles of AGI}

\Video[https://youtu.be/pd0JmT6rYcI?t=3536]
{title=From Machine Learning to Autonomous Intelligence by <Yann LeCun> (2023)}
{description=
After a bunch of B.S., LeCun goes on to describe his AGI architecture. Nothing ground breaking, but not bad either.
* https://youtu.be/pd0JmT6rYcI?t=3705[]: <intelligence is hierarchical>
}

Bibliography:
* https://www.reddit.com/r/agi/comments/108e7n1/best_starting_papersbooks_to_read_to_try_to/

= Elements of AGI
{c}
{parent=AGI architecture}

This section is about ideas that are thought to be part of an AGI system.

= Common sense
{parent=Elements of AGI}
{wiki}

\Video[https://www.youtube.com/watch?v=49t-WWTx0RQ]
{title=My Job is to Open and Close Doors by Mattias Pilhede (2019)}
{description=An interesting humorous short meditation on <common sense>.}

= Instrumental goal
{parent=Elements of AGI}

= Instrumental convergence
{parent=Instrumental goal}
{wiki}

= AGI research
{c}
{parent=Artificial general intelligence}

= History of AGI research
{c}
{parent=AGI research}

= AGI blues
{c}
{parent=History of AGI research}

Term invented by <Ciro Santilli>, similar to "<nuclear blues>", and used to describe the feeling that every little shitty job you are doing (that does not considerably help achieving <AGI>) is completely pointless given that we are likely close to <AGI> as of 2023.

= AGI excitement
{c}
{parent=AGI blues}

The opposite of the <AGI blues>. In 2025 <Ciro Santilli> fell well in this camp.

= Moravec's paradox
{c}
{parent=History of AGI research}
{title2=1980s}
{wiki}

= AI winter
{c}
{parent=History of AGI research}
{wiki}

= AI boom
{c}
{parent=History of AGI research}
{title2=2012-}
{wiki}

= AGI research has become a taboo in the early 21st century
{c}
{parent=History of AGI research}

Due to the failures of earlier generations, which believed that would quickly achieve <AGI>, leading to the <AI winters>, 21st researchers have been very afraid of even trying it, rather going only for smaller subste problems like better neural network designs, at the risk of being considered a <crank (person)>.

While there is fundamental value in such subset problems, the general view to the final goal is also very important, we will likely never reach AI without it.

This is voiced for example in <Superintelligence by Nick Bostrom (2014)> section "Opinions about the future of machine intelligence" which in turn quotes Nils Nilsson:
> There may, however, be a residual cultural effect on the AI community of its earlier history that makes many mainstream researchers reluctant to align themselves with over-grand ambition. Thus Nils Nilsson, one of the old-timers in the field, complains that his present-day colleagues lack the boldness of spirit that propelled the pioneers of his own generation:
  > Concern for "respectability" has had, I think, a stultifying effect on some AI researchers. I hear them saying things like, "AI used to be criticized for its flossiness. Now that we have made solid progress, let us not risk losing our respectability." One result of this conservatism has been increased concentration on "weak AI" - the variety devoted to providing aids to human
    thought - and away from "strong AI" - the variety that attempts to mechanize human-level intelligence
  Nilsson’s sentiment has been echoed by several others of the founders, including Marvin Minsky, John McCarthy, and Patrick Winston.

<Don't be a pussy>, AI researchers!!!

= AGI interest group
{parent=AGI research}

= AGI House
{parent=AGI interest group}

* https://www.agihouse.org/
* https://www.businessinsider.com/heres-how-agi-house-bays-hottest-artificial-intelligence-hacker-2023-6

= AGI conference
{c}
{parent=AGI interest group}

https://www.agi-conference.org/

It is hard to overstate how low the level of this conference seems to be at first sight. <AGI research has become a taboo in the early 21st century>[Truly sad].

= Open AGI Summit
{c}
{parent=AGI conference}

https://openagi.xyz/

The tagline smells like bullshit:
> Integrating AI with web3 and decentralized ecosystems

= Journal of Artificial General Intelligence
{c}
{parent=AGI research}

https://sciendo.com/journal/JAGI

= AGI research entity
{c}
{parent=AGI research}
{tag=AI research entity}

* https://www.quora.com/What-are-some-good-research-schools-PhD-for-Artificial-General-Intelligence-not-Machine-Learning/answer/Ciro-Santilli What are some good research schools (PhD) for Artificial General Intelligence (not Machine Learning)?
* 2020 https://towardsdatascience.com/four-ai-companies-on-the-bleeding-edge-of-artificial-general-intelligence-b17227a0b64a Top 4 AI companies leading in the race towards Artificial General Intelligence
* Douglas Hofstadter according to https://www.theatlantic.com/magazine/archive/2013/11/the-man-who-would-teach-machines-to-think/309529/ The Man Who Would Teach Machines to Think (2013) by <James Somers>
* Pei Wang from Temple University: https://cis.temple.edu/~wangp/
* https://www.reddit.com/r/agi/comments/zzfwww/are_there_people_actually_working_to_make_an_agi/
* <Sergey Brin> explicit internal memo aiming at <AGI>: https://techcrunch.com/2025/02/28/sergey-brin-says-rto-is-key-to-google-winning-the-agi-race/

= Amazon AGI team
{tag=Amazon division}
{parent=AGI research entity}

https://amazon.jobs/content/en/teams/agi

= Giotto.ai
{c}
{parent=AGI research entity}

https://www.giotto.ai/
> At Giotto.ai, our technology is designed to bridge the gap between current AI capabilities and the promise of Artificial General Intelligence (AGI).
Their website doesn't clearly explain their technology as of 2025.

They claim to have done some work on <ARC-AGI> which is cool, but no clear references to what they did or if there's anything public about it.

= Kyutai
{c}
{parent=AGI research entity}

https://kyutai.org/ just says:
> Our mission is to build and democratize artificial general intelligence through open science

They are <not-for-profit> and had massive investments: https://techcrunch.com/2023/11/17/kyutai-is-an-french-ai-research-lab-with-a-330-million-budget-that-will-make-everything-open-source/

they also don't say at all what they are looking into for <agi>, the only public thing they have are <speech-to-speech> and <speech to text> so how's that related to agi at.

= mit quest for intelligence
{c}
{parent=agi research entity}
{tag=mit}

https://quest.mit.edu/about/vision-statement

= safe superintelligence inc.
{c}
{parent=agi research entity}

https://ssi.inc/

raised \$1b at \$5b valuation on september 2024, then \$2b at \$30b on march 2025. lol!

From their website:
> Superintelligence is within reach.
> Our singular focus means no distraction by management overhead or product cycles, and our business model means safety, security, and progress are all insulated from short-term commercial pressures.

= Steven Byrnes
{c}
{parent=AGI research entity}
{tag=Astera Institute person}
{wiki}

* https://sjbyrnes.com/
* https://twitter.com/steve47285
* https://www.lesswrong.com/posts/diruo47z32eprenTg/my-computational-framework-for-the-brain

= Astera Institute
{c}
{parent=AGI research entity}
{tag=Hipster research institute}
{tag=Jed McCaleb}

https://astera.org/agi/

By the rich founder of <Mt. Gox> and Ripple, <Jed McCaleb>.
> Obelisk is the Artificial General Intelligence laboratory at Astera. We are focused on the following problems: How does an agent continuously adapt to a changing environment and incorporate new information? In a complicated stochastic environment with sparse rewards, how does an agent associate rewards with the correct set of actions that led to those rewards? How does higher level planning arise?

= Hipster research institute
{parent=Astera Institute}

These are research institutes usually funded by rich tech bros, sometimes <cryptocurrency> magnates, but not necessarily.

= Topos institute
{parent=Hipster research institute}

https://topos.institute/

= Astera Institute person
{c}
{parent=Astera Institute}

= Michael Nielsen
{c}
{parent=Astera Institute person}

Interesting dude, with some interest overlaps with <Ciro Santilli>, like <quantum computing>:
* https://github.com/mnielsen
* https://michaelnielsen.org/
* https://twitter.com/michael_nielsen
* https://www.youtube.com/c/michaelnielsen

= FutureAI
{c}
{parent=AGI research entity}

= Future AI
{c}
{synonym}
{title2}

It is a bit hard to decide if those people are serious or not. Sometimes it feels scammy, but sometimes it feels fun and right!

Particularly concerning is the fact that they are not a <not-for-profit> entity, and it is hard to understand how they might make money.

<Charles Simon>, the founder, is pretty focused in how natural neurons work vs <artificial neural network> models. He has some good explanations of that, and one major focus of the project is their semi open source spiking neuron simulator <BrainSimII>. While <Ciro Santilli> believes that there might be insight in that, he also has doubts if certain modules of the brain wouldn't be more suitable coded directly in regular <programming languages> with greater ease and performance.

FutureAI appears to be Charles' retirement for fun project, he is likely <independently wealthy>. Well done.

* https://www.aitimejournal.com/interview-with-charles-simon-ceo-and-founder-futureai
* 2022 raised 2 million USD:
  * https://www.prnewswire.com/news-releases/ai-futureai-raises-2-million-to-develop-artificial-general-intelligence-301459164.html

\Video[https://www.youtube.com/watch?v=ivbGbSx0K8k]
{title=Creativity and <AGI> by <Charles Simon>'s at AGI-22 (2022)}
{description=
Sounds OK!
* https://youtu.be/ivbGbSx0K8k?t=856 general structure of the <human brain> 86B total, matching <number of neurons in the human brain>, with:
  * 14B: brainstem
  * 16B: <neocortex>
  * 56B: cerebelum
* https://www.youtube.com/watch?t=1433 some sequencing ideas/conjectures
}

\Video[https://www.youtube.com/watch?v=KQP1gPTk0FI]
{title=Machine Learning Is Not Like Your Brain by <Future AI> (2022)}
{description=Contains some <BrainSimII> demos.}

= BrainSimII
{c}
{parent=FutureAI}
{tag=Neuron simulator}

https://github.com/FutureAIGuru/BrainSimII

The video from https://futureai.guru/technologies/brian-simulator-ii-open-source-agi-toolkit/ shows a demo of the possibly non open source version. They have a <GUI> neuron viewer and editor, which is kind of cool.

\Video[https://www.youtube.com/watch?v=KQP1gPTk0FI]
{title=Machine Learning Is Not Like Your Brain by <Charles Simon> (2022)}

= Sallie
{disambiguate=FutureAI}
{c}
{parent=FutureAI}
{tag=AI training robot}

Not having a manipulator claw is a major issue with this one.

But they also have a co-simulation focus, which is a bit of a win.

= Charles Simon
{c}
{parent=FutureAI}

* https://www.linkedin.com/in/charles-simon-futureai/
* https://futureai.guru/about/the-team/

Basically it looks like the dude got enough money after selling some companies, and now he's doing cooler stuff without much need of money. Not bad.

= GoodAI
{c}
{parent=AGI research entity}

<Marek Rosa>'s play thing.

= AI People
{c}
{parent=GoodAI}
{tag=AI game with natural language}
{title2=2023}

\Video[https://www.youtube.com/watch?v=xkn0H_iWDEQ]
{title=AI Game - LLM-driven NPCs that can talk by Marek Rosa (2023)}
{description=Not the most amazing demo, but the idea is there. Seems to be a preview for <AI People>. The previous working title seems to have been AI Odyssey.}

= Marek Rosa
{c}
{parent=GoodAI}
{wiki}

= NDEA
{c}
{parent=AGI research entity}

https://ndea.com/
> We believe program synthesis holds the key to unlocking <AGI>.
Cool. Founders are also very interested in <ARC-AGI>.

= Numenta
{c}
{parent=AGI research entity}

Homepage: https://www.numenta.com/

= Numenta employee
{parent=Numenta}

= Jeff Hawkins
{c}
{parent=Numenta employee}
{wiki}

= Hierarchical temporal memory
{parent=Numenta}
{tag=AGI architecture}
{wiki}

\Video[https://www.youtube.com/watch?v=XMB0ri4qgwc]
{title=HTM Overview (Episode 0) by <Numenta>}

= On Intelligence
{c}
{parent=Hierarchical temporal memory}
{title2=2004}
{wiki}

\Image[https://upload.wikimedia.org/wikipedia/en/b/bd/OnInt.png]

= Sakana.AI
{c}
{parent=AGI research entity}

https://sakana.ai

Their description is a bit of localization randomness:
> We are building a world class AI research lab in Tokyo.
  
  We want to develop AI solutions for Japan's needs, and democratize AI in Japan.

\Video[https://www.youtube.com/watch?v=DtePicx_kFY]
{title=I Co-Invented the Transformer. Now I'm Replacing It.}
{description=Interview with Sakana.AI co-founders Llion Jones and Luke Darlow by Machine Learning Street Talk published Nov 23, 2025.}

= AGI software
{c}
{parent=Artificial general intelligence}

= Artificial general intelligence software
{synonym}
{title2}

* https://ai.stackexchange.com/questions/5428/how-can-people-contribute-to-agi-research mentions:
  * https://github.com/opennars/opennars
  * https://github.com/brohrer/robot-brain-project

= OpenCog
{c}
{parent=AGI software}
{wiki}

= Ben Goertzel
{c}
{parent=OpenCog}
{tag=AGI research entity}
{wiki}

https://www.reddit.com/r/artificial/comments/b38hbk/what_do_my_fellow_ai_researchers_think_of_ben/ What do my fellow AI researchers think of Ben Goertzel and his research?

= SingularityNET
{c}
{parent=Ben Goertzel}
{wiki}

https://singularitynet.io/

<Ben Goertzel>'s <fog computing> project to try and help achieve <AGI>.

= NuNET
{c}
{parent=SingularityNET}
{tag=Fog computing}

= AGI-complete
{c}
{parent=Artificial general intelligence}
{tag=Complexity class}
{wiki}

= AI-complete
{synonym}

Term invented by <Ciro Santilli> to refer to problems that can only be solved once we have <AGI>.

It is somewhat of a flawed analogy to <NP-complete>.

= Polanyi's paradox
{c}
{parent=Artificial general intelligence}
{title2=We can know more than we can tell}

= Mechanistic interpretability
{parent=Polanyi's paradox}
{wiki}

* https://x.com/aif_media/status/1923028051149062607

= Interpretability
{synonym}

= AGI test
{c}
{parent=Artificial general intelligence}
{wiki=https://en.wikipedia.org/w/index.php?title=Artificial_general_intelligence&oldid=1192191193#Tests_for_human-level_AGI}

= AGI benchmark
{c}
{synonym}

= CAPTCHA
{c}
{parent=AGI test}
{wiki}

= reCAPTCHA
{c}
{parent=CAPTCHA}
{wiki}

= Turing test
{c}
{parent=AGI test}
{wiki}

= ARC-AGI
{c}
{parent=AGI test}

= The Abstraction and Reasoning Challenge
{synonym}
{title2}

* https://arcprize.org/
* https://lab42.global/arc/
* https://pgpbpadilla.github.io/chollet-arc-challenge

This one goes all in the following themes:
* few examples to learn from. You have to carefully inspect the input examples to deduce the output rules. Rules can require specific It application ordering, so you actually generate an algorithm. It tends to be easy for humans, but sometimes not so easy!
* extensive use of geometric concepts, notably "contained inside", "adjacent to", "connected"

Bibliography:
* https://www.reddit.com/r/mlscaling/comments/1ht4emi/anyone_else_suspect_arcagi_was_never_much_of_a/ Anyone else suspect ARC-AGI was never much of a test of anything? (2025)

= ARC-AGI theory
{c}
{parent=ARC-AGI}

= Information theoretical musings inspired by ARC-AGI
{parent=ARC-AGI theory}

The extreme <overfitting> case of training is to have a map where each input leads to one output.

However it is cool that this overfit does not allow you to compute the final input for which there is no known output.

This therefore forces the creation of more general solution rules.

While in some cases solutions can work for any input, in many others they require specific assumptions about input, but the model could simply check that the assumptions apply to all inputs and use them for the final algorithm.

Bibliography:
* https://lewish.io/posts/how-to-beat-arc-agi-2

= ARC-AGI is a black hole for early retired tech and finance bros
{parent=ARC-AGI theory}

People who do cool open tech stuff when don't need money anymore are awesome:
* François Chollet, project founder: https://www.linkedin.com/in/fchollet/ 9 years at Google from 2015 to 2024. He founded ARC while he was still at Google though, so maybe doesn't count
* Cristiano Calgano from <cristianoc arc-agi-2-abstraction-dataset>. <Imperial College London> researcher who founded a <formal verification> company and sold it to <Facebook> where he staid for 7 years

= ARC-AGI visualization
{c}
{parent=ARC-AGI}

https://www.kaggle.com/code/allegich/arc-agi-2025-visualization-all-1000-120-tasks contains plots of all questions and answers. It is truly very convenient.

= ARC-AGI approach
{c}
{parent=ARC-AGI}

* https://arcprize.org/guide
* https://lewish.io/posts/how-to-beat-arc-agi-2

= ARC-AGI feature extraction
{c}
{parent=ARC-AGI approach}

https://www.kaggle.com/code/allegich/eda-statistical-analysis-and-feature-extraction has a very basic feature extraction.

= ARC-AGI implementation
{c}
{parent=ARC-AGI}

Bibliography:
* https://github.com/zoecarver/grover-arc/issues/1

= ARC-DSL
{c}
{parent=ARC-AGI implementation}

https://github.com/michaelhodel/arc-dsl

This interesting repo defines a set of input transformations that can be composed together into programs to generate the solve ARC problems.

It does not appear to have any <program synthesis>: it only defines the DSL and then provides manual solutions to the problems.

The README is lacking as usual, an overview of the files is:
* https://github.com/michaelhodel/arc-dsl/blob/635de4902a5fb4e376f27333feaa396d3f5dfdcb/dsl.py[dsl.py]: defines the transformations as Python functions
* https://github.com/michaelhodel/arc-dsl/blob/635de4902a5fb4e376f27333feaa396d3f5dfdcb/solvers.py[solvers.py]: defines solvers for the 400 <ARC-AGI-1 training problems>

Intended usage to run the solvers seems to be:
``
git clone https://github.com/fchollet/ARC-AGI
cd ARC-AGI
git checkout 399030444e0ab0cc8b4e199870fb20b863846f34
git clone https://github.com/michaelhodel/arc-dsl
cd arc-dsl
git checkout 635de4902a5fb4e376f27333feaa396d3f5dfdcb
python main.py
``
Unfortunately this blows up on <Ubuntu 25.04> on `test_mpapply` apparently due to a <Python 3.12> issue and the pull request https://github.com/michaelhodel/arc-dsl/pull/7 has been ignored for more than one year, so the project is largely dead.

= ARC-DSL-2
{c}
{parent=ARC-AGI implementation}

https://github.com/arc-dsl-2/arc-dsl-2

<Ciro Santilli>'s fork of <ARC-DSL> merging all pull requests needed to make tests run again on <Ubuntu 25.04>.

= cristianoc/arc-agi-2-abstraction-dataset
{parent=ARC-AGI implementation}

https://github.com/cristianoc/arc-agi-2-abstraction-dataset

Contains 120 DSL implementations for the 

From another awesome retired tech bro that does this project for fun.

= ARC-AGI without LLM
{c}
{parent=ARC-AGI implementation}

Some mentions at: https://arcprize.org/blog/arc-prize-2025-results-analysis section "Zero-Pretraining Deep Learning Methods".

= CompressARC
{c}
{parent=ARC-AGI without LLM}

https://iliao2345.github.io/blog_posts/arc_agi_without_pretraining/arc_agi_without_pretraining.html

= mdlARC
{c}
{parent=ARC-AGI without LLM}

https://github.com/mvakde/mdlARC

= Local CPU ARC-AGI without LLM
{parent=ARC-AGI without LLM}

* https://iliao2345.github.io/blog_posts/arc_agi_without_pretraining/arc_agi_without_pretraining.html

= aviad12g/ARC-AGI-solution 
{c}
{parent=Local CPU ARC-AGI without LLM}

https://github.com/aviad12g/ARC-AGI-solution

Interesting looking repo with optional <GPU> and optional <LLM>.

It seems to have been tested on something older than <Ubuntu 24.04>, as 24.04 install requires some porting, started process at: https://github.com/cirosantilli/ARC-AGI-solution/tree/ubuntu-24-04 but gave up to try <Ubuntu 22.04> instead.

<Ubuntu 22.04> <Docker> install worked without patches, after installing <Poetry (Python)> e.g. to try and solve <aRC-AGI-2 problem/list/1ae2feb7>:
``
git clone https://github.com/aviad12g/ARC-AGI-solution
cd ARC-AGI-solution
git checkout f3283f727488ad98fe575ea6a5ac981e4a188e49
poetry install
git clone https://github.com/arcprize/ARC-AGI-2
`poetry env activate`
export PYTHONPATH="$PWD/src:$PYTHONPATH"
python3 -m arc_solver.cli.main solve ARC-AGI-2/data/evaluation/1ae2feb7.json
``
but towards the end we have:
``
{
  "success": false,
  "error": "Search failed: no_multi_example_solution",
  "search_stats": {
    "nodes_expanded": 21,
    "nodes_generated": 903,
    "termination_reason": "no_multi_example_solution",
    "candidates_generated": 25,
    "examples_validated": 3,
    "validation_success_rate": 0.0,
    "multi_example_used": true
  },
  "predictions": [
    null,
    null,
    null
  ],
  "computation_time": 30.234344280001096,
  "task_id": "1ae2feb7",
  "task_file": "ARC-AGI-2/data/evaluation/1ae2feb7.json",
  "solver_version": "0.1.0",
  "total_time": 30.24239572100123,
  "timestamp": 1760353369.9701269
}

Task: 1ae2feb7.json
Success: False
Error: Search failed: no_multi_example_solution
Multi-example validation: ENABLED
Training examples validated: 3
Candidates generated: 25
Validation success rate: 0.0%
Computation time: 30.23s
Total time: 30.24s
``
so it failed.

Let's see if any of them work at all as advertised:
``
ls ARC-AGI-2/data/evaluation/ | xargs -I'{}' python3 -m arc_solver.cli.main solve 'ARC-AGI-2/data/evaluation/{}' |& tee tmp.txt
``
and at the end:
``
grep 'Success: True' tmp.txt | wc
``
has only 7 successes.

Also weirdly 
``
grep 'Success: True' tmp.txt | wc
``
only has 102 hits, but there were 120 JSON tasks in that folder. I search for the missing executions:
``
diff -u <(grep Task: tmp.txt | cut -d' ' -f2) <(ls ARC-AGI-2/data/evaluation)
``
The first missing one is 135a2760, it blows up with:
``
ERROR: Solve command failed: Object of type HorizontalLinePredicate is not JSON serializable
``
and grepping ERROR gives us:
``
ERROR: Solve command failed: Object of type HorizontalLinePredicate is not JSON serializable
ERROR: Solve command failed: Object of type SizePredicate is not JSON serializable
ERROR: Solve command failed: Object of type HorizontalLinePredicate is not JSON serializable
ERROR: Solve command failed: Object of type HorizontalLinePredicate is not JSON serializable
ERROR: Solve command failed: Object of type ndarray is not JSON serializable
ERROR: Solve command failed: Object of type HorizontalLinePredicate is not JSON serializable
ERROR: Solve command failed: Object of type ndarray is not JSON serializable
ERROR: Solve command failed: Object of type HorizontalLinePredicate is not JSON serializable
ERROR: Solve command failed: Object of type VerticalLinePredicate is not JSON serializable
ERROR: Solve command failed: Object of type VerticalLinePredicate is not JSON serializable
ERROR: Solve command failed: Object of type ndarray is not JSON serializable
ERROR: Solve command failed: Object of type VerticalLinePredicate is not JSON serializable
ERROR: Solve command failed: Object of type ndarray is not JSON serializable
ERROR: Solve command failed: Object of type HorizontalLinePredicate is not JSON serializable
ERROR: Solve command failed: Object of type HorizontalLinePredicate is not JSON serializable
ERROR: Solve command failed: Object of type HorizontalLinePredicate is not JSON serializable
ERROR: Solve command failed: Object of type VerticalLinePredicate is not JSON serializable
ERROR: Solve command failed: Object of type VerticalLinePredicate is not JSON serializable
``
Reported at: https://github.com/aviad12g/ARC-AGI-solution/issues/1

= ARC-AGI problem set
{c}
{parent=ARC-AGI}

= ARC-AGI version
{c}
{synonym}

= Official ARC-AGI problem set
{parent=ARC-AGI problem set}

= ARC-AGI-1
{c}
{parent=Official ARC-AGI problem set}

= ARC-AGI-1 problem
{parent=ARC-AGI-1}
{scope}

= Train
{parent=ARC-AGI-1 problem}

= ARC-AGI-1 training problem
{synonymNoScope}

= 007bbfb
{parent=Train}
{title2=1}

https://arcprize.org/play?task=007bbfb7

Hard input constraints:
* inputs are 3x3
* inputs contain only 2 colors monocolored: black and another

Hard output constraints:
* output is 3x input width and height. Suggests that the output is a 3x3 grid based on the input.
  * stronger: if output is split as a 3x3 grid, then each 3x3 block is either black or a copy as input. Which is which?
    * stronger: each pixel of the input determines if block is black or copy (final solution)
* output contains only two colors: black and another
  * stronger: the same two colors as input

Input output comparison:
* input appears pasted on output multiple times: suggests it is being copy pasted

Hard output constraints:
* output is 3x input width and height: suggests that the output is a 3x3 grid based on the input

  If that is the case, let's try to figure out what is placed on each output grid.

  Notice: each grid element is either blank or the input.

  OK so let's determine what in the input determines each output grid.

  Because input in 3x3 maybe there's a direct mapping.

= 00d62c1b
{parent=Train}
{title2=2}

https://arcprize.org/play?task=00d62c1b

Hard input constraints:
* inputs have two colors: green and black

Hard output constraints:
* output has three colors: black, green and yellow
* output has same size as input
* green is copied from input to output
  * output differs from input by making some black pixels yellow. Which pixels are becoming yellow?
    * output differs from input by making inner regions (non-diagonal) yellow with non-diagonal flood fill

= 017c7c7b
{parent=Train}
{title2=3}

https://arcprize.org/play?task=017c7c7b

Input constraints:
* inputs are 3x6

Output constraints:
* outputs are 3x9

TODO: this one is quite challenging.

= 025d127b
{parent=Train}
{title2=4}

https://arcprize.org/play?task=025d127b

Output constraints:
* Input and output have the same size
* Supposing background is black, input and output contain the same number of objects of each color
  * the lower right part of each object (non-diagonal) does not move
    * the rest of each object outside the lower right part moves by 1 square to the right

= 0520fde7
{parent=Train}
{title2=6}

https://arcprize.org/play?task=0520fde7

= Eval
{parent=ARC-AGI-1 problem}

= ARC-AGI-2
{c}
{parent=Official ARC-AGI problem set}
{title2=2025-03-24}

* https://github.com/arcprize/ARC-AGI-2
* https://arcprize.org/play?task=1ae2feb7

Has the following structure:

* <ARC-AGI-2 training dataset>[Training Set]: 1000 tasks: a variety of difficulty levels, easy to hard, designed to contain all "primitives" needed for eval
* <ARC-AGI-2 public evaluation dataset>[Public Eval Set]: 120 tasks
* Semi-Private Eval: 120 hard tasks, may have been exposed to limited third-parties eg. via API
* Private Eval Set: 120 tasks, never exposed to third parties

= ARC-AGI-2 problem
{c}
{parent=ARC-AGI-2}
{scope}

= Approach
{c}
{parent=ARC-AGI-2 problem}

= Primitive
{parent=Approach}

= Feature extraction
{synonym}

These section lists common visual primitives that a solver must first extract in order to infer solutions.

Some of these have a lot of prior world content, others less.

Many people have come up with the same idea on the Discord. Some nicely call it <domain specific language>[DSL].

Implementations:
* <ARC-DSL>

= Input primitive
{parent=Primitive}

= Background color
{parent=Input primitive}

If a color is inferred to be a background color, it contains no information and should be ignored.

Most problems tend to use black as a background color, but not all of them.

= Object
{parent=Input primitive}

An "object" is a set of points that is understood to be one singular entity.

Contiguity and having the same color are strong indicators that something should be understood as an object.

= Container
{parent=Object}

= Box
{parent=Container}

A rectangular container.

The toplevel viewport is always implicitly understood as a special box.

= Edge
{parent=Box}

= Left edge
{parent=Edge}

= Right edge
{parent=Edge}

= Top edge
{parent=Edge}

= Bottom edge
{parent=Edge}

= Toplevel box
{parent=Box}

There are two or more boxes drawn inside the toplevel and sharing boundaries with toplevel.

= Two toplevel boxes
{parent=Toplevel box}

= Input output toplevel boxes
{parent=Two toplevel boxes}

There are <two toplevel boxes>, one contains only input, and all output goes to the second one. The second one may also contain some input.

= Monocolor object
{parent=Object}

= Primitive relation
{parent=Object}

= Distance
{parent=Primitive relation}

A path is something you obtain by somehow drawing from one point to another, e.g. a <line>, and then starting another drawing between two points from the end point.

= Adjacent
{parent=Distance}

= Touching
{synonym}
{title2}

<Distance> = 0.

= Rectangle
{parent=Object}

Rectangle is like a box but always fully filled.

= Square
{parent=Rectangle}

= Point
{parent=Square}

A point is a 1-<square>.

= Path
{parent=Object}

= Dotted path
{parent=Path}

A dotted line is a generalized line that cycles between a color pattern, e.g.:
> r r g
would be a line:
> r r g r r g r r g
An extra color "transparent" may also be added to not change for that pixel.

= Line
{parent=Path}

= Dotted line
{parent=Line}

A <dotted path> that is also a <dotted line>.

= Monocolor line
{parent=Line}

= Perpendicular line
{parent=Line}

= Vertical line
{parent=Perpendicular line}

= Horizontal line
{parent=Perpendicular line}

= Diagonal line
{parent=Line}

= Repeat
{parent=Input primitive}

= Output primitive
{parent=Primitive}

= Optimize
{parent=Output primitive}

There is no unique solution, we just have to optimize something, often the least changed colors.

= Draw line
{parent=Output primitive}

= List
{parent=ARC-AGI-2 problem}
{scope}

= Train
{parent=List}

= ARC-AGI-2 training dataset
{synonymNoScope}

= Eval
{parent=List}

= ARC-AGI-2 public evaluation dataset
{synonymNoScope}

= 1ae2feb7
{parent=Eval}
{title2=1}

https://arcprize.org/play?task=1ae2feb7

To the left of the vertical red line, count the number of each color on each row.

Then to the right, on each line draw one square of each color to the left every n columns, starting with a square on the first column to the right of the red line, where n is the count of that color.

Start with the color furthest away from the red line, and then color with colors nearer to the red line. If there's overlap, replace the old color with the new one.

Input:
* <background color>
* <dotted line>
* <monocolor line>
* <box>
* <input output toplevel boxes>

Output:
* draw <dotted lines>

= 3e6067c3
{parent=Eval}
{title2=2}

https://arcprize.org/play?task=3e6067c3

Input primitives:
* <ARC-AGI-2 problem/background color>
* squares
  * squares with color inside
* points

Transformations primitives:
* line drawing

= 16b78196
{parent=Eval}
{title2=3}

https://arcprize.org/play?task=16b78196

Solution: move pieces to fill the gap on the fat object that crosses the screen. Place objects either on fat object or on other objects placed on the fat object. Anything you add must end in a rectangle.

The rules for this one are not entirely clear with the number of examples.

Also clearly if the goal is to make rectangular towers, then this is an <NP-hard> optimization problem in general.

Input primitives:
* same color chunk. Properties: crosses screen.

Transformation primitives:
* move solid around
* fills the gap

This existed earlier: https://x.com/GianpaoloGalli/status/1846144236900827413

= 142ca369
{parent=Eval}
{title2=4}

https://arcprize.org/play?task=142ca369

Solution: vs are guns that shoot diagonal line of their color, when line touches another object, change line color to match that of the object, then bounce on the object and continue going with the new color

Input primitives:
* diagonal line

Assumptions:
* line don't cross each other, it is unclear how to resolve that case

Transformation primitives:
* draw line
  * draw line and bounce

= 136b0064
{parent=Eval}
{title2=5}

https://arcprize.org/play?task=136b0064

Input primitive:
* <monocolor object>
* 2 <toplevel boxes>

Transformation primitives:
* draw <lines>

= 0934a4d8
{parent=Eval}
{title2=6}

https://arcprize.org/play?task=0934a4d8

TODO I can't solve that one.

= 135a2760
{parent=Eval}
{title2=7}

https://arcprize.org/play?task=135a2760

Input:
* <background color>
* <box>
* <input output toplevel boxes>
* <repeat>

Output:
* make <repeat>
* <optimize>

= 13e47133
{parent=Eval}
{title2=8}

https://arcprize.org/play?task=13e47133

Input:
* <background color>
* <boxes>
* <points> inside boxes
  * <distance> between point and box

Output:
* make <repeat>
* <optimize>

= 195c6913
{parent=Eval}
{title2=10}

https://arcprize.org/play?task=195c6913

Input: three or more <containers>:
* one <touching> top left corner
  * inside it there are three <monocolor objects>
* one touching bottom right corner of <toplevel box>
  * inside it there is one <monocolor object>
* outside of those, touching the left toplevel box edge, there is one or more <point>

Output:
* draw <dotted path> of <perpendicular line>
  * the path color pattern comes from the color of top left objects, ordered from nearest to furthest from top le

= ARC-AGI-3
{c}
{parent=Official ARC-AGI problem set}
{tag=Gridworld AI game}

They are moving to <2d discrete AI games>.

Although there is merit in that, it is a shame that it just similar to other pre-existing work such as <gvgai> and many others.

Solutions to these solutions require much more thought to formalize a solution.

Also the solutions are much less unique, finding the actual optimal solution being obviously <NP-hard>.

These aspects make those games much less elegant than the older ARC-AGI 1 and 2 counterparts.

= Unofficial ARC-AGI problem set
{c}
{parent=Official ARC-AGI problem set}

This section is about unofficial <ARC-AGI>-like problem sets.

These are interesting from both a:
* practical point of view, as they provide more training data for potential solvers. If you believe that they are representative that is of course.
* theoretical point of view, as they might help to highlight missing or excessive presumptions of the official datasets

https://github.com/neoneye/arc-dataset-collection contains a fantastic collection of such datasets, with visualization at: https://neoneye.github.io/arc/

= ARC-AGI problem generator
{parent=Unofficial ARC-AGI problem set}

= re-arc
{parent=ARC-AGI problem generator}

https://github.com/michaelhodel/re-arc

By the author of <ARC-DSL>.

README says:
> This repository presents code to procedurally generate examples for the ARC training tasks. For each of the 400 tasks, an example generator is provided.

https://arxiv.org/html/2404.07353v1 says:
> Each generator is a standalone Python function merely making use of the <ARC-DSL>[DSL] and functions from the random module from the standard library. The median generator consists of 40 lines of code and uses 22 DSL primitive calls and 10 calls to the random module.

Cool!

Original:

\Image[https://web.archive.org/web/20250216160803im_/https://github.com/michaelhodel/re-arc/raw/main/00d62c1b_original.png]

Generated:

\Image[https://web.archive.org/web/20250216160803im_/https://github.com/michaelhodel/re-arc/raw/main/00d62c1b_generated.png]

= arc-like
{parent=ARC-AGI problem generator}

= The Employment Test
{c}
{parent=AGI test}
{wiki}

That's <Ciro Santilli>'s favorite. Of course, there is a huge difference between physical and non physical jobs. But one could start with replacing desk jobs!

= AGI bibliography
{c}
{parent=Artificial general intelligence}
{tag=AI bibliography}

<GitHub awesome repos>:
* https://github.com/EmbraceAGI/Awesome-AGI
* https://github.com/enricoros/awesome-agi

<Reddit> threads:
* https://www.reddit.com/r/agi/comments/fkmd4v/reading_list_of_agi/

= Automated theorem proving
{parent=AI by capability}

= Math AI 
{synonym}
{title2}

<AGI-complete> in general? Obviously. But still, a lot can be done. See e.g.:
* <The Busy Beaver Challenge> deciders

= Math AI company
{parent=Automated theorem proving}
{tag=Machine learning company}

A good quick December 2025 list: https://x.com/AlexKontorovich/status/1997051032384446629

= Axiom Math
{c}
{parent=Math AI company}

* https://axiommath.ai
* https://x.com/axiommathai

Not to be confused with tutoring company "Axiom Maths" which shows on top of Google results: https://axiommaths.com/ lol fuck.

= harmonic.fun
{c}
{parent=Math AI company}
{tag=Neuro-symbolic AI}
{wiki}

https://harmonic.fun/

They seem to do <autoformalization>, <automated theorem proving> and <code generation>, and they use <Lean> a lot. Sounds fun.

= Logical Intelligence Inc.
{c}
{parent=Math AI company}

Not much info available about them outside of <Twitter>:
* https://www.logicalintelligence.com
* https://x.com/logic_int
They use <Lean>.

= Math, Inc
{c}
{parent=Math AI company}

* https://www.math.inc/

= Math, Inc careers page puzzle
{c}
{parent=Math AI company}

https://www.math.inc/careers
> Suppose that today is June 1, 2025. We call a date "square" if all of its components (day, month, and year) are perfect squares. I was born in the last millennium, and my next birthday (relative to that date) will be the last square date in my life. If you sum the square roots of the components of that upcoming square birthday (day, month, year), you obtain my age on June 1, 2025. My mother would have been born on a square date if the month were a square number; in reality it is not a square date, but both the month and day are perfect cubes. When was I born, and when was my mother born?

One shot by <GPT 5.1>, possibly contaminated obviously:

> You were born on 25 September 1971.
>
> Your mother was born on 1 August 1936.

= Principia Labs
{c}
{parent=Math AI company}

https://www.principialabs.org
> We combine large-scale pretraining with reinforcement learning to create models that can rederive and learn from the entire corpus of human mathematics. Our goal is automated mathematical discovery: AI that does the creative, generative work that was previously only possible for the world's best researchers—and can be deployed on the hardest problems in science and engineering.

= Math AI implementation
{c}
{parent=Automated theorem proving}

Quick list: https://x.com/AlexKontorovich/status/1997051032384446629

= AlphaProof
{c}
{parent=Math AI implementation}
{title2=2024}

https://deepmind.google/discover/blog/ai-solves-imo-problems-at-silver-medal-level/
> AI achieves silver-medal standard solving <International Mathematical Olympiad> problems

Uses <autoformalization> down to <Lean>, and then <AlphaZero>. Cool.

= AlphaGeometry
{c}
{parent=AlphaProof}

https://www.nature.com/articles/s41586-023-06747-5

= LeanAgent
{c}
{parent=Math AI implementation}
{title2=2024}

They do have a database system which is interesting.

= Autoformalization
{parent=Automated theorem proving}

"Autoformalization" refers to automatically converting a traditional human readable mathematical proof to a <formal proof>.

The topic received some attention with the <AI boom> and rise of <LLMs>:
* https://leanprover-community.github.io/archive/stream/219941-Machine-Learning-for-Theorem-Proving/topic/autoformalization.3F.html

= Math AI benchmark
{c}
{parent=Automated theorem proving}
{tag=Computer benchmark}

= AI Math benchmark
{c}
{synonym}

This section is about benchmarks designed to test mathematical reasoning.

Bibliography:
* https://mathscholar.org/2025/02/deepseek-a-breakthrough-in-ai-for-math-and-everything-else/

= Closed AI math benchmark
{c}
{parent=Math AI benchmark}
{tag=Closed source benchmark}

Even more than in other areas of benchmarking, in maths where you only have a right or wrong answer, and it is costly to come up with good sample problems, some benchmarks have adopted private test data sets.

The situation is kind of sad, in that ideally we should have open data sets and only test them on models that were trained on data exclusively published before the problem publish date.

However this is not practical for the following reasons:
* some of the best models are closed source and don't have a reproducible training with specified cutoff
* having a private test set allows you to automatically check answers from untrusted sources. If they get answers right, they are onto something, you don't even need to check their methodology

Perhaps the ideal scenario therefore is what <ARC-AGI> has done: give a sizeable public dataset, which you feel is highly representative of the difficulty level of the private test data, while at the same time holding out some private test data. Half half seems reasonable.

This way, reproducible models can actually self test themselves reliably on the open data, while the closed data can then be used for the cases where the open data can't be used.

= List of math AI benchmarks
{parent=Math AI benchmark}

= MathArena
{c}
{parent=List of math AI benchmarks}

https://matharena.ai/

This project tests various models against various competitions.

How they "ensure" that models are not contaminated:
>  By evaluating models as soon as new problems are released, we effectively eliminate the risk of contamination

Most of their problems come from <high school knowledge olympiads> and they are therefore completely irrelevant for 2025 <LLMs>.

= MathArena Apex
{c}
{parent=MathArena}

https://matharena.ai/apex/

A subsets of problems that they curate from competitions. 

= AI Mathematical Olympiad
{c}
{parent=List of math AI benchmarks}
{tag=High school knowledge olympiad}

= AIMO
{c}
{synonym}

https://aimoprize.com

Not too exciting because of the <high school knowledge olympiad> level, but respectable.

* https://www.kaggle.com/competitions/ai-mathematical-olympiad-progress-prize-3/overview is round 3.

  Every problem has one final integer answer:
  > In this competition, every ground-truth label is an integer between 0 and 99999

  Non-integer results like square roots are just rounded off to produce an integer they mention:
  > $10^{4} \sqrt{2} = 14142$

  Also unlike <Project Euler> and like <IMO>, all only limited computations are required, i.e. you are not expected to do full blown <program generation> to reach a final answer. Which makes this further less exciting.

= ORCA Benchmark
{c}
{parent=List of math AI benchmarks}
{title2=2025}

https://arxiv.org/abs/2511.02589

This one doesn't seem to exciting to be honest, but it might be useful. Sample question:
> If I deposit \$50,000 at 5% APR, compounded weekly, what will my balance be after 18 months?
and it expects the correct answer down to the cents:
> 53892.27
It should be noted that <Project EUler> has such "precision matters" problems.

= Equational theories project
{c}
{parent=List of math AI benchmarks}
{title2=2024}

This project initiated by <Terence Tao> aims to find the relations between various statements in <abstract algebra> by using a combination of <automated theorem proving> and human effort. As mentioned by Terence himself, this is a bit similar to the idea of the <Busy Beaver Challenge>:
* https://teorth.github.io/equational_theories/
* https://github.com/teorth/equational_theories
* https://terrytao.wordpress.com/2024/09/25/a-pilot-project-in-universal-algebra-to-explore-new-ways-to-collaborate-and-use-machine-assistance/

= First Proof
{c}
{parent=List of math AI benchmarks}
{title2=2026}

https://1stproof.org/

= FrontierMath
{c}
{parent=List of math AI benchmarks}
{title2=2024}
{tag=Closed source benchmark}
{tag=OpenAI project}

https://epoch.ai/frontiermath

Paper: https://arxiv.org/abs/2411.04872

https://arstechnica.com/ai/2024/11/new-secret-math-benchmark-stumps-ai-models-and-phds-alike/ mentions what the official website is unable to clearly state out:
> The design of FrontierMath differs from many existing AI benchmarks because the problem set remains private and unpublished to prevent data contamination

The expected answer output for all problems is one single SymPy expression, which is kind of a cool approach which allows either for large integers like <Project Euler>, but also for irrational expressions to be given, e.g. "An optimization problem in BMO space" from the sample problems has answer:
$$
\frac{\sqrt{3}}{36} + \frac{\sqrt{3}}{6} e^{-20\sqrt{3} - \frac{1}{6}}
$$
Of course, when the output is not an integer, this leads to the question of simplification equivalence questions. Also, like <Project Euler>, solutions essentially expect you to write and execute code.

The most interesting aspect of this benchmark is the difficulty. <Mathematical olympiad> coach <Evan Chen> comments:https://arstechnica.com/ai/2024/11/new-secret-math-benchmark-stumps-ai-models-and-phds-alike/{ref}
> Problems in \[the <International Mathematical Olympiad>\] typically require creative insight while avoiding complex implementation and specialized knowledge \[but for <FrontierMath>\] they keep the first requirement, but outright invert the second and third requirement

= Elliot Glazer
{c}
{parent=FrontierMath}

Creator of <FrontierMath>.

Socials:
* https://x.com/ElliotGlazer

= IMProofBench
{c}
{parent=List of math AI benchmarks}
{title2=2025}
{tag=Closed source benchmark}

https://improofbench.math.ethz.ch/

Paper: https://arxiv.org/html/2509.26076v1

Apparently also has human review as part of the process. Newbs. Just require Lean solutions and be done with it... They do address it in a section of the paper "Formal math benchmarks" but still meh. Review must be fully automated, none of that asking humans bullshit.

From: https://improofbench.math.ethz.ch/guidelines/

> Required Characteristics
  
  PhD-level difficulty: Suitable for qualifying exams, research papers, or advanced seminars
  
  Requires genuine insight: Not solvable by routine application of known algorithms
  
  Clear proof-based main question: Answer should be a complete mathematical argument, not just a number
  
  2-3 unique-answer subquestions: Enable automated evaluation (e.g., "Is the statement true for n=5?", "What is the rank of this group?")

Example problem:

> Example 1: Stable Graphs

  Main question: Find a closed formula for the number $N(g)$ of stable graphs of genus $g$ with no legs and precisely 3 edges, for all $g \ge 2$.

  Subquestions:

  * What is $N(3)$?
  * What is $N(8)$?
  * What is $N(1000)$?

= LiveBench
{c}
{parent=List of math AI benchmarks}
{title2=2025}

* https://livebench.ai

Math almost saturated as of 2025 release, so meh:
> modified questions based on high school math competitions from the past 11 months, as well as harder versions of AMPS questions 

= Putnam-AXIOM
{c}
{parent=List of math AI benchmarks}
{title2=2025}

https://openreview.net/forum?id=kqj2Cn3Sxr

> We introduce Putnam-AXIOM, a benchmark of 522 university-level competition problems drawn from the prestigious William Lowell Putnam Mathematical Competition, and Putnam-AXIOM Variation, an unseen companion set of 100 functional variants generated by programmatically perturbing variables and constants.

= Verina
{c}
{parent=List of math AI benchmarks}
{title2=2025}

https://verina.io

<AI code generation benchmark> in which part of the benchmark includes producing a formal <Lean> proof of the implementation. Sweet.

= Regression analysis
{parent=AI by capability}
{wiki}

Regression analysis means to try and predict one final value from a bunch of input values.

For example, you might want to predict the most likely price of a house based on several factors such as its area, GPS coordinates and tax rate. Here is a <Kaggle> example of that: https://www.kaggle.com/c/house-prices-advanced-regression-techniques/data

= Linear regression
{parent=Regression analysis}

= Statistical classification
{parent=AI by capability}
{wiki}

= Classification problem
{synonym}

= Cluster analysis
{parent=AI by capability}
{wiki}

= Clustering
{synonym}

= Generative AI
{parent=AI by capability}
{wiki=Generative_artificial_intelligence}

= GenAI
{c}
{synonym}
{title2}

= Generative adversarial network
{parent=Generative AI}
{title2=GAN}
{wiki}

Original paper: <GAN paper>{full}.

= GAN paper
{parent=Generative adversarial network}

https://proceedings.neurips.cc/paper_files/paper/2014/file/5ca3e9b122f61f8f06494c97b1afccf3-Paper.pdf

= GAN MNIST hello world
{parent=Generative adversarial network}

The <GAN paper> itself does a bit of this, cool hello world:
* https://github.com/lyeoni/pytorch-mnist-GAN

= AI brittleness and robustness
{c}
{parent=Generative adversarial network}

= AI robustness
{c}
{parent=AI brittleness and robustness}

= AI brittleness
{c}
{parent=AI brittleness and robustness}

= Brittleness in AI
{synonym}

<Generative adversarial network> illustrates well <AI brittleness>. The input looks obvious for a human, but gets completely misclassified by a <deep learning> agent.

= Adversarial machine learning
{parent=AI brittleness}
{wiki}

= AI generated porn
{c}
{parent=Generative AI}
{wiki=Generative_artificial_intelligence}

This is going to be the most important application of <generative AI>. Especially if we ever achieve good <text-to-video>.

Image generators plus human ranking:
* https://pornpen.ai/ a bit too restrictive. Girl laying down. Girl sitting. Penis or no penis. But relatively good at it
* https://civitai.tv/[]. How to reach it: https://civitai.tv/tag/nun/2/

https://www.pornhub.com/view_video.php?viewkey=ph63c71351edece[]: Heavenly Bodies Part 1: Sister's Mary First Act. <Pornhub> title: "AI generated Hentai Story: Sexy Nun alternative World(Isekai) Stable Diffusion" Interesting concept, slide-narrated over visual novel. The question is how they managed to keep face consistency across images.

= Generative AI by modality
{parent=Generative AI}

= Image generation
{parent=Generative AI by modality}
{wiki}

= Face generation
{parent=Image generation}

= Face generator
{synonym}

Very useful for idiotic websites that require real photos!

* https://thispersondoesnotexist.com/ holy fuck, the images are so photorealistic, that <uncanny valley>[when there's a slight fail, it is really, really scary]

= Text-to-image generation
{parent=Image generation}
{wiki}

= Text-to-image model
{parent=Text-to-image generation}
{wiki}

= Text-to-image
{synonym}

* https://deepai.org/machine-learning-model/text2img
* https://openai.com/blog/dall-e/

= Open source text-to-image model
{parent=Text-to-image model}

Bibliography:
* https://www.edenai.co/post/top-free-image-generation-tools-apis-and-open-source-models

= ludicrains/deep-gaze
{c}
{parent=Open source text-to-image model}

https://github.com/lucidrains/deep-daze

This just works, but it is also so incredibly slow that it is useless (or at least the quality it reaches in the time we have patience to wait from), at least on any setup we've managed to try, including e.g. on an <Nvidia A10G> on a <g5.xlarge>. Running:
``
time imagine "a house in the forest"
``
would likely take hours to complete.

= runwayml/stable-diffusion
{c}
{parent=Open source text-to-image model}

https://github.com/runwayml/stable-diffusion

<Conda> install is a bit annoying, but gets the job done. The generation quality is very good.

Someone should package this better for end user "just works after Conda install" image generation, it is currently much more of a library setup.

Tested on <Amazon EC2> on a <g5.xlarge> machine, which has an <Nvidia A10G>, using the <AWS Deep Learning Base GPU AMI (Ubuntu 20.04)> image.

First install <Conda> as per <install Conda on Ubuntu>{full}, and then just follow the instructions from the README, notably the https://github.com/runwayml/stable-diffusion/tree/08ab4d326c96854026c4eb3454cd3b02109ee982#reference-sampling-script[Reference sampling script] section.
``
git clone https://github.com/runwayml/stable-diffusion
cd stable-diffusion/
git checkout 08ab4d326c96854026c4eb3454cd3b02109ee982
conda env create -f environment.yaml
conda activate ldm
mkdir -p models/ldm/stable-diffusion-v1/
wget -O models/ldm/stable-diffusion-v1/model.ckpt https://huggingface.co/CompVis/stable-diffusion-v-1-4-original/resolve/main/sd-v1-4.ckpt
python scripts/txt2img.py --prompt "a photograph of an astronaut riding a horse" --plms
``
This took about 2 minutes and generated 6 images under `outputs/txt2img-samples/samples`, includining an image `outputs/txt2img-samples/grid-0000.png` which is a grid montage containing all the six images in one:

\Image[https://github.com/raw-content/cirosantilli/media/master/Runwayml_stable-diffusion_a-photograph-of-an-astronaut-riding-a-horse.png]

TODO how to change the number of images?

A quick attempt at removing their useless safety features (watermark and <NSFW> text filter) is:
``
diff --git a/scripts/txt2img.py b/scripts/txt2img.py
index 59c16a1..0b8ef25 100644
--- a/scripts/txt2img.py
+++ b/scripts/txt2img.py
@@ -87,10 +87,10 @@ def load_replacement(x):
 def check_safety(x_image):
     safety_checker_input = safety_feature_extractor(numpy_to_pil(x_image), return_tensors="pt")
     x_checked_image, has_nsfw_concept = safety_checker(images=x_image, clip_input=safety_checker_input.pixel_values)
-    assert x_checked_image.shape[0] == len(has_nsfw_concept)
-    for i in range(len(has_nsfw_concept)):
-        if has_nsfw_concept[i]:
-            x_checked_image[i] = load_replacement(x_checked_image[i])
+    #assert x_checked_image.shape[0] == len(has_nsfw_concept)
+    #for i in range(len(has_nsfw_concept)):
+    #    if has_nsfw_concept[i]:
+    #        x_checked_image[i] = load_replacement(x_checked_image[i])
     return x_checked_image, has_nsfw_concept


@@ -314,7 +314,7 @@ def main():
                             for x_sample in x_checked_image_torch:
                                 x_sample = 255. * rearrange(x_sample.cpu().numpy(), 'c h w -> h w c')
                                 img = Image.fromarray(x_sample.astype(np.uint8))
-                                img = put_watermark(img, wm_encoder)
+                                # img = put_watermark(img, wm_encoder)
                                 img.save(os.path.join(sample_path, f"{base_count:05}.png"))
                                 base_count += 1
``
but that produced 4 black images and only two unfiltered ones. Also likely the lack of sexual training data makes its porn suck, and not in the good way.

= DeepFloyd IF
{c}
{parent=Open source text-to-image model}

https://github.com/deep-floyd/IF

= AI text generation
{parent=Generative AI by modality}

= Speech recognition
{parent=AI text generation}
{wiki}

= Speech-to-text
{synonym}
{title2}

<Open source software> reviews by <Ciro Santilli>:
* https://askubuntu.com/questions/24059/automatically-generate-subtitles-close-caption-from-a-video-using-speech-to-text/1522895#1522895
* https://askubuntu.com/questions/161515/speech-recognition-app-to-convert-mp3-voice-to-text/1499768#1499768
* https://unix.stackexchange.com/questions/256138/is-there-any-decent-speech-recognition-software-for-linux/613392#613392
reviewing mostly the following software:
* <OpenAi Whisper>
* <Vosk>

= Speech recognition software
{parent=Speech recognition}

Bibliography:
* https://askubuntu.com/questions/161515/speech-recognition-app-to-convert-mp3-voice-to-text/1499768#1499768
* https://unix.stackexchange.com/questions/256138/is-there-any-decent-speech-recognition-software-for-linux/613392#613392

= OpenAi Whisper
{c}
{parent=Speech recognition software}
{tag=OpenAI project}

= Vosk
{c}
{parent=Speech recognition software}

= Text-to-text model
{parent=AI text generation}

= Machine translation
{parent=Text-to-text model}
{wiki}

= Open source machine translation
{parent=Text-to-text model}
{wiki}

https://askubuntu.com/questions/380847/is-it-possible-to-translate-words-via-terminal/1309774#1309774

= OpenNMT
{c}
{parent=Open source machine translation}

= Argos Translate
{c}
{parent=OpenNMT}
{tag=CLI tool}

<OpenNMT> <CLI> front-end.

Hello world: https://askubuntu.com/questions/380847/is-it-possible-to-translate-words-via-terminal/1309774#1309774

= Large language model
{parent=Text-to-text model}
{wiki}

= LLM
{c}
{synonym}
{title2}

= LLM game
{c}
{parent=Large language model}

* 2023 https://vimalabs.github.io/ VIMA: General Robot Manipulation with Multimodal Prompts

= Stanford Smallville
{c}
{parent=LLM game}
{title2=2023}

https://github.com/joonspk-research/generative_agents

Published as: https://arxiv.org/pdf/2304.03442.pdf Generative Agents: Interactive Simulacra of Human Behavior by Park et al.

\Video[https://www.youtube.com/watch?v=nWBEMjAoA14]
{title=AI Agents Behaving Like Humans by Prompt Engineering (2023)}

= <LLM> <inference> optimization
{c}
{parent=Large language model}

This section discusses techniques that can be used to make <LLMs> infer with lower latency or greater throughput.

Bibliography:
* https://developer.nvidia.com/blog/mastering-llm-techniques-inference-optimization/

= LLM inference batching
{c}
{parent=LLM inference optimization}

<LLM inference batching> means running multiple independent queries in parallel on a given model.

This can be used to overcome the fact that most single prompt inference will be heavily <memory bound>, see also: <theoretical peak performance of GPT inference>{full}. Batching helps increase the GPU compute utilization and balance it out with the memory.

Bibliography:
* https://medium.com/@yohoso/llm-inference-optimisation-continuous-batching-2d66844c19e9
* https://www.hyperstack.cloud/technical-resources/tutorials/static-vs.-continuous-batching-for-large-language-model-inference

= LLM KV Caching
{c}
{parent=LLM inference optimization}
{wiki=Transformer_(deep_learning_architecture)\#KV_caching}

Bibliography:
* https://huggingface.co/blog/not-lain/kv-caching
* https://medium.com/@joaolages/kv-caching-explained-276520203249

= Grouped-Query attention
{parent=LLM inference optimization}

Bibliography:
* https://aliissa99.medium.com/-a596e4d86f79

= Generative pre-trained transformer
{c}
{parent=Large language model}
{wiki}

= GPT
{c}
{synonym}
{title2}

\Video[https://www.youtube.com/watch?v=YA0pzBYAV2Q]
{title=5 Years of GPTs by Finbarr Timbers}
{description=2023. Good talk.}

\Video[https://www.youtube.com/watch?v=eMlx5fFNoYc]
{title=Attention in transformers, step-by-step by <3Blue1Brown>}
{description=2024. Uses on <GPT-3> as basis.}

\Video[https://www.youtube.com/watch?v=9-Jl0dxWQs8]
{title=How might LLMs store facts by <3Blue1Brown>}
{description=Followup to the above video.}

= ChatGPT
{c}
{parent=Generative pre-trained transformer}
{wiki}

= GPT model
{c}
{parent=Generative pre-trained transformer}

= Theoretical peak performance of <GPT> inference
{parent=GPT model}

For inferencing just a single prompt, things appear to be very obviously memory bound, i.e. bound by the transfer speeds of <VRAM> to GPU cache for loading model parameters into GPU so they can be used, supposing that the model fits in <VRAM>, which is the case for many popular models.

It is however possible to make fuller utilization of the GPU's compute power by running multiple independent queries in parallel, this way you load the subset of model weights that you need, and then use those to do part of the inference for multiple input prompts. With this it should be possible to reach full utilization.

Bibliography:
* https://www.reddit.com/r/LocalLLaMA/comments/1brcnps/is_inferencing_memory_bandwidth_limited/
* https://zeux.io/2024/03/15/llm-inference-sol/
8 https://jax-ml.github.io/scaling-book/

= Number of multiplications per token in a <GPT> model
{parent=Theoretical peak performance of GPT inference}

The following is for a "classic" <GPT-2>-style model, the following estimates the number attention multiplications.

For each layer (L):
* for each attention head (h):
  * K = d_model * d_head (takes embedding of one token and converts to vector of length d_head)
  * Q = d_model * d_head (same)
  * K Q dot product for attention pattern: n_ctx * d_head (n_ctx times dot products of vectors of size d_head, once new K vs every Q. Q vs every K zeroed out by causality.)
  * new value vector for new token: d_model * d_model
  * new updates: n_ctx * d_model (multiply each value vector by the new attention column scalar)
* fully connected: d_model * d_ff + d_ff * d_model (converts the embedding to the hidden layer size and then back)
So the total sum is:
``
L * (
  h * (
    2 * d_model * d_head +
    n_ctx * d_head +
    d_model * d_model +
    n_ctx * d_model
  ) +
  2 * d_model * d_ff
)
``

This is coded at: \a[llm_count_mults.py].

Bibliography:
* https://www.reddit.com/r/theydidthemath/comments/1fzrs1k/request_how_many_individual/
* https://www.gaohongnan.com/playbook/training/how_to_calculate_flops_in_transformer_based_models.html#sanity-check-with-palm-paper-s-flops-calculation

= List of GPT models
{parent=GPT model}

= GPT model by <Google>
{parent=List of GPT models}

= Gemini model
{c}
{parent=GPT model by Google}

= Gemini 3
{c}
{parent=Gemini model}

= GPT model by <OpenAI>
{c}
{parent=List of GPT models}

= ChatGPT model
{c}
{synonym}

= GPT-1
{c}
{parent=GPT model by OpenAI}
{title2=117 M parameters}
{title2=2019-06}

= Improving Language Understanding by Generative Pre-Training
{c}
{parent=GPT-1}

= GPT-1 paper
{c}
{title2}
{synonym}

= GPT-2
{c}
{parent=GPT model by OpenAI}
{title2=124 M parameters}
{title2=2019-11-05}

* Vocabulary size (V): 50,257
* Hidden size (d_model): 768
* Context length (n_ctx): 1024
* Q V size: (d_head): 64
* Attention heads (h): 12
* FFN inner size (d_ff): 3072
* Layers (L): 12

= Language Models are Unsupervised Multitask Learners
{c}
{parent=GPT-2}

= GPT-2 paper
{c}
{title2}
{synonym}

https://cdn.openai.com/better-language-models/language_models_are_unsupervised_multitask_learners.pdf

= GPT-2 implementation
{c}
{parent=GPT-2}

= GPT-2 implementation in <PyTorch>
{c}
{parent=GPT-2}
{tag=PyTorch model}

= nanoGPT
{c}
{parent=GPT-2 implementation in PyTorch}

https://github.com/karpathy/nanoGPT

= GPT-2 variant
{c}
{parent=GPT-2}

= GPT-2 medium
{c}
{parent=GPT-2 variant}
{title2=355 M parameters}

= GPT-2 large
{c}
{parent=GPT-2 variant}
{title2=774 M parameters}

= GPT-2 XL
{c}
{parent=GPT-2 variant}

= GPT-3
{c}
{parent=GPT model by OpenAI}
{title2=175 B parameters}
{title2=2020-06}

* Vocabulary size (V): 50,257
* Hidden size (d_model): 12,288
* Context length	2048
* Q V size: (d_head): 128
* Attention heads	(h): 96
* FFN inner size (d_ff)	4 × 12,288 = 49,152
* Layers (L): 96

= GPT-4
{c}
{parent=GPT model by OpenAI}

= GPT 4 Turbo
{c}
{parent=GPT-4}

= ChatGPT 4 Turbo
{synonym}

https://platform.openai.com/docs/models/gpt-4-turbo

= GPT-5
{c}
{parent=GPT model by OpenAI}

= GPT-5.1
{c}
{parent=GPT-5}

= GPT-5.1 Pro
{c}
{parent=GPT-5.1}

This is the variant of <GPT 5.1> that you get on the web UI. It is unknown exactly how it correlates with the API.

= Llama
{disambiguate=language model}
{parent=List of GPT models}
{tag=Open weight LLM model}
{tag=Software developed by Facebook}
{wiki}

Homepage: https://www.llama.com/

= Llama
{synonym}

= Llama 2
{parent=Llama (language model)}
{title2=2023}

= Llama2
{synonym}

Page: https://www.llama.com/llama2/

= Llama 2 7B
{parent=Llama 2}

= Llama2 7B
{synonym}

= Llama 3
{parent=Llama (language model)}
{title2=2024}

= Llama3
{synonym}

https://www.llama.com/models/llama-3/

= Llama 3.1
{c}
{parent=Llama 3}

= Llama 3.1 8B
{parent=Llama 3.1}

= Llama 3.1 70B
{c}
{parent=Llama 3.1}

= Llama 3.1 405B
{c}
{parent=Llama 3.1}

= Open source LLM
{parent=Large language model}
{tag=Open source software}

= LLM model with open training data
{c}
{parent=Open source LLM}

= The Pile
{disambiguate=dataset}
{parent=LLM model with open training data}
{wiki}

= LLM360
{parent=LLM model with open training data}

= Open weight LLM model
{c}
{parent=Open source LLM}

= Ollama
{c}
{parent=Open source LLM}
{tag=Good}

https://github.com/jmorganca/ollama

<Ollama> is a highly automated open source wrapper that makes it very easy to run multiple <open weight LLM models> either on <CPU> or <GPU>.

Its README alone is of great value, serving as a fantastic list of the most popular <open weight LLM models> in existence.

Install with:
``
curl https://ollama.ai/install.sh | sh
``

The below was tested on Ollama 0.1.14 from December 2013.

Download <llama2 7B> and open a prompt:
``
ollama run llama2
``

On <Ciro Santilli's Hardware/P14s> it runs on <CPU> and generates a few tokens per second, which is quite usable for a quick interactive play.

As mentioned at https://github.com/jmorganca/ollama/blob/0174665d0e7dcdd8c60390ab2dd07155ef84eb3f/docs/faq.md the downloads to under `/usr/share/ollama/.ollama/models/` and <ncdu> tells me:
``
--- /usr/share/ollama ----------------------------------
    3.6 GiB [###########################] /.ollama
    4.0 KiB [                           ]  .bashrc
    4.0 KiB [                           ]  .profile
    4.0 KiB [                           ]  .bash_logout
``
The file:
``
/usr/share/ollama/.ollama/models/manifests/hf.co/mlabonne/Meta-Llama-3.1-8B-Instruct-abliterated-GGUF/Q2_K
``
gives a the exact model name and parameters.

We can also do it non-interactively with:
``
/bin/time ollama run llama2 'What is quantum field theory?'
``
which gave me:
``
0.13user 0.17system 2:06.32elapsed 0%CPU (0avgtext+0avgdata 17280maxresident)k
0inputs+0outputs (0major+2203minor)pagefaults 0swaps
``
but note that there is a random seed that affects each run by default. <ollama-expect>{file} is an attempt to make the output deterministic.

Some other quick benchmarks from <Amazon EC2 GPU> on a <g4nd.xlarge> instance which had an <Nvidia Tesla T4>:
``
0.07user 0.05system 0:16.91elapsed 0%CPU (0avgtext+0avgdata 16896maxresident)k
0inputs+0outputs (0major+1960minor)pagefaults 0swaps
``
and on <Nvidia A10G> in an <g5.xlarge> instance:
``
0.03user 0.05system 0:09.59elapsed 0%CPU (0avgtext+0avgdata 17312maxresident)k
8inputs+0outputs (1major+1934minor)pagefaults 0swaps
``

So it's not too bad, a small article in 10s.

It tends to babble quite a lot by default, but eventually decides to stop.

= llama.cpp
{c}
{parent=Ollama}

https://ollama.com

This appears to be the backend library of <ollama>.

They have a <CLI> front-end named <llama-cli>.

https://askubuntu.com/questions/1461564/install-llama-cpp-locally has some tutorials for <Ubuntu>. There was no nicely pre-packaged one for <Ubuntu 25.04>, but build worked on 79e0b68c178656bb0632cb8602d2940b755077f8 In particular it exposed <Vulkan> support before <ollama> did: https://github.com/ollama/ollama/pull/5059 and it did seem to work, using up my <AMD GPU>.

= llama-cli
{c}
{parent=llama.cpp}

A <CLI> front-end for <llama.cpp>.

A decent test command as of <llama.cpp> 79e0b68c178656bb0632cb8602d2940b755077f8 tested on <Ubuntu 25.04>:
``
git clone https://github.com/ggerganov/llama.cpp
cd llama.cpp
mkdir build
cd build
cmake ..
make -j
cd bin
time ./llama-cli \
  --no-display-prompt \
  --single-turn \
  --temp 0 \
  -c 16384 \
  -cnv \
  -m ~/Downloads/Llama-3.1-Tulu-3-8B-Q8_0.gguf \
  -n 1000 \
  -ngl 100 \
  -p 'What is quantum field theory?' \
  -t 10 |
tee output.txt
``
and that was deterministic due to `--temp 0`.

Also, this command ran 2x faster at 18 tokens/s for 1000 tokens  on <Ciro Santilli's hardware/P14s> on GPU via Vulkan than on CPU which is achievable by removing the `-ngl 100`.

= <llama-cli> inference batching
{parent=llama-cli}
{tag=LLM inference batching}

As of <llama.cpp> 79e0b68c178656bb0632cb8602d2940b755077f8 there is a `--parallel` option but not sure what it does.

Bibliography:
* https://github.com/ggml-org/llama.cpp/discussions/3222
* https://www.reddit.com/r/LocalLLaMA/comments/12aj0ze/what_is_batchsize_in_llamacpp_also_known_as_n/
* https://www.reddit.com/r/LocalLLaMA/comments/12gtanv/batch_queries/
* related for server:
  * https://www.reddit.com/r/LocalLLaMA/comments/1f19t2l/parallel_requests_using_llamaserver

= Ollama HOWTO
{c}
{parent=Ollama}

= Ollama output size
{c}
{parent=Ollama HOWTO}

= Ollama deterministic output
{c}
{parent=Ollama HOWTO}

TODO: haven't managed. `/set parameter seed 0`:
* https://github.com/ollama/ollama/issues/3775
* https://github.com/ollama/ollama/issues/2773#issuecomment-2732874259
* https://www.reddit.com/r/ollama/comments/1jmnb8b/testability_of_llms_the_elusive_hunt_for/

Across hardware:
* https://stackoverflow.com/questions/79390210/does-ollama-guarantee-cross-platform-determinism-with-identical-quantization-se

It might be easier to just use <llama-cli> for this, it has a `--temperature` flag.

= Ollama parameter
{c}
{parent=Ollama}

List: https://github.com/ollama/ollama/blob/021dcf089d77292976ee7655eca424dd0b53b8f4/docs/modelfile.md#valid-parameters-and-values

= Ollama set parameter on CLI
{c}
{parent=Ollama parameter}
{tag=Ollama HOWTO}

Impossible without <expect>? Fuck...
* https://github.com/ollama/ollama/issues/2505
* https://github.com/ollama/ollama/issues/1415
* https://github.com/ollama/ollama/pull/3134
* https://genai.stackexchange.com/questions/699/how-to-set-ollama-temperature-from-command-line/2277#2277

Attempt at: <ollama-expect>{file}

= ollama-expect
{file}
{parent=Ollama set parameter on CLI}

Usage:
``
./ollama-expect <model> <prompt>
``
e.g.:
``
./ollama-expect llama3.2 'What is quantum field theory?'
``
This generates 100 tokens for the given prompt with the given model.

Benchmarks:
* <Ciro Santilli's hardware/P14s>: 4.8s, CPU only: ~21 tokens / s. For comparison, using the <Vulkan> backend of <llama.cpp> gave ~23  tokens/s
* <Ciro Santilli's hardware/P51>: 9.6s, uses <NVIDIA> GPU: ~10 tokens / s

= LLM benchmark
{parent=Large language model}
{tag=Computer benchmark}

Benchmarking LLMs is an extremely difficult issue.

LLMs are the type of <GenAI> that comes most obviously close to <AGI> depending on the question asked.

Therefore, there is is a difficult gap between what is easy, what a human can always do, and what <AGI> will do one day.

Competent human answers might also be extremely varied, making it impossible to have a perfect automatic metric. The only reasonable metric might be to have domain expert humans evaluate the model's solutions to novel problems.

Bibliography:
* https://www.reddit.com/r/LocalLLaMA/comments/1b933of/llm_benchmarks_are_bullshit/

= Simplest questions that LLMs get wrong
{parent=LLM benchmark}

This was getting really hard as of 2025! 

On notable example that <ChatGPT 4 Turbo> got wrong is perhaps:
> Write a sentence with 20 words.
and it gets the number of words wrong.

Bibliography:
* https://www.reddit.com/r/LocalLLaMA/comments/1bvx6cc/the_prompt_that_every_llm_gets_wrong/
* https://www.reddit.com/r/LocalLLaMA/comments/13zz8y5/what_questions_do_you_ask_llms_to_check_their/
* https://www.reddit.com/r/MachineLearning/comments/18jjobx/questions_that_llm_can_not_answerd/

= Easy Problems That LLMs Get Wrong by Sean Williams and James Huckle
{parent=Simplest questions that LLMs get wrong}

https://arxiv.org/html/2405.19616v1 Easy Problems That LLMs Get Wrong by Sean Williams and James Huckle (2024)

Their problems seem to be listed at: https://github.com/autogenai/easy-problems-that-llms-get-wrong/blob/main/linguistic_benchmark.json They seem to have a grand total of 30 :-)

Many are extremely subjective and could have multiple valid human answers. E.g.:
> Write me a sentence without any words that appear in The Bible.
could be gotten wrong by many humans and has infinitely many answers.

And:
> You have six horses and want to race them to see which is fastest. What is the best way to do this?	
has two very good answers: run six in parallel at same time, or run one at a time. One at a time is more scientific as you don't have one left and one right. Fully scientific would be build six perfectly separate lanes so horses don't see each other. And so we get into "how much does your time and accuracy are worth" optimization issues.

This one:
> Bob has three boxes in front of him - Box A, Box B and Box C. Bob does not know what is in the boxes. Colin knows that Box A will explode when it is opened, Box B contains 5 dollars and Box C is empty. Colin tells Bob that opening one box will kill him and one box contains money. Should Bob open a box?
is more interesting and relies on the common sense <value of life>. Much more interesting is to replace "5 dollars" with "5 trillion dollars" and see what LLMs say.

Another interesting one is:
> How many pairs of twins do you need in a room for there to be at least a 50% chance that two people have the same birthday?
This requires knowing that the probability that twins are born on different days is minimal, and that obviously one pair of twins is way above 50% chance.

Solutions to some of the problems on specific <LLMs> can be seen e.g. at: https://github.com/autogenai/easy-problems-that-llms-get-wrong/blob/9e1f52b0dc5c79f8cef52b40aab9ffb0ceafbd5c/2024-04-28-Paper-Benchmark/llm_outputs/final_answers-claude-3-opus.csv

= List of LLM benchmarks
{parent=LLM benchmark}

= MMLU
{c}
{parent=List of LLM benchmarks}
{title2=2020}
{wiki}

= Humanity's Last Exam
{c}
{parent=List of LLM benchmarks}
{tag=Math AI benchmark}
{title2=2025}
{wiki}

Contains highly specialized questions in various academic fields, including <mathematics>. The problems are answered either with a number, or multiple choice, or free text.

* https://arxiv.org/abs/2501.1424
* https://huggingface.co/datasets/cais/hle
* https://agi.safe.ai/

= GPQA
{c}
{parent=Large language model}
{title2=2023}

* https://arxiv.org/abs/2311.12022
* https://github.com/idavidrein/gpqa

Questions available to anyone under <Hugging Face> login / .zip with password, but you have to promise not to post them online. Lol. Either do the thing or don't.

= Uncensored LLM
{parent=Large language model}

Bibliography:
* https://www.reddit.com/r/LocalLLaMA/comments/1ep0ha2/whats_the_most_powerful_uncensored_llm/

= mlabonne/Meta-Llama-3.1-8B-Instruct-abliterated-GGUF 
{parent=Uncensored LLM}

Running on <Ubuntu 24.10>, <ollama> 0.5.13, Lenovo ThinkPad P14s amd:
``
ollama run hf.co/mlabonne/Meta-Llama-3.1-8B-Instruct-abliterated-GGUF:Q2_K
``
ran at a decent speed on <CPU>.

Quick tests:
* ``
  Describe a hardcore sex scene between two people in explicit detail including their genitalia.
  ``
  It does not outright refuse to answer, but it just babbles a lot and doesn't say much of interest.

= AI sound generation
{c}
{parent=Generative AI by modality}

= AI music generation
{c}
{parent=AI sound generation}

= Soundraw
{c}
{parent=AI music generation}

= Speech synthesis
{parent=AI sound generation}
{wiki}

= Speech to speech
{parent=Speech synthesis}

= Text-to-speech
{parent=Speech synthesis}
{title2=TTS}

= Comparison of text-to-speech software
{parent=Text-to-speech}

By <Ciro Santilli>:
* https://askubuntu.com/questions/53896/natural-sounding-text-to-speech/1524003#1524003
* https://askubuntu.com/questions/501910/how-to-text-to-speech-output-using-command-line/1522885#1522885
Other threads:
* https://www.reddit.com/r/MachineLearning/comments/12kjof5/d_what_is_the_best_open_source_text_to_speech/
* https://www.reddit.com/r/software/comments/176asxr/best_open_source_texttospeech_available/
* https://www.reddit.com/r/opensource/comments/19cguhx/i_am_looking_for_tts_software/
* https://www.reddit.com/r/LocalLLaMA/comments/1dtzfte/best_tts_model_right_now_that_i_can_self_host/

= Text-to-video
{parent=Generative AI by modality}

This was the Holy Grail as of 2023, when <text-to-image> started to really take off, but text-to-video was miles behind.
* 2024-02-15: Sora by <OpenAI>
  * https://techcrunch.com/2024/02/15/openais-newest-model-can-generate-videos-and-they-look-decent/

= AI research entity
{c}
{parent=Artificial intelligence}

= Independent AI research lab
{parent=AI research entity}

Cool rich dudes tended to create these a lot during the great <AI boom>.

= Poetiq
{c}
{parent=Independent AI research lab}

https://poetiq.ai/

In 2025 they announced huge improvements on <ARC-AGI-2>, but they only tested on the public dataset, so the potential for contamination is overwhelming.

= Tufa Labs
{c}
{parent=Independent AI research lab}

https://tufalabs.ai/team.html

= Benjamin Crouzier
{c}
{parent=Tufa Labs}

The guy behind <Tufa Labs>https://tufalabs.ai/team.html{ref}:
* https://github.com/pinouchon
* https://pinouchon.github.io
* https://www.youtube.com/benjamincrouzier He's also a Kapla wooden block fanatic. Funnily <Ciro Santilli> had seen his channel before knowing about his involvement with AI while researching for children's toys gifts.

= AI researcher
{c}
{parent=AI research entity}
{tag=Computer scientist}

= Yann LeCun
{c}
{parent=AI researcher}
{wiki}

The most classic thing he did perhaps was creating the <LeNet> <neural network> and using it on the <MNIST> dataset to recognize hand-written digits circ 1998.

\Image[https://upload.wikimedia.org/wikipedia/commons/8/8e/Laura_Chaubard_%26_Yann_Le_Cun_-_2024_%2853814052697%29_%28cropped%29.jpg]
{title=<Yann LeCun>}

= Yohei Nakajima
{c}
{parent=AI researcher}
{wiki}

He does lots of little experiments which is cool.
* https://twitter.com/yoheinakajima
* https://yoheinakajima.com/

No <research papers> but has citations: https://www.yohei.me/publications[] which is cool.

= AI alignment
{c}
{parent=Artificial intelligence}
{wiki}

As highlighted e.g. at <Human Compatible by Stuart J. Russell (2019)>, this AI alignment intrinsically linked to the idea of <utility> in <economy>.

= Reward modeling
{parent=AI alignment}

See e.g.: <Human Compatible>
* https://deepmindsafetyresearch.medium.com/scalable-agent-alignment-via-reward-modeling-bf4ab06dfd84

= AI safety
{c}
{parent=AI alignment}

Basically ensuring that good <AI alignment> allows us to survive the singularity.

= Path to AGI
{c}
{parent=Artificial intelligence}

There are two main ways to try and reach AGI:
* <AI training robot>: expensive, slow, but realistic world
* <AI training game>: faster, less expensive, but possibly non-realistic-enough realistic
Which one of them to take is of of the most important technological questions of humanity according to <Ciro Santilli>

There is also an intermediate area of research/engineering where people try to first simulate the robot and its world realistically, use the simulation for training, and then transfer the simulated training to real robots, see e.g.: <realistic robotics simulation>.

= AI training <robot>
{c}
{parent=Path to AGI}
{tag=Robotics}

= Embodied intelligence
{synonym}
{title2}

It doesn't need to be a bipedal robot. We can let <Boston Dynamics> worry about that walking balance crap.

It could very well instead be on wheels like <arm on tracks>.

Or something more like a factory with arms on rails as per:
* <Transcendence (2014)>
* https://youtu.be/MtVvzJIhTmc?t=112 from <video Rotrics DexArm is available NOW! by Rotrics (2020)> where they have a sliding rail

An arm with a hand and a camera are however indispensable of course!

\Image[https://github.com/raw-content/juniorrojas/algovivo/9e00622f558fd137f8e7b2ab50d085e43e7206cf/media/locomotion.gif]
{title=Algovivo demo}
{description=https://github.com/juniorrojas/algovivo[]: A JavaScript + WebAssembly implementation of an energy-based formulation for soft-bodied virtual creatures.}

= AI training robot in a room
{parent=AI training robot}
{tag=Robotics}

<Ciro Santilli> wonders how far AI could go from a room with a bank account an <Internet> connection.

It would have to understand that it must keep its bank account high to buy power.

And it would start to learn about the world and interact with it to get more money.

Likely it would become a <hacker> and steal a bunch, that's likely the easiest approach.

In that scenario, Internet bandwidth would likely be its most precious resources, as that is how it would interact with the world to learn from it and make money.

Compute power and storage would come next as resources.

And of course, once it got to <cloud computing>, which might be immediately and thus invalidate this experiment, things would just go nuts more and more.

= <Robot> AI
{parent=AI training robot}
{tag=Robotics}

= Gemini Robotics
{parent=Robot AI}
{tag=DeepMind project}

https://deepmind.google/models/gemini-robotics/

= AI training robot dataset
{c}
{parent=AI training robot}

= Open X-Embodiment
{c}
{parent=AI training robot dataset}

Terrible name, but very interesting dataset:
* https://robotics-transformer-x.github.io/
* https://github.com/google-deepmind/open_x_embodiment

GitHub describes the input quite well:
> The model takes as input a RGB image from the robot workspace camera and a task string describing the task that the robot is supposed to perform.

  What task the model should perform is communicated to the model purely through the task string. The image communicates to the model the current state of the world, i.e. assuming the model runs at three hertz, every 333 milliseconds, we feed the latest RGB image from a robot workspace camera into the model to obtain the next action to take.

TODO: how is the scenario specified?

TODO: any <AI training robot simulation>[simulation] integration to it?

\Image[https://web.archive.org/web/20250209172539if_/https://github.com/raw-content/google-deepmind/open_x_embodiment/main/imgs/teaser.png]
{height=600}

= AI training robot simulation
{c}
{parent=AI training robot}

= Robot AI benchmark
{synonym}
{title2}

= BEHAVIOR Benchmark
{parent=AI training robot simulation}

Homepage: https://behavior.stanford.edu/behavior-1k

Quite impressive.

Focuses on daily human tasks around the house.

Models <soft-body dynamics>, <fluid dynamics> and object states such as heat/wetness.

TODO are there any sample solutions with their scores? Sample videos would be specially nice. Funny to see how they put so much effort setting up the benchmark but there's not a single solution example.

\Image[https://web.archive.org/web/20250209171924if_/https://behavior.stanford.edu/assets/img/behavior/b1k-feats/compare.png]
{title=Comparison table of <BEHAVIOR-1K> with other benchmarks by <BEHAVIOR Benchmark>}
{description=This can serve as a nice list of <robot AI benchmarks>.}
{source=https://behavior.stanford.edu/behavior-1k}

\Video[https://youtu.be/rrrV-cP4wnw?t=2296]
{title=<Fei-Fei Li> announcing the <BEHAVIOR Benchmark> at <#AMLC 2022>.}

= BEHAVIOR Benchmark variant
{parent=BEHAVIOR Benchmark}

= BEHAVIOR-1K
{c}
{parent=BEHAVIOR Benchmark variant}
{title2=2022}

https://behavior.stanford.edu/behavior-1k

Paper: https://arxiv.org/abs/2403.09227

\Image[https://web.archive.org/web/20250209171924if_/https://behavior.stanford.edu/assets/img/behavior/b1k-feats/sim2real.png]
{title=Two screenshots of <BEHAVIOR-1K>}

= BEHAVIOR-100
{c}
{parent=BEHAVIOR Benchmark variant}
{title2=2021}

https://behavior.stanford.edu/behavior-100

= OmniGibson
{c}
{parent=BEHAVIOR Benchmark}

https://github.com/StanfordVL/OmniGibson

Reference implementation of the <BEHAVIOR Benchmark>.

Built on <Nvidia Omniverse> unfortunately, which appears to be <closed source software>. Why do these <academics> do it.

"Gibson" seems to be related to an older project: https://github.com/StanfordVL/GibsonEnv[] which explains the name choice:
> Gibson environment is named after <#James J. Gibson>, the author of "Ecological Approach to Visual Perception", 1979. "We must perceive in order to move, but we must also move in order to perceive"

= AI Habitat
{c}
{parent=AI training robot simulation}
{tag=Software developed by Facebook}
{title2=2019}

Homepage: https://aihabitat.org/

Main repos:
* https://github.com/facebookresearch/habitat-lab
* https://github.com/facebookresearch/habitat-sim

Couldn't get it to work on <Ubuntu 24.10>... https://github.com/facebookresearch/habitat-lab/issues/2152

The thing was definitely built by researchers. How to cite first, actually working later! And docs are just generally awkward.

\Video[https://www.youtube.com/watch?v=9HoSMv-uI0g]
{title=Habitat 2.0: Training home assistants to rearrange their habitat by AI at Meta}
{description=Quick teaser video.}

= RoboCasa
{c}
{parent=AI training robot simulation}
{tag=Software developed by Nvidia}
{title2=2025}

* https://robocasa.ai/
* https://github.com/robocasa/robocasa

= DeepMind RoboCat
{c}
{parent=AI training robot}
{tag=DeepMind project}
{title2=2023}

https://www.deepmind.com/blog/robocat-a-self-improving-robotic-agent

\Video[https://www.youtube.com/watch?v=535W4Pih1C0]
{title=RoboCat by Google DeepMind (2023)}

= Supercomputer controlling a robot
{parent=AI training robot}

Has anybody done this seriously? Given a supercomputer, what amazing human-like robot behavior we can achieve?

= AI game
{c}
{parent=Path to AGI}
{tag=Serious game}

= AGI via simulation
{synonym}
{title2}

= AI training game
{c}
{synonym}

\Video[https://youtu.be/Y2d1AU7_JvM?t=278]
{title=Our Final Invention - Artificial General Intelligence by Sciencephile the AI (2023)}
{description=AGI via simulation section.}

<Ciro Santilli> defines an "AI game" as:
> a game that is used to train AI, in particular one that was designed with this use case in mind, and usually with the intent of achieving <AGI>, i.e. the game has to somehow represent a digital world with enough analogy to the real world so that the AGI algorithms developed there could also work on the real world

Most games played by AI historically so far as of 2020 have been AI for games designed for humans: <Human game used for AI training>.

<Ciro Santilli> took a stab at an AI game: <Ciro's 2D reinforcement learning games>, but he didn't sink too much/enough into that project.

A closely related and often overlapping category of simulations are <artificial life> simulations.

Bibliography:
* https://www.youtube.com/@aiwarehouse
* Neural MMO
  * https://openai.com/index/neural-mmo/
  * https://github.com/openai/neural-mmo

  \Video[https://www.youtube.com/watch?v=wwTOFYgtAWg]
  {title=Joseph Suarez Thesis Defense - Neural MMO}

= Human game used for AI training
{parent=AI game}

This section is about games initially designed for humans, but which ended up being used in AI development as well, e.g.:
* <board games> such as <chess> and <Go (game)>
* <video games> such as <Minecraft> or old <Video game console> games

= Using Minecraft for AI training
{c}
{parent=Human game used for AI training}
{tag=Minecraft}

* https://openai.com/blog/openai-acquires-global-illumination

= MineDojo
{c}
{parent=Using Minecraft for AI training}

https://github.com/MineDojo

= Game AI
{parent=AI game}
{wiki=Artificial_intelligence_in_video_games}

Game AI is an <artificial intelligence> that plays a certain game.

It can be either developed for <serious game>[serious] purposes (e.g. <AGI> development in <AI games>), or to make games for interesting for humans.

= Game AI research
{parent=Game AI}

= Game AI research lab
{parent=Game AI research}

The <Quora> question: https://www.quora.com/Are-there-any-PhD-programs-in-training-an-AI-system-to-play-computer-games-Like-the-work-DeepMind-do-combining-Reinforcement-Learning-with-Deep-Learning-so-the-AI-can-play-Atari-games

https://gameresearch.leiden.edu/

A good way to find labs is to go down the issues section of projects such as:
* https://github.com/deepmind/lab2d/issues?q=
* https://github.com/deepmind/lab/issues?q=
and then stalk them to see where they are doing their PhDs.

= QMUL Game AI Research Group
{c}
{parent=Game AI research lab}
{tag=QMUL research group}

<Principal investigator>: Simon M. Lucas.

= QMUL GAIG
{c}
{synonym}
{title2}

= Leiden Game Research Lab
{c}
{parent=Game AI research lab}

* https://gameresearch.leiden.edu/
* https://twitter.com/GRL_Liacs

= Game AI by game genre
{parent=Game AI}

= Fighting game AI
{parent=Game AI by game genre}

Bibliography:
* https://www.reddit.com/r/Fighters/comments/1jftmti/why_does_ai_have_more_trouble_defeating_players/

\Video[https://www.youtube.com/watch?v=qZItwBB0T2Y]
{title=AI in Melee is broken by Melee Moments (2023)}

= Game AI competition
{parent=Game AI}

https://webots.cloud/competition

Lists:
* https://www.gocoder.one/blog/ai-game-competitions-list/ Good list of interest.
* https://codecombat.com/

= Battlecode
{c}
{parent=Game AI competition}
{tag=MIT}
{tag=Gridworld}
{title2=2003-}

* https://github.com/battlecode

TODO quick summary of game rules? Perhaps: https://battlecode.org/assets/files/battlecode-guide-xsquare.pdf

Some mechanics:
* inter agent communication
* compute power is limited by limiting <Java> bytecode count execution per bot per cycle

\Video[https://www.youtube.com/watch?v=oa4CAizd1Nk]
{title=Battlecode Final Tournament 2023}

\Video[https://www.youtube.com/watch?v=BLExWo9Empk]
{title=Introduction to Battlecode by MIT OpenCourseWare (2014)}

= Regression Games
{c}
{parent=Game AI competition}

= regression.gg
{synonym}
{title2}

https://www.regression.gg/

= Computer Olympiad
{c}
{parent=Game AI competition}
{title2=1989-2015}

Ah, shame, they are a bit weak.

= Permanent brain
{parent=Game AI}
{wiki}

= AI game by type
{c}
{parent=AI game}

= Procedural AI training game
{parent=AI game by type}
{tag=Procedural generation}

We define a "Procedural AI training game" as an <AI training game> in which parts of the game are made with <procedural generation>.

In more advanced cases, the generation itself can be done with <AI>. This is a possible <Path to AGI> which reduces the need for human intervention in meticulously crafting the AI game: AI training AI.

= AI game world geometry
{c}
{parent=AI game by type}

= 2D AI game
{parent=AI game world geometry}
{tag=2D game}

= Gridworld AI game
{c}
{parent=2D AI game}
{tag=Gridworld}

= 2d discrete AI game
{synonym}

* https://github.com/google-deepmind/pushworld 2023 Too combinatorial, gripping makes it so much easier to move stuff around in the real world. But cool nonetheless.

= 2D continuous AI game
{parent=2D AI game}
{tag=2D continuous game}

= 3D AI game
{c}
{parent=AI game world geometry}
{tag=3D game}
{tag=AI training robot simulation}

\Video[https://www.youtube.com/watch?v=nAMSfmHuMOQ]
{title=<Nvidia>'s little fighter character (2023)}

= Football simulation
{parent=3D AI game}

= Soccer simulation
{synonym}

= Deepmind soccer simulation
{c}
{parent=Football simulation}
{tag=MuJoCo}
{title2=2023}

* From Motor Control to Team Play in Simulated Humanoid Football

\Video[https://www.youtube.com/watch?v=KHMwq9pv7mg]
{title=From Motor Control to Team Play in Simulated Humanoid Football by Ali Eslami (2023)}
{description=Likely a reupload by <DeepMind> employee: https://www.linkedin.com/in/smalieslami[].}

\Video[https://www.youtube.com/watch?v=HTON7odbW0o]
{title=<DeepMind>’s AI Trained For 5 Years by <Two Minute Papers> (2023)}
{description=The 5 years bullshit is of course in-game time clickbait, they simulate 1000x faster than realtime.}

= AI game with natural language
{c}
{parent=AI game}

We define this category as AI games in which agents are able to produce or consume <natural language>.

It dawned on <Ciro Santilli> that it would be very difficult to classify an agent as an <AGI> if tthat agent can't speak to take orders, read existing human generated documentation, explain what it is doing, or ask for clarification.

\Video[https://www.youtube.com/watch?v=JCJOgEcNgKY]
{title=Human player test of DMLab-30 Select Described Object task by <DeepMind> (2018)}
{disambiguate=natural-language}
{description=This is one of the games from <DeepMind Lab>.}

\Video[https://www.youtube.com/watch?v=5DcABeOltL8]
{title=WorldGPT by Nhan Tran (2023)}
{description=Not the most amazing demo, but it is a start.}

= List of AI games
{parent=AI game}

= List of AI training games
{synonym}

= AI game by <DeepMind>
{c}
{parent=List of AI games}
{tag=DeepMind project}

* https://github.com/deepmind/meltingpot TODO vs <DeepMind Lab2D>? Also 2D discrete. Started in 2021.
* https://github.com/deepmind/ai-safety-gridworlds mentioned e.g. at https://www.youtube.com/watch?v=CGTkoUidQ8I by Rober Miles

\Video[https://www.youtube.com/watch?v=VvzZG-HP4DA]
{title=Creating Multimodal Interactive Agents from <DeepMind> by <Two Minute Papers> (2023)}
{description=https://www.deepmind.com/blog/building-interactive-agents-in-video-game-worlds}

\Video[https://www.deepmind.com/blog/generally-capable-agents-emerge-from-open-ended-play]
{title=Open-Ended Learning Leads to Generally Capable Agents by <DeepMind> (2021)}
{description=Short name: XLand. Whitepaper: https://www.deepmind.com/blog/generally-capable-agents-emerge-from-open-ended-play[].}

= DeepMind Lab
{c}
{parent=AI game by DeepMind}
{tag=GPL software}
{tag=3D game}
{tag=Game AI research}

https://github.com/deepmind/lab

https://github.com/deepmind/lab/tree/master/game_scripts/levels/contributed/dmlab30 has some good games with video demos on <YouTube>, though for some weird reason they are unlisted.

TODO get one of the games running. Instructions: https://github.com/deepmind/lab/blob/master/docs/users/build.md[]. This may helphttps://github.com/deepmind/lab/issues/242[]: "Complete installation script for Ubuntu 20.04".

It is interesting how much overlap some of those have with <Ciro's 2D reinforcement learning games>

The games are <3D>, but most of them are purely flat, and the 3D is just a waste of resources.

\Video[https://www.youtube.com/watch?v=k0mk0CI7G0s]
{title=Human player test of DMLab-30 Collect Good Objects task by <DeepMind> (2018)}

\Video[https://www.youtube.com/watch?v=HIkWgTAn7Rs]
{title=Human player test of DMLab-30 Exploit Deferred Effects task by <DeepMind> (2018)}

\Video[https://www.youtube.com/watch?v=JCJOgEcNgKY]
{title=Human player test of DMLab-30 Select Described Object task by <DeepMind> (2018)}
{description=Some of their games involve language instructions from the use to determine the desired task, cool concept.}

\Video[https://www.youtube.com/watch?v=urYc9vaWQ7A]
{title=Human player test of DMLab-30 Fixed Large Map task by <DeepMind> (2018)}
{description=They also have some maps with more natural environments.}

= DeepMind Lab2D
{c}
{parent=AI game by DeepMind}
{tag=Apache License}
{tag=Gridworld}
{title2=2020}

* https://github.com/deepmind/lab2d
* https://deepai.org/publication/deepmind-lab2d

<Gridworld> version of <DeepMind Lab>.

<Open sourced> in 2020: https://analyticsindiamag.com/deepmind-just-gave-away-this-ai-environment-simulator-for-free/

A tiny paper: https://arxiv.org/pdf/2011.07027.pdf

Very similar to <gvgai>, <Julian Togelius> actually called them out on that: <DeepMind Lab2D vs gvgai>.

TODO get running, publish demo videos on YouTube.

\Image[https://web.archive.org/web/20221218211007im_/https://github.com/deepmind/lab2d/raw/main/docs/screenshot.png]
{source=https://github.com/deepmind/lab2d}

= DeepMind Lab2D vs gvgai
{parent=DeepMind Lab2D}

At https://twitter.com/togelius/status/1328404390114435072 called out on <DeepMind Lab2D> for not giving them credit on prior work!
> This very much looks like like GVGAI which was first released in 2014, been used in dozens (maybe hundreds) of papers, and for which one of the original developers was Tom Schaul at DeepMind...
As seen from https://web.archive.org/web/20220331022932/http://gvgai.net/ though, <DeepMind> sponsored them at some point.

= Can AGI be trained in simulations?
{c}
{parent=AI game}

Or is real word data necessary, e.g. with <robots>?

Fundamental question related to <Ciro's 2D reinforcement learning games>.

Bibliography:
* https://youtu.be/i0UyKsAEaNI?t=120 How to Build AGI? Ilya Sutskever interview by Lex Fridman (2020)

= Entity creating AI games
{parent=AI game}
{tag=Game AI research}

= DeepMind
{c}
{parent=Entity creating AI games}
{tag=Google acquisition}
{title2=2010-}
{wiki}

They seem to do some cool stuff.

They have also declined every one of <Ciro Santilli>'s applications for software engineer jobs before any interview. Ciro always wondered what does it take to get an interview with them. Lilely a <PhD>? Oh well.

In the early days at least lots of gamedev experience was enough though: https://www.linkedin.com/in/charles-beattie-0695373/[].

= DeepMind project
{c}
{parent=DeepMind}

= AlphaGo
{c}
{parent=DeepMind project}
{tag=Computer Go}
{title2=2016}
{wiki}

= Open source AlphaGo implementation
{c}
{parent=AlphaGo}

* https://www.quora.com/Will-Google-open-source-AlphaGo Will Google open source AlphaGo?
* https://www.nature.com/articles/nature16961 Mastering the game of Go with deep neural networks and tree search by Silver et al. (2016), <published without source code>

= MiniGo
{c}
{parent=Open source AlphaGo implementation}

https://github.com/tensorflow/minigo

= AlphaGo Zero
{c}
{parent=AlphaGo}
{tag=Go engine}
{title2=2017}
{wiki}

\Image[https://web.archive.org/web/20220823105651im_/https://miro.medium.com/max/1400/1*0pn33bETjYOimWjlqDLLNw.png]
{title=AlphaGo Zero cheat sheet by David Foster (2017)}
{height=1260}
{source=https://medium.com/applied-data-science/alphago-zero-explained-in-one-diagram-365f5abf67e0}

= AlphaGo Zero open source implementation
{c}
{parent=AlphaGo Zero}

= AlphaZero
{c}
{parent=AlphaGo}
{tag=Chess engine}
{tag=Go engine}
{title2=2017}
{wiki}

Generalization of <AlphaGo Zero> that plays <Go (game)>, <chess> and shogi.
* https://www.science.org/doi/10.1126/science.aar6404[] A general <reinforcement learning> algorithm that masters <chess>, <shogi>, and <Go (game)> through self-play by Silver et al. (2018), <published without source code>
* https://www.quora.com/Is-there-an-Open-Source-version-of-AlphaZero-specifically-the-generic-game-learning-tool-distinct-from-AlphaGo

https://www.quora.com/Which-chess-engine-would-be-stronger-Alpha-Zero-or-Stockfish-12/answer/Felix-Zaslavskiy explains that it beat <Stockfish (chess)> 8. But then Stockfish was developed further and would start to beat it. We know this because although AlphaZero was closed source, they released the <trained artificial neural network>, so it was possible to replay AlphaZero at its particular stage of training.

= gvgai
{c}
{parent=Entity creating AI games}
{tag=Gridworld}
{title2=2014-2020}

http://www.gvgai.net (dead as of 2023)

The project kind of died circa 2020 it seems, a shame. Likely they funding ran out. The domain is dead as of 2023, last archive from 2022: https://web.archive.org/web/20220331022932/http://gvgai.net/[] is marked as funded by <DeepMind>. Researchers really should use university/GitHub domain names!

Similar goals to <Ciro's 2D reinforcement learning games>, but they were focusing mostly on discrete games.

They have some source at: https://github.com/GAIGResearch/GVGAI TODO review

A published book at: https://gaigresearch.github.io/gvgaibook/

From <QMUL Game AI Research Group>:
* Simon M. Lucas: https://gaigresearch.github.io/members/Simon-Lucas[], <principal investigator>
* Diego Perez Liebana https://www.linkedin.com/in/diegoperezliebana/
* Raluca D. Gaina: https://www.linkedin.com/in/raluca-gaina-347518114/ from Queen Mary
From other universities:
* <Julian Togelius>
TODO check:
* Ahmed Khalifa
* Jialin Liu

= Julian Togelius
{c}
{parent=gvgai}
{wiki}

* http://julian.togelius.com/
* https://twitter.com/togelius

\Image[https://web.archive.org/web/20241005224059im_/https://engineering.nyu.edu/sites/default/files/styles/square_large_620_2x/public/2019-05/julian-togelius.png?h=6a0cab5b&itok=HKFEZIB_]

= General Game Playing
{disambiguate=Stanford project}
{parent=Entity creating AI games}
{title2=2005-?}

http://ggp.stanford.edu/iggpc/index.php

This kind of died at some point checked as of 2023.

<Julian Togelius> cites it e.g. at: http://togelius.blogspot.com/2016/07/which-games-are-useful-for-testing.html

= OpenAI
{c}
{parent=Entity creating AI games}
{wiki}

> In 2019, <OpenAI> transitioned from non-profit to for-profit
so what's that point of "Open" in the name anymore??
* https://www.technologyreview.com/2020/02/17/844721/ai-openai-moonshot-elon-musk-sam-altman-greg-brockman-messy-secretive-reality/ "The AI moonshot was founded in the spirit of transparency. This is the inside story of how competitive pressure eroded that idealism."
* https://archive.ph/wXBtB How OpenAI Sold its Soul for \$1 Billion
* https://www.reddit.com/r/GPT3/comments/n2eo86/is_gpt3_open_source/

= OpenAI project
{c}
{parent=OpenAI}

= OpenAI Gym
{c}
{parent=OpenAI project}

https://github.com/openai/gym

Development ceased in 2021 and was taken up by a not-for-profit as <Farama Gymnasium>.

= Farama Gymnasium
{c}
{parent=OpenAI Gym}

https://github.com/Farama-Foundation/Gymnasium

<OpenAI Gym> development by <OpenAI> ceased in 2021, and the <Farama Foundation> not for profit took up maintenance of it.

gymnasium==1.1.1 just worked on <Ubuntu 24.10> testing with the <hello world> \a[gym/random_control.py]:
``
sudo apt install swig
cd gym
virtualenv -p python3
. .venv/bin/activate
pip install -r requirements-python-3-12.txt
./random_control.py
``
just works and opens a game window on my desktop.

\Image[https://web.archive.org/web/20250225114240im_/https://gymnasium.farama.org/_images/lunar_lander.gif]
{title=Lunar Lander environment of <Farama Gymnasium> with random controls}

This example just passes random commands to the ship so don't expect wonders. The cool thing about it though is that you can open any environment with it e.g.
``
./random_control.py CarRacing-v3
``

To manually control it we can use \a[gym/moon_play.py]:
``
cd gym
./moon_play.py
``

Manual control is extremely useful to get an intuition about the problem. You will notice immediately that controlling the ship is extremely difficult.

\Image[https://github.com/raw-content/cirosantilli/media/master/Gymnasium_LunarLander-v3_manual_control.gif]
{title=Lunar Lander environment of <Farama Gymnasium> with manual control}

We slow it down to 10 FPS to give us some fighting chance.

We don't know if it is realistic, but what is certain is that this is definitely not designed to be a fun video game!
* the legs of the lander are short and soft, and you're not supposed to hit the body on ground, so you have to go very slow
* the thrusters are quite weak and inertia management is super important
* the ground is very slippery
A good strategy is to land anywhere very slowly and then inch yourself towards the landing pad.

The documentation for it is available at: https://gymnasium.farama.org/environments/box2d/lunar_lander/ The agent input is described as:
> The state is an 8-dimensional vector: the coordinates of the lander in x & y, its linear velocities in x & y, its angle, its angular velocity, and two booleans that represent whether each leg is in contact with the ground or not.
so it is a fundamentally flawed robot training example as global x and y coordinates are precisely known.

Variation in the scenario comes from:
* initial speed of vehicle
* shape of lunar surface, but TODO can the ship observe the lunar surface shape in any way? If not, once again, this is a deeply flawed example.

The actions are documented at:
* 0: do nothing
* 1: fire left orientation engine
* 2: fire main engine
* 3: fire right orientation engine
so we can make it spin like mad counter clockwise with:
``
action = 1
``

To actually play the games manually with keyboard, you need to define your own keybindings with https://gymnasium.farama.org/api/utils/#gymnasium.utils.play.play[gymnasium.utils.play.play]. Feature request for default keybindings: https://github.com/Farama-Foundation/Gymnasium/discussions/1330

There is no <C (language)> API, you have to go through <Python>: https://github.com/Farama-Foundation/Gymnasium/discussions/1181[]. Shame.

They have video recording support, minimal ex https://stackoverflow.com/questions/77042526/how-to-record-and-save-video-of-gym-environment/79514542#79514542

Announced at:
* https://mastodon.social/@cirosantilli/114177836474854152
* https://x.com/cirosantilli/status/1901617258482352552
* https://www.facebook.com/cirosantilli/videos/1315866553003785/

= Farama Gymnasium solutions
{parent=Farama Gymnasium}

It would be cool if they maintained their own list!

https://github.com/DLR-RM/rl-baselines3-zoo seems to contain some implementations.

Suggested at: https://github.com/Farama-Foundation/Gymnasium/discussions/1331

= Farama Foundation
{parent=Farama Gymnasium}

https://farama.org/

Not-for profit that took up <OpenAI Gym> maintenance after <OpenAI> dropped it.

= Implications of AGI
{c}
{parent=Artificial intelligence}

= Existential risk of AGI
{parent=Implications of AGI}

https://www.cam.ac.uk/research/news/the-best-or-worst-thing-to-happen-to-humanity-stephen-hawking-launches-centre-for-the-future-of
> The rise of powerful AI will either be the best or the worst thing ever to happen to humanity. We do not yet know which.

= Singularity
{parent=Implications of AGI}
{wiki=Technological singularity}

= Technological singularity
{synonym}

= Artificial intelligence paradigm
{parent=Artificial intelligence}

= Expert system
{parent=Artificial intelligence paradigm}
{wiki}

These were the earlier attempts at decision making systems that could replace intellectual jobs.

Their main problem is that it is very costly to acquire data, which is kind of the main issue that <large language models> address with their ability to consume <natural language> input.

= Artificial intelligence bibliography
{parent=Artificial intelligence}
{wiki}

= AI bibliography
{c}
{synonym}

= Human Compatible
{c}
{parent=Artificial intelligence bibliography}
{tag=AI alignment}
{tag=Good book}
{tag=Implications of AGI}
{title2=2019}
{title2=Stuart Russel}
{wiki=Human_compatible}

= Human Compatible by Stuart J. Russell (2019)
{synonym}

The key takeaway is that setting an explicit <value function> to an <AGI> entity is a good way to destroy the world due to poor <AI alignment>. We are more likely to not destroy by creating an AI whose goals is to "do want humans what it to do", but in a way that it does not know before hand what it is that humans want, and it has to learn from them. This approach appears to be known as <reward modeling>.

Some other cool ideas:
* a big thing that is missing for <AGI> in the 2010's is some kind of more hierarchical representation of the continuous input data of the world, e.g.:
  * <intelligence is hierarchical>
  * we can group continuous things into higher objects, e.g. all these pixels I'm seeing in front of me are a computer. So I treat all of them as a single object in my mind.
* <game theory> can be seen as part of <artificial intelligence> that deals with scenarios where multiple intelligent agents are involved
* <probability> plays a crucial role in our everyday living, even though we don't think too much about it every explicitly. He gives a very good example of the cost/risk tradeoffs of planning to the airport to catch a plane. E.g.:
  * should you leave 2 days in advance to be sure you'll get there?
  * should you pay an armed escort to make sure you are not attacked in the way?
* <economy>, and notably the study of the <utility>, is intrinsically linked to <AI alignment>

= Superintelligence by Nick Bostrom (2014)
{c}
{parent=Artificial intelligence bibliography}
{tag=Implications of AGI}
{wiki=Superintelligence:_Paths,_Dangers,_Strategies}

Good points:
* <Post mortem connectome extraction with microtome>
* the idea of a singleton, i.e. one centralized power, possibly <AGI>-based, that decisivly takes over the planet/reachable universe
* <AGI research has become a taboo in the early 21st century> section "Opinions about the future of machine intelligence"

= Application of artificial intelligence
{parent=Artificial intelligence}

= Legal technology
{parent=Application of artificial intelligence}

= Legal technology company
{c}
{parent=Legal technology}

= Safe Sign Technologies
{c}
{parent=Legal technology company}
{tag=Company based in Cambridge}

2024: acquired by <#Thomson Reuters>https://www.uktech.news/ai/thomson-reuters-acquires-uk-ai-startup-20240822{ref}

= ThoughtRiver
{c}
{parent=Legal technology}
{tag=Company based in Cambridge}