The Memo - 31/Jan/2024

Neuralink Telepathy, gold datasets, AjaxGPT on iOS, and much more!

Jan 31, 2024

∙ Paid

To:      US Govt, major govts, Microsoft, Apple, NVIDIA, Alphabet, Amazon, Meta, Tesla, Citi, Tencent, IBM, & 10,000+ more recipients…
From:    Dr Alan D. Thompson <LifeArchitect.ai>
Sent:    31/Jan/2024
Subject: The Memo - AI that matters, as it happens, in plain English
AGI:     64%

FOR IMMEDIATE RELEASE: 31/Jan/2024

Welcome back to The Memo.

You’re joining full subscribers from NVIDIA, IBM, Amazon, Microsoft, Apple, Intel, Twitter, Netflix, Google, and more…

There’s a 100% AI-generated soundtrack to this edition. The genre is close to synth-pop, the prompter used GPT-4 for the lyrics and Suno AI to generate the song from scratch (30/Jan/2024).

FatesWaltz - Ghosts in the machine:

1×

0:00

-4:57

The next roundtable will be 17/Feb/2024 (US), details at the end of this edition.

The BIG Stuff

First human Neuralink implant (28/Jan/2024)

Brain-machine interfaces + benefits (Sep/2021): https://lifearchitect.ai/bmi/

Elon Musk: ‘The first human received an implant from Neuralink yesterday [28/Jan/2024 US time] and is recovering well. Initial results show promising neuron spike detection.’ (30/Jan/2024)

He continued: ‘The first Neuralink product is called Telepathy… Enables control of your phone or computer, and through them almost any device, just by thinking. Initial users will be those who have lost the use of their limbs. Imagine if Stephen Hawking could communicate faster than a speed typist or auctioneer. That is the goal.’ (30/Jan/2024)

Read not very much more via The Verge.

The official Neuralink study brochure (PDF) notes that ‘The study will take approximately 6 years to complete… The long-term follow-up begins immediately after completion of the Primary Study and takes place over 5 years…’

While this would possibly take us out to 2035, the shortening of timescales may mean this technology is available to the public far more quickly (recall the recent condensing of regulation and timescales in the biomedical industry, ‘Operation Warp Speed’ wiki).

Watch a short official video on Neuralink (Nov/2023): https://youtu.be/z7o39CzHgug

Read my Sep/2021 page on brain-machine interfaces, including visualization of benefits and video walk-throughs: https://lifearchitect.ai/bmi/

Exclusive: GPT-5 and gold datasets (Jan/2024)

When raising a child prodigy, should we provide more learning and experiences or higher-quality learning and experiences?

When training frontier models like GPT-5, should we use more data or higher-quality data?

In Jun/2021, I published a paper called ‘Integrated AI: Dataset quality vs quantity via bonum (GPT-4 and beyond)’. It explored high-quality data aligned with ‘the ultimate good’ (in Latin, this is ‘summum bonum’).

OpenAI’s CEO recently spoke at a number of big venues including the 54th annual meeting of the World Economic Forum (WEF) at Davos-Klosters, Switzerland from 15th to 19th January 2024. He was recorded as making a very interesting comment:

As models become smarter and better at reasoning, we need less training data. For example, no one needs to read 2000 biology textbooks; you only need a small portion of extremely high-quality data and to deeply think and chew over it. The models will work harder on thinking through a small portion of known high-quality data. (Reddit, not verbatim, 22/Jan/2024)

One researcher (22/Jan/2024) similarly notes:

…potentially 'infinity efficient' because they may be one-time costs to create. Depending on the details, you may simply create them once and then never again. For example, in ‘AlphaGeometry’, it seems likely that for most problems there’s going to be one and only one best & shortest proof, and that any search process would converge upon it quickly, and now you can just train all future geometry models on that ideal proof. Similarly, in chess or Go I expect that in the overwhelming majority of positions (even excluding the opening book & endgame databases), the best move is known and the engines aren't going to change the choice no matter how long you run them. ‘Gold datasets’ may be a good moat.

For text training, we’ve now hit massive datasets like the 125TB (30 trillion token) RedPajama-Data-v2, and I continue to track the other highlights on the Datasets Table.

Nearly three years after my data quality paper, are we finally on the way to higher quality (and perhaps temporarily smaller) datasets rather than ‘more is better’?

Explore further in my Mar/2022 comprehensive analysis of datasets, ‘What’s in my AI?’.

Exclusive: Half of all LLM innovations in January were from China (31/Jan/2024)

This month, the US was fairly quiet. In fact, only a few US-based AI labs announced new models (generally, I don’t count Llama finetunes, nor OpenAI’s minor model updates). In America, Adept announced their closed-source commercial model Fuyu-Heavy (120B), and Meta showed off their largest CodeLlama-70B model. Several of the US model innovations—like DocLLM and MambaByte—were led by researchers from China. And many of the Chinese models revealed major innovations not yet explored by US-based labs.

JPMorgan DocLLM (7B) - USA
SUTD/Independent TinyLlama (1.1B) - Singapore
Tencent LLaMA Pro (8.3B) - China
DeepSeek-AI DeepSeek (67B) - China
DeepSeek-AI DeepSeekMoE (16B) - China
Zhipu AI (Tsinghua) GLM-4 (200B) - China
Adept Fuyu-Heavy (120B) - USA
Tencent FuseLLM (7B) - China
DeepSeek-AI DeepSeek-Coder (33B) - China
Cornell MambaByte (972M) - USA
LMU MaLA-500 (10B) - Germany
RWKV RWKV-v5 Eagle (7.52B) - International
Meta AI CodeLlama-70B - USA
iFlyTek Xinghuo 3.5 (Spark) - China
iFlytekSpark-13B - China

GPT-4 cooked up a handy viz for me (30/Jan/2024) but its maths were off. So here’s the real version generated by hand in Google Charts (oh, the humanity!):

I’m not sure what China’s prolific output means, but I’m concerned that most of these models do not seem to show up in a GPT-centric world, despite many of them significantly outperforming ChatGPT. Remember, the popular gpt-3.5-turbo model is based on the GPT-3 model from 2020, available via Emerson the same year, and showcased throughout 2021 with the Leta AI experiments (watch playlist). While the internet worships this four-year-old technology (maybe the equivalent of 40 years old with the AI time multiplier), large language models are screaming along in every corner of the globe…

The Models Table: https://lifearchitect.ai/models-table/

The Interesting Stuff

China approves over 40 AI models for public use in past six months (29/Jan/2024)

China has given regulatory approval to more than 40 artificial intelligence models for public service use in the past six months, signaling a push in the country's AI sector.

The Memo by LifeArchitect.ai

The Memo - 31/Jan/2024

Neuralink Telepathy, gold datasets, AjaxGPT on iOS, and much more!

The BIG Stuff

The Interesting Stuff

This post is for paid subscribers