The Memo - 31/Jan/2024

Neuralink Telepathy, gold datasets, AjaxGPT on iOS, and much more!

Jan 31, 2024

To:      US Govt, major govts, Microsoft, Apple, NVIDIA, Alphabet, Amazon, Meta, Tesla, Citi, Tencent, IBM, & 10,000+ more recipients…
From:    Dr Alan D. Thompson <LifeArchitect.ai>
Sent:    31/Jan/2024
Subject: The Memo - AI that matters, as it happens, in plain English
AGI:     64%

FOR IMMEDIATE RELEASE: 31/Jan/2024

Welcome back to The Memo.

You’re joining full subscribers from NVIDIA, IBM, Amazon, Microsoft, Apple, Intel, Twitter, Netflix, Google, and more…

There’s a 100% AI-generated soundtrack to this edition. The genre is close to synth-pop, the prompter used GPT-4 for the lyrics and Suno AI to generate the song from scratch (30/Jan/2024).

FatesWaltz - Ghosts in the machine:

1×

0:00

-4:57

The next roundtable will be 17/Feb/2024 (US), details at the end of this edition.

The BIG Stuff

First human Neuralink implant (28/Jan/2024)

Brain-machine interfaces + benefits (Sep/2021): https://lifearchitect.ai/bmi/

Elon Musk: ‘The first human received an implant from Neuralink yesterday [28/Jan/2024 US time] and is recovering well. Initial results show promising neuron spike detection.’ (30/Jan/2024)

He continued: ‘The first Neuralink product is called Telepathy… Enables control of your phone or computer, and through them almost any device, just by thinking. Initial users will be those who have lost the use of their limbs. Imagine if Stephen Hawking could communicate faster than a speed typist or auctioneer. That is the goal.’ (30/Jan/2024)

Read not very much more via The Verge.

The official Neuralink study brochure (PDF) notes that ‘The study will take approximately 6 years to complete… The long-term follow-up begins immediately after completion of the Primary Study and takes place over 5 years…’

While this would possibly take us out to 2035, the shortening of timescales may mean this technology is available to the public far more quickly (recall the recent condensing of regulation and timescales in the biomedical industry, ‘Operation Warp Speed’ wiki).

Watch a short official video on Neuralink (Nov/2023): https://youtu.be/z7o39CzHgug

Read my Sep/2021 page on brain-machine interfaces, including visualization of benefits and video walk-throughs: https://lifearchitect.ai/bmi/

Exclusive: GPT-5 and gold datasets (Jan/2024)

When raising a child prodigy, should we provide more learning and experiences or higher-quality learning and experiences?

When training frontier models like GPT-5, should we use more data or higher-quality data?

In Jun/2021, I published a paper called ‘Integrated AI: Dataset quality vs quantity via bonum (GPT-4 and beyond)’. It explored high-quality data aligned with ‘the ultimate good’ (in Latin, this is ‘summum bonum’).

OpenAI’s CEO recently spoke at a number of big venues including the 54th annual meeting of the World Economic Forum (WEF) at Davos-Klosters, Switzerland from 15th to 19th January 2024. He was recorded as making a very interesting comment:

As models become smarter and better at reasoning, we need less training data. For example, no one needs to read 2000 biology textbooks; you only need a small portion of extremely high-quality data and to deeply think and chew over it. The models will work harder on thinking through a small portion of known high-quality data. (Reddit, not verbatim, 22/Jan/2024)

One researcher (22/Jan/2024) similarly notes:

…potentially 'infinity efficient' because they may be one-time costs to create. Depending on the details, you may simply create them once and then never again. For example, in ‘AlphaGeometry’, it seems likely that for most problems there’s going to be one and only one best & shortest proof, and that any search process would converge upon it quickly, and now you can just train all future geometry models on that ideal proof. Similarly, in chess or Go I expect that in the overwhelming majority of positions (even excluding the opening book & endgame databases), the best move is known and the engines aren't going to change the choice no matter how long you run them. ‘Gold datasets’ may be a good moat.

For text training, we’ve now hit massive datasets like the 125TB (30 trillion token) RedPajama-Data-v2, and I continue to track the other highlights on the Datasets Table.

Nearly three years after my data quality paper, are we finally on the way to higher quality (and perhaps temporarily smaller) datasets rather than ‘more is better’?

Explore further in my Mar/2022 comprehensive analysis of datasets, ‘What’s in my AI?’.

Exclusive: Half of all LLM innovations in January were from China (31/Jan/2024)

This month, the US was fairly quiet. In fact, only a few US-based AI labs announced new models (generally, I don’t count Llama finetunes, nor OpenAI’s minor model updates). In America, Adept announced their closed-source commercial model Fuyu-Heavy (120B), and Meta showed off their largest CodeLlama-70B model. Several of the US model innovations—like DocLLM and MambaByte—were led by researchers from China. And many of the Chinese models revealed major innovations not yet explored by US-based labs.

JPMorgan DocLLM (7B) - USA
SUTD/Independent TinyLlama (1.1B) - Singapore
Tencent LLaMA Pro (8.3B) - China
DeepSeek-AI DeepSeek (67B) - China
DeepSeek-AI DeepSeekMoE (16B) - China
Zhipu AI (Tsinghua) GLM-4 (200B) - China
Adept Fuyu-Heavy (120B) - USA
Tencent FuseLLM (7B) - China
DeepSeek-AI DeepSeek-Coder (33B) - China
Cornell MambaByte (972M) - USA
LMU MaLA-500 (10B) - Germany
RWKV RWKV-v5 Eagle (7.52B) - International
Meta AI CodeLlama-70B - USA
iFlyTek Xinghuo 3.5 (Spark) - China
iFlytekSpark-13B - China

GPT-4 cooked up a handy viz for me (30/Jan/2024) but its maths were off. So here’s the real version generated by hand in Google Charts (oh, the humanity!):

I’m not sure what China’s prolific output means, but I’m concerned that most of these models do not seem to show up in a GPT-centric world, despite many of them significantly outperforming ChatGPT. Remember, the popular gpt-3.5-turbo model is based on the GPT-3 model from 2020, available via Emerson the same year, and showcased throughout 2021 with the Leta AI experiments (watch playlist). While the internet worships this four-year-old technology (maybe the equivalent of 40 years old with the AI time multiplier), large language models are screaming along in every corner of the globe…

The Models Table: https://lifearchitect.ai/models-table/

The Interesting Stuff

China approves over 40 AI models for public use in past six months (29/Jan/2024)

China has given regulatory approval to more than 40 artificial intelligence models for public service use in the past six months, signaling a push in the country's AI sector.

Policy

White House science chief signals US-China co-operation on AI safety (26/Jan/2024)

In November, China signed the UK’s Bletchley Park agreement on standards for the technology, while US President Joe Biden and his Chinese counterpart Xi Jinping discussed working together on AI at a summit in California the same month.
“We are at a moment where everyone understands that AI is the most powerful technology . . . every country is bracing to use it to build a future that reflects their values,” said Prabhakar, who advises Biden on issues including AI regulation.
“But I think the one place we can all really agree is we want to have a technology base that is safe and effective,” she added. “So I think that is a good place for collaboration.”
Earlier this month, the Financial Times reported that leading US AI companies including OpenAI had engaged in covert meetings with Chinese experts to discuss the emerging risks associated with the technology.

Toys to Play With

Agent: Open interpreter (Jan/2024)

Open Interpreter lets LLMs run code (Python, Javascript, Shell, and more) locally. You can chat with Open Interpreter through a ChatGPT-like interface in your terminal by running $ interpreter after installing.
This provides a natural-language interface to your computer's general-purpose capabilities:
Create and edit photos, videos, PDFs, etc.
Control a Chrome browser to perform research
Plot, clean, and analyze large datasets
...etc.

Given the power of these agentic systems, I will issue a warning to please be careful with this kind of application. There are guardrails in place, but giving LLMs access to your computer’s operating system should be approached with eyes open.

Take a look: https://github.com/KillianLucas/open-interpreter

LLM App Stack by Andreessen Horowitz (a16z) (Jan/2024)

a16z has compiled a list of available tools, projects, and vendors at each layer of the LLM app stack.

Take a look: https://github.com/a16z-infra/llm-app-stack

How I used Midjourney to design a brand identity (19/May/2023)

This older article from May/2023 steps through using Midjourney to create a unique brand identity, detailing the process from defining graphic criteria to post editing for refinement.

Flashback

Someone recently drew my attention back to my ‘Roadmap’ paper from Jul/2022, several months before the release of ChatGPT.

…this paper provides a high-level overview of how I see the world right now, and for the next few months and years. For clarity, this is not a decades-long view; this is happening right now. Perhaps consider this report as a 48-month view from 2022 to 2026.

Read it (again): https://lifearchitect.ai/roadmap

The next roundtable will be:

Life Architect - The Memo - Roundtable #7
Follows the Chatham House Rule (no recording, no outside discussion)
Saturday 17/Feb/2024 at 4PM Los Angeles
Saturday 17/Feb/2024 at 7PM New York
Sunday 18/Feb/2024 at 8AM Perth (primary/reference time zone)
or check your timezone via Google.

You don’t need to do anything for this; there’s no registration or forms to fill in, I don’t want your email, you don’t even need to turn on your camera or give your real name!

All my very best,

Alan
LifeArchitect.ai

Search | Archives

The Memo by LifeArchitect.ai

Discussion about this post