The Memo - 30/Aug/2023

Llama-3 and OLMo rumours, Consensus, The Line, and much more!

FOR IMMEDIATE RELEASE: 30/Aug/2023

Welcome back to The Memo.

You’re reading alongside policy writers and decision makers within governments, agencies, and intergovernmental organisations including the ███, ██████, the ████, the ███, the █████ Government, the ███, the Government of █████, the trillion-dollar foreign reserves management company behind the Government of █████████, and more…

The winner of the Who Moved My Cheese? AI Awards! for August 2023 is theoretical physicist Prof Michio Kaku, who (very wrongly) refers to LLMs as ‘glorified tape recorders’:

It takes snippets of what’s on the web created by a human, splices them together and passes it off as if it created these things… And people are saying, ‘Oh my God, it’s a human, it’s humanlike.’

An author once said: ‘Better to remain silent and be thought a fool than to speak and to remove all doubt.’

This is another very long edition, and if you’re interested in reading and/or listening, there are several hours of updates here. In the Toys to play with section, we look at the latest way to run Llama 2 on Mac, a new gold-standard LLM platform for academic research and writing, Harvey Castro MD’s new LLM one-pagers, a new free DALL-E 2 experiment, and a rehearsal version of my latest keynote for a major government.

I will be ramping up the livestreams over the next few weeks. The best way to be notified about those is to click some buttons (Subscribe ➜ Notify) on the YouTube channel:

https://www.youtube.com/c/DrAlanDThompson

The next roundtable for full subscribers will be:

Life Architect - The Memo - Roundtable #3 with Harvey Castro
Follows the Chatham House Rule (no recording, no outside discussion)
Saturday 23/Sep/2023 at 5PM Los Angeles
Saturday 23/Sep/2023 at 8PM New York
Sunday 24/Sep/2023 at 8AM Perth (primary/reference time zone)
or check your timezone via Google.

Details at the end of this edition.

The BIG Stuff

AI vs Human: The Creativity Experiment (29/Aug/2023)

I recently appeared on ABC Catalyst in Australia. The documentary is called AI vs Human: The Creativity Experiment, and also features my friend and colleague Prof Jeremy Howard (yes, the Aussie that invented large language model training and fine-tuning as we know it!).

In Australia, you can stream on ABC iview with some alternate viewing times on ABC TV.

Preview clip:

ChatGPT Enterprise (28/Aug/2023)

It’s the most common concern I hear during my keynotes: What about my data? While Microsoft Azure has offered a solution to that during the first half of 2023, now OpenAI has a direct offering: ChatGPT Enterprise.

You own and control your business data in ChatGPT Enterprise. We do not train on your business data or conversations, and our models don’t learn from your usage. ChatGPT Enterprise is also SOC 2 compliant and all conversations are encrypted in transit and at rest. Our new admin console lets you manage team members easily and offers domain verification, SSO, and usage insights, allowing for large-scale deployment into enterprise.

Notably, ChatGPT Enterprise provides GPT-4 as ‘unlimited’ and 2x faster than standard.

Read more: https://openai.com/blog/introducing-chatgpt-enterprise

See OpenAI’s pen test and compliance docs: https://trust.openai.com/

Increasing LLM training budgets (25/Aug/2023)

AnthropicAI’s CEO Dario Amodei says:

Right now the most expensive model [in Aug/2023] costs +/- $100m. Next year we will have $1B+ models. By 2025, we may have a $10B model. (24/Aug/2023, Twitter)

Here are my draft numbers for training spend on an LLM (in USD) over the years:

2018: BERT 330M (4 days x 64 TPUv2): $7K
2019: GPT-2 1.5B (7 days x 256 TPU v3): $43k
2020: GPT-3 175B (1 month x 1,024 V100s): $4.6M
2021: MT-NLG 530B (3 months x 2,000 A100s): $25M
2022: PaLM 540B: (64 days x 6,144 TPU v4): $23.1M
2022: BLOOM 176B (176 days x 384 A100s): $5M
2023: GPT-4/PaLM 2/Gemini (Gemini on TPUv5) 2T: $100M
2024 (4 months on 25,000 H100s): GPT-5: $1B
2025 (not yet announced hardware): GPT-6: $10B

GPT-4 (my experience is that this model is more like GPT-4.5 when using ‘Advanced Data Analysis’, which is the new name for the ‘Code Interpreter’ as of 28/Aug/2023) kindly generated this chart within seconds based on my rough working above:

And here’s a paper about model training costs up to 2021: https://epochai.org/blog/trends-in-the-dollar-training-cost-of-machine-learning-systems

For a little bit of history, see an article from all the way back in 2019, ‘The Staggering Cost of Training SOTA AI Models’ (Jun/2019). As pointed out there, ‘the cost of the compute used to train models is also expected to become significantly cheaper with the continuing advance of algorithms, computing devices, and engineering efforts.’

In plain English, as we move along the timeline, AI labs continue to increase their training spend, just as the capability of training hardware increases (and/or hardware cost decreases). A perfect storm for the evolution of humanity.

Llama-3 and Llama-4 rumours (26/Aug/2023)