The Memo - Special edition: Llama 3

Meta AI releases Llama 3 70B, 405B coming soon...

To:      US Govt, major govts, Microsoft, Apple, NVIDIA, Alphabet, Amazon, Meta, Tesla, Citi, Tencent, IBM, & 10,000+ more recipients…
From:    Dr Alan D. Thompson <LifeArchitect.ai>
Sent:    18/Apr/2024
Subject: The Memo - AI that matters, as it happens, in plain English
AGI:     72%

The BIG Stuff

Meta AI releases Llama 3

Once again, we have this out to The Memo readers within about an hour of model release.

Key points:

• Llama 3 includes 8B, 70B, and 405B (coming soon) models.

• 70B parameters trained on 15T tokens (215:1). ‘Llama 3 is pretrained on over 15T tokens that were all collected from publicly available sources. Our training dataset is seven times larger than that used for Llama 2, and it includes four times more code.’
  [Sidenote: I’ve recently updated my Chinchilla advisory note to account for new papers recommending increasing training data from 20:1 → 190:1. Llama 3 exceeds this recommendation.]

• Dataset used ‘some synthetic AI-generated data’ (The Verge, 18/Apr/2024).

• 70B Instruct MMLU=82.0. GPQA=39.5.

• A 405B parameter model is still in training, and scoring MMLU=84.8+.

• The original Llama models had more than 100 million downloads in a year, so this new open model ‘will be everywhere’.

MMLU scores for Llama 1, 2, and 3:
Llama 1 65B on 1.4T tokens (22:1) • Feb/2023 • MMLU=63.4
Llama 2 70B on 2T tokens (29:1) • Jul/2023 • MMLU=68.9
Llama 3 70B on 15T tokens (215:1) • Apr/2024 • MMLU=82.0

Read the Llama 3 announce: https://ai.meta.com/blog/meta-llama-3/

Model card: https://github.com/meta-llama/llama3/blob/main/MODEL_CARD.md
The paper will be released ‘in the coming months’.

Watch the Meta CEO talk about today’s Llama 3 release (link):

Download the weights: https://llama.meta.com/llama3/

Playgrounds will include HF and Poe.com shortly, and the model is already available on Azure and IBM Watsonx.

Llama 3 is built into the new Meta AI assistant (login): https://www.meta.ai/

Llama 3 is now on Poe (free, login): https://poe.com/Llama-3-70B-T

Sidenote: try my ALPrompt: https://lifearchitect.ai/ALPrompt/

See also my Models table, and Timeline.

I’d like to invite you to gift a subscription to someone in your world who needs AI that matters, as it happens, in plain English:

All my very best,

Alan
LifeArchitect.ai