The Memo - 5/Aug/2023
Google DeepMind RT-2, OpenAI G3PO, large language models coming to Alexa and Google assistants, and much more!
FOR IMMEDIATE RELEASE: 5/Aug/2023
Welcome back to The Memo.
You’re joining full subscribers from Reuters, The Associated Press (AP), many government departments and agencies, Accenture, Google, and more…
In this edition we look at my favourite paper of 2023 (so far), AI and legal advances, new open-source models, upcoming AI on-device with Alexa and Google assistants, new AI economic predictions measured in the quadrillions of dollars(!), and much more.
The BIG Stuff
Google DeepMind Robotics Transformer RT-2 (Jul/2023)
Google DeepMind’s latest robotics advance is a ‘vision-language-action’ model (VLA), and its capabilities are incredible. Hooked up to PaLI-X-55B or PaLM-E 12B, it is a significant evolution from the SayCan family of LLM-backed robots.
The table above says that language model-backed RT-2 robot can solve hard unseen problems—objects or backgrounds or environments they’ve never seen or been trained on before —a huge 62% of the time on average.
This bumped up my AGI countdown from 52 to 54%, as this is directly applicable to Woz's coffee test:
A machine is required to enter an average American home and figure out how to make coffee: find the coffee machine, find the coffee, add water, find a mug, and brew the coffee by pushing the proper buttons. (wiki)
My conservative AGI countdown: https://lifearchitect.ai/agi/
While there is no crossover in the project teams by authorship (my checks shown in the image above using duplicateword.com for RT-2, Chinchilla/Flamingo, PaLM 2), expect some of the general concepts presented in RT-2 to also be in the Gemini model before the end of the year... Coupled with the ‘Soft MoE’ advance on 2/Aug/2023 (paper), this is going to be a wild ride.
Gemini: https://lifearchitect.ai/gemini/
RT-2 project page with videos: https://robotics-transformer2.github.io/
RT-2 paper (PDF): https://robotics-transformer2.github.io/assets/rt2.pdf
Amazon Bedrock Agents (26/Jul/2023)
I’ve been waiting for a nice service like this since… 2021. Agentic AI can independently make informed decisions, take action, and adapt to changing circumstances, within a defined scope.
AI models have been exhibiting agentic capabilities for a while, and Amazon’s new service allows users to take advantage of independent and autonomous models that can go out and performs tasks independently.
Using agents for Amazon Bedrock, you can automate tasks for your internal or external customers, such as managing retail orders or processing insurance claims. For example, an agent-powered generative AI e-commerce application can not only respond to the question, “Do you have this jacket in blue?” with a simple answer but can also help you with the task of updating your order or managing an exchange.
The preview is closed/by application: https://aws.amazon.com/bedrock/
The Interesting Stuff
OpenAI may be preparing open-source model G3PO (25/Jul/2023)
[This is a non-English source:] According to foreign science and technology media… OpenAI in order to fight against Microsoft and Meta’s co-developed open source model Llama 2, is currently internal development codenamed “G3PO” of the new open source model, it is not clear when it will be released…
OpenAI currently uses a closed-source model, so it felt the pressure from the Llama 2 model, so it planned to release an open-source model two months ago, reportedly under the internal codename “G3PO”.
My estimate is that the G3PO model will be in the 45B-75B parameter range; perhaps triple GPT-3-scale with Chinchilla optimization (GPT-3 should have been only 15B parameters when trained on 300B tokens, to meet Chinchilla recommendations of 20 tokens per parameter). I also expect the name to be changed to something more palatable before release.
But it is an interesting project name! We have DALL-E (WALL-E + Dali), Megatron, many four-legged animals, most of the Muppets from Elmo to Big Bird, and you may recall that GPT-2 was internally known as ‘Snuffleupagus’. [Alan: This and more undercover information was part of my planned book about Integrated AI, with several publishing deals offered last year. Given the slow pace of publishing, I turned them all down, deciding instead to pen The Memo, allowing editions to be released in real-time to more readers.]
Jack Clark’s tweet from 26/Oct/2019:
Snuffleupagus, or Snuffy for short. We chose to name it GPT2 publicly as felt in poor taste to give muppet name while discussing reasons to be cautious with regard to increasingly powerful language models.
— https://twitter.com/jackclarkSF/status/1187824098916753408
I suppose G3PO is a little less cuddly, and unfortunately breaks my ‘no associating LLMs with sci-fi robots’ policy. Ah well, we wait for more info!
LightOn Alfred-40B-0723 (1/Aug/2023)
This is a great advance for open-source AI models. LightOn is an international team (10+ nationalities) with headquarters in downtown Paris. The company was founded in 2016. They’ve taken Abu Dhabi’s open-source May/2023 model Falcon 40B, and fine-tuned it with RLHF. It maintains the original architecture: 40B parameters on 1T tokens, for a 25:1 ratio. There is also enterprise access available.
Project page: https://www.lighton.ai/blog/lighton-s-blog-4/introducing-alfred-40b-0723-38
Try it: https://huggingface.co/lightonai/alfred-40b-0723
See it on the Models Table: https://lifearchitect.ai/models-table/
AI team sizes in the Western world (Mar/2023)
Here’s a look at the number of AI employees for some of the biggest AI labs. Amazon is the leader with over 10,000 AI staff. OpenAI is hidden within Microsoft, with only ~150 staff.
Source: https://www.glass.ai/glass-news/code-red-the-ai-armies-of-the-tech-giants
Amazon working on AI models for Alexa & all other businesses (3/Aug/2023)
According to the chart above, the largest AI lab in the Western world right now is Amazon. On their recent earnings call, CEO Andy Jassy went into how Amazon is applying large language models across every department.
On the AI question, what I would tell you, every single one of our businesses inside of Amazon, every single one has multiple generative AI initiatives going right now. And they range from things that help us be more cost-effective and streamlined in how we run operations in various businesses to the absolute heart of every customer experience in which we offer.
And so, it's true in our stores business. It's true in our AWS business. It's true in our advertising business. It's true in all our devices, and you can just imagine what we're working on with respect to Alexa there…
It is going to be at the heart of what we do. It's a significant investment and focus for us.
Read the Amazon Q2 2023 earnings call transcript.
Axios: Scoop: Google Assistant to get an AI makeover (1/Aug/2023)
Where are the language models on-device? Surely it doesn’t take 3+ years to whack an LLM like GPT-3 into some existing hardware! In the Western world, all the top devices are still using older knowledge engine + logic technology:
Apple Homepod and HomeKit and Siri (2B active Apple devices to Feb/2023).
Amazon Echo and Alexa (500M sold to May/2023).
Google Home and Nest.
More…
We first reported that Amazon was finally working on a new LLM to power Alexa in The Memo edition 30/Apr/2023. Now Google has made a similar revelation:
Google plans to overhaul its Assistant to focus on using generative AI technologies similar to those that power ChatGPT and its own Bard chatbot, according to an internal e-mail sent to employees Monday and seen by Axios…
The leaked Google internal email says:
‘We’ve seen the profound potential of generative AI to transform people's lives and see a huge opportunity to explore what a supercharged Assistant, powered by the latest LLM technology, would look like. (A portion of the team has already started working on this, beginning with mobile.)’
Read it: https://www.axios.com/2023/07/31/google-assistant-artificial-intelligence-news
To round this out, here are the remaining two big earnings calls from Aug/2023…
Apple: ‘AI and machine learning as core fundamental technologies that are integral to virtually every product that we build… it's absolutely critical to us… we've been doing research across a wide range of AI technologies, including generative AI for years. We're going to continue investing and innovating and responsibly advancing our products with these technologies with the goal of enriching people's lives.’
Read Apple’s Q2 2023 earnings call transcript.
Microsoft: ‘We had a solid close to our fiscal year. The Microsoft Cloud surpassed $110 billion in annual revenue, up 27% in constant currency, with Azure all-up accounting for more than 50% of the total for the first time. Every customer I speak with is asking not only how, but how fast, they can apply next generation AI to address the biggest opportunities and challenges they face – and to do so safely and responsibly. To that end, we remain focused on… investing to lead in the new AI platform shift by infusing AI across every layer of the tech stack.’
Read Microsoft’s Q2 2023 earnings call transcript.
[Alan: China is in a completely different world for AI on-device, which is why I’m careful to always add ‘in the Western world’ to this kind of analysis. Consider that when last reported four years ago, Baidu’s DuerOS home assistant technology—invisible in Western media—had more users than the entire population of the United States (2/Jul/2019).]
More AI + human performance charts (2/Aug/2023)
TIME magazine has provided an interesting visualization of AI’s progress. To me, some of those lines look nearly vertical, as we continue this exponential pace of change towards the Singularity (wiki).
Read the TIME source: https://time.com/6300942/ai-progress-charts/
See the TIME chart next to my AI + IQ charts: https://lifearchitect.ai/iq-testing-ai/
OpenAI trademarks GPT-5 term (18/Jul/2023)
I expect GPT-5 to begin training in Dec/2023, and to be released mid-2024. I’ve provided full coverage of this upcoming model in detail for readers around the world.