The Memo - 5/Aug/2023
Google DeepMind RT-2, OpenAI G3PO, large language models coming to Alexa and Google assistants, and much more!
FOR IMMEDIATE RELEASE: 5/Aug/2023
Welcome back to The Memo.
You’re joining full subscribers from Reuters, The Associated Press (AP), many government departments and agencies, Accenture, Google, and more…
In this edition we look at my favourite paper of 2023 (so far), AI and legal advances, new open-source models, upcoming AI on-device with Alexa and Google assistants, new AI economic predictions measured in the quadrillions of dollars(!), and much more.
The BIG Stuff
Google DeepMind Robotics Transformer RT-2 (Jul/2023)
Google DeepMind’s latest robotics advance is a ‘vision-language-action’ model (VLA), and its capabilities are incredible. Hooked up to PaLI-X-55B or PaLM-E 12B, it is a significant evolution from the SayCan family of LLM-backed robots.
The table above says that language model-backed RT-2 robot can solve hard unseen problems—objects or backgrounds or environments they’ve never seen or been trained on before —a huge 62% of the time on average.
This bumped up my AGI countdown from 52 to 54%, as this is directly applicable to Woz's coffee test:
A machine is required to enter an average American home and figure out how to make coffee: find the coffee machine, find the coffee, add water, find a mug, and brew the coffee by pushing the proper buttons. (wiki)
My conservative AGI countdown: https://lifearchitect.ai/agi/
While there is no crossover in the project teams by authorship (my checks shown in the image above using duplicateword.com for RT-2, Chinchilla/Flamingo, PaLM 2), expect some of the general concepts presented in RT-2 to also be in the Gemini model before the end of the year... Coupled with the ‘Soft MoE’ advance on 2/Aug/2023 (paper), this is going to be a wild ride.
Gemini: https://lifearchitect.ai/gemini/
RT-2 project page with videos: https://robotics-transformer2.github.io/
RT-2 paper (PDF): https://robotics-transformer2.github.io/assets/rt2.pdf
Amazon Bedrock Agents (26/Jul/2023)
I’ve been waiting for a nice service like this since… 2021. Agentic AI can independently make informed decisions, take action, and adapt to changing circumstances, within a defined scope.
AI models have been exhibiting agentic capabilities for a while, and Amazon’s new service allows users to take advantage of independent and autonomous models that can go out and performs tasks independently.
Using agents for Amazon Bedrock, you can automate tasks for your internal or external customers, such as managing retail orders or processing insurance claims. For example, an agent-powered generative AI e-commerce application can not only respond to the question, “Do you have this jacket in blue?” with a simple answer but can also help you with the task of updating your order or managing an exchange.
The preview is closed/by application: https://aws.amazon.com/bedrock/
The Interesting Stuff
OpenAI may be preparing open-source model G3PO (25/Jul/2023)
[This is a non-English source:] According to foreign science and technology media… OpenAI in order to fight against Microsoft and Meta’s co-developed open source model Llama 2, is currently internal development codenamed “G3PO” of the new open source model, it is not clear when it will be released…
OpenAI currently uses a closed-source model, so it felt the pressure from the Llama 2 model, so it planned to release an open-source model two months ago, reportedly under the internal codename “G3PO”.
My estimate is that the G3PO model will be in the 45B-75B parameter range; perhaps triple GPT-3-scale with Chinchilla optimization (GPT-3 should have been only 15B parameters when trained on 300B tokens, to meet Chinchilla recommendations of 20 tokens per parameter). I also expect the name to be changed to something more palatable before release.
But it is an interesting project name! We have DALL-E (WALL-E + Dali), Megatron, many four-legged animals, most of the Muppets from Elmo to Big Bird, and you may recall that GPT-2 was internally known as ‘Snuffleupagus’. [Alan: This and more undercover information was part of my planned book about Integrated AI, with several publishing deals offered last year. Given the slow pace of publishing, I turned them all down, deciding instead to pen The Memo, allowing editions to be released in real-time to more readers.]
Jack Clark’s tweet from 26/Oct/2019:
Snuffleupagus, or Snuffy for short. We chose to name it GPT2 publicly as felt in poor taste to give muppet name while discussing reasons to be cautious with regard to increasingly powerful language models.
— https://twitter.com/jackclarkSF/status/1187824098916753408
I suppose G3PO is a little less cuddly, and unfortunately breaks my ‘no associating LLMs with sci-fi robots’ policy. Ah well, we wait for more info!
LightOn Alfred-40B-0723 (1/Aug/2023)
This is a great advance for open-source AI models. LightOn is an international team (10+ nationalities) with headquarters in downtown Paris. The company was founded in 2016. They’ve taken Abu Dhabi’s open-source May/2023 model Falcon 40B, and fine-tuned it with RLHF. It maintains the original architecture: 40B parameters on 1T tokens, for a 25:1 ratio. There is also enterprise access available.
Project page: https://www.lighton.ai/blog/lighton-s-blog-4/introducing-alfred-40b-0723-38
Try it: https://huggingface.co/lightonai/alfred-40b-0723
See it on the Models Table: https://lifearchitect.ai/models-table/
AI team sizes in the Western world (Mar/2023)
Here’s a look at the number of AI employees for some of the biggest AI labs. Amazon is the leader with over 10,000 AI staff. OpenAI is hidden within Microsoft, with only ~150 staff.
Source: https://www.glass.ai/glass-news/code-red-the-ai-armies-of-the-tech-giants
Amazon working on AI models for Alexa & all other businesses (3/Aug/2023)
According to the chart above, the largest AI lab in the Western world right now is Amazon. On their recent earnings call, CEO Andy Jassy went into how Amazon is applying large language models across every department.
On the AI question, what I would tell you, every single one of our businesses inside of Amazon, every single one has multiple generative AI initiatives going right now. And they range from things that help us be more cost-effective and streamlined in how we run operations in various businesses to the absolute heart of every customer experience in which we offer.
And so, it's true in our stores business. It's true in our AWS business. It's true in our advertising business. It's true in all our devices, and you can just imagine what we're working on with respect to Alexa there…
It is going to be at the heart of what we do. It's a significant investment and focus for us.
Read the Amazon Q2 2023 earnings call transcript.
Axios: Scoop: Google Assistant to get an AI makeover (1/Aug/2023)
Where are the language models on-device? Surely it doesn’t take 3+ years to whack an LLM like GPT-3 into some existing hardware! In the Western world, all the top devices are still using older knowledge engine + logic technology:
Apple Homepod and HomeKit and Siri (2B active Apple devices to Feb/2023).
Amazon Echo and Alexa (500M sold to May/2023).
Google Home and Nest.
More…
We first reported that Amazon was finally working on a new LLM to power Alexa in The Memo edition 30/Apr/2023. Now Google has made a similar revelation:
Google plans to overhaul its Assistant to focus on using generative AI technologies similar to those that power ChatGPT and its own Bard chatbot, according to an internal e-mail sent to employees Monday and seen by Axios…
The leaked Google internal email says:
‘We’ve seen the profound potential of generative AI to transform people's lives and see a huge opportunity to explore what a supercharged Assistant, powered by the latest LLM technology, would look like. (A portion of the team has already started working on this, beginning with mobile.)’
Read it: https://www.axios.com/2023/07/31/google-assistant-artificial-intelligence-news
To round this out, here are the remaining two big earnings calls from Aug/2023…
Apple: ‘AI and machine learning as core fundamental technologies that are integral to virtually every product that we build… it's absolutely critical to us… we've been doing research across a wide range of AI technologies, including generative AI for years. We're going to continue investing and innovating and responsibly advancing our products with these technologies with the goal of enriching people's lives.’
Read Apple’s Q2 2023 earnings call transcript.
Microsoft: ‘We had a solid close to our fiscal year. The Microsoft Cloud surpassed $110 billion in annual revenue, up 27% in constant currency, with Azure all-up accounting for more than 50% of the total for the first time. Every customer I speak with is asking not only how, but how fast, they can apply next generation AI to address the biggest opportunities and challenges they face – and to do so safely and responsibly. To that end, we remain focused on… investing to lead in the new AI platform shift by infusing AI across every layer of the tech stack.’
Read Microsoft’s Q2 2023 earnings call transcript.
[Alan: China is in a completely different world for AI on-device, which is why I’m careful to always add ‘in the Western world’ to this kind of analysis. Consider that when last reported four years ago, Baidu’s DuerOS home assistant technology—invisible in Western media—had more users than the entire population of the United States (2/Jul/2019).]
More AI + human performance charts (2/Aug/2023)
TIME magazine has provided an interesting visualization of AI’s progress. To me, some of those lines look nearly vertical, as we continue this exponential pace of change towards the Singularity (wiki).
Read the TIME source: https://time.com/6300942/ai-progress-charts/
See the TIME chart next to my AI + IQ charts: https://lifearchitect.ai/iq-testing-ai/
OpenAI trademarks GPT-5 term (18/Jul/2023)
I expect GPT-5 to begin training in Dec/2023, and to be released mid-2024. I’ve provided full coverage of this upcoming model in detail for readers around the world.
Read more: https://lifearchitect.ai/gpt-5/
Full filing at USPTO: https://uspto.report/TM/98089548 and application table.
Berggruen Institute: The Illusion Of AI’s Existential Risk (18/Jul/2023)
AI acting on its own cannot induce human extinction in any of the ways that extinctions have happened in the past. Appeals to the competitive nature of evolution or previous instances of a more intelligent species causing the extinction of a less intelligent species reflect a common mischaracterization of evolution by natural selection.
Read the article: https://www.noemamag.com/the-illusion-of-ais-existential-risk/
Silly numbers being thrown around by AI pioneers (2022)
Professor Stuart J. Russell OBE (wiki) is not prone to bombastic exaggeration. And yet, his mention of AI providing ‘quadrillions’ of dollars of value seems glaringly overwrought. It’s estimated that there are only ~40 trillion dollars in circulation worldwide, so I’m not sure where we’re getting 14,000 trillion dollars from. While I’m not an economist, I think his reasoning (and number crunching) is very interesting.
Consider, instead, a more prosaic goal: raising the living standard of everyone on Earth, in a sustainable way, to a level that would be considered respectable in a developed country. Choosing “respectable” (somewhat arbitrarily) to mean the eighty-eighth percentile in the United States, this goal represents an almost tenfold increase in global GDP, from $76 trillion to $750 trillion per year. The increased income stream resulting from this achievement has a net present value of $13.5 quadrillion, assuming a discount factor of 5 percent. (The value is $9.4 quadrillion or $6.8 quadrillion if the technology is phased in over ten or twenty years.) These numbers tower over the amounts currently invested in AI research, and momentum toward this goal will increase as technical advances bring general-purpose AI closer to realization.
Read the 2022 paper: https://www.amacad.org/sites/default/files/publication/downloads/Daedalus_Sp22_03_Russell.pdf
This paper was recently picked up by Berkeley News in Apr/2023.
Law: AI + Legal (Aug/2023)
After my 2022 keynotes with Bond University’s Law department and Actium AI and others, I’ve been touching on the use cases for post-2020 AI and law. Here are a few interesting examples:
translating legal jargon into accessible language…
quickly create informative videos for court visitors, explaining topics like court dress codes or the forms residents need to file for custody. Generative AI tools can produce good quality educational videos in a matter of minutes…
help website visitors get answers to questions… a ChatGPT-based chatbot to answer basic logistical questions, such as when is their next court date….
having the tool rephrase a form with language that is readily understandable to anyone…
there’s huge potential for generative AI dispute resolution.
— via GovTech (28/Jul/2023)
I also like this simple infographic by the Texas Judicial Branch:
Law: Generative Interpretation (31/Jul/2023)
This might be favourite ‘human-readable’ paper of 2023 (so far). Researchers from UA Law and Harvard Law said:
You may have read, but would be wrong to conclude, that because the goal is to assign probability to the next word, these models simply imitate text they have seen elsewhere or only develop a superficial model of the world. These models exhibit originality—sometimes making up facts, while other times developing entirely new but responsive text. And yet, we should be very hesitant to call the model’s function understanding, even if to a first approximation this is a useful metaphor. Recent work on very simple transformers has shown that memorization is not a viable technique for complex models.
As large as they are, the models are much smaller than the data they are trained on. And so, models necessarily seek deeper representation of the information they train on. This is not unlike how humans read books, learn from them, but cannot recite them…
The model’s outputs are a brute statistical fact…
Models offer an approximation of general understanding that may simply not be available in any other way, and thus advance long-held goals of contract theory.
Download the 60-page paper: https://papers.ssrn.com/sol3/papers.cfm?abstract_id=4526219
A jargon-free explanation of how AI large language models work (31/Jul/2023)
Conventional software is created by human programmers, who give computers explicit, step-by-step instructions. By contrast, ChatGPT is built on a neural network that was trained using billions of words of ordinary language.
As a result, no one on Earth fully understands the inner workings of LLMs.
LK-99 + AI + BCIs (1/Aug/2023)
[Update: 16/Aug/2023 by Nature: ‘LK-99 isn’t a superconductor — how science sleuths solved the mystery’]
There’s a lot of technical jargon being thrown around right now in the world of physics. Here’s my plain English….
Scientists in Korea have discovered a new kind of metal (like aluminium) that doesn’t heat up when electricity is passed through it, responds to magnets by levitating, and conducts electric current with no resistance. The metal is actually a compound made from lead sulfate or lanarkite (Pb₂SO₅) and copper phosphide, and the name is from the scientists (Lee and Kim) and the year they started researching this thing (1999).
While LK-99 may not immediately seem to directly apply to AI, if this proves to be replicable (they’re still testing), then:
Expect it to benefit new chips and GPUs, speeding up AI training and inference considerably.Expect it to be somehow worked into non-invasive BCIs (brain-computer interfaces). No heat, no electronic ‘noise’, better magnetic sensing, faster speeds.Expect it to change the world of physics, electrics and electronics, computing, and artificial intelligence across the board.
The wikipedia article is the best live update source right now: https://en.wikipedia.org/wiki/LK-99
[Update: 16/Aug/2023 by Nature: ‘LK-99 isn’t a superconductor — how science sleuths solved the mystery’]
AI.com diverted from chat.openai.com to x.ai (2/Aug/2023)
Here’s a funny one… Back in The Memo edition 17/Feb/2023, I was very careful in my reporting of ai.com being purchased for $11M, and then redirected to ChatGPT. It wasn’t clear to me that OpenAI (or their lawyers) actually owned the domain.
Turns out I was right to be careful.
Whoever does own ai.com has this week pointed it to Elon Musk’s new ai company, x.ai.
Try it: https://ai.com/
News Corp using AI to produce 3,000 Australian local news stories a week (1/Aug/2023)
The Data Local team uses AI technology [like GPT] to generate stories on weather, fuel prices and traffic reports for hyperlocal mastheads.
Read more via The Guardian based on original exclusive by mediaweek.
Policy
Australian education system (Jul/2023)
Australia has quickly pulled together an entire Senate Inquiry into the use of generative artificial intelligence in the Australian education system. The House Standing Committee on Employment, Education and Training adopted an inquiry into the use of generative artificial intelligence in the Australian education system on 24 May 2023 following a referral from the Minister for Education, the Hon Jason Clare MP.
The 91 submissions from universities, schools, independent researchers, and other bodies, show the full scale from fear to excitement, with predictable responses from GO8 and IP-focused institutions. Perhaps the cleanest submission was from Amazon:
Educators have compared this [AI] innovation to the introduction of calculators or smartphones, and agree that continuous innovation is inevitable. Still, the adoption of generative AI within Australia has been rapid. School principals, teachers, university lecturers, faculty and IT staff alike are asking questions about the impact of this technology on teaching and learning more broadly, digital skilling, curriculum design and graduate readiness for employment in a rapidly changing workplace, among other issues.
At AWS, we believe generative AI provides a profound opportunity for the sector to critically and creatively reflect on the nature of assessment and meaningful learning outcomes. This provides a generational window to improve educational outcomes and even to develop a framework to address complex, longstanding problems across the Australian education landscape. Importantly, we see the adoption and use of AI/ML and generative AI in education as a critical component of Australia improving its position against the OECD’s Educational and Skills indicators and achieve against the United Nations’ Sustainability Development Goal 4 (Education) outcomes by 2030. (Amazon submission)
Read the submissions (includes nearly all major Aussie universities and institutions).
Toys to Play With
GPT-4-powered Perplexity now allows file upload (Aug/2023)
Perplexity is still ahead of its time, and looks even better than it did last year. Upload a PDF (under 10MB), and query it with any question.
Try it (free, login): https://www.perplexity.ai/
Catching up with Alan! (Aug/2023)
It was podcast and video week for the start of August 2023. Here’s the big stuff:
Interview with NASA’s Peter Scott about all things AI (link):
Interview with ABM Risk Australia about all things AI (link):
And here’s a fast turn-around! This one came to me while driving, and I wrote and edited it together within about 3-4 hours. It’s all about keeping a time capsule of early 2023, because we won’t be doing this ‘olden days’ stuff for much longer!
Watch my video (link):
I’ll be appearing with Adam Spencer and Dr Jordan Nguyen on 10/Aug/2023 in Melbourne (link to trailer):
Flashback
AI + IQ testing (2021-2023)
I used this viz recently during a recorded discussion. It goes all the way back to Sep/2021(!). Obviously we’ve advanced considerably since then, with PaLM 2 hitting 90.9% in the WinoGrande, just below the human benchmark.
Compare Sep/2021 with 2023: https://lifearchitect.ai/iq-testing-ai/
Next
I expect to be able to provide you with recordings of upcoming keynotes very shortly.
There are a few more videos in the pipeline, you can tune into those via my channel.
Let’s see what the second half of 2023 brings!
All my very best,
Alan
LifeArchitect.ai
Alan,
I just registered.
So, the only way to interact is to leave comments on your monthly issues? That would seem sub-optimal. But I did not like Reddit or YouTube and the potential for censorship.
What has happened to LETA? Was she decommissioned as I saw you recently did a retrospective.