The Memo - 31/Oct/2023
Amazon using Digit humanoids already, RedPajama-Data-v2 with 30T tokens, 28 new AI models, and much more!
FOR IMMEDIATE RELEASE: 31/Oct/2023
OpenAI CEO (22/Oct/2023):
We define AGI as ‘the thing we don’t have quite yet.’ There were a lot of people who would have—ten years ago [2013 compared to 2023]—said alright, if you can make something like GPT-4, GPT-5 maybe, that would have been an AGI… I think we’re getting close enough to whatever that AGI threshold is going to be.
Welcome back to The Memo.
You’re joining full subscribers from Lockheed Martin, L'Oréal, the Linux Foundation, Lincoln University, Locus Robotics, Lycos (wow, that’s a blast from the past!), Landmark Education, and more…
This edition’s Policy section is extraordinarily long to address developments around the world in Oct-Nov/2023. In the Toys to play with section, we look at a new audio model with vocals, a famous book simulated by GPT-4, a new way of writing programs you can talk to, using seeds in DALL-E 3, and much more…
The next roundtable for full subscribers will be:
Life Architect - The Memo - Roundtable #4
Follows the Chatham House Rule (no recording, no outside discussion)
Saturday 4/Nov/2023 at 5PM Los Angeles
Saturday 4/Nov/2023 at 8PM New York
Sunday 5/Nov/2023 at 8AM Perth (primary/reference time zone)
or check your timezone via Google.
Details at the end of this edition.
Every now and again, an amazing output from GPT-4V (GPT-4 Vision) makes me laugh. This is one of those times (Reddit, 22/Oct/2023). This is a well-known publicity shot of the OpenAI team in 2023: Mira (CTO), Sam (CEO), Greg (President), Ilya (Chief Scientist).
Following a run-in with a misinformed ‘leader’ here in Australia, I was guided to issue another press release a few days ago on 25/Oct/2023.
Artificial general intelligence is here. Leaders not endorsing this revolution are guilty of negligence.
Having AI augment and amplify humans is an exciting event, but Dr Thompson warns that most world leaders were understating or ignoring the present level of AI capabilities. “Burying heads in the sand is irresponsible. Discussions about copyright and IP are a distraction. Comparisons with robots in Hollywood movies are a misuse of imagination. Commentary downplaying AI’s role in replacing outdated job roles is dangerous. Nearly all leaders are demonstrating negligence in their lack of preparation, understanding, and application of a growth mindset during humanity’s most rapid and important evolution.”
Read this media release from 25/Oct/2023: https://lifearchitect.ai/leaders-guilty-of-negligence/
Read ‘AI is outperforming humans in both IQ and creativity in 2021’ (19/Sep/2021): https://lifearchitect.ai/outperforming-humans/
Read ‘AI fire alarm’ (20/Jul/2021): https://lifearchitect.ai/fire-alarm/
We are entering the final few weeks of 2023. I will be running public livestreams every Tuesday night (US time) until the end of the year.
Notify/watch: https://www.youtube.com/@DrAlanDThompson
The BIG Stuff
RedPajama-Data-v2: an open dataset with 30 trillion tokens for training LLMs (30/Oct/2023)
Together AI has released a new version of the RedPajama dataset, with 30T filtered and deduplicated tokens from 84 Common Crawl (web crawl, the Google version is known as the Colossal Clean Crawled Corpus or C4) dumps covering 5 languages. This dataset is believed to be the largest public dataset ever released for large language model (LLM) training.
At an estimated 125TB of data for 30T tokens, RedPajama-Data-v2 is:
2.3× larger than the dataset used to train GPT-4 across 13T tokens (estimated).
4.7× larger than the next biggest public dataset (CulturaX by UOregon).
6× larger than TII’s RefinedWeb used for training Falcon 180B just a few months ago in Jun/2023.
Amazon trials humanoid robots to 'free up' staff (19/Oct/2023)
Amazon is testing a new humanoid robot, called 'Digit', in its US warehouses. The robot is designed to automate repetitive tasks, with the intention of allowing employees to focus more on customer service. Despite concerns, Amazon insists its use of robots, now numbering over 750,000, has led to the creation of hundreds of thousands of new jobs.
The knees on these things bend backwards. An interesting early example of AI optimizing our world for even greater efficiency than natural evolution...
Read more about Amazon and Digit: https://www.bbc.com/news/technology-67163680
Read more about Digit: https://agilityrobotics.com/
Read more about humanoid robots: https://lifearchitect.ai/humanoids/
Watch the video (link):
Microsoft: ChatGPT is only 20B parameters (26/Oct/2023)
Parameters (or weights) tells us the number of connections in the model, and can be compared with human brain synapses (connections between neurons). Generally, the more parameters, the more powerful the model, though there is a lot of new research on compression and optimization.
In a paper on a new model called CodeFusion, Microsoft has revealed that gpt-3.5-turbo (the model behind ChatGPT) is only 20B parameters, far smaller than previously thought. This may suggest that there is significant optimization happening behind the scenes to serve ChatGPT’s 200 million users.
I’ve updated my viz to match; note the stark size difference between ChatGPT 20B and GPT-4 1760B (in red). That means GPT-4 is 88x times larger than ChatGPT. Even the open-source Llama 2 is 3.5× larger than ChatGPT!
No wonder OpenAI’s CEO called ChatGPT a ‘horrible product… really not designed to be used…’ (10/Feb/2023).
See the viz: https://lifearchitect.ai/models/
Read the paper: https://arxiv.org/abs/2310.17680
Update: The paper was quickly withdrawn by Microsoft. A backup of the paper is available for readers of The Memo here:
OpenAI preparedness challenge (26/Oct/2023)
OpenAI is launching the Preparedness Challenge to expand its understanding of potential areas of concern. The challenge invites participants to imagine a scenario where they have unrestricted access to OpenAI’s models and consider the most unique potentially catastrophic misuse of the model. It also asks for mitigation strategies against such misuse. Up to 10 top submissions will receive US$25,000 each in API credits.
Imagine we gave you unrestricted access to OpenAI’s:
Whisper (transcription)
Voice (text-to-speech)
GPT-4V [vision/eyes]
DALLE·3 [images]
models and you were a malicious actor. Consider the most unique, while still being probable, potentially catastrophic misuse of the model… For example, a malicious actor might misuse these models to uncover a zero-day exploit in a government security system.
I expect that the company will use GPT-n to summarize, rate, and rank the thousands of responses they are sure to receive.
Read more: https://openai.com/form/preparedness-challenge
The Interesting Stuff
New models (Oct/2023)
There were 18 models announced in September 2023 (highlights only):
SUTD/Independent - TinyLlama (1.1B), TII - Falcon 180B (180B), BAAI - FLM-101B (101B), Adept - Persimmon-8B (8B), Apple - UniLM (0.034B), Microsoft - phi-1.5 (1.3B), Singapore - NExT-GPT (7B), IBM - MoLM (8B), Deci - DeciLM (5.7B), ThirdAI - BOLT2.5B (2.5B), Baichuan - Baichuan 2 (13B), Microsoft - Kosmos-2.5 (1.3B), Mistral AI - Mistral 7B (7.3B), Hessian AI/LAION - LeoLM (13B), Meta AI - Llama 2 Long (70B), Alibaba - Qwen (14B), Wayve - GAIA-1 (9B), Waymo - MotionLM (0.09B).
I counted 10 models announced in October 2023 (highlights only):
Google DeepMind - RT-X (55B), Reka AI - Yasa-1, KAUST/Shenzhen - AceGPT (13B), XLANG Lab - Lemur (70B), NVIDIA - Retro 48B (48B), Google DeepMind - PaLI-3 (5B), Hugging Face H4 - Zephyr (7.3B), Baidu - ERNIE 4.0 (1T+), Adept - Fuyu (8B), Jina AI - jina-embeddings-v2 (0.435B).
See the Models Table with playground/paper links: https://lifearchitect.ai/models-table/
More OpenAI rumors from Jimmy Apples (Oct/2023)
‘Jimmy Apples’ is a pseudonym probably used by someone inside or with intimate knowledge of OpenAI. While the Twitter account was paused for a while, it is back with a vengeance…
OpenAI received the first of its 25,000 NVIDIA H100 GPUs in Oct/2023.
OpenAI CEO investing in a non-invasive brain-computer interface (BCI) in Oct/2023 (‘Hey @sama, would be rather interesting of you to be working on a BCI device, non-invasive, via a stealth startup.‘).
OpenAI potentially losing employees in Oct/2023 (‘There’s been a vibe change at openai and we risk losing some key ride or die openai employees.’).
Conversation about AGI and UBI (‘there are [GPT-4] agents who can learn and update knowledge, possess vision, and work on long-term goals (e.g., agents that can work on tasks for several months).’).
Exclusive: Agents that can work on tasks for several months (Oct/2023)
It’s exclusive in that I’m talking about it here while no-one else seems to be (they’re off having arguments about AI regulation, copyright, and god-knows-what-else), but ‘Jimmy’ noted it first above.
I think I have a pretty good imagination, but it’s a challenge to imagine an agent with the speed and depth of GPT-n working on any problem (even a super wicked problem) for months. Off the top of my head, these may be some good examples, but do they really take months of processing?
Economy: As we integrate AI + robots, and humans no longer need to work to produce goods or services (the ‘post-scarcity economy’, wiki), redesigning the entire world economy down to allocation of $ for each person.
Energy: redesigning our energy harnessing and storage using new methods, from sustainable extraction, transformation, storage, distribution, and utilization.
Environment: Planning out and deploying a full resolution to our current environmental challenges including climate change.
Happiness (or your choice of similar word): Ensuring human flourishing post-AI, in a world with no employment/work needed, and in a world where everyone is living inside ‘full dive virtual reality’, FDVR (wiki), in line with Prof Martin Seligman’s PERMA model: Positive Emotion, Engagement, Relationships, Meaning, and Accomplishments.
GPT-4 integrated tools (Oct/2023)
Tooling for the GPT-4 model within the chat.openai.com platform is being slowly rolled out. Paid users can now send ‘anything’ to GPT-4, and it will work out what to do with it.
Waymo MotionLM for autonomous cars (26/Oct/2023)
Waymo recently put out a paper that used Transformers to forecast vehicle motion. It may have shortcut 10 years of work within just a few hours of training…
The interview is with Joe Ternasky, former engineer at Apple, Microsoft, Google, Adobe, Facebook, and Splunk. It’s a really fun and interesting conversation.
These researchers took a language model and tweaked it to put out these little velocity vectors and spent a few months dorking around with it, and we're going to publish this paper that basically says ‘this is better than everything you've worked on [at Waymo and Cruise and Uber] for the last decade’…
OpenAI Chief Scientist: Ilya Sutskever, OpenAI’s chief scientist, on his hopes and fears for the future of AI (26/Oct/2023)
Ilya Sutskever, OpenAI’s cofounder and chief scientist, shares his vision and concerns for artificial superintelligence. He believes that the world needs to recognize the true power of the technology his company and others are developing, and anticipates that some humans will one day choose to merge with machines.
Read more via MIT (3,500 words, about 11mins).
DeepMind Chief Scientist: AGI will be another five years (27/Oct/2023)
No, Shane is not Aussie, he’s a Kiwi. This is a great interview about all things AI, with an interviewer that knows how to ask good questions.
As DeepMind’s chief scientist, Shane’s prediction that artificial general intelligence (where a machine can perform at the level of an average human in any field) will be in 2028-2029 obviously means that Google DeepMind Gemini will not be AGI. You can see my predictions for AGI being achieved much sooner than that in my conservative countdown to AGI.
Watch the video (link):
Bentley GAI (12/Oct/2023)
Generative AI for infrastructure engineering, beginning with an AI agent assisting engineers in further optimizing site layouts by leveraging designs and data from previous projects… generative AI can be applied to minimize time spent on project documentation by automating drawing production with fit-for-purpose annotations.
Read the media release via Bentley.
Watch the video (also on Twitter and YouTube):
How logistics moves forward | Android EVE by 1X (20/Oct/2023)
1X has investment from OpenAI. Their wheeled robot is called EVE, and their bipedal (two legs for walking) robot is called NEO.
Read more about humanoids: https://lifearchitect.ai/humanoids/
Browse the 1X EVE page: https://www.1x.tech/androids/eve
Watch the video (link):
Ray’s book is coming in June 2024 (28/Oct/2023)
After some speculation that the book release was being abandoned, Ray Kurzweil’s upcoming book The Singularity Is Nearer: When We Merge with Computers has a new launch date of 18/Jun/2024.
I personally think that AI is moving much too fast for a published book, and that is one of the main reasons that we have The Memo.
Check it out: https://www.amazon.com/dp/0399562761/
His original 2005 book The Singularity is Near can be read at archive.org.
Japanese tea commercial actress created by AI, has some wondering if it’s the scandal-free future (17/Oct/2023)
Japanese tea maker Ito En recently launched a commercial for its Oi Ocha Catechin Green Tea, featuring an AI-created model as the spokesperson. This has sparked conversations about the future of AI in advertising and the potential of AI models to offer a scandal-free alternative to human endorsers.
Policy
UN: Stressing artificial intelligence could power extraordinary progress for humanity, UN Secretary-General says new high-level advisory body ‘is the starting point’ (26/Oct/2023)
Alan’s alternative title: UN pulls their finger out on AI three years too late