The Memo - 29/Mar/2023
Figure 01, Cerebras-GPT, Neuralink seeking human trials, ChatGPT retrieval, and much more!
FOR IMMEDIATE RELEASE: 29/Mar/2023
Welcome back to The Memo.
The Policy section makes a comeback to analyze the Goldman Sachs report on AI and the economy.
In the Toys to play with section, we look at setting GPT-4 loose in a browser to do whatever it wants, Synthesia’s hidden AI avatar channel with millions of views, and demonstrating the incredible skills of ChatGPT using retrieval on UN documents.
The BIG Stuff
GPT-4 has 1T parameters, according to eight sources (26/Mar/2023)
I have significantly expanded my GPT-4 summary page, including Semafor’s recent disclosure (via eight sources) that GPT-4 has 1T (1,000B) parameters. It is still unclear whether this is a dense model like most other models, or a sparse MoE model.
This means that GPT-4 1T would be 6x larger than GPT-3 175B, and 14x larger than Chinchilla 70B.
With Chinchilla-scaling indicating training GPT-4 1T with 20T tokens, GPT-4 1T would have been trained on 66x more data than GPT-3 175B (300B tokens), 14x more data than Chinchilla (1.4T tokens).
Read more: https://lifearchitect.ai/gpt-4/
Neuralink seeking human trials (27/Mar/2023)
Elon Musk's brain implant company Neuralink has approached one of the biggest U.S. neurosurgery centers as a potential clinical trials partner as it prepares to test its devices on humans once regulators allow for it, according to six people familiar with the matter.
I’ve spoken a lot about how AI models (like GPT-3 or GPT-4) will be directly integrated with the brain, and the enormous benefits to doing so, from pain relief to restoring sight to thought transfer.
Read more: https://lifearchitect.ai/bmi/
Watch my video from a year ago—Apr/2022—simulating how a brain-computer interface might use a large-language model (timecode):
We recently played around with a short live-stream using GPT-4 in the same way!
Watch the video:
Future of Life Institute calling for a ‘pause’ on AI development (29/Mar/2023)
‘We call on all AI labs to immediately pause for at least 6 months the training of AI systems more powerful than GPT-4.’
This is laughable. Surprising signatories include:
Connor Leahy (first to re-create GPT-2)
Steve Wozniak (founded Apple)
Elon Musk (founded OpenAI)
Andrew Yang (UBI advocate)
1,000+ more…
EDIT: Confirmed by Reuters.
I will not be signing.
As Ray Kurzweil noted a few years ago:
You can’t stop the river of advances. These ethical debates are like stones in a stream. The water runs around them. You haven’t seen any of these… technologies held up for one week by any of these debates. — Dr Ray Kurzweil (January 2020, and further in his prediction here.)
Read the petition: https://futureoflife.org/open-letter/pause-giant-ai-experiments/
The Interesting Stuff
Cerebras-GPT 13B released (29/Mar/2023)
‘All Cerebras-GPT models are available on Hugging Face… All models in the Cerebras-GPT family have been trained in accordance with Chinchilla scaling laws (20 tokens per model parameter) which is compute-optimal. These models were trained on the Andromeda AI supercomputer comprised of 16 CS-2 wafer scale systems.’
Cerebras-GPT 13B underperforms similar models, and severely underperforms LLaMa 13B on benchmarks that matter (thanks to binarymax at HN):
Try it: https://huggingface.co/cerebras
Figure.ai is a contender for humanoid embodied AI (4/Mar/2023)
In The Memo 24/Mar/2023 edition, we mentioned Agility Robotics’ Digit v4 (video). There are several other contenders, and I have now listed some of these at my conservative countdown to AGI.
Figure 01 was unveiled at the beginning of Mar/2023, at 60kg and 170cm or 5’6”. It has five hours of electric battery life, can carry 20kg, and walks at around 4.3km/h (1.2m/s). The company says: ‘Figure is the first-of-its-kind AI robotics company bringing a general purpose humanoid to life… The world’s first commercially-viable autonomous humanoid robot’
With $100M in funding, this will be the missing piece between post-2020 AI models (language, visual, multimodal models) and physical embodiment. If I were looking to help steer a country (or world!), I would go ‘all in’ on this. Artificial general intelligence (AGI) is nearly here, and needs to be inside a physical form factor like a humanoid body. Once that is done, we will move into this AI revolution completely, changing the way all of humanity operates, removing the need for any and all ‘work,’ and opening up a whole new world.
‘The team is ex-Boston Dynamics, Tesla, Apple SPG, IHMC, Cruise [and Alphabet X]. Collectively we align on building a better future for humanity through the intersection of AI and robotics.’