The Memo - 3/Apr/2023
OpenAI's investment in 1X NEO robots, Gemini, BloombergGPT 50B, GPT4All-LoRa 7B, and much more!
FOR IMMEDIATE RELEASE: 3/Apr/2023
Welcome back to The Memo.
What a massive month we had for AI in March; publishing a record eight editions of The Memo! Don’t expect that to happen too often, as we aim for a monthly release cadence. Though, as usual, April is off to a noisy start. There is a fair amount of information triage in these editions now, otherwise this would be an encyclopedia.
Our first casual ‘roundtable’ group video call (on 1/Apr/2023) was a lot of fun! We had a small group of informed like minds, covering AI topics including Apple, agency and goals, and universal AI guidelines. Thanks to those who joined in, and if you’d like to be in the next roundtable, paid members can keep an eye out here in the coming months.
The winner of the Who Moved My Cheese? AI Awards! for April is Italy.
In the Toys to play with section, we look at one of my favorite post-2020 AI apps now for iPhone, a simple Siri replacement, and the funniest emulation of ChatGPT I’ve seen so far!
The BIG Stuff
OpenAI’s investment in 1X NEO (31/Mar/2023)
This is really big stuff. Embodiment of large language models has been on the waiting list for a little while, and is the next big milestone on my conservative countdown to AGI. Several players have started work on integrating robots with language models already, including Google and Microsoft.
Now, OpenAI have invested around US$25M in a company called 1X (formerly Halodi Robotics), enabling them to embody GPT-4 and GPT-n.
The NEO robot is beautiful, the images and videos are real—not CGI—and it is due for release some time in the US summer (Jun-Aug/2023). The previous version of the bot, EVE, used wheels to get around.
Read the press release by 1X quoting OpenAI.
Browse the 1X website: https://1x.tech/
This video of EVE is 3 years old, but gives a feel for the design. Note that the new body uses a soft fabric (see the photo of NEO above), which is new to me!
A quick note on AI progress coverage (Apr/2023)
Press outlets seem to be providing next to zero coverage of NEO (above), or many of the latest developments in other AI areas like open source releases, physical embodiment, brain-computer interfaces, AGI, and more.
The media will always focus on the past—and the drama—but this isn’t where the real action is. In the same way that I’ve revealed major releases sometimes months (Leta GPT-3 Episode 0 video from Apr/2021) before they become ‘mainstream,’ I’ll continue to provide visibility for the stuff that matters.
Right now, the stuff that matters is:
The discovery of GPT-4’s capabilities. Like GPT-3, this process will take years.
GPT-5 training right now through to Dec/2023.
Embodiment of AI models, and general robot design.
Connection of AI models to our biological brains via Stentrode and other options.
An intergovernmental and universal constitution for AI: both dev and alignment.
Project Gemini: Google and DeepMind play catchup (Mar/2023)
Google is the reason that we’re here right now. In AI, they’ve given us:
Transformer, BERT, LaMDA 137B, PaLM 540B, and many of the latest optimizations.
On the other side of the pond, DeepMind in London have worked diligently on:
Gopher 280B, Chinchilla 70B, Flamingo 80B, Gato 1.2B, Sparrow 70B, and more.
The two companies are coming together with secret project ‘Gemini’ to combine expertise and compute. The required hardware will run into the hundreds of millions of dollars, with rumours that they are using ‘tens of thousands’ of TPUs to train a trillion-parameter model. While Google has already given us the 1.6T-parameter Switch Transformer (Jan/2021, paper), the Gemini project should achieve a much more thorough and optimized multimodal result.
3rd anniversary of ‘The New Irrelevance of Intelligence’ (1/Apr/2023)
Well before GPT-3 and the explosion of post-2020 AI, this was an unnerving article to write, release, and then present. The draft was sent to my editor on 1/Apr/2020. You may enjoy reading the official ‘camera ready’ article published in the Journal of Australian Mensa the following month.
The Interesting Stuff
‘The AI Pause’ distraction (30/Mar/2023)
Further to my comments in The Memo edition 29/Mar/2023 that the AI pause petition is ‘laughable’, I now see the proposal as even more disenchanting. It seems to be a power and publicity grab from people feeling left out, and a very human misunderstanding of post-2020 AI.
As I quote in my new executive summary, ‘no-one here is smart enough. Including the rocket scientist’. AI will continue to progress, and its superintelligence will help us guide it.
OpenAI provided a key insight to this several years ago:
…an AGI will be a system capable of mastering a field of study to the world-expert level, and mastering more fields than any one human—like a tool which combines the skills of Curie, Turing, and Bach.
An AGI working on a problem would be able to see connections across disciplines that no human could. We want AGI to work with people to solve currently intractable multi-disciplinary problems, including global challenges such as climate change, affordable and high-quality healthcare, and personalized education. We think its impact should be to give everyone economic freedom to pursue what they find most fulfilling, creating new opportunities for all of our lives that are unimaginable today. (via OpenAI, 22/Jul/2019)
I also found a disappointing response issued by Drs Timnit Gebru and Emily Bender, authors of the infamous ‘stochastic parrots’ paper (PDF), which upset Leta AI all the way back in Episode 16, Aug/2021 (video timecode).
Read more: https://www.dair-institute.org/blog/letter-statement-March2023
LAION’s alternative petition (29/Mar/2023)
I did find a sliver of light in LAION’s alternative petition, ‘Securing Our Digital Future: Calling for CERN like international organization to transparently coordinate and progress on large-scale AI research and its safety.’
The recent proposition of decelerating AI research as a means to ensure safety and progress presents an understandable but untenable approach that will be detrimental to both objectives. Corporate or state actors will make advancements in the dark while simultaneously curtailing the public research community's ability to scrutinize the safety aspects of advanced AI systems thoroughly. Rather than impeding the momentum of public AI development, a more judicious and efficacious approach would be to foster a better-organized, transparent, safety-aware, and collaborative research environment.
To this end, they want a supercomputer with ‘at least’ 100,000 GPUs(!), which would make it 4x bigger than the high-ranking Microsoft/OpenAI supercomputer being used to train GPT-5.
Read more: https://laion.ai/blog/petition/
Judge asks GPT to decide bail in murder trial (29/Mar/2023)
You may be experiencing déjà vu, as a judge also used ChatGPT for a ruling in Colombia back in Jan/2023. This one is from India.
Prompt (human): What is the jurisprudence on bail when the assailant assaulted with cruelty?
Response (GPT): If the assailants have been charged with a violent crime that involves cruelty, they may be considered a danger to the community and a flight risk. In such cases, the judge may be less inclined to grant bail or may set the bail amount very high to ensure that the defendant appears in court and does not pose a risk to public safety. However, in general, if the assailants have been charged with a violent crime that involves cruelty, such as murder, aggravated assault, or torture, they may be considered a danger to the community and a flight risk.
The revolutionary decision occurred at the Punjab and Haryana High Court during the trial of Jaswinder Singh, who was arrested in 2020 for allegedly assaulting and killing an individual, Bar and Bench reported. He was reportedly seeking a stay of imprisonment during the trial.
Left at a loss for how to handle the situation, judge Anoop Chitkara decided to seek counsel from GPT-4… Armed with [GPT’s] legal expertise, Chitkara ultimately rejected the defendant’s bail bid on the grounds that they did act cruelly before the victim died.
See my full list of ChatGPT capabilities (Google Sheet).
Self-reflection emerging in GPT-4 (Apr/2023)
Building on recent research, we propose Reflexion, an approach that endows an agent with dynamic memory and self-reflection capabilities to enhance its existing reasoning trace and task-specific action choice abilities. To achieve full automation, we introduce a straightforward yet effective heuristic that enables the agent to pinpoint hallucination instances, avoid repetition in action sequences, and, in some environments, construct an internal memory map of the given environment. To assess our approach, we evaluate the agent's ability to complete decision-making tasks in AlfWorld environments and knowledge-intensive, search-based question-and-answer tasks in HotPotQA environments. We observe success rates of 97% and 51%, respectively, and provide a discussion on the emergent property of self-reflection.
Read the ‘reflexion’ paper: https://arxiv.org/abs/2303.11366
Othello-GPT has a concept of the world (28/Mar/2023)
The headline result is that Othello-GPT learns an emergent world representation - despite never being explicitly given the state of the board, and just being tasked to predict the next move, it learns to compute the state of the board at each move…Not only does the model learn an emergent world representation, it learns a linear emergent world representation, which can be causally intervened on in a linear way! But rather than representing "this square has a black/white piece", it represents "this square has my/their piece".
Read (19,000 words): https://www.neelnanda.io/mechanistic-interpretability/othello
GPT4All-LoRa 7B (29/Mar/2023)
Nomic’s latest model, GPT4All, is a chatbot trained on ~1M GPT-3.5-Turbo [ChatGPT] generations based on LLaMa.
LoRA only fine-tunes a small subset of parameters.
LoRA (Low-Rank Adaptation of Large Language Models) was initially proposed by Microsoft Research in a paper with a focus on addressing the challenge of fine-tuning large language models. These models, such as GPT-3, with billions of parameters, are prohibitively expensive to adapt to specific tasks or domains. LoRA addresses this issue by freezing the pre-trained model weights and introducing trainable layers (rank-decomposition matrices) in each transformer block, thereby reducing the number of trainable parameters and GPU memory requirements. This approach eliminates the need to compute gradients for most model weights, resulting in faster fine-tuning times. - via HF
GPT4All is Chinchilla-aligned, but the choice of fine-tuning data was poor (i.e. they didn’t clean any of the ChatGPT error/response messages ‘As a large language model…’)
You will be hearing more about this model, though!
Read more: https://github.com/nomic-ai/gpt4all
GPT4All dataset (Mar/2023)
Explore the dataset used to train GPT4All. This is a fascinating time sink!
Play with it: https://atlas.nomic.ai/map/gpt4all_data_clean_without_p3
BloombergGPT 50B (30/Mar/2023)
Bloomberg have trained their own model, using BLOOM’s architecture and methodology. It is not quite Chinchilla aligned (only 12:1 instead of 20:1). ‘We select a model size motivated by Hoffmann et al. (2022) and train a 50 billion parameter model on 569 billion tokens from our corpus of over 700 billion tokens to produce a model that is competitive with larger models.’
The model severely underperforms even GPT-3, and certainly doesn’t come close to the ‘superintelligence’ capabilities that we are just beginning to uncover in GPT-4.
Organizations throwing heavy resources at training their own proprietary models is an interesting discussion. Using an airplane analogy, they could lease a Boeing 787 Dreamliner (GPT-4) and fine-tune it on their financial data (just like Harvey AI did with legal data). But even though it’s the ‘best’ available model, they would never really own that aircraft. It’s a permanent lease that can be revoked or remotely shut down at any time…
Instead, Bloomberg (and some other enterprise/governments) are choosing to design and build their own little Cessna aircraft. It’s slow, short range, and really doesn’t compare at all to the big models, but at least they own it completely!
Read the paper: https://arxiv.org/abs/2303.17564
LAION OpenFlamingo-9B (28/Mar/2023)
LAION continues to provide bleeding-edge advances to the whole community. This release includes:
A Python framework to train Flamingo-style LMMs (based on Lucidrains' flamingo implementation and David Hansmair's flamingo-mini repository).
A large-scale [Common Crawl/C4 with 400M images + 38B tokens] multimodal dataset with interleaved image and text sequences.
An in-context learning evaluation benchmark for vision-language tasks.
A first version of the OpenFlamingo-9B model based on LLaMA [7B], with much better models to come!
Read the announce: https://laion.ai/blog/open-flamingo/
Explore via HF: https://huggingface.co/openflamingo/OpenFlamingo-9B
Try the demo with image upload: https://7164d2142d11.ngrok.app/
Microsoft introduces TaskMatrix.ai (29/Mar/2023)
We illustrate how TaskMatrix.AI can perform tasks in the physical world by [LLMs] interacting with robots and IoT devices… All these cases have been implemented in practice and will be supported by the online system of TaskMatrix.AI, which will be released soon… TaskMatrix.AI can help users to interact with the real world by instructing robots and IoT devices. MCFM can understand the environment with camera API, and transform user instructions to action APIs provided by robots and IoT devices. TaskMatrix.AI can facilitate the handling of physical work with the assistance of robots and the construction of smart homes by connecting IoT devices…
This made an appearance on my AGI countdown, but hasn’t moved the needle yet, as it mainly consolidates existing technologies.
Read the paper: https://arxiv.org/abs/2303.16434
Interview with Prof Geoffrey Hinton (25/Mar/2023)
(Edit: Thanks to The Memo reader David for pointing out that Geoffrey Hinton is unable to sit due to a back injury, and stands with the interviewer during this 1-hour video.)
Timecodes:
6:15: Big data + big compute
21:39: ChatGPT
33:12: Predictions in humans and AI
33:52: Transformer: translating ‘the trophy would not fit in the suitcase because it was too big’
40:37: Sentience
Watch the video:
Toys to Play With
Exclusive: GPT in 2 minutes (Apr/2023)
This video is unreleased, and exclusive to paid readers of The Memo for the next little while. This version has a voiceover.
Cat-GPT (Mar/2023)
Okay, it just copies the ChatGPT user interface, and doesn’t hit any GPT-like model (or any model at all), but the results are hilarious!
Try it (free, no login): https://cat-gpt.com/
Hey GPT for iPhone: Replace Siri with ChatGPT (28/Mar/2023)
Mckay Wrigley: ‘I created an iOS shortcut that allows you to replace Siri with ChatGPT.’
Download: https://www.icloud.com/shortcuts/8bcc2979d6e34a3d90a48c3f77f96d68
15 useful ChatGPT prompts for startups (2/Apr/2023)
My free workbook, The ChatGPT Prompt Book, now has well over a million views. As a business-specific alternative, try out some of these prompts for your own startup.
Try it: https://www.fastcompany.com/90875084/chatgpt-prompts-for-startups
Perplexity on iOS (Mar/2023)
I’m a huge fan of Perplexity on web, and it is one of my favorite applications to experience the power of post-2020 AI. Now it’s available on iPhone.
The big differentiators are:
Cited sources.
Web access.
Free.
Combines various models.
The company was the first to commercialize the WebGPT concept, and the list of investors is the ‘who’s who’ of Silicon Valley:
…Nat Friedman (Former CEO of GitHub), Jeff Dean (SVP, Google AI) and Bob Muglia (Former President of Microsoft), as well as new investors Susan Wojcicki (Former CEO of Youtube), Paul Buchheit (Creator of Gmail), Soleio (Designer of Messenger, Dropbox), and Databricks Ventures.
We are also grateful to our angel investors who participated in our $3.1 million seed round in September 2022, led by Elad Gil and Nat Friedman, with participation from Pieter Abbeel (UC Berkeley, AIX Ventures), Yann LeCun (Chief Scientist, Meta), Andrej Karpathy (Founding Member, OpenAI), Ashish Vaswani (Lead Inventor of Transformer), Amjad Masad (CEO, Replit), Clem Delangue (CEO, HuggingFace), Oriol Vinyals (VP of Research, DeepMind) and others.
Read the release: https://www.perplexity.ai/blog
Download: https://apps.apple.com/us/app/perplexity-ask-anything/id1668000334
Next
I was in Sydney recently to give a private keynote/workshop that was part of a very slick and well-produced multi-camera live and streamed paid event (I believe tickets were $4,999). I’m hoping to release the recording of that to you shortly. Thanks again for your trust and investment in The Memo!
All my very best,
Alan
LifeArchitect.ai
Archives | Unsubscribe new account | Unsubscribe old account (before Aug/2022)