FOR IMMEDIATE RELEASE: 7/Jul/2022
Welcome back to The Memo.
All the way back in 2019, OpenAI Co-founder and Chief Scientist Ilya Sutskever said that the GPT model outputs are 'like alchemy'.
alchemy /ˈalkɪmi/
noun. a type of chemistry, especially in the Middle Ages, that dealt with trying to find a way to change ordinary metals into gold and with trying to find a medicine that would cure any disease.
literary
a process that is so effective that it seems like magic.
The AI gold rush in 2022 has been and continues to be incredible...
The BIG Stuff
BLOOM 176B finished training (6/Jul/2022)
BLOOM is an open-source model created by 1,000 researchers in 60 countries, and aims to replicate GPT-3 using a more equal distribution of many languages. Training started on 11/Mar/2022, and ended on 6/Jul/2022. The model was trained on 384x A100 GPUs on the Jean Zay public supercomputer, cost $7M in compute, and 'speaks' 46 languages + even more programming languages. The final version of the model is now available, but we are still waiting on a proper playground and the paper to be released.
My latest video on the BLOOM model from today (Part 1):
My viz of the model languages: https://lifearchitect.ai/models/#bloom
Nature article: https://www.nature.com/articles/d41586-022-01705-z
Official notebook: https://bigscience.notion.site/BLOOM-BigScience-176B-Model-ad073ca07cdf479398d5f95d88e218c4
Demo of the full 176B model:
1. Go to: https://huggingface.co/bigscience/bloom-6b3?text=we+are+the
2. On the right side, enter your prompt for completion (e.g. We are the)
3. Click Compute. (The wait time should be instant for the model to load, and maybe 5-10sec for inference).
Yandex in Russia releases YaLM 100B (23/Jun/2022)
The 100B model was released by the Russian Google-equivalent, Yandex. It is about 50/50 English and Russian. YaLM is a great addition to the open-source model world, alongside other large models like GPT-NeoX-20B and BLOOM 176B (above). A hosted version of the model should be available soon.
Announcement: https://medium.com/yandex/yandex-publishes-yalm-100b-its-the-largest-gpt-like-neural-network-in-open-source-d1df53d0e9a6
Google Parti Text-to-image (Jun/2022)
Google Parti is similar to the original DALL-E, using an autoregressive process rather than the diffusion process of Google Imagen and OpenAI DALL-E 2. I think some of the images are 'state of the art'.
Blog: https://parti.research.google/
My video:
(Shout out to the Google Imagen and Parti teams for their kind words about these videos... “Thanks Alan for featuring both Parti and Imagen text-to-image work from Google! Nice video, neat and informative!” — Dr Jiahui Yu, Google Brain.)
OpenAI and other labs explore Minecraft as a physical embodiment tool (Jun/2022)
NVIDIA, Caltech, Stanford, Columbia, SJTU, UT Austin: https://minedojo.org/
OpenAI: https://openai.com/blog/vpt/
The Interesting Stuff
DeepMind Gato 2 research (1/Jul/2022)
DeepMind Co-founder and CEO Demis Hassabis was interviewed recently, and mentioned that Gato 2 is being designed and possibly trained right now...
"Gato predicts potentially any action or any token, and it's just the beginning really, it's our most general agent... that itself can be scaled up massively, more than we've done so far, obviously we're in the middle of doing that."
For more info, see my mid-year report video on SayCan and Gato. It is my belief that once we have a physically-embodied agent (robot) with a big Transformer-based AI ('big' means 1 Trillion+ parameters), we will be in the world of iRobot (but with much more friendliness!). With DeepMind, Google, and possibly Tesla and OpenAI working on such a combination, this world is just months away...
InstructGPT poetry (21/Jun/2022)
The New Yorker featured an article about the latest GPT-3 model writing poetry:
https://www.newyorker.com/culture/culture-desk/the-new-poem-making-machinery
Countdown to Optimus/Teslabot (23/Jun/2022)
Robert Scoble is being noisy about the possibility of Tesla revealing their rumoured Optimus robot in Sep/2022. You may recall my video on DeepMind Gato, where we mention the bot, and explore the groundbreaking integration of AI Transformer models and physical embodiment...
Mojo's contact lens (29/Jun/2022)
This contact lens is similar to having a heads-up display. When we finally combine Transformer-based AI into this, we will have reached 'integrated ai,' or what some have called 'hybrid humans'.
https://newatlas.com/wearables/mojo-vision-ar-contact-lens/
Toys to Play With
DALL-E 2 Faces (30/Jun/2022)
Exclusively for readers of The Memo, here's a 100-page PDF of faces generated by DALL-E 2 in the style of different famous photographers like Annie Leibovitz. The original prompt and generation is by Michael Green, and I've formatted the document to show all images at full resolution (1024x1024).
This might be the most confronting view of DALL-E 2 and its possibilities that I've ever seen: 'real' humans, created by AI in a second or two.