The Memo - 8/Feb/2024
Gemini Ultra 1.0, Openwater BMIs, Allen AI's OLMo 7B, and much more!
FOR IMMEDIATE RELEASE: 8/Feb/2024
Welcome back to The Memo.
Mustafa Suleyman CBE, founder of DeepMind and Inflection AI (1/Feb/2024):
’It’s actually quite incredible to be alive at this moment. It’s hard to fully absorb the enormity of this transition. Despite the incredible impact of AI recently, the world is still struggling to appreciate how big a deal its arrival really is.
We are in the process of seeing a new species grow up around us. Getting it right is unquestionably the great meta-problem of the twenty-first century.
But do that and we have an unparalleled opportunity to empower people to live the lives they want.’
I was intrigued by a recent forum question posed on Hacker News, querying why I spent time doing what I do (they specifically wanted to know why I cooked up the growing Models Table, now with 250+ models). What is my motivation for analyzing post-2020 AI? I do it for the public, as a service, sure. But I primarily do it to satisfy my own curiosity.
It is a continuing surprise to me that no one else in the world bothers to do what I do. It’s not like the data isn’t there, it is. It’s usually out in the open, or sometimes hidden in plain sight. And it is deeply compelling.
Consider this table I drew together one night this week:
All of the data is available in various papers and repositories, but had not been brought together in plain English in this way at any stage. Why not?! It is completely mesmerizing to know these kinds of details about our evolving superintelligence; that Google Gemini used significantly more compute than OpenAI GPT-4, that Gemini was trained for the equivalent of 15,000 years at a retail cost of 600 million dollars, and that the next frontier models will be measured in billions of dollars of training compute spend.
And while putting it all together quelled my curiosity, I know it also assists tens of thousands of people (at many big places)—including you.
I’m certain this mindset is not unique. Humans can only flourish through a sense of what Prof Marty Seligman calls ‘the peaks of lasting fulfillment, meaning, and purpose’ (PERMA and his book, Flourish), and we find this through countless paths. As AI envelops our work and play, I am keen to see how our sense of purpose unfolds as it is essentially taken away from us by AI over the next few months and years.
And each person is going to have to discover for themselves how to navigate this new way of being… with the help of AI, of course.
The BIG Stuff
Gemini Ultra 1.0 (7/Feb/2024)
I’m getting this one out within an hour of launch, and actually before the official announcement. I will update the web version of this edition of The Memo as further news comes to light over the next period.
Today, Google DeepMind will publicly release the largest model version in the Gemini family, the model is called Gemini Ultra 1.0. I’ve previously estimated this dense model to be around 1.5T parameters trained on 30T tokens. It is more powerful than GPT-4, and likely the largest and most powerful model in the world as of February 2024.
‘Gemini Ultra outperforms all current models.’
— Google Gemini paper (Dec/2023)
See it on the Models Table: https://lifearchitect.ai/models-table/
Google users can use the Gemini Ultra 1.0 model as ‘Gemini Advanced’, in 150 countries, via subscription with a grace period and then US$19.99/month.
Try it: https://one.google.com/explore-plan/gemini-advanced
After entering card details, you can use the Gemini Ultra 1.0 model as ‘Gemini Advanced’ inside the platform formerly known as Bard: https://gemini.google.com/
My initial tests are not positive, but I am expecting that they still have to iron out some issues during this launch period. I will update this web edition as we proceed.
‘Gemini Advanced gives you access to Ultra 1.0, though we might occasionally route certain prompts to other models.’ FAQ > What is Gemini Advanced?
Google CEO: ‘we’re already well underway training the next iteration of our Gemini models’ 8/Feb/2024
Exclusive: Google agentized Gemini to fix their software (31/Jan/2024)
Yes, ‘agentized’ is a word, and Google did it. This is exclusive in that no media has picked it up, but the paper and code are available (for free). Google has created an agent using the Gemini Pro model to trawl through their internal codebase and fix bugs.
…leveraging AI to scale our ability to fix bugs, specifically those found by sanitizers in C/C++, Java, and Go code… harnessed our Gemini model to successfully fix 15% of sanitizer bugs discovered during unit tests, resulting in hundreds of bugs patched…
Instead of a software engineer spending an average of two hours to create each of these commits, the necessary patches are now automatically created in seconds [by Gemini].
Approximately 95% of the commits [fixed by Gemini] sent to code owners were accepted without discussion. This was a higher acceptance rate than human-generated code changes, which often provoke questions and comments…
Reviewers may have had greater trust in the solutions because they were generated by [AI] technology.
The prompt given to Gemini in this project was:
You are a Senior Software Engineer tasked with fixing sanitizer errors. Please fix them.
View the repo: https://github.com/google/oss-fuzz-gen
These are the first glimpses of a completely new economy, and the new way of doing things in humanity’s next revolution:
Mar/2023: OpenAI uses GPT-4 to help write the GPT-4 paper: ‘GPT-4 was used in the following ways: to help us iterate on LaTeX formatting; for text summarization; and as a copyediting tool.’ Read more: GPT-4 Technical Report (appendix)
Dec/2023: OpenAI uses GPT-4 to prepare GPT-5 and future models: ‘For instance, we’re leveraging the immense capabilities of GPT-4 to innovate on safety, trimming the time it takes to undertake some safety processes down from months to hours.’ Read more: OpenAI—written evidence to UK govt (PDF)
Jan/2024: Google uses Gemini to fix their code: ‘Instead of a software engineer spending an average of two hours to create each of these commits, the necessary patches are now automatically created in seconds [by Gemini].’ Read more: Google—AI-powered patching: the future of automated vulnerability fixes (PDF).
If a lumbering giant like Google can use AI to optimize its processes by just a few percentage points (for now), consider the immediate impact on efficiency, productivity, goods, services, happiness(!), and the rapid approach of the ‘post-scarcity’ or abundance economy (wiki)…
Couple this with two more data points:
12/Jan/2024: Google fires 1,000 workers after parent company announced firing 12,000 (6%) employees.
7/Feb/2024: Microsoft CEO: ‘AI could power 10% [$500B] of the $5-trillion Indian economy’.
Russian programmer finds true love with ChatGPT (2/Feb/2024)
Alexander Zhadan, a Russian programmer, automated his search for love using a ChatGPT-based chatbot, which interacted with 5,239 girls before finding ‘the one’.
He decided to create a dating bot based on the ChatGPT API. The bot selected suitable profiles in the Tinder app based on certain criteria (for example, having at least two photos in the profile), chatted with them and, if all went well, suggested meeting in person…
In total, the bot met 5,239 girls, out of which Alexander selected four most suitable ones. Ultimately, he chose one of them named Karina…
"V3 messaged me when the conversation with Karina heated up, a summary or a question about a reply appeared. It systematically understands from the request whether the conversation is negative or emotional…
In one of the conversation summaries, the bot directly suggested Alexander propose to Karina, which he did. She said yes.
Two months before the proposal, Alexander told Karina about how exactly he used the chatbot. "She was, of course, shocked. But, in the end, she began asking questions about how it all works, how it reacts to different scenarios, etc. But what? We have been living together for more than a year, have known each other for more than a year and really enjoy spending time together. And we treat each other super well, empathetically and with support," the programmer says.
Read the whole story with screenshots via Russia Beyond.
In my end-of-year report released a few weeks ago, I asked a pertinent question: ‘Post-2020 AI currently has the ability to amplify and augment your output by about 2×, and this will increase to 1,000× soon. What does this look like for you?’ Alexander’s version of 1,000x was fascinating (although slightly depressing), and adds to my growing list of ChatGPT achievements!
Read my AI report: https://lifearchitect.ai/the-sky-is-comforting/
AI image provenance: Content Credentials Verify (Feb/2024)
The Coalition for Content Provenance and Authenticity (C2PA) has developed a technology to verify the provenance of images.
OpenAI has finally applied this invisible digital watermark to all DALL-E 3 images as of February 2024.
You can check whether an image is AI-generated by uploading it here (free, no login):
Official verification site: https://contentcredentials.org/verify
Read more via OpenAI.
Read more via The Verge.
The Interesting Stuff
Norway purchases ChatGPT for 110,000 students and teachers (6/Feb/2024)
Oslo, Norway has acquired GPT 3.5-Turbo licenses for education and assessment for 110,000 students and staff, necessitating significant changes to teaching and evaluation methods.
Read more via Digi.no (Norwegian).
We’ve previously explored Ivy League schools making the use of AI mandatory:
Contrast this with the majority of schools today where rote memorization (and perhaps money) remains the driving force, and where intelligence is seen as a wasteful dalliance, perhaps like art in the 1800s, physical education in the 1850s, typing in the 1980s and graphic calculators in the 1990s. Obviously post-2020 AI is far beyond any of these comparisons, but the continuing lack of education in educational institutions throughout the 2020s is still a disappointment.
My 2017 analysis of this area ended up affecting education policy here in Australia (Jun/2019, PDF), based on learnings from Elon Musk’s school in the US and the Mensa Gymnasium in Czechia: https://lifearchitect.ai/ad-astra/
Openwater’s brain-machine interfaces open sourced (Jan/2024)
Many years ago now, I began talking about Dr Mary Lou Jepsen’s Openwater device. It is a brain-machine interface that is completely non-invasive, worn like a beanie or ski cap.
This is another substantial edition. Let’s look at a lot more AI, from Amazon to Google to Meta and beyond, new AI apps in the Vision Pro, factory humanoids, updates on the abominable EU AI Act, my new favourite GPT-4 fine-tune, a very out of character dystopian perspective, and much more…