The Memo - 10/Dec/2022
Watermarking text in GPT models, the new Google LaMDA as Character.ai, DeepMind Dramatron playground, and much more!
FOR IMMEDIATE RELEASE: 10/Dec/2022
Welcome back to The Memo.
While The Memo is supposed to be sent out in monthly editions, there is a lot happening right now, so it has been a bit more frequent!
In the ‘toys to play with’ section at the end of this edition, we look at a new ChatGPT with web access (free), the first user interface for Stable Diffusion 2.1 (free), DeepMind’s Dramatron playground released today (using GPT-3 instead of Chinchilla), and the new Leta archives!
The BIG Stuff
OpenAI building in a watermarking layer to text generation (28/Nov/2022)
Mega-engineer and former gifted child Scott Aaronson at OpenAI has been working on adding text watermarking to GPT outputs. In my Jan/2022 report to the UN, I made a recommendation that this be put in place for all models:
Recommendation 3. Watermarking. Ensure that AI-generated content is clearly marked (or watermarked) in the early stages of AI development. This recommendation may become less relevant as AI is integrated with humanity more completely in the next few years.
OpenAI’s text watermarking prototype already works. In the plainest English, after the GPT model has generated a response, there will be a layer on top that shifts words/tokens very slightly and consistently. This is the output that will be given to the user (via API or Playground). This output can be measured and checked later by OpenAI/govt/potentially others. Scott says:
My main project so far has been a tool for statistically watermarking the outputs of a text model like GPT. Basically, whenever GPT generates some long text, we want there to be an otherwise unnoticeable secret signal in its choices of words, which you can use to prove later that, yes, this came from GPT.
…instead of selecting the next token randomly, the idea will be to select it pseudorandomly, using a cryptographic pseudorandom function, whose key is known only to OpenAI.
…we actually have a working prototype of the watermarking scheme, built by OpenAI engineer Hendrik Kirchner. It seems to work pretty well—empirically, a few hundred tokens seem to be enough to get a reasonable signal that yes, this text came from GPT.
Scott’s entire post is worth reading, and he links out to one of my papers. Warning: the post is over 10,000 words, plus more than 100 insightful comments.
Anthropic estimates AI model training costs will increase by 100x (8/Dec/2022)
In a recent live presentation, Dario Amodei and Anthropic (ex-OpenAI staff) commented on the large language model training space, including scaling. Even as we become much more efficient in training, Anthropic are estimating that organizations will continue to increase spending on compute, by a factor of 100x. (‘Cost on biggest model today [Dec/2022] is $10M. Expect $100M in a few years [~2026]. $1B by 2030.’)
via Rhys Lindmark.
The Interesting Stuff
OpenAI CEO talks about the Singularity (19/Nov/2022)
21 minutes worth watching! Sam says:
Imagine that all of the scientific progress since the beginning of the Enlightenment until now could happen in one year…
I think we're really just gonna have to think about how we share wealth in a very different way than we have in the past. Like the fundamental forces of capitalism and what make that work, I think break down a little bit… some version of like a basic income or basic wealth redistribution or some sort of version like that. We're trying to study that. I think it's collectively under-explored.
Watch the video:
I’ve previously defined the Singularity in this video.
The LifeArchitect.ai Who Moved My Cheese? AI Awards! (Dec/2022)
Look, I didn’t really want to do this. But the sheer lunacy of some of the noisiest critics deserves to be memorialized, much like the famous Darwin Awards.
Inspired by Spencer Johnson’s 1998 business classic, Who Moved My Cheese? (wiki link), these awards recognize those who are most change-resistant, their outrageous quotes and claims, and… that’s about it!
Take a look: https://lifearchitect.ai/cheese/
Character.ai provides some detail (5/Dec/2022)
Prompted by the noisy release of ChatGPT, the team have finally spoken up, after months of silence. Character.ai now generates 1 billion words per day (~692,000wpm), which is only ~22% of what GPT-3 was generating a year ago.
Still, that’s pretty good for a new project! Expect to see more of these guys in 2023. Noam Shazeer says:
As an early engineer at Google, I’ve built AI systems used by billions of people, from Google's Spell Checker ("did you mean?") to the first targeting system for Adsense. I also co-invented Transformers, scaled them to supercomputers for the first time, and pioneered large-scale pretraining, all of which are foundational to recent AI progress…
Two months after launching in September, our beta generates 1 billion words per day. We've been particularly impressed by the creativity of our community, who have created more than 350,0000 unique Characters, with applications spanning searching for information, coaching, education, entertainment and more.
Try it yourself: https://beta.character.ai/
Also, try interacting with ‘The Psychologist’.
OpenAI updates Whisper on GitHub (6/Dec/2022)
Just a small update to the large model (view). This improves transcription of languages other than English.
OpenAI Codespace rumor (8/Dec/2022)
There is a secret space inside ChatGPT called ‘Codespace’, only available to OpenAI staff at the moment.