The Memo - 13/Nov/2023
Amazon Olympus 2T, Cato’s smart approach to AI policy, xAI Grok-1, and much more!
FOR IMMEDIATE RELEASE: 13/Nov/2023
Bill Gates (9/Nov/2023):
’…[AI agents will] force humans to face profound questions about purpose. Imagine that agents become so good that everyone can have
a high quality of life without working nearly as much.
In a future like that, what would people do with their time?
Would anyone still want to get an education when an agent has all the answers? Can you have a safe and thriving society when most people have a lot of free time on their hands?’
Welcome back to The Memo.
You’re joining full subscribers from the big government departments and agencies, F500s, people that can see the power of the AI revolution, and more…
This is another long edition, with my highlights of OpenAI’s dev day, brief analysis of new models, Microsoft’s 11 prompt phrases to improve GPT outputs, further insights into the new-new NVIDIA GPUs for China, and more. In the Toys to play with section, we look at big prompts, camera hardware using GPT, new games by GPT-4, and much more…
The next roundtable will be Sat 9/Dec in US and Sun 10/Dec in Aus. Details at the end of this edition.
The BIG Stuff
Amazon Olympus 2T in training (8/Nov/2023)
In November 2023, Amazon is investing millions in training an ambitious large language model (LLM) codenamed ‘Olympus’ that could rival top models from OpenAI and Alphabet, according to anonymous sources. The model reportedly has 2 trillion parameters, potentially making it one of the largest models being trained.
Well, they’re all piling in now. Besides OpenAI, Google DeepMind, Anthropic, Meta AI, and Baidu, we’ve now got xAI, Apple, Amazon, and hundreds of other labs waking up to the power of large language models. Where were they in 2020 after GPT-3? They could have had a three-year head start!
By the way, these model family names are getting much better. Forget word-salad or acronyms. First we had muppets (BigBird), AI21 had geologic periods (Jurassic), DeepMind gave us animals (Chinchilla), OpenAI is trying out deserts (Gobi), and now… Olympus. In Greek mythology, Olympus was the home of the gods who lived in palaces of marble and gold. Mount Olympus was ruled by Zeus and his sister-wife, Hera (Greek mythology).
See my models viz: https://lifearchitect.ai/models/
Obama talks about UBI (3/Nov/2023)
It’s unusual to have a leader speaking about universal basic income, the post-scarcity (post-abundance) necessity where everything is done for us by AI and robotics, so money is distributed evenly.
At the Obama Foundation Democracy Forum in Chicago, Illinois, on 3/Nov/2023, President Obama spoke about universal basic income (UBI) post-AI:
‘With automation replacing so many blue-collar jobs, and pretty soon white-collar jobs, we may need to consider bigger changes. We should start talking about that now. Things like a shorter work week, or a universal basic income, or guaranteed income…
‘Some of these ideas are being tested as we speak. Right here in Chicago, there's a pilot program called 'Chicago Promise' that's been giving thousands of low to moderate income families $500 in cash every month to help them meet basic needs. And it's modeled after a similar program in Stockton, California, and it's showing early promise.
‘Because the thesis is, you improve financial stability, you improve the well-being of low-income families. They actually can fix a busted car, they don't have electricity turned off, their lives are more stable…
‘And that's the thing about giving people a safety net and helping them achieve their basic needs. Not only does it relieve hardship, not only does it strengthen our democracy, it can also empower people to be more productive and raise their ambitions, and set an example for their children. And that benefits all of us, and that benefits the economy, and that benefits business.’
Transcript thanks to https://youtubetranscript.com/?v=Ul1Ye2XXMfM and GPT-4.
Watch the video (link):
New viz: Six thinking hats (6/Nov/2023)
You’ll recognize this parallel thinking system by Dr Edward de Bono from the late 1980s. I’ve employed it here to help visualize different perspectives on artificial intelligence.
Grab it: https://lifearchitect.ai/hats/
Sidenote: Even famously-cautious philosopher Nick Bostrom agrees with me: AI panic and negativity is like a big wrecking ball destroying the future (11/Nov/2023).
Many leaders are a lost cause when it comes to AI (30/Oct/2023)
‘Alan, this is not news, we’ve known it for a long time.’ Well, maybe I expected more…
A lot of people (labs, media) have been asking for my opinion on the new executive order on AI issued by the US under President Joe Biden. The reality is that I don’t really have an opinion on the content.
The AI order is 3.5 years too late, the focus is nearly entirely cautious/negative, and the underlying aim is a waste of time. Further, America’s past—and the pendulum theory (Brookings, 2020)—comes back to haunt it (‘advancement of equity and civil rights’?). I look forward to the next few months and years as AI helps us create and shape law, rather than relying on committees of very-human humans. (See the Policy section of this edition for a bird’s-eye view on the order.)
The below insight into the leader of the free world was published in TIME magazine, by editors Josh Boak and Matt O’Brien for the Associated Press. Compare and contrast with the former President’s remarks above.
The issue of AI was seemingly inescapable for Biden. At Camp David one weekend [in the second half of 2023], he relaxed by watching the Tom Cruise film “Mission: Impossible — Dead Reckoning Part One.” The film's villain is a sentient and rogue AI known as “the Entity” that sinks a submarine and kills its crew in the movie's opening minutes.
“If he hadn’t already been concerned about what could go wrong with AI before that movie, he saw plenty more to worry about,” said Reed, who watched the film with the president.
Sidenote: DALL-E 3 was happy to give this prompt a go. (Apologies for the quick glance into politics; don’t expect it to happen again!)
The Interesting Stuff
xAI announces Grok-1 (4/Nov/2023)
Sidenote: The name means ‘to understand deeply and completely’. Oh, and Elon actually wants it pronounced ‘groak’ or ‘grook’ (4/Nov/2023).
Parameters: 33B for Grok-0 (confirmed), between 33B and 70B for Grok-1 (estimated).
Dataset: The Pile (~300B tokens) for several epochs, plus Tweets. Note that Twitter boasts 12TB of new text data per day, that’s around 3.6T tokens per day. If you want big numbers, here they are: each year, Twitter provides a text dataset of 4PB which is 1,314T or 1.3Qa tokens. Yes, 1.3 quadrillion tokens available per year.
This is definitely leveraged for live retrieval using retrieval augmented generation (RAG).
Besides its popularity due to being built-in to a large platform (Twitter still has 200M daily users), the model is not particularly noteworthy. The interface is nice (with a tree view of conversations), but the model is small.
Read the launch by xAI (archive.org).
Read more about The Pile in my 2022 paper What’s in my AI?.
Complimentary copy of the ChatGPT for Dummies book (Nov/2023)