The Memo - 18/Oct/2023
Baidu ERNIE 4.0, NVIDIA B100 and X100, UBI in North America, and much more!
FOR IMMEDIATE RELEASE: 18/Oct/2023
Marc Andreessen (16/Oct/2023):
Artificial Intelligence is our alchemy, our Philosopher’s Stone – we are literally making sand think… Any deceleration of AI will cost lives. Deaths that were preventable by the AI that was prevented from existing is a form of murder. We believe in Augmented Intelligence just as much as we believe in Artificial Intelligence. Intelligent machines augment intelligent humans, driving a geometric expansion of what humans can do.
Welcome back to The Memo.
You’re joining paid subscribers from Microsoft, MIT, Moodle, Monash, Mercado Libre, MYOB, Mastercard, and more…
This is another long edition. In the Toys to play with section, we look at a new collection of OpenAI’s hidden system prompts used every time you interact with their platforms, South Park’s AI episodes, Meta AI’s chatbots now in WhatsApp (US), and much more.
The next roundtable will be:
Life Architect - The Memo - Roundtable #4
Follows the Chatham House Rule (no recording, no outside discussion)
Saturday 4/Nov/2023 at 5PM Los Angeles
Saturday 4/Nov/2023 at 8PM New York
Sunday 5/Nov/2023 at 8AM Perth (primary/reference time zone)
or check your timezone via Google.
Details at the end of this edition.
Reggie Watts was dropping my name recently in an interview with some former UFC commentator (link).
I think he is referring to the entire body of work at LifeArchitect.ai (that’s around 100+ papers, 350+ videos, and 100+ editions of The Memo), but if you’re looking for policy references you’ll find them in every edition for the last year or so. Three big editions mentioning AI and government are:
The Memo - 7/Mar/2023: Romanian PM using a ChatGPT alternative behind a mirror.
The Memo - 30/Apr/2023: US/EU/Japan AI acts, Palantir AIP.
The Memo - 17/Aug/2023: Where will AGI begin?, US + China analysis.
The BIG Stuff
OpenAI: AGI is now a ‘core value’ (12/Oct/2023)
GPT creator OpenAI has subtly revised its ‘Core values’ on its website, placing a more significant emphasis on the development of AGI — artificial general intelligence.
Artificial general intelligence is the equivalent of a median human that you could hire as a co-worker.
— OpenAI CEO, NYMag (25/Sep/2023)
The organization's revamped values now include ‘AGI focus,’ stating ‘Anything that doesn't help with that is out of scope.’
Since the 2020 release of GPT-3 (watch Leta AI), I have been telling friends and colleagues as clearly as I can: ‘Drop everything, and focus on AI!’ Governments, companies, individuals… sit up and take notice. Anything outside of AI is just a distraction. (Of course, living life and being human is still important, but investing time in things like committees and land wars and climate change and voices in parliament and electric vehicle upgrades is a waste of time, when superintelligence will easily give us a 1,000x advantage in solving all of these and more.)
See the core values: https://openai.com/careers
Read more (paywall): https://www.semafor.com/article/10/12/2023/openai-quietly-changed-its-core-values
See my conservative countdown to AGI (at 55% in Oct/2023): https://lifearchitect.ai/agi/
OpenAI: Even more GPT-5 rumors (18/Oct/2023)
Users Jimmy Apples and FeltSteam are back. Here’s what I’ve gleaned:
The new OpenAI internal project names based on deserts may reflect the fact that these models are ‘sparse,’ like a desert.
Sahara = GPT-3.5 (ChatGPT).
Gobi = GPT-5-related.
Arrakis = GPT-5-related (maybe failed/suspended due to low model quality).
Screenshots of rumors via Twitter.
Read my GPT-5 summary: https://lifearchitect.ai/gpt-5/
NVIDIA Blackwell B100 GPUs launches in Q2 2024 due to rise in AI demand (15/Oct/2023)
NVIDIA has reportedly moved the launch of its next-gen Blackwell B100 GPUs up from Q4 to Q2 2024 following a huge surge in AI demand.
The market predicts that B100 will be a more powerful AI game changer than H100, Nvidia's current highest-spec GPU. This product is mainly used in AI cloud and supercomputing. NVIDIA accounts for more than 90% of the AI GPU market share.
These expanded timelines are big news for NVIDIA and the AI labs (and humanity!). Here is a rapid successor to their H100 already, and not just for next year (2024) with the B100, but the following year (2025) with the X100. This is not-so-good news for Intel and AMD, who are still playing catchup to the older NVIDIA A100.
Both the B100 and X100 GPUs will run the world. Count on it.
Read source: https://wccftech.com/nvidia-blackwell-b100-gpus-sk-hynix-hbm3e-memory-launches-q2-2024-rise-in-ai/
China's Baidu unveils new ERNIE AI version to rival GPT-4 (17/Oct/2023)
ERNIE stands for ‘Enhanced Representation from kNowledge IntEgration’. The Chinese name is 文心一言, or Wenxin Yiyan.
Chinese tech giant Baidu has introduced ERNIE 4.0, its newest generative AI model, which it claims is on par with OpenAI's GPT-4.
China now has at least 130 large language models (LLMs), representing 40% of the global total and behind only the United States' 50%, data from brokerage CLSA showed.
We covered 103 new Chinese LLMs in The Memo edition 27/Jul/2023.
Once again, another AI lab has chosen to guard its model architecture. It is assumed that the model is a sparse mixture-of-experts (MoE). While ERNIE 3.0 was only 260B parameters, ERNIE 4.0 is said to have more than 1 trillion parameters, on par with GPT-4 1.76T.
See Baidu’s launch video with English voiceover.
Read my brief ERNIE summary: https://lifearchitect.ai/ernie/
And see Baidu on the AGI viz: https://lifearchitect.ai/agi#where
The Interesting Stuff
AGI achieved internally: A re-written story (Life 3.0 annotated, with replacements for the Omegas and Prometheus) (16/Oct/2023)
I had a bit of fun working with GPT-4 to convert Max Tegmark’s famous story about achieving AGI from his generic names of ‘Omega’ and ‘Prometheus’ to OpenAI/Google and GPT-5/Gemini 2. It’s a 20-minute read, and I’m sure you’ll recognize some of it as happening already…
Over a timescale of months, the business empire controlled by the OpenAI team started gaining a foothold in ever more areas of the world economy, thanks to superhuman planning by GPT-5. By carefully analyzing the world’s data, it had already during its first week presented the OpenAI team with a detailed step-by-step growth plan, and it kept improving and refining this plan as its data and computer resources grew. Although GPT-5 was far from omniscient, its capabilities were now so far beyond human that the OpenAI team viewed it as the perfect oracle, dutifully providing brilliant answers and advice in response to all their questions.
Read the story (6,000 words, 20mins): https://lifearchitect.ai/agi-achieved-internally/
Exclusive: OpenAI may release text-embedding-ada-003 at dev day on 6/Nov/2023 (5/Oct/2023)
A recent GitHub pull request (5/Oct/2023) suggests that there is a new embeddings model, text-embedding-ada-003. It may be that OpenAI releases this during the upcoming dev day on 6/Nov/2023. ‘Ada’ is the smallest publicly-available model in the GPT-3 family, at only 350M parameters. The text-embedding-ada-002 model has been the standard consolidated embedding model since the beginning of 2023.
I am not expecting any major releases at OpenAI dev day, as the CEO has said that there will not be any: ‘on november 6, we’ll have some great stuff to show developers! (no gpt-5 or 4.5 or anything like that, calm down, but still i think people will be very happy…)‘ (7/Sep/2023).
Read the PR: https://github.com/microsoft/semantic-memory/pull/78
OpenAI dev day: https://openai.com/blog/announcing-openai-devday
See my older view (Mar/2023) of the GPT-3 family: https://lifearchitect.ai/gpt-3/
Exclusive: OpenAI partners with Dropbox, but your data is ‘never’ used to train models (Oct/2023)
I think some people (including myself, initially) were concerned about this partnership, and confused about the implications. They are now much clearer in the Dropbox T&Cs:
What [Dropbox] information is shared with third-party partners?
Your files within Dropbox are sent to a third-party AI only when you chose to interact with AI powered features. For example, when you ask a question about a file. At this time, we’re [Dropbox] partnered with one third-party AI partner, OpenAI. Open AI is an artificial intelligence research organization that develops cutting-edge language models and advanced AI technologies. Your data is never used to train their [OpenAI’s] internal models, and is deleted from OpenAI’s servers within 30 days.
Read the Dropbox T&C section on AI.
2023 State of AI report released (12/Oct/2023)
Every year since 2021, there are three big AI reports released, each with a different objective and focus:
The sky is… by LifeArchitect.ai (recent release for mid-year Jun/2023).
Artificial Intelligence Index Report by Stanford (recent release Apr/2023).
State of AI by Air Street Capital (recent release Oct/2023).
The State of AI report is early (12/Oct/2023), and a useful read for those involved in setting policy or making strategic decisions. I enjoyed this funding summary slide on page 119:
Read the 2023 State of AI report.
I expect to have my end-of-year AI report, ‘The sky is…’ released in about 7-8 weeks from now, in December 2023. As usual, paid subscribers of The Memo receive the report first.
New LLM leaderboard using a set of 60 prompts (Sep/2023)
The HuggingFace leaderboard is practically useless given its reliance on TruthfulQA (often negatively correlated with response quality). This alternative seems much more pragmatic.
Asking 60+ LLMs a set of 20 questions
This is a neat little project testing different LLMs with prompts that test for different capabilities like basic reasoning and instruction following.
I have a similar set of tests that I perform on LLMs but my tests are more robust than this list and more catered to real-world use cases. I still like the effort here as I find it useful to do simple tests like this these days, especially with all the LLMs being released.
See the benchmarks: https://benchmarks.llmonitor.com/
Conversation by former Meta AI engineer.
You can also use the quick parameters + tokens ‘AlScore’ on my Models Table.
Exclusive: MNBVC: Massive Never-ending BT Vast Chinese corpus (Oct/2023)
A new dataset called the ‘Massive Never-ending BT Vast Chinese corpus’ (MNBVC) is half-way through data collection.
As of Oct/2023, the team has collected 20TB of their 40TB aim. The large-scale Chinese corpus will be used for training Chinese models.
See the GitHub repo (Chinese).
See the official site (Chinese).
Legal: Google & Microsoft will pay your legal fees for AI copyright claims (12/Oct/2023)
Google says:
…generated output indemnity means that you can use content generated with a range of our products knowing Google will indemnify you for third-party IP claims, including copyright…
This follows on from Microsoft providing the same indemnification back in Sep/2023:
…if a third party sues a commercial customer for copyright infringement for using Microsoft’s Copilots or the output they generate, we will defend the customer and pay the amount of any adverse judgments or settlements that result from the lawsuit, as long as the customer used the guardrails and content filters we have built into our products.
Read more: https://blogs.microsoft.com/on-the-issues/2023/09/07/copilot-copyright-commitment-ai-legal-concerns/
From reading X-rays to decoding classified UFO reports, [GPT-4V] shows off its vision (11/Oct/2023)
Trying to fill gaps in a string of text is basically what LLMs do. The user did the next best thing when trying to test GPT-V’s capabilities and made it guess parts of a text that he censored. ‘Nearly 100% intent accuracy.’ he reported.
Of course, it's hard to verify whether its guess at what's otherwise obscured is accurate—it’s not like we can ask the CIA how well it did peering through the black lines.
Read more: https://decrypt.co/201060/chat-gpt-vision-visual-gpt4-multimodal
Read Slashdot’s analysis and commentary.
See the tweet about GPT-4V for un-redaction. (video below)
Think before you speak: Training Language Models With Pause Tokens: Carnegie Mellon & Google (3/Oct/2023)
[We] delay extracting the model's outputs until the last pause token is seen, thereby allowing the model to process extra computation before committing to an answer. We empirically evaluate pause-training on decoder-only models of 1B and 130M parameters with causal pretraining on C4, and on downstream tasks covering reasoning, question-answering, general understanding and fact recall. Our main finding is that inference-time delays show gains when the model is both pre-trained and finetuned with delays. For the 1B model, we witness gains on 8 of 9 tasks, most prominently, a gain of 18% EM score on the QA task of SQuAD, 8% on CommonSenseQA and 1% accuracy on the reasoning task of GSM8k. Our work raises a range of conceptual and practical future research questions on making delayed next-token prediction a widely applicable new paradigm.
Read the paper: https://arxiv.org/abs/2310.02226
We explored some of GPT-4V’s vision in my recent livestream (15/Oct/2023).
Google’s AI-powered search experience can now generate images (12/Oct/2023)
Google’s Search Generative Experience (SGE) now allows users to create images from a text prompt. The tool, powered by the Imagen family of AI models, also allows users to generate written drafts directly from the search bar.
Read more: https://www.theverge.com/2023/10/12/23913337/google-ai-powered-search-sge-images-written-drafts
OpenAI’s revenue crossed $1.3 billion annualized rate, CEO tells staff (12/Oct/2023)
OpenAI, the maker of ChatGPT, is generating revenue at a pace of US$1.3 billion a year, according to CEO Sam Altman. The revenue, largely from subscriptions to its conversational chatbot, represents significant growth from last year's revenue of $28 million. Altman’s remark implies the company is generating more than $100 million per month, up 30% from this summer [US summer is Jun-Aug 2023], when the Microsoft-backed startup generated revenue at a $1 billion-a-year pace.
Read more (paywall): https://www.theinformation.com/articles/openais-revenue-crossed-1-3-billion-annualized-rate-ceo-tells-staff
Google DeepMind PaLI-3 5B (13/Oct/2023)
Google continues its long journey using the Pathways architecture. Pathways Language and Image 3 (PaLI-3) is a stripped back version of the original PaLI 17B and PaLI 55B, this time with lean 5B parameters.
This paper presents PaLI-3, a smaller, faster, and stronger vision language model (VLM) that compares favorably to similar models that are 10x larger.
Read the paper: https://arxiv.org/abs/2310.09199
Read my Aug/2022 Google Pathways report: https://lifearchitect.ai/pathways/
China gives Ehang the first industry approval for fully autonomous, passenger-carrying air taxis (13/Oct/2023)
Guangzhou-based Ehang received an airworthiness 'type certificate' from the Civil Aviation Administration of China for its fully autonomous drone, the EH216-S AAV, that carries two human passengers. This makes Ehang the first in the world to get such a certificate, which allows it to fly passenger-carrying autonomous electric vertical take-off and landing (eVTOL) aircraft in China.
Can GPT models be Financial Analysts? An Evaluation of ChatGPT and GPT-4 on mock CFA Exams (12/Oct/2023)
A Chartered Financial Analyst (CFA) requires at least 4,000 hours of relevant professional experience. The average base salary is US$103,000/year (Payscale).
GPT-4 would have a decent chance of passing the CFA Level I and Level II if prompted with FS (few-shot) and/or CoT [chain-of-thought].
Read the paper: https://arxiv.org/abs/2310.08678
The Techno-Optimist Manifesto: Marc Andreessen (16/Oct/2023)
We believe Artificial Intelligence is our alchemy, our Philosopher’s Stone – we are literally making sand think.
We believe Artificial Intelligence is best thought of as a universal problem solver. And we have a lot of problems to solve.
We believe Artificial Intelligence can save lives – if we let it. Medicine, among many other fields, is in the stone age compared to what we can achieve with joined human and machine intelligence working on new cures. There are scores of common causes of death that can be fixed with AI, from car crashes to pandemics to wartime friendly fire.
We believe any deceleration of AI will cost lives. Deaths that were preventable by the AI that was prevented from existing is a form of murder.
We believe in Augmented Intelligence just as much as we believe in Artificial Intelligence. Intelligent machines augment intelligent humans, driving a geometric expansion of what humans can do.
We believe Augmented Intelligence drives marginal productivity which drives wage growth which drives demand which drives the creation of new supply… with no upper bound.
Read it (5,000 words): https://a16z.com/the-techno-optimist-manifesto/
Policy
A universal basic income is being considered by Canada's government (16/Oct/2023)
The Canadian Senate is scrutinizing a bill that intends to establish a framework for a universal basic income (UBI) policy, promising access to a livable income for everyone over 17. The bill, if passed, would mandate provincial ministers and Indigenous governing bodies to devise a feasible UBI plan, ensuring no cuts in other social services and no compulsory participation in education, training, or the labour market.
Saudi-China collaboration raises concerns about access to AI chips (10/Oct/2023)
Saudi-Chinese collaboration in artificial intelligence has stirred fears within the Gulf kingdom’s premier academic institution that the ties could jeopardise the university’s access to US-made chips needed to power the new technology.
ForeignPolicy.com: America Can’t Stop China’s Rise (19/Sep/2023)
…on Aug. 9, the Biden administration issued an executive order prohibiting American investments in China involving “sensitive technologies and products in the semiconductors and microelectronics, quantum information technologies, and artificial intelligence sectors” which “pose a particularly acute national security threat because of their potential to significantly advance the military, intelligence, surveillance, or cyber-enabled capabilities” of China.
All these actions confirm that the American government is trying to stop China’s growth. Yet, the big question is whether America can succeed in this campaign—and the answer is probably not. Fortunately, it is not too late for the United States to reorient its China policy toward an approach that would better serve Americans—and the rest of the world.
Read more: https://archive.md/peOEV
Toys to Play With
Meta AI’s chatbots now in WhatsApp in the US (Oct/2023)
These bots are built on the company’s Llama 2 open-source large language model (LLM), and can connect to the internet via Bing to deliver up-to-date answers to your questions.
All 28 chatbots have their own unique personality, and Meta wants you to chat with different bots for different conversations…
Fifteen of these chatbots are actually based on celebrities. Meta paid these actors, chefs, athletes, and personalities to use their likeness as AI bots. While the bots are text-only for now, meaning you can’t have an actual face-to-face conversation with Tom Brady, the idea is they’ll text like the celebrity. On top of that, they’ll appear in a floating windows above the chat, “reacting” to different parts of the conversation:
Here’s the full list of Meta’s AI bots you can chat with:
Lorena (Padma Lakshmi): Travel expert
Bru (Tom Brady): Confident sports debater
Dungeon Master (Snoop Dogg): Adventurous storyteller
Tamika (Naomi Osaka): Anime fanatic
Billie (Kendall Jenner): Ride-or-die older sister
Amber (Paris Hilton): Crime-solving detective
Max (Roy Choi): Seasoned sous chef
Coco (Charli D’Amelio): Dance enthusiast
Luiz (Isreal Adesanya): MMA expert
Perry (Chris Paul): Approachable golf pro
Dylan (LaurDIY): Quirky DIYer
Victor (Dwayne Wade): Motivational triathlete
Zach (Mr. Beast): Brotherly jokester
Sally (Sam Kerr): Free-spirited friend
Angie (Raven Ross): Fitness enthusiast
There are also non-celebrity AI chatbots you can chat with as well:
Meta AI: AI Assistant
Thalia: Fantasy adventure guide
Brian: Warm-hearted grandpa
Izzy: Aspiring singer-songwriter
Scarlett: Hype woman bestie
Becca: Devoted dog mom
Alvin the Alien: Quirky alien
Bob the robot: Sarcastic robot
Lily: Creative writing partner
Carter: Practical dating coach
Jane Austen (lol): Opinionated author
Leo: Career coach
Jade: Hip-hop obsessive
Liv: Open-hearted mom
As of this article, Meta’s chatbots should be live across its apps. To start, fire up Instagram, Messenger, or WhatsApp [Alan: this seems to be US-only, and WhatsApp says it is for ‘limited countries’], then start a new chat. Rather than picking one of your contacts however, choose “AI Chat.” Tap “Continue” on the pop-up, and you’ll be greeted by a “Chat with an AI” window. Here, you can choose from the entire cast of AI characters to chat with, including Meta’s AI assistant.
OpenAI system prompts (Oct/2023)
Here’s the DALL-E 3 on ChatGPT prompt:
https://lifearchitect.ai/alignment/#dall-e3
And here are all the others:
https://github.com/spdustin/ChatGPT-AutoExpert/blob/main/System%20Prompts.md
South Park tackling AI for next event special, releases teaser (11/Oct/2023)
Paramount+ has announced its fifth South Park event special, titled ‘South Park: Joining the Panderverse’. The episode, which is about AI ‘turning their world upside down’, streams on Friday 27/Oct/2023 in the US and Canada, and the next day in the UK and Australia.
Read more: https://www.hollywoodreporter.com/tv/tv-news/south-park-ai-joining-panderverse-1235615276/
Watch the trailer; no mention of AI (link):
And here’s a clip from the ChatGPT episode earlier this year (link):
Flashback
As large language models continue to saturate the public consciousness, I was thinking back to very recently when people were saying that these models are just parrots.
It seems like only yesterday that even ‘experts’ who studied AI from the 80s and 90s were giving their uninformed opinions about today’s artificial intelligence. Recall the absolute nonsense that was spewed by Sophia creator Ben Goertzel in Dec/2020. (Unfortunately, some people still listen to him…)
But what [GPT] did, it looked at all the multiplication problems online and memorized the answers. And then it came up with some weird extrapolations and let it do a few problems that weren’t in its training database. [Alan: this is wildly incorrect, and spelt out in detail in the GPT-3 paper.]
It doesn’t understand what multiplication is, or it would never get 15 or 20% multiplication problems right. And you can see that in many other cases. Ask it like, who were the best presidents of the U.S. it’ll answer a lot of good things then it’ll throw a few kings of England in there just for fun. But I mean, because it doesn’t know what ‘of the U.S’ means.
…in the end, [GPT] has no more to do with AGI than my toaster oven does. It’s not representing the knowledge in a way that will allow it to make consistently meaningful responses. And that’s not to say that everything in there is totally useless for AGI. It’s just you’re not going to make GPT-4, 5, 6, 7 and get AGI.
Next
The next roundtable will be:
Life Architect - The Memo - Roundtable #4
Follows the Chatham House Rule (no recording, no outside discussion)
Saturday 4/Nov/2023 at 5PM Los Angeles
Saturday 4/Nov/2023 at 8PM New York
Sunday 5/Nov/2023 at 8AM Perth (primary/reference time zone)
or check your timezone via Google.
You don’t need to do anything for this; there’s no registration or forms to fill in, I don’t want your email, you don’t even need to turn on your camera or give your real name!
All my very best,
Alan
LifeArchitect.ai