The Memo - 17/Aug/2023
Stability AI's new LLMs, chips and politics, Unitree H1 robot, and much more!
FOR IMMEDIATE RELEASE: 17/Aug/2023
Bill Gates on GPT-4 passing the AP Biology exam (11/Aug/2023 or
watch my video about this quote):
It's the most stunning demo I've ever seen in my life, right up there with seeing the Xerox PARC graphics user interface that set the agenda for Microsoft... this demo was so surprising to me, the emergent depth that as they scaled up the training set, its fluency and you have to say understanding…
I'm still, personally, in a state of shock at ‘Wow, it is so good’.
Welcome back to The Memo.
You’re joining full subscribers from BlackRock, Boeing, BAAI, Brainware.io, BAE Systems, BSH Hausgeräte, Berkeley, Boston Consulting Group, and more…
The chip edition! In this edition, we spend time putting the recent GPU chip distribution into plain English, looking at a new chip company to compete with NVIDIA, and hearing from the man who made GPT-3 and Claude. In the Toys to play with section, we look at new on-device Llama 2 models, slide decks designed by AI, a global map of robotaxis, and the coolest new audio AI model.
The next roundtable for full subscribers will be very soon:
Life Architect - The Memo - Roundtable #2 with Harvey Castro
Saturday 26/Aug/2023 at 5PM Los Angeles
Saturday 26/Aug/2023 at 8PM New York
Sunday 27/Aug/2023 at 8AM Perth (primary/reference time zone)
or check your timezone.
Details at the end of this edition.
The BIG Stuff
Where will AGI begin? (Aug/2023)
OpenAI defines artificial general intelligence (AGI) as ‘highly autonomous systems that outperform humans at most economically valuable work.’
We can now estimate with a little more certainty when AGI will be here (hint: in around 23 months from now; mid-2025). But what about where? I was discussing this recently with a colleague, and posing China as a potential leader in delivering AGI first. Here’s a new viz showing my top three picks for the exact location of the first AGI system.
If you live in one of these cities (perhaps with the exception of sunny London!), you’re already exposed to bleeding-edge technology like 24x7 driverless robotaxis, very high-speed Internet, streamlined access to society via mobile apps, automated deliveries—sometimes within minutes of purchase, and much more. It’s not such a stretch to imagine a human-level machine being produced in an AI lab in one of these cities very soon.
Bonus: Back in Jul/2016, OpenAI admitted that one of their special projects is to monitor for stealth AGI release by another lab: ‘Detect if someone is using a covert breakthrough AI system in the world… the probability increases that an organization will make an undisclosed AI breakthrough… It seems important to detect this. We can imagine a lot of ways to do this—looking at the news, financial markets, online games, etc.’
Read more: https://lifearchitect.ai/agi/
Stability AI unleashes many open-source models (Aug/2023)
Stability AI were on fire in the last few weeks. Their major language model announcements include:
Released a Japanese model, Japanese StableLM Alpha 7B, the best-performing openly available language model for Japanese speakers (link).
Released a code model, StableCode 3B trained on The Stack dataset, and with a context window of 16,384 tokens (link).
Released their ChatGPT competitor Stable Chat: research.stability.ai/chat.
Renamed their fine-tuned FreeWilly models to Stable Beluga 1 65B and Stable Beluga 2 70B (based on Llama 1 and Llama 2) (link).
Stability AI’s instruction tuned version of Llama 2 Stable Beluga 2 70B does very well in English assessments like the SAT, even outperforming OpenAI’s ChatGPT:
Make sure you try Stable Chat (free, login): research.stability.ai/chat
Advanced use cases for LLMS: Agriculture (15/Aug/2023)
I’m considering running a series about advanced use cases of LLMs around the world, on the way to having these models actually invent new things, present new theorems and proofs, discover new materials, and more broadly augment our human capacity.
The following article is about ‘genAI’ [what a lame buzzword; it just means AI based on large language models or other models that can generate completions] being applied to agriculture in India:
According to a recent report [ARK 2/Aug/2023], AI and precision agriculture have the potential to bring down annual agricultural operating costs by over 22% globally.
Plantix, a Germany-based agritech firm focused on India, is testing genAI on its plant disease diagnosis platform. Users can get a diagnosis as well as suggestions for remedies and preventive measures by capturing and uploading a plant image on the app.
Meanwhile, Cropin, a precision-farming firm established in 2010, is using genAI to rebuild satellite data obstructed by clouds.
On the other hand, products like Kissan AI, FarmerChat, and Jugalbandi are introducing ChatGPT-like conversational AI products for farming…
Some companies like Sagri, Cropin, and Satyukt are taking a different approach when it comes to integrating genAI into agriculture. Instead of focusing on chatbots, they use AI, machine learning, and remote sensing to create an intelligent, interconnected data platform…
Read more via TechInAsia: https://archive.md/fUdvA
Some of these advanced LLM use cases for agri seem similar to several earlier reports in The Memo of LLM use in defense: notably Palantir AIP (video) and Scale AI Donovan (video).
I’ve said it many times: human imagination is the main limitation of post-2020 AI. We still don’t know how these superhuman models work (and we don’t really need to), and we are still finding ways to apply this superhuman intelligence to our human challenges.
Unitree H1 humanoid robot (15/Aug/2023)
Here it is, China’s latest copy of a humanoid robot. H1 is 47kg, about 180cm, and available for under US$90,000.
This thing is FAST, the fastest robot I’ve seen, running at more than 18km/h (for humans, men run at 13km/h and women at 10km/h on average).
Read more: https://www.unitree.com/en/h1/
Watch the video (link):
The Interesting Stuff
China + GPUs (10/Aug/2023)
A graphics processing unit (GPU) is a computer chip that renders images by performing rapid mathematical calculations. Over the last few years, researchers have been using clusters of thousands of GPUs to accelerate AI and large language model workloads.
This stuff might seem nerdy, but consider that with the advent of AGI, GPUs will increasingly run our entire lives, becoming more important than the electrical transformer, internal combustion engine, or even the Internet (NVIDIA calls it ‘the Most Important Work of Our Time’).
Alibaba Group Holding Ltd. hasn’t been able to completely fulfill demand for AI training from clients because of global supply constraints, its top executive said, suggesting a shortage of critical components such as artificial intelligence chips is weighing on Chinese efforts to ramp up in the cutting-edge technology…
A shortage of high-powered semiconductors is undermining Chinese efforts to keep pace with the US in AI. Washington has banned Chinese firms from buying the most advanced chips made by Nvidia Corp., impeding attempts to build rivals to OpenAI’s ChatGPT. Nvidia has since created an inferior version of its most potent A100 chips for the country, and major Chinese tech firms including Alibaba have reportedly placed billions of dollars’ worth of orders.
Alibaba, Baidu, ByteDance, and Tencent have placed orders worth $1 billion to buy about 100,000 of Nvidia’s A800 processors, which will be delivered this year. The four Chinese tech giants also bought $4 billion worth of NVDA’s graphics processing units (GPUs) to be delivered in 2024.
Nvidia offers its A800 processor in China to abide by the U.S. export control rules, which banned the sale of two advanced chips – the A100 and H100. These two advanced chips are used for high-performance computing and for training AI models. The A800 GPUs have slower data transfer rates than the A100s… [Alan: see details below.]
ByteDance, which owns the video-sharing social media app TikTok, has reportedly stockpiled at least 10,000 Nvidia chips to support its AI projects. Additionally, it has placed orders for about 70,000 A800 chips for nearly $700 million, to be delivered in 2024.
The only real difference between the A100 and the ‘made for China’ A800 is the chip-to-chip data transfer rate, also known as the interconnect, or what NVIDIA marks as the NVLink. The interconnect on the A100 for US operates at 600GB/sec, where the interconnect on the A800 for China operates at 400GB/sec; the US model runs at 150% the speed of the Chinese model.
Generalising this interconnect speed to model training time (although this is not really a great assumption), and in plain English, this would translate to training a large language model the size of GPT-3 in 51 days instead of 34 days. A huge difference if you’re in a rush to compete with other nations during humanity’s biggest evolutionary process…
Here’s an exclusive comparison of the two major cards side-by-side using NVIDIA’s spec sheets:
Notably, NVIDIA’s most sought-after GPU for LLM training right now is the larger NVIDIA H100, with up to 900GB/sec interconnect [225% the speed of the A800] and up to 188GB GPU memory (link). This hardware will no longer be available to China.
Read the gold-standard summary of NVIDIA H100 supply by GPU Utils (Aug/2023).
Despite Thomas L. Friedman’s assertion that The World Is Flat (wiki), this is a massive geo-political-socio-cultural-economic footnote in history:
NVIDIA was founded by both Chinese (the Republic of China, ROC) and American members, with a Taiwan-born CEO.
NVIDIA is headquartered in the US (incorporated in Delaware and based in Santa Clara, California).
NVIDIA is ‘fabless,’ meaning they don’t have their own fabrication facilities (also called foundries) or factories, instead relying on partners to make the chips.
NVIDIA’s H100 chips are actually made in Anding District, Tainan, Taiwan (ROC) by the Taiwan Semiconductor Manufacturing Co Ltd (TSMC) in a $20B facility called Fab 18 (source, wiki).
Just a 2-hour flight across the pond is mainland China (the People’s Republic of China, PRC). Parts of the NVIDIA fab process are also undertaken at TSMC’s Fab 10 in Songjiang, China (PRC), or Fab 16 in Nanjiing, China (PRC).
The US has now said that China (PRC) is not allowed to buy the most powerful NVIDIA chips, despite the fact that China actually makes those chips (manufactured in ROC, with some fab in PRC). Read The White House Executive Order (9/Aug/2023).
I wonder how we’ll view this decision to slow down competitors in a year or two…
Having been a permanent resident in PRC, and spent even longer in ROC, I look forward to AI helping out with stabilising these very strange geo-political tensions, and bringing much-needed diplomacy to the world.
Bonus: OpenAI’s big models were designed by Russian-born and Israel-raised Canadian Dr Ilya Sutskever, with a multinational team of mostly American, Indian, and Chinese developers, programmed in the Python language invented by a Dutchman, trained on global data in 90+ languages using Taiwanese hardware… and I suppose the kicker is that you’re getting this analysis from an Australian with a bit of Italian heritage!
Saudi Arabia + UAE + GPUs (14/Aug/2023)