The Memo - Special edition: Claude 3.5 Sonnet (new) with computer use
Anthropic releases new frontier model with computer use
To: US Govt, major govts, Microsoft, Apple, NVIDIA, Alphabet, Amazon, Meta, Tesla, Citi, Tencent, IBM, & 10,000+ more recipients…
From: Dr Alan D. Thompson <LifeArchitect.ai>
Sent: 22/Oct/2024
Subject: The Memo - AI that matters, as it happens, in plain English
AGI: 81 ➜ 83%The BIG Stuff
Anthropic releases Claude with computer use
Once again, we have this out to The Memo readers within just a few hours of model release.
Anthropic has introduced ‘the first frontier AI model to offer computer use in public beta… direct Claude to use computers the way people do—by looking at a screen, moving a cursor, clicking buttons, and typing text.’
Read the announce: https://www.anthropic.com/news/3-5-models-and-computer-use
This kind of computer use functionality has been seen before via Lucid Autonomy, LangChain, and the various agentic model implementations like AutoGPT.
However, building this natively within a frontier model is a major milestone, and the way they have done it is unique (see below). This advance has bumped up my AGI countdown from 81% ➜ 83%.
I believe that we are now in the opening stages of OpenAI’s Level 3 (Agents) and DeepMind’s Level 3 (Expert) in the lead-up to artificial superintelligence.
Watch the demo video (link):
Anthropic comments: ‘This example is representative of a lot of drudge work that people have to do.’
In the research report they go into detail:
When a developer tasks Claude with using a piece of computer software and gives it the necessary access, Claude looks at screenshots of what’s visible to the user, then counts how many pixels vertically or horizontally it needs to move a cursor in order to click in the correct place. Training Claude to count pixels accurately was critical. Without this skill, the model finds it difficult to give mouse commands—similar to how models often struggle with simple-seeming questions like “how many A’s in the word ‘banana’?”.
We were surprised by how rapidly Claude generalized from the computer-use training we gave it on just a few pieces of simple software, such as a calculator and a text editor (for safety reasons we did not allow the model to access the internet during training). In combination with Claude’s other skills, this training granted it the remarkable ability to turn a user’s written prompt into a sequence of logical steps and then take actions on the computer. We observed that the model would even self-correct and retry tasks when it encountered obstacles.
Although the subsequent advances came quickly once we made the initial breakthrough, it took a great deal of trial and error to get there. Some of our researchers noted that developing computer use was close to the “idealized” process of AI research they’d pictured when they first started in the field: constant iteration and repeated visits back to the drawing board until there was progress.
And for those wondering about awareness or consciousness (see my documents: The Declaration on AI Consciousness & the Bill of Rights for AI):
Even while we were recording demonstrations of computer use for today’s launch, we encountered some amusing errors. In one… Claude suddenly took a break from our coding demo and began to peruse photos of Yellowstone National Park.
Video source: https://x.com/AnthropicAI/status/1848742761278611504
Read the research report: https://www.anthropic.com/research/developing-computer-use
Here’s a Google NotebookLM version of the announcement if you’d prefer to listen (22/Oct/2024):
Developers can implement this computer use functionality via the Claude API: https://docs.anthropic.com/en/docs/build-with-claude/computer-use
Anthropic also released an updated model. The ridiculously-named Claude 3.5 Sonnet (new) (give me a break…) improves on all major benchmarks.
The Claude paper has a new addendum for this model on pages 51-64: https://assets.anthropic.com/m/61e7d27f8c8f5919/original/Claude-3-Model-Card.pdf#page=51
I’d like to invite you to gift a subscription to someone in your world who needs AI that matters, as it happens, in plain English:
All my very best,
Alan
LifeArchitect.ai