From learning to earning,sight translated to sound.
Sense Square is building an AI-powered visual companion for blind and low-vision people—helping users read, locate what they need, understand their surroundings, and carry useful context from education into everyday work.
A world of information. A more natural way to access it.S² / 01
01The gap we're working on
Education opens the door. Independence helps people stay in the room.
Blind and low-vision people can encounter visual information throughout education and employment—from books and printed material to desks, screens, objects, unfamiliar layouts, and changing surroundings. When that information is inaccessible, everyday tasks can require unnecessary assistance.
We are focusing on four recurring challenges.
01
Reading workplace and learning material
Printed documents, books, notices, labels, screens, and other visual information are still difficult to access consistently.
02
Finding objects and navigating familiar spaces
Locating a desk, bottle, document, doorway, appliance, or commonly used object can interrupt otherwise independent work.
03
Understanding the surrounding environment
Rooms and workplaces constantly change—people move, objects shift, and new visual information appears.
04
Personal safety and situational context
Useful environmental context can help users make more informed decisions without overwhelming them with unnecessary narration.
02What we're building
Three capabilities. One continuous understanding.
We are concentrating the next stage of Sense Square on the tasks that can directly support learning, independence, and work.
01 Find mode
Prototype — improving
Find what you need.
Ask Sense Square to locate an object and receive repeatable audio guidance that helps you move toward it.
We are improving Find Mode for
Object localisation
Repeat assistance
Directional guidance
Desks and workstations
Familiar indoor spaces
Basic navigation around known environments
Find Mode is not autonomous or safety-critical navigation, and it does not promise centimetre-level precision.
Example interaction
ID
Slightly right · on the desk
You
“Where is my ID card?”
Sense Square
“It's on the desk, slightly to your right, beside the laptop.”
02 Reading mode
In development
Turn visual information into something you can work with.
Reading Mode is being designed for books, documents, printed material, labels, and screens—not just to read text aloud, but to help users understand and interact with it.
Example use cases
Read a printed page
Understand a workplace document
Read a screen or notice
Ask questions about what was read
Continue from the previous section
Summarise longer material
Example multi-turn reading interaction
You
“Read this page.”
Page read aloud
You
“What was the second requirement?”
Answer grounded in the page
You
“Continue from there.”
03 Agentic memory + situational awareness
R&D
Let context stay useful.
Instead of treating every camera interaction as a completely new moment, we are exploring an intentional memory layer that can retain useful context about places, objects, people, and surroundings.
Example requests
“Remember that this is my desk.”
“Remember where I normally keep my ID card.”
“Who did I meet here earlier?”
“What changed around my workspace?”
How intentional memory could support assistance
Camera / Voice
↓Current Understanding
↓Intentional Memory
↓Relevant Recall
↓Audio Assistance
Potential memory categories
Places
Objects
People
Tasks
Environmental context
Memory by intention, not by default.
Any memory layer must avoid covert recording, continuous surveillance, and permanent recognition of every person encountered.
User controlled
Intentional
Inspectable
Removable
Designed around explicit consent where people are involved
Future direction. Research may extend this foundation into safer mobility, richer navigation, and trusted assistance.
03Software × Hardware
The intelligence is only half the product.
For assistive AI to become genuinely useful throughout a working day, the interaction cannot always depend on holding up a phone. We are developing the software and hardware together so visual understanding can eventually become a more natural, hands-free experience.
Software
The Sense Square intelligence layer
01
Find Mode
Object localisation, repeat guidance, and familiar-space assistance.
02
Reading Mode
Books, documents, printed material, screens, and contextual questions.
03
Agentic Memory
Intentional memory for useful context about places, objects, people, and tasks.
04
Situational Awareness
Understanding the current environment and meaningful changes around the user.
Hardware
Hardware design in progress
Built around the person, not the phone.
Hardware engineering is currently focused on the complete device experience without presenting unfinished decisions as final specifications.
Complete 3D form factor
Camera placement and field of view
Microphones
Audio output
Battery
Connectivity
Compute requirements
Thermal constraints
Weight and comfort
Physical controls
Charging
Overall engineering architecture
Conceptual hardware architecture
Camera
Audio input and output
Power and compute
Controls and connectivity
Simplified product architecture
01See
Camera
↓
02Understand
Scene intelligence, reading, and spatial understanding
↓
03Remember
Intentional context
↓
04Respond
Natural audio
↓
05Device
Wearable interaction
Prototype roadmap
Building the system in deliberate stages.
Phase 01
Validate the intelligence
Improve Find Mode, Reading Mode, memory, and situational understanding using the existing prototype.
Phase 02
Engineer the device
Finalise the initial 3D form factor, core component selection, cameras, audio, battery, connectivity, and compute architecture.
Phase 03
First integrated hardware prototype
Bring the AI capabilities onto the first functional device and begin testing the software-hardware experience together.
Target: first integrated functional prototype during the next hardware build cycle.
04How it works
Point. Ask. Hear.
01
Point and ask
Open the prototype, aim the camera, and ask a question in your own words.
02
Understand the scene
Sense Square interprets the current view while the session remains open.
03
Hear and continue
Hear a concise answer, interrupt if needed, and ask a follow-up without starting over.
Prototype target: first spoken response in under one second.
Why conversation
A conversation, not a sequence of disconnected photos.
During an open session, users can interrupt, ask a follow-up, and keep the current scene in context instead of restarting every question from a new image.
05Responsible development
Useful context, spoken responsibly.
We are building toward more independent learning and work while treating uncertainty, consent, privacy, and user control as product requirements.
01
User control
The user chooses when a session starts, what to ask, and when the camera and microphone stop.
02
Spoken uncertainty
When the system cannot see clearly, it should say so instead of turning uncertainty into confident guidance.
03
Privacy by design
The current prototype does not store captured session video, audio, images, or transcripts on our server.
04
Built with users
Interaction choices are shaped around blind and low-vision users, not added as an accessibility layer later.
06Privacy
What happens to the camera.
The camera and microphone stream to OpenAI only while a session is open, using encrypted connections and secure APIs. Sense square does not record or store captured video, audio, images, or transcripts on its server. Bystanders may appear in frame.