Sight, translated to sound

From learning
to earning,
sight translated
to sound.

Sense Square is building an AI-powered visual companion for blind and low-vision people—helping users read, locate what they need, understand their surroundings, and carry useful context from education into everyday work.

  • Read independently
  • Find what you need
  • Understand your surroundings
Visual world. Spoken context.
A blind man with a white cane sits beside a mountain lake. Sound lines connect the landscape to an ear symbol. The illustration imagines hearing the scene: ‘Towering peaks kiss the sky, whispering calm through evergreen sighs.’
A world of information.
A more natural way to access it.

The gap we're working on

Education opens the door. Independence helps people stay in the room.

Blind and low-vision people can encounter visual information throughout education and employment—from books and printed material to desks, screens, objects, unfamiliar layouts, and changing surroundings. When that information is inaccessible, everyday tasks can require unnecessary assistance.

We are focusing on four recurring challenges.

  1. Reading workplace and learning material

    Printed documents, books, notices, labels, screens, and other visual information are still difficult to access consistently.

  2. Finding objects and navigating familiar spaces

    Locating a desk, bottle, document, doorway, appliance, or commonly used object can interrupt otherwise independent work.

  3. Understanding the surrounding environment

    Rooms and workplaces constantly change—people move, objects shift, and new visual information appears.

  4. Personal safety and situational context

    Useful environmental context can help users make more informed decisions without overwhelming them with unnecessary narration.

What we're building

Three capabilities.
One continuous understanding.

We are concentrating the next stage of Sense Square on the tasks that can directly support learning, independence, and work.

01 Find mode

Prototype — improving

Find what you need.

Ask Sense Square to locate an object and receive repeatable audio guidance that helps you move toward it.

We are improving Find Mode for

  • Object localisation
  • Repeat assistance
  • Directional guidance
  • Desks and workstations
  • Familiar indoor spaces
  • Basic navigation around known environments

Find Mode is not autonomous or safety-critical navigation, and it does not promise centimetre-level precision.

Example interaction

You

“Where is my ID card?”

Sense Square

“It's on the desk, slightly to your right, beside the laptop.”

02 Reading mode

In development

Turn visual information into something you can work with.

Reading Mode is being designed for books, documents, printed material, labels, and screens—not just to read text aloud, but to help users understand and interact with it.

Example use cases

  • Read a printed page
  • Understand a workplace document
  • Read a screen or notice
  • Ask questions about what was read
  • Continue from the previous section
  • Summarise longer material
Example multi-turn reading interaction

You

“Read this page.”

Page read aloud

You

“What was the second requirement?”

Answer grounded in the page

You

“Continue from there.”

03 Agentic memory + situational awareness

R&D

Let context stay useful.

Instead of treating every camera interaction as a completely new moment, we are exploring an intentional memory layer that can retain useful context about places, objects, people, and surroundings.

Example requests

  • “Remember that this is my desk.”
  • “Remember where I normally keep my ID card.”
  • “Who did I meet here earlier?”
  • “What changed around my workspace?”
How intentional memory could support assistance
  1. Camera / Voice
  2. Current Understanding
  3. Intentional Memory
  4. Relevant Recall
  5. Audio Assistance

Potential memory categories

  • Places
  • Objects
  • People
  • Tasks
  • Environmental context

Memory by intention, not by default.

Any memory layer must avoid covert recording, continuous surveillance, and permanent recognition of every person encountered.

  • User controlled
  • Intentional
  • Inspectable
  • Removable
  • Designed around explicit consent where people are involved

Future direction. Research may extend this foundation into safer mobility, richer navigation, and trusted assistance.

Software × Hardware

The intelligence is only
half the product.

For assistive AI to become genuinely useful throughout a working day, the interaction cannot always depend on holding up a phone. We are developing the software and hardware together so visual understanding can eventually become a more natural, hands-free experience.

Software

The Sense Square intelligence layer

  1. Find Mode

    Object localisation, repeat guidance, and familiar-space assistance.

  2. Reading Mode

    Books, documents, printed material, screens, and contextual questions.

  3. Agentic Memory

    Intentional memory for useful context about places, objects, people, and tasks.

  4. Situational Awareness

    Understanding the current environment and meaningful changes around the user.

Hardware

Hardware design in progress

Built around the person, not the phone.

Hardware engineering is currently focused on the complete device experience without presenting unfinished decisions as final specifications.

  • Complete 3D form factor
  • Camera placement and field of view
  • Microphones
  • Audio output
  • Battery
  • Connectivity
  • Compute requirements
  • Thermal constraints
  • Weight and comfort
  • Physical controls
  • Charging
  • Overall engineering architecture
Conceptual hardware architecture
  • Camera
  • Audio input and output
  • Power and compute
  • Controls and connectivity
Simplified product architecture
  1. See

    Camera

  2. Understand

    Scene intelligence, reading, and spatial understanding

  3. Remember

    Intentional context

  4. Respond

    Natural audio

  5. Device

    Wearable interaction

Prototype roadmap

Building the system in deliberate stages.

  1. Phase 01

    Validate the intelligence

    Improve Find Mode, Reading Mode, memory, and situational understanding using the existing prototype.

  2. Phase 02

    Engineer the device

    Finalise the initial 3D form factor, core component selection, cameras, audio, battery, connectivity, and compute architecture.

  3. Phase 03

    First integrated hardware prototype

    Bring the AI capabilities onto the first functional device and begin testing the software-hardware experience together.

Target: first integrated functional prototype during the next hardware build cycle.

How it works

Point. Ask. Hear.

  1. Point and ask

    Open the prototype, aim the camera, and ask a question in your own words.

  2. Understand the scene

    Sense Square interprets the current view while the session remains open.

  3. Hear and continue

    Hear a concise answer, interrupt if needed, and ask a follow-up without starting over.

    Prototype target: first spoken response in under one second.

Why conversation

A conversation, not a sequence of disconnected photos.

During an open session, users can interrupt, ask a follow-up, and keep the current scene in context instead of restarting every question from a new image.

Responsible development

Useful context,
spoken responsibly.

We are building toward more independent learning and work while treating uncertainty, consent, privacy, and user control as product requirements.

User control

The user chooses when a session starts, what to ask, and when the camera and microphone stop.

Spoken uncertainty

When the system cannot see clearly, it should say so instead of turning uncertainty into confident guidance.

Privacy by design

The current prototype does not store captured session video, audio, images, or transcripts on our server.

Built with users

Interaction choices are shaped around blind and low-vision users, not added as an accessibility layer later.

Privacy

What happens to the camera.

The camera and microphone stream to OpenAI only while a session is open, using encrypted connections and secure APIs. Sense square does not record or store captured video, audio, images, or transcripts on its server. Bystanders may appear in frame.

Read the complete privacy policy

Developers

Built openly.
Built with people.

Sense Square is an open-source project in active development, created to make the world easier to understand through sound.

Meet the developers

Contact · Help shape what comes next

More independence.
Built together.

Partner with us to co-design real-world pilots, host beta testing, or contribute expert feedback.