Recycled Sound · Engineering teardown

A phone that recognises a hearing aid

Point a camera at a used hearing aid the size of a coffee bean, and in a few seconds the app knows the brand, the model, and a dozen things an audiologist would need to fit it - running entirely on the phone, for effectively zero dollars a month. Here's exactly how every piece fits together.

~18 min read Flutter · Firebase · on-device ML 9 diagrams

The one-paragraph version

Recycled Sound collects donated hearing aids and gives them to refugees and low-income residents who can't afford them. The hard part is that a donated aid arrives as anonymous plastic - no box, no papers. So the hero feature is a Google-Lens-style scanner: it reads the tiny printed label, cross-references a bundled catalogue, and pre-fills a device record - then an audiologist confirms it. Around that scanner sits a mobile app, a web admin console, a Firebase backend, and an offline machine-learning pipeline that trains the models the phone runs. No automated system ever touches a recipient. That last sentence is a design decision, not an accident.

01The problem: anonymous plastic

A hearing aid is one of the most information-dense objects most people will never read. Whether it takes a size-312 battery or charges on a dock, whether its receiver is a slim tube or a wire, whether it can be tuned remotely or only in a clinic - all of that changes how, and whether, it can be given to a specific person. Get it wrong for a refugee who has waited months for help, and you haven't just wasted a device; you've broken a promise.

The catch: a donated aid shows up with none of that written down. No box. No fitting notes. Just a curved sliver of plastic with a model name printed on it in letters smaller than this full stop. Somebody has to turn that sliver into a 26-field clinical record - and the pool of "somebodies" is a handful of volunteer audiologists giving their evenings.

So the whole system is built around a single leverage point: let a phone do the reading, and let a human do the deciding. Everything below is in service of that split.

26
clinical fields per device record
<5s
to identify a known device, on-device
$0
/month cloud inference at rest
0
automated touches on a recipient

02The 10,000-foot view

Before any code, here's the shape of the whole thing: four kinds of people, two apps they use, one backend that holds the truth, and a set of cloud functions off to the side. Follow the arrows - an arrow means "sends data to."

flowchart LR
  subgraph P["People"]
    direction TB
    V["Volunteer / Donor"]
    A["Audiologist"]
    AD["Admin"]
  end
  subgraph AP["Two apps"]
    direction TB
    M["📱 Mobile app
scan + capture"] W["🖥️ Web admin
triage + register"] end subgraph FB["Firebase backend - the source of truth"] direction TB AU["Auth"] FS[("Firestore")] ST[("Storage")] end CF["☁️ Cloud Functions
Vision + CLIP (fallback)"] V --> M A --> W AD --> W M --> AU & FS & ST W --> AU & FS & ST M -. "gallery fallback only" .-> CF CF --> FS classDef app fill:#e7f3ec,stroke:#1c8a63,color:#0e5c40; classDef store fill:#fff,stroke:#c98a1e,color:#8a5e12; classDef cloud fill:#fbe9e2,stroke:#e0603c,color:#a13c1f; class M,W app; class AU,FS,ST store; class CF cloud;
The context map. Volunteers scan on mobile; audiologists and admins work the web console; both apps read and write the same Firebase backend. The cloud functions (dashed) are a fallback path - hold that thought for §5.

The most important thing this picture says: the two apps are genuinely different programs. They don't share a screen. The mobile app is a scanner that a volunteer points at a device; the web app is a spreadsheet-with-superpowers where an audiologist curates the register. They meet only in the backend.

03The four moving parts

Zoom in one level. There are exactly four things that run, each in its own place. In the codebase these are four directories; in production they are four independently deployable artifacts.

flowchart TB
  subgraph Phone["On the phone"]
    MA["Mobile app · Flutter
lib/main.dart
scanner · capture · device list"] end subgraph Browser["In a browser"] WA["Web admin · Flutter
lib/main_web.dart
triage queue · register · review"] end subgraph Google["On Google's servers"] FN["Cloud Functions
functions/ (TypeScript)
functions-clip/ (Python)"] BE["Firebase backend
Auth · Firestore · Storage
+ security rules"] end MA <--> BE WA <--> BE MA -.-> FN FN --> BE classDef a fill:#e7f3ec,stroke:#1c8a63,color:#0e5c40; classDef b fill:#eef1ec,stroke:#5f6f68,color:#2a3a34; class MA,WA a; class FN,BE b;
Four containers. Two Flutter builds from one codebase (different main() entrypoints), plus two things living in Google's cloud. The mobile app and web app never talk to each other directly - only through the backend.

One codebase, two apps, is a deliberate trick. The mobile and web builds share every model, repository, and security assumption - but main_web.dart deliberately omits the scanner, because the scanner depends on tflite_flutter, a native library that won't compile to web. The web app couldn't run the neural net if it wanted to; it doesn't want to. It's an admin tool.

The routes, if you like a map with street names

The mobile app boots at /boot (a diagnostic splash that waits for Firebase), lands on / (home), and the interesting screens are all full-screen takeovers: /scan (the live scanner), /scan/confirm (the 7-field review), /capture and /capture/uploading (the photo flow), and /devices/:id. There's a /scan/3d ARKit point-cloud experiment and a legacy /scan/analysing path we'll meet in §5.

The web app starts at /incoming (the triage queue), with /incoming/:id/review and /devices as the working surfaces. /matching and /users are honest placeholders. An auth guard bounces anyone not signed in to /login; the actual "can this person see this?" is enforced by the backend rules, not the router.

04The scanner: what happens in those five seconds

This is the heart of it, so we'll go slow. You open /scan, the camera comes alive, and you hold a hearing aid in frame. Somewhere between one and five seconds later, seven fields have filled themselves in. Here is every stage of that, in order.

sequenceDiagram
  autonumber
  participant Cam as 📷 Camera stream
  participant Pre as Frame filter
  participant OCR as ML Kit OCR
  participant BM as BrandMatcher
  participant DI as DeviceIndex
  participant NN as TFLite neural net
  participant HUD as 7-field HUD

  loop every frame (one in flight)
    Cam->>Pre: raw frame
    Pre->>OCR: RAW / ENHANCE / HI-CON (cycled)
    OCR->>BM: text lines
    BM->>DI: narrow(brand, model, …) - fuzzy match
    DI->>DI: intersect candidates · auto-lock singletons
    DI->>HUD: shrinking slot reels
  end
  Note over DI,NN: colour locks → grab one full-res still
  DI->>NN: full-res image
  NN-->>DI: brand vote (only if confident)
  DI->>HUD: brand + model locked
  HUD->>HUD: → route to /scan/confirm
    
The live pipeline. A tight per-frame loop (OCR → match → narrow) does the work; the heavier neural net fires once, on a single full-resolution still, only after the colour detector has settled. Nothing waits on the network.

Stage 1 - read the label (OCR)

Each camera frame is run through a small filter that cycles between raw, contrast-enhanced, and high-contrast versions - because a label that's illegible under one looks crisp under another - and then handed to Google's ML Kit text recogniser, which runs on the phone. No upload. The text it finds ("phonak", "audeo", a serial number) becomes the raw material for everything downstream.

◆ The jaw-drop

OCR accuracy is not monotonic in how much you zoom. Crop in tighter and a label the recogniser nailed at 60% can vanish at 80%, then reappear at 85%. There's a dead-band. So the certification harness reads the label at several crop scales at once - a "crop pyramid" from ~35% to 85% - and takes the best hit. A well-chosen pyramid of native center-crops beats any single crop, and beats naively upscaling the image. That one finding is what made "identify on the phone" viable at all.

Stage 2 - turn text into candidates (BrandMatcher)

Printed labels are tiny and cameras are shaky, so the OCR output is misspelled more often than not - "oticon" comes back as "oricon", "moxi" as "movi". The BrandMatcher expects this. It does fuzzy matching (Levenshtein edit-distance) against the known brand and model list, and - crucially - it can go backwards: spot a model name and infer the brand, even if the brand word never made it through the camera.

Stage 3 - the elimination tree (DeviceIndex)

This is the cleverest part, and the least obvious. Think of the whole catalogue of hearing aids as a room full of suspects. Every piece of evidence - a matched brand, a legible model, a detected colour - doesn't pick a device; it eliminates the ones that don't fit. When only one candidate remains for a field, that field locks itself. And locked fields cascade: pin the battery size and the power type often follows for free; pin the style and the tubing narrows.

Why an elimination tree and not "just pick the best match"? Because evidence arrives noisily, frame by frame, and contradicts itself. The DeviceIndex ranks every clue by how much to trust it - a clear OCR read outranks a hesitant neural-net guess - and refuses to let weaker evidence overturn a lock. But it isn't stubborn: if the same contradicting value shows up two frames running, it treats that as a real correction and re-opens the field. Confidence with a backspace key.

Stage 4 - the neural net (once, and only once)

Reading text handles most devices, but some labels are worn, embossed, or simply blank. For those, the app ships a trained neural network - an EfficientNet-B0 brand classifier, a 4.7 MB TensorFlow Lite model - that guesses the brand from what the device looks like. It's expensive relative to a frame filter, so it doesn't run on every frame. It fires exactly once, on a single full-resolution photo, after the colour detector has settled - and even then its vote only counts if it clears a confidence bar. It's a tie-breaker, not the star.

◆ The constraint that shapes everything

The camera pipeline is single-threaded. Every pretty animation, every extra filter, every eager neural-net call competes with OCR for the same frames. Early on, detection took 15+ seconds because visual polish was quietly eating the frame budget. The governing rule now: detection throughput is sacred. You don't add anything to the per-frame loop without measuring the milliseconds it costs. That's why the neural net runs off the hot path, and why colour detection doesn't even start until OCR has seen some text.

Stage 5 - the HUD, and the handoff

All of this is happening while a heads-up display shows seven fields - brand, model, style, tubing, power, battery, colour - as little slot-machine reels that shrink as candidates get eliminated. The moment brand and model are both locked, the scanner has done its job. It packages what it found, tears down the camera, and routes to the confirmation screen. It does not phone home.

05The plot twist: the cloud brain we retired

Here's the part that surprises engineers. Recycled Sound started as a cloud-inference app. Two Cloud Functions did the heavy lifting: a Python one (clip_encode) that turned a photo into a 512-number "visual fingerprint" using OpenCLIP, and a TypeScript one (analyzeHearingAid) that ran Google's Vision API for labels and text. The phone sent up a picture and got back an answer. It worked.

And then it got demoted to a fallback.

flowchart TB
  Start["📷 A photo"] --> Q{"Live camera,
or picked from gallery?"} Q -- "live scan" --> ON["On-device path
OCR → match → elimination → NN"] ON --> Conf["/scan/confirm"] Q -- "gallery pick
(no live detection)" --> CLOUD["Cloud path
clip_encode + analyzeHearingAid
fused on-device"] CLOUD --> Conf classDef on fill:#e7f3ec,stroke:#1c8a63,color:#0e5c40; classDef cl fill:#fbe9e2,stroke:#e0603c,color:#a13c1f; class ON,Conf on; class CLOUD cl;
Two paths, one destination. A live scan never touches the cloud. The cloud functions only run when you pick a still from the gallery - there's no live detection to work with, so it falls back to the old fused pipeline. Both roads end at the same confirmation screen.

Why demote a working cloud pipeline? Three reasons, and they're the moral of the whole project:

Speed. A network round-trip is seconds you don't have when a volunteer is holding a fiddly device up to a phone. On-device is instant.

Cost. The Python function's own source comment brags about it: on-device cosine-similarity search against bundled fingerprints is "$0/month infrastructure." A charity's cloud bill at rest should be zero, and it is.

The domain gap. The visual-fingerprint approach was trained on clean product photos. Real photos are hand-held, badly lit, and half-occluded by fingers. It turned out the printed text - read locally - was simply the stronger signal. So the architecture followed the evidence: OCR-first, with a visual fallback for the rare label you can't read.

The cloud functions aren't dead - they still deploy, and they still catch the gallery-pick case. They just aren't the main road anymore. It's a rare and honest thing to see in a codebase: a good idea, kept exactly as big as it deserves to be.

06From a scan to labelled training data

Identifying a device is half the point. The other half is photographing it well - both to document the physical item and to feed the next generation of models. This is the capture flow, and it hides a subtle idea.

sequenceDiagram
  autonumber
  participant Sc as Scanner
  participant Cf as Confirm screen
  participant Se as CaptureSeed
  participant Ca as Capture flow
  participant St as Storage
  participant Fs as Firestore

  Sc->>Cf: 7-field ScanResult
  Cf->>Cf: audiologist corrects & confirms
  Cf->>Se: label + scanId + box number
  Se->>Ca: seed the capture
  loop 12 shots - 6 angles × left/right
    Ca->>St: captures/{uid}/{deviceId}/{side}_{slot}.jpg
  end
  Ca->>Fs: incoming/{id} - fully labelled
    
The capture flow. The confirmed label rides along into the photo session via a CaptureSeed, so every one of the 12 photos lands tagged with the audiologist's ground truth - not just a box number.

The 12 shots aren't arbitrary. Each aid gets photographed from six named angles - the two broad sides, the thin front and back edges, the top where the hook or tube comes out, and the underside with the battery door. Two aids, six angles: twelve frames. The filenames encode the angle (left_medial.jpg), so a photo's identity survives even if the sequence is interrupted - an earlier position-based scheme silently mislabelled photos when a step was retaken.

One of those six is worth a note, because getting it wrong cost us real volunteer confusion. Earlier copy pinned the brand lettering to the medial face and told volunteers the other side was the wrong one. It isn't reliably true: the label can sit on either broad side depending on the make. The app now says "usually the inner side, but check both", and OCR runs on every shot rather than only the one we expected to be readable.

◆ The subtle idea

For a long time, the photos and the labels lived on two different paths. You could scan-and-confirm a device (getting a rich label but no photos), or you could capture-photos-for-later (getting 12 images tagged with nothing but a box number and a guessed brand). Pulling the photo bucket gave you "12 angles of box C25, maybe an Oticon" - useless as training data. The fix was to make the confirmed label travel with the pixels: the CaptureSeed carries the audiologist's make, model, style, battery - and a link back to the original scan - into the capture. Now every bundle is labelled ground truth. Recycled hardware, producing recycled data, to train better models. The loop closes.

07The backend: the spine we replaced, and a lifecycle

Everything the apps produce lands in Firebase. Two ideas make the backend worth explaining: how it handles identity - which we got wrong once, in an instructive way - and how a device moves through its life.

The spine we had to retire

The original design was elegant and we were fond of it. The very first thing the app did was sign you in anonymously: before you had typed a thing you had a stable user ID, everything you scanned was owned by that ID, and the rules could still enforce "you can only edit what you created." Zero friction at a working bee, real ownership underneath. If you signed up later, linkWithCredential upgraded the account in place and you kept every device you had logged.

The flaw is visible the moment you say it out loud: anyone who downloaded the app got a writable identity. Not a volunteer - anyone. The intake collection that the whole charity's records are built on was writable by any stranger who installed the app. Convenience for a volunteer and an open door for everyone else turned out to be the same mechanism.

So it is gone. Access is now invite-gated: an admin mints a single-use code, a volunteer redeems it (redeemInvite), and roles are granted rather than self-assigned. Signing up gets you an account in pending with the role you requested; an admin confirms it before you can do anything.

The subtle part: deleting the call would have done nothing. Removing signInAnonymously() is a no-op for every install that already exists - Firebase persists the anonymous uid across an app update, so those users would have opened the app already "signed in", with no profile, no role, and tightened rules silently denying every write. A working app, against a backend that refuses it, with no screen explaining why. Production measurement said it would have hit 100% of users: all 88 documents, 11 of 11 owner uids, anonymous. The fix had to evict the persisted session, not merely stop minting new ones - and the predicate is anonymity specifically, because a named volunteer signing back in between boxes is exactly the friction that would make the app unusable where it is actually used.
flowchart LR
  Open["Open the app"] --> Evict{"Persisted session
anonymous?"} Evict -- yes --> Out["Signed out
(evicted on boot)"] Evict -- "no, named" --> Keep["Session kept"] Out --> In["Sign in / sign up"] In --> Code["Redeem invite code"] Code --> Pend["users/{uid} - pending
+ requestedRole"] Pend --> Conf["Admin confirms"] Conf --> Role["Role claim
volunteer · audiologist · admin"] Keep --> Role classDef g fill:#e7f3ec,stroke:#1c8a63,color:#0e5c40; classDef r fill:#fdeaea,stroke:#c0392b,color:#7b241c; class Keep,Role,Conf g; class Out r;
The auth spine, after the cutover. No anonymous identity is minted, and any persisted one is evicted at boot. Roles are granted by an admin, never self-assigned, and are projected onto the auth token by a backend trigger.

Roles (volunteer, audiologist, admin) live in Firestore as the single source of truth, and a Cloud Function called syncRoleClaim mirrors your role onto the auth token that the security rules actually read - because a rule can only cheaply see the token, not the database. Role changes go through an admin-only callable, setUserRole; self-promotion is impossible by any write verb. A volunteer can create records; only an elevated role - audiologist or admin - can promote a device into the curated register.

Where the real security lives. Not in the app, and not in the router - in the Firestore and Storage rules, which run on Google's servers and can't be bypassed by a hacked client. They enforce genuinely tricky invariants: a device can only be promoted to "ready" if every empty clinical field has been explicitly declared a known-gap, and any "promote despite gaps" is a signed, audited override. Those rules were hardened by adversarial review until they held.

The device lifecycle

A donated aid isn't just "in the database" - it's somewhere along a journey, and the record knows where. This is a plain state machine, and it's the backbone of the whole charity's operations.

stateDiagram-v2
  direction LR
  [*] --> Donated
  Donated --> Reprogramming: audiologist claims it
  Reprogramming --> Ready: QA passed
  Ready --> Shipped: matched to a recipient
  Shipped --> Delivered
  Delivered --> Active: fitted & working
  Active --> [*]
    
The device lifecycle. Every record carries its status. The scanner creates it at Donated; an audiologist services and QA-checks it to Ready; admins match and track it the rest of the way to Active in someone's ear.
A note on the data model, for the type-system curious

A device is a 26-field record, and the same shape lives in two collections: incoming/ (what the scanner writes, pre-triage) and devices/ (the audiologist-curated register, post-triage). The closed-set clinical fields - style, battery size, tubing, power source - are real enum types with tolerant parsing, so a stray string from an old record heals to a known value instead of crashing.

There's a small, satisfying piece of type discipline here: the pre-saved draft type (DraftDevice) has no ID field at all. An ID only exists once the database assigns one, so the code makes it impossible to hold a half-real device with a fake empty ID. The transition is a one-way door: draft.toDevice(id). You can't accidentally lie about whether something's been saved.

08The brain, offline

The phone runs the models. But where do the models come from? That's a separate, offline pipeline - Python scripts that never ship to a user, whose only job is to produce the assets the app bundles.

flowchart LR
  Scrape["🌐 Scrape catalogue
~2,300 product photos"] --> Clip["CLIP embed
512-dim fingerprints"] Scrape --> Dna["Device DNA
ChromaDB vector store"] Clip --> Assets["📦 device_db.bin +
device_catalog.json"] Train["Train EfficientNet
brand classifier"] --> Tflite["brand_classifier.tflite
(4.7 MB)"] Certify["Certify OCR-first
crop-pyramid harness"] -.validates.-> Assets Assets --> App["📱 Bundled on-device"] Tflite --> App classDef d fill:#eef1ec,stroke:#5f6f68,color:#2a3a34; classDef a fill:#e7f3ec,stroke:#1c8a63,color:#0e5c40; class Scrape,Clip,Dna,Train,Certify d; class Assets,Tflite,App a;
The offline ML pipeline. Scrape catalogue photos, turn them into fingerprints and a structured "device DNA", train the brand classifier, certify the OCR approach - then pack it all into the assets the app carries. This runs on a laptop, not in production.

The centrepiece is the device DNA: for every catalogue device, the pipeline builds a rich profile with four facets - a visual signature, a technical "genome" (its specs), its capability bits (remote-tuning, app support, Auracast-readiness), and a suitability profile. That DNA lives in a vector database (ChromaDB) so the eventual matching step can ask "which donated devices are most like what this recipient needs?" - a nearest-neighbour search, not a rules engine.

Two other jobs matter. A linear probe is the honesty check: it trains a dead-simple classifier on the frozen visual fingerprints to ask "does CLIP even carry enough signal to tell brands apart?" - before anyone invests in the bigger model. And the certification harness is what proved the crop-pyramid OCR approach cleared a pre-committed 90% brand-accuracy bar, which is the evidence that justified going on-device-first in the first place. The team measured before it committed.

09How it all ships

Four artifacts, four release paths - and a deliberate split between "safe to automate" and "a human presses the button."

flowchart TB
  Main["git · main branch"] --> CI["ci.yml (on every push)"]
  CI --> Rules["✅ Firestore + Storage rules
auto-deploy"] CI --> Fns["✅ Cloud Functions (vision)
auto-deploy"] Main -. "manual button" .-> TF["🍎 iOS → TestFlight"] Main -. "manual button" .-> Play["🤖 Android → Play (internal)"] Main -. "manual build" .-> Host["🌐 Web → Firebase Hosting"] classDef auto fill:#e7f3ec,stroke:#1c8a63,color:#0e5c40; classDef man fill:#fbe9e2,stroke:#e0603c,color:#a13c1f; class Rules,Fns auto; class TF,Play,Host man;
Deployment topology. Backend rules and functions deploy automatically when tests pass - they're low-blast-radius and reversible. The user-facing app builds (iOS, Android, web) require a manual trigger, because every TestFlight upload burns a build number and notifies real testers.

The split is the interesting bit. Security rules and cloud functions auto-deploy on a green build - they're versioned, testable, and instantly reversible. But the mobile builds are gated behind a human, on purpose: an accidental release to real testers isn't something you can quietly undo. The pipeline treats "notify humans" as the higher-stakes action, which is exactly backwards from most people's intuition and exactly right.

10The human in the loop

We've spent nine sections on machinery that reads labels, eliminates candidates, runs neural nets, and matches devices to people by nearest-neighbour search. So it's worth ending on the one rule that overrides all of it:

No automated system ever touches a recipient.

The recipients here are refugees and asylum seekers - people for whom a wrong device or a mishandled message isn't a bad user-experience, it's a real harm to someone with very little margin. So the scanner, for all its cleverness, is explicitly a co-pilot. It pre-fills; the audiologist confirms. It suggests a match; a human approves it. Matching starts fully manual - an admin selects, the system merely notifies. Every recipient-facing workflow is human-mediated by design, and stays that way until the manual version has been proven safe.

That's the quiet engineering decision underneath the flashy one. The hard part wasn't teaching a phone to recognise a hearing aid - impressive as that is. The hard part was building all of that power and then drawing a firm line around where it's allowed to act alone. The AI removes the tedious reading. The human keeps the deciding. That's the whole design, in one sentence.


Recycled Sound is a community initiative of the Rotary Club of Springvale City and Arches Audiology, collecting, servicing, and redistributing donated hearing aids in Greater Dandenong, Victoria. If a diagram above made you want to poke at the code, that was the idea.