The one-paragraph version
Recycled Sound collects donated hearing aids and gives them to refugees and low-income residents who can't afford them. The hard part is that a donated aid arrives as anonymous plastic - no box, no papers. So the hero feature is a Google-Lens-style scanner: it reads the tiny printed label, cross-references a bundled catalogue, and pre-fills a device record - then an audiologist confirms it. Around that scanner sits a mobile app, a web admin console, a Firebase backend, and an offline machine-learning pipeline that trains the models the phone runs. No automated system ever touches a recipient. That last sentence is a design decision, not an accident.
01The problem: anonymous plastic
A hearing aid is one of the most information-dense objects most people will never read. Whether it takes a size-312 battery or charges on a dock, whether its receiver is a slim tube or a wire, whether it can be tuned remotely or only in a clinic - all of that changes how, and whether, it can be given to a specific person. Get it wrong for a refugee who has waited months for help, and you haven't just wasted a device; you've broken a promise.
The catch: a donated aid shows up with none of that written down. No box. No fitting notes. Just a curved sliver of plastic with a model name printed on it in letters smaller than this full stop. Somebody has to turn that sliver into a 26-field clinical record - and the pool of "somebodies" is a handful of volunteer audiologists giving their evenings.
So the whole system is built around a single leverage point: let a phone do the reading, and let a human do the deciding. Everything below is in service of that split.
02The 10,000-foot view
Before any code, here's the shape of the whole thing: four kinds of people, two apps they use, one backend that holds the truth, and a set of cloud functions off to the side. Follow the arrows - an arrow means "sends data to."
flowchart LR
subgraph P["People"]
direction TB
V["Volunteer / Donor"]
A["Audiologist"]
AD["Admin"]
end
subgraph AP["Two apps"]
direction TB
M["📱 Mobile app
scan + capture"]
W["🖥️ Web admin
triage + register"]
end
subgraph FB["Firebase backend - the source of truth"]
direction TB
AU["Auth"]
FS[("Firestore")]
ST[("Storage")]
end
CF["☁️ Cloud Functions
Vision + CLIP (fallback)"]
V --> M
A --> W
AD --> W
M --> AU & FS & ST
W --> AU & FS & ST
M -. "gallery fallback only" .-> CF
CF --> FS
classDef app fill:#e7f3ec,stroke:#1c8a63,color:#0e5c40;
classDef store fill:#fff,stroke:#c98a1e,color:#8a5e12;
classDef cloud fill:#fbe9e2,stroke:#e0603c,color:#a13c1f;
class M,W app; class AU,FS,ST store; class CF cloud;
The most important thing this picture says: the two apps are genuinely different programs. They don't share a screen. The mobile app is a scanner that a volunteer points at a device; the web app is a spreadsheet-with-superpowers where an audiologist curates the register. They meet only in the backend.
03The four moving parts
Zoom in one level. There are exactly four things that run, each in its own place. In the codebase these are four directories; in production they are four independently deployable artifacts.
flowchart TB
subgraph Phone["On the phone"]
MA["Mobile app · Flutter
lib/main.dart
scanner · capture · device list"]
end
subgraph Browser["In a browser"]
WA["Web admin · Flutter
lib/main_web.dart
triage queue · register · review"]
end
subgraph Google["On Google's servers"]
FN["Cloud Functions
functions/ (TypeScript)
functions-clip/ (Python)"]
BE["Firebase backend
Auth · Firestore · Storage
+ security rules"]
end
MA <--> BE
WA <--> BE
MA -.-> FN
FN --> BE
classDef a fill:#e7f3ec,stroke:#1c8a63,color:#0e5c40;
classDef b fill:#eef1ec,stroke:#5f6f68,color:#2a3a34;
class MA,WA a; class FN,BE b;
main() entrypoints), plus two things living in Google's cloud. The mobile app and web app never talk to each other directly - only through the backend.One codebase, two apps, is a deliberate trick. The mobile and web builds share every model, repository, and security assumption - but main_web.dart deliberately omits the scanner, because the scanner depends on tflite_flutter, a native library that won't compile to web. The web app couldn't run the neural net if it wanted to; it doesn't want to. It's an admin tool.
The routes, if you like a map with street names
The mobile app boots at /boot (a diagnostic splash that waits for Firebase), lands on / (home), and the interesting screens are all full-screen takeovers: /scan (the live scanner), /scan/confirm (the 7-field review), /capture and /capture/uploading (the photo flow), and /devices/:id. There's a /scan/3d ARKit point-cloud experiment and a legacy /scan/analysing path we'll meet in §5.
The web app starts at /incoming (the triage queue), with /incoming/:id/review and /devices as the working surfaces. /matching and /users are honest placeholders. An auth guard bounces anyone not signed in to /login; the actual "can this person see this?" is enforced by the backend rules, not the router.
04The scanner: what happens in those five seconds
This is the heart of it, so we'll go slow. You open /scan, the camera comes alive, and you hold a hearing aid in frame. Somewhere between one and five seconds later, seven fields have filled themselves in. Here is every stage of that, in order.
sequenceDiagram
autonumber
participant Cam as 📷 Camera stream
participant Pre as Frame filter
participant OCR as ML Kit OCR
participant BM as BrandMatcher
participant DI as DeviceIndex
participant NN as TFLite neural net
participant HUD as 7-field HUD
loop every frame (one in flight)
Cam->>Pre: raw frame
Pre->>OCR: RAW / ENHANCE / HI-CON (cycled)
OCR->>BM: text lines
BM->>DI: narrow(brand, model, …) - fuzzy match
DI->>DI: intersect candidates · auto-lock singletons
DI->>HUD: shrinking slot reels
end
Note over DI,NN: colour locks → grab one full-res still
DI->>NN: full-res image
NN-->>DI: brand vote (only if confident)
DI->>HUD: brand + model locked
HUD->>HUD: → route to /scan/confirm
Stage 1 - read the label (OCR)
Each camera frame is run through a small filter that cycles between raw, contrast-enhanced, and high-contrast versions - because a label that's illegible under one looks crisp under another - and then handed to Google's ML Kit text recogniser, which runs on the phone. No upload. The text it finds ("phonak", "audeo", a serial number) becomes the raw material for everything downstream.
◆ The jaw-drop
OCR accuracy is not monotonic in how much you zoom. Crop in tighter and a label the recogniser nailed at 60% can vanish at 80%, then reappear at 85%. There's a dead-band. So the certification harness reads the label at several crop scales at once - a "crop pyramid" from ~35% to 85% - and takes the best hit. A well-chosen pyramid of native center-crops beats any single crop, and beats naively upscaling the image. That one finding is what made "identify on the phone" viable at all.
Stage 2 - turn text into candidates (BrandMatcher)
Printed labels are tiny and cameras are shaky, so the OCR output is misspelled more often than not - "oticon" comes back as "oricon", "moxi" as "movi". The BrandMatcher expects this. It does fuzzy matching (Levenshtein edit-distance) against the known brand and model list, and - crucially - it can go backwards: spot a model name and infer the brand, even if the brand word never made it through the camera.
Stage 3 - the elimination tree (DeviceIndex)
This is the cleverest part, and the least obvious. Think of the whole catalogue of hearing aids as a room full of suspects. Every piece of evidence - a matched brand, a legible model, a detected colour - doesn't pick a device; it eliminates the ones that don't fit. When only one candidate remains for a field, that field locks itself. And locked fields cascade: pin the battery size and the power type often follows for free; pin the style and the tubing narrows.
DeviceIndex ranks every clue by how much to trust it - a clear OCR read outranks a hesitant neural-net guess - and refuses to let weaker evidence overturn a lock. But it isn't stubborn: if the same contradicting value shows up two frames running, it treats that as a real correction and re-opens the field. Confidence with a backspace key.
Stage 4 - the neural net (once, and only once)
Reading text handles most devices, but some labels are worn, embossed, or simply blank. For those, the app ships a trained neural network - an EfficientNet-B0 brand classifier, a 4.7 MB TensorFlow Lite model - that guesses the brand from what the device looks like. It's expensive relative to a frame filter, so it doesn't run on every frame. It fires exactly once, on a single full-resolution photo, after the colour detector has settled - and even then its vote only counts if it clears a confidence bar. It's a tie-breaker, not the star.
◆ The constraint that shapes everything
The camera pipeline is single-threaded. Every pretty animation, every extra filter, every eager neural-net call competes with OCR for the same frames. Early on, detection took 15+ seconds because visual polish was quietly eating the frame budget. The governing rule now: detection throughput is sacred. You don't add anything to the per-frame loop without measuring the milliseconds it costs. That's why the neural net runs off the hot path, and why colour detection doesn't even start until OCR has seen some text.
Stage 5 - the HUD, and the handoff
All of this is happening while a heads-up display shows seven fields - brand, model, style, tubing, power, battery, colour - as little slot-machine reels that shrink as candidates get eliminated. The moment brand and model are both locked, the scanner has done its job. It packages what it found, tears down the camera, and routes to the confirmation screen. It does not phone home.
05The plot twist: the cloud brain we retired
Here's the part that surprises engineers. Recycled Sound started as a cloud-inference app. Two Cloud Functions did the heavy lifting: a Python one (clip_encode) that turned a photo into a 512-number "visual fingerprint" using OpenCLIP, and a TypeScript one (analyzeHearingAid) that ran Google's Vision API for labels and text. The phone sent up a picture and got back an answer. It worked.
And then it got demoted to a fallback.
flowchart TB
Start["📷 A photo"] --> Q{"Live camera,
or picked from gallery?"}
Q -- "live scan" --> ON["On-device path
OCR → match → elimination → NN"]
ON --> Conf["/scan/confirm"]
Q -- "gallery pick
(no live detection)" --> CLOUD["Cloud path
clip_encode + analyzeHearingAid
fused on-device"]
CLOUD --> Conf
classDef on fill:#e7f3ec,stroke:#1c8a63,color:#0e5c40;
classDef cl fill:#fbe9e2,stroke:#e0603c,color:#a13c1f;
class ON,Conf on; class CLOUD cl;
Why demote a working cloud pipeline? Three reasons, and they're the moral of the whole project:
Speed. A network round-trip is seconds you don't have when a volunteer is holding a fiddly device up to a phone. On-device is instant.
Cost. The Python function's own source comment brags about it: on-device cosine-similarity search against bundled fingerprints is "$0/month infrastructure." A charity's cloud bill at rest should be zero, and it is.
The domain gap. The visual-fingerprint approach was trained on clean product photos. Real photos are hand-held, badly lit, and half-occluded by fingers. It turned out the printed text - read locally - was simply the stronger signal. So the architecture followed the evidence: OCR-first, with a visual fallback for the rare label you can't read.
The cloud functions aren't dead - they still deploy, and they still catch the gallery-pick case. They just aren't the main road anymore. It's a rare and honest thing to see in a codebase: a good idea, kept exactly as big as it deserves to be.
06From a scan to labelled training data
Identifying a device is half the point. The other half is photographing it well - both to document the physical item and to feed the next generation of models. This is the capture flow, and it hides a subtle idea.
sequenceDiagram
autonumber
participant Sc as Scanner
participant Cf as Confirm screen
participant Se as CaptureSeed
participant Ca as Capture flow
participant St as Storage
participant Fs as Firestore
Sc->>Cf: 7-field ScanResult
Cf->>Cf: audiologist corrects & confirms
Cf->>Se: label + scanId + box number
Se->>Ca: seed the capture
loop 12 shots - 6 angles × left/right
Ca->>St: captures/{uid}/{deviceId}/{side}_{slot}.jpg
end
Ca->>Fs: incoming/{id} - fully labelled
CaptureSeed, so every one of the 12 photos lands tagged with the audiologist's ground truth - not just a box number.The 12 shots aren't arbitrary. Each aid gets photographed from six named angles - the two broad sides, the thin front and back edges, the top where the hook or tube comes out, and the underside with the battery door. Two aids, six angles: twelve frames. The filenames encode the angle (left_medial.jpg), so a photo's identity survives even if the sequence is interrupted - an earlier position-based scheme silently mislabelled photos when a step was retaken.
One of those six is worth a note, because getting it wrong cost us real volunteer confusion. Earlier copy pinned the brand lettering to the medial face and told volunteers the other side was the wrong one. It isn't reliably true: the label can sit on either broad side depending on the make. The app now says "usually the inner side, but check both", and OCR runs on every shot rather than only the one we expected to be readable.
◆ The subtle idea
For a long time, the photos and the labels lived on two different paths. You could scan-and-confirm a device (getting a rich label but no photos), or you could capture-photos-for-later (getting 12 images tagged with nothing but a box number and a guessed brand). Pulling the photo bucket gave you "12 angles of box C25, maybe an Oticon" - useless as training data. The fix was to make the confirmed label travel with the pixels: the CaptureSeed carries the audiologist's make, model, style, battery - and a link back to the original scan - into the capture. Now every bundle is labelled ground truth. Recycled hardware, producing recycled data, to train better models. The loop closes.
07The backend: the spine we replaced, and a lifecycle
Everything the apps produce lands in Firebase. Two ideas make the backend worth explaining: how it handles identity - which we got wrong once, in an instructive way - and how a device moves through its life.
The spine we had to retire
The original design was elegant and we were fond of it. The very first thing the app did was sign you in anonymously: before you had typed a thing you had a stable user ID, everything you scanned was owned by that ID, and the rules could still enforce "you can only edit what you created." Zero friction at a working bee, real ownership underneath. If you signed up later, linkWithCredential upgraded the account in place and you kept every device you had logged.
The flaw is visible the moment you say it out loud: anyone who downloaded the app got a writable identity. Not a volunteer - anyone. The intake collection that the whole charity's records are built on was writable by any stranger who installed the app. Convenience for a volunteer and an open door for everyone else turned out to be the same mechanism.
So it is gone. Access is now invite-gated: an admin mints a single-use code, a volunteer redeems it (redeemInvite), and roles are granted rather than self-assigned. Signing up gets you an account in pending with the role you requested; an admin confirms it before you can do anything.
signInAnonymously() is a no-op for every install that already exists - Firebase persists the anonymous uid across an app update, so those users would have opened the app already "signed in", with no profile, no role, and tightened rules silently denying every write. A working app, against a backend that refuses it, with no screen explaining why. Production measurement said it would have hit 100% of users: all 88 documents, 11 of 11 owner uids, anonymous. The fix had to evict the persisted session, not merely stop minting new ones - and the predicate is anonymity specifically, because a named volunteer signing back in between boxes is exactly the friction that would make the app unusable where it is actually used.
flowchart LR
Open["Open the app"] --> Evict{"Persisted session
anonymous?"}
Evict -- yes --> Out["Signed out
(evicted on boot)"]
Evict -- "no, named" --> Keep["Session kept"]
Out --> In["Sign in / sign up"]
In --> Code["Redeem invite code"]
Code --> Pend["users/{uid} - pending
+ requestedRole"]
Pend --> Conf["Admin confirms"]
Conf --> Role["Role claim
volunteer · audiologist · admin"]
Keep --> Role
classDef g fill:#e7f3ec,stroke:#1c8a63,color:#0e5c40;
classDef r fill:#fdeaea,stroke:#c0392b,color:#7b241c;
class Keep,Role,Conf g;
class Out r;
Roles (volunteer, audiologist, admin) live in Firestore as the single source of truth, and a Cloud Function called syncRoleClaim mirrors your role onto the auth token that the security rules actually read - because a rule can only cheaply see the token, not the database. Role changes go through an admin-only callable, setUserRole; self-promotion is impossible by any write verb. A volunteer can create records; only an elevated role - audiologist or admin - can promote a device into the curated register.
The device lifecycle
A donated aid isn't just "in the database" - it's somewhere along a journey, and the record knows where. This is a plain state machine, and it's the backbone of the whole charity's operations.
stateDiagram-v2
direction LR
[*] --> Donated
Donated --> Reprogramming: audiologist claims it
Reprogramming --> Ready: QA passed
Ready --> Shipped: matched to a recipient
Shipped --> Delivered
Delivered --> Active: fitted & working
Active --> [*]
A note on the data model, for the type-system curious
A device is a 26-field record, and the same shape lives in two collections: incoming/ (what the scanner writes, pre-triage) and devices/ (the audiologist-curated register, post-triage). The closed-set clinical fields - style, battery size, tubing, power source - are real enum types with tolerant parsing, so a stray string from an old record heals to a known value instead of crashing.
There's a small, satisfying piece of type discipline here: the pre-saved draft type (DraftDevice) has no ID field at all. An ID only exists once the database assigns one, so the code makes it impossible to hold a half-real device with a fake empty ID. The transition is a one-way door: draft.toDevice(id). You can't accidentally lie about whether something's been saved.
08The brain, offline
The phone runs the models. But where do the models come from? That's a separate, offline pipeline - Python scripts that never ship to a user, whose only job is to produce the assets the app bundles.
flowchart LR Scrape["🌐 Scrape catalogue
~2,300 product photos"] --> Clip["CLIP embed
512-dim fingerprints"] Scrape --> Dna["Device DNA
ChromaDB vector store"] Clip --> Assets["📦 device_db.bin +
device_catalog.json"] Train["Train EfficientNet
brand classifier"] --> Tflite["brand_classifier.tflite
(4.7 MB)"] Certify["Certify OCR-first
crop-pyramid harness"] -.validates.-> Assets Assets --> App["📱 Bundled on-device"] Tflite --> App classDef d fill:#eef1ec,stroke:#5f6f68,color:#2a3a34; classDef a fill:#e7f3ec,stroke:#1c8a63,color:#0e5c40; class Scrape,Clip,Dna,Train,Certify d; class Assets,Tflite,App a;
The centrepiece is the device DNA: for every catalogue device, the pipeline builds a rich profile with four facets - a visual signature, a technical "genome" (its specs), its capability bits (remote-tuning, app support, Auracast-readiness), and a suitability profile. That DNA lives in a vector database (ChromaDB) so the eventual matching step can ask "which donated devices are most like what this recipient needs?" - a nearest-neighbour search, not a rules engine.
Two other jobs matter. A linear probe is the honesty check: it trains a dead-simple classifier on the frozen visual fingerprints to ask "does CLIP even carry enough signal to tell brands apart?" - before anyone invests in the bigger model. And the certification harness is what proved the crop-pyramid OCR approach cleared a pre-committed 90% brand-accuracy bar, which is the evidence that justified going on-device-first in the first place. The team measured before it committed.
09How it all ships
Four artifacts, four release paths - and a deliberate split between "safe to automate" and "a human presses the button."
flowchart TB Main["git · main branch"] --> CI["ci.yml (on every push)"] CI --> Rules["✅ Firestore + Storage rules
auto-deploy"] CI --> Fns["✅ Cloud Functions (vision)
auto-deploy"] Main -. "manual button" .-> TF["🍎 iOS → TestFlight"] Main -. "manual button" .-> Play["🤖 Android → Play (internal)"] Main -. "manual build" .-> Host["🌐 Web → Firebase Hosting"] classDef auto fill:#e7f3ec,stroke:#1c8a63,color:#0e5c40; classDef man fill:#fbe9e2,stroke:#e0603c,color:#a13c1f; class Rules,Fns auto; class TF,Play,Host man;
The split is the interesting bit. Security rules and cloud functions auto-deploy on a green build - they're versioned, testable, and instantly reversible. But the mobile builds are gated behind a human, on purpose: an accidental release to real testers isn't something you can quietly undo. The pipeline treats "notify humans" as the higher-stakes action, which is exactly backwards from most people's intuition and exactly right.
10The human in the loop
We've spent nine sections on machinery that reads labels, eliminates candidates, runs neural nets, and matches devices to people by nearest-neighbour search. So it's worth ending on the one rule that overrides all of it:
No automated system ever touches a recipient.
The recipients here are refugees and asylum seekers - people for whom a wrong device or a mishandled message isn't a bad user-experience, it's a real harm to someone with very little margin. So the scanner, for all its cleverness, is explicitly a co-pilot. It pre-fills; the audiologist confirms. It suggests a match; a human approves it. Matching starts fully manual - an admin selects, the system merely notifies. Every recipient-facing workflow is human-mediated by design, and stays that way until the manual version has been proven safe.
That's the quiet engineering decision underneath the flashy one. The hard part wasn't teaching a phone to recognise a hearing aid - impressive as that is. The hard part was building all of that power and then drawing a firm line around where it's allowed to act alone. The AI removes the tedious reading. The human keeps the deciding. That's the whole design, in one sentence.
Recycled Sound is a community initiative of the Rotary Club of Springvale City and Arches Audiology, collecting, servicing, and redistributing donated hearing aids in Greater Dandenong, Victoria. If a diagram above made you want to poke at the code, that was the idea.