False-Negatives Pipeline & Distributed Media Sync
Flow & Architecture
In AI-driven computer vision systems for forest fire detection, accuracy relies not only on automated thermal PTZ scans, but also on continuous feedback loops. When an automated camera model misses an active smoke column or an operator visually identifies an unclassified thermal anomaly in the field, a False-Negative report must be filed immediately.
To ensure seamless synchronization across distributed edge client installations and our central monitoring platform, we architected a resilient, asynchronous pipeline connecting Saffira Edge Clients (on-premise or on-demand customer servers) with the Saffira-Admin Central Hub.
This feature integrates MongoDB multi-tenant document collections, Redis-backed BullMQ message workers, an automated FFmpeg background transcoding pipeline, and dual MinIO object storage buckets with presigned media security.
System Topology Overview
The diagrams below illustrate the full lifecycle — from GIS map reporting and local media conversion, through asynchronous BullMQ event dispatches, to cross-cluster presigned URL handshakes and HTTP range video streaming for operators.
Figure 1.1: Saffira-Admin + Messaging Layer + Client Instances — overall distributed architecture.
Figure 1.2: Video flow / false-negative creation — full sequence diagram.
Operator logs false-negative at GIS coordinate on map. Submits metadata payload + image/video files.
POST /api/falsenegativesStores document in falsenegatives collection. Generates unique fnKey linking media attachments.
Images ➔ Direct to Local MinIO.
Videos ➔ Temp Bucket ➔ Transcoded ➔ Local MinIO.
Figure 1.3: Interactive topology canvas of the Saffira ↔ Saffira-Admin false-negative reporting & media synchronization pipeline.
Technical Details
1. Frontend Map Submission & Multi-Media Upload
Operators initiate a false-negative report directly from the OpenLayers GIS map component. Selecting coordinates populates spatial metadata, weather conditions, and thermal intensity estimates.
The frontend submits the form alongside raw media attachments (images and high-definition field videos). The Saffira backend persists the core document in the client's local MongoDB falsenegatives collection, attaching a unique UUID tracking key (fnKey).
| Field | Type | Constraint | Purpose |
|---|---|---|---|
| fnKey | String | unique · indexed | UUID linking all media attachments to the report |
| clientId | String | required | Identifies the originating Saffira edge instance |
| coordinates | { lat, lng } | required | GIS map coordinate where the false-negative was logged |
| status | Enum | default: CREATED | CREATED → TRANSCODING → SYNCED_ADMIN → MEDIA_READY |
| attachments[] | Array | nested docs | Each item holds type (IMAGE | VIDEO), minioKey, isConverted flag |
2. Dual MinIO Buckets & FFmpeg Transcoding Pipeline
Handling media uploads at the edge requires strict separation between raw file ingestion and optimized streaming formats:
- Images: Uploaded directly to the client's definitive local MinIO bucket (
saffira-media-store). - Videos: Uploaded first to a temporary staging bucket (
saffira-temp-video-staging). An asynchronous FFmpeg background worker process is triggered to transcode raw camera feeds into standardized MP4 containers with H.264 video codec and AAC audio encoding.
3. Asynchronous BullMQ & Redis Queue Topology
Once the false-negative is registered locally, the Saffira backend dispatches an event to the BullMQ Redis queue targeting the central saffira-admin server.
The BullMQ worker running on saffira-admin consumes the message, executes a data pass-through validation, and inserts the record into the central MongoDB unified false-negatives collection with status PENDING_MEDIA.
| Queue Step | Producer / Service | Consumer / Receiver | Payload Action |
|---|---|---|---|
| 01. Report Created | Saffira Client Backend | Redis Queue (`fn-sync-queue`) | Publish { fnKey, clientId, coords } |
| 02. Admin Ingestion | BullMQ Admin Worker | Saffira-Admin Central DB | Insert unified `false-negatives` record |
| 03. Media Handshake | Saffira-Admin Sync Dispatcher | Saffira Client Backend | Request presigned URLs for `fnKey` |
| 04. Media Transfer | BullMQ Media Worker | Saffira-Admin Local MinIO | Fetch via presigned URL & store locally |
4. Cross-Cluster Presigned URL Handshake Protocol
To prevent cross-network firewall blocks or exposing private edge storage nodes to the internet, we implement a time-limited Presigned URL Handshake Protocol:
- When
saffira-adminis ready to sync media, its Sync Dispatcher publishes a command to BullMQ requesting media access forclientIdandfnKey. - The Saffira client backend receives the request. If video transcoding is completed and images exist in MinIO, it generates read-only presigned URLs (valid for 15 minutes) using the MinIO SDK.
- The presigned URLs are posted back to BullMQ. The
saffira-adminbackground worker consumes the URLs, streams the files directly into the centralsaffira-adminlocal MinIO bucket, and updates the MongoDB record status toREADY.
5. Operator UI Exhibition & HTTP Range Video Streaming
When operators review reported false negatives on the Saffira-Admin frontend dashboard, the application requests media access from the admin backend. The backend generates presigned URLs pointed at the central MinIO instance.
For video attachments, the browser leverages native HTML5 video player streaming with HTTP 206 Partial Content range requests. Operators can seek through high-resolution clips with zero buffer stutter, giving them complete visual context to retrain AI detection models.
Engineering Lessons
Building the false-negatives pipeline reinforced the necessity of asynchronous decoupling in distributed multi-tenant environments. By isolating heavy media transformations (FFmpeg transcoding) into background worker pools and passing lightweight presigned URLs over BullMQ Redis channels, we eliminated UI response latencies for field dispatchers.
Furthermore, using MinIO presigned URL handshakes ensured strict network security while keeping edge customer installations air-gapped from public cloud ingress bottlenecks. This architecture provided Saffira with a robust feedback loop to continually improve model accuracy across millions of monitored hectares.