A sensor system is most valuable when conditions are least convenient: during a storm, a power cut, or an internet outage. That is also when a design that depends on a permanent cloud connection is most likely to lose the evidence we wanted it to capture.
This network therefore uses a local-first, centrally processed architecture. Raspberry Pi computers at the monitored locations collect and timestamp readings close to the sensors. They keep a local history and try to synchronize raw data to the public server every minute. If the route to the server disappears, collection continues; when connectivity returns, the queued history can catch up.
Why collect locally?
The Raspberry Pi is a boundary between small sensor devices and the wider internet. Ethernet, Power over Ethernet (PoE) and Wi-Fi devices can all report to a nearby collector. A 433 MHz radio path is also planned for environments where cabling or Wi-Fi is unsuitable.
This division lets the sensor hardware stay relatively simple and inexpensive. A node does not need a precise real-time clock of its own: the collector timestamps its reading using the Raspberry Pi clock, and that clock is aligned to network time through NTP. The important rule is that time is assigned consistently at the edge, before data is transported elsewhere.
Each collector retains up to 30 days of local data. Every minute it samples host health and relevant attached devices, then attempts to upload the raw files. An internet outage delays delivery but does not stop observation. Thirty days is not an arbitrary promise of infinite storage; it is a practical recovery window, enforced by a daily pruning job so memory and persistent storage remain predictable.
The active sensor-data tree on each collector is a RAM disk (tmpfs): 1 GB at the UK location and 1.5 GB at the Swedish location. Frequent samples, appends and regenerated files therefore occur in memory rather than continually writing to the Raspberry Pi storage. This reduces flash wear and makes it inexpensive to refresh working files often.
Because a RAM disk is volatile, it is paired with a persistent mirror. The collector copies its live sensor directories to persistent storage every day and during an orderly shutdown, then restores that mirror into RAM when it boots. An abrupt loss of power can still discard RAM-only changes made since the last persistent copy; in normal operation, however, the minute-by-minute upload means most of those readings have already reached the central server. The persistent mirror protects local restart continuity, while server synchronization and off-site archives provide the wider recovery path.
The one-minute data flow
- Measure: services read environmental, motion, power, daylight and system-health sensors.
- Timestamp: the local Raspberry Pi assigns time from its NTP-aligned clock.
- Persist locally: raw readings are written before any internet dependency enters the path.
- Synchronize: an upload job runs every minute. Existing files are synchronized so delayed readings can arrive after a connection recovers.
- Ingest and validate: the central server receives the raw hierarchy and checks whether fresh host data has arrived.
- Materialize and publish: the server creates live output every minute and performs fuller historical processing every 15 minutes.
There is an important distinction between measurement time and arrival time. A cached reading may reach the server later, but it retains the timestamp assigned when it was observed. Keeping both concepts makes delayed synchronization honest rather than making old data look new.
Outages become data too
The regular upload is also a heartbeat. The central server records a successful minute only when both the latest Raspberry Pi host sample and its uploaded file are no more than three minutes old. If those heartbeats stop, the gap can indicate an internet failure, an electrical outage, a failed collector, or a broken upload path.
A heartbeat alone cannot prove which of those causes occurred. Diagnosis comes from correlation: compare locations, inspect the last host metrics, check whether mains-powered devices stopped together, and see whether cached measurements arrive later. If the missing interval is backfilled after reconnecting, the sensors and collector were probably alive while the route to the server was not.
Why centralize the expensive work?
Local collection does not mean every edge device must build charts, compress history and calculate statistics. The public server has more CPU, storage and ingestion capacity, so it performs the heavier work for all locations. At present it regenerates recent live pages each minute, while compression, timestamp synchronization and full-page generation run every 15 minutes.
Compress first, without losing detail
Completed historical CSV files are compressed with Zstandard at level 10 before they move further through the storage lifecycle. Sensor CSV is highly repetitive—timestamps, identifiers and similar numeric values recur on almost every line—so it compresses particularly well. A sample of completed files from both collectors reduced their combined size by about 89%, making the compressed representation roughly nine times smaller than the original CSV.
This is lossless compression: decompression recreates the original readings rather than approximating them. It reduces collector memory use, synchronization traffic, central-server storage and upload time. The nightly raw archive uploads the completed files in their compressed .zst form to Backblaze B2, so the same reduction also lowers off-site storage and transfer requirements. The newest active CSV remains uncompressed while readings are still being appended.
Raw and still-changing data remains on the filesystem. This matters because statistics for the current hour, day, month and year are continuously revised as new readings—and sometimes delayed readings—arrive. A late reading can change every aggregate that contains its timestamp, from an hourly average through the annual high, low or total. Finalizing any of those periods too early would produce stale statistics.
Materialized data is being migrated to MySQL. The useful pattern is a staged lifecycle:
- keep recent mutable source data in a form that is cheap and safe to append or synchronize;
- recompute affected aggregates when late data arrives;
- materialize stable summaries in the database for efficient queries;
- retain enough raw history to rebuild derived values when processing logic changes.
The server also takes frequent filesystem snapshots, while sensor history is backed up off-site to Backblaze B2. Local cache, central storage and off-site backup solve different failure modes; none is a substitute for the others.
From raw reading to safely pruned history
Once a day, the server runs a serialized storage pipeline. Its order is deliberate:
- Archive raw files: completed raw sensor files are uploaded to Backblaze B2. This archive is append-only in normal operation: the job uploads objects that are missing and does not synchronize deletions to B2.
- Materialize settled data: historical values whose calculation is no longer expected to change are processed into MariaDB.
- Back up the database: the aggregate database is dumped, compressed, checksummed and uploaded to B2.
- Prune eligible local files: only raw files that have passed all of the safety checks may be removed from the server filesystem.
Deletion is intentionally conservative. A local raw file is eligible only when its source, unit and day have a completed record in MariaDB and that exact file is present in B2. If either check fails, pruning stops rather than guessing. The newest incomplete day is excluded from backup materialization and pruning because readings may still be arriving for it.
This creates three complementary storage tiers: B2 is the long-term raw source of truth, MariaDB is a fast and rebuildable aggregate cache, and the local filesystem holds current or unsettled raw data. The order prevents a derived database row from becoming the only surviving representation of a measurement.
Not every source settles at the same speed
A day boundary does not necessarily make data final. Some calculations depend on a longer moving or age-dependent window. Swedish wind data is a useful example: recent raw readings remain on the filesystem for 32 days because the wind aggregation algorithm uses different bucket sizes as the data ages. After that window, its hourly representation is stable enough to materialize and the normal archive-and-prune checks can apply.
This is a general design lesson: settlement should be defined per data source and algorithm, not by one global retention rule. If a calculation changes with context, keep its raw inputs until the result becomes reproducible and stable.
Database fallback and recovery
Normal page generation queries MariaDB first for materialized historical periods, then falls back to raw files for missing or unsettled periods. A database outage therefore does not make the raw format obsolete. If necessary, archived files can be restored from B2 into the normal data hierarchy, pages can be generated directly from those files, and MariaDB can be rebuilt by running the materialization process again.
The MariaDB backup has two retention roles. A replaceable latest dump supports routine recovery, while one snapshot per month provides longer-lived restore points. Each dump is accompanied by a SHA-256 checksum and metadata recording its creation time, row counts and newest materialized source day. Those details make it possible to verify both file integrity and how current the restored database will be.
Keeping scheduled work from colliding
Several jobs operate on the same data: live generation every minute, full generation and filesystem snapshots every 15 minutes, and the nightly archive, materialization, database-backup and pruning pipeline. The generation and storage workflows use flock lock files so a second copy of the same operation cannot overlap an active run; the raw and database backup steps also carry their own locks.
Live generation uses a non-blocking lock and skips a cycle if the previous one is still busy; more complete work can wait for its lock. The nightly pipeline also holds one outer lock while its ordered stages run. This small amount of concurrency control prevents two processes from rewriting generated output, materializing the same partition or pruning while another operation is inspecting the files.
One server, multiple locations and views
The central server processes the UK and Sweden independently. Each location has its own raw-data root, generation logs, locks and published directory, so a slow or unusual data source in one place does not merge the two histories.
Rather than performing expensive queries for every browser request, the processing jobs create specialized static pages. Depending on the sensors available at a location, these cover live readings, temperature, humidity, electrical power, internet and electrical outages, Raspberry Pi health, wind, daylight, barometric pressure and unit internals. Static publication makes browser delivery inexpensive while the server performs aggregation on its own controlled schedule.
This differs from systems that query a database separately for every visitor. Here, MariaDB is an internal input to the generation process: one build performs the historical lookups and produces a shared static view that can be served to everyone. A browser request therefore needs neither a new aggregate query nor a live MariaDB connection. That reduces response latency and database load, makes traffic spikes much cheaper to absorb, works especially well with Cloudflare caching, and allows already-generated pages to remain available during a temporary database failure.
Web-server operations
Lighttpd serves the generated pages over HTTPS. Certificate renewal is scheduled automatically, and HTTP compression is enabled to reduce transfer sizes—particularly useful for pages containing substantial chart data. Cloudflare remains the public-facing proxy layer, while Lighttpd is the origin responsible for serving the generated files and terminating protected connections that reach it.
Publishing without exposing the origin
Visitors reach the site through Cloudflare's proxy network rather than connecting directly to the origin web server. The proxy provides an outer layer of caching and traffic filtering, hides the normal origin path from browsers, and absorbs much of the unwanted traffic before it reaches the small server.
That distributed cache is also an important part of scaling the site. Static pages, SVG diagrams, photographs, video, CSS and JavaScript can be stored at Cloudflare edge locations close to visitors. After an asset has entered a regional cache, later requests can be answered by Cloudflare without fetching it from the origin again. This reduces origin bandwidth, connections, disk activity and CPU work while usually lowering latency for the visitor.
The static shared-view architecture is particularly cache-friendly. One server-side build produces a page that can be reused for every visitor; Cloudflare can then distribute that same result to a much larger audience without each browser causing page generation or a MariaDB query. A sudden increase in readers is therefore handled mainly by the edge network rather than becoming an equal increase in origin load.
Cache lifetime still needs to match the content. Versioned media and site assets can be cached for a long time because a changed file receives a new URL, while frequently regenerated live sensor pages need shorter expiry or explicit revalidation. Cached material may also remain available through some brief origin disruptions, but uncached and expired requests still depend on the origin.
This reduces the effect of common denial-of-service traffic, but it is not magic. A similar deployment should restrict the origin firewall to trusted administration paths and Cloudflare's published proxy ranges, keep software patched, authenticate ingestion separately from public reading, apply rate limits, and monitor origin traffic for bypass attempts.
A practical blueprint
If you are designing a similar system, begin with failure behaviour rather than a shopping list.
- Write down the outage contract. Decide how long the edge must run without internet, how much data it must hold and what happens when storage fills.
- Persist before transmitting. A successful sensor read should survive a failed upload or reboot.
- Use idempotent synchronization. Retrying the same file or record must not create duplicate measurements.
- Separate observed time from received time. Store timestamps in UTC, synchronize collector clocks, and record when the server actually received the data.
- Expose health as data. Upload collector CPU, memory, disk, clock and heartbeat measurements alongside environmental readings.
- Design for late arrivals. Recompute only the time partitions affected by backfilled data and finalize aggregates after an intentional delay.
- Test recovery. Disconnect the WAN, collect several hours of readings, reconnect it and verify that history fills in without duplicates or misleading timestamps.
- Back up and restore. Encrypt off-site backups, define retention, and rehearse restoring them to a clean machine.
The broad lesson is simple: cloud connectivity should improve a sensor network, not be a precondition for sensing. Collect close to the physical event, preserve raw evidence, move it when possible, and do expensive shared work where the resources are strongest.