This puts a lot of state into a new `LockedStream` that was formerly
guarded more coarsely by the recording lock or more granularly by
`RecordingToInsert` locks. And it restructures `LockedDatabase::flush`
to avoid holding this lock while performing SQLite operations.
Streamers write only into a recent frames buffer in RAM, which the
directory pool workers asynchronously write to disk approximately once
per GOP, where formerly there was a write call per frame.
Before, streamers could fall behind due to slow sample file directory
disk I/O calls or if writing got stuck in a retry loop, for example if
out of disk space. If Moonfire falls too far behind in receiving,
cameras tend to disconnect or (sadly) mis-frame messages because they do
not handle this case well. This removes a confusing potential cause of
such errors.
In the new model, streamers never wait for disk I/O. Frames are buffered
up to a limit. Recordings are aborted if the I/O pool falls too far
behind. There's no retry loop.
This roughly halves CPU usage as thread hand-offs and syscalls are less
common. It also enables several desirable future changes:
* audio recording (which radically increases the number of frames and
thus would otherwise significantly increase CPU usage).
* "monitor mode" (allowing live view without recording to disk),
* further CPU reductions for live view (by skipping all SQLite ops and
disk I/O).
In the new model, streamers also rarely wait for SQLite database
operations either—now only on video parameter changes. That might go
away in an upcoming change too.
The "syncer" (which synced files/dirs and kept track of what has to be
flushed into the SQLite database when) has mostly gone away too. The
directory pool workers handle the syncing now, and then they notify
a new "flusher" for the latter portion.
I'd like to go further still and have the database as a whole be owned
by a dedicated thread, the only one that uses the SQLite connection.
This prevents the tokio threads from blocking on SQLite operations.
I tried out doing so before making the streamers less dependent on the
db, but the extra thread handoffs were awful for performance.
* add `/var/tmp` to the image so it doesn't have to be mounted
* remove obsolete `version: 3` from compose snippet
* mention `/usr/share/zoneinfo` in error msg and troubleshooting guide
Since buildx v0.10.0, provenance attestations are enabled by default,
causing even single-platform `docker build --push` to produce a
manifest list (OCI image index) rather than a plain image manifest.
This breaks the subsequent `docker manifest create` step, which
expects plain image manifests as inputs:
ghcr.io/scottlamb/moonfire-nvr:v0.7.24-x86_64 is a manifest list
Fix by passing `--provenance=false` to suppress the attestation,
ensuring each per-arch tag is a plain image manifest.
Ref: https://github.com/docker/buildx/issues/1507
...to:
* 20 (maintenance LTS)
* 22 (maintenance LTS)
* 24 (active LTS release)
* 25 (current non-LTS release)
And address some "realm mismatch" problems in tests:
* It was using `AbortSignal` from `jsdom`'s realm, `fetch` from the Node
native `fetch`. Node 24 upgraded to undici v7 with stricter Fetch spec
compliance, which complained about this.
* It was doing `instanceof` checks from `myfetch` which failed for
similar reasons.
by updating vitest (see <https://github.com/vitest-dev/vitest/issues/8374>) and simplifying the `api.ts` checking.
Along the way, give error messages like `POST /api/login failed; see browser console` rather than the unhelpful `TypeError: Failed to fetch`.
This is a step toward H.265 recording support (#33) by using Retina
for the `VisualSampleEncoding` instead of the prior H.264-specific
logic. The only further change to Moonfire itself needed for H.265 is to
add `| "h265"` to the check for the right video stream.
This also adds support for recording MJPEG, although browser playback
almost certainly won't work.
* use `termion` rather than `ncurses` to limit runtime deps
* cross-compile with `cross` instead of our own dockerfiles/scripts
* update instructions
* update release procedure and GitHub actions to match
* prep changelog for `v0.7.8`
Fixes#160Closes#265
This is optional but now enabled for release builds.
Why?
* It shrinks the release docker images a bit, as the binary
includes only the gzipped version of files and uncompressed into RAM
at startup (which should be fast).
* It's a step toward #160.
For #210
* update changelog format so that `parse-changelog` is happy.
* add the example from taiki-e/create-gh-release-action verbatim.
Fingers crossed it works in my setup.
I did a full `cargo upgrade` and fixed what it broke:
* a couple things for the latest protobuf 3.0 alphas
(note alphas don't promise API stability)
* new minimum supported Rust version
This should have some other nice effects: parking_lot now uses inline
assembler, tokio has gotten faster, etc.
1c9a556 started depending on arrays implementing IntoIterator. I keep
forgetting that 1.52 doesn't support this. There's really no compelling
reason to keep building on 1.52, so don't.
It seems to be failing spuriously now, eg:
https://github.com/scottlamb/moonfire-nvr/runs/3957348093?check_suite_focus=true
```
error[E0277]: the trait bound `dyn view_trait::View: view_trait::View` is not satisfied
--> /home/runner/.cargo/registry/src/github.com-1ecc6299db9ec823/cursive_core-0.2.2/src/views/list_view.rs:98:14
|
98 | view.take_focus(direction::Direction::none());
| ^^^^^^^^^^ the trait `view_trait::View` is not implemented for `dyn view_trait::View`
For more information about this error, try `rustc --explain E0277`.
error: could not compile `cursive_core` due to previous error
warning: build failed, waiting for other jobs to finish...
error: build failed
Error: Process completed with exit code 101.
```
* run node 12, 14, and 16 (next to be supported) on CI. This will catch
node version-specific problems like that solved in dad9bdc.
* mention 12 and 14 in build instructions and link to instructions for
installing that version.
* follow this in Dockerfile, installing version 14. This addresses
a "Cannot find module 'worker_threads'" error introduced in
39a63e0, which (inadvisedly) upgraded gzipper 4->5 in addition to
the material-ui upgrade.
* use utf-8 encoding rather than ascii in live part parser. Those
builds apparently don't support ascii. iThey must use "small-icu" or
have ICU disabled, as described here:
https://nodejs.org/api/util.html#util_encodings_supported_when_node_js_is_built_with_the_small_icu_option
* send folks to the issue tracker with the same template as bug
reports. Gathering this extra info up front should help things
move more quickly.
* in the template, ask for camera info.
As noted in mylog's 2b1085c:
Looks like both the GNU tools' --color argument and cargo's
CARGO_TERM_COLOR expect always/never rather than on/off. Match that.
Might as well understand off/no/false and on/yes/true also.
I'm tired of all the boilerplate, so use the new
GPL-3.0-linking-exception license identifier instead in all the server
components.
I left the ui stuff alone because I'm just going to replace it (#111).
Add a checker for the header because it's easy to forget.
I want to make the project more accessible by not expecting folks to
match my idiosyncratic style. Now almost [1] everything is written
in the "standard" style. CI enforces this.
[1] "Almost": I used #[rustfmt::skip] in a few sections where I felt
aligning things in columns significantly improves readability.
This eases build setup. Where Yarn requires a separate package
repository, npm is available in the standard one.
yarn's package repository signature was recently expired, and apparently
will expire again in a year. Avoid dealing with that.
Fixes#110.
Besides being more clear about what belongs to which, this helps with
docker caching. The server and ui parts are only rebuilt when their
respective subdirectories change.
Extend this a bit further by making the webpack build not depend on
the target architecture. And adding cache dirs so parts of the server
and ui build process can be reused when layer-wide caching fails.
Scott Lamb
michioxd
Skye