* reset local start time at each recording boundary
* include the final frame's time when converting from wall to media time
Together at 30 fps these caused about 15 seconds of drift over 4 hours.
As noted in the changelog, this problem caused restarts to take a long
time, and the SQLite database to (slowly) grow without bound.
The problem was simple: the directory worker incorrectly aborted
`collect_garbage` if nothing was watching the result. For some
operations (e.g. reading from sample files) this is the correct
behavior, but in this case it contradicts the `Pool::collect_garbage`
contract.
This puts a lot of state into a new `LockedStream` that was formerly
guarded more coarsely by the recording lock or more granularly by
`RecordingToInsert` locks. And it restructures `LockedDatabase::flush`
to avoid holding this lock while performing SQLite operations.
Streamers write only into a recent frames buffer in RAM, which the
directory pool workers asynchronously write to disk approximately once
per GOP, where formerly there was a write call per frame.
Before, streamers could fall behind due to slow sample file directory
disk I/O calls or if writing got stuck in a retry loop, for example if
out of disk space. If Moonfire falls too far behind in receiving,
cameras tend to disconnect or (sadly) mis-frame messages because they do
not handle this case well. This removes a confusing potential cause of
such errors.
In the new model, streamers never wait for disk I/O. Frames are buffered
up to a limit. Recordings are aborted if the I/O pool falls too far
behind. There's no retry loop.
This roughly halves CPU usage as thread hand-offs and syscalls are less
common. It also enables several desirable future changes:
* audio recording (which radically increases the number of frames and
thus would otherwise significantly increase CPU usage).
* "monitor mode" (allowing live view without recording to disk),
* further CPU reductions for live view (by skipping all SQLite ops and
disk I/O).
In the new model, streamers also rarely wait for SQLite database
operations either—now only on video parameter changes. That might go
away in an upcoming change too.
The "syncer" (which synced files/dirs and kept track of what has to be
flushed into the SQLite database when) has mostly gone away too. The
directory pool workers handle the syncing now, and then they notify
a new "flusher" for the latter portion.
I'd like to go further still and have the database as a whole be owned
by a dedicated thread, the only one that uses the SQLite connection.
This prevents the tokio threads from blocking on SQLite operations.
I tried out doing so before making the streamers less dependent on the
db, but the extra thread handoffs were awful for performance.
Now, all IO happens on each directory's two-thread pool with threads
named `dir-<PATH>`. Formerly, IO could happen on the per-stream writer
threads, per-directory reader threads, and occasional on even the tokio
worker threads.
The recording path uses more CPU than before due to extra thread
hand-offs. That will change in the next commit, which will radically
change the approach to writing.
* add `/var/tmp` to the image so it doesn't have to be mounted
* remove obsolete `version: 3` from compose snippet
* mention `/usr/share/zoneinfo` in error msg and troubleshooting guide
...to:
* 20 (maintenance LTS)
* 22 (maintenance LTS)
* 24 (active LTS release)
* 25 (current non-LTS release)
And address some "realm mismatch" problems in tests:
* It was using `AbortSignal` from `jsdom`'s realm, `fetch` from the Node
native `fetch`. Node 24 upgraded to undici v7 with stricter Fetch spec
compliance, which complained about this.
* It was doing `instanceof` checks from `myfetch` which failed for
similar reasons.
by updating vitest (see <https://github.com/vitest-dev/vitest/issues/8374>) and simplifying the `api.ts` checking.
Along the way, give error messages like `POST /api/login failed; see browser console` rather than the unhelpful `TypeError: Failed to fetch`.
This is a step toward H.265 recording support (#33) by using Retina
for the `VisualSampleEncoding` instead of the prior H.264-specific
logic. The only further change to Moonfire itself needed for H.265 is to
add `| "h265"` to the check for the right video stream.
This also adds support for recording MJPEG, although browser playback
almost certainly won't work.
* strongly encourage the single-binary approach and say why.
* fix a broken link in troubleshooting guide (and regenerate toc).
* add a couple more comments to the docker compose snippet
* use a more recent screenshot for selecting tag on github UI, and
put in a `<details>` because it's annoyingly large.
* put a border around `<details>` so it's easy to see where it
starts/ends
* add troubleshooting info for docker setup, to help with e.g. #296.
Systemd syntax expects comment in a single line, hence the parsing failure
https://www.freedesktop.org/software/systemd/man/latest/systemd.syntax.html
Before;
```
[leandro@nuc ~]$ sudo systemd-analyze verify /etc/systemd/system/moonfire-nvr.service
/etc/systemd/system/moonfire-nvr.service:16: Failed to parse TimeoutStartSec= parameter, ignoring: 300 # large installations take a while to scan the sample file dirs
[leandro@nuc ~]$
```
After;
```
[leandro@nuc ~]$ sudo systemd-analyze verify /etc/systemd/system/moonfire-nvr.service
[leandro@nuc ~]$
```
* use `termion` rather than `ncurses` to limit runtime deps
* cross-compile with `cross` instead of our own dockerfiles/scripts
* update instructions
* update release procedure and GitHub actions to match
* prep changelog for `v0.7.8`
Fixes#160Closes#265
I think this is a big improvement in readability.
I removed the `lnav` config, which is a little sad, but I don't think it
supports this structured logging format well. Still seems worthwhile on
balance.
This gives much better information to the UI layer, getting rid of a
whole troubleshooting guide entry. See #119#132#218#219
I also restructured the code in anticipation of a new WebSocket event
stream (#40).
The build instructions currently fail if the user doesn't have a group
matching their user name (not a universally adopted convention). Update
the command to use the following functionality of the `chown` command:
"If a colon but no group name follows the user name, that user is made
the owner of the files and the group of the files is changed to that
user's login group."
* support cross-compiling an x86-64 target on an arm64 host. This
it turns out is a matter of *removing* an unnecessary dependency.
(aarch64-linux-gnu-pkg-config exists but x86_64-linux-gnu-pkg-config
doesn't. Turns out neither is necessary.) Added a comment explaining
where ${gcc_target}-pkg-config comes from now.
* documentation tweaks
* improve debug output a bit
Scott Lamb
michioxd
Leandro Silva
Skye
Tim Small