23 Commits
Author SHA1 Message Date
Scott Lamb 316d3d8482 allow inserting VSEs without the db lock 2026-02-17 11:55:55 -08:00
Scott Lamb fd7ae68657 overhaul streams!
This puts a lot of state into a new `LockedStream` that was formerly
guarded more coarsely by the recording lock or more granularly by
`RecordingToInsert` locks. And it restructures `LockedDatabase::flush`
to avoid holding this lock while performing SQLite operations.

Streamers write only into a recent frames buffer in RAM, which the
directory pool workers asynchronously write to disk approximately once
per GOP, where formerly there was a write call per frame.

Before, streamers could fall behind due to slow sample file directory
disk I/O calls or if writing got stuck in a retry loop, for example if
out of disk space. If Moonfire falls too far behind in receiving,
cameras tend to disconnect or (sadly) mis-frame messages because they do
not handle this case well. This removes a confusing potential cause of
such errors.

In the new model, streamers never wait for disk I/O. Frames are buffered
up to a limit. Recordings are aborted if the I/O pool falls too far
behind. There's no retry loop.

This roughly halves CPU usage as thread hand-offs and syscalls are less
common. It also enables several desirable future changes:

* audio recording (which radically increases the number of frames and
  thus would otherwise significantly increase CPU usage).
* "monitor mode" (allowing live view without recording to disk),
* further CPU reductions for live view (by skipping all SQLite ops and
  disk I/O).

In the new model, streamers also rarely wait for SQLite database
operations either—now only on video parameter changes. That might go
away in an upcoming change too.

The "syncer" (which synced files/dirs and kept track of what has to be
flushed into the SQLite database when) has mostly gone away too. The
directory pool workers handle the syncing now, and then they notify
a new "flusher" for the latter portion.

I'd like to go further still and have the database as a whole be owned
by a dedicated thread, the only one that uses the SQLite connection.
This prevents the tokio threads from blocking on SQLite operations.
I tried out doing so before making the streamers less dependent on the
db, but the extra thread handoffs were awful for performance.
2026-02-17 11:55:55 -08:00
Scott Lamb 0dc285b159 move all sample file dir I/O to pool
Now, all IO happens on each directory's two-thread pool with threads
named `dir-<PATH>`. Formerly, IO could happen on the per-stream writer
threads, per-directory reader threads, and occasional on even the tokio
worker threads.

The recording path uses more CPU than before due to extra thread
hand-offs. That will change in the next commit, which will radically
change the approach to writing.
2026-02-17 11:17:28 -08:00
Scott Lamb 39b6f6c49c build releases with mimalloc 2025-04-03 10:06:50 -07:00
Scott Lamb 2903b680df clippy fixes to tests 2025-04-03 09:16:44 -07:00
Scott Lamb cbb2c30b56 use jiff crate 2025-01-22 14:16:31 -08:00
Scott Lamb 86816e862a update some Rust dependencies
I didn't go to quite the latest version of everything, in an effort to
minimize duplicates in the cargo tree.
2024-01-06 11:41:28 -08:00
Scott Lamb 2a8c1bb632 let libtest capture logs 2023-08-08 18:24:50 -07:00
Scott Lamb aa60bc991c test and fix #282
Sessions' last use updates weren't getting persisted to the database
because the update statement wasn't passing through the hash.

Also address a TODO of tracing in tests by using the same tracing
setup as in production.
2023-07-04 20:53:53 -07:00
Scott Lamb ebcdd76084 switch from log to tracing
I think this is a big improvement in readability.

I removed the `lnav` config, which is a little sad, but I don't think it
supports this structured logging format well. Still seems worthwhile on
balance.
2023-07-04 16:51:22 -07:00
Scott Lamb ae502200c0 upgrade various Rust dependencies
This stops using parking_lot entirely. Since Rust 1.62, the std
implementations on Linux are direct futexes, not the boxed pthread
mutexes they used to be. No real reason to use parking_lot anymore, so
shed this dependency.
2022-09-28 22:20:48 -07:00
Scott Lamb 721141770f flexible config for sample_file_dir 2021-10-26 11:47:13 -07:00
Scott Lamb 4a7f22723c Merge branch 'master' into new-schema 2021-10-21 12:26:31 -07:00
Scott Lamb b41a6c43da shutdown better
After a frustrating search for a suitable channel to use for shutdown
(tokio::sync::watch::Receiver and
futures::future::Shared<tokio::sync::oneshot::Receiver> didn't look
quite right) in which I rethought my life decisions, I finally just made
my own (server/base/shutdown.rs). We can easily poll it or wait for it
in async or sync contexts. Most importantly, it's convenient; not that
it really matters here, but it's also efficient.

We now do a slightly better job of propagating a "graceful" shutdown
signal, and this channel will give us tools to improve it over time.

* Shut down even when writer or syncer operations are stuck. Fixes #117
* Not done yet: streamers should instantly shut down without waiting for
  a connection attempt or frame or something. I'll probably
  implement that when removing --rtsp-library=ffmpeg. The code should be
  cleaner then.
* Not done yet: fix a couple places that sleep for up to a second when
  they could shut down immediately. I just need to do the plumbing for
  mock clocks to work.

I also implemented an immediate shutdown mode, activated by a second
signal. I think this will mitigate the streamer wait situation.
2021-09-23 16:33:29 -07:00
Scott Lamb dafd9041d6 json-based config for cameras and streams
for #155

The config interface code for changing cameras is quite messy but
seems to work for now.
2021-09-16 16:13:41 -07:00
Scott Lamb 95dec9791c config: validate more in cameras dialog box
Fixes #152

This won't win any awards for best UI or cleanest UI code, but it's an
improvement. Long-term I want the web config UI instead.
2021-08-23 15:55:47 -07:00
Scott Lamb 032bd76577 support --rtsp-library=retina (#37)
This isn't well-tested and doesn't yet support an initial connection
timeout. But in a quick test, it successfully returns video!

I'd like to do some more aggressive code restructuring for zero-copy
and to have only one writer thread per sample file directory (rather
than the syncer thread + one writer thread per RTSP stream). But I'll
likely wait until I drop support for ffmpeg entirely.
2021-06-07 14:40:26 -07:00
Scott Lamb 54bd068706 address some no-op clippy warnings 2021-05-17 15:00:51 -07:00
Scott Lamb 603f02b686 stop using old tempdir crate 2021-05-17 13:08:18 -07:00
Scott Lamb 2936c138c5 various doc improvements
I bumped the minimum Rust version because I'm taking advantage of
the rustdoc linking added in Rust 1.48:
https://blog.rust-lang.org/2020/11/19/Rust-1.48.html#easier-linking-in-rustdoc
2021-04-10 17:34:52 -07:00
Scott Lamb c547a49ac8 shorten per-file copyright headers
I'm tired of all the boilerplate, so use the new
GPL-3.0-linking-exception license identifier instead in all the server
components.

I left the ui stuff alone because I'm just going to replace it (#111).

Add a checker for the header because it's easy to forget.
2021-02-17 15:39:17 -08:00
Scott Lamb 97678f42e4 rustfmt everything
I want to make the project more accessible by not expecting folks to
match my idiosyncratic style. Now almost [1] everything is written
in the "standard" style. CI enforces this.

[1] "Almost": I used #[rustfmt::skip] in a few sections where I felt
aligning things in columns significantly improves readability.
2021-02-17 09:17:24 -08:00
Scott Lamb dd66c7b0dd restructure into "server" and "ui" subdirs
Besides being more clear about what belongs to which, this helps with
docker caching. The server and ui parts are only rebuilt when their
respective subdirectories change.

Extend this a bit further by making the webpack build not depend on
the target architecture. And adding cache dirs so parts of the server
and ui build process can be reused when layer-wide caching fails.
2021-01-22 22:01:17 -08:00