Compare commits

...
Author SHA1 Message Date
guoyuqi 1fe78e4fe6 refactor: refactor website ssl delete text 2026-10-10 17:55:58 +08:00
Yuki Guo 8415a1ce28 refactor: refactor user info (#14021) 2026-10-10 17:50:05 +08:00
Yuki Guo faf4933a02 refactor: refactor website ssl keyType default value (#14023) 2026-10-10 17:49:14 +08:00
Yuki Guo c692a9f5e3 refactor: refactor website upload css (#14022)
* refactor: refactor website upload css

* refactor: refactor website upload css
2026-10-10 17:48:57 +08:00
CityFun e7f0c21715 fix(vllm): resolve stuck startup state blocking operations (#14017) 2026-10-10 11:13:17 +08:00
ssongliu da7e8f1dbf fix: Remove duplicate TaskScopeVm (#14020) 2026-10-10 10:59:22 +08:00
Yuki Guoandssongliu 486cd5d0ad feat(redis): enable remote terminal through tracked background tasks (#14012)
* feat(redis): enable remote terminal through tracked background tasks

* feat(redis): enable remote terminal through tracked background tasks

* refactor(redis): remove redundant CLI status tracking

* refactor(redis): retain existing task execution flow

---------

Co-authored-by: ssongliu <sloooop1x@gmail.com>
2026-10-09 18:20:51 +08:00
ssongliu 00ce5a7b36 fix(firewall): preserve IPv6 RA protection and simplify error messages (#14015) 2026-10-09 17:45:57 +08:00
ssongliu c17f055e06 fix: preserve VM task timeout cleanup and terminate command groups (#14008) 2026-10-09 17:45:41 +08:00
王贺 f22df4b29c fix(file): stabilize closable tab width to avoid layout jitter (#14013)
The closable card tabs animated the close icon width between active and inactive tabs, so each tab switch reflowed the tab bar and caused the page to jitter. Reserve the close icon slot and toggle it with opacity instead, matching the terminal tab implementation.
2026-10-09 17:16:10 +08:00
Yuki Guoandssongliu b9bab90a14 fix: fix clean network (#14009)
* fix: fix clean network

* refactor(container): simplify network cleanup

---------

Co-authored-by: ssongliu <sloooop1x@gmail.com>
2026-10-09 17:14:50 +08:00
ssongliu ce14a104cc feat(vm): move shared VM support to agent (#14007) 2026-10-09 14:09:53 +08:00
CN_CoreStebandssongliu 2eaabccf94 fix(dashboard): prevent overlapping df collection and share disk queries (#13971)
* fix(dashboard): keep disk info working when a mount is stale or broken

The dashboard, the file manager mount list and the alert disk list
enumerated mounts with df. df calls statfs on every mount, so one bad
mount took the whole disk list down: the request hung until the
frontend timed out, or the list came back empty.

- enumerate mounts from /proc/self/mountinfo instead of running df
- read each mount through a guard with a 3s timeout that keeps at most
  one call per mount in flight
- fail closed: a mount that cannot be read stays in the list with
  errorType and error, and an unreadable mount table is an error
  rather than an empty list
- show failed mounts and their reasons on the dashboard and in the
  file manager
- cmd: after killing a timed-out command, wait at most 5s for it
  instead of forever

* fix(dashboard): distinguish remounted and hidden filesystems

* fix(disk): simplify disk queries and bound timeout handling

* fix(disk): prevent overlapping df collection tasks

---------

Co-authored-by: ssongliu <sloooop1x@gmail.com>
2026-10-09 10:14:16 +08:00
ssongliu e119239d3d refactor: manage firewall rules through runtime state and file backups (#13998)
Replace database rule synchronization with native rule operations and file-based backup, import, and recovery. Update firewall management views, descriptions, whitelist handling, and translations.

Include terminal reconnect fixes and xpack build tags for Darwin build targets. Remove temporary regression test files before committing.
2026-10-08 18:19:30 +08:00
ssongliu 14a57af4e9 feat(vm): clarify network modes and add macvtap translations (#13996) 2026-10-08 18:18:17 +08:00
Yuki Guoandssongliu 2701af9054 refactor: select bind (#13993)
* refactor: select bind

* refactor: simplify table selection state handling

---------

Co-authored-by: ssongliu <sloooop1x@gmail.com>
2026-10-08 17:11:28 +08:00
Yuki Guoandssongliu 4954067736 refactor: search bind (#13992)
* refactor: search bind

* refactor: simplify search persistence

---------

Co-authored-by: ssongliu <sloooop1x@gmail.com>
2026-10-08 16:19:29 +08:00
ssongliu b962144e7d fix(ssh): validate imported key pairs and cancel stale file reads (#13991) 2026-10-08 14:06:40 +08:00
Yuki Guo 8f1be42b6b fix: fix log text (#13990) 2026-10-08 11:32:09 +08:00
Yuki Guo c6969f231b fix: fix operation log (#13989) 2026-10-08 11:08:04 +08:00
4kerccand4kercc 3571fb8d86 fix: speed up SFTP backup upload with concurrent writes (#13976)
The SFTP uploader used io.Copy(dstFile, srcFile). Since the source is an
*os.File it implements io.WriterTo, so io.Copy took the os.File.WriteTo
branch and issued one synchronous SFTP WRITE per packet, waiting for the
server reply every time. Throughput was therefore bounded by packetSize/RTT
instead of the link capacity: on a ~68ms RTT link a 440MB backup reached
only ~0.25 MB/s, while rclone / OpenSSH sftp reach 16-19 MB/s on the very
same link and target.

pkg/sftp already provides a pipelined write path (File.ReadFrom, enabled by
UseConcurrentWrites and bounded by MaxConcurrentRequestsPerFile, default 64).
Use it for uploads, and truncate the remote file to the number of bytes
actually written when the concurrent write fails, to avoid leaving holes.

Measured against the same 68ms RTT target: a 96MiB upload went from 6m40s
(~0.25 MB/s) to ~6s (~16 MB/s); a 950MB three-site backup job from ~2h40m to
~2m04s.

Co-authored-by: 4kercc <13767585+4kercc@users.noreply.github.com>
2026-10-08 09:45:51 +08:00
ssongliu d21288788e fix: harden WebSocket origin validation (#13953) 2026-09-30 15:21:21 +08:00
ssongliu f65e4b06ab feat(vm): add VNC keyboard control translations (#13951)
* feat(vm): add VNC keyboard control translations

* feat(vm): add Windows VirtIO driver guidance translations

* feat(vm): add disk expansion translations and operation logs
2026-09-30 15:20:19 +08:00
ssongliu d26bc9a3a4 chore(deps): update dependencies and address security advisories (#13945)
Update pending Go and frontend dependencies, including gRPC, OpenTelemetry
and SVGO security fixes. Raise the minimum Go version to 1.26.6 and update
x/image and x/mod to patched versions.

Use moby/go-archive directly for image build contexts so its vulnerable
v0.1.0 replacement can be removed and v0.3.0 used. Regenerate Go module
checksums with go mod tidy and refresh the npm lockfile.

Validation: Linux amd64 builds, existing Go tests and go mod verify pass
for core and agent; frontend production build passes; npm audit reports
zero vulnerabilities. Frontend type-check reports the same nine errors
with the original and updated lockfiles.

Go scanning still flags two advisories in the legacy Docker SDK, which
has no patched release on its current module path. A separate SDK
migration and host Docker daemon updates remain necessary.
2026-09-29 21:36:19 +08:00
ssongliu 7088a903fe fix(vllm): sample realtime metrics independently of history (#13944) 2026-09-29 17:29:29 +08:00
ssongliu 864ebeed82 fix: load domestic release notes from new docs (#13943) 2026-09-29 17:29:16 +08:00
CN_CoreStebandssongliu c1b2b92708 feat(monitor): add monitor data export with csv, json and xlsx formats (#13875)
- new export tab between monitor and settings with time range, metric,
  device/interface/gpu selection and format choice
- files are generated in the browser from existing monitor apis; no new
  backend endpoints
- csv with utf-8 bom, escaping and formula sanitization; multiple metrics
  are zipped; xlsx uses one worksheet per metric with real date cells and
  row-limit warnings; json emits flat row objects
- share downloadBlob in utils/file with delayed url revocation
- use the 60s monitor timeout and node header for gpu queries

Co-authored-by: ssongliu <sloooop1x@gmail.com>
2026-09-29 15:50:00 +08:00
CN_CoreStebandssongliu 2189d9229a perf(monitor): improve monitor query and chart performance for large ranges (#13874)
* perf(monitor): improve monitor query and chart performance for large ranges

- enable LTTB sampling in the shared line chart to bound rendered points
- empty io/network in monitor search now means no name filter, returning
  all device rows in one query instead of fanning out per device
- merge device names recorded in the monitor db into io/network/gpu
  options so removed devices stay selectable; move logic to service layer
- add composite indexes (name, created_at) on monitor_ios/monitor_networks
  and (product_name, created_at) on monitor_gpus
- stop migrating the unused MonitorGPU model into MonitorDB
- fix the memory param which was compared against "mem" and never matched

* chore(monitor): remove monitor test fixtures

---------

Co-authored-by: ssongliu <sloooop1x@gmail.com>
2026-09-29 15:34:16 +08:00
ssongliu d0183f460e feat: add vLLM monitoring (#13942) 2026-09-29 15:19:30 +08:00
ssongliu d27d6db1ea feat: enhance native GPU NPU and XPU monitoring (#13939) 2026-09-28 18:32:12 +08:00
蘭 d2bb3813d9 fix: enhance SSH login alert by adding local IP addresses to success login whitelist (#13920) 2026-09-24 18:23:59 +08:00
蘭 cac73e6d45 fix: add success alert message for cron jobs and enhance alert handling (#13919) 2026-09-24 18:23:39 +08:00
ssongliu 4712ceaf2c fix: allow read-only firewall APIs in demo mode (#13914) 2026-09-24 14:01:05 +08:00
ssongliu 65243c68c4 fix: allow licensed community restore progress route (#13913) 2026-09-24 10:18:01 +08:00
ssongliu 256e79ca81 fix: remove minimum upgrade backup copies limit (#13907) 2026-09-23 18:17:32 +08:00
ssongliu 4861eb69cb fix: separate firewalld batches by native option (#13896) 2026-09-22 18:20:18 +08:00
蘭 fb8cf15537 fix: Fix the automatic renewal certificate alert issue (#13895) 2026-09-22 15:06:48 +08:00
ssongliu c4a6791271 fix: allow updated SSH ports through firewall before restart (#13894) 2026-09-22 14:51:54 +08:00
蘭 f58e147636 fix: enhance file rename functionality (#13891) 2026-09-22 12:49:05 +08:00
ssongliu 387e9fbeed fix: correct firewall port allowances and whitelist sync reporting (#13892) 2026-09-22 12:48:40 +08:00
ssongliu 4eb627bc79 fix(vm): display additional virtual machine states (#13890) 2026-09-22 10:27:49 +08:00
ssongliu 850c86229c fix: log dashboard disk command failures (#13889) 2026-09-22 10:16:54 +08:00
CityFun f984917a66 fix(upgrade): resolve application upgrade failures (#13888) 2026-09-22 10:11:13 +08:00
A_Words 8588217fbf fix: prioritize HTTP-to-HTTPS redirects over website redirects (#13847) 2026-09-22 10:05:53 +08:00
ssongliu a2307c5f64 fix(firewall): improve batch operations and task log completion (#13886) 2026-09-22 09:46:46 +08:00
蘭 5923290de8 ref: add file task messages for copy, move, compress and decompress (#13885) 2026-09-21 17:32:14 +08:00
蘭 1c994fba4a ref: update documentation URL for the panel (#13884) 2026-09-21 17:28:23 +08:00
蘭 415ab96aab ref:improve the panel API key (#13883) 2026-09-21 17:18:15 +08:00
ssongliu 19bb823b05 fix(firewall): guard IPv6 forwarding against RA disruption (#13882) 2026-09-21 17:12:17 +08:00
ssongliu 3a5371652e refactor(firewall): simplify state reads and batch verification (#13881) 2026-09-21 15:09:13 +08:00
蘭 a267b4148a ref:Optimize and improve the panel API key (#13880) 2026-09-21 12:49:45 +08:00
蘭 36a01eb60d feat: The panel API key supports creating multiple tokens and enhance terminal capabilities (#13872) 2026-09-20 18:32:54 +08:00
王贺 65f6fdd045 fix(ssh): display SSH login logs in server timezone (#13869)
The SSH log page formatted dates with the browser timezone, so logs looked shifted when the client timezone differed from the server. Format the timestamp with the offset returned by the API so the page matches auth.log, the terminal and the CSV export. Refs #13860.
2026-09-20 18:02:46 +08:00
ssongliu 75b60b32e4 feat(vm): update multi-CD-ROM guidance translations (#13857) 2026-09-18 16:38:31 +08:00
CityFun 6cb65e2290 fix: Fixed with ollama page not work (#13855) 2026-09-18 11:35:46 +08:00
ssongliu 0bad1b471f feat(core): add runtime diagnostics and pprof capture (#13852) 2026-09-17 15:46:06 +08:00
ssongliu 3814525edd fix(core): avoid enumerating authorized IP subnets (#13851)
Use net.IPNet.Contains to check CIDR membership directly and remove the address increment loop. Large authorized subnets no longer cause per-request address enumeration and excessive CPU usage.
2026-09-17 15:45:57 +08:00
ssongliu 8162dd1856 fix(container): improve inspect panel theme and text layout (#13849) 2026-09-17 15:45:48 +08:00
ssongliu e833787020 fix(firewall): preserve whitelist priority and rule ordering (#13845) 2026-09-17 13:02:08 +08:00
ssongliu 673ffac516 refactor(firewall): simplify whitelist configuration and rule protection (#13838) 2026-09-16 22:09:49 +08:00
ssongliu e864610015 fix(firewall): validate whitelist ports and sources in form (#13835) 2026-09-16 21:31:06 +08:00
ssongliu 78402e1b7d refactor(firewall): consolidate utilities and flatten packages (#13833) 2026-09-16 16:48:48 +08:00
ssongliu 782bc1e67c fix(firewall): manage SSH access and queue stop operations (#13831)
* fix(firewall): manage SSH access and queue stop operations

* fix(firewall): reconcile whitelist rules and sync differences
2026-09-16 16:45:33 +08:00
ssongliu 86e4ed6f64 perf(firewall): optimize large rule sets and queue deletions (#13829) 2026-09-16 15:34:10 +08:00
CityFun ee8bac39af style: Optimize the website configuration UI (#13828) 2026-09-16 14:48:57 +08:00
ssongliu fe742b9f41 fix(firewall): improve whitelist management and rule lifecycle (#13826) 2026-09-15 23:55:01 +08:00
CityFun 9a5bd9bcba fix: Fix the issue where Brotli settings cannot be saved when enabled in OpenResty (#13822) 2026-09-15 18:36:51 +08:00
ssongliu b9c8e39560 fix: validate Docker IPv4 forwarding (#13820) 2026-09-15 18:36:36 +08:00
CityFun 6b20ff0b13 feat: OpenClaw supports configuring model Max Tokens (#13818) 2026-09-15 18:36:24 +08:00
ssongliu 89bd32b6d4 fix(terminal): isolate persistent shortcut sessions (#13810) 2026-09-15 10:05:16 +08:00
ssongliu 005f240fb7 fix: improve firewall lifecycle and sync (#13809) 2026-09-15 10:04:53 +08:00
蘭 75da53e374 feat: Remote download supports server file name options and improves error handling (#13808)
* feat: Remote download supports server file name options and improves error handling

* feat: Remote download supports server file name options and improves error handling
2026-09-15 10:04:44 +08:00
ssongliu 2485b0aa5e fix: remove debug logs (#13807) 2026-09-14 17:36:23 +08:00
ssongliu ed51a5e1fa fix(firewall): split UFW all-protocol port ranges (#13806) 2026-09-14 15:46:14 +08:00
ssongliu 56870504ac fix: restrict terminal dock to super administrators (#13804) 2026-09-14 11:36:35 +08:00
ssongliu 7aefb47cc3 fix(firewall): improve rule loading, task labels and panel port cleanup (#13798) 2026-09-14 09:40:20 +08:00
ssongliu aba41c0aea feat(terminal): share connection menu and add node quick connect (#13787) 2026-09-14 09:40:15 +08:00
ssongliu 9300bf4141 refactor(firewall): queue rule operations and simplify synchronization (#13786) 2026-09-11 15:16:44 +08:00
王贺 b7ec17b3e3 style(terminal): refine terminal shortcut translations (#13785) 2026-09-11 11:07:06 +08:00
ssongliu 2fcfe56a30 refactor(firewall): queue rule operations and simplify synchronization (#13784) 2026-09-10 23:54:46 +08:00
CityFun 63b2d4e4d5 fix: Fix the issue where installing extended channel plugins fails in OpenClaw (#13782) 2026-09-10 18:11:49 +08:00
ssongliu 4cd77d8ee1 fix(firewall): simplify rule editing and reject duplicate adoption (#13779) 2026-09-10 18:11:36 +08:00
CityFun 53a7347bea fix: Fix an issue with pulling images when upgrading OpenList. (#13776) 2026-09-10 18:11:26 +08:00
ssongliu a02c25ebcc feat(terminal): add terminal button visibility setting (#13772) 2026-09-09 18:39:08 +08:00
CityFun 605c8cc6db fix: optimize self-signed certificate format (#13771) 2026-09-09 18:27:17 +08:00
ssongliu 033cc7c2d1 fix: sort containers by name and backup uploads by time (#13770) 2026-09-09 16:58:53 +08:00
蘭 7c1ddb5b4c fix: add localized message for download records not removed (#13757) 2026-09-09 15:33:56 +08:00
ssongliu eb0f5264d7 refactor: simplify firewall rule management and whitelist updates (#13758) 2026-09-09 15:33:24 +08:00
蘭 5ad12c6fe4 fix: improve progress percentage calculation for download status (#13755) 2026-09-09 13:54:05 +08:00
ssongliu 61dacce5e0 refactor: replace freetype captcha with opentype renderer (#13753) 2026-09-09 13:53:56 +08:00
ssongliu e3f0381a26 fix: preserve table selections when clicking row content (#13752) 2026-09-09 13:53:46 +08:00
ssongliu 6bc9dd96af fix: simplify firewall rule editing and dual-stack port handling (#13748) 2026-09-08 22:46:35 +08:00
蘭 e23f338b31 feat: Processing synchronous alert settings (#13747) 2026-09-08 20:45:07 +08:00
蘭 6e08b50e3c fix: Fix file timeout and retry processing for long transmission tasks (#13746) 2026-09-08 18:34:21 +08:00
ssongliu 536712cd55 feat(vm): add localized license introduction (#13745) 2026-09-08 18:29:05 +08:00
CityFun 191ff0cda4 fix: Fix an error when selecting a runtime environment version. (#13742) 2026-09-08 17:10:56 +08:00
ssongliu 671f781564 feat: add terminal session rules hint (#13744) 2026-09-08 16:58:54 +08:00
ssongliu 30dc36b95d feat: integrate virtual machine migration into XPack (#13740)
* feat: integrate virtual machine migration into XPack

* chore: remove virtual machine test files
2026-09-08 14:20:26 +08:00
蘭 fac4aec680 feat: Enhance responsive design and layout adjustments for mobile devices (#13739) 2026-09-08 14:20:16 +08:00
ssongliu 8eac9a1808 fix(container): preserve IP allocation across container operations (#13738) 2026-09-08 14:20:03 +08:00
CityFun ce74d96617 feat: Add support for importing environment variables into the runtime environment. (#13737) 2026-09-08 09:52:40 +08:00
ssongliu a15e77d605 fix: harden terminal session lifecycle (#13736) 2026-09-08 09:29:01 +08:00
CityFun bad022f524 feat: Add support for switching image versions in runtime environments. (#13730) 2026-09-07 18:30:27 +08:00
CityFun 50a54d0613 feat: Add support for importing environment variables into the runtime environment. (#13727) 2026-09-07 18:26:35 +08:00
蘭 a47e41a8b7 feat: enhance download management with improved error handling and status tracking (#13734) 2026-09-07 18:25:15 +08:00
CityFun f938443e55 fix: issue self-signed certificates using the selected CA (#13728) 2026-09-07 18:06:20 +08:00
蘭 b90abd2b28 feat: enhance ZIP entry normalization and path compatibility checks (#13733) 2026-09-07 18:06:11 +08:00
ssongliu da5682a600 fix(firewall): harden port switching and rule synchronization (#13731) 2026-09-07 18:05:31 +08:00
HynoR 81b72d9b7d feat: Implement server-side SSH session persistence and recovery (#13707)
* feat(terminal): keep ssh sessions alive server-side with reattach

Split the terminal ws handling into a Session (pty + ssh backend) and an
Attachment (one websocket). A session outlives its websocket: a clean close
(1000) ends the pty, any other disconnect keeps it for a 30-minute grace
period and it can be reattached via `?session=<id>`. Output goes through a
fixed 128KB ring buffer so a reattaching client gets the recent tail, with a
truncation marker if it fell behind. Sessions are owner-scoped; a second
attachment kicks the first (4409), unknown ids get 4404.

New endpoints under /hosts/terminal/sessions (search, close) let the
frontend list and recover sessions after a tab or browser is closed.

* feat(terminal): floating terminal dock with session recovery

Terminals now live in a layout-level host and are teleported into whichever
view shows them, so leaving the terminal page no longer kills them. A dock
handle on the right edge opens a non-modal dialog from any page with every
live session, a picker for local shell / ssh hosts, minimize, and
close-all. On page load the store recovers sessions the server still holds,
so an accidentally closed tab or browser can resume within the grace period.
The menu-tab label shows the live session count.

* fix(terminal): page re-claims its slots under a locked menu tab

With the terminal menu tab locked (keep-alive), leaving the page deactivates
it instead of unmounting it, so the slot ref callback never re-runs on
return. After the dock had taken the Terminal over and released it, nobody
claimed it for the page again and it stayed parked in the hidden host.

Claim/release slots explicitly on mount, activated, deactivated and unmount,
the same ownership rule the dock uses, instead of relying on the ref callback.

* fix(terminal): logout closes every kept-alive terminal session

A logged-out panel has nobody watching it, so nothing it left running should
survive: core now tells the local agent to close all terminal sessions when the
user logs out, changes the password, or changes the bind domain. Until now the
teardown relied on the logging-out tab sending close code 1000; a second tab or
a websocket held outside the SPA kept its shell after logout.

Agent: terminal.CloseAll and POST /hosts/terminal/sessions/closeAll.
Core: LogOut / deleteCurrentSession / BindDomain call it via proxy_local,
best effort.

* fix(terminal): pin a local shell to the node it was opened on

The node a local shell connects to was resolved from the current node every
time the websocket was built, so after switching nodes a reconnect carried the
old session id to the new node (4404) and then opened a shell there instead.
Store the operateNode on the entry when it is created; ssh shells keep going to
the master. Shells on a non-master node get the node name in their title so a
restore in another node's view can tell them apart.
2026-09-07 15:01:35 +08:00
CityFun 6e13143286 feat: VllM add support for GB10 Deepseek V4 Flash Vision Exp (#13725) 2026-09-07 14:56:56 +08:00
Snrat 70fc628c81 feat(openresty): activate brotli when the module is enabled, and fix gzip defaults (#13639)
* feat(openresty): manage http-context directives via conf/http.d

Add a managed-file mechanism for http-context nginx directives, mirroring
the existing one for conf/modules-enabled.

A separate directory is required because load_module is a main-context
directive, so modules-enabled is included at the top level of nginx.conf and
cannot host http-context directives.

Files carry a 1panel-http- prefix; anything else in the directory is left
untouched. Writes are atomic via a temporary file plus rename, and the
directory is snapshotted so a failed nginx -t can be rolled back.

The mechanism is inert when conf/http.d does not exist, which is the case
for OpenResty installations predating the directory.

* fix(openresty): correct gzip defaults and add missing compressible types

Bring the embedded gzip template in line with how sites are actually served.
It was previously dead code: nothing referenced gzip.conf, so the values
never reached an installation. It is now embedded and used by the migration
that follows.

gzip_types was missing application/json, so JSON API responses were served
uncompressed. Also add ld+json, text/xml, xhtml+xml, rss+xml, atom+xml,
wasm, svg+xml and ttf/otf. Already compressed formats (images, woff2,
archives) stay out on purpose.

gzip_comp_level 6 -> 5, at the cost/ratio knee for gzip.

gzip_proxied any, so that proxied responses are compressed regardless of
their Cache-Control semantics.

gzip_static is intentionally not enabled: nginx does not verify that a .gz
file is newer than its source, so a stale artifact would be served
indefinitely with no error.

* feat(openresty): activate brotli directives when the module is enabled

Enabling ngx_brotli only emitted load_module, leaving the module loaded but
inert: no response was ever brotli-encoded until the user added
`brotli on` and `brotli_types` to nginx.conf by hand. The module is
prebuilt into the OpenResty image and listed in the catalog, so the only
missing step was the runtime configuration.

Enabling the module now also writes its http-context directives to
conf/http.d, and disabling or deleting it removes them. Removal matters:
leaving `brotli on` behind after the .so is unloaded makes nginx fail to
start on an unknown directive.

Both directory sets are written before nginx -t runs, so nginx only ever
observes a consistent state, and a failed check rolls back load_module
files and runtime directives together.

Runtime defaults are declared per module in a table, so other modules
needing http-context configuration can be added without touching the
reconcile logic.

brotli_types matches gzip_types so both encoders cover the same content.
brotli_comp_level is 5 rather than the nginx default of 6: level 5 reaches
roughly gzip level 9 ratio at a fraction of the cost, while 6 is tuned for
static assets and is too expensive for dynamic responses.

brotli_static is deliberately omitted, for the same reason gzip_static is:
nginx does not verify that a precompressed artifact is newer than its
source, so a stale file would be served indefinitely with no error.

Installations without conf/http.d keep the previous behaviour instead of
failing.

* feat(openresty): refresh stock gzip defaults on upgrade

Upgrades deliberately preserve the user's nginx.conf, so corrected gzip
defaults shipped with a new OpenResty version never reach existing
installations. Rewrite the values in place during upgrade, but only when the
block is provably untouched.

The rewrite requires every gzip directive to match the factory values byte
for byte, with none missing, none added and none duplicated. Any deviation
means the user tuned compression, and their configuration is left alone.

gzip stays in the http block of nginx.conf rather than moving to an included
file: nginx rejects a duplicate gzip directive across contexts, and the
compression settings page reads and writes these same keys in nginx.conf, so
a relocated block would be reintroduced on the next save and break nginx -t.

The config parser is not used either. Its dumper regenerates the whole file,
drops standalone comments and reorders proxy includes, which would be
destructive on a user's main config. Lines are edited individually so
everything outside the gzip block stays byte-identical.

The rewrite is idempotent, and a failed nginx -t restores the previous file.
A failure is logged as a warning instead of failing the upgrade.

* fix(website): preserve size units in nginx performance settings

The form stripped the unit suffix when reading a directive and then always
appended a fixed one when saving, so the unit was silently reinterpreted.

A config carrying `gzip_min_length 512;`, meaning 512 bytes, was read as 512
and written back as `512k`, inflating the threshold by 1024 and effectively
disabling compression for every response under 512 KB. The same applied to
client_header_buffer_size and client_max_body_size, where the value grew by
a factor of 1024 in the opposite, riskier direction.

Remember the unit that was read and write it back unchanged, defaulting to
the previous suffix only when the directive carries no unit information. The
input suffix now shows the unit actually in use instead of a hardcoded
label.

Also fix the value parsing itself: `Number(value.match(/\d+/g))` coerces a
multi-number match to NaN, so a directive such as `gzip_buffers 4 16k` would
blank the field. Take the first captured number instead.

* feat(website): expose brotli settings in the compression page

Brotli could be enabled as a module but never configured from the panel, so
its behaviour was invisible and unchangeable without editing nginx.conf by
hand.

The section appears only once the module is enabled and built, since the
directives are rejected by nginx while the module is not loaded. Values are
read from and written to the panel-managed http.d file rather than
nginx.conf, so they are removed together with the module.

brotli_types stays out of the form on purpose: it is kept aligned with
gzip_types so both encoders cover the same content, and exposing it would
invite the two lists to drift apart.

Saving reuses the existing scope endpoint with a dedicated brotli scope,
which keeps the managed file as the single source of truth instead of
duplicating the values into nginx.conf.

* fix(openresty): stop a stale build option from forcing a full rebuild

Manual builds and upgrades disagreed on when a full OpenResty image rebuild
is required. `executeNginxModuleBuild` used `staticNginxBuildRequired`, which
also treated a non-empty `RESTY_CONFIG_OPTIONS_MORE` in .env as a reason to
rebuild, while `buildNginx` looked only at the module list.

The env value is derived state, not an input: `configureStaticNginxModules`
rewrites it from the current module list, and every build path calls that
function before building. With no static module enabled it writes an empty
string, so the rebuild the latch triggered ran with an empty option list and
could only reproduce the image it started from — up to 120 minutes of build
time to arrive back where it began.

Decide on the module list alone, which is what the upgrade path already did.

An install that genuinely has an enabled static module is unaffected: both
predicates already agreed in that case. Leftover values are still cleared, by
`configureStaticNginxModules` on the next build or upgrade.

* feat(openresty): build modules on versions without a dynamic builder

Module state written before build modes existed carries no buildMode.
validateNginxModuleBuildMode rejects the empty value, which fails
loadNginxModules and with it every module operation and the upgrade itself —
the whole module subsystem, not just the static feature.

Infer the missing value from what the install can actually do instead:
dynamic when the builder and catalog are present, static when the compose
file still has a build section and build/Dockerfile to recompile the image.

Builds follow the same principle. Asking a pre-dynamic install to build a
module used to return "the installed OpenResty version does not support
dynamic module builds", which is a dead end: these versions produce modules
by compiling them into the image, and they still can. Such a build is now
retargeted to the static path, with --add-dynamic-module rewritten back to
--add-module and =dynamic switches reduced to their plain form. The error is
kept only for installs that reference a prebuilt image and genuinely cannot
compile anything, and it now says so and points at the upgrade.

The retarget applies to a copy that drives one build and is never persisted,
so the catalog stays authoritative and modules return to dynamic once the
install gains a builder.

Verified end to end against 1.27.1.2-5-1-focal, which ships no
Dockerfile.modules and no module.catalog.json: ngx_brotli compiles into the
image, nginx -t accepts the brotli directives with no load_module present,
and the server responds with Content-Encoding: br.

* feat(openresty): respect a hand-written brotli configuration

A user who enabled brotli before the panel managed it did so by editing their
configuration by hand. Emitting a managed file alongside it defined every
directive twice and nginx refused to start, so these users — the very ones
this feature is for — broke on upgrade.

Detection now scans every file nginx loads brotli from: nginx.conf and the
conf.d and default includes. Any active brotli* directive counts, so a lone
tuning directive is enough to treat the module as user-managed, and a
commented-out line never triggers it.

When the user owns the configuration, the panel stays out of the way:

- No managed http.d file is written, so the user's definition stays the only
  one and their values are never overridden.
- brotli_types diverging from gzip_types is left exactly as written; the panel
  does not widen them.
- The settings page reports their real values and shows a notice that brotli
  is managed manually, rather than presenting defaults that do not match the
  running configuration.
- Saving edits their own lines in place, keeping indentation and comments,
  instead of writing a second copy. The flag is localised in all 12 languages.

Detection re-runs on every reconcile, so once the user deletes their
hand-written config the panel takes over again automatically.

* feat(openresty): wire conf/http.d from the agent instead of upgrade scripts

Following review feedback: setup scripts no longer create conf/http.d or
inject its include into existing installations' nginx.conf. The agent owns
the directory, the include, the runtime directives and the rollback, and only
touches nginx.conf when a module that needs http-context configuration is
actually enabled.

Insertion is a line-level edit, never the config parser: the include lands
before the conf.d include, or at the top of the http block when that anchor
is absent, keeping the surrounding indentation and leaving the rest of the
file byte-identical. A config without a locatable http block degrades to the
previous behaviour — module loads, runtime directives skipped, warning logged
— instead of failing the operation. Detection re-runs on every reconcile, so
an install recovers on its own once nginx.conf can be edited again.

Rollback now covers three artefacts: modules-enabled, http.d, and the
inserted line in nginx.conf.

The include is kept when the last module is disabled. Pointing at an empty
directory is harmless, and removing it would mean another edit of the user's
main config with its own failure surface.

When the include is missing and cannot be inserted, the brotli settings
report ManagedUnavailable and the settings page warns that the values shown
will not take effect, instead of presenting inert settings as live.

* fix(openresty): tighten brotli ownership handling and build guards

The settings page could not save brotli values for users who wrote their own
directives after the panel had started managing the module: the stale managed
file was still on disk, so every save ended in a duplicate directive error.
That file is now removed before the in-place edit, and a failed nginx -t
rolls back both sides.

User-managed detection now also covers conf/default, which is included at
http scope like conf.d, and the http.d include check no longer depends on the
exact container path literal, so an include written in a slightly different
form is recognised instead of duplicated.

The dynamic-to-static build fallback is dropped. The catalog and the dynamic
builder ship together, and installs without the catalog fail to load their
module state earlier anyway, so the branch could never run; what remains is
an error that says the version cannot build modules and to upgrade first.

The embedded gzip template is no longer wired to an unused variable, and a
test keeps it in sync with the defaults the upgrade writes.

Smaller fixes in the same area: a custom module named ngx_brotli no longer
inherits the built-in runtime defaults; nginx.conf edits go through temp file
renames and inserted lines follow the file's own line endings; the gzip
rewrite keeps each line's own indentation; the settings page resets its unit
cache on load, warns when the brotli half of a save fails after gzip already
applied, and no longer coerces unrendered keys to zero.

* fix(openresty): prove brotli reached the running server, not just disk

nginx -t and a successful reload both pass even when the managed directory
never reaches the container: the include is a glob, so a missing bind mount
or an unrecognised include variant silently loads nothing. The brotli save
now reads the effective configuration back with nginx -T and rolls the write
back with an actionable error when the directives are not there, instead of
reporting success for settings nobody will ever see.

The include match also accepts the quoted form nginx permits, so a
hand-written or legacy variant no longer invites a second include of the same
directory.

Values written into nginx.conf are checked against a whitelist before any
file is touched. The UI only ever sends on/off, numbers and sizes, but the
endpoint is reachable directly, and an unfiltered value could inject a
directive or trip the group-reference expansion of regexp.ReplaceAllString in
the in-place rewrite.
2026-09-07 14:42:03 +08:00
Eric Curtin 9858881ce6 feat(ai): add llmman as a local model provider (#13717)
llmman (https://github.com/llmmanorg/llmman) is a local model runner
serving Ollama- and OpenAI-compatible routes on 127.0.0.1:17434.
Register it in the agent provider catalog next to Ollama and extend
every Ollama special case (no API key, verification skipped, OpenClaw
placeholder key, manual initial model) to cover it as well.
2026-09-07 11:41:13 +08:00
A_Words eb6a8c7646 fix: avoid website SSL port conflicts on creation (#13720) 2026-09-07 11:39:15 +08:00
ssongliu a71aea8aec fix(firewall): hide inactive Docker ports (#13716) 2026-09-04 21:43:05 +08:00
ssongliu 918c441f88 Revert "fix(fail2ban): treat process as active when client ping succeeds (#13…" (#13715)
This reverts commit a6e2efa6c9.
2026-09-04 16:46:47 +08:00
CityFun 960b4b0345 fix(b.ai): resolve account validation failure (#13712) 2026-09-04 16:40:48 +08:00
蘭 3aa4bfaa82 ref: streamline SSH login log handling and remove unused functions (#13705) 2026-09-03 18:33:07 +08:00
ssongliu b3bdf9ef7e fix: validate cronjob timeout as positive integer (#13706) 2026-09-03 18:26:17 +08:00
ssongliu 2948b8ffe8 fix: normalize Docker firewall rule sync (#13704) 2026-09-03 18:22:48 +08:00
ssongliu e99c6c08a5 feat: support menu tab session keep-alive (#13698) 2026-09-03 15:31:25 +08:00
ssongliu 6fb389b2ed fix(container): extend disk usage stats timeout (#13695) 2026-09-03 15:31:12 +08:00
ssongliu c09833cbde chore(deps): batch safe and security dependency updates (#13696)
* chore(deps): batch safe dependency updates

* chore(deps): address dependency alerts
2026-09-03 11:46:56 +08:00
ssongliu fa2ad69154 feat: improve community restore package guidance (#13694) 2026-09-03 09:45:40 +08:00
ssongliu 51d84455a3 fix(container): avoid pinning dynamic IPs on upgrade (#13693) 2026-09-02 17:23:00 +08:00
ssongliu d88d98d8a8 fix: warn on node version mismatch after login (#13691) 2026-09-02 14:57:10 +08:00
蘭 5aec466c8e feat: add support for custom webhook configuration (#13685) 2026-09-02 14:49:20 +08:00
ssongliu 0ee93774d5 fix(container): authenticate private registry image repulls (#13690) 2026-09-02 14:49:09 +08:00
ssongliu 7be7368bb9 fix: improve firewall lifecycle recovery (#13686) 2026-09-02 14:48:59 +08:00
ssongliu eab0bb4a94 fix: repair firewall forwarding migration (#13689) 2026-09-02 14:48:47 +08:00
9f74f2077a Fix/ssh disconnect session key (#13669)
* fix(ssh): correlate disconnect logs by client endpoint

* fix(ssh): scope endpoint correlation to active sessions

---------

Co-authored-by: JayLee-sre <1.18655426e+08+JayLee-sre@users.noreply.github.com>
Co-authored-by: ssongliu <sloooop1x@gmail.com>
2026-09-02 09:32:07 +08:00
zhudaguanrenandzhudaguaneren a6e2efa6c9 fix(fail2ban): treat process as active when client ping succeeds (#13679)
Fail2ban UI currently keys isActive only on systemd fail2ban.service.
If the daemon is alive under another process manager, whitelist and
blacklist stay disabled. Detect liveness with fail2ban-client ping.

Fixes #13678

Co-authored-by: zhudaguaneren <218366267+zhudaguaneren@users.noreply.github.com>
2026-09-01 18:15:32 +08:00
igophper 205ef3009a fix: deduplicate port bindings in container port mapping (#13663) 2026-09-01 18:05:07 +08:00
蘭 d7edbd1e95 feat: reconcile hide menu integrity (#13681) 2026-09-01 17:52:12 +08:00
蘭 b361f464c5 fix: enhance upload handling and disable actions during processing (#13676) 2026-09-01 17:52:03 +08:00
ssongliu fb377d2e99 fix(firewall): recover rules after upgrade (#13680) 2026-09-01 14:53:01 +08:00
ssongliu deddd392ba fix(firewall): handle Docker host input ports (#13675) 2026-09-01 09:33:29 +08:00
ssongliu 3ab10848c8 fix: restore firewall-dependent rules after reset (#13674) 2026-08-31 17:15:56 +08:00
ssongliu 433f1a940f fix: improve firewall abnormal state diagnostics (#13673) 2026-08-31 16:18:13 +08:00
ssongliu 1f12c09eb5 fix: improve firewall rule management (#13670) 2026-08-31 11:48:14 +08:00
ssongliu 31e6d523f9 fix: improve firewall runtime rule handling (#13667) 2026-08-31 09:28:15 +08:00
CityFun 3c0bd051bf feat: custom model account dashscope-images (#13664) 2026-08-28 18:22:52 +08:00
ssongliu 3c2d92dc5f fix: improve firewall backend rule handling (#13662) 2026-08-28 16:11:11 +08:00
ssongliu 262bd14bc8 chore(deps): batch dependency updates (#13650) 2026-08-28 09:56:01 +08:00
ssongliu f15ff46e34 fix: improve firewall rule management (#13648) 2026-08-27 22:24:15 +08:00
CityFun fc1ec4e1b0 feat: add gb10 vllm image support (#13647)
* feat: add gb10 vllm image support

* feat: add gb10 vllm image support
2026-08-27 18:36:37 +08:00
ssongliu 53f75826d8 refactor: simplify firewall service structure (#13646) 2026-08-27 14:00:43 +08:00
ssongliu ddfb816ef1 feat: improve firewall backend synchronization (#13645) 2026-08-27 10:37:09 +08:00
ssongliu 18428d108e feat: improve firewall backend synchronization (#13644) 2026-08-27 10:31:39 +08:00
蘭 3506c5dd3b feat: add footer navigation component and update translations (#13642) 2026-08-26 14:30:45 +08:00
ssongliu 2dffd06b1b chore: clarify agent Skill directory labels (#13640) 2026-08-26 14:30:34 +08:00
ssongliu 12f2484d12 feat: support firewall rule synchronization (#13637) 2026-08-25 18:50:27 +08:00
蘭 2dea44acf6 fix: prevent rate limit bypass in public file shares (#13632) 2026-08-24 21:50:48 +08:00
ssongliu 205f76c65d fix: update vulnerable dependencies (#13629) 2026-08-24 18:04:14 +08:00
ssongliu 86af4fbd4d feat: improve firewall status and UI translations (#13630) 2026-08-24 17:40:36 +08:00
ssongliu 7915230121 refactor: rebuild firewall management (#13628)
* refactor(firewall): rebuild rule management foundation

* refactor(firewall): streamline rule checks and inventory

* feat(firewall): improve native rule inventory

* refactor(firewall): refine rule management

* feat: add Docker port guard

* feat(firewall): support native nftables

* feat(firewall): add configurable firewall selection

* feat(firewall): support nftables docker port guard

* refactor(firewall): complete v2 rule management and migration

* refactor(firewall): align state and API contracts

* refactor(firewall): unify rule management operations

* feat: refine firewall v2 rules and forwarding

* fix(firewall): harden dual-stack rule management

* refactor(firewall): consolidate rule validation and persistence
2026-08-24 12:51:34 +08:00
ssongliu 1b27db7daa fix: honor configured ClamAV scan timeout (#13619) 2026-08-21 17:33:52 +08:00
ssongliu 7370dcaa55 fix(container): log startup failure before rollback (#13618) 2026-08-21 17:33:40 +08:00
ssongliu 6a378b6863 fix: improve enterprise license page compatibility (#13616) 2026-08-21 17:33:23 +08:00
ssongliu 6f6747a584 fix: support trusted proxies for allowed IPs (#13608) 2026-08-21 17:33:14 +08:00
蘭 825221b2bb ref: Optimize mobile editor (#13607) 2026-08-20 18:34:20 +08:00
ssongliu 1e9d4b592e fix: preserve failed compose deployments (#13603) 2026-08-20 18:28:51 +08:00
蘭 4a51db4764 fix: Fix the issue of menu loss caused by switching service regions (#13602)
* fix: Fix the issue of menu loss caused by switching service regions

* fix: Fix the issue of menu loss caused by switching service regions
2026-08-20 18:28:41 +08:00
ssongliu afea71c81c fix: sync system version on startup (#13601) 2026-08-20 16:06:12 +08:00
ssongliu 9c8ca2ab3c fix: center community restore dialog (#13600) 2026-08-20 15:34:17 +08:00
ssongliu a04875f64b fix: sync documentation source settings (#13599) 2026-08-20 15:01:27 +08:00
ssongliu 7ec0bdb3f7 feat: support multi-chip Ascend devices (#13593) 2026-08-20 15:01:15 +08:00
CityFun d2dbb6486e feat: add enterprise demo handle (#13592)
* feat: add enterprise demo handle

* feat: add enterprise demo handle
2026-08-20 15:01:04 +08:00
14728f889e fix(ssh): correlate disconnect logs by client endpoint (#13581)
* fix(ssh): correlate disconnect logs by client endpoint

* fix(ssh): scope endpoint correlation to active sessions

---------

Co-authored-by: JayLee-sre <1.18655426e+08+JayLee-sre@users.noreply.github.com>
Co-authored-by: ssongliu <sloooop1x@gmail.com>
2026-08-19 10:20:16 +08:00
CityFun ff199245b0 feat: add some translate (#13585) 2026-08-18 18:46:27 +08:00
蘭 667807e249 fix: Fix large file/slow network upload timeout in file management (#13584) 2026-08-18 18:12:42 +08:00
Jet.andJayLee-sre 63e09c8c47 docs: name Halo in website deployment overview (#13580)
Co-authored-by: JayLee-sre <1.18655426e+08+JayLee-sre@users.noreply.github.com>
2026-08-18 09:49:11 +08:00
ssongliu 17a8835d59 feat: support Ascend 910B GPU monitoring (#13579)
* feat: support Ascend 910B GPU monitoring

* chore: remove GPU test files
2026-08-17 17:38:43 +08:00
王贺 b306bfa77a fix: correct disk usage for device aliases (#13570) 2026-08-17 15:20:15 +08:00
蘭 b1eff2a893 feat: change some translate (#13568) 2026-08-17 13:58:53 +08:00
王贺 5ac7c80881 fix: support underscores and hyphens in website default documents (#13551) 2026-08-14 10:55:44 +08:00
CityFun d402f67fc3 feat: Optimize application upgrade logic (#13549)
* feat: Optimize application upgrade logic

* feat: Optimize application upgrade logic

* feat: Optimize application upgrade logic
2026-08-13 18:28:59 +08:00
Chen, Ting-An c13793c445 fix(i18n): polish Traditional Chinese agent copy (#13536) 2026-08-12 15:32:39 +08:00
CityFun daa3f6b206 chore: Update dependencies (#13528) 2026-08-12 15:29:40 +08:00
maninhill a2d85c911d Revise user statistics and AI agents limit in README (#13525)
Updated user statistics and modified AI agents limit in the feature table.
2026-08-11 09:39:10 +08:00
蘭 1b76c91e1b fix: Fix file loss issues when copying or moving large directories (#13524) 2026-08-10 22:07:45 +08:00
maninhill d663a4397a Update README for clarity and security features (#13516)
Removed redundant mention of AI gateway in the Full-Stack AI Management section and added WAF to the security features.
2026-08-10 10:04:41 +08:00
maninhill d1558c5eae Revise README for clarity and feature updates (#13515)
Updated the README to enhance the description of 1Panel's features, including AI management, security, and backup capabilities. Adjusted the Pro Edition feature comparison to include the Enterprise edition.
2026-08-09 09:14:52 +08:00
maninhill 0da4f77a2e Revise README.zh-Hans.md for feature updates (#13512)
Updated the README in Chinese to reflect new features and improvements in 1Panel, including AI management capabilities and enhanced security features.
2026-08-07 22:51:57 +08:00
ssongliu e7ef35740c fix: align compose project name handling (#13500) 2026-08-07 12:44:29 +08:00
ssongliu b0d561e33b feat: show license expiration alert on dashboard (#13499) 2026-08-07 10:22:23 +08:00
CityFun 75b362fa9b chore: update dependencies (#13497) 2026-08-06 21:47:20 +08:00
蘭 466f373ef6 feat: change some translate (#13496) 2026-08-06 18:27:27 +08:00
ssongliu 4489641b54 perf(snapshot): hard link local recovery archive (#13494) 2026-08-06 16:08:58 +08:00
蘭 1971d9dec2 fix: handle external login state and emit readiness event (#13493)
* fix: handle external login state and emit readiness event

* fix: handle external login state and improve SAML2 logout response handling
2026-08-06 13:53:36 +08:00
CityFun c38d741770 fix: Fix an issue where the website monitoring directory was not completely removed when deleting a website. (#13492) 2026-08-06 13:52:53 +08:00
蘭 122a474032 feat: enhance alert log search with start and end time filters (#13489) 2026-08-05 21:13:25 +08:00
CityFun 54854e99e7 feat: change deepseek api url (#13487) 2026-08-05 17:10:34 +08:00
CityFun e01fb7c905 fix: Fixed bug with website template (#13484) 2026-08-05 16:56:08 +08:00
CityFun 83a3675a0c feat: QwenPaw support config username/password (#13478) 2026-08-05 16:35:09 +08:00
ssongliu 9dfa451fae fix: adjust host column display (#13477) 2026-08-05 13:52:33 +08:00
ssongliu 9204a287fd fix: improve community restore loading state (#13475)
* fix: improve community restore loading state

* fix: resume license page initialization after restore
2026-08-05 13:52:14 +08:00
f027507f9a fix: preserve active cronjob records during cleanup (#13474)
* fix: preserve active cronjob records during cleanup

* fix: delete cronjob records before removing files

Co-authored-by: ssongliu <73214554+ssongliu@users.noreply.github.com>

---------

Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
2026-08-05 11:32:20 +08:00
ssongliu 01aee89f3b fix: correct process start time in LXC (#13473)
* fix: correct process start time in LXC

* fix: reject unresolved process start times
2026-08-05 11:22:26 +08:00
CityFun 17285d4397 fix: Fixed issue with upgrade openresty failed (#13470)
* fix: Fixed issue with upgrade openresty failed

* fix: Fixed issue with upgrade openresty failed
2026-08-05 09:37:02 +08:00
蘭 91ae846418 feat: enhance archiving and extraction capabilities with additional compression formats and ownership preservation (#13458)
* feat: enhance archiving and extraction capabilities with additional compression formats and ownership preservation

* feat: enhance archiving and extraction capabilities with additional compression formats and ownership preservation

* feat: enhance archiving and extraction capabilities with additional compression formats and ownership preservation
2026-08-04 17:34:50 +08:00
ssongliu 1eb429631d fix: resolve compose project names consistently (#13468) 2026-08-04 17:17:07 +08:00
CityFun 6584e3b868 chore: update dependencies (#13467) 2026-08-04 16:46:27 +08:00
ssongliu 14e2294db9 feat: refine node health check settings (#13463) 2026-08-04 15:09:34 +08:00
ssongliu 26b69bc208 fix: bypass cached route in user info navigation (#13462)
* fix: bypass cached route in user info navigation

* fix: avoid caching entrance route
2026-08-04 15:09:17 +08:00
CityFun 2111d4c16b style: change table page config (#13456) 2026-08-04 10:45:13 +08:00
蘭 b682835b4e fix: Implement file sharing password processing and remote download file name processing (#13452) 2026-08-04 09:37:26 +08:00
蘭 c34ac2f31e feat: change some translate (#13451) 2026-08-04 09:37:08 +08:00
CityFun c28047374a feat: Fix the issue where website configuration files fail to restore from backups. (#13445) 2026-08-03 15:26:20 +08:00
ssongliu 60d16609f7 fix: clarify API trusted proxy placeholder (#13444) 2026-08-03 14:30:50 +08:00
CityFun 02ca9347dc feat: Update Xiaomi Models (#13443) 2026-08-03 14:28:26 +08:00
ssongliu d0187994ee fix: ignore empty directories in device clean scan (#13442)
* fix: ignore empty directories in device clean scan

* fix: skip zero-size device clean entries
2026-08-03 14:06:56 +08:00
蘭 be672d604f feat:Support logging in with oidc and saml2 (#13441)
* feat(auth): implement OIDC authentication endpoints and error handling

* feat(auth): add OIDC provider discovery endpoint and related functionality

* feat(auth): add SAML2 authentication support and related functionality

* feat(auth): add LDAP authentication support and related functionality

* fix(auth): improve login keydown handler for better event handling
2026-08-03 13:43:01 +08:00
CityFun 16e3d496eb Merge branch 'dev-v2' into pr@dev-v2@common (#13436) 2026-08-01 14:08:03 +08:00
ssongliu 9159ab842d feat: support filtering app store composes (#13432)
* feat: support filtering app store composes

* fix: initialize app store filter during setup
2026-07-31 18:27:10 +08:00
ssongliu 0d8835d494 fix: constrain dashboard column height (#13433) 2026-07-31 18:26:02 +08:00
ssongliu 9f9e3aacfc feat: add Community Edition restore UI (#13431) 2026-07-31 18:15:24 +08:00
CityFun 7bd11fe73e feat: Keep Website List Order Stable After Editing (#13424) 2026-07-31 11:05:23 +08:00
BugPleaseGoandCityFun e11dc5fadd Add Website Template (#13400)
* feat: Add Website Template

* fix: Don't display the template list

* feat: Add Mcp TopList

* docs: Remove Mcp TopList

* Add more languages

---------

Co-authored-by: CityFun <31820853+zhengkunwang223@users.noreply.github.com>
2026-07-31 10:01:20 +08:00
HynoR f35b0deb29 refactor(firewall): extract port forwarding subsystem (#13347)
Port forwarding no longer shares the filter client. FilterClient keeps only
filter capabilities, and forwarding gets its own adapter, service and boot
replay:

- utils/firewall/forwarding holds the provider adapters. firewalld uses native
  forward-port, ufw and iptables share the NAT implementation moved out of
  client/iptables/forward.go.
- service/forwarding.go owns base info, search, operate, enable and replay.
  The API keeps its routes and dispatches on name/type/operate.
- init/firewall replays forwarding through that service instead of loading NAT
  rule files inline.

Also adds 1PANEL_FORWARD to the IptablesOp name enum: the frontend already
sends {"name":"1PANEL_FORWARD","operate":"init-forward"} and the validator
rejected it with 400 before reaching the service. Besides that, the only
observable difference is that a forward-tab search no longer triggers the
port/address record cleanup goroutine on the side.
2026-07-30 14:07:41 +08:00
ssongliu 33b3eecb95 feat: unify pin actions (#13417) 2026-07-30 09:39:47 +08:00
CityFun 6ac7a5f167 feat: Optimize the application upgrade logic. (#13416) 2026-07-29 17:44:50 +08:00
CityFun a5fbbfc460 feat: Optimize the application upgrade logic. (#13415) 2026-07-29 17:19:52 +08:00
ssongliu 563df3da71 fix: support trusted proxies for API allowlist (#13409) 2026-07-29 16:48:59 +08:00
ssongliu 13e6bc4fac feat: separate website and standalone FTP identities (#13412) 2026-07-29 16:41:43 +08:00
ssongliu 357d77856a fix: align dashboard uptime with boot time (#13410)
* fix: align dashboard uptime with boot time

* fix: handle invalid dashboard boot time
2026-07-29 16:36:05 +08:00
ssongliu 4279339189 fix: simplify cronjob record duration display (#13398) 2026-07-29 10:57:40 +08:00
ssongliuandCopilot Autofix powered by AI 380033dfe0 fix: support MySQL backup GTID options (#13407)
* fix: support MySQL backup GTID options

* Potential fix for pull request finding

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

* Potential fix for pull request finding

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

---------

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
2026-07-29 10:42:21 +08:00
ssongliu 97d383ed12 feat: add explicit FTP identity initialization (#13390) 2026-07-28 17:07:57 +08:00
ssongliu 52e6a63ebc fix: improve host system log pagination (#13395) 2026-07-28 17:07:22 +08:00
CityFun 7506e709e2 Add plugin management support to OpenClaw. (#13388) 2026-07-28 14:20:55 +08:00
蘭 cf37de66fc feat(auth): add LDAP authentication support and related configurations (#13385)
* feat(auth): add LDAP authentication support and related configurations

* feat(ldap): implement LDAP synchronization features and enhance user import logic

* feat(auth): add authSource and authSourceStatus to user information

* feat(i18n): update LDAP synchronization error messages in multiple languages

* feat(i18n): update LDAP synchronization error messages in multiple languages

* feat(i18n): update LDAP synchronization error messages in multiple languages
2026-07-28 14:20:42 +08:00
CityFun e2754b447d feat: Add support for text-to-image APIs. (#13380)
* feat: Add support for text-to-image APIs.

* feat: Add support for text-to-image APIs.
2026-07-28 09:42:23 +08:00
694 changed files with 97357 additions and 21175 deletions
+23 -37
View File
@@ -1,9 +1,6 @@
<p align="center"><a href="https://1panel.pro"><img src="https://resource.1panel.pro/img/1panel-logo.png" alt="1Panel" width="300" /></a></p> <p align="center"><a href="https://1panel.pro"><img src="https://resource.1panel.pro/img/1panel-logo.png" alt="1Panel" width="300" /></a></p>
<h3 align="center">The open-source VPS control panel with native AI agent support</h3>
<p align="center"> <p align="center">
Trusted by <strong>2,000,000+</strong> self-hosters worldwide Loved by a global community of <strong>2.5M+</strong> self-hosters.
</p> </p>
<p align="center"> <p align="center">
@@ -12,7 +9,6 @@
<p align="center"> <p align="center">
<a href="https://www.gnu.org/licenses/gpl-3.0.html"><img src="https://shields.io/github/license/1Panel-dev/1Panel?color=%231890FF" alt="License: GPL v3"></a> <a href="https://www.gnu.org/licenses/gpl-3.0.html"><img src="https://shields.io/github/license/1Panel-dev/1Panel?color=%231890FF" alt="License: GPL v3"></a>
<a href="https://app.codacy.com/gh/1Panel-dev/1Panel"><img src="https://app.codacy.com/project/badge/Grade/da67574fd82b473992781d1386b937ef" alt="Codacy"></a>
<a href="https://discord.gg/bUpUqWqdRr"><img src="https://img.shields.io/discord/1318846410149335080?logo=discord&labelColor=%20%235462eb&logoColor=%20%23f5f5f5&color=%20%235462eb" alt="Discord"></a> <a href="https://discord.gg/bUpUqWqdRr"><img src="https://img.shields.io/discord/1318846410149335080?logo=discord&labelColor=%20%235462eb&logoColor=%20%23f5f5f5&color=%20%235462eb" alt="Discord"></a>
<a href="https://github.com/1Panel-dev/1Panel/releases"><img src="https://img.shields.io/github/v/release/1Panel-dev/1Panel" alt="GitHub release"></a> <a href="https://github.com/1Panel-dev/1Panel/releases"><img src="https://img.shields.io/github/v/release/1Panel-dev/1Panel" alt="GitHub release"></a>
<a href="https://github.com/1Panel-dev/1Panel"><img src="https://img.shields.io/github/stars/1Panel-dev/1Panel?color=%231890FF&style=flat-square" alt="Stars"></a> <a href="https://github.com/1Panel-dev/1Panel"><img src="https://img.shields.io/github/stars/1Panel-dev/1Panel?color=%231890FF&style=flat-square" alt="Stars"></a>
@@ -41,34 +37,28 @@
## What is 1Panel? ## What is 1Panel?
1Panel is a modern, open-source VPS control panel — and the only one with **native AI agent support**. Run Ollama models, deploy OpenClaw agents, and manage your entire server stack from one clean web interface. No CLI memorization required. 1Panel is a modern, open-source Linux server management panel and a lightweight AI management platform. Through an intuitive web interface, it provides users with comprehensive, one-stop server management capabilities:
- **AI Management**: Offers a unified management platform from bare metal to agents (Metal-to-Agent). It integrates an AI gateway, and Skills Hub, while supporting centralized management of agents and models.
👉 Watch the [2-minute introduction](https://www.youtube.com/watch?v=Jl_wqp-XA08) - **Efficient Visual Operations**: Easily manage Linux servers through a web-based GUI, streamlining tasks such as host monitoring, file management, database management, and container management.
- **Rapid Website Deployment**: Deeply integrates with popular website builders like WordPress and Halo. It enables one-click domain binding and SSL certificate configuration, significantly lowering the barrier to website creation.
- **Curated App Store**: Features a built-in store of high-quality open-source applications, providing one-click installation and upgrade services to effortlessly extend server capabilities.
- **Enterprise-Grade Security**: Deploys applications based on container technology to effectively minimize vulnerability exposure. It also provides security features such as WAF and log auditing to ensure comprehensive server protection.
- **One-Click Data Backup**: Supports one-click backup and restoration, and integrates with various cloud storage solutions to ensure data security and prevent loss.
## Why 1Panel? ## Why 1Panel?
| | 1Panel | cPanel / Plesk | aaPanel | Webmin | | | 1Panel | cPanel / Plesk | aaPanel | Webmin |
|--|--------|----------------|---------|--------| |--|--------|----------------|---------|--------|
| Free & open source | ✅ | ❌ | Partial | ✅ | | Free & open source | ✅ | ❌ | Partial | ✅ |
| Native AI agent runtime | ✅ | ❌ | ❌ | ❌ | | AI management | ✅ | ❌ | ❌ | ❌ |
| One-click app marketplace | ✅ 165+ apps | ❌ | ✅ | ❌ | | One-click app marketplace | ✅ 165+ apps | ❌ | ✅ | ❌ |
| Modern UI (post-2020) | ✅ | ❌ | Partial | ❌ | | Modern UI (post-2020) | ✅ | ❌ | Partial | ❌ |
| Docker / container management | ✅ | ❌ | ❌ | ❌ | | Docker / container management | ✅ | ❌ | ❌ | ❌ |
| Active development | ✅ | ✅ | ✅ | Slow | | Active development | ✅ | ✅ | ✅ | Slow |
## Key Features
- **AI Agent Runtime**: Deploy Ollama LLMs, spin up OpenClaw personal agents, and monitor GPU utilization — all from the dashboard. No separate AI stack to manage.
- **One-Click Website Deployment**: Launch production-ready websites with automatic domain binding, SSL provisioning, and Nginx config — zero manual setup.
- **App Marketplace**: 165+ trusted open-source apps (Nextcloud, Bitwarden, Umami, NocoBase, and more) installed and updated with a single click.
- **Docker & Container Management**: Create, start, stop, and inspect containers, images, networks, and volumes through a visual UI — no CLI juggling.
- **Security Out of the Box**: Firewall rules, fail2ban, container isolation, WAF, and audit logs — configured and running from day one.
- **Backup & Restore**: Schedule automated backups to AWS S3, Cloudflare R2, or local storage. Restore any snapshot in one click.
## Quick Start ## Quick Start
> **Requirements:** Linux VPS (Debian / Ubuntu / CentOS / Rocky), 1 GB RAM, internet access. Prepare your Linux server and run the following script:
> Takes ~60 seconds.
```bash ```bash
bash -c "$(curl -sSL https://resource.1panel.pro/v2/quick_start.sh)" bash -c "$(curl -sSL https://resource.1panel.pro/v2/quick_start.sh)"
@@ -83,24 +73,20 @@ Run `1pctl user-info` via SSH if you need to retrieve your access credentials.
## Pro Edition ## Pro Edition
1Panel OSS is free forever. Pro adds features built for teams and production workloads: 1Panel OSS is free forever. 1Panel Pro and Ent adds features built for teams and production workloads:
| Feature | OSS | Pro | | Feature | OSS | Pro | Ent |
|---------|:---:|:---:| |---------|:---:|:---:|:---:|
| One-click app installs | ✅ | ✅ | | One-click app installs | ✅ | ✅ | ✅ |
| AI agents (OpenClaw) | 1 agent | Unlimited | | AI agents (OpenClaw) | 5 agent | Unlimited | ✅ |
| WAF & advanced security | Basic | ✅ | | WAF & advanced security | Basic | ✅ | ✅ |
| Website tamper protection | ❌ | ✅ | | Website tamper protection | ❌ | ✅ | ✅ |
| Website uptime monitoring | ❌ | ✅ | | Website uptime monitoring | ❌ | ✅ | ✅ |
| Multi-node management | ❌ | ✅ | | Multi-node management | ❌ | ✅ | ✅ |
| Custom logo & theme | ❌ | ✅ | | Custom logo & theme | ❌ | ✅ | ✅ |
| Priority support | ❌ | ✅ | | KVM Web UI | ❌ | ❌ | ✅ |
| AI Gateway | ❌ | ❌ | ✅ |
**From $80/year.** [Compare plans & start 30-day free trial →](https://1panel.pro/pricing) | Priority support | ❌ | ❌ | ✅ |
## Star History
[![Star History Chart](https://api.star-history.com/svg?repos=1Panel-dev/1Panel&type=Date)](https://star-history.com/#1Panel-dev/1Panel&Date)
## Community & Support ## Community & Support
+82
View File
@@ -1378,6 +1378,88 @@ func (b *BaseApi) UninstallAgentSkill(c *gin.Context) {
helper.Success(c) helper.Success(c)
} }
// @Tags AI
// @Summary List OpenClaw plugins
// @Accept json
// @Param request body dto.AgentPluginsReq true "request"
// @Success 200 {array} dto.AgentPluginItem
// @Security ApiKeyAuth
// @Security Timestamp
// @Router /ai/agents/plugins/list [post]
func (b *BaseApi) ListAgentPlugins(c *gin.Context) {
var req dto.AgentPluginsReq
if err := helper.CheckBindAndValidate(&req, c); err != nil {
return
}
data, err := agentService.ListPlugins(req)
if err != nil {
helper.BadRequest(c, err)
return
}
helper.SuccessWithData(c, data)
}
// @Tags AI
// @Summary Search OpenClaw plugins
// @Accept json
// @Param request body dto.AgentPluginSearchReq true "request"
// @Success 200 {array} dto.AgentPluginSearchItem
// @Security ApiKeyAuth
// @Security Timestamp
// @Router /ai/agents/plugins/search [post]
func (b *BaseApi) SearchAgentPlugins(c *gin.Context) {
var req dto.AgentPluginSearchReq
if err := helper.CheckBindAndValidate(&req, c); err != nil {
return
}
data, err := agentService.SearchPlugins(req)
if err != nil {
helper.BadRequest(c, err)
return
}
helper.SuccessWithData(c, data)
}
// @Tags AI
// @Summary Install an OpenClaw marketplace plugin
// @Accept json
// @Param request body dto.AgentPluginMarketInstallReq true "request"
// @Success 200
// @Security ApiKeyAuth
// @Security Timestamp
// @Router /ai/agents/plugins/install [post]
func (b *BaseApi) InstallAgentMarketPlugin(c *gin.Context) {
var req dto.AgentPluginMarketInstallReq
if err := helper.CheckBindAndValidate(&req, c); err != nil {
return
}
if err := agentService.InstallMarketPlugin(req); err != nil {
helper.BadRequest(c, err)
return
}
helper.Success(c)
}
// @Tags AI
// @Summary Operate an OpenClaw plugin
// @Accept json
// @Param request body dto.AgentPluginOperateReq true "request"
// @Success 200
// @Security ApiKeyAuth
// @Security Timestamp
// @Router /ai/agents/plugins/operate [post]
func (b *BaseApi) OperateAgentPlugin(c *gin.Context) {
var req dto.AgentPluginOperateReq
if err := helper.CheckBindAndValidate(&req, c); err != nil {
return
}
if err := agentService.OperatePlugin(req); err != nil {
helper.BadRequest(c, err)
return
}
helper.Success(c)
}
// @Tags AI // @Tags AI
// @Summary Login Agent Weixin channel // @Summary Login Agent Weixin channel
// @Accept json // @Accept json
+40
View File
@@ -2,11 +2,14 @@ package v2
import ( import (
"errors" "errors"
"net/http"
"net/url" "net/url"
"strings" "strings"
"github.com/1Panel-dev/1Panel/agent/app/api/v2/helper" "github.com/1Panel-dev/1Panel/agent/app/api/v2/helper"
"github.com/1Panel-dev/1Panel/agent/app/dto" "github.com/1Panel-dev/1Panel/agent/app/dto"
"github.com/1Panel-dev/1Panel/agent/app/repo"
"github.com/1Panel-dev/1Panel/agent/constant"
"github.com/gin-gonic/gin" "github.com/gin-gonic/gin"
) )
@@ -294,6 +297,34 @@ func (b *BaseApi) UpdateAlertConfig(c *gin.Context) {
return return
} }
if err := alertService.UpdateAlertConfig(req, loadAuditUser(c)); err != nil { if err := alertService.UpdateAlertConfig(req, loadAuditUser(c)); err != nil {
switch {
case errors.Is(err, repo.ErrAlertConfigRevisionConflict):
helper.ErrorWithBusinessCode(c, http.StatusConflict, "ALERT_CONFIG_REVISION_CONFLICT", "ErrInvalidParams", err)
case errors.Is(err, repo.ErrAlertConfigRevisionRequired):
helper.ErrorWithBusinessCode(c, http.StatusConflict, "ALERT_CONFIG_REVISION_REQUIRED", "ErrInvalidParams", err)
default:
helper.InternalServer(c, err)
}
return
}
helper.Success(c)
}
// @Tags Alert
// @Summary Update alert config status
// @Accept json
// @Param request body dto.AlertConfigStatusUpdate true "request"
// @Success 200
// @Security ApiKeyAuth
// @Security Timestamp
// @Router /alert/config/status [post]
// @x-panel-log {"bodyKeys":["id","status"],"paramKeys":[],"BeforeFunctions":[],"formatZH":"更新告警配置状态 [id][status]","formatEN":"update alert config status [id][status]"}
func (b *BaseApi) UpdateAlertConfigStatus(c *gin.Context) {
var req dto.AlertConfigStatusUpdate
if err := helper.CheckBindAndValidate(&req, c); err != nil {
return
}
if err := alertService.UpdateAlertConfigStatus(req, loadAuditUser(c)); err != nil {
helper.InternalServer(c, err) helper.InternalServer(c, err)
return return
} }
@@ -346,6 +377,15 @@ func (b *BaseApi) TestAlertConfig(c *gin.Context) {
if err := helper.CheckBindAndValidate(&req, c); err != nil { if err := helper.CheckBindAndValidate(&req, c); err != nil {
return return
} }
if req.Type == constant.Custom {
result, err := alertService.TestCustomAlertConfig(req)
if err != nil {
helper.InternalServer(c, err)
return
}
helper.SuccessWithData(c, result)
return
}
flag, err := alertService.TestAlertConfig(req) flag, err := alertService.TestAlertConfig(req)
if err != nil { if err != nil {
helper.InternalServer(c, err) helper.InternalServer(c, err)
+21 -1
View File
@@ -439,7 +439,7 @@ func (b *BaseApi) ContainerItemStats(c *gin.Context) {
return return
} }
data, err := containerService.ContainerItemStats(req) data, err := containerService.ContainerItemStats(c.Request.Context(), req)
if err != nil { if err != nil {
helper.InternalServer(c, err) helper.InternalServer(c, err)
return return
@@ -881,6 +881,26 @@ func (b *BaseApi) ComposeUpdate(c *gin.Context) {
helper.Success(c) helper.Success(c)
} }
// @Tags Container Compose
// @Summary Pin compose
// @Accept json
// @Param request body dto.ComposePin true "request"
// @Success 200
// @Security ApiKeyAuth
// @Security Timestamp
// @Router /containers/compose/pin [post]
func (b *BaseApi) ComposePin(c *gin.Context) {
var req dto.ComposePin
if err := helper.CheckBindAndValidate(&req, c); err != nil {
return
}
if err := containerService.ComposePin(req); err != nil {
helper.InternalServer(c, err)
return
}
helper.Success(c)
}
// @Tags Container Compose // @Tags Container Compose
// @Summary Load compose environment variables // @Summary Load compose environment variables
// @Accept json // @Accept json
+10 -4
View File
@@ -86,17 +86,23 @@ func (b *BaseApi) CheckHasCli(c *gin.Context) {
// @Tags Database Redis // @Tags Database Redis
// @Summary Install redis-cli // @Summary Install redis-cli
// @Success 200 // @Accept json
// @Param request body dto.RedisCliInstall true "request"
// @Success 200 {string} string
// @Security ApiKeyAuth // @Security ApiKeyAuth
// @Security Timestamp // @Security Timestamp
// @Router /databases/redis/install/cli [post] // @Router /databases/redis/install/cli [post]
func (b *BaseApi) InstallCli(c *gin.Context) { func (b *BaseApi) InstallCli(c *gin.Context) {
if err := redisService.InstallCli(); err != nil { var req dto.RedisCliInstall
if err := helper.CheckBindAndValidate(&req, c); err != nil {
return
}
data, err := redisService.InstallCli(req)
if err != nil {
helper.InternalServer(c, err) helper.InternalServer(c, err)
return return
} }
helper.SuccessWithData(c, data)
helper.Success(c)
} }
// @Tags Database Redis // @Tags Database Redis
+4 -1
View File
@@ -42,7 +42,9 @@ var (
fileShareService = service.NewIFileShareService() fileShareService = service.NewIFileShareService()
sshService = service.NewISSHService() sshService = service.NewISSHService()
firewallService = service.NewIFirewallService() firewallService = service.NewIFirewallService()
iptablesService = service.NewIIptablesService() firewallSettingService = service.NewIFirewallSettingService()
forwardingService = service.NewIForwardingService()
dockerPortGuardService = service.NewIDockerPortGuardService()
monitorService = service.NewIMonitorService() monitorService = service.NewIMonitorService()
systemService = service.NewISystemService() systemService = service.NewISystemService()
runtimeDiagnosticsService = service.NewIRuntimeDiagnosticsService() runtimeDiagnosticsService = service.NewIRuntimeDiagnosticsService()
@@ -60,6 +62,7 @@ var (
websiteDnsAccountService = service.NewIWebsiteDnsAccountService() websiteDnsAccountService = service.NewIWebsiteDnsAccountService()
websiteSSLService = service.NewIWebsiteSSLService() websiteSSLService = service.NewIWebsiteSSLService()
websiteAcmeAccountService = service.NewIWebsiteAcmeAccountService() websiteAcmeAccountService = service.NewIWebsiteAcmeAccountService()
websiteTemplateService = service.NewIWebsiteTemplateService()
nginxService = service.NewINginxService() nginxService = service.NewINginxService()
+470 -23
View File
@@ -35,6 +35,81 @@ var cancelledChunkUploads = struct {
ids map[string]struct{} ids map[string]struct{}
}{ids: make(map[string]struct{})} }{ids: make(map[string]struct{})}
type chunkUploadLock struct {
mutex sync.Mutex
refs int
}
var chunkUploadLocks = struct {
sync.Mutex
items map[string]*chunkUploadLock
}{items: make(map[string]*chunkUploadLock)}
type completedChunkUpload struct {
dstDir string
filename string
fileSize int64
}
var completedChunkUploads = struct {
sync.RWMutex
items map[string]completedChunkUpload
}{items: make(map[string]completedChunkUpload)}
var activeChunkUploadTTL = 24 * time.Hour
var (
errChunkUploadCancelled = errors.New("upload cancelled")
errInvalidChunkUpload = errors.New("invalid chunk upload")
)
type activeChunkUpload struct {
upload completedChunkUpload
expiresAt time.Time
timer *time.Timer
}
var activeChunkUploads = struct {
sync.RWMutex
items map[string]activeChunkUpload
}{items: make(map[string]activeChunkUpload)}
type resumableUploadChunk struct {
UploadID string
Filename string
DstDir string
ChunkIndex int
ChunkCount int
Offset int64
FileSize int64
Overwrite bool
}
func invalidChunkUploadError(message string) error {
return fmt.Errorf("%w: %s", errInvalidChunkUpload, message)
}
func isRetryableChunkUploadError(err error) bool {
if err == nil {
return false
}
if errors.Is(err, errChunkUploadCancelled) ||
errors.Is(err, errInvalidChunkUpload) ||
errors.Is(err, os.ErrExist) ||
errors.Is(err, os.ErrPermission) ||
errors.Is(err, os.ErrInvalid) ||
errors.Is(err, syscall.ENOSPC) ||
errors.Is(err, syscall.EDQUOT) ||
errors.Is(err, syscall.EROFS) ||
errors.Is(err, syscall.EFBIG) ||
errors.Is(err, syscall.ENAMETOOLONG) ||
errors.Is(err, syscall.ENOTDIR) ||
errors.Is(err, syscall.EISDIR) {
return false
}
return true
}
// @Tags File // @Tags File
// @Summary List files // @Summary List files
// @Accept json // @Accept json
@@ -474,11 +549,7 @@ func (b *BaseApi) UploadFiles(c *gin.Context) {
continue continue
} }
dstInfo, statErr := os.Stat(dstFilename) dstInfo, statErr := os.Stat(dstFilename)
if overwrite { err = finalizeUploadedFile(tmpFilename, dstFilename, overwrite)
_ = os.Remove(dstFilename)
}
err = os.Rename(tmpFilename, dstFilename)
if err != nil { if err != nil {
_ = os.Remove(tmpFilename) _ = os.Remove(tmpFilename)
e := fmt.Errorf("upload [%s] file failed, err: %v", file.Filename, err) e := fmt.Errorf("upload [%s] file failed, err: %v", file.Filename, err)
@@ -613,10 +684,35 @@ func (b *BaseApi) StopWget(c *gin.Context) {
return return
} }
files.CancelDownload(req.Key) if err := files.CancelDownload(req.Key); err != nil {
helper.InternalServer(c, err)
return
}
helper.Success(c) helper.Success(c)
} }
// @Tags File
// @Summary Remove finished download progress records without deleting files
// @Accept json
// @Param request body request.FileProcessRemoveReq true "request"
// @Success 200 {object} response.FileProcessKeys
// @Security ApiKeyAuth
// @Security Timestamp
// @Router /files/wget/process/remove [post]
// @x-panel-log {"bodyKeys":["keys"],"paramKeys":[],"BeforeFunctions":[],"formatZH":"移除已结束下载记录 [keys]","formatEN":"Remove finished download records [keys]"}
func (b *BaseApi) RemoveWgetRecords(c *gin.Context) {
var req request.FileProcessRemoveReq
if err := helper.CheckBindAndValidate(&req, c); err != nil {
return
}
keys, err := files.RemoveDownloadRecords(req.Keys)
if err != nil {
helper.BadRequest(c, err)
return
}
helper.SuccessWithData(c, response.FileProcessKeys{Keys: keys})
}
// @Tags File // @Tags File
// @Summary Move file // @Summary Move file
// @Accept json // @Accept json
@@ -638,6 +734,26 @@ func (b *BaseApi) MoveFile(c *gin.Context) {
helper.Success(c) helper.Success(c)
} }
// @Tags File
// @Summary Stop file move task
// @Accept json
// @Param request body request.FileMoveStopReq true "request"
// @Success 200
// @Security ApiKeyAuth
// @Security Timestamp
// @Router /files/move/stop [post]
func (b *BaseApi) StopMoveFile(c *gin.Context) {
var req request.FileMoveStopReq
if err := helper.CheckBindAndValidate(&req, c); err != nil {
return
}
if err := fileService.StopMvFile(req.TaskID); err != nil {
helper.InternalServer(c, err)
return
}
helper.Success(c)
}
// @Tags File // @Tags File
// @Summary Download file // @Summary Download file
// @Accept json // @Accept json
@@ -792,6 +908,289 @@ func (b *BaseApi) DepthDirSize(c *gin.Context) {
helper.SuccessWithData(c, res) helper.SuccessWithData(c, res)
} }
func lockChunkUpload(uploadID string) func() {
chunkUploadLocks.Lock()
lock, ok := chunkUploadLocks.items[uploadID]
if !ok {
lock = &chunkUploadLock{}
chunkUploadLocks.items[uploadID] = lock
}
lock.refs++
chunkUploadLocks.Unlock()
lock.mutex.Lock()
return func() {
lock.mutex.Unlock()
chunkUploadLocks.Lock()
lock.refs--
if lock.refs == 0 {
delete(chunkUploadLocks.items, uploadID)
}
chunkUploadLocks.Unlock()
}
}
func resumableUploadPartPath(dstDir, uploadID string) string {
return filepath.Join(dstDir, fmt.Sprintf(".1panel-upload-%s.part", uploadID))
}
func finalizeUploadedFile(tmpFile, dstFile string, overwrite bool) error {
if overwrite {
return os.Rename(tmpFile, dstFile)
}
if err := os.Link(tmpFile, dstFile); err != nil {
return err
}
if err := os.Remove(tmpFile); err != nil {
if rollbackErr := os.Remove(dstFile); rollbackErr != nil {
return fmt.Errorf("remove upload temporary file failed: %v, rollback destination failed: %w", err, rollbackErr)
}
return err
}
return nil
}
func registerActiveChunkUpload(uploadID string, upload completedChunkUpload) error {
activeChunkUploads.Lock()
defer activeChunkUploads.Unlock()
if active, ok := activeChunkUploads.items[uploadID]; ok {
if active.upload != upload {
return invalidChunkUploadError("upload ID is already used by another file")
}
active.timer.Stop()
}
expiresAt := time.Now().Add(activeChunkUploadTTL)
timer := time.AfterFunc(activeChunkUploadTTL, func() {
expireActiveChunkUpload(uploadID, expiresAt)
})
activeChunkUploads.items[uploadID] = activeChunkUpload{
upload: upload,
expiresAt: expiresAt,
timer: timer,
}
return nil
}
func loadActiveChunkUpload(uploadID string) (completedChunkUpload, bool) {
activeChunkUploads.RLock()
active, ok := activeChunkUploads.items[uploadID]
activeChunkUploads.RUnlock()
return active.upload, ok
}
func deleteActiveChunkUpload(uploadID string) {
activeChunkUploads.Lock()
if active, ok := activeChunkUploads.items[uploadID]; ok {
active.timer.Stop()
}
delete(activeChunkUploads.items, uploadID)
activeChunkUploads.Unlock()
}
func discardActiveChunkUpload(uploadID, partFile string) error {
deleteActiveChunkUpload(uploadID)
if err := os.Remove(partFile); err != nil && !os.IsNotExist(err) {
return fmt.Errorf("remove upload temporary file failed: %w", err)
}
return nil
}
func finalizeActiveChunkUpload(uploadID, partFile, dstFile string, overwrite bool) error {
if err := finalizeUploadedFile(partFile, dstFile, overwrite); err != nil {
if removeErr := discardActiveChunkUpload(uploadID, partFile); removeErr != nil {
return errors.Join(err, removeErr)
}
return err
}
return nil
}
func expireActiveChunkUpload(uploadID string, expiresAt time.Time) {
unlock := lockChunkUpload(uploadID)
defer unlock()
activeChunkUploads.Lock()
active, ok := activeChunkUploads.items[uploadID]
if !ok || !active.expiresAt.Equal(expiresAt) {
activeChunkUploads.Unlock()
return
}
delete(activeChunkUploads.items, uploadID)
activeChunkUploads.Unlock()
partFile := resumableUploadPartPath(active.upload.dstDir, uploadID)
if err := os.Remove(partFile); err != nil && !os.IsNotExist(err) {
global.LOG.Warnf("remove inactive upload part [%s] failed: %v", partFile, err)
}
}
func removeActiveResumableUploadPart(uploadID string) error {
unlock := lockChunkUpload(uploadID)
defer unlock()
upload, ok := loadActiveChunkUpload(uploadID)
if !ok {
return nil
}
err := os.Remove(resumableUploadPartPath(upload.dstDir, uploadID))
if err == nil || os.IsNotExist(err) {
deleteActiveChunkUpload(uploadID)
return nil
}
return err
}
func loadCompletedChunkUpload(uploadID string) (completedChunkUpload, bool) {
completedChunkUploads.RLock()
completed, ok := completedChunkUploads.items[uploadID]
completedChunkUploads.RUnlock()
return completed, ok
}
func markChunkUploadCompleted(uploadID string, completed completedChunkUpload) {
completedChunkUploads.Lock()
completedChunkUploads.items[uploadID] = completed
completedChunkUploads.Unlock()
time.AfterFunc(10*time.Minute, func() {
completedChunkUploads.Lock()
delete(completedChunkUploads.items, uploadID)
completedChunkUploads.Unlock()
})
}
func writeResumableUploadChunk(chunk resumableUploadChunk, chunkData []byte) error {
unlock := lockChunkUpload(chunk.UploadID)
defer unlock()
if chunkUploadCancelled(chunk.UploadID) {
return errChunkUploadCancelled
}
if chunk.UploadID == "" || filepath.Base(chunk.UploadID) != chunk.UploadID || strings.ContainsAny(chunk.UploadID, `/\`) {
return invalidChunkUploadError("invalid upload ID")
}
if chunk.Filename == "" || filepath.Base(chunk.Filename) != chunk.Filename || strings.ContainsAny(chunk.Filename, `/\`) {
return invalidChunkUploadError("invalid filename")
}
if strings.TrimSpace(chunk.DstDir) == "" {
return invalidChunkUploadError("upload destination is required")
}
dstDir := filepath.Clean(strings.TrimSpace(chunk.DstDir))
if chunk.ChunkCount <= 0 || chunk.ChunkIndex < 0 || chunk.ChunkIndex >= chunk.ChunkCount {
return invalidChunkUploadError("invalid chunk index")
}
if chunk.FileSize <= 0 || chunk.Offset < 0 || chunk.Offset > chunk.FileSize {
return invalidChunkUploadError("invalid upload offset")
}
chunkEnd := chunk.Offset + int64(len(chunkData))
if chunkEnd > chunk.FileSize {
return invalidChunkUploadError("chunk exceeds file size")
}
if chunk.ChunkIndex+1 == chunk.ChunkCount {
if chunkEnd != chunk.FileSize {
return invalidChunkUploadError("final chunk does not match file size")
}
} else if chunkEnd >= chunk.FileSize {
return invalidChunkUploadError("non-final chunk reaches file size")
}
if completed, ok := loadCompletedChunkUpload(chunk.UploadID); ok {
if completed.dstDir == dstDir && completed.filename == chunk.Filename && completed.fileSize == chunk.FileSize {
return nil
}
return invalidChunkUploadError("upload ID has already completed another file")
}
upload := completedChunkUpload{dstDir: dstDir, filename: chunk.Filename, fileSize: chunk.FileSize}
if err := registerActiveChunkUpload(chunk.UploadID, upload); err != nil {
return err
}
mode, err := files.GetParentMode(dstDir)
if err != nil {
return err
}
if err = os.MkdirAll(dstDir, mode); err != nil {
return err
}
dstDirInfo, err := os.Stat(dstDir)
if err != nil {
return err
}
if !dstDirInfo.IsDir() {
return invalidChunkUploadError(fmt.Sprintf("upload destination [%s] is not a directory", dstDir))
}
dstFile := filepath.Join(dstDir, chunk.Filename)
partFile := resumableUploadPartPath(dstDir, chunk.UploadID)
if dstFile == partFile {
return invalidChunkUploadError("filename conflicts with upload temporary file")
}
fileMode := dstDirInfo.Mode().Perm()
ownerInfo := dstDirInfo
if dstInfo, statErr := os.Stat(dstFile); statErr == nil {
if !chunk.Overwrite {
if err := discardActiveChunkUpload(chunk.UploadID, partFile); err != nil {
return errors.Join(os.ErrExist, err)
}
return os.ErrExist
}
fileMode = dstInfo.Mode().Perm()
ownerInfo = dstInfo
} else if !os.IsNotExist(statErr) {
return statErr
}
part, err := os.OpenFile(partFile, os.O_CREATE|os.O_RDWR, fileMode)
if err != nil {
return err
}
partClosed := false
defer func() {
if !partClosed {
_ = part.Close()
}
}()
if stat, statErr := part.Stat(); statErr != nil {
return statErr
} else if chunk.Offset > stat.Size() {
return invalidChunkUploadError(fmt.Sprintf("unexpected upload offset %d, current size is %d", chunk.Offset, stat.Size()))
} else if chunk.Offset < stat.Size() && chunkEnd > stat.Size() {
if err = part.Truncate(chunk.Offset); err != nil {
return err
}
}
if _, err = part.WriteAt(chunkData, chunk.Offset); err != nil {
return err
}
if chunk.ChunkIndex+1 != chunk.ChunkCount {
return nil
}
partInfo, err := part.Stat()
if err != nil {
return err
}
if partInfo.Size() != chunk.FileSize {
return invalidChunkUploadError(fmt.Sprintf("uploaded file size mismatch: expected %d, got %d", chunk.FileSize, partInfo.Size()))
}
if err = part.Close(); err != nil {
return err
}
partClosed = true
if err = os.Chmod(partFile, fileMode); err != nil {
return err
}
if stat, ok := ownerInfo.Sys().(*syscall.Stat_t); ok {
if err = os.Chown(partFile, int(stat.Uid), int(stat.Gid)); err != nil {
return err
}
}
if chunkUploadCancelled(chunk.UploadID) {
return errChunkUploadCancelled
}
if err = finalizeActiveChunkUpload(chunk.UploadID, partFile, dstFile, chunk.Overwrite); err != nil {
return err
}
markChunkUploadCompleted(chunk.UploadID, upload)
deleteActiveChunkUpload(chunk.UploadID)
return nil
}
func mergeChunks(fileName string, fileDir string, dstDir string, chunkCount int, overwrite bool) error { func mergeChunks(fileName string, fileDir string, dstDir string, chunkCount int, overwrite bool) error {
defer func() { defer func() {
_ = os.RemoveAll(fileDir) _ = os.RemoveAll(fileDir)
@@ -871,6 +1270,10 @@ func (b *BaseApi) UploadChunkFiles(c *gin.Context) {
helper.BadRequest(c, err) helper.BadRequest(c, err)
return return
} }
if chunkCount <= 0 || chunkIndex < 0 || chunkIndex >= chunkCount {
helper.BadRequest(c, errors.New("invalid chunk index"))
return
}
fileOp := files.NewFileOp() fileOp := files.NewFileOp()
tmpDir := path.Join(global.Dir.TmpDir, "upload") tmpDir := path.Join(global.Dir.TmpDir, "upload")
if !fileOp.Stat(tmpDir) { if !fileOp.Stat(tmpDir) {
@@ -885,20 +1288,25 @@ func (b *BaseApi) UploadChunkFiles(c *gin.Context) {
return return
} }
uploadID := strings.TrimSpace(c.PostForm("uploadID")) uploadID := strings.TrimSpace(c.PostForm("uploadID"))
resumable := c.PostForm("fileSize") != "" || c.PostForm("offset") != ""
cancellable := uploadID != "" cancellable := uploadID != ""
if cancellable && (filepath.Base(uploadID) != uploadID || strings.ContainsAny(uploadID, `/\\`)) { if cancellable && (filepath.Base(uploadID) != uploadID || strings.ContainsAny(uploadID, `/\\`)) {
helper.BadRequest(c, errors.New("invalid upload ID")) helper.BadRequest(c, errors.New("invalid upload ID"))
return return
} }
if resumable && !cancellable {
helper.BadRequest(c, errors.New("upload ID is required"))
return
}
if !cancellable { if !cancellable {
uploadID = filename uploadID = filename
} }
fileDir := filepath.Join(tmpDir, uploadID) fileDir := filepath.Join(tmpDir, uploadID)
if cancellable && chunkUploadCancelled(uploadID) { if cancellable && chunkUploadCancelled(uploadID) {
helper.BadRequest(c, errors.New("upload cancelled")) helper.BadRequest(c, errChunkUploadCancelled)
return return
} }
if chunkIndex == 0 { if !resumable && chunkIndex == 0 {
if fileOp.Stat(fileDir) { if fileOp.Stat(fileDir) {
_ = fileOp.DeleteDir(fileDir) _ = fileOp.DeleteDir(fileDir)
} }
@@ -907,32 +1315,67 @@ func (b *BaseApi) UploadChunkFiles(c *gin.Context) {
filePath := filepath.Join(fileDir, filename) filePath := filepath.Join(fileDir, filename)
defer func() { defer func() {
if err != nil { if !resumable && err != nil {
_ = os.RemoveAll(fileDir) _ = os.RemoveAll(fileDir)
} }
}() }()
var ( chunkData, err := io.ReadAll(uploadFile)
emptyFile *os.File
chunkData []byte
)
emptyFile, err = os.Create(filePath)
if err != nil {
helper.BadRequest(c, err)
return
}
defer emptyFile.Close()
chunkData, err = io.ReadAll(uploadFile)
if err != nil { if err != nil {
helper.InternalServer(c, buserr.WithMap("ErrFileUpload", map[string]interface{}{"name": filename, "detail": err.Error()}, err)) helper.InternalServer(c, buserr.WithMap("ErrFileUpload", map[string]interface{}{"name": filename, "detail": err.Error()}, err))
return return
} }
if cancellable && chunkUploadCancelled(uploadID) { if cancellable && chunkUploadCancelled(uploadID) {
err = errors.New("upload cancelled") err = errChunkUploadCancelled
helper.BadRequest(c, err) helper.BadRequest(c, err)
return return
} }
if resumable {
offset, parseErr := strconv.ParseInt(c.PostForm("offset"), 10, 64)
if parseErr != nil {
helper.BadRequest(c, parseErr)
return
}
fileSize, parseErr := strconv.ParseInt(c.PostForm("fileSize"), 10, 64)
if parseErr != nil {
helper.BadRequest(c, parseErr)
return
}
overwrite := true
if ow := c.PostForm("overwrite"); ow != "" {
overwrite, _ = strconv.ParseBool(ow)
}
err = writeResumableUploadChunk(resumableUploadChunk{
UploadID: uploadID,
Filename: filename,
DstDir: c.PostForm("path"),
ChunkIndex: chunkIndex,
ChunkCount: chunkCount,
Offset: offset,
FileSize: fileSize,
Overwrite: overwrite,
}, chunkData)
if err != nil {
uploadErr := buserr.WithMap("ErrFileUpload", map[string]interface{}{"name": filename, "detail": err.Error()}, err)
helper.ErrorWithDetailAndData(c, http.StatusInternalServerError, "ErrInternalServer", uploadErr, gin.H{
"retryable": isRetryableChunkUploadError(err),
})
return
}
if chunkIndex+1 == chunkCount {
cancelledChunkUploads.Lock()
delete(cancelledChunkUploads.ids, uploadID)
cancelledChunkUploads.Unlock()
}
helper.SuccessWithData(c, true)
return
}
emptyFile, err := os.Create(filePath)
if err != nil {
helper.BadRequest(c, err)
return
}
defer emptyFile.Close()
chunkPath := filepath.Join(fileDir, fmt.Sprintf("%s.%d", filename, chunkIndex)) chunkPath := filepath.Join(fileDir, fmt.Sprintf("%s.%d", filename, chunkIndex))
err = os.WriteFile(chunkPath, chunkData, constant.DirPerm) err = os.WriteFile(chunkPath, chunkData, constant.DirPerm)
@@ -985,6 +1428,10 @@ func (b *BaseApi) StopChunkUpload(c *gin.Context) {
helper.InternalServer(c, err) helper.InternalServer(c, err)
return return
} }
if err := removeActiveResumableUploadPart(uploadID); err != nil {
helper.InternalServer(c, err)
return
}
helper.Success(c) helper.Success(c)
} }
+587 -246
View File
@@ -1,26 +1,53 @@
package v2 package v2
import ( import (
"errors"
"github.com/1Panel-dev/1Panel/agent/buserr"
"net/http"
"github.com/1Panel-dev/1Panel/agent/app/api/v2/helper" "github.com/1Panel-dev/1Panel/agent/app/api/v2/helper"
"github.com/1Panel-dev/1Panel/agent/app/dto" "github.com/1Panel-dev/1Panel/agent/app/dto"
"github.com/1Panel-dev/1Panel/agent/global"
"github.com/1Panel-dev/1Panel/agent/utils/docker"
"github.com/1Panel-dev/1Panel/agent/utils/firewall/filter"
"github.com/gin-gonic/gin" "github.com/gin-gonic/gin"
) )
func (b *BaseApi) UpdatePanelFirewallPort(c *gin.Context) {
if !global.IsMaster {
c.AbortWithStatus(http.StatusForbidden)
return
}
var request struct {
OldPort uint `json:"oldPort" validate:"required,min=1,max=65535"`
NewPort uint `json:"newPort" validate:"required,min=1,max=65535"`
}
if err := helper.CheckBindAndValidate(&request, c); err != nil {
return
}
if err := firewallService.UpdatePanelPort(c.Request.Context(), request.OldPort, request.NewPort); err != nil {
helper.InternalServer(c, err)
return
}
helper.Success(c)
}
// @Tags Firewall // @Tags Firewall
// @Summary Load firewall base info // @Summary Load firewall base info
// @Accept json // @Accept json
// @Param request body dto.OperationWithName true "request" // @Param request body dto.OperationWithName true "request"
// @Success 200 {object} dto.FirewallBaseInfo // @Success 200 {object} dto.FirewallSubsystemStatus
// @Security ApiKeyAuth // @Security ApiKeyAuth
// @Security Timestamp // @Security Timestamp
// @Router /hosts/firewall/base [post] // @Router /hosts/firewall/base [post]
func (b *BaseApi) LoadFirewallBaseInfo(c *gin.Context) { func (b *BaseApi) LoadFirewallBaseInfo(c *gin.Context) {
var req dto.OperationWithName var request dto.OperationWithName
if err := helper.CheckBindAndValidate(&req, c); err != nil { if err := helper.CheckBindAndValidate(&request, c); err != nil {
return return
} }
data, err := firewallService.LoadBaseInfo(req.Name) data, err := firewallService.LoadBaseInfo(request.Name)
if err != nil { if err != nil {
helper.InternalServer(c, err) helper.InternalServer(c, err)
return return
@@ -29,311 +56,625 @@ func (b *BaseApi) LoadFirewallBaseInfo(c *gin.Context) {
helper.SuccessWithData(c, data) helper.SuccessWithData(c, data)
} }
// @Tags Firewall
// @Summary Page firewall rules
// @Accept json
// @Param request body dto.RuleSearch true "request"
// @Success 200 {object} dto.PageResult
// @Security ApiKeyAuth
// @Security Timestamp
// @Router /hosts/firewall/search [post]
func (b *BaseApi) SearchFirewallRule(c *gin.Context) {
var req dto.RuleSearch
if err := helper.CheckBindAndValidate(&req, c); err != nil {
return
}
total, list, err := firewallService.SearchWithPage(req)
if err != nil {
helper.InternalServer(c, err)
return
}
helper.SuccessWithData(c, dto.PageResult{
Items: list,
Total: total,
})
}
// @Tags Firewall // @Tags Firewall
// @Summary Operate firewall // @Summary Operate firewall
// @Accept json // @Accept json
// @Param request body dto.FirewallOperation true "request" // @Param request body dto.FirewallLifecycleOperation true "request"
// @Success 200 // @Success 200 {object} dto.FirewallLifecycleOperationResponse
// @Security ApiKeyAuth // @Security ApiKeyAuth
// @Security Timestamp // @Security Timestamp
// @Router /hosts/firewall/operate [post] // @Router /hosts/firewall/operate [post]
// @x-panel-log {"bodyKeys":["operation"],"paramKeys":[],"BeforeFunctions":[],"formatZH":"[operation] 防火墙","formatEN":"[operation] firewall"} // @x-panel-log {"bodyKeys":["operation"],"paramKeys":[],"BeforeFunctions":[],"formatZH":"[operation] 防火墙","formatEN":"[operation] firewall"}
func (b *BaseApi) OperateFirewall(c *gin.Context) { func (b *BaseApi) OperateFirewall(c *gin.Context) {
var req dto.FirewallOperation var request dto.FirewallLifecycleOperation
if err := helper.CheckBindAndValidate(&req, c); err != nil { if err := helper.CheckBindAndValidate(&request, c); err != nil {
return return
} }
if err := firewallService.OperateFirewall(req); err != nil { result, err := firewallService.QueueFirewallOperation(request)
helper.InternalServer(c, err)
return
}
helper.Success(c)
}
// @Tags Firewall
// @Summary Create group
// @Accept json
// @Param request body dto.PortRuleOperate true "request"
// @Success 200
// @Security ApiKeyAuth
// @Security Timestamp
// @Router /hosts/firewall/port [post]
// @x-panel-log {"bodyKeys":["port","strategy"],"paramKeys":[],"BeforeFunctions":[],"formatZH":"添加端口规则 [strategy] [port]","formatEN":"create port rules [strategy][port]"}
func (b *BaseApi) OperatePortRule(c *gin.Context) {
var req dto.PortRuleOperate
if err := helper.CheckBindAndValidate(&req, c); err != nil {
return
}
if err := firewallService.OperatePortRule(req, true); err != nil {
helper.InternalServer(c, err)
return
}
helper.Success(c)
}
// OperateForwardRule
// @Tags Firewall
// @Summary Operate forward rule
// @Accept json
// @Param request body dto.ForwardRuleOperate true "request"
// @Success 200
// @Security ApiKeyAuth
// @Security Timestamp
// @Router /hosts/firewall/forward [post]
// @x-panel-log {"bodyKeys":[],"paramKeys":[],"BeforeFunctions":[],"formatZH":"更新端口转发规则","formatEN":"update port forward rules"}
func (b *BaseApi) OperateForwardRule(c *gin.Context) {
var req dto.ForwardRuleOperate
if err := helper.CheckBindAndValidate(&req, c); err != nil {
return
}
if err := firewallService.OperateForwardRule(req); err != nil {
helper.InternalServer(c, err)
return
}
helper.Success(c)
}
// @Tags Firewall
// @Summary Operate Ip rule
// @Accept json
// @Param request body dto.AddrRuleOperate true "request"
// @Success 200
// @Security ApiKeyAuth
// @Security Timestamp
// @Router /hosts/firewall/ip [post]
// @x-panel-log {"bodyKeys":["strategy","address"],"paramKeys":[],"BeforeFunctions":[],"formatZH":"添加 ip 规则 [strategy] [address]","formatEN":"create address rules [strategy][address]"}
func (b *BaseApi) OperateIPRule(c *gin.Context) {
var req dto.AddrRuleOperate
if err := helper.CheckBindAndValidate(&req, c); err != nil {
return
}
if err := firewallService.OperateAddressRule(req, true); err != nil {
helper.InternalServer(c, err)
return
}
helper.Success(c)
}
// @Tags Firewall
// @Summary Batch operate rule
// @Accept json
// @Param request body dto.BatchRuleOperate true "request"
// @Success 200
// @Security ApiKeyAuth
// @Security Timestamp
// @Router /hosts/firewall/batch [post]
func (b *BaseApi) BatchOperateRule(c *gin.Context) {
var req dto.BatchRuleOperate
if err := helper.CheckBindAndValidate(&req, c); err != nil {
return
}
if err := firewallService.BatchOperateRule(req); err != nil {
helper.InternalServer(c, err)
return
}
helper.Success(c)
}
// @Tags Firewall
// @Summary Update rule description
// @Accept json
// @Param request body dto.UpdateFirewallDescription true "request"
// @Success 200
// @Security ApiKeyAuth
// @Security Timestamp
// @Router /hosts/firewall/update/description [post]
func (b *BaseApi) UpdateFirewallDescription(c *gin.Context) {
var req dto.UpdateFirewallDescription
if err := helper.CheckBindAndValidate(&req, c); err != nil {
return
}
if err := firewallService.UpdateDescription(req); err != nil {
helper.InternalServer(c, err)
return
}
helper.Success(c)
}
// @Tags Firewall
// @Summary Update port rule
// @Accept json
// @Param request body dto.PortRuleUpdate true "request"
// @Success 200
// @Security ApiKeyAuth
// @Security Timestamp
// @Router /hosts/firewall/update/port [post]
func (b *BaseApi) UpdatePortRule(c *gin.Context) {
var req dto.PortRuleUpdate
if err := helper.CheckBindAndValidate(&req, c); err != nil {
return
}
if err := firewallService.UpdatePortRule(req); err != nil {
helper.InternalServer(c, err)
return
}
helper.Success(c)
}
// @Tags Firewall
// @Summary Update Ip rule
// @Accept json
// @Param request body dto.AddrRuleUpdate true "request"
// @Success 200
// @Security ApiKeyAuth
// @Security Timestamp
// @Router /hosts/firewall/update/addr [post]
func (b *BaseApi) UpdateAddrRule(c *gin.Context) {
var req dto.AddrRuleUpdate
if err := helper.CheckBindAndValidate(&req, c); err != nil {
return
}
if err := firewallService.UpdateAddrRule(req); err != nil {
helper.InternalServer(c, err)
return
}
helper.Success(c)
}
// @Tags Firewall
// @Summary search iptables filter rules
// @Accept json
// @Param request body dto.SearchPageWithType true "request"
// @Success 200 {object} dto.PageResult
// @Security ApiKeyAuth
// @Security Timestamp
// @Router /hosts/firewall/filter/rule/search [post]
func (b *BaseApi) SearchFilterRules(c *gin.Context) {
var req dto.SearchPageWithType
if err := helper.CheckBindAndValidate(&req, c); err != nil {
return
}
total, list, err := iptablesService.Search(req)
if err != nil { if err != nil {
helper.InternalServer(c, err) helper.InternalServer(c, err)
return return
} }
helper.SuccessWithData(c, dto.PageResult{ helper.SuccessWithData(c, result)
Items: list,
Total: total,
})
} }
// @Tags Firewall // @Tags Firewall
// @Summary Operate iptables filter rule // @Summary Load forwarding base info
// @Accept json // @Accept json
// @Param request body dto.IptablesRuleOp true "request" // @Success 200 {object} dto.FirewallSubsystemStatus
// @Success 200
// @Security ApiKeyAuth // @Security ApiKeyAuth
// @Security Timestamp // @Security Timestamp
// @Router /hosts/firewall/filter/rule/operate [post] // @Router /hosts/firewall/forward/base [post]
// @x-panel-log {"bodyKeys":["operation","chain"],"paramKeys":[],"BeforeFunctions":[],"formatZH":"[operation] filter规则到 [chain]","formatEN":"[operation] filter rule to [chain]"} func (b *BaseApi) LoadForwardingBaseInfo(c *gin.Context) {
func (b *BaseApi) OperateFilterRule(c *gin.Context) { data, err := forwardingService.LoadBaseInfo(c.Request.Context())
var req dto.IptablesRuleOp if err != nil {
if err := helper.CheckBindAndValidate(&req, c); err != nil { helper.InternalServer(c, err)
return return
} }
if err := iptablesService.OperateRule(req, true); err != nil { helper.SuccessWithData(c, data)
}
// @Tags Firewall
// @Summary Page forwarding rules
// @Accept json
// @Param request body dto.ForwardRuleSearch true "request"
// @Success 200 {object} dto.PageResult
// @Security ApiKeyAuth
// @Security Timestamp
// @Router /hosts/firewall/forward/search [post]
func (b *BaseApi) SearchForwardingRules(c *gin.Context) {
var request dto.ForwardRuleSearch
if err := helper.CheckBindAndValidate(&request, c); err != nil {
return
}
total, items, err := forwardingService.SearchRules(c.Request.Context(), request)
if err != nil {
helper.InternalServer(c, err) helper.InternalServer(c, err)
return return
} }
helper.Success(c) helper.SuccessWithData(c, dto.PageResult{Items: items, Total: total})
} }
// @Tags Firewall // @Tags Firewall
// @Summary Batch operate iptables filter rules // @Summary Operate forwarding rules
// @Accept json // @Accept json
// @Param request body dto.IptablesBatchOperate true "request" // @Param request body dto.ForwardRuleOperate true "request"
// @Success 200 // @Success 200 {object} dto.FilterChainOperationResponse
// @Security ApiKeyAuth // @Security ApiKeyAuth
// @Security Timestamp // @Security Timestamp
// @Router /hosts/firewall/filter/rule/batch [post] // @Router /hosts/firewall/forward/operate [post]
func (b *BaseApi) BatchOperateFilterRule(c *gin.Context) { // @x-panel-log {"bodyKeys":[],"paramKeys":[],"BeforeFunctions":[],"formatZH":"更新端口转发规则","formatEN":"update port forward rules"}
var req dto.IptablesBatchOperate func (b *BaseApi) OperateForwardingRules(c *gin.Context) {
if err := helper.CheckBindAndValidate(&req, c); err != nil { var request dto.ForwardRuleOperate
if err := helper.CheckBindAndValidate(&request, c); err != nil {
return return
} }
if err := iptablesService.BatchOperate(req); err != nil { result, err := forwardingService.OperateRules(request)
if err != nil {
helper.InternalServer(c, err) helper.InternalServer(c, err)
return return
} }
helper.SuccessWithData(c, result)
helper.Success(c)
} }
// @Tags Firewall // @Tags Firewall
// @Summary Apply/Unload/Init iptables filter // @Summary Enable forwarding
// @Accept json // @Accept json
// @Param request body dto.IptablesOp true "request" // @Param request body dto.FirewallInitializationTask true "request"
// @Success 200 // @Success 200 {object} dto.FilterChainOperationResponse
// @Security ApiKeyAuth
// @Security Timestamp
// @Router /hosts/firewall/forward/enable [post]
// @x-panel-log {"bodyKeys":[],"paramKeys":[],"BeforeFunctions":[],"formatZH":"初始化并启用端口转发","formatEN":"initialize and enable port forwarding"}
func (b *BaseApi) EnableForwarding(c *gin.Context) {
var request dto.FirewallInitializationTask
if err := helper.CheckBindAndValidate(&request, c); err != nil {
return
}
result, err := forwardingService.QueueInitialization(request)
if err != nil {
helper.InternalServer(c, err)
return
}
helper.SuccessWithData(c, result)
}
// @Tags Firewall
// @Summary Apply/Unload/Init firewall filter chain
// @Accept json
// @Param request body dto.FilterChainOperation true "request"
// @Success 200 {object} dto.FilterChainOperationResponse
// @Security ApiKeyAuth // @Security ApiKeyAuth
// @Security Timestamp // @Security Timestamp
// @Router /hosts/firewall/filter/operate [post] // @Router /hosts/firewall/filter/operate [post]
// @x-panel-log {"bodyKeys":["operate"],"paramKeys":[],"BeforeFunctions":[],"formatZH":"[operate] iptables filter 防火墙","formatEN":"[operate] iptables filter firewall"} // @x-panel-log {"bodyKeys":["operate"],"paramKeys":[],"BeforeFunctions":[],"formatZH":"[operate] 防火墙过滤链","formatEN":"[operate] firewall filter chain"}
func (b *BaseApi) OperateFilterChain(c *gin.Context) { func (b *BaseApi) OperateFilterChain(c *gin.Context) {
var req dto.IptablesOp var request dto.FilterChainOperation
if err := helper.CheckBindAndValidate(&req, c); err != nil { if err := helper.CheckBindAndValidate(&request, c); err != nil {
return return
} }
if err := iptablesService.Operate(req); err != nil { if request.Operate == "init-base" {
result, err := firewallService.QueueFilterChainInitialization(request)
if err != nil {
helper.InternalServer(c, err)
return
}
helper.SuccessWithData(c, result)
return
}
if err := firewallService.OperateFilterChain(request); err != nil {
helper.InternalServer(c, err) helper.InternalServer(c, err)
return return
} }
helper.SuccessWithData(c, dto.FilterChainOperationResponse{})
}
// @Tags Firewall
// @Summary List unified firewall v2 rules
// @Accept json
// @Param request body dto.FirewallRuleInventory true "request"
// @Success 200 {object} dto.FirewallRuleInventoryResponse
// @Failure 400 {object} dto.Response
// @Security ApiKeyAuth
// @Security Timestamp
// @Router /hosts/firewall/rules/search [post]
func (b *BaseApi) SearchFirewallRules(c *gin.Context) {
var request dto.FirewallRuleInventory
if err := helper.CheckBindAndValidate(&request, c); err != nil {
return
}
inventory, err := firewallService.Inventory(c.Request.Context(), request)
if err != nil {
handleFirewallRuleError(c, err)
return
}
helper.SuccessWithData(c, inventory)
}
// @Tags Firewall
// @Summary Reset firewall rules
// @Accept json
// @Param request body dto.FirewallRuleReset true "request"
// @Success 200 {object} dto.FirewallRuleResetResponse
// @Failure 400 {object} dto.Response
// @Security ApiKeyAuth
// @Security Timestamp
// @Router /hosts/firewall/rules/reset [post]
// @x-panel-log {"bodyKeys":[],"paramKeys":[],"BeforeFunctions":[],"formatZH":"重置防火墙规则","formatEN":"reset firewall rules"}
func (b *BaseApi) ResetFirewallRules(c *gin.Context) {
var request dto.FirewallRuleReset
if err := helper.CheckBindAndValidate(&request, c); err != nil {
return
}
result, err := firewallService.Reset(c.Request.Context(), request)
if err != nil {
handleFirewallRuleError(c, err)
return
}
helper.SuccessWithData(c, result)
}
// @Tags Firewall
// @Summary Load one provider-native firewall object definition
// @Accept json
// @Param request body dto.FirewallNativeDetail true "request"
// @Success 200 {string} string
// @Failure 400 {object} dto.Response
// @Security ApiKeyAuth
// @Security Timestamp
// @Router /hosts/firewall/rules/native/detail [post]
func (b *BaseApi) LoadFirewallNativeDetail(c *gin.Context) {
var request dto.FirewallNativeDetail
if err := helper.CheckBindAndValidate(&request, c); err != nil {
return
}
info, err := firewallService.LoadFirewallNativeDetail(c.Request.Context(), request)
if err != nil {
handleFirewallRuleError(c, err)
return
}
helper.SuccessWithData(c, info)
}
// @Tags Firewall
// @Summary Queue firewall rule creation
// @Description Creation and import return a taskID immediately; validation and execution results are written to the task log.
// @Accept json
// @Param request body dto.FirewallRuleCreate true "request"
// @Success 200 {object} dto.FirewallRuleCreateResponse
// @Failure 400 {object} dto.Response
// @Failure 409 {object} dto.Response
// @Security ApiKeyAuth
// @Security Timestamp
// @Router /hosts/firewall/rules [post]
// @x-panel-log {"bodyKeys":[],"paramKeys":[],"BeforeFunctions":[],"formatZH":"添加防火墙规则","formatEN":"create firewall rules"}
func (b *BaseApi) CreateFirewallRules(c *gin.Context) {
var request dto.FirewallRuleCreate
if err := helper.CheckBindAndValidate(&request, c); err != nil {
return
}
result, err := firewallService.Create(c.Request.Context(), request)
if err != nil {
handleFirewallRuleError(c, err)
return
}
helper.SuccessWithData(c, result)
}
// @Tags Firewall
// @Summary Queue firewall rule deletion
// @Description Deletes non-whitelist rules by scope and instance key. Returns a taskID immediately; results are written to the task log.
// @Accept json
// @Param request body dto.FirewallRuleDelete true "request"
// @Success 200 {object} dto.FirewallRuleDeleteResponse
// @Failure 400 {object} dto.Response
// @Security ApiKeyAuth
// @Security Timestamp
// @Router /hosts/firewall/rules/delete [post]
// @x-panel-log {"bodyKeys":[],"paramKeys":[],"BeforeFunctions":[],"formatZH":"删除防火墙规则","formatEN":"delete firewall rules"}
func (b *BaseApi) DeleteFirewallRules(c *gin.Context) {
var request dto.FirewallRuleDelete
if err := helper.CheckBindAndValidate(&request, c); err != nil {
return
}
result, err := firewallService.Delete(c.Request.Context(), request)
if err != nil {
handleFirewallRuleError(c, err)
return
}
helper.SuccessWithData(c, result)
}
// @Tags Firewall
// @Summary Update a firewall rule
// @Accept json
// @Param request body dto.FirewallRuleUpdate true "request"
// @Success 200
// @Failure 400 {object} dto.Response
// @Security ApiKeyAuth
// @Security Timestamp
// @Router /hosts/firewall/rules/update [post]
// @x-panel-log {"bodyKeys":["instanceKey"],"paramKeys":[],"BeforeFunctions":[],"formatZH":"更新防火墙规则 [instanceKey]","formatEN":"update firewall rule [instanceKey]"}
func (b *BaseApi) UpdateFirewallRule(c *gin.Context) {
var request dto.FirewallRuleUpdate
if err := helper.CheckBindAndValidate(&request, c); err != nil {
return
}
if err := firewallService.Update(c.Request.Context(), request); err != nil {
handleFirewallRuleError(c, err)
return
}
helper.Success(c) helper.Success(c)
} }
// @Tags Firewall // @Tags Firewall
// @Summary load chain status with name // @Summary Reorder a firewall rule
// @Accept json // @Accept json
// @Param request body dto.OperationWithName true "request" // @Param request body dto.FirewallRuleReorder true "request"
// @Success 200
// @Failure 400 {object} dto.Response
// @Security ApiKeyAuth
// @Security Timestamp
// @Router /hosts/firewall/rules/reorder [post]
// @x-panel-log {"bodyKeys":["instanceKey"],"paramKeys":[],"BeforeFunctions":[],"formatZH":"调整防火墙规则顺序 [instanceKey]","formatEN":"reorder firewall rule [instanceKey]"}
func (b *BaseApi) ReorderFirewallRule(c *gin.Context) {
var request dto.FirewallRuleReorder
if err := helper.CheckBindAndValidate(&request, c); err != nil {
return
}
if err := firewallService.Reorder(c.Request.Context(), request); err != nil {
handleFirewallRuleError(c, err)
return
}
helper.Success(c)
}
func handleFirewallRuleError(c *gin.Context, err error) {
var businessErr buserr.BusinessError
isBusinessError := errors.As(err, &businessErr)
switch {
case errors.Is(err, filter.ErrProtectedRule):
helper.ErrorWithBusinessCode(c, http.StatusBadRequest, "FW_LOCKOUT_RISK", "ErrInvalidParams", err)
case errors.Is(err, filter.ErrRuleStale):
helper.ErrorWithBusinessCode(c, http.StatusConflict, "FW_RULE_STALE", "ErrInvalidParams", err)
case errors.Is(err, filter.ErrUnsupportedScope), errors.Is(err, filter.ErrInvalidScope),
errors.Is(err, filter.ErrProviderUnavailable), errors.Is(err, filter.ErrAdapterUnavailable):
helper.ErrorWithBusinessCode(c, http.StatusBadRequest, "FW_SCOPE_UNSUPPORTED", "ErrInvalidParams", err)
case errors.Is(err, filter.ErrInvalidRule), errors.Is(err, filter.ErrRuleOperation):
helper.ErrorWithBusinessCode(c, http.StatusBadRequest, "FW_RULE_UNSUPPORTED", "ErrInvalidParams", err)
case isBusinessError && businessErr.Msg == "ErrRecordExist":
c.JSON(http.StatusOK, dto.Response{Code: http.StatusConflict, ErrorCode: "FW_RULE_DUPLICATE", Message: err.Error()})
c.Abort()
case isBusinessError && businessErr.Msg == "ErrFirewallRuleConflict":
c.JSON(http.StatusOK, dto.Response{Code: http.StatusConflict, ErrorCode: "FW_RULE_CONFLICT", Message: err.Error()})
c.Abort()
case isBusinessError && businessErr.Msg == "ErrInvalidParams":
c.JSON(http.StatusOK, dto.Response{Code: http.StatusBadRequest, ErrorCode: "FW_RULE_UNSUPPORTED", Message: err.Error()})
c.Abort()
default:
helper.ErrorWithBusinessCode(c, http.StatusInternalServerError, "FW_APPLY_FAILED", "ErrInternalServer", err)
}
}
// @Tags Firewall
// @Summary Load firewall settings
// @Success 200 {object} dto.FirewallSettings
// @Security ApiKeyAuth
// @Security Timestamp
// @Router /hosts/firewall/settings [get]
func (b *BaseApi) LoadFirewallSettings(c *gin.Context) {
data, err := firewallSettingService.Load(c.Request.Context())
if err != nil {
helper.InternalServer(c, err)
return
}
helper.SuccessWithData(c, data)
}
// @Tags Firewall
// @Summary Create firewall port whitelist rules
// @Description Saves whitelist configuration and applies missing allowances; existing rules are not removed.
// @Accept json
// @Param request body dto.FirewallPortWhitelistCreate true "request"
// @Success 200 // @Success 200
// @Security ApiKeyAuth // @Security ApiKeyAuth
// @Security Timestamp // @Security Timestamp
// @Router /hosts/firewall/filter/chain/status [post] // @Router /hosts/firewall/settings/whitelist [post]
func (b *BaseApi) LoadChainStatus(c *gin.Context) { // @x-panel-log {"bodyKeys":["rule"],"paramKeys":[],"BeforeFunctions":[],"formatZH":"创建防火墙端口白名单","formatEN":"create firewall port whitelist"}
var req dto.OperationWithName func (b *BaseApi) CreateFirewallPortWhitelist(c *gin.Context) {
if err := helper.CheckBindAndValidate(&req, c); err != nil { var request dto.FirewallPortWhitelistCreate
if err := helper.CheckBindAndValidate(&request, c); err != nil {
return return
} }
if err := firewallSettingService.CreatePortWhitelist(c.Request.Context(), request); err != nil {
helper.InternalServer(c, err)
return
}
helper.Success(c)
}
helper.SuccessWithData(c, iptablesService.LoadChainStatus(req)) // @Tags Firewall
// @Summary Update firewall port whitelist rules
// @Description Saves whitelist configuration and applies missing allowances; existing rules are not removed.
// @Accept json
// @Param request body dto.FirewallPortWhitelistUpdate true "request"
// @Success 200
// @Security ApiKeyAuth
// @Security Timestamp
// @Router /hosts/firewall/settings/whitelist/update [post]
// @x-panel-log {"bodyKeys":["oldRule","rule"],"paramKeys":[],"BeforeFunctions":[],"formatZH":"编辑防火墙端口白名单","formatEN":"update firewall port whitelist"}
func (b *BaseApi) UpdateFirewallPortWhitelist(c *gin.Context) {
var request dto.FirewallPortWhitelistUpdate
if err := helper.CheckBindAndValidate(&request, c); err != nil {
return
}
if err := firewallSettingService.UpdatePortWhitelist(c.Request.Context(), request); err != nil {
helper.InternalServer(c, err)
return
}
helper.Success(c)
}
// @Tags Firewall
// @Summary Delete firewall port whitelist rules
// @Description Removes whitelist configuration; existing firewall rules are not removed.
// @Accept json
// @Param request body dto.FirewallPortWhitelistDelete true "request"
// @Success 200
// @Security ApiKeyAuth
// @Security Timestamp
// @Router /hosts/firewall/settings/whitelist/delete [post]
// @x-panel-log {"bodyKeys":["rules"],"paramKeys":[],"BeforeFunctions":[],"formatZH":"删除防火墙端口白名单","formatEN":"delete firewall port whitelist"}
func (b *BaseApi) DeleteFirewallPortWhitelist(c *gin.Context) {
var request dto.FirewallPortWhitelistDelete
if err := helper.CheckBindAndValidate(&request, c); err != nil {
return
}
if err := firewallSettingService.DeletePortWhitelist(c.Request.Context(), request); err != nil {
helper.InternalServer(c, err)
return
}
helper.Success(c)
}
// @Tags Firewall
// @Summary Operate firewall backend
// @Accept json
// @Param request body dto.FirewallBackendOperation true "request"
// @Success 200
// @Security ApiKeyAuth
// @Security Timestamp
// @Router /hosts/firewall/settings/operate [post]
// @x-panel-log {"bodyKeys":["subsystem","backend","operation"],"paramKeys":[],"BeforeFunctions":[],"formatZH":"防火墙子系统 [subsystem] 后端 [operation] [backend]","formatEN":"[operation] firewall [subsystem] backend [backend]"}
func (b *BaseApi) OperateFirewallBackend(c *gin.Context) {
var request dto.FirewallBackendOperation
if err := helper.CheckBindAndValidate(&request, c); err != nil {
return
}
if err := firewallSettingService.Operate(c.Request.Context(), request); err != nil {
var businessErr buserr.BusinessError
if errors.As(err, &businessErr) && businessErr.Msg == "ErrFirewallBackendCleanupRequired" {
c.JSON(http.StatusOK, dto.Response{Code: http.StatusConflict, ErrorCode: "FW_BACKEND_CLEANUP_REQUIRED", Message: err.Error()})
c.Abort()
return
}
helper.InternalServer(c, err)
return
}
helper.Success(c)
}
// @Tags Firewall
// @Summary List Docker port guard status and policies
// @Success 200 {object} dto.DockerPortGuardList
// @Security ApiKeyAuth
// @Security Timestamp
// @Router /hosts/firewall/docker/ports [get]
func (b *BaseApi) ListDockerPortGuard(c *gin.Context) {
data, err := dockerPortGuardService.LoadOverview(c.Request.Context())
if err != nil {
handleDockerPortGuardError(c, err)
return
}
helper.SuccessWithData(c, data)
}
// @Tags Firewall
// @Summary List Docker published ports
// @Success 200 {array} dto.DockerPortGuardContainer
// @Security ApiKeyAuth
// @Security Timestamp
// @Router /hosts/firewall/docker/endpoints [get]
func (b *BaseApi) ListDockerPublishedPorts(c *gin.Context) {
data, err := dockerPortGuardService.LoadPublishedPorts(c.Request.Context())
if err != nil {
handleDockerPortGuardError(c, err)
return
}
helper.SuccessWithData(c, data)
}
// @Tags Firewall
// @Summary Operate Docker port guard
// @Accept json
// @Param request body dto.DockerPortGuardOperation true "request"
// @Success 200 {object} dto.FilterChainOperationResponse
// @Security ApiKeyAuth
// @Security Timestamp
// @Router /hosts/firewall/docker/operate [post]
// @x-panel-log {"bodyKeys":["operation"],"paramKeys":[],"BeforeFunctions":[],"formatZH":"[operation] Docker 端口防护","formatEN":"[operation] Docker port guard"}
func (b *BaseApi) OperateDockerPortGuard(c *gin.Context) {
var request dto.DockerPortGuardOperation
if err := helper.CheckBindAndValidate(&request, c); err != nil {
return
}
if request.Operation == "initialize" {
result, err := dockerPortGuardService.QueueInitialization(request)
if err != nil {
handleDockerPortGuardError(c, err)
return
}
helper.SuccessWithData(c, result)
return
}
if err := dockerPortGuardService.Operate(c.Request.Context(), request); err != nil {
handleDockerPortGuardError(c, err)
return
}
helper.Success(c)
}
// @Tags Firewall
// @Summary Delete Docker port guard policies
// @Accept json
// @Param request body dto.DockerPortGuardPolicyBatchDelete true "request"
// @Success 200 {object} dto.FilterChainOperationResponse
// @Security ApiKeyAuth
// @Security Timestamp
// @Router /hosts/firewall/docker/policies/delete/batch [post]
// @x-panel-log {"bodyKeys":["uuids"],"paramKeys":[],"BeforeFunctions":[],"formatZH":"删除 Docker 端口防护策略 [uuids]","formatEN":"delete Docker port guard policies [uuids]"}
func (b *BaseApi) DeleteDockerPortGuardPolicies(c *gin.Context) {
var request dto.DockerPortGuardPolicyBatchDelete
if err := helper.CheckBindAndValidate(&request, c); err != nil {
return
}
result, err := dockerPortGuardService.DeletePolicies(request)
if err != nil {
handleDockerPortGuardError(c, err)
return
}
helper.SuccessWithData(c, result)
}
// @Tags Firewall
// @Summary Batch upsert Docker port guard policies
// @Accept json
// @Param request body dto.DockerPortGuardPolicyBatch true "request"
// @Success 200 {object} dto.FilterChainOperationResponse
// @Security ApiKeyAuth
// @Security Timestamp
// @Router /hosts/firewall/docker/policies/batch [post]
// @x-panel-log {"bodyKeys":[],"paramKeys":[],"BeforeFunctions":[],"formatZH":"批量更新 Docker 端口防护策略","formatEN":"batch update Docker port guard policies"}
func (b *BaseApi) UpsertDockerPortGuardPolicies(c *gin.Context) {
var request dto.DockerPortGuardPolicyBatch
if err := helper.CheckBindAndValidate(&request, c); err != nil {
return
}
result, err := dockerPortGuardService.UpsertPolicies(request)
if err != nil {
handleDockerPortGuardError(c, err)
return
}
helper.SuccessWithData(c, result)
}
func handleDockerPortGuardError(c *gin.Context, err error) {
var businessErr buserr.BusinessError
if errors.As(err, &businessErr) {
code, errorCode := http.StatusInternalServerError, ""
switch businessErr.Msg {
case "ErrDockerIptablesChainUnavailable":
code, errorCode = http.StatusServiceUnavailable, "FW_DOCKER_IPTABLES_CHAIN_UNAVAILABLE"
case "ErrDockerNftablesChainUnavailable":
code, errorCode = http.StatusServiceUnavailable, "FW_DOCKER_NFTABLES_CHAIN_UNAVAILABLE"
case "ErrInvalidParams":
code, errorCode = http.StatusBadRequest, "FW_DOCKER_GUARD_INVALID"
case "ErrDockerFailed":
code, errorCode = http.StatusServiceUnavailable, "FW_DOCKER_UNAVAILABLE"
}
if errorCode != "" {
c.JSON(http.StatusOK, dto.Response{Code: code, ErrorCode: errorCode, Message: err.Error()})
c.Abort()
return
}
}
if errors.Is(err, docker.ErrUnavailable) {
helper.ErrorWithBusinessCode(c, http.StatusServiceUnavailable, "FW_DOCKER_UNAVAILABLE", "ErrDockerFailed", err)
return
}
helper.ErrorWithBusinessCode(c, http.StatusInternalServerError, "FW_DOCKER_GUARD_FAILED", "ErrInternalServer", err)
}
// @Tags Firewall
// @Summary List firewall rule backups
// @Param subsystem query string false "Firewall subsystem" Enums(system,forwarding,docker) default(system)
// @Success 200 {object} dto.FirewallRuleBackups
// @Security ApiKeyAuth
// @Security Timestamp
// @Router /hosts/firewall/rules/backups [get]
func (b *BaseApi) ListFirewallRuleBackups(c *gin.Context) {
result, err := firewallService.ListRuleBackups(c.Request.Context(), c.DefaultQuery("subsystem", "system"))
if err != nil {
helper.InternalServer(c, err)
return
}
helper.SuccessWithData(c, result)
}
// @Tags Firewall
// @Summary Initialize, repair or bind one firewall address family
// @Accept json
// @Param request body dto.FirewallFamilyOperation true "request"
// @Success 200 {object} dto.FilterChainOperationResponse
// @Security ApiKeyAuth
// @Security Timestamp
// @Router /hosts/firewall/family/operate [post]
// @x-panel-log {"bodyKeys":["subsystem","family","operation"],"paramKeys":[],"BeforeFunctions":[],"formatZH":"[operation] [subsystem] [family] 防火墙链","formatEN":"[operation] [subsystem] [family] firewall chains"}
func (b *BaseApi) OperateFirewallFamily(c *gin.Context) {
var request dto.FirewallFamilyOperation
if err := helper.CheckBindAndValidate(&request, c); err != nil {
return
}
result, err := firewallSettingService.OperateFamily(request)
if err != nil {
helper.InternalServer(c, err)
return
}
helper.SuccessWithData(c, result)
}
// @Tags Firewall
// @Summary Update firewall IPv6 support
// @Accept json
// @Param request body dto.FirewallIPv6Operation true "request"
// @Success 200 {object} dto.FilterChainOperationResponse
// @Security ApiKeyAuth
// @Security Timestamp
// @Router /hosts/firewall/settings/ipv6 [post]
// @x-panel-log {"bodyKeys":["status"],"paramKeys":[],"BeforeFunctions":[],"formatZH":"设置防火墙 IPv6 支持为 [status]","formatEN":"Set firewall IPv6 support to [status]"}
func (b *BaseApi) OperateFirewallIPv6(c *gin.Context) {
var request dto.FirewallIPv6Operation
if err := helper.CheckBindAndValidate(&request, c); err != nil {
return
}
result, err := firewallSettingService.OperateIPv6(request)
if err != nil {
helper.InternalServer(c, err)
return
}
helper.SuccessWithData(c, result)
} }
+8 -16
View File
@@ -3,9 +3,8 @@ package v2
import ( import (
"github.com/1Panel-dev/1Panel/agent/app/api/v2/helper" "github.com/1Panel-dev/1Panel/agent/app/api/v2/helper"
"github.com/1Panel-dev/1Panel/agent/app/dto" "github.com/1Panel-dev/1Panel/agent/app/dto"
"github.com/1Panel-dev/1Panel/agent/utils/ai_tools/gpu" "github.com/1Panel-dev/1Panel/agent/global"
"github.com/1Panel-dev/1Panel/agent/utils/ai_tools/gpu/common" "github.com/1Panel-dev/1Panel/agent/utils/ai_tools/accelerator"
"github.com/1Panel-dev/1Panel/agent/utils/ai_tools/xpu"
"github.com/gin-gonic/gin" "github.com/gin-gonic/gin"
) )
@@ -17,27 +16,20 @@ import (
// @Security Timestamp // @Security Timestamp
// @Router /ai/gpu/load [get] // @Router /ai/gpu/load [get]
func (b *BaseApi) LoadGpuInfo(c *gin.Context) { func (b *BaseApi) LoadGpuInfo(c *gin.Context) {
ok, client := gpu.New() ok, client := accelerator.New()
if ok { if ok {
info, err := client.LoadGpuInfo() snapshot, err := client.Collect(c.Request.Context())
if err != nil { if err != nil {
helper.BadRequest(c, err) helper.BadRequest(c, err)
return return
} }
helper.SuccessWithData(c, info) if warning := snapshot.Warning(); warning != nil {
return global.LOG.Warnf("load realtime accelerator data partially failed, err: %v", warning)
}
xpuOK, xpuClient := xpu.New()
if xpuOK {
info, err := xpuClient.LoadGpuInfo()
if err != nil {
helper.BadRequest(c, err)
return
} }
helper.SuccessWithData(c, info) helper.SuccessWithData(c, &snapshot.Info)
return return
} }
helper.SuccessWithData(c, &common.GpuInfo{}) helper.SuccessWithData(c, &accelerator.Info{})
} }
// @Tags AI // @Tags AI
+20
View File
@@ -30,6 +30,26 @@ func ErrorWithDetail(ctx *gin.Context, code int, msgKey string, err error) {
ctx.Abort() ctx.Abort()
} }
func ErrorWithBusinessCode(ctx *gin.Context, code int, businessCode, msgKey string, err error) {
res := dto.Response{
Code: code,
ErrorCode: businessCode,
Message: i18n.GetMsgWithDetail(msgKey, err.Error()),
}
ctx.JSON(http.StatusOK, res)
ctx.Abort()
}
func ErrorWithDetailAndData(ctx *gin.Context, code int, msgKey string, err error, data interface{}) {
res := dto.Response{
Code: code,
Data: data,
}
res.Message = i18n.GetMsgWithDetail(msgKey, err.Error())
ctx.JSON(http.StatusOK, res)
ctx.Abort()
}
func InternalServer(ctx *gin.Context, err error) { func InternalServer(ctx *gin.Context, err error) {
ErrorWithDetail(ctx, http.StatusInternalServerError, "ErrInternalServer", err) ErrorWithDetail(ctx, http.StatusInternalServerError, "ErrInternalServer", err)
} }
+10 -23
View File
@@ -1,13 +1,9 @@
package v2 package v2
import ( import (
"sort"
"github.com/1Panel-dev/1Panel/agent/app/api/v2/helper" "github.com/1Panel-dev/1Panel/agent/app/api/v2/helper"
"github.com/1Panel-dev/1Panel/agent/app/dto" "github.com/1Panel-dev/1Panel/agent/app/dto"
"github.com/gin-gonic/gin" "github.com/gin-gonic/gin"
"github.com/shirou/gopsutil/v4/disk"
"github.com/shirou/gopsutil/v4/net"
) )
// @Tags Monitor // @Tags Monitor
@@ -32,14 +28,19 @@ func (b *BaseApi) LoadMonitor(c *gin.Context) {
} }
// @Tags Monitor // @Tags Monitor
// @Summary Clean monitor data // @Summary Clean host or GPU monitor data
// @Param request body dto.MonitorClean true "request"
// @Success 200 // @Success 200
// @Security ApiKeyAuth // @Security ApiKeyAuth
// @Security Timestamp // @Security Timestamp
// @Router /hosts/monitor/clean [post] // @Router /hosts/monitor/clean [post]
// @x-panel-log {"bodyKeys":[],"paramKeys":[],"BeforeFunctions":[],"formatZH":"清空监控数据","formatEN":"clean monitor datas"} // @x-panel-log {"bodyKeys":["type"],"paramKeys":[],"BeforeFunctions":[],"formatZH":"清空 [type] 监控数据","formatEN":"clean [type] monitoring data"}
func (b *BaseApi) CleanMonitor(c *gin.Context) { func (b *BaseApi) CleanMonitor(c *gin.Context) {
if err := monitorService.CleanData(); err != nil { var req dto.MonitorClean
if err := helper.CheckBindAndValidate(&req, c); err != nil {
return
}
if err := monitorService.CleanData(req.Type); err != nil {
helper.InternalServer(c, err) helper.InternalServer(c, err)
return return
} }
@@ -91,14 +92,7 @@ func (b *BaseApi) UpdateMonitorSetting(c *gin.Context) {
// @Security Timestamp // @Security Timestamp
// @Router /hosts/monitor/netoptions [get] // @Router /hosts/monitor/netoptions [get]
func (b *BaseApi) GetNetworkOptions(c *gin.Context) { func (b *BaseApi) GetNetworkOptions(c *gin.Context) {
netStat, _ := net.IOCounters(true) helper.SuccessWithData(c, monitorService.LoadNetworkOptions())
var options []string
options = append(options, "all")
for _, net := range netStat {
options = append(options, net.Name)
}
sort.Strings(options)
helper.SuccessWithData(c, options)
} }
// @Tags Monitor // @Tags Monitor
@@ -108,12 +102,5 @@ func (b *BaseApi) GetNetworkOptions(c *gin.Context) {
// @Security Timestamp // @Security Timestamp
// @Router /hosts/monitor/iooptions [get] // @Router /hosts/monitor/iooptions [get]
func (b *BaseApi) GetIOOptions(c *gin.Context) { func (b *BaseApi) GetIOOptions(c *gin.Context) {
diskStat, _ := disk.IOCounters() helper.SuccessWithData(c, monitorService.LoadIOOptions())
var options []string
options = append(options, "all")
for _, net := range diskStat {
options = append(options, net.Name)
}
sort.Strings(options)
helper.SuccessWithData(c, options)
} }
+45
View File
@@ -0,0 +1,45 @@
package v2
import (
"github.com/1Panel-dev/1Panel/agent/app/api/v2/helper"
"github.com/1Panel-dev/1Panel/agent/app/dto"
"github.com/gin-gonic/gin"
)
func (b *BaseApi) LoadVLLMMonitor(c *gin.Context) {
var req dto.MonitorVLLMSearch
if err := helper.CheckBindAndValidate(&req, c); err != nil {
return
}
data, err := monitorService.LoadVLLMMonitorData(req)
if err != nil {
helper.InternalServer(c, err)
return
}
helper.SuccessWithData(c, data)
}
func (b *BaseApi) LoadVLLMCurrent(c *gin.Context) {
var req dto.MonitorVLLMCurrent
if err := helper.CheckBindAndValidate(&req, c); err != nil {
return
}
data, err := monitorService.LoadVLLMCurrent(c.Request.Context(), req)
if err != nil {
helper.InternalServer(c, err)
return
}
helper.SuccessWithData(c, data)
}
func (b *BaseApi) CleanVLLMMonitor(c *gin.Context) {
var req dto.MonitorVLLMClean
if err := helper.CheckBindAndValidate(&req, c); err != nil {
return
}
if err := monitorService.CleanVLLMMonitor(req); err != nil {
helper.InternalServer(c, err)
return
}
helper.Success(c)
}
+1 -1
View File
@@ -169,7 +169,7 @@ func (b *BaseApi) GetNodePackageRunScript(c *gin.Context) {
// @Security ApiKeyAuth // @Security ApiKeyAuth
// @Security Timestamp // @Security Timestamp
// @Router /runtimes/operate [post] // @Router /runtimes/operate [post]
// @x-panel-log {"bodyKeys":["id"],"paramKeys":[],"BeforeFunctions":[],"formatZH":"操作运行环境 [id]","formatEN":"Operate runtime [id]"} // @x-panel-log {"bodyKeys":["ID"],"paramKeys":[],"BeforeFunctions":[{"input_column":"id","input_value":"ID","isList":false,"db":"runtimes","output_column":"name","output_value":"name"}],"formatZH":"操作运行环境 [name]","formatEN":"Operate runtime [name]"}
func (b *BaseApi) OperateRuntime(c *gin.Context) { func (b *BaseApi) OperateRuntime(c *gin.Context) {
var req request.RuntimeOperate var req request.RuntimeOperate
if err := helper.CheckBindAndValidate(&req, c); err != nil { if err := helper.CheckBindAndValidate(&req, c); err != nil {
+2
View File
@@ -83,6 +83,7 @@ func (b *BaseApi) CreateRootCert(c *gin.Context) {
} }
if err := loadCertAfterDecrypt(&req); err != nil { if err := loadCertAfterDecrypt(&req); err != nil {
helper.BadRequest(c, err) helper.BadRequest(c, err)
return
} }
if err := sshService.CreateRootCert(req); err != nil { if err := sshService.CreateRootCert(req); err != nil {
helper.InternalServer(c, err) helper.InternalServer(c, err)
@@ -107,6 +108,7 @@ func (b *BaseApi) EditRootCert(c *gin.Context) {
} }
if err := loadCertAfterDecrypt(&req); err != nil { if err := loadCertAfterDecrypt(&req); err != nil {
helper.BadRequest(c, err) helper.BadRequest(c, err)
return
} }
if err := sshService.EditRootCert(req); err != nil { if err := sshService.EditRootCert(req); err != nil {
helper.InternalServer(c, err) helper.InternalServer(c, err)
+153 -31
View File
@@ -1,11 +1,14 @@
package v2 package v2
import ( import (
"crypto/sha256"
"encoding/base64" "encoding/base64"
"encoding/hex"
"encoding/json" "encoding/json"
"fmt" "fmt"
"net/http" "net/http"
"strconv" "strconv"
"strings"
"time" "time"
"github.com/1Panel-dev/1Panel/agent/app/api/v2/helper" "github.com/1Panel-dev/1Panel/agent/app/api/v2/helper"
@@ -19,29 +22,35 @@ import (
"github.com/gin-gonic/gin" "github.com/gin-gonic/gin"
"github.com/gorilla/websocket" "github.com/gorilla/websocket"
"github.com/pkg/errors" "github.com/pkg/errors"
gossh "golang.org/x/crypto/ssh"
) )
// @Tags Terminal // @Tags Terminal
// @Summary Ws local terminal // @Summary Ws local terminal
// @Param command query string false "command" // @Param command query string false "command"
// @Param session query string false "session id to reattach"
// @Param terminalPersistent query boolean false "allow recovery after an unexpected disconnect"
// @Success 200 // @Success 200
// @Security ApiKeyAuth // @Security ApiKeyAuth
// @Security Timestamp // @Security Timestamp
// @Router /hosts/terminal/local [get] // @Router /hosts/terminal/local [get]
func (b *BaseApi) WsLocalTerminal(c *gin.Context) { func (b *BaseApi) WsLocalTerminal(c *gin.Context) {
b.runSSHSession(c, loadLocalConn, c.DefaultQuery("command", "")) b.runSSHSession(c, "local", loadLocalConn, c.DefaultQuery("command", ""))
} }
// @Tags Terminal // @Tags Terminal
// @Summary Ws host SSH // @Summary Ws host SSH
// @Param id query integer false "id" // @Param id query integer false "id"
// @Param command query string false "command" // @Param command query string false "command"
// @Param session query string false "session id to reattach"
// @Param title query string false "session title shown in the session list"
// @Param terminalPersistent query boolean false "allow recovery after an unexpected disconnect"
// @Success 200 // @Success 200
// @Security ApiKeyAuth // @Security ApiKeyAuth
// @Security Timestamp // @Security Timestamp
// @Router /hosts/terminal/ssh [get] // @Router /hosts/terminal/ssh [get]
func (b *BaseApi) WsHostSSH(c *gin.Context) { func (b *BaseApi) WsHostSSH(c *gin.Context) {
b.runSSHSession(c, func() (*ssh.SSHClient, error) { b.runSSHSession(c, "ssh", func() (*ssh.SSHClient, error) {
hostID, _ := strconv.Atoi(c.DefaultQuery("id", "0")) hostID, _ := strconv.Atoi(c.DefaultQuery("id", "0"))
if hostID <= 0 { if hostID <= 0 {
return nil, errors.New("missing host id") return nil, errors.New("missing host id")
@@ -65,26 +74,33 @@ func (b *BaseApi) WsContainerTerminal(c *gin.Context) {
return return
} }
defer wsConn.Close() defer wsConn.Close()
identity, ok := loadTerminalIdentity(c)
slave, err := loadContainerTerminalCommand(c) if !ok {
if wshandleError(wsConn, err) { _ = wshandleError(wsConn, errors.New("missing terminal identity"))
return
}
defer slave.Close()
tty, err := terminal.NewLocalWsSession(cols, rows, wsConn, slave, false)
if wshandleError(wsConn, err) {
return return
} }
quitChan := make(chan bool, 3) opts := terminal.SessionOptions{
tty.Start(quitChan) Identity: identity,
go slave.Wait(quitChan) Kind: "container",
Target: containerTerminalTarget(c),
Cols: cols,
Rows: rows,
}
if err := terminal.ServeCommand(wsConn, strings.TrimSpace(c.Query("session")), opts, func() (*terminal.LocalCommand, error) {
return loadContainerTerminalCommand(c)
}); err != nil {
_ = wshandleError(wsConn, err)
}
}
<-quitChan func containerTerminalTarget(c *gin.Context) string {
query := c.Request.URL.Query()
global.LOG.Info("websocket finished") for _, key := range []string{"cols", "rows", "session", "terminalRevalidate"} {
closeTerminalConn(wsConn) query.Del(key)
}
sum := sha256.Sum256([]byte(query.Encode()))
return hex.EncodeToString(sum[:])
} }
func prepareTerminalSession(c *gin.Context) (*websocket.Conn, int, int, bool) { func prepareTerminalSession(c *gin.Context) (*websocket.Conn, int, int, bool) {
@@ -115,32 +131,138 @@ func prepareTerminalSession(c *gin.Context) (*websocket.Conn, int, int, bool) {
return wsConn, cols, rows, true return wsConn, cols, rows, true
} }
func (b *BaseApi) runSSHSession(c *gin.Context, connect func() (*ssh.SSHClient, error), command string) { func (b *BaseApi) runSSHSession(c *gin.Context, kind string, connect func() (*ssh.SSHClient, error), command string) {
wsConn, cols, rows, ok := prepareTerminalSession(c) wsConn, cols, rows, ok := prepareTerminalSession(c)
if !ok { if !ok {
return return
} }
defer wsConn.Close() defer wsConn.Close()
identity, ok := loadTerminalIdentity(c)
client, clientErr := connect() if !ok {
if wshandleError(wsConn, errors.WithMessage(clientErr, "failed to set up the connection. Please check the host information")) { _ = wshandleError(wsConn, errors.New("missing terminal identity"))
return return
} }
defer client.Close()
sws, err := terminal.NewLogicSshWsSession(cols, rows, client.Client, wsConn, command) hostID := 0
if wshandleError(wsConn, err) { if kind == "ssh" {
hostID, _ = strconv.Atoi(c.DefaultQuery("id", "0"))
}
opts := terminal.SessionOptions{
Identity: identity,
Kind: kind,
Title: sanitizeTerminalTitle(c.Query("title")),
Persistent: c.Query("terminalPersistent") == "true",
HostID: uint(max(hostID, 0)),
Cols: cols,
Rows: rows,
InitCmd: command,
}
err := terminal.Serve(wsConn, strings.TrimSpace(c.Query("session")), opts, func() (*gossh.Client, error) {
client, err := connect()
if err != nil {
return nil, errors.WithMessage(err, "failed to set up the connection. Please check the host information")
}
return client.Client, nil
})
if err != nil {
_ = wshandleError(wsConn, err)
}
}
// @Tags Terminal
// @Summary List the caller's live terminal sessions
// @Success 200 {array} terminal.Info
// @Security ApiKeyAuth
// @Security Timestamp
// @Router /hosts/terminal/sessions/search [post]
func (b *BaseApi) SearchTerminalSessions(c *gin.Context) {
identity, ok := loadTerminalIdentity(c)
if !ok {
helper.BadRequest(c, errors.New("missing terminal identity"))
return return
} }
defer sws.Close() helper.SuccessWithData(c, terminal.List(identity))
}
quitChan := make(chan bool, 3) // @Tags Terminal
sws.Start(quitChan) // @Summary Close a terminal session
go sws.Wait(quitChan) // @Accept json
// @Param request body dto.TerminalSessionClose true "request"
// @Success 200
// @Security ApiKeyAuth
// @Security Timestamp
// @Router /hosts/terminal/sessions/close [post]
func (b *BaseApi) CloseTerminalSession(c *gin.Context) {
var req dto.TerminalSessionClose
if err := helper.CheckBindAndValidate(&req, c); err != nil {
return
}
identity, ok := loadTerminalIdentity(c)
if !ok {
helper.BadRequest(c, errors.New("missing terminal identity"))
return
}
if err := terminal.CloseSession(req.ID, identity); err != nil {
helper.BadRequest(c, err)
return
}
helper.Success(c)
}
<-quitChan // @Tags Terminal
// @Success 200
// @Security ApiKeyAuth
// @Security Timestamp
// @Router /hosts/terminal/sessions/closeAll [post]
func (b *BaseApi) CloseAllTerminalSessions(c *gin.Context) {
identity, ok := loadTerminalIdentity(c)
if !ok {
helper.BadRequest(c, errors.New("missing terminal identity"))
return
}
terminal.Revoke("auth_session", identity.UserID, identity.AuthSessionID)
helper.Success(c)
}
closeTerminalConn(wsConn) func (b *BaseApi) RevokeTerminalSessions(c *gin.Context) {
var req dto.TerminalSessionRevoke
if err := helper.CheckBindAndValidate(&req, c); err != nil {
return
}
if (req.Scope == "auth_session" && (req.UserID == "" || req.AuthSessionID == "")) ||
(req.Scope == "user" && req.UserID == "") {
helper.BadRequest(c, errors.New("missing terminal revocation identity"))
return
}
terminal.Revoke(req.Scope, req.UserID, req.AuthSessionID)
helper.Success(c)
}
func loadTerminalIdentity(c *gin.Context) (terminal.Identity, bool) {
identity := terminal.Identity{
UserID: strings.TrimSpace(c.GetHeader(terminal.HeaderUserID)),
AuthSessionID: strings.TrimSpace(c.GetHeader(terminal.HeaderAuthSessionID)),
}
if value := c.GetHeader(terminal.HeaderAuthLeaseUntil); value != "" {
millis, err := strconv.ParseInt(value, 10, 64)
if err != nil || millis <= 0 {
return terminal.Identity{}, false
}
identity.AuthLeaseUntil = time.UnixMilli(millis)
if maximum := time.Now().Add(90 * time.Second); identity.AuthLeaseUntil.After(maximum) {
identity.AuthLeaseUntil = maximum
}
}
return identity, identity.Valid()
}
// sanitizeTerminalTitle keeps the title a short single line.
func sanitizeTerminalTitle(title string) string {
title = strings.Join(strings.Fields(title), " ")
if r := []rune(title); len(r) > 64 {
title = string(r[:64])
}
return title
} }
func closeTerminalConn(wsConn *websocket.Conn) { func closeTerminalConn(wsConn *websocket.Conn) {
+10
View File
@@ -0,0 +1,10 @@
package v2
import (
"github.com/1Panel-dev/1Panel/agent/app/api/v2/helper"
"github.com/gin-gonic/gin"
)
func (b *BaseApi) TerminalCapabilities(c *gin.Context) {
helper.SuccessWithData(c, gin.H{"apiKeyLeaseVersion": 1})
}
+259
View File
@@ -0,0 +1,259 @@
package v2
import (
"io"
"github.com/1Panel-dev/1Panel/agent/app/api/v2/helper"
"github.com/1Panel-dev/1Panel/agent/app/dto"
"github.com/1Panel-dev/1Panel/agent/app/dto/request"
"github.com/gin-gonic/gin"
)
// @Tags Website Template
// @Summary Page website templates
// @Accept json
// @Param request body request.WebsiteTemplateSearch true "request"
// @Success 200 {object} dto.PageResult
// @Security ApiKeyAuth
// @Security Timestamp
// @Router /websites/templates/search [post]
func (b *BaseApi) PageWebsiteTemplate(c *gin.Context) {
var req request.WebsiteTemplateSearch
if err := helper.CheckBindAndValidate(&req, c); err != nil {
return
}
total, templates, err := websiteTemplateService.PageTemplate(req)
if err != nil {
helper.InternalServer(c, err)
return
}
helper.SuccessWithData(c, dto.PageResult{
Total: total,
Items: templates,
})
}
// @Tags Website Template
// @Summary Create website template
// @Accept json
// @Param request body request.WebsiteTemplateCreate true "request"
// @Success 200
// @Security ApiKeyAuth
// @Security Timestamp
// @Router /websites/templates [post]
// @x-panel-log {"bodyKeys":["name"],"paramKeys":[],"BeforeFunctions":[],"formatZH":"创建网站模板 [name]","formatEN":"Create website template [name]"}
func (b *BaseApi) CreateWebsiteTemplate(c *gin.Context) {
var req request.WebsiteTemplateCreate
if err := helper.CheckBindAndValidate(&req, c); err != nil {
return
}
if err := websiteTemplateService.CreateTemplate(req); err != nil {
helper.InternalServer(c, err)
return
}
helper.Success(c)
}
// @Tags Website Template
// @Summary Update website template
// @Accept json
// @Param request body request.WebsiteTemplateUpdate true "request"
// @Success 200
// @Security ApiKeyAuth
// @Security Timestamp
// @Router /websites/templates/update [post]
// @x-panel-log {"bodyKeys":["name"],"paramKeys":[],"BeforeFunctions":[],"formatZH":"更新网站模板 [name]","formatEN":"Update website template [name]"}
func (b *BaseApi) UpdateWebsiteTemplate(c *gin.Context) {
var req request.WebsiteTemplateUpdate
if err := helper.CheckBindAndValidate(&req, c); err != nil {
return
}
if err := websiteTemplateService.UpdateTemplate(req); err != nil {
helper.InternalServer(c, err)
return
}
helper.Success(c)
}
// @Tags Website Template
// @Summary Delete website template
// @Accept json
// @Param request body dto.OperateByID true "request"
// @Success 200
// @Security ApiKeyAuth
// @Security Timestamp
// @Router /websites/templates/del [post]
// @x-panel-log {"bodyKeys":["id"],"paramKeys":[],"BeforeFunctions":[{"input_column":"id","input_value":"id","isList":false,"db":"website_templates","output_column":"name","output_value":"name"}],"formatZH":"删除网站模板 [name]","formatEN":"Delete website template [name]"}
func (b *BaseApi) DeleteWebsiteTemplate(c *gin.Context) {
var req dto.OperateByID
if err := helper.CheckBindAndValidate(&req, c); err != nil {
return
}
if err := websiteTemplateService.DeleteTemplate(req.ID); err != nil {
helper.InternalServer(c, err)
return
}
helper.Success(c)
}
// @Tags Website Template
// @Summary Get website template
// @Accept json
// @Param request body dto.OperateByID true "request"
// @Success 200 {object} response.WebsiteTemplateDTO
// @Security ApiKeyAuth
// @Security Timestamp
// @Router /websites/templates/get [post]
func (b *BaseApi) GetWebsiteTemplate(c *gin.Context) {
var req dto.OperateByID
if err := helper.CheckBindAndValidate(&req, c); err != nil {
return
}
template, err := websiteTemplateService.GetTemplate(req.ID)
if err != nil {
helper.InternalServer(c, err)
return
}
helper.SuccessWithData(c, template)
}
// @Tags Website Template
// @Summary Upload website template zip
// @Accept multipart/form-data
// @Param file formData file true "file"
// @Success 200
// @Security ApiKeyAuth
// @Security Timestamp
// @Router /websites/templates/upload [post]
func (b *BaseApi) UploadTemplateZip(c *gin.Context) {
fileHeader, err := c.FormFile("file")
if err != nil {
helper.BadRequest(c, err)
return
}
file, err := fileHeader.Open()
if err != nil {
helper.InternalServer(c, err)
return
}
defer file.Close()
content, err := io.ReadAll(file)
if err != nil {
helper.InternalServer(c, err)
return
}
filePath, variables, err := websiteTemplateService.SaveUploadZip(fileHeader.Filename, content)
if err != nil {
helper.InternalServer(c, err)
return
}
helper.SuccessWithData(c, gin.H{"filePath": filePath, "variables": variables})
}
// @Tags Website Template
// @Summary Preview website template
// @Accept json
// @Param request body request.WebsitePreviewReq true "request"
// @Success 200 {object} response.WebsitePreviewDTO
// @Security ApiKeyAuth
// @Security Timestamp
// @Router /websites/templates/preview [post]
func (b *BaseApi) PreviewWebsiteTemplate(c *gin.Context) {
var req request.WebsitePreviewReq
if err := helper.CheckBindAndValidate(&req, c); err != nil {
return
}
preview, err := websiteTemplateService.Preview(req)
if err != nil {
helper.InternalServer(c, err)
return
}
helper.SuccessWithData(c, preview)
}
// @Tags Website Template
// @Summary Page website template outputs
// @Accept json
// @Param request body request.WebsiteTemplateOutputSearch true "request"
// @Success 200 {object} dto.PageResult
// @Security ApiKeyAuth
// @Security Timestamp
// @Router /websites/templates/outputs/search [post]
func (b *BaseApi) PageWebsiteTemplateOutput(c *gin.Context) {
var req request.WebsiteTemplateOutputSearch
if err := helper.CheckBindAndValidate(&req, c); err != nil {
return
}
total, outputs, err := websiteTemplateService.PageOutput(req)
if err != nil {
helper.InternalServer(c, err)
return
}
helper.SuccessWithData(c, dto.PageResult{
Total: total,
Items: outputs,
})
}
// @Tags Website Template
// @Summary Create website template output
// @Accept json
// @Param request body request.WebsiteTemplateOutputCreate true "request"
// @Success 200
// @Security ApiKeyAuth
// @Security Timestamp
// @Router /websites/templates/outputs [post]
// @x-panel-log {"bodyKeys":["name"],"paramKeys":[],"BeforeFunctions":[],"formatZH":"生成模板产物 [name]","formatEN":"Generate template output [name]"}
func (b *BaseApi) CreateWebsiteTemplateOutput(c *gin.Context) {
var req request.WebsiteTemplateOutputCreate
if err := helper.CheckBindAndValidate(&req, c); err != nil {
return
}
if err := websiteTemplateService.CreateOutput(req); err != nil {
helper.InternalServer(c, err)
return
}
helper.Success(c)
}
// @Tags Website Template
// @Summary Delete website template output
// @Accept json
// @Param request body dto.OperateByID true "request"
// @Success 200
// @Security ApiKeyAuth
// @Security Timestamp
// @Router /websites/templates/outputs/del [post]
// @x-panel-log {"bodyKeys":["id"],"paramKeys":[],"BeforeFunctions":[{"input_column":"id","input_value":"id","isList":false,"db":"website_template_outputs","output_column":"name","output_value":"name"}],"formatZH":"删除模板产物 [name]","formatEN":"Delete template output [name]"}
func (b *BaseApi) DeleteWebsiteTemplateOutput(c *gin.Context) {
var req dto.OperateByID
if err := helper.CheckBindAndValidate(&req, c); err != nil {
return
}
if err := websiteTemplateService.DeleteOutput(req.ID); err != nil {
helper.InternalServer(c, err)
return
}
helper.Success(c)
}
// @Tags Website Template
// @Summary Get website template output
// @Accept json
// @Param request body dto.OperateByID true "request"
// @Success 200 {object} response.WebsiteTemplateOutputDTO
// @Security ApiKeyAuth
// @Security Timestamp
// @Router /websites/templates/outputs/get [post]
func (b *BaseApi) GetWebsiteTemplateOutput(c *gin.Context) {
var req dto.OperateByID
if err := helper.CheckBindAndValidate(&req, c); err != nil {
return
}
output, err := websiteTemplateService.GetOutput(req.ID)
if err != nil {
helper.InternalServer(c, err)
return
}
helper.SuccessWithData(c, output)
}
+95 -34
View File
@@ -162,16 +162,25 @@ type AgentWebsiteBindReq struct {
} }
type AgentModelConfigUpdateReq struct { type AgentModelConfigUpdateReq struct {
AgentID uint `json:"agentId" validate:"required"` AgentID uint `json:"agentId" validate:"required"`
AccountID uint `json:"accountId" validate:"required"` AccountID uint `json:"accountId" validate:"required"`
Model string `json:"model" validate:"required"` Model string `json:"model" validate:"required"`
Fallbacks []string `json:"fallbacks"` Fallbacks []string `json:"fallbacks"`
Metadata []AgentModelMetadata `json:"metadata" validate:"dive"`
} }
type AgentModelConfig struct { type AgentModelConfig struct {
AccountID uint `json:"accountId"` AccountID uint `json:"accountId"`
Model string `json:"model"` Model string `json:"model"`
Fallbacks []string `json:"fallbacks"` Fallbacks []string `json:"fallbacks"`
Metadata []AgentModelMetadata `json:"metadata"`
}
type AgentModelMetadata struct {
Model string `json:"model" validate:"required"`
InputMode string `json:"inputMode" validate:"required,oneof=auto text image"`
ContextWindow int `json:"contextWindow" validate:"min=0"`
MaxTokens int `json:"maxTokens" validate:"min=0"`
} }
type AgentHermesChatSessionItem struct { type AgentHermesChatSessionItem struct {
@@ -322,29 +331,31 @@ type AgentAccountModelDeleteReq struct {
} }
type AgentAccountCreateReq struct { type AgentAccountCreateReq struct {
Provider string `json:"provider" validate:"required"` Provider string `json:"provider" validate:"required"`
Name string `json:"name" validate:"required"` Name string `json:"name" validate:"required"`
APIKey string `json:"apiKey" validate:"required"` APIKey string `json:"apiKey" validate:"required"`
RememberAPIKey bool `json:"rememberApiKey"` RememberAPIKey bool `json:"rememberApiKey"`
BaseURL string `json:"baseURL"` BaseURL string `json:"baseURL"`
Models []AgentAccountModel `json:"models"` Models []AgentAccountModel `json:"models"`
APIType string `json:"apiType" validate:"required"` APIType string `json:"apiType" validate:"required"`
AuthMode string `json:"authMode"` AuthMode string `json:"authMode"`
VerifyModel string `json:"verifyModel"` VerifyModel string `json:"verifyModel"`
Remark string `json:"remark"` ValidateAvailability *bool `json:"validateAvailability"`
Remark string `json:"remark"`
} }
type AgentAccountUpdateReq struct { type AgentAccountUpdateReq struct {
ID uint `json:"id" validate:"required"` ID uint `json:"id" validate:"required"`
Name string `json:"name" validate:"required"` Name string `json:"name" validate:"required"`
APIKey string `json:"apiKey" validate:"required"` APIKey string `json:"apiKey" validate:"required"`
RememberAPIKey bool `json:"rememberApiKey"` RememberAPIKey bool `json:"rememberApiKey"`
BaseURL string `json:"baseURL"` BaseURL string `json:"baseURL"`
APIType string `json:"apiType" validate:"required"` APIType string `json:"apiType" validate:"required"`
AuthMode string `json:"authMode"` AuthMode string `json:"authMode"`
VerifyModel string `json:"verifyModel"` VerifyModel string `json:"verifyModel"`
Remark string `json:"remark"` ValidateAvailability *bool `json:"validateAvailability"`
SyncAgents bool `json:"syncAgents"` Remark string `json:"remark"`
SyncAgents bool `json:"syncAgents"`
} }
type AgentAccountVerifyReq struct { type AgentAccountVerifyReq struct {
@@ -363,6 +374,8 @@ type AgentAccountDeleteReq struct {
type AgentAccountSearch struct { type AgentAccountSearch struct {
PageInfo PageInfo
Provider string `json:"provider"` Provider string `json:"provider"`
APIType string `json:"apiType"`
TextOnly bool `json:"textOnly"`
Name string `json:"name"` Name string `json:"name"`
} }
@@ -394,11 +407,13 @@ type ProviderModelInfo struct {
} }
type ProviderAPIInfo struct { type ProviderAPIInfo struct {
APIType string `json:"apiType"` APIType string `json:"apiType"`
BaseURL string `json:"baseUrl"` BaseURL string `json:"baseUrl"`
EditableBaseURL bool `json:"editableBaseUrl"` EditableBaseURL bool `json:"editableBaseUrl"`
DefaultAuthMode string `json:"defaultAuthMode"` SupportsModelDiscovery bool `json:"supportsModelDiscovery"`
AuthModes []string `json:"authModes"` DefaultAuthMode string `json:"defaultAuthMode"`
AuthModes []string `json:"authModes"`
Models []ProviderModelInfo `json:"models"`
} }
type ProviderInfo struct { type ProviderInfo struct {
@@ -598,6 +613,52 @@ type AgentPluginStatus struct {
Upgradable bool `json:"upgradable"` Upgradable bool `json:"upgradable"`
} }
type AgentPluginsReq struct {
AgentID uint `json:"agentId" validate:"required"`
}
type AgentPluginSearchReq struct {
AgentID uint `json:"agentId" validate:"required"`
Keyword string `json:"keyword" validate:"required,max=100"`
Limit int `json:"limit" validate:"omitempty,min=1,max=100"`
}
type AgentPluginMarketInstallReq struct {
AgentID uint `json:"agentId" validate:"required"`
Package string `json:"package" validate:"required,max=200"`
Version string `json:"version" validate:"required,max=100"`
TaskID string `json:"taskID" validate:"required"`
}
type AgentPluginOperateReq struct {
AgentID uint `json:"agentId" validate:"required"`
PluginID string `json:"pluginId" validate:"required,max=200"`
Operate string `json:"operate" validate:"required,oneof=enable disable update uninstall"`
TaskID string `json:"taskID" validate:"required"`
}
type AgentPluginItem struct {
ID string `json:"id"`
Name string `json:"name"`
Version string `json:"version"`
Origin string `json:"origin"`
Enabled bool `json:"enabled"`
}
type AgentPluginSearchItem struct {
Package string `json:"package"`
PluginID string `json:"pluginId"`
Name string `json:"name"`
Description string `json:"description"`
Version string `json:"version"`
Channel string `json:"channel"`
VerificationTier string `json:"verificationTier"`
Categories []string `json:"categories"`
Official bool `json:"official"`
Downloads int64 `json:"downloads"`
Score float64 `json:"score"`
}
type AgentDiscordConfigUpdateReq struct { type AgentDiscordConfigUpdateReq struct {
AgentID uint `json:"agentId" validate:"required"` AgentID uint `json:"agentId" validate:"required"`
Enabled bool `json:"enabled"` Enabled bool `json:"enabled"`
@@ -678,9 +739,9 @@ type AgentSecurityConfig struct {
type AgentOtherConfigUpdateReq struct { type AgentOtherConfigUpdateReq struct {
AgentID uint `json:"agentId" validate:"required"` AgentID uint `json:"agentId" validate:"required"`
UserTimezone string `json:"userTimezone" validate:"required"` UserTimezone string `json:"userTimezone"`
BrowserEnabled bool `json:"browserEnabled"` BrowserEnabled bool `json:"browserEnabled"`
NPMRegistry string `json:"npmRegistry" validate:"required"` NPMRegistry string `json:"npmRegistry"`
DashboardUsername string `json:"dashboardUsername"` DashboardUsername string `json:"dashboardUsername"`
DashboardPassword string `json:"dashboardPassword"` DashboardPassword string `json:"dashboardPassword"`
} }
+35 -18
View File
@@ -21,6 +21,7 @@ type AlertBase struct {
} }
type PushAlert struct { type PushAlert struct {
Result string `json:"result,omitempty"`
TaskName string `json:"taskName"` TaskName string `json:"taskName"`
AlertType string `json:"alertType"` AlertType string `json:"alertType"`
EntryID uint `json:"entryID"` EntryID uint `json:"entryID"`
@@ -53,6 +54,7 @@ type AlertDTO struct {
Method string `json:"method"` Method string `json:"method"`
Title string `json:"title"` Title string `json:"title"`
Project string `json:"project"` Project string `json:"project"`
TaskName string `json:"taskName,omitempty"`
Status string `json:"status"` Status string `json:"status"`
SendCount uint `json:"sendCount"` SendCount uint `json:"sendCount"`
AdvancedParams string `json:"advancedParams"` AdvancedParams string `json:"advancedParams"`
@@ -113,8 +115,10 @@ type DiskDTO struct {
type AlertLogSearch struct { type AlertLogSearch struct {
PageInfo PageInfo
Count uint `json:"count"` Count uint `json:"count"`
Status string `json:"status"` Status string `json:"status"`
StartTime time.Time `json:"startTime"`
EndTime time.Time `json:"endTime"`
} }
type AlertLogDTO struct { type AlertLogDTO struct {
@@ -149,16 +153,25 @@ type AlertLog struct {
} }
type AlertDetail struct { type AlertDetail struct {
LicenseId string `json:"licenseId"` LicenseId string `json:"licenseId"`
Type string `json:"type"` Type string `json:"type"`
SubType string `json:"subType"` SubType string `json:"subType"`
Title string `json:"title"` Title string `json:"title"`
Method string `json:"method"` Method string `json:"method"`
LicenseCode string `json:"licenseCode"` LicenseCode string `json:"licenseCode"`
DeviceId string `json:"deviceId"` DeviceId string `json:"deviceId"`
Project string `json:"project"` Project string `json:"project"`
Params []Param `json:"params"` Params []Param `json:"params"`
Phone string `json:"phone"` Phone string `json:"phone"`
Task *AlertTaskMetadata `json:"task,omitempty"`
}
type AlertTaskMetadata struct {
AlertID uint `json:"alertId"`
Type string `json:"type"`
Quota string `json:"quota"`
QuotaType string `json:"quotaType"`
Method string `json:"method"`
} }
type AlertRule struct { type AlertRule struct {
@@ -293,15 +306,19 @@ type OfflineQueryRequest struct {
} }
type AlertConfigUpdate struct { type AlertConfigUpdate struct {
ID uint `json:"id"` ID uint `json:"id"`
Type string `json:"type"` Type string `json:"type"`
Title string `json:"title"` Title string `json:"title"`
Status string `json:"status"` Status string `json:"status"`
Config string `json:"config"` Config string `json:"config"`
DisplayName string `json:"displayName"` DisplayName string `json:"displayName"`
Revision *time.Time `json:"revision"`
} }
type AlertConfigTest struct { type AlertConfigTest struct {
ID uint `json:"id"`
Type string `json:"type"`
Config string `json:"config"`
Host string `json:"host"` Host string `json:"host"`
Port int `json:"port"` Port int `json:"port"`
Sender string `json:"sender"` Sender string `json:"sender"`
+80
View File
@@ -0,0 +1,80 @@
package dto
const AlertCustomWebhookSchemaVersion = 1
type AlertConfigStatusUpdate struct {
ID uint `json:"id" validate:"required"`
Status string `json:"status" validate:"required,oneof=Enable Disable"`
}
type AlertCustomWebhookSecretMutation struct {
Action string `json:"action,omitempty"`
Value string `json:"value,omitempty"`
}
type AlertCustomWebhookURL struct {
AlertCustomWebhookSecretMutation
Configured bool `json:"configured"`
Masked string `json:"masked,omitempty"`
}
type AlertCustomWebhookBody struct {
Type string `json:"type"`
Template string `json:"template,omitempty"`
Fields []AlertCustomWebhookFormField `json:"fields,omitempty"`
}
type AlertCustomWebhookFormField struct {
Key string `json:"key"`
Value string `json:"value"`
}
type AlertCustomWebhookHeader struct {
UID string `json:"uid"`
Key string `json:"key"`
Secret bool `json:"secret"`
Action string `json:"action,omitempty"`
Value string `json:"value,omitempty"`
Configured bool `json:"configured,omitempty"`
Masked string `json:"masked,omitempty"`
}
type AlertCustomWebhookConfig struct {
SchemaVersion int `json:"schemaVersion"`
State string `json:"state,omitempty"`
DisplayName string `json:"displayName"`
Preset string `json:"preset"`
Method string `json:"method"`
URL AlertCustomWebhookURL `json:"url"`
Body AlertCustomWebhookBody `json:"body"`
Headers []AlertCustomWebhookHeader `json:"headers"`
}
type AlertCustomWebhookSecretConfig struct {
SchemaVersion int `json:"schemaVersion"`
URL string `json:"url"`
Headers map[string]string `json:"headers,omitempty"`
}
type AlertCustomWebhookResolvedConfig struct {
SchemaVersion int
DisplayName string
Preset string
Method string
URL string
Body AlertCustomWebhookBody
Headers []AlertCustomWebhookResolvedHeader
}
type AlertCustomWebhookResolvedHeader struct {
Key string
Value string
}
type AlertConfigTestResult struct {
Success bool `json:"success"`
StatusCode int `json:"statusCode,omitempty"`
Duration int64 `json:"duration,omitempty"` // milliseconds
Message string `json:"message,omitempty"`
Response string `json:"response,omitempty"`
}
+2 -1
View File
@@ -2,7 +2,8 @@ package dto
type SearchWithPage struct { type SearchWithPage struct {
PageInfo PageInfo
Info string `json:"info"` Info string `json:"info"`
ExcludeAppStore bool `json:"excludeAppStore"`
} }
type SearchPageWithType struct { type SearchPageWithType struct {
+4 -3
View File
@@ -6,9 +6,10 @@ type PageResult struct {
} }
type Response struct { type Response struct {
Code int `json:"code"` Code int `json:"code"`
Message string `json:"message"` ErrorCode string `json:"errorCode,omitempty"`
Data interface{} `json:"data"` Message string `json:"message"`
Data interface{} `json:"data"`
} }
type Options struct { type Options struct {
+6
View File
@@ -300,6 +300,7 @@ type ComposeInfo struct {
ConfigFile string `json:"configFile"` ConfigFile string `json:"configFile"`
Workdir string `json:"workdir"` Workdir string `json:"workdir"`
ComposeFileExists bool `json:"composeFileExists"` ComposeFileExists bool `json:"composeFileExists"`
IsPinned bool `json:"isPinned"`
Path string `json:"path"` Path string `json:"path"`
Containers []ComposeContainer `json:"containers"` Containers []ComposeContainer `json:"containers"`
Env string `json:"env"` Env string `json:"env"`
@@ -314,6 +315,7 @@ type ComposeContainer struct {
type ComposeCreate struct { type ComposeCreate struct {
TaskID string `json:"taskID"` TaskID string `json:"taskID"`
Name string `json:"name"` Name string `json:"name"`
DirName string `json:"dirName"`
From string `json:"from" validate:"required,oneof=edit path template"` From string `json:"from" validate:"required,oneof=edit path template"`
File string `json:"file"` File string `json:"file"`
Path string `json:"path"` Path string `json:"path"`
@@ -337,6 +339,10 @@ type ComposeUpdate struct {
Env string `json:"env"` Env string `json:"env"`
ForcePull bool `json:"forcePull"` ForcePull bool `json:"forcePull"`
} }
type ComposePin struct {
Name string `json:"name" validate:"required"`
IsPinned bool `json:"isPinned"`
}
type ComposeLogClean struct { type ComposeLogClean struct {
Name string `json:"name" validate:"required"` Name string `json:"name" validate:"required"`
Path string `json:"path" validate:"required"` Path string `json:"path" validate:"required"`
+11 -7
View File
@@ -51,9 +51,10 @@ type CronjobOperate struct {
Secret string `json:"secret"` Secret string `json:"secret"`
Args string `json:"args"` Args string `json:"args"`
AlertCount uint `json:"alertCount"` AlertCount uint `json:"alertCount"`
AlertTitle string `json:"alertTitle"` AlertTitle string `json:"alertTitle"`
AlertMethod string `json:"alertMethod"` AlertMethod string `json:"alertMethod"`
AlertTriggerMode string `json:"alertTriggerMode" validate:"omitempty,oneof=failed success both"`
CleanLogConfig CleanLogConfig
} }
@@ -126,7 +127,8 @@ type CronjobInfo struct {
Secret string `json:"secret"` Secret string `json:"secret"`
Args string `json:"args"` Args string `json:"args"`
AlertCount uint `json:"alertCount"` AlertCount uint `json:"alertCount"`
AlertTriggerMode string `json:"alertTriggerMode"`
} }
type CronjobImport struct { type CronjobImport struct {
@@ -169,9 +171,10 @@ type CronjobTrans struct {
SourceAccounts []string `json:"sourceAccounts"` SourceAccounts []string `json:"sourceAccounts"`
DownloadAccount string `json:"downloadAccount"` DownloadAccount string `json:"downloadAccount"`
AlertCount uint `json:"alertCount"` AlertCount uint `json:"alertCount"`
AlertTitle string `json:"alertTitle"` AlertTitle string `json:"alertTitle"`
AlertMethod string `json:"alertMethod"` AlertMethod string `json:"alertMethod"`
AlertTriggerMode string `json:"alertTriggerMode" validate:"omitempty,oneof=failed success both"`
} }
type TransHelper struct { type TransHelper struct {
Name string `json:"name"` Name string `json:"name"`
@@ -197,6 +200,7 @@ type SearchRecord struct {
type Record struct { type Record struct {
ID uint `json:"id"` ID uint `json:"id"`
CronjobID uint `json:"cronjobID"`
TaskID string `json:"taskID"` TaskID string `json:"taskID"`
StartTime string `json:"startTime"` StartTime string `json:"startTime"`
Records string `json:"records"` Records string `json:"records"`
+35 -7
View File
@@ -121,6 +121,7 @@ type DashboardCurrent struct {
NetBytesRecv uint64 `json:"netBytesRecv"` NetBytesRecv uint64 `json:"netBytesRecv"`
GPUData []GPUInfo `json:"gpuData"` GPUData []GPUInfo `json:"gpuData"`
NPUData []NPUInfo `json:"npuData"`
XPUData []XPUInfo `json:"xpuData"` XPUData []XPUInfo `json:"xpuData"`
TopCPUItems []Process `json:"topCPUItems"` TopCPUItems []Process `json:"topCPUItems"`
@@ -156,8 +157,12 @@ type DiskInfo struct {
} }
type GPUInfo struct { type GPUInfo struct {
Type string `json:"type"`
Index uint `json:"index"` Index uint `json:"index"`
NPUIndex uint `json:"npuIndex"`
ChipIndex uint `json:"chipIndex"`
ProductName string `json:"productName"` ProductName string `json:"productName"`
BusID string `json:"busID"`
GPUUtil string `json:"gpuUtil"` GPUUtil string `json:"gpuUtil"`
Temperature string `json:"temperature"` Temperature string `json:"temperature"`
PerformanceState string `json:"performanceState"` PerformanceState string `json:"performanceState"`
@@ -170,6 +175,27 @@ type GPUInfo struct {
FanSpeed string `json:"fanSpeed"` FanSpeed string `json:"fanSpeed"`
} }
type NPUInfo struct {
Type string `json:"type"`
Index uint `json:"index"`
NPUIndex uint `json:"npuIndex"`
ChipIndex uint `json:"chipIndex"`
ProductName string `json:"productName"`
BusID string `json:"busID"`
Health string `json:"health"`
Temperature string `json:"temperature"`
PowerDraw string `json:"powerDraw"`
AICore string `json:"aiCore"`
MemUsed string `json:"memUsed"`
MemTotal string `json:"memTotal"`
MemoryUsed string `json:"memoryUsed"`
MemoryTotal string `json:"memoryTotal"`
HBMUsed string `json:"hbmUsed"`
HBMTotal string `json:"hbmTotal"`
HugepagesUsed string `json:"hugepagesUsed"`
HugepagesTotal string `json:"hugepagesTotal"`
}
type AppLauncher struct { type AppLauncher struct {
Key string `json:"key"` Key string `json:"key"`
Type string `json:"type"` Type string `json:"type"`
@@ -202,11 +228,13 @@ type LauncherOption struct {
} }
type XPUInfo struct { type XPUInfo struct {
DeviceID int `json:"deviceID"` DeviceID int `json:"deviceID"`
DeviceName string `json:"deviceName"` DeviceName string `json:"deviceName"`
Memory string `json:"memory"` PciBdfAddress string `json:"pciBdfAddress"`
Temperature string `json:"temperature"` Memory string `json:"memory"`
MemoryUsed string `json:"memoryUsed"` Temperature string `json:"temperature"`
Power string `json:"power"` GPUUtil string `json:"gpuUtil"`
MemoryUtil string `json:"memoryUtil"` MemoryUsed string `json:"memoryUsed"`
Power string `json:"power"`
MemoryUtil string `json:"memoryUtil"`
} }
+4
View File
@@ -22,6 +22,10 @@ type DBBaseInfo struct {
Port int64 `json:"port"` Port int64 `json:"port"`
} }
type RedisCliInstall struct {
TaskID string `json:"taskID" validate:"omitempty,uuid"`
}
// mysql // mysql
type MysqlDBSearch struct { type MysqlDBSearch struct {
PageInfo PageInfo
+324 -84
View File
@@ -1,113 +1,353 @@
package dto package dto
type FirewallBaseInfo struct { import (
Name string `json:"name"` "github.com/1Panel-dev/1Panel/agent/utils/firewall"
IsExist bool `json:"isExist"` dockerfirewall "github.com/1Panel-dev/1Panel/agent/utils/firewall/docker_guard"
IsActive bool `json:"isActive"` "github.com/1Panel-dev/1Panel/agent/utils/firewall/filter"
IsInit bool `json:"isInit"` "github.com/1Panel-dev/1Panel/agent/utils/firewall/forwarding"
IsBind bool `json:"isBind"` )
Version string `json:"version"`
PingStatus string `json:"pingStatus"` type FirewallSubsystemStatus struct {
IPv6Enabled bool `json:"ipv6Enabled"`
Name string `json:"name"`
Backend string `json:"backend"`
ConflictBackend string `json:"conflictBackend,omitempty"`
IsExist bool `json:"isExist"`
IsActive bool `json:"isActive"`
IsInit bool `json:"isInit"`
IsBind bool `json:"isBind"`
Version string `json:"version"`
PingStatus string `json:"pingStatus"`
Message string `json:"message,omitempty"`
Reason string `json:"reason,omitempty"`
LifecycleTaskID string `json:"lifecycleTaskID,omitempty"`
IPv4 FirewallBackendFamilyStatus `json:"ipv4"`
IPv6 FirewallBackendFamilyStatus `json:"ipv6"`
} }
type RuleSearch struct { type FirewallLifecycleOperation struct {
PageInfo
Info string `json:"info"`
Status string `json:"status"`
Strategy string `json:"strategy"`
Type string `json:"type" validate:"required"`
}
type FirewallOperation struct {
Operation string `json:"operation" validate:"required,oneof=start stop restart disableBanPing enableBanPing"` Operation string `json:"operation" validate:"required,oneof=start stop restart disableBanPing enableBanPing"`
WithDockerRestart bool `json:"withDockerRestart"` WithDockerRestart bool `json:"withDockerRestart"`
} }
type PortRuleOperate struct { type FirewallLifecycleOperationResponse struct {
ID uint `json:"id"` TaskID string `json:"taskID,omitempty"`
Operation string `json:"operation" validate:"required,oneof=add remove"` Queued bool `json:"queued"`
Chain string `json:"chain"`
Address string `json:"address"`
Port string `json:"port" validate:"required"`
Protocol string `json:"protocol" validate:"required,oneof=tcp udp tcp/udp"`
Strategy string `json:"strategy" validate:"required,oneof=accept drop"`
Description string `json:"description"`
} }
type ForwardRuleOperate struct { type FirewallBackendOption struct {
ForceDelete bool `json:"forceDelete"` Name string `json:"name"`
Rules []struct { Installed bool `json:"installed"`
Operation string `json:"operation" validate:"required,oneof=add remove"` Active bool `json:"active"`
Num string `json:"num"` Initialized bool `json:"initialized"`
Protocol string `json:"protocol" validate:"required,oneof=tcp udp tcp/udp"` Bound bool `json:"bound"`
Interface string `json:"interface"` Supported bool `json:"supported"`
Port string `json:"port" validate:"required"` SupportReason string `json:"supportReason,omitempty"`
TargetIP string `json:"targetIP"` Implementation string `json:"implementation,omitempty"`
TargetPort string `json:"targetPort" validate:"required"` Message string `json:"message,omitempty"`
} `json:"rules"` IPv4 FirewallBackendFamilyStatus `json:"ipv4"`
IPv6 FirewallBackendFamilyStatus `json:"ipv6"`
} }
type UpdateFirewallDescription struct { type FirewallBackendFamilyStatus struct {
Type string `json:"type"` Partial bool `json:"partial"`
Chain string `json:"chain"` Available bool `json:"available"`
SrcIP string `json:"srcIP"` Initialized bool `json:"initialized"`
DstIP string `json:"dstIP"` Bound bool `json:"bound"`
SrcPort string `json:"srcPort"` Reason string `json:"reason,omitempty"`
DstPort string `json:"dstPort"` RAInterfaces []string `json:"raInterfaces,omitempty"`
Protocol string `json:"protocol"`
Strategy string `json:"strategy" validate:"required,oneof=accept drop"`
Description string `json:"description"`
} }
type AddrRuleOperate struct { type FirewallBackendGroup struct {
ID uint `json:"id"` Selected string `json:"selected"`
Operation string `json:"operation" validate:"required,oneof=add remove"` Current string `json:"current,omitempty"`
Address string `json:"address" validate:"required"` Options []FirewallBackendOption `json:"options"`
Strategy string `json:"strategy" validate:"required,oneof=accept drop"`
Description string `json:"description"`
} }
type PortRuleUpdate struct { type FirewallSettings struct {
OldRule PortRuleOperate `json:"oldRule"` IPv6Enabled bool `json:"ipv6Enabled"`
NewRule PortRuleOperate `json:"newRule"` System FirewallBackendGroup `json:"system"`
Forwarding FirewallBackendGroup `json:"forwarding"`
Docker FirewallBackendGroup `json:"docker"`
PingStatus string `json:"pingStatus"`
PortWhitelist []filter.PortWhitelist `json:"portWhiteList"`
PanelPort string `json:"panelPort"`
SSHPort string `json:"sshPort"`
} }
type AddrRuleUpdate struct { type FirewallPortWhitelistCreate struct {
OldRule AddrRuleOperate `json:"oldRule"` Rule filter.PortWhitelist `json:"rule" validate:"required"`
NewRule AddrRuleOperate `json:"newRule"`
} }
type BatchRuleOperate struct { type FirewallPortWhitelistUpdate struct {
Type string `json:"type" validate:"required"` OldRule filter.PortWhitelist `json:"oldRule" validate:"required"`
Rules []PortRuleOperate `json:"rules"` Rule filter.PortWhitelist `json:"rule" validate:"required"`
} }
type IptablesOp struct { type FirewallPortWhitelistDelete struct {
Name string `json:"name" validate:"required,oneof=1PANEL_INPUT 1PANEL_OUTPUT 1PANEL_BASIC"` Rule *filter.PortWhitelist `json:"rule" validate:"required"`
Operate string `json:"operate" validate:"required,oneof=init-base init-forward init-advance bind-base unbind-base bind unbind"`
} }
type IptablesRuleOp struct { type FirewallBackendOperation struct {
Operation string `json:"operation" validate:"required,oneof=add remove"` Subsystem string `json:"subsystem" validate:"required,oneof=system forwarding docker"`
ID uint `json:"id"` Backend string `json:"backend" validate:"required,oneof=firewalld ufw iptables nftables"`
Chain string `json:"chain" validate:"required,oneof=1PANEL_BASIC 1PANEL_BASIC_BEFORE 1PANEL_INPUT 1PANEL_OUTPUT"` Operation string `json:"operation" validate:"required,oneof=select initialize cleanup"`
Protocol string `json:"protocol"`
SrcIP string `json:"srcIP"`
SrcPort uint `json:"srcPort"`
DstIP string `json:"dstIP"`
DstPort uint `json:"dstPort"`
Strategy string `json:"strategy" validate:"required,oneof=accept drop reject"`
Description string `json:"description"`
} }
type IptablesBatchOperate struct { type FirewallIPv6Operation struct {
Rules []IptablesRuleOp `json:"rules"` Status string `json:"status" validate:"required,oneof=Enable Disable"`
} }
type IptablesChainStatus struct { type FirewallFamilyOperation struct {
IsBind bool `json:"isBind"` Subsystem string `json:"subsystem" validate:"required,oneof=system forwarding docker"`
DefaultStrategy string `json:"defaultStrategy"` Backend string `json:"backend" validate:"required,oneof=iptables nftables"`
Family string `json:"family" validate:"required,oneof=ipv4 ipv6"`
Operation string `json:"operation" validate:"required,oneof=initialize repair bind"`
}
type FilterChainOperation struct {
Name string `json:"name" validate:"required,eq=1PANEL_BASIC"`
Operate string `json:"operate" validate:"required,oneof=init-base bind-base unbind-base"`
TaskID string `json:"taskID,omitempty" validate:"omitempty,max=64"`
}
type FilterChainOperationResponse struct {
TaskID string `json:"taskID"`
Queued bool `json:"queued"`
}
type FirewallInitializationTask struct {
BackupFile string `json:"backupFile,omitempty" validate:"omitempty,max=255"`
TaskID string `json:"taskID,omitempty" validate:"omitempty,max=64"`
}
type FirewallSystemPort = firewall.SystemPort
type FirewallRuleInventoryResponse struct {
IPv4Range filter.PositionRange `json:"ipv4Range"`
IPv6Range filter.PositionRange `json:"ipv6Range"`
Total int64 `json:"total"`
AllTotal int64 `json:"allTotal"`
Items []filter.InventoryItem `json:"items"`
Notices []filter.ScopeNotice `json:"notices,omitempty"`
}
type FirewallRuleBackup struct {
Name string `json:"name"`
Provider filter.Provider `json:"provider"`
RuleCount int `json:"ruleCount"`
ModifiedAt int64 `json:"modifiedAt"`
}
type FirewallRuleBackups struct {
Directory string `json:"directory"`
Files []FirewallRuleBackup `json:"files"`
}
type FirewallRuleResetResponse struct {
BackupPath string `json:"backupPath"`
Removed int `json:"removed"`
Disabled bool `json:"disabled"`
}
type FirewallRuleReset struct {
Subsystem string `json:"subsystem,omitempty" validate:"omitempty,oneof=system forwarding docker"`
Backup *bool `json:"backup,omitempty" default:"true"`
Provider filter.Provider `json:"provider,omitempty" validate:"omitempty,oneof=firewalld ufw iptables nftables"`
WithDockerRestart bool `json:"withDockerRestart"`
}
type FirewallRuleInventory struct {
PageInfo
Scope filter.Scope `json:"scope,omitempty"`
Scopes []filter.Scope `json:"scopes,omitempty" validate:"max=16"`
All bool `json:"all,omitempty"`
Info string `json:"info"`
Families []filter.Family `json:"families,omitempty" validate:"omitempty,dive,oneof=ipv4 ipv6"`
Actions []string `json:"actions,omitempty" validate:"omitempty,dive,oneof=accept deny"`
ExcludeChains []string `json:"excludeChains,omitempty" validate:"omitempty,dive,oneof=1PANEL_BASIC_BEFORE 1PANEL_BASIC 1PANEL_BASIC_AFTER"`
}
type FirewallNativeDetail struct {
Provider filter.Provider `json:"provider" validate:"required,oneof=firewalld ufw"`
NativeKind filter.NativeKind `json:"nativeKind" validate:"required,oneof=zone_service ufw_application"`
Name string `json:"name" validate:"required"`
Permanent bool `json:"permanent"`
}
type DockerPortGuardBase struct {
IPv6Enabled bool `json:"ipv6Enabled"`
Name string `json:"name"`
Version string `json:"version"`
IsExist bool `json:"isExist"`
Initialized bool `json:"initialized"`
Bound bool `json:"bound"`
IPv4 DockerPortGuardFamilyStatus `json:"ipv4"`
IPv6 DockerPortGuardFamilyStatus `json:"ipv6"`
Backend string `json:"backend"`
Message string `json:"message,omitempty"`
}
type DockerPortGuardFamilyStatus struct {
Partial bool `json:"partial"`
State string `json:"state"`
Reason string `json:"reason,omitempty"`
Initialized bool `json:"initialized"`
Bound bool `json:"bound"`
Effective bool `json:"effective"`
}
type DockerPortGuardEndpoint struct {
Family string `json:"family"`
HostIP string `json:"hostIP"`
HostPort uint16 `json:"hostPort"`
Protocol string `json:"protocol"`
ContainerID string `json:"containerID"`
ContainerName string `json:"containerName"`
ContainerState string `json:"containerState,omitempty"`
ContainerPort uint16 `json:"containerPort"`
Compose string `json:"compose,omitempty"`
Application string `json:"application,omitempty"`
PolicyUUID string `json:"policyUUID,omitempty"`
Mode string `json:"mode,omitempty"`
Sources []string `json:"sources"`
Effective bool `json:"effective"`
TrafficPath string `json:"trafficPath"`
ManagementTarget string `json:"managementTarget"`
ManagementReason string `json:"managementReason,omitempty"`
}
type DockerPortGuardPortGroup struct {
Key string `json:"key"`
Label string `json:"label"`
Endpoint DockerPortGuardEndpoint `json:"endpoint"`
Endpoints []DockerPortGuardEndpoint `json:"endpoints"`
}
type DockerPortGuardContainer struct {
Key string `json:"key"`
Name string `json:"name"`
Compose string `json:"compose,omitempty"`
Application string `json:"application,omitempty"`
Endpoints []DockerPortGuardEndpoint `json:"endpoints"`
PortGroups []DockerPortGuardPortGroup `json:"portGroups"`
}
type DockerPortGuardList struct {
Base DockerPortGuardBase `json:"base"`
Containers []DockerPortGuardContainer `json:"containers"`
OrphanPolicies []DockerPortGuardEndpoint `json:"orphanPolicies"`
}
type DockerPortGuardEndpointIdentity struct {
Family string `json:"family" validate:"required,oneof=ipv4 ipv6"`
HostIP string `json:"hostIP" validate:"required,max=45"`
HostPort uint16 `json:"hostPort" validate:"required,min=1"`
Protocol string `json:"protocol" validate:"required,oneof=tcp udp"`
}
type DockerPortGuardPolicyBatch struct {
Policies []DockerPortGuardPolicy `json:"policies" validate:"required,min=1,dive"`
Import bool `json:"import"`
}
type DockerPortGuardPolicyBatchDelete struct {
UUIDs []string `json:"uuids" validate:"required,min=1,dive,required,max=64"`
}
type DockerPortGuardPolicy struct {
DockerPortGuardEndpointIdentity
Mode string `json:"mode" validate:"required,oneof=deny_sources allow_sources deny_all accept_sources accept_all"`
Sources []string `json:"sources" validate:"dive,required,max=64"`
}
type DockerPortGuardOperation struct {
BackupFile string `json:"backupFile,omitempty" validate:"omitempty,max=255"`
Operation string `json:"operation" validate:"required,oneof=initialize bind unbind"`
TaskID string `json:"taskID,omitempty" validate:"omitempty,max=64"`
}
type FirewallRuleCreateItem struct {
Raw string `json:"raw,omitempty"`
ParseStatus filter.ParseStatus `json:"parseStatus,omitempty"`
Rule filter.FirewallRule `json:"rule" validate:"required"`
SourceKind string `json:"sourceKind" validate:"omitempty,oneof=user imported"`
}
type FirewallRuleCreate struct {
BackupFile string `json:"backupFile,omitempty" validate:"omitempty,max=255"`
Initialize bool `json:"initialize"`
Items []FirewallRuleCreateItem `json:"items" validate:"dive"`
}
type FirewallRuleCreateResponse struct {
TaskID string `json:"taskID,omitempty"`
Queued bool `json:"queued,omitempty"`
Succeeded int `json:"succeeded"`
Failed int `json:"failed"`
Skipped int `json:"skipped"`
Errors []FirewallRuleCreateFailure `json:"errors,omitempty"`
}
type FirewallRuleCreateFailure struct {
Index int `json:"index"`
Status string `json:"status"`
Rule filter.FirewallRule `json:"rule"`
Error string `json:"error,omitempty"`
}
type FirewallRuleDelete struct {
Targets []FirewallRuleDeleteItem `json:"targets" validate:"required,min=1,dive"`
}
type FirewallRuleDeleteItem struct {
FirewallRuleDeleteTarget
Observed filter.ObservedRule `json:"observed" validate:"required"`
}
type FirewallRuleDeleteTarget struct {
Scope filter.Scope `json:"scope" validate:"required"`
InstanceKey string `json:"instanceKey" validate:"required,max=128"`
}
type FirewallRuleDeleteResponse struct {
TaskID string `json:"taskID,omitempty"`
Queued bool `json:"queued,omitempty"`
Succeeded int `json:"succeeded"`
Failed int `json:"failed"`
Errors []FirewallRuleDeleteFailure `json:"errors,omitempty"`
}
type FirewallRuleDeleteFailure struct {
Index int `json:"index"`
InstanceKey string `json:"instanceKey"`
Error string `json:"error"`
}
type FirewallRuleUpdate struct {
FirewallRuleDeleteTarget
Rule *filter.FirewallRule `json:"rule,omitempty" validate:"required_without_all=Description OrderIndex Priority,excluded_with=Description OrderIndex Priority"`
Description *string `json:"description,omitempty" validate:"excluded_with=Rule"`
OrderIndex *int64 `json:"orderIndex,omitempty" validate:"excluded_with=Rule Priority"`
Priority *int `json:"priority,omitempty" validate:"excluded_with=Rule OrderIndex"`
}
type FirewallRuleReorder struct {
FirewallRuleDeleteTarget
TargetPosition *int64 `json:"targetPosition"`
Priority *int `json:"priority"`
}
type FirewallRuleExportItem struct {
filter.FirewallRule
Raw string `json:"raw,omitempty"`
ParseStatus filter.ParseStatus `json:"parseStatus,omitempty"`
}
type FirewallSubsystemBackup struct {
Families []string `json:"families,omitempty"`
Subsystem string `json:"subsystem"`
Provider filter.Provider `json:"provider"`
Forwarding []forwarding.Rule `json:"forwarding"`
Docker *dockerfirewall.PolicyInventory `json:"docker,omitempty"`
} }
+43
View File
@@ -0,0 +1,43 @@
package dto
type ForwardRuleSearch struct {
PageInfo
All bool `json:"all,omitempty"`
Info string `json:"info"`
Status string `json:"status"`
Strategy string `json:"strategy"`
}
type ForwardRule struct {
ID uint `json:"id"`
Chain string `json:"chain"`
Family string `json:"family"`
Address string `json:"address"`
Port string `json:"port"`
Protocol string `json:"protocol"`
Strategy string `json:"strategy"`
Num string `json:"num"`
TargetIP string `json:"targetIP"`
TargetPort string `json:"targetPort"`
Interface string `json:"interface"`
UsedStatus string `json:"usedStatus"`
Description string `json:"description"`
}
type ForwardRuleOperate struct {
Import bool `json:"import"`
Rules []ForwardRuleOperation `json:"rules" validate:"required,min=1,dive"`
}
type ForwardRuleOperation struct {
Operation string `json:"operation" validate:"required,oneof=add remove"`
Num string `json:"num"`
Family string `json:"family" validate:"omitempty,oneof=ipv4 ipv6"`
Protocol string `json:"protocol" validate:"required,oneof=tcp udp tcp/udp"`
Interface string `json:"interface"`
Port string `json:"port" validate:"required"`
TargetIP string `json:"targetIP"`
TargetPort string `json:"targetPort" validate:"required"`
}
+89 -18
View File
@@ -1,6 +1,10 @@
package dto package dto
import "time" import (
"time"
"github.com/1Panel-dev/1Panel/agent/app/model"
)
type MonitorSearch struct { type MonitorSearch struct {
Param string `json:"param" validate:"required,oneof=all cpu memory load io network"` Param string `json:"param" validate:"required,oneof=all cpu memory load io network"`
@@ -26,50 +30,92 @@ type Process struct {
} }
type MonitorSetting struct { type MonitorSetting struct {
MonitorStatus string `json:"monitorStatus"` GPUMonitorStatus string `json:"gpuMonitorStatus"`
MonitorStoreDays string `json:"monitorStoreDays"` GPUMonitorStoreDays string `json:"gpuMonitorStoreDays"`
MonitorInterval string `json:"monitorInterval"` GPUMonitorInterval string `json:"gpuMonitorInterval"`
DefaultNetwork string `json:"defaultNetwork"` VLLMMonitorStatus string `json:"vllmMonitorStatus"`
DefaultIO string `json:"defaultIO"` VLLMMonitorStoreDays string `json:"vllmMonitorStoreDays"`
VLLMMonitorInterval string `json:"vllmMonitorInterval"`
MonitorStatus string `json:"monitorStatus"`
MonitorStoreDays string `json:"monitorStoreDays"`
MonitorInterval string `json:"monitorInterval"`
DefaultNetwork string `json:"defaultNetwork"`
DefaultIO string `json:"defaultIO"`
} }
type MonitorSettingUpdate struct { type MonitorSettingUpdate struct {
Key string `json:"key" validate:"required,oneof=MonitorStatus MonitorStoreDays MonitorInterval DefaultNetwork DefaultIO"` Key string `json:"key" validate:"required,oneof=MonitorStatus MonitorStoreDays MonitorInterval GPUMonitorStatus GPUMonitorStoreDays GPUMonitorInterval VLLMMonitorStatus VLLMMonitorStoreDays VLLMMonitorInterval DefaultNetwork DefaultIO"`
Value string `json:"value"` Value string `json:"value"`
} }
type MonitorGPUOptions struct { type MonitorGPUOptions struct {
Supported bool `json:"supported"`
GPUType string `json:"gpuType"` GPUType string `json:"gpuType"`
ChartHide []GPUChartHide `json:"chartHide"` ChartHide []GPUChartHide `json:"chartHide"`
Options []string `json:"options"` Options []string `json:"options"`
} }
type GPUChartHide struct { type GPUChartHide struct {
DeviceID string `json:"deviceID"`
Legacy bool `json:"legacy"`
ProductName string `json:"productName"` ProductName string `json:"productName"`
Type string `json:"type"`
Process bool `json:"process"` Process bool `json:"process"`
GPU bool `json:"gpu"` GPU bool `json:"gpu"`
Memory bool `json:"memory"` Memory bool `json:"memory"`
Power bool `json:"power"` Power bool `json:"power"`
PowerLimit bool `json:"powerLimit"`
Temperature bool `json:"temperature"` Temperature bool `json:"temperature"`
Speed bool `json:"speed"` Speed bool `json:"speed"`
} }
type MonitorGPUSearch struct { type MonitorGPUSearch struct {
Aggregation string `json:"aggregation" validate:"omitempty,oneof=avg max"`
DeviceID string `json:"deviceID"`
Legacy bool `json:"legacy"`
ProductName string `json:"productName"` ProductName string `json:"productName"`
StartTime time.Time `json:"startTime"` StartTime time.Time `json:"startTime"`
EndTime time.Time `json:"endTime"` EndTime time.Time `json:"endTime"`
} }
type MonitorGPUData struct { type MonitorGPUData struct {
Date []time.Time `json:"date"` MemoryActivity []*float64 `json:"memoryActivity"`
GPUValue []float64 `json:"gpuValue"` EncoderUtil []*float64 `json:"encoderUtil"`
TemperatureValue []float64 `json:"temperatureValue"` DecoderUtil []*float64 `json:"decoderUtil"`
PowerTotal []float64 `json:"powerTotal"` JPEGUtil []*float64 `json:"jpegUtil"`
PowerUsed []float64 `json:"powerUsed"` OFAUtil []*float64 `json:"ofaUtil"`
PowerPercent []float64 `json:"powerPercent"` MediaUtil []*float64 `json:"mediaUtil"`
MemoryTotal []float64 `json:"memoryTotal"` ComputeUtil []*float64 `json:"computeUtil"`
MemoryUsed []float64 `json:"memoryUsed"` CopyUtil []*float64 `json:"copyUtil"`
MemoryPercent []float64 `json:"memoryPercent"` HotspotTemperature []*float64 `json:"hotspotTemperature"`
SpeedValue []int `json:"speedValue"` FanRPM []*float64 `json:"fanRPM"`
AICPUUtil []*float64 `json:"aiCPUUtil"`
CtrlCPUUtil []*float64 `json:"ctrlCPUUtil"`
DDRUsed []*float64 `json:"ddrUsed"`
DDRTotal []*float64 `json:"ddrTotal"`
HBMUsed []*float64 `json:"hbmUsed"`
HBMTotal []*float64 `json:"hbmTotal"`
DDRBandwidth []*float64 `json:"ddrBandwidth"`
HBMBandwidth []*float64 `json:"hbmBandwidth"`
MemoryBandwidth []*float64 `json:"memoryBandwidth"`
MediaFrequency []*float64 `json:"mediaFrequency"`
HugepagesUsed []*float64 `json:"hugepagesUsed"`
HugepagesTotal []*float64 `json:"hugepagesTotal"`
ProcessCount []int `json:"processCount"` BucketSeconds int64 `json:"bucketSeconds"`
SampleCount int64 `json:"sampleCount"`
MemoryTemperatureValue []*float64 `json:"memoryTemperatureValue"`
FrequencyValue []*float64 `json:"frequencyValue"`
MemoryFrequencyValue []*float64 `json:"memoryFrequencyValue"`
Date []time.Time `json:"date"`
GPUValue []*float64 `json:"gpuValue"`
TemperatureValue []*float64 `json:"temperatureValue"`
PowerTotal []*float64 `json:"powerTotal"`
PowerUsed []*float64 `json:"powerUsed"`
PowerPercent []*float64 `json:"powerPercent"`
MemoryTotal []*float64 `json:"memoryTotal"`
MemoryUsed []*float64 `json:"memoryUsed"`
MemoryPercent []*float64 `json:"memoryPercent"`
SpeedValue []*float64 `json:"speedValue"`
ProcessCount []*float64 `json:"processCount"`
GPUProcesses [][]GPUProcess `json:"gpuProcesses"` GPUProcesses [][]GPUProcess `json:"gpuProcesses"`
} }
@@ -79,3 +125,28 @@ type GPUProcess struct {
ProcessName string `json:"processName"` ProcessName string `json:"processName"`
UsedMemory string `json:"usedMemory"` UsedMemory string `json:"usedMemory"`
} }
type MonitorVLLMSearch struct {
AppInstallID uint `json:"appInstallID" validate:"required"`
StartTime time.Time `json:"startTime" validate:"required"`
EndTime time.Time `json:"endTime" validate:"required"`
Aggregation string `json:"aggregation" validate:"omitempty,oneof=avg max"`
}
type MonitorVLLMData struct {
SampleCount int64 `json:"sampleCount"`
BucketSeconds int64 `json:"bucketSeconds"`
Points []model.MonitorVLLM `json:"points"`
}
type MonitorVLLMCurrent struct {
AppInstallID uint `json:"appInstallID" validate:"required"`
}
type MonitorVLLMClean struct {
AppInstallID uint `json:"appInstallID" validate:"required"`
}
type MonitorClean struct {
Type string `json:"type" validate:"required,oneof=host gpu"`
}
+7 -1
View File
@@ -51,13 +51,19 @@ const (
CACHE NginxKey = "cache" CACHE NginxKey = "cache"
HttpPer NginxKey = "http-per" HttpPer NginxKey = "http-per"
ProxyCache NginxKey = "proxy-cache" ProxyCache NginxKey = "proxy-cache"
Brotli NginxKey = "brotli"
) )
// BrotliKeys are served from the panel-managed http.d file rather than
// nginx.conf, because the module is optional: its directives must disappear
// together with the module, otherwise nginx refuses to start.
var BrotliKeys = []string{"brotli", "brotli_comp_level", "brotli_min_length", "brotli_types"}
var ScopeKeyMap = map[NginxKey][]string{ var ScopeKeyMap = map[NginxKey][]string{
Index: {"index"}, Index: {"index"},
LimitConn: {"limit_conn", "limit_rate", "limit_conn_zone"}, LimitConn: {"limit_conn", "limit_rate", "limit_conn_zone"},
SSL: {"ssl_certificate", "ssl_certificate_key"}, SSL: {"ssl_certificate", "ssl_certificate_key"},
HttpPer: {"server_names_hash_bucket_size", "client_header_buffer_size", "client_max_body_size", "keepalive_timeout", "gzip", "gzip_min_length", "gzip_comp_level"}, HttpPer: {"server_names_hash_bucket_size", "client_header_buffer_size", "client_max_body_size", "keepalive_timeout", "gzip", "gzip_min_length", "gzip_comp_level", "gzip_types", "gzip_vary", "gzip_proxied"},
} }
var StaticFileKeyMap = map[NginxKey]struct { var StaticFileKeyMap = map[NginxKey]struct {
+9
View File
@@ -50,6 +50,10 @@ type AppContainerConfig struct {
Type string `json:"type"` Type string `json:"type"`
SpecifyIP string `json:"specifyIP"` SpecifyIP string `json:"specifyIP"`
RestartPolicy string `json:"restartPolicy" validate:"omitempty,oneof=always unless-stopped no on-failure"` RestartPolicy string `json:"restartPolicy" validate:"omitempty,oneof=always unless-stopped no on-failure"`
KeepServiceName bool `json:"-"`
SkipComposeCommonConfig bool `json:"-"`
UseLifecycleScripts bool `json:"-"`
} }
type AppInstalledSearch struct { type AppInstalledSearch struct {
@@ -92,6 +96,8 @@ type AppInstalledOperate struct {
TaskID string `json:"taskID"` TaskID string `json:"taskID"`
DeleteImage bool `json:"deleteImage"` DeleteImage bool `json:"deleteImage"`
Favorite bool `json:"favorite"` Favorite bool `json:"favorite"`
UseLifecycleScripts bool `json:"-"`
} }
type AppInstallUpgrade struct { type AppInstallUpgrade struct {
@@ -111,11 +117,14 @@ type AppInstallDelete struct {
DeleteDB bool `json:"deleteDB"` DeleteDB bool `json:"deleteDB"`
DeleteImage bool `json:"deleteImage"` DeleteImage bool `json:"deleteImage"`
TaskID string `json:"taskID"` TaskID string `json:"taskID"`
UseLifecycleScripts bool `json:"-"`
} }
type AppInstalledUpdate struct { type AppInstalledUpdate struct {
InstallId uint `json:"installId" validate:"required"` InstallId uint `json:"installId" validate:"required"`
Params map[string]interface{} `json:"params" validate:"required"` Params map[string]interface{} `json:"params" validate:"required"`
TaskID string `json:"-"`
AppContainerConfig AppContainerConfig
} }
+10
View File
@@ -122,6 +122,7 @@ type FileWget struct {
Name string `json:"name" validate:"required"` Name string `json:"name" validate:"required"`
IgnoreCertificate bool `json:"ignoreCertificate"` IgnoreCertificate bool `json:"ignoreCertificate"`
UseProxy bool `json:"useProxy"` UseProxy bool `json:"useProxy"`
UseServerFilename bool `json:"useServerFilename"`
} }
type FileMove struct { type FileMove struct {
@@ -131,6 +132,11 @@ type FileMove struct {
Name string `json:"name"` Name string `json:"name"`
Cover bool `json:"cover"` Cover bool `json:"cover"`
CoverPaths []string `json:"coverPaths"` CoverPaths []string `json:"coverPaths"`
TaskID string `json:"taskID"`
}
type FileMoveStopReq struct {
TaskID string `json:"taskID" validate:"required"`
} }
type FileDownload struct { type FileDownload struct {
@@ -153,6 +159,10 @@ type FileProcessReq struct {
Key string `json:"key"` Key string `json:"key"`
} }
type FileProcessRemoveReq struct {
Keys []string `json:"keys" validate:"required,min=1,max=1000"`
}
type FileRoleUpdate struct { type FileRoleUpdate struct {
Path string `json:"path" validate:"required"` Path string `json:"path" validate:"required"`
User string `json:"user" validate:"required"` User string `json:"user" validate:"required"`
+9 -8
View File
@@ -66,14 +66,15 @@ type RuntimeDelete struct {
} }
type RuntimeUpdate struct { type RuntimeUpdate struct {
Name string `json:"name"` AppDetailID uint `json:"appDetailId"`
ID uint `json:"id"` Name string `json:"name"`
Image string `json:"image"` ID uint `json:"id"`
Version string `json:"version"` Image string `json:"image"`
Rebuild bool `json:"rebuild"` Version string `json:"version"`
Source string `json:"source"` Rebuild bool `json:"rebuild"`
CodeDir string `json:"codeDir"` Source string `json:"source"`
Remark string `json:"remark"` CodeDir string `json:"codeDir"`
Remark string `json:"remark"`
Params map[string]interface{} `json:"params"` Params map[string]interface{} `json:"params"`
NodeConfig NodeConfig
+2
View File
@@ -34,6 +34,8 @@ type WebsiteCreate struct {
SiteDir string `json:"siteDir"` SiteDir string `json:"siteDir"`
TemplateOutputID uint `json:"templateOutputID"`
RuntimeConfig RuntimeConfig
FtpConfig FtpConfig
DataBaseConfig DataBaseConfig
+46
View File
@@ -0,0 +1,46 @@
package request
import (
"github.com/1Panel-dev/1Panel/agent/app/dto"
)
type WebsiteTemplateSearch struct {
dto.PageInfo
Name string `json:"name"`
Type string `json:"type"`
}
type WebsiteTemplateCreate struct {
Name string `json:"name" validate:"required"`
Type string `json:"type" validate:"required,oneof=single multi"`
Content string `json:"content"`
FilePath string `json:"filePath"`
Variables string `json:"variables"`
Remark string `json:"remark"`
}
type WebsiteTemplateUpdate struct {
ID uint `json:"id" validate:"required"`
Name string `json:"name" validate:"required"`
Type string `json:"type" validate:"required,oneof=single multi"`
Content string `json:"content"`
FilePath string `json:"filePath"`
Variables string `json:"variables"`
Remark string `json:"remark"`
}
type WebsiteTemplateOutputSearch struct {
dto.PageInfo
TemplateID uint `json:"templateID"`
}
type WebsiteTemplateOutputCreate struct {
TemplateID uint `json:"templateID" validate:"required"`
Name string `json:"name" validate:"required"`
VariableValues map[string]string `json:"variableValues"`
}
type WebsitePreviewReq struct {
TemplateID uint `json:"templateID" validate:"required"`
VariableValues map[string]string `json:"variableValues"`
}
+11
View File
@@ -17,6 +17,17 @@ type NginxParam struct {
Params []string `json:"params"` Params []string `json:"params"`
} }
// NginxBrotliRes carries the brotli settings together with where they live.
// ManagedExternally is true when the user defined brotli by hand, in which
// case the panel only reports the values and must not write its own copy.
// ManagedUnavailable is true when the panel could not wire the managed
// configuration into nginx.conf at all, so the reported values are inert.
type NginxBrotliRes struct {
Params []NginxParam `json:"params"`
ManagedExternally bool `json:"managedExternally"`
ManagedUnavailable bool `json:"managedUnavailable"`
}
type NginxAuthRes struct { type NginxAuthRes struct {
Enable bool `json:"enable"` Enable bool `json:"enable"`
Items []dto.NginxAuth `json:"items"` Items []dto.NginxAuth `json:"items"`
@@ -0,0 +1,18 @@
package response
import (
"github.com/1Panel-dev/1Panel/agent/app/model"
)
type WebsiteTemplateDTO struct {
model.WebsiteTemplate
}
type WebsiteTemplateOutputDTO struct {
model.WebsiteTemplateOutput
TemplateName string `json:"templateName"`
}
type WebsitePreviewDTO struct {
HTML string `json:"html"`
}
+1 -1
View File
@@ -35,7 +35,7 @@ type SettingUpdate struct {
} }
type AgentSettingUpdate struct { type AgentSettingUpdate struct {
Key string `json:"key" validate:"required,oneof=SystemIP DockerSockPath FileRecycleBin FirewallPortWhiteList"` Key string `json:"key" validate:"required,oneof=SystemIP DockerSockPath FileRecycleBin"`
Value string `json:"value"` Value string `json:"value"`
} }
+1 -1
View File
@@ -25,7 +25,7 @@ type RootCertOperate struct {
ID uint `json:"id"` ID uint `json:"id"`
Name string `json:"name"` Name string `json:"name"`
Mode string `json:"mode"` Mode string `json:"mode"`
EncryptionMode string `json:"encryptionMode" validate:"required,oneof=rsa ed25519 ecdsa dsa"` EncryptionMode string `json:"encryptionMode"`
PassPhrase string `json:"passPhrase"` PassPhrase string `json:"passPhrase"`
PublicKey string `json:"publicKey"` PublicKey string `json:"publicKey"`
PrivateKey string `json:"privateKey"` PrivateKey string `json:"privateKey"`
+11
View File
@@ -0,0 +1,11 @@
package dto
type TerminalSessionClose struct {
ID string `json:"id" validate:"required"`
}
type TerminalSessionRevoke struct {
Scope string `json:"scope" validate:"required,oneof=auth_session user all"`
UserID string `json:"userId"`
AuthSessionID string `json:"authSessionId"`
}
+27 -10
View File
@@ -1,5 +1,12 @@
package model package model
import (
"strings"
"github.com/google/uuid"
"gorm.io/gorm"
)
type Alert struct { type Alert struct {
BaseModel BaseModel
@@ -18,10 +25,11 @@ type Alert struct {
type AlertTask struct { type AlertTask struct {
BaseModel BaseModel
Type string `gorm:"type:varchar(64);not null" json:"type"` Type string `gorm:"type:varchar(64);not null" json:"type"`
Quota string `gorm:"type:varchar(64)" json:"quota"` Quota string `gorm:"type:varchar(64)" json:"quota"`
QuotaType string `gorm:"type:varchar(64)" json:"quotaType"` QuotaType string `gorm:"type:varchar(64)" json:"quotaType"`
Method string `gorm:"type:varchar(128);not null;default:'sms'" json:"method"` Method string `gorm:"type:varchar(128);not null;default:'sms'" json:"method"`
DeliveryLogID *uint `gorm:"uniqueIndex" json:"-"`
} }
type AlertLog struct { type AlertLog struct {
@@ -41,12 +49,21 @@ type AlertLog struct {
type AlertConfig struct { type AlertConfig struct {
BaseModel BaseModel
Type string `gorm:"type:varchar(64);not null" json:"type"` UID string `gorm:"type:varchar(64);not null;uniqueIndex" json:"uid"`
Title string `gorm:"type:varchar(64);not null" json:"title"` Type string `gorm:"type:varchar(64);not null" json:"type"`
Status string `gorm:"type:varchar(64);not null" json:"status"` Title string `gorm:"type:varchar(64);not null" json:"title"`
Config string `gorm:"type:varchar(256);not null" json:"config"` Status string `gorm:"type:varchar(64);not null" json:"status"`
CreateUser string `gorm:"type:varchar(256)" json:"createUser"` Config string `gorm:"type:text;not null" json:"config"`
UpdateUser string `gorm:"type:varchar(256)" json:"updateUser"` SecretConfig string `gorm:"type:text;not null;default:''" json:"-"`
CreateUser string `gorm:"type:varchar(256)" json:"createUser"`
UpdateUser string `gorm:"type:varchar(256)" json:"updateUser"`
}
func (a *AlertConfig) BeforeCreate(_ *gorm.DB) error {
if strings.TrimSpace(a.UID) == "" {
a.UID = uuid.NewString()
}
return nil
} }
type LoginLog struct { type LoginLog struct {
+1
View File
@@ -31,4 +31,5 @@ type BackupRecord struct {
Status string `json:"status"` Status string `json:"status"`
Message string `json:"message"` Message string `json:"message"`
Description string `json:"description"` Description string `json:"description"`
Args string `gorm:"not null;default:''" json:"args"`
} }
+3 -2
View File
@@ -11,6 +11,7 @@ type ComposeTemplate struct {
type Compose struct { type Compose struct {
BaseModel BaseModel
Name string `json:"name"` Name string `json:"name"`
Path string `json:"path"` Path string `json:"path"`
IsPinned bool `json:"isPinned"`
} }
-18
View File
@@ -1,18 +0,0 @@
package model
type Firewall struct {
BaseModel
Type string `json:"type"`
Port string `json:"port"` // Deprecated
Address string `json:"address"` // Deprecated
Chain string `json:"chain"`
Protocol string `json:"protocol"`
SrcIP string `json:"srcIP"`
SrcPort string `json:"srcPort"`
DstIP string `json:"dstIP"`
DstPort string `json:"dstPort"`
Strategy string `gorm:"not null" json:"strategy"`
Description string `json:"description"`
}
+2
View File
@@ -8,4 +8,6 @@ type Ftp struct {
Status string `gorm:"not null" json:"status"` Status string `gorm:"not null" json:"status"`
Path string `gorm:"not null" json:"path"` Path string `gorm:"not null" json:"path"`
Description string `gorm:"not null" json:"description"` Description string `gorm:"not null" json:"description"`
UID uint `gorm:"column:uid;not null;default:1000" json:"-"`
GID uint `gorm:"column:gid;not null;default:1000" json:"-"`
} }
+75 -9
View File
@@ -1,5 +1,7 @@
package model package model
import "time"
type MonitorBase struct { type MonitorBase struct {
BaseModel BaseModel
Cpu float64 `json:"cpu"` Cpu float64 `json:"cpu"`
@@ -33,14 +35,78 @@ type MonitorNetwork struct {
} }
type MonitorGPU struct { type MonitorGPU struct {
MemoryUtil *float64 `json:"memoryUtil"`
MemoryActivity *float64 `json:"memoryActivity"`
EncoderUtil *float64 `json:"encoderUtil"`
DecoderUtil *float64 `json:"decoderUtil"`
JPEGUtil *float64 `json:"jpegUtil"`
OFAUtil *float64 `json:"ofaUtil"`
MediaUtil *float64 `json:"mediaUtil"`
ComputeUtil *float64 `json:"computeUtil"`
CopyUtil *float64 `json:"copyUtil"`
HotspotTemperature *float64 `json:"hotspotTemperature"`
FanRPM *float64 `json:"fanRPM"`
AICPUUtil *float64 `json:"aiCPUUtil"`
CtrlCPUUtil *float64 `json:"ctrlCPUUtil"`
DDRUsed *float64 `json:"ddrUsed"`
DDRTotal *float64 `json:"ddrTotal"`
HBMUsed *float64 `json:"hbmUsed"`
HBMTotal *float64 `json:"hbmTotal"`
DDRBandwidth *float64 `json:"ddrBandwidth"`
HBMBandwidth *float64 `json:"hbmBandwidth"`
MemoryBandwidth *float64 `json:"memoryBandwidth"`
MediaFrequency *float64 `json:"mediaFrequency"`
HugepagesUsed *float64 `json:"hugepagesUsed"`
HugepagesTotal *float64 `json:"hugepagesTotal"`
MemoryTemperature *float64 `json:"memoryTemperature"`
DeviceID string `json:"deviceID"`
DeviceType string `json:"deviceType"`
ProcessStatus string `json:"processStatus"`
Frequency *float64 `json:"frequency"`
MemoryFrequency *float64 `json:"memoryFrequency"`
IntervalSeconds int `json:"intervalSeconds"`
BaseModel BaseModel
ProductName string `json:"productName"` ProductName string `json:"productName"`
GPUUtil float64 `json:"gpuUtil"` GPUUtil *float64 `json:"gpuUtil"`
Temperature float64 `json:"temperature"` Temperature *float64 `json:"temperature"`
PowerDraw float64 `json:"powerDraw"` PowerDraw *float64 `json:"powerDraw"`
MaxPowerLimit float64 `json:"maxPowerLimit"` MaxPowerLimit *float64 `json:"maxPowerLimit"`
MemUsed float64 `json:"memUsed"` MemUsed *float64 `json:"memUsed"`
MemTotal float64 `json:"memTotal"` MemTotal *float64 `json:"memTotal"`
FanSpeed int `json:"fanSpeed"` FanSpeed *float64 `json:"fanSpeed"`
Processes string `json:"processes"` Processes string `json:"processes"`
}
type MonitorVLLM struct {
ID uint `json:"-" gorm:"primarykey;autoIncrement"`
CreatedAt time.Time `json:"createdAt"`
AppInstallID uint `json:"appInstallID"`
Status string `json:"status"`
RawMetrics string `json:"-"`
HistogramDeltas string `json:"-"`
Running *float64 `json:"running"`
Waiting *float64 `json:"waiting"`
CacheUsage *float64 `json:"cacheUsage"`
PromptThroughput *float64 `json:"promptThroughput"`
GenerationThroughput *float64 `json:"generationThroughput"`
RequestThroughput *float64 `json:"requestThroughput"`
TimeToFirstToken *float64 `json:"timeToFirstToken"`
TimePerOutputToken *float64 `json:"timePerOutputToken"`
RequestLatency *float64 `json:"requestLatency"`
PrefillTime *float64 `json:"prefillTime"`
DecodeTime *float64 `json:"decodeTime"`
TimeToFirstTokenP50 *float64 `json:"timeToFirstTokenP50"`
TimeToFirstTokenP90 *float64 `json:"timeToFirstTokenP90"`
TimeToFirstTokenP95 *float64 `json:"timeToFirstTokenP95"`
TimeToFirstTokenP99 *float64 `json:"timeToFirstTokenP99"`
TimePerOutputTokenP50 *float64 `json:"timePerOutputTokenP50"`
TimePerOutputTokenP90 *float64 `json:"timePerOutputTokenP90"`
TimePerOutputTokenP95 *float64 `json:"timePerOutputTokenP95"`
TimePerOutputTokenP99 *float64 `json:"timePerOutputTokenP99"`
RequestLatencyP50 *float64 `json:"requestLatencyP50"`
RequestLatencyP90 *float64 `json:"requestLatencyP90"`
RequestLatencyP95 *float64 `json:"requestLatencyP95"`
RequestLatencyP99 *float64 `json:"requestLatencyP99"`
} }
+28
View File
@@ -0,0 +1,28 @@
package model
type WebsiteTemplate struct {
BaseModel
Name string `gorm:"not null" json:"name"`
Type string `gorm:"not null" json:"type"` // single | multi
Content string `gorm:"type:longtext" json:"content"`
FilePath string `json:"filePath"`
Variables string `gorm:"type:text" json:"variables"`
Remark string `json:"remark"`
}
func (w WebsiteTemplate) TableName() string {
return "website_templates"
}
type WebsiteTemplateOutput struct {
BaseModel
Name string `gorm:"not null" json:"name"`
TemplateID uint `gorm:"not null" json:"templateID"`
TemplateType string `json:"templateType"`
VariableValues string `gorm:"type:text" json:"variableValues"`
OutputPath string `json:"outputPath"`
}
func (w WebsiteTemplateOutput) TableName() string {
return "website_template_outputs"
}
+153 -13
View File
@@ -10,8 +10,10 @@ type APIConfig struct {
APIType string APIType string
BaseURL string BaseURL string
EditableBaseURL bool EditableBaseURL bool
DiscoverModels bool
DefaultAuthMode string DefaultAuthMode string
AuthModes []string AuthModes []string
Models []Model
} }
const ( const (
@@ -27,6 +29,7 @@ type Model struct {
type Meta struct { type Meta struct {
Key string Key string
DisplayName string DisplayName string
DisplayNameKey string
Sort uint Sort uint
DefaultAPIType string DefaultAPIType string
APIConfigs []APIConfig APIConfigs []APIConfig
@@ -37,26 +40,36 @@ type Meta struct {
var catalog = map[string]Meta{ var catalog = map[string]Meta{
"custom": { "custom": {
Key: "custom", DisplayName: "Custom", Sort: 10, DefaultAPIType: "openai-completions", EnvKey: "CUSTOM_API_KEY", Key: "custom", DisplayName: "Custom", Sort: 10, DefaultAPIType: "openai-completions", EnvKey: "CUSTOM_API_KEY",
APIConfigs: editableAPIConfigs("openai-completions", "openai-responses", "anthropic-messages"), APIConfigs: editableAPIConfigs(true, "openai-completions", "openai-responses", "anthropic-messages", "openai-images", "dashscope-images", "openai-embeddings"),
}, },
"ollama": { "ollama": {
Key: "ollama", DisplayName: "Ollama", Sort: 15, DefaultAPIType: "openai-responses", Key: "ollama", DisplayName: "Ollama", Sort: 15, DefaultAPIType: "openai-responses",
APIConfigs: editableAPIConfigs("openai-responses", "openai-completions"), APIConfigs: editableAPIConfigs(false, "openai-responses", "openai-completions", "openai-embeddings"),
},
// llmman (https://github.com/llmmanorg/llmman): local runner with Ollama/OpenAI-compatible routes on 127.0.0.1:17434.
"llmman": {
Key: "llmman", DisplayName: "llmman", Sort: 16, DefaultAPIType: "openai-responses",
APIConfigs: []APIConfig{
{APIType: "openai-responses", BaseURL: "http://127.0.0.1:17434/v1", EditableBaseURL: true},
{APIType: "openai-completions", BaseURL: "http://127.0.0.1:17434/v1", EditableBaseURL: true},
{APIType: "openai-embeddings", BaseURL: "http://127.0.0.1:17434/v1", EditableBaseURL: true},
},
}, },
"vllm": { "vllm": {
Key: "vllm", DisplayName: "vLLM", Sort: 20, DefaultAPIType: "openai-completions", EnvKey: "VLLM_API_KEY", Key: "vllm", DisplayName: "vLLM", Sort: 20, DefaultAPIType: "openai-completions", EnvKey: "VLLM_API_KEY",
APIConfigs: editableAPIConfigs("openai-completions", "openai-responses", "anthropic-messages"), APIConfigs: editableAPIConfigs(false, "openai-completions", "openai-responses", "anthropic-messages", "openai-images", "openai-embeddings"),
}, },
"deepseek": { "deepseek": {
Key: "deepseek", DisplayName: "DeepSeek", Sort: 25, DefaultAPIType: "openai-completions", EnvKey: "DEEPSEEK_API_KEY", Key: "deepseek", DisplayName: "DeepSeek", Sort: 25, DefaultAPIType: "openai-completions", EnvKey: "DEEPSEEK_API_KEY",
APIConfigs: []APIConfig{ APIConfigs: []APIConfig{
{APIType: "openai-completions", BaseURL: "https://api.deepseek.com/v1"}, {APIType: "openai-completions", BaseURL: "https://api.deepseek.com"},
{APIType: "openai-responses", BaseURL: "https://api.deepseek.com"},
anthropicAPIConfig("https://api.deepseek.com/anthropic", AuthModeXAPIKey), anthropicAPIConfig("https://api.deepseek.com/anthropic", AuthModeXAPIKey),
}, },
Models: []Model{{ID: "deepseek-v4-flash", Name: "deepseek-v4-flash"}, {ID: "deepseek-v4-pro", Name: "deepseek-v4-pro"}}, Models: []Model{{ID: "deepseek-v4-flash", Name: "deepseek-v4-flash"}, {ID: "deepseek-v4-pro", Name: "deepseek-v4-pro"}},
}, },
"bailian-coding-plan": { "bailian-coding-plan": {
Key: "bailian-coding-plan", DisplayName: "阿里云百炼 Coding Plan", Sort: 30, DefaultAPIType: "openai-completions", EnvKey: "QWEN_API_KEY", Key: "bailian-coding-plan", DisplayNameKey: "AIProviderBailianCodingPlan", Sort: 30, DefaultAPIType: "openai-completions", EnvKey: "QWEN_API_KEY",
APIConfigs: []APIConfig{ APIConfigs: []APIConfig{
{APIType: "openai-completions", BaseURL: "https://coding.dashscope.aliyuncs.com/v1"}, {APIType: "openai-completions", BaseURL: "https://coding.dashscope.aliyuncs.com/v1"},
anthropicAPIConfig("https://coding.dashscope.aliyuncs.com/apps/anthropic", AuthModeBearer), anthropicAPIConfig("https://coding.dashscope.aliyuncs.com/apps/anthropic", AuthModeBearer),
@@ -75,7 +88,7 @@ var catalog = map[string]Meta{
}, },
}, },
"ark-coding-plan": { "ark-coding-plan": {
Key: "ark-coding-plan", DisplayName: "方舟 Coding Plan", Sort: 35, DefaultAPIType: "openai-completions", EnvKey: "ARK_API_KEY", Key: "ark-coding-plan", DisplayNameKey: "AIProviderArkCodingPlan", Sort: 35, DefaultAPIType: "openai-completions", EnvKey: "ARK_API_KEY",
APIConfigs: []APIConfig{ APIConfigs: []APIConfig{
{APIType: "openai-completions", BaseURL: "https://ark.cn-beijing.volces.com/api/coding/v3"}, {APIType: "openai-completions", BaseURL: "https://ark.cn-beijing.volces.com/api/coding/v3"},
anthropicAPIConfig("https://ark.cn-beijing.volces.com/api/coding", AuthModeBearer), anthropicAPIConfig("https://ark.cn-beijing.volces.com/api/coding", AuthModeBearer),
@@ -88,14 +101,18 @@ var catalog = map[string]Meta{
}, },
"zai": { "zai": {
Key: "zai", DisplayName: "Z.ai", Sort: 40, DefaultAPIType: "openai-completions", EnvKey: "ZAI_API_KEY", Key: "zai", DisplayName: "Z.ai", Sort: 40, DefaultAPIType: "openai-completions", EnvKey: "ZAI_API_KEY",
APIConfigs: []APIConfig{{APIType: "openai-completions", BaseURL: "https://open.bigmodel.cn/api/paas/v4", EditableBaseURL: true}}, APIConfigs: []APIConfig{
Models: []Model{{ID: "glm-5", Name: "GLM-5"}, {ID: "glm-4.7", Name: "GLM-4.7"}, {ID: "glm-4.7-flash", Name: "GLM-4.7-Flash"}, {ID: "glm-4.7-flashx", Name: "GLM-4.7-FlashX"}}, {APIType: "openai-completions", BaseURL: "https://open.bigmodel.cn/api/paas/v4", EditableBaseURL: true},
{APIType: "openai-images", BaseURL: "https://open.bigmodel.cn/api/paas/v4", EditableBaseURL: true},
},
Models: []Model{{ID: "glm-5", Name: "GLM-5"}, {ID: "glm-4.7", Name: "GLM-4.7"}, {ID: "glm-4.7-flash", Name: "GLM-4.7-Flash"}, {ID: "glm-4.7-flashx", Name: "GLM-4.7-FlashX"}},
}, },
"minimax": { "minimax": {
Key: "minimax", DisplayName: "MiniMax (CN)", Sort: 45, DefaultAPIType: "anthropic-messages", EnvKey: "MINIMAX_API_KEY", Key: "minimax", DisplayName: "MiniMax (CN)", Sort: 45, DefaultAPIType: "anthropic-messages", EnvKey: "MINIMAX_API_KEY",
APIConfigs: []APIConfig{ APIConfigs: []APIConfig{
anthropicAPIConfig("https://api.minimaxi.com/anthropic", AuthModeXAPIKey, AuthModeBearer), anthropicAPIConfig("https://api.minimaxi.com/anthropic", AuthModeXAPIKey, AuthModeBearer),
{APIType: "openai-completions", BaseURL: "https://api.minimaxi.com/v1"}, {APIType: "openai-completions", BaseURL: "https://api.minimaxi.com/v1"},
{APIType: "minimax-images", BaseURL: "https://api.minimaxi.com"},
}, },
Models: []Model{{ID: "MiniMax-M3", Name: "MiniMax M3"}, {ID: "MiniMax-M2.7", Name: "MiniMax M2.7"}, {ID: "MiniMax-M2.7-highspeed", Name: "MiniMax M2.7 highspeed"}}, Models: []Model{{ID: "MiniMax-M3", Name: "MiniMax M3"}, {ID: "MiniMax-M2.7", Name: "MiniMax M2.7"}, {ID: "MiniMax-M2.7-highspeed", Name: "MiniMax M2.7 highspeed"}},
}, },
@@ -103,9 +120,10 @@ var catalog = map[string]Meta{
Key: "xiaomi", DisplayName: "Xiaomi", Sort: 46, DefaultAPIType: "openai-completions", EnvKey: "XIAOMI_API_KEY", Key: "xiaomi", DisplayName: "Xiaomi", Sort: 46, DefaultAPIType: "openai-completions", EnvKey: "XIAOMI_API_KEY",
APIConfigs: []APIConfig{ APIConfigs: []APIConfig{
{APIType: "openai-completions", BaseURL: "https://api.xiaomimimo.com/v1"}, {APIType: "openai-completions", BaseURL: "https://api.xiaomimimo.com/v1"},
{APIType: "openai-responses", BaseURL: "https://api.xiaomimimo.com/v1"},
anthropicAPIConfig("https://api.xiaomimimo.com/anthropic", AuthModeBearer), anthropicAPIConfig("https://api.xiaomimimo.com/anthropic", AuthModeBearer),
}, },
Models: []Model{{ID: "mimo-v2-flash", Name: "Xiaomi MiMo V2 Flash"}, {ID: "mimo-v2-pro", Name: "Xiaomi MiMo V2 Pro"}, {ID: "mimo-v2-omni", Name: "Xiaomi MiMo V2 Omni"}}, Models: []Model{{ID: "mimo-v2.5", Name: "Xiaomi MiMo V2.5"}, {ID: "mimo-v2.5-pro", Name: "Xiaomi MiMo V2.5 Pro"}},
}, },
"kimi": { "kimi": {
Key: "kimi", DisplayName: "Kimi (CN)", Sort: 50, DefaultAPIType: "openai-completions", EnvKey: "KIMI_API_KEY", Key: "kimi", DisplayName: "Kimi (CN)", Sort: 50, DefaultAPIType: "openai-completions", EnvKey: "KIMI_API_KEY",
@@ -122,13 +140,21 @@ var catalog = map[string]Meta{
APIConfigs: []APIConfig{ APIConfigs: []APIConfig{
{APIType: "openai-responses", BaseURL: "https://api.openai.com/v1"}, {APIType: "openai-responses", BaseURL: "https://api.openai.com/v1"},
{APIType: "openai-completions", BaseURL: "https://api.openai.com/v1"}, {APIType: "openai-completions", BaseURL: "https://api.openai.com/v1"},
{APIType: "openai-images", BaseURL: "https://api.openai.com/v1"},
{APIType: "openai-embeddings", BaseURL: "https://api.openai.com/v1", Models: []Model{
{ID: "text-embedding-3-small", Name: "text-embedding-3-small"},
{ID: "text-embedding-3-large", Name: "text-embedding-3-large"},
}},
}, },
Models: []Model{{ID: "gpt-5.4", Name: "gpt-5.4"}, {ID: "gpt-5.4-pro", Name: "gpt-5.4-pro"}, {ID: "gpt-5.4-mini", Name: "gpt-5.4-mini"}, {ID: "gpt-5.4-nano", Name: "gpt-5.4-nano"}}, Models: []Model{{ID: "gpt-5.4", Name: "gpt-5.4"}, {ID: "gpt-5.4-pro", Name: "gpt-5.4-pro"}, {ID: "gpt-5.4-mini", Name: "gpt-5.4-mini"}, {ID: "gpt-5.4-nano", Name: "gpt-5.4-nano"}},
}, },
"openrouter": { "openrouter": {
Key: "openrouter", DisplayName: "OpenRouter", Sort: 56, DefaultAPIType: "openai-completions", EnvKey: "OPENROUTER_API_KEY", Key: "openrouter", DisplayName: "OpenRouter", Sort: 56, DefaultAPIType: "openai-completions", EnvKey: "OPENROUTER_API_KEY",
APIConfigs: []APIConfig{{APIType: "openai-completions", BaseURL: "https://openrouter.ai/api/v1"}}, APIConfigs: []APIConfig{
Models: []Model{{ID: "openrouter/free", Name: "openrouter/free"}, {ID: "openrouter/auto", Name: "openrouter/auto"}}, {APIType: "openai-completions", BaseURL: "https://openrouter.ai/api/v1"},
{APIType: "openrouter-images", BaseURL: "https://openrouter.ai"},
},
Models: []Model{{ID: "openrouter/free", Name: "openrouter/free"}, {ID: "openrouter/auto", Name: "openrouter/auto"}},
}, },
"anthropic": { "anthropic": {
Key: "anthropic", DisplayName: "Anthropic", Sort: 60, DefaultAPIType: "anthropic-messages", EnvKey: "ANTHROPIC_API_KEY", Key: "anthropic", DisplayName: "Anthropic", Sort: 60, DefaultAPIType: "anthropic-messages", EnvKey: "ANTHROPIC_API_KEY",
@@ -145,9 +171,62 @@ var catalog = map[string]Meta{
APIConfigs: []APIConfig{{APIType: "openai-completions", BaseURL: "https://api.moonshot.ai/v1"}}, APIConfigs: []APIConfig{{APIType: "openai-completions", BaseURL: "https://api.moonshot.ai/v1"}},
Models: []Model{{ID: "kimi-k2.5", Name: "Kimi K2.5"}, {ID: "kimi-k2-0905-preview", Name: "Kimi K2 0905 Preview"}, {ID: "kimi-k2-thinking", Name: "Kimi K2 Thinking"}}, Models: []Model{{ID: "kimi-k2.5", Name: "Kimi K2.5"}, {ID: "kimi-k2-0905-preview", Name: "Kimi K2 0905 Preview"}, {ID: "kimi-k2-thinking", Name: "Kimi K2 Thinking"}},
}, },
"bailian": {
Key: "bailian", DisplayNameKey: "AIProviderBailian", Sort: 31, DefaultAPIType: "openai-completions", EnvKey: "DASHSCOPE_API_KEY",
APIConfigs: []APIConfig{
{
APIType: "openai-completions", BaseURL: "https://dashscope.aliyuncs.com/compatible-mode/v1",
DiscoverModels: true,
Models: []Model{{ID: "qwen3.7-plus", Name: "qwen3.7-plus"}, {ID: "qwen3.6-plus", Name: "qwen3.6-plus"}, {ID: "qwen3.6-flash", Name: "qwen3.6-flash"}},
},
{
APIType: "openai-responses", BaseURL: "https://dashscope.aliyuncs.com/compatible-mode/v1",
DiscoverModels: true,
Models: []Model{{ID: "qwen3.7-plus", Name: "qwen3.7-plus"}, {ID: "qwen3.6-plus", Name: "qwen3.6-plus"}, {ID: "qwen3.6-flash", Name: "qwen3.6-flash"}},
},
{
APIType: "anthropic-messages", BaseURL: "https://dashscope.aliyuncs.com/apps/anthropic",
DefaultAuthMode: AuthModeBearer,
AuthModes: []string{AuthModeBearer},
Models: []Model{{ID: "qwen3.7-plus", Name: "qwen3.7-plus"}, {ID: "qwen3.6-plus", Name: "qwen3.6-plus"}, {ID: "qwen3.6-flash", Name: "qwen3.6-flash"}},
},
{
APIType: "dashscope-images", BaseURL: "https://dashscope.aliyuncs.com",
Models: []Model{
{ID: "qwen-image-2.0-pro", Name: "qwen-image-2.0-pro"},
{ID: "qwen-image-2.0", Name: "qwen-image-2.0"},
{ID: "wan2.7-image-pro", Name: "wan2.7-image-pro"},
{ID: "wan2.7-image", Name: "wan2.7-image"},
},
},
},
},
"ark": {
Key: "ark", DisplayNameKey: "AIProviderArk", Sort: 36, DefaultAPIType: "openai-completions", EnvKey: "ARK_API_KEY",
APIConfigs: []APIConfig{
{
APIType: "openai-completions", BaseURL: "https://ark.cn-beijing.volces.com/api/v3",
DiscoverModels: true,
Models: []Model{{ID: "doubao-seed-2-0-pro-260215", Name: "doubao-seed-2-0-pro-260215"}, {ID: "doubao-seed-2-0-lite-260215", Name: "doubao-seed-2-0-lite-260215"}},
},
{
APIType: "openai-responses", BaseURL: "https://ark.cn-beijing.volces.com/api/v3",
DiscoverModels: true,
Models: []Model{{ID: "doubao-seed-2-0-pro-260215", Name: "doubao-seed-2-0-pro-260215"}, {ID: "doubao-seed-2-0-lite-260215", Name: "doubao-seed-2-0-lite-260215"}},
},
{
APIType: "openai-images", BaseURL: "https://ark.cn-beijing.volces.com/api/v3",
Models: []Model{
{ID: "doubao-seedream-5-0-260128", Name: "doubao-seedream-5-0-260128"},
{ID: "doubao-seedream-5-0-lite-260128", Name: "doubao-seedream-5-0-lite-260128"},
{ID: "doubao-seedream-4-5-251128", Name: "doubao-seedream-4-5-251128"},
},
},
},
},
} }
func editableAPIConfigs(apiTypes ...string) []APIConfig { func editableAPIConfigs(discoverModels bool, apiTypes ...string) []APIConfig {
configs := make([]APIConfig, 0, len(apiTypes)) configs := make([]APIConfig, 0, len(apiTypes))
for _, apiType := range apiTypes { for _, apiType := range apiTypes {
if apiType == "anthropic-messages" { if apiType == "anthropic-messages" {
@@ -156,7 +235,11 @@ func editableAPIConfigs(apiTypes ...string) []APIConfig {
configs = append(configs, config) configs = append(configs, config)
continue continue
} }
configs = append(configs, APIConfig{APIType: apiType, EditableBaseURL: true}) configs = append(configs, APIConfig{
APIType: apiType,
EditableBaseURL: true,
DiscoverModels: discoverModels && (apiType == "openai-completions" || apiType == "openai-responses"),
})
} }
return configs return configs
} }
@@ -203,12 +286,38 @@ func FindAPIConfig(key, apiType string) (APIConfig, bool) {
} }
for _, config := range meta.APIConfigs { for _, config := range meta.APIConfigs {
if config.APIType == target { if config.APIType == target {
config.AuthModes = append([]string(nil), config.AuthModes...)
config.Models = append([]Model(nil), config.Models...)
return config, true return config, true
} }
} }
return APIConfig{}, false return APIConfig{}, false
} }
func DefaultModels(key, apiType string) []Model {
meta, ok := catalog[key]
if !ok {
return nil
}
target := strings.TrimSpace(apiType)
if target == "" {
target = meta.DefaultAPIType
}
for _, config := range meta.APIConfigs {
if config.APIType != target {
continue
}
if len(config.Models) > 0 {
return append([]Model(nil), config.Models...)
}
if IsImageAPIType(config.APIType) || IsEmbeddingAPIType(config.APIType) {
return nil
}
break
}
return append([]Model(nil), meta.Models...)
}
func ResolveAuthMode(provider, apiType, requested string) (string, error) { func ResolveAuthMode(provider, apiType, requested string) (string, error) {
config, ok := FindAPIConfig(provider, apiType) config, ok := FindAPIConfig(provider, apiType)
if !ok { if !ok {
@@ -264,6 +373,9 @@ func ResolveBaseURL(key, apiType, requested string) (string, error) {
if err != nil || parsed.Scheme == "" || parsed.Host == "" { if err != nil || parsed.Scheme == "" || parsed.Host == "" {
return "", fmt.Errorf("invalid base url") return "", fmt.Errorf("invalid base url")
} }
if key == "custom" && (IsImageAPIType(config.APIType) || IsEmbeddingAPIType(config.APIType)) {
return baseURL, nil
}
parsed.Path = normalizeEndpointPath(config.APIType, parsed.Path) parsed.Path = normalizeEndpointPath(config.APIType, parsed.Path)
parsed.RawQuery = "" parsed.RawQuery = ""
parsed.Fragment = "" parsed.Fragment = ""
@@ -271,6 +383,9 @@ func ResolveBaseURL(key, apiType, requested string) (string, error) {
} }
func normalizeEndpointPath(apiType, value string) string { func normalizeEndpointPath(apiType, value string) string {
if IsImageAPIType(apiType) {
return strings.TrimRight(value, "/")
}
path := strings.TrimRight(value, "/") path := strings.TrimRight(value, "/")
suffixes := []string{} suffixes := []string{}
switch apiType { switch apiType {
@@ -280,6 +395,8 @@ func normalizeEndpointPath(apiType, value string) string {
suffixes = []string{"/responses"} suffixes = []string{"/responses"}
case "anthropic-messages": case "anthropic-messages":
suffixes = []string{"/v1/messages", "/messages"} suffixes = []string{"/v1/messages", "/messages"}
case "openai-embeddings":
suffixes = []string{"/v1/embeddings", "/embeddings"}
} }
for _, suffix := range suffixes { for _, suffix := range suffixes {
if strings.HasSuffix(strings.ToLower(path), suffix) { if strings.HasSuffix(strings.ToLower(path), suffix) {
@@ -289,6 +406,19 @@ func normalizeEndpointPath(apiType, value string) string {
return path return path
} }
func IsEmbeddingAPIType(apiType string) bool {
return apiType == "openai-embeddings"
}
func IsImageAPIType(apiType string) bool {
switch apiType {
case "openai-images", "dashscope-images", "minimax-images", "openrouter-images":
return true
default:
return false
}
}
func EnvKey(key string) string { func EnvKey(key string) string {
meta, ok := catalog[key] meta, ok := catalog[key]
if !ok { if !ok {
@@ -305,6 +435,14 @@ func DisplayName(key string) string {
return meta.DisplayName return meta.DisplayName
} }
func DisplayNameKey(key string) string {
meta, ok := catalog[key]
if !ok {
return ""
}
return meta.DisplayNameKey
}
func NormalizeModelID(provider, modelID string) string { func NormalizeModelID(provider, modelID string) string {
target := strings.TrimLeft(strings.TrimSpace(modelID), "/") target := strings.TrimLeft(strings.TrimSpace(modelID), "/")
for _, prefix := range legacyModelPrefixes[provider] { for _, prefix := range legacyModelPrefixes[provider] {
@@ -324,6 +462,7 @@ var legacyModelPrefixes = map[string][]string{
"custom": {"custom"}, "custom": {"custom"},
"vllm": {"custom"}, "vllm": {"custom"},
"ollama": {"ollama"}, "ollama": {"ollama"},
"llmman": {"llmman"},
"deepseek": {"deepseek"}, "deepseek": {"deepseek"},
"bailian-coding-plan": {"bailian-coding-plan"}, "bailian-coding-plan": {"bailian-coding-plan"},
"ark-coding-plan": {"ark-coding-plan"}, "ark-coding-plan": {"ark-coding-plan"},
@@ -344,6 +483,7 @@ func cloneMeta(meta Meta) Meta {
for index, config := range meta.APIConfigs { for index, config := range meta.APIConfigs {
clone.APIConfigs[index] = config clone.APIConfigs[index] = config
clone.APIConfigs[index].AuthModes = append([]string(nil), config.AuthModes...) clone.APIConfigs[index].AuthModes = append([]string(nil), config.AuthModes...)
clone.APIConfigs[index].Models = append([]Model(nil), config.Models...)
} }
clone.Models = append([]Model(nil), meta.Models...) clone.Models = append([]Model(nil), meta.Models...)
return clone return clone
+12 -2
View File
@@ -4,6 +4,7 @@ import (
"encoding/json" "encoding/json"
"fmt" "fmt"
"net/http" "net/http"
"strconv"
"strings" "strings"
) )
@@ -54,11 +55,20 @@ func buildModelDiscoveryURL(baseURL string) string {
base = normalizeEndpointPath(apiType, base) base = normalizeEndpointPath(apiType, base)
} }
switch { switch {
case strings.HasSuffix(base, "/v1/models"): case strings.HasSuffix(base, "/models"):
return base return base
case strings.HasSuffix(base, "/v1"): case hasAPIVersionSuffix(base):
return base + "/models" return base + "/models"
default: default:
return base + "/v1/models" return base + "/v1/models"
} }
} }
func hasAPIVersionSuffix(value string) bool {
segment := value[strings.LastIndex(value, "/")+1:]
if len(segment) < 2 || segment[0] != 'v' {
return false
}
_, err := strconv.Atoi(segment[1:])
return err == nil
}
+5 -2
View File
@@ -23,6 +23,9 @@ func BuildOpenClawProviderPatch(provider, modelName, apiType, authMode, baseURL,
if _, ok := FindAPIConfig(provider, resolvedAPIType); !ok { if _, ok := FindAPIConfig(provider, resolvedAPIType); !ok {
resolvedAPIType = DefaultAPIType(provider) resolvedAPIType = DefaultAPIType(provider)
} }
if IsImageAPIType(resolvedAPIType) || IsEmbeddingAPIType(resolvedAPIType) {
return nil, fmt.Errorf("api type %s does not support text generation", resolvedAPIType)
}
resolvedAuthMode, err := ResolveAuthMode(provider, resolvedAPIType, authMode) resolvedAuthMode, err := ResolveAuthMode(provider, resolvedAPIType, authMode)
if err != nil { if err != nil {
return nil, err return nil, err
@@ -40,8 +43,8 @@ func BuildOpenClawProviderPatch(provider, modelName, apiType, authMode, baseURL,
providerKey = "moonshot" providerKey = "moonshot"
resolvedAPIType = "openai-completions" resolvedAPIType = "openai-completions"
usesBearer = false usesBearer = false
case "ollama": case "ollama", "llmman":
apiKey = "ollama" apiKey = provider
usesBearer = false usesBearer = false
case "openai", "openrouter", "anthropic": case "openai", "openrouter", "anthropic":
preserveQualifiedModel = strings.Contains(modelName, "/") preserveQualifiedModel = strings.Contains(modelName, "/")
+57 -7
View File
@@ -27,11 +27,14 @@ type verifyErrorResponse struct {
Message string `json:"message"` Message string `json:"message"`
} }
const defaultVerifyTimeout = 30 * time.Second const (
defaultVerifyTimeout = 30 * time.Second
defaultVerifyMaxTokens = 16
)
func SkipVerification(provider string) bool { func SkipVerification(provider string) bool {
switch provider { switch provider {
case "vllm", "ollama", "kimi-coding": case "vllm", "ollama", "llmman", "kimi-coding":
return true return true
default: default:
return false return false
@@ -64,7 +67,10 @@ func VerifyAccount(provider, apiType, authMode, baseURL, apiKey, model string) e
} }
func BuildVerifyRequest(provider, apiType, authMode, baseURL, apiKey, model string) VerifyRequest { func BuildVerifyRequest(provider, apiType, authMode, baseURL, apiKey, model string) VerifyRequest {
baseURL = strings.TrimRight(strings.TrimSpace(baseURL), "/") baseURL = strings.TrimSpace(baseURL)
if provider != "custom" || !IsImageAPIType(apiType) {
baseURL = strings.TrimRight(baseURL, "/")
}
headers := map[string]string{"Content-Type": "application/json"} headers := map[string]string{"Content-Type": "application/json"}
request := VerifyRequest{Method: http.MethodPost, Headers: headers} request := VerifyRequest{Method: http.MethodPost, Headers: headers}
@@ -78,6 +84,32 @@ func BuildVerifyRequest(provider, apiType, authMode, baseURL, apiKey, model stri
} }
switch apiType { switch apiType {
case "openai-embeddings":
request.URL = embeddingVerifyURL(baseURL)
headers["Authorization"] = "Bearer " + apiKey
request.Body = mustJSON(map[string]interface{}{"model": model, "input": "ping"})
case "openai-images":
request.URL = imageVerifyURL(provider, baseURL, "/images/generations")
headers["Authorization"] = "Bearer " + apiKey
request.Body = mustJSON(map[string]interface{}{"model": model, "prompt": "test", "n": 1, "response_format": "url"})
case "dashscope-images":
request.URL = imageVerifyURL(provider, baseURL, "/api/v1/services/aigc/multimodal-generation/generation")
headers["Authorization"] = "Bearer " + apiKey
request.Body = mustJSON(map[string]interface{}{
"model": model,
"input": map[string]interface{}{"messages": []map[string]interface{}{
{"role": "user", "content": []map[string]string{{"text": "test"}}},
}},
"parameters": map[string]interface{}{"n": 1},
})
case "minimax-images":
request.URL = imageVerifyURL(provider, baseURL, "/v1/image_generation")
headers["Authorization"] = "Bearer " + apiKey
request.Body = mustJSON(map[string]interface{}{"model": model, "prompt": "test", "n": 1, "response_format": "url"})
case "openrouter-images":
request.URL = imageVerifyURL(provider, baseURL, "/api/v1/images")
headers["Authorization"] = "Bearer " + apiKey
request.Body = mustJSON(map[string]interface{}{"model": model, "prompt": "test", "n": 1, "response_format": "url"})
case "anthropic-messages": case "anthropic-messages":
request.URL = baseURL + "/v1/messages" request.URL = baseURL + "/v1/messages"
if authMode == AuthModeBearer { if authMode == AuthModeBearer {
@@ -87,25 +119,43 @@ func BuildVerifyRequest(provider, apiType, authMode, baseURL, apiKey, model stri
} }
headers["anthropic-version"] = "2023-06-01" headers["anthropic-version"] = "2023-06-01"
request.Body = mustJSON(map[string]interface{}{ request.Body = mustJSON(map[string]interface{}{
"model": model, "max_tokens": 1, "stream": false, "model": model, "max_tokens": defaultVerifyMaxTokens, "stream": false,
"messages": []map[string]interface{}{{"role": "user", "content": []map[string]string{{"type": "text", "text": "test"}}}}, "messages": []map[string]interface{}{{"role": "user", "content": []map[string]string{{"type": "text", "text": "test"}}}},
}) })
case "openai-responses": case "openai-responses":
request.URL = baseURL + "/responses" request.URL = baseURL + "/responses"
headers["Authorization"] = "Bearer " + apiKey headers["Authorization"] = "Bearer " + apiKey
request.Body = mustJSON(map[string]interface{}{"model": model, "input": "test", "max_output_tokens": 1, "stream": false}) request.Body = mustJSON(map[string]interface{}{"model": model, "input": "test", "max_output_tokens": defaultVerifyMaxTokens, "stream": false})
default: default:
request.URL = baseURL + "/chat/completions" request.URL = baseURL + "/chat/completions"
if provider != "ollama" || strings.TrimSpace(apiKey) != "" { if (provider != "ollama" && provider != "llmman") || strings.TrimSpace(apiKey) != "" {
headers["Authorization"] = "Bearer " + apiKey headers["Authorization"] = "Bearer " + apiKey
} }
request.Body = mustJSON(map[string]interface{}{ request.Body = mustJSON(map[string]interface{}{
"model": model, "messages": []map[string]string{{"role": "user", "content": "test"}}, "max_tokens": 1, "stream": false, "model": model, "messages": []map[string]string{{"role": "user", "content": "test"}}, "max_tokens": defaultVerifyMaxTokens, "stream": false,
}) })
} }
return request return request
} }
func embeddingVerifyURL(baseURL string) string {
lowerBaseURL := strings.ToLower(baseURL)
if strings.HasSuffix(lowerBaseURL, "/embeddings") {
return baseURL
}
if strings.HasSuffix(lowerBaseURL, "/v1") {
return baseURL + "/embeddings"
}
return baseURL + "/v1/embeddings"
}
func imageVerifyURL(provider, baseURL, endpoint string) string {
if provider == "custom" || strings.HasSuffix(strings.ToLower(baseURL), endpoint) {
return baseURL
}
return baseURL + endpoint
}
func verifyHTTPError(statusCode int, body []byte) string { func verifyHTTPError(statusCode int, body []byte) string {
message := strings.TrimSpace(string(body)) message := strings.TrimSpace(string(body))
var payload verifyErrorResponse var payload verifyErrorResponse
+3 -2
View File
@@ -17,7 +17,7 @@ type IAgentAccountRepo interface {
Save(account *model.AgentAccount) error Save(account *model.AgentAccount) error
DeleteByID(id uint) error DeleteByID(id uint) error
List(opts ...DBOption) ([]model.AgentAccount, error) List(opts ...DBOption) ([]model.AgentAccount, error)
CountByProviders(providers []string) (map[string]int64, error) CountTextByProviders(providers []string) (map[string]int64, error)
} }
func NewIAgentAccountRepo() IAgentAccountRepo { func NewIAgentAccountRepo() IAgentAccountRepo {
@@ -67,7 +67,7 @@ func (a AgentAccountRepo) List(opts ...DBOption) ([]model.AgentAccount, error) {
return accounts, nil return accounts, nil
} }
func (a AgentAccountRepo) CountByProviders(providers []string) (map[string]int64, error) { func (a AgentAccountRepo) CountTextByProviders(providers []string) (map[string]int64, error) {
normalizedProviders := normalizeProviders(providers) normalizedProviders := normalizeProviders(providers)
counts := make(map[string]int64, len(normalizedProviders)) counts := make(map[string]int64, len(normalizedProviders))
for _, provider := range normalizedProviders { for _, provider := range normalizedProviders {
@@ -86,6 +86,7 @@ func (a AgentAccountRepo) CountByProviders(providers []string) (map[string]int64
Model(&model.AgentAccount{}). Model(&model.AgentAccount{}).
Select("provider, COUNT(*) as count"). Select("provider, COUNT(*) as count").
Where("provider IN ?", normalizedProviders). Where("provider IN ?", normalizedProviders).
Scopes(WithTextAPIType()).
Group("provider"). Group("provider").
Scan(&rows).Error; err != nil { Scan(&rows).Error; err != nil {
return nil, err return nil, err
+217 -9
View File
@@ -1,20 +1,30 @@
package repo package repo
import ( import (
"encoding/base64"
"encoding/json" "encoding/json"
"errors"
"fmt"
"strconv"
"strings" "strings"
"time"
"github.com/1Panel-dev/1Panel/agent/app/model" "github.com/1Panel-dev/1Panel/agent/app/model"
"github.com/1Panel-dev/1Panel/agent/constant" "github.com/1Panel-dev/1Panel/agent/constant"
"github.com/1Panel-dev/1Panel/agent/global" "github.com/1Panel-dev/1Panel/agent/global"
"github.com/google/uuid"
"google.golang.org/genproto/googleapis/type/date" "google.golang.org/genproto/googleapis/type/date"
"gorm.io/gorm" "gorm.io/gorm"
"strconv" "gorm.io/gorm/clause"
"time"
) )
type AlertRepo struct{} type AlertRepo struct{}
var (
ErrAlertConfigRevisionConflict = errors.New("alert config revision conflict")
ErrAlertConfigRevisionRequired = errors.New("alert config revision is required")
)
type IAlertRepo interface { type IAlertRepo interface {
WithByType(alertType string) DBOption WithByType(alertType string) DBOption
WithByStatusIn(status []string) DBOption WithByStatusIn(status []string) DBOption
@@ -24,6 +34,7 @@ type IAlertRepo interface {
WithByCreateAt(date *date.Date) DBOption WithByCreateAt(date *date.Date) DBOption
WithByLicenseId(licenseId string) DBOption WithByLicenseId(licenseId string) DBOption
WithByRecordId(recordId uint) DBOption WithByRecordId(recordId uint) DBOption
WithByDeliveryLogID(logID uint) DBOption
WithByAlertMethodContainsConfigID(id uint) DBOption WithByAlertMethodContainsConfigID(id uint) DBOption
WithByMethodConfigIDs(ids []uint) DBOption WithByMethodConfigIDs(ids []uint) DBOption
@@ -45,6 +56,8 @@ type IAlertRepo interface {
CleanAlertLogs() error CleanAlertLogs() error
CreateAlertTask(alertTaskBase *model.AlertTask) error CreateAlertTask(alertTaskBase *model.AlertTask) error
CreatePendingAlertTask(logID, alertID uint, alertTask *model.AlertTask) (bool, error)
FinalizePendingAlertTask(logID uint, succeeded bool, message string, fallback *model.AlertTask) (bool, error)
DeleteAlertTask(opts ...DBOption) error DeleteAlertTask(opts ...DBOption) error
GetAlertTask(opts ...DBOption) (model.AlertTask, error) GetAlertTask(opts ...DBOption) (model.AlertTask, error)
LoadTaskCount(alertType string, project string, method string) (uint, uint, error) LoadTaskCount(alertType string, project string, method string) (uint, uint, error)
@@ -55,6 +68,7 @@ type IAlertRepo interface {
GetConfigById(id uint) (model.AlertConfig, error) GetConfigById(id uint) (model.AlertConfig, error)
AlertConfigList(opts ...DBOption) ([]model.AlertConfig, error) AlertConfigList(opts ...DBOption) ([]model.AlertConfig, error)
UpdateAlertConfig(maps map[string]interface{}, opts ...DBOption) error UpdateAlertConfig(maps map[string]interface{}, opts ...DBOption) error
UpdateAlertConfigWithRevision(maps map[string]interface{}, revision *time.Time, opts ...DBOption) error
CreateAlertConfig(config *model.AlertConfig) error CreateAlertConfig(config *model.AlertConfig) error
DeleteAlertConfig(opts ...DBOption) error DeleteAlertConfig(opts ...DBOption) error
@@ -223,13 +237,78 @@ func (a *AlertRepo) DeleteLog(opts ...DBOption) error {
} }
func (a *AlertRepo) CleanAlertLogs() error { func (a *AlertRepo) CleanAlertLogs() error {
return global.AlertDB.Where("1 = 1").Delete(&model.AlertLog{}).Error return global.AlertDB.Where("status <> ?", constant.AlertPushing).Delete(&model.AlertLog{}).Error
} }
func (a *AlertRepo) CreateAlertTask(alertTaskBase *model.AlertTask) error { func (a *AlertRepo) CreateAlertTask(alertTaskBase *model.AlertTask) error {
return global.AlertDB.Model(&model.AlertTask{}).Create(&alertTaskBase).Error return global.AlertDB.Model(&model.AlertTask{}).Create(&alertTaskBase).Error
} }
func (a *AlertRepo) CreatePendingAlertTask(logID, alertID uint, alertTask *model.AlertTask) (bool, error) {
if alertTask == nil {
return false, fmt.Errorf("pending alert task is required")
}
created := false
err := global.AlertDB.Transaction(func(tx *gorm.DB) error {
var log model.AlertLog
if err := tx.Where("id = ? AND status = ?", logID, constant.AlertPushing).First(&log).Error; err != nil {
return err
}
if log.AlertId != alertID || log.Type != alertTask.Type || log.Method != alertTask.Method {
return fmt.Errorf("pending alert task does not match delivery log %d", logID)
}
alertTask.DeliveryLogID = &logID
result := tx.Clauses(clause.OnConflict{
Columns: []clause.Column{{Name: "delivery_log_id"}},
DoNothing: true,
}).Create(alertTask)
if result.Error != nil {
return result.Error
}
created = result.RowsAffected > 0
return nil
})
return created, err
}
func (a *AlertRepo) FinalizePendingAlertTask(logID uint, succeeded bool, message string, fallback *model.AlertTask) (bool, error) {
finalized := false
err := global.AlertDB.Transaction(func(tx *gorm.DB) error {
status := constant.AlertError
if succeeded {
status = constant.AlertSuccess
message = ""
}
result := tx.Model(&model.AlertLog{}).
Where("id = ? AND status = ?", logID, constant.AlertPushing).
Updates(map[string]interface{}{"status": status, "message": message})
if result.Error != nil {
return result.Error
}
if result.RowsAffected == 0 {
return nil
}
finalized = true
if !succeeded {
return tx.Where("delivery_log_id = ?", logID).Delete(&model.AlertTask{}).Error
}
var count int64
if err := tx.Model(&model.AlertTask{}).Where("delivery_log_id = ?", logID).Count(&count).Error; err != nil {
return err
}
if count > 0 {
return nil
}
if fallback == nil {
return fmt.Errorf("pending alert task metadata is unavailable for delivery log %d", logID)
}
fallback.DeliveryLogID = &logID
return tx.Create(fallback).Error
})
return finalized, err
}
func (a *AlertRepo) DeleteAlertTask(opts ...DBOption) error { func (a *AlertRepo) DeleteAlertTask(opts ...DBOption) error {
db, _ := getAlertDB(opts...) db, _ := getAlertDB(opts...)
return db.Delete(&model.AlertTask{}).Error return db.Delete(&model.AlertTask{}).Error
@@ -310,7 +389,23 @@ func (a *AlertRepo) UpdateAlertConfig(maps map[string]interface{}, opts ...DBOpt
return db.Model(&model.AlertConfig{}).Updates(maps).Error return db.Model(&model.AlertConfig{}).Updates(maps).Error
} }
func (a *AlertRepo) UpdateAlertConfigWithRevision(maps map[string]interface{}, revision *time.Time, opts ...DBOption) error {
if revision == nil {
return a.UpdateAlertConfig(maps, opts...)
}
db, _ := getAlertDB(opts...)
result := db.Model(&model.AlertConfig{}).Where("updated_at = ?", *revision).Updates(maps)
if result.Error != nil {
return result.Error
}
if result.RowsAffected == 0 {
return ErrAlertConfigRevisionConflict
}
return nil
}
func (a *AlertRepo) CreateAlertConfig(config *model.AlertConfig) error { func (a *AlertRepo) CreateAlertConfig(config *model.AlertConfig) error {
ensureAlertConfigUID(config)
return global.AlertDB.Model(&model.AlertConfig{}).Create(config).Error return global.AlertDB.Model(&model.AlertConfig{}).Create(config).Error
} }
@@ -338,6 +433,12 @@ func (a *AlertRepo) WithByTypeNotIn(types []string) DBOption {
} }
} }
func (a *AlertRepo) WithByDeliveryLogID(logID uint) DBOption {
return func(g *gorm.DB) *gorm.DB {
return g.Where("delivery_log_id = ?", logID)
}
}
func (a *AlertRepo) PageAlertConfig(page, size int, opts ...DBOption) (int64, []model.AlertConfig, error) { func (a *AlertRepo) PageAlertConfig(page, size int, opts ...DBOption) (int64, []model.AlertConfig, error) {
var configs []model.AlertConfig var configs []model.AlertConfig
db := global.AlertDB.Model(&model.AlertConfig{}) db := global.AlertDB.Model(&model.AlertConfig{})
@@ -378,26 +479,44 @@ func (a *AlertRepo) SyncAll(data []model.AlertConfig) error {
return err return err
} }
oldConfigMap := make(map[string]uint) oldConfigMap := make(map[string]model.AlertConfig)
oldConfigByUID := make(map[string]model.AlertConfig)
oldConfigByType := make(map[string][]model.AlertConfig) oldConfigByType := make(map[string][]model.AlertConfig)
oldConfigByKey := make(map[string][]model.AlertConfig) oldConfigByKey := make(map[string][]model.AlertConfig)
consumedConfigIDs := make(map[uint]struct{}) consumedConfigIDs := make(map[uint]struct{})
for _, item := range oldConfigs { for _, item := range oldConfigs {
if strings.TrimSpace(item.UID) != "" {
oldConfigByUID[item.UID] = item
}
if singletonTypes[item.Type] { if singletonTypes[item.Type] {
oldConfigMap[item.Type] = item.ID oldConfigMap[item.Type] = item
continue continue
} }
oldConfigByType[item.Type] = append(oldConfigByType[item.Type], item) oldConfigByType[item.Type] = append(oldConfigByType[item.Type], item)
oldConfigByKey[alertConfigSyncKey(item)] = append(oldConfigByKey[alertConfigSyncKey(item)], item) oldConfigByKey[alertConfigSyncKey(item)] = append(oldConfigByKey[alertConfigSyncKey(item)], item)
} }
for _, item := range data { for _, item := range data {
if uid := strings.TrimSpace(item.UID); uid != "" {
if matched, ok := oldConfigByUID[uid]; ok && matched.Type != item.Type {
tx.Rollback()
return fmt.Errorf("alert config UID %q belongs to type %q, not %q", uid, matched.Type, item.Type)
}
}
if singletonTypes[item.Type] { if singletonTypes[item.Type] {
if val, ok := oldConfigMap[item.Type]; ok { if matched, ok := oldConfigMap[item.Type]; ok {
item.ID = val if err := inheritAlertConfigSyncState(&item, matched); err != nil {
tx.Rollback()
return err
}
delete(oldConfigMap, item.Type) delete(oldConfigMap, item.Type)
consumedConfigIDs[item.ID] = struct{}{} consumedConfigIDs[item.ID] = struct{}{}
} else { } else {
item.ID = 0 item.ID = 0
ensureAlertConfigUID(&item)
if err := validateAlertConfigSyncSecret(&item); err != nil {
tx.Rollback()
return err
}
} }
if item.ID == 0 { if item.ID == 0 {
if err := tx.Create(&item).Error; err != nil { if err := tx.Create(&item).Error; err != nil {
@@ -411,9 +530,31 @@ func (a *AlertRepo) SyncAll(data []model.AlertConfig) error {
continue continue
} }
if strings.TrimSpace(item.UID) != "" {
if matched, ok := oldConfigByUID[item.UID]; ok {
delete(oldConfigByUID, item.UID)
if err := inheritAlertConfigSyncState(&item, matched); err != nil {
tx.Rollback()
return err
}
consumedConfigIDs[item.ID] = struct{}{}
if err := tx.Save(&item).Error; err != nil {
tx.Rollback()
return err
}
deleteAlertConfigByID(oldConfigByType, matched.ID)
deleteAlertConfigByID(oldConfigByKey, matched.ID)
continue
}
}
key := alertConfigSyncKey(item) key := alertConfigSyncKey(item)
if matched, ok := popAlertConfigByKey(oldConfigByKey, key); ok { if matched, ok := popAlertConfigByKey(oldConfigByKey, key); ok {
item.ID = matched.ID delete(oldConfigByUID, matched.UID)
if err := inheritAlertConfigSyncState(&item, matched); err != nil {
tx.Rollback()
return err
}
consumedConfigIDs[item.ID] = struct{}{} consumedConfigIDs[item.ID] = struct{}{}
if err := tx.Save(&item).Error; err != nil { if err := tx.Save(&item).Error; err != nil {
tx.Rollback() tx.Rollback()
@@ -424,7 +565,12 @@ func (a *AlertRepo) SyncAll(data []model.AlertConfig) error {
} }
if matched, ok := popUnusedAlertConfigByType(oldConfigByType, usedConfigIDs, item.Type); ok { if matched, ok := popUnusedAlertConfigByType(oldConfigByType, usedConfigIDs, item.Type); ok {
item.ID = matched.ID delete(oldConfigByUID, matched.UID)
deleteAlertConfigByID(oldConfigByKey, matched.ID)
if err := inheritAlertConfigSyncState(&item, matched); err != nil {
tx.Rollback()
return err
}
consumedConfigIDs[item.ID] = struct{}{} consumedConfigIDs[item.ID] = struct{}{}
if err := tx.Save(&item).Error; err != nil { if err := tx.Save(&item).Error; err != nil {
tx.Rollback() tx.Rollback()
@@ -434,6 +580,11 @@ func (a *AlertRepo) SyncAll(data []model.AlertConfig) error {
} }
item.ID = 0 item.ID = 0
ensureAlertConfigUID(&item)
if err := validateAlertConfigSyncSecret(&item); err != nil {
tx.Rollback()
return err
}
if err := tx.Create(&item).Error; err != nil { if err := tx.Create(&item).Error; err != nil {
tx.Rollback() tx.Rollback()
return err return err
@@ -458,6 +609,63 @@ func (a *AlertRepo) SyncAll(data []model.AlertConfig) error {
return nil return nil
} }
func ensureAlertConfigUID(config *model.AlertConfig) {
if config != nil && strings.TrimSpace(config.UID) == "" {
config.UID = uuid.NewString()
}
}
func inheritAlertConfigSyncState(incoming *model.AlertConfig, existing model.AlertConfig) error {
if incoming.Type != existing.Type {
return fmt.Errorf("alert config UID %q belongs to type %q, not %q", incoming.UID, existing.Type, incoming.Type)
}
preserveExistingCustom := incoming.Type == constant.Custom &&
existing.Status == constant.AlertDisable &&
incoming.Title == existing.Title &&
incoming.Status == existing.Status &&
incoming.Config == existing.Config &&
(incoming.SecretConfig == "" || incoming.SecretConfig == existing.SecretConfig)
incoming.ID = existing.ID
if strings.TrimSpace(incoming.UID) == "" {
incoming.UID = existing.UID
}
if incoming.Type == constant.Custom && incoming.SecretConfig == "" {
incoming.SecretConfig = existing.SecretConfig
}
if preserveExistingCustom {
return nil
}
return validateAlertConfigSyncSecret(incoming)
}
func validateAlertConfigSyncSecret(incoming *model.AlertConfig) error {
if incoming.Type != constant.Custom {
incoming.SecretConfig = ""
return nil
}
if strings.TrimSpace(incoming.SecretConfig) == "" {
return fmt.Errorf("custom webhook sync secret is missing")
}
var version struct {
SchemaVersion int `json:"schemaVersion"`
}
if err := json.Unmarshal([]byte(incoming.Config), &version); err != nil || version.SchemaVersion != 1 {
return fmt.Errorf("custom webhook sync config must use schemaVersion 1")
}
secret := incoming.SecretConfig
for _, prefix := range []string{"core:v1:", "agent:v1:"} {
if !strings.HasPrefix(secret, prefix) {
continue
}
ciphertext, err := base64.StdEncoding.DecodeString(strings.TrimPrefix(secret, prefix))
if err != nil || len(ciphertext) < 32 || len(ciphertext)%16 != 0 {
return fmt.Errorf("custom webhook sync secret envelope is invalid")
}
return nil
}
return fmt.Errorf("custom webhook sync secret must use a versioned envelope")
}
func loadUsedAlertConfigIDs(tx *gorm.DB) (map[uint]struct{}, error) { func loadUsedAlertConfigIDs(tx *gorm.DB) (map[uint]struct{}, error) {
var alerts []model.Alert var alerts []model.Alert
if err := tx.Select("method").Find(&alerts).Error; err != nil { if err := tx.Select("method").Find(&alerts).Error; err != nil {
+18
View File
@@ -49,6 +49,12 @@ func WithByName(name string) DBOption {
} }
} }
func WithByPath(path string) DBOption {
return func(g *gorm.DB) *gorm.DB {
return g.Where("path = ?", path)
}
}
func WithByAddr(addr string) DBOption { func WithByAddr(addr string) DBOption {
return func(g *gorm.DB) *gorm.DB { return func(g *gorm.DB) *gorm.DB {
return g.Where("addr = ?", addr) return g.Where("addr = ?", addr)
@@ -94,6 +100,18 @@ func WithByProvider(provider string) DBOption {
} }
} }
func WithByAPIType(apiType string) DBOption {
return func(g *gorm.DB) *gorm.DB {
return g.Where("api_type = ?", apiType)
}
}
func WithTextAPIType() DBOption {
return func(g *gorm.DB) *gorm.DB {
return g.Where("api_type NOT LIKE ? AND api_type <> ?", "%-images", "openai-embeddings")
}
}
func WithByModel(model string) DBOption { func WithByModel(model string) DBOption {
return func(g *gorm.DB) *gorm.DB { return func(g *gorm.DB) *gorm.DB {
if len(model) == 0 { if len(model) == 0 {
-72
View File
@@ -22,11 +22,6 @@ type IHostRepo interface {
WithByPort(port uint) DBOption WithByPort(port uint) DBOption
WithByUser(user string) DBOption WithByUser(user string) DBOption
GetFirewallRecord(opts ...DBOption) (model.Firewall, error)
ListFirewallRecord(opts ...DBOption) ([]model.Firewall, error)
SaveFirewallRecord(firewall *model.Firewall) error
DeleteFirewallRecordByID(id uint) error
SyncCert(data []model.RootCert) error SyncCert(data []model.RootCert) error
GetCert(opts ...DBOption) (model.RootCert, error) GetCert(opts ...DBOption) (model.RootCert, error)
PageCert(limit, offset int, opts ...DBOption) (int64, []model.RootCert, error) PageCert(limit, offset int, opts ...DBOption) (int64, []model.RootCert, error)
@@ -34,8 +29,6 @@ type IHostRepo interface {
SaveCert(cert *model.RootCert) error SaveCert(cert *model.RootCert) error
UpdateCert(id uint, vars map[string]interface{}) error UpdateCert(id uint, vars map[string]interface{}) error
DeleteCert(opts ...DBOption) error DeleteCert(opts ...DBOption) error
WithByChain(chain string) DBOption
} }
func NewIHostRepo() IHostRepo { func NewIHostRepo() IHostRepo {
@@ -116,65 +109,6 @@ func (h *HostRepo) Delete(opts ...DBOption) error {
return db.Delete(&model.Host{}).Error return db.Delete(&model.Host{}).Error
} }
func (h *HostRepo) GetFirewallRecord(opts ...DBOption) (model.Firewall, error) {
var firewall model.Firewall
db := global.DB
for _, opt := range opts {
db = opt(db)
}
err := db.First(&firewall).Error
return firewall, err
}
func (h *HostRepo) ListFirewallRecord(opts ...DBOption) ([]model.Firewall, error) {
var firewalls []model.Firewall
db := global.DB
for _, opt := range opts {
db = opt(db)
}
if err := global.DB.Find(&firewalls).Error; err != nil {
return firewalls, nil
}
return firewalls, nil
}
func (h *HostRepo) SaveFirewallRecord(firewall *model.Firewall) error {
if firewall.ID != 0 {
return global.DB.Save(firewall).Error
}
var data model.Firewall
switch firewall.Type {
case "port":
_ = global.DB.Where("type = ? AND dst_port = ? AND protocol = ? AND src_ip = ? AND strategy = ?", "port",
firewall.DstPort,
firewall.Protocol,
firewall.SrcIP,
firewall.Strategy,
).First(&data).Error
case "ip":
_ = global.DB.Where("type = ? AND src_ip = ? AND strategy = ?", "address", firewall.SrcIP, firewall.Strategy).First(&data)
default:
_ = global.DB.Where("type = ? AND chain = ? AND src_port = ? AND dst_port = ? AND protocol = ? AND src_ip = ? AND dst_ip = ? AND strategy = ?",
firewall.Type,
firewall.Chain,
firewall.SrcPort,
firewall.DstPort,
firewall.Protocol,
firewall.SrcIP,
firewall.DstIP,
firewall.Strategy,
).First(&data).Error
}
if data.ID != 0 {
firewall.ID = data.ID
}
return global.DB.Save(firewall).Error
}
func (h *HostRepo) DeleteFirewallRecordByID(id uint) error {
return global.DB.Where("id = ?", id).Delete(&model.Firewall{}).Error
}
func (u *HostRepo) GetCert(opts ...DBOption) (model.RootCert, error) { func (u *HostRepo) GetCert(opts ...DBOption) (model.RootCert, error) {
var cert model.RootCert var cert model.RootCert
db := global.DB db := global.DB
@@ -253,9 +187,3 @@ func (u *HostRepo) SyncCert(data []model.RootCert) error {
tx.Commit() tx.Commit()
return nil return nil
} }
func (u *HostRepo) WithByChain(chain string) DBOption {
return func(g *gorm.DB) *gorm.DB {
return g.Where("chain = ?", chain)
}
}
+106 -13
View File
@@ -1,6 +1,8 @@
package repo package repo
import ( import (
"fmt"
"strings"
"time" "time"
"github.com/1Panel-dev/1Panel/agent/app/model" "github.com/1Panel-dev/1Panel/agent/app/model"
@@ -10,11 +12,25 @@ import (
type MonitorRepo struct{} type MonitorRepo struct{}
type GPUHistoryPoint struct {
model.MonitorGPU
Bucket int64
PowerPercent *float64
MemoryPercent *float64
ProcessCount *float64
}
type IMonitorRepo interface { type IMonitorRepo interface {
CleanHost() error
CleanGPU() error
GetBase(opts ...DBOption) ([]model.MonitorBase, error) GetBase(opts ...DBOption) ([]model.MonitorBase, error)
GetGPU(opts ...DBOption) ([]model.MonitorGPU, error) CountGPU(opts ...DBOption) (int64, error)
GetGPUHistory(start time.Time, bucketSeconds int64, aggregation string, opts ...DBOption) ([]GPUHistoryPoint, error)
GetGPUDevices() ([]model.MonitorGPU, error)
GetIO(opts ...DBOption) ([]model.MonitorIO, error) GetIO(opts ...DBOption) ([]model.MonitorIO, error)
GetNetwork(opts ...DBOption) ([]model.MonitorNetwork, error) GetNetwork(opts ...DBOption) ([]model.MonitorNetwork, error)
GetIONames() ([]string, error)
GetNetworkNames() ([]string, error)
CreateMonitorBase(model model.MonitorBase) error CreateMonitorBase(model model.MonitorBase) error
BatchCreateMonitorGPU(list []model.MonitorGPU) error BatchCreateMonitorGPU(list []model.MonitorGPU) error
@@ -25,13 +41,26 @@ type IMonitorRepo interface {
DelMonitorIO(timeForDelete time.Time) error DelMonitorIO(timeForDelete time.Time) error
DelMonitorNet(timeForDelete time.Time) error DelMonitorNet(timeForDelete time.Time) error
WithByProductName(name string) DBOption WithByGPUDevice(deviceID, name string, legacy bool) DBOption
} }
func NewIMonitorRepo() IMonitorRepo { func NewIMonitorRepo() IMonitorRepo {
return &MonitorRepo{} return &MonitorRepo{}
} }
func (s *MonitorRepo) CleanHost() error {
for _, item := range []interface{}{&model.MonitorBase{}, &model.MonitorIO{}, &model.MonitorNetwork{}} {
if err := global.MonitorDB.Where("1 = 1").Delete(item).Error; err != nil {
return err
}
}
return nil
}
func (s *MonitorRepo) CleanGPU() error {
return global.GPUMonitorDB.Where("1 = 1").Delete(&model.MonitorGPU{}).Error
}
func (u *MonitorRepo) GetBase(opts ...DBOption) ([]model.MonitorBase, error) { func (u *MonitorRepo) GetBase(opts ...DBOption) ([]model.MonitorBase, error) {
var data []model.MonitorBase var data []model.MonitorBase
db := global.MonitorDB db := global.MonitorDB
@@ -59,20 +88,25 @@ func (u *MonitorRepo) GetNetwork(opts ...DBOption) ([]model.MonitorNetwork, erro
err := db.Find(&data).Error err := db.Find(&data).Error
return data, err return data, err
} }
func (u *MonitorRepo) GetGPU(opts ...DBOption) ([]model.MonitorGPU, error) { func (u *MonitorRepo) GetIONames() ([]string, error) {
var data []model.MonitorGPU var names []string
db := global.GPUMonitorDB err := global.MonitorDB.Model(&model.MonitorIO{}).Distinct().Pluck("name", &names).Error
for _, opt := range opts { return names, err
db = opt(db) }
}
err := db.Find(&data).Error func (u *MonitorRepo) GetNetworkNames() ([]string, error) {
return data, err var names []string
err := global.MonitorDB.Model(&model.MonitorNetwork{}).Distinct().Pluck("name", &names).Error
return names, err
} }
func (u *MonitorRepo) CreateMonitorBase(model model.MonitorBase) error { func (u *MonitorRepo) CreateMonitorBase(model model.MonitorBase) error {
return global.MonitorDB.Create(&model).Error return global.MonitorDB.Create(&model).Error
} }
func (s *MonitorRepo) BatchCreateMonitorGPU(list []model.MonitorGPU) error { func (s *MonitorRepo) BatchCreateMonitorGPU(list []model.MonitorGPU) error {
if len(list) == 0 {
return nil
}
return global.GPUMonitorDB.CreateInBatches(&list, len(list)).Error return global.GPUMonitorDB.CreateInBatches(&list, len(list)).Error
} }
func (u *MonitorRepo) BatchCreateMonitorIO(ioList []model.MonitorIO) error { func (u *MonitorRepo) BatchCreateMonitorIO(ioList []model.MonitorIO) error {
@@ -94,8 +128,67 @@ func (s *MonitorRepo) DelMonitorGPU(timeForDelete time.Time) error {
return global.GPUMonitorDB.Where("created_at < ?", timeForDelete).Delete(&model.MonitorGPU{}).Error return global.GPUMonitorDB.Where("created_at < ?", timeForDelete).Delete(&model.MonitorGPU{}).Error
} }
func (s *MonitorRepo) WithByProductName(name string) DBOption { func (u *MonitorRepo) GetGPUDevices() ([]model.MonitorGPU, error) {
return func(g *gorm.DB) *gorm.DB { var data []model.MonitorGPU
return g.Where("product_name = ?", name) err := global.GPUMonitorDB.Model(&model.MonitorGPU{}).Select("device_id, product_name, device_type").Group("device_id, product_name, device_type").Order("product_name, device_id").Find(&data).Error
return data, err
}
func (u *MonitorRepo) WithByGPUDevice(deviceID, name string, legacy bool) DBOption {
return func(db *gorm.DB) *gorm.DB {
if deviceID != "" {
return db.Where("device_id = ?", deviceID)
}
db = db.Where("product_name = ?", name)
if legacy {
db = db.Where("device_id IS NULL OR device_id = ''")
}
return db
} }
} }
func (u *MonitorRepo) CountGPU(opts ...DBOption) (int64, error) {
db := global.GPUMonitorDB.Model(&model.MonitorGPU{})
for _, opt := range opts {
db = opt(db)
}
var count int64
err := db.Count(&count).Error
return count, err
}
func (u *MonitorRepo) GetGPUHistory(start time.Time, bucketSeconds int64, aggregation string, opts ...DBOption) ([]GPUHistoryPoint, error) {
db := global.GPUMonitorDB.Model(&model.MonitorGPU{})
for _, opt := range opts {
db = opt(db)
}
expressions := []string{
"CASE WHEN max_power_limit > 0 THEN 100.0 * power_draw / max_power_limit END",
"CASE WHEN mem_total > 0 AND mem_used IS NOT NULL THEN 100.0 * mem_used / mem_total ELSE memory_util END",
"CASE WHEN (process_status = 'ok' OR process_status IS NULL OR process_status = '') AND json_valid(processes) THEN CASE WHEN json_type(processes) = 'array' THEN json_array_length(processes) END END",
}
aliases := []string{"power_percent", "memory_percent", "process_count"}
columns := []string{"*"}
if bucketSeconds > 0 {
operation := "AVG"
if aggregation == "max" {
operation = "MAX"
}
columns = []string{fmt.Sprintf("(CAST(strftime('%%s', created_at) AS INTEGER) - %d) / %d AS bucket", start.Unix(), bucketSeconds)}
for _, column := range []string{"memory_activity", "encoder_util", "decoder_util", "jpeg_util", "ofa_util", "media_util", "compute_util", "copy_util", "hotspot_temperature", "fan_rpm", "ai_cpu_util", "ctrl_cpu_util", "ddr_used", "ddr_total", "hbm_used", "hbm_total", "ddr_bandwidth", "hbm_bandwidth", "memory_bandwidth", "media_frequency", "hugepages_used", "hugepages_total", "gpu_util", "temperature", "memory_temperature", "power_draw", "max_power_limit", "mem_used", "mem_total", "frequency", "memory_frequency", "fan_speed"} {
columns = append(columns, operation+"("+column+") AS "+column)
}
for i := range expressions {
expressions[i] = operation + "(" + expressions[i] + ")"
}
db = db.Group("bucket").Order("bucket ASC")
} else {
db = db.Order("created_at ASC, id ASC")
}
for i, expression := range expressions {
columns = append(columns, expression+" AS "+aliases[i])
}
var data []GPUHistoryPoint
err := db.Select(strings.Join(columns, ", ")).Scan(&data).Error
return data, err
}
+78
View File
@@ -0,0 +1,78 @@
package repo
import (
"errors"
"fmt"
"strings"
"time"
"github.com/1Panel-dev/1Panel/agent/app/model"
"github.com/1Panel-dev/1Panel/agent/global"
"gorm.io/gorm"
)
type VLLMMonitorRepo struct{}
type VLLMHistoryPoint struct {
model.MonitorVLLM
Bucket int64
HistogramSamples string
}
func (r *VLLMMonitorRepo) Create(point *model.MonitorVLLM) error {
return global.VLLMMonitorDB.Create(point).Error
}
func (r *VLLMMonitorRepo) Latest(id uint) (model.MonitorVLLM, error) {
var point model.MonitorVLLM
db := global.VLLMMonitorDB.Where("app_install_id = ?", id)
err := db.Order("created_at DESC, id DESC").First(&point).Error
if errors.Is(err, gorm.ErrRecordNotFound) {
return point, nil
}
return point, err
}
func (r *VLLMMonitorRepo) CleanTarget(id uint) error {
return global.VLLMMonitorDB.Where("app_install_id = ?", id).Delete(&model.MonitorVLLM{}).Error
}
func (r *VLLMMonitorRepo) DeleteBefore(before time.Time) error {
return global.VLLMMonitorDB.Where("created_at < ?", before).Delete(&model.MonitorVLLM{}).Error
}
func (r *VLLMMonitorRepo) Count(id uint, start, end time.Time) (int64, error) {
var count int64
db := global.VLLMMonitorDB.Model(&model.MonitorVLLM{}).Where("app_install_id = ? AND created_at >= ? AND created_at <= ?", id, start, end)
err := db.Count(&count).Error
return count, err
}
func (r *VLLMMonitorRepo) History(id uint, start, end time.Time, seconds int64, aggregation string) ([]VLLMHistoryPoint, error) {
db := global.VLLMMonitorDB.Model(&model.MonitorVLLM{}).Where("app_install_id = ? AND created_at >= ? AND created_at <= ?", id, start, end)
metrics := []string{"running", "waiting", "cache_usage", "prompt_throughput", "generation_throughput", "request_throughput", "time_to_first_token", "time_per_output_token", "request_latency", "prefill_time", "decode_time", "time_to_first_token_p50", "time_to_first_token_p90", "time_to_first_token_p95", "time_to_first_token_p99", "time_per_output_token_p50", "time_per_output_token_p90", "time_per_output_token_p95", "time_per_output_token_p99", "request_latency_p50", "request_latency_p90", "request_latency_p95", "request_latency_p99"}
var columns []string
if seconds > 0 {
operation := "AVG"
if aggregation == "max" {
operation = "MAX"
}
columns = []string{fmt.Sprintf("(CAST(strftime('%%s', created_at) AS INTEGER) - %d) / %d AS bucket", start.Unix(), seconds)}
for _, column := range metrics {
if aggregation != "max" && (strings.HasPrefix(column, "time_to_first_token_p") || strings.HasPrefix(column, "time_per_output_token_p") || strings.HasPrefix(column, "request_latency_p")) {
continue
}
columns = append(columns, operation+"("+column+") AS "+column)
}
if aggregation != "max" {
columns = append(columns, "json_group_array(json(NULLIF(histogram_deltas, ''))) AS histogram_samples")
}
db = db.Group("bucket").Order("bucket ASC")
} else {
columns = append([]string{"id", "created_at", "app_install_id", "status"}, metrics...)
db = db.Order("created_at ASC, id ASC")
}
var points []VLLMHistoryPoint
err := db.Select(strings.Join(columns, ", ")).Scan(&points).Error
return points, err
}
+53
View File
@@ -1,11 +1,13 @@
package repo package repo
import ( import (
"context"
"errors" "errors"
"github.com/1Panel-dev/1Panel/agent/app/model" "github.com/1Panel-dev/1Panel/agent/app/model"
"github.com/1Panel-dev/1Panel/agent/global" "github.com/1Panel-dev/1Panel/agent/global"
"gorm.io/gorm" "gorm.io/gorm"
"gorm.io/gorm/clause"
) )
type SettingRepo struct{} type SettingRepo struct{}
@@ -19,12 +21,16 @@ type ISettingRepo interface {
WithByKey(key string) DBOption WithByKey(key string) DBOption
UpdateOrCreate(key, value string) error UpdateOrCreate(key, value string) error
UpdateValues(map[string]string) error
GetDescription(opts ...DBOption) (model.CommonDescription, error) GetDescription(opts ...DBOption) (model.CommonDescription, error)
GetDescriptionList(opts ...DBOption) ([]model.CommonDescription, error) GetDescriptionList(opts ...DBOption) ([]model.CommonDescription, error)
CreateDescription(data *model.CommonDescription) error CreateDescription(data *model.CommonDescription) error
SaveDescriptions(context.Context, []model.CommonDescription) error
UpdateDescription(id string, val map[string]interface{}) error UpdateDescription(id string, val map[string]interface{}) error
DelDescription(id string) error DelDescription(id string) error
DeleteDescriptions(context.Context, string, []string, bool) (int64, error)
WithDescriptionIDs(ids []string) DBOption
WithByDescriptionID(id string) DBOption WithByDescriptionID(id string) DBOption
} }
@@ -90,6 +96,25 @@ func (s *SettingRepo) UpdateOrCreate(key, value string) error {
return global.DB.Model(&setting).UpdateColumn("value", value).Error return global.DB.Model(&setting).UpdateColumn("value", value).Error
} }
func (s *SettingRepo) UpdateValues(values map[string]string) error {
return global.DB.Transaction(func(tx *gorm.DB) error {
for key, value := range values {
var setting model.Setting
err := tx.Where("key = ?", key).First(&setting).Error
if errors.Is(err, gorm.ErrRecordNotFound) {
if err := tx.Create(&model.Setting{Key: key, Value: value}).Error; err != nil {
return err
}
} else if err != nil {
return err
} else if err := tx.Model(&setting).UpdateColumn("value", value).Error; err != nil {
return err
}
}
return nil
})
}
func (s *SettingRepo) GetDescriptionList(opts ...DBOption) ([]model.CommonDescription, error) { func (s *SettingRepo) GetDescriptionList(opts ...DBOption) ([]model.CommonDescription, error) {
var lists []model.CommonDescription var lists []model.CommonDescription
db := global.DB.Model(&model.CommonDescription{}) db := global.DB.Model(&model.CommonDescription{})
@@ -111,14 +136,42 @@ func (s *SettingRepo) GetDescription(opts ...DBOption) (model.CommonDescription,
func (s *SettingRepo) CreateDescription(data *model.CommonDescription) error { func (s *SettingRepo) CreateDescription(data *model.CommonDescription) error {
return global.DB.Create(data).Error return global.DB.Create(data).Error
} }
func (s *SettingRepo) SaveDescriptions(ctx context.Context, descriptions []model.CommonDescription) error {
return global.DB.WithContext(ctx).Clauses(clause.OnConflict{
Columns: []clause.Column{{Name: "id"}},
DoUpdates: clause.AssignmentColumns([]string{"description"}),
}).CreateInBatches(&descriptions, 100).Error
}
func (s *SettingRepo) UpdateDescription(id string, val map[string]interface{}) error { func (s *SettingRepo) UpdateDescription(id string, val map[string]interface{}) error {
return global.DB.Model(&model.CommonDescription{}).Where("id = ?", id).Updates(val).Error return global.DB.Model(&model.CommonDescription{}).Where("id = ?", id).Updates(val).Error
} }
func (s *SettingRepo) DelDescription(id string) error { func (s *SettingRepo) DelDescription(id string) error {
return global.DB.Where("id = ?", id).Delete(&model.CommonDescription{}).Error return global.DB.Where("id = ?", id).Delete(&model.CommonDescription{}).Error
} }
func (s *SettingRepo) DeleteDescriptions(ctx context.Context, kind string, ids []string, emptyOnly bool) (int64, error) {
var deleted int64
for start := 0; start < len(ids); start += 500 {
query := global.DB.WithContext(ctx).Where("type = ? AND id IN ?", kind, ids[start:min(start+500, len(ids))])
if emptyOnly {
query = query.Where("description = ? AND is_pinned = ?", "", false)
}
result := query.Delete(&model.CommonDescription{})
deleted += result.RowsAffected
if result.Error != nil {
return deleted, result.Error
}
}
return deleted, nil
}
func (s *SettingRepo) WithByDescriptionID(id string) DBOption { func (s *SettingRepo) WithByDescriptionID(id string) DBOption {
return func(g *gorm.DB) *gorm.DB { return func(g *gorm.DB) *gorm.DB {
return g.Where("id = ?", id) return g.Where("id = ?", id)
} }
} }
func (s *SettingRepo) WithDescriptionIDs(ids []string) DBOption {
return func(db *gorm.DB) *gorm.DB { return db.Where("id IN ?", ids) }
}
+4 -1
View File
@@ -2,6 +2,7 @@ package repo
import ( import (
"context" "context"
"time"
"github.com/1Panel-dev/1Panel/agent/constant" "github.com/1Panel-dev/1Panel/agent/constant"
"github.com/1Panel-dev/1Panel/agent/global" "github.com/1Panel-dev/1Panel/agent/global"
@@ -110,7 +111,9 @@ func (t TaskRepo) Update(ctx context.Context, task *model.Task) error {
} }
func (t TaskRepo) UpdateRunningTaskToFailed() error { func (t TaskRepo) UpdateRunningTaskToFailed() error {
return getTaskDb(t.WithByStatus(constant.StatusExecuting)).Model(&model.Task{}).Updates(map[string]interface{}{"status": constant.StatusFailed, "error_msg": "1Panel restart causes failure"}).Error return getTaskDb(t.WithByStatus(constant.StatusExecuting)).Model(&model.Task{}).Updates(map[string]interface{}{
"status": constant.StatusFailed, "error_msg": constant.InterruptedMsg, "end_at": time.Now(),
}).Error
} }
func (t TaskRepo) CountExecutingTask() (int64, error) { func (t TaskRepo) CountExecutingTask() (int64, error) {
+131
View File
@@ -0,0 +1,131 @@
package repo
import (
"github.com/1Panel-dev/1Panel/agent/app/model"
"gorm.io/gorm"
)
type IWebsiteTemplateRepo interface {
Page(page, size int, opts ...DBOption) (int64, []model.WebsiteTemplate, error)
GetFirst(opts ...DBOption) (*model.WebsiteTemplate, error)
List(opts ...DBOption) ([]model.WebsiteTemplate, error)
Create(template *model.WebsiteTemplate) error
Save(template *model.WebsiteTemplate) error
DeleteBy(opts ...DBOption) error
WithName(name string) DBOption
WithType(templateType string) DBOption
}
func NewIWebsiteTemplateRepo() IWebsiteTemplateRepo {
return &WebsiteTemplateRepo{}
}
type WebsiteTemplateRepo struct {
}
func (w *WebsiteTemplateRepo) WithName(name string) DBOption {
return func(db *gorm.DB) *gorm.DB {
return db.Where("name like ?", "%"+name+"%")
}
}
func (w *WebsiteTemplateRepo) WithType(templateType string) DBOption {
return func(db *gorm.DB) *gorm.DB {
return db.Where("type = ?", templateType)
}
}
func (w *WebsiteTemplateRepo) Page(page, size int, opts ...DBOption) (int64, []model.WebsiteTemplate, error) {
var templates []model.WebsiteTemplate
db := getDb(opts...).Model(&model.WebsiteTemplate{})
count := int64(0)
db = db.Count(&count)
err := db.Limit(size).Offset(size * (page - 1)).Find(&templates).Error
return count, templates, err
}
func (w *WebsiteTemplateRepo) GetFirst(opts ...DBOption) (*model.WebsiteTemplate, error) {
var template model.WebsiteTemplate
db := getDb(opts...).Model(&model.WebsiteTemplate{})
if err := db.First(&template).Error; err != nil {
return nil, err
}
return &template, nil
}
func (w *WebsiteTemplateRepo) List(opts ...DBOption) ([]model.WebsiteTemplate, error) {
var templates []model.WebsiteTemplate
err := getDb(opts...).Model(&model.WebsiteTemplate{}).Find(&templates).Error
return templates, err
}
func (w *WebsiteTemplateRepo) Create(template *model.WebsiteTemplate) error {
return getDb().Create(template).Error
}
func (w *WebsiteTemplateRepo) Save(template *model.WebsiteTemplate) error {
return getDb().Save(template).Error
}
func (w *WebsiteTemplateRepo) DeleteBy(opts ...DBOption) error {
return getDb(opts...).Delete(&model.WebsiteTemplate{}).Error
}
type IWebsiteTemplateOutputRepo interface {
Page(page, size int, opts ...DBOption) (int64, []model.WebsiteTemplateOutput, error)
GetFirst(opts ...DBOption) (*model.WebsiteTemplateOutput, error)
List(opts ...DBOption) ([]model.WebsiteTemplateOutput, error)
Create(output *model.WebsiteTemplateOutput) error
Save(output *model.WebsiteTemplateOutput) error
DeleteBy(opts ...DBOption) error
WithByTemplateID(templateID uint) DBOption
}
func NewIWebsiteTemplateOutputRepo() IWebsiteTemplateOutputRepo {
return &WebsiteTemplateOutputRepo{}
}
type WebsiteTemplateOutputRepo struct {
}
func (w *WebsiteTemplateOutputRepo) WithByTemplateID(templateID uint) DBOption {
return func(db *gorm.DB) *gorm.DB {
return db.Where("template_id = ?", templateID)
}
}
func (w *WebsiteTemplateOutputRepo) Page(page, size int, opts ...DBOption) (int64, []model.WebsiteTemplateOutput, error) {
var outputs []model.WebsiteTemplateOutput
db := getDb(opts...).Model(&model.WebsiteTemplateOutput{})
count := int64(0)
db = db.Count(&count)
err := db.Limit(size).Offset(size * (page - 1)).Find(&outputs).Error
return count, outputs, err
}
func (w *WebsiteTemplateOutputRepo) GetFirst(opts ...DBOption) (*model.WebsiteTemplateOutput, error) {
var output model.WebsiteTemplateOutput
db := getDb(opts...).Model(&model.WebsiteTemplateOutput{})
if err := db.First(&output).Error; err != nil {
return nil, err
}
return &output, nil
}
func (w *WebsiteTemplateOutputRepo) List(opts ...DBOption) ([]model.WebsiteTemplateOutput, error) {
var outputs []model.WebsiteTemplateOutput
err := getDb(opts...).Model(&model.WebsiteTemplateOutput{}).Find(&outputs).Error
return outputs, err
}
func (w *WebsiteTemplateOutputRepo) Create(output *model.WebsiteTemplateOutput) error {
return getDb().Create(output).Error
}
func (w *WebsiteTemplateOutputRepo) Save(output *model.WebsiteTemplateOutput) error {
return getDb().Save(output).Error
}
func (w *WebsiteTemplateOutputRepo) DeleteBy(opts ...DBOption) error {
return getDb(opts...).Delete(&model.WebsiteTemplateOutput{}).Error
}
+72 -25
View File
@@ -105,6 +105,10 @@ type IAgentService interface {
UpgradePlugin(req dto.AgentPluginUpgradeReq) error UpgradePlugin(req dto.AgentPluginUpgradeReq) error
UninstallPlugin(req dto.AgentPluginUninstallReq) error UninstallPlugin(req dto.AgentPluginUninstallReq) error
CheckPlugin(req dto.AgentPluginCheckReq) (*dto.AgentPluginStatus, error) CheckPlugin(req dto.AgentPluginCheckReq) (*dto.AgentPluginStatus, error)
ListPlugins(req dto.AgentPluginsReq) ([]dto.AgentPluginItem, error)
SearchPlugins(req dto.AgentPluginSearchReq) ([]dto.AgentPluginSearchItem, error)
InstallMarketPlugin(req dto.AgentPluginMarketInstallReq) error
OperatePlugin(req dto.AgentPluginOperateReq) error
ApproveChannelPairing(req dto.AgentChannelPairingApproveReq) error ApproveChannelPairing(req dto.AgentChannelPairingApproveReq) error
} }
@@ -177,7 +181,7 @@ func (a AgentService) Create(req dto.AgentCreateReq) (*dto.AgentItem, error) {
var allowedOrigins []string var allowedOrigins []string
var account *model.AgentAccount var account *model.AgentAccount
var installHooks *appInstallHooks var installHooks *appInstallHooks
var hermesAuth hermesDashboardAuth var dashboardAuth agentDashboardAuth
if agentType == constant.AppOpenclaw || agentType == constant.AppHermesAgent { if agentType == constant.AppOpenclaw || agentType == constant.AppHermesAgent {
if req.AccountID == 0 { if req.AccountID == 0 {
@@ -222,15 +226,17 @@ func (a AgentService) Create(req dto.AgentCreateReq) (*dto.AgentItem, error) {
}, },
} }
} else if agentType == constant.AppHermesAgent { } else if agentType == constant.AppHermesAgent {
hermesAuth = normalizeHermesDashboardAuth(req.DashboardUsername, req.DashboardPassword) dashboardAuth = normalizeAgentDashboardAuth(req.DashboardUsername, req.DashboardPassword)
installHooks = &appInstallHooks{ installHooks = &appInstallHooks{
AfterCopyData: func(appInstall *model.AppInstall) error { AfterCopyData: func(appInstall *model.AppInstall) error {
if err := prepareHermesInstallFiles(appInstall, account, storedModel); err != nil { if err := prepareHermesInstallFiles(appInstall, account, storedModel); err != nil {
return err return err
} }
return writeHermesDashboardAuthEnv(path.Join(appInstall.GetPath(), ".env"), hermesAuth, false) return writeAgentDashboardAuthEnv(appInstall.GetEnvPath(), agentType, dashboardAuth, false)
}, },
} }
} else if agentType == constant.AppCopaw {
dashboardAuth = normalizeAgentDashboardAuth(req.DashboardUsername, req.DashboardPassword)
} }
params := map[string]interface{}{ params := map[string]interface{}{
@@ -250,9 +256,12 @@ func (a AgentService) Create(req dto.AgentCreateReq) (*dto.AgentItem, error) {
params["API_KEY"] = apiKey params["API_KEY"] = apiKey
params["OPENCLAW_GATEWAY_TOKEN"] = token params["OPENCLAW_GATEWAY_TOKEN"] = token
} }
if agentType == constant.AppHermesAgent { if usernameKey, passwordKey, ok := agentDashboardAuthEnvKeys(agentType); ok {
params[hermesDashboardUsernameEnvKey] = hermesAuth.Username params[usernameKey] = dashboardAuth.Username
params[hermesDashboardPasswordEnvKey] = hermesAuth.Password params[passwordKey] = dashboardAuth.Password
if agentType == constant.AppCopaw {
params[qwenPawAuthEnabledEnvKey] = "true"
}
} }
if req.EditCompose && strings.TrimSpace(req.DockerCompose) == "" { if req.EditCompose && strings.TrimSpace(req.DockerCompose) == "" {
@@ -927,6 +936,7 @@ func (a AgentService) GetModelConfig(req dto.AgentIDReq) (*dto.AgentModelConfig,
AccountID: agent.AccountID, AccountID: agent.AccountID,
Model: model, Model: model,
Fallbacks: extractOpenclawFallbackModelIDs(conf, account, models, model), Fallbacks: extractOpenclawFallbackModelIDs(conf, account, models, model),
Metadata: extractOpenclawModelMetadata(conf, account, models),
}, nil }, nil
} }
@@ -958,7 +968,7 @@ func (a AgentService) UpdateModelConfig(req dto.AgentModelConfigUpdateReq) error
if agent.AgentType != constant.AppOpenclaw { if agent.AgentType != constant.AppOpenclaw {
return fmt.Errorf("%s does not support", agent.AgentType) return fmt.Errorf("%s does not support", agent.AgentType)
} }
if err := writeOpenclawConfig(confDir, account, modelName, agent.Token, nil, req.Fallbacks); err != nil { if err := writeOpenclawConfig(confDir, account, modelName, agent.Token, nil, req.Fallbacks, req.Metadata); err != nil {
return err return err
} }
} }
@@ -984,19 +994,28 @@ func (a AgentService) GetProviders() ([]dto.ProviderInfo, error) {
} }
apiTypes := make([]dto.ProviderAPIInfo, 0, len(def.APIConfigs)) apiTypes := make([]dto.ProviderAPIInfo, 0, len(def.APIConfigs))
for _, item := range def.APIConfigs { for _, item := range def.APIConfigs {
apiModels := make([]dto.ProviderModelInfo, 0, len(item.Models))
for _, model := range item.Models {
apiModels = append(apiModels, dto.ProviderModelInfo{
ID: model.ID,
Name: model.Name,
})
}
apiTypes = append(apiTypes, dto.ProviderAPIInfo{ apiTypes = append(apiTypes, dto.ProviderAPIInfo{
APIType: item.APIType, APIType: item.APIType,
BaseURL: item.BaseURL, BaseURL: item.BaseURL,
EditableBaseURL: item.EditableBaseURL, EditableBaseURL: item.EditableBaseURL,
DefaultAuthMode: item.DefaultAuthMode, SupportsModelDiscovery: item.DiscoverModels,
AuthModes: item.AuthModes, DefaultAuthMode: item.DefaultAuthMode,
AuthModes: item.AuthModes,
Models: apiModels,
}) })
} }
baseURL, _ := providercatalog.DefaultBaseURL(key) baseURL, _ := providercatalog.DefaultBaseURL(key)
providers = append(providers, dto.ProviderInfo{ providers = append(providers, dto.ProviderInfo{
Sort: def.Sort, Sort: def.Sort,
Provider: key, Provider: key,
DisplayName: def.DisplayName, DisplayName: localizedAgentProviderName(key),
BaseURL: baseURL, BaseURL: baseURL,
DefaultAPIType: def.DefaultAPIType, DefaultAPIType: def.DefaultAPIType,
APITypes: apiTypes, APITypes: apiTypes,
@@ -1016,7 +1035,7 @@ func (a AgentService) CreateAccount(req dto.AgentAccountCreateReq) error {
if err := ensureAgentAccountNameAvailable(provider, req.Name, 0); err != nil { if err := ensureAgentAccountNameAvailable(provider, req.Name, 0); err != nil {
return err return err
} }
initialModels, err := buildInitialAgentAccountModels(&model.AgentAccount{Provider: provider}, req.Models) initialModels, err := buildInitialAgentAccountModels(&model.AgentAccount{Provider: provider, APIType: req.APIType}, req.Models)
if err != nil { if err != nil {
return err return err
} }
@@ -1024,7 +1043,8 @@ func (a AgentService) CreateAccount(req dto.AgentAccountCreateReq) error {
if err != nil { if err != nil {
return err return err
} }
resolvedInput, err := resolveAgentAccountInput(provider, req.APIType, req.AuthMode, req.APIKey, req.BaseURL, verifyModel) validateAvailability := req.ValidateAvailability == nil || *req.ValidateAvailability
resolvedInput, err := resolveAgentAccountInput(provider, req.APIType, req.AuthMode, req.APIKey, req.BaseURL, verifyModel, validateAvailability)
if err != nil { if err != nil {
return err return err
} }
@@ -1062,6 +1082,9 @@ func (a AgentService) UpdateAccount(req dto.AgentAccountUpdateReq) error {
if err != nil { if err != nil {
return err return err
} }
if req.APIType != account.APIType {
return buserr.WithDetail("ErrInvalidParams", "API type cannot be changed", nil)
}
provider := account.Provider provider := account.Provider
if err := ensureAgentAccountNameAvailable(provider, req.Name, account.ID); err != nil { if err := ensureAgentAccountNameAvailable(provider, req.Name, account.ID); err != nil {
return err return err
@@ -1078,7 +1101,8 @@ func (a AgentService) UpdateAccount(req dto.AgentAccountUpdateReq) error {
if err != nil { if err != nil {
return err return err
} }
resolvedInput, err := resolveAgentAccountInput(provider, req.APIType, req.AuthMode, req.APIKey, req.BaseURL, verifyModel) validateAvailability := req.ValidateAvailability == nil || *req.ValidateAvailability
resolvedInput, err := resolveAgentAccountInput(provider, req.APIType, req.AuthMode, req.APIKey, req.BaseURL, verifyModel, validateAvailability)
if err != nil { if err != nil {
return err return err
} }
@@ -1110,6 +1134,12 @@ func (a AgentService) PageAccounts(req dto.AgentAccountSearch) (int64, []dto.Age
if strings.TrimSpace(req.Provider) != "" { if strings.TrimSpace(req.Provider) != "" {
opts = append(opts, repo.WithByProvider(req.Provider)) opts = append(opts, repo.WithByProvider(req.Provider))
} }
if apiType := strings.TrimSpace(req.APIType); apiType != "" {
opts = append(opts, repo.WithByAPIType(apiType))
}
if req.TextOnly {
opts = append(opts, repo.WithTextAPIType())
}
if strings.TrimSpace(req.Name) != "" { if strings.TrimSpace(req.Name) != "" {
opts = append(opts, repo.WithByLikeName(req.Name)) opts = append(opts, repo.WithByLikeName(req.Name))
} }
@@ -1127,7 +1157,7 @@ func (a AgentService) PageAccounts(req dto.AgentAccountSearch) (int64, []dto.Age
ID: item.ID, ID: item.ID,
MasterAccountID: item.MasterAccountID, MasterAccountID: item.MasterAccountID,
Provider: item.Provider, Provider: item.Provider,
ProviderName: providercatalog.DisplayName(item.Provider), ProviderName: localizedAgentProviderName(item.Provider),
Name: item.Name, Name: item.Name,
APIKey: apiKey, APIKey: apiKey,
RememberAPIKey: item.RememberAPIKey, RememberAPIKey: item.RememberAPIKey,
@@ -1166,7 +1196,7 @@ func (a AgentService) PageAccounts(req dto.AgentAccountSearch) (int64, []dto.Age
} }
func (a AgentService) CountAccountsByProviders(req dto.AgentAccountProviderCountReq) (map[string]int64, error) { func (a AgentService) CountAccountsByProviders(req dto.AgentAccountProviderCountReq) (map[string]int64, error) {
return agentAccountRepo.CountByProviders(req.Providers) return agentAccountRepo.CountTextByProviders(req.Providers)
} }
func (a AgentService) GetAccountModels(req dto.AgentAccountModelReq) ([]dto.AgentAccountModel, error) { func (a AgentService) GetAccountModels(req dto.AgentAccountModelReq) ([]dto.AgentAccountModel, error) {
@@ -1178,7 +1208,8 @@ func (a AgentService) GetAccountModels(req dto.AgentAccountModelReq) ([]dto.Agen
} }
func (a AgentService) DiscoverAccountModels(req dto.AgentAccountModelDiscoverReq) ([]dto.AgentAccountModel, error) { func (a AgentService) DiscoverAccountModels(req dto.AgentAccountModelDiscoverReq) ([]dto.AgentAccountModel, error) {
if req.APIType != "openai-completions" && req.APIType != "openai-responses" { config, ok := providercatalog.FindAPIConfig(req.Provider, req.APIType)
if !ok || !config.DiscoverModels {
return nil, buserr.New("ErrAgentAccountModelsRequired") return nil, buserr.New("ErrAgentAccountModelsRequired")
} }
baseURL, err := providercatalog.ResolveBaseURL(req.Provider, req.APIType, req.BaseURL) baseURL, err := providercatalog.ResolveBaseURL(req.Provider, req.APIType, req.BaseURL)
@@ -1320,7 +1351,7 @@ func (a AgentService) SyncAgentsByAccount(account *model.AgentAccount) error {
} }
func (a AgentService) VerifyAccount(req dto.AgentAccountVerifyReq) error { func (a AgentService) VerifyAccount(req dto.AgentAccountVerifyReq) error {
_, err := resolveAgentAccountInput(req.Provider, req.APIType, req.AuthMode, req.APIKey, req.BaseURL, req.Model) _, err := resolveAgentAccountInput(req.Provider, req.APIType, req.AuthMode, req.APIKey, req.BaseURL, req.Model, true)
return err return err
} }
@@ -1410,7 +1441,7 @@ func (a AgentService) GetOtherConfig(req dto.AgentIDReq) (*dto.AgentOtherConfig,
if err != nil { if err != nil {
return nil, err return nil, err
} }
auth := readHermesDashboardAuthFromInstall(install) auth := readAgentDashboardAuthFromInstall(install, agent.AgentType)
return &dto.AgentOtherConfig{ return &dto.AgentOtherConfig{
UserTimezone: cfg.Timezone, UserTimezone: cfg.Timezone,
BrowserEnabled: true, BrowserEnabled: true,
@@ -1419,6 +1450,13 @@ func (a AgentService) GetOtherConfig(req dto.AgentIDReq) (*dto.AgentOtherConfig,
DashboardPassword: auth.Password, DashboardPassword: auth.Password,
}, nil }, nil
} }
if agent.AgentType == constant.AppCopaw {
auth := readAgentDashboardAuthFromInstall(install, agent.AgentType)
return &dto.AgentOtherConfig{
DashboardUsername: auth.Username,
DashboardPassword: auth.Password,
}, nil
}
conf, err := readOpenclawConfig(agent.ConfigPath) conf, err := readOpenclawConfig(agent.ConfigPath)
if err != nil { if err != nil {
return nil, err return nil, err
@@ -1437,16 +1475,19 @@ func (a AgentService) UpdateOtherConfig(req dto.AgentOtherConfigUpdateReq) error
return err return err
} }
if agent.AgentType == constant.AppHermesAgent { if agent.AgentType == constant.AppHermesAgent {
if strings.TrimSpace(req.UserTimezone) == "" {
return buserr.New("ErrInvalidParams")
}
account, err := agentAccountRepo.GetFirst(repo.WithByID(agent.AccountID)) account, err := agentAccountRepo.GetFirst(repo.WithByID(agent.AccountID))
if err != nil { if err != nil {
return err return err
} }
previousAuth := readHermesDashboardAuthFromInstall(install) previousAuth := readAgentDashboardAuthFromInstall(install, agent.AgentType)
nextAuth := normalizeHermesDashboardAuth(req.DashboardUsername, req.DashboardPassword) nextAuth := normalizeAgentDashboardAuth(req.DashboardUsername, req.DashboardPassword)
if err := writeHermesConfig(path.Dir(agent.ConfigPath), account, agent.Model, strings.TrimSpace(req.UserTimezone)); err != nil { if err := writeHermesConfig(path.Dir(agent.ConfigPath), account, agent.Model, strings.TrimSpace(req.UserTimezone)); err != nil {
return err return err
} }
if err := writeHermesDashboardAuthEnv(path.Join(install.GetPath(), ".env"), nextAuth, true); err != nil { if err := writeAgentDashboardAuthEnv(install.GetEnvPath(), agent.AgentType, nextAuth, true); err != nil {
return err return err
} }
operate := constant.Restart operate := constant.Restart
@@ -1458,6 +1499,12 @@ func (a AgentService) UpdateOtherConfig(req dto.AgentOtherConfigUpdateReq) error
Operate: operate, Operate: operate,
}) })
} }
if agent.AgentType == constant.AppCopaw {
return updateQwenPawDashboardAuth(install, normalizeAgentDashboardAuth(req.DashboardUsername, req.DashboardPassword))
}
if strings.TrimSpace(req.UserTimezone) == "" || strings.TrimSpace(req.NPMRegistry) == "" {
return buserr.New("ErrInvalidParams")
}
if err := ensureContainerRunning(install.ContainerName); err != nil { if err := ensureContainerRunning(install.ContainerName); err != nil {
return err return err
} }
@@ -1638,7 +1685,7 @@ func (a AgentService) syncAgentsByAccount(account *model.AgentAccount) error {
return err return err
} }
fallbacks := extractOpenclawFallbackModelIDs(conf, account, accountModels, selectedAccountModel.ID) fallbacks := extractOpenclawFallbackModelIDs(conf, account, accountModels, selectedAccountModel.ID)
if err := writeOpenclawConfig(confDir, account, modelName, agent.Token, nil, fallbacks); err != nil { if err := writeOpenclawConfig(confDir, account, modelName, agent.Token, nil, fallbacks, nil); err != nil {
return err return err
} }
case constant.AppHermesAgent: case constant.AppHermesAgent:
+18 -1
View File
@@ -5,6 +5,7 @@ import (
"fmt" "fmt"
"os" "os"
"path" "path"
"slices"
"sort" "sort"
"strings" "strings"
"time" "time"
@@ -1320,6 +1321,10 @@ func appendPluginAllow(conf map[string]interface{}, pluginID string) {
} }
func installOpenclawPlugin(mgr *cmd.CommandHelper, containerName, spec, pluginID string) error { func installOpenclawPlugin(mgr *cmd.CommandHelper, containerName, spec, pluginID string) error {
help, err := cmd.RunDockerExecWithStdout(time.Minute, containerName, "openclaw", "plugins", "install", "--help")
if err != nil {
return err
}
workdir := path.Join(openclawPluginPackageTmpDir, pluginID) workdir := path.Join(openclawPluginPackageTmpDir, pluginID)
defer func() { defer func() {
_ = mgr.Run("docker", "exec", containerName, "rm", "-rf", workdir) _ = mgr.Run("docker", "exec", containerName, "rm", "-rf", workdir)
@@ -1341,7 +1346,19 @@ func installOpenclawPlugin(mgr *cmd.CommandHelper, containerName, spec, pluginID
if pkgPath == "" { if pkgPath == "" {
return fmt.Errorf("openclaw plugin package not found") return fmt.Errorf("openclaw plugin package not found")
} }
return mgr.Run("docker", "exec", containerName, "openclaw", "plugins", "install", pkgPath, "--dangerously-force-unsafe-install") args := []string{"exec", containerName, "openclaw", "plugins", "install", pkgPath}
// Newer CLIs require source confirmation; older releases do not support --force.
options := strings.Fields(help)
if slices.Contains(options, "--force") {
args = append(args, "--force")
} else if slices.Contains(options, "--dangerously-force-unsafe-install") {
args = append(args, "--dangerously-force-unsafe-install")
}
// Source confirmation does not grant the selected channel plugin's capabilities.
if slices.Contains(options, "--accept-capabilities") {
args = append(args, "--accept-capabilities")
}
return mgr.Run("docker", args...)
} }
func uninstallOpenclawPlugin(mgr *cmd.CommandHelper, containerName, pluginID string) error { func uninstallOpenclawPlugin(mgr *cmd.CommandHelper, containerName, pluginID string) error {
+128
View File
@@ -0,0 +1,128 @@
package service
import (
"bytes"
"context"
"encoding/json"
"errors"
"fmt"
"io"
"net/http"
"strings"
"time"
"github.com/1Panel-dev/1Panel/agent/app/model"
"github.com/1Panel-dev/1Panel/agent/buserr"
"github.com/1Panel-dev/1Panel/agent/constant"
)
type qwenPawAuthStatus struct {
Enabled bool `json:"enabled"`
HasUsers bool `json:"has_users"`
}
type qwenPawLoginResponse struct {
Token string `json:"token"`
}
func updateQwenPawDashboardAuth(install *model.AppInstall, next agentDashboardAuth) error {
if install == nil || install.ID == 0 {
return buserr.New("ErrRecordNotFound")
}
current, err := readAgentDashboardAuthEnv(install.GetEnvPath(), constant.AppCopaw)
if err != nil {
return err
}
if current == next {
return writeAgentDashboardAuthEnv(install.GetEnvPath(), constant.AppCopaw, next, true)
}
if err := ensureContainerRunning(install.ContainerName); err != nil {
return err
}
baseURL := fmt.Sprintf("http://127.0.0.1:%d/api/auth", install.HttpPort)
var status qwenPawAuthStatus
if _, err := requestQwenPawAuth(http.MethodGet, baseURL+"/status", nil, "", &status); err != nil {
return buserr.WithMap("ErrQwenPawAuthRequest", map[string]interface{}{"err": err.Error()}, err)
}
if !status.Enabled {
return buserr.New("ErrQwenPawAuthDisabled")
}
if !status.HasUsers {
payload := map[string]string{"username": next.Username, "password": next.Password}
if _, err := requestQwenPawAuth(http.MethodPost, baseURL+"/register", payload, "", nil); err != nil {
return buserr.WithMap("ErrQwenPawAuthRequest", map[string]interface{}{"err": err.Error()}, err)
}
} else {
var login qwenPawLoginResponse
payload := map[string]string{"username": current.Username, "password": current.Password}
statusCode, err := requestQwenPawAuth(http.MethodPost, baseURL+"/login", payload, "", &login)
if statusCode == http.StatusUnauthorized {
return buserr.New("ErrQwenPawAuthOutOfSync")
}
if err != nil {
return buserr.WithMap("ErrQwenPawAuthRequest", map[string]interface{}{"err": err.Error()}, err)
}
payload = map[string]string{"current_password": current.Password}
if current.Username != next.Username {
payload["new_username"] = next.Username
}
if current.Password != next.Password {
payload["new_password"] = next.Password
}
if _, err := requestQwenPawAuth(http.MethodPost, baseURL+"/update-profile", payload, login.Token, nil); err != nil {
return buserr.WithMap("ErrQwenPawAuthRequest", map[string]interface{}{"err": err.Error()}, err)
}
}
return writeAgentDashboardAuthEnv(install.GetEnvPath(), constant.AppCopaw, next, true)
}
func requestQwenPawAuth(method, reqURL string, payload interface{}, token string, result interface{}) (int, error) {
var body io.Reader
if payload != nil {
data, err := json.Marshal(payload)
if err != nil {
return 0, err
}
body = bytes.NewReader(data)
}
ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
defer cancel()
req, err := http.NewRequestWithContext(ctx, method, reqURL, body)
if err != nil {
return 0, err
}
req.Header.Set("Content-Type", "application/json")
if token != "" {
req.Header.Set("Authorization", "Bearer "+token)
}
resp, err := (&http.Client{Timeout: 10 * time.Second}).Do(req)
if err != nil {
return 0, err
}
defer resp.Body.Close()
data, err := io.ReadAll(io.LimitReader(resp.Body, 1<<20))
if err != nil {
return resp.StatusCode, err
}
if resp.StatusCode < http.StatusOK || resp.StatusCode >= http.StatusMultipleChoices {
detail := strings.TrimSpace(string(data))
var errorResponse struct {
Detail string `json:"detail"`
}
if json.Unmarshal(data, &errorResponse) == nil && strings.TrimSpace(errorResponse.Detail) != "" {
detail = strings.TrimSpace(errorResponse.Detail)
}
if detail == "" {
detail = resp.Status
}
return resp.StatusCode, errors.New(detail)
}
if result != nil && len(data) > 0 {
if err := json.Unmarshal(data, result); err != nil {
return resp.StatusCode, err
}
}
return resp.StatusCode, nil
}
-53
View File
@@ -19,13 +19,6 @@ import (
const hermesWorkspaceDir = "/opt/data/workspace" const hermesWorkspaceDir = "/opt/data/workspace"
const hermesExecutablePath = "/opt/hermes/.venv/bin/hermes" const hermesExecutablePath = "/opt/hermes/.venv/bin/hermes"
const hermesDashboardUsernameEnvKey = "HERMES_DASHBOARD_USERNAME"
const hermesDashboardPasswordEnvKey = "HERMES_DASHBOARD_PASSWORD"
type hermesDashboardAuth struct {
Username string
Password string
}
type hermesConfig struct { type hermesConfig struct {
Model hermesModelConfig `yaml:"model"` Model hermesModelConfig `yaml:"model"`
@@ -117,52 +110,6 @@ func prepareHermesInstallFiles(appInstall *model.AppInstall, account *model.Agen
return files.NewFileOp().ChownR(dataDir, "1000", "1000", true) return files.NewFileOp().ChownR(dataDir, "1000", "1000", true)
} }
func normalizeHermesDashboardAuth(username, password string) hermesDashboardAuth {
auth := hermesDashboardAuth{
Username: strings.TrimSpace(username),
Password: strings.TrimSpace(password),
}
if auth.Username == "" {
auth.Username = "admin"
}
if auth.Password == "" {
auth.Password = common.RandStr(8)
}
return auth
}
func writeHermesDashboardAuthEnv(envPath string, auth hermesDashboardAuth, overwrite bool) error {
return upsertAgentEnv(envPath, map[string]string{
hermesDashboardUsernameEnvKey: auth.Username,
hermesDashboardPasswordEnvKey: auth.Password,
}, []string{
hermesDashboardUsernameEnvKey,
hermesDashboardPasswordEnvKey,
}, overwrite)
}
func readHermesDashboardAuthEnv(envPath string) (hermesDashboardAuth, error) {
envMap, err := readAgentEnvMap(envPath)
if err != nil {
return hermesDashboardAuth{}, err
}
return hermesDashboardAuth{
Username: strings.TrimSpace(envMap[hermesDashboardUsernameEnvKey]),
Password: strings.TrimSpace(envMap[hermesDashboardPasswordEnvKey]),
}, nil
}
func readHermesDashboardAuthFromInstall(appInstall *model.AppInstall) hermesDashboardAuth {
if appInstall == nil || appInstall.ID == 0 {
return hermesDashboardAuth{}
}
auth, err := readHermesDashboardAuthEnv(path.Join(appInstall.GetPath(), ".env"))
if err != nil {
return hermesDashboardAuth{}
}
return auth
}
func readHermesConfig(configPath string) (*hermesConfig, error) { func readHermesConfig(configPath string) (*hermesConfig, error) {
content, err := files.NewFileOp().GetContent(configPath) content, err := files.NewFileOp().GetContent(configPath)
if err != nil { if err != nil {
+315
View File
@@ -0,0 +1,315 @@
package service
import (
"database/sql"
"encoding/json"
"fmt"
"path/filepath"
"regexp"
"strings"
"time"
"github.com/1Panel-dev/1Panel/agent/app/dto"
"github.com/1Panel-dev/1Panel/agent/app/model"
"github.com/1Panel-dev/1Panel/agent/app/task"
"github.com/1Panel-dev/1Panel/agent/buserr"
"github.com/1Panel-dev/1Panel/agent/global"
"github.com/1Panel-dev/1Panel/agent/i18n"
"github.com/1Panel-dev/1Panel/agent/utils/cmd"
"github.com/1Panel-dev/1Panel/agent/utils/compose"
)
var (
openclawPluginPackagePattern = regexp.MustCompile(`^(@[a-z0-9][a-z0-9._-]*/)?[a-z0-9][a-z0-9._-]*$`)
openclawPluginVersionPattern = regexp.MustCompile(`^[0-9A-Za-z][0-9A-Za-z._-]*$`)
openclawPluginIDPattern = regexp.MustCompile(`^(@[A-Za-z0-9][A-Za-z0-9._-]*/)?[A-Za-z0-9][A-Za-z0-9._-]*$`)
)
type openclawPluginListOutput struct {
Plugins []struct {
ID string `json:"id"`
Name string `json:"name"`
Version string `json:"version"`
Origin string `json:"origin"`
Enabled bool `json:"enabled"`
} `json:"plugins"`
}
type openclawPluginIndexItem struct {
PluginID string `json:"pluginId"`
PackageName string `json:"packageName"`
PackageVersion string `json:"packageVersion"`
Origin string `json:"origin"`
Enabled bool `json:"enabled"`
}
type openclawPluginSearchOutput struct {
Results []struct {
Score float64 `json:"score"`
Package struct {
Name string `json:"name"`
RuntimeID string `json:"runtimeId"`
DisplayName string `json:"displayName"`
Summary string `json:"summary"`
LatestVersion string `json:"latestVersion"`
Categories []string `json:"categories"`
Channel string `json:"channel"`
IsOfficial bool `json:"isOfficial"`
VerificationTier string `json:"verificationTier"`
Stats struct {
Downloads int64 `json:"downloads"`
} `json:"stats"`
} `json:"package"`
} `json:"results"`
}
func (a AgentService) ListPlugins(req dto.AgentPluginsReq) ([]dto.AgentPluginItem, error) {
agent, install, err := a.loadOpenclawAgentAndInstall(req.AgentID)
if err != nil {
return nil, err
}
if plugins, err := readOpenclawPluginIndex(filepath.Join(filepath.Dir(agent.ConfigPath), "state", "openclaw.sqlite")); err == nil {
return plugins, nil
}
output, err := cmd.RunDockerExecWithStdout(2*time.Minute, install.ContainerName, "openclaw", "plugins", "list", "--json")
if err != nil {
return nil, err
}
return parseOpenclawPluginList([]byte(output))
}
func (a AgentService) SearchPlugins(req dto.AgentPluginSearchReq) ([]dto.AgentPluginSearchItem, error) {
_, install, err := a.loadOpenclawAgentAndInstall(req.AgentID)
if err != nil {
return nil, err
}
limit := req.Limit
if limit == 0 {
limit = 20
}
output, err := cmd.RunDockerExecWithStdout(
2*time.Minute,
install.ContainerName,
"openclaw", "plugins", "search", strings.TrimSpace(req.Keyword), "--limit", fmt.Sprint(limit), "--json",
)
if err != nil {
return nil, err
}
return parseOpenclawPluginSearch([]byte(output))
}
func (a AgentService) InstallMarketPlugin(req dto.AgentPluginMarketInstallReq) error {
spec, err := buildOpenclawPluginInstallSpec(req.Package, req.Version)
if err != nil {
return err
}
_, install, err := a.loadOpenclawAgentAndInstall(req.AgentID)
if err != nil {
return err
}
if err := task.CheckScopeTaskIsExecuting(task.TaskScopeAI, req.AgentID); err != nil {
return err
}
taskName := fmt.Sprintf("%s [%s]", i18n.GetMsgByKey("AgentPluginInstall"), req.Package)
installTask, err := task.NewTask(taskName, task.TaskInstall, task.TaskScopeAI, req.TaskID, req.AgentID)
if err != nil {
return err
}
installTask.AddSubTask(taskName, func(t *task.Task) error {
mgr := cmd.NewCommandMgr(cmd.WithTask(*t), cmd.WithContext(t.TaskCtx), cmd.WithTimeout(10*time.Minute))
return mgr.Run("docker", "exec", install.ContainerName, "openclaw", "plugins", "install", spec)
}, nil)
addOpenclawPluginRestartTask(installTask, install)
go executeAgentPluginTask(installTask)
return nil
}
func (a AgentService) OperatePlugin(req dto.AgentPluginOperateReq) error {
if !openclawPluginIDPattern.MatchString(req.PluginID) {
return buserr.New("ErrInvalidChar")
}
agent, install, err := a.loadOpenclawAgentAndInstall(req.AgentID)
if err != nil {
return err
}
if err := task.CheckScopeTaskIsExecuting(task.TaskScopeAI, req.AgentID); err != nil {
return err
}
if req.Operate == "update" || req.Operate == "uninstall" {
plugins, err := a.ListPlugins(dto.AgentPluginsReq{AgentID: req.AgentID})
if err != nil {
return err
}
for _, plugin := range plugins {
if plugin.ID == req.PluginID && plugin.Origin == "bundled" {
return buserr.WithName("ErrNotSupportType", req.Operate)
}
}
}
taskType := map[string]string{
"enable": task.TaskUpdate,
"disable": task.TaskUpdate,
"update": task.TaskUpgrade,
"uninstall": task.TaskUninstall,
}[req.Operate]
taskName := fmt.Sprintf("%s [%s]", i18n.GetMsgByKey(map[string]string{
"enable": "AgentPluginEnable",
"disable": "AgentPluginDisable",
"update": "AgentPluginUpdate",
"uninstall": "AgentPluginUninstall",
}[req.Operate]), req.PluginID)
operateTask, err := task.NewTask(taskName, taskType, task.TaskScopeAI, req.TaskID, req.AgentID)
if err != nil {
return err
}
operateTask.AddSubTask(taskName, func(t *task.Task) error {
mgr := cmd.NewCommandMgr(cmd.WithTask(*t), cmd.WithContext(t.TaskCtx), cmd.WithTimeout(10*time.Minute))
if req.Operate == "uninstall" {
if err := uninstallOpenclawPlugin(mgr, install.ContainerName, req.PluginID); err != nil {
return err
}
return cleanupManagedOpenclawPlugin(agent, req.PluginID)
}
return mgr.Run("docker", "exec", install.ContainerName, "openclaw", "plugins", req.Operate, req.PluginID)
}, nil)
addOpenclawPluginRestartTask(operateTask, install)
go executeAgentPluginTask(operateTask)
return nil
}
func parseOpenclawPluginList(raw []byte) ([]dto.AgentPluginItem, error) {
payload, err := extractEmbeddedJSON(string(raw))
if err != nil {
return nil, err
}
if len(payload) == 0 {
return []dto.AgentPluginItem{}, nil
}
var output openclawPluginListOutput
if err := json.Unmarshal(payload, &output); err != nil {
return nil, err
}
items := make([]dto.AgentPluginItem, 0, len(output.Plugins))
for _, plugin := range output.Plugins {
items = append(items, dto.AgentPluginItem{
ID: plugin.ID,
Name: plugin.Name,
Version: plugin.Version,
Origin: plugin.Origin,
Enabled: plugin.Enabled,
})
}
return items, nil
}
func readOpenclawPluginIndex(dbPath string) ([]dto.AgentPluginItem, error) {
db, err := sql.Open("sqlite", "file:"+filepath.ToSlash(dbPath)+"?mode=ro")
if err != nil {
return nil, err
}
defer db.Close()
var raw []byte
if err := db.QueryRow(
"SELECT plugins_json FROM installed_plugin_index WHERE index_key = ?",
"installed-plugin-index",
).Scan(&raw); err != nil {
return nil, err
}
var plugins []openclawPluginIndexItem
if err := json.Unmarshal(raw, &plugins); err != nil {
return nil, err
}
items := make([]dto.AgentPluginItem, 0, len(plugins))
for _, plugin := range plugins {
name := plugin.PackageName
if name == "" {
name = plugin.PluginID
}
items = append(items, dto.AgentPluginItem{
ID: plugin.PluginID,
Name: name,
Version: plugin.PackageVersion,
Origin: plugin.Origin,
Enabled: plugin.Enabled,
})
}
return items, nil
}
func parseOpenclawPluginSearch(raw []byte) ([]dto.AgentPluginSearchItem, error) {
payload, err := extractEmbeddedJSON(string(raw))
if err != nil {
return nil, err
}
if len(payload) == 0 {
return []dto.AgentPluginSearchItem{}, nil
}
var output openclawPluginSearchOutput
if err := json.Unmarshal(payload, &output); err != nil {
return nil, err
}
items := make([]dto.AgentPluginSearchItem, 0, len(output.Results))
for _, result := range output.Results {
items = append(items, dto.AgentPluginSearchItem{
Package: result.Package.Name,
PluginID: result.Package.RuntimeID,
Name: result.Package.DisplayName,
Description: result.Package.Summary,
Version: result.Package.LatestVersion,
Channel: result.Package.Channel,
VerificationTier: result.Package.VerificationTier,
Categories: append([]string{}, result.Package.Categories...),
Official: result.Package.IsOfficial,
Downloads: result.Package.Stats.Downloads,
Score: result.Score,
})
}
return items, nil
}
func buildOpenclawPluginInstallSpec(packageName, version string) (string, error) {
packageName = strings.TrimSpace(packageName)
version = strings.TrimSpace(version)
if !openclawPluginPackagePattern.MatchString(packageName) || !openclawPluginVersionPattern.MatchString(version) {
return "", buserr.New("ErrInvalidChar")
}
return "clawhub:" + packageName + "@" + version, nil
}
func cleanupManagedOpenclawPlugin(agent *model.Agent, pluginID string) error {
pluginType := map[string]string{
"openclaw-lark": "feishu",
"openclaw-qqbot": "qqbot",
"wecom-openclaw-plugin": "wecom",
"dingtalk-connector": "dingtalk",
"openclaw-weixin": "weixin",
}[pluginID]
if pluginType == "" {
return nil
}
conf, err := readOpenclawConfig(agent.ConfigPath)
if err != nil {
return err
}
cleanupOpenclawPluginConfig(conf, pluginType)
return writeOpenclawConfigRaw(agent.ConfigPath, conf)
}
func addOpenclawPluginRestartTask(t *task.Task, install *model.AppInstall) {
t.AddSubTask(task.GetTaskName("OpenClaw", task.TaskRestart, task.TaskScopeAI), func(t *task.Task) error {
output, err := compose.Restart(install.GetComposePath())
if output != "" {
t.Log(output)
}
return err
}, nil)
}
func executeAgentPluginTask(t *task.Task) {
if err := t.Execute(); err != nil {
global.LOG.Errorf("operate openclaw plugin failed: %v", err)
}
}
+252 -15
View File
@@ -10,6 +10,7 @@ import (
"net/url" "net/url"
"path" "path"
"regexp" "regexp"
"slices"
"strconv" "strconv"
"strings" "strings"
"time" "time"
@@ -21,6 +22,7 @@ import (
"github.com/1Panel-dev/1Panel/agent/buserr" "github.com/1Panel-dev/1Panel/agent/buserr"
"github.com/1Panel-dev/1Panel/agent/constant" "github.com/1Panel-dev/1Panel/agent/constant"
"github.com/1Panel-dev/1Panel/agent/global" "github.com/1Panel-dev/1Panel/agent/global"
"github.com/1Panel-dev/1Panel/agent/i18n"
"github.com/1Panel-dev/1Panel/agent/utils/common" "github.com/1Panel-dev/1Panel/agent/utils/common"
agentenv "github.com/1Panel-dev/1Panel/agent/utils/env" agentenv "github.com/1Panel-dev/1Panel/agent/utils/env"
"github.com/1Panel-dev/1Panel/agent/utils/files" "github.com/1Panel-dev/1Panel/agent/utils/files"
@@ -96,7 +98,7 @@ func ensureContainerRunning(containerName string) error {
return nil return nil
} }
func resolveAgentAccountInput(provider, apiType, authMode, apiKey, baseURL, modelID string) (resolvedAgentAccountInput, error) { func resolveAgentAccountInput(provider, apiType, authMode, apiKey, baseURL, modelID string, validateAvailability bool) (resolvedAgentAccountInput, error) {
resolvedAPIKey := strings.TrimSpace(apiKey) resolvedAPIKey := strings.TrimSpace(apiKey)
resolvedAPIType := strings.TrimSpace(apiType) resolvedAPIType := strings.TrimSpace(apiType)
resolvedAuthMode, err := providercatalog.ResolveAuthMode(provider, resolvedAPIType, authMode) resolvedAuthMode, err := providercatalog.ResolveAuthMode(provider, resolvedAPIType, authMode)
@@ -114,7 +116,8 @@ func resolveAgentAccountInput(provider, apiType, authMode, apiKey, baseURL, mode
if modelID == "" { if modelID == "" {
return resolvedAgentAccountInput{}, buserr.New("ErrAgentAccountModelsRequired") return resolvedAgentAccountInput{}, buserr.New("ErrAgentAccountModelsRequired")
} }
if !providercatalog.SkipVerification(provider) { imageAPI := providercatalog.IsImageAPIType(resolvedAPIType)
if validateAvailability && (imageAPI || providercatalog.IsEmbeddingAPIType(resolvedAPIType) || !providercatalog.SkipVerification(provider)) {
if err := providercatalog.VerifyAccount(provider, resolvedAPIType, resolvedAuthMode, resolvedBaseURL, resolvedAPIKey, modelID); err != nil { if err := providercatalog.VerifyAccount(provider, resolvedAPIType, resolvedAuthMode, resolvedBaseURL, resolvedAPIKey, modelID); err != nil {
return resolvedAgentAccountInput{}, err return resolvedAgentAccountInput{}, err
} }
@@ -386,7 +389,7 @@ func buildAgentItem(agent *model.Agent, appInstall *model.AppInstall, envMap map
Remark: agent.Remark, Remark: agent.Remark,
AgentType: agentType, AgentType: agentType,
Provider: agent.Provider, Provider: agent.Provider,
ProviderName: providercatalog.DisplayName(agent.Provider), ProviderName: localizedAgentProviderName(agent.Provider),
Model: agent.Model, Model: agent.Model,
APIType: agent.APIType, APIType: agent.APIType,
BaseURL: agent.BaseURL, BaseURL: agent.BaseURL,
@@ -420,8 +423,8 @@ func buildAgentItem(agent *model.Agent, appInstall *model.AppInstall, envMap map
item.BridgePort = toInt(bridge) item.BridgePort = toInt(bridge)
} }
} }
if agentType == constant.AppHermesAgent { if _, _, ok := agentDashboardAuthEnvKeys(agentType); ok {
auth := readHermesDashboardAuthFromInstall(appInstall) auth := readAgentDashboardAuthFromInstall(appInstall, agentType)
item.DashboardUsername = auth.Username item.DashboardUsername = auth.Username
item.DashboardPassword = auth.Password item.DashboardPassword = auth.Password
} }
@@ -429,6 +432,15 @@ func buildAgentItem(agent *model.Agent, appInstall *model.AppInstall, envMap map
return item return item
} }
func localizedAgentProviderName(provider string) string {
if key := providercatalog.DisplayNameKey(provider); key != "" {
if name := strings.TrimSpace(i18n.GetMsgByKey(key)); name != "" {
return name
}
}
return providercatalog.DisplayName(provider)
}
func isAgentAppKey(appKey string) bool { func isAgentAppKey(appKey string) bool {
return appKey == constant.AppOpenclaw || appKey == constant.AppCopaw || appKey == constant.AppHermesAgent return appKey == constant.AppOpenclaw || appKey == constant.AppCopaw || appKey == constant.AppHermesAgent
} }
@@ -726,9 +738,11 @@ type modelProvider struct {
} }
type modelEntry struct { type modelEntry struct {
ID string `json:"id"` ID string `json:"id"`
Name string `json:"name"` Name string `json:"name"`
Input []string `json:"input,omitempty"` Input []string `json:"input,omitempty"`
ContextWindow int `json:"contextWindow,omitempty"`
MaxTokens int `json:"maxTokens,omitempty"`
} }
func requiresOpenclawProviderModels(provider string) bool { func requiresOpenclawProviderModels(provider string) bool {
@@ -756,7 +770,7 @@ type browserConfig struct {
DefaultProfile string `json:"defaultProfile"` DefaultProfile string `json:"defaultProfile"`
} }
func writeOpenclawConfig(confDir string, account *model.AgentAccount, modelName, token string, allowedOrigins []string, fallbacks []string) error { func writeOpenclawConfig(confDir string, account *model.AgentAccount, modelName, token string, allowedOrigins []string, fallbacks []string, metadata []dto.AgentModelMetadata) error {
if strings.TrimSpace(confDir) == "" { if strings.TrimSpace(confDir) == "" {
return fmt.Errorf("config dir is required") return fmt.Errorf("config dir is required")
} }
@@ -841,6 +855,7 @@ func writeOpenclawConfig(confDir string, account *model.AgentAccount, modelName,
} }
conf = initial conf = initial
} else { } else {
preserveOpenclawModelMetadata(conf, cfg.Models)
if err := applyOpenclawModelsConfig(conf, cfg.Models); err != nil { if err := applyOpenclawModelsConfig(conf, cfg.Models); err != nil {
return err return err
} }
@@ -895,6 +910,9 @@ func writeOpenclawConfig(confDir string, account *model.AgentAccount, modelName,
if allowedOrigins != nil { if allowedOrigins != nil {
setSecurityConfig(conf, dto.AgentSecurityConfig{AllowedOrigins: allowedOrigins}) setSecurityConfig(conf, dto.AgentSecurityConfig{AllowedOrigins: allowedOrigins})
} }
if err := applyOpenclawModelMetadata(conf, account, metadata); err != nil {
return err
}
if err := writeOpenclawConfigRaw(configPath, conf); err != nil { if err := writeOpenclawConfigRaw(configPath, conf); err != nil {
return err return err
} }
@@ -909,6 +927,144 @@ func writeOpenclawConfig(confDir string, account *model.AgentAccount, modelName,
return writeAgentEnvMap(path.Join(confDir, ".env"), envMap, order) return writeAgentEnvMap(path.Join(confDir, ".env"), envMap, order)
} }
func readOpenclawModelsConfig(conf map[string]interface{}) *modelsConfig {
raw, ok := conf["models"]
if !ok {
return nil
}
payload, err := json.Marshal(raw)
if err != nil {
return nil
}
var models modelsConfig
if err := json.Unmarshal(payload, &models); err != nil {
return nil
}
return &models
}
func preserveOpenclawModelMetadata(conf map[string]interface{}, next *modelsConfig) {
current := readOpenclawModelsConfig(conf)
if current == nil || next == nil {
return
}
for providerID, nextProvider := range next.Providers {
currentProvider, ok := current.Providers[providerID]
if !ok {
continue
}
byID := make(map[string]modelEntry, len(currentProvider.Models))
for _, entry := range currentProvider.Models {
byID[entry.ID] = entry
}
for index := range nextProvider.Models {
currentEntry, ok := byID[nextProvider.Models[index].ID]
if !ok {
continue
}
nextProvider.Models[index].Input = currentEntry.Input
nextProvider.Models[index].ContextWindow = currentEntry.ContextWindow
nextProvider.Models[index].MaxTokens = currentEntry.MaxTokens
}
next.Providers[providerID] = nextProvider
}
}
func extractOpenclawModelMetadata(conf map[string]interface{}, account *model.AgentAccount, accountModels []dto.AgentAccountModel) []dto.AgentModelMetadata {
result := make([]dto.AgentModelMetadata, 0, len(accountModels))
configured := readOpenclawModelsConfig(conf)
for _, item := range accountModels {
_, inferred, providerID, _, err := buildOpenclawAccountModelConfig(account, item)
if err != nil {
continue
}
metadata := dto.AgentModelMetadata{Model: item.ID, InputMode: "auto"}
if configured != nil {
for _, entry := range configured.Providers[providerID].Models {
if entry.ID != inferred.ID {
continue
}
metadata.ContextWindow = entry.ContextWindow
metadata.MaxTokens = entry.MaxTokens
if len(entry.Input) > 0 && !slices.Equal(entry.Input, inferred.Input) {
if slices.Contains(entry.Input, "image") {
metadata.InputMode = "image"
} else {
metadata.InputMode = "text"
}
}
break
}
}
result = append(result, metadata)
}
return result
}
func applyOpenclawModelMetadata(conf map[string]interface{}, account *model.AgentAccount, requested []dto.AgentModelMetadata) error {
if len(requested) == 0 {
return nil
}
configured := readOpenclawModelsConfig(conf)
if configured == nil {
return fmt.Errorf("model metadata is not supported for provider %s", account.Provider)
}
accountModels, err := loadAgentAccountModels(account)
if err != nil {
return err
}
available := make(map[string]dto.AgentAccountModel, len(accountModels))
for _, item := range accountModels {
available[item.ID] = item
}
seen := make(map[string]struct{}, len(requested))
for _, metadata := range requested {
item, ok := available[metadata.Model]
if !ok {
return buserr.New("ErrAgentModelNotInAccount")
}
if _, ok := seen[metadata.Model]; ok {
return fmt.Errorf("duplicate model metadata: %s", metadata.Model)
}
seen[metadata.Model] = struct{}{}
_, inferred, providerID, _, err := buildOpenclawAccountModelConfig(account, item)
if err != nil {
return err
}
provider := configured.Providers[providerID]
found := false
for index := range provider.Models {
if provider.Models[index].ID != inferred.ID {
continue
}
found = true
provider.Models[index].ContextWindow = metadata.ContextWindow
provider.Models[index].MaxTokens = metadata.MaxTokens
switch metadata.InputMode {
case "auto":
provider.Models[index].Input = inferred.Input
case "text":
provider.Models[index].Input = []string{"text"}
case "image":
provider.Models[index].Input = []string{"text", "image"}
default:
return fmt.Errorf("unsupported model input mode: %s", metadata.InputMode)
}
break
}
if !found {
return buserr.New("ErrAgentModelNotInAccount")
}
configured.Providers[providerID] = provider
}
modelsMap, err := structToMap(configured)
if err != nil {
return err
}
conf["models"] = modelsMap
return nil
}
func resolveOpenclawFallbackModels(account *model.AgentAccount, primaryModel string, fallbackIDs []string) ([]string, error) { func resolveOpenclawFallbackModels(account *model.AgentAccount, primaryModel string, fallbackIDs []string) ([]string, error) {
accountModels, err := loadAgentAccountModels(account) accountModels, err := loadAgentAccountModels(account)
if err != nil { if err != nil {
@@ -1030,7 +1186,7 @@ func prepareOpenclawInstallFiles(appInstall *model.AppInstall, account *model.Ag
return fmt.Errorf("app install is required") return fmt.Errorf("app install is required")
} }
confDir := path.Join(appInstall.GetPath(), "data", "conf") confDir := path.Join(appInstall.GetPath(), "data", "conf")
if err := writeOpenclawConfig(confDir, account, modelName, token, allowedOrigins, nil); err != nil { if err := writeOpenclawConfig(confDir, account, modelName, token, allowedOrigins, nil, nil); err != nil {
return err return err
} }
dataDir := path.Join(appInstall.GetPath(), "data") dataDir := path.Join(appInstall.GetPath(), "data")
@@ -1159,15 +1315,15 @@ func buildInitialAgentAccountModels(account *model.AgentAccount, requested []dto
if len(requested) > 0 { if len(requested) > 0 {
return normalizeAgentAccountModels(account, requested) return normalizeAgentAccountModels(account, requested)
} }
meta, ok := providercatalog.Get(account.Provider) defaultModels := providercatalog.DefaultModels(account.Provider, account.APIType)
if !ok || len(meta.Models) == 0 { if len(defaultModels) == 0 {
if requiresInitialAgentAccountModels(account.Provider) { if requiresInitialAgentAccountModels(account.Provider) {
return nil, buserr.New("ErrAgentAccountModelsRequired") return nil, buserr.New("ErrAgentAccountModelsRequired")
} }
return nil, nil return nil, nil
} }
requested = make([]dto.AgentAccountModel, 0, len(meta.Models)) requested = make([]dto.AgentAccountModel, 0, len(defaultModels))
for _, item := range meta.Models { for _, item := range defaultModels {
requested = append(requested, dto.AgentAccountModel{ requested = append(requested, dto.AgentAccountModel{
ID: item.ID, ID: item.ID,
Name: item.Name, Name: item.Name,
@@ -1327,7 +1483,7 @@ func normalizeAgentAccountModel(account *model.AgentAccount, model dto.AgentAcco
func requiresInitialAgentAccountModels(provider string) bool { func requiresInitialAgentAccountModels(provider string) bool {
switch provider { switch provider {
case "custom", "vllm", "ollama": case "custom", "vllm", "ollama", "llmman":
return true return true
default: default:
return false return false
@@ -1452,6 +1608,87 @@ func readInstallEnv(envStr string) map[string]interface{} {
return data return data
} }
const (
hermesDashboardUsernameEnvKey = "HERMES_DASHBOARD_USERNAME"
hermesDashboardPasswordEnvKey = "HERMES_DASHBOARD_PASSWORD"
qwenPawAuthEnabledEnvKey = "QWENPAW_AUTH_ENABLED"
qwenPawAuthUsernameEnvKey = "QWENPAW_AUTH_USERNAME"
qwenPawAuthPasswordEnvKey = "QWENPAW_AUTH_PASSWORD"
)
type agentDashboardAuth struct {
Username string
Password string
}
func normalizeAgentDashboardAuth(username, password string) agentDashboardAuth {
auth := agentDashboardAuth{
Username: strings.TrimSpace(username),
Password: strings.TrimSpace(password),
}
if auth.Username == "" {
auth.Username = "admin"
}
if auth.Password == "" {
auth.Password = common.RandStr(8)
}
return auth
}
func agentDashboardAuthEnvKeys(agentType string) (string, string, bool) {
switch agentType {
case constant.AppHermesAgent:
return hermesDashboardUsernameEnvKey, hermesDashboardPasswordEnvKey, true
case constant.AppCopaw:
return qwenPawAuthUsernameEnvKey, qwenPawAuthPasswordEnvKey, true
default:
return "", "", false
}
}
func writeAgentDashboardAuthEnv(envPath, agentType string, auth agentDashboardAuth, overwrite bool) error {
usernameKey, passwordKey, ok := agentDashboardAuthEnvKeys(agentType)
if !ok {
return fmt.Errorf("dashboard auth is not supported for %s", agentType)
}
values := map[string]string{
usernameKey: auth.Username,
passwordKey: auth.Password,
}
order := []string{usernameKey, passwordKey}
if agentType == constant.AppCopaw {
values[qwenPawAuthEnabledEnvKey] = "true"
order = append([]string{qwenPawAuthEnabledEnvKey}, order...)
}
return upsertAgentEnv(envPath, values, order, overwrite)
}
func readAgentDashboardAuthEnv(envPath, agentType string) (agentDashboardAuth, error) {
usernameKey, passwordKey, ok := agentDashboardAuthEnvKeys(agentType)
if !ok {
return agentDashboardAuth{}, fmt.Errorf("dashboard auth is not supported for %s", agentType)
}
envMap, err := readAgentEnvMap(envPath)
if err != nil {
return agentDashboardAuth{}, err
}
return agentDashboardAuth{
Username: strings.TrimSpace(envMap[usernameKey]),
Password: strings.TrimSpace(envMap[passwordKey]),
}, nil
}
func readAgentDashboardAuthFromInstall(appInstall *model.AppInstall, agentType string) agentDashboardAuth {
if appInstall == nil || appInstall.ID == 0 {
return agentDashboardAuth{}
}
auth, err := readAgentDashboardAuthEnv(appInstall.GetEnvPath(), agentType)
if err != nil {
return agentDashboardAuth{}
}
return auth
}
func readAgentEnvMap(envPath string) (map[string]string, error) { func readAgentEnvMap(envPath string) (map[string]string, error) {
fileOp := files.NewFileOp() fileOp := files.NewFileOp()
if !fileOp.Stat(envPath) { if !fileOp.Stat(envPath) {
+434 -146
View File
@@ -4,11 +4,8 @@ import (
"encoding/json" "encoding/json"
"fmt" "fmt"
"mime" "mime"
"sort"
"strconv" "strconv"
"strings" "strings"
"sync"
"time"
"github.com/1Panel-dev/1Panel/agent/app/dto" "github.com/1Panel-dev/1Panel/agent/app/dto"
"github.com/1Panel-dev/1Panel/agent/app/model" "github.com/1Panel-dev/1Panel/agent/app/model"
@@ -17,11 +14,13 @@ import (
"github.com/1Panel-dev/1Panel/agent/constant" "github.com/1Panel-dev/1Panel/agent/constant"
"github.com/1Panel-dev/1Panel/agent/global" "github.com/1Panel-dev/1Panel/agent/global"
"github.com/1Panel-dev/1Panel/agent/i18n" "github.com/1Panel-dev/1Panel/agent/i18n"
"github.com/1Panel-dev/1Panel/agent/utils/cmd" alertUtil "github.com/1Panel-dev/1Panel/agent/utils/alert"
alertconfig "github.com/1Panel-dev/1Panel/agent/utils/alert_config"
alertwebhook "github.com/1Panel-dev/1Panel/agent/utils/alert_webhook"
"github.com/1Panel-dev/1Panel/agent/utils/copier" "github.com/1Panel-dev/1Panel/agent/utils/copier"
"github.com/1Panel-dev/1Panel/agent/utils/email" "github.com/1Panel-dev/1Panel/agent/utils/email"
"github.com/1Panel-dev/1Panel/agent/utils/xpack" "github.com/1Panel-dev/1Panel/agent/utils/xpack"
"github.com/shirou/gopsutil/v4/disk" "github.com/1Panel-dev/1Panel/agent/utils/xpack/providers"
) )
type AlertService struct{} type AlertService struct{}
@@ -34,6 +33,28 @@ var communityAlertMethodTypeNames = map[string]string{
constant.SMS: "SMS", constant.SMS: "SMS",
} }
var legacyAlertMethodTypeMap = map[string]string{
"mail": constant.Email,
constant.Email: constant.Email,
constant.SMS: constant.SMS,
constant.Bark: constant.Bark,
constant.WeChat: constant.WeCom,
constant.WeCom: constant.WeCom,
constant.DingTalk: constant.DingTalk,
constant.FeiShu: constant.FeiShu,
constant.Custom: constant.Custom,
}
var supportedAlertMethodTypes = map[string]struct{}{
constant.Email: {},
constant.SMS: {},
constant.Bark: {},
constant.WeCom: {},
constant.DingTalk: {},
constant.FeiShu: {},
constant.Custom: {},
}
type IAlertService interface { type IAlertService interface {
PageAlert(req dto.AlertSearch) (int64, []dto.AlertDTO, error) PageAlert(req dto.AlertSearch) (int64, []dto.AlertDTO, error)
GetAlerts() ([]dto.AlertDTO, error) GetAlerts() ([]dto.AlertDTO, error)
@@ -53,8 +74,10 @@ type IAlertService interface {
GetAlertConfig(req dto.AlertConfigQuery) ([]model.AlertConfig, error) GetAlertConfig(req dto.AlertConfigQuery) ([]model.AlertConfig, error)
PageAlertConfig(req dto.AlertConfigPageReq) (int64, []model.AlertConfig, error) PageAlertConfig(req dto.AlertConfigPageReq) (int64, []model.AlertConfig, error)
UpdateAlertConfig(req dto.AlertConfigUpdate, operator string) error UpdateAlertConfig(req dto.AlertConfigUpdate, operator string) error
UpdateAlertConfigStatus(req dto.AlertConfigStatusUpdate, operator string) error
DeleteAlertConfig(id uint) error DeleteAlertConfig(id uint) error
TestAlertConfig(req dto.AlertConfigTest) (bool, error) TestAlertConfig(req dto.AlertConfigTest) (bool, error)
TestCustomAlertConfig(req dto.AlertConfigTest) (dto.AlertConfigTestResult, error)
} }
func NewIAlertService() IAlertService { func NewIAlertService() IAlertService {
@@ -82,7 +105,38 @@ func (a AlertService) PageAlert(search dto.AlertSearch) (int64, []dto.AlertDTO,
return 0, nil, err return 0, nil, err
} }
cronjobProjects := make(map[string]uint)
var cronjobIDs []uint
for _, item := range alerts { for _, item := range alerts {
if alertUtil.GetCronJobType(item.Type) != "cronJob" {
continue
}
if _, exists := cronjobProjects[item.Project]; exists {
continue
}
id, parseErr := strconv.ParseUint(item.Project, 10, strconv.IntSize)
if parseErr != nil || id == 0 {
continue
}
cronjobProjects[item.Project] = uint(id)
cronjobIDs = append(cronjobIDs, uint(id))
}
cronjobsByID := make(map[uint]model.Cronjob)
if len(cronjobIDs) > 0 {
cronjobs, err := cronjobRepo.List(repo.WithByIDs(cronjobIDs))
if err != nil {
return 0, nil, err
}
for _, cronjob := range cronjobs {
cronjobsByID[cronjob.ID] = cronjob
}
}
for _, item := range alerts {
var taskName string
if cronjob, exists := cronjobsByID[cronjobProjects[item.Project]]; exists && cronjob.Type == item.Type {
taskName = cronjob.Name
}
result = append(result, dto.AlertDTO{ result = append(result, dto.AlertDTO{
ID: item.ID, ID: item.ID,
@@ -92,6 +146,7 @@ func (a AlertService) PageAlert(search dto.AlertSearch) (int64, []dto.AlertDTO,
Method: item.Method, Method: item.Method,
Title: item.Title, Title: item.Title,
Project: item.Project, Project: item.Project,
TaskName: taskName,
Status: item.Status, Status: item.Status,
SendCount: item.SendCount, SendCount: item.SendCount,
AdvancedParams: item.AdvancedParams, AdvancedParams: item.AdvancedParams,
@@ -163,6 +218,16 @@ func (a AlertService) CreateAlert(create dto.AlertCreate, operator string) error
return err return err
} }
} else { } else {
advanced, err := prepareCronJobAlertParams(create.Type, "", create.AdvancedParams)
if err != nil {
return err
}
create.AdvancedParams = advanced
if create.Status != constant.AlertDisable {
if err := a.validateCronJobAlertChannels(create.Type, advanced, create.Method); err != nil {
return err
}
}
alertInfo.Status = constant.AlertEnable alertInfo.Status = constant.AlertEnable
if err := copier.Copy(&alertInfo, &create); err != nil { if err := copier.Copy(&alertInfo, &create); err != nil {
return buserr.WithErr("ErrStructTransform", err) return buserr.WithErr("ErrStructTransform", err)
@@ -180,9 +245,28 @@ func (a AlertService) CreateAlert(create dto.AlertCreate, operator string) error
} }
func (a AlertService) UpdateAlert(req dto.AlertUpdate, operator string) error { func (a AlertService) UpdateAlert(req dto.AlertUpdate, operator string) error {
if err := a.validateCommunityAlertMethod(req.Method); err != nil { if alertUtil.GetCronJobType(req.Type) == "cronJob" {
previous, err := alertRepo.Get(repo.WithByID(req.ID))
if err != nil {
return err
}
req.AdvancedParams, err = prepareCronJobAlertParams(req.Type, previous.AdvancedParams, req.AdvancedParams)
if err != nil {
return err
}
}
methodTypes, err := a.validateAlertMethodReferences(req.Method)
if err != nil {
return err return err
} }
if req.Status != constant.AlertDisable {
if err := a.validateCronJobAlertChannels(req.Type, req.AdvancedParams, req.Method); err != nil {
return err
}
if err := a.validateAlertMethodEntitlement(methodTypes); err != nil {
return err
}
}
upMap := make(map[string]interface{}) upMap := make(map[string]interface{})
upMap["id"] = req.ID upMap["id"] = req.ID
@@ -240,7 +324,19 @@ func (a AlertService) UpdateStatus(id uint, status string) error {
if alertInfo.ID == 0 { if alertInfo.ID == 0 {
return buserr.New("ErrRecordNotFound") return buserr.New("ErrRecordNotFound")
} }
err := alertRepo.Update(map[string]interface{}{"status": status}, repo.WithByID(alertInfo.ID)) methodTypes, err := a.validateAlertMethodReferences(alertInfo.Method)
if err != nil {
return err
}
if status == constant.AlertEnable {
if err := a.validateCronJobAlertChannels(alertInfo.Type, alertInfo.AdvancedParams, alertInfo.Method); err != nil {
return err
}
if err := a.validateAlertMethodEntitlement(methodTypes); err != nil {
return err
}
}
err = alertRepo.Update(map[string]interface{}{"status": status}, repo.WithByID(alertInfo.ID))
if err != nil { if err != nil {
return err return err
} }
@@ -257,122 +353,14 @@ func (a AlertService) UpdateStatus(id uint, status string) error {
} }
func (a AlertService) GetDisks() ([]dto.DiskDTO, error) { func (a AlertService) GetDisks() ([]dto.DiskDTO, error) {
var disks []dto.DiskDTO infos := loadDiskInfo(true)
excludes := map[string]struct{}{ disks := make([]dto.DiskDTO, 0, len(infos))
"/mnt/cdrom": {}, "/boot": {}, "/boot/efi": {}, "/dev": {}, "/dev/shm": {}, for _, item := range infos {
"/run/lock": {}, "/run": {}, "/run/shm": {}, "/run/user": {}, disks = append(disks, dto.DiskDTO(item))
} }
stdout, err := executeDiskCommand()
if err != nil {
return disks, nil
}
lines := strings.Split(stdout, "\n")
var mounts []dto.AlertDiskInfo
for _, line := range lines {
fields := strings.Fields(line)
if len(fields) < 7 {
continue
}
mountPoint := strings.Join(fields[6:], " ")
if shouldExclude(fields, mountPoint, excludes) {
continue
}
mounts = append(mounts, dto.AlertDiskInfo{Type: fields[1], Device: fields[0], Mount: mountPoint})
}
var (
wg sync.WaitGroup
mu sync.Mutex
)
wg.Add(len(mounts))
for i := 0; i < len(mounts); i++ {
go func(timeoutCh <-chan time.Time, mount dto.AlertDiskInfo) {
defer wg.Done()
var itemData dto.DiskDTO
itemData.Path = mount.Mount
itemData.Type = mount.Type
itemData.Device = mount.Device
select {
case <-timeoutCh:
mu.Lock()
disks = append(disks, itemData)
mu.Unlock()
global.LOG.Errorf("load disk info from %s failed, err: timeout", mount.Mount)
default:
state, err := disk.Usage(mount.Mount)
if err != nil {
mu.Lock()
disks = append(disks, itemData)
mu.Unlock()
global.LOG.Errorf("load disk info from %s failed, err: %v", mount.Mount, err)
return
}
itemData.Total = state.Total
itemData.Free = state.Free
itemData.Used = state.Used
itemData.UsedPercent = state.UsedPercent
itemData.InodesTotal = state.InodesTotal
itemData.InodesUsed = state.InodesUsed
itemData.InodesFree = state.InodesFree
itemData.InodesUsedPercent = state.InodesUsedPercent
mu.Lock()
disks = append(disks, itemData)
mu.Unlock()
}
}(time.After(5*time.Second), mounts[i])
}
wg.Wait()
sort.Slice(disks, func(i, j int) bool {
return disks[i].Path < disks[j].Path
})
return disks, nil return disks, nil
} }
func executeDiskCommand() (string, error) {
cmdMgr := cmd.NewCommandMgr(cmd.WithTimeout(2 * time.Second))
stdout, err := cmdMgr.RunWithStdout("df", "-hT", "-P")
if err != nil {
cmdMgr2 := cmd.NewCommandMgr(cmd.WithTimeout(1 * time.Second))
stdout, err = cmdMgr2.RunWithStdout("df", "-lhT", "-P")
}
if err != nil {
return stdout, err
}
var lines []string
for _, line := range strings.Split(stdout, "\n") {
if !strings.Contains(line, "/") || strings.Contains(line, "tmpfs") || strings.Contains(line, "snap/core") || strings.Contains(line, "udev") {
continue
}
lines = append(lines, line)
}
if len(lines) == 0 {
return "", nil
}
return strings.Join(lines, "\n"), nil
}
func shouldExclude(fields []string, mountPoint string, excludes map[string]struct{}) bool {
if strings.HasPrefix(mountPoint, "/snap") || len(strings.Split(mountPoint, "/")) > 10 {
return true
}
if strings.TrimSpace(fields[1]) == "tmpfs" {
return true
}
if strings.Contains(fields[2], "K") {
return true
}
if strings.Contains(mountPoint, "docker") {
return true
}
_, excluded := excludes[mountPoint]
return excluded
}
func (a AlertService) PageAlertLogs(search dto.AlertLogSearch) (int64, []dto.AlertLogDTO, error) { func (a AlertService) PageAlertLogs(search dto.AlertLogSearch) (int64, []dto.AlertLogDTO, error) {
var ( var (
opts []repo.DBOption opts []repo.DBOption
@@ -384,6 +372,9 @@ func (a AlertService) PageAlertLogs(search dto.AlertLogSearch) (int64, []dto.Ale
if search.Count != 0 { if search.Count != 0 {
opts = append(opts, alertRepo.WithByCount(search.Count)) opts = append(opts, alertRepo.WithByCount(search.Count))
} }
if !search.StartTime.IsZero() && !search.EndTime.IsZero() {
opts = append(opts, repo.WithByCreatedAt(search.StartTime, search.EndTime))
}
opts = append(opts, repo.WithOrderDesc("created_at")) opts = append(opts, repo.WithOrderDesc("created_at"))
total, alerts, err := alertRepo.PageLog(search.Page, search.PageSize, opts...) total, alerts, err := alertRepo.PageLog(search.Page, search.PageSize, opts...)
@@ -409,6 +400,7 @@ func (a AlertService) parseAlertLog(item model.AlertLog) (dto.AlertLogDTO, error
if err := unmarshalAlertInfo(item.AlertDetail, &alertDetail); err != nil { if err := unmarshalAlertInfo(item.AlertDetail, &alertDetail); err != nil {
return dto.AlertLogDTO{}, err return dto.AlertLogDTO{}, err
} }
alertDetail.Task = nil
if err := unmarshalAlertInfo(item.AlertRule, &alertRule); err != nil { if err := unmarshalAlertInfo(item.AlertRule, &alertRule); err != nil {
return dto.AlertLogDTO{}, err return dto.AlertLogDTO{}, err
} }
@@ -491,7 +483,13 @@ func (a AlertService) GetAlertConfig(req dto.AlertConfigQuery) ([]model.AlertCon
} }
opts = append(opts, repo.WithByStatus(constant.AlertEnable)) opts = append(opts, repo.WithByStatus(constant.AlertEnable))
configs, err := alertRepo.AlertConfigList(opts...) configs, err := alertRepo.AlertConfigList(opts...)
return configs, err if err != nil {
return nil, err
}
if err := exposeCustomAlertConfigSecrets(configs); err != nil {
return nil, err
}
return configs, nil
} }
func (a AlertService) PageAlertConfig(req dto.AlertConfigPageReq) (int64, []model.AlertConfig, error) { func (a AlertService) PageAlertConfig(req dto.AlertConfigPageReq) (int64, []model.AlertConfig, error) {
@@ -502,13 +500,49 @@ func (a AlertService) PageAlertConfig(req dto.AlertConfigPageReq) (int64, []mode
if len(req.ExcludeTypes) > 0 { if len(req.ExcludeTypes) > 0 {
opts = append(opts, alertRepo.WithByTypeNotIn(req.ExcludeTypes)) opts = append(opts, alertRepo.WithByTypeNotIn(req.ExcludeTypes))
} }
return alertRepo.PageAlertConfig(req.Page, req.PageSize, opts...) total, configs, err := alertRepo.PageAlertConfig(req.Page, req.PageSize, opts...)
if err != nil {
return 0, nil, err
}
if err := exposeCustomAlertConfigSecrets(configs); err != nil {
return 0, nil, err
}
return total, configs, nil
} }
func (a AlertService) UpdateAlertConfig(req dto.AlertConfigUpdate, operator string) error { func (a AlertService) UpdateAlertConfig(req dto.AlertConfigUpdate, operator string) error {
if req.Type == constant.Custom {
if req.ID != 0 && req.Revision == nil {
return repo.ErrAlertConfigRevisionRequired
}
return a.updateCustomAlertConfig(req, operator)
}
usesMutation, err := alertconfig.UsesMutation(req.Type, req.Config)
if err != nil {
return err
}
if req.ID != 0 && usesMutation && req.Revision == nil {
return repo.ErrAlertConfigRevisionRequired
}
var existing *model.AlertConfig
if req.ID != 0 {
stored, err := alertRepo.GetConfigById(req.ID)
if err != nil {
return err
}
if stored.Type != req.Type {
return fmt.Errorf("alert config %d has type %s, not %s", req.ID, stored.Type, req.Type)
}
existing = &stored
}
if err := a.validateCommunityAlertConfigType(req.Type); err != nil { if err := a.validateCommunityAlertConfigType(req.Type); err != nil {
return err return err
} }
prepared, err := alertconfig.Prepare(req.Type, req.Config, req.Status, existing)
if err != nil {
return err
}
req.Config = prepared
if err := a.checkAlertConfigDisplayNameUnique(req); err != nil { if err := a.checkAlertConfigDisplayNameUnique(req); err != nil {
return err return err
} }
@@ -523,7 +557,7 @@ func (a AlertService) UpdateAlertConfig(req dto.AlertConfigUpdate, operator stri
upMap["status"] = req.Status upMap["status"] = req.Status
upMap["config"] = req.Config upMap["config"] = req.Config
upMap["update_user"] = operator upMap["update_user"] = operator
if err := alertRepo.UpdateAlertConfig(upMap, repo.WithByID(req.ID)); err != nil { if err := alertRepo.UpdateAlertConfigWithRevision(upMap, req.Revision, repo.WithByID(req.ID)); err != nil {
return err return err
} }
} else { } else {
@@ -541,6 +575,99 @@ func (a AlertService) UpdateAlertConfig(req dto.AlertConfigUpdate, operator stri
return nil return nil
} }
func (a AlertService) updateCustomAlertConfig(req dto.AlertConfigUpdate, operator string) error {
if err := validateAlertConfigStatus(req.Status); err != nil {
return err
}
var existing *model.AlertConfig
if req.ID != 0 {
config, err := alertRepo.GetConfigById(req.ID)
if err != nil {
return err
}
if config.Type != constant.Custom {
return fmt.Errorf("alert config %d is not a custom webhook", req.ID)
}
existing = &config
}
prepared, err := alertwebhook.Prepare(req.Config, req.Status, existing)
if err != nil {
return err
}
validatedReq := req
validatedReq.Config = prepared.Config
if err := a.checkAlertConfigDisplayNameUnique(validatedReq); err != nil {
return err
}
if existing != nil {
return alertRepo.UpdateAlertConfigWithRevision(map[string]interface{}{
"type": constant.Custom,
"title": req.Title,
"status": req.Status,
"config": prepared.Config,
"secret_config": prepared.SecretConfig,
"update_user": operator,
}, req.Revision, repo.WithByID(req.ID))
}
return alertRepo.CreateAlertConfig(&model.AlertConfig{
Type: constant.Custom,
Title: req.Title,
Status: req.Status,
Config: prepared.Config,
SecretConfig: prepared.SecretConfig,
CreateUser: operator,
UpdateUser: operator,
})
}
func (a AlertService) UpdateAlertConfigStatus(req dto.AlertConfigStatusUpdate, operator string) error {
if err := validateAlertConfigStatus(req.Status); err != nil {
return err
}
config, err := alertRepo.GetConfigById(req.ID)
if err != nil {
return err
}
if req.Status == constant.AlertEnable {
if err := a.validateCommunityAlertConfigType(config.Type); err != nil {
return err
}
if config.Type == constant.Custom {
if _, err := alertwebhook.Resolve(config); err != nil {
return err
}
}
}
return alertRepo.UpdateAlertConfig(map[string]interface{}{
"status": req.Status,
"update_user": operator,
}, repo.WithByID(req.ID))
}
func validateAlertConfigStatus(status string) error {
if status != constant.AlertEnable && status != constant.AlertDisable {
return fmt.Errorf("alert config status must be Enable or Disable")
}
return nil
}
func exposeCustomAlertConfigSecrets(configs []model.AlertConfig) error {
for index := range configs {
if configs[index].Type != constant.Custom {
continue
}
view, err := alertwebhook.PlainView(configs[index])
if err != nil {
return fmt.Errorf("build editable custom alert config %d: %w", configs[index].ID, err)
}
configs[index].Config = view
}
return nil
}
func (a AlertService) checkAlertConfigSMSPhoneUnique(req dto.AlertConfigUpdate) error { func (a AlertService) checkAlertConfigSMSPhoneUnique(req dto.AlertConfigUpdate) error {
if req.Type != constant.SMSConfig { if req.Type != constant.SMSConfig {
return nil return nil
@@ -565,6 +692,9 @@ func (a AlertService) checkAlertConfigSMSPhoneUnique(req dto.AlertConfigUpdate)
} }
func (a AlertService) checkAlertConfigDisplayNameUnique(req dto.AlertConfigUpdate) error { func (a AlertService) checkAlertConfigDisplayNameUnique(req dto.AlertConfigUpdate) error {
if req.Type != constant.Custom && (global.CONF.Base.IsEnterprise || global.CONF.Base.Edition == "cn") {
return nil
}
displayName := alertConfigDisplayName(req.Type, req.Config) displayName := alertConfigDisplayName(req.Type, req.Config)
if displayName == "" { if displayName == "" {
return nil return nil
@@ -588,37 +718,67 @@ func (a AlertService) checkAlertConfigDisplayNameUnique(req dto.AlertConfigUpdat
} }
func (a AlertService) validateCommunityAlertMethod(method string) error { func (a AlertService) validateCommunityAlertMethod(method string) error {
if global.CONF.Base.IsEnterprise || global.CONF.Base.Edition == "cn" { methodTypes, err := a.validateAlertMethodReferences(method)
return nil if err != nil {
} return err
if strings.TrimSpace(method) == "" {
return nil
} }
return a.validateAlertMethodEntitlement(methodTypes)
}
func (a AlertService) validateAlertMethodReferences(method string) ([]string, error) {
if strings.TrimSpace(method) == "" {
return nil, buserr.WithErr("ErrAlertMethodNotSupported", nil)
}
methodTypes := make([]string, 0)
for _, item := range strings.Split(method, ",") { for _, item := range strings.Split(method, ",") {
item = strings.TrimSpace(item) item = strings.TrimSpace(item)
if item == "" { if item == "" {
continue continue
} }
configType := ""
if configID, err := strconv.ParseUint(item, 10, 64); err == nil { if configID, err := strconv.ParseUint(item, 10, 64); err == nil {
config, err := alertRepo.GetConfigById(uint(configID)) config, err := alertRepo.GetConfigById(uint(configID))
if err != nil { if err != nil {
return err return nil, err
} }
if _, ok := communityAlertMethodTypeNames[config.Type]; ok { configType = config.Type
return buserr.WithErr("ErrAlertMethodNotSupported", nil) } else {
var ok bool
configType, ok = legacyAlertMethodTypeMap[item]
if !ok {
return nil, buserr.WithErr("ErrAlertMethodNotSupported", nil)
} }
}
if _, ok := supportedAlertMethodTypes[configType]; !ok {
return nil, buserr.WithErr("ErrAlertMethodNotSupported", nil)
}
methodTypes = append(methodTypes, configType)
}
if len(methodTypes) == 0 {
return nil, buserr.WithErr("ErrAlertMethodNotSupported", nil)
}
return methodTypes, nil
}
func (a AlertService) validateAlertMethodEntitlement(methodTypes []string) error {
for _, configType := range methodTypes {
if configType == constant.Custom {
continue continue
} }
if _, ok := communityAlertMethodTypeNames[item]; ok { if global.CONF.Base.IsEnterprise || global.CONF.Base.Edition == "cn" {
continue
}
if _, ok := communityAlertMethodTypeNames[configType]; ok {
return buserr.WithErr("ErrAlertMethodNotSupported", nil) return buserr.WithErr("ErrAlertMethodNotSupported", nil)
} }
} }
return nil return nil
} }
func (a AlertService) validateCommunityAlertConfigType(configType string) error { func (a AlertService) validateCommunityAlertConfigType(configType string) error {
if configType == constant.Custom {
return nil
}
if global.CONF.Base.IsEnterprise || global.CONF.Base.Edition == "cn" { if global.CONF.Base.IsEnterprise || global.CONF.Base.Edition == "cn" {
return nil return nil
} }
@@ -630,7 +790,7 @@ func (a AlertService) validateCommunityAlertConfigType(configType string) error
func alertConfigDisplayName(configType, configData string) string { func alertConfigDisplayName(configType, configData string) string {
switch configType { switch configType {
case constant.Email, constant.WeCom, constant.DingTalk, constant.FeiShu, constant.Bark, constant.SMS: case constant.Email, constant.WeCom, constant.DingTalk, constant.FeiShu, constant.Bark, constant.SMS, constant.Custom:
var cfg struct { var cfg struct {
DisplayName string `json:"displayName"` DisplayName string `json:"displayName"`
} }
@@ -669,20 +829,24 @@ func (a AlertService) DeleteAlertConfig(id uint) error {
} }
func (a AlertService) TestAlertConfig(req dto.AlertConfigTest) (bool, error) { func (a AlertService) TestAlertConfig(req dto.AlertConfigTest) (bool, error) {
username := req.UserName emailConfig, err := resolveEmailTestConfig(req)
if username == "" { if err != nil {
username = req.Sender return false, err
} }
encodedDisplayName := mime.BEncoding.Encode("UTF-8", req.DisplayName) username := emailConfig.UserName
if username == "" {
username = emailConfig.Sender
}
encodedDisplayName := mime.BEncoding.Encode("UTF-8", emailConfig.DisplayName)
cfg := email.SMTPConfig{ cfg := email.SMTPConfig{
Host: req.Host, Host: emailConfig.Host,
Port: req.Port, Port: emailConfig.Port,
Sender: req.Sender, Sender: emailConfig.Sender,
Username: username, Username: username,
Password: req.Password, Password: emailConfig.Password,
From: fmt.Sprintf(`"%s" <%s>`, encodedDisplayName, req.Sender), From: fmt.Sprintf(`"%s" <%s>`, encodedDisplayName, emailConfig.Sender),
Encryption: req.Encryption, Encryption: emailConfig.Encryption,
Recipient: req.Recipient, Recipient: emailConfig.Recipient,
} }
msg := email.EmailMessage{ msg := email.EmailMessage{
@@ -697,9 +861,94 @@ func (a AlertService) TestAlertConfig(req dto.AlertConfigTest) (bool, error) {
return true, nil return true, nil
} }
func resolveEmailTestConfig(req dto.AlertConfigTest) (dto.AlertEmailConfig, error) {
emailConfig := dto.AlertEmailConfig{
Host: req.Host,
Port: req.Port,
Sender: req.Sender,
UserName: req.UserName,
Password: req.Password,
DisplayName: req.DisplayName,
Encryption: req.Encryption,
Recipient: req.Recipient,
}
if strings.TrimSpace(req.Config) != "" {
configType := req.Type
if configType == "" {
configType = constant.EmailConfig
}
if configType != constant.EmailConfig {
return dto.AlertEmailConfig{}, fmt.Errorf("alert config test type must be email")
}
var existing *model.AlertConfig
if req.ID != 0 {
stored, err := alertRepo.GetConfigById(req.ID)
if err != nil {
return dto.AlertEmailConfig{}, err
}
existing = &stored
}
prepared, err := alertconfig.Prepare(configType, req.Config, constant.AlertEnable, existing)
if err != nil {
return dto.AlertEmailConfig{}, err
}
if err := json.Unmarshal([]byte(prepared), &emailConfig); err != nil {
return dto.AlertEmailConfig{}, fmt.Errorf("decode email alert config: %w", err)
}
}
return emailConfig, nil
}
func (a AlertService) TestCustomAlertConfig(req dto.AlertConfigTest) (dto.AlertConfigTestResult, error) {
if req.Type != constant.Custom {
return dto.AlertConfigTestResult{}, fmt.Errorf("alert config test type must be custom")
}
var existing *model.AlertConfig
if req.ID != 0 {
config, err := alertRepo.GetConfigById(req.ID)
if err != nil {
return dto.AlertConfigTestResult{}, err
}
if config.Type != constant.Custom {
return dto.AlertConfigTestResult{}, fmt.Errorf("alert config %d is not a custom webhook", req.ID)
}
existing = &config
}
prepared, err := alertwebhook.Prepare(req.Config, constant.AlertEnable, existing)
if err != nil {
return dto.AlertConfigTestResult{}, err
}
resolved, err := alertwebhook.Resolve(model.AlertConfig{
Type: constant.Custom,
Config: prepared.Config,
SecretConfig: prepared.SecretConfig,
})
if err != nil {
return dto.AlertConfigTestResult{}, err
}
tester, ok := xpack.AlertProvider.(providers.CustomWebhookTester)
if !ok {
return dto.AlertConfigTestResult{
Success: false,
Message: providers.ErrCustomWebhookUnsupported.Error(),
}, nil
}
return tester.TestCustomWebhook(resolved)
}
func (a AlertService) ExternalUpdateAlert(updateAlert dto.AlertCreate, operator string) error { func (a AlertService) ExternalUpdateAlert(updateAlert dto.AlertCreate, operator string) error {
if err := a.validateCommunityAlertMethod(updateAlert.Method); err != nil { var methodTypes []string
return err if updateAlert.SendCount != 0 || strings.TrimSpace(updateAlert.Method) != "" {
var err error
methodTypes, err = a.validateAlertMethodReferences(updateAlert.Method)
if err != nil {
return err
}
}
if updateAlert.SendCount != 0 {
if err := a.validateAlertMethodEntitlement(methodTypes); err != nil {
return err
}
} }
upMap := make(map[string]interface{}) upMap := make(map[string]interface{})
var newStatus string var newStatus string
@@ -718,6 +967,23 @@ func (a AlertService) ExternalUpdateAlert(updateAlert dto.AlertCreate, operator
alertRepo.WithByType(updateAlert.Type), alertRepo.WithByType(updateAlert.Type),
alertRepo.WithByProject(updateAlert.Project), alertRepo.WithByProject(updateAlert.Project),
) )
advanced, err := prepareCronJobAlertParams(updateAlert.Type, alertInfo.AdvancedParams, updateAlert.AdvancedParams)
if err != nil {
return err
}
updateAlert.AdvancedParams = advanced
if alertUtil.GetCronJobType(updateAlert.Type) == "cronJob" {
upMap["advanced_params"] = advanced
}
if newStatus == constant.AlertEnable {
method := updateAlert.Method
if method == "" {
method = alertInfo.Method
}
if err := a.validateCronJobAlertChannels(updateAlert.Type, advanced, method); err != nil {
return err
}
}
if alertInfo.ID > 0 { if alertInfo.ID > 0 {
shouldUpdate := false shouldUpdate := false
@@ -731,6 +997,9 @@ func (a AlertService) ExternalUpdateAlert(updateAlert dto.AlertCreate, operator
if val, ok := upMap["method"]; ok && val != "" && val != alertInfo.Method { if val, ok := upMap["method"]; ok && val != "" && val != alertInfo.Method {
shouldUpdate = true shouldUpdate = true
} }
if val, ok := upMap["advanced_params"]; ok && val != alertInfo.AdvancedParams {
shouldUpdate = true
}
if shouldUpdate { if shouldUpdate {
if err := alertRepo.Update( if err := alertRepo.Update(
@@ -752,3 +1021,22 @@ func (a AlertService) ExternalUpdateAlert(updateAlert dto.AlertCreate, operator
return nil return nil
} }
func prepareCronJobAlertParams(alertType, previous, incoming string) (string, error) {
if alertUtil.GetCronJobType(alertType) != "cronJob" {
return incoming, nil
}
return alertUtil.MergeCronJobAlertParams(previous, incoming)
}
func (a AlertService) validateCronJobAlertChannels(alertType, advanced, method string) error {
if alertUtil.GetCronJobType(alertType) != "cronJob" {
return nil
}
mode, err := alertUtil.CronJobAlertTriggerMode(advanced)
if err != nil || mode != alertUtil.CronJobAlertSuccess {
return err
}
_, err = a.validateAlertMethodReferences(method)
return err
}
+134 -53
View File
@@ -2,6 +2,7 @@ package service
import ( import (
"encoding/json" "encoding/json"
"errors"
"fmt" "fmt"
"math" "math"
"net" "net"
@@ -29,9 +30,11 @@ import (
) )
const ( const (
ResourceAlertInterval = 30 ResourceAlertInterval = 30
CheckIntervalSec = 3 CheckIntervalSec = 3
LoadCheckIntervalMin = 5 LoadCheckIntervalMin = 5
sshIPLoginWindow = 30 * time.Minute
sslAutoRenewAlertSkipDays = 31
) )
type AlertTaskHelper struct { type AlertTaskHelper struct {
@@ -512,10 +515,32 @@ func loadPanelLogin(alert dto.AlertDTO) {
} }
func loadSSHLogin(alert dto.AlertDTO) { func loadSSHLogin(alert dto.AlertDTO) {
count, isAlert, err := alertUtil.CountRecentFailedSSHLog(alert.Cycle, alert.Count) now := time.Now()
if err != nil { failedWindow := time.Duration(alert.Cycle) * time.Minute
global.LOG.Errorf("Failed to count recent failed ssh login logs: %v", err) loadWindow := failedWindow
if loadWindow < sshIPLoginWindow {
loadWindow = sshIPLoginWindow
} }
location, err := time.LoadLocation(common.LoadTimeZoneByCmd())
if err != nil {
global.LOG.Errorf("Failed to load timezone for ssh login logs: %v", err)
location = time.Local
}
histories, err := loadSSHAlertHistories(defaultSSHLogDir, now.Add(-loadWindow), now, location)
if err != nil {
global.LOG.Errorf("Failed to load ssh login logs: %v", err)
}
interfaceAddrs, err := net.InterfaceAddrs()
if err != nil {
global.LOG.Warnf("Failed to load local IP addresses for ssh login alert: %v", err)
}
count, records := summarizeSSHLoginHistories(
histories,
now,
failedWindow,
sshSuccessLoginWhitelist(alert.AdvancedParams, interfaceAddrs),
)
isAlert := count >= int(alert.Count)
if isAlert { if isAlert {
params := []dto.Param{ params := []dto.Param{
{ {
@@ -531,12 +556,6 @@ func loadSSHLogin(alert dto.AlertDTO) {
} }
sendAlerts(alert, "sshLogin", strconv.Itoa(count), "sshLogin", params) sendAlerts(alert, "sshLogin", strconv.Itoa(count), "sshLogin", params)
} }
whitelist := strings.Split(strings.TrimSpace(alert.AdvancedParams), "\n")
records, err := alertUtil.FindRecentSuccessLoginNotInWhitelist(30, whitelist)
if err != nil {
global.LOG.Errorf("Failed to check recent failed ip ssh login logs: %v", err)
}
records = filterSSHLoginEntriesNotInWhitelist(records, whitelist)
if len(records) > 0 { if len(records) > 0 {
quota := strings.Join(records, "\n") quota := strings.Join(records, "\n")
params := []dto.Param{ params := []dto.Param{
@@ -555,6 +574,19 @@ func loadSSHLogin(alert dto.AlertDTO) {
} }
} }
func sshSuccessLoginWhitelist(configured string, interfaceAddrs []net.Addr) []string {
whitelist := strings.Split(strings.TrimSpace(configured), "\n")
whitelist = append(whitelist, "127.0.0.0/8", "::1")
for _, addr := range interfaceAddrs {
ipNet, ok := addr.(*net.IPNet)
if !ok || ipNet.IP == nil || ipNet.IP.IsUnspecified() {
continue
}
whitelist = append(whitelist, ipNet.IP.String())
}
return whitelist
}
func filterLoginLogsNotInWhitelist(records []model.LoginLog, whitelist []string) []model.LoginLog { func filterLoginLogsNotInWhitelist(records []model.LoginLog, whitelist []string) []model.LoginLog {
filtered := make([]model.LoginLog, 0, len(records)) filtered := make([]model.LoginLog, 0, len(records))
for _, record := range records { for _, record := range records {
@@ -565,20 +597,6 @@ func filterLoginLogsNotInWhitelist(records []model.LoginLog, whitelist []string)
return filtered return filtered
} }
func filterSSHLoginEntriesNotInWhitelist(records []string, whitelist []string) []string {
filtered := make([]string, 0, len(records))
for _, record := range records {
ip := record
if idx := strings.Index(record, "-"); idx >= 0 {
ip = record[:idx]
}
if !isIPInWhitelist(ip, whitelist) {
filtered = append(filtered, record)
}
}
return filtered
}
func isIPInWhitelist(ip string, whitelist []string) bool { func isIPInWhitelist(ip string, whitelist []string) bool {
targetIP := net.ParseIP(strings.TrimSpace(ip)) targetIP := net.ParseIP(strings.TrimSpace(ip))
if targetIP == nil { if targetIP == nil {
@@ -695,9 +713,10 @@ func sendAlertsByConfigId(alert dto.AlertDTO, alertType, quota, quotaType string
func sendAlertsByLegacyMethod(alert dto.AlertDTO, alertType, quota, quotaType string, params []dto.Param, method string) { func sendAlertsByLegacyMethod(alert dto.AlertDTO, alertType, quota, quotaType string, params []dto.Param, method string) {
typeMap := map[string]string{ typeMap := map[string]string{
"mail": constant.Email, "mail": constant.Email,
constant.Bark: constant.Bark, constant.Bark: constant.Bark,
constant.SMS: constant.SMS, constant.SMS: constant.SMS,
constant.Custom: constant.Custom,
} }
configType, ok := typeMap[method] configType, ok := typeMap[method]
if !ok { if !ok {
@@ -785,7 +804,7 @@ func doSendAlert(alert dto.AlertDTO, alertType, quota, quotaType string, params
} }
alertUtil.CreateNewAlertTask(quota, alertType, quotaType, methodStr) alertUtil.CreateNewAlertTask(quota, alertType, quotaType, methodStr)
case constant.WeCom, constant.DingTalk, constant.FeiShu: case constant.WeCom, constant.DingTalk, constant.FeiShu, constant.Custom:
todayCount, isValid := canSendAlertToday(alertType, quotaType, alert.SendCount, methodStr) todayCount, isValid := canSendAlertToday(alertType, quotaType, alert.SendCount, methodStr)
if !isValid { if !isValid {
return return
@@ -798,12 +817,31 @@ func doSendAlert(alert dto.AlertDTO, alertType, quota, quotaType string, params
} }
transport := xpack.MultiNodeProvider.LoadRequestTransport() transport := xpack.MultiNodeProvider.LoadRequestTransport()
agentInfo, _ := xpack.MultiNodeProvider.GetAgentInfo() agentInfo, _ := xpack.MultiNodeProvider.GetAgentInfo()
alertErr := xpack.AlertProvider.CreateWebhookAlertLog(alertType, alert, create, quotaType, params, config, transport, agentInfo) queued := false
var alertErr error
if config.Type == constant.Custom {
task := dto.AlertTaskMetadata{
AlertID: alert.ID,
Type: alertType,
Quota: quota,
QuotaType: quotaType,
Method: methodStr,
}
result, deliveryErr := xpack.DeliverCustomWebhookAlertLog(alertType, alert, create, quotaType, params, config, transport, agentInfo, task)
queued, alertErr = result.Queued, deliveryErr
if alertErr == nil && result.Queued {
_, alertErr = alertUtil.RecordQueuedAlertTask(result.LogID, task)
}
} else {
alertErr = xpack.AlertProvider.CreateWebhookAlertLog(alertType, alert, create, quotaType, params, config, transport, agentInfo)
}
if alertErr != nil { if alertErr != nil {
global.LOG.Infof("%s alert webhook %s push faild, err: %v", alertType, methodStr, alertErr) global.LOG.Infof("%s alert webhook %s push faild, err: %v", alertType, methodStr, alertErr)
return return
} }
alertUtil.CreateNewAlertTask(quota, alertType, quotaType, methodStr) if !queued {
alertUtil.CreateNewAlertTask(quota, alertType, quotaType, methodStr)
}
} }
} }
@@ -844,7 +882,7 @@ func calculateSSLExpiryDays(sslList []model.WebsiteSSL, cycle uint) (map[int][]s
daysDiff := int(math.Ceil( daysDiff := int(math.Ceil(
ssl.ExpireDate.Sub(currentDate).Hours() / 24, ssl.ExpireDate.Sub(currentDate).Hours() / 24,
)) ))
if daysDiff > 0 && int(cycle) >= daysDiff { if daysDiff > 0 && int(cycle) >= daysDiff && !shouldSuppressSSLExpiryAlert(ssl, daysDiff) {
daysDiffMap[daysDiff] = append(daysDiffMap[daysDiff], ssl.PrimaryDomain) daysDiffMap[daysDiff] = append(daysDiffMap[daysDiff], ssl.PrimaryDomain)
projectMap[ssl.ID] = append(projectMap[ssl.ID], ssl.ExpireDate) projectMap[ssl.ID] = append(projectMap[ssl.ID], ssl.ExpireDate)
} }
@@ -852,6 +890,10 @@ func calculateSSLExpiryDays(sslList []model.WebsiteSSL, cycle uint) (map[int][]s
return daysDiffMap, projectMap return daysDiffMap, projectMap
} }
func shouldSuppressSSLExpiryAlert(ssl model.WebsiteSSL, remainingDays int) bool {
return ssl.AutoRenew && remainingDays < sslAutoRenewAlertSkipDays
}
func calculateWebsiteExpiryDays(websites []model.Website, cycle uint) (map[int][]string, map[uint][]time.Time) { func calculateWebsiteExpiryDays(websites []model.Website, cycle uint) (map[int][]string, map[uint][]time.Time) {
currentDate := time.Now() currentDate := time.Now()
daysDiffMap := make(map[int][]string) daysDiffMap := make(map[int][]string)
@@ -1024,50 +1066,37 @@ func processAllDisks(alert dto.AlertDTO) error {
global.LOG.Errorf("error getting disk list, err: %v", err) global.LOG.Errorf("error getting disk list, err: %v", err)
return err return err
} }
var errMsgs []string
for _, item := range diskList { for _, item := range diskList {
err := checkAndCreateDiskAlert(alert, item.Path) if item.Total == 0 {
if err != nil {
errMsg := fmt.Sprintf("disk path %s process failed: %v", item.Path, err)
errMsgs = append(errMsgs, errMsg)
global.LOG.Errorf("%s", errMsg)
continue continue
} }
} checkAndCreateDiskAlert(alert, item.Path, &disk.UsageStat{Used: item.Used, UsedPercent: item.UsedPercent})
if len(errMsgs) > 0 {
return fmt.Errorf("batch process disks failed, error count: %d, details: %s", len(errMsgs), strings.Join(errMsgs, "; "))
} }
return nil return nil
} }
func processSingleDisk(alert dto.AlertDTO) error { func processSingleDisk(alert dto.AlertDTO) error {
err := checkAndCreateDiskAlert(alert, alert.Project) usageStat, err := loadDiskUsageWithTimeout(alert.Project, true)
if err != nil { if err != nil {
global.LOG.Errorf("%s", err.Error()) global.LOG.Errorf("error getting disk usage for %s, err: %v", alert.Project, err)
return err return err
} }
checkAndCreateDiskAlert(alert, alert.Project, usageStat)
return nil return nil
} }
func checkAndCreateDiskAlert(alert dto.AlertDTO, path string) error { func checkAndCreateDiskAlert(alert dto.AlertDTO, path string, usageStat *disk.UsageStat) {
usageStat, err := psutil.DISK.GetUsage(path, false)
if err != nil {
global.LOG.Errorf("error getting disk usage for %s, err: %v", path, err)
return err
}
usedTotal, usedStr := calculateUsedTotal(alert.Cycle, usageStat) usedTotal, usedStr := calculateUsedTotal(alert.Cycle, usageStat)
commonTotal := float64(alert.Count) commonTotal := float64(alert.Count)
if alert.Cycle == 1 { if alert.Cycle == 1 {
commonTotal *= 1024 * 1024 * 1024 commonTotal *= 1024 * 1024 * 1024
} }
if usedTotal < commonTotal { if usedTotal < commonTotal {
return nil return
} }
params := createAlertDiskParams(path, usedStr) params := createAlertDiskParams(path, usedStr)
sender := NewAlertSender(alert, alert.Project) sender := NewAlertSender(alert, alert.Project)
sender.ResourceSend(path, params) sender.ResourceSend(path, params)
return nil
} }
func calculateUsedTotal(cycle uint, usageStat *disk.UsageStat) (float64, string) { func calculateUsedTotal(cycle uint, usageStat *disk.UsageStat) (float64, string) {
@@ -1097,3 +1126,55 @@ func calculateMinutesDifference(newDate time.Time) int {
minutesDifference := int(now.Sub(newDate).Minutes()) minutesDifference := int(now.Sub(newDate).Minutes())
return minutesDifference return minutesDifference
} }
func loadSSHAlertHistories(
baseDir string,
startTime, endTime time.Time,
location *time.Location,
) ([]dto.SSHHistory, error) {
fileList, err := listSSHLogFiles(baseDir)
if err != nil {
return nil, err
}
var (
histories []dto.SSHHistory
loadErr error
)
for _, file := range fileList {
items, err := loadSSHHistoriesFromFile(file.Name, "", "", startTime, endTime, file.Year, location)
if err != nil {
loadErr = errors.Join(loadErr, fmt.Errorf("load SSH log file %s: %w", file.Name, err))
continue
}
histories = append(histories, items...)
}
return histories, loadErr
}
func summarizeSSHLoginHistories(
histories []dto.SSHHistory,
now time.Time,
failedWindow time.Duration,
whitelist []string,
) (int, []string) {
failedStartTime := now.Add(-failedWindow)
successStartTime := now.Add(-sshIPLoginWindow)
failedCount := 0
var abnormalLogins []string
for _, item := range histories {
switch item.Status {
case constant.StatusFailed:
if isSSHLogWithinTimeRange(item.Date, failedStartTime, now) {
failedCount++
}
case constant.StatusSuccess:
if !isSSHLogWithinTimeRange(item.Date, successStartTime, now) || isIPInWhitelist(item.Address, whitelist) {
continue
}
abnormalLogins = append(abnormalLogins, fmt.Sprintf("%s-%s", item.Address, item.Date.Format(constant.DateTimeLayout)))
}
}
return failedCount, abnormalLogins
}
+45 -6
View File
@@ -75,7 +75,7 @@ func (s *AlertSender) sendByConfig(config model.AlertConfig, quota string, param
} else { } else {
s.sendBarkWithConfig(config, quota, params) s.sendBarkWithConfig(config, quota, params)
} }
case constant.WeCom, constant.DingTalk, constant.FeiShu: case constant.WeCom, constant.DingTalk, constant.FeiShu, constant.Custom:
if isResource { if isResource {
s.sendResourceWebhookWithConfig(config, quota, params) s.sendResourceWebhookWithConfig(config, quota, params)
} else { } else {
@@ -86,7 +86,7 @@ func (s *AlertSender) sendByConfig(config model.AlertConfig, quota string, param
func (s *AlertSender) sendByLegacyMethod(method string, quota string, params []dto.Param, isResource bool) { func (s *AlertSender) sendByLegacyMethod(method string, quota string, params []dto.Param, isResource bool) {
alertRepo := repo.NewIAlertRepo() alertRepo := repo.NewIAlertRepo()
typeMap := map[string]string{"mail": constant.Email, constant.Bark: constant.Bark, constant.SMS: constant.SMS} typeMap := map[string]string{"mail": constant.Email, constant.Bark: constant.Bark, constant.SMS: constant.SMS, constant.Custom: constant.Custom}
configType := method configType := method
if mapped, ok := typeMap[method]; ok { if mapped, ok := typeMap[method]; ok {
configType = mapped configType = mapped
@@ -308,12 +308,31 @@ func (s *AlertSender) sendWebhookWithConfig(config model.AlertConfig, quota stri
} }
transport := xpack.MultiNodeProvider.LoadRequestTransport() transport := xpack.MultiNodeProvider.LoadRequestTransport()
agentInfo, _ := xpack.MultiNodeProvider.GetAgentInfo() agentInfo, _ := xpack.MultiNodeProvider.GetAgentInfo()
err := xpack.AlertProvider.CreateWebhookAlertLog(s.alert.Type, s.alert, create, quota, params, config, transport, agentInfo) queued := false
var err error
if config.Type == constant.Custom {
task := dto.AlertTaskMetadata{
AlertID: s.alert.ID,
Type: s.alert.Type,
Quota: quota,
QuotaType: s.quotaType,
Method: strconv.Itoa(int(config.ID)),
}
result, deliveryErr := xpack.DeliverCustomWebhookAlertLog(s.alert.Type, s.alert, create, quota, params, config, transport, agentInfo, task)
queued, err = result.Queued, deliveryErr
if err == nil && result.Queued {
_, err = alertUtil.RecordQueuedAlertTask(result.LogID, task)
}
} else {
err = xpack.AlertProvider.CreateWebhookAlertLog(s.alert.Type, s.alert, create, quota, params, config, transport, agentInfo)
}
if err != nil { if err != nil {
global.LOG.Errorf("%s alert %s webhook push failed: %v", s.alert.Type, config.Type, err) global.LOG.Errorf("%s alert %s webhook push failed: %v", s.alert.Type, config.Type, err)
return return
} }
alertUtil.CreateNewAlertTask(quota, s.alert.Type, s.quotaType, strconv.Itoa(int(config.ID))) if !queued {
alertUtil.CreateNewAlertTask(quota, s.alert.Type, s.quotaType, strconv.Itoa(int(config.ID)))
}
} }
func (s *AlertSender) sendResourceWebhookWithConfig(config model.AlertConfig, quota string, params []dto.Param) { func (s *AlertSender) sendResourceWebhookWithConfig(config model.AlertConfig, quota string, params []dto.Param) {
@@ -334,11 +353,31 @@ func (s *AlertSender) sendResourceWebhookWithConfig(config model.AlertConfig, qu
} }
transport := xpack.MultiNodeProvider.LoadRequestTransport() transport := xpack.MultiNodeProvider.LoadRequestTransport()
agentInfo, _ := xpack.MultiNodeProvider.GetAgentInfo() agentInfo, _ := xpack.MultiNodeProvider.GetAgentInfo()
if err := xpack.AlertProvider.CreateWebhookAlertLog(s.alert.Type, s.alert, create, quota, params, config, transport, agentInfo); err != nil { queued := false
var err error
if config.Type == constant.Custom {
task := dto.AlertTaskMetadata{
AlertID: s.alert.ID,
Type: s.alert.Type,
Quota: quota,
QuotaType: s.quotaType,
Method: strconv.Itoa(int(config.ID)),
}
result, deliveryErr := xpack.DeliverCustomWebhookAlertLog(s.alert.Type, s.alert, create, quota, params, config, transport, agentInfo, task)
queued, err = result.Queued, deliveryErr
if err == nil && result.Queued {
_, err = alertUtil.RecordQueuedAlertTask(result.LogID, task)
}
} else {
err = xpack.AlertProvider.CreateWebhookAlertLog(s.alert.Type, s.alert, create, quota, params, config, transport, agentInfo)
}
if err != nil {
global.LOG.Errorf("%s alert %s webhook push failed: %v", s.alert.Type, config.Type, err) global.LOG.Errorf("%s alert %s webhook push failed: %v", s.alert.Type, config.Type, err)
return return
} }
alertUtil.CreateNewAlertTask(quota, s.alert.Type, s.quotaType, strconv.Itoa(int(config.ID))) if !queued {
alertUtil.CreateNewAlertTask(quota, s.alert.Type, s.quotaType, strconv.Itoa(int(config.ID)))
}
} }
func (s *AlertSender) sendWebhook(quota string, params []dto.Param, method string) { func (s *AlertSender) sendWebhook(quota string, params []dto.Param, method string) {
+26 -14
View File
@@ -223,6 +223,9 @@ func (a AppService) GetAppDetailByKey(appKey, version string) (response.AppDetai
if err != nil { if err != nil {
return appDetailDTO, err return appDetailDTO, err
} }
if err = checkVllmVersionAccess(app.Key, version); err != nil {
return appDetailDTO, err
}
appDetail, err := appDetailRepo.GetFirst(appDetailRepo.WithAppId(app.ID), appDetailRepo.WithVersion(version)) appDetail, err := appDetailRepo.GetFirst(appDetailRepo.WithAppId(app.ID), appDetailRepo.WithVersion(version))
if err != nil { if err != nil {
return appDetailDTO, err return appDetailDTO, err
@@ -241,14 +244,17 @@ func (a AppService) GetAppDetail(appID uint, version, appType string) (response.
if err != nil { if err != nil {
return appDetailDTO, err return appDetailDTO, err
} }
app, err := appRepo.GetFirst(repo.WithByID(detail.AppId))
if err != nil {
return appDetailDTO, err
}
if err = checkVllmVersionAccess(app.Key, detail.Version); err != nil {
return appDetailDTO, err
}
appDetailDTO.AppDetail = detail appDetailDTO.AppDetail = detail
appDetailDTO.Enable = true appDetailDTO.Enable = true
if appType == "runtime" { if appType == "runtime" {
app, err := appRepo.GetFirst(repo.WithByID(appID))
if err != nil {
return appDetailDTO, err
}
fileOp := files.NewFileOp() fileOp := files.NewFileOp()
versionPath := filepath.Join(app.GetAppResourcePath(), detail.Version) versionPath := filepath.Join(app.GetAppResourcePath(), detail.Version)
@@ -319,10 +325,6 @@ func (a AppService) GetAppDetail(appID uint, version, appType string) (response.
appDetailDTO.HostMode = isHostModel(appDetailDTO.DockerCompose) appDetailDTO.HostMode = isHostModel(appDetailDTO.DockerCompose)
app, err := appRepo.GetFirst(repo.WithByID(detail.AppId))
if err != nil {
return appDetailDTO, err
}
if err := checkLimit(app); err != nil { if err := checkLimit(app); err != nil {
appDetailDTO.Enable = false appDetailDTO.Enable = false
} }
@@ -374,6 +376,9 @@ func (a AppService) installWithHooks(req request.AppInstallCreate, executeScript
if err != nil { if err != nil {
return return
} }
if err = checkVllmVersionAccess(app.Key, appDetail.Version); err != nil {
return
}
if DatabaseKeys[app.Key] > 0 { if DatabaseKeys[app.Key] > 0 {
if existDatabases, _ := databaseRepo.GetList(repo.WithByName(req.Name)); len(existDatabases) > 0 { if existDatabases, _ := databaseRepo.GetList(repo.WithByName(req.Name)); len(existDatabases) > 0 {
err = buserr.New("ErrRemoteExist") err = buserr.New("ErrRemoteExist")
@@ -483,15 +488,17 @@ func (a AppService) installWithHooks(req request.AppInstallCreate, executeScript
index++ index++
} }
newServiceName := strings.ToLower(appInstall.Name) newServiceName := strings.ToLower(appInstall.Name)
if app.Limit == 0 && newServiceName != serviceName && len(servicesMap) == 1 { if app.Limit == 0 && newServiceName != serviceName && len(servicesMap) == 1 && !req.KeepServiceName {
servicesMap[newServiceName] = servicesMap[serviceName] servicesMap[newServiceName] = servicesMap[serviceName]
delete(servicesMap, serviceName) delete(servicesMap, serviceName)
serviceName = newServiceName serviceName = newServiceName
} }
appInstall.ServiceName = serviceName appInstall.ServiceName = serviceName
if err = addDockerComposeCommonParam(composeMap, appInstall.ServiceName, req.AppContainerConfig, req.Params); err != nil { if !req.SkipComposeCommonConfig {
return if err = addDockerComposeCommonParam(composeMap, appInstall.ServiceName, req.AppContainerConfig, req.Params); err != nil {
return
}
} }
var ( var (
composeByte []byte composeByte []byte
@@ -543,6 +550,11 @@ func (a AppService) installWithHooks(req request.AppInstallCreate, executeScript
installTask, err := task.NewTaskWithOps(appInstall.Name, task.TaskInstall, task.TaskScopeApp, req.TaskID, appInstall.ID) installTask, err := task.NewTaskWithOps(appInstall.Name, task.TaskInstall, task.TaskScopeApp, req.TaskID, appInstall.ID)
if err != nil { if err != nil {
appInstall.Status = constant.StatusInstallErr
appInstall.Message = err.Error()
if saveErr := appInstallRepo.Save(context.Background(), appInstall); saveErr != nil {
err = fmt.Errorf("%w; save failed install status: %v", err, saveErr)
}
return return
} }
@@ -550,7 +562,7 @@ func (a AppService) installWithHooks(req request.AppInstallCreate, executeScript
return return
} }
installApp := func(t *task.Task) error { installApp := func(t *task.Task) (err error) {
if err = copyData(t, app, appDetail, appInstall, req); err != nil { if err = copyData(t, app, appDetail, appInstall, req); err != nil {
return err return err
} }
@@ -559,7 +571,7 @@ func (a AppService) installWithHooks(req request.AppInstallCreate, executeScript
return err return err
} }
} }
if executeScript { if executeScript || req.UseLifecycleScripts {
if err = runScript(t, appInstall, "init"); err != nil { if err = runScript(t, appInstall, "init"); err != nil {
return err return err
} }
@@ -572,7 +584,7 @@ func (a AppService) installWithHooks(req request.AppInstallCreate, executeScript
return err return err
} }
} }
if err = upApp(t, appInstall, req.PullImage); err != nil { if err = upApp(t, appInstall, req.PullImage, req.UseLifecycleScripts); err != nil {
return err return err
} }
updateToolApp(appInstall) updateToolApp(appInstall)
+147 -17
View File
@@ -4,6 +4,7 @@ import (
"context" "context"
"encoding/json" "encoding/json"
"fmt" "fmt"
"maps"
"math" "math"
"net/http" "net/http"
"os" "os"
@@ -13,12 +14,14 @@ import (
"sort" "sort"
"strconv" "strconv"
"strings" "strings"
"time"
"github.com/1Panel-dev/1Panel/agent/app/dto" "github.com/1Panel-dev/1Panel/agent/app/dto"
"github.com/1Panel-dev/1Panel/agent/app/dto/request" "github.com/1Panel-dev/1Panel/agent/app/dto/request"
"github.com/1Panel-dev/1Panel/agent/app/dto/response" "github.com/1Panel-dev/1Panel/agent/app/dto/response"
"github.com/1Panel-dev/1Panel/agent/app/model" "github.com/1Panel-dev/1Panel/agent/app/model"
"github.com/1Panel-dev/1Panel/agent/app/repo" "github.com/1Panel-dev/1Panel/agent/app/repo"
"github.com/1Panel-dev/1Panel/agent/app/task"
"github.com/1Panel-dev/1Panel/agent/buserr" "github.com/1Panel-dev/1Panel/agent/buserr"
"github.com/1Panel-dev/1Panel/agent/constant" "github.com/1Panel-dev/1Panel/agent/constant"
"github.com/1Panel-dev/1Panel/agent/global" "github.com/1Panel-dev/1Panel/agent/global"
@@ -252,6 +255,9 @@ func (a *AppInstallService) Operate(req request.AppInstalledOperate) error {
return buserr.New("ErrInstallDirNotFound") return buserr.New("ErrInstallDirNotFound")
} }
dockerComposePath := install.GetComposePath() dockerComposePath := install.GetComposePath()
if req.UseLifecycleScripts && (req.Operate == constant.Start || req.Operate == constant.Stop || req.Operate == constant.Restart) {
return operateAppWithLifecycleScripts(install, req, nil)
}
switch req.Operate { switch req.Operate {
case constant.Rebuild: case constant.Rebuild:
return rebuildApp(install) return rebuildApp(install)
@@ -275,12 +281,13 @@ func (a *AppInstallService) Operate(req request.AppInstalledOperate) error {
return syncAppInstallStatus(&install, false) return syncAppInstallStatus(&install, false)
case constant.Delete: case constant.Delete:
deleteReq := request.AppInstallDelete{ deleteReq := request.AppInstallDelete{
Install: install, Install: install,
DeleteBackup: req.DeleteBackup, DeleteBackup: req.DeleteBackup,
ForceDelete: req.ForceDelete, ForceDelete: req.ForceDelete,
DeleteDB: req.DeleteDB, DeleteDB: req.DeleteDB,
DeleteImage: req.DeleteImage, DeleteImage: req.DeleteImage,
TaskID: req.TaskID, TaskID: req.TaskID,
UseLifecycleScripts: req.UseLifecycleScripts,
} }
if err = deleteAppInstall(deleteReq); err != nil && !req.ForceDelete { if err = deleteAppInstall(deleteReq); err != nil && !req.ForceDelete {
return err return err
@@ -312,6 +319,75 @@ func (a *AppInstallService) Operate(req request.AppInstalledOperate) error {
} }
} }
func operateAppWithLifecycleScripts(install model.AppInstall, req request.AppInstalledOperate, onFailure func(error)) error {
taskType := task.TaskUpdate
switch req.Operate {
case constant.Start:
install.Status = constant.StatusStarting
case constant.Restart:
taskType = task.TaskRestart
install.Status = constant.StatusRestarting
case constant.Stop:
install.Status = constant.StatusWaiting
default:
return errors.New("lifecycle script operation not supported")
}
install.Message = ""
if err := appInstallRepo.Save(context.Background(), &install); err != nil {
return err
}
operationTask, err := task.NewTaskWithOps(install.Name, taskType, task.TaskScopeApp, req.TaskID, install.ID)
if err != nil {
install.Status = constant.StatusUpErr
install.Message = err.Error()
if saveErr := appInstallRepo.Save(context.Background(), &install); saveErr != nil {
return fmt.Errorf("%w; save failed operation status: %v", err, saveErr)
}
return err
}
operation := string(req.Operate)
operationTask.AddSubTaskWithOps(
task.GetTaskName(install.Name, taskType, task.TaskScopeApp),
func(t *task.Task) error {
if err := runScript(t, &install, operation); err != nil {
return err
}
if req.Operate == constant.Stop {
install.Status = constant.StatusStopped
install.Message = ""
return appInstallRepo.Save(context.Background(), &install)
}
containerNames, err := getContainerNames(install)
if err != nil {
return err
}
if len(containerNames) == 0 {
return buserr.WithName("ErrContainerNotFound", install.Name)
}
install.ContainerName = strings.Join(containerNames, ",")
install.Status = constant.StatusRunning
install.Message = ""
return appInstallRepo.Save(context.Background(), &install)
},
nil,
0,
time.Hour,
)
go func() {
if taskErr := operationTask.Execute(); taskErr != nil {
if onFailure != nil {
onFailure(taskErr)
return
}
install.Status = constant.StatusUpErr
install.Message = taskErr.Error()
_ = appInstallRepo.Save(context.Background(), &install)
}
}()
return nil
}
func (a *AppInstallService) UpdateAppConfig(req request.AppConfigUpdate) error { func (a *AppInstallService) UpdateAppConfig(req request.AppConfigUpdate) error {
installed, err := appInstallRepo.GetFirst(repo.WithByID(req.InstallID)) installed, err := appInstallRepo.GetFirst(repo.WithByID(req.InstallID))
if err != nil { if err != nil {
@@ -374,8 +450,10 @@ func (a *AppInstallService) Update(req request.AppInstalledUpdate) error {
return err return err
} }
} }
if err = addDockerComposeCommonParam(composeMap, installed.ServiceName, req.AppContainerConfig, req.Params); err != nil { if !req.SkipComposeCommonConfig {
return err if err = addDockerComposeCommonParam(composeMap, installed.ServiceName, req.AppContainerConfig, req.Params); err != nil {
return err
}
} }
composeByte, err := yaml.Marshal(composeMap) composeByte, err := yaml.Marshal(composeMap)
if err != nil { if err != nil {
@@ -408,7 +486,7 @@ func (a *AppInstallService) Update(req request.AppInstalledUpdate) error {
if err != nil { if err != nil {
return err return err
} }
backupEnvMaps := oldEnvMaps backupEnvMaps := maps.Clone(oldEnvMaps)
handleMap(req.Params, oldEnvMaps) handleMap(req.Params, oldEnvMaps)
paramByte, err := json.Marshal(oldEnvMaps) paramByte, err := json.Marshal(oldEnvMaps)
if err != nil { if err != nil {
@@ -420,13 +498,32 @@ func (a *AppInstallService) Update(req request.AppInstalledUpdate) error {
} }
fileOp := files.NewFileOp() fileOp := files.NewFileOp()
_ = fileOp.WriteFile(installed.GetComposePath(), strings.NewReader(installed.DockerCompose), constant.DirPerm) _ = fileOp.WriteFile(installed.GetComposePath(), strings.NewReader(installed.DockerCompose), constant.DirPerm)
if err := rebuildApp(installed); err != nil { restoreConfig := func(operationErr error) {
_ = env.Write(backupEnvMaps, envPath) _ = env.Write(backupEnvMaps, envPath)
_ = fileOp.WriteFile(installed.GetComposePath(), strings.NewReader(backupDockerCompose), constant.DirPerm) _ = fileOp.WriteFile(installed.GetComposePath(), strings.NewReader(backupDockerCompose), constant.DirPerm)
failed := oldInstalled
failed.Status = constant.StatusUpErr
failed.Message = operationErr.Error()
_ = appInstallRepo.Save(context.Background(), &failed)
}
if req.UseLifecycleScripts {
err = operateAppWithLifecycleScripts(installed, request.AppInstalledOperate{
InstallId: installed.ID,
Operate: constant.Restart,
TaskID: req.TaskID,
UseLifecycleScripts: true,
}, restoreConfig)
} else {
err = rebuildApp(installed)
}
if err != nil {
restoreConfig(err)
return err return err
} }
installed.Status = constant.StatusRunning if !req.UseLifecycleScripts {
_ = appInstallRepo.Save(context.Background(), &installed) installed.Status = constant.StatusRunning
_ = appInstallRepo.Save(context.Background(), &installed)
}
proxyChanged := hasAppInstallProxyPassChanged(&oldInstalled, &installed) proxyChanged := hasAppInstallProxyPassChanged(&oldInstalled, &installed)
currentProxy, currentProxyErr := getAppInstallProxyPass(&installed) currentProxy, currentProxyErr := getAppInstallProxyPass(&installed)
@@ -475,11 +572,15 @@ func (a *AppInstallService) SyncAll(systemInit bool) error {
return err return err
} }
for _, i := range allList { for _, i := range allList {
if i.Status == constant.StatusInstalling || i.Status == constant.StatusUpgrading || i.Status == constant.StatusRebuilding || i.Status == constant.StatusUninstalling { if appInstallOperationInterruptedOnRestart(i.Status) {
if systemInit { if systemInit {
i.Status = constant.StatusError i.Status = appInstallOperationFailureStatus(i.Status)
i.Message = "1Panel restart causes the task to terminate" i.Message = constant.InterruptedMsg
_ = appInstallRepo.Save(context.Background(), &i) if err := appInstallRepo.Save(context.Background(), &i); err != nil {
return err
}
} else if err := syncAppInstallStatus(&i, false); err != nil {
global.LOG.Errorf("sync install app[%s] error,mgs: %s", i.Name, err.Error())
} }
continue continue
} }
@@ -583,6 +684,9 @@ func (a *AppInstallService) GetUpdateVersions(req request.AppUpdateVersion) ([]d
return versions, err return versions, err
} }
for _, detail := range details { for _, detail := range details {
if !canAccessVllmVersion(app.Key, detail.Version) {
continue
}
ignores, _ := appIgnoreUpgradeRepo.List(runtimeRepo.WithDetailId(detail.ID), appIgnoreUpgradeRepo.WithScope("version")) ignores, _ := appIgnoreUpgradeRepo.List(runtimeRepo.WithDetailId(detail.ID), appIgnoreUpgradeRepo.WithScope("version"))
if len(ignores) > 0 { if len(ignores) > 0 {
continue continue
@@ -836,8 +940,34 @@ func (a *AppInstallService) GetParams(id uint) (*response.AppConfig, error) {
} }
func syncAppInstallStatus(appInstall *model.AppInstall, force bool) error { func syncAppInstallStatus(appInstall *model.AppInstall, force bool) error {
if appInstall.Status == constant.StatusInstalling || appInstall.Status == constant.StatusRebuilding || appInstall.Status == constant.StatusUpgrading || appInstall.Status == constant.StatusUninstalling { operation := ""
switch appInstall.Status {
case constant.StatusInstalling:
operation = task.TaskInstall
case constant.StatusUpgrading:
operation = task.TaskUpgrade
case constant.StatusRebuilding:
operation = task.TaskBuild
case constant.StatusUninstalling:
operation = task.TaskUninstall
case constant.StatusStarting, constant.StatusWaiting:
operation = task.TaskUpdate
case constant.StatusRestarting:
operation = task.TaskRestart
}
if operation != "" {
if err := reconcileAppInstallTaskFailure(appInstall, operation, global.DB, global.TaskDB); err != nil {
return err
}
}
switch appInstall.Status {
case constant.StatusInstalling, constant.StatusRebuilding, constant.StatusUpgrading, constant.StatusUninstalling,
constant.StatusStarting, constant.StatusRestarting, constant.StatusWaiting:
return nil return nil
case constant.StatusInstallErr, constant.StatusUpgradeErr, constant.StatusUpErr:
if !force {
return nil
}
} }
cli, err := docker.NewClient() cli, err := docker.NewClient()
if err != nil { if err != nil {
+76
View File
@@ -0,0 +1,76 @@
package service
import (
"errors"
"github.com/1Panel-dev/1Panel/agent/app/model"
"github.com/1Panel-dev/1Panel/agent/constant"
"gorm.io/gorm"
)
func appInstallOperationFailureStatus(status string) string {
switch status {
case constant.StatusInstalling:
return constant.StatusInstallErr
case constant.StatusUpgrading:
return constant.StatusUpgradeErr
case constant.StatusUninstalling:
return constant.StatusError
default:
return constant.StatusUpErr
}
}
func appInstallOperationPending(status string) bool {
switch status {
case constant.StatusInstalling, constant.StatusRebuilding, constant.StatusUpgrading, constant.StatusUninstalling,
constant.StatusStarting, constant.StatusRestarting, constant.StatusWaiting:
return true
default:
return false
}
}
func appInstallOperationInterruptedOnRestart(status string) bool {
// ReStarting can also describe a container's Docker restart policy.
return appInstallOperationPending(status) && status != constant.StatusRestarting
}
func reconcileAppInstallTaskFailure(install *model.AppInstall, operation string, appDB, taskDB *gorm.DB) error {
if !appInstallOperationPending(install.Status) {
return nil
}
var operationTask model.Task
err := taskDB.Where("type = ? AND resource_id = ? AND operate = ?", "App", install.ID, operation).
Order("created_at DESC").First(&operationTask).Error
if errors.Is(err, gorm.ErrRecordNotFound) {
return nil
}
if err != nil {
return err
}
if operationTask.Status != constant.StatusFailed && operationTask.Status != constant.StatusCanceled {
return nil
}
// A previous operation must not turn a newly queued retry into a failure.
if operationTask.CreatedAt.Before(install.UpdatedAt) &&
(operationTask.EndAt.IsZero() || operationTask.EndAt.Before(install.UpdatedAt)) {
return nil
}
message := operationTask.ErrorMsg
if message == "" {
message = "the application operation task failed"
}
status := appInstallOperationFailureStatus(install.Status)
result := appDB.Model(&model.AppInstall{}).
Where("id = ? AND status = ? AND updated_at = ?", install.ID, install.Status, install.UpdatedAt).
Updates(map[string]interface{}{"status": status, "message": message})
if result.Error != nil {
return result.Error
}
if result.RowsAffected > 0 {
install.Status = status
install.Message = message
}
return nil
}
@@ -0,0 +1,147 @@
package service
import (
"path/filepath"
"testing"
"time"
"github.com/1Panel-dev/1Panel/agent/app/model"
"github.com/1Panel-dev/1Panel/agent/constant"
"github.com/glebarez/sqlite"
"gorm.io/gorm"
"gorm.io/gorm/clause"
"gorm.io/gorm/logger"
)
func operationStatusTestDB(t *testing.T) (*gorm.DB, *gorm.DB, model.AppInstall) {
t.Helper()
open := func(name string, schema interface{}) *gorm.DB {
db, err := gorm.Open(sqlite.Open(filepath.Join(t.TempDir(), name)), &gorm.Config{
Logger: logger.Default.LogMode(logger.Silent),
})
if err != nil {
t.Fatal(err)
}
if err := db.AutoMigrate(schema); err != nil {
t.Fatal(err)
}
t.Cleanup(func() {
sqlDB, _ := db.DB()
if sqlDB != nil {
_ = sqlDB.Close()
}
})
return db
}
appDB := open("apps.db", &model.AppInstall{})
taskDB := open("tasks.db", &model.Task{})
install := model.AppInstall{
Name: "gb10-ds-vision", AppId: 1, AppDetailId: 1,
Version: "nvidia-gb10-dspark-0.1.1", ContainerName: "vllm-dspark-1", ServiceName: "vllm-dspark",
Status: constant.StatusStarting, Env: `{"MODEL_DIR":"/models/vision"}`,
}
if err := appDB.Omit(clause.Associations).Create(&install).Error; err != nil {
t.Fatal(err)
}
return appDB, taskDB, install
}
func TestAppInstallFailedTaskRecoversPendingState(t *testing.T) {
for _, state := range []string{constant.StatusStarting, constant.StatusWaiting, constant.StatusRestarting, constant.StatusInstalling} {
t.Run(state, func(t *testing.T) {
appDB, taskDB, install := operationStatusTestDB(t)
install.Status = state
if err := appDB.Omit(clause.Associations).Save(&install).Error; err != nil {
t.Fatal(err)
}
task := model.Task{ID: "failed", Type: "App", Operate: "TaskUpdate", ResourceID: install.ID,
Status: constant.StatusFailed, ErrorMsg: "model startup script failed", CreatedAt: install.UpdatedAt.Add(time.Second), EndAt: install.UpdatedAt.Add(2 * time.Second)}
if err := taskDB.Create(&task).Error; err != nil {
t.Fatal(err)
}
if err := reconcileAppInstallTaskFailure(&install, task.Operate, appDB, taskDB); err != nil {
t.Fatal(err)
}
want := constant.StatusUpErr
if state == constant.StatusInstalling {
want = constant.StatusInstallErr
}
var stored model.AppInstall
if err := appDB.First(&stored, install.ID).Error; err != nil {
t.Fatal(err)
}
if install.Status != want || stored.Status != want || stored.Message != task.ErrorMsg || stored.Env != install.Env {
t.Fatalf("recovered state = %s/%s, message %q, env %q", install.Status, stored.Status, stored.Message, stored.Env)
}
})
}
}
func TestAppInstallPendingStateKeepsActiveAndUnrelatedTasks(t *testing.T) {
for _, taskStatus := range []string{constant.StatusExecuting, constant.StatusSuccess, "missing", "previous-failure", "other-resource"} {
t.Run(taskStatus, func(t *testing.T) {
appDB, taskDB, install := operationStatusTestDB(t)
task := model.Task{ID: "task", Type: "App", Operate: "TaskUpdate", ResourceID: install.ID,
Status: taskStatus, CreatedAt: install.UpdatedAt.Add(time.Second), EndAt: install.UpdatedAt.Add(2 * time.Second)}
if taskStatus == "previous-failure" {
task.Status = constant.StatusFailed
task.CreatedAt = install.UpdatedAt.Add(-2 * time.Hour)
task.EndAt = install.UpdatedAt.Add(-time.Hour)
}
if taskStatus == "other-resource" {
task.Status = constant.StatusFailed
task.ResourceID++
}
if taskStatus != "missing" {
if err := taskDB.Create(&task).Error; err != nil {
t.Fatal(err)
}
}
if err := reconcileAppInstallTaskFailure(&install, task.Operate, appDB, taskDB); err != nil {
t.Fatal(err)
}
if install.Status != constant.StatusStarting {
t.Fatalf("unrelated or active task changed pending state to %s", install.Status)
}
})
}
}
func TestAppInstallRecoveryDoesNotOverwriteConcurrentRetry(t *testing.T) {
appDB, taskDB, install := operationStatusTestDB(t)
task := model.Task{ID: "failed", Type: "App", Operate: "TaskUpdate", ResourceID: install.ID,
Status: constant.StatusCanceled, ErrorMsg: "interrupted", CreatedAt: install.UpdatedAt.Add(time.Second), EndAt: install.UpdatedAt.Add(2 * time.Second)}
if err := taskDB.Create(&task).Error; err != nil {
t.Fatal(err)
}
if err := appDB.Callback().Update().Before("gorm:update").Register("new-operation", func(db *gorm.DB) {
if err := db.Exec("UPDATE app_installs SET updated_at = ?, message = ? WHERE id = ?", install.UpdatedAt.Add(time.Hour), "new attempt", install.ID).Error; err != nil {
t.Error(err)
}
}); err != nil {
t.Fatal(err)
}
if err := reconcileAppInstallTaskFailure(&install, task.Operate, appDB, taskDB); err != nil {
t.Fatal(err)
}
var stored model.AppInstall
if err := appDB.First(&stored, install.ID).Error; err != nil {
t.Fatal(err)
}
if stored.Status != constant.StatusStarting || stored.Message != "new attempt" {
t.Fatalf("recovery overwrote a newer operation: %s, %s", stored.Status, stored.Message)
}
}
func TestAppInstallRestartRecoveryCoversLifecycleStates(t *testing.T) {
for _, state := range []string{constant.StatusStarting, constant.StatusWaiting, constant.StatusInstalling} {
if !appInstallOperationInterruptedOnRestart(state) {
t.Errorf("restart recovery omitted %s", state)
}
}
for _, state := range []string{constant.StatusRunning, constant.StatusStopped, constant.StatusRestarting} {
if appInstallOperationInterruptedOnRestart(state) {
t.Errorf("restart recovery changes a Docker runtime state: %s", state)
}
}
}
+918
View File
@@ -0,0 +1,918 @@
package service
import (
"context"
"encoding/json"
"errors"
"fmt"
"maps"
"os"
"path"
"sort"
"strings"
"sync"
"time"
"github.com/1Panel-dev/1Panel/agent/app/dto/request"
"github.com/1Panel-dev/1Panel/agent/app/model"
"github.com/1Panel-dev/1Panel/agent/app/repo"
"github.com/1Panel-dev/1Panel/agent/app/task"
"github.com/1Panel-dev/1Panel/agent/buserr"
"github.com/1Panel-dev/1Panel/agent/constant"
"github.com/1Panel-dev/1Panel/agent/global"
"github.com/1Panel-dev/1Panel/agent/i18n"
"github.com/1Panel-dev/1Panel/agent/utils/common"
"github.com/1Panel-dev/1Panel/agent/utils/compose"
"github.com/1Panel-dev/1Panel/agent/utils/docker"
"github.com/1Panel-dev/1Panel/agent/utils/files"
"github.com/docker/docker/api/types/container"
"github.com/docker/docker/api/types/filters"
"github.com/joho/godotenv"
)
type appUpgradePhase int
const (
appUpgradePreparing appUpgradePhase = iota
appUpgradePrepared
appUpgradeStopped
appUpgradeBackedUp
appUpgradeDown
appUpgradeMutated
appUpgradeStarted
appUpgradeCommitted
)
var appUpgradeLocks sync.Map
const composeServiceLabel = "com.docker.compose.service"
type appUpgradeSnapshot interface {
Restore() error
Cleanup()
}
type upgradeFileSnapshot struct {
installPath string
backupPath string
paths []string
existing map[string]bool
}
type appUpgradeContext struct {
req request.AppInstallUpgrade
original model.AppInstall
candidate model.AppInstall
detail model.AppDetail
phase appUpgradePhase
stopAttempted bool
downAttempted bool
rollbackErr error
detailDir string
stageDir string
envContent []byte
oldEnvContent []byte
oldDockerCompose string
oldImageIDs []appImageID
backupFile string
snapshot appUpgradeSnapshot
createdPaths []string
}
func upgradeInstall(req request.AppInstallUpgrade) error {
install, err := appInstallRepo.GetFirst(repo.WithByID(req.InstallID))
if err != nil {
return err
}
if install.Status == constant.StatusUpgrading {
return buserr.New("TaskIsExecuting")
}
if err = task.CheckScopeTaskIsExecuting(task.TaskScopeApp, install.ID); err != nil {
return err
}
if _, loaded := appUpgradeLocks.LoadOrStore(install.ID, struct{}{}); loaded {
return buserr.New("TaskIsExecuting")
}
releaseLock := true
defer func() {
if releaseLock {
appUpgradeLocks.Delete(install.ID)
}
}()
detail, err := appDetailRepo.GetFirst(repo.WithByID(req.DetailID))
if err != nil {
return err
}
if err = checkVllmVersionAccess(install.App.Key, detail.Version); err != nil {
return err
}
if install.App.Key == vllmAppKeyForUpgrade && !isVllmUpgradeVersionAllowed(install.Version, detail.Version, loadVllmImageFromEnv(install.Env)) {
return errors.New("vLLM can only upgrade within the same image type")
}
if install.Version == detail.Version {
return errors.New("two version is same")
}
upgradeTask, err := task.NewTaskWithOps(install.Name, task.TaskUpgrade, task.TaskScopeApp, req.TaskID, install.ID)
if err != nil {
return err
}
ctx := &appUpgradeContext{
req: req,
original: install,
candidate: install,
detail: detail,
phase: appUpgradePreparing,
oldDockerCompose: install.DockerCompose,
}
upgradeTask.AddSubTaskWithOps(i18n.GetMsgByKey("UpgradePrepare"), ctx.prepare, nil, 0, 0)
upgradeTask.AddSubTaskWithOps(
task.GetTaskName(install.Name, task.TaskUpgrade, task.TaskScopeApp),
ctx.cutover,
func(t *task.Task) {
ctx.rollbackErr = ctx.rollback(t)
},
0,
0,
)
upgradingInstall := install
upgradingInstall.Status = constant.StatusUpgrading
upgradingInstall.Message = ""
if err = appInstallRepo.Save(context.Background(), &upgradingInstall); err != nil {
return err
}
releaseLock = false
go func() {
defer appUpgradeLocks.Delete(install.ID)
defer ctx.cleanup()
taskErr := upgradeTask.Execute()
if taskErr == nil {
return
}
if ctx.rollbackErr != nil {
taskErr = fmt.Errorf("%w; %s: %v", taskErr, i18n.GetMsgByKey("UpgradeRollbackFailed"), ctx.rollbackErr)
upgradeTask.Task.ErrorMsg = taskErr.Error()
_ = repo.NewITaskRepo().Update(context.Background(), upgradeTask.Task)
}
if !ctx.stopAttempted || ctx.rollbackErr == nil {
restored := ctx.original
_ = appInstallRepo.Save(context.Background(), &restored)
return
}
failed := ctx.original
failed.Status = constant.StatusUpgradeErr
failed.Message = taskErr.Error()
_ = appInstallRepo.Save(context.Background(), &failed)
}()
return nil
}
func (u *appUpgradeContext) prepare(t *task.Task) error {
fileOp := files.NewFileOp()
u.detailDir = path.Join(u.original.App.GetAppResourcePath(), u.detail.Version)
if u.original.App.Resource == constant.AppResourceRemote {
if err := downloadApp(u.original.App, u.detail, nil, t.Logger); err != nil {
return err
}
}
if !fileOp.Stat(u.detailDir) {
return buserr.WithName("ErrFileNotFound", u.detailDir)
}
if u.detail.DockerCompose == "" {
composeContent, err := fileOp.GetContent(path.Join(u.detailDir, "docker-compose.yml"))
if err != nil {
return err
}
u.detail.DockerCompose = string(composeContent)
_ = appDetailRepo.Update(context.Background(), u.detail)
}
if strings.TrimSpace(u.detail.DockerCompose) == "" && strings.TrimSpace(u.req.DockerCompose) == "" {
return buserr.WithName("ErrFileNotFound", "docker-compose.yml")
}
var err error
u.oldEnvContent, err = fileOp.GetContent(u.original.GetEnvPath())
if err != nil {
return err
}
u.stageDir, err = os.MkdirTemp(u.original.GetAppPath(), "."+u.original.Name+"-upgrade-")
if err != nil {
return err
}
if err = fileOp.CopyDirWithNewName(u.detailDir, u.stageDir, "."); err != nil {
return err
}
if err = copyUpgradeStageFile(u.original.GetPath(), u.stageDir, ".env"); err != nil {
return err
}
if u.original.App.Key == constant.AppOpenclaw {
if err = copyUpgradeStageFile(u.original.GetPath(), u.stageDir, path.Join("data", "conf", "openclaw.json")); err != nil {
return err
}
}
if u.original.App.Key == constant.AppOpenresty {
for _, relativePath := range []string{
nginxModuleBuildDir,
nginxModuleModulesDir,
path.Join(nginxModuleConfDir, nginxModuleEnabledConfDir),
} {
if err = copyUpgradeStageFile(u.original.GetPath(), u.stageDir, relativePath); err != nil {
return err
}
}
}
stagedInstall := u.original
stagedInstall.Name = path.Base(u.stageDir)
stagedInstall.Version = u.detail.Version
stagedInstall.AppDetailId = u.req.DetailID
if stagedInstall.App.Key == vllmAppKeyForUpgrade {
envs := make(map[string]interface{})
if err = json.Unmarshal([]byte(stagedInstall.Env), &envs); err != nil {
return err
}
image := buildVllmUpgradeImage(loadVllmImageFromEnv(stagedInstall.Env), u.original.Version, u.detail.Version)
envs[vllmImageEnvKey] = image
paramBytes, marshalErr := json.Marshal(envs)
if marshalErr != nil {
return marshalErr
}
stagedInstall.Env = string(paramBytes)
}
if err = migrateOpenclawProtocolUpgrade(&stagedInstall, u.original.Version, u.detail.Version); err != nil {
return err
}
u.candidate = stagedInstall
u.candidate.Name = u.original.Name
u.candidate.DockerCompose, err = renderUpgradeCompose(u.candidate, u.detail, u.req.DockerCompose)
if err != nil {
return err
}
if strings.TrimSpace(u.candidate.DockerCompose) == "" {
return buserr.WithName("ErrFileNotFound", "docker-compose.yml")
}
u.envContent, err = renderUpgradeEnv(&u.candidate, u.oldEnvContent)
if err != nil {
return err
}
if err = writeUpgradeFile(path.Join(u.stageDir, ".env"), u.envContent, constant.FilePerm); err != nil {
return err
}
if err = writeUpgradeFile(path.Join(u.stageDir, "docker-compose.yml"), []byte(u.candidate.DockerCompose), constant.FilePerm); err != nil {
return err
}
project, err := docker.GetComposeProject(u.original.Name, u.stageDir, []byte(u.candidate.DockerCompose), u.envContent, false)
if err != nil {
return err
}
hasBuild := false
for _, service := range project.Services {
if service.Image == "" && service.Build == nil {
return fmt.Errorf("compose service %s has neither image nor build configuration", service.Name)
}
hasBuild = hasBuild || service.Build != nil
}
if u.req.DeleteImage {
dockerClient, clientErr := docker.NewClient()
if clientErr != nil {
return clientErr
}
u.oldImageIDs, err = getAppImageIDsByCompose(dockerClient, u.oldEnvContent, []byte(u.oldDockerCompose))
dockerClient.Close()
if err != nil {
return err
}
}
images := make([]string, 0, len(project.Services))
for _, service := range project.Services {
if service.Image != "" {
images = append(images, service.Image)
}
}
if err = prepareUpgradeImages(t, images, u.req.PullImage); err != nil {
return err
}
if u.candidate.App.Key == constant.AppOpenresty {
if err = u.prepareOpenresty(t, stagedInstall); err != nil {
return err
}
if err = verifyUpgradeImages(images); err != nil {
return err
}
} else if hasBuild {
logStr := fmt.Sprintf("%s %s", i18n.GetMsgByKey("TaskBuild"), i18n.GetMsgByKey("Image"))
t.LogStart(logStr)
if err = compose.BuildWithTask(path.Join(u.stageDir, "docker-compose.yml"), project.Name, t); err != nil {
t.LogFailedWithErr(logStr, err)
return err
}
t.LogSuccess(logStr)
if err = verifyUpgradeImages(images); err != nil {
return err
}
}
if u.original.App.Resource == constant.AppResourceRemote {
go RequestDownloadCallBack(u.detail.DownloadCallBackUrl)
}
u.phase = appUpgradePrepared
return nil
}
func (u *appUpgradeContext) prepareOpenresty(t *task.Task, stagedInstall model.AppInstall) error {
fileOp := files.NewFileOp()
detailBuildDir := path.Join(u.detailDir, nginxModuleBuildDir)
installBuildDir := path.Join(u.stageDir, nginxModuleBuildDir)
if !fileOp.Stat(installBuildDir) {
if err := fileOp.CreateDir(installBuildDir, constant.DirPerm); err != nil {
return err
}
}
if err := copyAppDetailMissing(fileOp, detailBuildDir, installBuildDir); err != nil {
return err
}
if err := fileOp.DeleteDir(path.Join(installBuildDir, nginxModuleTmpDir)); err != nil {
return err
}
if err := fileOp.CopyDir(path.Join(detailBuildDir, nginxModuleTmpDir), installBuildDir); err != nil {
return err
}
for _, fileName := range []string{"Dockerfile", "nginx.conf", "nginx.vh.default.conf"} {
if err := fileOp.CopyFile(path.Join(detailBuildDir, fileName), installBuildDir); err != nil {
return err
}
}
if err := syncNginxModuleBuilder(detailBuildDir, installBuildDir); err != nil {
return err
}
targetCatalogSource := path.Join(detailBuildDir, nginxModuleCatalogFile)
if !fileOp.Stat(targetCatalogSource) {
return fmt.Errorf("target OpenResty module catalog not found: %s", targetCatalogSource)
}
targetCatalogPath := path.Join(installBuildDir, nginxModuleCatalogPendingFile)
if err := stageNginxModuleCatalog(targetCatalogSource, targetCatalogPath); err != nil {
return err
}
stagedInstall.Name = path.Base(u.stageDir)
stagedInstall.Version = u.candidate.Version
stagedInstall.Env = u.candidate.Env
stagedInstall.DockerCompose = u.candidate.DockerCompose
return buildNginx(t, stagedInstall, targetCatalogPath)
}
func (u *appUpgradeContext) cutover(t *task.Task) error {
u.stopAttempted = true
t.LogStart(i18n.GetMsgByKey("UpgradeStop"))
if out, err := compose.Stop(u.original.GetComposePath()); err != nil {
if out != "" {
err = fmt.Errorf("%s: %w", out, err)
}
t.LogFailedWithErr(i18n.GetMsgByKey("UpgradeStop"), err)
return err
}
t.LogSuccess(i18n.GetMsgByKey("UpgradeStop"))
u.phase = appUpgradeStopped
var err error
if u.original.App.Key == constant.AppOpenresty {
u.snapshot, err = createOpenrestyUpgradeSnapshot(u.original.GetPath())
} else {
snapshotPaths := []string{".env", "docker-compose.yml", "scripts"}
if u.original.App.Key == constant.AppOpenclaw {
snapshotPaths = append(snapshotPaths, path.Join("data", "conf", "openclaw.json"))
}
u.snapshot, err = createUpgradeFileSnapshot(u.original.GetPath(), snapshotPaths)
}
if err != nil {
return err
}
if u.req.Backup {
if err = u.backup(t); err != nil {
return err
}
u.phase = appUpgradeBackedUp
} else {
t.Log(i18n.GetMsgByKey("UpgradeBackupDisabled"))
}
u.downAttempted = true
if out, downErr := compose.Down(u.original.GetComposePath()); downErr != nil {
if out != "" {
downErr = fmt.Errorf("%s: %w", out, downErr)
}
return downErr
}
u.phase = appUpgradeDown
u.phase = appUpgradeMutated
if err = u.applyStagedFiles(); err != nil {
return err
}
if err = writeUpgradeFile(u.original.GetEnvPath(), u.envContent, constant.FilePerm); err != nil {
return err
}
if err = runScript(t, &u.candidate, "upgrade"); err != nil {
return err
}
if err = writeUpgradeFile(u.original.GetComposePath(), []byte(u.candidate.DockerCompose), constant.FilePerm); err != nil {
return err
}
logStr := fmt.Sprintf("%s %s", i18n.GetMsgByKey("Run"), i18n.GetMsgByKey("App"))
t.LogStart(logStr)
if out, upErr := compose.UpWithoutBuild(u.original.GetComposePath()); upErr != nil {
if out != "" {
upErr = fmt.Errorf("%s: %w", out, upErr)
}
t.LogFailedWithErr(logStr, upErr)
return upErr
}
t.LogSuccess(logStr)
u.phase = appUpgradeStarted
containerNames, discoverErr := discoverUpgradeContainerNames(u.candidate, u.envContent)
if discoverErr != nil {
t.Logf("WARNING: discover upgraded application containers failed: %v", discoverErr)
} else if len(containerNames) > 0 {
u.candidate.ContainerName = strings.Join(containerNames, ",")
} else {
t.Log("WARNING: no containers found for the upgraded application")
}
u.candidate.Status = constant.StatusRunning
u.candidate.Message = ""
if u.candidate.App.Key == constant.AppOpenresty {
liveCatalogPath := path.Join(u.candidate.GetPath(), nginxModuleBuildDir, nginxModuleCatalogPendingFile)
if err = commitStaticNginxModuleBuilds(u.candidate, liveCatalogPath, t); err != nil {
return err
}
activeCatalogPath := path.Join(u.candidate.GetPath(), nginxModuleBuildDir, nginxModuleCatalogFile)
if err = activateNginxModuleCatalogAndCommit(liveCatalogPath, activeCatalogPath, func() error {
return appInstallRepo.Save(context.Background(), &u.candidate)
}); err != nil {
return err
}
// Upgrades deliberately keep the user's nginx.conf, so corrected gzip
// defaults shipped with a new version would never reach existing
// installations. Rewrite only an untouched factory configuration, and
// never fail the upgrade over it.
if gzipErr := upgradeStockNginxGzipConfig(u.candidate); gzipErr != nil {
t.Logf("WARNING: update stock gzip configuration failed, keeping the current one: %v", gzipErr)
}
} else if err = appInstallRepo.Save(context.Background(), &u.candidate); err != nil {
return err
}
if discoverErr == nil && len(containerNames) > 0 {
if syncErr := syncAppInstallStatus(&u.candidate, true); syncErr != nil {
t.Logf("WARNING: sync upgraded application status failed: %v", syncErr)
}
}
u.phase = appUpgradeCommitted
u.deleteOldImages(t)
return nil
}
func (u *appUpgradeContext) backup(t *task.Task) error {
fileName := fmt.Sprintf("upgrade_backup_%s_%s.tar.gz", u.original.Name, time.Now().Format(constant.DateTimeSlimLayout)+common.RandStrAndNum(5))
record, err := backupAppWithParentTask(&u.original, t, fileName)
if err != nil {
return buserr.WithNameAndErr("ErrAppBackup", u.original.Name, err)
}
u.backupFile = path.Join(global.Dir.LocalBackupDir, record.FileDir, record.FileName)
info, err := os.Stat(u.backupFile)
if err != nil || info.Size() == 0 || record.Status != constant.StatusSuccess {
if err == nil {
err = errors.New("backup archive is empty or incomplete")
}
markBackupFailed(record.ID, err)
return buserr.WithNameAndErr("ErrAppBackup", u.original.Name, err)
}
backupRecordService := NewIBackupRecordService()
backups, _ := backupRecordService.ListAppRecords(u.original.App.Key, u.original.Name, "upgrade_backup")
if len(backups) > 3 {
deleteIDs := make([]uint, 0, len(backups)-3)
for _, backup := range backups[:len(backups)-3] {
deleteIDs = append(deleteIDs, backup.ID)
}
_ = backupRecordService.BatchDeleteRecord(deleteIDs)
}
return nil
}
func (u *appUpgradeContext) applyStagedFiles() error {
fileOp := files.NewFileOp()
if err := copyAppDetailMissingTracked(fileOp, u.detailDir, u.original.GetPath(), &u.createdPaths); err != nil {
return err
}
if err := replaceUpgradePath(u.stageDir, u.original.GetPath(), "scripts"); err != nil {
return err
}
if u.original.App.Key == constant.AppOpenclaw {
if err := replaceUpgradePath(u.stageDir, u.original.GetPath(), path.Join("data", "conf", "openclaw.json")); err != nil {
return err
}
}
if u.original.App.Key == constant.AppOpenresty {
for _, relativePath := range []string{
nginxModuleBuildDir,
nginxModuleModulesDir,
path.Join(nginxModuleConfDir, nginxModuleEnabledConfDir),
} {
if err := replaceUpgradePath(u.stageDir, u.original.GetPath(), relativePath); err != nil {
return err
}
}
}
return nil
}
func (u *appUpgradeContext) rollback(t *task.Task) (rollbackErr error) {
if !u.stopAttempted {
return nil
}
logStr := i18n.GetWithName("AppRecover", u.original.Name)
t.LogStart(logStr)
defer func() {
if rollbackErr != nil {
t.LogFailedWithErr(logStr, rollbackErr)
} else {
t.LogSuccess(logStr)
}
}()
if !u.downAttempted {
if out, err := compose.Operate(u.original.GetComposePath(), "start"); err != nil {
if out != "" {
err = fmt.Errorf("%s: %w", out, err)
}
return err
}
return u.finishRollback()
}
if u.phase < appUpgradeMutated {
if out, err := compose.UpWithoutBuild(u.original.GetComposePath()); err != nil {
if out != "" {
err = fmt.Errorf("%s: %w", out, err)
}
return err
}
return u.finishRollback()
}
if out, err := compose.Down(u.original.GetComposePath()); err != nil {
if out != "" {
err = fmt.Errorf("%s: %w", out, err)
}
rollbackErr = err
}
if u.backupFile != "" {
_ = u.restoreManagedFiles()
if err := handleAppRecover(&u.original, t, u.backupFile, true, "", ""); err != nil {
_, _ = compose.UpWithoutBuild(u.original.GetComposePath())
return errors.Join(rollbackErr, err)
}
} else {
if err := u.restoreManagedFiles(); err != nil {
return errors.Join(rollbackErr, err)
}
if out, err := compose.UpWithoutBuild(u.original.GetComposePath()); err != nil {
if out != "" {
err = fmt.Errorf("%s: %w", out, err)
}
return errors.Join(rollbackErr, err)
}
}
return errors.Join(rollbackErr, u.finishRollback())
}
func (u *appUpgradeContext) finishRollback() error {
restored := u.original
if err := appInstallRepo.Save(context.Background(), &restored); err != nil {
return err
}
return nil
}
func (u *appUpgradeContext) restoreManagedFiles() error {
var restoreErr error
if u.snapshot != nil {
restoreErr = u.snapshot.Restore()
}
for index := len(u.createdPaths) - 1; index >= 0; index-- {
if err := os.RemoveAll(u.createdPaths[index]); err != nil {
restoreErr = errors.Join(restoreErr, err)
}
}
return restoreErr
}
func (u *appUpgradeContext) deleteOldImages(t *task.Task) {
if !u.req.DeleteImage {
return
}
excludeImages, err := docker.GetImagesFromDockerCompose(u.envContent, []byte(u.candidate.DockerCompose))
if err != nil {
t.LogFailedWithErr(i18n.GetMsgByKey("TaskDelete")+i18n.GetMsgByKey("Image"), err)
return
}
dockerClient, err := docker.NewClient()
if err != nil {
t.LogFailedWithErr(i18n.GetMsgByKey("TaskDelete")+i18n.GetMsgByKey("Image"), err)
return
}
defer dockerClient.Close()
if err = deleteAppImagesByIDs(t, dockerClient, u.oldImageIDs, excludeImages); err != nil {
t.LogFailedWithErr(i18n.GetMsgByKey("TaskDelete")+i18n.GetMsgByKey("Image"), err)
}
}
func (u *appUpgradeContext) cleanup() {
if u.snapshot != nil {
u.snapshot.Cleanup()
}
if u.stageDir != "" {
_ = os.RemoveAll(u.stageDir)
}
}
type upgradeImageClient interface {
PullImageWithProcess(*task.Task, string) error
ImageExists(string) (bool, error)
Close()
}
func prepareUpgradeImages(t *task.Task, images []string, pull bool) error {
dockerClient, err := docker.NewClient()
if err != nil {
return err
}
return prepareUpgradeImagesWithClient(t, dockerClient, images, pull)
}
func prepareUpgradeImagesWithClient(t *task.Task, dockerClient upgradeImageClient, images []string, pull bool) error {
defer dockerClient.Close()
seen := make(map[string]struct{}, len(images))
for _, image := range images {
image = strings.TrimSpace(image)
if image == "" {
continue
}
if _, ok := seen[image]; ok {
continue
}
seen[image] = struct{}{}
if pull {
if t != nil {
t.Log(i18n.GetWithName("PullImageStart", image))
}
if pullErr := dockerClient.PullImageWithProcess(t, image); pullErr != nil {
if exists, _ := dockerClient.ImageExists(image); exists {
if t != nil {
t.Log(i18n.GetMsgByKey("UseExistImage"))
}
continue
}
return buserr.WithNameAndErr("ErrDockerPullImage", "", pullErr)
}
}
exists, inspectErr := dockerClient.ImageExists(image)
if inspectErr != nil || !exists {
return buserr.WithNameAndErr("ErrDockerPullImage", "", fmt.Errorf("image %s is not available locally: %v", image, inspectErr))
}
if pull && t != nil {
t.LogSuccess(i18n.GetMsgByKey("PullImage"))
}
}
return nil
}
func verifyUpgradeImages(images []string) error {
dockerClient, err := docker.NewClient()
if err != nil {
return err
}
defer dockerClient.Close()
for _, image := range images {
exists, inspectErr := dockerClient.ImageExists(image)
if inspectErr != nil || !exists {
return buserr.WithNameAndErr("ErrDockerPullImage", "", fmt.Errorf("image %s is not available locally: %v", image, inspectErr))
}
}
return nil
}
func renderUpgradeEnv(install *model.AppInstall, original []byte) ([]byte, error) {
originalEnv := make(map[string]string)
if len(original) > 0 {
var err error
originalEnv, err = godotenv.UnmarshalBytes(original)
if err != nil {
return nil, err
}
}
params := make(map[string]string, len(originalEnv))
maps.Copy(params, originalEnv)
envs := make(map[string]interface{})
if err := json.Unmarshal([]byte(install.Env), &envs); err != nil {
return nil, err
}
handleMap(envs, params)
if install.App.Key == "openlist" {
// The upgrade script updates this too late for the pre-pull phase.
image := "openlistteam/openlist:v" + strings.TrimPrefix(install.Version, "v")
if preInstalled := params["PRE_INSTALLED"]; preInstalled != "" {
image += "-" + preInstalled
}
params["OPENLIST_IMAGE"] = image
envs["OPENLIST_IMAGE"] = image
content, err := json.Marshal(envs)
if err != nil {
return nil, err
}
install.Env = string(content)
}
if install.App.Key == constant.AppOpenresty {
for _, key := range []string{"CONTAINER_PACKAGE_URL", "RESTY_ADD_PACKAGE_BUILDDEPS", "RESTY_CONFIG_OPTIONS_MORE"} {
if value, ok := originalEnv[key]; ok {
params[key] = value
}
}
if websiteDir := strings.TrimSpace(originalEnv["WEBSITE_DIR"]); websiteDir != "" {
params["WEBSITE_DIR"] = websiteDir
}
websiteDir := strings.TrimSpace(params["WEBSITE_DIR"])
if websiteDir == "" {
websiteDir = NewISettingService().GetWebsiteDir()
}
if !path.IsAbs(websiteDir) {
websiteDir = path.Join(global.Dir.DataDir, websiteDir)
}
params["WEBSITE_DIR"] = websiteDir
envs["WEBSITE_DIR"] = websiteDir
content, marshalErr := json.Marshal(envs)
if marshalErr != nil {
return nil, marshalErr
}
install.Env = string(content)
}
content, err := godotenv.Marshal(params)
if err != nil {
return nil, err
}
return []byte(content), nil
}
func renderUpgradeCompose(install model.AppInstall, detail model.AppDetail, customCompose string) (string, error) {
if customCompose != "" {
return customCompose, nil
}
if install.App.Key == vllmAppKeyForUpgrade {
return install.DockerCompose, nil
}
return getUpgradeCompose(install, detail)
}
func writeUpgradeFile(filePath string, content []byte, mode os.FileMode) error {
tmp, err := os.CreateTemp(path.Dir(filePath), "."+path.Base(filePath)+".*")
if err != nil {
return err
}
tmpPath := tmp.Name()
defer os.Remove(tmpPath)
if err = tmp.Chmod(mode); err == nil {
_, err = tmp.Write(content)
}
if err == nil {
err = tmp.Sync()
}
if closeErr := tmp.Close(); err == nil {
err = closeErr
}
if err != nil {
return err
}
return os.Rename(tmpPath, filePath)
}
func copyUpgradeStageFile(sourceRoot, targetRoot, relativePath string) error {
source := path.Join(sourceRoot, relativePath)
if _, err := os.Stat(source); err != nil {
if os.IsNotExist(err) {
return nil
}
return err
}
target := path.Join(targetRoot, relativePath)
_ = os.RemoveAll(target)
return copyOpenrestyUpgradeSnapshotEntry(source, target)
}
func replaceUpgradePath(sourceRoot, targetRoot, relativePath string) error {
source := path.Join(sourceRoot, relativePath)
if _, err := os.Stat(source); err != nil {
if os.IsNotExist(err) {
return nil
}
return err
}
target := path.Join(targetRoot, relativePath)
if err := os.RemoveAll(target); err != nil {
return err
}
return copyOpenrestyUpgradeSnapshotEntry(source, target)
}
func createUpgradeFileSnapshot(installPath string, paths []string) (*upgradeFileSnapshot, error) {
backupPath, err := os.MkdirTemp("", "1panel-app-upgrade-*")
if err != nil {
return nil, err
}
snapshot := &upgradeFileSnapshot{
installPath: installPath,
backupPath: backupPath,
paths: paths,
existing: make(map[string]bool, len(paths)),
}
for _, relativePath := range paths {
source := path.Join(installPath, relativePath)
if _, err = os.Stat(source); err != nil {
if os.IsNotExist(err) {
continue
}
snapshot.Cleanup()
return nil, err
}
snapshot.existing[relativePath] = true
if err = copyOpenrestyUpgradeSnapshotEntry(source, path.Join(backupPath, relativePath)); err != nil {
snapshot.Cleanup()
return nil, err
}
}
return snapshot, nil
}
func (s *upgradeFileSnapshot) Restore() error {
for _, relativePath := range s.paths {
target := path.Join(s.installPath, relativePath)
if err := os.RemoveAll(target); err != nil {
return err
}
if !s.existing[relativePath] {
continue
}
if err := copyOpenrestyUpgradeSnapshotEntry(path.Join(s.backupPath, relativePath), target); err != nil {
return err
}
}
return nil
}
func (s *upgradeFileSnapshot) Cleanup() {
if s != nil && s.backupPath != "" {
_ = os.RemoveAll(s.backupPath)
}
}
func discoverUpgradeContainerNames(install model.AppInstall, envContent []byte) ([]string, error) {
project, err := docker.GetComposeProject(install.Name, install.GetPath(), []byte(install.DockerCompose), envContent, false)
if err != nil {
return nil, err
}
expectedServices := make(map[string]struct{})
for _, service := range project.Services {
if !skipCheckStatus(service) {
expectedServices[service.Name] = struct{}{}
}
}
if len(expectedServices) == 0 {
return strings.Split(install.ContainerName, ","), nil
}
client, err := docker.NewDockerClient()
if err != nil {
return nil, err
}
defer client.Close()
containers, err := client.ContainerList(context.Background(), container.ListOptions{
All: true,
Filters: filters.NewArgs(filters.Arg("label", composeWorkdirLabel+"="+install.GetPath())),
})
if err != nil {
return nil, err
}
containerNames := make([]string, 0, len(containers))
for _, item := range containers {
if _, ok := expectedServices[item.Labels[composeServiceLabel]]; ok && len(item.Names) > 0 {
containerNames = append(containerNames, strings.TrimPrefix(item.Names[0], "/"))
}
}
sort.Strings(containerNames)
return containerNames, nil
}
+60 -424
View File
@@ -9,7 +9,6 @@ import (
"math" "math"
"net/http" "net/http"
"os" "os"
"os/exec"
"path" "path"
"path/filepath" "path/filepath"
"reflect" "reflect"
@@ -353,15 +352,33 @@ func deleteAppInstall(deleteReq request.AppInstallDelete) error {
if dir != nil { if dir != nil {
logStr := i18n.GetMsgByKey("Stop") + i18n.GetMsgByKey("App") logStr := i18n.GetMsgByKey("Stop") + i18n.GetMsgByKey("App")
t.Log(logStr) t.Log(logStr)
cleanupFailed := false
out, err := compose.Down(install.GetComposePath()) if deleteReq.UseLifecycleScripts {
if err != nil && !deleteReq.ForceDelete { if scriptErr := runScript(t, &install, "uninstall"); scriptErr != nil {
return handleErr(install, err, out) cleanupFailed = true
if !deleteReq.ForceDelete {
return scriptErr
}
}
} else {
out, downErr := compose.Down(install.GetComposePath())
if downErr != nil {
cleanupFailed = true
if !deleteReq.ForceDelete {
return handleErr(install, downErr, out)
}
}
if scriptErr := runScript(t, &install, "uninstall"); scriptErr != nil {
cleanupFailed = true
if !deleteReq.ForceDelete {
_, _ = compose.Up(install.GetComposePath())
return scriptErr
}
}
} }
t.LogSuccess(logStr) if !cleanupFailed {
if err = runScript(t, &install, "uninstall"); err != nil { t.LogSuccess(logStr)
_, _ = compose.Up(install.GetComposePath())
return err
} }
if deleteReq.DeleteImage { if deleteReq.DeleteImage {
content, err := op.GetContent(install.GetEnvPath()) content, err := op.GetContent(install.GetEnvPath())
@@ -461,8 +478,9 @@ func deleteAppInstall(deleteReq request.AppInstallDelete) error {
} }
uninstallTask.AddSubTask(task.GetTaskName(install.Name, task.TaskUninstall, task.TaskScopeApp), uninstall, nil) uninstallTask.AddSubTask(task.GetTaskName(install.Name, task.TaskUninstall, task.TaskScopeApp), uninstall, nil)
go func() { go func() {
if err := uninstallTask.Execute(); err != nil && !deleteReq.ForceDelete { if err := uninstallTask.Execute(); err != nil {
install.Status = constant.StatusError install.Status = constant.StatusError
install.Message = err.Error()
_ = appInstallRepo.Save(context.Background(), &install) _ = appInstallRepo.Save(context.Background(), &install)
} }
}() }()
@@ -783,416 +801,6 @@ func buildNginx(parentTask *task.Task, nginxInstall model.AppInstall, catalogPat
return commitNginxModuleBuilds(nginxInstall, previousModules, modules, false, catalogPath) return commitNginxModuleBuilds(nginxInstall, previousModules, modules, false, catalogPath)
} }
func upgradeInstall(req request.AppInstallUpgrade) error {
install, err := appInstallRepo.GetFirst(repo.WithByID(req.InstallID))
if err != nil {
return err
}
originalInstall := install
oldVersion := install.Version
detail, err := appDetailRepo.GetFirst(repo.WithByID(req.DetailID))
if err != nil {
return err
}
if install.App.Key == vllmAppKeyForUpgrade && !isVllmUpgradeVersionAllowed(install.Version, detail.Version, loadVllmImageFromEnv(install.Env)) {
return errors.New("vLLM can only upgrade within the same image type")
}
if install.Version == detail.Version {
return errors.New("two version is same")
}
upgradeTask, err := task.NewTaskWithOps(install.Name, task.TaskUpgrade, task.TaskScopeApp, req.TaskID, install.ID)
if err != nil {
return err
}
install.Status = constant.StatusUpgrading
var (
upErr error
backupFile string
nginxUpgradeSnapshot *openrestyUpgradeSnapshot
)
backUpApp := func(t *task.Task) error {
backupService := NewIBackupService()
backupRecordService := NewIBackupRecordService()
fileName := fmt.Sprintf("upgrade_backup_%s_%s.tar.gz", install.Name, time.Now().Format(constant.DateTimeSlimLayout)+common.RandStrAndNum(5))
backupRecord, err := backupService.AppBackup(dto.CommonBackup{Name: install.App.Key, DetailName: install.Name, FileName: fileName})
if err == nil {
backups, _ := backupRecordService.ListAppRecords(install.App.Key, install.Name, "upgrade_backup")
if len(backups) > 3 {
backupsToDelete := backups[:len(backups)-3]
var deleteIDs []uint
for _, backup := range backupsToDelete {
deleteIDs = append(deleteIDs, backup.ID)
}
_ = backupRecordService.BatchDeleteRecord(deleteIDs)
}
backupFile = path.Join(global.Dir.LocalBackupDir, backupRecord.FileDir, backupRecord.FileName)
} else {
return buserr.WithNameAndErr("ErrAppBackup", install.Name, err)
}
return nil
}
if req.Backup {
upgradeTask.AddSubTask(task.GetTaskName(install.Name, task.TaskBackup, task.TaskScopeApp), backUpApp, nil)
}
upgradeApp := func(t *task.Task) error {
fileOp := files.NewFileOp()
detailDir := path.Join(global.Dir.ResourceDir, "apps", install.App.Resource, install.App.Key, detail.Version)
if install.App.Resource == constant.AppResourceRemote {
if err = downloadApp(install.App, detail, &install, t.Logger); err != nil {
return err
}
if detail.DockerCompose == "" {
composeDetail, err := fileOp.GetContent(path.Join(detailDir, "docker-compose.yml"))
if err != nil {
return err
}
detail.DockerCompose = string(composeDetail)
_ = appDetailRepo.Update(context.Background(), detail)
}
go func() {
RequestDownloadCallBack(detail.DownloadCallBackUrl)
}()
}
if install.App.Resource == constant.AppResourceLocal {
detailDir = path.Join(global.Dir.ResourceDir, "apps", "local", strings.TrimPrefix(install.App.Key, "local"), detail.Version)
}
content, err := fileOp.GetContent(install.GetEnvPath())
if err != nil {
return err
}
oldEnvContent := append([]byte(nil), content...)
oldDockerCompose := install.DockerCompose
targetNginxCatalogPath := ""
if install.App.Key == constant.AppOpenresty {
nginxUpgradeSnapshot, err = createOpenrestyUpgradeSnapshot(install.GetPath())
if err != nil {
return err
}
}
if install.App.Key == vllmAppKeyForUpgrade {
envs := make(map[string]interface{})
if err = json.Unmarshal([]byte(install.Env), &envs); err != nil {
return err
}
image := buildVllmUpgradeImage(loadVllmImageFromEnv(install.Env), oldVersion, detail.Version)
envs[vllmImageEnvKey] = image
paramByte, err := json.Marshal(envs)
if err != nil {
return err
}
install.Env = string(paramByte)
content = setVllmImageInEnvContent(content, image)
}
_ = copyAppDetailMissing(fileOp, detailDir, install.GetPath())
if install.App.Key == constant.AppOpenresty {
installBuildDir := path.Join(install.GetPath(), nginxModuleBuildDir)
detailBuildDir := path.Join(detailDir, nginxModuleBuildDir)
if !fileOp.Stat(installBuildDir) {
if err := fileOp.CreateDir(installBuildDir, constant.DirPerm); err != nil {
return err
}
}
if err := fileOp.DeleteDir(path.Join(installBuildDir, nginxModuleTmpDir)); err != nil {
return err
}
if err := fileOp.CopyDir(path.Join(detailBuildDir, nginxModuleTmpDir), installBuildDir); err != nil {
return err
}
if err := fileOp.CopyFile(path.Join(detailBuildDir, "Dockerfile"), installBuildDir); err != nil {
return err
}
if err := syncNginxModuleBuilder(detailBuildDir, installBuildDir); err != nil {
return err
}
targetCatalogSource := path.Join(detailBuildDir, nginxModuleCatalogFile)
if !fileOp.Stat(targetCatalogSource) {
return fmt.Errorf("target OpenResty module catalog not found: %s", targetCatalogSource)
}
targetNginxCatalogPath = path.Join(installBuildDir, nginxModuleCatalogPendingFile)
if err := stageNginxModuleCatalog(targetCatalogSource, targetNginxCatalogPath); err != nil {
return err
}
if err := fileOp.CopyFile(path.Join(detailBuildDir, "nginx.conf"), installBuildDir); err != nil {
return err
}
if err := fileOp.CopyFile(path.Join(detailBuildDir, "nginx.vh.default.conf"), installBuildDir); err != nil {
return err
}
}
sourceScripts := path.Join(detailDir, "scripts")
if fileOp.Stat(sourceScripts) {
dstScripts := path.Join(install.GetPath(), "scripts")
_ = fileOp.DeleteDir(dstScripts)
_ = fileOp.CreateDir(dstScripts, constant.DirPerm)
scriptCmd := exec.Command("cp", "-rf", sourceScripts+"/.", dstScripts+"/")
_, _ = scriptCmd.CombinedOutput()
}
var newCompose string
if err = migrateOpenclawProtocolUpgrade(&install, oldVersion, detail.Version); err != nil {
return err
}
if req.DockerCompose == "" {
if install.App.Key == vllmAppKeyForUpgrade {
newCompose = install.DockerCompose
} else {
newCompose, err = getUpgradeCompose(install, detail)
if err != nil {
return err
}
}
} else {
newCompose = req.DockerCompose
}
install.DockerCompose = newCompose
install.Version = detail.Version
install.AppDetailId = req.DetailID
var oldImageIDs []appImageID
if req.DeleteImage {
dockerCLi, err := docker.NewClient()
if err != nil {
return err
}
oldImageIDs, err = getAppImageIDsByCompose(dockerCLi, oldEnvContent, []byte(oldDockerCompose))
dockerCLi.Close()
if err != nil {
return err
}
}
if req.PullImage {
images, err := docker.GetImagesFromDockerCompose(content, []byte(install.DockerCompose))
if err != nil {
return err
}
dockerCLi, err := docker.NewClient()
if err != nil {
return err
}
defer dockerCLi.Close()
for _, image := range images {
t.Log(i18n.GetWithName("PullImageStart", image))
if pullErr := dockerCLi.PullImageWithProcess(t, image); pullErr != nil {
if exist, _ := dockerCLi.ImageExists(image); exist {
t.Log(i18n.GetMsgByKey("UseExistImage"))
continue
}
return buserr.WithNameAndErr("ErrDockerPullImage", "", pullErr)
}
exist, err := dockerCLi.ImageExists(image)
if err != nil || !exist {
return buserr.WithNameAndErr("ErrDockerPullImage", "", fmt.Errorf("image %s does not exist after pull: %v", image, err))
}
t.LogSuccess(i18n.GetMsgByKey("PullImage"))
}
}
if install.App.Key == constant.AppOpenresty {
modules, moduleErr := loadNginxModulesWithCatalog(install, targetNginxCatalogPath)
if moduleErr != nil {
return moduleErr
}
// Build dynamic modules for the target version before stopping the
// current container. Static modules retain the full rebuild path.
if !hasEnabledStaticNginxModules(modules) {
previousModules := cloneNginxModules(modules)
modules, moduleErr = buildDynamicNginxModules(install, modules, nil, false, "", targetNginxCatalogPath, t)
if moduleErr != nil {
return moduleErr
}
if moduleErr = saveNginxModulesWithCatalog(install, modules, targetNginxCatalogPath); moduleErr != nil {
removeNginxModuleOutputsNotReferenced(install, modules, previousModules)
return moduleErr
}
}
}
if out, err := compose.Down(install.GetComposePath()); err != nil {
if out != "" {
upErr = errors.New(out)
return upErr
}
return err
}
envs := make(map[string]interface{})
if err = json.Unmarshal([]byte(install.Env), &envs); err != nil {
return err
}
envParams := make(map[string]string, len(envs))
if install.App.Key == constant.AppOpenresty {
packageUrl, _ := env.GetEnvValueByKey(install.GetEnvPath(), "CONTAINER_PACKAGE_URL")
addPackage, _ := env.GetEnvValueByKey(install.GetEnvPath(), "RESTY_ADD_PACKAGE_BUILDDEPS")
options, _ := env.GetEnvValueByKey(install.GetEnvPath(), "RESTY_CONFIG_OPTIONS_MORE")
envParams["CONTAINER_PACKAGE_URL"] = packageUrl
envParams["RESTY_ADD_PACKAGE_BUILDDEPS"] = addPackage
envParams["RESTY_CONFIG_OPTIONS_MORE"] = options
}
handleMap(envs, envParams)
if err = env.Write(envParams, install.GetEnvPath()); err != nil {
return err
}
if err = runScript(t, &install, "upgrade"); err != nil {
return err
}
if err = fileOp.WriteFile(install.GetComposePath(), strings.NewReader(install.DockerCompose), constant.FilePerm); err != nil {
return err
}
if install.App.Key == constant.AppOpenresty {
if err = buildNginx(t, install, targetNginxCatalogPath); err != nil {
t.Log(err.Error())
return err
}
}
logStr := fmt.Sprintf("%s %s", i18n.GetMsgByKey("Run"), i18n.GetMsgByKey("App"))
t.Log(logStr)
if out, err := compose.Up(install.GetComposePath()); err != nil {
if out != "" {
return errors.New(out)
}
return err
}
t.LogSuccess(logStr)
install.Status = constant.StatusRunning
if install.App.Key == constant.AppOpenresty {
if err = commitStaticNginxModuleBuilds(install, targetNginxCatalogPath, t); err != nil {
return err
}
activeCatalogPath := path.Join(install.GetPath(), nginxModuleBuildDir, nginxModuleCatalogFile)
if err = activateNginxModuleCatalogAndCommit(targetNginxCatalogPath, activeCatalogPath, func() error {
return appInstallRepo.Save(context.Background(), &install)
}); err != nil {
return err
}
} else {
if err = appInstallRepo.Save(context.Background(), &install); err != nil {
return err
}
}
if nginxUpgradeSnapshot != nil {
nginxUpgradeSnapshot.Cleanup()
nginxUpgradeSnapshot = nil
}
if req.DeleteImage {
newEnvContent, err := fileOp.GetContent(install.GetEnvPath())
if err != nil {
t.LogFailedWithErr(i18n.GetMsgByKey("TaskDelete")+i18n.GetMsgByKey("Image"), err)
return nil
}
excludeImages, err := docker.GetImagesFromDockerCompose(newEnvContent, []byte(install.DockerCompose))
if err != nil {
t.LogFailedWithErr(i18n.GetMsgByKey("TaskDelete")+i18n.GetMsgByKey("Image"), err)
return nil
}
dockerCLi, err := docker.NewClient()
if err != nil {
t.LogFailedWithErr(i18n.GetMsgByKey("TaskDelete")+i18n.GetMsgByKey("Image"), err)
return nil
}
defer dockerCLi.Close()
if err = deleteAppImagesByIDs(t, dockerCLi, oldImageIDs, excludeImages); err != nil {
t.LogFailedWithErr(i18n.GetMsgByKey("TaskDelete")+i18n.GetMsgByKey("Image"), err)
}
}
return nil
}
rollBackApp := func(t *task.Task) {
if req.Backup {
t.Log(i18n.GetWithName("AppRecover", install.Name))
recoverErr := NewIBackupService().AppRecover(dto.CommonRecover{
Name: install.App.Key, DetailName: install.Name, Type: "app", DownloadAccountID: 1, File: backupFile,
})
if recoverErr == nil {
if nginxUpgradeSnapshot != nil {
nginxUpgradeSnapshot.Cleanup()
nginxUpgradeSnapshot = nil
}
t.LogSuccess(i18n.GetWithName("AppRecover", install.Name))
return
}
t.LogFailedWithErr(i18n.GetWithName("AppRecover", install.Name), recoverErr)
if install.App.Key != constant.AppOpenresty {
return
}
}
if install.App.Key == constant.AppOpenresty && nginxUpgradeSnapshot != nil {
if out, rollbackErr := compose.Down(install.GetComposePath()); rollbackErr != nil {
if out != "" {
rollbackErr = fmt.Errorf("%s: %w", out, rollbackErr)
}
t.LogFailedWithErr(i18n.GetWithName("AppRecover", install.Name), rollbackErr)
}
if rollbackErr := nginxUpgradeSnapshot.Restore(); rollbackErr != nil {
t.LogFailedWithErr(i18n.GetWithName("AppRecover", install.Name), rollbackErr)
return
}
nginxUpgradeSnapshot.Cleanup()
nginxUpgradeSnapshot = nil
if out, rollbackErr := compose.Up(originalInstall.GetComposePath()); rollbackErr != nil {
if out != "" {
rollbackErr = fmt.Errorf("%s: %w", out, rollbackErr)
}
t.LogFailedWithErr(i18n.GetWithName("AppRecover", install.Name), rollbackErr)
return
}
originalInstall.Status = constant.StatusRunning
originalInstall.Message = ""
if rollbackErr := appInstallRepo.Save(context.Background(), &originalInstall); rollbackErr != nil {
t.LogFailedWithErr(i18n.GetWithName("AppRecover", install.Name), rollbackErr)
return
}
install = originalInstall
t.LogSuccess(i18n.GetWithName("AppRecover", install.Name))
return
}
if install.App.Key == constant.AppOpenresty {
if rollbackErr := appInstallRepo.Save(context.Background(), &originalInstall); rollbackErr != nil {
t.LogFailedWithErr(i18n.GetWithName("AppRecover", install.Name), rollbackErr)
return
}
install = originalInstall
t.LogSuccess(i18n.GetWithName("AppRecover", install.Name))
}
}
upgradeTimeout := 1 * time.Hour
if install.App.Key == constant.AppOpenresty {
// Dynamic modules are built serially and each Docker build has its own
// timeout. An outer deadline would start rollback while upgradeApp is
// still mutating the installation because SubTask does not stop its
// action goroutine on timeout.
upgradeTimeout = 0
}
upgradeTask.AddSubTaskWithOps(task.GetTaskName(install.Name, task.TaskUpgrade, task.TaskScopeApp), upgradeApp, rollBackApp, 0, upgradeTimeout)
upgradingInstall := install
if err = appInstallRepo.Save(context.Background(), &upgradingInstall); err != nil {
return err
}
go func() {
if taskErr := upgradeTask.Execute(); taskErr != nil {
existInstall, _ := appInstallRepo.GetFirst(repo.WithByID(req.InstallID))
if existInstall.ID > 0 && existInstall.Status != constant.StatusRunning {
existInstall.Status = constant.StatusUpgradeErr
existInstall.Message = taskErr.Error()
_ = appInstallRepo.Save(context.Background(), &existInstall)
}
}
}()
return nil
}
func skipCheckStatus(service types.ServiceConfig) bool { func skipCheckStatus(service types.ServiceConfig) bool {
for key := range service.Labels { for key := range service.Labels {
if key == "skipStatusCheck" { if key == "skipStatusCheck" {
@@ -1410,6 +1018,12 @@ func runScript(task *task.Task, appInstall *model.AppInstall, operate string) er
scriptPath = path.Join(workDir, "scripts", "upgrade.sh") scriptPath = path.Join(workDir, "scripts", "upgrade.sh")
case "uninstall": case "uninstall":
scriptPath = path.Join(workDir, "scripts", "uninstall.sh") scriptPath = path.Join(workDir, "scripts", "uninstall.sh")
case "start":
scriptPath = path.Join(workDir, "scripts", "start.sh")
case "stop":
scriptPath = path.Join(workDir, "scripts", "stop.sh")
case "restart":
scriptPath = path.Join(workDir, "scripts", "restart.sh")
} }
fileOp := files.NewFileOp() fileOp := files.NewFileOp()
if !fileOp.Stat(scriptPath) { if !fileOp.Stat(scriptPath) {
@@ -1419,7 +1033,11 @@ func runScript(task *task.Task, appInstall *model.AppInstall, operate string) er
logStr := i18n.GetWithName("ExecShell", operate) logStr := i18n.GetWithName("ExecShell", operate)
task.LogStart(logStr) task.LogStart(logStr)
cmdMgr := cmd.NewCommandMgr(cmd.WithTimeout(10*time.Minute), cmd.WithWorkDir(workDir)) timeout := 10 * time.Minute
if operate == "start" || operate == "restart" {
timeout = time.Hour
}
cmdMgr := cmd.NewCommandMgr(cmd.WithTimeout(timeout), cmd.WithWorkDir(workDir), cmd.WithTask(*task))
if err := cmdMgr.Run("bash", scriptPath); err != nil { if err := cmdMgr.Run("bash", scriptPath); err != nil {
task.LogFailedWithErr(logStr, err) task.LogFailedWithErr(logStr, err)
return err return err
@@ -1454,12 +1072,15 @@ func checkContainerNameIsExist(containerName, appDir string) (bool, error) {
return false, nil return false, nil
} }
func upApp(task *task.Task, appInstall *model.AppInstall, pullImages bool) error { func upApp(task *task.Task, appInstall *model.AppInstall, pullImages, useLifecycleScripts bool) error {
upProject := func(appInstall *model.AppInstall) (err error) { upProject := func(appInstall *model.AppInstall) (err error) {
var ( var (
out string out string
errMsg string errMsg string
) )
if useLifecycleScripts {
return runScript(task, appInstall, "start")
}
if pullImages && appInstall.App.Type != "php" { if pullImages && appInstall.App.Type != "php" {
envByte, err := files.NewFileOp().GetContent(appInstall.GetEnvPath()) envByte, err := files.NewFileOp().GetContent(appInstall.GetEnvPath())
if err != nil { if err != nil {
@@ -1785,8 +1406,8 @@ func handleErr(install model.AppInstall, err error, out string) error {
} }
func doNotNeedSync(installed model.AppInstall) bool { func doNotNeedSync(installed model.AppInstall) bool {
return installed.Status == constant.StatusInstalling || installed.Status == constant.StatusRebuilding || installed.Status == constant.StatusUpgrading || return appInstallOperationPending(installed.Status) || installed.Status == constant.StatusSyncing ||
installed.Status == constant.StatusSyncing || installed.Status == constant.StatusUninstalling || installed.Status == constant.StatusInstallErr installed.Status == constant.StatusInstallErr || installed.Status == constant.StatusUpgradeErr || installed.Status == constant.StatusUpErr
} }
func synAppInstall(containers map[string]container.Summary, appInstall *model.AppInstall, force bool) { func synAppInstall(containers map[string]container.Summary, appInstall *model.AppInstall, force bool) {
@@ -1880,6 +1501,11 @@ func handleInstalled(appInstallList []model.AppInstall, updated, sync, checkUpda
} }
for _, installed := range appInstallList { for _, installed := range appInstallList {
if sync && appInstallOperationPending(installed.Status) {
if err := syncAppInstallStatus(&installed, false); err != nil {
return nil, err
}
}
if updated && ignoreUpdate(installed) { if updated && ignoreUpdate(installed) {
continue continue
} }
@@ -2233,6 +1859,10 @@ func isHostModel(dockerCompose string) bool {
} }
func copyAppDetailMissing(fileOp files.FileOp, srcDir, dstDir string) error { func copyAppDetailMissing(fileOp files.FileOp, srcDir, dstDir string) error {
return copyAppDetailMissingTracked(fileOp, srcDir, dstDir, nil)
}
func copyAppDetailMissingTracked(fileOp files.FileOp, srcDir, dstDir string, createdPaths *[]string) error {
entries, err := os.ReadDir(srcDir) entries, err := os.ReadDir(srcDir)
if err != nil { if err != nil {
return err return err
@@ -2244,6 +1874,9 @@ func copyAppDetailMissing(fileOp files.FileOp, srcDir, dstDir string) error {
srcPath := path.Join(srcDir, entry.Name()) srcPath := path.Join(srcDir, entry.Name())
dstPath := path.Join(dstDir, entry.Name()) dstPath := path.Join(dstDir, entry.Name())
if !fileOp.Stat(dstPath) { if !fileOp.Stat(dstPath) {
if createdPaths != nil {
*createdPaths = append(*createdPaths, dstPath)
}
if entry.IsDir() { if entry.IsDir() {
if err := fileOp.CopyDir(srcPath, dstDir); err != nil { if err := fileOp.CopyDir(srcPath, dstDir); err != nil {
return err return err
@@ -2258,7 +1891,7 @@ func copyAppDetailMissing(fileOp files.FileOp, srcDir, dstDir string) error {
if !entry.IsDir() { if !entry.IsDir() {
continue continue
} }
if err := copyAppDetailMissing(fileOp, srcPath, dstPath); err != nil { if err := copyAppDetailMissingTracked(fileOp, srcPath, dstPath, createdPaths); err != nil {
return err return err
} }
} }
@@ -2633,6 +2266,9 @@ func getAppVersions(key string, details []model.AppDetail) []string {
hasLatest := false hasLatest := false
latestVersion := "" latestVersion := ""
for _, detail := range details { for _, detail := range details {
if !canAccessVllmVersion(key, detail.Version) {
continue
}
if key != "mssql" && strings.Contains(detail.Version, "latest") { if key != "mssql" && strings.Contains(detail.Version, "latest") {
hasLatest = true hasLatest = true
latestVersion = detail.Version latestVersion = detail.Version
+33 -1
View File
@@ -90,6 +90,34 @@ func (u *BackupService) AppBackup(req dto.CommonBackup) (*model.BackupRecord, er
return record, nil return record, nil
} }
func backupAppWithParentTask(install *model.AppInstall, parentTask *task.Task, fileName string) (*model.BackupRecord, error) {
itemDir := fmt.Sprintf("app/%s/%s", install.App.Key, install.Name)
backupDir := path.Join(global.Dir.LocalBackupDir, itemDir)
record := &model.BackupRecord{
Type: "app",
Name: install.App.Key,
DetailName: install.Name,
SourceAccountIDs: "1",
DownloadAccountID: 1,
FileDir: itemDir,
FileName: fileName,
TaskID: parentTask.TaskID,
Status: constant.StatusWaiting,
}
if err := backupRepo.CreateRecord(record); err != nil {
return nil, err
}
if err := handleAppBackup(install, parentTask, record.ID, backupDir, fileName, "", "", parentTask.TaskID); err != nil {
markBackupFailed(record.ID, err)
record.Status = constant.StatusFailed
record.Message = err.Error()
return record, err
}
backupRepo.UpdateRecordByMap(record.ID, map[string]interface{}{"status": constant.StatusSuccess})
record.Status = constant.StatusSuccess
return record, nil
}
func (u *BackupService) AppRecover(req dto.CommonRecover) error { func (u *BackupService) AppRecover(req dto.CommonRecover) error {
app, err := appRepo.GetFirst(appRepo.WithKey(req.Name)) app, err := appRepo.GetFirst(appRepo.WithKey(req.Name))
if err != nil { if err != nil {
@@ -203,7 +231,11 @@ func handleAppRecover(install *model.AppInstall, parentTask *task.Task, recoverF
return err return err
} }
defer func() { defer func() {
_, _ = compose.Up(install.GetComposePath()) if isRollback {
_, _ = compose.UpWithoutBuild(install.GetComposePath())
} else {
_, _ = compose.Up(install.GetComposePath())
}
_ = os.RemoveAll(strings.ReplaceAll(recoverFile, ".tar.gz", "")) _ = os.RemoveAll(strings.ReplaceAll(recoverFile, ".tar.gz", ""))
}() }()
+28 -54
View File
@@ -582,6 +582,10 @@ func stepRecreateContainer(recoverCtx *containerRecoverContext, taskItem *task.T
if config.Image == "" { if config.Image == "" {
return fmt.Errorf("container image not found in backup file") return fmt.Errorf("container image not found in backup file")
} }
networkConf, extraNetworks := buildContainerRecoverNetworkConfig(recoverCtx.inspectInfo.NetworkSettings, hostConfig)
if err := normalizeContainerEndpointSettings(ctx, recoverCtx.client, networkConf, extraNetworks); err != nil {
return err
}
if !checkImageExist(recoverCtx.client, config.Image) { if !checkImageExist(recoverCtx.client, config.Image) {
if err := pullImages(taskItem, recoverCtx.client, config.Image); err != nil { if err := pullImages(taskItem, recoverCtx.client, config.Image); err != nil {
return err return err
@@ -596,7 +600,7 @@ func stepRecreateContainer(recoverCtx *containerRecoverContext, taskItem *task.T
return err return err
} }
createRes, err := createContainerWithOldNetworks(ctx, recoverCtx.client, config, hostConfig, recoverCtx.inspectInfo.NetworkSettings, recoverCtx.targetName) createRes, err := createContainerWithNetworks(ctx, recoverCtx.client, config, hostConfig, networkConf, extraNetworks, recoverCtx.targetName)
if err != nil { if err != nil {
return err return err
} }
@@ -604,7 +608,7 @@ func stepRecreateContainer(recoverCtx *containerRecoverContext, taskItem *task.T
return nil return nil
} }
func normalizeContainerEndpointSettings(ctx context.Context, cli *client.Client, primary *network.NetworkingConfig, extras map[string]*network.EndpointSettings) { func normalizeContainerEndpointSettings(ctx context.Context, cli *client.Client, primary *network.NetworkingConfig, extras map[string]*network.EndpointSettings) error {
if cli.NewVersionError(ctx, "1.44", "specify mac-address per network") != nil { if cli.NewVersionError(ctx, "1.44", "specify mac-address per network") != nil {
removeEndpointMacAddresses(primary, extras) removeEndpointMacAddresses(primary, extras)
} }
@@ -619,11 +623,14 @@ func normalizeContainerEndpointSettings(ctx context.Context, cli *client.Client,
} }
info, err := cli.NetworkInspect(ctx, netName, network.InspectOptions{}) info, err := cli.NetworkInspect(ctx, netName, network.InspectOptions{})
if err != nil { if err != nil {
continue return fmt.Errorf("inspect network %s failed: %w", netName, err)
}
if err := validateContainerEndpointStaticIP(netName, info, endpoint); err != nil {
return err
} }
removeUnsupportedEndpointStaticIP(netName, info, endpoint)
} }
} }
return nil
} }
func removeEndpointMacAddresses(primary *network.NetworkingConfig, extras map[string]*network.EndpointSettings) { func removeEndpointMacAddresses(primary *network.NetworkingConfig, extras map[string]*network.EndpointSettings) {
@@ -641,24 +648,28 @@ func removeEndpointMacAddresses(primary *network.NetworkingConfig, extras map[st
} }
} }
func removeUnsupportedEndpointStaticIP(netName string, info network.Inspect, endpoint *network.EndpointSettings) { func validateContainerEndpointStaticIP(netName string, info network.Inspect, endpoint *network.EndpointSettings) error {
if endpoint == nil || endpoint.IPAMConfig == nil { if endpoint == nil || endpoint.IPAMConfig == nil {
return return nil
} }
if isDefaultBridgeNetwork(netName, info) { ipam := endpoint.IPAMConfig
endpoint.IPAMConfig = nil if err := ipam.Validate(); err != nil {
return return fmt.Errorf("invalid IP configuration for network %s: %w", netName, err)
}
if ipam.IPv4Address == "" && ipam.IPv6Address == "" {
return nil
}
if netName == "host" || netName == "none" || isDefaultBridgeNetwork(netName, info) {
return fmt.Errorf("network %s does not support static IP configuration", netName)
} }
if endpoint.IPAMConfig.IPv4Address != "" && !networkSupportsStaticIP(info, endpoint.IPAMConfig.IPv4Address, false) { if ipam.IPv4Address != "" && !networkSupportsStaticIP(info, ipam.IPv4Address, false) {
endpoint.IPAMConfig.IPv4Address = "" return fmt.Errorf("static IPv4 address %s is not in a configured subnet of network %s", ipam.IPv4Address, netName)
} }
if endpoint.IPAMConfig.IPv6Address != "" && !networkSupportsStaticIP(info, endpoint.IPAMConfig.IPv6Address, true) { if ipam.IPv6Address != "" && !networkSupportsStaticIP(info, ipam.IPv6Address, true) {
endpoint.IPAMConfig.IPv6Address = "" return fmt.Errorf("static IPv6 address %s is not in a configured subnet of network %s", ipam.IPv6Address, netName)
}
if endpoint.IPAMConfig.IPv4Address == "" && endpoint.IPAMConfig.IPv6Address == "" && len(endpoint.IPAMConfig.LinkLocalIPs) == 0 {
endpoint.IPAMConfig = nil
} }
return nil
} }
func isDefaultBridgeNetwork(netName string, info network.Inspect) bool { func isDefaultBridgeNetwork(netName string, info network.Inspect) bool {
@@ -673,6 +684,7 @@ func networkSupportsStaticIP(info network.Inspect, ip string, isIPv6 bool) bool
if err != nil { if err != nil {
return false return false
} }
addr = addr.Unmap()
if addr.Is6() != isIPv6 { if addr.Is6() != isIPv6 {
return false return false
} }
@@ -813,11 +825,6 @@ func buildContainerRecoverNetworkConfig(networkSettings *container.NetworkSettin
IPv6Address: endpoint.IPAMConfig.IPv6Address, IPv6Address: endpoint.IPAMConfig.IPv6Address,
LinkLocalIPs: append([]string(nil), endpoint.IPAMConfig.LinkLocalIPs...), LinkLocalIPs: append([]string(nil), endpoint.IPAMConfig.LinkLocalIPs...),
} }
} else if name != "bridge" && (endpoint.IPAddress != "" || endpoint.GlobalIPv6Address != "") {
endpointSetting.IPAMConfig = &network.EndpointIPAMConfig{
IPv4Address: endpoint.IPAddress,
IPv6Address: endpoint.GlobalIPv6Address,
}
} }
if name == primaryName { if name == primaryName {
config.EndpointsConfig[name] = endpointSetting config.EndpointsConfig[name] = endpointSetting
@@ -831,39 +838,6 @@ func buildContainerRecoverNetworkConfig(networkSettings *container.NetworkSettin
return config, extraNetworks return config, extraNetworks
} }
const unsupportedUserSpecifiedIPAddress = "user specified IP address is supported only when connecting to networks with user configured subnets"
func clearUnsupportedDynamicEndpointIPAM(err error, endpoints map[string]*network.EndpointSettings, networkSettings *container.NetworkSettings) bool {
if err == nil || !strings.Contains(err.Error(), unsupportedUserSpecifiedIPAddress) {
return false
}
for name, endpoint := range endpoints {
if !isDynamicContainerNetwork(networkSettings, name) || endpoint == nil || endpoint.IPAMConfig == nil {
continue
}
if strings.Contains(err.Error(), "network "+name+":") {
endpoint.IPAMConfig = nil
return true
}
}
cleared := false
for name, endpoint := range endpoints {
if isDynamicContainerNetwork(networkSettings, name) && endpoint != nil && endpoint.IPAMConfig != nil {
endpoint.IPAMConfig = nil
cleared = true
}
}
return cleared
}
func isDynamicContainerNetwork(networkSettings *container.NetworkSettings, name string) bool {
if networkSettings == nil || name == "bridge" {
return false
}
endpoint := networkSettings.Networks[name]
return endpoint != nil && endpoint.IPAMConfig == nil && (endpoint.IPAddress != "" || endpoint.GlobalIPv6Address != "")
}
func cloneContainerConfig(config *container.Config) *container.Config { func cloneContainerConfig(config *container.Config) *container.Config {
if config == nil { if config == nil {
return &container.Config{} return &container.Config{}
+41
View File
@@ -2,6 +2,7 @@ package service
import ( import (
"context" "context"
"encoding/json"
"fmt" "fmt"
"os" "os"
"path" "path"
@@ -39,6 +40,7 @@ func (u *BackupService) MysqlBackup(req dto.CommonBackup) error {
TaskID: req.TaskID, TaskID: req.TaskID,
Status: constant.StatusWaiting, Status: constant.StatusWaiting,
Description: req.Description, Description: req.Description,
Args: encodeBackupArgs(req.Args),
} }
if err := backupRepo.CreateRecord(record); err != nil { if err := backupRepo.CreateRecord(record); err != nil {
global.LOG.Errorf("save backup record failed, err: %v", err) global.LOG.Errorf("save backup record failed, err: %v", err)
@@ -143,6 +145,14 @@ func handleMysqlRecover(req dto.CommonRecover, parentTask *task.Task, isRollback
if !isRollback { if !isRollback {
rollbackFile := path.Join(global.Dir.TmpDir, fmt.Sprintf("database/%s/%s_%s.sql.gz", req.Type, req.DetailName, time.Now().Format(constant.DateTimeSlimLayout))) rollbackFile := path.Join(global.Dir.TmpDir, fmt.Sprintf("database/%s/%s_%s.sql.gz", req.Type, req.DetailName, time.Now().Format(constant.DateTimeSlimLayout)))
var rollbackArgs []string
if req.BackupRecordID != 0 {
record, err := backupRepo.GetRecord(repo.WithByID(req.BackupRecordID))
if err != nil {
return err
}
rollbackArgs = decodeBackupArgs(record.Args)
}
if err := cli.Backup(client.BackupInfo{ if err := cli.Backup(client.BackupInfo{
Name: req.DetailName, Name: req.DetailName,
Type: req.Type, Type: req.Type,
@@ -150,6 +160,7 @@ func handleMysqlRecover(req dto.CommonRecover, parentTask *task.Task, isRollback
Format: dbInfo.Format, Format: dbInfo.Format,
TargetDir: path.Dir(rollbackFile), TargetDir: path.Dir(rollbackFile),
FileName: path.Base(rollbackFile), FileName: path.Base(rollbackFile),
Args: rollbackArgs,
}); err != nil { }); err != nil {
return fmt.Errorf("backup mysql db %s for rollback before recover failed, err: %v", req.DetailName, err) return fmt.Errorf("backup mysql db %s for rollback before recover failed, err: %v", req.DetailName, err)
} }
@@ -242,6 +253,36 @@ func doMysqlBackup(db DatabaseHelper, targetDir, fileName, secret string) error
return nil return nil
} }
func encodeBackupArgs(args []string) string {
var items []string
for _, arg := range args {
if len(arg) != 0 {
items = append(items, arg)
}
}
if len(items) == 0 {
return ""
}
data, err := json.Marshal(items)
if err != nil {
global.LOG.Warnf("marshal backup args failed: %v", err)
return ""
}
return string(data)
}
func decodeBackupArgs(value string) []string {
if len(value) == 0 {
return nil
}
var args []string
if err := json.Unmarshal([]byte(value), &args); err != nil {
global.LOG.Warnf("unmarshal backup args failed: %v", err)
return nil
}
return args
}
func loadSqlFile(file string) (string, error) { func loadSqlFile(file string) (string, error) {
if !strings.HasSuffix(file, ".tar.gz") && !strings.HasSuffix(file, ".zip") { if !strings.HasSuffix(file, ".tar.gz") && !strings.HasSuffix(file, ".zip") {
return file, nil return file, nil
+72 -139
View File
@@ -16,6 +16,7 @@ import (
"path" "path"
"path/filepath" "path/filepath"
"regexp" "regexp"
"slices"
"sort" "sort"
"strconv" "strconv"
"strings" "strings"
@@ -24,6 +25,7 @@ import (
"time" "time"
"github.com/1Panel-dev/1Panel/agent/app/dto" "github.com/1Panel-dev/1Panel/agent/app/dto"
"github.com/1Panel-dev/1Panel/agent/app/model"
"github.com/1Panel-dev/1Panel/agent/app/repo" "github.com/1Panel-dev/1Panel/agent/app/repo"
"github.com/1Panel-dev/1Panel/agent/app/task" "github.com/1Panel-dev/1Panel/agent/app/task"
"github.com/1Panel-dev/1Panel/agent/buserr" "github.com/1Panel-dev/1Panel/agent/buserr"
@@ -71,6 +73,7 @@ type IContainerService interface {
ComposeOperation(req dto.ComposeOperation) error ComposeOperation(req dto.ComposeOperation) error
TestCompose(req dto.ComposeCreate) (bool, error) TestCompose(req dto.ComposeCreate) (bool, error)
ComposeUpdate(req dto.ComposeUpdate) error ComposeUpdate(req dto.ComposeUpdate) error
ComposePin(req dto.ComposePin) error
ComposeLogClean(req dto.ComposeLogClean) error ComposeLogClean(req dto.ComposeLogClean) error
ContainerCreate(req dto.ContainerOperate, inThread bool) error ContainerCreate(req dto.ContainerOperate, inThread bool) error
@@ -78,7 +81,7 @@ type IContainerService interface {
ContainerUpgrade(req dto.ContainerUpgrade) error ContainerUpgrade(req dto.ContainerUpgrade) error
ContainerInfo(req dto.OperationWithName) (*dto.ContainerOperate, error) ContainerInfo(req dto.OperationWithName) (*dto.ContainerOperate, error)
ContainerListStats() ([]dto.ContainerListStats, error) ContainerListStats() ([]dto.ContainerListStats, error)
ContainerItemStats(req dto.OperationWithName) (dto.ContainerItemStats, error) ContainerItemStats(ctx context.Context, req dto.OperationWithName) (dto.ContainerItemStats, error)
LoadResourceLimit() (*dto.ResourceLimit, error) LoadResourceLimit() (*dto.ResourceLimit, error)
ContainerRename(req dto.ContainerRename) error ContainerRename(req dto.ContainerRename) error
ContainerCommit(req dto.ContainerCommit) error ContainerCommit(req dto.ContainerCommit) error
@@ -245,15 +248,15 @@ func (u *ContainerService) LoadStatus() (dto.ContainerStatus, error) {
} }
return data, nil return data, nil
} }
func (u *ContainerService) ContainerItemStats(req dto.OperationWithName) (dto.ContainerItemStats, error) { func (u *ContainerService) ContainerItemStats(ctx context.Context, req dto.OperationWithName) (dto.ContainerItemStats, error) {
var data dto.ContainerItemStats var data dto.ContainerItemStats
client, err := docker.NewDockerClient() client, err := docker.NewDockerClient()
if err != nil { if err != nil {
return data, err return data, err
} }
defer client.Close()
if req.Name != "system" { if req.Name != "system" {
defer client.Close() containerInfo, _, err := client.ContainerInspectWithRaw(ctx, req.Name, true)
containerInfo, _, err := client.ContainerInspectWithRaw(context.Background(), req.Name, true)
if err != nil { if err != nil {
return data, err return data, err
} }
@@ -262,7 +265,7 @@ func (u *ContainerService) ContainerItemStats(req dto.OperationWithName) (dto.Co
return data, nil return data, nil
} }
usage, err := client.DiskUsage(context.Background(), types.DiskUsageOptions{}) usage, err := client.DiskUsage(ctx, types.DiskUsageOptions{})
if err != nil { if err != nil {
return data, err return data, err
} }
@@ -380,11 +383,6 @@ func (u *ContainerService) Inspect(req dto.InspectReq) (string, error) {
} }
func (u *ContainerService) Prune(req dto.ContainerPrune) error { func (u *ContainerService) Prune(req dto.ContainerPrune) error {
client, err := docker.NewDockerClient()
if err != nil {
return err
}
defer client.Close()
name := "" name := ""
switch req.PruneType { switch req.PruneType {
case "container": case "container":
@@ -405,6 +403,14 @@ func (u *ContainerService) Prune(req dto.ContainerPrune) error {
} }
taskItem.AddSubTask(i18n.GetMsgByKey("TaskClean"), func(t *task.Task) error { taskItem.AddSubTask(i18n.GetMsgByKey("TaskClean"), func(t *task.Task) error {
if err := t.TaskCtx.Err(); err != nil {
return err
}
client, err := docker.NewDockerClient()
if err != nil {
return err
}
defer client.Close()
pruneFilters := filters.NewArgs() pruneFilters := filters.NewArgs()
if req.WithTagAll { if req.WithTagAll {
pruneFilters.Add("dangling", "false") pruneFilters.Add("dangling", "false")
@@ -428,10 +434,7 @@ func (u *ContainerService) Prune(req dto.ContainerPrune) error {
} }
SpaceReclaimed = int(rep.SpaceReclaimed) SpaceReclaimed = int(rep.SpaceReclaimed)
case "network": case "network":
_, err := client.NetworksPrune(context.Background(), pruneFilters) return cleanUnusedNetworks(t, client)
if err != nil {
return err
}
case "volume": case "volume":
versions, err := client.ServerVersion(context.Background()) versions, err := client.ServerVersion(context.Background())
if err != nil { if err != nil {
@@ -533,7 +536,9 @@ func (u *ContainerService) ContainerCreate(req dto.ContainerOperate, inThread bo
if err != nil { if err != nil {
return err return err
} }
normalizeContainerEndpointSettings(ctx, client, networkConf, nil) if err := normalizeContainerEndpointSettings(ctx, client, networkConf, nil); err != nil {
return err
}
con, err := client.ContainerCreate(ctx, config, hostConf, networkConf, &v1.Platform{}, req.Name) con, err := client.ContainerCreate(ctx, config, hostConf, networkConf, &v1.Platform{}, req.Name)
if err != nil { if err != nil {
taskItem.Log(i18n.GetMsgByKey("ContainerCreateFailed")) taskItem.Log(i18n.GetMsgByKey("ContainerCreateFailed"))
@@ -643,14 +648,9 @@ func loadContainerNetworkInfo(name string, endpoint *network.EndpointSettings) d
if endpoint.IPAMConfig != nil { if endpoint.IPAMConfig != nil {
item.LinkLocalIPs = append([]string(nil), endpoint.IPAMConfig.LinkLocalIPs...) item.LinkLocalIPs = append([]string(nil), endpoint.IPAMConfig.LinkLocalIPs...)
} }
if name != "bridge" { if name != "bridge" && endpoint.IPAMConfig != nil {
if endpoint.IPAMConfig != nil { item.Ipv4 = endpoint.IPAMConfig.IPv4Address
item.Ipv4 = endpoint.IPAMConfig.IPv4Address item.Ipv6 = endpoint.IPAMConfig.IPv6Address
item.Ipv6 = endpoint.IPAMConfig.IPv6Address
} else {
item.Ipv4 = endpoint.IPAddress
item.Ipv6 = endpoint.GlobalIPv6Address
}
} }
return item return item
} }
@@ -1677,30 +1677,42 @@ func checkImageLike(client *client.Client, imageName string) bool {
func pullImages(task *task.Task, client *client.Client, imageName string) error { func pullImages(task *task.Task, client *client.Client, imageName string) error {
dockerCli := docker.NewClientWithExist(client) dockerCli := docker.NewClientWithExist(client)
repos, err := imageRepoRepo.List()
if err != nil {
return err
}
imageRepo := selectImageRepo(imageName, repos)
if imageRepo == nil || !imageRepo.Auth {
return dockerCli.PullImageWithProcess(task, imageName)
}
options := image.PullOptions{} options := image.PullOptions{}
repos, _ := imageRepoRepo.List() authConfig := registry.AuthConfig{
if len(repos) != 0 { Username: imageRepo.Username,
for _, repo := range repos { Password: imageRepo.Password,
if strings.HasPrefix(imageName, repo.DownloadUrl) && repo.Auth { }
authConfig := registry.AuthConfig{ encodedJSON, err := json.Marshal(authConfig)
Username: repo.Username, if err != nil {
Password: repo.Password, return err
} }
encodedJSON, err := json.Marshal(authConfig) options.RegistryAuth = base64.URLEncoding.EncodeToString(encodedJSON)
if err != nil { return dockerCli.PullImageWithProcessAndOptions(task, imageName, options)
return err }
}
authStr := base64.URLEncoding.EncodeToString(encodedJSON) func selectImageRepo(imageName string, repos []model.ImageRepo) *model.ImageRepo {
options.RegistryAuth = authStr var selected *model.ImageRepo
} selectedURLLength := 0
for i := range repos {
downloadURL := strings.TrimRight(strings.TrimSpace(repos[i].DownloadUrl), "/")
if downloadURL == "" || !strings.HasPrefix(imageName, downloadURL+"/") {
continue
} }
} else { if len(downloadURL) > selectedURLLength {
hasAuth, authStr := loadAuthInfo(imageName) selected = &repos[i]
if hasAuth { selectedURLLength = len(downloadURL)
options.RegistryAuth = authStr
} }
} }
return dockerCli.PullImageWithProcessAndOptions(task, imageName, options) return selected
} }
func loadCpuAndMem(client *client.Client, containerItem string) dto.ContainerListStats { func loadCpuAndMem(client *client.Client, containerItem string) dto.ContainerListStats {
@@ -1757,7 +1769,10 @@ func checkPortStats(ports []dto.PortHelper, checkInUse bool) (nat.PortMap, error
} }
for i := 0; i <= hostEnd-hostStart; i++ { for i := 0; i <= hostEnd-hostStart; i++ {
bindItem := nat.PortBinding{HostPort: strconv.Itoa(hostStart + i), HostIP: port.HostIP} bindItem := nat.PortBinding{HostPort: strconv.Itoa(hostStart + i), HostIP: port.HostIP}
portMap[nat.Port(fmt.Sprintf("%d/%s", containerStart+i, port.Protocol))] = []nat.PortBinding{bindItem} portKey := nat.Port(fmt.Sprintf("%d/%s", containerStart+i, port.Protocol))
if !slices.Contains(portMap[portKey], bindItem) {
portMap[portKey] = append(portMap[portKey], bindItem)
}
} }
for i := hostStart; i <= hostEnd; i++ { for i := hostStart; i <= hostEnd; i++ {
if checkInUse && common.ScanPortWithIP(port.HostIP, i) { if checkInUse && common.ScanPortWithIP(port.HostIP, i) {
@@ -1775,7 +1790,10 @@ func checkPortStats(ports []dto.PortHelper, checkInUse bool) (nat.PortMap, error
return portMap, buserr.WithDetail("ErrPortInUsed", portItem, nil) return portMap, buserr.WithDetail("ErrPortInUsed", portItem, nil)
} }
bindItem := nat.PortBinding{HostPort: strconv.Itoa(portItem), HostIP: port.HostIP} bindItem := nat.PortBinding{HostPort: strconv.Itoa(portItem), HostIP: port.HostIP}
portMap[nat.Port(fmt.Sprintf("%s/%s", port.ContainerPort, port.Protocol))] = []nat.PortBinding{bindItem} portKey := nat.Port(fmt.Sprintf("%s/%s", port.ContainerPort, port.Protocol))
if !slices.Contains(portMap[portKey], bindItem) {
portMap[portKey] = append(portMap[portKey], bindItem)
}
} }
} }
return portMap, nil return portMap, nil
@@ -1921,90 +1939,7 @@ func loadPortByInspect(id string, client *client.Client) ([]container.Port, erro
return itemPorts, nil return itemPorts, nil
} }
func transPortToStr(ports []container.Port) []string { func transPortToStr(ports []container.Port) []string {
var ( return docker.SimplifyPorts(ports)
ipv4Ports []container.Port
ipv6Ports []container.Port
)
for _, port := range ports {
if strings.Contains(port.IP, ":") {
ipv6Ports = append(ipv6Ports, port)
} else {
ipv4Ports = append(ipv4Ports, port)
}
}
list1 := simplifyPort(ipv4Ports)
list2 := simplifyPort(ipv6Ports)
return append(list1, list2...)
}
func simplifyPort(ports []container.Port) []string {
var datas []string
if len(ports) == 0 {
return datas
}
if len(ports) == 1 {
ip := ""
if len(ports[0].IP) != 0 {
ip = ports[0].IP + ":"
}
itemPortStr := fmt.Sprintf("%s%v/%s", ip, ports[0].PrivatePort, ports[0].Type)
if ports[0].PublicPort != 0 {
itemPortStr = fmt.Sprintf("%s%v->%v/%s", ip, ports[0].PublicPort, ports[0].PrivatePort, ports[0].Type)
}
datas = append(datas, itemPortStr)
return datas
}
sort.Slice(ports, func(i, j int) bool {
return ports[i].PrivatePort < ports[j].PrivatePort
})
start := ports[0]
for i := 1; i < len(ports); i++ {
if ports[i].PrivatePort != ports[i-1].PrivatePort+1 || ports[i].IP != ports[i-1].IP || ports[i].PublicPort != ports[i-1].PublicPort+1 || ports[i].Type != ports[i-1].Type {
if ports[i-1].PrivatePort == start.PrivatePort {
itemPortStr := fmt.Sprintf("%s:%v/%s", start.IP, start.PrivatePort, start.Type)
if start.PublicPort != 0 {
itemPortStr = fmt.Sprintf("%s:%v->%v/%s", start.IP, start.PublicPort, start.PrivatePort, start.Type)
}
if len(start.IP) == 0 {
itemPortStr = strings.TrimPrefix(itemPortStr, ":")
}
datas = append(datas, itemPortStr)
} else {
itemPortStr := fmt.Sprintf("%s:%v-%v/%s", start.IP, start.PrivatePort, ports[i-1].PrivatePort, start.Type)
if start.PublicPort != 0 {
itemPortStr = fmt.Sprintf("%s:%v-%v->%v-%v/%s", start.IP, start.PublicPort, ports[i-1].PublicPort, start.PrivatePort, ports[i-1].PrivatePort, start.Type)
}
if len(start.IP) == 0 {
itemPortStr = strings.TrimPrefix(itemPortStr, ":")
}
datas = append(datas, itemPortStr)
}
start = ports[i]
}
if i == len(ports)-1 {
if ports[i].PrivatePort == start.PrivatePort {
itemPortStr := fmt.Sprintf("%s:%v/%s", start.IP, start.PrivatePort, start.Type)
if start.PublicPort != 0 {
itemPortStr = fmt.Sprintf("%s:%v->%v/%s", start.IP, start.PublicPort, start.PrivatePort, start.Type)
}
if len(start.IP) == 0 {
itemPortStr = strings.TrimPrefix(itemPortStr, ":")
}
datas = append(datas, itemPortStr)
} else {
itemPortStr := fmt.Sprintf("%s:%v-%v/%s", start.IP, start.PrivatePort, ports[i].PrivatePort, start.Type)
if start.PublicPort != 0 {
itemPortStr = fmt.Sprintf("%s:%v-%v->%v-%v/%s", start.IP, start.PublicPort, ports[i].PublicPort, start.PrivatePort, ports[i].PrivatePort, start.Type)
}
if len(start.IP) == 0 {
itemPortStr = strings.TrimPrefix(itemPortStr, ":")
}
datas = append(datas, itemPortStr)
}
}
}
return datas
} }
func loadComposeCount(client *client.Client) int { func loadComposeCount(client *client.Client) int {
@@ -2025,6 +1960,9 @@ func loadComposeCount(client *client.Client) int {
} }
} }
for _, compose := range composeCreatedByLocal { for _, compose := range composeCreatedByLocal {
if len(compose.Path) == 0 {
continue
}
if _, has := composeMap[compose.Name]; !has { if _, has := composeMap[compose.Name]; !has {
composeMap[compose.Name] = struct{}{} composeMap[compose.Name] = struct{}{}
} }
@@ -2034,7 +1972,7 @@ func loadComposeCount(client *client.Client) int {
} }
func loadContainerPortForInfo(itemPorts []container.Port) []dto.PortHelper { func loadContainerPortForInfo(itemPorts []container.Port) []dto.PortHelper {
var exposedPorts []dto.PortHelper var exposedPorts []dto.PortHelper
samePortMap := make(map[string]dto.PortHelper) seenPorts := make(map[dto.PortHelper]struct{})
ports := transPortToStr(itemPorts) ports := transPortToStr(itemPorts)
for _, item := range ports { for _, item := range ports {
itemStr := strings.Split(item, "->") itemStr := strings.Split(item, "->")
@@ -2055,16 +1993,11 @@ func loadContainerPortForInfo(itemPorts []container.Port) []dto.PortHelper {
} }
itemPort.ContainerPort = itemContainer[0] itemPort.ContainerPort = itemContainer[0]
itemPort.Protocol = itemContainer[1] itemPort.Protocol = itemContainer[1]
keyItem := fmt.Sprintf("%s->%s/%s", itemPort.HostPort, itemPort.ContainerPort, itemPort.Protocol) if _, exists := seenPorts[itemPort]; exists {
if val, ok := samePortMap[keyItem]; ok { continue
val.HostIP = ""
samePortMap[keyItem] = val
} else {
samePortMap[keyItem] = itemPort
} }
} seenPorts[itemPort] = struct{}{}
for _, val := range samePortMap { exposedPorts = append(exposedPorts, itemPort)
exposedPorts = append(exposedPorts, val)
} }
return exposedPorts return exposedPorts
} }
+331 -40
View File
@@ -6,6 +6,7 @@ import (
"errors" "errors"
"fmt" "fmt"
"os" "os"
"os/exec"
"path" "path"
"path/filepath" "path/filepath"
"sort" "sort"
@@ -23,8 +24,11 @@ import (
"github.com/1Panel-dev/1Panel/agent/utils/cmd" "github.com/1Panel-dev/1Panel/agent/utils/cmd"
"github.com/1Panel-dev/1Panel/agent/utils/compose" "github.com/1Panel-dev/1Panel/agent/utils/compose"
"github.com/1Panel-dev/1Panel/agent/utils/docker" "github.com/1Panel-dev/1Panel/agent/utils/docker"
"github.com/1Panel-dev/1Panel/agent/utils/re"
"github.com/docker/docker/api/types/container" "github.com/docker/docker/api/types/container"
"github.com/docker/docker/api/types/filters" "github.com/docker/docker/api/types/filters"
"gopkg.in/yaml.v3"
"gorm.io/gorm"
) )
const composeProjectLabel = "com.docker.compose.project" const composeProjectLabel = "com.docker.compose.project"
@@ -52,7 +56,15 @@ func (u *ContainerService) PageCompose(req dto.SearchWithPage) (int64, interface
return 0, nil, err return 0, nil, err
} }
composeCreatedByLocal, _ := composeRepo.ListRecord() composeRecords, _ := composeRepo.ListRecord()
pinnedByName := make(map[string]bool, len(composeRecords))
composeCreatedByLocal := make([]model.Compose, 0, len(composeRecords))
for _, record := range composeRecords {
pinnedByName[record.Name] = record.IsPinned
if len(record.Path) != 0 {
composeCreatedByLocal = append(composeCreatedByLocal, record)
}
}
composeLocalMap := make(map[string]dto.ComposeInfo) composeLocalMap := make(map[string]dto.ComposeInfo)
for _, localItem := range composeCreatedByLocal { for _, localItem := range composeCreatedByLocal {
composeItemLocal := dto.ComposeInfo{ composeItemLocal := dto.ComposeInfo{
@@ -136,6 +148,7 @@ func (u *ContainerService) PageCompose(req dto.SearchWithPage) (int64, interface
for key, value := range mergedMap { for key, value := range mergedMap {
value.Name = key value.Name = key
value.ComposeFileExists = composeFileExists(value.Workdir, value.ConfigFile) value.ComposeFileExists = composeFileExists(value.Workdir, value.ConfigFile)
value.IsPinned = pinnedByName[key]
records = append(records, value) records = append(records, value)
} }
if len(req.Info) != 0 { if len(req.Info) != 0 {
@@ -149,7 +162,21 @@ func (u *ContainerService) PageCompose(req dto.SearchWithPage) (int64, interface
} }
} }
} }
if req.ExcludeAppStore {
length, count := len(records), 0
for count < length {
if records[count].CreatedBy == "Apps" {
records = append(records[:count], records[(count+1):]...)
length--
} else {
count++
}
}
}
sort.Slice(records, func(i, j int) bool { sort.Slice(records, func(i, j int) bool {
if records[i].IsPinned != records[j].IsPinned {
return records[i].IsPinned
}
return records[i].CreatedAt > records[j].CreatedAt return records[i].CreatedAt > records[j].CreatedAt
}) })
total, start, end := len(records), (req.Page-1)*req.PageSize, req.Page*req.PageSize total, start, end := len(records), (req.Page-1)*req.PageSize, req.Page*req.PageSize
@@ -189,54 +216,56 @@ func composeFileExists(workdir, configFile string) bool {
} }
func (u *ContainerService) TestCompose(req dto.ComposeCreate) (bool, error) { func (u *ContainerService) TestCompose(req dto.ComposeCreate) (bool, error) {
if cmd.CheckIllegal(req.Path) { if err := validateComposeCreateName(req); err != nil {
return false, err
}
if hasIllegalComposeCreateInput(req) {
return false, buserr.New("ErrCmdIllegal") return false, buserr.New("ErrCmdIllegal")
} }
composeItem, _ := composeRepo.GetRecord(repo.WithByName(req.Name)) projectName, err := resolveComposeCreateProjectName(req)
if composeItem.ID != 0 {
return false, buserr.New("ErrRecordExist")
}
if err := u.loadPath(&req); err != nil {
return false, err
}
if err := newComposeEnv(req.Path, req.Env); err != nil {
return false, err
}
cmd := getComposeCmd(req.Path, "config")
stdout, err := cmd.CombinedOutput()
if err != nil { if err != nil {
return false, fmt.Errorf("docker-compose config failed, std: %s, err: %v", string(stdout), err) return false, err
}
if err := checkComposeCreateDuplicate(req, projectName); err != nil {
return false, err
} }
return true, nil return true, nil
} }
func (u *ContainerService) CreateCompose(req dto.ComposeCreate) error { func (u *ContainerService) CreateCompose(req dto.ComposeCreate) error {
if cmd.CheckIllegal(req.Name, req.Path) { if err := validateComposeCreateName(req); err != nil {
return err
}
if hasIllegalComposeCreateInput(req) {
return buserr.New("ErrCmdIllegal") return buserr.New("ErrCmdIllegal")
} }
projectName, err := resolveComposeCreateProjectName(req)
if err != nil {
return err
}
if err := checkComposeCreateDuplicate(req, projectName); err != nil {
return err
}
if err := u.loadPath(&req); err != nil { if err := u.loadPath(&req); err != nil {
return err return err
} }
if req.From == "path" { if err := newComposeEnv(req.Path, req.Env); err != nil {
req.Name = path.Base(path.Dir(req.Path)) return err
}
req.Name = projectName
recordName := strings.ToLower(req.Name)
if err := saveComposeRecord(recordName, req.Path); err != nil {
return fmt.Errorf("save compose record failed, err: %v", err)
} }
taskItem, err := task.NewTaskWithOps(req.Name, task.TaskCreate, task.TaskScopeCompose, req.TaskID, 1) taskItem, err := task.NewTaskWithOps(req.Name, task.TaskCreate, task.TaskScopeCompose, req.TaskID, 1)
if err != nil { if err != nil {
return fmt.Errorf("new task for image build failed, err: %v", err) return fmt.Errorf("new task for image build failed, err: %v", err)
} }
if err := newComposeEnv(req.Path, req.Env); err != nil {
return err
}
go func() { go func() {
taskItem.AddSubTask(i18n.GetMsgByKey("ComposeCreate"), func(t *task.Task) error { taskItem.AddSubTask(i18n.GetMsgByKey("ComposeCreate"), func(t *task.Task) error {
err := compose.UpWithTask(req.Path, t, req.ForcePull) err := compose.UpWithTask(req.Path, t, req.ForcePull, req.Name)
t.LogWithStatus(i18n.GetMsgByKey("ComposeCreate"), err) t.LogWithStatus(i18n.GetMsgByKey("ComposeCreate"), err)
if err != nil { return err
_, _ = compose.Down(req.Path)
return err
}
_ = composeRepo.CreateRecord(&model.Compose{Name: strings.ToLower(req.Name), Path: req.Path})
return nil
}, nil) }, nil)
_ = taskItem.Execute() _ = taskItem.Execute()
}() }()
@@ -244,6 +273,254 @@ func (u *ContainerService) CreateCompose(req dto.ComposeCreate) error {
return nil return nil
} }
func saveComposeRecord(name, composePath string) error {
record, err := composeRepo.GetRecord(repo.WithByName(name))
if err != nil && !errors.Is(err, gorm.ErrRecordNotFound) {
return err
}
if record.ID == 0 {
return composeRepo.CreateRecord(&model.Compose{Name: name, Path: composePath})
}
return composeRepo.UpdateRecord(name, map[string]interface{}{"path": composePath})
}
func checkComposeRecordName(name string) error {
composeItem, _ := composeRepo.GetRecord(repo.WithByName(name))
if composeItem.ID != 0 && len(composeItem.Path) != 0 {
return buserr.New("ErrRecordExist")
}
return nil
}
func checkComposeCreateDuplicate(req dto.ComposeCreate, projectName string) error {
if err := checkComposeRecordName(projectName); err != nil {
return err
}
if req.From == "path" {
return nil
}
composeItem, _ := composeRepo.GetRecord(repo.WithByPath(composeCreatePath(req)))
if composeItem.ID != 0 && composeItem.Path != "" {
return buserr.New("ErrRecordExist")
}
return nil
}
func validateComposeCreateName(req dto.ComposeCreate) error {
if req.From == "path" {
name := strings.TrimSpace(req.Name)
if name != "" && !re.GetRegex(re.ComposeNamePattern).MatchString(name) {
return buserr.New("ErrComposeNameInvalid")
}
return nil
}
if !re.GetRegex(re.ComposeNamePattern).MatchString(composeCreateDirName(req)) {
return buserr.New("ErrComposeNameInvalid")
}
return nil
}
func hasIllegalComposeCreateInput(req dto.ComposeCreate) bool {
if req.From == "path" {
return cmd.CheckIllegal(req.Name, req.Path)
}
return cmd.CheckIllegal(composeCreateDirName(req))
}
func composeCreateDirName(req dto.ComposeCreate) string {
dirName := strings.TrimSpace(req.DirName)
if dirName == "" {
// Keep compatibility with callers that used name as both the directory and
// Compose project name before dirName was introduced.
return strings.TrimSpace(req.Name)
}
return dirName
}
func composeCreatePath(req dto.ComposeCreate) string {
return filepath.Join(global.Dir.DataDir, "docker", "compose", composeCreateDirName(req), "docker-compose.yml")
}
func resolveComposeCreateProjectName(req dto.ComposeCreate) (string, error) {
if req.From == "path" {
envPath, err := createComposeTempFile(
filepath.Dir(primaryComposePath(req.Path)),
".1panel-compose-*.env",
req.Env,
)
if err != nil {
return "", err
}
defer os.Remove(envPath)
return resolveComposeProjectName(req.Path, req.Name, envPath)
}
dir := filepath.Dir(composeCreatePath(req))
cleanupDir, err := prepareComposeStagingDir(dir)
if err != nil {
return "", err
}
defer cleanupDir()
composePath, err := createComposeTempFile(dir, ".1panel-compose-*.yml", req.File)
if err != nil {
return "", err
}
defer os.Remove(composePath)
envPath, err := createComposeTempFile(dir, ".1panel-compose-*.env", req.Env)
if err != nil {
return "", err
}
defer os.Remove(envPath)
return resolveComposeProjectName(composePath, "", envPath)
}
func createComposeTempFile(dir, pattern, content string) (string, error) {
file, err := os.CreateTemp(dir, pattern)
if err != nil {
return "", err
}
filePath := file.Name()
if _, err := file.WriteString(content); err != nil {
_ = file.Close()
_ = os.Remove(filePath)
return "", err
}
if err := file.Close(); err != nil {
_ = os.Remove(filePath)
return "", err
}
return filePath, nil
}
func prepareComposeStagingDir(dir string) (func(), error) {
if err := os.MkdirAll(filepath.Dir(dir), os.ModePerm); err != nil {
return nil, err
}
created := false
if err := os.Mkdir(dir, os.ModePerm); err != nil {
if !errors.Is(err, os.ErrExist) {
return nil, err
}
} else {
created = true
}
return func() {
if created {
_ = os.Remove(dir)
}
}, nil
}
func resolveComposeProjectName(composePath, fallbackName, envFile string) (string, error) {
// Preserve the name resolved by Compose (including a top-level name) so the
// container label and the local record always use the same project identity.
parentName := normalizeComposeProjectName(path.Base(path.Dir(primaryComposePath(composePath))))
fallbackName = strings.TrimSpace(fallbackName)
stdout, err := runComposeConfig(composePath, "", envFile)
if err == nil {
projectName, parseErr := loadComposeProjectName(stdout)
if parseErr != nil {
return "", parseErr
}
if projectName != "" {
if !re.GetRegex(re.ComposeNamePattern).MatchString(projectName) {
return "", buserr.New("ErrComposeNameInvalid")
}
return projectName, nil
}
if parentName != "" {
return parentName, nil
}
if fallbackName != "" {
if _, fallbackErr := runComposeConfig(composePath, fallbackName, envFile); fallbackErr != nil {
return "", fallbackErr
}
return fallbackName, nil
}
return "", buserr.New("ErrComposeProjectNameEmpty")
}
if !isComposeProjectNameEmptyError(err) {
return "", err
}
resolveErr := err
if parentName != "" {
if _, parentErr := runComposeConfig(composePath, parentName, envFile); parentErr == nil {
return parentName, nil
} else {
resolveErr = parentErr
}
}
if fallbackName != "" && fallbackName != parentName {
if _, fallbackErr := runComposeConfig(composePath, fallbackName, envFile); fallbackErr == nil {
return fallbackName, nil
} else {
return "", fallbackErr
}
}
if parentName == "" && fallbackName == "" {
return "", buserr.New("ErrComposeProjectNameEmpty")
}
return "", resolveErr
}
func runComposeConfig(composePath, projectName, envFile string) ([]byte, error) {
configCmd := getComposeCmdWithEnv(composePath, "config", envFile, projectName)
stdout, err := configCmd.Output()
if err != nil {
var stderr []byte
if exitErr, ok := err.(*exec.ExitError); ok {
stderr = exitErr.Stderr
}
return nil, fmt.Errorf("docker-compose config failed, std: %s, err: %v", mergeComposeOutput(stdout, stderr), err)
}
return stdout, nil
}
func mergeComposeOutput(stdout, stderr []byte) string {
outputs := make([]string, 0, 2)
if output := strings.TrimSpace(string(stdout)); output != "" {
outputs = append(outputs, output)
}
if output := strings.TrimSpace(string(stderr)); output != "" {
outputs = append(outputs, output)
}
return strings.Join(outputs, "\n")
}
func loadComposeProjectName(config []byte) (string, error) {
var project struct {
Name string `yaml:"name"`
}
if err := yaml.Unmarshal(config, &project); err != nil {
return "", buserr.WithDetail("ErrComposeProjectNameParse", err.Error(), err)
}
return strings.TrimSpace(project.Name), nil
}
func primaryComposePath(composePath string) string {
if index := strings.Index(composePath, ","); index >= 0 {
return composePath[:index]
}
return composePath
}
func normalizeComposeProjectName(name string) string {
name = re.GetRegex(re.ComposeDisallowedCharsPattern).
ReplaceAllString(strings.ToLower(strings.TrimSpace(name)), "")
return strings.TrimLeft(name, "_-")
}
func isComposeProjectNameEmptyError(err error) bool {
message := strings.ToLower(err.Error())
return strings.Contains(message, "project name must not be empty") ||
strings.Contains(message, "project name can't be empty")
}
func (u *ContainerService) ComposeOperation(req dto.ComposeOperation) error { func (u *ContainerService) ComposeOperation(req dto.ComposeOperation) error {
if len(req.Path) == 0 && req.Operation == "delete" { if len(req.Path) == 0 && req.Operation == "delete" {
_ = composeRepo.DeleteRecord(repo.WithByName(req.Name)) _ = composeRepo.DeleteRecord(repo.WithByName(req.Name))
@@ -267,15 +544,15 @@ func (u *ContainerService) ComposeOperation(req dto.ComposeOperation) error {
return nil return nil
} }
if req.Operation == "up" { if req.Operation == "up" {
if stdout, err := compose.Up(req.Path); err != nil { if stdout, err := compose.Up(req.Path, req.Name); err != nil {
return fmt.Errorf("docker-compose up failed, std: %s, err: %v", stdout, err) return fmt.Errorf("docker-compose up failed, std: %s, err: %v", stdout, err)
} }
} else if req.Operation == "rebuild" { } else if req.Operation == "rebuild" {
if stdout, err := compose.DownAndUp(req.Path); err != nil { if stdout, err := compose.DownAndUp(req.Path, req.Name); err != nil {
return fmt.Errorf("docker-compose rebuild failed, std: %s, err: %v", stdout, err) return fmt.Errorf("docker-compose rebuild failed, std: %s, err: %v", stdout, err)
} }
} else { } else {
if stdout, err := compose.Operate(req.Path, req.Operation); err != nil { if stdout, err := compose.Operate(req.Path, req.Operation, req.Name); err != nil {
return fmt.Errorf("docker-compose %s failed, std: %s, err: %v", req.Operation, stdout, err) return fmt.Errorf("docker-compose %s failed, std: %s, err: %v", req.Operation, stdout, err)
} }
} }
@@ -311,9 +588,9 @@ func (u *ContainerService) ComposeUpdate(req dto.ComposeUpdate) error {
return err return err
} }
if err := compose.UpWithTask(req.Path, t, req.ForcePull); err != nil { if err := compose.UpWithTask(req.Path, t, req.ForcePull, req.Name); err != nil {
global.LOG.Errorf("update failed when handle compose up, err: %s, now try to recreate the old compose file", err) global.LOG.Errorf("update failed when handle compose up, err: %s, now try to recreate the old compose file", err)
if err := recreateCompose(string(oldFile), req.Path); err != nil { if err := recreateCompose(string(oldFile), req.Path, req.Name); err != nil {
return fmt.Errorf("update failed and recreate old compose file also failed, err: %v", err) return fmt.Errorf("update failed and recreate old compose file also failed, err: %v", err)
} }
return fmt.Errorf("update failed when handle compose up, err: %s", err) return fmt.Errorf("update failed when handle compose up, err: %s", err)
@@ -327,6 +604,20 @@ func (u *ContainerService) ComposeUpdate(req dto.ComposeUpdate) error {
return nil return nil
} }
func (u *ContainerService) ComposePin(req dto.ComposePin) error {
record, _ := composeRepo.GetRecord(repo.WithByName(req.Name))
if record.ID == 0 {
if !req.IsPinned {
return nil
}
return composeRepo.CreateRecord(&model.Compose{Name: req.Name, IsPinned: true})
}
if !req.IsPinned && len(record.Path) == 0 {
return composeRepo.DeleteRecord(repo.WithByName(req.Name))
}
return composeRepo.UpdateRecord(req.Name, map[string]interface{}{"is_pinned": req.IsPinned})
}
func (u *ContainerService) ComposeLogClean(req dto.ComposeLogClean) error { func (u *ContainerService) ComposeLogClean(req dto.ComposeLogClean) error {
client, err := docker.NewDockerClient() client, err := docker.NewDockerClient()
if err != nil { if err != nil {
@@ -389,15 +680,15 @@ func (u *ContainerService) LoadComposeEnv(name string) (string, error) {
func (u *ContainerService) loadPath(req *dto.ComposeCreate) error { func (u *ContainerService) loadPath(req *dto.ComposeCreate) error {
if req.From == "template" || req.From == "edit" { if req.From == "template" || req.From == "edit" {
dir := fmt.Sprintf("%s/docker/compose/%s", global.Dir.DataDir, req.Name) composePath := composeCreatePath(*req)
dir := filepath.Dir(composePath)
if _, err := os.Stat(dir); err != nil && os.IsNotExist(err) { if _, err := os.Stat(dir); err != nil && os.IsNotExist(err) {
if err = os.MkdirAll(dir, os.ModePerm); err != nil { if err = os.MkdirAll(dir, os.ModePerm); err != nil {
return err return err
} }
} }
path := fmt.Sprintf("%s/docker-compose.yml", dir) file, err := os.OpenFile(composePath, os.O_WRONLY|os.O_CREATE|os.O_TRUNC, constant.FilePerm)
file, err := os.OpenFile(path, os.O_WRONLY|os.O_CREATE|os.O_TRUNC, constant.FilePerm)
if err != nil { if err != nil {
return err return err
} }
@@ -405,14 +696,14 @@ func (u *ContainerService) loadPath(req *dto.ComposeCreate) error {
write := bufio.NewWriter(file) write := bufio.NewWriter(file)
_, _ = write.WriteString(string(req.File)) _, _ = write.WriteString(string(req.File))
write.Flush() write.Flush()
req.Path = path req.Path = composePath
} }
return nil return nil
} }
func removeContainerForCompose(composeName, composePath string) error { func removeContainerForCompose(composeName, composePath string) error {
if _, err := os.Stat(composePath); err == nil { if _, err := os.Stat(composePath); err == nil {
if stdout, err := compose.Operate(composePath, "down"); err != nil { if stdout, err := compose.Operate(composePath, "down", composeName); err != nil {
return errors.New(stdout) return errors.New(stdout)
} }
return nil return nil
@@ -437,7 +728,7 @@ func removeContainerForCompose(composeName, composePath string) error {
return nil return nil
} }
func recreateCompose(content, path string) error { func recreateCompose(content, path, projectName string) error {
file, err := os.OpenFile(path, os.O_WRONLY|os.O_TRUNC, 0640) file, err := os.OpenFile(path, os.O_WRONLY|os.O_TRUNC, 0640)
if err != nil { if err != nil {
return err return err
@@ -447,7 +738,7 @@ func recreateCompose(content, path string) error {
_, _ = write.WriteString(content) _, _ = write.WriteString(content)
write.Flush() write.Flush()
if stdout, err := compose.Up(path); err != nil { if stdout, err := compose.Up(path, projectName); err != nil {
return errors.New(string(stdout)) return errors.New(string(stdout))
} }
return nil return nil
+119
View File
@@ -2,16 +2,24 @@ package service
import ( import (
"context" "context"
"errors"
"fmt" "fmt"
"sort" "sort"
"strings" "strings"
"github.com/1Panel-dev/1Panel/agent/app/dto" "github.com/1Panel-dev/1Panel/agent/app/dto"
"github.com/1Panel-dev/1Panel/agent/app/task"
"github.com/1Panel-dev/1Panel/agent/buserr" "github.com/1Panel-dev/1Panel/agent/buserr"
"github.com/1Panel-dev/1Panel/agent/i18n"
"github.com/1Panel-dev/1Panel/agent/utils/docker" "github.com/1Panel-dev/1Panel/agent/utils/docker"
"github.com/docker/docker/api/types/container"
"github.com/docker/docker/api/types/network" "github.com/docker/docker/api/types/network"
"github.com/docker/docker/client"
"github.com/docker/docker/errdefs"
) )
var networkCleanupSlot = make(chan struct{}, 1)
func (u *ContainerService) PageNetwork(req dto.SearchWithPage) (int64, interface{}, error) { func (u *ContainerService) PageNetwork(req dto.SearchWithPage) (int64, interface{}, error) {
client, err := docker.NewDockerClient() client, err := docker.NewDockerClient()
if err != nil { if err != nil {
@@ -172,3 +180,114 @@ func (u *ContainerService) CreateNetwork(req dto.NetworkCreate) error {
} }
return nil return nil
} }
func cleanUnusedNetworks(t *task.Task, cli *client.Client) error {
ctx := t.TaskCtx
select {
case networkCleanupSlot <- struct{}{}:
defer func() { <-networkCleanupSlot }()
case <-ctx.Done():
return ctx.Err()
}
if err := ctx.Err(); err != nil {
return err
}
networks, err := cli.NetworkList(ctx, network.ListOptions{})
if err != nil {
if ctx.Err() != nil {
return ctx.Err()
}
return err
}
deleted, skipped, failed := 0, 0, 0
defer func() {
t.Log(i18n.GetMsgWithMap("NetworkCleanupSummary", map[string]interface{}{
"deleted": deleted, "skipped": skipped, "failed": failed,
}))
}()
var used map[string]bool
sort.Slice(networks, func(i, j int) bool { return networks[i].Name < networks[j].Name })
for _, n := range networks {
if err := ctx.Err(); err != nil {
return err
}
switch n.Name {
case "none", "host", "bridge", "1panel-network":
skipped++
continue
}
if n.Scope != "local" || n.Ingress || n.ConfigOnly {
skipped++
continue
}
values := map[string]interface{}{"name": n.Name, "id": n.ID}
if used == nil {
containers, err := cli.ContainerList(ctx, container.ListOptions{All: true})
if err != nil {
if ctx.Err() != nil {
return ctx.Err()
}
return err
}
used = make(map[string]bool)
for _, c := range containers {
if c.NetworkSettings == nil {
continue
}
for name, endpoint := range c.NetworkSettings.Networks {
used[name] = true
if endpoint != nil {
used[endpoint.NetworkID] = true
}
}
}
}
if used[n.Name] || used[n.ID] {
skipped++
t.Log(i18n.GetMsgWithMap("NetworkCleanupConnected", values))
continue
}
inspected, err := cli.NetworkInspect(ctx, n.ID, network.InspectOptions{})
if err != nil {
if ctx.Err() != nil {
return ctx.Err()
}
if errdefs.IsNotFound(err) {
skipped++
} else {
failed++
t.Logf("Failed to inspect network [%s] (%s): %v", n.Name, n.ID, err)
}
continue
}
if len(inspected.Containers) > 0 {
skipped++
t.Log(i18n.GetMsgWithMap("NetworkCleanupConnected", values))
continue
}
if err := cli.NetworkRemove(ctx, n.ID); err != nil {
if ctx.Err() != nil {
return ctx.Err()
}
switch {
case errdefs.IsNotFound(err):
skipped++
case errdefs.IsConflict(err):
skipped++
t.Log(i18n.GetMsgWithMap("NetworkCleanupConnected", values))
default:
failed++
t.Logf("Failed to remove network [%s] (%s): %v", n.Name, n.ID, err)
}
continue
}
deleted++
}
if err := ctx.Err(); err != nil {
return err
}
if failed > 0 {
return errors.New(i18n.GetMsgByKey("NetworkCleanupPartialFailure"))
}
return nil
}
+92 -67
View File
@@ -1,9 +1,11 @@
package service package service
import ( import (
"bytes"
"context" "context"
"errors" "errors"
"fmt" "fmt"
"io"
"sort" "sort"
"strings" "strings"
"sync" "sync"
@@ -18,6 +20,7 @@ import (
"github.com/docker/docker/api/types/mount" "github.com/docker/docker/api/types/mount"
"github.com/docker/docker/api/types/network" "github.com/docker/docker/api/types/network"
"github.com/docker/docker/client" "github.com/docker/docker/client"
"github.com/docker/docker/pkg/stdcopy"
v1 "github.com/opencontainers/image-spec/specs-go/v1" v1 "github.com/opencontainers/image-spec/specs-go/v1"
) )
@@ -64,13 +67,13 @@ func (u *ContainerService) ContainerUpdate(req dto.ContainerOperate) error {
if err != nil { if err != nil {
return err return err
} }
normalizeContainerEndpointSettings(ctx, client, networkConf, nil) if err := normalizeContainerEndpointSettings(ctx, client, networkConf, nil); err != nil {
return err
}
cleanupErr, err := switchContainer(ctx, client, req.Name, oldContainer, func() (container.CreateResponse, error) { cleanupErr, err := switchContainer(ctx, client, req.Name, oldContainer, func() (container.CreateResponse, error) {
return createContainerWithDynamicIPFallback(func() (container.CreateResponse, error) { return client.ContainerCreate(ctx, config, hostConf, networkConf, &v1.Platform{}, req.Name)
return client.ContainerCreate(ctx, config, hostConf, networkConf, &v1.Platform{}, req.Name) }, config.Tty, t)
}, networkConf.EndpointsConfig, oldContainer.NetworkSettings)
}, newContainerSwitchTaskLogger(t))
if err != nil { if err != nil {
return fmt.Errorf("update container failed, err: %v", err) return fmt.Errorf("update container failed, err: %v", err)
} }
@@ -135,9 +138,15 @@ func (u *ContainerService) ContainerUpgrade(req dto.ContainerUpgrade) error {
config.Image = req.Image config.Image = req.Image
hostConf := cloneContainerHostConfig(oldContainer.HostConfig) hostConf := cloneContainerHostConfig(oldContainer.HostConfig)
preserveContainerVolumeMounts(hostConf, oldContainer.Mounts) preserveContainerVolumeMounts(hostConf, oldContainer.Mounts)
networkConf, extraNetworks := buildContainerRecoverNetworkConfig(oldContainer.NetworkSettings, hostConf)
if err := normalizeContainerEndpointSettings(ctx, client, networkConf, extraNetworks); err != nil {
upgradeErr := fmt.Errorf("prepare networks for container %s failed: %w", item, err)
upgradeErrors = append(upgradeErrors, upgradeErr)
return upgradeErr
}
cleanupErr, err := switchContainer(ctx, client, item, oldContainer, func() (container.CreateResponse, error) { cleanupErr, err := switchContainer(ctx, client, item, oldContainer, func() (container.CreateResponse, error) {
return createContainerWithOldNetworks(ctx, client, config, hostConf, oldContainer.NetworkSettings, item) return createContainerWithNetworks(ctx, client, config, hostConf, networkConf, extraNetworks, item)
}, newContainerSwitchTaskLogger(t)) }, config.Tty, t)
if err != nil { if err != nil {
upgradeErr := fmt.Errorf("upgrade container %s failed: %w", item, err) upgradeErr := fmt.Errorf("upgrade container %s failed: %w", item, err)
upgradeErrors = append(upgradeErrors, upgradeErr) upgradeErrors = append(upgradeErrors, upgradeErr)
@@ -166,10 +175,15 @@ type containerSwitchClient interface {
ContainerStart(context.Context, string, container.StartOptions) error ContainerStart(context.Context, string, container.StartOptions) error
ContainerRemove(context.Context, string, container.RemoveOptions) error ContainerRemove(context.Context, string, container.RemoveOptions) error
ContainerInspect(context.Context, string) (container.InspectResponse, error) ContainerInspect(context.Context, string) (container.InspectResponse, error)
ContainerLogs(context.Context, string, container.LogsOptions) (io.ReadCloser, error)
NetworkConnect(context.Context, string, string, *network.EndpointSettings) error NetworkConnect(context.Context, string, string, *network.EndpointSettings) error
NetworkDisconnect(context.Context, string, string, bool) error NetworkDisconnect(context.Context, string, string, bool) error
} }
type containerInspectClient interface {
ContainerInspect(context.Context, string) (container.InspectResponse, error)
}
type containerOperationMutex struct { type containerOperationMutex struct {
mutex sync.Mutex mutex sync.Mutex
locks map[string]*containerOperationLockEntry locks map[string]*containerOperationLockEntry
@@ -234,22 +248,18 @@ func (l *containerOperationMutex) lock(names ...string) func() {
} }
type containerNetworkAttachment struct { type containerNetworkAttachment struct {
name string name string
endpoint *network.EndpointSettings endpoint *network.EndpointSettings
isDynamic bool
} }
type containerSwitchLogFunc func(messageKey, containerName string, err error) type containerSwitchLogger interface {
LogWithStatus(string, error)
func newContainerSwitchTaskLogger(t *task.Task) containerSwitchLogFunc { Log(string)
return func(messageKey, containerName string, err error) {
t.LogWithStatus(i18n.GetWithName(messageKey, containerName), err)
}
} }
func logContainerSwitchStep(logger containerSwitchLogFunc, messageKey, containerName string, err error) { func logContainerSwitchStep(logger containerSwitchLogger, messageKey, containerName string, err error) {
if logger != nil { if logger != nil {
logger(messageKey, containerName, err) logger.LogWithStatus(i18n.GetWithName(messageKey, containerName), err)
} }
} }
@@ -260,7 +270,8 @@ func switchContainer(
name string, name string,
oldContainer container.InspectResponse, oldContainer container.InspectResponse,
createNew func() (container.CreateResponse, error), createNew func() (container.CreateResponse, error),
logger containerSwitchLogFunc, tty bool,
logger containerSwitchLogger,
) (cleanupErr error, err error) { ) (cleanupErr error, err error) {
if oldContainer.ID == "" { if oldContainer.ID == "" {
return nil, fmt.Errorf("original container ID is empty") return nil, fmt.Errorf("original container ID is empty")
@@ -310,6 +321,7 @@ func switchContainer(
} }
if err := cli.ContainerStart(ctx, created.ID, container.StartOptions{}); err != nil { if err := cli.ContainerStart(ctx, created.ID, container.StartOptions{}); err != nil {
logContainerSwitchStep(logger, "ContainerStartReplacement", name, err) logContainerSwitchStep(logger, "ContainerStartReplacement", name, err)
logContainerStartupLogs(ctx, cli, created.ID, name, tty, logger)
rollbackErr := restoreOriginalContainer(ctx, cli, oldContainer.ID, name, wasRunning, created.ID, disconnectedNetworks, logger) rollbackErr := restoreOriginalContainer(ctx, cli, oldContainer.ID, name, wasRunning, created.ID, disconnectedNetworks, logger)
return nil, errors.Join(fmt.Errorf("start new container failed: %w", err), rollbackErr) return nil, errors.Join(fmt.Errorf("start new container failed: %w", err), rollbackErr)
} }
@@ -317,6 +329,7 @@ func switchContainer(
if wasRunning { if wasRunning {
if err := waitContainerReady(ctx, cli, created.ID); err != nil { if err := waitContainerReady(ctx, cli, created.ID); err != nil {
logContainerSwitchStep(logger, "ContainerWaitReplacement", name, err) logContainerSwitchStep(logger, "ContainerWaitReplacement", name, err)
logContainerStartupLogs(ctx, cli, created.ID, name, tty, logger)
rollbackErr := restoreOriginalContainer(ctx, cli, oldContainer.ID, name, wasRunning, created.ID, disconnectedNetworks, logger) rollbackErr := restoreOriginalContainer(ctx, cli, oldContainer.ID, name, wasRunning, created.ID, disconnectedNetworks, logger)
return nil, errors.Join(fmt.Errorf("new container readiness check failed: %w", err), rollbackErr) return nil, errors.Join(fmt.Errorf("new container readiness check failed: %w", err), rollbackErr)
} }
@@ -333,9 +346,49 @@ const (
containerStartPollInterval = time.Second containerStartPollInterval = time.Second
containerHealthCheckMinWait = 30 * time.Second containerHealthCheckMinWait = 30 * time.Second
containerHealthCheckMaxWait = 10 * time.Minute containerHealthCheckMaxWait = 10 * time.Minute
containerDiagnosticLogTail = "200"
) )
func waitContainerReady(ctx context.Context, cli containerSwitchClient, containerID string) error { func logContainerStartupLogs(ctx context.Context, cli containerSwitchClient, containerID, name string, tty bool, logger containerSwitchLogger) {
if logger == nil {
return
}
logger.Log(fmt.Sprintf("========== %s ==========", i18n.GetWithName("ContainerStartupDiagnostic", name)))
diagnosticCtx, cancel := context.WithTimeout(ctx, 5*time.Second)
defer cancel()
reader, err := cli.ContainerLogs(diagnosticCtx, containerID, container.LogsOptions{
ShowStdout: true,
ShowStderr: true,
Timestamps: true,
Tail: containerDiagnosticLogTail,
})
if err != nil {
logger.Log(i18n.GetWithNameAndErr("ContainerDiagnosticLogsFailed", name, err))
return
}
defer reader.Close()
var output bytes.Buffer
if tty {
_, err = io.Copy(&output, reader)
} else {
_, err = stdcopy.StdCopy(&output, &output, reader)
}
if err != nil {
logger.Log(i18n.GetWithNameAndErr("ContainerDiagnosticLogsFailed", name, err))
return
}
logs := strings.TrimSpace(output.String())
logger.Log(fmt.Sprintf("---------- %s ----------", i18n.GetMsgByKey("ContainerRecentLogs")))
if logs == "" {
logger.Log(i18n.GetMsgByKey("ContainerDiagnosticLogsEmpty"))
return
}
logger.Log(logs)
}
func waitContainerReady(ctx context.Context, cli containerInspectClient, containerID string) error {
info, err := cli.ContainerInspect(ctx, containerID) info, err := cli.ContainerInspect(ctx, containerID)
if err != nil { if err != nil {
return err return err
@@ -347,14 +400,15 @@ func waitContainerReady(ctx context.Context, cli containerSwitchClient, containe
return waitContainerStable(ctx, cli, containerID, info) return waitContainerStable(ctx, cli, containerID, info)
} }
initialRestartCount := info.RestartCount
timeout := containerHealthCheckTimeout(info.Config) timeout := containerHealthCheckTimeout(info.Config)
deadline := time.NewTimer(timeout) deadline := time.NewTimer(timeout)
ticker := time.NewTicker(time.Second) ticker := time.NewTicker(time.Second)
defer deadline.Stop() defer deadline.Stop()
defer ticker.Stop() defer ticker.Stop()
for { for {
if info.State.Restarting || info.RestartCount != 0 { if info.State.Restarting || info.RestartCount != initialRestartCount {
return fmt.Errorf("container restarted %d times during startup", info.RestartCount) return fmt.Errorf("container restart count changed from %d to %d during startup", initialRestartCount, info.RestartCount)
} }
if info.State.Health == nil { if info.State.Health == nil {
return fmt.Errorf("container health status is unavailable") return fmt.Errorf("container health status is unavailable")
@@ -382,9 +436,10 @@ func waitContainerReady(ctx context.Context, cli containerSwitchClient, containe
} }
} }
func waitContainerStable(ctx context.Context, cli containerSwitchClient, containerID string, initial container.InspectResponse) error { func waitContainerStable(ctx context.Context, cli containerInspectClient, containerID string, initial container.InspectResponse) error {
startedAt := initial.State.StartedAt startedAt := initial.State.StartedAt
if err := checkContainerStableState(initial, startedAt); err != nil { restartCount := initial.RestartCount
if err := checkContainerStableState(initial, startedAt, restartCount); err != nil {
return err return err
} }
deadline := time.NewTimer(containerStartStabilization) deadline := time.NewTimer(containerStartStabilization)
@@ -400,25 +455,25 @@ func waitContainerStable(ctx context.Context, cli containerSwitchClient, contain
if err != nil { if err != nil {
return err return err
} }
return checkContainerStableState(info, startedAt) return checkContainerStableState(info, startedAt, restartCount)
case <-ticker.C: case <-ticker.C:
info, err := cli.ContainerInspect(ctx, containerID) info, err := cli.ContainerInspect(ctx, containerID)
if err != nil { if err != nil {
return err return err
} }
if err := checkContainerStableState(info, startedAt); err != nil { if err := checkContainerStableState(info, startedAt, restartCount); err != nil {
return err return err
} }
} }
} }
} }
func checkContainerStableState(info container.InspectResponse, startedAt string) error { func checkContainerStableState(info container.InspectResponse, startedAt string, restartCount int) error {
if err := checkContainerRunningState(info); err != nil { if err := checkContainerRunningState(info); err != nil {
return err return err
} }
if info.State.Restarting || info.RestartCount != 0 { if info.State.Restarting || info.RestartCount != restartCount {
return fmt.Errorf("container restarted %d times during startup", info.RestartCount) return fmt.Errorf("container restart count changed from %d to %d during startup", restartCount, info.RestartCount)
} }
if startedAt != "" && info.State.StartedAt != startedAt { if startedAt != "" && info.State.StartedAt != startedAt {
return fmt.Errorf("container start time changed during startup") return fmt.Errorf("container start time changed during startup")
@@ -532,13 +587,13 @@ func disconnectOriginalContainerNetworks(ctx context.Context, cli containerSwitc
endpoints := make(map[string]*network.EndpointSettings, len(extras)+1) endpoints := make(map[string]*network.EndpointSettings, len(extras)+1)
if primary != nil { if primary != nil {
for name, endpoint := range primary.EndpointsConfig { for name, endpoint := range primary.EndpointsConfig {
if name != "bridge" && endpoint != nil && endpoint.IPAMConfig != nil { if name != "bridge" && endpoint != nil {
endpoints[name] = endpoint endpoints[name] = endpoint
} }
} }
} }
for name, endpoint := range extras { for name, endpoint := range extras {
if name != "bridge" && endpoint != nil && endpoint.IPAMConfig != nil { if name != "bridge" && endpoint != nil {
endpoints[name] = endpoint endpoints[name] = endpoint
} }
} }
@@ -554,9 +609,8 @@ func disconnectOriginalContainerNetworks(ctx context.Context, cli containerSwitc
return disconnected, fmt.Errorf("disconnect original container from network %s failed: %w", name, err) return disconnected, fmt.Errorf("disconnect original container from network %s failed: %w", name, err)
} }
disconnected = append(disconnected, containerNetworkAttachment{ disconnected = append(disconnected, containerNetworkAttachment{
name: name, name: name,
endpoint: endpoints[name], endpoint: endpoints[name],
isDynamic: isDynamicContainerNetwork(oldContainer.NetworkSettings, name),
}) })
} }
return disconnected, nil return disconnected, nil
@@ -566,10 +620,6 @@ func reconnectOriginalContainerNetworks(ctx context.Context, cli containerSwitch
var reconnectErr error var reconnectErr error
for _, attachment := range attachments { for _, attachment := range attachments {
err := cli.NetworkConnect(ctx, attachment.name, containerID, attachment.endpoint) err := cli.NetworkConnect(ctx, attachment.name, containerID, attachment.endpoint)
if err != nil && attachment.isDynamic && strings.Contains(err.Error(), unsupportedUserSpecifiedIPAddress) {
attachment.endpoint.IPAMConfig = nil
err = cli.NetworkConnect(ctx, attachment.name, containerID, attachment.endpoint)
}
if err != nil { if err != nil {
reconnectErr = errors.Join(reconnectErr, fmt.Errorf("reconnect original container to network %s failed: %w", attachment.name, err)) reconnectErr = errors.Join(reconnectErr, fmt.Errorf("reconnect original container to network %s failed: %w", attachment.name, err))
} }
@@ -577,7 +627,7 @@ func reconnectOriginalContainerNetworks(ctx context.Context, cli containerSwitch
return reconnectErr return reconnectErr
} }
func restoreOriginalContainer(ctx context.Context, cli containerSwitchClient, oldContainerID, originalName string, wasRunning bool, newContainer string, disconnectedNetworks []containerNetworkAttachment, logger containerSwitchLogFunc) error { func restoreOriginalContainer(ctx context.Context, cli containerSwitchClient, oldContainerID, originalName string, wasRunning bool, newContainer string, disconnectedNetworks []containerNetworkAttachment, logger containerSwitchLogger) error {
var rollbackErr error var rollbackErr error
backupName := containerSwitchBackupName(oldContainerID) backupName := containerSwitchBackupName(oldContainerID)
if newContainer != "" { if newContainer != "" {
@@ -602,7 +652,7 @@ func restoreOriginalContainer(ctx context.Context, cli containerSwitchClient, ol
reconnectErr := reconnectOriginalContainerNetworks(ctx, cli, oldContainerID, disconnectedNetworks) reconnectErr := reconnectOriginalContainerNetworks(ctx, cli, oldContainerID, disconnectedNetworks)
logContainerSwitchStep(logger, "ContainerRollbackReconnectOld", currentName, reconnectErr) logContainerSwitchStep(logger, "ContainerRollbackReconnectOld", currentName, reconnectErr)
rollbackErr = errors.Join(rollbackErr, reconnectErr) rollbackErr = errors.Join(rollbackErr, reconnectErr)
if wasRunning { if wasRunning && reconnectErr == nil {
restartErr := restartOriginalContainer(ctx, cli, oldContainerID) restartErr := restartOriginalContainer(ctx, cli, oldContainerID)
logContainerSwitchStep(logger, "ContainerRollbackRestartOld", currentName, restartErr) logContainerSwitchStep(logger, "ContainerRollbackRestartOld", currentName, restartErr)
rollbackErr = errors.Join(rollbackErr, restartErr) rollbackErr = errors.Join(rollbackErr, restartErr)
@@ -610,17 +660,8 @@ func restoreOriginalContainer(ctx context.Context, cli containerSwitchClient, ol
return rollbackErr return rollbackErr
} }
func createContainerWithOldNetworks(ctx context.Context, client *client.Client, config *container.Config, hostConf *container.HostConfig, networkSettings *container.NetworkSettings, name string) (container.CreateResponse, error) { func createContainerWithNetworks(ctx context.Context, client *client.Client, config *container.Config, hostConf *container.HostConfig, networkConf *network.NetworkingConfig, extraNetworks map[string]*network.EndpointSettings, name string) (container.CreateResponse, error) {
networkConf, extraNetworks := buildContainerRecoverNetworkConfig(networkSettings, hostConf) created, err := client.ContainerCreate(ctx, config, hostConf, networkConf, nil, name)
normalizeContainerEndpointSettings(ctx, client, networkConf, extraNetworks)
var primaryEndpoints map[string]*network.EndpointSettings
if networkConf != nil {
primaryEndpoints = networkConf.EndpointsConfig
}
created, err := createContainerWithDynamicIPFallback(func() (container.CreateResponse, error) {
return client.ContainerCreate(ctx, config, hostConf, networkConf, nil, name)
}, primaryEndpoints, networkSettings)
if err != nil { if err != nil {
return created, err return created, err
} }
@@ -632,9 +673,6 @@ func createContainerWithOldNetworks(ctx context.Context, client *client.Client,
sort.Strings(extraNames) sort.Strings(extraNames)
for _, item := range extraNames { for _, item := range extraNames {
err := client.NetworkConnect(ctx, item, created.ID, extraNetworks[item]) err := client.NetworkConnect(ctx, item, created.ID, extraNetworks[item])
if clearUnsupportedDynamicEndpointIPAM(err, map[string]*network.EndpointSettings{item: extraNetworks[item]}, networkSettings) {
err = client.NetworkConnect(ctx, item, created.ID, extraNetworks[item])
}
if err != nil { if err != nil {
_ = client.ContainerRemove(ctx, created.ID, container.RemoveOptions{Force: true}) _ = client.ContainerRemove(ctx, created.ID, container.RemoveOptions{Force: true})
return created, err return created, err
@@ -642,16 +680,3 @@ func createContainerWithOldNetworks(ctx context.Context, client *client.Client,
} }
return created, nil return created, nil
} }
func createContainerWithDynamicIPFallback(
create func() (container.CreateResponse, error),
endpoints map[string]*network.EndpointSettings,
networkSettings *container.NetworkSettings,
) (container.CreateResponse, error) {
for {
created, err := create()
if err == nil || created.ID != "" || !clearUnsupportedDynamicEndpointIPAM(err, endpoints, networkSettings) {
return created, err
}
}
}
+98 -29
View File
@@ -16,6 +16,7 @@ import (
"github.com/1Panel-dev/1Panel/agent/buserr" "github.com/1Panel-dev/1Panel/agent/buserr"
"github.com/1Panel-dev/1Panel/agent/constant" "github.com/1Panel-dev/1Panel/agent/constant"
"github.com/1Panel-dev/1Panel/agent/global" "github.com/1Panel-dev/1Panel/agent/global"
alertUtil "github.com/1Panel-dev/1Panel/agent/utils/alert"
"github.com/1Panel-dev/1Panel/agent/utils/docker" "github.com/1Panel-dev/1Panel/agent/utils/docker"
"github.com/jinzhu/copier" "github.com/jinzhu/copier"
"github.com/pkg/errors" "github.com/pkg/errors"
@@ -75,6 +76,7 @@ func (u *CronjobService) SearchWithPage(search dto.PageCronjob) (int64, interfac
EntryID: cronjob.ID, EntryID: cronjob.ID,
} }
alertInfo, _ := alertRepo.Get(alertRepo.WithByType(alertBase.AlertType), alertRepo.WithByProject(strconv.Itoa(int(alertBase.EntryID))), repo.WithByStatus(constant.AlertEnable)) alertInfo, _ := alertRepo.Get(alertRepo.WithByType(alertBase.AlertType), alertRepo.WithByProject(strconv.Itoa(int(alertBase.EntryID))), repo.WithByStatus(constant.AlertEnable))
item.AlertTriggerMode, _ = alertUtil.CronJobAlertTriggerMode(alertInfo.AdvancedParams)
if alertInfo.SendCount != 0 { if alertInfo.SendCount != 0 {
item.AlertCount = alertInfo.SendCount item.AlertCount = alertInfo.SendCount
} else { } else {
@@ -98,9 +100,11 @@ func (u *CronjobService) LoadInfo(req dto.OperateByID) (*dto.CronjobOperate, err
AlertType: cronjob.Type, AlertType: cronjob.Type,
EntryID: cronjob.ID, EntryID: cronjob.ID,
} }
alertInfo, _ := alertRepo.Get(alertRepo.WithByType(alertBase.AlertType), alertRepo.WithByProject(strconv.Itoa(int(alertBase.EntryID))), repo.WithByStatus(constant.AlertEnable)) alertInfo, _ := alertRepo.Get(alertRepo.WithByType(alertBase.AlertType), alertRepo.WithByProject(strconv.Itoa(int(alertBase.EntryID))))
item.AlertMethod = alertInfo.Method item.AlertMethod = alertInfo.Method
if alertInfo.SendCount != 0 { item.AlertTitle = alertInfo.Title
item.AlertTriggerMode, _ = alertUtil.CronJobAlertTriggerMode(alertInfo.AdvancedParams)
if alertInfo.Status == constant.AlertEnable {
item.AlertCount = alertInfo.SendCount item.AlertCount = alertInfo.SendCount
} else { } else {
item.AlertCount = 0 item.AlertCount = 0
@@ -195,11 +199,12 @@ func (u *CronjobService) Export(req dto.OperateByIDs) (string, error) {
} }
} }
item.SourceAccounts, item.DownloadAccount, _ = loadBackupNamesByID(cronjob.SourceAccountIDs, cronjob.DownloadAccountID) item.SourceAccounts, item.DownloadAccount, _ = loadBackupNamesByID(cronjob.SourceAccountIDs, cronjob.DownloadAccountID)
alertInfo, _ := alertRepo.Get(alertRepo.WithByType(cronjob.Type), alertRepo.WithByProject(strconv.Itoa(int(cronjob.ID))), repo.WithByStatus(constant.AlertEnable)) alertInfo, _ := alertRepo.Get(alertRepo.WithByType(cronjob.Type), alertRepo.WithByProject(strconv.Itoa(int(cronjob.ID))))
if alertInfo.SendCount != 0 { item.AlertTitle = alertInfo.Title
item.AlertMethod = alertInfo.Method
item.AlertTriggerMode, _ = alertUtil.CronJobAlertTriggerMode(alertInfo.AdvancedParams)
if alertInfo.Status == constant.AlertEnable {
item.AlertCount = alertInfo.SendCount item.AlertCount = alertInfo.SendCount
item.AlertTitle = alertInfo.Title
item.AlertMethod = alertInfo.Method
} else { } else {
item.AlertCount = 0 item.AlertCount = 0
} }
@@ -213,6 +218,17 @@ func (u *CronjobService) Export(req dto.OperateByIDs) (string, error) {
} }
func (u *CronjobService) Import(req []dto.CronjobTrans, operator string) error { func (u *CronjobService) Import(req []dto.CronjobTrans, operator string) error {
for _, item := range req {
advanced, err := cronJobAlertAdvancedParams(item.AlertTriggerMode)
if err != nil {
return err
}
if item.AlertCount != 0 {
if err := (AlertService{}).validateCronJobAlertChannels(item.Type, advanced, item.AlertMethod); err != nil {
return err
}
}
}
for _, item := range req { for _, item := range req {
cronjobItem, _ := cronjobRepo.Get(repo.WithByName(item.Name)) cronjobItem, _ := cronjobRepo.Get(repo.WithByName(item.Name))
if cronjobItem.ID != 0 { if cronjobItem.ID != 0 {
@@ -395,17 +411,27 @@ func (u *CronjobService) Import(req []dto.CronjobTrans, operator string) error {
} else { } else {
cronjob.Status = constant.StatusDisable cronjob.Status = constant.StatusDisable
} }
_ = cronjobRepo.Create(&cronjob) if err := cronjobRepo.Create(&cronjob); err != nil {
if item.AlertCount != 0 && item.AlertTitle != "" && item.AlertMethod != "" { return err
}
if item.AlertTitle != "" && item.AlertMethod != "" {
advanced, _ := cronJobAlertAdvancedParams(item.AlertTriggerMode)
status := constant.AlertEnable
if item.AlertCount == 0 {
status = constant.AlertDisable
}
createAlert := dto.AlertCreate{ createAlert := dto.AlertCreate{
Title: item.AlertTitle, Title: item.AlertTitle,
SendCount: item.AlertCount, SendCount: item.AlertCount,
Method: item.AlertMethod, Method: item.AlertMethod,
Type: cronjob.Type, Type: cronjob.Type,
Project: strconv.Itoa(int(cronjob.ID)), Project: strconv.Itoa(int(cronjob.ID)),
Status: constant.AlertEnable, Status: status,
AdvancedParams: advanced,
}
if err := NewIAlertService().CreateAlert(createAlert, operator); err != nil {
return err
} }
_ = NewIAlertService().CreateAlert(createAlert, operator)
} }
} }
return nil return nil
@@ -538,11 +564,14 @@ func (u *CronjobService) CleanRecord(req dto.CronjobClean) error {
return err return err
} }
for _, del := range delRecords { for _, del := range delRecords {
if del.Status == constant.StatusWaiting || del.Status == constant.StatusRunning {
continue
}
if err := cronjobRepo.DeleteRecord(repo.WithByID(del.ID)); err != nil {
return err
}
_ = os.RemoveAll(del.Records) _ = os.RemoveAll(del.Records)
} }
if err := cronjobRepo.DeleteRecord(cronjobRepo.WithByJobID(int(req.CronjobID))); err != nil {
return err
}
return nil return nil
} }
@@ -559,6 +588,15 @@ func (u *CronjobService) HandleOnce(id uint) error {
} }
func (u *CronjobService) Create(req dto.CronjobOperate, operator string) error { func (u *CronjobService) Create(req dto.CronjobOperate, operator string) error {
advanced, err := cronJobAlertAdvancedParams(req.AlertTriggerMode)
if err != nil {
return err
}
if req.AlertCount != 0 {
if err := (AlertService{}).validateCronJobAlertChannels(req.Type, advanced, req.AlertMethod); err != nil {
return err
}
}
cronjob, _ := cronjobRepo.Get(repo.WithByName(req.Name)) cronjob, _ := cronjobRepo.Get(repo.WithByName(req.Name))
if cronjob.ID != 0 { if cronjob.ID != 0 {
return buserr.New("ErrRecordExist") return buserr.New("ErrRecordExist")
@@ -600,12 +638,13 @@ func (u *CronjobService) Create(req dto.CronjobOperate, operator string) error {
} }
if req.AlertCount != 0 && req.AlertTitle != "" && req.AlertMethod != "" { if req.AlertCount != 0 && req.AlertTitle != "" && req.AlertMethod != "" {
createAlert := dto.AlertCreate{ createAlert := dto.AlertCreate{
Title: req.AlertTitle, Title: req.AlertTitle,
SendCount: req.AlertCount, SendCount: req.AlertCount,
Method: req.AlertMethod, Method: req.AlertMethod,
Type: cronjob.Type, Type: cronjob.Type,
Project: strconv.Itoa(int(cronjob.ID)), Project: strconv.Itoa(int(cronjob.ID)),
Status: constant.AlertEnable, Status: constant.AlertEnable,
AdvancedParams: advanced,
} }
err := NewIAlertService().CreateAlert(createAlert, operator) err := NewIAlertService().CreateAlert(createAlert, operator)
if err != nil { if err != nil {
@@ -679,6 +718,10 @@ func (u *CronjobService) Delete(req dto.CronjobBatchDelete) error {
} }
func (u *CronjobService) Update(id uint, req dto.CronjobOperate, operator string) error { func (u *CronjobService) Update(id uint, req dto.CronjobOperate, operator string) error {
advanced, err := cronJobAlertAdvancedParams(req.AlertTriggerMode)
if err != nil {
return err
}
var cronjob model.Cronjob var cronjob model.Cronjob
if err := copier.Copy(&cronjob, &req); err != nil { if err := copier.Copy(&cronjob, &req); err != nil {
return buserr.WithDetail("ErrStructTransform", err.Error(), nil) return buserr.WithDetail("ErrStructTransform", err.Error(), nil)
@@ -694,6 +737,20 @@ func (u *CronjobService) Update(id uint, req dto.CronjobOperate, operator string
if err != nil { if err != nil {
return buserr.New("ErrRecordNotFound") return buserr.New("ErrRecordNotFound")
} }
if req.AlertCount != 0 {
previous, _ := alertRepo.Get(alertRepo.WithByType(cronModel.Type), alertRepo.WithByProject(strconv.Itoa(int(id))))
merged, err := prepareCronJobAlertParams(cronModel.Type, previous.AdvancedParams, advanced)
if err != nil {
return err
}
method := req.AlertMethod
if method == "" {
method = previous.Method
}
if err := (AlertService{}).validateCronJobAlertChannels(cronModel.Type, merged, method); err != nil {
return err
}
}
upMap := make(map[string]interface{}) upMap := make(map[string]interface{})
cronjob.EntryIDs = cronModel.EntryIDs cronjob.EntryIDs = cronModel.EntryIDs
cronjob.Type = cronModel.Type cronjob.Type = cronModel.Type
@@ -750,11 +807,12 @@ func (u *CronjobService) Update(id uint, req dto.CronjobOperate, operator string
return err return err
} }
updateAlert := dto.AlertCreate{ updateAlert := dto.AlertCreate{
Title: req.AlertTitle, Title: req.AlertTitle,
SendCount: req.AlertCount, SendCount: req.AlertCount,
Method: req.AlertMethod, Method: req.AlertMethod,
Type: cronjob.Type, Type: cronjob.Type,
Project: strconv.Itoa(int(cronModel.ID)), Project: strconv.Itoa(int(cronModel.ID)),
AdvancedParams: advanced,
} }
err = NewIAlertService().ExternalUpdateAlert(updateAlert, operator) err = NewIAlertService().ExternalUpdateAlert(updateAlert, operator)
if err != nil { if err != nil {
@@ -763,6 +821,17 @@ func (u *CronjobService) Update(id uint, req dto.CronjobOperate, operator string
return nil return nil
} }
func cronJobAlertAdvancedParams(mode string) (string, error) {
if mode == "" {
return "", nil
}
data, err := json.Marshal(map[string]string{"alertTriggerMode": mode})
if err != nil {
return "", err
}
return alertUtil.MergeCronJobAlertParams("", string(data))
}
func (u *CronjobService) UpdateStatus(id uint, status string) error { func (u *CronjobService) UpdateStatus(id uint, status string) error {
cronjob, _ := cronjobRepo.Get(repo.WithByID(id)) cronjob, _ := cronjobRepo.Get(repo.WithByID(id))
if cronjob.ID == 0 { if cronjob.ID == 0 {
+2
View File
@@ -163,6 +163,7 @@ func (u *CronjobService) handleDatabase(cronjob model.Cronjob, startTime time.Ti
record.Name = dbInfo.Database record.Name = dbInfo.Database
record.DetailName = dbInfo.Name record.DetailName = dbInfo.Name
record.DownloadAccountID, record.SourceAccountIDs = cronjob.DownloadAccountID, cronjob.SourceAccountIDs record.DownloadAccountID, record.SourceAccountIDs = cronjob.DownloadAccountID, cronjob.SourceAccountIDs
record.Args = encodeBackupArgs(dbInfo.Args)
backupDir := path.Join(global.Dir.LocalBackupDir, fmt.Sprintf("tmp/database/%s/%s/%s", dbInfo.DBType, record.Name, dbInfo.Name)) backupDir := path.Join(global.Dir.LocalBackupDir, fmt.Sprintf("tmp/database/%s/%s/%s", dbInfo.DBType, record.Name, dbInfo.Name))
switch dbInfo.DBType { switch dbInfo.DBType {
@@ -411,6 +412,7 @@ func addSkipTask(source string, taskItem *task.Task) {
taskItem.Log(i18n.GetMsgByKey("NoSuchResource")) taskItem.Log(i18n.GetMsgByKey("NoSuchResource"))
return nil return nil
}, nil) }, nil)
taskItem.SubTasks[len(taskItem.SubTasks)-1].StepAlias = cronJobSkippedStep
} }
func loadDbsForJob(cronjob model.Cronjob) []DatabaseHelper { func loadDbsForJob(cronjob model.Cronjob) []DatabaseHelper {
+35 -6
View File
@@ -4,6 +4,7 @@ import (
"bufio" "bufio"
"context" "context"
"encoding/json" "encoding/json"
"errors"
"fmt" "fmt"
"io" "io"
"net/http" "net/http"
@@ -23,6 +24,7 @@ import (
"github.com/1Panel-dev/1Panel/agent/constant" "github.com/1Panel-dev/1Panel/agent/constant"
"github.com/1Panel-dev/1Panel/agent/global" "github.com/1Panel-dev/1Panel/agent/global"
"github.com/1Panel-dev/1Panel/agent/i18n" "github.com/1Panel-dev/1Panel/agent/i18n"
alertUtil "github.com/1Panel-dev/1Panel/agent/utils/alert"
"github.com/1Panel-dev/1Panel/agent/utils/cmd" "github.com/1Panel-dev/1Panel/agent/utils/cmd"
"github.com/1Panel-dev/1Panel/agent/utils/files" "github.com/1Panel-dev/1Panel/agent/utils/files"
"github.com/1Panel-dev/1Panel/agent/utils/ntp" "github.com/1Panel-dev/1Panel/agent/utils/ntp"
@@ -56,10 +58,11 @@ func (u *CronjobService) HandleJob(cronjob *model.Cronjob) {
_ = taskRepo.Save(context.Background(), taskItem.Task) _ = taskRepo.Save(context.Background(), taskItem.Task)
} }
cronjobRepo.EndRecords(record, constant.StatusFailed, err.Error(), record.Records) cronjobRepo.EndRecords(record, constant.StatusFailed, err.Error(), record.Records)
handleCronJobAlert(cronjob) handleCronJobAlert(cronjob, cronJobAlertResult(taskItem, err))
return return
} }
cronjobRepo.EndRecords(record, constant.StatusSuccess, "", record.Records) cronjobRepo.EndRecords(record, constant.StatusSuccess, "", record.Records)
handleCronJobAlert(cronjob, cronJobAlertResult(taskItem, nil))
}() }()
return return
} }
@@ -70,19 +73,20 @@ func (u *CronjobService) HandleJob(cronjob *model.Cronjob) {
record.TaskID = "" record.TaskID = ""
} }
cronjobRepo.EndRecords(record, constant.StatusFailed, err.Error(), record.Records) cronjobRepo.EndRecords(record, constant.StatusFailed, err.Error(), record.Records)
handleCronJobAlert(cronjob) handleCronJobAlert(cronjob, cronJobAlertResult(taskItem, err))
return return
} }
go func() { go func() {
if err := taskItem.Execute(); err != nil { if err := taskItem.Execute(); err != nil {
taskItem, _ := taskRepo.GetFirst(taskRepo.WithByID(record.TaskID)) storedTask, _ := taskRepo.GetFirst(taskRepo.WithByID(record.TaskID))
if len(taskItem.ID) == 0 { if len(storedTask.ID) == 0 {
record.TaskID = "" record.TaskID = ""
} }
cronjobRepo.EndRecords(record, constant.StatusFailed, err.Error(), record.Records) cronjobRepo.EndRecords(record, constant.StatusFailed, err.Error(), record.Records)
handleCronJobAlert(cronjob) handleCronJobAlert(cronjob, cronJobAlertResult(taskItem, err))
} else { } else {
cronjobRepo.EndRecords(record, constant.StatusSuccess, "", record.Records) cronjobRepo.EndRecords(record, constant.StatusSuccess, "", record.Records)
handleCronJobAlert(cronjob, cronJobAlertResult(taskItem, nil))
} }
}() }()
} }
@@ -482,8 +486,33 @@ func hasBackup(cronjobType string) bool {
return cronjobType == "app" || cronjobType == "database" || cronjobType == "website" || cronjobType == "directory" || cronjobType == "snapshot" || cronjobType == "log" || cronjobType == "cutWebsiteLog" return cronjobType == "app" || cronjobType == "database" || cronjobType == "website" || cronjobType == "directory" || cronjobType == "snapshot" || cronjobType == "log" || cronjobType == "cutWebsiteLog"
} }
func handleCronJobAlert(cronjob *model.Cronjob) { const cronJobSkippedStep = "cronjob-skipped"
func cronJobAlertResult(taskItem *task.Task, err error) string {
if errors.Is(err, context.Canceled) || taskItem.Task.Status == constant.StatusCanceled ||
(taskItem.TaskCtx != nil && taskItem.TaskCtx.Err() != nil) {
return ""
}
if err != nil {
return alertUtil.CronJobAlertFailed
}
if taskItem.Task.Status != constant.StatusSuccess {
return ""
}
for _, subTask := range taskItem.SubTasks {
if subTask.StepAlias != cronJobSkippedStep {
return alertUtil.CronJobAlertSuccess
}
}
return ""
}
func handleCronJobAlert(cronjob *model.Cronjob, result string) {
if result == "" {
return
}
pushAlert := dto.PushAlert{ pushAlert := dto.PushAlert{
Result: result,
TaskName: cronjob.Name, TaskName: cronjob.Name,
AlertType: cronjob.Type, AlertType: cronjob.Type,
EntryID: cronjob.ID, EntryID: cronjob.ID,
+168 -110
View File
@@ -18,8 +18,7 @@ import (
"github.com/1Panel-dev/1Panel/agent/buserr" "github.com/1Panel-dev/1Panel/agent/buserr"
"github.com/1Panel-dev/1Panel/agent/constant" "github.com/1Panel-dev/1Panel/agent/constant"
"github.com/1Panel-dev/1Panel/agent/global" "github.com/1Panel-dev/1Panel/agent/global"
"github.com/1Panel-dev/1Panel/agent/utils/ai_tools/gpu" "github.com/1Panel-dev/1Panel/agent/utils/ai_tools/accelerator"
"github.com/1Panel-dev/1Panel/agent/utils/ai_tools/xpu"
"github.com/1Panel-dev/1Panel/agent/utils/cmd" "github.com/1Panel-dev/1Panel/agent/utils/cmd"
"github.com/1Panel-dev/1Panel/agent/utils/common" "github.com/1Panel-dev/1Panel/agent/utils/common"
"github.com/1Panel-dev/1Panel/agent/utils/controller" "github.com/1Panel-dev/1Panel/agent/utils/controller"
@@ -184,10 +183,30 @@ func (u *DashboardService) LoadBaseInfo(ioOption string, netOption string) (*dto
func (u *DashboardService) LoadCurrentInfo(ioOption string, netOption string) *dto.DashboardCurrent { func (u *DashboardService) LoadCurrentInfo(ioOption string, netOption string) *dto.DashboardCurrent {
var currentInfo dto.DashboardCurrent var currentInfo dto.DashboardCurrent
hostInfo, _ := psutil.HOST.GetHostInfo(false) shotTime := time.Now()
currentInfo.Uptime = hostInfo.Uptime hostInfo, err := psutil.HOST.GetHostInfo(false)
currentInfo.TimeSinceUptime = time.Unix(int64(hostInfo.BootTime), 0).Format(constant.DateTimeLayout) if err != nil {
currentInfo.RunningTime = loadRunningTime(hostInfo.Uptime) global.LOG.Errorf("load host info failed: %v", err)
currentInfo.ShotTime = shotTime
return &currentInfo
}
uptime := hostInfo.Uptime
var bootTime uint64
if now := shotTime.Unix(); now > 0 {
nowUnix := uint64(now)
if hostInfo.BootTime > 0 && hostInfo.BootTime <= nowUnix {
bootTime = hostInfo.BootTime
uptime = nowUnix - bootTime
} else if uptime <= nowUnix {
bootTime = nowUnix - uptime
}
}
currentInfo.Uptime = uptime
currentInfo.RunningTime = loadRunningTime(uptime)
if bootTime > 0 {
currentInfo.TimeSinceUptime = time.Unix(int64(bootTime), 0).Format(constant.DateTimeLayout)
}
currentInfo.Procs = hostInfo.Procs currentInfo.Procs = hostInfo.Procs
currentInfo.CPUTotal, _ = psutil.CPUInfo.GetLogicalCores(false) currentInfo.CPUTotal, _ = psutil.CPUInfo.GetLogicalCores(false)
@@ -223,9 +242,8 @@ func (u *DashboardService) LoadCurrentInfo(ioOption string, netOption string) *d
currentInfo.SwapMemoryUsed = swapInfo.Used currentInfo.SwapMemoryUsed = swapInfo.Used
currentInfo.SwapMemoryUsedPercent = swapInfo.UsedPercent currentInfo.SwapMemoryUsedPercent = swapInfo.UsedPercent
currentInfo.DiskData = loadDiskInfo() currentInfo.DiskData = loadDiskInfo(false)
currentInfo.GPUData = loadGPUInfo() currentInfo.GPUData, currentInfo.NPUData, currentInfo.XPUData = loadAcceleratorInfo()
currentInfo.XPUData = loadXpuInfo()
if ioOption == "all" { if ioOption == "all" {
diskInfo, _ := disk.IOCounters() diskInfo, _ := disk.IOCounters()
@@ -263,7 +281,7 @@ func (u *DashboardService) LoadCurrentInfo(ioOption string, netOption string) *d
} }
} }
currentInfo.ShotTime = time.Now() currentInfo.ShotTime = shotTime
return &currentInfo return &currentInfo
} }
@@ -437,24 +455,9 @@ type diskInfo struct {
Device string Device string
} }
func loadDiskInfo() []dto.DiskInfo { func loadDiskInfo(forceRefresh bool) []dto.DiskInfo {
var datas []dto.DiskInfo var datas []dto.DiskInfo
cmdMgr := cmd.NewCommandMgr(cmd.WithTimeout(2 * time.Second)) stdout := loadDiskMounts()
format := `NR>1 && !/tmpfs|snap\/core|udev/ {printf "%s\t%s\t%s\t%s\t%s\t%s\t%s\n", $1, $2, $3, $4, $5, $6, $7}`
stdout, err := cmdMgr.RunPipe(
cmd.PipeCommand{Name: "df", Args: []string{"-hT", "-P"}},
cmd.PipeCommand{Name: "awk", Args: []string{format}},
)
if err != nil {
cmdMgr2 := cmd.NewCommandMgr(cmd.WithTimeout(1 * time.Second))
stdout, err = cmdMgr2.RunPipe(
cmd.PipeCommand{Name: "df", Args: []string{"-lhT", "-P"}},
cmd.PipeCommand{Name: "awk", Args: []string{format}},
)
if err != nil {
return datas
}
}
lines := strings.Split(stdout, "\n") lines := strings.Split(stdout, "\n")
var mounts []diskInfo var mounts []diskInfo
@@ -502,43 +505,22 @@ func loadDiskInfo() []dto.DiskInfo {
itemData.Type = mount.Type itemData.Type = mount.Type
itemData.Device = mount.Device itemData.Device = mount.Device
type diskResult struct { state, err := loadDiskUsageWithTimeout(mount.Mount, forceRefresh)
state *disk.UsageStat if err != nil {
err error global.LOG.Errorf("load disk info from %s failed, err: %v", mount.Mount, err)
} } else {
resultCh := make(chan diskResult, 1) itemData.Total = state.Total
itemData.Free = state.Free
go func() { itemData.Used = state.Used
state, err := psutil.DISK.GetUsage(mount.Mount, false) itemData.UsedPercent = state.UsedPercent
resultCh <- diskResult{state: state, err: err} itemData.InodesTotal = state.InodesTotal
}() itemData.InodesUsed = state.InodesUsed
itemData.InodesFree = state.InodesFree
select { itemData.InodesUsedPercent = state.InodesUsedPercent
case <-time.After(5 * time.Second):
mu.Lock()
datas = append(datas, itemData)
mu.Unlock()
global.LOG.Errorf("load disk info from %s failed, err: timeout", mount.Mount)
case result := <-resultCh:
if result.err != nil {
mu.Lock()
datas = append(datas, itemData)
mu.Unlock()
global.LOG.Errorf("load disk info from %s failed, err: %v", mount.Mount, result.err)
return
}
itemData.Total = result.state.Total
itemData.Free = result.state.Free
itemData.Used = result.state.Used
itemData.UsedPercent = result.state.UsedPercent
itemData.InodesTotal = result.state.InodesTotal
itemData.InodesUsed = result.state.InodesUsed
itemData.InodesFree = result.state.InodesFree
itemData.InodesUsedPercent = result.state.InodesUsedPercent
mu.Lock()
datas = append(datas, itemData)
mu.Unlock()
} }
mu.Lock()
datas = append(datas, itemData)
mu.Unlock()
}(mounts[i]) }(mounts[i])
} }
wg.Wait() wg.Wait()
@@ -549,32 +531,134 @@ func loadDiskInfo() []dto.DiskInfo {
return datas return datas
} }
func loadGPUInfo() []dto.GPUInfo { var diskMountsMu sync.Mutex
ok, client := gpu.New()
var list []interface{} func loadDiskMounts() string {
if ok { if !diskMountsMu.TryLock() {
info, err := client.LoadGpuInfo() return ""
if err != nil || len(info.GPUs) == 0 {
return nil
}
for _, item := range info.GPUs {
list = append(list, item)
}
} }
if len(list) == 0 { resultCh := make(chan string, 1)
return nil go func() {
var stdout string
defer func() {
diskMountsMu.Unlock()
resultCh <- stdout
}()
cmdMgr := cmd.NewCommandMgr(cmd.WithTimeout(2 * time.Second))
format := `NR>1 && !/tmpfs|snap\/core|udev/ {printf "%s\t%s\t%s\t%s\t%s\t%s\t%s\n", $1, $2, $3, $4, $5, $6, $7}`
output, err := cmdMgr.RunPipe(
cmd.PipeCommand{Name: "df", Args: []string{"-hT", "-P"}},
cmd.PipeCommand{Name: "awk", Args: []string{format}},
)
if err != nil {
global.LOG.Errorf("load disk info with df -hT -P failed, err: %v", err)
cmdMgr2 := cmd.NewCommandMgr(cmd.WithTimeout(1 * time.Second))
output, err = cmdMgr2.RunPipe(
cmd.PipeCommand{Name: "df", Args: []string{"-lhT", "-P"}},
cmd.PipeCommand{Name: "awk", Args: []string{format}},
)
if err != nil {
global.LOG.Errorf("load disk info with df -lhT -P failed, err: %v", err)
return
}
}
stdout = output
}()
timer := time.NewTimer(3 * time.Second)
defer timer.Stop()
select {
case stdout := <-resultCh:
return stdout
case <-timer.C:
global.LOG.Error("load disk mounts timed out; df collection is still running")
return ""
} }
var data []dto.GPUInfo }
for _, gpu := range list {
var dataItem dto.GPUInfo func loadDiskUsageWithTimeout(path string, forceRefresh bool) (*disk.UsageStat, error) {
if err := copier.Copy(&dataItem, &gpu); err != nil { type diskResult struct {
state *disk.UsageStat
err error
}
resultCh := make(chan diskResult, 1)
go func() {
state, err := psutil.DISK.GetUsage(path, forceRefresh)
resultCh <- diskResult{state: state, err: err}
}()
select {
case <-time.After(5 * time.Second):
return nil, fmt.Errorf("load disk usage from %s: timeout", path)
case result := <-resultCh:
return result.state, result.err
}
}
func loadAcceleratorInfo() ([]dto.GPUInfo, []dto.NPUInfo, []dto.XPUInfo) {
ok, client := accelerator.New()
if !ok {
return nil, nil, nil
}
snapshot, err := client.Collect(context.Background())
if err != nil || len(snapshot.Devices) == 0 {
return nil, nil, nil
}
if warning := snapshot.Warning(); warning != nil {
global.LOG.Warnf("load accelerator dashboard data partially failed, err: %v", warning)
}
var (
gpuData []dto.GPUInfo
npuData []dto.NPUInfo
xpuData []dto.XPUInfo
)
for _, device := range snapshot.Devices {
if device.ParentID != "" {
continue continue
} }
dataItem.PowerUsage = dataItem.PowerDraw + " / " + dataItem.MaxPowerLimit switch device.Kind {
dataItem.MemoryUsage = dataItem.MemUsed + " / " + dataItem.MemTotal case accelerator.KindGPU:
data = append(data, dataItem) if device.GPU == nil {
continue
}
var dataItem dto.GPUInfo
if err := copier.Copy(&dataItem, device.GPU); err != nil {
continue
}
dataItem.MaxPowerLimit = device.GPU.PowerLimit
dataItem.PowerUsage = dataItem.PowerDraw
if dataItem.MaxPowerLimit != "" {
dataItem.PowerUsage += " / " + dataItem.MaxPowerLimit
}
dataItem.MemoryUsage = dataItem.MemUsed + " / " + dataItem.MemTotal
gpuData = append(gpuData, dataItem)
case accelerator.KindNPU:
if device.NPU == nil {
continue
}
var dataItem dto.NPUInfo
if err := copier.Copy(&dataItem, device.NPU); err != nil {
continue
}
npuData = append(npuData, dataItem)
case accelerator.KindXPU:
if device.XPU == nil {
continue
}
xpuData = append(xpuData, dto.XPUInfo{
DeviceID: device.Index,
DeviceName: device.Name,
PciBdfAddress: device.BusID,
Memory: device.XPU.Basic.Memory,
Temperature: device.Metrics.Temperature.Display,
GPUUtil: device.Metrics.Utilization.Display,
MemoryUsed: device.Metrics.MemoryUsed.Display,
Power: device.Metrics.Power.Display,
MemoryUtil: device.Metrics.MemoryUtil.Display,
})
}
} }
return data return gpuData, npuData, xpuData
} }
type AppLauncher struct { type AppLauncher struct {
@@ -590,32 +674,6 @@ func ArryContains(arr []string, element string) bool {
return false return false
} }
func loadXpuInfo() []dto.XPUInfo {
var list []interface{}
ok, xpuClient := xpu.New()
if ok {
xpus, err := xpuClient.LoadDashData()
if err != nil || len(xpus) == 0 {
return nil
}
for _, item := range xpus {
list = append(list, item)
}
}
if len(list) == 0 {
return nil
}
var data []dto.XPUInfo
for _, gpu := range list {
var dataItem dto.XPUInfo
if err := copier.Copy(&dataItem, &gpu); err != nil {
continue
}
data = append(data, dataItem)
}
return data
}
func loadOutboundIP() string { func loadOutboundIP() string {
conn, err := network.Dial("udp", "8.8.8.8:80") conn, err := network.Dial("udp", "8.8.8.8:80")
+63 -19
View File
@@ -8,6 +8,12 @@ import (
"os" "os"
"os/exec" "os/exec"
"strings" "strings"
"sync"
"github.com/1Panel-dev/1Panel/agent/app/task"
"github.com/1Panel-dev/1Panel/agent/buserr"
"github.com/google/uuid"
"gorm.io/gorm"
"github.com/1Panel-dev/1Panel/agent/app/repo" "github.com/1Panel-dev/1Panel/agent/app/repo"
"github.com/1Panel-dev/1Panel/agent/global" "github.com/1Panel-dev/1Panel/agent/global"
@@ -17,12 +23,15 @@ import (
"github.com/1Panel-dev/1Panel/agent/utils/compose" "github.com/1Panel-dev/1Panel/agent/utils/compose"
"github.com/1Panel-dev/1Panel/agent/utils/docker" "github.com/1Panel-dev/1Panel/agent/utils/docker"
"github.com/1Panel-dev/1Panel/agent/utils/encrypt" "github.com/1Panel-dev/1Panel/agent/utils/encrypt"
"github.com/docker/docker/api/types/container" dockerclient "github.com/docker/docker/client"
_ "github.com/go-sql-driver/mysql" _ "github.com/go-sql-driver/mysql"
) )
type RedisService struct{} type RedisService struct{}
// The CLI container is shared by all remote Redis databases on this node.
var redisCliInstallMutex sync.Mutex
type IRedisService interface { type IRedisService interface {
UpdateConf(req dto.RedisConfUpdate) error UpdateConf(req dto.RedisConfUpdate) error
UpdatePersistenceConf(req dto.RedisConfPersistenceUpdate) error UpdatePersistenceConf(req dto.RedisConfPersistenceUpdate) error
@@ -33,7 +42,7 @@ type IRedisService interface {
LoadPersistenceConf(req dto.LoadRedisStatus) (*dto.RedisPersistence, error) LoadPersistenceConf(req dto.LoadRedisStatus) (*dto.RedisPersistence, error)
CheckHasCli() bool CheckHasCli() bool
InstallCli() error InstallCli(req dto.RedisCliInstall) (string, error)
} }
func NewIRedisService() IRedisService { func NewIRedisService() IRedisService {
@@ -61,30 +70,65 @@ func (u *RedisService) UpdateConf(req dto.RedisConfUpdate) error {
} }
func (u *RedisService) CheckHasCli() bool { func (u *RedisService) CheckHasCli() bool {
client, err := docker.NewDockerClient() installed, _ := u.checkCliInstalled()
if err != nil { return installed
return false
}
defer client.Close()
containerLists, err := client.ContainerList(context.Background(), container.ListOptions{})
if err != nil {
return false
}
for _, item := range containerLists {
if strings.ReplaceAll(item.Names[0], "/", "") == "1Panel-redis-cli-tools" {
return true
}
}
return false
} }
func (u *RedisService) InstallCli() error { func (u *RedisService) checkCliInstalled() (bool, error) {
client, err := docker.NewDockerClient()
if err != nil {
return false, err
}
defer client.Close()
info, err := client.ContainerInspect(context.Background(), "1Panel-redis-cli-tools")
if dockerclient.IsErrNotFound(err) {
return false, nil
}
if err != nil {
return false, err
}
return info.ContainerJSONBase != nil && info.State != nil && info.State.Running, nil
}
func (u *RedisService) InstallCli(req dto.RedisCliInstall) (string, error) {
if !redisCliInstallMutex.TryLock() {
return "", buserr.New("TaskIsExecuting")
}
defer redisCliInstallMutex.Unlock()
item := dto.ContainerOperate{ item := dto.ContainerOperate{
TaskID: req.TaskID,
Name: "1Panel-redis-cli-tools", Name: "1Panel-redis-cli-tools",
Image: "redis:7.4.4", Image: "redis:7.4.4",
Networks: []dto.ContainerNetwork{{Network: "1panel-network"}}, Networks: []dto.ContainerNetwork{{Network: "1panel-network"}},
} }
return NewIContainerService().ContainerCreate(item, false) running, err := taskRepo.GetFirst(
repo.WithByName(task.GetTaskName(item.Name, task.TaskCreate, task.TaskScopeContainer)),
repo.WithByType(task.TaskScopeContainer),
taskRepo.WithByStatus(constant.StatusExecuting),
)
if err != nil && !errors.Is(err, gorm.ErrRecordNotFound) {
return "", err
}
if running.ID != "" {
return running.ID, nil
}
installed, err := u.checkCliInstalled()
if err != nil || installed {
return "", err
}
if item.TaskID == "" {
item.TaskID = uuid.NewString()
}
if _, err := taskRepo.GetFirst(taskRepo.WithByID(item.TaskID)); !errors.Is(err, gorm.ErrRecordNotFound) {
if err != nil {
return "", err
}
return "", buserr.New("TaskIsExecuting")
}
if err := NewIContainerService().ContainerCreate(item, true); err != nil {
return "", err
}
return item.TaskID, nil
} }
func (u *RedisService) ChangePassword(req dto.ChangeRedisPass) error { func (u *RedisService) ChangePassword(req dto.ChangeRedisPass) error {
+1 -1
View File
@@ -73,7 +73,7 @@ func (u *DeviceService) LoadBaseInfo() (dto.DeviceBaseInfo, error) {
if baseInfo.SwapMemoryTotal != 0 { if baseInfo.SwapMemoryTotal != 0 {
baseInfo.SwapDetails = loadSwap() baseInfo.SwapDetails = loadSwap()
} }
disks := loadDiskInfo() disks := loadDiskInfo(false)
for _, item := range disks { for _, item := range disks {
baseInfo.MaxSize += item.Free baseInfo.MaxSize += item.Free
} }
+69 -20
View File
@@ -28,10 +28,11 @@ import (
) )
const ( const (
rollbackPath = "1panel/tmp" rollbackPath = "1panel/tmp"
upgradePath = "1panel/tmp/upgrade" communityRestorePath = "1panel/tmp/community-restore"
uploadPath = "1panel/uploads" upgradePath = "1panel/tmp/upgrade"
downloadPath = "1panel/download" uploadPath = "1panel/uploads"
downloadPath = "1panel/download"
) )
func (u *DeviceService) Scan() dto.CleanData { func (u *DeviceService) Scan() dto.CleanData {
@@ -58,7 +59,7 @@ func (u *DeviceService) Scan() dto.CleanData {
SystemClean.BackupClean = loadBackupTree(fileOp) SystemClean.BackupClean = loadBackupTree(fileOp)
rollBackTree := loadRollBackTree(fileOp) rollBackTree := loadRollBackTree()
rollbackSize := uint64(0) rollbackSize := uint64(0)
for _, rollback := range rollBackTree { for _, rollback := range rollBackTree {
rollbackSize += rollback.Size rollbackSize += rollback.Size
@@ -113,12 +114,15 @@ func (u *DeviceService) Clean(req []dto.Clean) {
dropFileOrDir(path.Join(global.Dir.BaseDir, rollbackPath, "app")) dropFileOrDir(path.Join(global.Dir.BaseDir, rollbackPath, "app"))
dropFileOrDir(path.Join(global.Dir.BaseDir, rollbackPath, "database")) dropFileOrDir(path.Join(global.Dir.BaseDir, rollbackPath, "database"))
dropFileOrDir(path.Join(global.Dir.BaseDir, rollbackPath, "website")) dropFileOrDir(path.Join(global.Dir.BaseDir, rollbackPath, "website"))
dropFileOrDir(path.Join(global.Dir.BaseDir, communityRestorePath))
case "rollback_app": case "rollback_app":
dropFileOrDir(path.Join(global.Dir.BaseDir, rollbackPath, "app", item.Name)) dropFileOrDir(path.Join(global.Dir.BaseDir, rollbackPath, "app", item.Name))
case "rollback_database": case "rollback_database":
dropFileOrDir(path.Join(global.Dir.BaseDir, rollbackPath, "database", item.Name)) dropFileOrDir(path.Join(global.Dir.BaseDir, rollbackPath, "database", item.Name))
case "rollback_website": case "rollback_website":
dropFileOrDir(path.Join(global.Dir.BaseDir, rollbackPath, "website", item.Name)) dropFileOrDir(path.Join(global.Dir.BaseDir, rollbackPath, "website", item.Name))
case "rollback_community_restore":
dropFileOrDir(path.Join(global.Dir.BaseDir, communityRestorePath, item.Name))
case "upload": case "upload":
dropFileOrDir(path.Join(global.Dir.BaseDir, uploadPath, item.Name)) dropFileOrDir(path.Join(global.Dir.BaseDir, uploadPath, item.Name))
@@ -214,6 +218,7 @@ func doSystemClean(taskItem *task.Task) func(t *task.Task) error {
dropWithTask(path.Join(global.Dir.BaseDir, rollbackPath, "app"), taskItem, &size, &fileCount) dropWithTask(path.Join(global.Dir.BaseDir, rollbackPath, "app"), taskItem, &size, &fileCount)
dropWithTask(path.Join(global.Dir.BaseDir, rollbackPath, "website"), taskItem, &size, &fileCount) dropWithTask(path.Join(global.Dir.BaseDir, rollbackPath, "website"), taskItem, &size, &fileCount)
dropWithTask(path.Join(global.Dir.BaseDir, rollbackPath, "database"), taskItem, &size, &fileCount) dropWithTask(path.Join(global.Dir.BaseDir, rollbackPath, "database"), taskItem, &size, &fileCount)
dropWithTask(path.Join(global.Dir.BaseDir, communityRestorePath), taskItem, &size, &fileCount)
upgrades := path.Join(global.Dir.BaseDir, upgradePath) upgrades := path.Join(global.Dir.BaseDir, upgradePath)
oldUpgradeFiles, _ := os.ReadDir(upgrades) oldUpgradeFiles, _ := os.ReadDir(upgrades)
@@ -606,20 +611,21 @@ func isExactPathMatch(path string, excludePaths []string) bool {
return false return false
} }
func loadRollBackTree(fileOp fileUtils.FileOp) []dto.CleanTree { func loadRollBackTree() []dto.CleanTree {
var treeData []dto.CleanTree var treeData []dto.CleanTree
treeData = loadTreeWithCheck(treeData, path.Join(global.Dir.BaseDir, rollbackPath, "app"), "rollback_app", fileOp) treeData = loadTreeWithCheck(treeData, path.Join(global.Dir.BaseDir, rollbackPath, "app"), "rollback_app")
treeData = loadTreeWithCheck(treeData, path.Join(global.Dir.BaseDir, rollbackPath, "website"), "rollback_website", fileOp) treeData = loadTreeWithCheck(treeData, path.Join(global.Dir.BaseDir, rollbackPath, "website"), "rollback_website")
treeData = loadTreeWithCheck(treeData, path.Join(global.Dir.BaseDir, rollbackPath, "database"), "rollback_database", fileOp) treeData = loadTreeWithCheck(treeData, path.Join(global.Dir.BaseDir, rollbackPath, "database"), "rollback_database")
treeData = loadTreeWithCheck(treeData, path.Join(global.Dir.BaseDir, communityRestorePath), "rollback_community_restore")
return treeData return treeData
} }
func loadUploadTree(fileOp fileUtils.FileOp) []dto.CleanTree { func loadUploadTree(fileOp fileUtils.FileOp) []dto.CleanTree {
var treeData []dto.CleanTree var treeData []dto.CleanTree
treeData = loadTreeWithCheck(treeData, path.Join(global.Dir.BaseDir, uploadPath, "app"), "upload_app", fileOp) treeData = loadTreeWithCheck(treeData, path.Join(global.Dir.BaseDir, uploadPath, "app"), "upload_app")
treeData = loadTreeWithCheck(treeData, path.Join(global.Dir.BaseDir, uploadPath, "website"), "upload_website", fileOp) treeData = loadTreeWithCheck(treeData, path.Join(global.Dir.BaseDir, uploadPath, "website"), "upload_website")
treeData = loadTreeWithCheck(treeData, path.Join(global.Dir.BaseDir, uploadPath, "database"), "upload_database", fileOp) treeData = loadTreeWithCheck(treeData, path.Join(global.Dir.BaseDir, uploadPath, "database"), "upload_database")
path5 := path.Join(global.Dir.BaseDir, uploadPath) path5 := path.Join(global.Dir.BaseDir, uploadPath)
uploadTreeData := loadTreeWithAllFile(true, path5, "upload", path5, fileOp) uploadTreeData := loadTreeWithAllFile(true, path5, "upload", path5, fileOp)
@@ -630,9 +636,9 @@ func loadUploadTree(fileOp fileUtils.FileOp) []dto.CleanTree {
func loadDownloadTree(fileOp fileUtils.FileOp) []dto.CleanTree { func loadDownloadTree(fileOp fileUtils.FileOp) []dto.CleanTree {
var treeData []dto.CleanTree var treeData []dto.CleanTree
treeData = loadTreeWithCheck(treeData, path.Join(global.Dir.BaseDir, downloadPath, "app"), "download_app", fileOp) treeData = loadTreeWithCheck(treeData, path.Join(global.Dir.BaseDir, downloadPath, "app"), "download_app")
treeData = loadTreeWithCheck(treeData, path.Join(global.Dir.BaseDir, downloadPath, "website"), "download_website", fileOp) treeData = loadTreeWithCheck(treeData, path.Join(global.Dir.BaseDir, downloadPath, "website"), "download_website")
treeData = loadTreeWithCheck(treeData, path.Join(global.Dir.BaseDir, downloadPath, "database"), "download_database", fileOp) treeData = loadTreeWithCheck(treeData, path.Join(global.Dir.BaseDir, downloadPath, "database"), "download_database")
path5 := path.Join(global.Dir.BaseDir, downloadPath) path5 := path.Join(global.Dir.BaseDir, downloadPath)
uploadTreeData := loadTreeWithAllFile(true, path5, "download", path5, fileOp) uploadTreeData := loadTreeWithAllFile(true, path5, "download", path5, fileOp)
@@ -814,16 +820,59 @@ func loadContainerTree() []dto.CleanTree {
return treeData return treeData
} }
func loadTreeWithCheck(treeData []dto.CleanTree, pathItem, treeType string, fileOp fileUtils.FileOp) []dto.CleanTree { func loadTreeWithCheck(treeData []dto.CleanTree, pathItem, treeType string) []dto.CleanTree {
size, _ := fileOp.GetDirSize(pathItem) list, size := loadTreeWithFileSize(true, pathItem, treeType, pathItem)
if size == 0 { if len(list) == 0 || size == 0 {
return treeData return treeData
} }
list := loadTreeWithAllFile(true, pathItem, treeType, pathItem, fileOp) treeData = append(treeData, dto.CleanTree{ID: uuid.NewString(), Label: treeType, Size: size, IsCheck: size > 0, Children: list, Type: treeType, IsRecommend: true, CanDelete: false})
treeData = append(treeData, dto.CleanTree{ID: uuid.NewString(), Label: treeType, Size: uint64(size), IsCheck: size > 0, Children: list, Type: treeType, IsRecommend: true, CanDelete: false})
return treeData return treeData
} }
func loadTreeWithFileSize(isCheck bool, originalPath, treeType, pathItem string) ([]dto.CleanTree, uint64) {
var (
lists []dto.CleanTree
total uint64
)
entries, err := os.ReadDir(pathItem)
if err != nil {
return lists, total
}
for _, entry := range entries {
item := dto.CleanTree{
ID: uuid.NewString(),
Label: entry.Name(),
Type: treeType,
Name: strings.TrimPrefix(path.Join(pathItem, entry.Name()), originalPath+"/"),
IsCheck: isCheck,
IsRecommend: isCheck,
CanDelete: true,
}
entryPath := path.Join(pathItem, entry.Name())
if entry.IsDir() {
children, size := loadTreeWithFileSize(isCheck, originalPath, treeType, entryPath)
if len(children) == 0 {
continue
}
item.Children = children
item.Size = size
} else {
info, err := entry.Info()
if err != nil {
continue
}
item.Size = uint64(info.Size())
}
if item.Size == 0 {
continue
}
total += item.Size
lists = append(lists, item)
}
return lists, total
}
func loadTreeWithDir(isCheck bool, treeType, pathItem string, fileOp fileUtils.FileOp) []dto.CleanTree { func loadTreeWithDir(isCheck bool, treeType, pathItem string, fileOp fileUtils.FileOp) []dto.CleanTree {
var lists []dto.CleanTree var lists []dto.CleanTree
files, err := os.ReadDir(pathItem) files, err := os.ReadDir(pathItem)
+47 -27
View File
@@ -61,18 +61,21 @@ func parseDevice(dev LsblkDevice) []response.DiskBasicInfo {
var used, avail, totalSize string var used, avail, totalSize string
var usePercent int var usePercent int
isMounted := mountPoint != "" isMounted := mountPoint != "" && mountPoint != "-"
isSystem := false isSystem := false
if dev.Fstype == "LVM2_member" && len(dev.Children) > 0 { if dev.Fstype == "LVM2_member" && len(dev.Children) > 0 {
for _, child := range dev.Children { for _, child := range dev.Children {
if child.Type == "lvm" && child.Mountpoint != "" { if child.Type == "lvm" && child.Mountpoint != "" && child.Mountpoint != "-" {
devicePath := "/dev/mapper/" + child.Name totalSize, used, avail, usePercent, _ := getDiskUsageInfo(child.Mountpoint)
totalSize, used, avail, usePercent, _ := getDiskUsageInfo(devicePath) childSize := child.Size
if totalSize != "" {
childSize = totalSize
}
childInfo := response.DiskBasicInfo{ childInfo := response.DiskBasicInfo{
Device: dev.Name, Device: dev.Name,
Size: totalSize, Size: childSize,
Model: dev.Model, Model: dev.Model,
DiskType: diskType, DiskType: diskType,
Filesystem: child.Fstype, Filesystem: child.Fstype,
@@ -91,8 +94,7 @@ func parseDevice(dev LsblkDevice) []response.DiskBasicInfo {
return list return list
} else if isMounted { } else if isMounted {
isSystem = isSystemDisk(mountPoint) isSystem = isSystemDisk(mountPoint)
devicePath := "/dev/" + dev.Name totalSize, used, avail, usePercent, _ = getDiskUsageInfo(mountPoint)
totalSize, used, avail, usePercent, _ = getDiskUsageInfo(devicePath)
if totalSize != "" { if totalSize != "" {
size = totalSize size = totalSize
} }
@@ -229,9 +231,14 @@ func parseLsblkOutput(output string) ([]response.DiskBasicInfo, error) {
size := fields["SIZE"] size := fields["SIZE"]
if diskType == "lvm" { if diskType == "lvm" {
total, used, avail, usePercent, _ := getDiskUsageInfo("/dev/mapper/" + name) var total, used, avail string
if total != "" && fsType != "" { var usePercent int
size = total isMounted := mountPoint != "" && mountPoint != "-"
if isMounted {
total, used, avail, usePercent, _ = getDiskUsageInfo(mountPoint)
if total != "" && fsType != "" {
size = total
}
} }
lvmInfo := response.DiskBasicInfo{ lvmInfo := response.DiskBasicInfo{
@@ -246,7 +253,7 @@ func parseLsblkOutput(output string) ([]response.DiskBasicInfo, error) {
Avail: avail, Avail: avail,
UsePercent: usePercent, UsePercent: usePercent,
MountPoint: mountPoint, MountPoint: mountPoint,
IsMounted: mountPoint != "" && mountPoint != "-", IsMounted: isMounted,
Serial: fields["SERIAL"], Serial: fields["SERIAL"],
} }
lvmMap[name] = lvmInfo lvmMap[name] = lvmInfo
@@ -269,8 +276,8 @@ func parseLsblkOutput(output string) ([]response.DiskBasicInfo, error) {
used, avail, totalSize string used, avail, totalSize string
usePercent int usePercent int
) )
if mountPoint != "" { if mountPoint != "" && mountPoint != "-" {
totalSize, used, avail, usePercent, _ = getDiskUsageInfo("/dev/" + name) totalSize, used, avail, usePercent, _ = getDiskUsageInfo(mountPoint)
if totalSize != "" { if totalSize != "" {
size = totalSize size = totalSize
} }
@@ -387,26 +394,39 @@ func getParentDevice(device string) string {
return device return device
} }
func getDiskUsageInfo(device string) (size, used, avail string, usePercent int, err error) { func getDiskUsageInfo(mountPoint string) (size, used, avail string, usePercent int, err error) {
output, err := cmd.NewCommandMgr(cmd.WithTimeout(20*time.Second)).RunWithStdout("df", "-h", device) // Query by mount point instead of a reconstructed device path. The mount table may record
// a different device alias such as /dev/root.
output, err := cmd.NewCommandMgr(cmd.WithTimeout(20*time.Second)).RunWithStdout("df", "-h", "-P", mountPoint)
if err != nil { if err != nil {
return "", "", "", 0, nil return "", "", "", 0, nil
} }
return parseDiskUsageOutput(output)
}
func parseDiskUsageOutput(output string) (size, used, avail string, usePercent int, err error) {
lines := strings.Split(strings.TrimSpace(output), "\n") lines := strings.Split(strings.TrimSpace(output), "\n")
if len(lines) > 1 { for i := len(lines) - 1; i >= 0; i-- {
output = lines[len(lines)-1] fields := strings.Fields(lines[i])
for index, field := range fields {
if index < 3 || !strings.HasSuffix(field, "%") {
continue
}
percent, parseErr := strconv.Atoi(strings.TrimSuffix(field, "%"))
if parseErr != nil {
continue
}
return fields[index-3], fields[index-2], fields[index-1], percent, nil
}
for index, field := range fields {
if index < 3 || index+1 >= len(fields) || field != "-" || !strings.HasPrefix(fields[index+1], "/") {
continue
}
return fields[index-3], fields[index-2], fields[index-1], 0, nil
}
} }
fields := strings.Fields(output) return "", "", "", 0, nil
if len(fields) >= 5 {
size = fields[1]
used = fields[2]
avail = fields[3]
usePercentStr := strings.TrimSuffix(fields[4], "%")
usePercent, _ = strconv.Atoi(usePercentStr)
}
return size, used, avail, usePercent, nil
} }
func formatDisk(req dto.DiskFormatRequest) error { func formatDisk(req dto.DiskFormatRequest) error {
+123 -1
View File
@@ -2,6 +2,7 @@ package service
import ( import (
"bufio" "bufio"
"bytes"
"context" "context"
"encoding/json" "encoding/json"
"fmt" "fmt"
@@ -14,14 +15,20 @@ import (
"github.com/1Panel-dev/1Panel/agent/constant" "github.com/1Panel-dev/1Panel/agent/constant"
"github.com/1Panel-dev/1Panel/agent/global" "github.com/1Panel-dev/1Panel/agent/global"
"github.com/1Panel-dev/1Panel/agent/utils/cmd" "github.com/1Panel-dev/1Panel/agent/utils/cmd"
"github.com/1Panel-dev/1Panel/agent/utils/common"
"github.com/1Panel-dev/1Panel/agent/utils/controller" "github.com/1Panel-dev/1Panel/agent/utils/controller"
"github.com/1Panel-dev/1Panel/agent/utils/docker" "github.com/1Panel-dev/1Panel/agent/utils/docker"
dockerfirewall "github.com/1Panel-dev/1Panel/agent/utils/firewall/docker_guard"
) )
const dockerNftablesMinVersion = "29.0.0"
type DockerService struct{} type DockerService struct{}
type IDockerService interface { type IDockerService interface {
UpdateConf(req dto.SettingUpdate, withRestart bool) error UpdateConf(req dto.SettingUpdate, withRestart bool) error
UpdateFirewallBackend(backend string) error
UpdateLogOption(req dto.LogOption) error UpdateLogOption(req dto.LogOption) error
UpdateIpv6Option(req dto.Ipv6Option) error UpdateIpv6Option(req dto.Ipv6Option) error
UpdateConfByFile(info dto.DaemonJsonUpdateByFile) error UpdateConfByFile(info dto.DaemonJsonUpdateByFile) error
@@ -30,6 +37,115 @@ type IDockerService interface {
OperateDocker(req dto.DockerOperation) error OperateDocker(req dto.DockerOperation) error
} }
func loadDockerEngineVersion(ctx context.Context) string {
client, err := docker.NewDockerClient()
if err == nil {
defer client.Close()
if version, versionErr := client.ServerVersion(ctx); versionErr == nil && version.Version != "" {
return version.Version
}
}
if !cmd.Which("dockerd") {
return ""
}
stdout, err := cmd.NewCommandMgr(cmd.WithTimeout(20*time.Second)).RunWithStdout("dockerd", "--version")
if err != nil {
return ""
}
return strings.TrimSpace(stdout)
}
func dockerNftablesSupported(version string) bool {
return version != "" && common.CompareAppVersion(version, dockerNftablesMinVersion)
}
func applyDockerFirewallBackendConfig(daemonMap map[string]interface{}, backend, version string) error {
switch backend {
case constant.FirewallProviderNftables:
if !dockerNftablesSupported(version) {
return fmt.Errorf("Docker Engine %s or later is required for the nftables firewall backend", dockerNftablesMinVersion)
}
daemonMap["experimental"] = true
daemonMap["firewall-backend"] = constant.FirewallProviderNftables
case constant.FirewallProviderIptables:
if dockerNftablesSupported(version) {
daemonMap["firewall-backend"] = constant.FirewallProviderIptables
} else {
delete(daemonMap, "firewall-backend")
}
default:
return fmt.Errorf("unsupported Docker firewall backend %q", backend)
}
return nil
}
func (u *DockerService) UpdateFirewallBackend(backend string) error {
version := loadDockerEngineVersion(context.Background())
if backend == constant.FirewallProviderNftables && !dockerNftablesSupported(version) {
return fmt.Errorf("Docker Engine %s or later is required for the nftables firewall backend", dockerNftablesMinVersion)
}
if backend == constant.FirewallProviderNftables {
if err := dockerfirewall.CheckIPv4Forwarding(); err != nil {
return err
}
}
original, readErr := os.ReadFile(constant.DaemonJsonPath)
existed := readErr == nil
if readErr != nil && !os.IsNotExist(readErr) {
return readErr
}
daemonMap := make(map[string]interface{})
if len(bytes.TrimSpace(original)) > 0 {
if err := json.Unmarshal(original, &daemonMap); err != nil {
return fmt.Errorf("failed to parse Docker configuration: %w", err)
}
}
if err := applyDockerFirewallBackendConfig(daemonMap, backend, version); err != nil {
return err
}
updated, err := json.MarshalIndent(daemonMap, "", "\t")
if err != nil {
return err
}
if existed && bytes.Equal(bytes.TrimSpace(original), bytes.TrimSpace(updated)) {
return nil
}
if err := os.MkdirAll(path.Dir(constant.DaemonJsonPath), 0755); err != nil {
return err
}
if err := os.WriteFile(constant.DaemonJsonPath, updated, 0640); err != nil {
return err
}
restore := func() error {
if existed {
return os.WriteFile(constant.DaemonJsonPath, original, 0640)
}
err := os.Remove(constant.DaemonJsonPath)
if os.IsNotExist(err) {
return nil
}
return err
}
if err := validateDockerConfig(); err != nil {
if restoreErr := restore(); restoreErr != nil {
return fmt.Errorf("%v; failed to restore Docker configuration: %w", err, restoreErr)
}
return err
}
if err := controller.HandleRestart("docker"); err != nil {
cause := fmt.Errorf("failed to restart Docker: %w", err)
if restoreErr := restore(); restoreErr != nil {
return fmt.Errorf("%v; failed to restore Docker configuration: %w", cause, restoreErr)
}
if restoreRestartErr := controller.HandleRestart("docker"); restoreRestartErr != nil {
return fmt.Errorf("%v; the previous configuration was restored but Docker could not be restarted: %w", cause, restoreRestartErr)
}
return cause
}
return nil
}
func NewIDockerService() IDockerService { func NewIDockerService() IDockerService {
return &DockerService{} return &DockerService{}
} }
@@ -167,7 +283,10 @@ func (u *DockerService) UpdateConf(req dto.SettingUpdate, withRestart bool) erro
delete(daemonMap, "ipv6") delete(daemonMap, "ipv6")
delete(daemonMap, "fixed-cidr-v6") delete(daemonMap, "fixed-cidr-v6")
delete(daemonMap, "ip6tables") delete(daemonMap, "ip6tables")
delete(daemonMap, "experimental") backend, _ := settingRepo.GetValueByKey(constant.FirewallDockerBackendKey)
if !strings.EqualFold(strings.TrimSpace(backend), constant.FirewallProviderNftables) {
delete(daemonMap, "experimental")
}
} }
case "LogOption": case "LogOption":
if req.Value == "disable" { if req.Value == "disable" {
@@ -368,6 +487,9 @@ func (u *DockerService) OperateDocker(req dto.DockerOperation) error {
if err := controller.Handle(req.Operation, service); err != nil { if err := controller.Handle(req.Operation, service); err != nil {
return err return err
} }
if req.Operation == "start" || req.Operation == "restart" {
return RestoreDockerPortGuard(context.Background())
}
return nil return nil
} }
+13 -10
View File
@@ -32,20 +32,23 @@ var (
scriptRepo = repo.NewIScriptRepo() scriptRepo = repo.NewIScriptRepo()
cronjobRepo = repo.NewICronjobRepo() cronjobRepo = repo.NewICronjobRepo()
hostRepo = repo.NewIHostRepo() hostRepo = repo.NewIHostRepo()
ftpRepo = repo.NewIFtpRepo() ftpRepo = repo.NewIFtpRepo()
clamRepo = repo.NewIClamRepo() clamRepo = repo.NewIClamRepo()
monitorRepo = repo.NewIMonitorRepo() monitorRepo = repo.NewIMonitorRepo()
vllmMonitorRepo = &repo.VLLMMonitorRepo{}
settingRepo = repo.NewISettingRepo() settingRepo = repo.NewISettingRepo()
backupRepo = repo.NewIBackupRepo() backupRepo = repo.NewIBackupRepo()
websiteRepo = repo.NewIWebsiteRepo() websiteRepo = repo.NewIWebsiteRepo()
websiteDomainRepo = repo.NewIWebsiteDomainRepo() websiteDomainRepo = repo.NewIWebsiteDomainRepo()
websiteDnsRepo = repo.NewIWebsiteDnsAccountRepo() websiteDnsRepo = repo.NewIWebsiteDnsAccountRepo()
websiteSSLRepo = repo.NewISSLRepo() websiteSSLRepo = repo.NewISSLRepo()
websiteAcmeRepo = repo.NewIAcmeAccountRepo() websiteAcmeRepo = repo.NewIAcmeAccountRepo()
websiteCARepo = repo.NewIWebsiteCARepo() websiteCARepo = repo.NewIWebsiteCARepo()
websiteTemplateRepo = repo.NewIWebsiteTemplateRepo()
websiteTemplateOutputRepo = repo.NewIWebsiteTemplateOutputRepo()
snapshotRepo = repo.NewISnapshotRepo() snapshotRepo = repo.NewISnapshotRepo()
+1 -2
View File
@@ -9,7 +9,6 @@ import (
"github.com/1Panel-dev/1Panel/agent/app/dto" "github.com/1Panel-dev/1Panel/agent/app/dto"
"github.com/1Panel-dev/1Panel/agent/buserr" "github.com/1Panel-dev/1Panel/agent/buserr"
"github.com/1Panel-dev/1Panel/agent/utils/firewall"
"github.com/1Panel-dev/1Panel/agent/utils/toolbox" "github.com/1Panel-dev/1Panel/agent/utils/toolbox"
) )
@@ -109,7 +108,7 @@ func (u *Fail2BanService) UpdateConf(req dto.Fail2BanUpdate) error {
if req.Value == "firewallcmd-ipset" { if req.Value == "firewallcmd-ipset" {
itemName = "firewalld" itemName = "firewalld"
} }
client, err := firewall.NewFirewallClient() client, err := NewSelectedSystemFirewallClient()
if err != nil { if err != nil {
return err return err
} }
+150 -28
View File
@@ -16,6 +16,7 @@ import (
"sort" "sort"
"strconv" "strconv"
"strings" "strings"
"syscall"
"time" "time"
"unicode/utf8" "unicode/utf8"
@@ -51,6 +52,8 @@ type FileService struct {
const fileHistorySnapshotMaxSize = 10 * 1024 * 1024 const fileHistorySnapshotMaxSize = 10 * 1024 * 1024
var fileTransferLocks = newFileTransferLocks()
type IFileService interface { type IFileService interface {
GetFileList(op request.FileOption) (response.FileInfo, error) GetFileList(op request.FileOption) (response.FileInfo, error)
SearchUploadWithPage(req request.SearchUploadWithPage) (int64, interface{}, error) SearchUploadWithPage(req request.SearchUploadWithPage) (int64, interface{}, error)
@@ -71,6 +74,7 @@ type IFileService interface {
ChangeName(req request.FileRename) error ChangeName(req request.FileRename) error
Wget(w request.FileWget) (string, error) Wget(w request.FileWget) (string, error)
MvFile(m request.FileMove) error MvFile(m request.FileMove) error
StopMvFile(taskID string) error
ChangeOwner(req request.FileRoleUpdate) error ChangeOwner(req request.FileRoleUpdate) error
ChangeMode(op request.FileCreate) error ChangeMode(op request.FileCreate) error
BatchChangeModeAndOwner(op request.FileRoleReq) error BatchChangeModeAndOwner(op request.FileRoleReq) error
@@ -155,6 +159,10 @@ func (f *FileService) SearchUploadWithPage(req request.SearchUploadWithPage) (in
}) })
} }
sort.SliceStable(files, func(i, j int) bool {
return files[i].CreatedAt > files[j].CreatedAt
})
total, start, end := len(files), (req.Page-1)*req.PageSize, req.Page*req.PageSize total, start, end := len(files), (req.Page-1)*req.PageSize, req.Page*req.PageSize
if start > total { if start > total {
backData = make([]response.UploadInfo, 0) backData = make([]response.UploadInfo, 0)
@@ -431,13 +439,15 @@ func (f *FileService) Compress(c request.FileCompress) error {
if err := preflightCompressTool(files.CompressType(c.Type)); err != nil { if err := preflightCompressTool(files.CompressType(c.Type)); err != nil {
return err return err
} }
taskItem, err := task.NewTask(c.Name, task.TaskExec, task.TaskScopeTask, c.TaskID, 1) taskName := i18n.GetMsgWithMap("FileTaskCompress", map[string]interface{}{"dst": strconv.Quote(filepath.Join(c.Dst, c.Name))})
taskItem, err := task.NewTask(taskName, task.TaskExec, task.TaskScopeTask, c.TaskID, 1)
if err != nil { if err != nil {
return err return err
} }
go func() { go func() {
taskItem.AddSubTask(c.Name, func(t *task.Task) error { taskItem.AddSubTask(taskName, func(t *task.Task) error {
t.LogStart(c.Name) logFileTaskSources(t, c.Files)
t.Log(i18n.GetMsgWithMap("FileTaskFormat", map[string]interface{}{"format": strconv.Quote(c.Type)}))
compressType := files.CompressType(c.Type) compressType := files.CompressType(c.Type)
dstFile := filepath.Join(c.Dst, c.Name) dstFile := filepath.Join(c.Dst, c.Name)
success := false success := false
@@ -466,7 +476,7 @@ func (f *FileService) Compress(c request.FileCompress) error {
func preflightCompressTool(compressType files.CompressType) error { func preflightCompressTool(compressType files.CompressType) error {
switch compressType { switch compressType {
case files.TarGz, files.Rar, files.X7z: case files.Tar, files.Gz, files.Bz2, files.TarBz2, files.Tgz, files.TarGz, files.Xz, files.TarXz, files.Rar, files.X7z:
_, err := files.NewShellArchiver(compressType) _, err := files.NewShellArchiver(compressType)
return err return err
default: default:
@@ -476,7 +486,7 @@ func preflightCompressTool(compressType files.CompressType) error {
func preflightDecompressTool(decompressType files.CompressType) error { func preflightDecompressTool(decompressType files.CompressType) error {
switch decompressType { switch decompressType {
case files.Rar, files.X7z: case files.Rar:
_, err := files.NewExtractShellArchiver(decompressType) _, err := files.NewExtractShellArchiver(decompressType)
return err return err
default: default:
@@ -508,13 +518,15 @@ func (f *FileService) DeCompress(c request.FileDeCompress) error {
if err := preflightDecompressTool(files.CompressType(c.Type)); err != nil { if err := preflightDecompressTool(files.CompressType(c.Type)); err != nil {
return err return err
} }
taskItem, err := task.NewTask(c.Path, task.TaskExec, task.TaskScopeTask, c.TaskID, 1) taskName := i18n.GetMsgWithMap("FileTaskDecompress", map[string]interface{}{"dst": strconv.Quote(c.Dst)})
taskItem, err := task.NewTask(taskName, task.TaskExec, task.TaskScopeTask, c.TaskID, 1)
if err != nil { if err != nil {
return err return err
} }
go func() { go func() {
taskItem.AddSubTask(c.Path, func(t *task.Task) error { taskItem.AddSubTask(taskName, func(t *task.Task) error {
t.LogStart(c.Path) logFileTaskSources(t, []string{c.Path})
t.Log(i18n.GetMsgWithMap("FileTaskFormat", map[string]interface{}{"format": strconv.Quote(c.Type)}))
dstExisted := fo.Stat(c.Dst) dstExisted := fo.Stat(c.Dst)
parentDir := filepath.Dir(c.Dst) parentDir := filepath.Dir(c.Dst)
if !fo.Stat(parentDir) { if !fo.Stat(parentDir) {
@@ -533,7 +545,10 @@ func (f *FileService) DeCompress(c request.FileDeCompress) error {
_ = os.RemoveAll(c.Dst) _ = os.RemoveAll(c.Dst)
} }
}() }()
if err := fo.Decompress(t.TaskCtx, c.Path, tempDst, files.CompressType(c.Type), c.Secret); err != nil { if err := fo.DecompressWithOptions(t.TaskCtx, c.Path, tempDst, files.CompressType(c.Type), c.Secret, files.DecompressOptions{
PreserveOwner: true,
AllowCLIReextract: true,
}); err != nil {
return err return err
} }
if err := fo.CreateDir(c.Dst, constant.DirPerm); err != nil { if err := fo.CreateDir(c.Dst, constant.DirPerm); err != nil {
@@ -551,19 +566,42 @@ func (f *FileService) DeCompress(c request.FileDeCompress) error {
} }
func copyDecompressTree(ctx context.Context, srcDir, dstDir string) error { func copyDecompressTree(ctx context.Context, srcDir, dstDir string) error {
state := decompressCopyState{hardlinks: make(map[decompressFileIdentity]string)}
entries, err := os.ReadDir(srcDir) entries, err := os.ReadDir(srcDir)
if err != nil { if err != nil {
return err return err
} }
for _, entry := range entries { for _, entry := range entries {
if err := copyDecompressEntry(ctx, filepath.Join(srcDir, entry.Name()), filepath.Join(dstDir, entry.Name())); err != nil { if err := copyDecompressEntryWithState(ctx, filepath.Join(srcDir, entry.Name()), filepath.Join(dstDir, entry.Name()), &state); err != nil {
return err return err
} }
} }
return nil return nil
} }
type decompressFileIdentity struct {
device uint64
inode uint64
}
type decompressCopyState struct {
hardlinks map[decompressFileIdentity]string
}
func decompressHardlinkIdentity(info os.FileInfo) (decompressFileIdentity, bool) {
stat, ok := info.Sys().(*syscall.Stat_t)
if !ok || stat.Nlink < 2 {
return decompressFileIdentity{}, false
}
return decompressFileIdentity{device: uint64(stat.Dev), inode: uint64(stat.Ino)}, true
}
func copyDecompressEntry(ctx context.Context, srcPath, dstPath string) (retErr error) { func copyDecompressEntry(ctx context.Context, srcPath, dstPath string) (retErr error) {
state := decompressCopyState{hardlinks: make(map[decompressFileIdentity]string)}
return copyDecompressEntryWithState(ctx, srcPath, dstPath, &state)
}
func copyDecompressEntryWithState(ctx context.Context, srcPath, dstPath string, state *decompressCopyState) (retErr error) {
if err := ctx.Err(); err != nil { if err := ctx.Err(); err != nil {
return err return err
} }
@@ -605,13 +643,16 @@ func copyDecompressEntry(ctx context.Context, srcPath, dstPath string) (retErr e
if err := applyDecompressOwnership(srcPath, dstPath); err != nil { if err := applyDecompressOwnership(srcPath, dstPath); err != nil {
return err return err
} }
if err := os.Chmod(dstPath, info.Mode().Perm()); err != nil {
return err
}
} }
entries, err := os.ReadDir(srcPath) entries, err := os.ReadDir(srcPath)
if err != nil { if err != nil {
return err return err
} }
for _, entry := range entries { for _, entry := range entries {
if err := copyDecompressEntry(ctx, filepath.Join(srcPath, entry.Name()), filepath.Join(dstPath, entry.Name())); err != nil { if err := copyDecompressEntryWithState(ctx, filepath.Join(srcPath, entry.Name()), filepath.Join(dstPath, entry.Name()), state); err != nil {
return err return err
} }
} }
@@ -634,6 +675,15 @@ func copyDecompressEntry(ctx context.Context, srcPath, dstPath string) (retErr e
if err := os.MkdirAll(filepath.Dir(dstPath), constant.DirPerm); err != nil { if err := os.MkdirAll(filepath.Dir(dstPath), constant.DirPerm); err != nil {
return err return err
} }
identity, isHardlink := decompressHardlinkIdentity(info)
if !keepExistingFile && isHardlink {
if existingPath, ok := state.hardlinks[identity]; ok {
if err := os.Link(existingPath, dstPath); err != nil {
return err
}
return os.Chtimes(dstPath, info.ModTime(), info.ModTime())
}
}
srcFile, err := os.Open(srcPath) srcFile, err := os.Open(srcPath)
if err != nil { if err != nil {
@@ -658,6 +708,12 @@ func copyDecompressEntry(ctx context.Context, srcPath, dstPath string) (retErr e
if err := applyDecompressOwnership(srcPath, dstPath); err != nil { if err := applyDecompressOwnership(srcPath, dstPath); err != nil {
return err return err
} }
if err := os.Chmod(dstPath, info.Mode().Perm()); err != nil {
return err
}
if isHardlink {
state.hardlinks[identity] = dstPath
}
} }
return os.Chtimes(dstPath, info.ModTime(), info.ModTime()) return os.Chtimes(dstPath, info.ModTime(), info.ModTime())
} }
@@ -667,7 +723,7 @@ func applyDecompressOwnership(srcPath, dstPath string) error {
if err != nil { if err != nil {
return err return err
} }
stat, ok := info.Sys().(*unix.Stat_t) stat, ok := info.Sys().(*syscall.Stat_t)
if !ok { if !ok {
return nil return nil
} }
@@ -844,6 +900,7 @@ func (f *FileService) Wget(w request.FileWget) (string, error) {
key := "file-wget-" + common.GetUuid() key := "file-wget-" + common.GetUuid()
options := files.DownloadOptions{ options := files.DownloadOptions{
IgnoreCertificate: w.IgnoreCertificate, IgnoreCertificate: w.IgnoreCertificate,
UseServerFilename: w.UseServerFilename,
} }
if w.UseProxy { if w.UseProxy {
systemProxy, err := NewISettingService().GetSystemProxy() systemProxy, err := NewISettingService().GetSystemProxy()
@@ -861,19 +918,79 @@ func (f *FileService) Wget(w request.FileWget) (string, error) {
return key, fo.DownloadFileWithProcess(w.Url, filepath.Join(w.Path, w.Name), key, options) return key, fo.DownloadFileWithProcess(w.Url, filepath.Join(w.Path, w.Name), key, options)
} }
func logFileTaskSources(t *task.Task, sources []string) {
for _, source := range sources {
t.Log(i18n.GetMsgWithMap("FileTaskSource", map[string]interface{}{"path": strconv.Quote(source)}))
}
}
func (f *FileService) MvFile(m request.FileMove) error { func (f *FileService) MvFile(m request.FileMove) error {
fo := files.NewFileOp() fo := files.NewFileOp()
if err := validateFileMove(fo, m); err != nil {
return err
}
if m.TaskID == "" {
m.TaskID = common.GetUuid()
}
if !fileTransferLocks.Acquire(m.TaskID, getFileTransferPaths(m)) {
return buserr.New("TaskIsExecuting")
}
nameKey := "FileTaskCopy"
if m.Type == "cut" {
nameKey = "FileTaskMove"
}
taskName := i18n.GetMsgWithMap(nameKey, map[string]interface{}{"dst": strconv.Quote(m.NewPath)})
taskItem, err := task.NewTask(taskName, task.TaskExec, task.TaskScopeTask, m.TaskID, 1)
if err != nil {
fileTransferLocks.Release(m.TaskID)
return err
}
go func() {
defer fileTransferLocks.Release(m.TaskID)
taskItem.AddSubTaskWithOps(taskName, func(t *task.Task) error {
logFileTaskSources(t, m.OldPaths)
logFileTaskSources(t, m.CoverPaths)
if m.Name != "" {
t.Log(i18n.GetMsgWithMap("FileTaskRename", map[string]interface{}{"name": strconv.Quote(m.Name)}))
}
err := f.moveFileWithContext(t.TaskCtx, m)
if err != nil && t.TaskCtx.Err() != nil {
return t.TaskCtx.Err()
}
return err
}, nil, 0, 0)
_ = taskItem.Execute()
}()
return nil
}
func (f *FileService) StopMvFile(taskID string) error {
if cancel, ok := global.LoadTaskCancel(taskID); ok {
cancel()
return nil
}
return buserr.New("TaskNotFound")
}
func validateFileMove(fo files.FileOp, m request.FileMove) error {
if !fo.Stat(m.NewPath) { if !fo.Stat(m.NewPath) {
return buserr.New("ErrPathNotFound") return buserr.New("ErrPathNotFound")
} }
for _, oldPath := range m.OldPaths { for _, oldPath := range append(append([]string{}, m.OldPaths...), m.CoverPaths...) {
if !fo.Stat(oldPath) { if !fo.Stat(oldPath) {
return buserr.WithName("ErrFileNotFound", oldPath) return buserr.WithName("ErrFileNotFound", oldPath)
} }
if oldPath == m.NewPath || strings.Contains(m.NewPath, filepath.Clean(oldPath)+"/") { oldPath = filepath.Clean(oldPath)
newPath := filepath.Clean(m.NewPath)
if oldPath == newPath || strings.HasPrefix(newPath, oldPath+string(filepath.Separator)) {
return buserr.New("ErrMovePathFailed") return buserr.New("ErrMovePathFailed")
} }
} }
return nil
}
func (f *FileService) moveFileWithContext(ctx context.Context, m request.FileMove) error {
fo := files.NewFileOp()
type moveSnapshot struct { type moveSnapshot struct {
path string path string
content []byte content []byte
@@ -889,13 +1006,25 @@ func (f *FileService) MvFile(m request.FileMove) error {
} }
if len(m.CoverPaths) > 0 { if len(m.CoverPaths) > 0 {
for _, src := range m.CoverPaths { for _, src := range m.CoverPaths {
if err := fo.CopyAndReName(src, m.NewPath, "", true); err != nil { if err := ctx.Err(); err != nil {
return err
}
if err := fo.CopyAndReNameWithContext(ctx, src, m.NewPath, "", true); err != nil {
errs = append(errs, err) errs = append(errs, err)
global.LOG.Errorf("cut copy file [%s] to [%s] failed, err: %s", src, m.NewPath, err.Error()) global.LOG.Errorf("cut copy file [%s] to [%s] failed, err: %s", src, m.NewPath, err.Error())
continue
}
if err := ctx.Err(); err != nil {
return err
}
if err := fo.DeleteDir(src); err != nil {
removeErr := fmt.Errorf("remove merged source [%s] failed: %w", src, err)
errs = append(errs, removeErr)
global.LOG.Errorf("%s", removeErr.Error())
} }
} }
} }
if err := fo.Cut(m.OldPaths, m.NewPath, m.Name, m.Cover); err != nil { if err := fo.CutWithContext(ctx, m.OldPaths, m.NewPath, m.Name, m.Cover); err != nil {
return err return err
} }
for _, snapshot := range snapshots { for _, snapshot := range snapshots {
@@ -906,18 +1035,18 @@ func (f *FileService) MvFile(m request.FileMove) error {
} }
} }
} }
return nil return aggregateFileMoveErrors(errs)
} }
if m.Type == "copy" { if m.Type == "copy" {
for _, src := range m.OldPaths { for _, src := range m.OldPaths {
if err := fo.CopyAndReName(src, m.NewPath, m.Name, m.Cover); err != nil { if err := fo.CopyAndReNameWithContext(ctx, src, m.NewPath, m.Name, m.Cover); err != nil {
errs = append(errs, err) errs = append(errs, err)
global.LOG.Errorf("copy file [%s] to [%s] failed, err: %s", src, m.NewPath, err.Error()) global.LOG.Errorf("copy file [%s] to [%s] failed, err: %s", src, m.NewPath, err.Error())
} }
} }
if len(m.CoverPaths) > 0 { if len(m.CoverPaths) > 0 {
for _, src := range m.CoverPaths { for _, src := range m.CoverPaths {
if err := fo.CopyAndReName(src, m.NewPath, "", true); err != nil { if err := fo.CopyAndReNameWithContext(ctx, src, m.NewPath, "", true); err != nil {
errs = append(errs, err) errs = append(errs, err)
global.LOG.Errorf("copy file [%s] to [%s] failed, err: %s", src, m.NewPath, err.Error()) global.LOG.Errorf("copy file [%s] to [%s] failed, err: %s", src, m.NewPath, err.Error())
} }
@@ -925,14 +1054,7 @@ func (f *FileService) MvFile(m request.FileMove) error {
} }
} }
var errString string return aggregateFileMoveErrors(errs)
for _, err := range errs {
errString += err.Error() + "\n"
}
if errString != "" {
return errors.New(errString)
}
return nil
} }
func readEditableFileHistoryContent(filePath string) ([]byte, os.FileMode, bool) { func readEditableFileHistoryContent(filePath string) ([]byte, os.FileMode, bool) {
@@ -1188,7 +1310,7 @@ func (f *FileService) BatchCheckFiles(req request.FilePathsCheck) []response.Exi
} }
func (f *FileService) GetHostMount() []dto.DiskInfo { func (f *FileService) GetHostMount() []dto.DiskInfo {
return loadDiskInfo() return loadDiskInfo(false)
} }
func (f *FileService) GetUsersAndGroups() (*response.UserGroupResponse, error) { func (f *FileService) GetUsersAndGroups() (*response.UserGroupResponse, error) {
+77
View File
@@ -0,0 +1,77 @@
package service
import (
"errors"
"path/filepath"
"strings"
"sync"
"github.com/1Panel-dev/1Panel/agent/app/dto/request"
)
type fileTransferLockSet struct {
mu sync.Mutex
paths map[string][]string
}
func newFileTransferLocks() *fileTransferLockSet {
return &fileTransferLockSet{paths: make(map[string][]string)}
}
func (s *fileTransferLockSet) Acquire(taskID string, transferPaths []string) bool {
s.mu.Lock()
defer s.mu.Unlock()
for _, activePaths := range s.paths {
for _, activePath := range activePaths {
for _, transferPath := range transferPaths {
if fileTransferPathsOverlap(activePath, transferPath) {
return false
}
}
}
}
s.paths[taskID] = transferPaths
return true
}
func (s *fileTransferLockSet) Release(taskID string) {
s.mu.Lock()
defer s.mu.Unlock()
delete(s.paths, taskID)
}
func getFileTransferPaths(req request.FileMove) []string {
paths := make([]string, 0, 1+len(req.OldPaths)+len(req.CoverPaths))
paths = append(paths, req.NewPath)
paths = append(paths, req.OldPaths...)
paths = append(paths, req.CoverPaths...)
unique := make(map[string]struct{}, len(paths))
result := make([]string, 0, len(paths))
for _, item := range paths {
item = filepath.Clean(item)
if _, ok := unique[item]; ok {
continue
}
unique[item] = struct{}{}
result = append(result, item)
}
return result
}
func fileTransferPathsOverlap(first, second string) bool {
return first == second || strings.HasPrefix(first, second+string(filepath.Separator)) || strings.HasPrefix(second, first+string(filepath.Separator))
}
func aggregateFileMoveErrors(errs []error) error {
if len(errs) == 0 {
return nil
}
var errString strings.Builder
for _, err := range errs {
errString.WriteString(err.Error())
errString.WriteByte('\n')
}
return errors.New(errString.String())
}
File diff suppressed because it is too large Load Diff
+490
View File
@@ -0,0 +1,490 @@
package service
import (
"context"
"errors"
"fmt"
"os"
"slices"
"sort"
"strconv"
"strings"
"sync"
"github.com/1Panel-dev/1Panel/agent/app/dto"
"github.com/1Panel-dev/1Panel/agent/app/task"
"github.com/1Panel-dev/1Panel/agent/buserr"
"github.com/1Panel-dev/1Panel/agent/constant"
"github.com/1Panel-dev/1Panel/agent/i18n"
dockerfirewall "github.com/1Panel-dev/1Panel/agent/utils/firewall/docker_guard"
"github.com/1Panel-dev/1Panel/agent/utils/firewall/filter"
"github.com/docker/docker/client"
"github.com/google/uuid"
)
const (
dockerGuardComposeProjectLabel = "com.docker.compose.project"
dockerGuardComposeCreatedBy = "createdBy"
dockerTrafficPathForward = "forward"
dockerTrafficPathInput = "input"
dockerTrafficPathUnknown = "unknown"
dockerManagementContainerGuard = "container_guard"
dockerManagementHostFirewall = "host_firewall"
dockerManagementNeedsDiagnosis = "needs_diagnosis"
dockerReasonNATInspectFailed = "nat_inspect_failed"
dockerReasonNATChainUnreachable = "nat_chain_unreachable"
dockerReasonProxyInspectFailed = "proxy_inspect_failed"
dockerReasonNoMatchingPath = "no_matching_path"
)
type DockerPortGuardService struct {
runtime dockerfirewall.Runtime
runtimeForBackend func(context.Context, string) dockerfirewall.Runtime
client func() (*client.Client, error)
version func(string) string
}
var dockerPortGuardServiceMu sync.Mutex
type IDockerPortGuardService interface {
LoadOverview(context.Context) (dto.DockerPortGuardList, error)
ExportBackup(context.Context, filter.Provider) (dto.FirewallSubsystemBackup, error)
LoadPublishedPorts(context.Context) ([]dto.DockerPortGuardContainer, error)
Operate(context.Context, dto.DockerPortGuardOperation) error
QueueInitialization(dto.DockerPortGuardOperation) (dto.FilterChainOperationResponse, error)
DeletePolicies(dto.DockerPortGuardPolicyBatchDelete) (dto.FilterChainOperationResponse, error)
UpsertPolicies(dto.DockerPortGuardPolicyBatch) (dto.FilterChainOperationResponse, error)
Restore(context.Context) error
}
func NewIDockerPortGuardService() IDockerPortGuardService {
return newDockerPortGuardService()
}
func (s *DockerPortGuardService) LoadOverview(ctx context.Context) (dto.DockerPortGuardList, error) {
families, err := loadFirewallFamilies()
if err != nil {
return dto.DockerPortGuardList{}, err
}
backend := selectedDockerFirewallBackend("")
inventory, err := s.guardRuntime(ctx, backend).ListPolicies()
if err != nil {
return dto.DockerPortGuardList{}, err
}
policies := dockerGuardInventoryEndpoints(inventory)
unavailable := func() dto.DockerPortGuardList {
backend := selectedDockerFirewallBackend("")
base := s.runtimeStatus(s.guardRuntime(ctx, backend), backend, len(families) > 1)
base.Message = i18n.Get("ErrDockerFailed")
return dto.DockerPortGuardList{Base: base, Containers: []dto.DockerPortGuardContainer{}, OrphanPolicies: policies}
}
cli, err := s.client()
if err != nil {
return unavailable(), nil
}
defer cli.Close()
info, err := cli.Info(ctx)
if err != nil {
return unavailable(), nil
}
detectedBackend := dockerFirewallBackend(info)
backend = selectedDockerFirewallBackend(detectedBackend)
base := s.runtimeStatus(s.guardRuntime(ctx, backend), backend, len(families) > 1)
endpoints, err := discoverDockerEndpoints(ctx, cli, true)
if err != nil {
return dto.DockerPortGuardList{}, err
}
annotateDockerEndpointManagement(ctx, endpoints, detectedBackend)
endpoints, orphanPolicies := matchDockerGuardPolicies(base, policies, endpoints)
sort.Slice(endpoints, func(i, j int) bool {
return fmt.Sprintf("%s|%s|%d|%s", endpoints[i].Family, endpoints[i].HostIP, endpoints[i].HostPort, endpoints[i].Protocol) < fmt.Sprintf("%s|%s|%d|%s", endpoints[j].Family, endpoints[j].HostIP, endpoints[j].HostPort, endpoints[j].Protocol)
})
sort.Slice(orphanPolicies, func(i, j int) bool {
return fmt.Sprintf("%s|%s|%d|%s", orphanPolicies[i].Family, orphanPolicies[i].HostIP, orphanPolicies[i].HostPort, orphanPolicies[i].Protocol) < fmt.Sprintf("%s|%s|%d|%s", orphanPolicies[j].Family, orphanPolicies[j].HostIP, orphanPolicies[j].HostPort, orphanPolicies[j].Protocol)
})
return dto.DockerPortGuardList{Base: base, Containers: groupDockerGuardContainers(endpoints), OrphanPolicies: orphanPolicies}, nil
}
func (s *DockerPortGuardService) ExportBackup(ctx context.Context, provider filter.Provider) (dto.FirewallSubsystemBackup, error) {
backend := string(provider)
if backend == "" {
backend = selectedDockerFirewallBackend("")
}
if backend != constant.FirewallProviderIptables && backend != constant.FirewallProviderNftables {
return dto.FirewallSubsystemBackup{}, filter.ErrInvalidRule
}
inventory, err := s.guardRuntime(ctx, backend).ListPolicies()
if err != nil {
return dto.FirewallSubsystemBackup{}, err
}
if inventory.Policies == nil {
inventory.Policies = []dockerfirewall.Policy{}
}
return dto.FirewallSubsystemBackup{Subsystem: "docker", Provider: filter.Provider(backend), Docker: &inventory}, nil
}
func (s *DockerPortGuardService) LoadPublishedPorts(ctx context.Context) ([]dto.DockerPortGuardContainer, error) {
cli, err := s.client()
if err != nil {
return nil, buserr.WithDetail("ErrDockerFailed", err.Error(), err)
}
defer cli.Close()
if socketPath, local := strings.CutPrefix(cli.DaemonHost(), "unix://"); local {
if _, statErr := os.Stat(socketPath); errors.Is(statErr, os.ErrNotExist) {
return []dto.DockerPortGuardContainer{}, nil
}
}
endpoints, err := discoverDockerEndpoints(ctx, cli, false)
if err != nil {
return nil, err
}
backend := selectedDockerFirewallBackend("")
if info, infoErr := cli.Info(ctx); infoErr == nil {
backend = dockerFirewallBackend(info)
}
annotateDockerEndpointManagement(ctx, endpoints, backend)
return groupDockerGuardContainers(endpoints), nil
}
func (s *DockerPortGuardService) Operate(ctx context.Context, request dto.DockerPortGuardOperation) error {
dockerPortGuardServiceMu.Lock()
defer dockerPortGuardServiceMu.Unlock()
switch request.Operation {
case "initialize":
return s.initialize(ctx, request, nil)
case "bind":
runtime, _, err := s.runtimeForDocker(ctx)
if err != nil {
return err
}
families, err := loadFirewallFamilies()
if err != nil {
return err
}
if err := errors.Join(runtime.Bind(families...), ctx.Err()); err != nil {
return err
}
return settingRepo.UpdateOrCreate(constant.FirewallDockerPortGuardStatusKey, constant.StatusEnable)
case "unbind":
var err error
if s.runtime != nil {
err = s.runtime.Unbind()
} else {
err = errors.Join(dockerfirewall.NewIptables(ctx).Unbind(), dockerfirewall.NewNftables(ctx).Unbind())
}
if err = errors.Join(err, ctx.Err()); err != nil {
return err
}
return settingRepo.UpdateOrCreate(constant.FirewallDockerPortGuardStatusKey, constant.StatusDisable)
default:
return fmt.Errorf("unsupported Docker port guard operation: %s", request.Operation)
}
}
func (s *DockerPortGuardService) QueueInitialization(request dto.DockerPortGuardOperation) (dto.FilterChainOperationResponse, error) {
if err := task.CheckScopeTaskIsExecuting(task.TaskScopeFirewall, 0); err != nil {
return dto.FilterChainOperationResponse{}, err
}
if request.Operation != "initialize" {
return dto.FilterChainOperationResponse{}, filter.ErrInvalidRule
}
if request.BackupFile != "" {
if _, err := readFirewallSubsystemBackup(request.BackupFile, "docker"); err != nil {
return dto.FilterChainOperationResponse{}, err
}
}
return queueFirewallRuleTask(firewallTaskDocker, task.TaskExec, request.TaskID, []string{firewallTaskDocker}, func(t *task.Task) error {
dockerPortGuardServiceMu.Lock()
defer dockerPortGuardServiceMu.Unlock()
return s.initialize(t.TaskCtx, request, t)
})
}
func (s *DockerPortGuardService) DeletePolicies(request dto.DockerPortGuardPolicyBatchDelete) (dto.FilterChainOperationResponse, error) {
uuids, err := normalizeDockerFirewallUUIDs(request.UUIDs)
if err != nil {
return dto.FilterChainOperationResponse{}, err
}
return queueFirewallRuleTask(firewallTaskDocker, task.TaskDelete, "", uuids, func(t *task.Task) error {
ctx := t.TaskCtx
dockerPortGuardServiceMu.Lock()
defer dockerPortGuardServiceMu.Unlock()
runtime, backend, err := s.runtimeForDocker(ctx)
if err != nil {
return err
}
inventory, err := runtime.ListPolicies()
if err != nil {
return err
}
wanted := make(map[string]bool, len(uuids))
for _, id := range uuids {
wanted[id] = true
}
remaining := make([]dockerfirewall.Policy, 0, len(inventory.Policies))
for _, policy := range inventory.Policies {
if wanted[policy.UUID] {
delete(wanted, policy.UUID)
} else {
remaining = append(remaining, policy)
}
}
if len(wanted) > 0 {
return filter.ErrRuleStale
}
return applyDockerPolicies(ctx, runtime, backend, inventory, remaining)
})
}
func (s *DockerPortGuardService) UpsertPolicies(request dto.DockerPortGuardPolicyBatch) (dto.FilterChainOperationResponse, error) {
if len(request.Policies) > filter.MaxAtomicExpansion {
return dto.FilterChainOperationResponse{}, fmt.Errorf("create or import at most %d rules per batch (after expansion)", filter.MaxAtomicExpansion)
}
labels := make([]string, len(request.Policies))
policies := make([]dockerfirewall.Policy, 0, len(request.Policies))
endpoints := make([]dto.DockerPortGuardEndpointIdentity, 0, len(request.Policies))
count := 0
for i, policy := range request.Policies {
labels[i] = fmt.Sprintf("[%d/%d] %s %s %s:%d %s", i+1, len(request.Policies), policy.Family, policy.Protocol, policy.HostIP, policy.HostPort, policy.Mode)
normalized, err := normalizeDockerFirewallPolicy(dockerfirewall.Policy{
Family: policy.Family, HostIP: policy.HostIP, HostPort: policy.HostPort,
Protocol: policy.Protocol, Mode: policy.Mode, Sources: policy.Sources,
})
if err != nil {
return dto.FilterChainOperationResponse{}, fmt.Errorf("%s: %w", labels[i], err)
}
if normalized.Mode == dockerfirewall.ModeAll || normalized.Mode == dockerfirewall.ModeAcceptAll {
count++
} else {
count += len(normalized.Sources)
if normalized.Mode == dockerfirewall.ModeAllow {
count++
}
}
if count > filter.MaxAtomicExpansion {
return dto.FilterChainOperationResponse{}, fmt.Errorf("create or import at most %d rules per batch (after expansion)", filter.MaxAtomicExpansion)
}
normalized.UUID = uuid.NewString()
policies = append(policies, normalized)
}
return queueFirewallRuleTask(firewallTaskDocker, task.TaskUpdate, "", labels, func(t *task.Task) error {
ctx := t.TaskCtx
dockerPortGuardServiceMu.Lock()
defer dockerPortGuardServiceMu.Unlock()
if err := ctx.Err(); err != nil {
return err
}
runtime, backend, err := s.runtimeForDocker(ctx)
if err != nil {
return err
}
inventory, err := runtime.ListPolicies()
if err != nil {
return err
}
current := append([]dockerfirewall.Policy(nil), inventory.Policies...)
if request.Import {
backup := dto.FirewallSubsystemBackup{Provider: filter.Provider(backend), Docker: &dockerfirewall.PolicyInventory{Policies: policies}}
merged, err := mergeDockerBackup(inventory, backup, backend, t)
if err != nil {
return err
}
current = merged.Policies
}
byEndpoint := make(map[string]int, len(current))
for i, policy := range current {
byEndpoint[dockerPolicyEndpointKey(policy)] = i
}
for i := range policies {
key := dockerPolicyEndpointKey(policies[i])
index, exists := byEndpoint[key]
if request.Import {
if !exists || current[index].UUID != policies[i].UUID {
labels[i] = ""
continue
}
} else if exists {
policies[i].UUID = current[index].UUID
current[index] = policies[i]
} else {
byEndpoint[key] = len(current)
current = append(current, policies[i])
}
endpoints = append(endpoints, dto.DockerPortGuardEndpointIdentity{
Family: policies[i].Family, HostIP: policies[i].HostIP, HostPort: policies[i].HostPort, Protocol: policies[i].Protocol,
})
}
if len(endpoints) == 0 {
return nil
}
if err := s.rejectHostInputDockerGuardEndpoints(ctx, endpoints); err != nil {
return err
}
if err := applyDockerPolicies(ctx, runtime, backend, inventory, current); err != nil {
return err
}
return nil
})
}
func (s *DockerPortGuardService) Restore(ctx context.Context) error {
dockerPortGuardServiceMu.Lock()
defer dockerPortGuardServiceMu.Unlock()
enabled, err := dockerPortGuardPersistedEnabled()
if err != nil || !enabled {
return err
}
runtime, backend, err := s.runtimeForDocker(ctx)
if err != nil {
return err
}
backup, err := readFirewallSubsystemBackup("docker-"+backend+".rules", "docker")
if errors.Is(err, os.ErrNotExist) {
return nil
}
if err != nil {
return err
}
families, err := loadFirewallFamilies()
if err != nil {
return err
}
if !slices.Contains(families, constant.FirewallFamilyIPv6) {
before := len(backup.Docker.Policies)
backup.Docker.Policies = slices.DeleteFunc(backup.Docker.Policies, func(policy dockerfirewall.Policy) bool { return policy.Family == constant.FirewallFamilyIPv6 })
logFirewallIPv6Skipped(nil, "Docker startup", before-len(backup.Docker.Policies))
}
missing := make(map[string]bool)
needsBind := false
for _, family := range families {
status := runtime.Status(family)
if status.Reason == dockerfirewall.ReasonInspectFailed {
return fmt.Errorf("inspect Docker guard %s failed", family)
}
if status.Reason == dockerfirewall.ReasonCommandMissing {
continue
}
missing[family] = !status.Initialized
needsBind = needsBind || (status.Initialized && !status.Effective)
}
if !missing[dockerfirewall.FamilyIPv4] && !missing[dockerfirewall.FamilyIPv6] {
if needsBind {
return runtime.Bind(families...)
}
return nil
}
inventory, err := runtime.ListPolicies()
if err != nil {
return err
}
for _, policy := range inventory.Policies {
missing[policy.Family] = false
}
if inventory.RuleOrders == nil {
inventory.RuleOrders = make(map[string][]int64)
}
for _, policy := range backup.Docker.Policies {
if !missing[policy.Family] {
continue
}
inventory.Policies = append(inventory.Policies, policy)
key := policy.Family + "\x00" + policy.UUID
inventory.RuleOrders[key] = backup.Docker.RuleOrders[key]
}
return runtime.Initialize(inventory.Policies, inventory, families...)
}
func (s *DockerPortGuardService) runtimeStatus(runtime dockerfirewall.Runtime, backend string, ipv6Enabled bool) dto.DockerPortGuardBase {
ipv4 := runtime.Status(dockerfirewall.FamilyIPv4)
var ipv6 dockerfirewall.FamilyStatus
if ipv6Enabled {
ipv6 = runtime.Status(dockerfirewall.FamilyIPv6)
}
version := "-"
if s.version != nil {
version = s.version(backend)
}
name := "iptables-docker"
if strings.EqualFold(strings.TrimSpace(backend), constant.FirewallProviderNftables) {
name = "nftables-docker"
}
return dto.DockerPortGuardBase{
IPv6Enabled: ipv6Enabled,
Name: name,
Version: version,
Backend: backend,
IsExist: ipv4.Reason != dockerfirewall.ReasonCommandMissing || (ipv6Enabled && ipv6.Reason != dockerfirewall.ReasonCommandMissing),
Initialized: ipv4.Initialized || ipv6.Initialized,
Bound: ipv4.Bound || ipv6.Bound,
IPv4: dto.DockerPortGuardFamilyStatus{Partial: ipv4.Partial, State: ipv4.State, Reason: ipv4.Reason, Initialized: ipv4.Initialized, Bound: ipv4.Bound, Effective: ipv4.Effective},
IPv6: dto.DockerPortGuardFamilyStatus{Partial: ipv6.Partial, State: ipv6.State, Reason: ipv6.Reason, Initialized: ipv6.Initialized, Bound: ipv6.Bound, Effective: ipv6.Effective},
}
}
func matchDockerGuardPolicies(base dto.DockerPortGuardBase, policies []dto.DockerPortGuardEndpoint, endpoints []dto.DockerPortGuardEndpoint) ([]dto.DockerPortGuardEndpoint, []dto.DockerPortGuardEndpoint) {
matched := make(map[int]bool, len(policies))
byEndpoint := make(map[string]int, len(policies))
for index, policy := range policies {
key := strings.Join([]string{policy.Family, policy.HostIP, strconv.Itoa(int(policy.HostPort)), policy.Protocol}, "\x00")
if _, exists := byEndpoint[key]; !exists {
byEndpoint[key] = index
}
}
for i := range endpoints {
key := strings.Join([]string{endpoints[i].Family, endpoints[i].HostIP, strconv.Itoa(int(endpoints[i].HostPort)), endpoints[i].Protocol}, "\x00")
index, exists := byEndpoint[key]
if !exists {
continue
}
policy := policies[index]
endpoints[i].PolicyUUID, endpoints[i].Mode, endpoints[i].Sources = policy.PolicyUUID, policy.Mode, policy.Sources
endpoints[i].Effective = endpoints[i].ManagementTarget == dockerManagementContainerGuard && ((policy.Family == dockerfirewall.FamilyIPv4 && base.IPv4.Effective) || (policy.Family == dockerfirewall.FamilyIPv6 && base.IPv6.Effective))
matched[index] = true
}
orphans := make([]dto.DockerPortGuardEndpoint, 0)
for i, policy := range policies {
if !matched[i] {
orphans = append(orphans, policy)
}
}
return endpoints, orphans
}
func (s *DockerPortGuardService) rejectHostInputDockerGuardEndpoints(ctx context.Context, requested []dto.DockerPortGuardEndpointIdentity) error {
if s.client == nil || len(requested) == 0 {
return ctx.Err()
}
cli, err := s.client()
if err != nil {
return ctx.Err()
}
defer cli.Close()
info, err := cli.Info(ctx)
if err != nil {
return ctx.Err()
}
endpoints, err := discoverDockerEndpoints(ctx, cli, true)
if err != nil {
return ctx.Err()
}
annotateDockerEndpointManagement(ctx, endpoints, dockerFirewallBackend(info))
if err := ctx.Err(); err != nil {
return err
}
targets := make(map[string]string, len(endpoints))
for _, endpoint := range endpoints {
targets[fmt.Sprintf("%s|%s|%d|%s", endpoint.Family, endpoint.HostIP, endpoint.HostPort, endpoint.Protocol)] = endpoint.ManagementTarget
}
for _, endpoint := range requested {
target := targets[fmt.Sprintf("%s|%s|%d|%s", endpoint.Family, endpoint.HostIP, endpoint.HostPort, endpoint.Protocol)]
if target == dockerManagementHostFirewall {
return buserr.WithDetail("ErrInvalidParams", "endpoint traffic is handled by the host input firewall", nil)
}
if target == dockerManagementNeedsDiagnosis {
return buserr.WithDetail("ErrInvalidParams", "endpoint traffic management target requires diagnosis", nil)
}
}
return nil
}
@@ -1,78 +0,0 @@
package service
import (
"fmt"
"os"
"github.com/1Panel-dev/1Panel/agent/global"
"github.com/1Panel-dev/1Panel/agent/utils/controller"
)
const fail2BanRestoreWithFirewallMarker = "/run/1panel_fail2ban_restore_with_firewall"
type firewallFail2BanState struct {
markerPath string
isExist func(string) bool
isActive func(string) bool
start func(string) error
}
func newFirewallFail2BanState() *firewallFail2BanState {
return &firewallFail2BanState{
markerPath: fail2BanRestoreWithFirewallMarker,
isExist: func(serviceName string) bool {
exists, err := controller.CheckExist(serviceName)
if err != nil {
global.LOG.Warnf("check %s installation before stopping the firewall failed: %v", serviceName, err)
}
return exists
},
isActive: func(serviceName string) bool {
active, err := controller.CheckActive(serviceName)
if err != nil {
global.LOG.Warnf("check %s status before stopping the firewall failed: %v", serviceName, err)
}
return active
},
start: controller.HandleStart,
}
}
func (s *firewallFail2BanState) rememberBeforeFirewallStop() error {
if !s.isExist("fail2ban.service") {
return nil
}
if !s.isActive("fail2ban.service") {
return nil
}
return s.markForRestore()
}
func (s *firewallFail2BanState) markForRestore() error {
if err := os.WriteFile(s.markerPath, nil, 0600); err != nil {
return fmt.Errorf("mark Fail2Ban for restoration with the firewall: %w", err)
}
return nil
}
func (s *firewallFail2BanState) restoreAfterFirewallStart() error {
_, err := os.Stat(s.markerPath)
if err != nil {
if os.IsNotExist(err) {
return nil
}
return fmt.Errorf("load Fail2Ban restore marker after starting the firewall: %w", err)
}
if err := s.start("fail2ban.service"); err != nil {
return fmt.Errorf("restore Fail2Ban after starting the firewall: %w", err)
}
return s.clearRestoreMarker()
}
func (s *firewallFail2BanState) clearRestoreMarker() error {
if err := os.Remove(s.markerPath); err != nil && !os.IsNotExist(err) {
return fmt.Errorf("clear Fail2Ban firewall restore status: %w", err)
}
return nil
}
+649 -136
View File
@@ -1,175 +1,688 @@
package service package service
import ( import (
"context"
"encoding/json"
"errors"
"fmt" "fmt"
"strconv" "slices"
"strings" "strings"
"sync"
"github.com/1Panel-dev/1Panel/agent/app/dto"
"github.com/1Panel-dev/1Panel/agent/app/task"
"github.com/1Panel-dev/1Panel/agent/buserr"
"github.com/1Panel-dev/1Panel/agent/constant" "github.com/1Panel-dev/1Panel/agent/constant"
"github.com/1Panel-dev/1Panel/agent/global"
"github.com/1Panel-dev/1Panel/agent/utils/cmd"
"github.com/1Panel-dev/1Panel/agent/utils/firewall" "github.com/1Panel-dev/1Panel/agent/utils/firewall"
fireClient "github.com/1Panel-dev/1Panel/agent/utils/firewall/client" dockerfirewall "github.com/1Panel-dev/1Panel/agent/utils/firewall/docker_guard"
"github.com/1Panel-dev/1Panel/agent/utils/firewall/client/iptables" "github.com/1Panel-dev/1Panel/agent/utils/firewall/filter"
"github.com/1Panel-dev/1Panel/agent/utils/firewall/forwarding"
"github.com/1Panel-dev/1Panel/agent/utils/firewall/iptables_helper"
"github.com/1Panel-dev/1Panel/agent/utils/firewall/lifecycle"
"github.com/1Panel-dev/1Panel/agent/utils/firewall/nftables_helper"
"gorm.io/gorm"
) )
type firewallPortWhitelist struct { type IFirewallSettingService interface {
Port string CreatePortWhitelist(context.Context, dto.FirewallPortWhitelistCreate) error
Protocol string UpdatePortWhitelist(context.Context, dto.FirewallPortWhitelistUpdate) error
DeletePortWhitelist(context.Context, dto.FirewallPortWhitelistDelete) error
Load(context.Context) (dto.FirewallSettings, error)
Operate(context.Context, dto.FirewallBackendOperation) error
OperateFamily(dto.FirewallFamilyOperation) (dto.FilterChainOperationResponse, error)
OperateIPv6(dto.FirewallIPv6Operation) (dto.FilterChainOperationResponse, error)
} }
func loadConfiguredFirewallPortWhiteList() ([]firewallPortWhitelist, error) { type FirewallSettingService struct{}
value, err := settingRepo.GetValueByKey(constant.FirewallPortWhiteList)
if err != nil {
value = constant.FirewallPortWhiteListValue
if err := settingRepo.UpdateOrCreate(constant.FirewallPortWhiteList, value); err != nil {
return nil, err
}
}
return parseFirewallPortWhiteList(value)
}
func loadFirewallPortWhiteList() ([]firewallPortWhitelist, error) { var firewallWhitelistMu sync.Mutex
portWhiteList, err := loadConfiguredFirewallPortWhiteList()
if err != nil {
return nil, err
}
requiredPorts, err := loadRequiredFirewallPortWhiteList()
if err != nil {
return nil, err
}
return normalizeFirewallPortWhiteList(append(portWhiteList, requiredPorts...)), nil
}
func loadRequiredFirewallPortWhiteList() ([]firewallPortWhitelist, error) { func (s *FirewallSettingService) CreatePortWhitelist(ctx context.Context, request dto.FirewallPortWhitelistCreate) (result error) {
panelPort := LoadPanelPort() firewallWhitelistMu.Lock()
if panelPort == "" { firewallRuleMutationMu.Lock()
return nil, fmt.Errorf("find 1panel service port failed") defer func() {
} firewallRuleMutationMu.Unlock()
return normalizeFirewallPortWhiteList([]firewallPortWhitelist{ firewallWhitelistMu.Unlock()
{Port: panelPort, Protocol: "tcp"}, if result == nil {
{Port: loadSSHPort(), Protocol: "tcp"}, result = newFirewallService().SyncPortWhitelist(ctx)
}), nil
}
func parseFirewallPortWhiteList(value string) ([]firewallPortWhitelist, error) {
items := strings.FieldsFunc(value, func(r rune) bool {
return r == ',' || r == '\n' || r == ';' || r == ' '
})
ports := make([]firewallPortWhitelist, 0, len(items))
exists := make(map[string]struct{})
for _, item := range items {
item = strings.TrimSpace(item)
if item == "" {
continue
} }
port, protocol, ok := strings.Cut(item, "/") }()
if !ok { current, err := loadFirewallPortWhiteList()
protocol = "tcp"
}
port = strings.TrimSpace(port)
protocol = strings.ToLower(strings.TrimSpace(protocol))
if protocol != "tcp" && protocol != "udp" {
return nil, fmt.Errorf("invalid firewall port whitelist protocol: %s", item)
}
portNum, err := strconv.Atoi(port)
if err != nil || portNum < 1 || portNum > 65535 {
return nil, fmt.Errorf("invalid firewall port whitelist: %s", item)
}
key := fmt.Sprintf("%d/%s", portNum, protocol)
if _, ok := exists[key]; ok {
continue
}
exists[key] = struct{}{}
ports = append(ports, firewallPortWhitelist{Port: strconv.Itoa(portNum), Protocol: protocol})
}
return ports, nil
}
func normalizeFirewallPortWhiteList(portWhiteList []firewallPortWhitelist) []firewallPortWhitelist {
ports := make([]firewallPortWhitelist, 0, len(portWhiteList))
exists := make(map[string]struct{})
for _, item := range portWhiteList {
if item.Port == "" {
continue
}
key := fmt.Sprintf("%s/%s", item.Port, item.Protocol)
if _, ok := exists[key]; ok {
continue
}
exists[key] = struct{}{}
ports = append(ports, item)
}
return ports
}
func syncFirewallPortWhiteListAfterUpdate(oldValue string) error {
client, err := firewall.NewFirewallClient()
if err != nil { if err != nil {
return err return err
} }
if client.Name() == "iptables" { current = append(current, request.Rule)
isInit, _ := iptables.LoadInitStatus("iptables", "base") current, err = firewall.ValidatePortWhitelist(current)
if !isInit { if err != nil {
return nil return err
}
if err := validateFirewallWhitelistFamilies(current); err != nil {
return err
}
value, err := json.Marshal(current)
if err != nil {
return err
}
return settingRepo.UpdateOrCreate(constant.FirewallPortWhiteList, string(value))
}
func (s *FirewallSettingService) UpdatePortWhitelist(ctx context.Context, request dto.FirewallPortWhitelistUpdate) (result error) {
firewallWhitelistMu.Lock()
firewallRuleMutationMu.Lock()
defer func() {
firewallRuleMutationMu.Unlock()
firewallWhitelistMu.Unlock()
if result == nil {
result = newFirewallService().SyncPortWhitelist(ctx)
} }
oldPortWhiteList, err := parseFirewallPortWhiteList(oldValue) }()
current, err := loadFirewallPortWhiteList()
if err != nil {
return err
}
index, err := findPortWhitelistRule(current, request.OldRule)
if err != nil {
return err
}
current[index] = request.Rule
current, err = firewall.ValidatePortWhitelist(current)
if err != nil {
return err
}
if err := validateFirewallWhitelistFamilies(current); err != nil {
return err
}
value, err := json.Marshal(current)
if err != nil {
return err
}
return settingRepo.UpdateOrCreate(constant.FirewallPortWhiteList, string(value))
}
func (s *FirewallSettingService) DeletePortWhitelist(ctx context.Context, request dto.FirewallPortWhitelistDelete) error {
firewallWhitelistMu.Lock()
firewallRuleMutationMu.Lock()
defer func() {
firewallRuleMutationMu.Unlock()
firewallWhitelistMu.Unlock()
}()
current, err := loadFirewallPortWhiteList()
if err != nil {
return err
}
if request.Rule == nil {
return filter.ErrInvalidRule
}
index, err := findPortWhitelistRule(current, *request.Rule)
if err != nil {
return err
}
current = slices.Delete(current, index, index+1)
current, err = firewall.ValidatePortWhitelist(current)
if err != nil {
return err
}
if err := validateFirewallWhitelistFamilies(current); err != nil {
return err
}
value, err := json.Marshal(current)
if err != nil {
return err
}
return settingRepo.UpdateOrCreate(constant.FirewallPortWhiteList, string(value))
}
func (s *FirewallSettingService) Load(ctx context.Context) (dto.FirewallSettings, error) {
families, err := loadFirewallFamilies()
if err != nil {
return dto.FirewallSettings{}, err
}
result := dto.FirewallSettings{PingStatus: firewall.LoadPingStatus(), IPv6Enabled: slices.Contains(families, constant.FirewallFamilyIPv6)}
installed := make(map[string]bool)
for _, name := range lifecycle.InstalledProviders() {
installed[name] = true
}
systemBackend, _ := settingRepo.GetValueByKey(constant.FirewallSystemBackendKey)
result.System.Selected = strings.TrimSpace(systemBackend)
if result.System.Selected == "" {
if client, err := lifecycle.NewClient(""); err == nil {
result.System.Selected = client.Name()
}
}
result.System.Current = result.System.Selected
for _, name := range []string{
constant.FirewallProviderFirewalld,
constant.FirewallProviderUFW,
constant.FirewallProviderIptables,
constant.FirewallProviderNftables,
} {
option := dto.FirewallBackendOption{Name: name, Installed: installed[name], Supported: true}
if option.Installed && name == result.System.Selected {
client, err := lifecycle.NewClient(name)
if err != nil {
option.Message = err.Error()
} else if name == constant.FirewallProviderIptables || name == constant.FirewallProviderNftables {
overview, err := loadSystemFirewallOverview(name, "base", families)
if err != nil {
option.Message = err.Error()
}
option.Initialized, option.Bound = overview.IsInit, overview.IsBind
option.IPv4, option.IPv6 = overview.IPv4, overview.IPv6
} else if option.Active, err = client.Status(); err != nil {
option.Message = err.Error()
}
}
if name == result.System.Selected && name == constant.FirewallProviderIptables {
if commands, err := lifecycle.ResolveIptablesCommands(); err == nil {
option.Implementation = commands.IPv4
}
}
result.System.Options = append(result.System.Options, option)
}
forwardingBackend, _ := settingRepo.GetValueByKey(constant.FirewallForwardingBackendKey)
result.Forwarding.Selected = strings.TrimSpace(forwardingBackend)
if result.Forwarding.Selected == "" {
result.Forwarding.Selected = constant.FirewallProviderIptables
}
result.Forwarding.Current = result.Forwarding.Selected
for _, name := range []string{constant.FirewallProviderIptables, constant.FirewallProviderNftables} {
option := dto.FirewallBackendOption{Name: name, Installed: installed[name], Supported: true}
if option.Installed && name == result.Forwarding.Selected {
manager, err := newForwardingAdapterFor(ctx, name)
if err != nil {
option.Message = err.Error()
} else {
status, statusErr := loadForwardingFirewallOverview(manager, families)
option.IPv4, option.IPv6 = status.IPv4, status.IPv6
if statusErr != nil {
option.Message = statusErr.Error()
} else {
option.Initialized, option.Bound = status.IsInit, status.IsBind
}
if result.IPv6Enabled && name == constant.FirewallProviderIptables && !option.IPv6.Available {
if commands, err := lifecycle.ResolveIptablesCommands(); err == nil && !commands.IPv6Available() {
option.IPv6.Reason = dockerfirewall.ReasonCommandMissing
}
}
}
}
if name == result.Forwarding.Selected && name == constant.FirewallProviderIptables {
if commands, err := lifecycle.ResolveIptablesCommands(); err == nil {
option.Implementation = commands.IPv4
}
}
result.Forwarding.Options = append(result.Forwarding.Options, option)
}
dockerInstalled := cmd.Which("docker")
dockerVersion := ""
if dockerInstalled {
dockerVersion = loadDockerEngineVersion(ctx)
}
dockerBackend, _ := settingRepo.GetValueByKey(constant.FirewallDockerBackendKey)
dockerBackend = strings.ToLower(strings.TrimSpace(dockerBackend))
if dockerBackend == constant.FirewallProviderIptables || dockerBackend == constant.FirewallProviderNftables {
result.Docker.Selected = dockerBackend
}
result.Docker.Current = result.Docker.Selected
for _, name := range []string{constant.FirewallProviderIptables, constant.FirewallProviderNftables} {
option := dto.FirewallBackendOption{
Name: name, Installed: installed[name], Supported: dockerInstalled,
Active: dockerInstalled && installed[name] && result.Docker.Selected == name,
}
if name == constant.FirewallProviderNftables && dockerInstalled && !dockerNftablesSupported(dockerVersion) {
option.Supported = false
option.SupportReason = "docker_version_unsupported"
option.Active = false
}
if option.Active {
guard := newDockerFirewallRuntime(ctx, name)
for _, family := range families {
status := guard.Status(family)
option.Initialized = option.Initialized || status.Initialized
option.Bound = option.Bound || status.Bound
info := dto.FirewallBackendFamilyStatus{
Available: status.Reason != dockerfirewall.ReasonCommandMissing,
Initialized: status.Initialized, Bound: status.Bound, Reason: status.Reason,
}
if family == constant.FirewallFamilyIPv4 {
option.IPv4 = info
} else {
option.IPv6 = info
}
}
}
result.Docker.Options = append(result.Docker.Options, option)
}
result.PortWhitelist, err = loadPortWhitelistSetting()
if err != nil {
return result, err
}
result.PanelPort = LoadPanelPort()
sshPort, sshErr := loadSSHWhitelistPortFrom(sshPath)
if sshErr != nil {
global.LOG.Warnf("load SSH port for firewall settings: %v", sshErr)
} else {
result.SSHPort = sshPort
}
return result, err
}
func (s *FirewallSettingService) Operate(ctx context.Context, request dto.FirewallBackendOperation) error {
if request.Operation == "cleanup" {
_, err := newFirewallService().Reset(ctx, dto.FirewallRuleReset{Subsystem: request.Subsystem, Provider: filter.Provider(request.Backend)})
return err
}
if err := lockFirewallLifecycleIdle(); err != nil {
return err
}
defer firewallLifecycleTaskMu.Unlock()
if request.Subsystem != "system" && request.Backend != constant.FirewallProviderIptables && request.Backend != constant.FirewallProviderNftables {
return fmt.Errorf("%s only supports iptables or nftables", request.Subsystem)
}
if request.Subsystem == "system" && (request.Backend != constant.FirewallProviderIptables && request.Backend != constant.FirewallProviderNftables) && request.Operation != "select" {
return fmt.Errorf("%s does not support initialization or cleanup", request.Backend)
}
switch request.Subsystem {
case "system":
if err := s.operateSystem(request); err != nil {
return err
}
if request.Operation == "initialize" {
service := newFirewallService()
whitelistErr := service.SyncPortWhitelist(ctx)
return whitelistErr
}
return nil
case "forwarding":
return s.operateForwarding(ctx, request)
case "docker":
return s.operateDocker(ctx, request)
default:
return fmt.Errorf("unsupported firewall subsystem %q", request.Subsystem)
}
}
func (s *FirewallSettingService) OperateFamily(request dto.FirewallFamilyOperation) (dto.FilterChainOperationResponse, error) {
if request.Family != constant.FirewallFamilyIPv4 && request.Family != constant.FirewallFamilyIPv6 {
return dto.FilterChainOperationResponse{}, filter.ErrInvalidScope
}
if request.Backend != constant.FirewallProviderIptables && request.Backend != constant.FirewallProviderNftables {
return dto.FilterChainOperationResponse{}, filter.ErrUnsupportedScope
}
if request.Operation != "initialize" && request.Operation != "repair" && request.Operation != "bind" {
return dto.FilterChainOperationResponse{}, filter.ErrRuleOperation
}
subsystem := ""
switch request.Subsystem {
case "system":
subsystem = firewallTaskHost
case "forwarding":
subsystem = firewallTaskForwarding
case "docker":
subsystem = firewallTaskDocker
default:
return dto.FilterChainOperationResponse{}, filter.ErrInvalidScope
}
if err := task.CheckScopeTaskIsExecuting(task.TaskScopeFirewall, 0); err != nil {
return dto.FilterChainOperationResponse{}, err
}
return queueFirewallRuleTask(subsystem, task.TaskExec, "", nil, func(t *task.Task) error {
if err := lockFirewallLifecycleIdle(); err != nil {
return err
}
defer firewallLifecycleTaskMu.Unlock()
t.Logf("backend=%s family=%s operation=%s", request.Backend, request.Family, request.Operation)
families, err := loadFirewallFamilies()
if err != nil { if err != nil {
return err return err
} }
return syncIptablesFirewallPortWhiteList(true, oldPortWhiteList) if !slices.Contains(families, request.Family) {
} return fmt.Errorf("IPv6 firewall support is disabled")
}
initialize := request.Operation != "bind"
switch request.Subsystem {
case "system":
firewallWhitelistMu.Lock()
defer firewallWhitelistMu.Unlock()
if err := newFirewallService().checkSelectedProvider(t.TaskCtx, filter.Provider(request.Backend)); err != nil {
return err
}
ports, err := loadFirewallPortWhiteList()
if err != nil {
return err
}
required, err := firewall.RequiredPortWhitelist(ports)
if err != nil {
return err
}
firewallRuleMutationMu.Lock()
if request.Backend == constant.FirewallProviderIptables {
err = iptables_helper.OperateFamily(request.Family, initialize, required)
} else {
err = nftables_helper.OperateFamily(filter.Family(request.Family), initialize, required)
}
firewallRuleMutationMu.Unlock()
if err != nil {
return err
}
if initialize {
if err := newFirewallService().applyPortWhitelist(t.TaskCtx, ports, nil, filter.Family(request.Family)); err != nil {
return err
}
}
return settingRepo.UpdateOrCreate("IptablesStatus", constant.StatusEnable)
case "forwarding":
forwardingMutationMu.Lock()
defer forwardingMutationMu.Unlock()
manager, err := newForwardingAdapter(t.TaskCtx)
if err != nil {
return err
}
if manager.Name() != request.Backend {
return filter.ErrProviderUnavailable
}
if err := manager.OperateFamily(request.Family, initialize); err != nil {
return err
}
rules, err := manager.List()
if err != nil {
return err
}
if err := persistForwardingRules(manager, rules); err != nil {
return err
}
return settingRepo.UpdateOrCreate(constant.FirewallForwardingInitializedKey, constant.StatusEnable)
default:
dockerPortGuardServiceMu.Lock()
defer dockerPortGuardServiceMu.Unlock()
runtime, backend, err := newDockerPortGuardService().runtimeForDocker(t.TaskCtx)
if err != nil {
return err
}
if backend != request.Backend {
return filter.ErrProviderUnavailable
}
if err := runtime.OperateFamily(request.Family, initialize); err != nil {
return err
}
inventory, err := runtime.ListPolicies()
if err != nil {
return err
}
if err := persistDockerRules(backend, inventory); err != nil {
return err
}
return settingRepo.UpdateOrCreate(constant.FirewallDockerPortGuardStatusKey, constant.StatusEnable)
}
})
}
isActive, _ := client.Status() func (s *FirewallSettingService) OperateIPv6(request dto.FirewallIPv6Operation) (dto.FilterChainOperationResponse, error) {
if !isActive { if request.Status != constant.StatusEnable && request.Status != constant.StatusDisable {
return dto.FilterChainOperationResponse{}, filter.ErrInvalidRule
}
if err := task.CheckScopeTaskIsExecuting(task.TaskScopeFirewall, 0); err != nil {
return dto.FilterChainOperationResponse{}, err
}
return queueFirewallRuleTask(firewallTaskHost, task.TaskExec, "", nil, func(t *task.Task) error {
if err := lockFirewallLifecycleIdle(); err != nil {
return err
}
defer firewallLifecycleTaskMu.Unlock()
firewallWhitelistMu.Lock()
defer firewallWhitelistMu.Unlock()
firewallRuleMutationMu.Lock()
defer firewallRuleMutationMu.Unlock()
forwardingMutationMu.Lock()
defer forwardingMutationMu.Unlock()
dockerPortGuardServiceMu.Lock()
defer dockerPortGuardServiceMu.Unlock()
if request.Status == constant.StatusEnable {
return settingRepo.UpdateOrCreate(constant.FirewallIPv6SupportKey, request.Status)
}
ports, err := loadFirewallPortWhiteList()
if err != nil {
return err
}
value, err := json.Marshal(ipv4PortWhitelist(ports))
if err != nil {
return err
}
installed := lifecycle.InstalledProviders()
for _, selection := range []struct{ subsystem, key string }{
{"system", constant.FirewallSystemBackendKey},
{"forwarding", constant.FirewallForwardingBackendKey},
{"docker", constant.FirewallDockerBackendKey},
} {
backend, err := settingRepo.GetValueByKey(selection.key)
if err != nil && !errors.Is(err, gorm.ErrRecordNotFound) {
return err
}
backend = strings.ToLower(strings.TrimSpace(backend))
if backend == "" {
if selection.subsystem == "system" {
if len(installed) == 0 {
continue
}
client, err := NewSelectedSystemFirewallClient()
if err != nil {
return err
}
backend = client.Name()
} else {
backend = constant.FirewallProviderIptables
}
}
if !slices.Contains(installed, backend) || (backend != constant.FirewallProviderIptables && backend != constant.FirewallProviderNftables) {
continue
}
t.Logf("disable IPv6 firewall bindings: subsystem=%s backend=%s", selection.subsystem, backend)
switch selection.subsystem {
case "system":
if backend == constant.FirewallProviderIptables {
err = iptables_helper.UnbindIPv6BaseChains()
} else {
err = nftables_helper.SetTableDormant(t.TaskCtx, "ip6", nftables_helper.TableName)
if err == nil {
err = nftables_helper.PersistRuleset(t.TaskCtx)
}
}
case "forwarding":
var manager forwarding.Adapter
manager, err = newForwardingAdapterFor(t.TaskCtx, backend)
if err == nil {
err = manager.UnbindFamily(constant.FirewallFamilyIPv6)
}
case "docker":
err = newDockerFirewallRuntime(t.TaskCtx, backend).Unbind(constant.FirewallFamilyIPv6)
}
if err != nil && !errors.Is(err, filter.ErrFamilyUnavailable) {
return err
}
}
if err := t.TaskCtx.Err(); err != nil {
return err
}
return settingRepo.UpdateValues(map[string]string{constant.FirewallIPv6SupportKey: request.Status, constant.FirewallPortWhiteList: string(value)})
})
}
func NewIFirewallSettingService() IFirewallSettingService {
return &FirewallSettingService{}
}
func (s *FirewallSettingService) operateSystem(request dto.FirewallBackendOperation) error {
firewallRuleMutationMu.Lock()
defer firewallRuleMutationMu.Unlock()
if _, err := lifecycle.NewClient(request.Backend); err != nil {
return err
}
previous, _ := settingRepo.GetValueByKey(constant.FirewallSystemBackendKey)
if previous == "" {
if client, err := lifecycle.NewClient(""); err == nil {
previous = client.Name()
}
}
if request.Operation == "select" && previous != "" && previous != request.Backend {
initialized, err := systemFirewallBackendInitialized(previous)
if err != nil {
return err
}
if initialized {
return buserr.WithMap("ErrFirewallBackendCleanupRequired", map[string]interface{}{"current": previous, "target": request.Backend}, nil)
}
}
if err := settingRepo.UpdateOrCreate(constant.FirewallSystemBackendKey, request.Backend); err != nil {
return err
}
rollback := func(err error) error {
if err == nil {
return nil
}
_ = settingRepo.UpdateOrCreate(constant.FirewallSystemBackendKey, previous)
return err
}
if request.Operation == "select" {
return nil return nil
} }
portWhiteList, err := loadFirewallPortWhiteList() initErr := newFirewallService().operateFilterChainBaseLocked(request.Backend, dto.FilterChainOperation{
if err != nil { Name: constant.FirewallBasicChain, Operate: string(firewall.BaseOperationInit),
return err })
if initErr != nil {
return rollback(initErr)
} }
oldPortWhiteList, err := parseFirewallPortWhiteList(oldValue) return settingRepo.UpdateOrCreate(constant.FirewallFilterInitializedKey, constant.StatusEnable)
if err != nil {
return err
}
requiredPorts, err := loadRequiredFirewallPortWhiteList()
if err != nil {
return err
}
oldPortWhiteList = normalizeFirewallPortWhiteList(append(oldPortWhiteList, requiredPorts...))
return syncFirewallClientPortWhiteList(client, oldPortWhiteList, portWhiteList)
} }
func syncFirewallClientPortWhiteList(client firewall.FirewallClient, oldPortWhiteList, portWhiteList []firewallPortWhitelist) error { func systemFirewallBackendInitialized(backend string) (bool, error) {
oldPorts := firewallPortWhiteListMap(oldPortWhiteList) client, err := lifecycle.NewClient(backend)
newPorts := firewallPortWhiteListMap(portWhiteList) if err != nil {
for _, item := range oldPortWhiteList { if errors.Is(err, lifecycle.ErrNotInstalled) {
key := firewallPortWhiteListKey(item) return false, nil
if _, ok := newPorts[key]; ok {
continue
} }
if err := client.Port(fireClient.FireInfo{Port: item.Port, Protocol: item.Protocol, Strategy: "accept"}, "remove"); err != nil { return false, err
}
if backend == constant.FirewallProviderIptables || backend == constant.FirewallProviderNftables {
for _, family := range []string{constant.FirewallFamilyIPv4, constant.FirewallFamilyIPv6} {
initialized, _, err := loadSystemFirewallFamilyStatus(backend, family)
if family == constant.FirewallFamilyIPv6 && errors.Is(err, filter.ErrFamilyUnavailable) {
continue
}
if err != nil {
return false, err
}
if initialized {
return true, nil
}
}
return false, nil
}
return client.Status()
}
func (s *FirewallSettingService) operateForwarding(ctx context.Context, request dto.FirewallBackendOperation) error {
if _, err := newForwardingAdapterFor(ctx, request.Backend); err != nil {
return err
}
previous, _ := settingRepo.GetValueByKey(constant.FirewallForwardingBackendKey)
if request.Operation == "select" {
current := previous
if current == "" {
detected, err := newForwardingAdapter(ctx)
if err != nil {
return err
}
current = detected.Name()
}
initialized, err := forwardingBackendInitialized(ctx, current)
if err != nil {
return err
}
if current != request.Backend && initialized {
return buserr.WithMap("ErrFirewallBackendCleanupRequired", map[string]interface{}{"current": current, "target": request.Backend}, nil)
}
}
if err := settingRepo.UpdateOrCreate(constant.FirewallForwardingBackendKey, request.Backend); err != nil {
return err
}
if request.Operation == "initialize" {
return newForwardingService().Enable(ctx)
}
return nil
}
func forwardingBackendInitialized(ctx context.Context, backend string) (bool, error) {
manager, err := newForwardingAdapterFor(ctx, backend)
if err != nil {
if errors.Is(err, lifecycle.ErrNotInstalled) {
return false, nil
}
return false, err
}
for _, family := range []string{constant.FirewallFamilyIPv4, constant.FirewallFamilyIPv6} {
initialized, _, err := manager.FamilyStatus(family)
if err != nil {
return false, err
}
if initialized {
return true, nil
}
}
return false, nil
}
func (s *FirewallSettingService) operateDocker(ctx context.Context, request dto.FirewallBackendOperation) error {
previous, _ := settingRepo.GetValueByKey(constant.FirewallDockerBackendKey)
if request.Operation == "select" {
current := previous
if current == "" {
current = constant.FirewallProviderNftables
if request.Backend == constant.FirewallProviderNftables {
current = constant.FirewallProviderIptables
}
}
initialized, err := dockerGuardBackendInitialized(ctx, current)
if err != nil {
return err
}
if current != request.Backend && initialized {
return buserr.WithMap("ErrFirewallBackendCleanupRequired", map[string]interface{}{"current": current, "target": request.Backend}, nil)
}
}
if err := settingRepo.UpdateOrCreate(constant.FirewallDockerBackendKey, request.Backend); err != nil {
return err
}
if request.Operation == "select" {
if err := (&DockerService{}).UpdateFirewallBackend(request.Backend); err != nil {
_ = settingRepo.UpdateOrCreate(constant.FirewallDockerBackendKey, previous)
return err return err
} }
} }
for _, item := range portWhiteList { if request.Operation == "initialize" {
key := firewallPortWhiteListKey(item) if err := newDockerPortGuardService().Operate(ctx, dto.DockerPortGuardOperation{Operation: "initialize"}); err != nil {
if _, ok := oldPorts[key]; ok { _ = settingRepo.UpdateOrCreate(constant.FirewallDockerBackendKey, previous)
continue
}
if err := client.Port(fireClient.FireInfo{Port: item.Port, Protocol: item.Protocol, Strategy: "accept"}, "add"); err != nil {
return err return err
} }
} }
return client.Reload() return nil
} }
func firewallPortWhiteListMap(portWhiteList []firewallPortWhitelist) map[string]struct{} { func dockerGuardBackendInitialized(ctx context.Context, backend string) (bool, error) {
ports := make(map[string]struct{}) guard := newDockerFirewallRuntime(ctx, backend)
for _, item := range portWhiteList { for _, family := range []string{dockerfirewall.FamilyIPv4, dockerfirewall.FamilyIPv6} {
ports[firewallPortWhiteListKey(item)] = struct{}{} initialized, err := guard.Initialized(family)
if err != nil {
return false, err
}
if initialized {
return true, nil
}
} }
return ports return false, nil
}
func firewallPortWhiteListKey(item firewallPortWhitelist) string {
return item.Port + "/" + item.Protocol
} }
File diff suppressed because it is too large Load Diff

Some files were not shown because too many files have changed in this diff Show More