feat: background tools for ai agent steps (#11827)

* feat(ai-agent): background tools that wake the agent when they finish

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* feat(ai-agent): run a tool in the background after a delay

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* refactor(ai-agent): park and background tool queries as sqlx macros

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* feat(flow-chat): show background tool calls as one running card, no typing dots while waiting

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* feat(flow-chat): say when a call moved to the background and when results came back

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(flow-chat): one card per background call, without the model's note as its result

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01F4rzFpemqD4pqLnAp4qRqF

* fix(flow-chat): a background result streamed in early still folds its row into the call's card

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01F4rzFpemqD4pqLnAp4qRqF

* fix(flow-chat): a stopped background call stays finished in later turns, and a late row read keeps a streamed result

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01F4rzFpemqD4pqLnAp4qRqF

* fix(chat-sdk): a late background row keeps a streamed failure failed

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01F4rzFpemqD4pqLnAp4qRqF

* chore: restore sqlx cache entries the main merge dropped

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01F4rzFpemqD4pqLnAp4qRqF

* fix(chat-sdk): a background result row stays with its own call when the tool is called again; drop stale sqlx entries

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01F4rzFpemqD4pqLnAp4qRqF

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
hugocasa
2026-11-13 13:12:29 +01:00
committed by GitHub
co-authored by Claude Opus 5.5
parent 59a8657426
commit 10a8577fab
50 changed files with 1724 additions and 174 deletions
@@ -0,0 +1,23 @@
{
"db_name": "PostgreSQL",
"query": "SELECT id FROM resume_job WHERE job = $1 AND id <> ALL($2)",
"describe": {
"columns": [
{
"ordinal": 0,
"name": "id",
"type_info": "Uuid"
}
],
"parameters": {
"Left": [
"Uuid",
"UuidArray"
]
},
"nullable": [
false
]
},
"hash": "0177ee2ab01adcfe59cc879f1e7f8a8cfb42ad5ebc71e649b17ab39822a08f55"
}
@@ -0,0 +1,17 @@
{
"db_name": "PostgreSQL",
"query": "UPDATE v2_job_status SET\n flow_status = jsonb_set(\n flow_status,\n array['modules', $2::int::TEXT, 'agent_actions_success'],\n COALESCE((\n SELECT jsonb_agg(e ORDER BY i)\n FROM jsonb_array_elements(\n flow_status->'modules'->$2::int->'agent_actions_success'\n ) WITH ORDINALITY AS t(e, i)\n WHERE i <= $3\n ), '[]'::jsonb) || CASE WHEN $4::bool IS NULL THEN '[]'::jsonb\n ELSE to_jsonb(ARRAY[$4::bool]) END\n )\n WHERE id = $1",
"describe": {
"columns": [],
"parameters": {
"Left": [
"Uuid",
"Int4",
"Int8",
"Bool"
]
},
"nullable": []
},
"hash": "38755d1d28e0d6db5f9dd89f14cb7f6a74b1125069684aa22aafd9c7b4878714"
}
@@ -0,0 +1,41 @@
{
"db_name": "PostgreSQL",
"query": "SELECT id, value AS \"value: sqlx::types::Json<serde_json::Value>\", approver, approved\n FROM resume_job WHERE job = $1 AND id <> ALL($2) ORDER BY created_at ASC",
"describe": {
"columns": [
{
"ordinal": 0,
"name": "id",
"type_info": "Uuid"
},
{
"ordinal": 1,
"name": "value: sqlx::types::Json<serde_json::Value>",
"type_info": "Jsonb"
},
{
"ordinal": 2,
"name": "approver",
"type_info": "Varchar"
},
{
"ordinal": 3,
"name": "approved",
"type_info": "Bool"
}
],
"parameters": {
"Left": [
"Uuid",
"UuidArray"
]
},
"nullable": [
false,
false,
true,
false
]
},
"hash": "6204479b0fdb9492aaa4fed3bb128153cb03925398c43ce5c2dbe111399a34c4"
}
@@ -0,0 +1,22 @@
{
"db_name": "PostgreSQL",
"query": "SELECT checkpoint AS \"checkpoint: sqlx::types::Json<Box<serde_json::value::RawValue>>\"\n FROM ai_agent_checkpoint WHERE job_id = $1",
"describe": {
"columns": [
{
"ordinal": 0,
"name": "checkpoint: sqlx::types::Json<Box<serde_json::value::RawValue>>",
"type_info": "Jsonb"
}
],
"parameters": {
"Left": [
"Uuid"
]
},
"nullable": [
false
]
},
"hash": "71b94ff615d2c5bf4f5d2ec8d02dd96732c9960c0bd654b4fd1d6322e5ba3343"
}
@@ -0,0 +1,17 @@
{
"db_name": "PostgreSQL",
"query": "UPDATE v2_job_status SET\n flow_status = jsonb_set(\n flow_status,\n array['modules', $2::int::TEXT, 'agent_actions_success'],\n (\n SELECT jsonb_agg(CASE WHEN i = $3 THEN to_jsonb($4::bool) ELSE arr->i END ORDER BY i)\n FROM (SELECT COALESCE(\n flow_status->'modules'->$2::int->'agent_actions_success', '[]'::jsonb\n ) AS arr) a,\n generate_series(0, GREATEST(jsonb_array_length(arr) - 1, $3)) AS i\n )\n )\n WHERE id = $1",
"describe": {
"columns": [],
"parameters": {
"Left": [
"Uuid",
"Int4",
"Int4",
"Bool"
]
},
"nullable": []
},
"hash": "76f0d44dc51f391d7563c2a5a1ff674a0d953d4a4709ab21f6651f0438c351d5"
}
@@ -0,0 +1,35 @@
{
"db_name": "PostgreSQL",
"query": "SELECT id, status = 'success' AS \"success!\", result AS \"result: Json<Box<RawValue>>\"\n FROM v2_job_completed WHERE workspace_id = $1 AND id = ANY($2)",
"describe": {
"columns": [
{
"ordinal": 0,
"name": "id",
"type_info": "Uuid"
},
{
"ordinal": 1,
"name": "success!",
"type_info": "Bool"
},
{
"ordinal": 2,
"name": "result: Json<Box<RawValue>>",
"type_info": "Jsonb"
}
],
"parameters": {
"Left": [
"Text",
"UuidArray"
]
},
"nullable": [
false,
null,
true
]
},
"hash": "789c3f6d29f46fcb17a22fe97405a1754ccb8ddeb08c23795dfe14e9f332e9dc"
}
@@ -0,0 +1,15 @@
{
"db_name": "PostgreSQL",
"query": "UPDATE v2_job_status SET\n flow_status = jsonb_set(\n flow_status,\n array['modules', $2::int::TEXT, 'agent_actions_success'],\n (\n SELECT jsonb_agg(CASE WHEN e = 'null'::jsonb THEN 'false'::jsonb ELSE e END\n ORDER BY i)\n FROM jsonb_array_elements(\n flow_status->'modules'->$2::int->'agent_actions_success'\n ) WITH ORDINALITY AS t(e, i)\n )\n )\n WHERE id = $1\n AND flow_status->'modules'->$2::int->'agent_actions_success' @> '[null]'::jsonb",
"describe": {
"columns": [],
"parameters": {
"Left": [
"Uuid",
"Int4"
]
},
"nullable": []
},
"hash": "9f729dd19fa4155d769eec3ed55ef0b15539aa9c0fef1559820384aa7864056b"
}
@@ -0,0 +1,17 @@
{
"db_name": "PostgreSQL",
"query": "INSERT INTO ai_agent_checkpoint (job_id, workspace_id, checkpoint, waiting_on)\n VALUES ($1, $2, $3, $4)\n ON CONFLICT (job_id) DO UPDATE\n SET checkpoint = EXCLUDED.checkpoint, waiting_on = EXCLUDED.waiting_on",
"describe": {
"columns": [],
"parameters": {
"Left": [
"Uuid",
"Varchar",
"Jsonb",
"UuidArray"
]
},
"nullable": []
},
"hash": "be2bcf0fb9a3c2fa50ad43d4a2d766f073e96a25d307db70fa6560a1f40b4602"
}
@@ -0,0 +1,22 @@
{
"db_name": "PostgreSQL",
"query": "SELECT EXISTS (SELECT 1 FROM v2_job_completed WHERE id = ANY($1)) AS \"finished!\"",
"describe": {
"columns": [
{
"ordinal": 0,
"name": "finished!",
"type_info": "Bool"
}
],
"parameters": {
"Left": [
"UuidArray"
]
},
"nullable": [
null
]
},
"hash": "beca1893fb61eb1d825af909345ea816ca2b0f6cb8b557dcbf9b0a538fb720f5"
}
@@ -0,0 +1,12 @@
{
"db_name": "PostgreSQL",
"query": "UPDATE v2_job_queue q SET suspend = 0\n FROM ai_agent_checkpoint c\n WHERE c.waiting_on <> '{}' AND q.id = c.job_id\n AND q.suspend > 0 AND q.suspend_until IS NOT NULL\n AND EXISTS (SELECT 1 FROM v2_job_completed jc WHERE jc.id = ANY(c.waiting_on))",
"describe": {
"columns": [],
"parameters": {
"Left": []
},
"nullable": []
},
"hash": "e63877925691e80d722f5fc4384c0ac00f6f4389224dca3feee1b879d4009f81"
}
@@ -0,0 +1,12 @@
{
"db_name": "PostgreSQL",
"query": "DELETE FROM ai_agent_checkpoint c\n WHERE NOT EXISTS (SELECT 1 FROM v2_job_queue q WHERE q.id = c.job_id)",
"describe": {
"columns": [],
"parameters": {
"Left": []
},
"nullable": []
},
"hash": "f688cb133dc6e916ef575d4ea7a55ac041db620c5bd881110efcb8ceb6fd1f8b"
}
@@ -0,0 +1 @@
ALTER TABLE ai_agent_checkpoint DROP COLUMN IF EXISTS waiting_on;
@@ -0,0 +1,3 @@
-- The background tool jobs a parked AI agent waits on: the monitor wakes it once one of them has
-- completed. Empty while it waits on an approval, whose answer comes first.
ALTER TABLE ai_agent_checkpoint ADD COLUMN IF NOT EXISTS waiting_on UUID[] NOT NULL DEFAULT '{}';
+33 -1
View File
@@ -4578,6 +4578,17 @@ pub async fn monitor_db(
}
};
// Every pass: a parked AI agent waits on this for its background tool jobs.
let wake_ai_agents_f = async {
if server_mode && !initial_load {
if let Some(db) = conn.as_sql() {
if let Err(e) = wake_ai_agents_with_finished_background_jobs(&db).await {
tracing::error!("Error waking AI agents on background jobs: {:?}", e);
}
}
}
};
// run every 30 iterations (~5min at the default LISTEN_NEW_EVENTS_INTERVAL_SEC).
let stranded_jobs_f = async {
if server_mode && iteration.is_some() && iteration.as_ref().unwrap().should_run(30) {
@@ -4732,6 +4743,7 @@ pub async fn monitor_db(
pipeline_freshness_watchdog_f,
reconcile_unarmed_schedules_f,
stranded_jobs_f,
wake_ai_agents_f,
);
}
@@ -7881,10 +7893,30 @@ async fn cleanup_job_result_stream_orphaned_jobs(db: &DB) -> error::Result<()> {
Ok(())
}
/// Zero the `suspend` of each parked AI agent with a completed job in its `waiting_on`, so the
/// suspended pull takes it back. Each job is listed by one park only, and an agent no longer
/// parked (`suspend_until` cleared by its pull) is left alone.
async fn wake_ai_agents_with_finished_background_jobs(db: &DB) -> error::Result<()> {
let woken = sqlx::query!(
"UPDATE v2_job_queue q SET suspend = 0
FROM ai_agent_checkpoint c
WHERE c.waiting_on <> '{}' AND q.id = c.job_id
AND q.suspend > 0 AND q.suspend_until IS NOT NULL
AND EXISTS (SELECT 1 FROM v2_job_completed jc WHERE jc.id = ANY(c.waiting_on))",
)
.execute(db)
.await?
.rows_affected();
if woken > 0 {
tracing::info!("Woke {woken} AI agents whose background tool jobs finished");
}
Ok(())
}
/// A parked agent cancelled from outside, or a worker that died mid-segment, never deletes its
/// checkpoint itself.
async fn cleanup_ai_agent_checkpoint_orphaned_jobs(db: &DB) -> error::Result<()> {
let deleted = sqlx::query(
let deleted = sqlx::query!(
"DELETE FROM ai_agent_checkpoint c
WHERE NOT EXISTS (SELECT 1 FROM v2_job_queue q WHERE q.id = c.job_id)",
)
+1 -1
View File
@@ -37,7 +37,7 @@ _sqlx_migrations: version(bigint), description(text), installed_on(ts), success(
account: workspace_id(char), id(int), expires_at(ts), refresh_token(char), client(char), refresh_error(text), grant_type(char), cc_client_id(char), cc_client_secret(char), cc_token_url(char), mcp_server_url(text), scopes(text[])
FK: (workspace_id) -> workspace(id)
agent_token_blacklist: token(char), expires_at(ts), blacklisted_at(ts), blacklisted_by(char)
ai_agent_checkpoint: job_id(uuid), workspace_id(char), checkpoint(jsonb), created_at(ts)
ai_agent_checkpoint: job_id(uuid), workspace_id(char), checkpoint(jsonb), created_at(ts), waiting_on(uuid[])
ai_agent_memory: workspace_id(char), conversation_id(uuid), step_id(char), messages(jsonb), created_at(ts), updated_at(ts)
ai_free_token_daily_usage: day(date), cost_nanos(bigint), updated_at(ts)
ai_free_token_usage: email(char), cost_nanos(bigint), updated_at(ts)
+53 -1
View File
@@ -612,7 +612,7 @@ pub fn openai_messages_to_bedrock(
})?;
bedrock_messages.push(tool_result_message);
}
bedrock_messages.push(convert_message(msg)?);
push_alternating(&mut bedrock_messages, convert_message(msg)?)?;
}
"tool" => {
// Accumulate tool results - they will be flushed as a single message
@@ -646,6 +646,27 @@ pub fn openai_messages_to_bedrock(
Ok((bedrock_messages, system_prompts))
}
/// Push a message, folding it into the previous one when both are the user's: an agent's
/// background-tool results arrive as user messages, right after tool results or each other.
fn push_alternating(messages: &mut Vec<Message>, message: Message) -> Result<(), Error> {
match messages.last_mut() {
Some(last)
if *last.role() == ConversationRole::User
&& *message.role() == ConversationRole::User =>
{
let mut content = last.content().to_vec();
content.extend(message.content().iter().cloned());
*last = Message::builder()
.role(ConversationRole::User)
.set_content(Some(content))
.build()
.map_err(|e| Error::internal_err(format!("Failed to merge user messages: {e}")))?;
}
_ => messages.push(message),
}
Ok(())
}
/// Helper to extract text from OpenAIContent (ignoring images)
///
/// This is public so it can be reused by the worker module.
@@ -1511,6 +1532,37 @@ mod tests {
));
}
#[test]
fn openai_messages_to_bedrock_keeps_roles_alternating() {
let tool_result = OpenAIMessage {
role: "tool".to_string(),
content: Some(OpenAIContent::Text("started".to_string())),
tool_call_id: Some("call_1".to_string()),
..Default::default()
};
let messages = vec![
text_message("user", "Export the report"),
text_message("assistant", "Exporting"),
tool_result,
text_message("user", "Background tool finished"),
text_message("user", "Another background tool finished"),
];
let (bedrock_messages, _) =
openai_messages_to_bedrock(&messages, false).expect("bedrock conversion succeeds");
let roles: Vec<_> = bedrock_messages.iter().map(|m| m.role().clone()).collect();
assert_eq!(
roles,
vec![
ConversationRole::User,
ConversationRole::Assistant,
ConversationRole::User
]
);
assert_eq!(bedrock_messages[2].content().len(), 3);
}
#[test]
fn build_tool_config_adds_cache_point_when_enabled() {
let tools = vec![test_tool()];
+6
View File
@@ -65,6 +65,9 @@ pub struct Tool {
/// Set on the built-in question tool, which the worker answers by parking the agent until
/// someone answers or this many seconds pass.
pub question_timeout_secs: Option<u32>,
/// A flow module tool whose call is answered once its job is queued (`Some(0)`) or, unfinished,
/// this many seconds after; see `AgentTool::background`.
pub background_after_secs: Option<u32>,
}
#[derive(Deserialize, Serialize, Debug, Clone, PartialEq)]
@@ -553,6 +556,9 @@ pub enum StreamingEvent {
ToolExecution { call_id: String, function_name: String },
/// Tool execution result
ToolResult { call_id: String, function_name: String, result: String, success: bool },
/// The agent has answered and waits, without a worker, on tools still running in the
/// background: nothing more streams until one of them finishes and wakes it.
BackgroundWait,
}
#[derive(Serialize, Deserialize, Clone, Debug)]
File diff suppressed because one or more lines are too long
+5 -5
View File
@@ -198,7 +198,7 @@ struct UntaggedFlowStatusModule {
failed_retries: Option<Vec<Uuid>>,
skipped: Option<bool>,
agent_actions: Option<Vec<AgentAction>>,
agent_actions_success: Option<Vec<bool>>,
agent_actions_success: Option<Vec<Option<bool>>>,
}
#[derive(Serialize, Deserialize, Debug, Clone)]
@@ -284,7 +284,7 @@ pub enum FlowStatusModule {
#[serde(skip_serializing_if = "Option::is_none")]
agent_actions: Option<Vec<AgentAction>>,
#[serde(skip_serializing_if = "Option::is_none")]
agent_actions_success: Option<Vec<bool>>,
agent_actions_success: Option<Vec<Option<bool>>>,
},
Success {
id: String,
@@ -306,7 +306,7 @@ pub enum FlowStatusModule {
#[serde(skip_serializing_if = "Option::is_none")]
agent_actions: Option<Vec<AgentAction>>,
#[serde(skip_serializing_if = "Option::is_none")]
agent_actions_success: Option<Vec<bool>>,
agent_actions_success: Option<Vec<Option<bool>>>,
},
Failure {
id: String,
@@ -324,7 +324,7 @@ pub enum FlowStatusModule {
#[serde(skip_serializing_if = "Option::is_none")]
agent_actions: Option<Vec<AgentAction>>,
#[serde(skip_serializing_if = "Option::is_none")]
agent_actions_success: Option<Vec<bool>>,
agent_actions_success: Option<Vec<Option<bool>>>,
},
}
@@ -556,7 +556,7 @@ impl FlowStatusModule {
}
}
pub fn agent_actions_success(&self) -> Option<Vec<bool>> {
pub fn agent_actions_success(&self) -> Option<Vec<Option<bool>>> {
match self {
FlowStatusModule::InProgress { agent_actions_success, .. } => {
agent_actions_success.clone()
+8
View File
@@ -960,6 +960,14 @@ pub struct AgentTool {
/// Flow module and MCP tools only.
#[serde(default, skip_serializing_if = "std::ops::Not::not")]
pub requires_approval: bool,
/// The agent is answered as soon as the tool's job starts, and gets its result in a later
/// message once it finishes. Only a flow module tool can run in the background.
#[serde(default, skip_serializing_if = "std::ops::Not::not")]
pub background: bool,
/// With `background` unset, the agent waits this many seconds for the tool's job, then is
/// answered as for a background tool. Flow module tools only.
#[serde(default, skip_serializing_if = "Option::is_none")]
pub background_after_secs: Option<u32>,
pub value: ToolValue,
}
+106 -33
View File
@@ -1,8 +1,15 @@
//! A call of a tool that requires approval, or of the built-in question tool, parks the agent
//! instead of running. The agent stores its conversation, releases its worker the way a
//! The agent parks, releasing its worker, while it waits on a person or on its background tools.
//! A call of a tool that requires approval, or of the built-in question tool, parks it until
//! someone answers; an answer the model gave while background tool jobs still run parks it until
//! one of them finishes. It stores its conversation and parks its own queue row the way a
//! workflow-as-code parent does (`suspend_wac_parent`), and is pulled again when `suspend` reaches
//! 0 (an answer, or a rejection) or `suspend_until` passes (the request timing out).
//! `resume_suspended` answers it like a WAC approval, since its queue row is the one suspended.
//! 0 or `suspend_until` passes.
//!
//! An answer or a rejection reaches `suspend` through `resume_suspended`, which takes the agent's
//! row for a WAC approval. A finished background job reaches it through the monitor, which zeroes
//! the `suspend` of a parked agent once a job in its checkpoint's `waiting_on` has completed. Not
//! through `record_child_completion`: it clears every pending id of a parent it wakes, which fits a
//! WAC round but would drop the agent's other background jobs.
//!
//! The checkpoint row outlives the segment that reads it, so a job restarted after its worker died
//! resumes from it again. It is deleted by the monitor once the job leaves the queue.
@@ -24,6 +31,7 @@ use windmill_common::{
use windmill_queue::MiniPulledJob;
use crate::ai::compaction::HistoryCheckpoint;
use crate::ai::tools::BackgroundJob;
use crate::ai::utils::find_unique_tool_name;
use crate::wac_executor::{end_wac_segment, suspend_wac_parent, WacPark};
@@ -33,6 +41,8 @@ pub(crate) const QUESTION_ENABLED_NAME: &str = "__wm_question";
/// Matches a suspended flow step's default.
const DEFAULT_TIMEOUT_SECS: u32 = 1800;
/// A park on background jobs only is woken by the monitor; this only bounds the park.
const BACKGROUND_RECHECK_SECS: f64 = 3600.0;
pub(crate) fn question_timeout(value: &QuestionToolValue) -> u32 {
value.timeout.unwrap_or(DEFAULT_TIMEOUT_SECS).max(1)
@@ -79,6 +89,7 @@ pub(crate) fn question_tool(timeout_secs: u32, existing: &[ToolDef]) -> Tool {
mcp_source: None,
requires_approval: false,
question_timeout_secs: Some(timeout_secs),
background_after_secs: None,
}
}
@@ -180,7 +191,17 @@ pub(crate) struct AgentCheckpoint {
pub events: String,
pub reasoning: String,
pub next_iteration: usize,
pub pending: PendingPause,
/// The approval the agent waits on, if any.
pub pending: Option<PendingPause>,
#[serde(default)]
pub background: Vec<BackgroundJob>,
/// The model had given its answer: with nothing new from a background job, there is nothing
/// to ask it, so the agent parks again.
#[serde(default)]
pub answered: bool,
/// The model gave a structured answer, which stands unless it gives another.
#[serde(default)]
pub structured_answer: bool,
/// `resume_job` rows never leave while the agent is queued, so each answer must skip the
/// ones earlier parks already read.
#[serde(default)]
@@ -191,11 +212,13 @@ pub(crate) async fn load_checkpoint(
db: &DB,
job_id: &Uuid,
) -> error::Result<Option<AgentCheckpoint>> {
let raw: Option<sqlx::types::Json<Box<serde_json::value::RawValue>>> =
sqlx::query_scalar("SELECT checkpoint FROM ai_agent_checkpoint WHERE job_id = $1")
.bind(job_id)
.fetch_optional(db)
.await?;
let raw = sqlx::query_scalar!(
r#"SELECT checkpoint AS "checkpoint: sqlx::types::Json<Box<serde_json::value::RawValue>>"
FROM ai_agent_checkpoint WHERE job_id = $1"#,
job_id
)
.fetch_optional(db)
.await?;
raw.map(|r| {
serde_json::from_str(r.0.get())
.map_err(|e| Error::internal_err(format!("Invalid AI agent checkpoint: {e}")))
@@ -207,34 +230,63 @@ pub(crate) enum ParkOutcome {
Parked,
/// A cancel reached the queue row first, so nothing was written.
Cancelled,
/// A background job finished before the park could be written, so the agent continues.
JobFinished,
}
/// Park the agent until its approval is answered or expires.
/// Park the agent until its approval is answered or expires or, with no approval open, until the
/// monitor finds one of its `waiting_on` jobs completed.
pub(crate) async fn park(
db: &DB,
conn: &Connection,
job: &MiniPulledJob,
checkpoint: &AgentCheckpoint,
) -> error::Result<ParkOutcome> {
let remaining_ms = (checkpoint.pending.expires_at - Utc::now()).num_milliseconds();
let suspend_secs = (remaining_ms as f64 / 1000.0).max(1.0);
let suspend_secs = match &checkpoint.pending {
Some(pending) => {
let remaining_ms = (pending.expires_at - Utc::now()).num_milliseconds();
(remaining_ms as f64 / 1000.0).max(1.0)
}
None => BACKGROUND_RECHECK_SECS,
};
let running: Vec<Uuid> = checkpoint.background.iter().map(|b| b.job_id).collect();
let waits_on_approval = checkpoint.pending.is_some();
let checkpoint = serde_json::to_value(checkpoint).map_err(|e| {
Error::internal_err(format!("Failed to serialize AI agent checkpoint: {e}"))
})?;
let mut tx = db.begin().await?;
// One event: an answer, or a rejection, which zeroes `suspend` outright.
// One event wakes it: an answer, a rejection (which zeroes `suspend` outright), or a
// background job finishing.
let segment_ms =
match suspend_wac_parent(&mut tx, &job.id, &job.workspace_id, 1, suspend_secs).await? {
WacPark::Parked(ms) => ms,
WacPark::Cancelled(_) => return Ok(ParkOutcome::Cancelled),
};
sqlx::query(
"INSERT INTO ai_agent_checkpoint (job_id, workspace_id, checkpoint) VALUES ($1, $2, $3)
ON CONFLICT (job_id) DO UPDATE SET checkpoint = EXCLUDED.checkpoint",
// A job that completed since the loop last looked is delivered now rather than after a park
// and a monitor tick. While an approval is open, results wait for the answer anyway.
let waiting_on: Vec<Uuid> = if waits_on_approval { vec![] } else { running };
if !waiting_on.is_empty() {
let finished = sqlx::query_scalar!(
r#"SELECT EXISTS (SELECT 1 FROM v2_job_completed WHERE id = ANY($1)) AS "finished!""#,
&waiting_on
)
.fetch_one(&mut *tx)
.await?;
if finished {
tx.rollback().await?;
return Ok(ParkOutcome::JobFinished);
}
}
sqlx::query!(
"INSERT INTO ai_agent_checkpoint (job_id, workspace_id, checkpoint, waiting_on)
VALUES ($1, $2, $3, $4)
ON CONFLICT (job_id) DO UPDATE
SET checkpoint = EXCLUDED.checkpoint, waiting_on = EXCLUDED.waiting_on",
job.id,
&job.workspace_id,
checkpoint,
&waiting_on
)
.bind(&job.id)
.bind(&job.workspace_id)
.bind(&checkpoint)
.execute(&mut *tx)
.await?;
tx.commit().await?;
@@ -242,6 +294,30 @@ pub(crate) async fn park(
Ok(ParkOutcome::Parked)
}
/// Resume an agent that waits on no approval. `suspend` is zeroed, so `resume_suspended` refuses
/// answers while it runs, as `answer_pause` does for one with an approval. An answer the parked
/// row still accepted is marked read: no request was open, so none may take it later.
pub(crate) async fn resume_without_approval(
db: &DB,
job_id: &Uuid,
consumed_resume_rows: &mut Vec<Uuid>,
) -> error::Result<()> {
let mut tx = db.begin().await?;
sqlx::query!("UPDATE v2_job_queue SET suspend = 0 WHERE id = $1", job_id)
.execute(&mut *tx)
.await?;
let stray = sqlx::query_scalar!(
"SELECT id FROM resume_job WHERE job = $1 AND id <> ALL($2)",
job_id,
&*consumed_resume_rows
)
.fetch_all(&mut *tx)
.await?;
tx.commit().await?;
consumed_resume_rows.extend(stray);
Ok(())
}
pub(crate) enum Resumed {
/// The answer, and whether it counts as a success on the run page.
Answered(Answer, bool),
@@ -272,23 +348,20 @@ pub(crate) async fn answer_pause(
consumed_resume_rows: &mut Vec<Uuid>,
) -> error::Result<Resumed> {
let mut tx = db.begin().await?;
sqlx::query("UPDATE v2_job_queue SET suspend = 0 WHERE id = $1")
.bind(job_id)
sqlx::query!("UPDATE v2_job_queue SET suspend = 0 WHERE id = $1", job_id)
.execute(&mut *tx)
.await?;
let rows: Vec<(
Uuid,
sqlx::types::Json<serde_json::Value>,
Option<String>,
bool,
)> = sqlx::query_as(
"SELECT id, value, approver, approved FROM resume_job
WHERE job = $1 AND id <> ALL($2) ORDER BY created_at ASC",
let rows = sqlx::query!(
r#"SELECT id, value AS "value: sqlx::types::Json<serde_json::Value>", approver, approved
FROM resume_job WHERE job = $1 AND id <> ALL($2) ORDER BY created_at ASC"#,
job_id,
&*consumed_resume_rows
)
.bind(job_id)
.bind(&*consumed_resume_rows)
.fetch_all(&mut *tx)
.await?;
.await?
.into_iter()
.map(|r| (r.id, r.value, r.approver, r.approved))
.collect::<Vec<_>>();
tx.commit().await?;
consumed_resume_rows.extend(rows.iter().map(|(id, ..)| *id));
+276 -15
View File
@@ -2,8 +2,8 @@ use crate::ai::pause::{start_approval, start_question, PendingPause};
use crate::ai::stream_event_processor::StreamEventProcessor;
use crate::ai::utils::{
add_message_to_conversation, execute_mcp_tool, get_step_name_from_flow,
update_flow_status_module_with_actions, update_flow_status_module_with_actions_success,
FlowContext,
set_flow_status_module_action_success_at, update_flow_status_module_with_actions,
update_flow_status_module_with_actions_success, FlowContext,
};
use crate::common::{get_root_job_id, OccupancyMetrics};
use crate::result_processor::handle_non_flow_job_error;
@@ -14,9 +14,14 @@ use crate::worker_flow::{
use crate::{create_job_dir, handle_queued_job, JobCompletedSender};
use anyhow::Context;
use mappable_rc::Marc;
use serde::{Deserialize, Serialize};
use serde_json::value::RawValue;
use sqlx::types::Json;
use std::{collections::HashMap, sync::Arc, time::Duration};
use std::{
collections::HashMap,
sync::Arc,
time::{Duration, Instant},
};
use uuid::Uuid;
use windmill_ai::{ai_types::OpenAIToolCall, query_builder::StreamEventSink, types::*};
use windmill_common::jobs::JobPayload;
@@ -126,6 +131,7 @@ pub async fn execute_tool_calls(
refuse_tool_call(&mut ctx, call, &refusal, &mut messages, final_events_str).await?;
}
let mut started = Vec::new();
let mut calls = to_run.into_iter().peekable();
while let Some(tool_call) = calls.next() {
// Stream tool call progress
@@ -151,6 +157,25 @@ pub async fn execute_tool_calls(
tool_call_id: Some(tool_call.id.clone()),
..Default::default()
});
// The answer ends the turn, but the history can be sent again after a background
// result, and a provider refuses a call left unanswered.
for skipped in calls.by_ref() {
messages.push(OpenAIMessage {
role: "tool".to_string(),
content: Some(OpenAIContent::Text(
"Not run: the structured answer ended this turn.".to_string(),
)),
tool_call_id: Some(skipped.id.clone()),
..Default::default()
});
}
// An action like a text answer's: the run page matches tagged messages to actions by
// position, and with background tools a run can go on after this answer.
actions.push(AgentAction::Message {});
if let Some(parent_job) = ctx.parent_job {
update_flow_status_module_with_actions(ctx.db, parent_job, actions).await?;
update_flow_status_module_with_actions_success(ctx.db, parent_job, true).await?;
}
messages.push(OpenAIMessage {
role: "assistant".to_string(),
content: Some(OpenAIContent::Text(tool_call.function.arguments.clone())),
@@ -189,6 +214,11 @@ pub async fn execute_tool_calls(
final_events_str,
)
.await?;
} else if tool.module.is_some() && tool.background_after_secs == Some(0) {
started.push(
start_background_tool(&mut ctx, tool_call, tool, actions, &mut messages)
.await?,
);
} else if tool.module.is_some() {
let mut batch = vec![(tool_call, tool)];
while let Some(next_call) = calls.peek() {
@@ -201,6 +231,7 @@ pub async fn execute_tool_calls(
t.def.function.name == next_call.function.name
&& t.mcp_source.is_none()
&& t.module.is_some()
&& t.background_after_secs != Some(0)
}) else {
break;
};
@@ -209,8 +240,15 @@ pub async fn execute_tool_calls(
}
batch.push((calls.next().unwrap(), next_tool));
}
execute_windmill_tools(&mut ctx, &batch, actions, &mut messages, final_events_str)
.await?;
execute_windmill_tools(
&mut ctx,
&batch,
actions,
&mut messages,
&mut started,
final_events_str,
)
.await?;
} else {
return Err(Error::internal_err(format!(
"Tool type not supported: {}",
@@ -225,7 +263,7 @@ pub async fn execute_tool_calls(
}
}
Ok(ToolCallsOutcome { messages, final_content, used_structured_output_tool, pause })
Ok(ToolCallsOutcome { messages, final_content, used_structured_output_tool, pause, started })
}
/// A turn's calls: run now, refused with a message, or held for a person's answer.
@@ -325,6 +363,182 @@ pub struct ToolCallsOutcome {
pub used_structured_output_tool: bool,
/// The turn asked a person something: the agent parks, those calls unanswered.
pub pause: Option<PendingPause>,
/// Background tool calls this turn started.
pub started: Vec<BackgroundJob>,
}
/// A background tool call, answered when its job started and still owed its result.
#[derive(Serialize, Deserialize, Clone, Debug)]
pub(crate) struct BackgroundJob {
pub call_id: String,
pub function_name: String,
pub arguments: String,
pub module_id: String,
pub job_id: Uuid,
/// Its position in the run's actions, which is where its outcome is recorded.
pub action_index: usize,
/// A nested agent's result is an envelope; the model gets its `output`.
pub is_agent: bool,
}
/// A background job that finished: whether it succeeded, and what the model is told it returned.
pub(crate) struct FinishedBackgroundJob {
pub job: BackgroundJob,
pub success: bool,
pub content: String,
}
impl FinishedBackgroundJob {
/// What the model reads. Its call was answered when the job started, so the result comes as a
/// message of its own, tagged with the call's action for the run's trace.
pub fn message(&self) -> OpenAIMessage {
let job = &self.job;
let outcome = if self.success { "finished" } else { "failed" };
OpenAIMessage {
role: "user".to_string(),
content: Some(OpenAIContent::Text(format!(
"Background tool `{}` (call {}, job {}) {outcome}. Result:\n{}",
job.function_name, job.call_id, job.job_id, self.content
))),
agent_action: Some(AgentAction::ToolCall {
job_id: job.job_id,
function_name: job.function_name.clone(),
module_id: job.module_id.clone(),
}),
..Default::default()
}
}
/// The chat transcript row, shaped like a foreground tool's.
pub fn chat_row(&self) -> (String, MessageExtras) {
let job = &self.job;
let content = if self.success {
format!("Used {} tool", job.function_name)
} else {
format!("Error executing {}", job.function_name)
};
let extras = MessageExtras {
tool_arguments: Some(job.arguments.clone()),
tool_result: Some(self.content.clone()),
..Default::default()
};
(content, extras)
}
}
/// Take the background jobs that have finished out of `running`.
pub(crate) async fn take_finished_background_jobs(
db: &DB,
workspace_id: &str,
running: &mut Vec<BackgroundJob>,
) -> Result<Vec<FinishedBackgroundJob>, Error> {
let ids: Vec<Uuid> = running.iter().map(|b| b.job_id).collect();
let completed = sqlx::query!(
"SELECT id, status = 'success' AS \"success!\", result AS \"result: Json<Box<RawValue>>\"
FROM v2_job_completed WHERE workspace_id = $1 AND id = ANY($2)",
workspace_id,
&ids,
)
.fetch_all(db)
.await?;
let mut finished = Vec::with_capacity(completed.len());
for (id, success, result) in completed.into_iter().map(|c| (c.id, c.success, c.result)) {
let Some(index) = running.iter().position(|b| b.job_id == id) else {
continue;
};
let job = running.remove(index);
let result = result
.map(|r| r.0)
.unwrap_or_else(|| to_raw_value(&serde_json::Value::Null));
let content = if job.is_agent && success {
extract_ai_agent_output(&result).unwrap_or_else(|| result.get().to_string())
} else {
result.get().to_string()
};
finished.push(FinishedBackgroundJob { job, success, content });
}
// Call order, so a run reads the same however the completions raced.
finished.sort_by_key(|f| f.job.action_index);
Ok(finished)
}
async fn start_background_tool(
ctx: &mut ToolExecutionContext<'_>,
tool_call: &OpenAIToolCall,
tool: &Tool,
actions: &mut Vec<AgentAction>,
messages: &mut Vec<OpenAIMessage>,
) -> Result<BackgroundJob, Error> {
let job_id = ulid::Ulid::new().into();
// Never reserved for this worker: it is busy with the agent, which may park before the job
// is pulled.
enqueue_windmill_tool(ctx, tool_call, tool, actions, job_id, false).await?;
let action_index = actions.len() - 1;
let content = format!(
"Started in the background as job {job_id}. Its result arrives in a later message: keep \
working meanwhile, or finish your answer and you will continue once it is in."
);
let row = format!("Started {} in the background", tool_call.function.name);
answer_in_background(
ctx,
tool_call,
tool,
actions,
messages,
job_id,
action_index,
content,
row,
)
.await
}
/// Answer a call whose job keeps running, and hand the job to the agent's background list.
async fn answer_in_background(
ctx: &mut ToolExecutionContext<'_>,
tool_call: &OpenAIToolCall,
tool: &Tool,
actions: &[AgentAction],
messages: &mut Vec<OpenAIMessage>,
job_id: Uuid,
action_index: usize,
content: String,
row: String,
) -> Result<BackgroundJob, Error> {
if let Some(parent_job) = ctx.parent_job {
set_flow_status_module_action_success_at(ctx.db, parent_job, action_index, None).await?;
}
messages.push(OpenAIMessage {
role: "tool".to_string(),
content: Some(OpenAIContent::Text(content.clone())),
tool_call_id: Some(tool_call.id.clone()),
agent_action: actions.get(action_index).cloned(),
..Default::default()
});
let extras = MessageExtras {
tool_arguments: Some(tool_call.function.arguments.clone()),
tool_result: Some(content),
..Default::default()
};
add_tool_message_to_chat(ctx, Some(job_id), &row, true, Some(extras)).await;
Ok(BackgroundJob {
call_id: tool_call.id.clone(),
function_name: tool_call.function.name.clone(),
arguments: tool_call.function.arguments.clone(),
module_id: tool
.module
.as_ref()
.map(|m| m.id.clone())
.unwrap_or_default(),
job_id,
action_index,
is_agent: tool.module.as_ref().is_some_and(|module| {
matches!(
module.get_value(),
Ok(FlowModuleValue::AIAgent { .. } | FlowModuleValue::AIDecision { .. })
)
}),
})
}
pub enum ApprovalGate<'a> {
@@ -514,6 +728,7 @@ async fn enqueue_windmill_tool(
tool_call: &OpenAIToolCall,
tool: &Tool,
actions: &mut Vec<AgentAction>,
job_id: Uuid,
reserved: bool,
) -> Result<(Uuid, bool), Error> {
// Regular Windmill tools must have a module
@@ -521,7 +736,6 @@ async fn enqueue_windmill_tool(
Error::internal_err(format!("Tool {} has no module", tool_call.function.name))
})?;
let job_id = ulid::Ulid::new().into();
actions.push(AgentAction::ToolCall {
job_id,
function_name: tool_call.function.name.clone(),
@@ -969,9 +1183,14 @@ async fn execute_windmill_tools(
batch: &[(&OpenAIToolCall, &Tool)],
actions: &mut Vec<AgentAction>,
messages: &mut Vec<OpenAIMessage>,
started: &mut Vec<BackgroundJob>,
final_events_str: &mut String,
) -> Result<(), Error> {
let mut job_ids = Vec::with_capacity(batch.len());
let mut action_indexes = Vec::with_capacity(batch.len());
// When a call still running is answered as a background one. Such a job never runs in this
// worker's process, which could not release the worker while it runs: the agent may park.
let mut detach_at = Vec::with_capacity(batch.len());
let mut local = None;
for (index, (call, tool)) in batch.iter().enumerate() {
if index > 0 {
@@ -987,8 +1206,21 @@ async fn execute_windmill_tools(
.await?;
}
}
let (id, reserved) = enqueue_windmill_tool(ctx, call, tool, actions, index == 0).await?;
let job_id = ulid::Ulid::new().into();
let detach_after = tool.background_after_secs;
let (id, reserved) = enqueue_windmill_tool(
ctx,
call,
tool,
actions,
job_id,
index == 0 && detach_after.is_none(),
)
.await?;
job_ids.push(id);
action_indexes.push(actions.len() - 1);
detach_at
.push(detach_after.map(|secs| Instant::now() + Duration::from_secs(u64::from(secs))));
if reserved {
let job = get_mini_pulled_job(ctx.db, &id)
.await?
@@ -1067,10 +1299,39 @@ async fn execute_windmill_tools(
// Transcript rows, model messages, and positional action statuses share call order.
while next_result < batch.len() {
let job_id = job_ids[next_result];
let Some((success, content)) = results.remove(&job_id) else {
break;
};
let (call, tool) = batch[next_result];
let Some((success, content)) = results.remove(&job_id) else {
if !detach_at[next_result].is_some_and(|at| Instant::now() >= at) {
break;
}
let secs = tool.background_after_secs.unwrap_or_default();
let content = format!(
"Not finished after {secs}s, so it continues in the background as job \
{job_id}. Its result arrives in a later message: keep working meanwhile, or \
finish your answer and you will continue once it is in."
);
let row = format!(
"Moved {} to the background after {secs}s",
call.function.name
);
let action_index = action_indexes[next_result];
started.push(
answer_in_background(
ctx,
call,
tool,
actions,
messages,
job_id,
action_index,
content,
row,
)
.await?,
);
next_result += 1;
continue;
};
let module = tool
.module
.as_ref()
@@ -1095,10 +1356,9 @@ async fn execute_windmill_tools(
}
if local.is_none() && next_result < batch.len() {
let pending: Vec<Uuid> = job_ids[next_result..]
.iter()
.filter(|id| !results.contains_key(id))
.copied()
let pending: Vec<Uuid> = (next_result..batch.len())
.filter(|&i| detach_at[i].is_none() && !results.contains_key(&job_ids[i]))
.map(|i| job_ids[i])
.collect();
if !pending.is_empty() {
local = claim_local_tool(ctx, &pending).await?;
@@ -1292,6 +1552,7 @@ mod tests {
mcp_source: None,
requires_approval,
question_timeout_secs: question.then_some(60),
background_after_secs: None,
};
let call = |id: &str, name: &str| OpenAIToolCall {
id: id.to_string(),
+80 -13
View File
@@ -307,34 +307,100 @@ pub async fn update_flow_status_module_with_actions_success(
Ok(())
}
/// Record the outcome of the last of `action_count` actions, dropping any recorded past it.
pub async fn set_flow_status_module_last_action_success(
/// Record the outcome of the action at `index`, `None` while it runs. Outcomes are positional and
/// a background call's comes after later actions' own, so it is set in place, padded with nulls.
pub async fn set_flow_status_module_action_success_at(
db: &DB,
parent_job: &Uuid,
action_count: usize,
action_success: bool,
index: usize,
action_success: Option<bool>,
) -> Result<(), Error> {
if let Step::Step { idx: step, .. } = get_step_of_flow_status(db, parent_job.to_owned()).await?
{
sqlx::query(
sqlx::query!(
"UPDATE v2_job_status SET
flow_status = jsonb_set(
flow_status,
array['modules', $2::TEXT, 'agent_actions_success'],
array['modules', $2::int::TEXT, 'agent_actions_success'],
(
SELECT jsonb_agg(CASE WHEN i = $3 THEN to_jsonb($4::bool) ELSE arr->i END ORDER BY i)
FROM (SELECT COALESCE(
flow_status->'modules'->$2::int->'agent_actions_success', '[]'::jsonb
) AS arr) a,
generate_series(0, GREATEST(jsonb_array_length(arr) - 1, $3)) AS i
)
)
WHERE id = $1",
parent_job,
step as i32,
index as i32,
action_success
)
.execute(db)
.await?;
}
Ok(())
}
/// Mark as failed the outcomes still `null`: background calls whose agent ended before they did,
/// and whose jobs it cancels.
pub async fn fail_unfinished_action_successes(db: &DB, parent_job: &Uuid) -> Result<(), Error> {
if let Step::Step { idx: step, .. } = get_step_of_flow_status(db, parent_job.to_owned()).await?
{
sqlx::query!(
"UPDATE v2_job_status SET
flow_status = jsonb_set(
flow_status,
array['modules', $2::int::TEXT, 'agent_actions_success'],
(
SELECT jsonb_agg(CASE WHEN e = 'null'::jsonb THEN 'false'::jsonb ELSE e END
ORDER BY i)
FROM jsonb_array_elements(
flow_status->'modules'->$2::int->'agent_actions_success'
) WITH ORDINALITY AS t(e, i)
)
)
WHERE id = $1
AND flow_status->'modules'->$2::int->'agent_actions_success' @> '[null]'::jsonb",
parent_job,
step as i32
)
.execute(db)
.await?;
}
Ok(())
}
/// Keep the first `keep` action outcomes, dropping any recorded past them, then append `append`.
/// A resumed segment calls it so that one replayed after its worker died records nothing twice.
pub async fn reset_flow_status_module_action_successes(
db: &DB,
parent_job: &Uuid,
keep: usize,
append: Option<bool>,
) -> Result<(), Error> {
if let Step::Step { idx: step, .. } = get_step_of_flow_status(db, parent_job.to_owned()).await?
{
sqlx::query!(
"UPDATE v2_job_status SET
flow_status = jsonb_set(
flow_status,
array['modules', $2::int::TEXT, 'agent_actions_success'],
COALESCE((
SELECT jsonb_agg(e ORDER BY i)
FROM jsonb_array_elements(
flow_status->'modules'->$2->'agent_actions_success'
flow_status->'modules'->$2::int->'agent_actions_success'
) WITH ORDINALITY AS t(e, i)
WHERE i < $3
), '[]'::jsonb) || to_jsonb(ARRAY[$4::bool])
WHERE i <= $3
), '[]'::jsonb) || CASE WHEN $4::bool IS NULL THEN '[]'::jsonb
ELSE to_jsonb(ARRAY[$4::bool]) END
)
WHERE id = $1",
parent_job,
step as i32,
keep as i64,
append
)
.bind(parent_job)
.bind(step as i32)
.bind(action_count as i64)
.bind(action_success)
.execute(db)
.await?;
}
@@ -441,6 +507,7 @@ fn convert_mcp_tools_to_windmill_tools(
}),
requires_approval: false,
question_timeout_secs: None,
background_after_secs: None,
})
})
.collect()
+227 -52
View File
@@ -3,18 +3,20 @@ use crate::ai::compaction::{
Compactor,
};
use crate::ai::pause::{
answer_pause, load_checkpoint, park, question_timeout, question_tool, AgentCheckpoint, Answer,
ParkOutcome, PendingRequest, Resumed, QUESTION_ENABLED_NAME,
answer_pause, load_checkpoint, park, question_timeout, question_tool, resume_without_approval,
AgentCheckpoint, Answer, ParkOutcome, PendingRequest, Resumed, QUESTION_ENABLED_NAME,
};
use crate::ai::tools::{
execute_tool_calls, ApprovalGate, ToolAbortHandles, ToolCallsOutcome, ToolExecutionContext,
execute_tool_calls, take_finished_background_jobs, ApprovalGate, ToolAbortHandles,
ToolCallsOutcome, ToolExecutionContext,
};
use crate::ai::utils::{
add_message_to_conversation, any_tool_needs_previous_result, cleanup_mcp_clients,
filter_schema_by_input_transforms, find_unique_tool_name, get_flow_context,
get_flow_job_runnable_and_raw_flow, get_step_name_from_flow, load_mcp_tools,
parse_raw_script_schema, set_flow_status_module_last_action_success,
update_flow_status_module_with_actions, update_flow_status_module_with_actions_success,
fail_unfinished_action_successes, filter_schema_by_input_transforms, find_unique_tool_name,
get_flow_context, get_flow_job_runnable_and_raw_flow, get_step_name_from_flow, load_mcp_tools,
parse_raw_script_schema, reset_flow_status_module_action_successes,
set_flow_status_module_action_success_at, update_flow_status_module_with_actions,
update_flow_status_module_with_actions_success,
};
use crate::memory_common::MAX_MEMORY_SIZE_BYTES;
use crate::memory_oss::{memory_storage_capacity_bytes, read_from_memory, write_to_memory};
@@ -978,6 +980,8 @@ pub async fn handle_ai_agent_job(
// Explicit per-tool descriptions keyed by tool id. When set, these override the
// description auto-derived from the underlying runnable when building tool definitions.
let mut tool_descriptions: HashMap<String, String> = HashMap::new();
// A nested agent cannot park to wait for one, so its background tools run like any other.
let mut background_tools: HashMap<String, u32> = HashMap::new();
#[allow(unused_mut)]
let mut mcp_configs: Vec<crate::ai::utils::McpResourceConfig> = Vec::new();
let mut has_websearch = false;
@@ -1027,6 +1031,16 @@ pub async fn handle_ai_agent_job(
{
tool_descriptions.insert(tool.id.clone(), description.to_string());
}
let background_after_secs = if tool.background {
Some(0)
} else {
tool.background_after_secs
};
if let Some(secs) = background_after_secs {
if direct_parent_job_kind != JobKind::AIAgent {
background_tools.insert(tool.id.clone(), secs);
}
}
if let Some(flow_module) = Option::<FlowModule>::from(&tool) {
windmill_modules.push(flow_module);
}
@@ -1049,6 +1063,7 @@ pub async fn handle_ai_agent_job(
let job = job;
let user_description = tool_descriptions.get(&t.id).cloned();
let requires_approval = approval_required.contains(&t.id);
let background_after_secs = background_tools.get(&t.id).copied();
async move {
let summary = flow_module_tool_name(t.summary.as_deref())?;
@@ -1184,6 +1199,7 @@ pub async fn handle_ai_agent_job(
module: Some(t),
mcp_source: None,
question_timeout_secs: None,
background_after_secs,
})
}
}))
@@ -1341,24 +1357,28 @@ pub async fn handle_ai_agent_job(
handle.abort();
}
cleanup_orphaned_tool_jobs(db, &job.id, &job.workspace_id, canceled_by.clone()).await;
if let Some(parent_job) = flow_status_job.as_ref() {
if let Err(e) = fail_unfinished_action_successes(db, parent_job).await {
tracing::error!("Failed to close the background calls of {}: {e}", job.id);
}
}
}
cleanup_mcp_clients(mcp_clients).await;
result
}
/// Park the agent on its pending approval, as the error `run_agent` returns.
/// Park the agent, as the error `run_agent` returns. `None` when a background job finished
/// before the park was written, so the agent goes on instead; never for a park on an approval.
async fn park_agent(
db: &DB,
conn: &Connection,
job: &MiniPulledJob,
checkpoint: &AgentCheckpoint,
cancel_rx: &tokio::sync::watch::Receiver<bool>,
) -> Error {
match park(db, conn, job, checkpoint).await {
Ok(ParkOutcome::Parked) => Error::WacSuspended(format!(
"AI agent {} parked on {}",
job.id, checkpoint.pending.call_id
)),
) -> Option<Error> {
Some(match park(db, conn, job, checkpoint).await {
Ok(ParkOutcome::JobFinished) => return None,
Ok(ParkOutcome::Parked) => Error::WacSuspended(format!("AI agent {} parked", job.id)),
Ok(ParkOutcome::Cancelled) => {
// The cancel is on the queue row but not yet seen by the poller, which is what
// completes the job as cancelled rather than failed. Wait for it to notice.
@@ -1371,7 +1391,12 @@ async fn park_agent(
Error::ExecutionErr("Job cancelled".to_string())
}
Err(e) => e,
}
})
}
/// A park on an approval is never stood down by a background job.
fn approval_park(parked: Option<Error>) -> Error {
parked.unwrap_or_else(|| Error::internal_err("AI agent approval park was skipped".to_string()))
}
/// OpenAI rejects a `prompt_cache_key` over 64 characters
@@ -1479,24 +1504,35 @@ pub async fn run_agent(
// Fetch flow context for input transforms context, chat and memory
let mut flow_context = get_flow_context(db, job).await;
// Set when the job parked on an approval or a question and has been pulled again. Read
// Set when the job parked on a request or on background jobs and has been pulled again. Read
// whatever the roster says now: a linked agent can lose a tool while one of its runs is parked.
let resumed = match load_checkpoint(db, &job.id).await? {
Some(mut checkpoint) => {
let answer = answer_pause(
db,
&job.id,
&checkpoint.pending,
&checkpoint.actions,
&mut checkpoint.consumed_resume_rows,
)
.await?;
match answer {
Resumed::StillPending => {
return Err(park_agent(db, conn, job, &checkpoint, &cancel_rx).await)
let answer = match checkpoint.pending.clone() {
Some(pending) => {
let answer = answer_pause(
db,
&job.id,
&pending,
&checkpoint.actions,
&mut checkpoint.consumed_resume_rows,
)
.await?;
match answer {
Resumed::StillPending => {
let parked = park_agent(db, conn, job, &checkpoint, &cancel_rx).await;
return Err(approval_park(parked));
}
Resumed::Answered(message, success) => Some((pending, message, success)),
}
}
Resumed::Answered(message, success) => Some((checkpoint, message, success)),
}
None => {
resume_without_approval(db, &job.id, &mut checkpoint.consumed_resume_rows)
.await?;
None
}
};
Some((checkpoint, answer))
}
None => None,
};
@@ -1663,13 +1699,20 @@ pub async fn run_agent(
let mut first_iteration = 0;
let mut consumed_resume_rows = vec![];
let mut resumed_answer = None;
let mut background = vec![];
let mut answered = false;
let mut keep_structured_answer = false;
let was_resumed = resumed.is_some();
let (mut messages, mut actions, mut content, mut final_usage) = match resumed {
Some((checkpoint, message, success)) => {
Some((checkpoint, answer)) => {
final_events_str = checkpoint.events;
final_reasoning = checkpoint.reasoning;
first_iteration = checkpoint.next_iteration;
consumed_resume_rows = checkpoint.consumed_resume_rows;
resumed_answer = Some((checkpoint.pending, message, success));
background = checkpoint.background;
answered = checkpoint.answered;
keep_structured_answer = checkpoint.structured_answer;
resumed_answer = answer;
(
AgentHistory::from_checkpoint(checkpoint.history),
checkpoint.actions,
@@ -1734,17 +1777,22 @@ pub async fn run_agent(
*has_stream = user_wants_streaming && is_text_output;
// Always create a StreamEventProcessor for text output (use silent mode if user doesn't want streaming)
let mut stream_event_processor = if is_text_output {
if user_wants_streaming {
// Each streamed event is one line of the accumulated string.
let streamed = final_events_str.lines().count() as i32;
Some(StreamEventProcessor::new(conn, job, streamed))
// Each streamed event is one line of the accumulated string, so a segment streams on from
// where the last one stopped.
let new_stream_event_processor = |events: &str| {
if !is_text_output {
None
} else if user_wants_streaming {
Some(StreamEventProcessor::new(
conn,
job,
events.lines().count() as i32,
))
} else {
Some(StreamEventProcessor::new_silent())
}
} else {
None
};
let mut stream_event_processor = new_stream_event_processor(&final_events_str);
let chat_enabled = flow_context
.flow_status
@@ -1806,14 +1854,15 @@ pub async fn run_agent(
windmill_ai::providers::anthropic::FederatedAccessToken,
> = None;
if let (true, Some(parent_job)) = (was_resumed, parent_job) {
// Rewritten whole rather than appended: a job restarted after its worker died resumes
// from the same checkpoint and must not record anything twice.
update_flow_status_module_with_actions(db, parent_job, &actions).await?;
let answer = resumed_answer.as_ref().map(|(_, _, success)| *success);
let keep = actions.len() - usize::from(answer.is_some());
reset_flow_status_module_action_successes(db, parent_job, keep, answer).await?;
}
if let Some((pending, answer, success)) = resumed_answer {
if let Some(parent_job) = parent_job {
// Rewritten whole rather than appended: a job restarted after its worker died
// resumes from the same checkpoint and must not record the answer twice.
update_flow_status_module_with_actions(db, parent_job, &actions).await?;
set_flow_status_module_last_action_success(db, parent_job, actions.len(), success)
.await?;
}
match (pending.request, answer) {
(PendingRequest::Question { function_name }, Answer::Question(message)) => {
if let (Some(processor), Some(OpenAIContent::Text(result))) =
@@ -1870,6 +1919,7 @@ pub async fn run_agent(
)
.await?;
messages.extend(outcome.messages);
background.extend(outcome.started);
}
(PendingRequest::Question { .. }, _) => {
return Err(Error::internal_err(
@@ -1880,14 +1930,105 @@ pub async fn run_agent(
}
// Main agent loop
for i in first_iteration..max_iterations {
// Advanced by each model call that answers the run's own work; the call that follows a
// background result reuses the iteration of the answer it revisits.
let mut i = first_iteration;
while i < max_iterations {
// Check if parent was canceled — stop iterating but let current tool calls finish
if *cancel_rx.borrow() {
return Err(Error::ExecutionErr("Job cancelled".to_string()));
}
if used_structured_output_tool {
break;
if !background.is_empty() {
let finished =
take_finished_background_jobs(db, &job.workspace_id, &mut background).await?;
if !finished.is_empty() {
answered = false;
// The model is asked again, but its structured answer stands unless it gives
// another one.
keep_structured_answer |= used_structured_output_tool;
used_structured_output_tool = false;
}
for finished in finished {
messages.push(finished.message());
if let Some(parent_job) = parent_job {
set_flow_status_module_action_success_at(
db,
parent_job,
finished.job.action_index,
Some(finished.success),
)
.await?;
}
if let Some(processor) = stream_event_processor.as_ref() {
let event = StreamingEvent::ToolResult {
call_id: finished.job.call_id.clone(),
function_name: finished.job.function_name.clone(),
result: finished.content.clone(),
success: finished.success,
};
processor.send(event, &mut final_events_str).await?;
}
if let (true, Some(conversation_id)) =
(persist_output_to_conversation, conversation_id)
{
let (row, extras) = finished.chat_row();
if let Err(e) = add_message_to_conversation(
db,
&conversation_id,
Some(finished.job.job_id),
get_root_job_id(job),
&row,
MessageType::Tool,
&step_name,
finished.success,
Some(&extras),
)
.await
{
tracing::warn!(
"Failed to add tool message to conversation {conversation_id}: {e}"
);
}
}
}
}
if answered || used_structured_output_tool {
if background.is_empty() {
break;
}
// The model has answered but background jobs still run: wait for one, without a
// worker. This iteration runs again, starting with what finished.
if let Some(processor) = stream_event_processor.as_ref() {
processor
.send(StreamingEvent::BackgroundWait, &mut final_events_str)
.await?;
}
if let Some(handle) = stream_event_processor.take().and_then(|p| p.to_handle()) {
let _ = handle.await;
}
let checkpoint = AgentCheckpoint {
history: messages.checkpoint(),
actions: actions.clone(),
content: content.clone(),
usage: final_usage.clone(),
events: final_events_str.clone(),
reasoning: final_reasoning.clone(),
next_iteration: i,
pending: None,
background: background.clone(),
answered: true,
structured_answer: keep_structured_answer || used_structured_output_tool,
consumed_resume_rows: consumed_resume_rows.clone(),
};
match park_agent(db, conn, job, &checkpoint, &cancel_rx).await {
Some(parked) => return Err(parked),
None => {
stream_event_processor = new_stream_event_processor(&final_events_str);
continue;
}
}
}
compact_if_needed(
@@ -2372,7 +2513,9 @@ pub async fn run_agent(
.await?;
}
content = Some(OpenAIContent::Text(response_content.clone()));
if !keep_structured_answer {
content = Some(OpenAIContent::Text(response_content.clone()));
}
// Add assistant message to conversation if chat_input_enabled
if persist_output_to_conversation && !response_content.is_empty() {
@@ -2406,7 +2549,8 @@ pub async fn run_agent(
}
if tool_calls.is_empty() {
break;
answered = true;
continue;
} else if i == max_iterations - 1 {
#[derive(serde::Serialize)]
struct MaxIterError<'a> {
@@ -2491,6 +2635,7 @@ pub async fn run_agent(
final_content: tool_content,
used_structured_output_tool: tool_used_structured_output,
pause,
started,
} = execute_tool_calls(
tool_execution_ctx,
&tool_calls,
@@ -2544,6 +2689,12 @@ pub async fn run_agent(
content = Some(tc);
}
used_structured_output_tool = tool_used_structured_output;
background.extend(started);
if used_structured_output_tool {
// An answer, like a text one: it keeps its iteration for the call that
// revisits it once a background result arrives.
continue;
}
if let Some(mut pending) = pause {
actions.extend(pending.action.take());
@@ -2561,10 +2712,14 @@ pub async fn run_agent(
events: final_events_str,
reasoning: final_reasoning,
next_iteration: i + 1,
pending,
pending: Some(pending),
background,
answered: false,
structured_answer: keep_structured_answer,
consumed_resume_rows,
};
let parked = park_agent(db, conn, job, &checkpoint, &cancel_rx).await;
let parked =
approval_park(park_agent(db, conn, job, &checkpoint, &cancel_rx).await);
// Written once parked: the run page offers an answer only for a request that
// `resume_suspended` can take. The park is committed, so a failure here only
// costs the page its banner.
@@ -2632,6 +2787,16 @@ pub async fn run_agent(
return Ok(content);
}
}
i += 1;
}
if !background.is_empty() {
// Resumed past its iteration limit, which a linked agent edited while parked can lower:
// nothing would read the background jobs' results.
cleanup_orphaned_tool_jobs(db, &job.id, &job.workspace_id, None).await;
if let Some(parent_job) = parent_job {
fail_unfinished_action_successes(db, parent_job).await?;
}
}
if let Some(capacity_bytes) =
@@ -3319,6 +3484,8 @@ mod tests {
AgentTool {
job_token_scopes: None,
requires_approval: false,
background: false,
background_after_secs: None,
id: id.to_string(),
summary: None,
description: None,
@@ -3348,6 +3515,8 @@ mod tests {
AgentTool {
job_token_scopes: None,
requires_approval: false,
background: false,
background_after_secs: None,
id: "m".to_string(),
summary: None,
description: None,
@@ -3397,6 +3566,8 @@ mod tests {
AgentTool {
job_token_scopes: None,
requires_approval: false,
background: false,
background_after_secs: None,
id: id.to_string(),
summary: Some(summary.to_string()),
description: None,
@@ -3414,6 +3585,8 @@ mod tests {
AgentTool {
job_token_scopes: None,
requires_approval: false,
background: false,
background_after_secs: None,
id: id.to_string(),
summary: Some(summary.to_string()),
description: None,
@@ -3428,6 +3601,8 @@ mod tests {
AgentTool {
job_token_scopes: None,
requires_approval: false,
background: false,
background_after_secs: None,
id: id.to_string(),
summary: summary.map(str::to_string),
description: None,
+11
View File
@@ -5020,6 +5020,17 @@ pub async fn handle_queued_job(
let _masks = RunningJobMasks::register(job.id, &client.token);
if job.canceled_by.is_some() {
// An AI agent cancelled while parked ends here, never reaching the handler that marks its
// background calls still running as failed.
if job.kind == JobKind::AIAgent {
if let (Some(parent_job), Some(db)) = (job.parent_job.as_ref(), conn.as_sql()) {
if let Err(e) =
crate::ai::utils::fail_unfinished_action_successes(&db, parent_job).await
{
tracing::error!("Failed to close the background calls of {}: {e}", job.id);
}
}
}
return Err(Error::JsonErr(canceled_job_to_result(&job)));
}
if let Some(e) = &job.pre_run_error {
+4
View File
@@ -245,6 +245,7 @@ interface ChatState {
loadingMessages: boolean
hasMoreMessages: boolean
runningJobId?: string // the flow job the running turn follows
waitingOnBackground?: boolean // the agent answered and waits on tools running in the background
}
interface ChatMessage {
@@ -267,6 +268,9 @@ A turn goes `submitted` (the flow is queued) → `streaming` (the answer is arri
`success` or `error`. A flow that fails still completes the turn: its error is the
answer, an `assistant` message with `success: false`. `status: 'error'` (with `error`
set) means the turn could not run or be followed at all, such as a refused request.
An agent that answered while some of its tools still run in the background keeps its turn
`streaming` with `waitingOnBackground` set until one of them finishes and wakes it: its answer
is given, and those tools' messages stay `running` until their results arrive.
Methods: `sendMessage(text, { inputs?, attachments?, attachmentsInput? })`, `stop()`,
`newConversation()`, `selectConversation(id)`, `loadConversations({ page?, perPage?, kind? })`,
+51 -8
View File
@@ -328,7 +328,9 @@ class ChatImpl implements Chat {
this.#failTurn(turn, e)
} finally {
if (this.#turn === turn) this.#turn = undefined
if (this.#state.runningJobId === turn.jobId) this.#set({ runningJobId: undefined })
if (this.#state.runningJobId === turn.jobId) {
this.#set({ runningJobId: undefined, waitingOnBackground: undefined })
}
}
if (nextTurn) await this.resumeTurn(nextTurn)
}
@@ -644,15 +646,21 @@ class ChatImpl implements Chat {
patch: Partial<ToolInvocation> & { content?: string; success?: boolean }
) => {
const { content, success, ...toolPatch } = patch
// Only this turn's tool messages are pending; a provider may reuse call ids across turns.
// Only this turn's: a provider may reuse call ids across turns. Those are the pending
// ones and those past the latest question, which a mid-turn row read settles while the
// call still waits on its background job.
let asked = messages.length - 1
while (asked >= 0 && messages[asked].role !== 'user') asked--
const i = messages.findIndex(
(m) => m.pending && m.role === 'tool' && m.tool?.callId === callId
(m, j) => (m.pending || j > asked) && m.role === 'tool' && m.tool?.callId === callId
)
if (i >= 0) {
const existing = messages[i]
messages[i] = {
...existing,
content: content ?? existing.content,
// A stored row keeps its text: one that says the call went to the background is what
// its later result row is matched to.
content: existing.seq === undefined ? content ?? existing.content : existing.content,
success: success ?? existing.success,
tool: { ...existing.tool!, ...toolPatch }
}
@@ -697,7 +705,10 @@ class ChatImpl implements Chat {
})
}
}
// Anything the agent streams after it means it woke up.
let waitingOnBackground = false
for (const event of events) {
waitingOnBackground = event.type === 'background_wait'
switch (event.type) {
case 'token_delta':
turn.streamedText = true
@@ -729,7 +740,17 @@ class ChatImpl implements Chat {
break
}
}
this.#set({ messages, status: 'streaming' })
const startsWaiting = waitingOnBackground && !this.#state.waitingOnBackground
this.#set({ messages, status: 'streaming', waitingOnBackground })
// Rows are otherwise read when the turn ends, which a wait on background jobs postpones; the
// rows of the calls it waits on are written by then, and say how each went to the background.
if (startsWaiting && this.#state.history === 'server') {
this.#rowsAfterLastSeq(turn.conversationId, turn.controller.signal)
.then((rows) => {
if (this.#turnActive(turn)) this.#mergeRows(rows)
})
.catch(() => {})
}
}
async #finishTurn(turn: Turn, result: unknown, isNew: boolean): Promise<RunningTurn | undefined> {
@@ -977,8 +998,15 @@ class ChatImpl implements Chat {
for (const row of rows.map(fromRow)) {
if (known.has(row.id)) continue
known.add(row.id)
// The result of a call whose background start is already held is that call's own row:
// matched by tool name, it would take a later call of the same tool's place.
const backgroundResult =
row.role === 'tool' &&
row.jobId !== undefined &&
messages.some((m) => m.seq !== undefined && m.jobId === row.jobId && BACKGROUND_START.test(m.content))
let i = messages.findIndex(
(m) =>
!backgroundResult &&
m.seq === undefined &&
m.role === row.role &&
(m.content === row.content || (row.tool !== undefined && m.tool?.name === row.tool.name))
@@ -996,9 +1024,13 @@ class ChatImpl implements Chat {
}
if (i >= 0) {
const m = messages[i]
// A row saying the call went to the background, read after its result streamed in,
// does not set it running again nor undo how it went.
const outrun = row.tool?.status === 'running' && !!m.tool && m.tool.status !== 'running'
messages[i] = {
...row,
id: m.id,
success: outrun ? m.success : row.success,
reasoning: m.reasoning ?? row.reasoning,
// The stream's call wins where it has a value; a stream cut short leaves gaps the row fills.
tool:
@@ -1007,7 +1039,7 @@ class ChatImpl implements Chat {
...m.tool,
arguments: m.tool.arguments ?? row.tool?.arguments,
result: m.tool.result ?? row.tool?.result,
status: row.tool?.status ?? m.tool.status
status: outrun ? m.tool.status : (row.tool?.status ?? m.tool.status)
}
: row.tool
}
@@ -1135,10 +1167,21 @@ function latestSeq(messages: readonly ChatMessage[]): number {
return messages.reduce((newest, m) => (m.seq !== undefined && m.seq > newest ? m.seq : newest), 0)
}
const BACKGROUND_START = /^(?:Started (.+) in the background|Moved (.+) to the background(?: after \d+s)?)$/
function fromRow(row: FlowConversationMessage): ChatMessage {
// A call that went to the background: its result comes as a later row of the same job.
const background =
row.message_type === 'tool'
? BACKGROUND_START.exec(row.content)
?.slice(1)
.find((name) => name !== undefined)
: undefined
const toolName =
row.message_type === 'tool'
? /^Used (.+) tool$/.exec(row.content)?.[1] ?? /^Error executing (.+)$/.exec(row.content)?.[1]
? /^Used (.+) tool$/.exec(row.content)?.[1] ??
/^Error executing (.+)$/.exec(row.content)?.[1] ??
background
: undefined
const success = row.success ?? true
return {
@@ -1160,7 +1203,7 @@ function fromRow(row: FlowConversationMessage): ChatMessage {
tool: toolName
? {
name: toolName,
status: success ? 'success' : 'error',
status: background ? 'running' : success ? 'success' : 'error',
arguments: row.tool_arguments ?? undefined,
result: row.tool_result ?? undefined
}
+3
View File
@@ -12,8 +12,11 @@ export type AgentStreamEvent =
result: string
success: boolean
}
/** The agent answered and waits on tools still running in the background. */
| { type: 'background_wait' }
const KNOWN_TYPES = new Set([
'background_wait',
'token_delta',
'reasoning_token_delta',
'tool_call',
+6
View File
@@ -82,6 +82,12 @@ export interface ChatState {
hasMoreMessages: boolean
/** The flow job the running turn follows, once it is known. */
runningJobId?: string
/**
* The running turn's AI agent has answered and waits, without a worker, on tools still running
* in the background: nothing streams until one finishes. Set by its `background_wait` event,
* cleared by the next event and when the turn ends.
*/
waitingOnBackground?: boolean
}
export type TokenSource = string | (() => string | Promise<string>)
+204
View File
@@ -26,6 +26,34 @@ function options(extra: Partial<ChatOptions>, fetch: ChatOptions['fetch']): Chat
}
describe('createChat with local history', () => {
test('marks a turn waiting on background tools until the agent streams again', async () => {
const { fetch } = fetchMock(run, (c) =>
c.url.pathname === streamPath
? sseTimed([
{
type: 'update',
new_result_stream: ndjson({ type: 'token_delta', content: 'On it' }, { type: 'background_wait' }),
stream_offset: 1
},
200,
{
type: 'update',
new_result_stream: ndjson({ type: 'token_delta', content: ' and done' }),
stream_offset: 2,
completed: true,
only_result: { windmill_chat_answer: 'On it and done' }
}
])
: undefined
)
const chat = createChat(options({}, fetch))
const sent = chat.sendMessage('go')
await until(() => chat.getState().waitingOnBackground === true, 'the background wait')
expect(chat.getState().status).toBe('streaming')
await sent
expect(chat.getState().waitingOnBackground).toBeFalsy()
})
test('streams text and tool calls, then finalizes and persists the turn', async () => {
const storage = memoryStorage()
const { fetch, calls } = fetchMock(run, (c) =>
@@ -955,6 +983,182 @@ describe('createChat with server history', () => {
])
})
test('reads a row of a call that went to the background as a call of that tool', async () => {
const { fetch } = fetchMock((c) =>
c.url.pathname.endsWith('/messages')
? json([
messageRow(1, 'user', 'go'),
messageRow(2, 'tool', 'Started report in the background', { job_id: 'bg-1' }),
messageRow(3, 'tool', 'Moved export to the background after 5s', { job_id: 'bg-2' })
])
: undefined
)
const chat = createChat(options({}, fetch))
await chat.selectConversation('conv')
expect(chat.getState().messages.map((m) => m.tool?.name)).toEqual([undefined, 'report', 'export'])
})
test.each([true, false])('a row read after a background result streamed in keeps how it went (success: %p)', async (success) => {
let reads = 0
const { fetch } = fetchMock(
run,
(c) =>
c.url.pathname === streamPath
? sseTimed([
{
type: 'update',
new_result_stream: ndjson(
{ type: 'tool_call', call_id: 'c1', function_name: 'export' },
{ type: 'background_wait' }
),
stream_offset: 2
},
50,
{
type: 'update',
new_result_stream: ndjson({
type: 'tool_result',
call_id: 'c1',
function_name: 'export',
result: '"ok"',
success
}),
stream_offset: 3
},
600,
{
type: 'update',
new_result_stream: ndjson({ type: 'token_delta', content: 'Done' }),
stream_offset: 4,
completed: true,
only_result: { output: 'Done', messages: [] }
}
])
: undefined,
async (c) => {
if (c.method !== 'GET' || !c.url.pathname.endsWith('/messages')) return undefined
// The read the wait starts answers after the result streamed in.
if (reads++ === 0) await new Promise((resolve) => setTimeout(resolve, 300))
return json([
messageRow(1, 'user', 'go'),
messageRow(2, 'tool', 'Moved export to the background after 5s', { job_id: 'bg-1' })
])
}
)
const chat = createChat(options({}, fetch))
const sent = chat.sendMessage('go')
await until(() => reads > 0, 'the read')
await new Promise((resolve) => setTimeout(resolve, 450))
const call = chat.getState().messages.find((m) => m.role === 'tool')
expect([call?.content, call?.tool?.status, call?.success]).toEqual([
'Moved export to the background after 5s',
success ? 'success' : 'error',
success
])
await sent
})
test('a background result streams onto the card its row was read into while waiting', async () => {
let waited = false
const { fetch } = fetchMock(
run,
(c) =>
c.url.pathname === streamPath
? sseTimed([
{
type: 'update',
new_result_stream: ndjson(
{ type: 'tool_call', call_id: 'c1', function_name: 'export' },
{ type: 'background_wait' }
),
stream_offset: 2
},
100,
{
type: 'update',
new_result_stream: ndjson(
{ type: 'tool_result', call_id: 'c1', function_name: 'export', result: '"ok"', success: true },
{ type: 'token_delta', content: 'Done' }
),
stream_offset: 4,
completed: true,
only_result: { output: 'Done', messages: [] }
}
])
: undefined,
(c) => {
if (c.method !== 'GET' || !c.url.pathname.endsWith('/messages')) return undefined
if (!waited) {
waited = true
return json([
messageRow(1, 'user', 'go'),
messageRow(2, 'tool', 'Moved export to the background after 5s', { job_id: 'bg-1' })
])
}
return json([
messageRow(1, 'user', 'go'),
messageRow(2, 'tool', 'Moved export to the background after 5s', { job_id: 'bg-1' }),
messageRow(3, 'tool', 'Used export tool', { job_id: 'bg-1', tool_result: '"ok"' }),
messageRow(4, 'assistant', 'Done')
])
}
)
const chat = createChat(options({}, fetch))
await chat.sendMessage('go')
// The call's card, filled by the stream and still worded as its row, then the result's row,
// which a display folds into that card by job.
const tools = chat.getState().messages.filter((m) => m.role === 'tool')
expect(tools.map((m) => [m.content, m.tool?.callId, m.tool?.result])).toEqual([
['Moved export to the background after 5s', 'c1', '"ok"'],
['Used export tool', undefined, '"ok"']
])
})
test('repeated background calls of one tool each keep their own result', async () => {
const call = (id: string) => ({ type: 'tool_call', call_id: id, function_name: 'export' })
const result = (id: string, r: string) => ({
type: 'tool_result',
call_id: id,
function_name: 'export',
result: r,
success: true
})
const { fetch } = fetchMock(
run,
(c) =>
c.url.pathname === streamPath
? sse([
{
type: 'update',
new_result_stream: ndjson(call('c1'), result('c1', '"A"'), call('c2'), result('c2', '"B"')),
stream_offset: 4,
completed: true,
only_result: { output: 'Done', messages: [] }
}
])
: undefined,
(c) =>
c.method === 'GET' && c.url.pathname.endsWith('/messages')
? json([
messageRow(1, 'user', 'go'),
messageRow(2, 'tool', 'Started export in the background', { job_id: 'bg-1' }),
messageRow(3, 'tool', 'Used export tool', { job_id: 'bg-1', tool_result: '"A"' }),
messageRow(4, 'tool', 'Started export in the background', { job_id: 'bg-2' }),
messageRow(5, 'tool', 'Used export tool', { job_id: 'bg-2', tool_result: '"B"' })
])
: undefined
)
const chat = createChat(options({}, fetch))
await chat.sendMessage('go')
const tools = chat.getState().messages.filter((m) => m.role === 'tool')
expect(tools.map((m) => [m.jobId, m.tool?.callId, m.tool?.result])).toEqual([
['bg-1', 'c1', '"A"'],
['bg-1', undefined, '"A"'],
['bg-2', 'c2', '"B"'],
['bg-2', undefined, '"B"']
])
})
test('a message refused because a turn is running leaves nothing behind and names that turn', async () => {
const { fetch } = fetchMock(
(c) =>
+7 -2
View File
File diff suppressed because one or more lines are too long
+32
View File
@@ -218,6 +218,38 @@ Both live in `windmill-worker/src/ai/pause.rs`.
inside a loop or branch is answered from its graph node. The chat transcript of a chat flow
records neither the request nor its answer.
## Background tools
A `flowmodule` tool with `background: true` is answered as soon as its job is queued ("started in
the background as job X"), and the agent keeps working. The job's result reaches the model later as
a user message tagged with the call's `tool_call` action, since the call itself was already
answered. Its outcome in `agent_actions_success` is `null` until then, set in place by position.
With `background_after_secs: N` instead, the call is waited on like a foreground one, and still
unfinished N seconds after it was queued it is answered the same way ("not finished after Ns") and
continues in the background. Such a job never runs in the agent's own process, which could not release the worker
while it ran, so it needs another worker free: with none, it only starts once the agent parks.
- **Parking.** The loop checks for finished jobs before each model call. When the model answers
with jobs still running, the agent parks like it does for an approval (`suspend = 1`, the same
checkpoint table) instead of ending, with the jobs in the checkpoint row's `waiting_on`. The park
first checks, under its queue row lock, whether one already finished, and delivers it instead.
- **Waking.** Every monitor pass zeroes the `suspend` of a parked agent with a completed job in
`waiting_on`, so a result waits at most one pass (10s by default) once its job ends. Not
`record_child_completion`, although tool jobs are the agent's children: when it wakes a parent it
clears all of its pending ids, which fits a workflow-as-code round but would drop the agent's
other background jobs.
- **Iterations.** The model call that follows a background result reuses the iteration of the
answer it revisits, so waiting never runs an agent out of iterations. An agent that does run out
cancels the jobs still running. With structured output, an earlier structured answer stands unless
the model gives another.
- **With an approval.** While an approval is open, `waiting_on` is empty: results wait for the
answer. An answer the row accepts while the agent waits on background jobs only is marked read on
resume, so it cannot answer a later request.
- **Limits.** A nested agent runs its background tools in the foreground, since it cannot park. The
job is never reserved for the agent's own worker. Like a foreground tool, a job that never
finishes (a tag no worker serves) keeps the agent waiting, here parked and re-checked hourly.
## Drafts
The agent editor edits the resource through a **per-user resource draft** (`draft` table,
@@ -942,16 +942,16 @@
function getAgentActionContent(agentResult: any, actionIndex: number): unknown | undefined {
if (!agentResult?.messages || !Array.isArray(agentResult.messages)) return undefined
let agentActionIdx = 0
for (const m of agentResult.messages) {
if (m.agent_action) {
if (agentActionIdx === actionIndex) {
return m.content
}
agentActionIdx++
}
}
return undefined
// A background tool's result arrives later as a user message tagged with the action its
// job started: it is not an action of its own, and it answers that one.
const isBackgroundResult = (m: any) => m.role === 'user' && m.agent_action?.type === 'tool_call'
const tagged = agentResult.messages.filter((m: any) => m.agent_action)
const message = tagged.filter((m: any) => !isBackgroundResult(m))[actionIndex]
const jobId = message?.agent_action?.job_id
const result = jobId
? tagged.find((m: any) => isBackgroundResult(m) && m.agent_action.job_id === jobId)
: undefined
return (result ?? message)?.content
}
function isSuccess(arg: any): boolean | undefined {
+5 -1
View File
@@ -238,6 +238,10 @@ function isWellFormedEvent(event: AgentStreamEvent): boolean {
if (event.type === 'token_delta' || event.type === 'reasoning_token_delta') {
return typeof event.content === 'string'
}
// Says the agent waits on its background tools, which the run page shows on their rows.
if (event.type === 'background_wait') {
return false
}
return (
typeof event.call_id === 'string' &&
event.call_id !== '' &&
@@ -286,7 +290,7 @@ export function advanceAgentStream(
}
const stream: AgentStream = { ...previous.stream, entries: [...previous.stream.entries] }
for (const event of parseStreamEvents(raw.slice(previous.consumed, complete))) {
if (!isWellFormedEvent(event)) {
if (!isWellFormedEvent(event) || event.type === 'background_wait') {
continue
}
if (event.type === 'token_delta') {
@@ -345,8 +345,11 @@
// Also shown for a run held by another tab, labeled with where it is: the
// dots say a turn is in flight even before the reader reaches the footer
// note. Remote runs pause nothing and offer no Stop — this tab can't cancel.
const showTypingIndicator = $derived(chatHost.loading || chatHost.runHeldElsewhere)
// note. Remote runs pause nothing and offer no Stop — this tab can't cancel. A turn
// that answered and waits on background work writes nothing, so it shows none.
const showTypingIndicator = $derived(
(chatHost.loading && !chatHost.backgroundWait) || chatHost.runHeldElsewhere
)
// The manual `@` context-picker button. Shown in SCRIPT/FLOW (workspace items +
// code blocks) and APP (datatables, frontend files). Hidden in GLOBAL — there
@@ -55,6 +55,9 @@ export interface ChatViewHost {
* pins away from the navigated one. */
readonly operatingWorkspace: string | undefined
loading: boolean
/** The turn has answered and waits on work running in the background: still `loading`, so
* Stop stays, but nothing is being written. */
readonly backgroundWait?: boolean
/** A send waiting for the chat's transcript to be restored; queue behind it. */
sendPending: boolean
/** A turn this tab can neither follow nor stop, held by another tab on the same chat. */
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
@@ -66,6 +66,10 @@
bind:toolRequiresApproval={
() => tool.requires_approval, (v) => (tool.requires_approval = v || undefined)
}
bind:toolBackground={() => tool.background, (v) => (tool.background = v || undefined)}
bind:toolBackgroundAfterSecs={
() => tool.background_after_secs, (v) => (tool.background_after_secs = v)
}
{siblingToolNames}
/>
{:else if isMcpTool(tool)}
@@ -26,6 +26,9 @@
import { getFailureStepPropPicker, getStepPropPicker } from '../previousResults'
import { deepEqual } from 'fast-equals'
import Section from '$lib/components/Section.svelte'
import SecondsInput from '$lib/components/common/seconds/SecondsInput.svelte'
import ToggleButtonGroup from '$lib/components/common/toggleButton-v2/ToggleButtonGroup.svelte'
import ToggleButton from '$lib/components/common/toggleButton-v2/ToggleButton.svelte'
import Button from '$lib/components/common/button/Button.svelte'
import FlowPathViewer from './FlowPathViewer.svelte'
@@ -72,7 +75,7 @@
signDebugRequest,
getDebugErrorMessage
} from '$lib/components/debug'
import { Bug, Hand, Terminal } from 'lucide-svelte'
import { Bug, Hand, Hourglass, Terminal } from 'lucide-svelte'
import Popover from '$lib/components/Popover.svelte'
import { sendUserToast } from '$lib/utils'
import { useOperatingWorkspace } from '$lib/components/operatingWorkspace.svelte'
@@ -137,6 +140,10 @@
toolDescription?: string | undefined
/** The tool's `requires_approval` setting; only an agent tool has one. */
toolRequiresApproval?: boolean
/** The tool's `background` setting; only an agent tool has one. */
toolBackground?: boolean
/** The tool's `background_after_secs` setting; only an agent tool has one. */
toolBackgroundAfterSecs?: number
siblingToolNames?: string[]
}
@@ -158,9 +165,21 @@
flowModuleSchemaMap = undefined,
toolDescription = $bindable(undefined),
toolRequiresApproval = $bindable(),
toolBackground = $bindable(),
toolBackgroundAfterSecs = $bindable(),
siblingToolNames = undefined
}: Props = $props()
const DEFAULT_BACKGROUND_AFTER_SECS = 30
type BackgroundMode = 'never' | 'after' | 'always'
let backgroundMode: BackgroundMode = $derived(
toolBackground ? 'always' : toolBackgroundAfterSecs != undefined ? 'after' : 'never'
)
function setBackgroundMode(mode: BackgroundMode) {
toolBackground = mode === 'always'
toolBackgroundAfterSecs = mode === 'after' ? DEFAULT_BACKGROUND_AFTER_SECS : undefined
}
setAiDecisionStep({
get id() {
return flowModule.value.type === 'aidecision' && !isAgentTool ? flowModule.id : undefined
@@ -1160,12 +1179,22 @@
{#if isAgentTool}
<Tab value="tool_settings" label="Tool settings">
{#snippet extra()}
{#if toolRequiresApproval}
{#if toolRequiresApproval || backgroundMode !== 'never'}
<div class="flex flex-row items-center gap-1 text-secondary">
<Popover notClickable>
<Hand size={12} />
{#snippet text()}Require approval · On{/snippet}
</Popover>
{#if toolRequiresApproval}
<Popover notClickable>
<Hand size={12} />
{#snippet text()}Require approval · On{/snippet}
</Popover>
{/if}
{#if backgroundMode !== 'never'}
<Popover notClickable>
<Hourglass size={12} />
{#snippet text()}Run in background · {backgroundMode === 'always'
? 'Always'
: `After ${toolBackgroundAfterSecs}s`}{/snippet}
</Popover>
{/if}
</div>
{/if}
{/snippet}
@@ -1418,7 +1447,7 @@
</Section>
</div>
{:else if visibleSelected === 'tool_settings' && isAgentTool}
<div class="flex-1 overflow-auto p-4">
<div class="flex flex-1 flex-col gap-6 overflow-auto p-4">
<Section label="Approval">
<Toggle
size="xs"
@@ -1434,6 +1463,48 @@
}}
/>
</Section>
<Section
label="Run in background"
tooltip="A call in the background answers the agent with its job id, and the agent keeps working. The result reaches it in a later message. If it finishes its answer first, it waits for the result without holding a worker."
>
<div class="flex flex-col gap-2">
<ToggleButtonGroup
disabled={noEditor}
selected={backgroundMode}
onSelected={setBackgroundMode}
>
{#snippet children({ item })}
<ToggleButton
value="never"
label="Never"
tooltip="The agent waits for the result."
{item}
/>
<ToggleButton
value="after"
label="After a delay"
tooltip="The agent waits for the result up to a delay, then the call continues in the background."
{item}
/>
<ToggleButton
value="always"
label="Always"
tooltip="The agent is answered as soon as the job starts."
{item}
/>
{/snippet}
</ToggleButtonGroup>
{#if backgroundMode === 'after'}
<SecondsInput
disabled={noEditor}
bind:seconds={
() => toolBackgroundAfterSecs ?? DEFAULT_BACKGROUND_AFTER_SECS,
(v) => (toolBackgroundAfterSecs = Math.max(1, v ?? 1))
}
/>
{/if}
</div>
</Section>
</div>
{:else if visibleSelected === 'advanced'}
<FlowRunSettings
@@ -146,6 +146,20 @@ export type DisplayLookups = {
revealed?: (message: ChatMessage) => Revealed | undefined
/** The approval or question the running turn's agent waits on. */
request?: AgentRequest
/** The running turn's agent answered and waits on tools running in the background: the calls
* it shows as running are those. */
backgroundWait?: boolean
}
/** The row the worker writes when a call goes to the background. Its result is a later row of
* the same job, which this call's card shows in its place. */
const BACKGROUND_START =
/^(?:Started .+ in the background|Moved .+ to the background( after (\d+)s)?)$/
/** How long a call ran before it moved to the background, for its card. */
function movedAfter(content: string): string {
const secs = BACKGROUND_START.exec(content)?.[2]
return secs ? ` (took over ${secs}s)` : ''
}
const QUESTION_CALL = /^ask_question(_\d+)?$/
@@ -260,7 +274,66 @@ export function toDisplayMessages(
}
const request = lookups.request
let requestShown = false
const shown = messages.flatMap((message, i): DisplayMessage[] => {
// A background call's result row, by the index of the row that started it.
const backgroundResults = new Map<number, number>()
const backgroundStarts = new Map<string, number>()
messages.forEach((message, i) => {
if (message.role !== 'tool' || !message.jobId) return
const start = backgroundStarts.get(message.jobId)
if (start !== undefined) backgroundResults.set(start, i)
else if (BACKGROUND_START.test(message.content)) backgroundStarts.set(message.jobId, i)
})
const shownElsewhere = new Set(backgroundResults.values())
const shown = messages.flatMap((row, i): DisplayMessage[] => {
// Where results came back, a bubble says why the agent goes on, as the session chat's
// auto-resume does; the results themselves are on their calls' cards.
if (shownElsewhere.has(i)) {
if (shownElsewhere.has(i - 1)) return []
let count = 1
while (shownElsewhere.has(i + count)) count++
return [
{
role: 'user',
index: -1,
synthetic: true,
content:
count === 1
? 'A background job just finished.'
: `${count} background jobs just finished.`
}
]
}
const resultRow = backgroundResults.get(i)
const started =
row.role === 'tool' && resultRow === undefined && BACKGROUND_START.test(row.content)
// Its result streamed in ahead of the row that stores it: shown as that row will be.
const streamedBack = started && !!row.tool && row.tool.status !== 'running'
// The result in the call's place, with the thinking that led to the call, stored on the
// row that started it.
const message =
resultRow !== undefined
? { ...messages[resultRow], id: row.id, reasoning: row.reasoning }
: streamedBack
? {
...row,
content:
row.success === false
? `Error executing ${row.tool!.name}`
: `Used ${row.tool!.name} tool`
}
: row
// Started in the background, its result not in yet. Once the turn that started it is no
// longer running, it never will be: its agent was stopped and its job with it. A later turn
// running does not bring it back.
const backgroundStart = started && !streamedBack
const ownTurnRunning = busy && (i > latestUser || !!row.pending)
const backgroundUnfinished = backgroundStart && !ownTurnRunning
const inBackground =
(backgroundStart && ownTurnRunning) ||
(message.role === 'tool' &&
!!lookups.backgroundWait &&
!!message.pending &&
message.tool?.status === 'running')
switch (message.role) {
case 'user': {
const index = userIndex++
@@ -305,7 +378,9 @@ export function toDisplayMessages(
// worker stored them on the row. A row stored without them shows its name and job.
const toolName = message.tool?.name
const parameters = parseToolPayload(message.tool?.arguments)
const result = parseToolPayload(message.tool?.result)
// A call started in the background stores, as its result, what the model was told
// about it; its card shows the real result once it comes back.
const result = backgroundStart ? undefined : parseToolPayload(message.tool?.result)
const failed = message.success === false
// A call the turn finished without, stopped or lost before its row was written. The
// call a structured answer streams as never gets a result of its own, the answer
@@ -313,7 +388,7 @@ export function toDisplayMessages(
const unfinished =
message.tool &&
!message.pending &&
!message.content &&
(!message.content || backgroundUnfinished) &&
!STRUCTURED_OUTPUT_CALL.test(message.tool.name)
? `${message.tool.name} did not finish`
: undefined
@@ -322,7 +397,12 @@ export function toDisplayMessages(
tool_call_id: message.id,
// The card's header is the row's text, which the server only words once the
// tool has returned; until then the row says what is running.
content: message.content || unfinished || (toolName ? `Running ${toolName}` : ''),
content: inBackground
? `Running ${toolName ?? 'tool'} in the background${movedAfter(message.content)}`
: (backgroundUnfinished && unfinished) ||
message.content ||
unfinished ||
(toolName ? `Running ${toolName}` : ''),
// Withheld for the copilot's two plan-mode names: `toolName` is what makes
// ToolExecutionDisplay render a plan card, and an agent tool that happened to
// share one would silently become one.
@@ -331,7 +411,7 @@ export function toDisplayMessages(
result,
showDetails: parameters !== undefined || result !== undefined,
error: failed ? (toolErrorText(result) ?? message.content) : unfinished,
isLoading: message.pending && message.tool?.status === 'running',
isLoading: inBackground || (message.pending && message.tool?.status === 'running'),
jobId: message.jobId
}
if (request && i > latestUser && call.isLoading && holds(request, toolName)) {
@@ -538,7 +618,8 @@ export class FlowChatViewHost implements ChatViewHost, DraftSender<ComposerAttac
toDisplayMessages(this.#state.messages, isBusy(this.#state.status), this.#stoppedTurns, {
workspace: this.#options.workspace?.(),
revealed: (message) => this.#revealed[message.id],
request: this.#agentRequest?.action
request: this.#agentRequest?.action,
backgroundWait: this.backgroundWait
})
)
@@ -576,6 +657,9 @@ export class FlowChatViewHost implements ChatViewHost, DraftSender<ComposerAttac
get loading(): boolean {
return isBusy(this.#state.status)
}
get backgroundWait(): boolean {
return isBusy(this.#state.status) && !!this.#state.waitingOnBackground
}
// No transcript restore to wait for: a flow chat renders from its rows.
sendPending = false
runHeldElsewhere = false
@@ -352,6 +352,139 @@ describe('toDisplayMessages', () => {
})
})
it('shows a background call as one card, running until its result row comes in', () => {
const started = message({
id: 'started',
role: 'tool',
content: 'Started report in the background',
jobId: 'bg-job',
reasoning: 'This takes a while',
tool: { name: 'report', status: 'running' }
})
const later = [
message({ role: 'assistant', content: 'On it' }),
message({ role: 'user', content: 'meanwhile' }),
message({
role: 'tool',
content: 'Used report tool',
jobId: 'bg-job',
tool: { name: 'report', status: 'success', result: '"42 rows"' }
})
]
expect(toDisplayMessages([started, later[0]], true)[1]).toMatchObject({
tool_call_id: 'started',
content: 'Running report in the background',
isLoading: true
})
// No turn running any more: its agent was stopped, and its job with it.
expect(toDisplayMessages([started, later[0]])[1]).toMatchObject({
content: 'report did not finish',
error: 'report did not finish',
isLoading: false
})
// Nor when a later turn runs.
expect(toDisplayMessages([started, later[0], later[1]], true)[1]).toMatchObject({
content: 'report did not finish',
isLoading: false
})
const display = toDisplayMessages([started, ...later])
expect(display.filter((m) => m.role === 'tool')).toHaveLength(1)
// The thinking that led to the call stays beside it.
expect(display[0]).toMatchObject({ role: 'assistant', reasoning: 'This takes a while' })
expect(display[1]).toMatchObject({
tool_call_id: 'started',
content: 'Used report tool',
result: '42 rows',
isLoading: false
})
// Where the result came back, what the session chat says when it resumes on one.
expect(display.at(-1)).toMatchObject({
role: 'user',
synthetic: true,
content: 'A background job just finished.'
})
})
it('counts the background results that came back together', () => {
const call = (job: string, content: string) =>
message({
role: 'tool',
content,
jobId: job,
tool: { name: 'report', status: content.startsWith('Used') ? 'success' : 'running' }
})
const display = toDisplayMessages([
call('a', 'Started report in the background'),
call('b', 'Moved report to the background after 5s'),
message({ role: 'assistant', content: 'Both started' }),
call('a', 'Used report tool'),
call('b', 'Used report tool'),
message({ role: 'assistant', content: 'Both done' })
])
expect(display.map((m) => m.content)).toEqual([
'Used report tool',
'Used report tool',
'Both started',
'2 background jobs just finished.',
'Both done'
])
})
it('says how long a call ran before it moved to the background, without what the model was told', () => {
const moved = message({
role: 'tool',
content: 'Moved export to the background after 5s',
jobId: 'bg-job',
tool: { name: 'export', status: 'running', result: 'Not finished after 5s, so it continues…' }
})
const [card] = toDisplayMessages([moved], true)
expect(card).toMatchObject({
content: 'Running export in the background (took over 5s)',
isLoading: true
})
expect((card as { result?: unknown }).result).toBeUndefined()
// Its result streamed in before its row: the card shows it, and the row folds in later.
const streamed = {
...moved,
tool: { name: 'export', status: 'success' as const, result: '"ok"' }
}
expect(toDisplayMessages([streamed], true)[0]).toMatchObject({
content: 'Used export tool',
result: 'ok',
isLoading: false
})
const display = toDisplayMessages([
streamed,
message({ role: 'assistant', content: 'Started it' }),
message({
role: 'tool',
content: 'Used export tool',
jobId: 'bg-job',
tool: { name: 'export', status: 'success', result: '"ok"' }
})
])
expect(display.map((m) => m.content)).toEqual([
'Used export tool',
'Started it',
'A background job just finished.'
])
})
it('shows the calls still running as in the background while the agent waits on them', () => {
const running = message({
role: 'tool',
pending: true,
tool: { callId: 'c1', name: 'report', status: 'running' }
})
expect(toDisplayMessages([running])[0]).toMatchObject({ content: 'Running report' })
expect(
toDisplayMessages([running], true, new Set(), { backgroundWait: true })[0]
).toMatchObject({
content: 'Running report in the background',
isLoading: true
})
})
// A row stored before the worker kept the call has only its sentence: the card names the
// tool and links its job, and has no details to open.
it('builds a tool card from the row alone, with its job link', () => {
+2 -2
View File
@@ -66,8 +66,8 @@ export type GraphModuleState = {
isListJob?: boolean
skipped?: boolean
agent_actions?: FlowStatusModule['agent_actions']
/** Positionally aligned with `agent_actions`: every push of an action appends one entry, so a
* missing entry means that action has not finished yet. */
/** Positionally aligned with `agent_actions`: every action gets one entry, so a missing or
* `null` entry (a background tool call still running) means it has not finished yet. */
agent_actions_success?: FlowStatusModule['agent_actions_success']
script_hash?: string
workflow_as_code_status?: WorkflowStatus
@@ -382,7 +382,7 @@
return
count++
const success = agentState?.agent_actions_success?.[index]
if (success === undefined) pending++
if (success == null) pending++
else if (!success) failed++
})
if (count === 0) return undefined
@@ -60,14 +60,14 @@
actions.forEach((action, index) => {
const entry = tally.get(action.type) ?? { count: 0, pending: 0 }
entry.count++
// A missing entry in the parallel success array means the action is still running.
// A missing or null entry in the parallel success array means the action is still running.
const success = agentState?.agent_actions_success?.[index]
if (success === undefined) entry.pending++
if (success == null) entry.pending++
else if (!success) failed++
tally.set(action.type, entry)
if (action.type !== 'message') {
calls++
if (success === undefined) pending++
if (success == null) pending++
}
})
const detail: string[] = []
+9
View File
@@ -1052,6 +1052,13 @@ components:
requires_approval:
type: boolean
description: "Flowmodule and mcp tools only. Each call waits for a person to approve or reject it from the flow's run page or chat, without the agent holding a worker. Calls in the same turn that do not require approval run right away. A rejected call, or one nobody answers within 30 minutes, is reported to the AI as not run."
background:
type: boolean
description: "Flowmodule tools only. The agent is answered as soon as the tool's job starts and keeps working; the job's result reaches it in a later message. If the agent finishes its answer first, it stops without holding a worker until the job is done, then continues with the result."
background_after_secs:
type: integer
minimum: 1
description: "Flowmodule tools only, with background unset. The agent waits up to this many seconds for the tool's job and gets its result as usual; a job still running then continues as a background one, its result reaching the agent in a later message."
value:
$ref: '#/components/schemas/ToolValue'
required:
@@ -1606,6 +1613,8 @@ components:
- type
agent_actions_success:
type: array
description: The outcome of each agent action, by position. null while a background tool call is still running.
items:
type: boolean
nullable: true
required: [type]
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
+6 -1
View File
@@ -72,7 +72,12 @@ agent or AI decision), `mcp` for an MCP server tool, `websearch` for web search,
built-in `ask_question` tool (the agent stops, without holding a worker, until someone answers from
the run page or the chat; optional `value.timeout` in seconds, default 1800). Set
`requires_approval: true` on a `flowmodule` or `mcp` tool to have a person approve or reject each of
its calls before it runs.
its calls before it runs. Set `background: true` on a `flowmodule` tool that runs long: the agent is
answered as soon as its job starts and keeps working, and gets the result in a later message,
waiting for it without holding a worker if it finishes its answer first. Prefer it to having the
agent start jobs through a script or `runScriptByPath`, whose results it cannot wait on. For a tool
that is usually quick but sometimes long, set `background_after_secs: <seconds>` instead: the agent
waits that long for the result, then the call continues in the background.
```json
{