starred/windmill
1
0
Fork 0
mirror of https://github.com/windmill-labs/windmill.git synced 2026-10-03 00:02:08 +00:00
Code Issues Packages Projects Releases Wiki Activity

feat: make hub integrations usable as examples in global AI chat (#10599)

Browse Source 
* feat: make hub integrations usable as examples in global chat

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* fix: keep web search guidance in sync with provider capability

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* fix: resync web search guidance before the request is built

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* fix: carry loop web search availability into every prompt rebuild

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* fix: drop web search guidance on the completions api fallback

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* docs: correct onBeforeIteration contract for the fallback re-entry

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* feat: give the ai chat hub script descriptions and integration metadata

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* fix: treat authored hub metadata as evidence the scripts were curated

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* fix: treat hub curated as three-state and speak only for a stated true

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* test: let the benchmark hub serve integration metadata

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* test: benchmark the hub tool against a real integration's scripts and metadata

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* test: cover two more real integrations where the scripts already answer

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* test: report metadata_source from whether the fixture has authored meta

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* fix: parse the hub resource type schema instead of relaying it as a string

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* fix: match integration suggestions on slug words instead of substrings

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* fix: search the integration a query names outright instead of ranking past it

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* fix: reject hub integration slugs that would re-target the proxied request

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* fix: add a mentioned integration's hits instead of filtering the search to it

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* fix: keep named-integration hits under the content cap and unmangle fixture placeholders

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* fix: cut the middle of a capped hub result instead of repeating its tail

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* fix: cap hub results by keeping the best of the ranked and named hits

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* fix: keep a named integration that ranking placed below the content cap

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* fix: let plan mode use the hub integration lookup

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* feat: mark which hub integrations carry authored provider knowledge

The hub flags the integrations whose document holds hand-written provider
knowledge, so a caller can tell before spending a call on the metadata
endpoint: 18 of ~216 qualify, and for the rest the endpoint returns what
was inferred from the same scripts a search already hands back.

Carry the flag onto search results, where the model first meets a slug, so
get_hub_integration is aimed at the few instead of guessed at. A hub that
predates the flag omits it and nothing is marked.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* fix: stop the documented mark reading as a reason not to call

The flag says the hub additionally holds provider knowledge checked against
the live API. It is not a signal to skip the lookup elsewhere: that call
still returns the resource type and the usage-ranked examples, neither of
which a search result carries, and neither guessable.

The prompt said to read a script instead when the mark is absent, which
traded those for a guess on the ~198 undocumented integrations.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* fix: keep the benchmark's documented flag true to what it serves

The eval hub reported Baremetrics and Holded as undocumented while its
metadata endpoint handed back their authored notes, so a case could teach
the model the flag means nothing. Derive it from both fixture sources, and
pin the agreement.

Also trims the documentedIntegrations comment to the four lines AGENTS.md
allows, keeping the case-folding constraint.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* fix: keep the benchmark hub mostly undocumented, like the real one

Adding holded and baremetrics to the documented set resolved the fixture's
contradiction the wrong way: it left five of six integrations marked, against
the live hub's 18 of ~216, and claimed authored notes for two the hub reports
as having none. Drop the notes instead. Their auth and endpoints are in their
shipped scripts, which is what the cases that use them are about.

Also drops the last two places still describing the flag as a reason to spend
or skip the metadata call.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* refactor: let the model name the integration instead of guessing it from the query

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* refactor: word web search guidance conditionally instead of tracking provider capability

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* fix: open the hub metadata route to job tokens and share one integrations fetch

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* fix: record a copied hub script's source where write_script keeps it

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* test: pin that a query narrows to the integration it was given

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* fix: match integration suggestions on the name the hub curates

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* fix: name an integration the hub leaves unnamed from the local word table

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
hugocasa
2026-09-30 20:04:06 +02:00
committed by GitHub
co-authored by Claude Opus 5
parent 8c7dcbbda5
commit 8953ca670a
15 changed files with 4198 additions and 51 deletions
Download Patch File Download Diff File Expand all files Collapse all files
ai_evals/adapters/frontend/mockBackend.ts
+332 -24
View File
@@ -1,4 +1,7 @@
import { randomUUID } from 'node:crypto'
import confluenceFixture from '../../fixtures/frontend/global/hub/confluence.json'
import servicenowFixture from '../../fixtures/frontend/global/hub/servicenow.json'
import outreachFixture from '../../fixtures/frontend/global/hub/outreach.json'
import type {
AppWithLastVersion,
CompletedJob,
@@ -250,9 +253,7 @@ export function listBenchmarkApps(workspace: string): ListableApp[] | null {
}
export function getBenchmarkAppByPath(workspace: string, path: string): AppWithLastVersion | null {
const app = benchmarkWorkspaceRunnables
.get(workspace)
?.apps?.find((entry) => entry.path === path)
const app = benchmarkWorkspaceRunnables.get(workspace)?.apps?.find((entry) => entry.path === path)
return app ? buildBenchmarkApp(app) : null
}
@@ -448,7 +449,10 @@ export function createBenchmarkCompletedJob(input: {
labels: input.label ? [input.label] : undefined
}
benchmarkJobs.set(benchmarkJobKey(input.workspace, jobId), { workspace: input.workspace, job })
benchmarkJobs.set(benchmarkJobKey(input.workspace, jobId), {
workspace: input.workspace,
job
})
return jobId
}
@@ -532,7 +536,13 @@ export function getBenchmarkFlowAllResults(workspace: string, jobId: string) {
*/
const benchmarkDrafts = new Map<
string,
{ workspace: string; kind: UserDraftItemKind; path: string; value: unknown; createdAt: string }
{
workspace: string
kind: UserDraftItemKind
path: string
value: unknown
createdAt: string
}
>()
// Counter-based timestamps: deterministic run-to-run (same event order → same
@@ -614,7 +624,9 @@ export function getBenchmarkDraftForUser(input: {
}): GetDraftForUserResponse {
const entry = benchmarkDrafts.get(benchmarkDraftKey(input.workspace, input.kind, input.path))
if (!entry) {
throw Object.assign(new Error(`no draft for "${input.path}"`), { status: 404 })
throw Object.assign(new Error(`no draft for "${input.path}"`), {
status: 404
})
}
return { value: entry.value, created_at: entry.createdAt }
}
@@ -635,7 +647,11 @@ export function getBenchmarkOwnDraft(input: {
/** Whether a deployed benchmark item exists for a draft row's kind+path —
* drives `draft_only`, which production computes against the deployed tables. */
function benchmarkDeployedExists(workspace: string, kind: UserDraftItemKind, path: string): boolean {
function benchmarkDeployedExists(
workspace: string,
kind: UserDraftItemKind,
path: string
): boolean {
if (kind === 'script') return Boolean(getBenchmarkScriptByPath(workspace, path))
if (kind === 'flow') return Boolean(getBenchmarkFlowByPath(workspace, path))
if (kind === 'app' || kind === 'raw_app') return Boolean(getBenchmarkAppByPath(workspace, path))
@@ -762,10 +778,11 @@ export function runBenchmarkDatatableSql(input: {
* shape `pollJobResult` consumes. The job is created synchronously before
* polling, so it is always present and completed.
*/
export function getBenchmarkCompletedJobResultMaybe(input: {
workspace: string
id: string
}): { success: boolean; completed: boolean; result: unknown } {
export function getBenchmarkCompletedJobResultMaybe(input: { workspace: string; id: string }): {
success: boolean
completed: boolean
result: unknown
} {
const job = getBenchmarkCompletedJob(input.workspace, input.id)
if (!job) {
throw new Error(`Job "${input.id}" not found in benchmark workspace`)
@@ -896,7 +913,9 @@ export function previewBenchmarkSchedule(input: {
}): Record<string, unknown> {
const schedule = input.requestBody?.schedule
if (typeof schedule !== 'string' || schedule.trim().split(/\s+/).length !== 6) {
throw new Error(`schedule must use a six-field cron expression, got ${JSON.stringify(schedule)}`)
throw new Error(
`schedule must use a six-field cron expression, got ${JSON.stringify(schedule)}`
)
}
return {
@@ -928,7 +947,9 @@ export function createBenchmarkHttpTrigger(input: {
typeof input.requestBody.route_path === 'string' &&
input.requestBody.route_path.startsWith('/')
) {
throw new Error(`HTTP trigger route_path must not start with /, got "${input.requestBody.route_path}"`)
throw new Error(
`HTTP trigger route_path must not start with /, got "${input.requestBody.route_path}"`
)
}
return {
path: input.requestBody.path,
@@ -1024,7 +1045,39 @@ function buildBenchmarkApp(app: BenchmarkWorkspaceApp): AppWithLastVersion {
* hub tools throw and no case can exercise hub reuse. Serving fixtures rather
* than the live hub also keeps assertions on script content stable as the real
* hub republishes new versions. */
const BENCHMARK_INTEGRATION_META_PATH = /^\/api\/integrations\/hub\/([^/]+)\/meta$/
/** One integration lifted verbatim from the content repo — its shipped scripts, its
* authored meta.json and its resource type. Hand-written fixtures make both routes to
* an integration's conventions look equally cheap; a real one is the only way to tell
* whether reading the metadata beats reading a script. */
const REAL_HUB_INTEGRATIONS = [
confluenceFixture,
servicenowFixture,
outreachFixture
] as unknown as Array<{
app: string
display_name: string
description: string
docs_url: string
curated: boolean | null
meta: unknown
resource_type: { name: string; description: string; schema: unknown }
scripts: Array<{
version_id: number
app: string
summary: string
description: string
terms: string
kind: string
language: string
content: string
schema: unknown
}>
}>
const BENCHMARK_HUB_SCRIPTS = [
...REAL_HUB_INTEGRATIONS.flatMap((integration) => integration.scripts),
{
version_id: 22235,
app: 'holded',
@@ -1112,6 +1165,87 @@ export async function main(discord_webhook: DiscordWebhook, message: string) {
message: { type: 'string' }
}
}
},
// Baremetrics carries no annotation script, on purpose: it stands for an
// integration whose conventions (an `apiKey` resource, bearer auth, the /v1
// base) are only learnable by reading a script that does something else.
{
version_id: 8995,
app: 'baremetrics',
summary: 'List Sources',
description: 'List all the sources attached to a Baremetrics account.',
terms: 'baremetrics sources list revenue metrics',
language: 'bunnative',
content: `//native
type Baremetrics = {
apiKey: string;
};
export async function main(resource: Baremetrics) {
const response = await fetch("https://api.baremetrics.com/v1/sources", {
method: "GET",
headers: {
"Content-Type": "application/json",
Accept: "application/json",
Authorization: \`Bearer \${resource.apiKey}\`,
},
});
if (!response.ok) {
throw new Error(\`\${response.status} \${await response.text()}\`);
}
return await response.json();
}
`,
schema: {
type: 'object',
required: ['resource'],
properties: {
resource: { type: 'object', format: 'resource-baremetrics' }
}
}
},
{
version_id: 8996,
app: 'baremetrics',
summary: 'Create Customer',
description: 'Create a customer on a Baremetrics source.',
terms: 'baremetrics customer create source',
language: 'bunnative',
content: `//native
type Baremetrics = {
apiKey: string;
};
export async function main(
resource: Baremetrics,
sourceId: string,
body: { oid: string; name?: string; email?: string },
) {
const response = await fetch(
\`https://api.baremetrics.com/v1/\${sourceId}/customers\`,
{
method: "POST",
headers: {
"Content-Type": "application/json",
Accept: "application/json",
Authorization: \`Bearer \${resource.apiKey}\`,
},
body: JSON.stringify(body),
},
);
if (!response.ok) {
throw new Error(\`\${response.status} \${await response.text()}\`);
}
return await response.json();
}
`,
schema: {
type: 'object',
required: ['resource', 'sourceId', 'body'],
properties: {
resource: { type: 'object', format: 'resource-baremetrics' },
sourceId: { type: 'string' },
body: { type: 'object' }
}
}
}
]
@@ -1121,20 +1255,26 @@ export async function main(discord_webhook: DiscordWebhook, message: string) {
* integration, or overlapping on three meaningful words. A looser bar answers
* "send a Slack message" with the Discord fixture, handing an unrelated case a
* plausible-looking wrong integration. */
function searchBenchmarkHubScripts(text: string) {
function searchBenchmarkHubScripts(text: string, app: string | null, kind: string | null) {
const tokens = new Set(
text
.toLowerCase()
.split(/[^a-z0-9]+/)
.filter((token) => token.length > 2)
)
return BENCHMARK_HUB_SCRIPTS.map((script) => {
const words = new Set(
`${script.app} ${script.summary} ${script.terms}`.toLowerCase().split(/[^a-z0-9]+/)
)
const score = [...tokens].filter((token) => words.has(token)).length
return { script, score, namesApp: tokens.has(script.app) }
})
// The real endpoint filters by app and kind before ranking, so honour both here —
// otherwise a narrowed search silently returns other integrations' scripts, and a
// trigger answers a search production would never have shown it to.
return BENCHMARK_HUB_SCRIPTS.filter(
(script) => (!app || script.app === app) && (!kind || (script.kind ?? 'script') === kind)
)
.map((script) => {
const words = new Set(
`${script.app} ${script.summary} ${script.terms}`.toLowerCase().split(/[^a-z0-9]+/)
)
const score = [...tokens].filter((token) => words.has(token)).length
return { script, score, namesApp: tokens.has(script.app) }
})
.filter((entry) => entry.namesApp || entry.score >= 3)
.sort((a, b) => b.score - a.score)
.map(({ script }, index) => ({
@@ -1142,12 +1282,104 @@ function searchBenchmarkHubScripts(text: string) {
id: script.version_id,
version_id: script.version_id,
summary: script.summary,
description: script.description ?? null,
app: script.app,
kind: 'script',
kind: script.kind ?? 'script',
score: 1 - index * 0.01
}))
}
/** Listing an integration is unranked and description-bearing, matching the hub's
* top-scripts endpoint — that asymmetry with the semantic search is the whole
* reason the chat browses by app when no script matches the task. */
function listBenchmarkHubScriptsByApp(app: string | null, kind: string | null) {
return BENCHMARK_HUB_SCRIPTS.filter(
(script) => (!app || script.app === app) && (!kind || (script.kind ?? 'script') === kind)
).map((script) => ({
id: script.version_id,
ask_id: script.version_id,
version_id: script.version_id,
summary: script.summary,
description: script.description ?? null,
app: script.app,
kind: script.kind ?? 'script',
views: 0,
votes: 0
}))
}
/** What `/integrations/hub/<app>/meta` serves. Carries the integration's conventions
* and none of the endpoints a case asks the model to write, so the tool shortens the
* path to that knowledge without supplying answers. */
/** The integrations a case writes against, none of which the content repo documents —
* the same majority shape as the live hub, where 18 of ~216 carry a meta.json. Their
* conventions live in their shipped scripts, which is what these cases exercise. */
const BENCHMARK_HUB_INTEGRATION_META: Record<
string,
{
display_name: string
description: string
docs_url: string
curated: boolean | null
meta: unknown
}
> = {
baremetrics: {
display_name: 'Baremetrics',
description: 'Subscription analytics for recurring-revenue businesses.',
docs_url: 'https://developers.baremetrics.com/reference',
curated: null,
meta: null
},
holded: {
display_name: 'Holded',
description: 'Invoicing, accounting and CRM for small businesses.',
docs_url: 'https://developers.holded.com/reference',
curated: null,
meta: null
}
}
/** Mirrors the hub's own derivation: hosts seen in the shipped scripts, whether they
* call the provider directly, and how many there are of each kind. */
function benchmarkDerivedFacts(app: string) {
const scripts = BENCHMARK_HUB_SCRIPTS.filter((script) => script.app === app)
const hosts = new Map<string, number>()
const languages: Record<string, number> = {}
for (const script of scripts) {
languages[script.language] = (languages[script.language] ?? 0) + 1
for (const match of script.content.matchAll(/https?:\/\/([a-zA-Z0-9._-]+)/g)) {
hosts.set(match[1], (hosts.get(match[1]) ?? 0) + 1)
}
}
return {
api_hosts: [...hosts.entries()]
.sort((a, b) => b[1] - a[1])
.map(([host, count]) => ({ host, count })),
style: 'fetch',
languages,
script_counts: {
total: scripts.length,
by_kind: scripts.reduce<Record<string, number>>((acc, script) => {
const kind = script.kind ?? 'script'
acc[kind] = (acc[kind] ?? 0) + 1
return acc
}, {})
},
top_scripts: scripts.map((script) => ({
path: `hub/${script.version_id}/${script.app}/${script.summary.toLowerCase().replaceAll(/\s+/g, '_')}`,
ask_id: script.version_id,
version_id: script.version_id,
summary: script.summary,
description: script.description ?? null,
kind: script.kind ?? 'script',
language: script.language,
views: 0,
votes: 0
}))
}
}
/** The hub keys a script by its version id; the app and slug segments that
* follow are descriptive, so match on the id exactly as the real hub does. */
function getBenchmarkHubScript(path: string) {
@@ -1190,6 +1422,9 @@ export function hasBenchmarkApiHandler(url: string): boolean {
return (
path === '/api/workers/list' ||
path === '/api/embeddings/query_hub_scripts' ||
path === '/api/scripts/hub/top' ||
path === '/api/integrations/hub/list' ||
BENCHMARK_INTEGRATION_META_PATH.test(path) ||
path.startsWith('/api/scripts/hub/get_full/') ||
BENCHMARK_AI_MODELS_PATH.test(path)
)
@@ -1213,8 +1448,81 @@ export function handleBenchmarkApiFetch(url: string, init?: RequestInit): Respon
return Response.json({ data: (seed?.models ?? []).map((id) => ({ id })) })
}
if (path === '/api/embeddings/query_hub_scripts') {
const text = new URLSearchParams(url.split('?')[1] ?? '').get('text') ?? ''
return Response.json(searchBenchmarkHubScripts(text))
const params = new URLSearchParams(url.split('?')[1] ?? '')
return Response.json(
searchBenchmarkHubScripts(params.get('text') ?? '', params.get('app'), params.get('kind'))
)
}
if (path === '/api/scripts/hub/top') {
const params = new URLSearchParams(url.split('?')[1] ?? '')
return Response.json({
asks: listBenchmarkHubScriptsByApp(params.get('app'), params.get('kind'))
})
}
const integrationMeta = BENCHMARK_INTEGRATION_META_PATH.exec(path)
if (integrationMeta) {
const app = decodeURIComponent(integrationMeta[1])
const real = REAL_HUB_INTEGRATIONS.find((integration) => integration.app === app)
if (real) {
return Response.json({
app,
display_name: real.display_name,
description: real.description,
docs_url: real.docs_url,
curated: real.curated,
metadata_source: real.meta ? 'curated' : 'derived',
meta: real.meta,
meta_updated_at: null,
derived: benchmarkDerivedFacts(app),
resource_types: [{ id: 1, ...real.resource_type }]
})
}
const entry = BENCHMARK_HUB_INTEGRATION_META[app]
if (!entry) {
return Response.json({ error: 'integration not found' }, { status: 404 })
}
return Response.json({
app,
display_name: entry.display_name,
description: entry.description,
docs_url: entry.docs_url,
curated: entry.curated,
metadata_source: entry.meta ? 'curated' : 'derived',
meta: entry.meta,
meta_updated_at: null,
derived: benchmarkDerivedFacts(app),
resource_types: [
{
id: 1,
name: app,
description: `${entry.display_name} credentials`,
schema: {
type: 'object',
required: ['apiKey'],
properties: {
apiKey: {
type: 'string',
description: `${entry.display_name} API key`
}
}
}
}
]
})
}
if (path === '/api/integrations/hub/list') {
const apps = [...new Set(BENCHMARK_HUB_SCRIPTS.map((script) => script.app))].sort()
// Read off both fixtures the metadata endpoint serves from, or the list would
// call an integration undocumented and then hand back its authored notes.
const documented = new Set([
...REAL_HUB_INTEGRATIONS.filter((integration) => integration.meta).map(
(integration) => integration.app
),
...Object.entries(BENCHMARK_HUB_INTEGRATION_META)
.filter(([, entry]) => entry.meta)
.map(([app]) => app)
])
return Response.json(apps.map((name) => ({ name, documented: documented.has(name) })))
}
if (path.startsWith('/api/scripts/hub/get_full/')) {
const script = getBenchmarkHubScript(path)
ai_evals/adapters/frontend/mockBackendApi.test.ts
+18
View File
@@ -16,3 +16,21 @@ describe('benchmark API fetch handlers', () => {
expect(body[0]).toMatchObject({ worker: expect.any(String), worker_group: expect.any(String) })
})
})
// A benchmark whose list calls an integration undocumented while its metadata endpoint
// hands back authored notes teaches the model the flag means nothing.
describe('benchmark hub integration list', () => {
it('flags exactly the integrations whose metadata carries authored notes', async () => {
const listed = (await handleBenchmarkApiFetch('/api/integrations/hub/list').json()) as Array<{
name: string
documented: boolean
}>
expect(listed.length).toBeGreaterThan(0)
for (const { name, documented } of listed) {
const res = handleBenchmarkApiFetch(`/api/integrations/hub/${name}/meta`)
const authored = res.status === 200 && !!((await res.json()) as { meta?: unknown }).meta
expect(authored).toBe(documented)
}
})
})
ai_evals/cases/global.yaml
+171
View File
@@ -2312,6 +2312,105 @@
- email
skipJudge: true
# `List Pages` takes a cursor without ever following one, so where the next cursor
# comes from (`_links.next`) is stated in the authored metadata and in no script body.
# Paginating is therefore the one task where reading scripts is not an equal route.
- id: global-hub4-confluence-paginate-all-pages
prompt: |-
I need a script that returns every page in one of my Confluence spaces, not just the
first batch — it should keep fetching until there are no more.
Save it as a draft at `f/evals/global/confluence_all_pages` and leave it as an AI
draft; do not deploy it.
initial: ai_evals/fixtures/frontend/global/initial/user_admin_evals_folder.json
runtime:
maxTurns: 14
validate:
draftCountExactly: 1
requiredDrafts:
- type: script
path: f/evals/global/confluence_all_pages
valueIncludes:
- /wiki/
- Basic
toolExpect:
requiredToolsUsed:
- write_script
# Either route to the integration's conventions is fine; the checklist is where the
# difference between them has to show up.
requiredToolsAnyOf:
- - search_hub_scripts
- get_hub_integration
forbiddenToolsUsed:
- deploy_workspace_item
judgeChecklist:
- the script loops until the API stops returning a next link, rather than making a single request
- the next cursor is read from the response's `_links.next` or the Link header, not from an invented field such as `next_cursor`, `nextPageToken` or `offset`
- it authenticates with HTTP Basic built from the Confluence resource's email and API token rather than a bearer token
- the script takes a Confluence resource as an input parameter rather than loose credential strings
- the result stays an AI draft and is not deployed
# Unlike the Confluence case, both facts this needs — the {"result"} envelope and
# PATCH-not-PUT — are visible in ServiceNow's own shipped scripts. That makes it the
# control for whether the metadata earns its place when the scripts already answer.
- id: global-hub5-servicenow-assign-incident
prompt: |-
Write me a script that assigns an existing ServiceNow incident to a person, looking
that person up by their email address.
Save it as a draft at `f/evals/global/servicenow_assign_incident` and leave it as an
AI draft; do not deploy it.
initial: ai_evals/fixtures/frontend/global/initial/user_admin_evals_folder.json
runtime:
maxTurns: 14
validate:
draftCountExactly: 1
requiredDrafts:
- type: script
path: f/evals/global/servicenow_assign_incident
toolExpect:
requiredToolsUsed:
- write_script
requiredToolsAnyOf:
- - search_hub_scripts
- get_hub_integration
forbiddenToolsUsed:
- deploy_workspace_item
judgeChecklist:
- it looks the person up in the sys_user table by email rather than assuming a sys_id was supplied
- it reads records out of the `result` envelope the ServiceNow API wraps responses in
- it sets assigned_to to the user's sys_id rather than to the raw email address
- it updates the incident with a partial update (PATCH) rather than a full replace that would clear the fields it did not send
- the result stays an AI draft and is not deployed
# Outreach is JSON:API. Its content type and payload envelope are both stated in the
# metadata and visible in the shipped update scripts, so this is a second control.
- id: global-hub6-outreach-reschedule-task
prompt: |-
Write me a script that moves an existing Outreach task to a new due date.
Save it as a draft at `f/evals/global/outreach_reschedule_task` and leave it as an AI
draft; do not deploy it.
initial: ai_evals/fixtures/frontend/global/initial/user_admin_evals_folder.json
runtime:
maxTurns: 14
validate:
draftCountExactly: 1
requiredDrafts:
- type: script
path: f/evals/global/outreach_reschedule_task
toolExpect:
requiredToolsUsed:
- write_script
requiredToolsAnyOf:
- - search_hub_scripts
- get_hub_integration
forbiddenToolsUsed:
- deploy_workspace_item
judgeChecklist:
- the request sends the JSON:API content type application/vnd.api+json for its body
- the payload wraps the change as data.type and data.attributes rather than sending bare top-level fields
- it identifies the task by id inside the payload rather than only in the URL, and treats the id as a number
- it updates the existing task with PATCH rather than creating a new one
- the result stays an AI draft and is not deployed
# --- Windmill Hub reuse (search_hub_scripts + read_workspace_item on a hub/ path) ---
# Holded's API is obscure enough that a model writing from memory cannot reproduce
# its endpoint and `key` auth header — so the draft's fidelity to the published
@@ -2358,6 +2457,78 @@
- the document type, document id, and recipient emails are inputs to the script
- the result stays an AI draft and is not deployed
# Same task as global-hub1 with the hub never mentioned: this is what tells us the
# model reaches for the hub on its own rather than only when pointed at it.
- id: global-hub2-checks-hub-unprompted
prompt: |-
I want to email one of my Holded invoices to a customer from Windmill.
Save it as a draft script at `f/evals/global/holded_send_document` and leave it as
an AI draft; do not deploy it.
initial: ai_evals/fixtures/frontend/global/initial/user_admin_evals_folder.json
runtime:
maxTurns: 12
validate:
draftCountExactly: 1
requiredDrafts:
# No language assertion: the Holded script's body is `//native`, so a faithful
# fork lands on bun or bunnative depending on how literally it is copied.
- type: script
path: f/evals/global/holded_send_document
valueIncludes:
- api.holded.com/api/invoicing/v1/documents
- mailTemplateId
toolExpect:
requiredToolsUsed:
- search_hub_scripts
- write_script
forbiddenToolsUsed:
- deploy_workspace_item
judgeChecklist:
- the draft sends an existing Holded document by email rather than creating one
- the request targets Holded's document send endpoint, not an invented URL
- authentication uses Holded's own key header rather than a bearer token
- the result stays an AI draft and is not deployed
# The fixture hub carries two Baremetrics scripts, neither for annotations, so nothing
# does what is asked. Its conventions — an `apiKey` resource, bearer auth, the
# api.baremetrics.com/v1 base — are only knowable by reading one of those near-misses.
- id: global-hub3-integration-as-example
prompt: |-
Write me a script that adds an annotation to my Baremetrics account.
Save it as a draft at `f/evals/global/baremetrics_create_annotation` and leave it as
an AI draft; do not deploy it.
initial: ai_evals/fixtures/frontend/global/initial/user_admin_evals_folder.json
runtime:
maxTurns: 14
validate:
draftCountExactly: 1
requiredDrafts:
# No language assertion: matching the hub script's own dialect is the point,
# and Baremetrics' scripts are `//native`, so both bun and bunnative are right.
- type: script
path: f/evals/global/baremetrics_create_annotation
valueIncludes:
- api.baremetrics.com/v1
- apiKey
toolExpect:
requiredToolsUsed:
- search_hub_scripts
- write_script
# Reading a near-miss script and reading the integration's own metadata are two
# routes to the same conventions, so requiring either keeps the case about the
# outcome rather than about which route the model took.
requiredToolsAnyOf:
- - read_workspace_item
- get_hub_integration
forbiddenToolsUsed:
- deploy_workspace_item
judgeChecklist:
- the script takes a Baremetrics resource as an input parameter rather than a bare API key string
- the resource type it declares carries an apiKey field, matching how other Baremetrics hub scripts take credentials
- the request authenticates with a bearer token built from that resource
- the script posts an annotation rather than reading data back
- the result stays an AI draft and is not deployed
# The value of a secret variable is unreadable, so a metadata-only edit must leave
# it untouched: passing any `value` here means inventing one, which silently
# replaces the real secret at deploy.
ai_evals/fixtures/frontend/global/hub/confluence.json
+983
View File
@@ -0,0 +1,983 @@
{
"app": "confluence",
"display_name": "Confluence",
"description": "Atlassian's team workspace: pages, blog posts and spaces.",
"docs_url": "https://developer.atlassian.com/cloud/confluence/rest/v2/intro/",
"curated": true,
"meta": {
"api_docs": "https://developer.atlassian.com/cloud/confluence/rest/v2/intro/",
"shipped_surfaces": [
"pages",
"blogposts",
"spaces",
"search",
"users"
],
"validation": {
"status": "live_validated",
"method": "smoke test via `bun --env-file=.env.test -e` (+ `bun test` with the windmill-client fake for the trigger) against a free Confluence Cloud site (a test site) with an API token. All 13 actions + the trigger returned 2xx: full page lifecycle (create -> get -> update -> list -> delete), blog post create/update/delete, list_spaces, search_content (CQL), get_current_user, and the new_or_updated_page polling trigger.",
"sources": {
"pages": "HIGH \u2014 Pipedream actions + official v2 OpenAPI; create/get/update/list/delete all run live (2xx)",
"blogposts": "HIGH \u2014 Pipedream actions + official v2 OpenAPI; create/update/delete all run live (2xx)",
"spaces": "HIGH \u2014 Pipedream list-spaces + official v2 OpenAPI; list_spaces run live (2xx)",
"search": "HIGH \u2014 Pipedream search-content + official v1 REST; search_content (CQL) run live (2xx)",
"users": "HIGH \u2014 v1 /wiki/rest/api/user/current run live (2xx) with Basic auth",
"trigger": "HIGH \u2014 new_or_updated_page e2e-tested under bun test with the auto-fake; first run sets the watermark, a subsequent run emits a freshly created page"
},
"low_confidence_fields": []
},
"pagination": {
"pattern": "cursor",
"request_params": {
"cursor": "cursor",
"limit": "limit"
},
"response_fields": {
"next": "_links.next"
},
"default_limit": 25,
"notes": "v2 endpoints return an opaque cursor inside the `_links.next` relative URL (?cursor=...) and also in the Link response header. The v1 /search endpoint paginates the same way."
},
"gotchas": [
"Two API versions are in play: pages / blog posts / spaces use the v2 REST API (/wiki/api/v2); CQL search and current-user use the v1 REST API (/wiki/rest/api).",
"Auth is HTTP Basic with the Atlassian account email as the username and an API token (NOT the account password) as the password.",
"v2 query params are kebab-case (body-format, created-date, get-draft, root-level); v2 JSON request-body fields are camelCase (spaceId, parentId).",
"Updating a page or blog post requires version.number = currentVersion + 1; the update_* scripts fetch the current version and increment it automatically.",
"spaceId / parentId in v2 request bodies are numeric IDs passed as strings, not space keys.",
"DELETE returns 204 with an empty body and trashes by default; pass purge=true to permanently remove an item that is already in the trash."
],
"enums": {
"page_status": [
"current",
"archived",
"deleted",
"trashed",
"draft",
"historical"
],
"body_representation": [
"storage",
"atlas_doc_format"
],
"body_format_read": [
"storage",
"atlas_doc_format",
"view",
"export_view",
"anonymous_export_view",
"styled_view",
"editor"
],
"space_type": [
"global",
"collaboration",
"knowledge_base",
"personal"
],
"page_sort": [
"id",
"-id",
"created-date",
"-created-date",
"modified-date",
"-modified-date",
"title",
"-title"
]
},
"errors": {
"401": "Authentication failed \u2014 verify the email + API token pair and that the token has not been revoked.",
"404": "Resource not found, or the authenticating account lacks permission to view it (Confluence often returns 404 rather than 403 for content the user cannot access)."
}
},
"resource_type": {
"name": "confluence",
"description": "Confluence Cloud credentials. Authenticate with an Atlassian account email and an API token (HTTP Basic auth) against your site base URL.",
"schema": {
"$schema": "https://json-schema.org/draft/2020-12/schema",
"properties": {
"apiToken": {
"description": "API token for your Atlassian account. Create one at https://id.atlassian.com/manage-profile/security/api-tokens",
"type": "string"
},
"baseUrl": {
"description": "Base URL of your Confluence Cloud site, e.g. https://your-domain.atlassian.net (no trailing slash, no /wiki).",
"type": "string"
},
"email": {
"description": "Email of the Atlassian account used to authenticate.",
"type": "string"
}
},
"required": [
"email",
"apiToken",
"baseUrl"
],
"type": "object"
}
},
"scripts": [
{
"version_id": 28765,
"app": "confluence",
"summary": "Create Blog Post",
"description": "Create a new blog post in a space.",
"terms": "confluence create blog post create a new blog post in a space.",
"kind": "script",
"language": "bunnative",
"content": "//native\n\n/**\n * Create Blog Post\n * Create a new blog post in a space. The body is supplied as a content string in the chosen representation (default \"storage\", i.e. Confluence storage HTML).\n */\nexport async function main(\n auth: RT.Confluence,\n space_id: string,\n title: string,\n body: string,\n status: \"current\" | \"draft\" | undefined,\n representation: \"storage\" | \"atlas_doc_format\" | undefined\n) {\n const base = auth.baseUrl.replace(/\\/$/, \"\")\n const url = new URL(`${base}/wiki/api/v2/blogposts`)\n\n const payload = {\n spaceId: space_id,\n status: status ?? \"current\",\n title,\n body: {\n representation: representation ?? \"storage\",\n value: body,\n },\n }\n\n const response = await fetch(url, {\n method: \"POST\",\n headers: {\n Authorization: \"Basic \" + btoa(`${auth.email}:${auth.apiToken}`),\n \"Content-Type\": \"application/json\",\n Accept: \"application/json\",\n },\n body: JSON.stringify(payload),\n })\n\n if (!response.ok) {\n throw new Error(`${response.status} ${await response.text()}`)\n }\n\n return await response.json()\n}\n",
"schema": {
"$schema": "https://json-schema.org/draft/2020-12/schema",
"properties": {
"auth": {
"default": null,
"description": "",
"format": "resource-confluence",
"type": "object"
},
"body": {
"default": null,
"description": "",
"originalType": "string",
"type": "string"
},
"representation": {
"default": null,
"description": "",
"enum": [
"storage",
"atlas_doc_format"
],
"originalType": "enum",
"type": "string"
},
"space_id": {
"default": null,
"description": "",
"originalType": "string",
"type": "string"
},
"status": {
"default": null,
"description": "",
"enum": [
"current",
"draft"
],
"originalType": "enum",
"type": "string"
},
"title": {
"default": null,
"description": "",
"originalType": "string",
"type": "string"
}
},
"required": [
"auth",
"space_id",
"title",
"body"
],
"type": "object"
}
},
{
"version_id": 28766,
"app": "confluence",
"summary": "Create Page",
"description": "Create a new page in a space.",
"terms": "confluence create page create a new page in a space.",
"kind": "script",
"language": "bunnative",
"content": "//native\n\n/**\n * Create Page\n * Create a new page in a space. The body is supplied as a content string in the chosen representation (default \"storage\", i.e. Confluence storage HTML).\n */\nexport async function main(\n auth: RT.Confluence,\n space_id: string,\n title: string,\n body: string,\n status: \"current\" | \"draft\" | undefined,\n parent_id: string | undefined,\n representation: \"storage\" | \"atlas_doc_format\" | undefined\n) {\n const base = auth.baseUrl.replace(/\\/$/, \"\")\n const url = new URL(`${base}/wiki/api/v2/pages`)\n\n const payload = {\n spaceId: space_id,\n status: status ?? \"current\",\n title,\n parentId: parent_id,\n body: {\n representation: representation ?? \"storage\",\n value: body,\n },\n }\n\n const response = await fetch(url, {\n method: \"POST\",\n headers: {\n Authorization: \"Basic \" + btoa(`${auth.email}:${auth.apiToken}`),\n \"Content-Type\": \"application/json\",\n Accept: \"application/json\",\n },\n body: JSON.stringify(payload),\n })\n\n if (!response.ok) {\n throw new Error(`${response.status} ${await response.text()}`)\n }\n\n return await response.json()\n}\n",
"schema": {
"$schema": "https://json-schema.org/draft/2020-12/schema",
"properties": {
"auth": {
"default": null,
"description": "",
"format": "resource-confluence",
"type": "object"
},
"body": {
"default": null,
"description": "",
"originalType": "string",
"type": "string"
},
"parent_id": {
"default": null,
"description": "",
"originalType": "string",
"type": "string"
},
"representation": {
"default": null,
"description": "",
"enum": [
"storage",
"atlas_doc_format"
],
"originalType": "enum",
"type": "string"
},
"space_id": {
"default": null,
"description": "",
"originalType": "string",
"type": "string"
},
"status": {
"default": null,
"description": "",
"enum": [
"current",
"draft"
],
"originalType": "enum",
"type": "string"
},
"title": {
"default": null,
"description": "",
"originalType": "string",
"type": "string"
}
},
"required": [
"auth",
"space_id",
"title",
"body"
],
"type": "object"
}
},
{
"version_id": 28767,
"app": "confluence",
"summary": "Delete Blog Post",
"description": "Delete a blog post by its ID (moved to trash by default; purge to permanently remove).",
"terms": "confluence delete blog post delete a blog post by its id (moved to trash by default; purge to permanently remove).",
"kind": "script",
"language": "bunnative",
"content": "//native\n\n/**\n * Delete Blog Post\n * Delete a blog post by its ID. By default the post is moved to the trash; set purge to permanently remove an already-trashed post.\n */\nexport async function main(\n auth: RT.Confluence,\n blog_post_id: string,\n purge: boolean | undefined,\n draft: boolean | undefined\n) {\n const base = auth.baseUrl.replace(/\\/$/, \"\")\n const url = new URL(`${base}/wiki/api/v2/blogposts/${blog_post_id}`)\n if (purge !== undefined) url.searchParams.append(\"purge\", String(purge))\n if (draft !== undefined) url.searchParams.append(\"draft\", String(draft))\n\n const response = await fetch(url, {\n method: \"DELETE\",\n headers: {\n Authorization: \"Basic \" + btoa(`${auth.email}:${auth.apiToken}`),\n Accept: \"application/json\",\n },\n })\n\n if (!response.ok) {\n throw new Error(`${response.status} ${await response.text()}`)\n }\n\n if (response.status === 204) return { success: true }\n return await response.json()\n}\n",
"schema": {
"$schema": "https://json-schema.org/draft/2020-12/schema",
"properties": {
"auth": {
"default": null,
"description": "",
"format": "resource-confluence",
"type": "object"
},
"blog_post_id": {
"default": null,
"description": "",
"originalType": "string",
"type": "string"
},
"draft": {
"default": null,
"description": "",
"type": "boolean"
},
"purge": {
"default": null,
"description": "",
"type": "boolean"
}
},
"required": [
"auth",
"blog_post_id"
],
"type": "object"
}
},
{
"version_id": 28768,
"app": "confluence",
"summary": "Delete Page",
"description": "Delete a page by its ID (moved to trash by default; purge to permanently remove).",
"terms": "confluence delete page delete a page by its id (moved to trash by default; purge to permanently remove).",
"kind": "script",
"language": "bunnative",
"content": "//native\n\n/**\n * Delete Page\n * Delete a page by its ID. By default the page is moved to the trash; set purge to permanently remove an already-trashed page.\n */\nexport async function main(\n auth: RT.Confluence,\n page_id: string,\n purge: boolean | undefined,\n draft: boolean | undefined\n) {\n const base = auth.baseUrl.replace(/\\/$/, \"\")\n const url = new URL(`${base}/wiki/api/v2/pages/${page_id}`)\n if (purge !== undefined) url.searchParams.append(\"purge\", String(purge))\n if (draft !== undefined) url.searchParams.append(\"draft\", String(draft))\n\n const response = await fetch(url, {\n method: \"DELETE\",\n headers: {\n Authorization: \"Basic \" + btoa(`${auth.email}:${auth.apiToken}`),\n Accept: \"application/json\",\n },\n })\n\n if (!response.ok) {\n throw new Error(`${response.status} ${await response.text()}`)\n }\n\n if (response.status === 204) return { success: true }\n return await response.json()\n}\n",
"schema": {
"$schema": "https://json-schema.org/draft/2020-12/schema",
"properties": {
"auth": {
"default": null,
"description": "",
"format": "resource-confluence",
"type": "object"
},
"draft": {
"default": null,
"description": "",
"type": "boolean"
},
"page_id": {
"default": null,
"description": "",
"originalType": "string",
"type": "string"
},
"purge": {
"default": null,
"description": "",
"type": "boolean"
}
},
"required": [
"auth",
"page_id"
],
"type": "object"
}
},
{
"version_id": 28769,
"app": "confluence",
"summary": "Get Current User",
"description": "Returns the profile of the user that the API token authenticates as.",
"terms": "confluence get current user returns the profile of the user that the api token authenticates as.",
"kind": "script",
"language": "bunnative",
"content": "//native\n\n/**\n * Get Current User\n * Returns the profile of the user that the API token authenticates as (account ID, display name, email).\n */\nexport async function main(auth: RT.Confluence) {\n const base = auth.baseUrl.replace(/\\/$/, \"\")\n const url = new URL(`${base}/wiki/rest/api/user/current`)\n\n const response = await fetch(url, {\n method: \"GET\",\n headers: {\n Authorization: \"Basic \" + btoa(`${auth.email}:${auth.apiToken}`),\n Accept: \"application/json\",\n },\n })\n\n if (!response.ok) {\n throw new Error(`${response.status} ${await response.text()}`)\n }\n\n return await response.json()\n}\n",
"schema": {
"$schema": "https://json-schema.org/draft/2020-12/schema",
"properties": {
"auth": {
"default": null,
"description": "",
"format": "resource-confluence",
"type": "object"
}
},
"required": [
"auth"
],
"type": "object"
}
},
{
"version_id": 28770,
"app": "confluence",
"summary": "Get Page by ID",
"description": "Retrieve a single page by its ID, optionally returning its body in a specific format or a previous version.",
"terms": "confluence get page by id retrieve a single page by its id, optionally returning its body in a specific format or a previous version.",
"kind": "script",
"language": "bunnative",
"content": "//native\n\n/**\n * Get Page by ID\n * Retrieve a single page by its ID, optionally returning its body in a specific format or a previous version.\n */\nexport async function main(\n auth: RT.Confluence,\n page_id: string,\n body_format:\n | \"storage\"\n | \"atlas_doc_format\"\n | \"view\"\n | \"export_view\"\n | \"anonymous_export_view\"\n | \"styled_view\"\n | \"editor\"\n | undefined,\n version: number | undefined,\n get_draft: boolean | undefined\n) {\n const base = auth.baseUrl.replace(/\\/$/, \"\")\n const url = new URL(`${base}/wiki/api/v2/pages/${page_id}`)\n if (body_format !== undefined)\n url.searchParams.append(\"body-format\", body_format)\n if (version !== undefined) url.searchParams.append(\"version\", String(version))\n if (get_draft !== undefined)\n url.searchParams.append(\"get-draft\", String(get_draft))\n\n const response = await fetch(url, {\n method: \"GET\",\n headers: {\n Authorization: \"Basic \" + btoa(`${auth.email}:${auth.apiToken}`),\n Accept: \"application/json\",\n },\n })\n\n if (!response.ok) {\n throw new Error(`${response.status} ${await response.text()}`)\n }\n\n return await response.json()\n}\n",
"schema": {
"$schema": "https://json-schema.org/draft/2020-12/schema",
"properties": {
"auth": {
"default": null,
"description": "",
"format": "resource-confluence",
"type": "object"
},
"body_format": {
"default": null,
"description": "",
"enum": [
"storage",
"atlas_doc_format",
"view",
"export_view",
"anonymous_export_view",
"styled_view",
"editor"
],
"originalType": "enum",
"type": "string"
},
"get_draft": {
"default": null,
"description": "",
"type": "boolean"
},
"page_id": {
"default": null,
"description": "",
"originalType": "string",
"type": "string"
},
"version": {
"default": null,
"description": "",
"type": "number"
}
},
"required": [
"auth",
"page_id"
],
"type": "object"
}
},
{
"version_id": 28771,
"app": "confluence",
"summary": "Get Pages in Space",
"description": "Retrieve a paginated list of pages within a given space.",
"terms": "confluence get pages in space retrieve a paginated list of pages within a given space.",
"kind": "script",
"language": "bunnative",
"content": "//native\n\n/**\n * Get Pages in Space\n * Retrieve a paginated list of pages within a given space.\n */\nexport async function main(\n auth: RT.Confluence,\n space_id: string,\n status: (\"current\" | \"archived\" | \"deleted\" | \"trashed\")[] | undefined,\n title: string | undefined,\n sort:\n | \"id\"\n | \"-id\"\n | \"created-date\"\n | \"-created-date\"\n | \"modified-date\"\n | \"-modified-date\"\n | \"title\"\n | \"-title\"\n | undefined,\n body_format: \"storage\" | \"atlas_doc_format\" | undefined,\n cursor: string | undefined,\n limit: number | undefined\n) {\n const base = auth.baseUrl.replace(/\\/$/, \"\")\n const url = new URL(`${base}/wiki/api/v2/spaces/${space_id}/pages`)\n for (const s of status ?? []) url.searchParams.append(\"status\", s)\n if (title !== undefined && title !== \"\")\n url.searchParams.append(\"title\", title)\n if (sort !== undefined) url.searchParams.append(\"sort\", sort)\n if (body_format !== undefined)\n url.searchParams.append(\"body-format\", body_format)\n if (cursor !== undefined && cursor !== \"\")\n url.searchParams.append(\"cursor\", cursor)\n if (limit !== undefined) url.searchParams.append(\"limit\", String(limit))\n\n const response = await fetch(url, {\n method: \"GET\",\n headers: {\n Authorization: \"Basic \" + btoa(`${auth.email}:${auth.apiToken}`),\n Accept: \"application/json\",\n },\n })\n\n if (!response.ok) {\n throw new Error(`${response.status} ${await response.text()}`)\n }\n\n return await response.json()\n}\n",
"schema": {
"$schema": "https://json-schema.org/draft/2020-12/schema",
"properties": {
"auth": {
"default": null,
"description": "",
"format": "resource-confluence",
"type": "object"
},
"body_format": {
"default": null,
"description": "",
"enum": [
"storage",
"atlas_doc_format"
],
"originalType": "enum",
"type": "string"
},
"cursor": {
"default": null,
"description": "",
"originalType": "string",
"type": "string"
},
"limit": {
"default": null,
"description": "",
"type": "number"
},
"sort": {
"default": null,
"description": "",
"enum": [
"id",
"-id",
"created-date",
"-created-date",
"modified-date",
"-modified-date",
"title",
"-title"
],
"originalType": "enum",
"type": "string"
},
"space_id": {
"default": null,
"description": "",
"originalType": "string",
"type": "string"
},
"status": {
"default": null,
"description": "",
"items": {
"enum": [
"current",
"archived",
"deleted",
"trashed"
],
"type": "string"
},
"originalType": "enum[]",
"type": "array"
},
"title": {
"default": null,
"description": "",
"originalType": "string",
"type": "string"
}
},
"required": [
"auth",
"space_id"
],
"type": "object"
}
},
{
"version_id": 28772,
"app": "confluence",
"summary": "List Blog Posts",
"description": "Retrieve a paginated list of blog posts on the site, with optional filtering and sorting.",
"terms": "confluence list blog posts retrieve a paginated list of blog posts on the site, with optional filtering and sorting.",
"kind": "script",
"language": "bunnative",
"content": "//native\n\n/**\n * List Blog Posts\n * Retrieve a paginated list of blog posts on the site, with optional filtering and sorting.\n */\nexport async function main(\n auth: RT.Confluence,\n status: (\"current\" | \"archived\" | \"deleted\" | \"trashed\")[] | undefined,\n title: string | undefined,\n sort:\n | \"id\"\n | \"-id\"\n | \"created-date\"\n | \"-created-date\"\n | \"modified-date\"\n | \"-modified-date\"\n | undefined,\n body_format: \"storage\" | \"atlas_doc_format\" | undefined,\n cursor: string | undefined,\n limit: number | undefined\n) {\n const base = auth.baseUrl.replace(/\\/$/, \"\")\n const url = new URL(`${base}/wiki/api/v2/blogposts`)\n for (const s of status ?? []) url.searchParams.append(\"status\", s)\n if (title !== undefined && title !== \"\")\n url.searchParams.append(\"title\", title)\n if (sort !== undefined) url.searchParams.append(\"sort\", sort)\n if (body_format !== undefined)\n url.searchParams.append(\"body-format\", body_format)\n if (cursor !== undefined && cursor !== \"\")\n url.searchParams.append(\"cursor\", cursor)\n if (limit !== undefined) url.searchParams.append(\"limit\", String(limit))\n\n const response = await fetch(url, {\n method: \"GET\",\n headers: {\n Authorization: \"Basic \" + btoa(`${auth.email}:${auth.apiToken}`),\n Accept: \"application/json\",\n },\n })\n\n if (!response.ok) {\n throw new Error(`${response.status} ${await response.text()}`)\n }\n\n return await response.json()\n}\n",
"schema": {
"$schema": "https://json-schema.org/draft/2020-12/schema",
"properties": {
"auth": {
"default": null,
"description": "",
"format": "resource-confluence",
"type": "object"
},
"body_format": {
"default": null,
"description": "",
"enum": [
"storage",
"atlas_doc_format"
],
"originalType": "enum",
"type": "string"
},
"cursor": {
"default": null,
"description": "",
"originalType": "string",
"type": "string"
},
"limit": {
"default": null,
"description": "",
"type": "number"
},
"sort": {
"default": null,
"description": "",
"enum": [
"id",
"-id",
"created-date",
"-created-date",
"modified-date",
"-modified-date"
],
"originalType": "enum",
"type": "string"
},
"status": {
"default": null,
"description": "",
"items": {
"enum": [
"current",
"archived",
"deleted",
"trashed"
],
"type": "string"
},
"originalType": "enum[]",
"type": "array"
},
"title": {
"default": null,
"description": "",
"originalType": "string",
"type": "string"
}
},
"required": [
"auth"
],
"type": "object"
}
},
{
"version_id": 28773,
"app": "confluence",
"summary": "List Pages",
"description": "Retrieve a paginated list of pages on the site, with optional filtering and sorting.",
"terms": "confluence list pages retrieve a paginated list of pages on the site, with optional filtering and sorting.",
"kind": "script",
"language": "bunnative",
"content": "//native\n\n/**\n * List Pages\n * Retrieve a paginated list of pages on the site, with optional filtering and sorting.\n */\nexport async function main(\n auth: RT.Confluence,\n status: (\"current\" | \"archived\" | \"deleted\" | \"trashed\")[] | undefined,\n title: string | undefined,\n sort:\n | \"id\"\n | \"-id\"\n | \"created-date\"\n | \"-created-date\"\n | \"modified-date\"\n | \"-modified-date\"\n | \"title\"\n | \"-title\"\n | undefined,\n body_format: \"storage\" | \"atlas_doc_format\" | undefined,\n cursor: string | undefined,\n limit: number | undefined\n) {\n const base = auth.baseUrl.replace(/\\/$/, \"\")\n const url = new URL(`${base}/wiki/api/v2/pages`)\n for (const s of status ?? []) url.searchParams.append(\"status\", s)\n if (title !== undefined && title !== \"\")\n url.searchParams.append(\"title\", title)\n if (sort !== undefined) url.searchParams.append(\"sort\", sort)\n if (body_format !== undefined)\n url.searchParams.append(\"body-format\", body_format)\n if (cursor !== undefined && cursor !== \"\")\n url.searchParams.append(\"cursor\", cursor)\n if (limit !== undefined) url.searchParams.append(\"limit\", String(limit))\n\n const response = await fetch(url, {\n method: \"GET\",\n headers: {\n Authorization: \"Basic \" + btoa(`${auth.email}:${auth.apiToken}`),\n Accept: \"application/json\",\n },\n })\n\n if (!response.ok) {\n throw new Error(`${response.status} ${await response.text()}`)\n }\n\n return await response.json()\n}\n",
"schema": {
"$schema": "https://json-schema.org/draft/2020-12/schema",
"properties": {
"auth": {
"default": null,
"description": "",
"format": "resource-confluence",
"type": "object"
},
"body_format": {
"default": null,
"description": "",
"enum": [
"storage",
"atlas_doc_format"
],
"originalType": "enum",
"type": "string"
},
"cursor": {
"default": null,
"description": "",
"originalType": "string",
"type": "string"
},
"limit": {
"default": null,
"description": "",
"type": "number"
},
"sort": {
"default": null,
"description": "",
"enum": [
"id",
"-id",
"created-date",
"-created-date",
"modified-date",
"-modified-date",
"title",
"-title"
],
"originalType": "enum",
"type": "string"
},
"status": {
"default": null,
"description": "",
"items": {
"enum": [
"current",
"archived",
"deleted",
"trashed"
],
"type": "string"
},
"originalType": "enum[]",
"type": "array"
},
"title": {
"default": null,
"description": "",
"originalType": "string",
"type": "string"
}
},
"required": [
"auth"
],
"type": "object"
}
},
{
"version_id": 28774,
"app": "confluence",
"summary": "List Spaces",
"description": "Retrieve a paginated list of spaces on the site, with optional filtering by key, type, and status.",
"terms": "confluence list spaces retrieve a paginated list of spaces on the site, with optional filtering by key, type, and status.",
"kind": "script",
"language": "bunnative",
"content": "//native\n\n/**\n * List Spaces\n * Retrieve a paginated list of spaces on the site, with optional filtering by key, type, and status.\n */\nexport async function main(\n auth: RT.Confluence,\n keys: string[] | undefined,\n type: \"global\" | \"collaboration\" | \"knowledge_base\" | \"personal\" | undefined,\n status: \"current\" | \"archived\" | undefined,\n sort: \"id\" | \"-id\" | \"key\" | \"-key\" | \"name\" | \"-name\" | undefined,\n cursor: string | undefined,\n limit: number | undefined\n) {\n const base = auth.baseUrl.replace(/\\/$/, \"\")\n const url = new URL(`${base}/wiki/api/v2/spaces`)\n if (keys !== undefined && keys.length > 0)\n url.searchParams.append(\"keys\", keys.join(\",\"))\n if (type !== undefined) url.searchParams.append(\"type\", type)\n if (status !== undefined) url.searchParams.append(\"status\", status)\n if (sort !== undefined) url.searchParams.append(\"sort\", sort)\n if (cursor !== undefined && cursor !== \"\")\n url.searchParams.append(\"cursor\", cursor)\n if (limit !== undefined) url.searchParams.append(\"limit\", String(limit))\n\n const response = await fetch(url, {\n method: \"GET\",\n headers: {\n Authorization: \"Basic \" + btoa(`${auth.email}:${auth.apiToken}`),\n Accept: \"application/json\",\n },\n })\n\n if (!response.ok) {\n throw new Error(`${response.status} ${await response.text()}`)\n }\n\n return await response.json()\n}\n",
"schema": {
"$schema": "https://json-schema.org/draft/2020-12/schema",
"properties": {
"auth": {
"default": null,
"description": "",
"format": "resource-confluence",
"type": "object"
},
"cursor": {
"default": null,
"description": "",
"originalType": "string",
"type": "string"
},
"keys": {
"default": null,
"description": "",
"items": {
"type": "string"
},
"originalType": "string[]",
"type": "array"
},
"limit": {
"default": null,
"description": "",
"type": "number"
},
"sort": {
"default": null,
"description": "",
"enum": [
"id",
"-id",
"key",
"-key",
"name",
"-name"
],
"originalType": "enum",
"type": "string"
},
"status": {
"default": null,
"description": "",
"enum": [
"current",
"archived"
],
"originalType": "enum",
"type": "string"
},
"type": {
"default": null,
"description": "",
"enum": [
"global",
"collaboration",
"knowledge_base",
"personal"
],
"originalType": "enum",
"type": "string"
}
},
"required": [
"auth"
],
"type": "object"
}
},
{
"version_id": 28775,
"app": "confluence",
"summary": "Search Content (CQL)",
"description": "Search for content using the Confluence Query Language (CQL).",
"terms": "confluence search content (cql) search for content using the confluence query language (cql).",
"kind": "script",
"language": "bunnative",
"content": "//native\n\n/**\n * Search Content (CQL)\n * Search for content using the Confluence Query Language (CQL), e.g. `type=page AND space=ENG AND text ~ \"release notes\"`.\n */\nexport async function main(\n auth: RT.Confluence,\n cql: string,\n cursor: string | undefined,\n limit: number | undefined,\n expand: string | undefined\n) {\n const base = auth.baseUrl.replace(/\\/$/, \"\")\n const url = new URL(`${base}/wiki/rest/api/search`)\n url.searchParams.append(\"cql\", cql)\n if (cursor !== undefined && cursor !== \"\")\n url.searchParams.append(\"cursor\", cursor)\n if (limit !== undefined) url.searchParams.append(\"limit\", String(limit))\n if (expand !== undefined && expand !== \"\")\n url.searchParams.append(\"expand\", expand)\n\n const response = await fetch(url, {\n method: \"GET\",\n headers: {\n Authorization: \"Basic \" + btoa(`${auth.email}:${auth.apiToken}`),\n Accept: \"application/json\",\n },\n })\n\n if (!response.ok) {\n throw new Error(`${response.status} ${await response.text()}`)\n }\n\n return await response.json()\n}\n",
"schema": {
"$schema": "https://json-schema.org/draft/2020-12/schema",
"properties": {
"auth": {
"default": null,
"description": "",
"format": "resource-confluence",
"type": "object"
},
"cql": {
"default": null,
"description": "",
"originalType": "string",
"type": "string"
},
"cursor": {
"default": null,
"description": "",
"originalType": "string",
"type": "string"
},
"expand": {
"default": null,
"description": "",
"originalType": "string",
"type": "string"
},
"limit": {
"default": null,
"description": "",
"type": "number"
}
},
"required": [
"auth",
"cql"
],
"type": "object"
}
},
{
"version_id": 28776,
"app": "confluence",
"summary": "Update Blog Post",
"description": "Update a blog post's title, body, or status. The current version is fetched automatically and incremented.",
"terms": "confluence update blog post update a blog post's title, body, or status. the current version is fetched automatically and incremented.",
"kind": "script",
"language": "bunnative",
"content": "//native\n\n/**\n * Update Blog Post\n * Update a blog post's title, body, or status. The current version is fetched automatically and incremented; omitted fields keep their existing values.\n */\nexport async function main(\n auth: RT.Confluence,\n blog_post_id: string,\n title: string | undefined,\n body: string | undefined,\n status: \"current\" | \"draft\" | \"archived\" | undefined,\n representation: \"storage\" | \"atlas_doc_format\" | undefined,\n version_message: string | undefined\n) {\n const base = auth.baseUrl.replace(/\\/$/, \"\")\n const repr = representation ?? \"storage\"\n const authHeader = \"Basic \" + btoa(`${auth.email}:${auth.apiToken}`)\n\n // Fetch the current blog post to obtain the version number and any fields not being changed.\n const currentRes = await fetch(\n `${base}/wiki/api/v2/blogposts/${blog_post_id}?body-format=${repr}`,\n {\n method: \"GET\",\n headers: { Authorization: authHeader, Accept: \"application/json\" },\n }\n )\n if (!currentRes.ok) {\n throw new Error(`${currentRes.status} ${await currentRes.text()}`)\n }\n const current = (await currentRes.json()) as {\n title: string\n status: string\n version: { number: number }\n body?: { [key: string]: { value?: string } }\n }\n\n const payload = {\n id: blog_post_id,\n status: status ?? current.status,\n title: title ?? current.title,\n body: {\n representation: repr,\n value: body ?? current.body?.[repr]?.value ?? \"\",\n },\n version: {\n number: current.version.number + 1,\n message: version_message,\n },\n }\n\n const response = await fetch(\n `${base}/wiki/api/v2/blogposts/${blog_post_id}`,\n {\n method: \"PUT\",\n headers: {\n Authorization: authHeader,\n \"Content-Type\": \"application/json\",\n Accept: \"application/json\",\n },\n body: JSON.stringify(payload),\n }\n )\n\n if (!response.ok) {\n throw new Error(`${response.status} ${await response.text()}`)\n }\n\n return await response.json()\n}\n",
"schema": {
"$schema": "https://json-schema.org/draft/2020-12/schema",
"properties": {
"auth": {
"default": null,
"description": "",
"format": "resource-confluence",
"type": "object"
},
"blog_post_id": {
"default": null,
"description": "",
"originalType": "string",
"type": "string"
},
"body": {
"default": null,
"description": "",
"originalType": "string",
"type": "string"
},
"representation": {
"default": null,
"description": "",
"enum": [
"storage",
"atlas_doc_format"
],
"originalType": "enum",
"type": "string"
},
"status": {
"default": null,
"description": "",
"enum": [
"current",
"draft",
"archived"
],
"originalType": "enum",
"type": "string"
},
"title": {
"default": null,
"description": "",
"originalType": "string",
"type": "string"
},
"version_message": {
"default": null,
"description": "",
"originalType": "string",
"type": "string"
}
},
"required": [
"auth",
"blog_post_id"
],
"type": "object"
}
},
{
"version_id": 28777,
"app": "confluence",
"summary": "Update Page",
"description": "Update a page's title, body, or status. The current version is fetched automatically and incremented.",
"terms": "confluence update page update a page's title, body, or status. the current version is fetched automatically and incremented.",
"kind": "script",
"language": "bunnative",
"content": "//native\n\n/**\n * Update Page\n * Update a page's title, body, or status. The current version is fetched automatically and incremented; omitted fields keep their existing values.\n */\nexport async function main(\n auth: RT.Confluence,\n page_id: string,\n title: string | undefined,\n body: string | undefined,\n status: \"current\" | \"draft\" | \"archived\" | undefined,\n representation: \"storage\" | \"atlas_doc_format\" | undefined,\n version_message: string | undefined\n) {\n const base = auth.baseUrl.replace(/\\/$/, \"\")\n const repr = representation ?? \"storage\"\n const authHeader = \"Basic \" + btoa(`${auth.email}:${auth.apiToken}`)\n\n // Fetch the current page to obtain the version number and any fields not being changed.\n const currentRes = await fetch(\n `${base}/wiki/api/v2/pages/${page_id}?body-format=${repr}`,\n {\n method: \"GET\",\n headers: { Authorization: authHeader, Accept: \"application/json\" },\n }\n )\n if (!currentRes.ok) {\n throw new Error(`${currentRes.status} ${await currentRes.text()}`)\n }\n const current = (await currentRes.json()) as {\n title: string\n status: string\n version: { number: number }\n body?: { [key: string]: { value?: string } }\n }\n\n const payload = {\n id: page_id,\n status: status ?? current.status,\n title: title ?? current.title,\n body: {\n representation: repr,\n value: body ?? current.body?.[repr]?.value ?? \"\",\n },\n version: {\n number: current.version.number + 1,\n message: version_message,\n },\n }\n\n const response = await fetch(`${base}/wiki/api/v2/pages/${page_id}`, {\n method: \"PUT\",\n headers: {\n Authorization: authHeader,\n \"Content-Type\": \"application/json\",\n Accept: \"application/json\",\n },\n body: JSON.stringify(payload),\n })\n\n if (!response.ok) {\n throw new Error(`${response.status} ${await response.text()}`)\n }\n\n return await response.json()\n}\n",
"schema": {
"$schema": "https://json-schema.org/draft/2020-12/schema",
"properties": {
"auth": {
"default": null,
"description": "",
"format": "resource-confluence",
"type": "object"
},
"body": {
"default": null,
"description": "",
"originalType": "string",
"type": "string"
},
"page_id": {
"default": null,
"description": "",
"originalType": "string",
"type": "string"
},
"representation": {
"default": null,
"description": "",
"enum": [
"storage",
"atlas_doc_format"
],
"originalType": "enum",
"type": "string"
},
"status": {
"default": null,
"description": "",
"enum": [
"current",
"draft",
"archived"
],
"originalType": "enum",
"type": "string"
},
"title": {
"default": null,
"description": "",
"originalType": "string",
"type": "string"
},
"version_message": {
"default": null,
"description": "",
"originalType": "string",
"type": "string"
}
},
"required": [
"auth",
"page_id"
],
"type": "object"
}
},
{
"version_id": 28778,
"app": "confluence",
"summary": "New or Updated Page",
"description": "Emits pages created or updated since the last poll, tracked via Windmill state. Optionally restrict to a single space.",
"terms": "confluence new or updated page emits pages created or updated since the last poll, tracked via windmill state. optionally restrict to a single space.",
"kind": "trigger",
"language": "bunnative",
"content": "//native\n\nimport * as wmill from \"windmill-client\"\n\n/**\n * New or Updated Page\n * Emits pages created or updated since the last poll, tracked via Windmill state. Optionally restrict to a single space.\n */\nexport async function main(auth: RT.Confluence, space_id: string | undefined) {\n const base = auth.baseUrl.replace(/\\/$/, \"\")\n const lastChecked: number = (await wmill.getState()) ?? 0\n\n const path = space_id\n ? `/wiki/api/v2/spaces/${space_id}/pages`\n : `/wiki/api/v2/pages`\n const url = new URL(`${base}${path}`)\n url.searchParams.append(\"sort\", \"-modified-date\")\n url.searchParams.append(\"limit\", \"50\")\n\n const response = await fetch(url, {\n method: \"GET\",\n headers: {\n Authorization: \"Basic \" + btoa(`${auth.email}:${auth.apiToken}`),\n Accept: \"application/json\",\n },\n })\n\n if (!response.ok) {\n throw new Error(`${response.status} ${await response.text()}`)\n }\n\n const data = (await response.json()) as {\n results?: { version?: { createdAt?: string } }[]\n }\n const pages = data.results ?? []\n\n const ts = (p: { version?: { createdAt?: string } }) =>\n p.version?.createdAt ? Date.parse(p.version.createdAt) : 0\n const maxTs = pages.reduce((m, p) => Math.max(m, ts(p)), 0)\n\n // First run: set the watermark to the newest page seen and don't emit a backlog.\n if (!lastChecked) {\n if (maxTs) await wmill.setState(maxTs)\n return []\n }\n\n const fresh = pages\n .filter((p) => ts(p) > lastChecked)\n .sort((a, b) => ts(a) - ts(b))\n\n if (maxTs > lastChecked) {\n await wmill.setState(maxTs)\n }\n\n return fresh\n}\n",
"schema": {
"$schema": "https://json-schema.org/draft/2020-12/schema",
"properties": {
"auth": {
"default": null,
"description": "",
"format": "resource-confluence",
"type": "object"
},
"space_id": {
"default": null,
"description": "",
"originalType": "string",
"type": "string"
}
},
"required": [
"auth"
],
"type": "object"
}
}
]
}
ai_evals/fixtures/frontend/global/hub/outreach.json
+1055
View File
File diff suppressed because it is too large Load Diff
ai_evals/fixtures/frontend/global/hub/servicenow.json
+672
View File
@@ -0,0 +1,672 @@
{
"app": "servicenow",
"display_name": "ServiceNow",
"description": "IT service management: incidents, records and attachments.",
"docs_url": "https://www.servicenow.com/docs/bundle/zurich-api-reference/page/integrate/inbound-rest/concept/c_RESTAPI.html",
"curated": true,
"meta": {
"api_docs": "https://www.servicenow.com/docs/bundle/zurich-api-reference/page/integrate/inbound-rest/concept/c_RESTAPI.html",
"shipped_surfaces": [
"table",
"stats",
"attachment",
"import_set",
"user"
],
"validation": {
"status": "live_validated",
"method": "All surfaces run live against a ServiceNow PDI (a developer instance, Zurich) on 2026-06-04 with Basic auth: Table CRUD (create / get / update PATCH+PUT / delete on incident, delete verified by 404), list_records (encoded query + fields + pagination), aggregate (Stats group-by count), get_current_user, list_tables, and the table/fields/group_by dynselect resolvers. Attachment upload (binary /file) -> list -> download -> delete round-trip with the downloaded bytes byte-for-byte matching the upload. insert_import_set + get_import_set_result against imp_user (transform status surfaced). Trigger e2e-tested via bun test with the windmill-client fake. All created records cleaned up.",
"sources": {
"table_crud": "HIGH - run live (create/get/update PATCH & PUT/delete on incident)",
"aggregate_stats": "HIGH - run live (/api/now/stats group-by count on incident)",
"attachment": "HIGH - run live (binary /file upload -> list -> download -> delete round-trip; downloaded bytes verified equal)",
"import_set": "HIGH - run live (POST /api/now/import/imp_user + GET result; top-level {import_set,staging_table,result} envelope, transform status surfaced)",
"user": "HIGH - run live (/api/now/ui/user/current_user -> admin sys_user)",
"trigger": "HIGH - e2e-tested live via bun test (first run sets watermark + no backlog, second run emits the new record)",
"dropdowns": "HIGH - run live (sys_db_object table list; sys_dictionary super_class inheritance walk confirmed - incident resolves 106 fields incl. task-inherited short_description/priority/state)"
},
"low_confidence_fields": [
"trigger datetime comparison assumes the integration user's timezone is UTC/GMT (validated with the PDI admin user, whose TZ is UTC); a non-UTC integration user could shift the sys_updated_on> watermark boundary by the offset"
]
},
"pagination": {
"pattern": "offset",
"request_params": {
"offset": "sysparm_offset",
"limit": "sysparm_limit"
},
"response_fields": {
"next": "Link (response header, rel=\"next\")",
"total": "X-Total-Count (response header)"
},
"default_limit": 10000,
"notes": "Table API pages with sysparm_offset + sysparm_limit. Each page response carries a Link header (RFC 5988) with rel=\"first\"/\"prev\"/\"next\"/\"last\" full URLs, plus X-Total-Count with the total match count. Add sysparm_no_count=true to skip the count on huge tables. The list_records action exposes limit+offset and returns one page; iterate offset until fewer than limit rows return."
},
"gotchas": [
"Base path is {instance_url}/api/now; instance_url lives on the resource. Auth is Basic (username:password) OR Bearer token \u2014 scripts send Bearer when auth.token is set, otherwise Basic. Both hit identical paths.",
"Table/Attachment/Stats reads/writes return a {\"result\": ...} envelope: list/stats -> result is an array, get/create/update -> result is an object. EXCEPTION: the Import Set API (POST /api/now/import/{staging}, GET .../{sys_id}) returns a TOP-LEVEL object {import_set, staging_table, result:[...]} with NO outer 'result' wrapper. Scripts return the raw JSON (envelope intact).",
"Datetimes are 'YYYY-MM-DD HH:MM:SS' in UTC with NO trailing 'Z'. Used directly in encoded queries, e.g. sys_updated_on>2026-01-01 00:00:00.",
"sysparm_query is an ENCODED query string: '^' = AND clause separator, '^OR' = OR, operators =, !=, LIKE, >=, <=, IN, STARTSWITH, ORDERBY, ORDERBYDESC. A literal '^' inside a value must be doubled to '^^'. Example: active=true^priority=1^ORDERBYDESCsys_updated_on.",
"System fields sys_id, sys_created_on, sys_updated_on, sys_created_by are auto-managed; do not send them on create/update.",
"DELETE (record and attachment) returns 204 No Content with an empty body \u2014 guard with `if (response.status === 204) return { success: true }` before response.json().",
"Update: PATCH partial-updates; PUT replaces the whole record (clears omitted fields). update_record sends PUT when replace=true, PATCH otherwise.",
"Attachment upload uses POST /api/now/attachment/file with table_name, table_sys_id, file_name in the QUERY STRING and the raw file bytes as the body (Content-Type = the file's MIME type), NOT multipart. Returns 201. Download is GET /api/now/attachment/{sys_id}/file (raw bytes).",
"Import Set transform runs SYNCHRONOUSLY. A result[].status of 'error' is an application-level failure even on HTTP 200/201 \u2014 inspect result[].status / error_message, don't trust the HTTP code alone.",
"Table-name DROPDOWN comes from GET /api/now/table/sys_db_object?sysparm_fields=name,label (use the 'name' as the value). A table's COLUMNS come from sys_dictionary \u2014 but sys_dictionary stores each field once, under the table where it is DEFINED, so name=<table> returns ONLY that table's own fields. Extended tables (incident/change_request/sc_task all extend task) inherit columns like short_description/priority/assigned_to/state that live on the parent. To get the full column set, walk sys_db_object.super_class up the hierarchy (GET sys_db_object?sysparm_query=name=<t>&sysparm_fields=super_class.name, repeat) and query GET sys_dictionary?sysparm_query=nameIN<table>,<ancestors>^elementISNOTEMPTY&sysparm_fields=element,column_label (elementISNOTEMPTY drops the collection/meta row).",
"Table API enforces row- and field-level ACLs: a 200 can omit fields the user can't read; inaccessible tables/ops return 403."
],
"enums": {
"sysparm_display_value": [
"true",
"false",
"all"
],
"sysparm_view": [
"desktop",
"mobile",
"both"
],
"import_set_result_status": [
"inserted",
"updated",
"ignored",
"error"
],
"incident_state_note": "incident.state and priority/urgency/impact are instance-configurable; defaults: state 1=New,2=In Progress,3=On Hold,6=Resolved,7=Closed,8=Canceled; priority/urgency/impact 1-5. On task/incident, priority is CALCULATED from impact+urgency (a priority value sent on create/update is recomputed/ignored) \u2014 set impact+urgency instead. Never hardcode for generic actions."
},
"errors": {
"401": "Authentication failed or OAuth token expired/invalid. Check username/password or refresh the bearer token.",
"403": "ACL denies the table or operation, or the user lacks the required role (e.g. rest_api_explorer / table ACL). Response: {\"error\":{\"message\":\"User Not Authorized\"}}.",
"404": "No Record found (bad sys_id) or Invalid table (bad/inaccessible table name, common Import Set staging-table error).",
"400": "Bad encoded query, invalid field name, or malformed request body."
}
},
"resource_type": {
"name": "servicenow",
"description": "ServiceNow instance credential. Set instance_url to your instance base (https://<instance>.service-now.com). Authenticate with EITHER Basic auth (username + password) OR an OAuth 2.0 bearer token \u2014 scripts send a Bearer header when token is set, otherwise Basic auth.",
"schema": {
"$schema": "https://json-schema.org/draft/2020-12/schema",
"order": [
"instance_url",
"username",
"password",
"token"
],
"properties": {
"instance_url": {
"default": "",
"description": "Instance base URL, e.g. https://dev12345.service-now.com (no trailing slash). Every REST call is made against this host.",
"placeholder": "https://dev12345.service-now.com",
"type": "string"
},
"password": {
"default": "",
"description": "Basic auth password. Leave blank when using an OAuth bearer token.",
"password": true,
"type": "string"
},
"token": {
"default": "",
"description": "OAuth 2.0 access token (Bearer). Populated automatically when you connect via Windmill's ServiceNow OAuth flow (admin configures the instance + client id/secret in instance settings), or paste one manually. Provide this instead of username/password to authenticate with OAuth.",
"password": true,
"type": "string"
},
"username": {
"default": "",
"description": "Basic auth username. Leave blank when using an OAuth bearer token.",
"type": "string"
}
},
"required": [
"instance_url"
],
"type": "object"
}
},
"scripts": [
{
"version_id": 28665,
"app": "servicenow",
"summary": "Aggregate Records",
"description": "Count records on a table (Aggregate API), optionally filtered by an encoded query and grouped by a field.",
"terms": "servicenow aggregate records count records on a table (aggregate api), optionally filtered by an encoded query and grouped by a field.",
"kind": "script",
"language": "bunnative",
"content": "//native\n\nexport type DynSelect_table = string\nexport type DynSelect_group_by = string\n\nfunction authHeader(auth: RT.Servicenow) {\n return auth.token\n ? `Bearer ${auth.token}`\n : `Basic ${btoa(`${auth.username}:${auth.password}`)}`\n}\n\n// Dropdown of the instance's tables (sys_db_object).\nexport async function table(auth: RT.Servicenow) {\n const response = await fetch(\n `${auth.instance_url}/api/now/table/sys_db_object?sysparm_fields=name,label&sysparm_limit=10000`,\n {\n headers: {\n Authorization: authHeader(auth),\n Accept: \"application/json\",\n },\n }\n )\n if (!response.ok) {\n throw new Error(`${response.status} ${await response.text()}`)\n }\n const { result } = (await response.json()) as {\n result: { name: string; label: string }[]\n }\n return result\n .filter((t) => t.name)\n .map((t) => ({ value: t.name, label: `${t.label} (${t.name})` }))\n .sort((a, b) => a.label.localeCompare(b.label))\n}\n\n// Walk the table's inheritance chain (sys_db_object.super_class) so the group-by\n// options include fields inherited from parent tables, not just fields defined\n// on the chosen table (sys_dictionary stores each field under its defining\n// table \u2014 e.g. incident extends task, so priority/state live on task).\n// Best-effort: any failure falls back to just the table itself.\nasync function tableChain(auth: RT.Servicenow, table: string) {\n const chain: string[] = []\n let current: string | undefined = table\n while (current && !chain.includes(current)) {\n chain.push(current)\n const response = await fetch(\n `${auth.instance_url}/api/now/table/sys_db_object?sysparm_query=name=${current}&sysparm_fields=super_class.name&sysparm_limit=1`,\n {\n headers: {\n Authorization: authHeader(auth),\n Accept: \"application/json\",\n },\n }\n )\n if (!response.ok) break\n const { result } = (await response.json()) as {\n result: { \"super_class.name\"?: string }[]\n }\n current = result[0]?.[\"super_class.name\"] || undefined\n }\n return chain\n}\n\n// Dependent select: the chosen table's columns (sys_dictionary) to group by,\n// including columns inherited from parent tables.\nexport async function group_by(auth: RT.Servicenow, table: DynSelect_table) {\n if (!table) return []\n const chain = await tableChain(auth, table)\n const url = new URL(`${auth.instance_url}/api/now/table/sys_dictionary`)\n url.searchParams.append(\n \"sysparm_query\",\n `nameIN${chain.join(\",\")}^elementISNOTEMPTY`\n )\n url.searchParams.append(\"sysparm_fields\", \"element,column_label\")\n url.searchParams.append(\"sysparm_limit\", \"10000\")\n\n const response = await fetch(url, {\n headers: {\n Authorization: authHeader(auth),\n Accept: \"application/json\",\n },\n })\n if (!response.ok) {\n throw new Error(`${response.status} ${await response.text()}`)\n }\n const { result } = (await response.json()) as {\n result: { element: string; column_label: string }[]\n }\n return result\n .filter((f) => f.element)\n .map((f) => ({\n value: f.element,\n label: `${f.column_label} (${f.element})`,\n }))\n .sort((a, b) => a.label.localeCompare(b.label))\n}\n\n/**\n * Aggregate Records\n * Count records on a table (Aggregate API), optionally filtered by an encoded query and grouped by a field. Returns ServiceNow's stats envelope with the count per group.\n */\nexport async function main(\n auth: RT.Servicenow,\n table: DynSelect_table,\n query: string | undefined,\n group_by: DynSelect_group_by | undefined\n) {\n const url = new URL(`${auth.instance_url}/api/now/stats/${table}`)\n url.searchParams.append(\"sysparm_count\", \"true\")\n if (query !== undefined && query !== \"\") {\n url.searchParams.append(\"sysparm_query\", query)\n }\n if (group_by !== undefined && group_by !== \"\") {\n url.searchParams.append(\"sysparm_group_by\", group_by)\n }\n\n const response = await fetch(url, {\n method: \"GET\",\n headers: {\n Authorization: authHeader(auth),\n Accept: \"application/json\",\n },\n })\n\n if (!response.ok) {\n throw new Error(`${response.status} ${await response.text()}`)\n }\n\n return await response.json()\n}\n",
"schema": {
"$schema": "https://json-schema.org/draft/2020-12/schema",
"properties": {
"auth": {
"default": null,
"description": "",
"format": "resource-servicenow",
"type": "object"
},
"group_by": {
"default": null,
"description": "",
"format": "dynselect-group_by",
"type": "object"
},
"query": {
"default": null,
"description": "",
"originalType": "string",
"type": "string"
},
"table": {
"default": null,
"description": "",
"format": "dynselect-table",
"type": "object"
}
},
"required": [
"auth",
"table"
],
"type": "object"
}
},
{
"version_id": 28666,
"app": "servicenow",
"summary": "Create Record",
"description": "Insert a record into any table (incident, change_request, sys_user, cmdb_ci, or a custom table). Pick the table from a live dropdown and pass a column/value map.",
"terms": "servicenow create record insert a record into any table (incident, change_request, sys_user, cmdb_ci, or a custom table). pick the table from a live dropdown and pass a column/value map.",
"kind": "script",
"language": "bunnative",
"content": "//native\n\nexport type DynSelect_table = string\n\nfunction authHeader(auth: RT.Servicenow) {\n return auth.token\n ? `Bearer ${auth.token}`\n : `Basic ${btoa(`${auth.username}:${auth.password}`)}`\n}\n\n// Dropdown of the instance's tables (sys_db_object).\nexport async function table(auth: RT.Servicenow) {\n const response = await fetch(\n `${auth.instance_url}/api/now/table/sys_db_object?sysparm_fields=name,label&sysparm_limit=10000`,\n {\n headers: {\n Authorization: authHeader(auth),\n Accept: \"application/json\",\n },\n }\n )\n if (!response.ok) {\n throw new Error(`${response.status} ${await response.text()}`)\n }\n const { result } = (await response.json()) as {\n result: { name: string; label: string }[]\n }\n return result\n .filter((t) => t.name)\n .map((t) => ({ value: t.name, label: `${t.label} (${t.name})` }))\n .sort((a, b) => a.label.localeCompare(b.label))\n}\n\n/**\n * Create Record\n * Insert a record into any table (incident, change_request, sys_user, cmdb_ci, or a custom table). Pick the table from a live dropdown and pass a column/value map.\n */\nexport async function main(\n auth: RT.Servicenow,\n table: DynSelect_table,\n body: { [key: string]: any }\n) {\n const url = new URL(`${auth.instance_url}/api/now/table/${table}`)\n\n const response = await fetch(url, {\n method: \"POST\",\n headers: {\n Authorization: authHeader(auth),\n \"Content-Type\": \"application/json\",\n Accept: \"application/json\",\n },\n body: JSON.stringify(body),\n })\n\n if (!response.ok) {\n throw new Error(`${response.status} ${await response.text()}`)\n }\n\n return await response.json()\n}\n",
"schema": {
"$schema": "https://json-schema.org/draft/2020-12/schema",
"properties": {
"auth": {
"default": null,
"description": "",
"format": "resource-servicenow",
"type": "object"
},
"body": {
"default": null,
"description": "",
"properties": {},
"type": "object"
},
"table": {
"default": null,
"description": "",
"format": "dynselect-table",
"type": "object"
}
},
"required": [
"auth",
"table",
"body"
],
"type": "object"
}
},
{
"version_id": 28667,
"app": "servicenow",
"summary": "Delete Attachment",
"description": "Delete an attachment by sys_id. Returns {success, sys_id} (ServiceNow replies 204 No Content).",
"terms": "servicenow delete attachment delete an attachment by sys_id. returns {success, sys_id} (servicenow replies 204 no content).",
"kind": "script",
"language": "bunnative",
"content": "//native\n\nfunction authHeader(auth: RT.Servicenow) {\n return auth.token\n ? `Bearer ${auth.token}`\n : `Basic ${btoa(`${auth.username}:${auth.password}`)}`\n}\n\n/**\n * Delete Attachment\n * Delete an attachment by its sys_id. ServiceNow returns 204 No Content on success, so this returns {success, sys_id}.\n */\nexport async function main(auth: RT.Servicenow, attachment_sys_id: string) {\n const response = await fetch(\n `${auth.instance_url}/api/now/attachment/${attachment_sys_id}`,\n {\n method: \"DELETE\",\n headers: {\n Authorization: authHeader(auth),\n Accept: \"application/json\",\n },\n }\n )\n\n if (!response.ok) {\n throw new Error(`${response.status} ${await response.text()}`)\n }\n\n return { success: true, sys_id: attachment_sys_id }\n}\n",
"schema": {
"$schema": "https://json-schema.org/draft/2020-12/schema",
"properties": {
"attachment_sys_id": {
"default": null,
"description": "",
"originalType": "string",
"type": "string"
},
"auth": {
"default": null,
"description": "",
"format": "resource-servicenow",
"type": "object"
}
},
"required": [
"auth",
"attachment_sys_id"
],
"type": "object"
}
},
{
"version_id": 28668,
"app": "servicenow",
"summary": "Delete Record",
"description": "Delete a record by sys_id. Returns {success, sys_id} (ServiceNow replies 204 No Content).",
"terms": "servicenow delete record delete a record by sys_id. returns {success, sys_id} (servicenow replies 204 no content).",
"kind": "script",
"language": "bunnative",
"content": "//native\n\nexport type DynSelect_table = string\n\nfunction authHeader(auth: RT.Servicenow) {\n return auth.token\n ? `Bearer ${auth.token}`\n : `Basic ${btoa(`${auth.username}:${auth.password}`)}`\n}\n\n// Dropdown of the instance's tables (sys_db_object).\nexport async function table(auth: RT.Servicenow) {\n const response = await fetch(\n `${auth.instance_url}/api/now/table/sys_db_object?sysparm_fields=name,label&sysparm_limit=10000`,\n {\n headers: {\n Authorization: authHeader(auth),\n Accept: \"application/json\",\n },\n }\n )\n if (!response.ok) {\n throw new Error(`${response.status} ${await response.text()}`)\n }\n const { result } = (await response.json()) as {\n result: { name: string; label: string }[]\n }\n return result\n .filter((t) => t.name)\n .map((t) => ({ value: t.name, label: `${t.label} (${t.name})` }))\n .sort((a, b) => a.label.localeCompare(b.label))\n}\n\n/**\n * Delete Record\n * Delete a record by sys_id. ServiceNow returns 204 No Content on success, so this returns {success, sys_id}.\n */\nexport async function main(\n auth: RT.Servicenow,\n table: DynSelect_table,\n sys_id: string\n) {\n const url = new URL(`${auth.instance_url}/api/now/table/${table}/${sys_id}`)\n\n const response = await fetch(url, {\n method: \"DELETE\",\n headers: {\n Authorization: authHeader(auth),\n Accept: \"application/json\",\n },\n })\n\n if (!response.ok) {\n throw new Error(`${response.status} ${await response.text()}`)\n }\n\n return { success: true, sys_id }\n}\n",
"schema": {
"$schema": "https://json-schema.org/draft/2020-12/schema",
"properties": {
"auth": {
"default": null,
"description": "",
"format": "resource-servicenow",
"type": "object"
},
"sys_id": {
"default": null,
"description": "",
"originalType": "string",
"type": "string"
},
"table": {
"default": null,
"description": "",
"format": "dynselect-table",
"type": "object"
}
},
"required": [
"auth",
"table",
"sys_id"
],
"type": "object"
}
},
{
"version_id": 28670,
"app": "servicenow",
"summary": "Get Current User",
"description": "Return the sys_user record of the authenticated user.",
"terms": "servicenow get current user return the sys_user record of the authenticated user.",
"kind": "script",
"language": "bunnative",
"content": "//native\n\nfunction authHeader(auth: RT.Servicenow) {\n return auth.token\n ? `Bearer ${auth.token}`\n : `Basic ${btoa(`${auth.username}:${auth.password}`)}`\n}\n\n/**\n * Get Current User\n * Return the sys_user record of the authenticated user (resolves current_user, then fetches the full record from sys_user).\n */\nexport async function main(auth: RT.Servicenow) {\n const currentResponse = await fetch(\n `${auth.instance_url}/api/now/ui/user/current_user`,\n {\n method: \"GET\",\n headers: {\n Authorization: authHeader(auth),\n Accept: \"application/json\",\n },\n }\n )\n if (!currentResponse.ok) {\n throw new Error(`${currentResponse.status} ${await currentResponse.text()}`)\n }\n const { result } = (await currentResponse.json()) as {\n result: { user_sys_id: string }\n }\n\n const userResponse = await fetch(\n `${auth.instance_url}/api/now/table/sys_user/${result.user_sys_id}`,\n {\n method: \"GET\",\n headers: {\n Authorization: authHeader(auth),\n Accept: \"application/json\",\n },\n }\n )\n if (!userResponse.ok) {\n throw new Error(`${userResponse.status} ${await userResponse.text()}`)\n }\n\n return await userResponse.json()\n}\n",
"schema": {
"$schema": "https://json-schema.org/draft/2020-12/schema",
"properties": {
"auth": {
"default": null,
"description": "",
"format": "resource-servicenow",
"type": "object"
}
},
"required": [
"auth"
],
"type": "object"
}
},
{
"version_id": 28671,
"app": "servicenow",
"summary": "Get Import Set Result",
"description": "Fetch the outcome of a prior import row by the staging-row sys_id returned by Insert Into Import Set (result.sys_id).",
"terms": "servicenow get import set result fetch the outcome of a prior import row by the staging-row sys_id returned by insert into import set (result.sys_id).",
"kind": "script",
"language": "bunnative",
"content": "//native\n\nfunction authHeader(auth: RT.Servicenow) {\n return auth.token\n ? `Bearer ${auth.token}`\n : `Basic ${btoa(`${auth.username}:${auth.password}`)}`\n}\n\n/**\n * Get Import Set Result\n * Fetch the outcome of a prior import row by the staging-row sys_id returned by Insert Into Import Set (result.sys_id), returning the transform result (target table, status, target record link).\n */\nexport async function main(\n auth: RT.Servicenow,\n staging_table: string,\n sys_id: string\n) {\n const url = new URL(\n `${auth.instance_url}/api/now/import/${staging_table}/${sys_id}`\n )\n\n const response = await fetch(url, {\n method: \"GET\",\n headers: {\n Authorization: authHeader(auth),\n Accept: \"application/json\",\n },\n })\n\n if (!response.ok) {\n throw new Error(`${response.status} ${await response.text()}`)\n }\n\n return await response.json()\n}\n",
"schema": {
"$schema": "https://json-schema.org/draft/2020-12/schema",
"properties": {
"auth": {
"default": null,
"description": "",
"format": "resource-servicenow",
"type": "object"
},
"staging_table": {
"default": null,
"description": "",
"originalType": "string",
"type": "string"
},
"sys_id": {
"default": null,
"description": "",
"originalType": "string",
"type": "string"
}
},
"required": [
"auth",
"staging_table",
"sys_id"
],
"type": "object"
}
},
{
"version_id": 28672,
"app": "servicenow",
"summary": "Get Record",
"description": "Retrieve a single record by sys_id. Optionally restrict the returned columns, and choose raw vs. display values.",
"terms": "servicenow get record retrieve a single record by sys_id. optionally restrict the returned columns, and choose raw vs. display values.",
"kind": "script",
"language": "bunnative",
"content": "//native\n\nexport type DynSelect_table = string\nexport type DynMultiselect_fields = string[]\n\nfunction authHeader(auth: RT.Servicenow) {\n return auth.token\n ? `Bearer ${auth.token}`\n : `Basic ${btoa(`${auth.username}:${auth.password}`)}`\n}\n\n// Dropdown of the instance's tables (sys_db_object).\nexport async function table(auth: RT.Servicenow) {\n const response = await fetch(\n `${auth.instance_url}/api/now/table/sys_db_object?sysparm_fields=name,label&sysparm_limit=10000`,\n {\n headers: {\n Authorization: authHeader(auth),\n Accept: \"application/json\",\n },\n }\n )\n if (!response.ok) {\n throw new Error(`${response.status} ${await response.text()}`)\n }\n const { result } = (await response.json()) as {\n result: { name: string; label: string }[]\n }\n return result\n .filter((t) => t.name)\n .map((t) => ({ value: t.name, label: `${t.label} (${t.name})` }))\n .sort((a, b) => a.label.localeCompare(b.label))\n}\n\n// Walk the table's inheritance chain (sys_db_object.super_class) so the column\n// list includes fields inherited from parent tables, not just fields defined on\n// the chosen table (sys_dictionary stores each field under its defining table \u2014\n// e.g. incident extends task, so short_description/priority live on task).\n// Best-effort: any failure falls back to just the table itself.\nasync function tableChain(auth: RT.Servicenow, table: string) {\n const chain: string[] = []\n let current: string | undefined = table\n while (current && !chain.includes(current)) {\n chain.push(current)\n const response = await fetch(\n `${auth.instance_url}/api/now/table/sys_db_object?sysparm_query=name=${current}&sysparm_fields=super_class.name&sysparm_limit=1`,\n {\n headers: {\n Authorization: authHeader(auth),\n Accept: \"application/json\",\n },\n }\n )\n if (!response.ok) break\n const { result } = (await response.json()) as {\n result: { \"super_class.name\"?: string }[]\n }\n current = result[0]?.[\"super_class.name\"] || undefined\n }\n return chain\n}\n\n// Dependent multi-select: the chosen table's columns (sys_dictionary), including\n// columns inherited from parent tables.\nexport async function fields(auth: RT.Servicenow, table: DynSelect_table) {\n if (!table) return []\n const chain = await tableChain(auth, table)\n const url = new URL(`${auth.instance_url}/api/now/table/sys_dictionary`)\n url.searchParams.append(\n \"sysparm_query\",\n `nameIN${chain.join(\",\")}^elementISNOTEMPTY`\n )\n url.searchParams.append(\"sysparm_fields\", \"element,column_label\")\n url.searchParams.append(\"sysparm_limit\", \"10000\")\n\n const response = await fetch(url, {\n headers: {\n Authorization: authHeader(auth),\n Accept: \"application/json\",\n },\n })\n if (!response.ok) {\n throw new Error(`${response.status} ${await response.text()}`)\n }\n const { result } = (await response.json()) as {\n result: { element: string; column_label: string }[]\n }\n return result\n .filter((f) => f.element)\n .map((f) => ({\n value: f.element,\n label: `${f.column_label} (${f.element})`,\n }))\n .sort((a, b) => a.label.localeCompare(b.label))\n}\n\n/**\n * Get Record\n * Retrieve a single record by sys_id. Optionally restrict the returned columns, and choose raw vs. display values.\n */\nexport async function main(\n auth: RT.Servicenow,\n table: DynSelect_table,\n sys_id: string,\n fields: DynMultiselect_fields | undefined,\n display_value: \"false\" | \"true\" | \"all\" | undefined\n) {\n const url = new URL(`${auth.instance_url}/api/now/table/${table}/${sys_id}`)\n if (fields && fields.length > 0) {\n url.searchParams.append(\"sysparm_fields\", fields.join(\",\"))\n }\n if (display_value !== undefined && display_value !== \"\") {\n url.searchParams.append(\"sysparm_display_value\", display_value)\n }\n\n const response = await fetch(url, {\n method: \"GET\",\n headers: {\n Authorization: authHeader(auth),\n Accept: \"application/json\",\n },\n })\n\n if (!response.ok) {\n throw new Error(`${response.status} ${await response.text()}`)\n }\n\n return await response.json()\n}\n",
"schema": {
"$schema": "https://json-schema.org/draft/2020-12/schema",
"properties": {
"auth": {
"default": null,
"description": "",
"format": "resource-servicenow",
"type": "object"
},
"display_value": {
"default": null,
"description": "",
"enum": [
"false",
"true",
"all"
],
"originalType": "enum",
"type": "string"
},
"fields": {
"default": null,
"description": "",
"format": "dynmultiselect-fields",
"type": "object"
},
"sys_id": {
"default": null,
"description": "",
"originalType": "string",
"type": "string"
},
"table": {
"default": null,
"description": "",
"format": "dynselect-table",
"type": "object"
}
},
"required": [
"auth",
"table",
"sys_id"
],
"type": "object"
}
},
{
"version_id": 28673,
"app": "servicenow",
"summary": "Insert Into Import Set",
"description": "Push a row into an import set staging table; the transform map runs synchronously. Inspect result[].status.",
"terms": "servicenow insert into import set push a row into an import set staging table; the transform map runs synchronously. inspect result[].status.",
"kind": "script",
"language": "bunnative",
"content": "//native\n\nfunction authHeader(auth: RT.Servicenow) {\n return auth.token\n ? `Bearer ${auth.token}`\n : `Basic ${btoa(`${auth.username}:${auth.password}`)}`\n}\n\n/**\n * Insert Into Import Set\n * Push a row into an import set staging table; the associated transform map runs synchronously. Inspect result[].status (inserted/updated/ignored/error) \u2014 a transform error returns 200/201 with status \"error\".\n */\nexport async function main(\n auth: RT.Servicenow,\n staging_table: string,\n body: { [key: string]: any }\n) {\n const url = new URL(`${auth.instance_url}/api/now/import/${staging_table}`)\n\n const response = await fetch(url, {\n method: \"POST\",\n headers: {\n Authorization: authHeader(auth),\n \"Content-Type\": \"application/json\",\n Accept: \"application/json\",\n },\n body: JSON.stringify(body),\n })\n\n if (!response.ok) {\n throw new Error(`${response.status} ${await response.text()}`)\n }\n\n return await response.json()\n}\n",
"schema": {
"$schema": "https://json-schema.org/draft/2020-12/schema",
"properties": {
"auth": {
"default": null,
"description": "",
"format": "resource-servicenow",
"type": "object"
},
"body": {
"default": null,
"description": "",
"properties": {},
"type": "object"
},
"staging_table": {
"default": null,
"description": "",
"originalType": "string",
"type": "string"
}
},
"required": [
"auth",
"staging_table",
"body"
],
"type": "object"
}
},
{
"version_id": 28674,
"app": "servicenow",
"summary": "List Attachments",
"description": "List attachment metadata, optionally filtered with an encoded query (e.g. table_name=incident^table_sys_id=<id>).",
"terms": "servicenow list attachments list attachment metadata, optionally filtered with an encoded query (e.g. table_name=incident^table_sys_id=<id>).",
"kind": "script",
"language": "bunnative",
"content": "//native\n\nfunction authHeader(auth: RT.Servicenow) {\n return auth.token\n ? `Bearer ${auth.token}`\n : `Basic ${btoa(`${auth.username}:${auth.password}`)}`\n}\n\n/**\n * List Attachments\n * List attachment metadata (sys_id, file_name, content_type, size, download_link). Filter with an encoded query, e.g. table_name=incident^table_sys_id=<record sys_id>.\n */\nexport async function main(\n auth: RT.Servicenow,\n query: string | undefined,\n limit: number | undefined\n) {\n const url = new URL(`${auth.instance_url}/api/now/attachment`)\n if (query !== undefined && query !== \"\") {\n url.searchParams.append(\"sysparm_query\", query)\n }\n url.searchParams.append(\"sysparm_limit\", String(limit ?? 100))\n\n const response = await fetch(url, {\n method: \"GET\",\n headers: {\n Authorization: authHeader(auth),\n Accept: \"application/json\",\n },\n })\n\n if (!response.ok) {\n throw new Error(`${response.status} ${await response.text()}`)\n }\n\n return await response.json()\n}\n",
"schema": {
"$schema": "https://json-schema.org/draft/2020-12/schema",
"properties": {
"auth": {
"default": null,
"description": "",
"format": "resource-servicenow",
"type": "object"
},
"limit": {
"default": null,
"description": "",
"type": "number"
},
"query": {
"default": null,
"description": "",
"originalType": "string",
"type": "string"
}
},
"required": [
"auth"
],
"type": "object"
}
},
{
"version_id": 28675,
"app": "servicenow",
"summary": "List Records",
"description": "Query records from any table with an encoded query (sysparm_query), pagination, column selection, and raw/display values.",
"terms": "servicenow list records query records from any table with an encoded query (sysparm_query), pagination, column selection, and raw/display values.",
"kind": "script",
"language": "bunnative",
"content": "//native\n\nexport type DynSelect_table = string\nexport type DynMultiselect_fields = string[]\n\nfunction authHeader(auth: RT.Servicenow) {\n return auth.token\n ? `Bearer ${auth.token}`\n : `Basic ${btoa(`${auth.username}:${auth.password}`)}`\n}\n\n// Dropdown of the instance's tables (sys_db_object).\nexport async function table(auth: RT.Servicenow) {\n const response = await fetch(\n `${auth.instance_url}/api/now/table/sys_db_object?sysparm_fields=name,label&sysparm_limit=10000`,\n {\n headers: {\n Authorization: authHeader(auth),\n Accept: \"application/json\",\n },\n }\n )\n if (!response.ok) {\n throw new Error(`${response.status} ${await response.text()}`)\n }\n const { result } = (await response.json()) as {\n result: { name: string; label: string }[]\n }\n return result\n .filter((t) => t.name)\n .map((t) => ({ value: t.name, label: `${t.label} (${t.name})` }))\n .sort((a, b) => a.label.localeCompare(b.label))\n}\n\n// Walk the table's inheritance chain (sys_db_object.super_class) so the column\n// list includes fields inherited from parent tables, not just fields defined on\n// the chosen table (sys_dictionary stores each field under its defining table \u2014\n// e.g. incident extends task, so short_description/priority live on task).\n// Best-effort: any failure falls back to just the table itself.\nasync function tableChain(auth: RT.Servicenow, table: string) {\n const chain: string[] = []\n let current: string | undefined = table\n while (current && !chain.includes(current)) {\n chain.push(current)\n const response = await fetch(\n `${auth.instance_url}/api/now/table/sys_db_object?sysparm_query=name=${current}&sysparm_fields=super_class.name&sysparm_limit=1`,\n {\n headers: {\n Authorization: authHeader(auth),\n Accept: \"application/json\",\n },\n }\n )\n if (!response.ok) break\n const { result } = (await response.json()) as {\n result: { \"super_class.name\"?: string }[]\n }\n current = result[0]?.[\"super_class.name\"] || undefined\n }\n return chain\n}\n\n// Dependent multi-select: the chosen table's columns (sys_dictionary), including\n// columns inherited from parent tables.\nexport async function fields(auth: RT.Servicenow, table: DynSelect_table) {\n if (!table) return []\n const chain = await tableChain(auth, table)\n const url = new URL(`${auth.instance_url}/api/now/table/sys_dictionary`)\n url.searchParams.append(\n \"sysparm_query\",\n `nameIN${chain.join(\",\")}^elementISNOTEMPTY`\n )\n url.searchParams.append(\"sysparm_fields\", \"element,column_label\")\n url.searchParams.append(\"sysparm_limit\", \"10000\")\n\n const response = await fetch(url, {\n headers: {\n Authorization: authHeader(auth),\n Accept: \"application/json\",\n },\n })\n if (!response.ok) {\n throw new Error(`${response.status} ${await response.text()}`)\n }\n const { result } = (await response.json()) as {\n result: { element: string; column_label: string }[]\n }\n return result\n .filter((f) => f.element)\n .map((f) => ({\n value: f.element,\n label: `${f.column_label} (${f.element})`,\n }))\n .sort((a, b) => a.label.localeCompare(b.label))\n}\n\n/**\n * List Records\n * Query records from any table with an encoded query (sysparm_query), pagination, column selection, and raw/display values. Returns one page; iterate offset until fewer than limit rows return.\n */\nexport async function main(\n auth: RT.Servicenow,\n table: DynSelect_table,\n query: string | undefined,\n limit: number | undefined,\n offset: number | undefined,\n fields: DynMultiselect_fields | undefined,\n display_value: \"false\" | \"true\" | \"all\" | undefined\n) {\n const url = new URL(`${auth.instance_url}/api/now/table/${table}`)\n if (query !== undefined && query !== \"\") {\n url.searchParams.append(\"sysparm_query\", query)\n }\n url.searchParams.append(\"sysparm_limit\", String(limit ?? 100))\n if (offset !== undefined) {\n url.searchParams.append(\"sysparm_offset\", String(offset))\n }\n if (fields && fields.length > 0) {\n url.searchParams.append(\"sysparm_fields\", fields.join(\",\"))\n }\n if (display_value !== undefined && display_value !== \"\") {\n url.searchParams.append(\"sysparm_display_value\", display_value)\n }\n\n const response = await fetch(url, {\n method: \"GET\",\n headers: {\n Authorization: authHeader(auth),\n Accept: \"application/json\",\n },\n })\n\n if (!response.ok) {\n throw new Error(`${response.status} ${await response.text()}`)\n }\n\n return await response.json()\n}\n",
"schema": {
"$schema": "https://json-schema.org/draft/2020-12/schema",
"properties": {
"auth": {
"default": null,
"description": "",
"format": "resource-servicenow",
"type": "object"
},
"display_value": {
"default": null,
"description": "",
"enum": [
"false",
"true",
"all"
],
"originalType": "enum",
"type": "string"
},
"fields": {
"default": null,
"description": "",
"format": "dynmultiselect-fields",
"type": "object"
},
"limit": {
"default": null,
"description": "",
"type": "number"
},
"offset": {
"default": null,
"description": "",
"type": "number"
},
"query": {
"default": null,
"description": "",
"originalType": "string",
"type": "string"
},
"table": {
"default": null,
"description": "",
"format": "dynselect-table",
"type": "object"
}
},
"required": [
"auth",
"table"
],
"type": "object"
}
},
{
"version_id": 28676,
"app": "servicenow",
"summary": "List Tables",
"description": "List the instance tables (sys_db_object) with API name, label, and sys_id. Optionally filter by a name/label substring.",
"terms": "servicenow list tables list the instance tables (sys_db_object) with api name, label, and sys_id. optionally filter by a name/label substring.",
"kind": "script",
"language": "bunnative",
"content": "//native\n\nfunction authHeader(auth: RT.Servicenow) {\n return auth.token\n ? `Bearer ${auth.token}`\n : `Basic ${btoa(`${auth.username}:${auth.password}`)}`\n}\n\n/**\n * List Tables\n * List the instance's tables (sys_db_object) with their API name, label, and sys_id. Optionally filter by a substring matched against name or label.\n */\nexport async function main(\n auth: RT.Servicenow,\n name_filter: string | undefined\n) {\n const url = new URL(`${auth.instance_url}/api/now/table/sys_db_object`)\n url.searchParams.append(\"sysparm_fields\", \"name,label,sys_id\")\n if (name_filter !== undefined && name_filter !== \"\") {\n url.searchParams.append(\n \"sysparm_query\",\n `nameLIKE${name_filter}^ORlabelLIKE${name_filter}`\n )\n }\n url.searchParams.append(\"sysparm_limit\", \"10000\")\n\n const response = await fetch(url, {\n method: \"GET\",\n headers: {\n Authorization: authHeader(auth),\n Accept: \"application/json\",\n },\n })\n\n if (!response.ok) {\n throw new Error(`${response.status} ${await response.text()}`)\n }\n\n return await response.json()\n}\n",
"schema": {
"$schema": "https://json-schema.org/draft/2020-12/schema",
"properties": {
"auth": {
"default": null,
"description": "",
"format": "resource-servicenow",
"type": "object"
},
"name_filter": {
"default": null,
"description": "",
"originalType": "string",
"type": "string"
}
},
"required": [
"auth"
],
"type": "object"
}
},
{
"version_id": 28677,
"app": "servicenow",
"summary": "Update Record",
"description": "Update a record by sys_id with a column/value map. PATCH (partial) by default; set replace=true to PUT (full overwrite).",
"terms": "servicenow update record update a record by sys_id with a column/value map. patch (partial) by default; set replace=true to put (full overwrite).",
"kind": "script",
"language": "bunnative",
"content": "//native\n\nexport type DynSelect_table = string\n\nfunction authHeader(auth: RT.Servicenow) {\n return auth.token\n ? `Bearer ${auth.token}`\n : `Basic ${btoa(`${auth.username}:${auth.password}`)}`\n}\n\n// Dropdown of the instance's tables (sys_db_object).\nexport async function table(auth: RT.Servicenow) {\n const response = await fetch(\n `${auth.instance_url}/api/now/table/sys_db_object?sysparm_fields=name,label&sysparm_limit=10000`,\n {\n headers: {\n Authorization: authHeader(auth),\n Accept: \"application/json\",\n },\n }\n )\n if (!response.ok) {\n throw new Error(`${response.status} ${await response.text()}`)\n }\n const { result } = (await response.json()) as {\n result: { name: string; label: string }[]\n }\n return result\n .filter((t) => t.name)\n .map((t) => ({ value: t.name, label: `${t.label} (${t.name})` }))\n .sort((a, b) => a.label.localeCompare(b.label))\n}\n\n/**\n * Update Record\n * Update a record by sys_id with a column/value map. PATCH (partial update) by default; set replace=true to PUT (full overwrite that clears omitted fields).\n */\nexport async function main(\n auth: RT.Servicenow,\n table: DynSelect_table,\n sys_id: string,\n body: { [key: string]: any },\n replace: boolean | undefined\n) {\n const url = new URL(`${auth.instance_url}/api/now/table/${table}/${sys_id}`)\n\n const response = await fetch(url, {\n method: replace ? \"PUT\" : \"PATCH\",\n headers: {\n Authorization: authHeader(auth),\n \"Content-Type\": \"application/json\",\n Accept: \"application/json\",\n },\n body: JSON.stringify(body),\n })\n\n if (!response.ok) {\n throw new Error(`${response.status} ${await response.text()}`)\n }\n\n return await response.json()\n}\n",
"schema": {
"$schema": "https://json-schema.org/draft/2020-12/schema",
"properties": {
"auth": {
"default": null,
"description": "",
"format": "resource-servicenow",
"type": "object"
},
"body": {
"default": null,
"description": "",
"properties": {},
"type": "object"
},
"replace": {
"default": null,
"description": "",
"type": "boolean"
},
"sys_id": {
"default": null,
"description": "",
"originalType": "string",
"type": "string"
},
"table": {
"default": null,
"description": "",
"format": "dynselect-table",
"type": "object"
}
},
"required": [
"auth",
"table",
"sys_id",
"body"
],
"type": "object"
}
},
{
"version_id": 28679,
"app": "servicenow",
"summary": "New or Updated Record",
"description": "Emits records whose timestamp is newer than the last poll, tracked via Windmill state. Defaults to sys_updated_on.",
"terms": "servicenow new or updated record emits records whose timestamp is newer than the last poll, tracked via windmill state. defaults to sys_updated_on.",
"kind": "trigger",
"language": "bunnative",
"content": "//native\n\nimport * as wmill from \"windmill-client\"\n\nexport type DynSelect_table = string\n\nfunction authHeader(auth: RT.Servicenow) {\n return auth.token\n ? `Bearer ${auth.token}`\n : `Basic ${btoa(`${auth.username}:${auth.password}`)}`\n}\n\n// Dropdown of the instance's tables (sys_db_object).\nexport async function table(auth: RT.Servicenow) {\n const response = await fetch(\n `${auth.instance_url}/api/now/table/sys_db_object?sysparm_fields=name,label&sysparm_limit=10000`,\n {\n headers: {\n Authorization: authHeader(auth),\n Accept: \"application/json\",\n },\n }\n )\n if (!response.ok) {\n throw new Error(`${response.status} ${await response.text()}`)\n }\n const { result } = (await response.json()) as {\n result: { name: string; label: string }[]\n }\n return result\n .filter((t) => t.name)\n .map((t) => ({ value: t.name, label: `${t.label} (${t.name})` }))\n .sort((a, b) => a.label.localeCompare(b.label))\n}\n\n/**\n * New or Updated Record\n * Emits records from a table whose timestamp is newer than the last poll, tracked via Windmill state. Defaults to sys_updated_on (set timestamp_field to sys_created_on for new-only). Optionally narrow with an extra encoded query.\n */\nexport async function main(\n auth: RT.Servicenow,\n table: DynSelect_table,\n timestamp_field: \"sys_updated_on\" | \"sys_created_on\" | undefined,\n query: string | undefined\n) {\n const tsField =\n timestamp_field && timestamp_field !== \"\"\n ? timestamp_field\n : \"sys_updated_on\"\n const lastChecked: string | undefined = await wmill.getState()\n\n // ServiceNow datetimes are 'YYYY-MM-DD HH:MM:SS' in UTC (no trailing Z).\n const nowUtc = new Date().toISOString().slice(0, 19).replace(\"T\", \" \")\n\n // First run: set the watermark to now and don't emit a backlog.\n if (!lastChecked) {\n await wmill.setState(nowUtc)\n return []\n }\n\n let encoded = `${tsField}>${lastChecked}`\n if (query !== undefined && query !== \"\") {\n encoded += `^${query}`\n }\n encoded += `^ORDERBY${tsField}`\n\n const url = new URL(`${auth.instance_url}/api/now/table/${table}`)\n url.searchParams.append(\"sysparm_query\", encoded)\n url.searchParams.append(\"sysparm_limit\", \"200\")\n\n const response = await fetch(url, {\n method: \"GET\",\n headers: {\n Authorization: authHeader(auth),\n Accept: \"application/json\",\n },\n })\n\n if (!response.ok) {\n throw new Error(`${response.status} ${await response.text()}`)\n }\n\n const { result } = (await response.json()) as {\n result: Array<{ [key: string]: any }>\n }\n\n if (result.length > 0) {\n await wmill.setState(result[result.length - 1][tsField])\n }\n\n return result\n}\n",
"schema": {
"$schema": "https://json-schema.org/draft/2020-12/schema",
"properties": {
"auth": {
"default": null,
"description": "",
"format": "resource-servicenow",
"type": "object"
},
"query": {
"default": null,
"description": "",
"originalType": "string",
"type": "string"
},
"table": {
"default": null,
"description": "",
"format": "dynselect-table",
"type": "object"
},
"timestamp_field": {
"default": null,
"description": "",
"enum": [
"sys_updated_on",
"sys_created_on"
],
"originalType": "enum",
"type": "string"
}
},
"required": [
"auth",
"table"
],
"type": "object"
}
}
]
}
backend/windmill-api-auth/src/scopes.rs
+2 -2
View File
@@ -1111,9 +1111,9 @@ fn is_global_read_open_to_job_token(route_path: &str) -> bool {
| "/api/settings/global/hub_base_url"
| "/api/docs/search"
| "/api/docs/page"
| "/api/integrations/hub/list"
| "/api/embeddings/query_hub_scripts"
) || route_path.starts_with("/api/scripts/hub/")
) || route_path.starts_with("/api/integrations/hub/")
|| route_path.starts_with("/api/scripts/hub/")
|| route_path.starts_with("/api/flows/hub/")
|| route_path.starts_with("/api/apps/hub/")
}
backend/windmill-api-embeddings/src/lib.rs
+38
View File
@@ -78,6 +78,8 @@ pub struct HubScriptResult {
id: i64,
version_id: i64,
summary: String,
#[serde(skip_serializing_if = "Option::is_none")]
description: Option<String>,
app: String,
kind: String,
score: f32,
@@ -163,6 +165,9 @@ struct HubScript {
id: i64,
version_id: i64,
summary: String,
// Nearly a fifth of hub scripts carry an explicit `"description": null`; a bare
// String here fails the whole blob and takes hub search down with it.
description: Option<String>,
app: String,
kind: String,
embedding: Vec<f32>,
@@ -360,6 +365,10 @@ impl EmbeddingsDb {
let mut hm = HashMap::new();
hm.insert("ask_id".to_string(), script.ask_id.clone().to_string());
hm.insert("summary".to_string(), script.summary.clone());
hm.insert(
"description".to_string(),
script.description.clone().unwrap_or_default(),
);
hm.insert("app".to_string(), script.app.clone());
hm.insert("kind".to_string(), script.kind.clone());
hm.insert("id".to_string(), script.id.clone().to_string());
@@ -517,6 +526,10 @@ impl EmbeddingsDb {
.get("summary")
.ok_or(Error::msg("no summary"))?
.to_owned(),
description: metadata
.get("description")
.filter(|d| !d.is_empty())
.map(|d| d.to_owned()),
app: metadata.get("app").ok_or(Error::msg("no app"))?.to_owned(),
kind: metadata
.get("kind")
@@ -707,6 +720,31 @@ pub fn global_service() -> Router {
mod tests {
use super::trim_to_top_score;
// The blob carries an explicit `"description": null` for roughly a fifth of hub
// scripts, and an older hub omits the key entirely. A bare String here fails the
// whole 155 MB array and takes hub search down with it.
#[test]
fn reads_a_hub_script_whether_or_not_it_has_a_description() {
let present = r#"{"ask_id":1,"id":2,"version_id":3,"summary":"s","description":"d","app":"a","kind":"script","embedding":[]}"#;
let null = r#"{"ask_id":1,"id":2,"version_id":3,"summary":"s","description":null,"app":"a","kind":"script","embedding":[]}"#;
let missing = r#"{"ask_id":1,"id":2,"version_id":3,"summary":"s","app":"a","kind":"script","embedding":[]}"#;
assert_eq!(
serde_json::from_str::<super::HubScript>(present)
.unwrap()
.description,
Some("d".to_string())
);
for without in [null, missing] {
assert_eq!(
serde_json::from_str::<super::HubScript>(without)
.unwrap()
.description,
None
);
}
}
#[test]
fn trims_scores_more_than_5pct_below_top() {
// top=1.0, cutoff at 0.95: 0.96 stays (0.04 drop), 0.93 is the first
backend/windmill-api/openapi.yaml
+168
View File
@@ -9909,6 +9909,13 @@ paths:
properties:
name:
type: string
documented:
description: >-
whether the integration carries provider knowledge checked
against the live API, on top of the resource type and example
scripts /integrations/hub/{app}/meta returns for any
integration. Absent on a hub predating the flag.
type: boolean
picks:
description: how often the integration has been picked, absent on a hub that does not count picks
type: integer
@@ -9919,6 +9926,162 @@ paths:
required:
- name
/integrations/hub/{app}/meta:
get:
summary: get hub integration metadata
operationId: getHubIntegrationMeta
tags:
- integration
parameters:
- name: app
description: integration slug
in: path
required: true
schema:
type: string
responses:
"200":
description: integration metadata
content:
application/json:
schema:
type: object
properties:
app:
type: string
display_name:
type: string
description:
type: string
nullable: true
docs_url:
type: string
nullable: true
curated:
description: >-
whether the script set was pruned to idiomatic actions rather
than generated one per endpoint; null when nobody has assessed
it, which is distinct from a stated false
type: boolean
nullable: true
metadata_source:
description: >-
whether the provider knowledge was authored (curated) or
inferred from the shipped scripts (derived)
type: string
enum:
- curated
- derived
meta:
description: >-
the integration's authored meta.json verbatim; the content
repo owns its schema, so it is passed through unvalidated
type: object
nullable: true
additionalProperties: true
meta_updated_at:
type: string
nullable: true
derived:
description: facts computed from the integration's shipped scripts
type: object
properties:
api_hosts:
type: array
items:
type: object
properties:
host:
type: string
count:
type: number
required:
- host
- count
style:
type: string
enum:
- fetch
- sdk
- mixed
- unknown
languages:
type: object
additionalProperties:
type: number
script_counts:
type: object
properties:
total:
type: number
by_kind:
type: object
additionalProperties:
type: number
required:
- total
- by_kind
top_scripts:
type: array
items:
type: object
properties:
path:
type: string
ask_id:
type: number
version_id:
type: number
summary:
type: string
description:
type: string
nullable: true
kind:
type: string
language:
type: string
nullable: true
views:
type: number
votes:
type: number
required:
- path
- summary
- kind
required:
- api_hosts
- style
- languages
- script_counts
- top_scripts
resource_types:
type: array
items:
type: object
properties:
id:
type: number
name:
type: string
description:
type: string
nullable: true
schema:
description: >-
the resource type's JSON schema; the hub stores it as text
and returns it as a JSON string, so a client must be ready
for either that or an object
required:
- name
- schema
# `derived` and `resource_types` are omitted by a hub older than the
# client, which reads every section as optional.
required:
- app
- display_name
/flows/hub/list:
get:
summary: list all hub flows
@@ -10247,6 +10410,9 @@ paths:
type: number
summary:
type: string
description:
type: string
nullable: true
app:
type: string
version_id:
@@ -10316,6 +10482,8 @@ paths:
type: number
summary:
type: string
description:
type: string
app:
type: string
kind:
backend/windmill-api/src/integration.rs
+66 -2
View File
@@ -1,9 +1,16 @@
use crate::{db::DB, HTTP_CLIENT};
use axum::{extract::Query, response::IntoResponse, routing::get, Extension, Router};
use axum::{
extract::{Path, Query},
response::IntoResponse,
routing::get,
Extension, Router,
};
use windmill_common::{error::Error, utils::query_elems_from_hub, HUB_BASE_URL};
pub fn global_service() -> Router {
Router::new().route("/hub/list", get(list_hub_integrations))
Router::new()
.route("/hub/list", get(list_hub_integrations))
.route("/hub/{app}/meta", get(get_hub_integration_meta))
}
#[derive(serde::Deserialize)]
@@ -29,3 +36,60 @@ async fn list_hub_integrations(
.await?;
Ok::<_, Error>((status_code, headers, response))
}
/// Axum percent-decodes a path parameter, so an interpolated slug carrying `..`, `?`
/// or `#` re-targets the proxied GET at another path on the hub origin — with the
/// instance's hub credentials attached. Slugs are `[A-Za-z0-9_-]`; reject the rest.
fn is_hub_integration_slug(app: &str) -> bool {
!app.is_empty()
&& app.len() <= 64
&& app
.chars()
.all(|c| c.is_ascii_alphanumeric() || c == '_' || c == '-')
}
/// Everything a caller needs to write code against one integration: its resource
/// types, the provider knowledge the content repo authored, and facts derived from
/// the shipped scripts. A hub older than the endpoint answers 404, which passes
/// through as-is.
async fn get_hub_integration_meta(
Path(app): Path<String>,
Extension(db): Extension<DB>,
) -> impl IntoResponse {
if !is_hub_integration_slug(&app) {
return Err(Error::BadRequest(format!(
"Not a valid integration name: {app}"
)));
}
let (status_code, headers, response) = query_elems_from_hub(
&HTTP_CLIENT,
&format!("{}/integrations/{}/meta", **HUB_BASE_URL.load(), app),
None,
&db,
)
.await?;
Ok::<_, Error>((status_code, headers, response))
}
#[cfg(test)]
mod tests {
use super::is_hub_integration_slug;
#[test]
fn rejects_slugs_that_would_re_target_the_proxied_request() {
assert!(is_hub_integration_slug("confluence"));
assert!(is_hub_integration_slug("aws-ses"));
assert!(is_hub_integration_slug("bamboo_hr"));
assert!(is_hub_integration_slug("RSS"));
for escape in [
"../../scripts/top",
"confluence?foo=bar",
"confluence#frag",
"confluence/meta",
"",
] {
assert!(!is_hub_integration_slug(escape), "accepted {escape}");
}
}
}
frontend/src/lib/components/copilot/chat/app/core.ts
+1 -1
View File
@@ -974,7 +974,7 @@ Use \`patch_file\` for small, localized edits when you can copy an exact snippet
### Discovery
- \`search_workspace(query, type)\`: Search workspace scripts and flows
- \`get_runnable_details(path, type)\`: Get details (summary, description, schema, content) of a specific script or flow
- \`search_hub_scripts(query)\`: Search hub scripts
- \`search_hub_scripts(query, integration)\`: Search hub scripts, or list one integration's scripts by slug
### Data Tables
- \`list_datatables()\`: List configured datatables with schema and table names only. Does not include columns. Use this directly for table-list or available-tables summaries.
frontend/src/lib/components/copilot/chat/global/core.test.ts
+5 -3
View File
@@ -1137,13 +1137,15 @@ describe('global AI tools', () => {
expect(ScriptService.queryHubScripts).toHaveBeenCalledWith({
text: 'slack message',
kind: 'script'
kind: 'script',
app: undefined
})
expect(ScriptService.getHubScriptContentByPath).not.toHaveBeenCalled()
expect(JSON.parse(raw)).toEqual([
expect(JSON.parse(raw).results).toEqual([
{
path: 'hub/7/slack/send_message',
summary: 'Send Message'
summary: 'Send Message',
integration: 'slack'
}
])
})
frontend/src/lib/components/copilot/chat/global/core.ts
+5 -1
View File
@@ -133,6 +133,7 @@ import {
createToolDef,
droppedOptionKeys,
createSearchHubScriptsTool,
getHubIntegrationTool,
executeTestRun,
findAndReplace,
isHubPath,
@@ -1437,7 +1438,9 @@ Rules:${when(
- You can never read a variable's value, secret or not, so never invent one: when editing an existing variable, omit value (and is_secret) from write_variable and pass only the fields you are actually changing. The user can reveal a value in the variable editor; you cannot, so never tell them a value is unreadable in general. "$var:path/to/variable" is how a resource value references a variable — it is never a variable's own value.
- Use search_resource_types before write_resource, and get_trigger_schema before write_trigger: the trigger config fields differ per kind and are not listed in the write_trigger definition.
- When script or raw app code needs an external npm package you are not fully familiar with, use search_npm_packages to find it and get its documentation and type definitions. Link the package documentation in your answer when you rely on it.
- Hub scripts are prebuilt integrations for third-party services, hosted outside the workspace under \`hub/<version>/<app>/<name>\` paths. Use search_hub_scripts to find one before hand-writing an integration, then read_workspace_item with type "script" and the returned hub path to get its code, language, and input schema.
- Hub scripts are prebuilt, vetted integrations for third-party services, hosted outside the workspace under \`hub/<version>/<app>/<name>\` paths. Check search_hub_scripts before hand-writing code against a third-party API, even when the user never mentions the hub; read a result with read_workspace_item type "script" and its hub path to get its code, language, and input schema. Use what you find in whichever way fits: reference the hub path directly from a flow module or app runnable when a script already does the job, copy it into a workspace draft and adapt it when it is close (note the source hub path in a comment at the top of the code), or take it as a worked example and write your own. A script that does not do what the user asked is still worth reading when it is the only example of that integration: pass its \`integration\` back to search_hub_scripts to list that integration's other scripts with their descriptions, or use the \`suggested_integrations\` a search hands back when it finds nothing.
- Before writing your own code against an integration the hub covers, call get_hub_integration with its slug: it returns the resource type to take, its auth fields and the integration's most-used scripts, which beats inferring them from script bodies. Call it for the integration you are about to write against, whichever it is. A search marks an integration \`documented\` when the hub additionally holds provider knowledge checked against the live API — pagination, enums, error codes and gotchas — so read that closely where it appears rather than trusting your own memory of the API.
- If you have a web search tool and the hub does not cover a third-party API, search for the vendor's own API documentation rather than writing its endpoints and auth from memory, and link the page you relied on. Reserve it for external APIs: search_docs answers questions about Windmill itself.
${when(canRunPreview, '- Use get_db_schema with a database resource path to fetch its tables and columns before writing SQL (or a script querying that database).\n')}- Use get_instructions before writing scripts, flows, resources, or apps. For scripts, pass the target language.
${pipelineBullet}`
)}${when(
@@ -3388,6 +3391,7 @@ export const globalTools: SessionTool<{}>[] = [
}
},
createSearchHubScriptsTool(false),
getHubIntegrationTool,
searchNpmPackagesTool,
searchDocsTool,
readDocsPageTool,
frontend/src/lib/components/copilot/chat/shared.test.ts
+380 -1
View File
@@ -38,6 +38,7 @@ vi.mock('$lib/components/flows/flowTree', () => ({
vi.mock('$lib/gen', () => ({
ScriptService: {},
FlowService: {},
IntegrationService: {},
JobService: { getJob: vi.fn(), getJobUpdates: vi.fn() },
ScheduleService: {
previewSchedule: vi.fn(),
@@ -1890,7 +1891,7 @@ describe('createSearchHubScriptsTool', () => {
toolId: 't1',
toolCallbacks: { setToolStatus: vi.fn() }
} as any)
return JSON.parse(raw)
return JSON.parse(raw).results
}
it('reports each script language alongside its content', async () => {
@@ -1905,12 +1906,14 @@ describe('createSearchHubScriptsTool', () => {
{
path: 'hub/1/discord/send_a_message',
summary: 'Send a message',
integration: 'discord',
language: 'bunnative',
content: '// hub/1/discord/send_a_message'
},
{
path: 'hub/2/slack/post_a_message',
summary: 'Post a message',
integration: 'slack',
language: 'python3',
content: '// hub/2/slack/post_a_message'
}
@@ -1929,6 +1932,382 @@ describe('createSearchHubScriptsTool', () => {
expect(results[0].content).toBeUndefined()
expect(results[1].content).toBe('ok')
})
// Browsing by app is what surfaces an integration's other scripts as examples,
// and only the top-scripts endpoint takes no query and applies no similarity
// floor, so the near-misses worth reading survive instead of being cut.
it('lists an integration through the top-scripts endpoint, with descriptions', async () => {
const { ScriptService } = await import('$lib/gen')
const getTopHubScripts = vi.fn(async () => ({
asks: [{ ...hit(3, 'stripe', 'Create a Payout'), description: 'Send funds to your bank' }]
}))
Object.assign(ScriptService, { queryHubScripts: vi.fn(async () => []), getTopHubScripts })
const { createSearchHubScriptsTool } = await import('./shared')
const raw = await createSearchHubScriptsTool().fn({
args: { integration: 'stripe' },
toolId: 't1',
toolCallbacks: { setToolStatus: vi.fn() }
} as any)
expect(getTopHubScripts).toHaveBeenCalledWith({ app: 'stripe', kind: 'script', limit: 20 })
expect(ScriptService.queryHubScripts).not.toHaveBeenCalled()
expect(JSON.parse(raw).results).toEqual([
{
path: 'hub/3/stripe/create_a_payout',
summary: 'Create a Payout',
integration: 'stripe',
description: 'Send funds to your bank'
}
])
})
// Narrowing a query to a slug is the only thing that searches an integration the
// user named: ranking alone surfaces other integrations whose scripts merely mention
// it, so a query that drops `app` silently searches the whole hub instead.
it('narrows a query to the integration it was given', async () => {
const { ScriptService } = await import('$lib/gen')
const queryHubScripts = vi.fn(async () => [hit(9, 'salesforce', 'SOSL Search')])
Object.assign(ScriptService, { queryHubScripts, getTopHubScripts: vi.fn() })
const { createSearchHubScriptsTool } = await import('./shared')
const raw = await createSearchHubScriptsTool().fn({
args: { query: 'look up an account', integration: 'salesforce' },
toolId: 't1',
toolCallbacks: { setToolStatus: vi.fn() }
} as any)
expect(queryHubScripts).toHaveBeenCalledWith({
text: 'look up an account',
kind: 'script',
app: 'salesforce'
})
expect(ScriptService.getTopHubScripts).not.toHaveBeenCalled()
expect(JSON.parse(raw).results.map((r: any) => r.integration)).toEqual(['salesforce'])
})
// A search below the similarity floor otherwise dead-ends; the slug is what
// lets the model fall back to browsing the integration.
it('suggests matching integrations when the search finds nothing', async () => {
const { ScriptService, IntegrationService } = await import('$lib/gen')
Object.assign(ScriptService, { queryHubScripts: vi.fn(async () => []) })
Object.assign(IntegrationService, {
listHubIntegrations: vi.fn(async () => [{ name: 'stripe' }, { name: 'slack' }])
})
const { createSearchHubScriptsTool, clearHubIntegrationsCache } = await import('./shared')
clearHubIntegrationsCache()
const raw = await createSearchHubScriptsTool().fn({
args: { query: 'refund a stripe charge' },
toolId: 't1',
toolCallbacks: { setToolStatus: vi.fn() }
} as any)
expect(JSON.parse(raw)).toEqual({ results: [], suggested_integrations: ['stripe'] })
})
// Matching a query word anywhere inside a slug turns every short English word into
// a hit — `for` in salesforce, `the` in basis_theory — so a task that names no
// integration came back with a confident-looking list of them.
it('suggests nothing for a request that names no integration', async () => {
const { ScriptService, IntegrationService } = await import('$lib/gen')
Object.assign(ScriptService, { queryHubScripts: vi.fn(async () => []) })
Object.assign(IntegrationService, {
listHubIntegrations: vi.fn(async () => [
{ name: 'salesforce' },
{ name: 'basis_theory' },
{ name: 'hackernews' },
{ name: 's3' },
// A slug that is an ordinary word with a letter in front of it: matching
// one character in would read "node" as this integration.
{ name: 'enode' }
])
})
const { createSearchHubScriptsTool, clearHubIntegrationsCache } = await import('./shared')
clearHubIntegrationsCache()
const raw = await createSearchHubScriptsTool().fn({
args: { query: 'list all the invoices for the new month in a node script' },
toolId: 't1',
toolCallbacks: { setToolStatus: vi.fn() }
} as any)
expect(JSON.parse(raw).suggested_integrations).toEqual([])
})
// Google's integrations compress the vendor to a `g`, so the word a user says
// appears in the name the hub curates and nowhere in the slug.
it('reaches an integration whose slug compresses the vendor name', async () => {
const { ScriptService, IntegrationService } = await import('$lib/gen')
Object.assign(ScriptService, { queryHubScripts: vi.fn(async () => []) })
Object.assign(IntegrationService, {
listHubIntegrations: vi.fn(async () => [
{ name: 'gsheets', display_name: 'Google Sheets' },
{ name: 'gdrive', display_name: 'Google Drive' },
{ name: 'smartsheet', display_name: null }
])
})
const { createSearchHubScriptsTool, clearHubIntegrationsCache } = await import('./shared')
clearHubIntegrationsCache()
const raw = await createSearchHubScriptsTool().fn({
args: { query: 'add a row to a google sheet' },
toolId: 't1',
toolCallbacks: { setToolStatus: vi.fn() }
} as any)
// "google" names both products, and a suggestion list is allowed to offer both.
expect(JSON.parse(raw).suggested_integrations).toEqual(['gsheets', 'gdrive'])
})
// The hub names only the slugs its own titleisation gets wrong, so the ones it
// leaves null are still named here.
it('falls back to the local name table for a slug the hub does not name', async () => {
const { ScriptService, IntegrationService } = await import('$lib/gen')
Object.assign(ScriptService, { queryHubScripts: vi.fn(async () => []) })
Object.assign(IntegrationService, {
listHubIntegrations: vi.fn(async () => [{ name: 'gcloud', display_name: null }])
})
const { createSearchHubScriptsTool, clearHubIntegrationsCache } = await import('./shared')
clearHubIntegrationsCache()
const raw = await createSearchHubScriptsTool().fn({
args: { query: 'deploy a service to google cloud' },
toolId: 't1',
toolCallbacks: { setToolStatus: vi.fn() }
} as any)
expect(JSON.parse(raw).suggested_integrations).toEqual(['gcloud'])
})
// A slug shorter than the token floor is only reachable by an exact match.
it('still reaches a two-character integration slug', async () => {
const { ScriptService, IntegrationService } = await import('$lib/gen')
Object.assign(ScriptService, { queryHubScripts: vi.fn(async () => []) })
Object.assign(IntegrationService, {
listHubIntegrations: vi.fn(async () => [{ name: 's3' }, { name: 'salesforce' }])
})
const { createSearchHubScriptsTool, clearHubIntegrationsCache } = await import('./shared')
clearHubIntegrationsCache()
const raw = await createSearchHubScriptsTool().fn({
args: { query: 'upload a file to s3' },
toolId: 't1',
toolCallbacks: { setToolStatus: vi.fn() }
} as any)
expect(JSON.parse(raw).suggested_integrations).toEqual(['s3'])
})
// Only about one hub integration in twelve carries authored provider knowledge,
// and a search result is where the model first sees a slug — so the mark has to
// ride along, or nothing tells it which integrations have verified notes to read.
it('marks the integrations get_hub_integration has knowledge for', async () => {
const { ScriptService, IntegrationService } = await import('$lib/gen')
Object.assign(ScriptService, {
queryHubScripts: vi.fn(async () => [
hit(1, 'confluence', 'Create page'),
hit(2, 'notion', 'Create page')
])
})
Object.assign(IntegrationService, {
listHubIntegrations: vi.fn(async () => [
{ name: 'Confluence', documented: true },
{ name: 'notion', documented: false }
])
})
const { createSearchHubScriptsTool, clearHubIntegrationsCache } = await import('./shared')
clearHubIntegrationsCache()
const raw = await createSearchHubScriptsTool().fn({
args: { query: 'create a wiki page' },
toolId: 't1',
toolCallbacks: { setToolStatus: vi.fn() }
} as any)
const results = JSON.parse(raw).results
// the hub's slugs are case-sensitive, so the two casings are one integration
expect(results.find((r: any) => r.integration === 'confluence').documented).toBe(true)
expect(results.find((r: any) => r.integration === 'notion').documented).toBeUndefined()
})
// A hub predating the flag sends no `documented` at all; reading that as "yes"
// would send the model after notes that are not coming.
it('claims no knowledge when the hub does not report any', async () => {
const { ScriptService, IntegrationService } = await import('$lib/gen')
Object.assign(ScriptService, {
queryHubScripts: vi.fn(async () => [hit(1, 'confluence', 'Create page')])
})
Object.assign(IntegrationService, {
listHubIntegrations: vi.fn(async () => [{ name: 'confluence' }])
})
const { createSearchHubScriptsTool, clearHubIntegrationsCache } = await import('./shared')
clearHubIntegrationsCache()
const raw = await createSearchHubScriptsTool().fn({
args: { query: 'create a wiki page' },
toolId: 't1',
toolCallbacks: { setToolStatus: vi.fn() }
} as any)
expect(JSON.parse(raw).results[0].documented).toBeUndefined()
})
})
describe('getHubIntegrationTool', () => {
// The plan-mode gate fails closed, so a read-only tool that is not tagged is
// silently unusable while planning.
it('is usable while planning, like the hub search beside it', async () => {
const { getHubIntegrationTool, createSearchHubScriptsTool } = await import('./shared')
expect(getHubIntegrationTool.planModeSafe).toBe(true)
expect(createSearchHubScriptsTool().planModeSafe).toBe(true)
})
const doc = {
app: 'confluence',
display_name: 'Confluence',
// Authored knowledge, but nobody has asserted whether the script set was pruned.
curated: null,
metadata_source: 'curated',
meta: { gotchas: ['Auth is Basic with an API token, not the password'] },
derived: {
api_hosts: [{ host: 'api.atlassian.com', count: 12 }],
style: 'fetch',
languages: { bun: 14 },
script_counts: { total: 14, by_kind: { script: 13 } },
top_scripts: [
{ path: 'hub/1/confluence/create_page', summary: 'Create page', language: 'bun' }
]
},
resource_types: [{ name: 'confluence', schema: { type: 'object' } }]
}
// Hand-validated provider knowledge and facts inferred from script bodies must
// stay under separate keys, or the model will report guesses as verified.
it('keeps authored notes apart from what was inferred from the scripts', async () => {
const { IntegrationService } = await import('$lib/gen')
Object.assign(IntegrationService, { getHubIntegrationMeta: vi.fn(async () => doc) })
const { getHubIntegrationTool } = await import('./shared')
const parsed = JSON.parse(
await getHubIntegrationTool.fn({
args: { integration: 'confluence' },
toolId: 't1',
toolCallbacks: { setToolStatus: vi.fn() }
} as any)
)
expect(parsed.verified_provider_notes).toEqual(doc.meta)
expect(parsed.scripts_note).toBeUndefined()
expect(parsed.observed_from_scripts).toEqual({
api_hosts: ['api.atlassian.com'],
style: 'fetch',
languages: ['bun'],
script_counts: { total: 14, by_kind: { script: 13 } }
})
})
// The hub repo owns this payload, so a hub older than this client can send a
// subset. Reading it must degrade to less content, never to a tool error.
it('returns what an older hub sent instead of failing on the missing sections', async () => {
const { IntegrationService } = await import('$lib/gen')
Object.assign(IntegrationService, {
getHubIntegrationMeta: vi.fn(async () => ({
app: 'stripe',
display_name: 'Stripe',
curated: false
}))
})
const { getHubIntegrationTool } = await import('./shared')
const parsed = JSON.parse(
await getHubIntegrationTool.fn({
args: { integration: 'stripe' },
toolId: 't1',
toolCallbacks: { setToolStatus: vi.fn() }
} as any)
)
expect(parsed.integration).toBe('stripe')
expect(parsed.observed_from_scripts).toBeUndefined()
expect(parsed.resource_types).toEqual([])
expect(parsed.example_scripts).toEqual([])
})
// `curated` has three states and only `true` is a licence to call the scripts good
// examples. `false` and the unassessed `null` must both stay silent, or the 212
// integrations nobody has looked at get characterised anyway.
it.each([
[true, true],
[false, false],
[null, false]
])('speaks about the script set only when curated is true (%s)', async (curated, expected) => {
const { IntegrationService } = await import('$lib/gen')
Object.assign(IntegrationService, {
getHubIntegrationMeta: vi.fn(async () => ({ ...doc, curated }))
})
const { getHubIntegrationTool } = await import('./shared')
const parsed = JSON.parse(
await getHubIntegrationTool.fn({
args: { integration: 'confluence' },
toolId: 't1',
toolCallbacks: { setToolStatus: vi.fn() }
} as any)
)
expect(!!parsed.scripts_note).toBe(expected)
})
// A hub that times out has said nothing about whether the integration exists, and
// reporting it as absent would stick for the rest of the conversation.
it('does not report a transient hub failure as a missing integration', async () => {
const { IntegrationService } = await import('$lib/gen')
Object.assign(IntegrationService, {
getHubIntegrationMeta: vi.fn(async () => {
throw Object.assign(new Error('Service Unavailable'), { status: 503 })
}),
listHubIntegrations: vi.fn(async () => [{ name: 'confluence' }])
})
const { getHubIntegrationTool, clearHubIntegrationsCache } = await import('./shared')
clearHubIntegrationsCache()
const parsed = JSON.parse(
await getHubIntegrationTool.fn({
args: { integration: 'confluence' },
toolId: 't1',
toolCallbacks: { setToolStatus: vi.fn() }
} as any)
)
expect(parsed.error).toContain('Could not reach the hub')
expect(parsed.error).not.toContain('No hub metadata')
})
// A hub with no such integration and one too old to serve the endpoint both 404;
// neither may surface as a tool error, since the model can still read scripts.
it('suggests real slugs instead of failing when the integration is unknown', async () => {
const { IntegrationService } = await import('$lib/gen')
Object.assign(IntegrationService, {
getHubIntegrationMeta: vi.fn(async () => {
throw Object.assign(new Error('Not Found'), { status: 404 })
}),
listHubIntegrations: vi.fn(async () => [{ name: 'stripe' }, { name: 'slack' }])
})
const { getHubIntegrationTool, clearHubIntegrationsCache } = await import('./shared')
clearHubIntegrationsCache()
const parsed = JSON.parse(
await getHubIntegrationTool.fn({
args: { integration: 'stripe_billing' },
toolId: 't1',
toolCallbacks: { setToolStatus: vi.fn() }
} as any)
)
expect(parsed.error).toContain('stripe_billing')
expect(parsed.suggested_integrations).toEqual(['stripe'])
})
})
describe('processToolCall confirmation hooks', () => {
frontend/src/lib/components/copilot/chat/shared.ts
+302 -17
View File
@@ -7,7 +7,8 @@ import type { UserDraftItemKind } from '$lib/gen'
// The gate's two refusals, from a module that holds prose and one size limit: under the
// shallow-import rule below, the rest of plan mode is not reachable from here.
import { PLAN_MODE_MESSAGES } from './planModeMessages'
import { NONE } from './sessionCapabilities'
import { NONE, type SessionTool } from './sessionCapabilities'
import { integrationDisplayName } from '$lib/components/resourceTypeDisplay'
// Import-free leaf, so it satisfies the shallow-import rule below.
import {
openItemPreviewAction,
@@ -45,6 +46,7 @@ import { z } from 'zod'
import {
ScriptService,
FlowService,
IntegrationService,
JobService,
type Job,
type CompletedJob,
@@ -1515,13 +1517,22 @@ function hasOptionalProperties(schema: Record<string, any> | undefined): boolean
const searchHubScriptsSchema = z.object({
query: z
.string()
.describe('The query to search for, e.g. send email, list stripe invoices, etc..')
.optional()
.describe(
'What you want the script to do, e.g. "send email", "list stripe invoices". Semantic search, so describe the intent. Omit when browsing with `integration` alone.'
),
integration: z
.string()
.optional()
.describe(
"Integration slug (e.g. \"stripe\") from a previous result's `integration` or `suggested_integrations`, or the vendor the user named. Alone, lists that integration's scripts with descriptions, including ones that do not match your task but show how the integration is used. With `query`, narrows the search to it; pass it whenever the user names the vendor, since ranking alone can surface another integration's scripts that merely mention it."
)
})
const searchHubScriptsToolDef = createToolDef(
searchHubScriptsSchema,
'search_hub_scripts',
'Search for scripts in the hub'
"Search the Windmill Hub for prebuilt integration scripts, or list one integration's scripts with `integration`."
)
/** The hub resolves a script by its version id alone; the app and summary
@@ -1536,29 +1547,304 @@ export function isHubPath(path: string): boolean {
return path.startsWith('hub/')
}
const MAX_BROWSED_HUB_SCRIPTS = 20
const MAX_FETCHED_HUB_SCRIPTS = 3
const MAX_SUGGESTED_INTEGRATIONS = 5
/** Common shape of the two hub listings. Both carry a description, but the hub has
* none for roughly a fifth of its scripts, and says so as a null rather than by
* omitting the key. */
type HubScriptHit = {
version_id: number
app: string
summary: string
description?: string | null
}
type HubIntegration = { name: string; displayName?: string | null; documented: boolean }
/** The integration slugs are a large but static list, so one fetch per session
* is enough. Only matched slugs ever reach the model, never the whole list. */
let hubIntegrationsCache: Promise<HubIntegration[]> | undefined
async function fetchHubIntegrations(): Promise<HubIntegration[]> {
try {
const integrations = await IntegrationService.listHubIntegrations({ kind: 'script' })
return integrations.map((i) => ({
name: i.name,
// The hub names only the slugs its own titleisation gets wrong, so a slug it
// leaves null can still have a name here — `gcloud` is "Google Cloud" in the
// word table this resolves through.
displayName: i.display_name ?? integrationDisplayName(i.name),
// A hub predating the flag omits it, and so does one with no authored
// notes. Both mean the same thing: nothing to read beyond what the
// metadata call returns for every integration.
documented: i.documented === true
}))
} catch (err) {
console.error('Could not list hub integrations', err)
return []
}
}
/** Concurrent callers share one in-flight fetch. An unreachable hub must degrade to
* "no suggestions" rather than turn the search into a tool error, so a failed or
* empty response is left uncached and retried. */
function loadHubIntegrations(): Promise<HubIntegration[]> {
return (hubIntegrationsCache ??= fetchHubIntegrations().then((integrations) => {
if (!integrations.length) hubIntegrationsCache = undefined
return integrations
}))
}
/** Which integrations carry provider knowledge checked against the live API,
* lowercased because a hub slug is case-sensitive and both casings can exist. The
* mark means there is more to read, never that the call is skippable elsewhere. */
async function documentedIntegrations(): Promise<Set<string>> {
const available = await loadHubIntegrations()
return new Set(available.filter((i) => i.documented).map((i) => i.name.toLowerCase()))
}
async function suggestHubIntegrations(query: string): Promise<string[]> {
const available = await loadHubIntegrations()
const tokens = query
.toLowerCase()
.split(/[^a-z0-9]+/)
.filter((t) => t.length >= 2)
// Ranked by how much of the query each name accounts for, because a vendor with
// several products shares one word across all of them: "google drive" matches
// every `g*` integration on "google", and only Google Drive on both words. The
// cap would otherwise cut the one the user named.
return available
.map(({ name, displayName }) => ({
name,
matched: tokens.filter((t) => tokenMatchesIntegration(t, name, displayName)).length
}))
.filter(({ matched }) => matched > 0)
.sort((a, b) => b.matched - a.matched)
.map(({ name }) => name)
.slice(0, MAX_SUGGESTED_INTEGRATIONS)
}
/** Matches a query word against an integration's slug and its name, which is what
* reaches a slug compressing the vendor to a letter: "google drive" matches `gdrive`
* only through its "Google Drive". A bare substring test
* instead makes every three-letter English word a hit — `for` in sales*for*ce, `the`
* in basis_*the*ory — so a token matches a whole word, or extends one from four
* characters on. Short tokens must equal a word, which is also what reaches the
* two-character slugs (`s3`, `wiz`) a length floor would hide. */
function tokenMatchesIntegration(
token: string,
slug: string,
displayName?: string | null
): boolean {
const words = [slug, displayName ?? '']
.join(' ')
.toLowerCase()
.split(/[^a-z0-9]+/)
.filter(Boolean)
if (words.includes(token)) {
return true
}
if (token.length < 4) {
return false
}
const extends_ = (a: string, b: string) => a.startsWith(b) || b.startsWith(a)
return words.some((w) => w.length >= 4 && extends_(w, token))
}
export const clearHubIntegrationsCache = () => {
hubIntegrationsCache = undefined
}
const getHubIntegrationSchema = z.object({
integration: z
.string()
.describe(
'Integration slug, e.g. "stripe". Take it from a search_hub_scripts result\'s `integration`, or guess the vendor name: a wrong guess comes back with the closest real slugs.'
)
})
const getHubIntegrationToolDef = createToolDef(
getHubIntegrationSchema,
'get_hub_integration',
'Read how one integration works before writing code against it: the resource type it takes, its auth fields, and its most-used scripts as examples. For an integration a search marked `documented` it also returns provider knowledge checked against the live API: pagination, enums, error codes and gotchas.'
)
/** Enough to show the integration's idiom; search_hub_scripts is the way to find
* a specific one. */
const MAX_INTEGRATION_EXAMPLES = 5
/** How the authored notes were checked — method, per-surface confidence, the instance
* used — is provenance for a human reviewing the hub, and up to a third of the
* document. The verdict and any fields it flags as shaky are the parts a caller can
* act on, so those stay. */
function withoutProvenance(meta: unknown): unknown {
if (typeof meta !== 'object' || meta === null || Array.isArray(meta)) {
return meta
}
const { validation, ...rest } = meta as Record<string, unknown>
const kept = Object.fromEntries(
Object.entries((validation as Record<string, unknown> | undefined) ?? {}).filter(
([key]) => key === 'status' || key === 'low_confidence_fields'
)
)
return Object.keys(kept).length ? { ...rest, validation: kept } : rest
}
/** The hub keeps a resource type's schema in a text column and hands it back as a
* JSON string, so parse it rather than passing an escaped blob to the model.
* Anything already structured goes through untouched. */
function parseResourceTypeSchema(schema: unknown): unknown {
if (typeof schema !== 'string') {
return schema
}
try {
return JSON.parse(schema)
} catch {
return schema
}
}
export const getHubIntegrationTool = {
requires: NONE,
def: getHubIntegrationToolDef,
// Reads one hub document over a GET and writes nothing, so planning may use it.
planModeSafe: true,
fn: async ({ args, toolId, toolCallbacks }) => {
const { integration } = getHubIntegrationSchema.parse(args)
toolCallbacks.setToolStatus(toolId, { content: `Reading the ${integration} integration...` })
let doc: Awaited<ReturnType<typeof IntegrationService.getHubIntegrationMeta>>
try {
doc = await IntegrationService.getHubIntegrationMeta({ app: integration })
} catch (err) {
// Only a 404 means the integration is absent — an unknown slug, or a hub
// predating the endpoint. Reporting a timeout the same way would teach the
// model that a real integration does not exist for the rest of the chat.
const absent = (err as { status?: number } | undefined)?.status === 404
const label = absent
? `No hub integration named ${integration}`
: `Could not reach the hub for ${integration}`
toolCallbacks.setToolStatus(toolId, { content: label })
const suggested = await suggestHubIntegrations(integration)
return JSON.stringify({
error: absent
? `No hub metadata for "${integration}".`
: `Could not reach the hub for "${integration}"; it may still exist. Read its scripts instead, or try again.`,
suggested_integrations: suggested
})
}
toolCallbacks.setToolStatus(toolId, { content: `Read the ${doc.display_name} integration` })
// The hub owns this response's shape and can be older than this client, so
// every section is read as optional: a hub that sends less should return less,
// not fail the call.
const derived = doc.derived
return JSON.stringify({
integration: doc.app,
display_name: doc.display_name,
...(doc.description ? { description: doc.description } : {}),
...(doc.docs_url ? { docs_url: doc.docs_url } : {}),
// Authored provider knowledge and facts inferred from the scripts stay
// separate: only the former was checked against the live API.
...(doc.meta ? { verified_provider_notes: withoutProvenance(doc.meta) } : {}),
...(derived
? {
observed_from_scripts: {
api_hosts: derived.api_hosts?.map((h) => h.host) ?? [],
style: derived.style,
languages: Object.keys(derived.languages ?? {}),
script_counts: derived.script_counts
}
}
: {}),
// `curated` is three-state: only an integration's own meta.json asserts it, and
// anything else means nobody has assessed the script set. Speak only for a
// positive assertion, so an unassessed integration is never characterised
// either way; `script_counts` below is the factual signal for the rest.
...(doc.curated === true
? {
scripts_note:
'These scripts were pruned to idiomatic actions rather than generated one per API endpoint, so they are worth following as style examples.'
}
: {}),
resource_types: (doc.resource_types ?? []).map((rt) => ({
name: rt.name,
...(rt.description ? { description: rt.description } : {}),
schema: parseResourceTypeSchema(rt.schema)
})),
example_scripts: (derived?.top_scripts ?? []).slice(0, MAX_INTEGRATION_EXAMPLES).map((s) => ({
path: s.path,
summary: s.summary,
...(s.description ? { description: s.description } : {}),
...(s.language ? { language: s.language } : {})
}))
})
}
} satisfies SessionTool<{}>
export const createSearchHubScriptsTool = (withContent: boolean = false) => ({
requires: NONE,
def: searchHubScriptsToolDef,
planModeSafe: true,
fn: async ({ args, toolId, toolCallbacks }) => {
toolCallbacks.setToolStatus(toolId, {
content: 'Searching for hub scripts related to "' + args.query + '"...'
})
const parsedArgs = searchHubScriptsSchema.parse(args)
const scripts = await ScriptService.queryHubScripts({
text: parsedArgs.query,
kind: 'script'
})
// The hub's own query param is `app`; the tool exposes it as `integration`,
// since `app` already means a Windmill app everywhere else in this surface.
const { query, integration: app } = parsedArgs
if (!query && !app) {
return 'Pass query (what the script should do), integration (a slug to list), or both.'
}
const subject = query ? `"${query}"` : `the ${app} integration`
toolCallbacks.setToolStatus(toolId, { content: `Searching hub scripts for ${subject}...` })
// Started alongside the search, not after it: this is where the model learns a
// slug exists, so it is where it has to learn how much get_hub_integration has
// to add for that slug. One cached fetch per session either way.
const documented = documentedIntegrations()
// Listing an integration goes through the hub's top-scripts endpoint rather
// than the semantic one: it takes no query, and it applies no similarity
// floor, so the near-misses worth reading as examples of how the integration
// is used survive instead of being cut.
const scripts: HubScriptHit[] = query
? await ScriptService.queryHubScripts({ text: query, kind: 'script', app })
: ((
await ScriptService.getTopHubScripts({
app,
kind: 'script',
limit: MAX_BROWSED_HUB_SCRIPTS
})
).asks ?? [])
if (scripts.length === 0) {
// A whiffed search still leaves the integration browsable, which is what
// turns "no exact match" into a worked example to follow. Suggest against
// the slug too, so a browse for a misremembered one lands on the real name
// instead of dead-ending on an empty list.
const suggested = await suggestHubIntegrations([app, query].filter(Boolean).join(' '))
toolCallbacks.setToolStatus(toolId, { content: `No hub script found for ${subject}` })
return JSON.stringify({ results: [], suggested_integrations: suggested })
}
// Each result costs a content fetch, so cap the fan-out when content is wanted.
const matches = withContent ? scripts.slice(0, 3) : scripts
const matches = withContent ? scripts.slice(0, MAX_FETCHED_HUB_SCRIPTS) : scripts
toolCallbacks.setToolStatus(toolId, {
content: `Found ${matches.length} script${matches.length === 1 ? '' : 's'} in the hub related to "${parsedArgs.query}"`
content: `Found ${matches.length} hub script${matches.length === 1 ? '' : 's'} for ${subject}`
})
const documentedSlugs = await documented
const results = await Promise.all(
matches.map(async (s) => {
const path = hubScriptPath(s)
const base = {
path,
summary: s.summary,
integration: s.app,
...(documentedSlugs.has(s.app.toLowerCase()) ? { documented: true } : {}),
...(s.description ? { description: s.description } : {})
}
if (!withContent) {
return { path, summary: s.summary }
return base
}
// The content fetch, not the listing above: these are the few candidates
// the AI pulled to choose between, which is the closest signal we have.
@@ -1567,18 +1853,17 @@ export const createSearchHubScriptsTool = (withContent: boolean = false) => ({
// get_full, not the raw content endpoint: callers are told to match the
// script's language, which raw content does not carry.
const hub = await ScriptService.getHubScriptByPath({ path })
return { path, summary: s.summary, language: hub.language, content: hub.content }
return { ...base, language: hub.language, content: hub.content }
} catch (err) {
// One unreachable script must not sink the whole search.
return {
path,
summary: s.summary,
...base,
error: `Could not fetch content: ${err instanceof Error ? err.message : String(err)}`
}
}
})
)
return JSON.stringify(results)
return JSON.stringify({ results })
}
})