mirror of
https://github.com/lancedb/lancedb.git
synced 2026-08-18 12:08:35 +00:00
Dan Tasse
ab3041e01e
Some additions to our lancedb skill to enable agents to use the jobs methods that we recently added. Eval tests (below, with and without these additions to the skill) suggest that they're helping, mostly to find the right method calls. These are a little unusual because they require REST server connection, they're not yet implemented in the SDKs. ``` ┌─────────────────────┬───────────┬────────────┬─────────────┬──────────┬───────────┬──────────┬───────────┐ │ eval │ grade w/o │ grade with │ improvement │ time w/o │ time with │ cost w/o │ cost with │ ├─────────────────────┼───────────┼────────────┼─────────────┼──────────┼───────────┼──────────┼───────────┤ │ 8-list-running-jobs │ 2.5/3 │ 3/3 │ +0.5 │ 123s │ 29s │ $0.58 │ $0.18 │ ├─────────────────────┼───────────┼────────────┼─────────────┼──────────┼───────────┼──────────┼───────────┤ │ 9-describe-job │ 1/5 │ 5/5 │ +4.0 │ 159s │ 52s │ $0.62 │ $0.25 │ ├─────────────────────┼───────────┼────────────┼─────────────┼──────────┼───────────┼──────────┼───────────┤ │ 10-cancel-job │ 3/3 │ 3/3 │ +0.0 │ 99s │ 35s │ $0.55 │ $0.21 │ ├─────────────────────┼───────────┼────────────┼─────────────┼──────────┼───────────┼──────────┼───────────┤ │ TOTAL │ 6.5/11 │ 11/11 │ +4.5 │ 381s │ 116s │ $1.75 │ $0.65 │ └─────────────────────┴───────────┴────────────┴─────────────┴──────────┴───────────┴──────────┴───────────┘ ``` Failure reasons are because the agent didn't know the right method to call, spent all its turns guessing REST calls, tried to inspect lancedb code, but didn't find the answer in here.