* test(sqlness): run environments concurrently with external backends
The runner forced both environment and instance parallelism to 1 whenever
etcd/PG/MySQL or an external Kafka was set up, so the standalone and
distributed environments ran one after the other.
Only distributed uses the kv backend, and the two environments use
different Kafka topic prefixes, so they can run at the same time. Keep one
instance per environment, since instances would share the metadata table
and Kafka topics. Runs with a runner-managed Kafka, an external server, a
test filter, or `-j 1` stay fully serial.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* test(sqlness): drop redundant restarts and sleeps
- flow_rebuild: drop the R2 block, which repeats R1 from the same state,
and the restart in R5, whose result R1 already asserts.
- flow_advance_ttl, flow_view, ttl_instant: drop sleeps that no assertion
depends on.
- flow_basic, flow_null, flow_call_df_func: drop the bytes_log section
(covered by flow_insert), a duplicate state_size query, an unchecked
insert, and flushes that run with no new data.
- alter_table_options, skip_wal: share restarts between independent tables.
- session_skip_wal, copy_skip_wal: check every case after a single restart.
Tables copied or inserted with skip_wal = true first get a WAL row that is
truncated, so the check that truncated WAL entries are not replayed stays.
- region_statistics: wait once for the statistics of all three tables.
- build_index_table: fold its index_size checks into
build_index_table_restart, which runs the same fixture and waits.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* test(sqlness): restore session_skip_wal and copy_skip_wal
Checking every case after a single restart dropped two state transitions
the original cases cover: recovering a region whose memtable only held
skipped rows, and writing to the recovered region before restarting again.
Restore the original cases.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* test(sqlness): restore the one-restart check in flow_auto_sink_table
#5112 checked the auto-created sink before a restart and the flow and sink
after it. #5987 moved the restart in front of the first SHOW and added a
second one. Flow recovery only creates the sink when it is missing, so the
second restart recovers from the same persisted state as the first.
Restore the original before/after check with one restart.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* test(sqlness): keep one instance per environment with external store addresses
Instances of the distributed environment share the etcd behind --store-addrs, so run one instance per environment when it is set, same as --setup-etcd.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* test(sqlness): restore the false-branch batch in flow_basic and assert it
The kept batch (20, 22) is all above the threshold. The second batch (10, 23) is the only input that hits the false branch of the CASE and makes the flow compute again. Restore it and check the result, which the original case never did.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
---------
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* feat: add experimental Jev SQL filtering
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* feat: gate Jev filtering behind an opt-in Cargo feature
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* ci: verify Jev registration with default features
Run the existing registry regression without the jev feature in both PR tests and merge-queue coverage, alongside the existing feature-enabled test runs.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* docs: clarify Jev concurrency scope and stabilization work
Document the per-expression/batch concurrency bound and track process-wide limiting, rate-limit backoff, and request budgets and metrics as stabilization prerequisites.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* feat: enable renamed ai-functions feature by default
Rename the Jev Cargo feature across the command, query, and function crates and enable it in their defaults. Update CI and documentation, retaining an isolated no-default-features registry check and the runtime API opt-in.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* ci: remove extra AI feature-off checks
Use the regular AI-enabled unit and coverage runs for the default feature configuration. Keep feature-off validation available locally and update the usage guide to match.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* refactor: rename AI feature to ai_functions
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
---------
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(tests): reject overlay directories before opening on all platforms
DatanodeOverlay::load() opened the target before checking is_file(). On
Unix, File::open on a directory succeeds and the loader rejects it with
"must be a regular file". On Windows, File::open on a directory fails up
front with "Access is denied", so the type check was never reached and
the rejects_directories_and_parse_errors test failed 4/4 in Nightly CI
(issue #8837).
Check std::fs::metadata before File::open: metadata succeeds on
directories on both platforms, so the error message is now identical
everywhere and the test assertion holds on Windows too.
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* test(datanode): make test_region_error deterministic across platforms
The second phase raced a 100ms mock handle delay against a 200ms
replay_timeout; on busy Windows CI runners the error could land after the
timeout fired, flaking reply.error.is_some() (Nightly CI, issue #8837).
Use a mock handle that returns the error on its first poll with no delay:
the catchup future completes before replay_timeout can ever fire, so the
test no longer depends on wall-clock scheduling.
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
---------
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* feat(table): add entity semantic declarations
Define open-ended greptime.semantic.entity.* options, validate entity columns at DDL time, and stamp OTLP trace tables with the service entity declaration.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* feat: add read-time entity relationships graph
Add computed semantic graph tables, typed DataFusion derivation plans for entity registry and trace calls edges, and streaming read-time execution.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* test: exclude semantic graph tables from table constraints
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* refactor(operator): name the plan-builder source groupings
Review feedback: build_registry_plan / build_calls_plan took anonymous
(declarations, DataFrame) tuples while the caller already grouped the same
fields. Introduce RegistrySource { declarations, scan } and CallsSource
{ service, scan } next to the builders and flow them through the frontend
caller and tests. The frontend-side EntitySource keeps holding a TableRef
(the operator builders stay pure over already-built scans), so the named
structs live in operator rather than reusing that type. No behavior change.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
---------
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* chore: gate soft-drop table behind the enterprise feature
Soft-drop table becomes an enterprise-only feature:
- metasrv rejects gc.experimental_soft_drop.enable=true at startup in
non-enterprise builds, and ddl_soft_drop_enabled is hard-disabled
without the enterprise feature as a second line of defense
- the UNDROP TABLE parser/AST/statement variant, ADMIN purge_table()
registration, and information_schema.recycle_bin registration are
compiled out unless the enterprise feature is enabled
- common-meta procedures, tombstone keys, and DdlTask serde stay
unconditional for persisted-procedure recovery and wire compatibility
- the [gc.experimental_soft_drop] section is removed from the OSS
example config and generated docs (moving to the enterprise repo)
- the soft-drop sqlness cases and their CI job are removed from OSS
(moving to the enterprise repo); affected information_schema .result
files are regenerated
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* refactor: limit unused_variables allow to non-enterprise builds
Addresses review comment: apply the allow via cfg_attr so enterprise
builds still catch accidental unused variables in register_admin_only.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* refactor: include the config key in the soft-drop enterprise gate error
Addresses review comment: name gc.experimental_soft_drop.enable in the
startup validation error so users can locate the setting quickly when
it is set via env vars or layered config.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* test: limit unused_mut allow to non-enterprise builds
Addresses review comment: apply the allow via cfg_attr so enterprise
builds still catch unused mut in the table_ddl_event test setup.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* feat: reject soft-drop DDL submissions in non-enterprise builds
Addresses review comment: clients could bypass the SQL-level gates by
submitting DdlTask::UndropTable or DdlTask::PurgeDroppedTable directly
to the procedure service. Reject fresh submissions at the DdlManager
boundary in non-enterprise builds while keeping the procedure loaders
registered for crash recovery and wire compatibility.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* test: stop --enable-gc from enabling soft drop in the sqlness template
Addresses review comment: the metasrv test template rendered
[gc.experimental_soft_drop] enable = true under the generic --enable-gc
flag, which non-enterprise metasrv now rejects at startup, making the
documented --enable-gc mode unusable in OSS. Keep the flag scoped to
plain GC; enterprise soft-drop coverage moves to the enterprise repo.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix: gate fresh soft-drop procedures
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* test: gate soft-drop fallback coverage
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix: gate soft-drop procedure implementation
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* refactor: gate drop table soft-drop behavior
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* refactor: gate expired soft-drop gc behavior
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* ci: test enterprise table ddl lifecycle
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* chore: mark purge_table as enterprise licensed
The purge_table module is compiled only with the enterprise feature, so
apply the Enterprise License header and register it with both license
header configurations.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* chore: mark recycle_bin as enterprise licensed
The recycle_bin module is compiled only with the enterprise feature, so
apply the Enterprise License header and register it with both license
header configurations.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* chore: mark soft-drop procedure sources as enterprise licensed
The purge and undrop procedure implementations plus the recycle-bin test
module compile only with the enterprise feature. Apply the Enterprise
License header and register them with both license configurations.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
---------
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* feat: support alter from primary_key to flat
Signed-off-by: evenyag <realevenyag@gmail.com>
* chore: alter flat to primary_key
Signed-off-by: evenyag <realevenyag@gmail.com>
* feat: change default_experimental_flat_format to true
Signed-off-by: evenyag <realevenyag@gmail.com>
* feat: compute channel size from splitted batch size
Signed-off-by: evenyag <realevenyag@gmail.com>
* test: add tests for split and channel size
Signed-off-by: evenyag <realevenyag@gmail.com>
* fix: always set sst_format from manifest on region open
sanitize_region_options did not set options.sst_format when the
default (PrimaryKey) matched the manifest value, leaving it as None
after reopen. This caused the alter format change to appear lost.
Signed-off-by: evenyag <realevenyag@gmail.com>
* test: fix tests
Signed-off-by: evenyag <realevenyag@gmail.com>
* test: show create table after alteration
Signed-off-by: evenyag <realevenyag@gmail.com>
* refactor!: rename default_experimental_flat_format to default_flat_format
The flat format is no longer experimental. Remove "experimental" from
the config field name, doc comments, and all references.
Signed-off-by: evenyag <realevenyag@gmail.com>
* chore: fix clippy
Signed-off-by: evenyag <realevenyag@gmail.com>
---------
Signed-off-by: evenyag <realevenyag@gmail.com>
* refactor: add datafusion-postgres dependency
* refactor: move and include pg_catalog udfs
* chore: update upstream
* feat: register table function pg_get_keywords
* feat: bridge CatalogInfo for our CatalogManager
Signed-off-by: Ning Sun <sunning@greptime.com>
* feat: convert pg_catalog table to our system table
* feat: bridge system catalog with datafusion-postgres
Signed-off-by: Ning Sun <sunning@greptime.com>
* feat: add more udfs
* feat: add compatibility rewriter to postgres handler
* fix: various fix
* fmt: fix
* fix: use functions from pg_catalog library
* fmt
* fix: sqlness runner
Signed-off-by: Ning Sun <sunning@greptime.com>
* test: adopt arrow 56.0 to 56.1 memory size change
* fix: add additional udfs
* chore: format
* refactor: return None when creating system table failed
Signed-off-by: Ning Sun <sunning@greptime.com>
* chore: provide safety comments about expect usage
---------
Signed-off-by: Ning Sun <sunning@greptime.com>
* feat: use flow batching engine
broken: try using logical plan
fix: use dummy catalog for logical plan
fix: insert plan exec&sqlness grpc addr
feat: use frontend instance in flownode in standalone
feat: flow type in metasrv&fix: flush flow out of sync& column name alias
tests: sqlness update
tests: sqlness flow rebuild udpate
chore: per review
refactor: keep chnl mgr
refactor: use catalog mgr for get table
tests: use valid sql
fix: add more check
refactor: put flow type determine to frontend
* chore: update proto
* chore: update proto to main branch
* fix: add locks for create/drop flow&docs: update docs
* feat: flush_flow flush all ranges now
* test: add align time window test
* docs: explain `nodeid` use in check task
* refactor: AddAutoColumnRewriter check for Projection
* refactor: per review
* fix: query without time window also clean dirty time window
* chore: better logging
* chore: add comments per review
* refactor: per review
* chore: per review
* chore: per review rename args
* refactor: per review partially
* chore: update docs
* chore: use better error variant
* chore: better error variant
* refactor: rename FlowWorkerManager to FlowStreamingEngine
* rename again
* refactor: per review
* chore: rebase after #5963 merged
* refactor: rename all flow_worker_manager occurs
* docs: rm resolved TODO
* feat: add mysql election
* feat: add mysql election
* chore: fix deps
* chore: fix deps
* fix: duplicate container
* fix: duplicate setup for sqlness
* fix: call once
* fix: do not use NOWAIT for mysql 5.7
* chore: apply comments
* fix: no parallel sqlness for mysql
* chore: comments and minor revert
* chore: apply comments
* chore: apply comments
* chore: add to table name
* ci: use 2 metasrv to detect election bugs
* refactor: better election logic
* chore: apply comments
* chore: apply comments
* feat: version check before startup
* feat(flow): (Part 1) refill utils
* chore: after rebase fix
* chore: more rebase
* rm refill.rs to reduce pr size
* chore: simpler args
* refactor: per review
* docs: more explain for instant requests
* refactor: per review
* feat: simple version switch
* chore: remove debug print
* chore: add common folder
* tests: add drop table
* feat: pull versioned binary
* chore: don't use native-tls
* chore: rm outdated docs
* chore: new line
* fix: save old bin dir
* fix: switch version restart all node
* feat: use etcd
* fix: wait for election
* fix: normal sqlness
* refactor: hashmap for bin dir
* test: past 3 major version compat crate table
* refactor: allow using without setup etcd
* feat: use `Inserter` as Frontend
* fix: enable procedure in flownode
* docs: remove `frontend_addr` opts
* chore: rm fe addr in test runner
* refactor: int test also use inserter invoker
* feat: flow shutdown&refactor: remove `Frontendinvoker`
* refactor: rename `RemoteFrontendInvoker` to `FrontendInvoker`
* refactor: per review
* refactor: remove a layer of box
* fix: standalone use `node_manager`
* fix: remove a `Arc` cycle
feat: flownode frontend client&test
feat: Frontend Client
feat: set frontend invoker for flownode
feat: set frontend invoker for flownode
chore: test script
WIP: test flow distributed
feat: hard coded demo
docs: flownode example toml
feat: add flownode support in runner
docs: comments for node
chore: after rebase
docs: add a todo
tests: move flow tests to common
fix: flownode sqlness dist test
chore: per review
docs: make
fix: make doc
dennis zhuang
Lei, HUANG
discord9
Weny Xu
jeremyhi
Yingwen
Ruihang Xia
QuakeWang
LFC
yihong
fys
Ning Sun
Cason Kervis
liyang
discord9
Yohan Wal
JohnsonLeeandYingwen
Lanqing Yang