Commit Graph
27 Commits
Author SHA1 Message Date
discord9 c3469fad51 fix(flow): retain tracked write completion across transient failures (#9525)
* fix(flow): retain tracked write completion across transient failures

Use receiver-issued single-write capabilities at the frontend and datanode. Rejoin exact admitted writes across unknown transport outcomes, then close and drain before allowing subsequent rounds. Keep ordinary clients and reads on their existing paths.

Separate schema map equality from encoded row identity so fresh protobuf decodes replay consistently even with NaN values. Cover receiver admission, owner cancellation, all-child drain, and retired capabilities.

Local feature-worktree checkpoint only. Main verified core enterprise check and 484 tests, plus DN enterprise check and all four tracked owner tests. Corrected seven-case timer acceptance remains to be rerun; this checkpoint does not authorize publishing or integration.

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix(flow): preserve upstream dependencies after rebase

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix(flow): fail fast on definitive tracked-write open rejection

A tracked write treats every application-level error from the frontend as
an ambiguous transport failure and retries the same pinned frontend
forever. During a rolling upgrade an older frontend does not recognize the
tracked-write command extension and processes Open as an ordinary empty
SQL query, rejecting it with InvalidSql (InvalidArguments); a non-internal
endpoint rejects the extension outright. Both are definitive rejections
that no same-frontend retry can ever resolve, so the flow stalls.

Classify the Open-phase errors by status code: a definitive rejection
(Unsupported / InvalidArguments / PermissionDenied) fails fast with a
terminal pre-write error, while genuinely ambiguous transport failures
(Unavailable, timeout, connection reset, ...) keep the same-capability
retry. Write/Close rejections are deliberately NOT failed fast: once a
write may have been admitted, a same-code rejection (e.g. WriterLost, or
UnknownCapability after a VIP moved Open->A/Write->B) leaves the outcome
ambiguous, and failing fast would let the caller replay and duplicate
rows.

Add a regression test that a legacy empty-SQL Open rejection fails fast
with a single attempt and surfaces as Error::External carrying a
TerminalError.

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix(flow): preflight tracked writes without submitting data

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test(flow): retain tracked command extensions while decoding

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

---------

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
2026-10-15 10:49:09 +00:00
discord9 28697c9e55 feat(datanode): execute nested MergeScan plans on the datanode (#9206)
* feat(datanode): execute nested MergeScan plans on the datanode

Give a datanode the ability to decode and execute a plan whose MergeScan
carries a nested plan (e.g. an inner MergeScan over a build table), so a
distributed sub-plan pushed to a datanode can itself fan out to other
datanodes. This is the execution/codec capability only; it is not wired
to any user-facing switch and produces no plan by itself.

Codec (src/query/src/query_engine/default_serializer.rs):
- MergeScanAwareSerializer keeps the request's SessionState and the
  engine catalog manager, and decodes a MergeScan payload's tables by
  (catalog, schema, table) name through the engine catalog rather than
  the region-aware request catalog list (which binds every table to the
  request's region and breaks on mismatched columns).

Datanode (src/datanode):
- New DatanodeRegionQueryHandler (region_query.rs) serves region queries
  for the MergeScan nodes of a received plan, resolving region leaders
  via PartitionRuleManager and dispatching to peer datanodes.
- Wire KvBackendCatalogManager + PartitionRuleManager + NodeClients into
  the datanode's catalog/cache registry (with_dist_planner stays false:
  a datanode never rewrites its own plan).

Bug fixes uncovered while wiring this up:
- cache registry: register the table/table_info/table_name/table_route/
  partition_info caches the datanode's catalog manager and partition
  rule manager require (previously missing -> panic on lookup).
- default_serializer: decode_sub_plan bridge no longer drives a future
  on a current-thread runtime from within another executor (spawn a
  dedicated thread + runtime when not on a multi-thread runtime).

Tested by query_engine::default_serializer unit tests: nested MergeScan
round-trip and engine-catalog resolution with mismatched column names.
No user-facing behavior change.

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix(query): keep GreptimeDB function bindings after rebuilding decode state

SessionStateBuilder::new_from_existing collects the function registry
(HashMap) into a Vec and build() re-registers each entry; register_udf
registers aliases before the function's own name and the Vec order is
unstable. Rebuilding the session state to install MergeScanAwareSerializer
after the GreptimeDB functions were registered could therefore rebind a
name/alias (e.g. date_format, an alias of DataFusion's to_char) back to
the built-in function. This path runs for every decoded plan, not only
nested MergeScan ones.

Extract the GreptimeDB function registration into register_greptime_functions
and re-apply it after the final build of both the top-level decode state and
the MergeScan payload state. Add regression tests asserting date_format stays
bound to the GreptimeDB implementation on both paths.

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix(datanode): address review on nested MergeScan capability

Three follow-ups from review of the nested MergeScan execution capability:

1. Layer the datanode caches (P2b). table_cache derives from
   table_info_cache + table_name_cache, and partition_info_cache derives
   from table_route_cache, but they were all registered in a single flat
   CacheRegistry whose invalidate() runs concurrently with no ordering.
   A derived cache could be invalidated and refilled from a base cache
   that had not been invalidated yet, leaving stale metadata. Build the
   base caches and the derived caches as separate layers of a
   LayeredCacheRegistry (reusing the existing layering mechanism) so base
   caches finish invalidating before derived ones. Both the production
   path (cmd) and the test scaffolding now build the same layered
   registry; the flat build_datanode_cache_registry is removed.

2. Don't let a nested MergeScan's internal region query compete for the
   concurrency permit (P2a). An outer query holds its permit until its
   stream is fully consumed, and a nested MergeScan makes the datanode
   issue region queries to peer datanodes as execution stages of that
   same query. If such an inner stage also took a permit, the outer query
   waiting on it would deadlock against its own held permit (or time out
   when max_concurrent_queries > 0). Mark inner DN->DN region queries as
   internal stages: QueryRequest gains an  flag carried over
   the wire in the request header's query_context extensions (reserved
   key, not injectable via hints), and the region server skips acquiring
   a concurrency permit for internal-stage requests on both acquire
   sites. Outer queries still acquire normally.

3. Add an end-to-end capability test
   (tests-integration/.../nested_merge_scan_capability_test.rs): hand-build
   Join(<local probe region>, MergeScan(build regions on both datanodes))
   with mismatched column names and duplicate build keys, ship it to a
   datanode over a real TCP Flight endpoint, consume the stream fully and
   compare the row multiset against the frontend's reference join result,
   assert a real cross-datanode DoGet via an RPC counter, and assert an
   inner region failure fails the whole query. Also migrate the
   with_real_datanode_grpc_addr + DatanodeRpcStats test scaffolding.

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix(query): adapt date_format regression test to the updated DataFusion fork

ScalarUDFImpl's trait bounds changed so its impl can no longer be
downcast via as_any(); assert the function equals the GreptimeDB
implementation instead, which to_char can never satisfy.

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test(datanode): cover the internal stage under the concurrency limit

Add an end-to-end regression test for the nested MergeScan capability
under max_concurrent_queries = 1: both datanodes' only permit is held by
the outer queries' streams while their inner MergeScans fetch the build
regions from the same datanodes over real Flight connections. The result
consumption is bounded by a timeout, so a broken link anywhere in the
chain (marking the inner request internal, carrying the marker in the
request header, honoring it at the receiving acquire sites) fails the
test instead of deadlocking.

Also add a minimal with_datanode_options_override hook to the cluster
builder, and fix the test module doc: the outer request is handed to
RegionServer::handle_remote_read in process; only the inner region
queries go over the wire.

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test(datanode): cover cold route loads under unrelated invalidations

The table route cache uses InitStrategy::VersionChecked, whose version
counter belongs to the whole CacheContainer rather than to a single
table: any TableId invalidation bumps the shared version, so a cold
load of one table's route retries when an unrelated table is
invalidated concurrently. Document this as a known limitation on the
region query handler, and pin the behavior with an integration test:
run the nested plans while a background task invalidates an unrelated
table id on every datanode, assert the results still match the expected
multiset (retries only add work, they never return a stale route), and
report the observed load amplification via the cache miss counter
(2 loads without invalidations, 8 over three runs with 24 unrelated
invalidations in this run). Also expose the datanode cache registries
on the test cluster so a test can drive the local invalidation path
that the heartbeat handler drives after a metasrv broadcast.

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* docs(query): trim nested plan decoder commentary

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* refactor: remove redundant nested scan support plumbing

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix(query): decode nested MergeScan payloads asynchronously

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test(query): avoid unnecessary extension name allocation

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix(query): retain extension decoding for manual session states

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* refactor(datanode): keep nested scan query admission unchanged

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix(datanode): honor query handler factories and trim nested scan fixtures

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix(datanode): retain required cache error import

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* refactor(query): reuse registered state for nested scan decoding

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test(query): verify payload information schema table functions

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test(query): retain nested scan runtime and cold-route guards

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* refactor(query): address nested scan review feedback

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix(session): preserve read preference across region query hops

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test(session): verify strict context rejection at Flow ingress

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test(query): remove redundant serializer error case

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

---------

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
2026-10-10 06:49:38 +00:00
dennis zhuang 079ffec116 chore: remove dead code left behind by removed features (#9377)
* chore: remove dead code left behind by removed features

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* chore: remove unused RouteInfoCorrupted error

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

---------

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
2026-09-28 12:35:47 +00:00
1d8d95c12a chore(deps): replace cargo-udeps with cargo-shear for unused dependency checks (#9294)
* chore(deps): replace cargo-udeps with cargo-shear for unused dependency checks

cargo-udeps requires a nightly toolchain and its pinned version (0.1.61)
no longer detects unused dependencies against current cargo internals —
unused deps have landed on main undetected (e.g. humantime in
common-frontend since #6689). cargo-shear is a standalone static analyzer
that runs on any toolchain.

- Swap 'make check-udeps' / 'make fix-udeps' recipes to 'cargo shear' /
  'cargo shear --fix' and retire scripts/fix-udeps.py
- CI: install cargo-shear in the check-udeps job; drop the build cache
  and protoc steps (cargo-shear never compiles)
- Remove ~150 unused dependency declarations found by cargo-shear, move
  misplaced deps to the correct sections, drop orphaned
  [workspace.dependencies] entries (arrow-cast, rustc-hash)
- Add [package.metadata.cargo-shear] ignored entries with explanations
  for dependencies that are structurally required despite no textual
  reference: sqlparser (required by sqlparser_derive expansions in
  datatypes, common-query), common-error (required by common-macro's
  stack_trace_debug expansions in session, tests-fuzz), k8s-openapi
  (feature-pinning for the transitive kube dependency in tests-fuzz),
  tikv-jemalloc-sys (link-only, enables jemalloc profiling features in
  common-mem-prof), protobuf (required by build.rs-generated bindings in
  log-store)
- Drop the obsolete [package.metadata.cargo-udeps.ignore] sections

Part of #9289

Signed-off-by: Ning Sun <sunning@greptime.com>

* fix(meta): populate physical metric table column ids (#9286)

* fix(meta): populate physical metric table column ids

Signed-off-by: dhruvxvaishnav <dhruvvaishnav687@gmail.com>

* test(meta): verify physical metric column ids

Signed-off-by: dhruvxvaishnav <dhruvvaishnav687@gmail.com>

---------

Signed-off-by: dhruvxvaishnav <dhruvvaishnav687@gmail.com>

* fix(postgres): return empty responses for comment-only SQL (#9295)

fix(postgres): handle parsed empty queries in both protocols

Signed-off-by: houyuwushang <180804215+houyuwushang@users.noreply.github.com>

* ci: create docs follow-up issue on PR merge instead of on label (#9237)

* ci: create docs follow-up issue on PR merge instead of on label

The docbot workflow previously created a docs-repo issue as soon as the
'docs-required' condition was detected (PR opened/edited with the docs
checkbox ticked), even if the PR was never merged.

Now the workflow also triggers on PR 'closed':
- opened/edited: only manage the docs-required/docs-not-required labels
- closed: create the docs issue only when the PR was actually merged and
  carries the docs-required label

This also lets maintainers control issue creation by manually adding or
removing the docs-required label before merging.

Signed-off-by: Ning Sun <sunning@greptime.com>

* fix: address review comments on docs issue creation timing

- Only touch docs labels when the docs checkbox state actually changed
  in an edit. Previously, editing any other part of the PR body while
  the checkbox stayed checked removed the docs-required label, silently
  dropping the docs follow-up now that issue creation happens at merge.
  Unchanged checkbox now leaves labels untouched, which also preserves
  manual label overrides.
- Do not trust the closed event's stale label snapshot at merge time:
  re-read the live PR via the API and create the docs issue if the
  docs-required label is present OR the checkbox is ticked in the
  current body.
- Make the workflow concurrency group action-aware so a merge run does
  not cancel an in-flight label update from an edit run.

Signed-off-by: Ning Sun <sunning@greptime.com>

* fix: make docs-required label the single source of truth at merge

The label-OR-checkbox merge condition could not distinguish an
intentional opt-out from an unfinished label update: removing
docs-required while the checkbox stayed checked still produced an
issue, and unchecking the box could still produce one if the merge
read the stale label before the edit run removed it.

At merge time, wait for any pending docbot runs on the PR head SHA to
finish their label updates (bounded to 5 minutes), then decide solely
by the live docs-required label. Adds actions: read permission for
listing workflow runs.

Signed-off-by: Ning Sun <sunning@greptime.com>

---------

Signed-off-by: Ning Sun <sunning@greptime.com>

* perf(promql): push label filters into grouped join inputs (#9280)

* perf(promql): propagate matching filters through grouped joins

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* perf(promql): check matcher safety on the receiving operand

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* refactor(promql): spell out the shapes a filter may cross

`preserves_filter` ended in `_ => true`, which was only sound because
`selector_matchers` independently rejects label rewriting, `count_values`,
subqueries and non-rollup calls on the same operand. Loosening the latter
alone would have silently pushed a matcher below a label rewrite. List the
shapes that carry a scan filter instead and default to `false`.

Cite #9207 for the result labels the grouped cases record: the join
projects the right operand's tag set, so `zone` is missing wherever the
right side aggregates it away.

No behavior change.

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* test(promql): assert the new pushdowns reach the scan

The grouped-join unit tests feed tag columns by hand and the SQLness case
only checks results, which are identical whether or not the rewrite fires.
Nothing would have failed if scalar arithmetic, ranking or grouped
matching stopped propagating. Assert through the planner that the matcher
reaches both scans, with a global topk one-side as the counter-example.

Also state that the duplicate-one-side cases record a cross product
Prometheus rejects (#9209), so the baseline is not read as intended
semantics.

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

---------

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* fix(ci): build tests-integration lib with meta-srv/mock (#9299)

* fix(ci): build tests-integration lib with meta-srv/mock

tests-integration's lib code (src/cluster.rs) uses meta_srv::mocks, but
the dependency carrying the mock feature sits in [dev-dependencies].
Builds that only touch the lib, such as the apidoc job's cargo doc
--workspace, resolve meta-srv without mock and fail with E0432.
--all-targets builds unify dev-dependency features, which is why check,
clippy and nextest stayed green.

Move the mock-enabled meta-srv entry back to [dependencies]. The other
testing features moved out in #9072 are not needed by the lib and stay
in [dev-dependencies].

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* test(repartition): split per-case repartition tests

test_repartition_metric ran four format/primary-key-encoding cases in a
single test function, and test_repartition_mito ran two format cases.
Each case builds its own 3-datanode cluster and runs a full repartition
plus GC cycle, so on S3 the metric test took 165-178s against the 180s
nextest terminate-after. Merge queue runs failed on it at random.

Split each case into its own test. Cases were already independent, so
they now run in parallel and each stays far inside the timeout, and a
failure points at one encoding instead of four.

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

---------

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* feat(json2): support altering JSON2 settings (#9029)

* feat(sql): support alter syntax for JSON2 columns

Signed-off-by: fys <fengys1996@gmail.com>

* fix(json2): preserve rows on type hint mismatch during compaction

* refactor(json2): simplify alter settings handling

* fix(json2): preserve coerced values during compaction

* chore: remove unnecessary clone

* chor: reduce memory allocations

* fix: cargo clippy

* chore: update greptime-proto to main branch

* refactor(datatypes): unify string handling with other JSON type hints

* fix: cr

---------

Signed-off-by: fys <fengys1996@gmail.com>

* fix: keep compaction pruning, metadata, and index work on compact runtime (#9304)

* fix: run compaction pruner tasks on compact runtime

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix: keep compaction metadata and index work on compact runtime

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

---------

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* feat: add AI matching, classification, and scoring functions (#9300)

* feat: return matching scores from jev

Replace the experimental three-argument Boolean function with jev(text, prompt) returning a Float64 probability in [0, 1]. Move threshold comparisons into SQL and update tests and migration examples.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* feat: add Jev choice and score functions

Share asynchronous execution across Noul, Choice, and Score. Validate JSON criteria before requests and return typed scalar answers. Add SQL and HTTP mock coverage with usage examples.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* refactor: use generic AI SQL function names

Expose ai_match, ai_choose, and ai_score and move their implementation, tests, and usage guide under generic AI names. Document the current unreleased interface without migration history.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix: share constant AI criteria within each batch

Borrow scalar string arguments and lazily parse constant criteria once per batch. Share the parsed allocation across requests while preserving NULL propagation and batch validation before HTTP calls.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* feat: preserve AI score uncertainty in JSONB results

Return score, confidence, and probabilities in criteria-level order from one evaluation. Validate the distribution and preserve provider precision. Add JSON extraction, uncertainty, and single-request regressions, and document confidence-aware ranking.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* docs: explain reuse of volatile AI evaluations

Document repeated SELECT and WHERE evaluation costs as N + M requests, and show subquery aliases for reusing scalar or structured AI results without additional model calls.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

---------

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* feat: share logical table batching with OTLP metrics (#9288)

* feat: share logical table batching with OTLP metrics

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix: unify pending rows batch acknowledgement policy

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix: align logical batcher example configuration expectations

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix: align batcher worker channel defaults to 65536

Signed-off-by: WenyXu <wenymedia@gmail.com>

---------

Signed-off-by: WenyXu <wenymedia@gmail.com>

* perf(mito2): lazily decode dense primary key columns (#9226)

* perf(mito2): lazily decode dense primary key columns

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* perf(mito2): bypass lazy decoding for full primary keys

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix(mito-codec): preserve prefix decoding errors

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* refactor(mito-codec): align encoded length helper naming

Rename encoded_length to encoded_len and update all callers to match the other length helpers in the module.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* refactor(mito2): clarify conditional dense key decoding

Rename decode_dense_pk to ensure_dense_pk_decoded so callers can see that existing decoded values are preserved and only missing caches are populated.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* refactor(mito-codec): share string framing in row converter

Move encoded_string_len to the parent module so Dense and Sparse use the same framing helper without depending on each other. Preserve its implementation and visibility.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

---------

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* chore: sync lock

* fix: shear and check issues

---------

Signed-off-by: Ning Sun <sunning@greptime.com>
Signed-off-by: dhruvxvaishnav <dhruvvaishnav687@gmail.com>
Signed-off-by: houyuwushang <180804215+houyuwushang@users.noreply.github.com>
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
Signed-off-by: fys <fengys1996@gmail.com>
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
Signed-off-by: WenyXu <wenymedia@gmail.com>
Co-authored-by: Dhruv Vaishnav <dhruvvaishnav687@gmail.com>
Co-authored-by: houyuwushang <180804215+houyuwushang@users.noreply.github.com>
Co-authored-by: dennis zhuang <killme2008@gmail.com>
Co-authored-by: fys <40801205+fengys1996@users.noreply.github.com>
Co-authored-by: Lei, HUANG <6406592+v0y4g3r@users.noreply.github.com>
Co-authored-by: Weny Xu <wenymedia@gmail.com>
2026-09-23 04:56:44 +00:00
Weny Xu 60c7bbcaf3 refactor: share query channel definition (#8825)
refactor: share query channel protocol mapping

Signed-off-by: WenyXu <wenymedia@gmail.com>
2026-08-10 10:47:50 +00:00
discord9 923c4227a3 feat: remote dyn filter basics (#7979)
* feat: dyn filter update abi

Signed-off-by: discord9 <discord9@163.com>

* feat: add remote dyn filter region rpc scaffolding

Signed-off-by: discord9 <discord9@163.com>

* feat: query id

Signed-off-by: discord9 <discord9@163.com>

* fix: prevent spoof

Signed-off-by: discord9 <discord9@163.com>

* fix: more proofing

Signed-off-by: discord9 <discord9@163.com>

* feat: make payload base64

Signed-off-by: discord9 <discord9@163.com>

* chore

Signed-off-by: discord9 <discord9@163.com>

* update

Signed-off-by: discord9 <discord9@163.com>

* chore: resolve conflict

Signed-off-by: discord9 <discord9@163.com>

* chore: update to main

Signed-off-by: discord9 <discord9@163.com>

* fix: add back defensive check

Signed-off-by: discord9 <discord9@163.com>

---------

Signed-off-by: discord9 <discord9@163.com>
2026-05-14 04:05:56 +00:00
LFC f1abe5d215 feat: suspend frontend and datanode (#7370)
Signed-off-by: luofucong <luofc@foxmail.com>
2025-12-10 12:18:24 +00:00
dennis zhuang 1234911ed3 refactor: query config options (#6781)
* feat: refactor columnar and vector conversion

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* feat: initialize config options from query context

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* fix: failure tests

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* chore: revert ColumnarValue::try_from_vector

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

---------

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
2025-09-01 07:00:26 +00:00
shuiyisong 51dc393371 chore(otlp_metric): support attr list in header opts (#6617)
* chore: support attr list in OTLP metrics

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* chore: fix cr issue

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

---------

Signed-off-by: shuiyisong <xixing.sys@gmail.com>
2025-07-31 08:54:45 +00:00
Weny Xu 4ef9afd8d8 feat: introduce read preference (#5783)
* feat: introduce read preference

* feat: introduce `RegionQueryHandlerFactory`

* feat: extract ReadPreference from http header

* test: add more tests

* chore: apply suggestions from CR

* chore: apply suggestions from CR
2025-04-01 09:17:01 +00:00
yihong 369b59c84a fix: drop unused dep using udeps to minial the size (#5301)
* fix: drop unused dep using udeps to minial the size

Signed-off-by: yihong0618 <zouzou0208@gmail.com>

* fix: adress comments fix the problem

Signed-off-by: yihong0618 <zouzou0208@gmail.com>

---------

Signed-off-by: yihong0618 <zouzou0208@gmail.com>
2025-01-08 06:36:22 +00:00
Ning Sun 3133f3fb4e feat: add cursor statements (#5094)
* feat: add sql parsers for cursor operations

* feat: cursor operator

* feat: implement RecordBatchStreamCursor

* feat: implement cursor storage and execution

* test: add tests

* chore: update docstring

* feat: add a temporary sql rewrite for cast in limit

this issue is described in #5097

* test: add more sql for cursor integration test

* feat: reject non-select query for cursor statement

* refactor: address review issues

* test: add empty result case

* feat: address review comments
2024-12-06 09:32:22 +00:00
Ning Sunandshuiyisong b741a7181b feat: track channels with query context and w/rcu (#4448)
* feat: add source channel to meter recorders

* feat: provide channel for query context

* fix: testing and extension get for query context

* chore: revert cargo toml structure changes

* fix: querycontext modification for prometheus and pipeline

* chore: switch git dependency to main branches

* chore: remove TODO

* refactor: rename other to unknown

---------

Co-authored-by: shuiyisong <113876041+shuiyisong@users.noreply.github.com>
2024-07-31 07:30:50 +00:00
JohnsonLeeandtison 83643eb195 feat: Support printing postgresql's bytea data type in its "hex" and "escape" format (#3567)
* feat: support set variable statement of session

* feat: support printing postgresql's bytea data type in its "hex" and "escape" format in ugly way

* refactor: add 'SessionConfigValue' type and unify the name

* doc: add license header

* refactor: confine coupling with 'sql::ast::Value' in SessionConfigValue

* refactor: move all bytea wrapper into bytea.rs

* fix: remove unused import in context.rs and postgres.rs

* refactor: rename 'set_configuration_parameter' to 'set_session_config'

rename 'set_configuration_parameter' in statement_.rs to 'set_session_config'

* refactor: use mod to organize options via macro

* refactor: re-model the session config value with static type

* test: add integration test

* refactor: move the encode bytea by format type logic into encoder

refactor: use Arc<DashMap> instead of DashMap in QueryContext

refactor: use Arc<DashMap> instead of DashMap in QueryContext

    Avoid expensive clone

refactor: use unreachable!() instead of unimplemented!()

refactor: move the encode bytea by format type logic into encoder

test: add binary format integration test case

* test: add ut for byte related type

* doc: remove TODO of bytea_output

* refactor: simplify the implementation with simple struct instead of complex typing

* fix: typo of 'Available'

* fix compile

Signed-off-by: tison <wander4096@gmail.com>

---------

Signed-off-by: tison <wander4096@gmail.com>
Co-authored-by: tison <wander4096@gmail.com>
2024-03-27 01:54:41 +00:00
dimbtp dd18d8c97b build(deps): remove some unused dependencies (#3582)
* build(deps): remove some unused dependencies

* add `arc-swap` dependency back
2024-03-26 12:48:28 +00:00
4b36c285f1 feat: flush or compact table and region functions (#3363)
* feat: adds Requester to process table flush and compaction request

* feat: admin_fn macros for administration functions

* test: add query result

* feat: impl flush_region, flush_table, compact_region, and flush_region functions

* docs: add Arguments to admin_fn macro

* chore: apply suggestion

Co-authored-by: Zhenchi <zhongzc_arch@outlook.com>

* chore: apply suggestion

Co-authored-by: Zhenchi <zhongzc_arch@outlook.com>

* fix: group_requests_by_peer and adds log

* Update src/common/macro/src/admin_fn.rs

Co-authored-by: Ruihang Xia <waynestxia@gmail.com>

* feat: adds todo for spawan thread

* feat: rebase with main

---------

Co-authored-by: Zhenchi <zhongzc_arch@outlook.com>
Co-authored-by: Ruihang Xia <waynestxia@gmail.com>
2024-02-27 08:57:38 +00:00
SteveLauC e9a2b0a9ee chore: use workspace-wide lints (#3352)
* chore: use workspace-wide lints

* respond to review
2024-02-22 01:01:10 +00:00
Ruihang Xia 93f21b188d chore: unify workspace dependencies and remove some unused (#2713)
* log-store

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* base64

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* udeps

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* snafu

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* snafu style

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* change the workspace deps' style

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

---------

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
2023-11-09 07:34:22 +00:00
shuiyisong 067917845f fix: carry dbname from frontend to datanode (#2520)
* chore: add dbname in region request header for tracking purpose

* chore: fix handle read

* chore: add write meter

* chore: add meter-core to dep

* chore: add converter between RegionRequestHeader and QueryContext & update proto version
2023-10-08 06:30:23 +00:00
shuiyisong 7f51141ed0 refactor: auth crate (#2148)
* chore: move user_info to auth crate

* chore: temp commit before resolving tests compile error

* chore: fix compile issue

* chore: minor fix

* chore: tmp save

* chore: change user_info to trait

* chore: minor change & use auth result user info in pg session setup

* chore: add as_any to user_info

* chore: rename user_info

* chore: remove ice file

* chore: add permission checker

* chore: add grpc permission check

* chore: add session spawn user_info to query_ctx

* chore: minor update

* chore: add permission checker to sql handler & temp save

* chore: add permission checker to prometheus handler

* chore: add permission checker to opentsdb handler

* chore: add permission checker to other handlers

* chore: add test

* chore: add user_info setting on http entrance

* chore: fix toml

* chore: remove box in permission req

* chore: cr issue

* chore: cr issue
2023-08-14 02:51:26 +00:00
Ning Sun c39de9072f refactor: use workspace dependencies for internal modules (#2119)
* refactor: use workspace dependencies for internal modules

* fix: resolve issue with mock module in datanode

* refactor: update test modules
2023-08-08 11:02:34 +00:00
shuiyisong 9c1f0234de refactor: query context (#2022)
* chore: refactor query_context

* chore: remove use statement

* chore: make query_context build return arc

* fix: sqlness test

* fix: cr issue

* fix: use unwrap or else
2023-07-25 06:11:34 +00:00
dennis zhuang ab5dfd31ec feat: sql dialect for different protocols (#1631)
* feat: add SqlDialect to query context

* feat: use session in postgrel handlers

* chore: refactor sql dialect

* feat: use different dialects for different sql protocols

* feat: adds GreptimeDbDialect

* refactor: replace GenericDialect with GreptimeDbDialect

* feat: save user info to session

* fix: compile error

* fix: test
2023-05-30 09:52:35 +08:00
Ning Sunanddennis zhuang 067c5ee7ce feat: time_zone variable for mysql connections (#1607)
* feat: add timezone info to query context

* feat: parse mysql compatible time zone string

* feat: add method to timestamp for rendering timezone aware string

* feat: use timezone from session for time string rendering

* refactor: use querycontectref

* feat: implement session/timezone variable read/write

* style: resolve toml format

* test: update tests

* Apply suggestions from code review

Co-authored-by: dennis zhuang <killme2008@gmail.com>

* Update src/session/src/context.rs

Co-authored-by: dennis zhuang <killme2008@gmail.com>

* refactor: address review issues

---------

Co-authored-by: dennis zhuang <killme2008@gmail.com>
2023-05-22 18:30:23 +08:00
LFC af935671b2 feat: support "use" in GRPC requests (#922)
* feat: support "use catalog and schema"(behave like the "use" in MySQL) in GRPC requests

* fix: rebase develop
2023-02-02 20:02:56 +08:00
LFCandluofucong dc52a51576 chore: upgrade to Arrow 29.0 and use workspace package and dependencies (#782)
* chore: upgrade to Arrow 29.0 and use workspace package and dependencies

* fix: resolve PR comments

Co-authored-by: luofucong <luofucong@greptime.com>
2022-12-23 14:28:37 +08:00
LFC 6127706b5b feat: support "use" stmt part 1 (#672)
* feat: a bare sketch of session; support "use" in MySQL server; modify insertion and selection related codes in Datanode
2022-12-01 17:05:32 +08:00