When every partition channel is full, the distributor walked the senders
round-robin with a 100us send timeout each. Tokio timers fire on a 1ms
tick, so a consumer that leaves some partitions unpolled costs about one
millisecond per unpolled partition for every series batch.
SeriesScan::build_stream and scan_all_partitions drain partitions one
after another, and SortPreservingMergeExec above a multi-partition
SeriesScan only pulls the partition holding the current minimum key.
With 16 partitions and one batch per series after a merge, a 40-series
scan read partition by partition took ~480ms (measured on the native
PromQL two-input route).
Wait on all senders at once and send to the first one that frees a slot.
The send-timeout metric goes away with the timeout.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
Introduce the metasrv side procedure that fences a datanode before it is
drained, and collect the regions placed on it:
- mark the datanode unschedulable through the node scheduling manager
- build the drain plan from the datanode table index and the authoritative
table routes, splitting leader regions from follower regions
- keep the plan in the volatile context: it is derived from the table
routes, so a restarted procedure rebuilds it instead of reading a
persisted copy
The leader regions are not migrated yet: the migration state machine lands
separately. The procedure has no caller yet, so it stays inert.
Signed-off-by: WenyXu <wenymedia@gmail.com>
* fix: acknowledge writes only within the region's leader lease
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* fix: drop cached logical columns after a failed logical drop
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* fix: keep the prepared region for the lease check after the WAL append
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* fix: serialize a logical drop with fills of the logical column cache
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* test: check the drop lock and grant long leases deterministically
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* test: cover a bulk insert that loses its lease during the WAL append
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
---------
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* fix(log-store): refuse an object store that ignores conditional create
The object store WAL checks only the write_with_if_not_exists capability
before it opens, and OpenDAL declares that capability per service. A store
behind the S3 service can accept If-None-Match and overwrite the object
anyway, so the WAL could overwrite its own objects and lose acknowledged
writes without an error.
Before recovery, the store now creates a probe key, unique to the open,
twice under <node prefix>/probes/, outside the objects it lists, and
deletes it. The second create must be refused with a condition failure,
otherwise the open fails with an error naming the object store. The
probe costs two writes and one delete per open.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* test(log-store): drop a comment that restates the keys helper
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* fix(log-store): accept AlreadyExists from the second probe create
A backend may report an existing conditional-create target as AlreadyExists. The first create stored the probe key, so that error can only refer to it.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
---------
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* fix(meta): fence datanode processes replaced by a newer process
Two datanode processes can run with the same node id, and metasrv
grants leases and delivers instructions by node id only. Every datanode
heartbeat already carries node_epoch, the time the process started.
The metasrv leader records the highest accepted epoch per node id under
__datanode_epoch/{node_id}, a prefix of the leader cache, and raises it
with compare-and-put. The heartbeat session checks each datanode request
with a non-zero epoch before the handler chain: a lower epoch gets a
reply without running the chain, so no lease, keep-alive, stats or
mailbox reply. An epoch more than 5 minutes ahead of the leader's clock
is not recorded and is fenced too.
Datanode pushers carry the epoch of their process: a lower epoch cannot
replace a higher one, and a stream removes only its own pusher.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* fix(meta): keep the leader cache load and pusher updates coherent
A write made while the leader cache is disabled, as during the load at
leader start, went to the store without the cache noticing, so the load
could install a value read before the write. Such writes now create new
cache versions around the store call, and the load drops what it loaded
for a prefix if a write happened meanwhile. This also covers node
addresses, which heartbeats write during the load.
A stream's pusher that takes its first epoch is updated in place instead
of being replaced, so messages waiting on it are not cancelled, and the
connection gauge counts only a pusher that fills an empty slot.
The epoch check moves into the heartbeat service.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* fix(meta): keep the leader cache coherent with writes while it is disabled
A write that started while the cache was disabled could commit after the
load had read the old value and the cache had been enabled, leaving the
old value cached. Writes now maintain the cache whether or not it is
enabled, as enabled writes already did: new versions around the store
call, then the key is updated or invalidated.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* test(meta): pin the epoch 0 exemption and pusher order, and update docs
The datanode test without an epoch now runs after another process has
recorded one, and the pusher test checks that a stream's first accepted
request receives what its handlers broadcast. The leader cache load no
longer claims that writes must not run while it loads, and the meta-srv
guide states that epoch 0 is not checked.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* fix(meta): register a datanode pusher only after its first request passes
A datanode stream registered an epoch-0 pusher on its handshake, so an
older process's reconnecting stream could take an empty slot, or a newer
process's pusher before its first heartbeat, for about a round trip. A
datanode stream now registers its pusher on its first request that
passes the epoch check (epoch-0 requests after the handshake included),
still before the handlers run.
Pusher registration is logged only when the insert happens. Tests cover
a fenced stream removing its own pusher, a handshake not taking an empty
slot, and a KV error failing the request; two steps that could not fail
are removed.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* fix(meta): check a datanode epoch again before replying
A request could pass the epoch check, wait in the handler chain while a
newer process of the node was recorded, and still return a region lease.
After the handlers run, the session checks the epoch again against the
cached record and, if a newer process has been recorded, replies as to a
fenced request and removes the stream's own pusher. This costs one more
read of the cached record per checked datanode request.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* fix(meta): handle a datanode's requests one at a time
A request of an older process could pass the epoch check, wait in the
handlers while a newer process was admitted, and then keep the node
alive with its own address. Checking the epoch again after the handlers
only dropped the reply.
The heartbeat session now takes an in-memory lock for the node id
before checking a request that carries an epoch, and releases it after
the handlers, before the reply is sent. A newer process is therefore
admitted only after the handlers of an older request finish. The check
after the handlers is removed.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
---------
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* fix: reclaim HDFS temporary files left behind by a datanode crash
Signed-off-by: lczllx <2181719471@qq.com>
* fix(mito2): stop the temporary file sweep from starving GC jobs
Signed-off-by: lczllx <2181719471@qq.com>
* feat(mito2): add metrics for the temporary file sweep
Signed-off-by: lczllx <2181719471@qq.com>
* refactor(object-store): share one definition of the temporary file name
Signed-off-by: lczllx <2181719471@qq.com>
* fix(mito2): skip the temporary file sweep when no GC job is allowed
Signed-off-by: lczllx <2181719471@qq.com>
* refactor: gate the temporary file sweep behind the hdfs-object-store feature
Signed-off-by: lczllx <2181719471@qq.com>
* fix(mito2): sweep temporary files when a create opens an existing region
Signed-off-by: lczllx <2181719471@qq.com>
* docs(config): document the temporary file sweep in the GC settings
Signed-off-by: lczllx <2181719471@qq.com>
* refactor: align the temporary file sweep code with the surrounding style
Signed-off-by: lczllx <2181719471@qq.com>
---------
Signed-off-by: lczllx <2181719471@qq.com>
* feat: postgres copy-in support
* feat: add postgres copy-in framework
* feat(postgres): route COPY FROM STDIN flushes through the pending-rows batcher
Follow-up to #9302: begin_copy_in now uses the batching-aware
new_query_context, so copy-in batches are submitted to the
pending-rows batcher like SQL INSERTs when PostgreSQL batching is
enabled for the connection, and keep the direct write path when it
is not. The batcher acknowledges synchronously, so the
CommandComplete row count stays exact in both modes.
Adds test_postgres_copy_in_with_batching asserting the batcher
flush metric increments only when batching is enabled.
Signed-off-by: Ning Sun <sunning@greptime.com>
* refactor(postgres): open up the COPY FROM STDIN WITH clause for format-specific options
The WITH (...) option list is now open instead of restricted to
sqlparser's COPY grammar:
- COPY ... FROM STDIN statements are parsed textually (new
copy_in::parse module, tokenizer-based) so arbitrary KEY value
entries reach the codecs; sqlparser would reject options it does
not know before we ever see them. WITH is optional, matching
PostgreSQL.
- CopyOptionSet becomes an open ordered map of KEY -> String |
Number | Ident | Boolean | Flag with typed take_* accessors
(take_string, take_single_byte, take_flag, raw take_value) and
case-insensitive keys; duplicate options are rejected.
- Each format codec takes the options it understands and rejects the
leftovers with one message naming them ('COPY option(s) X are not
supported by format Y'), so a typo cannot be silently ignored.
The text format now also rejects HEADER/QUOTE/ESCAPE instead of
ignoring them, matching PostgreSQL.
A future format (e.g. influxdb line protocol with
WITH (FORMAT influxdb_line, precision 'ns')) only needs its codec
and a registry entry; no parser changes.
Signed-off-by: Ning Sun <sunning@greptime.com>
* refactor(postgres): reuse arrow-csv decoder for COPY FROM STDIN csv parsing
Replace the hand-rolled CSV record parser and per-value conversion
with arrow-csv's incremental Decoder — the same engine backing
COPY ... FROM 'file.csv' in common-datasource — so both COPY routes
accept identical data. The decoder handles record splitting (quoting,
escaped quotes, custom delimiters, CRLF, headers, chunk boundaries)
and typed conversion against the targeted columns' schema in one
pass; decoded batches become insert rows via the existing
api::helper::vectors_to_rows and datatypes Helper::try_into_vectors,
and the decode schema comes from the greptime Schema's arrow
projection.
The codec framework now has parsers emit complete insertable rows
(api Row) instead of raw byte fields: create_parser takes the
targeted columns and is fallible; text converts with the shared
text-representation helpers, csv with the arrow decoder.
Documented consequences of sharing the engine: JSON and Binary
columns and bare epoch-integer timestamps are rejected for csv, and
an empty field — quoted or not — is NULL; a custom NULL string is
rejected accordingly.
Signed-off-by: Ning Sun <sunning@greptime.com>
* docs(postgres): make the copy-in framework explicitly format-open
Reframe the codec framework docs so nothing assumes csv is the only
other text format: the shared pieces (Field, convert_text_field,
check_text_representable_type) are described as building blocks for
any format whose records carry values as text, while formats with
their own typed decoding (csv today) build rows themselves. State
explicitly that formats are not limited to PostgreSQL's own —
GreptimeDB-specific formats (e.g. a line protocol) register like any
other and define their own options through the open WITH clause.
No code changes beyond a test rename and a doc clarifier.
Signed-off-by: Ning Sun <sunning@greptime.com>
* test(postgres): driver-level coverage for CSV copy-in options
Add an end-to-end test through tokio-postgres exercising HEADER,
custom DELIMITER, quoted fields with embedded newlines, escaped
quotes and CRLF record separators, with read-back verification
through the real insert path. WITH is omitted, matching PostgreSQL.
Signed-off-by: Ning Sun <sunning@greptime.com>
* test(postgres): drop the raw-socket copy-in tests
The simple-protocol raw client duplicated what tokio-postgres and the
integration tests already cover; keep the driver-based tests only.
Signed-off-by: Ning Sun <sunning@greptime.com>
* refactor(postgres): address review: fix text-format O(n^2) drain, share type checks, hex::decode
- text feed: take the longest complete line prefix in one drain and
iterate lines over the slice, instead of draining per line (each
drain memmoved the whole buffer; ~1300 drains for a 64KB chunk of
short lines)
- csv check_column_type: delegate to the shared
check_text_representable_type and additionally reject Binary/Json
- decode_bytea: use the hex crate instead of a hand-rolled decoder
- semantic_type_of: use ColumnSchema::is_time_index() like the flow
adapter's column_schemas_to_proto, instead of index lookups
- parse: drop the unused expect_ident parameter; document the
identifier-case decision
- PostgresServer:🆕 document that query_handler and copy_in_handler
are the same frontend instance
Signed-off-by: Ning Sun <sunning@greptime.com>
* chore: remove unrelated files
* feat(postgres)!: drop the text codec; csv is the only COPY FROM STDIN format
Remove the hand-rolled PostgreSQL text-format codec (backslash
escapes, octal, \N nulls) that arrow-csv cannot express, keeping
arrow-csv-based csv as the only implemented format.
- COPY FROM STDIN without FORMAT now fails with guidance ('requires
an explicit format in this version; use WITH (FORMAT csv)') rather
than silently defaulting to the removed format
- FORMAT text/txt and FORMAT binary report 'not supported yet' via
registry stubs instead of unknown-format errors
- codec.rs shrinks to the trait framework: the text-representation
helpers (Field, convert_text_field, bytea hex, timestamp parsing)
are removed with the codec; the module docs note where to find
their shape should a future text-serializing format need them
- csv implements check_column_type directly (no shared default)
Breaking for clients relying on the PostgreSQL default text format;
explicit WITH (FORMAT csv) is required.
Signed-off-by: Ning Sun <sunning@greptime.com>
* perf(postgres): raise COPY FROM STDIN flush batch to 32768 rows and harden flush diagnostics
- COPY_IN_FLUSH_ROWS 8192 -> 32768: without the pending-rows batcher
each flush pays the full insert pipeline (catalog lookup, schema
check, partition split, region write round trip), so 4x larger
batches cut round trips 4x at linearly-bounded memory; the constant
also sizes the arrow decoder batch. Comment documents the trade-off
and points high-rate users at the pending-rows batcher, which turns
flushes into enqueues.
- rows buffer pre-allocates COPY_IN_FLUSH_ROWS.
- flush_copy_in warns on unexpected output variants instead of
silently counting 0 rows.
- batching wire test feeds 70000 rows to stay above the new threshold.
Verified against a real psql 18.6 client: \copy ... WITH (FORMAT csv)
sends FORMAT csv and succeeds; default \copy (text) gets the
'requires an explicit format' guidance; COPY TO STDOUT round-trips.
Signed-off-by: Ning Sun <sunning@greptime.com>
* fix(postgres): reset copy-in state on successful CopyDone
on_copy_done left the CopyInState in place (inert but holding buffers)
until the next COPY overwrote it; take it like on_copy_fail does, so
buffers are freed eagerly and a stray post-done CopyData gets the
protocol-violation error instead of a finished parser.
Signed-off-by: Ning Sun <sunning@greptime.com>
* chore: bypass typo check
* chore: format change
* fix(postgres): multi-chunk COPY FROM STDIN corrupted at chunk boundaries
The csv feed loop flushed the arrow-csv decoder at every CopyData
chunk end — including partial batches at record boundaries — but
arrow-csv's contract permits flush() only after decode() returned 0
(a full batch of complete records). Flushing outside that point
corrupts subsequent decoding: fields mis-parse shifted by the size
of the record prefix buffered so far ('000001,host-001,...' parses
as '00001h,ost-001u,...'), either erroring (22P02/22P04) or, worse,
silently writing wrong values. Which chunks trigger it depends on
where record boundaries land, so it appears intermittent with
random-length data (psql 8KB / pgx ~64KB chunking both hit it).
Adopt the strict decoder protocol: feed decodes until input is
exhausted and flushes only when decode returns 0 (batch full);
trailing partial batches surface at the EOF signal in finish().
With batch_size == COPY_IN_FLUSH_ROWS the flush threshold is
unchanged; sub-threshold streams now emit their rows at CopyDone
instead of per chunk, which is semantically identical.
Adds test_csv_chunked_feeds_are_alignment_safe: the same payload
fed at chunk sizes 1..65531 (plus shifted starts) with every row's
content verified — reproduces the corruption on the old loop at
chunk sizes 7, 8, 13, 64 and 1000.
Verified with the reporter's Go reproducer (pgx, content-checked):
10/10 clean at -chunk 8192, and 4096/1000/97/65531 plus the
previously failing benchmark sizes 500/1000/2000/200/800 all round-
trip exactly.
Signed-off-by: Ning Sun <sunning@greptime.com>
* chore: bypass spell check
* chore: bypass spell check
* fix(postgres): address PR review — cross-catalog guard, identifier folding, tab delimiter
Three review findings (shuiyisong, PR #9297):
[P1] Enforce the catalog restriction on COPY targets. copy_in_table now
applies validate_catalog_and_schema when QueryOptions::
disallow_cross_catalog_query is set, mirroring SQL INSERT's
check_permission path, BEFORE begin_copy_in switches the query context
to the target catalog (which previously made the restriction
unenforceable downstream). Cross-catalog COPY is now rejected with
'Illegal access to catalog' like INSERT.
[P2] Canonicalize unquoted identifiers like the SQL parser
(ParserContext::canonicalize_identifier): unquoted table-name parts and
columns fold to lowercase, quoted ones keep their case. COPY Metrics
(Host, Ts) now addresses the table CREATE TABLE Metrics created, and
no longer silently misroutes into a coexisting quoted "Metrics".
[P2] Allow a tab as CSV delimiter: take_single_byte accepts a literal
tab while still rejecting newlines and other control characters, and
the option parser now also accepts E'...' escape-string constants
(DELIMITER E'\t'), processing PostgreSQL's backslash escapes.
Verified end-to-end with psql 18.6: unquoted/quoted mixed-case round-
trip into the right tables, cross-catalog rejection asserted in a new
frontend unit test (QueryOptions plugin + second catalog), and both
DELIMITER '<tab>' and DELIMITER E'\t' import tab-separated data.
Signed-off-by: Ning Sun <sunning@greptime.com>
* fix(postgres): accept PG boolean spellings in COPY IN; drop copy state on failure
Two review findings (shuiyisong, PR #9297):
[P2] Boolean columns round-trip between the COPY directions. arrow-csv
only parses true/false while PostgreSQL COPY OUT emits t/f, so
boolean tables failed to re-import their own export (22P02). Boolean
columns now decode as Utf8 in the arrow schema and flush_batch
converts them with PostgreSQL's boolean input grammar (true/false,
t/f, yes/no, y/n, on/off, 1/0, case-insensitive, whitespace
trimmed). Round-trip covered end-to-end: COPY ... TO STDOUT's exact
bytes are fed back via COPY FROM STDIN and verified.
[P2] Drop the COPY state on every server-side failure. on_copy_data
and on_copy_done now take ownership of the state up front, so parse,
conversion or insert errors drop the buffered data instead of
retaining it on long-lived or pooled connections until the next COPY
or connection close (the reviewer measured ~17 MiB retained after a
failed 8 MiB row); on_copy_data success puts the state back. An
e2e test drives a failed COPY followed by a successful one on the
same connection.
Signed-off-by: Ning Sun <sunning@greptime.com>
* fix(postgres): interpret offset-free COPY timestamps in the session timezone
Review finding (shuiyisong, PR #9297): the arrow decode schema carried
Timestamp fields with no timezone, so the CSV decoder parsed
offset-free timestamp strings as UTC while SQL INSERT parses them in
the session timezone — the same input string silently stored a
different instant depending on the ingestion method (8h apart in
Asia/Shanghai).
CopyInCodec::create_parser now receives the session timezone and the
csv decode schema attaches it to Timestamp fields, so naive strings
parse like INSERT; explicit offsets in the input (Z, +08:00) still
take precedence, and UTC sessions are unchanged.
Covered by test_csv_timestamp_session_timezone (naive vs +08:00 vs Z)
and the end-to-end test_postgres_copy_in_session_timezone_parity,
which SETs TIMEZONE TO 'Asia/Shanghai', inserts and COPY-loads the
same string, and asserts both CAST(ts AS BIGINT) readbacks agree on
1704038400000.
Signed-off-by: Ning Sun <sunning@greptime.com>
* fix(postgres): don't unescape E-string COPY options twice
Review finding (shuiyisong, PR #9297): sqlparser's tokenizer already
decodes PostgreSQL escape-string constants before returning
Token::EscapedStringLiteral, so the second unescape_pg_string pass
altered the decoded value — DELIMITER E'\\t' (a literal backslash
followed by t, two characters) was turned into a tab and wrongly
accepted as a single-byte delimiter.
Use the EscapedStringLiteral payload as-is, like SingleQuotedString,
and delete unescape_pg_string. Regression coverage distinguishes the
three cases: E'\t' -> tab, E'\\' -> single backslash, and E'\\t'
-> rejected as a multi-character delimiter.
Signed-off-by: Ning Sun <sunning@greptime.com>
* fix(postgres): COPY OUT timestamps carry session-tz offset; quoted empty strings survive COPY IN
Two review findings (shuiyisong, PR #9297):
[P2] Preserve timestamp instants across COPY OUT/IN in non-UTC
sessions. COPY OUT rendered naive UTC text while COPY IN (since the
session-tz fix) interprets offset-free strings as local time, so
re-importing an export shifted instants by the zone offset. COPY OUT
now renders timestamps with an explicit session-timezone offset
(2024-01-01 08:00:00.000000+08): the copy-out arrow schema carries the
session tz (arrow-pg converts to a tz-aware DateTime) and the pg field
type becomes TIMESTAMPTZ so the offset-bearing text format applies.
Embedded offsets win on re-import regardless of session, keeping the
OUT -> IN round trip exact; UTC sessions are unchanged.
[P2] Preserve quoted empty strings. The shared arrow-csv decoder
nulls every empty field, so a quoted empty string ("" — an empty
STRING in PostgreSQL) imported as NULL. A shadow quote-mask scanner
runs over the same bytes with the decoder's own splitting rules and
records per field whether it was quoted; flush_batch restores quoted
empties to empty strings for string columns, while unquoted empty
fields stay NULL. Masks track records across chunk boundaries and
align with decoder batches.
Covered end-to-end: test_postgres_copy_out_in_timestamp_round_trip
(Asia/Shanghai export asserted to carry +08, exact-epoch re-import
comparison) and test_postgres_copy_in_quoted_empty_string (NULL,
quoted empty and non-empty in one import), plus unit tests including
a chunk-boundary sweep for the quoted-empty mask alignment.
Signed-off-by: Ning Sun <sunning@greptime.com>
* fix(postgres): keep COPY IN quote masks aligned after doubled quotes; allow quoted empties in NOT NULL columns
Two review findings (fengjiachun, PR #9297):
[P2] Doubled quotes desynchronized the shadow quote-mask scanner. In
QuoteInQuoted the doubled-quote branch skipped the second quote AND the
following byte, so a closing quote or delimiter directly after an
escaped quote was swallowed and later fields lost their masks --
import semantics depended on CopyData chunk boundaries. The branch now
consumes only the escaped quote and returns to the in-quoted state,
mirroring the decoder. Regression: "a""b","" ... verified across
chunk sizes 1-64.
[P2] A NOT NULL string column rejected quoted empty strings before
restoration could run: the decode schema preserved the target
nullability, and arrow-csv's batch validation rejects the temporary
NULL a quoted empty decodes to. String fields now decode as nullable
(safe: quoted-vs-unquoted is exactly the restoration's job), and the
target NOT NULL constraint is enforced in flush_batch after
restoration, so "" imports as an empty string while an unquoted empty
field fails with a not-null-constraint error. Regressions at unit and
e2e level: NOT NULL column accepting "" and rejecting unquoted NULL.
Signed-off-by: Ning Sun <sunning@greptime.com>
* fix(postgres): align COPY IN quote-mask scanner with the CSV decoder on blank lines and CR terminators
Two review findings (fengjiachun, PR #9297):
[P2] Blank lines emitted a spurious empty mask. csv-core skips blank
lines, but the scanner queued one mask per blank record, so every
subsequent decoded row consumed the wrong mask (a leading blank line
turned a quoted empty into NULL and an unquoted empty into an empty
string, and tripped NOT NULL on the wrong row). A blank line before the
header also shifted the header expectation. end_record now returns no
mask for a completely blank record without consuming the header
expectation.
[P2] Standalone CR was not a record terminator. The decoder accepts CR,
CRLF and LF alike; the scanner only recognized CR before LF or at EOF,
so CR-separated input desynchronized masks and flush_batch failed with
'quote masks out of sync'. is_record_end now treats CR alone as a
terminator, consuming one byte (two for CRLF), while a CR at a chunk
boundary still defers via carry so CRLF split across CopyData chunks
stays one terminator.
Regressions: blank lines before/inside records and before a header
(NULL/empty preservation plus NOT NULL behavior), and mixed
CR/CRLF/LF terminators across chunk sizes 1-64 with a trailing
CR-only record.
Signed-off-by: Ning Sun <sunning@greptime.com>
* refactor(postgres)!: drop the quote-mask scanner; COPY IN CSV behaves exactly like file COPY
The shadow quote-mask scanner — kept to preserve PostgreSQL's
quoted-empty-string vs unquoted-NULL distinction — mirrored the
decoder's record-splitting grammar by hand and required a fix for every
rule it drifted on (doubled quotes, blank lines, CR terminators, NOT
NULL interplay). Per maintainers' decision, that complexity is not
worth the divergence: COPY FROM STDIN now behaves EXACTLY like the
file-based COPY ... FROM 'file.csv' — an empty field, quoted or not,
imports as NULL (arrow-csv's engine semantics, identical in both
routes).
Removed: QuoteMaskScanner and helpers, the always-nullable string
decode schema, the restoration + NOT NULL enforcement in flush_batch,
set_string_at, and the scanner-era unit tests. The decode schema again
preserves target nullability, so the decoder enforces NOT NULL itself,
same as the file route.
Kept (small, schema-level, no second grammar): boolean spellings
(t/f) and session-timezone timestamp decoding, both addressing
separate review findings.
E2E: test_postgres_copy_in_empty_field_is_null pins the parity —
quoted empty, unquoted empty and non-empty in one import.
Signed-off-by: Ning Sun <sunning@greptime.com>
* chore: remove empty file
* chore: lint issues
---------
Signed-off-by: Ning Sun <sunning@greptime.com>
* perf(log-store): read object store WAL footers while the recovery listing goes on
Recovery listed the whole prefix before it read any footer, so its LIST
pages, about 80-120 ms each on S3 and one per 1000 objects, added to the
footer reads. The listing now runs in a task of its own and feeds the
footer reads as its pages arrive; a listing that fails part-way still
fails recovery, and a failed footer read stops the listing.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* perf(log-store): fail object store WAL recovery as soon as its listing fails
A listing that failed part-way was reported only after every object it had
already listed was read, which on S3 can be tens of thousands of reads. The
stream now settles a finished listing before it yields another object, so
its failure goes first. Tests cover a large listed backlog and a listing
task that panics; the counting and parking test object accesses forward
list_ahead.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* test(log-store): hold the listing until the first footer reads wait in the backlog test
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* test(log-store): check a failed listing settles before buffered objects with a finished task
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* perf(log-store): bound the objects the recovery listing holds ahead of the reads
The listing fed the footer reads through an unbounded channel, so a listing
much faster than the reads could hold most of the prefix in memory. It now
holds at most 4096 objects, four LIST pages of S3, and waits for the reads
while that many are queued.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* test(log-store): poll the recovery listing directly to show it waits on a full backlog
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* test(log-store): check the bound with the channel the recovery listing uses
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
---------
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* fix(mito2): keep SSTs when a manifest save ends with RateLimited
The object store's retry layer retries temporary errors and returns the
last attempt's error. A manifest delta PUT can be stored by the server
while the client sees a timeout, reset or 5xx, and the retries that
follow can be throttled, so the save ends with RateLimited even though
the delta exists. may_have_persisted_manifest_update treated RateLimited
as proof that nothing was written, so flush and compaction deleted the
SSTs the stored delta references. A later open or follower sync replays
the delta and reads fail on the missing files.
Treat RateLimited like other ambiguous errors: the SSTs are kept and
become orphans when the save really failed.
Fixes#9897
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* fix(mito2): keep SSTs when a manifest save ends with PermissionDenied
The retry layer returns the last attempt's error, so a 403 from a revoked
key, a changed bucket policy or clock skew can follow a delta PUT that
the server already stored. Like RateLimited, PermissionDenied does not
prove that nothing was written, so it no longer lets flush or compaction
delete the SSTs; they become orphans when the save really failed.
The compaction test that rejects a manifest write before persistence now
injects Unsupported, which is raised before any request is sent.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
---------
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
Co-authored-by: Yingwen <realevenyag@gmail.com>
feat: track per-node scheduling state and fence region placement
Add a cluster-wide, persistent "can this node receive new regions" flag and
make every metasrv placement path respect it. This is the fence half of
`ADMIN drain_node`; the drain procedure, gRPC entry and region evacuation land
separately.
State layout
- New namespace `__node_scheduling/{datanode,flownode}/{node_id}` holding
`{"schedulable": false}`. A missing key means schedulable, so undrain is a
delete and no stale `schedulable: true` records are left behind.
- `NodeSchedulingKey`/`NodeSchedulingValue` mirror the existing KV conventions
(`NodeInfoKey`'s `{role, node_id}` shape, `LeaseValue`'s `*Value` suffix).
The role is part of the prefix, so extending to flownodes needs no key change.
- `NodeSchedulingManager` writes with `put_if_not_exists`/`compare_and_put`, so
concurrent marks change the state exactly once, and reads a single prefix
range per placement decision (DDL/failover only, not per query).
Placement
- `NodeInfo` gains `#[serde(skip)] schedulable: Option<bool>` plus
`is_schedulable()`; `None` (unresolved) counts as schedulable, and the field
never reaches heartbeat bytes, so the wire/in-memory format is unchanged.
- `SchedulingAwarePeerDiscovery` stamps the flag on `active_datanodes`. It is
additive (never hides nodes) because failover still needs the full active set
to compute its exclusion list.
- All three built-in selectors now fetch candidates through
`selector::common::active_schedulable_datanodes`; `RegionSupervisor` adds
unschedulable datanodes to `exclude_peer_ids` for both selector variants.
Enforcing the single fence
- `MetaPeerClient` no longer implements `PeerDiscovery`; its discovery methods
are inherent, and `DatabaseOperator` (frontends only) takes the concrete
client. `SchedulingAwarePeerDiscovery`, built once in `MetasrvBuilder`, is
therefore the only way to obtain a `PeerDiscoveryRef`, so bypassing the fence
is a compile error rather than a silent misplacement.
Compatibility
- New key namespace only; `NodeInfo` serialization is byte-identical, so old
metasrv nodes and persisted heartbeats keep working. The flag itself is stored
in the durable KV, so a follower metasrv sees marks written by the leader.
Verification
- `cargo nextest run -p meta-srv node_scheduling` (13), `selector` (12),
`supervisor` (14), `-p common-meta cluster` (9), `-p meta-client` (36),
`-p standalone` (compiles, no tests).
- `cargo clippy -p meta-srv -p common-meta -p meta-client -p standalone
--all-targets -- -D warnings`, `cargo fmt --all -- --check`.
Signed-off-by: WenyXu <wenymedia@gmail.com>
The `datanodeStorage` field belongs directly under `monitoring.standalone`
(GreptimeDBStandaloneSpec), not under `monitoring.standalone.base`
(PodTemplateSpec). Nesting it under `base` renders an invalid
GreptimeDBCluster resource and the CRD schema rejects it during
server-side apply:
.spec.monitoring.standalone.base.datanodeStorage: field not declared in schema
* ci: install protoc with taiki-e/install-action
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* ci: run multi-language SDK tests on Python 3.13
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* ci: drop the unused libfuzzer-14-dev package from fuzz builds
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* ci: build fuzz targets with the bundled libFuzzer
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
---------
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* feat(config): support overriding region engine options from env
`region_engine` is a `Vec` of externally tagged enums, so plain dotted
environment keys cannot address it: `config` walks the path, replaces the
default sequence with a table and loading fails with `invalid type: map,
expected a sequence`.
Address the entries by tag name instead
(`<PREFIX>__REGION_ENGINE__<TAG>__<FIELD>`) and merge the overrides onto the
default entries in a dedicated source, so the engines and fields that are not
mentioned keep their configured value. An entry the default sequence does not
have is appended; a numeric index, an override of the whole sequence, a field
below a non-table default and two variables that set a path and a field below
it are reported with a dedicated error instead of a serde type error.
A configuration file that configures the sequence as a whole still replaces
the merged value, which the example configs document.
Closes#8620
Signed-off-by: NightG1owZ <316247938+NightG1owZ@users.noreply.github.com>
* docs(config): clarify that the metric engine is always enabled
`region_engine` defaults to [mito, file], but the datanode builds and
registers the metric engine unconditionally. Reword the example configs so
`mito` and `file` read as the default configuration entries and naming
`metric` only supplies its options, then regenerate config.md.
Signed-off-by: NightG1owZ <316247938+NightG1owZ@users.noreply.github.com>
* docs(config): drop the env-var explanation from the config reference
Environment variables are documented in the GreptimeTeam/docs repository, so the example configs and the generated config reference only describe the config file. Keep the note that the metric engine is always registered.
Signed-off-by: NightG1owZ <316247938+NightG1owZ@users.noreply.github.com>
* test(config): cover tagged sequence environment collection
Add unit tests for TaggedSeqEnvSource::collect: merging by tag, appending unknown tags, empty and null defaults, flat key removal, the error paths and the take_entries and is_index_like helpers. The variables are injected through Environment::source, so the tests do not touch the process environment.
Signed-off-by: NightG1owZ <316247938+NightG1owZ@users.noreply.github.com>
---------
Signed-off-by: NightG1owZ <316247938+NightG1owZ@users.noreply.github.com>
Co-authored-by: NightG1owZ <316247938+NightG1owZ@users.noreply.github.com>
* refactor(mito2): pick TWCS L1 merges by closed units
TWCS could keep rewriting L1 files without new writes. Overlap-only picks
could split their inputs into one more file than they read, and L1 file
reduction then rewrote up to 16 files to remove that single file. The
outputs, split by key, created new overlaps and the cycle repeated.
Replace the three global pick phases with one pick per window, scheduled by
class across windows. Every pick now decreases a termination measure when the
window receives no writes.
- Active windows only take merges that remove files. L1 files are grouped by
conflicts and merged in key order, so size-split tails are merged together.
A reduction rewrites at most 4 output files' worth of bytes per removed file.
- Inactive windows absorb L0 files, then take closed merges: no other file
intrudes on the inputs' bounding box, so outputs cannot add overlaps. Each
merge resolves conflicts or removes files, decreasing (conflicts, files).
Files that do not take part in an overlap are not rewritten.
- Overlaps that only share a boundary key no longer count as conflicts, so a
size-split sorted run does not look overlapping.
- Closed merges may take 4x the input limit to resolve a whole window once.
- Cache primary key ranges per pick instead of resolving them per comparison.
- Log the pick branch of each compaction output.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* docs(mito2): state the compactor assumptions behind TWCS termination
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* feat(mito2): resolve active window overlaps on manual compaction
A window that stops receiving writes keeps being the active window, whose
picks only remove files, so its overlaps were never resolved. Manual
compactions now pick every window as inactive, so a closed merge can resolve
them. Automatic compactions are unchanged.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(mito2): defer overlap resolution in the window before the active one
Late data keeps landing in the window right before the active window. A
closed merge that resolves its overlaps early rewrites the whole window,
then the late data brings the overlaps back, and the balanced run check
keeps them there. Overlap-only merges now skip that window until it is
older. File-reducing merges and manual compactions are unchanged.
On the 12h/1GB out-of-order bench this reaches overlap depth 1.0 instead
of 2.08 at the same write amplification (3.01 vs 3.02).
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* refactor(mito2): only check earlier-starting files when closing units
Closing units only checks files that start before a unit ends, and the
overlap checks compare key bounds without cloning them.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(mito2): keep settling windows from rewriting L1 files
The window before the active one still receives late data. Skipping only
its overlap-only merges was not enough: file-reducing merges of
conflicting files and mixed L0/L1 merges still rewrote its L1 files,
and a mixed merge could rewrite a nearly full output for each small late
file. A settling window now only merges L0 files among themselves and
conflict-free files; the window resolves its overlaps once it is older.
On the out-of-order bench, write amplification drops from 3.88 to 3.30
at 2GB/6h, 3.05 to 2.94 at 1GB/6h and 3.01 to 2.96 at 1GB/12h, with
overlap depth 1.0 in all three.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(mito2): resolve late overlaps once a window is quiet
Late data that lands in a window older than the settling one is merged
into a small file covering the whole key range. Resolving it rewrites
the window's dominant sorted run, which the balanced run check refuses,
so the overlap stayed. Removing the check instead rewrites the window
for every late flush.
A window is quiet once the region has written at least as many rows as
the window holds since its newest file. Overlap merges in a quiet window
may rewrite a dominant run, so each quiet period pays for at most one
such rewrite. Sequences are per row and kept by compaction, so this
needs no state.
On the bench with late data up to 2 hours, mean overlap depth drops
from 1.67 to 1.17 at 2GB/6h and from 2.33 to 1.83 at 1GB/6h, at write
amplification 3.70 to 4.25 and 3.26 to 3.74; all inactive windows of
2GB/6h realtime reach depth 1 (write amplification 2.80 to 3.30). The
remaining depth is in the final active window.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* test(mito2): check closed merges of quiet and settling windows terminate
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(mito2): split TWCS closures over the input limit
A file spanning the keys of a large sorted run joins every run file into one
closure. Once a closure held more files than one compaction accepts, closed
merges skipped it, so the window never merged those files again and late
files kept piling up.
Split such closures into single files, like active-window picks do, and
replace the closed check with a conflict bound: outputs only conflict with
files intruding on the inputs' box, so a merge is taken if `outputs · I`
does not exceed the conflicts its inputs had with other files. Intervals of
whole closures have no such conflicts and still need to be closed, so other
windows pick as before, and `(conflicts, files)` still strictly decreases.
Small run files and late files over a large run can be merged again. A late
file over a run that cannot shrink stays: resolving it would rewrite the
whole run.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(mito2): address TWCS picker review findings
- Manual compactions may rewrite a dominant sorted run to resolve overlaps.
The active window is never quiet, so manual compactions could not resolve
its late-data overlaps before.
- Single-key intervals predict one output: the writer never splits a series,
so large files of one series were never merged.
- The active window falls back to single files when no group of conflicting
files qualifies, so a large file no longer keeps its small peers unmerged.
- Compute the settling window distance with checked_sub, which overflowed for
windows at the timestamp limits.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(mito2): count inner keys as TWCS conflicts and check balance per group
- A single key strictly inside another file's key range now conflicts, so
a late file of one series inside a wide file is resolved like other
overlaps. Files sharing only a boundary key still do not conflict.
Intrusion covers the new pairs, which a randomized test checks.
- Check the dominant-run guard per group of conflicting files. Runs of
separate groups never overlap, so pairing one group's run with another
group's peer made an interval of two dominant runs look balanced.
- A window with unknown row counts, written before they were recorded, is
not quiet: its threshold would drop to a single row.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* perf(mito2): index TWCS files by key to avoid quadratic grouping
Grouping a window's files compared every file with all files sharing its
time range, and closing groups and checking intrusions scanned the whole
window. A sorted run split by key shares one time range, so a window of N
such files took O(N²) checks without any conflict. The 1024-file fallback
only covers the sorted-run path, not these groups.
Index the files by lower primary key in a segment tree that keeps each
subtree's largest upper key and its time bounds. Grouping, closing, and the
conflict bound only check the files whose key and time ranges meet, so a
file of a key-sliced run only meets its neighbors. Conflicts among split
files are precomputed per file and counted per interval with a binary
search instead of pairwise.
On a run of 20,000 key-sliced files with 5 late files spanning all keys,
a closed merge pick takes ~62 ms instead of ~3.9 s and an active pick
~19 ms instead of ~3.5 s.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* test(object-store): retry transient S3 transport failures
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(mito2): bound TWCS closed-merge conflict graph work
Budget the candidate files visited while building conflict units and split-unit adjacency. When the budget is exceeded, treat every file as a split unit and pick a bounded file reduction instead of materializing a quadratic graph.
The fallback candidate is admitted only through adds_no_conflicts, so outputs do not add conflict pairs and the (P, N) termination invariant still holds. Dense picks are logged with their own branch.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(mito2): charge TWCS index nodes to the conflict-graph budget
The conflict-graph work budget only counted files that reached a callback, so a traversal with poor pruning could still visit a quadratic number of internal nodes without triggering the dense fallback. Charge every ExtentIndex node, including pruned nodes, and stop the traversal as soon as the shared budget is exhausted.
Add a regression test where the root straddles the query time but neither leaf matches, so a callback-only budget would never fire.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(mito2): retry smaller TWCS dense compaction candidates
If a dense fallback candidate fails admission, try one single-output subinterval within it before giving up. Reuse the reduction guards and check both candidates against the full window's external conflicts, preserving bounded work and termination.\n\nCover the chain regression through both fallback unit constructions and reject one-output retries that introduce conflicts across time gaps.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(mito2): compact shared primary-key boundaries
Count shared primary-key endpoints as conflicts and apply the same overlap rule to bounding-box intrusions. Existing compaction writers keep series intact, so fresh output ranges remain disjoint.
Align simulated output splits with the writer and cover boundary-only cleanup in inactive and manual windows, plus stable real output files after a subsequent TWCS request.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(mito2): compact small L1 files in inactive windows
Allow bounded small-L1 reductions in settling and older inactive windows when normal picks cannot proceed. Preserve full-window conflict checks, large-file protection, and LastNonNull closure semantics.\n\nCover selection limits, termination, and real SST compaction with tombstones and reopen checks.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* docs(mito2): clarify TWCS unit grouping guarantees
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
---------
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
sqlx 0.8 pins the unmaintained webpki-roots 0.26 line for its rustls
trust store, which still contains the recently compromised e-Tugra CA
root. Upgrading sqlx to 0.9 switches its dependency to webpki-roots
1.x, and the lockfile now resolves to 1.0.10 where the distrusted
root has been removed from the compiled-in Mozilla trust store.
sqlx 0.9 requires dynamic SQL strings to be explicitly audited via
the new SqlSafeStr trait: only &'static str literals are accepted
as-is. All affected call sites (MySQL KV backend, MySQL election,
integration tests and fuzz targets) now wrap their internally
generated SQL templates in AssertSqlSafe; the wrapped SQL is built
from operator config or test-local identifiers only, and dynamic
data keeps flowing through bind parameters.
The mysql-rsa feature is enabled alongside mysql to preserve 0.8
authentication behavior: sqlx 0.9 gates non-TLS RSA password auth
(caching_sha2_password / sha256_password) behind it.
Signed-off-by: Ning Sun <sunning@greptime.com>
* fix(mito): return from manual index builds only after the index is published
A manual build_index could report success while the index was missing:
the request was rejected or coalesced behind another build, the manifest
commit failed, the indexer dropped an error, or the region was staging.
Manual builds now wait for active builds of the same SST, keep their
waiters across coalescing and schema-change retries, are notified after
the worker applies the edit, and verify the files against the manifest
before returning.
The consistency check also honors inverted_index.ignore_column_ids and a
zero segment row count, so such SSTs are no longer rebuilt on every call.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* test(mito): cover manual index build waiting for a schema-change retry
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix(mito): select manual index build files by the applied version
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* chore(mito): log why index builds are skipped or aborted
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* refactor(mito): split IndexBuildTask::index_build into phases
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix(mito): stop scheduling index builds when the region state changes
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* chore(mito): document region file id and silent aborts in index builds
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix(mito): log why a manual index build is incomplete
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix(mito): expose incomplete index builds and treat compacted files as done
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* test(mito): pin index build concurrency in the compaction test
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
---------
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* feat: use pgwire auth mechanisms API to simplify startuphandler
* chore: update pgwire
* chore: update pgwire revision to merged one
* refactor: model based on upstream api
* refactor: simplify implementation by introducing authsource
* refactor: update to latest pgwire apis
* refactor: adopt pgwire api changes
* chore: switch to released pgwire
* test: harden pg scram mixed-auth integration test and reject channel-binding clients
- Fix MixedAuthProvider to return a SCRAM verifier for the denied user,
so the test exercises rejection by authorization after successful
SCRAM verification; record the selected mechanism from the returned
variant and assert authorize ran exactly once per verified user.
- Assert the denied connection fails with 28P01 before any terminal
message, protecting the completion-ordering introduced by pgwire's
authentication gate.
- Add a raw-protocol test proving a client-first with a p= channel
binding flag is rejected (RFC 5802 section 6), while n-flag clients
(tokio_postgres) continue normally.
Bump pgwire to 0.42.1, which enforces RFC 5802 channel binding
negotiation and rejects short/overlong SCRAM proofs on the verifier
path (sunng87/pgwire#476).
Signed-off-by: Ning Sun <sunning@greptime.com>
---------
Signed-off-by: Ning Sun <sunning@greptime.com>
* fix(mysql): cap timestamp output at microsecond precision
Truncate MySQL text timestamp results to microseconds while preserving the existing second and millisecond formatting. Keep stored nanoseconds and explicit integer/string representations unchanged.
Fixes#9828
Signed-off-by: luofucong <luofc@foxmail.com>
* chore: ignore local output directory
Signed-off-by: luofucong <luofc@foxmail.com>
---------
Signed-off-by: luofucong <luofc@foxmail.com>
CREATE OR REPLACE FLOW deleted and then put the same flow route,
flownode flow and table flow keys in a single txn when the replacement
kept the partitions, flownodes or source tables, which etcd rejects
with "duplicate key given in txn request".
Only delete the old keys that are not put again. Closes#9810.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
dennis zhuang
shuiyisong
Weny Xu
discord9
jeremyhi
B. Naut
Yingwen
Dragon Roar
Dhruv VaishnavandWeny Xu
Ning Sun
LFC
liyang
NightG1owZandNightG1owZ
Lei, HUANG
fys
Chenglong Zheng