Kaifeng Zheng
a283e13da7
feat: set max log files to 720 by default, info log only ( #4787 )
...
* feat: set max log files to 720 by default, info log only
* expose max_log_files in tomls
* include dir info when panicing, limit max_log_files of err_log to 30, and that of slow_queries to opt.max_log_files
* fix clippy
* update config.md
* update expected config str
* limit err_log max files size to `max_log_files` too, include err info when panicing, put `max_l_f` in right position
* fix typos
* chore: config
Co-authored-by: Lei, HUANG <6406592+v0y4g3r@users.noreply.github.com >
---------
Co-authored-by: dennis zhuang <killme2008@gmail.com >
Co-authored-by: Lei, HUANG <6406592+v0y4g3r@users.noreply.github.com >
2024-10-04 18:05:40 +00:00
JohnsonLee
47a3277d12
feat: customize channel information for sqlness tests ( #4729 )
...
* feat: add pg and mysql server_address options
* feat: start pg and mysql server(standalone)
* feat: start pg and mysql in distribute
* feat: finally get there, specify postgres sqlness
* feat: support mysql sqlness
* fix: license
* fix: remove unused import
* fix: toml
* fix: clippy
* refactor: BeginProtocolInterceptorFactory to ProtocolInterceptorFactory
* fix: sqlness pg connect
* fix: clippy
* Apply suggestions from code review
Co-authored-by: Yingwen <realevenyag@gmail.com >
* fix: rustfmt
* fix: reconnect pg and mysql when restart
* test: add mysql related sqlness
* fix: wait for start while restarting
* fix: clippy
* fix: cargo lock conflict
fix: Cargo.lock conflict
* fix: usage of '@@tx_isolation' in sqlness
* fix: typos
* feat: retry with backoff when create client
* fix: use millisecond rather than microseconds in backoff
---------
Co-authored-by: Yingwen <realevenyag@gmail.com >
2024-10-04 09:26:57 +00:00
Yingwen
caf5f2c7a5
docs: add TM to logos ( #4789 )
2024-10-01 03:35:04 +00:00
Weny Xu
c1e8084af6
feat: add add_handler_after, add_handler_before, replace_handler ( #4788 )
...
* feat: add `add_handler_after`, `add_handler_before`, `replace_handler`
* chore: apply suggestions from CR
* test: add more tests
* feat: use `Vec` instead of `LinkedList`
* Update src/meta-srv/src/lib.rs
Co-authored-by: Yingwen <realevenyag@gmail.com >
---------
Co-authored-by: Yingwen <realevenyag@gmail.com >
2024-09-30 08:53:59 +00:00
Weny Xu
6e776d5f98
feat: support to reject write after flushing ( #4759 )
...
* refactor: use `RegionRoleState` instead of `RegionState`
* feat: introducing `RegionLeaderState::Downgrading`
* refactor: introduce `set_region_role_state_gracefully`
* refactor: use `set_region_role` instead of `set_writable`
* feat: support to reject write after flushing
* fix: fix unit tests
* test: add unit test for `should_reject_write`
* chore: add comments
* chore: refine comments
* fix: fix unit test
* test: enable fuzz tests for Local WAL
* chore: add logs
* chore: rename `RegionStatus` to `RegionState`
* feat: introduce `DowngradingLeader`
* chore: rename
* refactor: refactor `set_role_state` tests
* test: ensure downgrading region will reject write
* chore: enhance logs
* chore: refine name
* chore: refine comment
* test: add tests for `set_role_role_state`
* fix: fix unit tests
* chore: apply suggestions from CR
* chore: apply suggestions from CR
2024-09-30 08:28:51 +00:00
zyy17
e39a9e6feb
feat: add StatementStatistics for slow query logging implementation ( #4719 )
...
* feat: log slow query
* feat: log slow query for sql
* refactor: add slow query logging options
* ci: fix errors
* feat: add StatementStatistics
* chore: revert modification of servers crate
* docs: update config docs
* fix: clippy errors
2024-09-30 03:26:50 +00:00
Weny Xu
77af4fd981
refactor: introduce HeartbeatHandlerGroupBuilder ( #4785 )
2024-09-30 02:56:53 +00:00
Yingwen
cd55202136
feat: unordered scanner scans data by time ranges ( #4757 )
...
* feat: define range meta
* feat: group ranges
* feat: split range
* feat: build ranges from the scan input
* feat: get partition range from range meta
* feat: build file range
* feat: unordered scan read by ranges
* feat: wip for mem ranges
* feat: build ranges
* feat: remove unused codes
* chore: update comments
* feat: update metrics
* chore: address review comments
* chore: debug assertion
2024-09-29 07:14:48 +00:00
Lei, HUANG
50cb59587d
chore: replace anymap with anymap2 ( #4781 )
2024-09-29 06:27:35 +00:00
Lei, HUANG
0a82b12d08
fix(sqlness): sqlness isolation ( #4780 )
...
* fix: isolate logs
* fix: copy cases
* fix: clippy
2024-09-29 05:54:01 +00:00
LFC
d9f2f0ccf0
feat: add a new status code for "external" errors ( #4775 )
...
* feat: add a new status code for "external" errors
* Update src/auth/src/error.rs
Co-authored-by: shuiyisong <113876041+shuiyisong@users.noreply.github.com >
* support mysql cli cleartext auth
* resolve PR comments
---------
Co-authored-by: shuiyisong <113876041+shuiyisong@users.noreply.github.com >
2024-09-29 03:38:50 +00:00
Ning Sun
cedbbcf2b8
fix: update pgwire for potential issue with connection establish ( #4783 )
2024-09-28 19:27:31 +00:00
Ning Sun
d6be44bc7f
fix: dead loop on detecting postgres ssl handshake ( #4778 )
2024-09-28 01:39:29 +00:00
JohnsonLee
3a46c1b235
fix: use information_schema returns Unknown database ( #4774 )
...
* fix: use information_schema returns Unknown database 'information_schema'
* test: make sure 'use information_schma' successful
2024-09-27 23:15:48 +00:00
Lei, HUANG
934bc13967
feat(mito): limit compaction output file size ( #4754 )
...
* Commit Message
Clarify documentation for CompactionOutput struct
Updated the documentation for the `CompactionOutput` struct to specify that the output time range is only relevant for windowed compaction.
* Add max_output_file_size to TwcsPicker and TwcsOptions
- Introduced `max_output_file_size` to `TwcsPicker` struct and its logic to enforce output file size limits during compaction.
- Updated `TwcsOptions` to include `max_output_file_size` and adjusted related tests.
- Modified `new_picker` function to initialize `TwcsPicker` with the new `max_output_file_size` field.
* feat/limit-compaction-output-size:
Refactor compaction picker and TWCS to support append mode and improve options handling
- Update compaction picker to accept a reference to options and append mode flag
- Modify TWCS picker logic to consider append mode when filtering deleted rows
- Remove VersionControl usage in compactor and simplify return type
- Adjust enforce_max_output_size logic in TWCS picker to handle max output file size
- Add append mode flag to TwcsPicker struct
- Fix incorrect condition in TWCS picker for enforcing max output size
- Update region options tests to reflect new max output file size format (1GB and 7MB)
- Simplify InvalidTableOptionSnafu error handling in create_parser
- Add `compaction.twcs.max_output_file_size` to mito engine option keys
* resolve some comments
2024-09-27 11:17:36 +00:00
Weny Xu
4045298cb2
feat: add region_statistics table ( #4771 )
...
* refactor: introduce `region_statistic`
* refactor: move DatanodeStat related structs to common_meta
* chore: add comments
* feat: implement `list_region_stats` for `ClusterInfo` trait
* feat: add `region_statistics` table
* feat: add table_id and region_number fields
* chore: rename unused snafu
* chore: udpate sqlness results
* chore: avoid to print source in error msg
* chore: move `procedure_info` under `greptime` catalog
* chore: apply suggestions from CR
* Update src/common/meta/src/datanode.rs
Co-authored-by: jeremyhi <jiachun_feng@proton.me >
---------
Co-authored-by: jeremyhi <jiachun_feng@proton.me >
2024-09-27 09:54:52 +00:00
Lanqing Yang
cc4106cbd2
feat: protect datanode with concurrency limit. ( #4699 )
...
Adding parallelism in region server to protect datanode from query overload.
2024-09-27 06:12:57 +00:00
localhost
627a326273
refactor: Change the error type in the pipeline crate from String to Error ( #4763 )
...
* chore: in process
* chore: change pipeline crate error type
* chore: improve event error
* chore: fix by pr comment
* chore: use snafu context replace ok_or_else
* refactor: update snafu usage
---------
Co-authored-by: Ning Sun <sunng@protonmail.com >
2024-09-25 19:32:34 +00:00
discord9
0274e752ae
chore: make sure aws-lc-sys wouldn't be built ( #4767 )
...
* chore: make sure aws-lc-sys wouldn't be built
* fix: bash
* refactor: multiple line bash
2024-09-25 17:11:37 +00:00
Lei, HUANG
cd4bf239d0
chore: relax table name constraint ( #4766 )
...
chore/relax-table-name-constraint: Updated NAME_PATTERN to allow '@' and '#' characters and adjusted tests for new table name validation rules.
2024-09-25 02:45:18 +00:00
Ning Sun
e3c0b5482f
feat: returning warning instead of error on unsupported SET statement ( #4761 )
...
* feat: add capability to send warning to pgclient
* fix: refactor query context to carry query scope data
* feat: return a warning for unsupported postgres statement
2024-09-24 08:45:55 +00:00
Weny Xu
d1b252736d
refactor: unify the styling in create_or_alter_tables_on_demand ( #4756 )
...
* refactor: refactor `create_or_alter_tables_on_demand`
* chore: apply suggestions from CR
2024-09-24 03:48:16 +00:00
discord9
54f6e13d13
fix: Release CI & make rustls use ring ( #4750 )
...
* feat: new dev-build
* fix: copy
* chore: rm dbg run
* fix: binstall install script
* chore: typos
* fix: update useable image
* fix: properly uninstall gcc-9
* chore: another try
* chore: print current cc version
* chore: another try to release
* fix: use gcc-10 by `update-alternatives`
* chore: update dev-build image
* remove gcc-9 again and install make/cmake
* use new image
* alias gcc-10/g++-10 to every variant
* again.....
* fix....
* again release ....
* chore: remove auto remove
* feat: rustls default to ring
* chore: update Cargo.lock
* chore: update Cargo.lock
* Apply suggestions from code review
---------
Co-authored-by: Ruihang Xia <waynestxia@gmail.com >
2024-09-24 03:11:12 +00:00
Ruihang Xia
5c64f0ce09
refactor!: simplify NativeType trait and remove percentile UDAF ( #4758 )
...
* refactor!: simplify NativeType trait and remove percentile UDAF
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* remove NativeType
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* recover a mis-deleted case
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
---------
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
2024-09-23 10:55:20 +00:00
Ruihang Xia
2feddca1cb
feat: include order by to commutativity rule set ( #4753 )
...
* feat: include order by to commutativity rule set
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* tune sqlness replace interceptor
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
---------
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
2024-09-23 08:35:06 +00:00
Ning Sun
0f99218386
feat: list/array/timezone support for postgres output ( #4727 )
...
* feat: list/array support for postgres output
* fix: implement time zone support for postgrsql
* feat: add a geohash function that returns array
* fix: typo
* fix: lint warnings
* test: add sqlness test
* refactor: check resolution range before convert value
* fix: test result for sqlness
* feat: upgrade pgwire apis
2024-09-22 02:39:38 +00:00
Weny Xu
163cea81c2
feat: migrate local WAL regions ( #4715 )
...
* feat: allow to flush region before migrating
* fix: fix unit tests
* feat: allow to set `flush_timeout`
* feat: skip to replay memtable
* fix: fix unit tests
* test: add more tests
* refactor: simplify timeout logical
* test: add unit tests
* test: add unit tests
* chore: update comments
* fix: fix unit tests
* fix: fmt and clippy
* feat: change default timeout to 30s
* fix: throw `ExceededDeadline` error
* test: add tests for `downgrade_region_with_retry`
* chore: apply suggestions from CR
* chore: apply suggestions from CR
* chore: apply suggestions from CR
* chore: update proto to `3633474`
* refactor: refactor `upgrade_region_with_retry`
* chore: apply suggestions from CR
2024-09-20 08:27:20 +00:00
taobo
0c9b8eb0d2
feat: improve observability for procedure ( #4675 )
...
* feat: improve observability for procedure
* fix: test error
* test: add sqlness test for information_schema.procedure_info
* fix: sqlness test error
* fix: cr comment
* chore: update proto version
* fix: apply cr comment
* update version
* fix: cr comment
* optimize procedure type output format
* upgrade dep version
* fix: clippy error
* fix: `procedure` borrowed error
* fix: optimize code
2024-09-20 06:07:53 +00:00
Ning Sun
75c6fad1a3
feat: add more h3 scalar functions ( #4707 )
...
* feat: add more h3 scalar functions
* chore: comment up
2024-09-20 04:19:50 +00:00
Yingwen
e12ffbeb2f
feat: flush other workers if still need flush ( #4746 )
2024-09-20 02:55:31 +00:00
discord9
c4e52ebf91
feat: use new image for gcc-10 ( #4748 )
...
feat: use new image
2024-09-20 02:34:45 +00:00
Yingwen
f02410c39b
fix: disable field pruning in last non null mode ( #4740 )
...
* fix: don't prune fields in last non null mode
* test: add sqlness test for field pruning
* test: add flush
* refine implementation
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
---------
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
Co-authored-by: Ruihang Xia <waynestxia@gmail.com >
2024-09-20 00:35:37 +00:00
Ruihang Xia
f5cf25b0db
refactor: remove DfPlan wrapper ( #4733 )
...
* refactor: remove DfPlan wrapper
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* clean up
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* remove unused errors
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* fix test assertion
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
---------
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
2024-09-19 12:29:33 +00:00
liyang
1acda74c26
fix: cannot input tag for the dev-builder image ( #4743 )
2024-09-19 11:14:34 +00:00
Yohan Wal
95787825f1
build(deps): use original jsonb repo ( #4742 )
2024-09-19 09:44:44 +00:00
Weny Xu
49004391d3
chore(fuzz): print table name for debugging ( #4738 )
...
* chore(fuzz): print table name for debugging
* chore: apply suggestions
2024-09-19 09:40:10 +00:00
discord9
d0f5b2ad7d
fix: use gcc-10 in release dev build ( #4741 )
2024-09-19 09:34:06 +00:00
Yohan Wal
0295f8dbea
docs: json datatype rfc ( #4515 )
...
* docs: json datatype rfc
* docs: turn to a jsonb proposal
* chore: fix typo
* feat: add store and query process
* fix: typo
* fix: use query nodes instead of query plans
* feat: a detailed overview of query
* fix: grammar
* fix: use independent cast function
* fix: unify cast function
* fix: refine, make statements clear
* docs: update rfc according to impl
* docs: refine
* docs: fix wrong arrows
* docs: refine
* docs: fix some errors qaq
2024-09-19 05:49:10 +00:00
Ning Sun
8786624515
feat: improve support for postgres extended protocol ( #4721 )
...
* feat: improve support for postgres extended protocol
* fix: lint fix
* fix: test code
* fix: adopt upstream
* refactor: remove dup code
* refactor: avoid copy on error message
2024-09-19 05:30:56 +00:00
shuiyisong
52d627e37d
chore: add log ingest interceptor ( #4734 )
...
* chore: add log ingest interceptor
* chore: rename
* chore: update interceptor signature
2024-09-19 05:14:47 +00:00
Lei, HUANG
b5f7138d33
refactor(tables): improve tables performance ( #4737 )
...
* chore: cherrypick 52e8eebb2dbbbe81179583c05094004a5eedd7fd
* refactor/tables: Change variable from immutable to mutable in KvBackendCatalogManager's method
* refactor/tables: Replace unbounded channel with bounded and use semaphore for concurrency control in KvBackendCatalogManager
* refactor/tables: Add common-runtime dependency and update KvBackendCatalogManager to use common_runtime::spawn_global
* refactor/tables: Await on sending error through channel in KvBackendCatalogManager
2024-09-19 04:44:02 +00:00
Ning Sun
08bd40333c
feat: add an option to turn on compression for arrow output ( #4730 )
...
* feat: add an option to turn on compression for arrow output
* fix: typo
2024-09-19 04:38:41 +00:00
discord9
d1e0602c76
fix: opensrv Use After Free update ( #4732 )
...
* chore: version skew
* fix: even more version skew
* feat: use `ring` instead of `aws-lc` for remove nasm assembler on windows
* feat: use `ring` for pgwire
* feat: change to use `aws-lc-sys` on windows instead
* feat: change back to use `ring`
* chore: provide CryptoProvider
* feat: use upstream repo
* feat: install ring crypto lib in main
* chore: use same fn to install in tests
* feat: make pgwire use `ring`
2024-09-19 04:12:13 +00:00
Weny Xu
befb6d85f0
fix: determine region role by using is_readonly ( #4725 )
...
fix: correct `is_writable` behavior
2024-09-18 22:17:39 +00:00
Yohan Wal
f73fb82133
feat: add respective json_is UDFs for JSON type ( #4726 )
...
* feat: add respective json_is UDFs
* refactor: rename to_json to parse_json
* chore: happy clippy
* chore: some rename
* fix: small fixes
2024-09-18 11:07:30 +00:00
shuiyisong
50b3bb4c0d
fix: sort cargo toml ( #4735 )
2024-09-18 09:19:05 +00:00
zyy17
0847ff36ce
fix: config test failed and use similar_asserts::assert_eq to replace assert_eq for long string compare ( #4731 )
...
* fix: config test failed and use 'similar_asserts::assert_eq' to replace 'assert_eq' for long string compare
* Update Cargo.toml
Co-authored-by: Yingwen <realevenyag@gmail.com >
* Update src/cmd/tests/load_config_test.rs
Co-authored-by: Yingwen <realevenyag@gmail.com >
---------
Co-authored-by: Ruihang Xia <waynestxia@gmail.com >
Co-authored-by: Yingwen <realevenyag@gmail.com >
2024-09-18 07:41:25 +00:00
shuiyisong
c014e875f3
chore: add auto-decompression layer for otlp http request ( #4723 )
...
* chore: add auto-decompression for http request
* test: otlp
2024-09-18 04:32:00 +00:00
Zhenchi
3b5b906543
feat(index): add explicit adapter between RangeReader and AsyncRead ( #4724 )
...
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
2024-09-18 03:33:55 +00:00
Weny Xu
d1dfffcdaf
chore: enable fuzz test for append table ( #4702 )
...
* chore: enable fuzz test for append table
* fix: fix mysql translator
2024-09-18 03:01:30 +00:00
