Ruihang Xia
8bea853954
refactor(mito2): implement RegionManifestManager ( #1984 )
...
* finilise manager and related API
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* impl manifest initialize and update
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* more test and utils
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* resolve CR comments
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
---------
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
2023-07-18 14:03:35 +00:00
Lei, HUANG
97cfa3d6c9
feat: support append entries from multiple regions at a time ( #1959 )
...
* feat: support append entries from multiple regions at a time
* chore: add some tests
* fix: false postive mutable_key warning
* fix: append_batch api
* fix: remove unused clippy allows
2023-07-14 09:57:17 +00:00
Ben Baoyi
9ff7670adf
refactor:remove common_error::prelude ( #1946 )
...
* feat:Remove common_error::prelude
* fix merge error
* cr comment
* fix error
2023-07-13 10:36:36 +08:00
Lei, HUANG
a7ea3bbc16
feat: manual compact api ( #1912 )
...
* merge develop
* chore: merge develop
* fix: some cr commentx
* fix: cr comments
2023-07-11 04:00:39 +00:00
Yingwen
f731193ddc
refactor: Define RegionId as a new type ( #1903 )
...
* refactor: Define RegionId as a new type
* chore: use into
* feat: custom debug print for region id
fix: test_show_create_table
2023-07-07 21:26:03 +08:00
Lei, HUANG
3b6f70cde3
feat: initial twcs impl ( #1851 )
...
* feat: initial twcs impl
* chore: rename SimplePicker to LeveledPicker
* rename some structs
* Remove Compaction strategy
* make compaction picker a trait object
* make compaction picker configurable for every region
* chore: add some test for ttl
* add some tests
* fix: some style issues in cr
* feat: enable twcs when creating tables
* feat: allow config time window when creating tables
* fix: some cr comments
2023-07-04 16:42:27 +08:00
Vanish
563f6e05e2
feat: remove all the manifests in drop_region. ( #1834 )
...
* feat: drop_region delete manifest file
* chore: remove redundant code
* chore: fmt
* chore: clippy
* chore: clippy
* feat: support delete_all in manifest.
* chore:CR
* test: test_drop_basic, test_drop_reopen
* chore: cr
* fix: typo
* chore: cr
2023-06-30 17:42:11 +08:00
LFC
fcff66e039
chore: deny unused results ( #1825 )
...
* chore: deny unused results
* rebase
2023-06-27 15:33:53 +08:00
Lei, HUANG
7961de25ad
feat: persist compaction time window ( #1757 )
...
* feat: persist compaction time window
* refactor: remove useless compaction window fields
* chore: revert some useless change
* fix: some CR comments
* fix: comment out unstable sqlness test
* revert commented sqlness
2023-06-13 10:15:42 +08:00
Ruihang Xia
2615718999
feat: merge scan for distributed execution ( #1660 )
...
* generate exec plan
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* move DatanodeClients to client crate
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* wip MergeScanExec::to_stream
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* fix compile errors
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* fix default catalog
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* fix expand order of new stage
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* move sqlness cases contains plan out of common dir
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* refactor information schema to allow duplicated scan call
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* fix: ignore two cases due to substrait
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* reorganise sqlness common cases
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* fix typos
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* redact round robin partition number
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* Apply suggestions from code review
Co-authored-by: Yingwen <realevenyag@gmail.com >
* skip tranforming projection
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* update sqlness result
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* revert common/order
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* fix clippy
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* Update src/query/src/dist_plan/merge_scan.rs
Co-authored-by: JeremyHi <jiachun_feng@proton.me >
* update sqlness result
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* update sqlness result again
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* resolve CR comments
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* ignore region failover IT
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* update sqlness result again and again
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* unignore some tests about projection
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* enable failover tests
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
---------
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
Co-authored-by: Yingwen <realevenyag@gmail.com >
Co-authored-by: JeremyHi <jiachun_feng@proton.me >
2023-06-02 06:42:54 +00:00
Ruihang Xia
e5b6f8654a
feat: optimizer rule to pass expected output ordering hint ( #1675 )
...
* move type convertsion rule into optimizer dir
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* implement order_hint rule
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* add test
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* fix clippy
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* it works!
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* fix clippy
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* use column name instead
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* fix clippy
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* accomplish test case
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* update lock file
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
---------
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
2023-06-02 03:43:51 +00:00
Weny Xu
0460f3ae30
test: add write test for region failover ( #1673 )
...
* test: add write test for region failover
* test: add written data assertion after failover
* test: support more storage types
2023-05-31 15:42:00 +08:00
Ruihang Xia
b27c569ae0
refactor: add scan_to_stream() to Table trait to postpone the stream generation ( #1639 )
...
* add scan_to_stream to Table
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* impl parquet stream
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* reorganise adapters
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* implement scan_to_stream for mito table
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* clean up
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* add location info
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* fix: table scan
* UT pass
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* impl project record batch
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* fix information schema
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* fix clippy
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* resolve CR comments
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* remove one todo
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* fix errors generated by merge commit
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* add output_ordering method to record batch stream
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* fix rustfmt
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* enhance error types
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
---------
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
Co-authored-by: Lei, HUANG <mrsatangel@gmail.com >
2023-05-29 20:03:47 +08:00
Weny Xu
9f67ad8bce
fix: fix doesn't release closed regions issue ( #1596 )
...
* fix: fix close region issue
* chore: apply suggestion from CR
* chore: apply suggestion from CR
* chore: apply suggestion from CR
* chore: apply suggestion from CR
* refactor: remove close method from Region trait
* chore: remove PartialEq from CloseTableResult
2023-05-23 11:40:12 +08:00
Yingwen
6dbceb1ad5
feat: Trigger flush based on global write buffer size ( #1585 )
...
* feat(storage): Add AllocTracker
* feat(storage): flush request wip
* feat(storage): support global write buffer size
* fix(storage): Test and fix size based strategy
* test(storage): Test AllocTracker
* test(storage): Test pick_by_write_buffer_full
* docs: Add flush config example
* test(storage): Test schedule_engine_flush
* feat(storage): Add metrics for write buffer size
* chore(flush): Add log when triggering flush by global buffer
* chore(storage): track allocation in update_stats
2023-05-22 19:00:30 +08:00
Vanish
8764ce7845
feat: add delete WAL in drop_region ( #1577 )
...
* feat: add delete WAL in drop_region
* chore: fix typo err.
* feat: mark all SSTs deleted and remove the region from StorageEngine's region map.
* test: add test_drop_region for StorageEngine.
* chore: make clippy happy
* fix: fix conflict
* chore: CR.
* chore: CR
* chore: fix clippy
* fix: temp file life time
2023-05-18 18:02:34 +08:00
Yingwen
8d54d40b21
feat: Add FlushPicker to flush regions periodically ( #1559 )
...
* feat: Add FlushPicker
* feat(storage): Add close to StorageEngine
* style(storage): fix clippy
* feat(storage): Close regions in StorageEngine::close
* chore(storage): Clear requests on scheduler stop
* test(storage): Test flush picker
* feat(storage): Add metrics for auto flush
* feat(storage): Add flush reason and record it in metrics
* feat: Expose flush config
docs(config): Update config example
* refactor(storage): Run auto flush task in FlushScheduler
* refactor(storage): Add FlushItem trait to make FlushPicker easy to test
2023-05-15 17:29:28 +08:00
Lei, HUANG
cfcfc72681
refactor: remove version column ( #1576 )
2023-05-15 11:03:37 +08:00
discord9
2c82ded975
feat: table metrics ( #1469 )
...
* feat: Statistic
* add todo
* fmt: cargo fmt
* feat: some simple impl for MemTable
* chore: a try on adding statistics
* Update src/table/src/stats.rs
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
* docs: fix typo
* newlines unnecessary
---------
Co-authored-by: Ruihang Xia <waynestxia@gmail.com >
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2023-05-06 14:59:49 +08:00
Ruihang Xia
d4f3f617e4
chore(toolchain): update rust-toolchain to 2023-05-03 ( #1524 )
...
* chore(toolchain): update rust-toolchain to 2023-05-03
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* update workflow yaml
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
---------
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
2023-05-06 11:34:09 +08:00
Yingwen
479ef9d379
fix: checkpoint GC task also deletes the file with the last version ( #1491 )
...
* test(storage): use assert_eq to check scan result
* feat(storage): Add more info to manifest log
* feat: Avoid error log when unable to delete
* fix: The manifest gc task should delete files <= last_version
* feat(storage): Don't log if the error kind is not found
* feat: Add keep_last_checkpoint option
2023-05-04 14:18:38 +08:00
dennis zhuang
2a9f482bc7
feat: show create table ( #1336 )
...
* temp commit
* feat: impl Display for CreateTable statement
* feat: impl show create table for standalone
* fix: forgot show.rs
* feat: clean code
* fix: typo
* feat: impl show create table for distributed
* test: add show create table sqlness test
* fix: typo
* fix: sqlness tests
* feat: render partition rules for distributed table
* Update src/sql/src/statements.rs
Co-authored-by: Yingwen <realevenyag@gmail.com >
* Update src/sql/src/statements.rs
Co-authored-by: Yingwen <realevenyag@gmail.com >
* Update src/sql/src/statements.rs
Co-authored-by: Yingwen <realevenyag@gmail.com >
* Update src/sql/src/statements/create.rs
Co-authored-by: Yingwen <realevenyag@gmail.com >
* chore: by CR comments
* fix: compile error
* fix: missing column comments and extra table options
* test: add show create table test
* test: add show create table test
* chore: timestamp precision
* fix: test
---------
Co-authored-by: Yingwen <realevenyag@gmail.com >
2023-04-21 11:37:16 +08:00
Ruihang Xia
da66138e80
refactor(error): remove backtrace, and introduce call-site location for debugging ( #1329 )
...
* wip: global replace
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* fix compile
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* fix warnings
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* remove unneeded tests of errors
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* fix ErrorExt trait implementator
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* fix warnings
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* fix format
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* fix pyo3 tests
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
---------
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
2023-04-06 04:06:00 +00:00
Eugene Tolbakov
59f7630000
feat: initial changes for compaction_time_window field support ( #1083 )
...
* feat(compaction_time_window): initial changes for compaction_time_window field support
* feat(compaction_time_window): move PickerContext creation
* feat(compaction_time_window): update region descriptor, fix formatting
* feat(compaction_time_window): add minor enhancements
* feat(compaction_time_window): fix failing test
* feat(compaction_time_window): return an error instead silently skip for the user provided compaction_time_window
* feat(compaction_time_window): add TODO reminder
2023-04-06 10:32:41 +08:00
dennis zhuang
563adbabe9
feat!: improve region manifest service ( #1268 )
...
* feat: try to use batch delete in ManifestLogStorage
* feat: clean temp dir when startup with file backend
* refactor: export region manifest checkpoint actions magin and refactor storage options
* feat: purge unused manifest and checkpoint files by repeat gc task
* chore: debug deleted logs
* feat: adds RepeatedTask and refactor all gc tasks
* chore: clean code
* feat: export gc_duration to manifest config
* test: assert gc works
* fix: typo
* Update src/common/runtime/src/error.rs
Co-authored-by: LFC <bayinamine@gmail.com >
* Update src/common/runtime/src/repeated_task.rs
Co-authored-by: LFC <bayinamine@gmail.com >
* Update src/common/runtime/src/repeated_task.rs
Co-authored-by: LFC <bayinamine@gmail.com >
* fix: format
* Update src/common/runtime/src/repeated_task.rs
Co-authored-by: Yingwen <realevenyag@gmail.com >
* chore: by CR comments
* chore: by CR comments
* fix: serde default for StorageConfig
* chore: remove compaction config in StandaloneOptions
---------
Co-authored-by: LFC <bayinamine@gmail.com >
Co-authored-by: Yingwen <realevenyag@gmail.com >
2023-03-31 10:42:00 +08:00
dennis zhuang
4f15b26b28
feat: region manifest checkpoint ( #1202 )
...
* chore: adds log when manifest protocol is changed
* chore: refactor region manifest
* temp commit
* feat: impl region manifest checkpoint
* feat: recover region version from manifest snapshot
* test: adds region snapshot test
* test: region manifest checkpoint
* test: alter region with manifest checkpoint
* fix: revert storage api
* feat: delete old snapshot
* refactor: manifest log storage
* Update src/storage/src/version.rs
Co-authored-by: Ruihang Xia <waynestxia@gmail.com >
* Update src/storage/src/manifest/checkpoint.rs
Co-authored-by: Ruihang Xia <waynestxia@gmail.com >
* Update src/storage/src/manifest/region.rs
Co-authored-by: Ruihang Xia <waynestxia@gmail.com >
* Update src/storage/src/manifest/region.rs
Co-authored-by: Ruihang Xia <waynestxia@gmail.com >
* chore: by CR comments
* refactor: by CR comments
* fix: typo
* chore: tweak start_version
---------
Co-authored-by: Ruihang Xia <waynestxia@gmail.com >
2023-03-27 11:15:52 +08:00
Yingwen
ad5c47185d
feat: wait flush until the flush is done ( #1188 )
...
* feat: Add wait argument to flush
* test(storage): Fix flush tests
2023-03-20 11:25:19 +08:00
Weny Xu
296c6dfcbf
feat: implement table flush ( #1121 )
...
* feat: add flush method for trait
* feat: implement flush via grpc
* chore: move table_dir/region_name/region_id to table crate
* chore: Update src/mito/src/table.rs
---------
Co-authored-by: Yingwen <realevenyag@gmail.com >
2023-03-13 20:10:37 +08:00
fys
3414ac46b0
fix: remove unnecessary braces around block return value ( #1147 )
2023-03-09 03:42:04 +00:00
Zheming Li
b70672be77
feat: track disk usage of regions ( #1125 )
...
* feat: track disk usage of regions
Signed-off-by: Zheming Li <nkdudu@126.com >
* calculate disk usage when call
* add default on file meta
---------
Signed-off-by: Zheming Li <nkdudu@126.com >
2023-03-07 17:13:12 +08:00
Lei, HUANG
e17d5a1c41
feat: support table options ( #1044 )
...
* feat: change table options from string map to a struct, add ttl and write_buffer_size
* fix: also pass table options to table meta
* feat: pass table options when opening/creating regions
* fix: CR comments
2023-02-21 08:10:23 +00:00
Weny Xu
2f39a77137
feat: add close method for the region trait ( #970 )
...
feat: add close for region trait
2023-02-17 11:32:55 +08:00
Lei, HUANG
374acc8830
feat: compaction reader and writer ( #972 )
...
* feat: compaction reader and writer
* feat: make ParquetWrite accept both memtable iterator and chunk reader
* feat: adapt ParquetWriter to accomodate ChunkReaderImpl
* chore: rebase develop
* wip: compile
* wip: task logic
* feat: version and manifest update
* fix: remove useless as_inner from Timestamp vectors
* feat: mark file compacting
* fix: unit test
* fix: clippy warnings
* fix: CR comment
* chore: according to cr comments, remove visit_levels from LevelMetas
* fix: some CR comments
* fix: add PlainTimestampRowFilter for correctness
* fix: cr comments
* fix: some typos
2023-02-14 17:32:00 +08:00
Lei, HUANG
5d62e193bd
feat: support multi regions on datanode ( #653 )
...
* wip: fix compile errors
* chore: move splitter to partition crate
* fix: remove useless variants in frontend errors
* chore: move more partition related code to partition manager
* fix: license header
* wip: move WriteSplitter to PartitionRuleManager
* fix: clippy warnings
* chore: remove useless error variant and format toml
* fix: cr comments
* chore: resolve conflicts
* chore: rebase develop
* fix: cr comments
* feat: support multi regions on datanode
* chore: rebase onto develop
* chore: rebase develop
* chore: rebase develop
* wip
* fix: compile errors
* feat: multi region
* fix: CR comments
* feat: allow stat existing regions without actually open it
* fix: use table meta in manifest to recover region info
2023-02-07 10:46:18 +08:00
Ruihang Xia
36a13dafb7
build(deps): bump tokio to 1.24.2 ( #900 )
...
deps: bump tokio to 1.24.2
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
2023-01-29 11:13:37 +08:00
Lei, HUANG
627d444723
fix: remove start from LogStore; fix error message ( #837 )
2023-01-06 12:21:00 +08:00
Lei, HUANG
8f5ecefc90
feat: use raft-engine crate to reimplement logstore ( #799 )
...
* chore: remove useless method in Entry trait, add proto definition for entry and namespace
* feat: add proto definition for raft-engine based logstore
* feat: introduce RaftEngineLogstore
* feat: impl read for raft engine log store
* feat: impl raft engine logstore
* feat: raft engine logstore start and stop
* feat: add purge bg task
* fix: license header
* fix: clippy
* fix: toml files
* feat: add some test cases
* fix: CR comments
* fix: CR comments
* fix: check namespace validity and state of logstore
* fix: CR comments; add config item to control sync/async flush per write
* fix: remove unused error variants
* fix: unit tests
* fix: use compare and exchange to stop logstore
* fix: CR comments
2023-01-05 17:18:51 +08:00
Lei, HUANG
8ffc078f88
fix: license header ( #815 )
2023-01-03 15:09:49 +08:00
Yingwen
4d56d896ca
feat: Implement delete for the storage engine ( #777 )
...
* docs: Fix incorrect comment of Vector::only_null
* feat: Add delete to WriteRequest and WriteBatch
* feat: Filter deleted rows
* fix: Fix panic after reopening engine
This is detected by adding a reopen step to the delete test for region.
* fix: Fix OpType::min_type()
* test: Add delete absent key test
* chore: Address CR comments
2022-12-30 17:12:18 +08:00
LFC
04df80e640
fix: further ease the restriction of executing SQLs in new GRPC interface ( #797 )
...
* fix: carry not recordbatch result in FlightData, to allow executing SQLs other than selection in new GRPC interface
* Update src/datanode/src/instance/flight/stream.rs
Co-authored-by: Jiachun Feng <jiachun_feng@proton.me >
2022-12-28 16:43:21 +08:00
Ruihang Xia
26a3e93ca7
chore: util workspace deps in more places ( #792 )
...
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
2022-12-27 16:26:59 +08:00
Yingwen
f8500e54c1
refactor: Remove PutOperation and Simplify WriteRequest API ( #775 )
...
* chore: Remove unused MutationExtra
* refactor(storage): Refactor Mutation and Payload
Change Mutation from enum to a struct that holds op type and record
batches so the encoder don't need to convert the mutation into record
batch. Now The Payload is no more an enum, it just holds the data, to
be serialized to the WAL, of the WriteBatch. The encoder and decoder
now deal with the Payload instead of the WriteBatch, so we could hold
more information not necessary to be stored to the WAL in the
WriteBatch.
This commit also merge variants in write_batch::Error to storage::Error
as some variants of them denote the same error.
* test(storage): Pass all tests in storage
* chore: Remove unused codes then format codes
* test(storage): Fix test_put_unknown_column test
* style(storage): Fix clippy
* chore: Remove some unused codes
* chore: Rebase upstream and fix clippy
* chore(storage): Remove unused codes
* chore(storage): Update comments
* feat: Remove PayloadType from wal.proto
* chore: Address CR comments
* chore: Remove unused write_batch.proto
2022-12-26 13:11:24 +08:00
LFC
dc52a51576
chore: upgrade to Arrow 29.0 and use workspace package and dependencies ( #782 )
...
* chore: upgrade to Arrow 29.0 and use workspace package and dependencies
* fix: resolve PR comments
Co-authored-by: luofucong <luofucong@greptime.com >
2022-12-23 14:28:37 +08:00
Lei, HUANG
0653301754
feat: replace arrow2 with official implementation 🎉 ( #753 )
...
* chore: kick off. change datafusion/arrow/parquet to target version
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* chore: replace one last datafusion dep
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* feat: arrow_array switch to arrow
* chore: update dep of binary vector
* chore: fix wrong merge commit
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* feat: Switch to datatypes2
* feat: Make recordbatch compile
* chore: sort Cargo.toml
* feat: Fix common::recordbatch compiler errors
* feat: Fix recordbatch test compiling issue
* fix: api crate (#708 )
* fix: rename ConcreteDataType::timestamp_millis_type to ConcreteDataType::timestamp_millisecond_type. fix other warnings regarding timestamp
* fix: revert changes in datatypes2
* fix: helper
* chore: delete datatypes based on arrow2
* feat: Fix some compiler errors in common::query (#710 )
* feat: Fix some compiler errors in common::query
* feat: test_collect use vectors api
* fix: common-query subcrate (#712 )
* fix: record batch adapter
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* fix error enum
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* fix: Fix common::query compiler errors (#713 )
* feat: Move conversion to ScalarValue to value.rs
* fix: Fix common::query compiler errors
This commit also make InnerError pub(crate)
* feat: Implements diff accumulator using WrapperType (#715 )
* feat: Remove usage of opaque error from common::recordbatch
* feat: Remove opaque error from common::query
* feat: Fix diff compiler errors
Now common_function just use common_query's Error and Result. Adds
a LargestType associated type to LogicalPrimitiveType to get the largest
type a logical primitive type can cast to.
* feat: Remove LargestType from NativeType trait
* chore: Update comments
* feat: Restrict Scalar::RefType of WrapperType to itself
Add trait bound `for<'a> Scalar<RefType<'a> = Self>` to WrapperType
* chore: Address CR comments
* chore: Format codes
* fix: fix compile error for mean/polyval/pow/interp ops
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* Revert "fix: fix compile error for mean/polyval/pow/interp ops"
This reverts commit fb0b4eb826#716 )
* fix: Fix compiler errors in argmax/rate/median/norm_cdf
* chore: Address CR comments
* fix: fix compile error for mean/polyval/pow/interp ops (#717 )
* fix: fix compile error for mean/polyval/pow/interp ops
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* simplify type bounds
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* fix: fix argmin/percentile/clip/interp/scipy_stats_norm_pdf errors (#718 )
fix: fix argmin/percentile/clip/interp/scipy_stats_norm_pdf compiler errors
* fix: fix other compile error in common-function (#719 )
* further fixing
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* fix all compile errors in common function
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* fix: Fix tests and clippy for common-function subcrate (#726 )
* further fixing
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* fix all compile errors in common function
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* fix tests
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* fix clippy
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* revert test changes
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* fix: row group pruning (#725 )
* fix: row group pruning
* chore: use macro to simplify stats implemetation
* fxi: CR comments
* fix: row group metadata length mismatch
* fix: simplify code
* fix: Fix common::grpc compiler errors (#722 )
* fix: Fix common::grpc compiler errors
This commit refactors RecordBatch and holds vectors in the RecordBatch
struct, so we don't need to cast the array to vector when doing
serialization or iterating the batch.
Now we use the vector API instead of the arrow API in grpc crate.
* chore: Address CR comments
* fix common record batch
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* fix: Fix compile error in server subcrate (#727 )
* fix: Fix compile error in server subcrate
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* remove unused type alias
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* explicitly panic
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* Update src/storage/src/sst/parquet.rs
Co-authored-by: Yingwen <realevenyag@gmail.com >
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
Co-authored-by: Yingwen <realevenyag@gmail.com >
* fix: Fix common grpc expr (#730 )
* fix compile errors
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* rename fn names
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* fix styles
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* fix wranings in common-time
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* fix: pre-cast to avoid tremendous match arms (#734 )
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* feat: upgrade storage crate to arrow and parquet offcial impl (#738 )
* fix: compile erros
* fix: parquet reader and writer
* fix: parquet reader and writer
* fix: WriteBatch IPC encode/decode
* fix: clippy errors in storage subcrate
* chore: remove suspicious unwrap
* fix: some cr comments
* fix: CR comments
* fix: CR comments
* fix: Fix compiler errors in catalog and mito crates (#742 )
* fix: Fix compiler errors in mito
* fix: Fix compiler errors in catalog crate
* style: Fix clippy
* chore: Fix use
* Merge pull request #745
* fix nyc-taxi and util
* Merge branch 'replace-arrow2' into fix-others
* fix substrait
* fix warnings and error in test
* fix: Fix imports in optimizer.rs
* fix: errors in optimzer
* fix: remove unwrap
* fix: Fix compiler errors in query crate (#746 )
* fix: Fix compiler errors in state.rs
* fix: fix compiler errors in state
* feat: upgrade sqlparser to 0.26
* fix: fix datafusion engine compiler errors
* fix: Fix some tests in query crate
* fix: Fix all warnings in tests
* feat: Remove `Type` from timestamp's type name
* fix: fix query tests
Now datafusion already supports median, so this commit also remove the
median function
* style: Fix clippy
* feat: Remove RecordBatch::pretty_print
* chore: Address CR comments
* Update src/query/src/query_engine/state.rs
Co-authored-by: Ruihang Xia <waynestxia@gmail.com >
* fix: frontend compile errors (#747 )
fix: fix compile errors in frontend
* fix: Fix compiler errors in script crate (#749 )
* fix: Fix compiler errors in state.rs
* fix: fix compiler errors in state
* feat: upgrade sqlparser to 0.26
* fix: fix datafusion engine compiler errors
* fix: Fix some tests in query crate
* fix: Fix all warnings in tests
* feat: Remove `Type` from timestamp's type name
* fix: fix query tests
Now datafusion already supports median, so this commit also remove the
median function
* style: Fix clippy
* feat: Remove RecordBatch::pretty_print
* chore: Address CR comments
* feat: Add column_by_name to RecordBatch
* feat: modify select_from_rb
* feat: Fix some compiler errors in vector.rs
* feat: Fix more compiler errors in vector.rs
* fix: fix table.rs
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* fix: Fix compiler errors in coprocessor
* fix: Fix some compiler errors
* fix: Fix compiler errors in script
* chore: Remove unused imports and format code
* test: disable interval tests
* test: Fix test_compile_execute test
* style: Fix clippy
* feat: Support interval
* feat: Add RecordBatch::columns and fix clippy
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
Co-authored-by: Ruihang Xia <waynestxia@gmail.com >
* fix: Fix All The Tests! (#752 )
* fix: Fix several tests compile errors
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* fix: some compile errors in tests
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* fix: compile errors in frontend tests
* fix: compile errors in frontend tests
* test: Fix tests in api and common-query
* test: Fix test in sql crate
* fix: resolve substrait error
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* chore: add more test
* test: Fix tests in servers
* fix instance_test
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* test: Fix tests in tests-integration
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
Co-authored-by: Lei, HUANG <mrsatangel@gmail.com >
Co-authored-by: evenyag <realevenyag@gmail.com >
* fix: clippy errors
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
Co-authored-by: Ruihang Xia <waynestxia@gmail.com >
Co-authored-by: evenyag <realevenyag@gmail.com >
2022-12-15 18:49:12 +08:00
Lei, HUANG
756c068166
feat: logstore compaction ( #740 )
...
* feat: add benchmark for wal
* add bin
* feat: impl wal compaction
* chore: This reverts commit ef9f2326
* chore: This reverts commit 9142ec0e
* fix: remove empty files
* fix: failing tests
* fix: CR comments
* fix: Mark log as stable after writer applies manifest
* fix: some cr comments and namings
* chore: rename all stable_xxx to obsolete_xxx
* chore: error message
2022-12-14 16:15:29 +08:00
Ruihang Xia
7ba512980a
chore: add APACHE-2.0 license header ( #518 )
...
* feat: add license checker workflow
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* fix existing header
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* specify license for internal sub-crate
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* fix rustfmt
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
2022-11-15 18:05:46 +08:00
Ruihang Xia
1565c8d236
chore: specify import style in rustfmt ( #460 )
...
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
2022-11-15 15:58:54 +08:00
dennis zhuang
74ea529d1a
feat: move time index metadata from schema into field ( #444 )
...
* feat: move time index metadata from schema into field
* chore: remove useless code
* test: test select with column alias
* fix: conflicts with develop branch
* test: add test
* test: order by timestamp to ensure query results order
* fix: comment
2022-11-11 15:36:27 +08:00
Lei, Huang
6288fdb6bc
feat: frontend catalog ( #437 )
...
* feat: add frontend catalog
2022-11-10 11:52:57 +08:00
Ruihang Xia
af1df2066c
perf: enlarge write row group size ( #413 )
...
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
2022-11-08 11:23:10 +08:00
