* feat/objbench-subcmd:
### Add Object Storage Benchmark Tool and Update Dependencies
- **`Cargo.lock` & `Cargo.toml`**: Added dependencies for `colored`, `parquet`, and `pprof` to support new features.
- **`datanode.rs`**: Introduced `ObjbenchCommand` for benchmarking object storage, including command-line options for configuration and execution. Added `StorageConfig` and `StorageConfigWrapper` for storage engine configuration.
- **`datanode.rs`**: Implemented a stub for `build_object_store` function to initialize object storage.
These changes introduce a new subcommand for object storage benchmarking and update dependencies to support additional functionality.
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* init
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* fix: code style and clippy
* feat/objbench-subcmd:
Improve error handling in `objbench.rs`
- Enhanced error handling in `parse_config` and `parse_file_dir_components` functions by replacing `unwrap` with `OptionExt` and `context` for better error messages.
- Updated `build_access_layer_simple` and `build_cache_manager` functions to use `map_err` for more descriptive error handling.
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* chore: rebase main
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
---------
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
again
false by default
test: config api
refactor: per code review
less info!
even less info!!
docs: gc regions instr
refactor: grp by region id
per code review
per review
error handling?
test: fix
todos
aft rebase fix
after refactor
Signed-off-by: discord9 <discord9@163.com>
* feat/gdump:
### Add Support for Jemalloc Gdump Flag
- **`jemalloc.rs`**: Introduced `PROF_GDUMP` constant and added functions `set_gdump_active` and `is_gdump_active` to manage the gdump flag.
- **`error.rs`**: Added error handling for reading and updating the jemalloc gdump flag with `ReadGdump` and `UpdateGdump` errors.
- **`lib.rs`**: Exposed `is_gdump_active` and `set_gdump_active` functions for non-Windows platforms.
- **`http.rs`**: Added HTTP routes for checking and toggling the jemalloc gdump flag status.
- **`mem_prof.rs`**: Implemented handlers `gdump_toggle_handler` and `gdump_status_handler` for managing gdump flag via HTTP requests.
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* Update docs/how-to/how-to-profile-memory.md
Co-authored-by: shuiyisong <113876041+shuiyisong@users.noreply.github.com>
* fix: typo in docs
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
---------
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
Co-authored-by: shuiyisong <113876041+shuiyisong@users.noreply.github.com>
* feat: adds format, regex_extract function and more type tests
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix: forgot functions
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* chore: forgot null type
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* test: forgot date type
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* feat: remove format function
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* test: update results after upgrading datafusion
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
---------
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* perf: only decode primary keys in the batch
Signed-off-by: evenyag <realevenyag@gmail.com>
* fix: don't push none to creator
Signed-off-by: evenyag <realevenyag@gmail.com>
* chore: implement method to filter __table_id for sparse encoding
Signed-off-by: evenyag <realevenyag@gmail.com>
* feat: filter table id for sparse encoding separately
The __table_id doesn't present in projection so we have to filter it
manually
Signed-off-by: evenyag <realevenyag@gmail.com>
* fix: decode tags for sparse encoding when building bloom filter
Signed-off-by: evenyag <realevenyag@gmail.com>
* feat: support inverted index for tags under sparse encoding
Signed-off-by: evenyag <realevenyag@gmail.com>
* feat: skip tag columns in fulltext index
Signed-off-by: evenyag <realevenyag@gmail.com>
* chore: fix warnings
Signed-off-by: evenyag <realevenyag@gmail.com>
* style: fix clippy
Signed-off-by: evenyag <realevenyag@gmail.com>
* test: fix list index metadata test
Signed-off-by: evenyag <realevenyag@gmail.com>
* fix: decode primary key columns to filter
When primary key columns are not in projection but in filters, we need
to decode them in compute_filter_mask_flat
Signed-off-by: evenyag <realevenyag@gmail.com>
* refactor: reuse filter method
Signed-off-by: evenyag <realevenyag@gmail.com>
* fix: only use dictionary for string type in compat
Signed-off-by: evenyag <realevenyag@gmail.com>
* refactor: safe to get column by creator's column id
Signed-off-by: evenyag <realevenyag@gmail.com>
---------
Signed-off-by: evenyag <realevenyag@gmail.com>
* test: support flat in basic_test
Signed-off-by: evenyag <realevenyag@gmail.com>
* test: support flat in alter_test
Signed-off-by: evenyag <realevenyag@gmail.com>
* test: support flat for append_mode_test
Signed-off-by: evenyag <realevenyag@gmail.com>
* refactor: update bump_committed_sequence_test to test both formats
Signed-off-by: evenyag <realevenyag@gmail.com>
* refactor: update close_test to test both formats
Signed-off-by: evenyag <realevenyag@gmail.com>
* refactor: update compaction_test to test both formats
Signed-off-by: evenyag <realevenyag@gmail.com>
* refactor: update create_test to test both formats
Signed-off-by: evenyag <realevenyag@gmail.com>
* refactor: update edit_region_test to test both formats
Signed-off-by: evenyag <realevenyag@gmail.com>
* refactor: update merge_mode_test to test both formats
Signed-off-by: evenyag <realevenyag@gmail.com>
* refactor: update parallel_test to test both formats
Signed-off-by: evenyag <realevenyag@gmail.com>
* refactor: update projection_test to test both formats
Signed-off-by: evenyag <realevenyag@gmail.com>
* refactor: update prune_test to test both formats
Signed-off-by: evenyag <realevenyag@gmail.com>
* refactor: update row_selector_test to test both formats
Signed-off-by: evenyag <realevenyag@gmail.com>
* refactor: update scan_test to test both formats
Signed-off-by: evenyag <realevenyag@gmail.com>
* refactor: update drop_test to test both formats
Signed-off-by: evenyag <realevenyag@gmail.com>
* refactor: update filter_deleted_test to test both formats
Signed-off-by: evenyag <realevenyag@gmail.com>
* refactor: update sync_test to test both formats
Signed-off-by: evenyag <realevenyag@gmail.com>
* refactor: update set_role_state_test to test both formats
Signed-off-by: evenyag <realevenyag@gmail.com>
* refactor: update staging_test to test both formats
Signed-off-by: evenyag <realevenyag@gmail.com>
* refactor: update truncate_test to test both formats
Signed-off-by: evenyag <realevenyag@gmail.com>
* refactor: update catchup_test to test both formats
Signed-off-by: evenyag <realevenyag@gmail.com>
* refactor: update flush_test to test both formats
Signed-off-by: evenyag <realevenyag@gmail.com>
* refactor: update open_test to test both formats
Signed-off-by: evenyag <realevenyag@gmail.com>
* refactor: update batch_open_test to test both formats
Signed-off-by: evenyag <realevenyag@gmail.com>
* test: fix all flat format tests
Signed-off-by: evenyag <realevenyag@gmail.com>
---------
Signed-off-by: evenyag <realevenyag@gmail.com>
