ci: make preview releases (#1302)

This PR changes the release process. Some parts are more complex, and
other parts I've simplified.

## Simplifications

* Combined `Create Release Commit` and `Create Python Release Commit`
into a single workflow. By default, it does a release of all packages,
but you can still choose to make just a Python or just Node/Rust release
through the arguments. This will make it rarer that we create a Node
release but forget about Python or vice-versa.
* Releases are automatically generated once a tag is pushed. This
eliminates the manual step of creating the release.
* Release notes are automatically generated and changes are categorized
based on the PR labels.
* Removed the use of `LANCEDB_RELEASE_TOKEN` in favor of just using
`GITHUB_TOKEN` where it wasn't necessary. In the one place it is
necessary, I left a comment as to why it is.
* Reused the version in `python/Cargo.toml` so we don't have two
different versions in Python LanceDB.

## New changes

* We now can create `preview` / `beta` releases. By default `Create
Release Commit` will create a preview release, but you can select a
"stable" release type and it will create a full stable release.
  * For Python, pre-releases go to fury.io instead of PyPI
* `bump2version` was deprecated, so upgraded to `bump-my-version`. This
also seems to better support semantic versioning with pre-releases.
* `ci` changes will now be shown in the changelog, allowing changes like
this to be visible to users. `chore` is still hidden.

## Versioning

**NOTE**: unlike how it is in lance repo right now, the version in main
is the last one released, including beta versions.

---------

Co-authored-by: Lance Release <lance-dev@lancedb.com>
Co-authored-by: Weston Pace <weston.pace@gmail.com>

release_process.md

@@ -8,6 +8,51 @@ The Python package is versioned and released separately from the Rust and Node.j
 ones. For Rust and Node.js, the release process is shared between `lancedb` and
 `vectordb` for now.
 
+## Preview releases
+
+LanceDB has full releases about every 2 weeks, but in between we make frequent
+preview releases. These are released as `0.x.y.betaN` versions. They receive the
+same level of testing as normal releases and let you get access to the latest
+features. However, we do not guarantee that preview releases will be available
+more than 6 months after they are released. We may delete the preview releases
+from the packaging index after a while. Once your application is stable, we
+recommend switching to full releases, which will never be removed from package
+indexes.
+
+## Making releases
+
+The release process uses a handful of GitHub actions to automate the process.
+
+```text
+  ┌─────────────────────┐                                                
+  │Create Release Commit│                                                
+  └─┬───────────────────┘                                                
+    │                           ┌────────────┐ ┌──►Python GH Release     
+    ├──►(tag) python-vX.Y.Z ───►│PyPI Publish├─┤                         
+    │                           └────────────┘ └──►Python Wheels         
+    │                                                                    
+    │                           ┌───────────┐                            
+    └──►(tag) vX.Y.Z ───┬──────►│NPM Publish├──┬──►Rust/Node GH Release  
+                        │       └───────────┘  │                         
+                        │                      └──►NPM Packages          
+                        │       ┌─────────────┐                          
+                        └──────►│Cargo Publish├───►Cargo Release         
+                                └─────────────┘                          
+```
+
+To start a release, trigger a `Create Release Commit` action from
+[the workflows page](https://github.com/lancedb/lancedb/actions/workflows/make-release-commit.yml)
+(Click on "Run workflow").
+
+* **For a preview release**, leave the default parameters.
+* **For a stable release**, set the `release_type` input to `stable`.
+
+> [!IMPORTANT]
+> If there was a breaking change since the last stable release, and we haven't
+> done so yet, we should increment the minor version. The CI will detect if this
+> is needed and fail the `Create Release Commit` job. To fix, select the
+> "bump minor version" option.
+
 ## Breaking changes
 
 We try to avoid breaking changes, but sometimes they are necessary. When there
@@ -21,12 +66,10 @@ body of the PR. A CI job will add a `breaking-change` label to the PR, which is
 what will ultimately be used to CI to determine if the minor version should be
 incremented.
 
-A CI job will validate that if a `breaking-change` label is added, the minor
-version is incremented in the `Cargo.toml` and `pyproject.toml` files. The only
-exception is if it has already been incremented since the last stable release.
-
-**It is the responsibility of the PR author to increment the minor version when
-appropriate.**
+> [!IMPORTANT]
+> Reviewers should check that PRs with breaking changes receive the `breaking-change`
+> label. If a PR is missing the label, please add it, even if after it was merged.
+> This label is used in the release process.
 
 Some things that are considered breaking changes: