rust/lancedb
1
0
Fork 0
mirror of https://github.com/lancedb/lancedb.git synced 2026-11-14 14:46:53 +00:00
Code Issues Packages Projects Releases Wiki Activity

refactor(fts): narrow sdk support and test coverage

Browse Source
This commit is contained in:
Yang Cen
2026-07-29 02:13:55 +08:00
parent 34d5c1175a
commit 1a50b697c5
18 changed files with 163 additions and 1963 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/src/java/java.md
+3 -93
View File
@@ -539,91 +539,6 @@ CreateTableScalarIndexResponse response = namespaceClient.createTableScalarIndex
System.out.println("Index transaction: " + response.getTransactionId());
```
### Custom FTS Stop Words
`LanceDbFtsIndexRequest` accepts either an inline list or a newline-delimited
UTF-8 file on the Java client. The client resolves the source to a concrete
snapshot before the request is sent; a client-local path is never sent to
LanceDB Cloud or Enterprise.
Custom stop words replace the built-in list for `language` and are only applied
when `removeStopWords` is true. `null` keeps the built-in language list, while
an empty list explicitly replaces it with no stop words. Exact empty strings
are ignored and exact duplicates retain their first occurrence. Other content,
including case and surrounding whitespace, is preserved.
```java
import com.lancedb.LanceDbFtsIndexClient;
import com.lancedb.LanceDbFtsIndexRequest;
import com.lancedb.LanceDbNamespaceClientBuilder;
import org.lance.namespace.model.CreateTableIndexResponse;
import java.util.Arrays;
public final class CustomStopWordsExample {
public static void main(String[] args) throws Exception {
LanceDbNamespaceClientBuilder clientBuilder =
LanceDbNamespaceClientBuilder.newBuilder()
.apiKey("your_lancedb_api_key")
.database("your_database_name");
LanceDbFtsIndexRequest request = new LanceDbFtsIndexRequest();
request.setId(Arrays.asList("my_namespace", "my_table"));
request.setColumn("text_column");
request.setName("text_idx");
request.setRemoveStopWords(true);
request.setCustomStopWords(Arrays.asList("copyright", "reserved"));
try (LanceDbFtsIndexClient indexClient = clientBuilder.buildFtsIndexClient()) {
CreateTableIndexResponse response = indexClient.createTableIndex(request);
System.out.println("Index transaction: " + response.getTransactionId());
}
}
}
```
To read the snapshot from a file instead, configure the file source before
calling `createTableIndex`:
```java
import java.nio.file.Paths;
request.setCustomStopWordsFile(Paths.get("./stop-words.txt"));
```
The file is decoded as strict UTF-8 and read on the first
`createTableIndex` call. Lines use LF or CRLF terminators; a lone carriage
return is preserved as part of the stop word. The resolved list is then
retained on the request, so a retry uses the same snapshot even if the file
changes.
!!! warning "Use the dedicated FTS index client for custom stop words"
Do not pass `LanceDbFtsIndexRequest` to
`namespaceClient.createTableIndex` or
`namespaceClient.createTableScalarIndex`. The current generated Lance
Namespace model has no `custom_stop_words` field, and its JNI boundary would
silently discard the extra JSON property. `LanceDbFtsIndexClient` uses the
official HTTP client directly and preserves the resolved snapshot.
The helper accepts `endpoint`, API key, database, `delimiter`, and
`header.*`/`headers.*` builder configuration. It rejects transport settings
it cannot safely reproduce, such as `tls.*`; callers needing a custom TLS
stack can construct it with a preconfigured Namespace Apache `ApiClient`.
It also rejects the `uri`, `header.x-api-key`, and
`header.x-lancedb-database` configuration aliases instead of silently
overriding them; use `endpoint()`, `apiKey()`, and `database()`,
respectively.
!!! note "Java table and tokenization APIs"
The current Java SDK is a remote Namespace client and has no embedded/local
LanceDB `Table` API. It therefore cannot prove a complete table-column
snapshot and intentionally does not accept remote table references as a
stop-word source. Materialize those words into an inline list or a local
UTF-8 file first.
Java also has no public standalone tokenization API today, so there is no
Java tokenization entry point to extend with custom stop words.
### Listing Indexes
```java
@@ -641,14 +556,9 @@ for (IndexContent index : response.getIndexes()) {
```
!!! note
The current Java namespace API exposes index type, index name, distance type,
and the generated FTS fields `withPosition`, `baseTokenizer`, `language`,
`maxTokenLength`, `lowerCase`, `stem`, `removeStopWords`, and
`asciiFolding`. `LanceDbFtsIndexRequest` adds the custom stop-word snapshot
while the upstream generated request catches up. IVF training parameters
such as `num_partitions` are not exposed by `CreateTableIndexRequest` yet.
To make those configurable from Java, the namespace API must add those
fields first.
The current Java namespace API exposes index type, index name, distance type, and full text search tokenizer options.
IVF training parameters such as `num_partitions` are not exposed by `CreateTableIndexRequest` yet.
To make those configurable from Java, the namespace API must add those fields first.
## Reading Query Results
java/lancedb-core/pom.xml
-11
View File
@@ -23,17 +23,6 @@
<artifactId>lance-core</artifactId>
</dependency>
<!--
The JNI-backed RestNamespace cannot forward extension fields that are
absent from its generated Rust request model. The official Apache
client is used by LanceDbFtsIndexClient to send the resolved
custom_stop_words snapshot without that lossy JNI round trip.
-->
<dependency>
<groupId>org.lance</groupId>
<artifactId>lance-namespace-apache-client</artifactId>
</dependency>
<dependency>
<groupId>org.apache.arrow</groupId>
<artifactId>arrow-vector</artifactId>
java/lancedb-core/src/main/java/com/lancedb/LanceDbFtsIndexClient.java
-118
View File
@@ -1,118 +0,0 @@
/*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.lancedb;
import org.lance.namespace.client.apache.ApiClient;
import org.lance.namespace.client.apache.ApiException;
import org.lance.namespace.client.apache.api.IndexApi;
import org.lance.namespace.model.CreateTableIndexResponse;
import java.io.Closeable;
import java.io.IOException;
import java.util.List;
import java.util.Objects;
/**
* HTTP client for creating LanceDB FTS indexes with custom stop words.
*
* <p>This deliberately bypasses the current JNI-backed namespace index method. That method
* deserializes requests into an upstream generated Rust model which does not yet have {@code
* custom_stop_words}, silently dropping the resolved snapshot before the HTTP request.
*
* <p>Prefer {@link LanceDbNamespaceClientBuilder#buildFtsIndexClient()}. The public constructor is
* available for callers that need to supply a custom Apache HTTP stack, such as mTLS. This client
* owns the supplied {@link ApiClient}'s HTTP client and closes it from {@link #close()}.
*/
public final class LanceDbFtsIndexClient implements Closeable {
private final ApiClient apiClient;
private final IndexApi indexApi;
private final String delimiter;
/**
* Create a client using a fully configured official Lance Namespace Apache client.
*
* @param apiClient configured HTTP client; ownership transfers to this instance
* @param delimiter namespace identifier delimiter, normally {@code $}
*/
public LanceDbFtsIndexClient(ApiClient apiClient, String delimiter) {
this(apiClient, new IndexApi(Objects.requireNonNull(apiClient, "apiClient")), delimiter);
}
LanceDbFtsIndexClient(ApiClient apiClient, IndexApi indexApi, String delimiter) {
this.apiClient = Objects.requireNonNull(apiClient, "apiClient");
this.indexApi = Objects.requireNonNull(indexApi, "indexApi");
if (delimiter == null || delimiter.isEmpty()) {
throw new IllegalArgumentException("delimiter cannot be null or empty");
}
this.delimiter = delimiter;
}
/**
* Resolve any client-local stop-word file and create the FTS index.
*
* <p>The outgoing JSON contains only the concrete {@code custom_stop_words} list (including a
* meaningful distinction between {@code null} and an empty list), never a client-local path.
*
* @param request FTS index request
* @return asynchronous index-creation transaction response
* @throws IOException if a local stop-word file cannot be read as strict UTF-8
* @throws ApiException if the LanceDB HTTP request fails
*/
public CreateTableIndexResponse createTableIndex(LanceDbFtsIndexRequest request)
throws IOException, ApiException {
validateRequest(request);
request.resolveCustomStopWordsSnapshot();
String id = joinIdentifier(request.getId());
return indexApi.createTableIndex(id, request, delimiter);
}
private void validateRequest(LanceDbFtsIndexRequest request) {
if (request == null) {
throw new IllegalArgumentException("FTS index request cannot be null");
}
if (!"FTS".equals(request.getIndexType())) {
throw new IllegalArgumentException("FTS index request index type must be FTS");
}
if (request.getColumn() == null || request.getColumn().trim().isEmpty()) {
throw new IllegalArgumentException("FTS index request requires a non-empty column");
}
List<String> id = request.getId();
if (id == null || id.isEmpty()) {
throw new IllegalArgumentException("FTS index request requires a non-empty table id");
}
for (int index = 0; index < id.size(); index++) {
String part = id.get(index);
if (part == null || part.isEmpty()) {
throw new IllegalArgumentException(
"FTS index request table id part at index " + index + " cannot be null or empty");
}
}
}
private String joinIdentifier(List<String> id) {
StringBuilder result = new StringBuilder();
for (String part : id) {
if (result.length() > 0) {
result.append(delimiter);
}
result.append(part);
}
return result.toString();
}
@Override
public void close() throws IOException {
apiClient.getHttpClient().close();
}
}
java/lancedb-core/src/main/java/com/lancedb/LanceDbFtsIndexRequest.java
-315
View File
@@ -1,315 +0,0 @@
/*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.lancedb;
import com.fasterxml.jackson.annotation.JsonIgnore;
import com.fasterxml.jackson.annotation.JsonInclude;
import com.fasterxml.jackson.annotation.JsonProperty;
import org.lance.namespace.model.CreateTableIndexRequest;
import java.io.IOException;
import java.nio.ByteBuffer;
import java.nio.charset.CharacterCodingException;
import java.nio.charset.CodingErrorAction;
import java.nio.charset.StandardCharsets;
import java.nio.file.Files;
import java.nio.file.Path;
import java.util.ArrayList;
import java.util.Collections;
import java.util.LinkedHashSet;
import java.util.List;
import java.util.Locale;
import java.util.Objects;
import java.util.Set;
/**
* FTS index request with a resolved custom stop-word snapshot.
*
* <p>Custom stop words replace the built-in list for {@link #getLanguage()} and only affect
* tokenization when {@link #getRemoveStopWords()} is {@code true}. A {@code null} list keeps the
* built-in language list. An empty list explicitly replaces it with no stop words.
*
* <p>Use exactly one source: {@link #setCustomStopWords(List)} for an inline list or {@link
* #setCustomStopWordsFile(Path)} for a newline-delimited UTF-8 file on the client. Exact empty
* strings are removed and exact duplicates retain their first occurrence. Other content, including
* case and surrounding whitespace, is preserved verbatim. File lines use LF or CRLF terminators; a
* lone carriage return is part of the stop word.
*
* <p>The file is read when {@link LanceDbFtsIndexClient#createTableIndex(LanceDbFtsIndexRequest)}
* is first called. The resulting owned list is then reused, so retries cannot silently pick up a
* different file. Client-local paths are never serialized.
*
* <p>Do not pass this request to the JNI-backed {@code LanceNamespace.createTableIndex} method.
* Its upstream generated request model does not yet contain {@code custom_stop_words}, and the JNI
* layer would discard the extension field. Use {@link LanceDbFtsIndexClient}.
*
* <p>A LanceDB table source is not exposed here because the current Java SDK has no embedded/local
* Table API from which it could prove and materialize a complete snapshot. Remote table references
* are intentionally not sent to the server.
*/
public final class LanceDbFtsIndexRequest extends CreateTableIndexRequest {
public static final String JSON_PROPERTY_CUSTOM_STOP_WORDS = "custom_stop_words";
private enum StopWordsSource {
NONE,
INLINE,
FILE
}
private List<String> customStopWords;
@JsonIgnore private transient StopWordsSource stopWordsSource = StopWordsSource.NONE;
@JsonIgnore private transient Path customStopWordsFile;
@JsonIgnore private transient boolean fileSnapshotResolved;
/** Create an FTS request. The index type is always {@code FTS}. */
public LanceDbFtsIndexRequest() {
super.setIndexType("FTS");
}
/**
* Return the resolved custom stop-word snapshot.
*
* <p>For a file source this remains {@code null} until the FTS index client resolves the file.
*
* @return {@code null} for the built-in language list, or the immutable custom snapshot
*/
@javax.annotation.Nullable
@JsonProperty(JSON_PROPERTY_CUSTOM_STOP_WORDS)
@JsonInclude(value = JsonInclude.Include.ALWAYS)
public List<String> getCustomStopWords() {
return customStopWords;
}
/**
* Set inline custom stop words.
*
* @param words custom stop words, {@code null} to use the built-in language list, or an empty
* list to explicitly use no stop words
* @throws IllegalArgumentException if an element is null or is not a string at runtime
* @throws IllegalStateException if a file source is already configured
*/
@JsonProperty(JSON_PROPERTY_CUSTOM_STOP_WORDS)
@JsonInclude(value = JsonInclude.Include.ALWAYS)
public void setCustomStopWords(@javax.annotation.Nullable List<String> words) {
if (words == null) {
clearCustomStopWords();
return;
}
requireCompatibleSource(StopWordsSource.INLINE);
customStopWords = snapshot(words);
customStopWordsFile = null;
fileSnapshotResolved = false;
stopWordsSource = StopWordsSource.INLINE;
}
/**
* Fluent form of {@link #setCustomStopWords(List)}.
*
* @param words inline stop words
* @return this request
*/
public LanceDbFtsIndexRequest customStopWords(@javax.annotation.Nullable List<String> words) {
setCustomStopWords(words);
return this;
}
/**
* Configure a newline-delimited UTF-8 file source.
*
* <p>The path is client-local and is not serialized. It is read exactly once, on the first call
* to the FTS index client. Pass {@code null} to clear the source and use the built-in language
* list.
*
* @param path client-local stop-word file
* @throws IllegalArgumentException if the path is empty
* @throws IllegalStateException if an inline source is already configured
*/
@JsonIgnore
public void setCustomStopWordsFile(@javax.annotation.Nullable Path path) {
if (path == null) {
clearCustomStopWords();
return;
}
if (path.toString().isEmpty()) {
throw new IllegalArgumentException(
"custom stop words file source requires a non-empty path");
}
requireCompatibleSource(StopWordsSource.FILE);
customStopWords = null;
customStopWordsFile = path;
fileSnapshotResolved = false;
stopWordsSource = StopWordsSource.FILE;
}
/**
* Fluent form of {@link #setCustomStopWordsFile(Path)}.
*
* @param path client-local stop-word file
* @return this request
*/
@JsonIgnore
public LanceDbFtsIndexRequest customStopWordsFile(@javax.annotation.Nullable Path path) {
setCustomStopWordsFile(path);
return this;
}
/**
* Keep this specialized request constrained to FTS.
*
* @param indexType must be {@code FTS}
*/
@Override
public void setIndexType(String indexType) {
if (!"FTS".equals(indexType)) {
throw new IllegalArgumentException("LanceDbFtsIndexRequest index type must be FTS");
}
super.setIndexType(indexType);
}
/**
* Keep this specialized request constrained to FTS.
*
* @param indexType must be {@code FTS}
* @return this request
*/
@Override
public LanceDbFtsIndexRequest indexType(String indexType) {
setIndexType(indexType);
return this;
}
void resolveCustomStopWordsSnapshot() throws IOException {
if (stopWordsSource != StopWordsSource.FILE || fileSnapshotResolved) {
return;
}
byte[] bytes;
try {
bytes = Files.readAllBytes(customStopWordsFile);
} catch (IOException e) {
throw new IOException(
"failed to read custom stop words file `" + customStopWordsFile + "`: " + e.getMessage(),
e);
}
String contents;
try {
contents =
StandardCharsets.UTF_8
.newDecoder()
.onMalformedInput(CodingErrorAction.REPORT)
.onUnmappableCharacter(CodingErrorAction.REPORT)
.decode(ByteBuffer.wrap(bytes))
.toString();
} catch (CharacterCodingException e) {
throw new IOException(
"custom stop words file `" + customStopWordsFile + "` is not valid UTF-8", e);
}
customStopWords = snapshot(splitLines(contents));
fileSnapshotResolved = true;
}
/**
* Match Rust {@code str::lines()}: split only on LF and remove a CR only when it belongs to a
* CRLF terminator. A lone CR is stop-word content and must be preserved.
*/
private static List<String> splitLines(String contents) {
List<String> lines = new ArrayList<>();
int lineStart = 0;
for (int index = 0; index < contents.length(); index++) {
if (contents.charAt(index) != '\n') {
continue;
}
int lineEnd = index;
if (lineEnd > lineStart && contents.charAt(lineEnd - 1) == '\r') {
lineEnd--;
}
lines.add(contents.substring(lineStart, lineEnd));
lineStart = index + 1;
}
if (lineStart < contents.length()) {
lines.add(contents.substring(lineStart));
}
return lines;
}
private void clearCustomStopWords() {
customStopWords = null;
customStopWordsFile = null;
fileSnapshotResolved = false;
stopWordsSource = StopWordsSource.NONE;
}
private void requireCompatibleSource(StopWordsSource requested) {
if (stopWordsSource != StopWordsSource.NONE && stopWordsSource != requested) {
throw new IllegalStateException(
"custom stop words inline and file sources are mutually exclusive; clear the "
+ stopWordsSource.name().toLowerCase(Locale.ROOT)
+ " source before configuring a "
+ requested.name().toLowerCase(Locale.ROOT)
+ " source");
}
}
private static List<String> snapshot(List<String> words) {
Set<String> seen = new LinkedHashSet<>();
for (int index = 0; index < words.size(); index++) {
Object value = words.get(index);
if (value == null) {
throw new IllegalArgumentException(
"custom stop words inline value at index " + index + " cannot be null");
}
if (!(value instanceof String)) {
throw new IllegalArgumentException(
"custom stop words inline value at index "
+ index
+ " must be a string, but was "
+ value.getClass().getName());
}
String word = (String) value;
if (!word.isEmpty()) {
seen.add(word);
}
}
return Collections.unmodifiableList(new ArrayList<>(seen));
}
@Override
public boolean equals(Object other) {
if (this == other) {
return true;
}
if (!(other instanceof LanceDbFtsIndexRequest)) {
return false;
}
LanceDbFtsIndexRequest that = (LanceDbFtsIndexRequest) other;
return super.equals(other)
&& Objects.equals(customStopWords, that.customStopWords)
&& stopWordsSource == that.stopWordsSource
&& Objects.equals(customStopWordsFile, that.customStopWordsFile)
&& fileSnapshotResolved == that.fileSnapshotResolved;
}
@Override
public int hashCode() {
return Objects.hash(
super.hashCode(),
customStopWords,
stopWordsSource,
customStopWordsFile,
fileSnapshotResolved);
}
}
java/lancedb-core/src/main/java/com/lancedb/LanceDbNamespaceClientBuilder.java
+14 -116
View File
@@ -14,7 +14,6 @@
package com.lancedb;
import org.lance.namespace.LanceNamespace;
import org.lance.namespace.client.apache.ApiClient;
import java.util.HashMap;
import java.util.Map;
@@ -137,130 +136,29 @@ public class LanceDbNamespaceClientBuilder {
* @throws IllegalStateException if required parameters are missing
*/
public LanceNamespace build() {
validateRequiredFields();
// Build configuration map
Map<String, String> config = new HashMap<>(additionalConfig);
config.put("header.x-lancedb-database", database);
config.put("header.x-api-key", apiKey);
config.put("uri", resolveUri());
return LanceNamespace.connect("rest", config, null);
}
/**
* Build a client for creating FTS indexes with custom stop words.
*
* <p>The generated Lance Namespace request model currently has no {@code custom_stop_words}
* field. Passing a subclass through the JNI-backed {@link LanceNamespace} client would silently
* discard that field. This client uses the official HTTP transport directly so the resolved
* client-side snapshot reaches LanceDB Cloud or Enterprise.
*
* <p>The helper supports the same endpoint, API key, database, identifier delimiter, and {@code
* header.*}/{@code headers.*} configuration as {@link #build()}. TLS and other transport
* configuration cannot be copied safely to the Apache HTTP client and is rejected with an
* actionable error instead of being ignored. Callers needing a custom HTTP stack can construct
* {@link LanceDbFtsIndexClient} with a preconfigured {@link ApiClient}.
*
* @return A configured custom-stop-word FTS index client
* @throws IllegalStateException if required parameters are missing or an unsupported
* configuration key is present
*/
public LanceDbFtsIndexClient buildFtsIndexClient() {
validateRequiredFields();
String delimiter = additionalConfig.getOrDefault("delimiter", "$");
if (delimiter == null || delimiter.isEmpty()) {
throw new IllegalStateException("FTS index client delimiter cannot be null or empty");
}
String uri = resolveUri();
if (uri.trim().isEmpty()) {
throw new IllegalStateException("FTS index client endpoint cannot be empty");
}
Map<String, String> headers = new HashMap<>();
for (Map.Entry<String, String> entry : additionalConfig.entrySet()) {
String key = entry.getKey();
String value = entry.getValue();
if (key == null) {
throw new IllegalStateException("FTS index client configuration key cannot be null");
}
if ("delimiter".equals(key)) {
continue;
}
if ("uri".equals(key)) {
throw new IllegalStateException(
"FTS index client configuration key 'uri' is not accepted; use endpoint() instead");
}
String headerName = headerName(key);
if (headerName != null) {
if (headerName.isEmpty()) {
throw new IllegalStateException(
"FTS index client header configuration requires a non-empty header name");
}
if (value == null) {
throw new IllegalStateException(
"FTS index client header '" + headerName + "' cannot have a null value");
}
if (isReservedHeader(headerName)) {
String builderMethod =
"x-api-key".equalsIgnoreCase(headerName) ? "apiKey()" : "database()";
throw new IllegalStateException(
"FTS index client header '"
+ headerName
+ "' is reserved; configure it with "
+ builderMethod
+ " instead");
}
headers.put(headerName, value);
continue;
}
throw new IllegalStateException(
"FTS index client cannot safely apply configuration key '"
+ key
+ "'. Configure an org.lance.namespace.client.apache.ApiClient explicitly and "
+ "pass it to LanceDbFtsIndexClient instead.");
}
ApiClient apiClient = new ApiClient().setBasePath(uri).setApiKey(apiKey);
for (Map.Entry<String, String> header : headers.entrySet()) {
apiClient.addDefaultHeader(header.getKey(), header.getValue());
}
apiClient.addDefaultHeader("x-lancedb-database", database);
return new LanceDbFtsIndexClient(apiClient, delimiter);
}
private void validateRequiredFields() {
// Validate required fields
if (apiKey == null) {
throw new IllegalStateException("API key is required");
}
if (database == null) {
throw new IllegalStateException("Database is required");
}
}
private String resolveUri() {
// Build configuration map
Map<String, String> config = new HashMap<>(additionalConfig);
config.put("header.x-lancedb-database", database);
config.put("header.x-api-key", apiKey);
// Determine base URL
String uri;
if (endpoint.isPresent()) {
return endpoint.get();
uri = endpoint.get();
} else {
String effectiveRegion = region.orElse(DEFAULT_REGION);
uri = String.format(CLOUD_URL_PATTERN, database, effectiveRegion);
}
String effectiveRegion = region.orElse(DEFAULT_REGION);
return String.format(CLOUD_URL_PATTERN, database, effectiveRegion);
}
config.put("uri", uri);
private static String headerName(String key) {
if (key.startsWith("header.")) {
return key.substring("header.".length());
}
if (key.startsWith("headers.")) {
return key.substring("headers.".length());
}
return null;
}
private static boolean isReservedHeader(String headerName) {
return "x-api-key".equalsIgnoreCase(headerName)
|| "x-lancedb-database".equalsIgnoreCase(headerName);
return LanceNamespace.connect("rest", config, null);
}
}
java/lancedb-core/src/test/java/com/lancedb/LanceDbFtsIndexClientTest.java
-133
View File
@@ -1,133 +0,0 @@
/*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.lancedb;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.io.TempDir;
import org.lance.namespace.client.apache.ApiClient;
import org.lance.namespace.client.apache.ApiException;
import org.lance.namespace.client.apache.api.IndexApi;
import org.lance.namespace.model.CreateTableIndexRequest;
import org.lance.namespace.model.CreateTableIndexResponse;
import java.nio.charset.StandardCharsets;
import java.nio.file.Files;
import java.nio.file.Path;
import java.util.Arrays;
import java.util.Collections;
import static org.junit.jupiter.api.Assertions.assertEquals;
import static org.junit.jupiter.api.Assertions.assertSame;
import static org.junit.jupiter.api.Assertions.assertThrows;
import static org.junit.jupiter.api.Assertions.assertTrue;
/** Unit tests for {@link LanceDbFtsIndexClient}. */
public class LanceDbFtsIndexClientTest {
@TempDir Path tempDir;
@Test
public void testCreateIndexResolvesFileAndReusesSnapshotOnRetry() throws Exception {
ApiClient apiClient = new ApiClient();
CreateTableIndexResponse response = new CreateTableIndexResponse();
CapturingIndexApi indexApi = new CapturingIndexApi(apiClient, response);
LanceDbFtsIndexRequest request = request();
Path path = tempDir.resolve("stop-words.txt");
Files.write(path, "cat\ncat\n dog \n".getBytes(StandardCharsets.UTF_8));
request.setCustomStopWordsFile(path);
try (LanceDbFtsIndexClient client = new LanceDbFtsIndexClient(apiClient, indexApi, "$")) {
CreateTableIndexResponse actual = client.createTableIndex(request);
Files.write(path, "changed\n".getBytes(StandardCharsets.UTF_8));
CreateTableIndexResponse retry = client.createTableIndex(request);
assertSame(response, actual);
assertSame(response, retry);
assertEquals(Arrays.asList("cat", " dog "), request.getCustomStopWords());
assertEquals("namespace$table", indexApi.id);
assertSame(request, indexApi.request);
assertEquals("$", indexApi.delimiter);
assertEquals(2, indexApi.calls);
}
}
@Test
public void testCreateIndexValidatesRequiredFieldsBeforeNetworkCall() throws Exception {
ApiClient apiClient = new ApiClient();
CapturingIndexApi indexApi =
new CapturingIndexApi(apiClient, new CreateTableIndexResponse());
try (LanceDbFtsIndexClient client = new LanceDbFtsIndexClient(apiClient, indexApi, "$")) {
LanceDbFtsIndexRequest noColumn = new LanceDbFtsIndexRequest();
noColumn.setId(Collections.singletonList("table"));
IllegalArgumentException columnError =
assertThrows(IllegalArgumentException.class, () -> client.createTableIndex(noColumn));
assertTrue(columnError.getMessage().contains("non-empty column"));
LanceDbFtsIndexRequest noId = new LanceDbFtsIndexRequest();
noId.setColumn("text");
IllegalArgumentException idError =
assertThrows(IllegalArgumentException.class, () -> client.createTableIndex(noId));
assertTrue(idError.getMessage().contains("non-empty table id"));
assertEquals(0, indexApi.calls);
}
}
@Test
public void testDelimiterMustBeNonEmpty() throws Exception {
ApiClient apiClient = new ApiClient();
try {
CapturingIndexApi indexApi =
new CapturingIndexApi(apiClient, new CreateTableIndexResponse());
IllegalArgumentException error =
assertThrows(
IllegalArgumentException.class,
() -> new LanceDbFtsIndexClient(apiClient, indexApi, ""));
assertTrue(error.getMessage().contains("delimiter"));
} finally {
apiClient.getHttpClient().close();
}
}
private static LanceDbFtsIndexRequest request() {
LanceDbFtsIndexRequest request = new LanceDbFtsIndexRequest();
request.setId(Arrays.asList("namespace", "table"));
request.setColumn("text");
request.setRemoveStopWords(true);
return request;
}
private static final class CapturingIndexApi extends IndexApi {
private final CreateTableIndexResponse response;
private String id;
private CreateTableIndexRequest request;
private String delimiter;
private int calls;
private CapturingIndexApi(ApiClient apiClient, CreateTableIndexResponse response) {
super(apiClient);
this.response = response;
}
@Override
public CreateTableIndexResponse createTableIndex(
String id, CreateTableIndexRequest request, String delimiter) throws ApiException {
this.id = id;
this.request = request;
this.delimiter = delimiter;
calls++;
return response;
}
}
}
java/lancedb-core/src/test/java/com/lancedb/LanceDbFtsIndexRequestTest.java
-239
View File
@@ -1,239 +0,0 @@
/*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.lancedb;
import com.fasterxml.jackson.databind.JsonNode;
import com.fasterxml.jackson.databind.ObjectMapper;
import org.apache.hc.core5.http.ContentType;
import org.apache.hc.core5.http.HttpEntity;
import org.apache.hc.core5.http.io.entity.EntityUtils;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.io.TempDir;
import org.lance.namespace.client.apache.ApiClient;
import java.io.IOException;
import java.nio.charset.StandardCharsets;
import java.nio.file.Files;
import java.nio.file.Path;
import java.nio.file.Paths;
import java.util.ArrayList;
import java.util.Arrays;
import java.util.Collections;
import java.util.List;
import static org.junit.jupiter.api.Assertions.assertEquals;
import static org.junit.jupiter.api.Assertions.assertFalse;
import static org.junit.jupiter.api.Assertions.assertNull;
import static org.junit.jupiter.api.Assertions.assertThrows;
import static org.junit.jupiter.api.Assertions.assertTrue;
/** Unit tests for {@link LanceDbFtsIndexRequest}. */
public class LanceDbFtsIndexRequestTest {
@TempDir Path tempDir;
@Test
public void testInlineSnapshotPreservesContentAndCanonicalizesExactValues() {
List<String> input =
new ArrayList<>(Arrays.asList("cat", "", " dog ", "cat", "CAT", " dog "));
LanceDbFtsIndexRequest request = new LanceDbFtsIndexRequest();
request.setCustomStopWords(input);
input.set(0, "mutated");
assertEquals(Arrays.asList("cat", " dog ", "CAT"), request.getCustomStopWords());
assertThrows(
UnsupportedOperationException.class,
() -> request.getCustomStopWords().add("later mutation"));
}
@Test
public void testNullAndEmptySnapshotsRemainDistinctInApacheClientJson() throws Exception {
ApiClient apiClient = new ApiClient();
try {
LanceDbFtsIndexRequest request = new LanceDbFtsIndexRequest();
request.setColumn("text");
JsonNode nullJson = serialize(apiClient, request);
assertTrue(nullJson.has("custom_stop_words"));
assertTrue(nullJson.get("custom_stop_words").isNull());
request.setCustomStopWords(Collections.emptyList());
JsonNode emptyJson = serialize(apiClient, request);
assertTrue(emptyJson.get("custom_stop_words").isArray());
assertEquals(0, emptyJson.get("custom_stop_words").size());
} finally {
apiClient.getHttpClient().close();
}
}
@Test
public void testInlineNullAndNonStringValuesAreRejected() {
LanceDbFtsIndexRequest request = new LanceDbFtsIndexRequest();
IllegalArgumentException nullError =
assertThrows(
IllegalArgumentException.class,
() -> request.setCustomStopWords(Arrays.asList("cat", null)));
assertTrue(nullError.getMessage().contains("index 1"));
assertTrue(nullError.getMessage().contains("cannot be null"));
@SuppressWarnings({"rawtypes", "unchecked"})
List<String> invalid = (List) Arrays.asList("cat", 42);
IllegalArgumentException typeError =
assertThrows(IllegalArgumentException.class, () -> request.setCustomStopWords(invalid));
assertTrue(typeError.getMessage().contains("index 1"));
assertTrue(typeError.getMessage().contains("must be a string"));
}
@Test
public void testInlineAndFileSourcesAreMutuallyExclusive() {
Path path = tempDir.resolve("stop-words.txt");
LanceDbFtsIndexRequest inline = new LanceDbFtsIndexRequest();
inline.setCustomStopWords(Collections.emptyList());
IllegalStateException inlineError =
assertThrows(IllegalStateException.class, () -> inline.setCustomStopWordsFile(path));
assertTrue(inlineError.getMessage().contains("mutually exclusive"));
LanceDbFtsIndexRequest file = new LanceDbFtsIndexRequest();
file.setCustomStopWordsFile(path);
IllegalStateException fileError =
assertThrows(
IllegalStateException.class,
() -> file.setCustomStopWords(Collections.singletonList("cat")));
assertTrue(fileError.getMessage().contains("mutually exclusive"));
inline.setCustomStopWords(null);
inline.setCustomStopWordsFile(path);
}
@Test
public void testFileSnapshotUsesStrictUtf8AndIsStable() throws Exception {
Path path = tempDir.resolve("stop-words.txt");
Files.write(path, "cat\n\n dog \r\ncat\nCAT\n".getBytes(StandardCharsets.UTF_8));
LanceDbFtsIndexRequest request = new LanceDbFtsIndexRequest();
request.setCustomStopWordsFile(path);
assertNull(request.getCustomStopWords());
request.resolveCustomStopWordsSnapshot();
assertEquals(Arrays.asList("cat", " dog ", "CAT"), request.getCustomStopWords());
Files.write(path, "changed\n".getBytes(StandardCharsets.UTF_8));
request.resolveCustomStopWordsSnapshot();
assertEquals(Arrays.asList("cat", " dog ", "CAT"), request.getCustomStopWords());
}
@Test
public void testFilePathIsNeverSerialized() throws Exception {
Path path = tempDir.resolve("private-client-path.txt");
Files.write(path, "cat\n".getBytes(StandardCharsets.UTF_8));
LanceDbFtsIndexRequest request = new LanceDbFtsIndexRequest();
request.setColumn("text");
request.setCustomStopWordsFile(path);
request.resolveCustomStopWordsSnapshot();
ApiClient apiClient = new ApiClient();
try {
JsonNode json = serialize(apiClient, request);
assertEquals(Collections.singletonList("cat"), request.getCustomStopWords());
assertEquals("cat", json.get("custom_stop_words").get(0).asText());
assertFalse(json.has("customStopWordsFile"));
assertFalse(json.has("custom_stop_words_file"));
assertFalse(json.toString().contains(path.toString()));
} finally {
apiClient.getHttpClient().close();
}
}
@Test
public void testEmptyFileProducesExplicitEmptySnapshot() throws Exception {
Path path = tempDir.resolve("empty.txt");
Files.write(path, new byte[0]);
LanceDbFtsIndexRequest request = new LanceDbFtsIndexRequest();
request.setCustomStopWordsFile(path);
request.resolveCustomStopWordsSnapshot();
assertEquals(Collections.emptyList(), request.getCustomStopWords());
}
@Test
public void testLoneCarriageReturnIsPreservedAsContent() throws Exception {
Path path = tempDir.resolve("lone-cr.txt");
Files.write(path, "cat\rdog\ncat\r\nbird\r".getBytes(StandardCharsets.UTF_8));
LanceDbFtsIndexRequest request = new LanceDbFtsIndexRequest();
request.setCustomStopWordsFile(path);
request.resolveCustomStopWordsSnapshot();
assertEquals(Arrays.asList("cat\rdog", "cat", "bird\r"), request.getCustomStopWords());
}
@Test
public void testMissingFileIsRejectedWithPath() {
Path path = tempDir.resolve("missing.txt");
LanceDbFtsIndexRequest request = new LanceDbFtsIndexRequest();
request.setCustomStopWordsFile(path);
IOException error =
assertThrows(IOException.class, request::resolveCustomStopWordsSnapshot);
assertTrue(error.getMessage().contains("failed to read custom stop words file"));
assertTrue(error.getMessage().contains(path.toString()));
}
@Test
public void testEmptyFilePathIsRejectedDescriptively() {
LanceDbFtsIndexRequest request = new LanceDbFtsIndexRequest();
IllegalArgumentException error =
assertThrows(
IllegalArgumentException.class,
() -> request.setCustomStopWordsFile(Paths.get("")));
assertTrue(error.getMessage().contains("file source requires a non-empty path"));
}
@Test
public void testMalformedUtf8IsRejected() throws Exception {
Path path = tempDir.resolve("invalid.txt");
Files.write(path, new byte[] {(byte) 0xc3, (byte) 0x28});
LanceDbFtsIndexRequest request = new LanceDbFtsIndexRequest();
request.setCustomStopWordsFile(path);
IOException error =
assertThrows(IOException.class, request::resolveCustomStopWordsSnapshot);
assertTrue(error.getMessage().contains("not valid UTF-8"));
}
@Test
public void testRequestCannotBeChangedToAnotherIndexType() {
LanceDbFtsIndexRequest request = new LanceDbFtsIndexRequest();
assertEquals("FTS", request.getIndexType());
IllegalArgumentException error =
assertThrows(IllegalArgumentException.class, () -> request.setIndexType("BTREE"));
assertTrue(error.getMessage().contains("must be FTS"));
}
private static JsonNode serialize(ApiClient apiClient, LanceDbFtsIndexRequest request)
throws Exception {
HttpEntity entity =
apiClient.serialize(request, Collections.emptyMap(), ContentType.APPLICATION_JSON);
String json = EntityUtils.toString(entity);
return new ObjectMapper().readTree(json);
}
}
java/lancedb-core/src/test/java/com/lancedb/LanceDbNamespaceClientBuilderTest.java
-87
View File
@@ -93,91 +93,4 @@ public class LanceDbNamespaceClientBuilderTest {
assertNotSame(builder1, builder2);
}
@Test
public void testFtsIndexClientRequiresApiKey() {
LanceDbNamespaceClientBuilder builder =
LanceDbNamespaceClientBuilder.newBuilder().database("test-db");
IllegalStateException exception =
assertThrows(IllegalStateException.class, builder::buildFtsIndexClient);
assertEquals("API key is required", exception.getMessage());
}
@Test
public void testFtsIndexClientRequiresDatabase() {
LanceDbNamespaceClientBuilder builder =
LanceDbNamespaceClientBuilder.newBuilder().apiKey("test-api-key");
IllegalStateException exception =
assertThrows(IllegalStateException.class, builder::buildFtsIndexClient);
assertEquals("Database is required", exception.getMessage());
}
@Test
public void testFtsIndexClientRejectsUnsupportedTransportConfiguration() {
LanceDbNamespaceClientBuilder builder =
LanceDbNamespaceClientBuilder.newBuilder()
.apiKey("test-api-key")
.database("test-db")
.config("tls.cert_file", "/tmp/client.pem");
IllegalStateException exception =
assertThrows(IllegalStateException.class, builder::buildFtsIndexClient);
assertTrue(exception.getMessage().contains("tls.cert_file"));
assertTrue(exception.getMessage().contains("ApiClient"));
}
@Test
public void testFtsIndexClientRejectsEmptyDelimiter() {
LanceDbNamespaceClientBuilder builder =
LanceDbNamespaceClientBuilder.newBuilder()
.apiKey("test-api-key")
.database("test-db")
.config("delimiter", "");
IllegalStateException exception =
assertThrows(IllegalStateException.class, builder::buildFtsIndexClient);
assertTrue(exception.getMessage().contains("delimiter"));
}
@Test
public void testFtsIndexClientRejectsUriConfigurationAlias() {
LanceDbNamespaceClientBuilder builder =
LanceDbNamespaceClientBuilder.newBuilder()
.apiKey("test-api-key")
.database("test-db")
.config("uri", "https://example.invalid");
IllegalStateException exception =
assertThrows(IllegalStateException.class, builder::buildFtsIndexClient);
assertTrue(exception.getMessage().contains("uri"));
assertTrue(exception.getMessage().contains("endpoint()"));
}
@Test
public void testFtsIndexClientRejectsReservedHeaders() {
LanceDbNamespaceClientBuilder apiKeyBuilder =
LanceDbNamespaceClientBuilder.newBuilder()
.apiKey("test-api-key")
.database("test-db")
.config("header.x-api-key", "other-key");
IllegalStateException apiKeyError =
assertThrows(IllegalStateException.class, apiKeyBuilder::buildFtsIndexClient);
assertTrue(apiKeyError.getMessage().contains("reserved"));
assertTrue(apiKeyError.getMessage().contains("apiKey()"));
LanceDbNamespaceClientBuilder databaseBuilder =
LanceDbNamespaceClientBuilder.newBuilder()
.apiKey("test-api-key")
.database("test-db")
.config("headers.X-LanceDB-Database", "other-db");
IllegalStateException databaseError =
assertThrows(IllegalStateException.class, databaseBuilder::buildFtsIndexClient);
assertTrue(databaseError.getMessage().contains("reserved"));
assertTrue(databaseError.getMessage().contains("database()"));
}
}
java/pom.xml
-7
View File
@@ -29,8 +29,6 @@
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
<arrow.version>15.0.0</arrow.version>
<lance-core.version>10.0.0-beta.5</lance-core.version>
<!-- Keep aligned with the Lance Namespace client used by lance-core. -->
<lance-namespace.version>0.7.7</lance-namespace.version>
<spotless.skip>false</spotless.skip>
<spotless.version>2.30.0</spotless.version>
<spotless.java.googlejavaformat.version>1.7</spotless.java.googlejavaformat.version>
@@ -69,11 +67,6 @@
<artifactId>lance-core</artifactId>
<version>${lance-core.version}</version>
</dependency>
<dependency>
<groupId>org.lance</groupId>
<artifactId>lance-namespace-apache-client</artifactId>
<version>${lance-namespace.version}</version>
</dependency>
<dependency>
<groupId>org.apache.arrow</groupId>
<artifactId>arrow-vector</artifactId>
nodejs/__test__/remote.test.ts
+22 -135
View File
@@ -1,10 +1,8 @@
// SPDX-License-Identifier: Apache-2.0
// SPDX-FileCopyrightText: Copyright The LanceDB Authors
import * as fs from "fs";
import * as http from "http";
import { RequestListener } from "http";
import * as path from "path";
import * as tmp from "tmp";
import {
ClientConfig,
@@ -13,7 +11,6 @@ import {
MatchQuery,
TlsConfig,
connect,
tokenize,
} from "../lancedb";
import {
HeaderProvider,
@@ -231,70 +228,20 @@ describe("remote connection", () => {
);
});
it("sends the FTS posting block size to remote tables", async () => {
let createIndexBody: Record<string, unknown> | undefined;
await withMockDatabase(
(req, res) => {
const path = req.url ?? "";
if (path.endsWith("/describe/")) {
res.writeHead(200, { "Content-Type": "application/json" }).end(
JSON.stringify({
name: "t",
version: 1,
schema: {
fields: [
{ name: "text", type: { type: "string" }, nullable: false },
],
},
}),
);
return;
}
if (path.endsWith("/create_index/")) {
let raw = "";
req.on("data", (chunk) => {
raw += chunk;
});
req.on("end", () => {
createIndexBody = JSON.parse(raw);
res.writeHead(200).end();
});
return;
}
res.writeHead(404).end();
},
async (db) => {
const table = await db.openTable("t");
await table.createIndex("text", {
config: Index.fts({ blockSize: 256 }),
});
},
);
expect(createIndexBody?.["column"]).toBe("text");
expect(createIndexBody?.["index_type"]).toBe("FTS");
expect(createIndexBody?.["block_size"]).toBe(256);
});
it("resolves custom stop word sources before a remote request", async () => {
it("serializes FTS options and rejects unsupported remote use", async () => {
const tmpDir = tmp.dirSync({ unsafeCleanup: true });
try {
const localDb = await connect(path.join(tmpDir.name, "local-db"));
const localDb = await connect(tmpDir.name);
const stopWordsTable = await localDb.createTable("dictionary_table", [
{ word: "the" },
{ word: "" },
{ word: "the" },
]);
const stopWordsPath = path.join(tmpDir.name, "stop-words.txt");
fs.writeFileSync(stopWordsPath, "lance\n\nlance\n data \n", "utf8");
const requestBodies: Record<string, unknown>[] = [];
let createIndexBody: Record<string, unknown> | undefined;
let unexpectedRequestCount = 0;
await withMockDatabase(
(req, res) => {
if (req.url?.endsWith("/describe/")) {
const path = req.url ?? "";
if (path.endsWith("/describe/")) {
res.writeHead(200, { "Content-Type": "application/json" }).end(
JSON.stringify({
name: "t",
@@ -313,39 +260,26 @@ describe("remote connection", () => {
return;
}
if (req.url?.endsWith("/create_index/")) {
if (path.endsWith("/create_index/")) {
let raw = "";
req.on("data", (chunk) => {
raw += chunk;
});
req.on("end", () => {
requestBodies.push(JSON.parse(raw));
createIndexBody = JSON.parse(raw);
res.writeHead(200).end();
});
return;
}
unexpectedRequestCount += 1;
res.writeHead(404).end();
},
async (remoteDb) => {
const table = await remoteDb.openTable("t");
await table.createIndex("text", {
config: Index.fts({ removeStopWords: true }),
});
await table.createIndex("text", {
config: Index.fts({
removeStopWords: true,
customStopWords: [],
}),
});
await table.createIndex("text", {
config: Index.fts({
removeStopWords: true,
customStopWords: { source: "file", path: stopWordsPath },
}),
});
await table.createIndex("text", {
config: Index.fts({
blockSize: 256,
removeStopWords: true,
customStopWords: {
source: "table",
@@ -354,6 +288,7 @@ describe("remote connection", () => {
},
}),
});
await expect(
table.createIndex("text", {
config: Index.fts({
@@ -368,80 +303,32 @@ describe("remote connection", () => {
).rejects.toThrow(
"remote table sources cannot guarantee a complete snapshot",
);
await expect(
tokenize("the lance", {
removeStopWords: true,
customStopWords: {
source: "table",
table,
column: "text",
},
}),
table
.search(new MatchQuery("alph", "text", { fuzziness: 1 }))
.toArray(),
).rejects.toThrow(
"remote table sources cannot guarantee a complete snapshot",
"explicit fuzzy FTS queries are not supported for remote tables",
);
},
);
expect(requestBodies).toHaveLength(4);
expect(requestBodies[0]["custom_stop_words"]).toBeNull();
expect(requestBodies[1]["custom_stop_words"]).toEqual([]);
expect(requestBodies[2]["custom_stop_words"]).toEqual([
"lance",
" data ",
]);
expect(requestBodies[3]["custom_stop_words"]).toEqual(["the"]);
const serialized = JSON.stringify(requestBodies);
expect(serialized).not.toContain(stopWordsPath);
expect(createIndexBody?.["column"]).toBe("text");
expect(createIndexBody?.["index_type"]).toBe("FTS");
expect(createIndexBody?.["block_size"]).toBe(256);
expect(createIndexBody?.["custom_stop_words"]).toEqual(["the"]);
const serialized = JSON.stringify(createIndexBody);
expect(serialized).not.toContain("dictionary_table");
expect(serialized).not.toContain('"source"');
expect(serialized).not.toContain('"path"');
expect(serialized).not.toContain('"table"');
expect(serialized).not.toContain('"column":"word"');
expect(unexpectedRequestCount).toBe(0);
} finally {
tmpDir.removeCallback();
}
});
it("rejects explicit remote fuzzy FTS queries before the query request", async () => {
let requestCount = 0;
await withMockDatabase(
(req, res) => {
if (req.url?.endsWith("/describe/")) {
res.writeHead(200, { "Content-Type": "application/json" }).end(
JSON.stringify({
name: "t",
version: 1,
schema: {
fields: [
{
name: "text",
type: { type: "string" },
nullable: false,
},
],
},
}),
);
return;
}
requestCount += 1;
res.writeHead(500).end();
},
async (remoteDb) => {
const table = await remoteDb.openTable("t");
await expect(
table
.search(new MatchQuery("alph", "text", { fuzziness: 1 }))
.toArray(),
).rejects.toThrow(
"explicit fuzzy FTS queries are not supported for remote tables",
);
},
);
expect(requestCount).toBe(0);
});
it("diffs and merges remote branches", async () => {
const sampleDiff = {
fromBranch: "exp",
nodejs/__test__/table.test.ts
+21 -332
View File
@@ -42,7 +42,7 @@ import {
getRegistry,
register,
} from "../lancedb/embedding";
import { type CustomStopWordsSource, Index } from "../lancedb/indices";
import { Index } from "../lancedb/indices";
import {
BooleanQuery,
Occur,
@@ -2791,345 +2791,34 @@ describe("custom FTS stop words", () => {
];
}
test("preserves undefined, empty, and custom snapshots", async () => {
const data = [{ text: "the alpha" }, { text: "lance beta" }];
const builtin = await db.createTable("builtin", data);
await builtin.createIndex("text", {
config: Index.fts({
stem: false,
removeStopWords: true,
}),
});
expect(await stopWordsSnapshot(builtin)).toBeNull();
expect(
(await builtin.tokenize("the lance", { column: "text" })).map(
(token) => token.text,
),
).toEqual(["lance"]);
test.each<[string, string[] | undefined, string[] | null]>([
["undefined", undefined, null],
["empty", [], []],
["inline", ["lance"], ["lance"]],
])(
"passes %s stop words through the FTS options",
async (name, customStopWords, expected) => {
const table = await db.createTable(name, [{ text: "the lance" }]);
await table.createIndex("text", {
config: Index.fts({ removeStopWords: true, customStopWords }),
});
expect(await stopWordsSnapshot(table)).toEqual(expected);
},
);
const empty = await db.createTable("empty", data);
await empty.createIndex("text", {
config: Index.fts({
stem: false,
removeStopWords: true,
customStopWords: [],
}),
});
expect(await stopWordsSnapshot(empty)).toEqual([]);
expect(
(await empty.tokenize("the lance", { column: "text" })).map(
(token) => token.text,
),
).toEqual(["the", "lance"]);
const custom = await db.createTable("custom", data);
await custom.createIndex("text", {
config: Index.fts({
stem: false,
removeStopWords: true,
customStopWords: ["", "lance", "lance", " data "],
}),
});
expect(await stopWordsSnapshot(custom)).toEqual(["lance", " data "]);
expect(
(await custom.tokenize("the lance data", { column: "text" })).map(
(token) => token.text,
),
).toEqual(["the", "data"]);
await expect(
custom.search("the", "fts").limit(10).toArray(),
).resolves.toHaveLength(1);
await expect(
custom
.search(new MatchQuery("alpha", "text", { fuzziness: 1 }))
.toArray(),
).rejects.toThrow("uses a custom stop-word snapshot");
await expect(
custom
.search(new MatchQuery("alpha", "text", { fuzziness: 0 }))
.toArray(),
).resolves.toHaveLength(1);
await expect(
builtin
.search(new MatchQuery("alpha", "text", { fuzziness: 1 }))
.toArray(),
).resolves.toBeDefined();
custom.close();
const reopened = await db.openTable("custom");
expect(
(await reopened.tokenize("the lance data", { column: "text" })).map(
(token) => token.text,
),
).toEqual(["the", "data"]);
await expect(
reopened.search("the", "fts").limit(10).toArray(),
).resolves.toHaveLength(1);
const disabled = await db.createTable("disabled", data);
await disabled.createIndex("text", {
config: Index.fts({
stem: false,
removeStopWords: false,
customStopWords: ["lance"],
}),
});
expect(
(await disabled.tokenize("the lance data", { column: "text" })).map(
(token) => token.text,
),
).toEqual(["the", "lance", "data"]);
await expect(
disabled
.search(new MatchQuery("alpha", "text", { fuzziness: 1 }))
.toArray(),
).resolves.toHaveLength(1);
});
test("uses inline, file, and table sources in standalone tokenize", async () => {
test("passes a file source to standalone tokenize", async () => {
const stopWordsPath = path.join(tmpDir.name, "stop-words.txt");
fs.writeFileSync(stopWordsPath, "the\n\nlance\nthe\n", "utf8");
const stopWordsTable = await db.createTable("stop_words", [
{ word: "the" },
{ word: "" },
{ word: "lance" },
{ word: "the" },
]);
const common = {
baseTokenizer: "simple" as const,
stem: false,
removeStopWords: true,
};
const texts = async (customStopWords?: CustomStopWordsSource) =>
(
await tokenize("the lance data", {
...common,
customStopWords,
})
).map((token) => token.text);
expect(await texts()).toEqual(["lance", "data"]);
expect(await texts([])).toEqual(["the", "lance", "data"]);
expect(await texts(["lance"])).toEqual(["the", "data"]);
expect(await texts({ source: "file", path: stopWordsPath })).toEqual([
"data",
]);
expect(
await texts({
source: "table",
table: stopWordsTable,
column: "word",
}),
).toEqual(["data"]);
fs.writeFileSync(stopWordsPath, "lance\n", "utf8");
expect(
(
await tokenize("the lance data", {
...common,
removeStopWords: false,
customStopWords: ["lance"],
baseTokenizer: "simple",
stem: false,
removeStopWords: true,
customStopWords: { source: "file", path: stopWordsPath },
})
).map((token) => token.text),
).toEqual(["the", "lance", "data"]);
});
test("snapshots file and table sources when creating an index", async () => {
const stopWordsPath = path.join(tmpDir.name, "index-stop-words.txt");
fs.writeFileSync(stopWordsPath, "lance\nlance\n\n data \n", "utf8");
const fileTarget = await db.createTable("file_target", [
{ text: "the lance data" },
]);
await fileTarget.createIndex("text", {
config: Index.fts({
stem: false,
removeStopWords: true,
customStopWords: { source: "file", path: stopWordsPath },
}),
});
expect(await stopWordsSnapshot(fileTarget)).toEqual(["lance", " data "]);
fs.writeFileSync(stopWordsPath, "the\n", "utf8");
fileTarget.close();
const reopenedFileTarget = await db.openTable("file_target");
expect(
(
await reopenedFileTarget.tokenize("the lance data", { column: "text" })
).map((token) => token.text),
).toEqual(["the", "data"]);
const sourceTable = await db.createTable("table_stop_words", [
{ word: "lance" },
{ word: "" },
{ word: "lance" },
]);
const tableTarget = await db.createTable("table_target", [
{ text: "the lance data" },
]);
await tableTarget.createIndex("text", {
config: Index.fts({
stem: false,
removeStopWords: true,
customStopWords: {
source: "table",
table: sourceTable,
column: "word",
},
}),
});
expect(await stopWordsSnapshot(tableTarget)).toEqual(["lance"]);
await sourceTable.add([{ word: "the" }]);
expect(
(await tableTarget.tokenize("the lance data", { column: "text" })).map(
(token) => token.text,
),
).toEqual(["the", "data"]);
});
test("rejects malformed sources and source read errors", async () => {
expect(() =>
Index.fts({
customStopWords: ["valid", 42] as unknown as string[],
}),
).toThrow("customStopWords[1] must be a string");
expect(() =>
Index.fts({ customStopWords: null as unknown as string[] }),
).toThrow("customStopWords must be");
expect(() =>
Index.fts({
customStopWords: {
source: "file",
path: "words.txt",
column: "word",
} as never,
}),
).toThrow("mutually exclusive");
expect(() =>
Index.fts({
customStopWords: { source: "file", path: "" },
}),
).toThrow("non-empty string 'path'");
expect(() =>
Index.fts({
customStopWords: {
source: "table",
table: {} as Table,
column: "",
},
}),
).toThrow("non-empty string 'column'");
await expect(
tokenize("hello", {
removeStopWords: true,
customStopWords: {
source: "file",
path: path.join(tmpDir.name, "missing.txt"),
},
}),
).rejects.toThrow("failed to read custom stop words file");
const reusableConfig = Index.fts({
stem: false,
removeStopWords: true,
customStopWords: {
source: "file",
path: path.join(tmpDir.name, "retry-stop-words.txt"),
},
});
const retryTarget = await db.createTable("retry_target", [
{ text: "hello world" },
]);
await expect(
retryTarget.createIndex("text", { config: reusableConfig }),
).rejects.toThrow("failed to read custom stop words file");
fs.writeFileSync(
path.join(tmpDir.name, "retry-stop-words.txt"),
"hello\n",
"utf8",
);
await expect(
retryTarget.createIndex("text", { config: reusableConfig }),
).resolves.toBeUndefined();
const invalidUtf8Path = path.join(tmpDir.name, "invalid-utf8.txt");
fs.writeFileSync(invalidUtf8Path, Buffer.from([0xff, 0xfe]));
await expect(
tokenize("hello", {
removeStopWords: true,
customStopWords: { source: "file", path: invalidUtf8Path },
}),
).rejects.toThrow("is not valid UTF-8");
const words = await db.createTable("error_words", [{ word: "lance" }]);
await expect(
tokenize("hello", {
removeStopWords: true,
customStopWords: {
source: "table",
table: words,
column: "missing",
},
}),
).rejects.toThrow("custom stop words column");
const numbers = await db.createTable("number_words", [{ word: 1 }]);
await expect(
tokenize("hello", {
removeStopWords: true,
customStopWords: {
source: "table",
table: numbers,
column: "word",
},
}),
).rejects.toThrow("must have type Utf8");
const nullableSchema = new arrow.Schema([
new arrow.Field("word", new arrow.Utf8(), true),
]);
const nullableWords = await db.createTable(
"nullable_words",
[{ word: "lance" }, { word: null }],
{ schema: nullableSchema },
);
await expect(
tokenize("hello", {
removeStopWords: true,
customStopWords: {
source: "table",
table: nullableWords,
column: "word",
},
}),
).rejects.toThrow("contains NULL");
const target = await db.createTable("closed_source_target", [
{ text: "hello" },
]);
words.close();
await expect(
target.createIndex("text", {
config: Index.fts({
removeStopWords: true,
customStopWords: {
source: "table",
table: words,
column: "word",
},
}),
}),
).rejects.toThrow("is closed; open the table");
await expect(
target.createIndex("text", {
config: Index.fts({
removeStopWords: true,
customStopWords: {
source: "table",
table: {} as Table,
column: "word",
},
}),
}),
).rejects.toThrow("created by this @lancedb/lancedb package instance");
});
});
nodejs/examples/full_text_search.test.ts
+1 -1
View File
@@ -48,4 +48,4 @@ test("full text search", async () => {
expect(result.length).toBe(10);
// --8<-- [end:full_text_search]
});
}, 30_000);
}, 10_000);
python/python/tests/docs/test_search.py
+13 -26
View File
@@ -358,38 +358,25 @@ def test_fts_custom_stop_word_sources(tmp_path):
"documents",
data=[{"text": "the lance database"}, {"text": "searchable data"}],
)
# Inline values
documents.create_index(
"text",
config=FTS(stem=False, custom_stop_words=["lance", "searchable"]),
)
# A strict UTF-8 file with one stop word per line
stop_words_path = tmp_path / "stop-words.txt"
stop_words_path.write_text("lance\nsearchable\n", encoding="utf-8")
documents.create_index(
"text",
config=FTS(
stem=False,
custom_stop_words=FtsStopWordsFile(stop_words_path),
),
replace=True,
)
# A string column from another open LanceDB table
stop_words = db.create_table(
"stop_words",
data=[{"word": "lance"}, {"word": "searchable"}],
)
documents.create_index(
"text",
config=FTS(
stem=False,
custom_stop_words=FtsStopWordsTable(stop_words, "word"),
),
replace=True,
)
# Inline values, a strict UTF-8 newline-delimited file, or a local table column.
sources = [
["lance", "searchable"],
FtsStopWordsFile(stop_words_path),
FtsStopWordsTable(stop_words, "word"),
]
for source in sources:
documents.create_index(
"text",
config=FTS(stem=False, custom_stop_words=source),
replace=True,
)
# --8<-- [end:fts_custom_stop_words]
tokens = documents.tokenize("the lance database", column="text")
python/python/tests/test_fts.py
+23 -263
View File
@@ -244,21 +244,19 @@ def test_create_inverted_index_rejects_invalid_block_size(table):
@pytest.mark.parametrize(
("custom_stop_words", "expected_the", "expected_lance"),
("custom_stop_words", "expected"),
[
(None, 0, 2),
([], 1, 2),
(["lance"], 1, 0),
(None, ["lance", "data"]),
([], ["the", "lance", "data"]),
(["lance"], ["the", "data"]),
],
ids=["builtin", "explicit-empty", "custom"],
)
def test_custom_stop_words_inline_none_vs_empty(
tmp_path, custom_stop_words, expected_the, expected_lance
):
def test_custom_stop_words_inline_none_vs_empty(tmp_path, custom_stop_words, expected):
db = ldb.connect(tmp_path)
table = db.create_table(
"documents",
data=pa.table({"text": ["the lance database", "lance data"]}),
data=pa.table({"text": ["the lance data"]}),
)
table.create_index(
"text",
@@ -266,92 +264,23 @@ def test_custom_stop_words_inline_none_vs_empty(
)
assert (
len(table.search("the", query_type="fts").limit(10).to_list()) == expected_the
table.list_indices()[0].index_details["custom_stop_words"] == custom_stop_words
)
assert (
len(table.search("lance", query_type="fts").limit(10).to_list())
== expected_lance
)
details = table.list_indices()[0].index_details
assert details["custom_stop_words"] == custom_stop_words
def test_custom_stop_words_are_inactive_when_removal_is_disabled(tmp_path):
db = ldb.connect(tmp_path)
table = db.create_table(
"documents",
data=pa.table({"text": ["the lance database", "lance data"]}),
)
table.create_index(
"text",
config=FTS(
stem=False,
remove_stop_words=False,
custom_stop_words=["lance"],
),
)
assert len(table.search("lance", query_type="fts").limit(10).to_list()) == 2
tokens = table.tokenize("the lance data", column="text")
assert [token.text for token in tokens] == ["the", "lance", "data"]
assert [token.text for token in tokens] == expected
def test_custom_stop_words_file_is_snapshotted_and_reopened(tmp_path):
def test_tokenize_custom_stop_words_file(tmp_path):
stop_words_path = tmp_path / "stop-words.txt"
stop_words_path.write_bytes(
"lance\r\n\r\nlance\r\nKeep Case\r\n spaced \r\n".encode()
)
db = ldb.connect(tmp_path / "db")
table = db.create_table(
"documents",
data=pa.table({"text": ["the lance database", "lance data"]}),
)
table.create_index(
"text",
config=FTS(
stem=False,
custom_stop_words=FtsStopWordsFile(stop_words_path),
),
stop_words_path.write_text("lance\n", encoding="utf-8")
tokens = ldb.tokenize(
"the lance data",
stem=False,
custom_stop_words=FtsStopWordsFile(stop_words_path),
)
stop_words_path.write_text("the\n", encoding="utf-8")
reopened = db.open_table("documents")
details = reopened.list_indices()[0].index_details
assert details["custom_stop_words"] == ["lance", "Keep Case", " spaced "]
tokens = reopened.tokenize("the lance data", column="text")
assert [token.text for token in tokens] == ["the", "data"]
assert len(reopened.search("the", query_type="fts").limit(10).to_list()) == 1
assert len(reopened.search("lance", query_type="fts").limit(10).to_list()) == 0
def test_custom_stop_words_table_is_snapshotted(tmp_path):
db = ldb.connect(tmp_path)
stop_words = db.create_table(
"stop_words",
data=pa.table({"word": ["lance", "", "lance", "data"]}),
)
documents = db.create_table(
"documents",
data=pa.table({"text": ["the lance database", "lance data"]}),
)
documents.create_index(
"text",
config=FTS(
stem=False,
custom_stop_words=FtsStopWordsTable(stop_words, "word"),
),
)
db.drop_table("stop_words")
reopened = db.open_table("documents")
assert reopened.list_indices()[0].index_details["custom_stop_words"] == [
"lance",
"data",
]
tokens = reopened.tokenize("the lance data", column="text")
assert [token.text for token in tokens] == ["the"]
@pytest.mark.asyncio
@@ -359,11 +288,11 @@ async def test_custom_stop_words_table_async(tmp_path):
db = await ldb.connect_async(tmp_path)
stop_words = await db.create_table(
"stop_words",
data=pa.table({"word": ["lance", "", "lance"]}),
data=pa.table({"word": ["lance"]}),
)
documents = await db.create_table(
"documents",
data=pa.table({"text": ["the lance database", "lance data"]}),
data=pa.table({"text": ["the lance data"]}),
)
await documents.create_index(
@@ -378,18 +307,7 @@ async def test_custom_stop_words_table_async(tmp_path):
assert [token.text for token in tokens] == ["the", "data"]
def test_custom_stop_words_deprecated_create_fts_index(table):
with pytest.warns(DeprecationWarning, match="create_fts_index"):
table.create_fts_index(
"text",
stem=False,
custom_stop_words=["puppy"],
)
assert len(table.search("puppy").limit(5).to_list()) == 0
def test_custom_stop_words_tokenizer_name_compatibility_is_persisted(table):
def test_custom_stop_words_legacy_tokenizer_name(table):
with pytest.warns(DeprecationWarning, match="create_fts_index"):
table.create_fts_index(
"text",
@@ -400,117 +318,13 @@ def test_custom_stop_words_tokenizer_name_compatibility_is_persisted(table):
details = table.list_indices()[0].index_details
assert details["custom_stop_words"] == ["puppy"]
assert details["remove_stop_words"] is False
assert len(table.search("puppy").limit(5).to_list()) == 5
assert [token.text for token in table.tokenize("puppy runs", column="text")] == [
"puppy",
"runs",
]
@pytest.mark.parametrize(
("source_factory", "match"),
[
(lambda _tmp, _table: "not-a-sequence", "not a string"),
(lambda _tmp, _table: ["valid", 42], r"custom_stop_words\[1\]"),
(
lambda tmp, _table: FtsStopWordsFile(tmp / "missing.txt"),
"failed to read custom stop words file",
),
(
lambda _tmp, _table: FtsStopWordsFile(""),
"path must be a non-empty string path",
),
(
lambda _tmp, table: FtsStopWordsTable(table, "missing"),
"failed to resolve custom stop words column",
),
(
lambda _tmp, table: FtsStopWordsTable(table, ""),
"column must be a non-empty string",
),
(
lambda _tmp, table: FtsStopWordsTable(table, "number"),
"must have type Utf8",
),
],
)
def test_custom_stop_words_validation_errors(tmp_path, source_factory, match):
db = ldb.connect(tmp_path / "db")
documents = db.create_table(
"documents",
data=pa.table(
{
"text": ["the lance database"],
"number": [1],
}
),
)
with pytest.raises((TypeError, ValueError), match=match):
documents.create_index(
"text",
config=FTS(custom_stop_words=source_factory(tmp_path, documents)),
)
def test_custom_stop_words_file_rejects_invalid_utf8(tmp_path):
stop_words_path = tmp_path / "stop-words.txt"
stop_words_path.write_bytes(b"valid\n\xff")
db = ldb.connect(tmp_path / "db")
documents = db.create_table(
"documents",
data=pa.table({"text": ["the lance database"]}),
)
with pytest.raises(ValueError, match="not valid UTF-8"):
documents.create_index(
"text",
config=FTS(custom_stop_words=FtsStopWordsFile(stop_words_path)),
)
@pytest.mark.parametrize(
("values", "match"),
[
(pa.array(["valid", None]), "contains NULL"),
(pa.array([1, 2]), "must have type Utf8"),
],
)
def test_custom_stop_words_table_rejects_invalid_values(tmp_path, values, match):
db = ldb.connect(tmp_path)
stop_words = db.create_table("stop_words", data=pa.table({"word": values}))
documents = db.create_table(
"documents",
data=pa.table({"text": ["the lance database"]}),
)
with pytest.raises(ValueError, match=match):
documents.create_index(
"text",
config=FTS(
custom_stop_words=FtsStopWordsTable(stop_words, "word"),
),
)
def test_custom_stop_words_table_rejects_closed_table(tmp_path):
db = ldb.connect(tmp_path)
stop_words = db.create_table(
"stop_words",
data=pa.table({"word": ["lance"]}),
)
documents = db.create_table(
"documents",
data=pa.table({"text": ["the lance database"]}),
)
stop_words._table.close()
with pytest.raises(RuntimeError, match="Table stop_words is closed"):
documents.create_index(
"text",
config=FTS(
custom_stop_words=FtsStopWordsTable(stop_words, "word"),
),
def test_custom_stop_words_rejects_non_string_binding_value():
with pytest.raises(TypeError, match=r"custom_stop_words\[1\]"):
ldb.tokenize(
"the lance data",
custom_stop_words=["lance", 42],
)
@@ -1165,60 +979,6 @@ def test_tokenize_uses_explicit_icu_tokenizer():
]
@pytest.mark.parametrize(
("custom_stop_words", "expected"),
[
(None, ["lance", "data"]),
([], ["the", "lance", "data"]),
(["lance"], ["the", "data"]),
],
ids=["builtin", "explicit-empty", "custom"],
)
def test_tokenize_custom_stop_words_none_vs_empty(custom_stop_words, expected):
tokens = ldb.tokenize(
"the lance data",
stem=False,
custom_stop_words=custom_stop_words,
)
assert [token.text for token in tokens] == expected
def test_tokenize_custom_stop_words_file_and_table(tmp_path):
stop_words_path = tmp_path / "stop-words.txt"
stop_words_path.write_text("lance\nlance\n\n", encoding="utf-8")
file_tokens = ldb.tokenize(
"the lance data",
stem=False,
custom_stop_words=FtsStopWordsFile(stop_words_path),
)
db = ldb.connect(tmp_path / "db")
stop_words = db.create_table(
"stop_words",
data=pa.table({"word": ["lance", "", "lance"]}),
)
table_tokens = ldb.tokenize(
"the lance data",
stem=False,
custom_stop_words=FtsStopWordsTable(stop_words, "word"),
)
assert [token.text for token in file_tokens] == ["the", "data"]
assert [token.text for token in table_tokens] == ["the", "data"]
def test_tokenize_custom_stop_words_disabled():
tokens = ldb.tokenize(
"the lance data",
stem=False,
remove_stop_words=False,
custom_stop_words=["lance"],
)
assert [token.text for token in tokens] == ["the", "lance", "data"]
def test_fts_on_list(mem_db: DBConnection):
data = pa.table(
{
python/python/tests/test_remote_db.py
+6 -84
View File
@@ -17,7 +17,7 @@ from packaging.version import Version
import lancedb
from lancedb.conftest import MockTextEmbeddingFunction
from lancedb.query import ColumnOrdering, MatchQuery
from lancedb.query import ColumnOrdering
from lancedb.remote import ClientConfig
from lancedb.remote.errors import HttpError, RetryError
import pytest
@@ -881,11 +881,6 @@ def test_remote_create_index_new_api(tmp_path):
with mock_lancedb_connection(handler) as db:
table = db.create_table("test", [{"id": 1}])
local_db = lancedb.connect(tmp_path / "local-source")
local_stop_words = local_db.create_table(
"stop_words",
data=[{"word": "table-word"}, {"word": ""}, {"word": "table-word"}],
)
# New API: column-first, config= kwarg. Should NOT emit DeprecationWarning.
import warnings as _warnings
@@ -895,25 +890,11 @@ def test_remote_create_index_new_api(tmp_path):
table.create_index("category", config=BTree())
table.create_index("text", config=FTS(block_size=256))
table.create_index("text", config=FTS(custom_stop_words=[]))
table.create_index(
"text",
config=FTS(
custom_stop_words=["cloud", "", "cloud", "Keep Case"],
),
)
stop_words_path = tmp_path / "stop-words.txt"
stop_words_path.write_text("file-word\n\nfile-word\n", encoding="utf-8")
stop_words_path.write_text("file-word\n", encoding="utf-8")
table.create_index(
"text",
config=FTS(
custom_stop_words=FtsStopWordsFile(stop_words_path),
),
)
table.create_index(
"text",
config=FTS(
custom_stop_words=FtsStopWordsTable(local_stop_words, "word"),
),
config=FTS(custom_stop_words=FtsStopWordsFile(stop_words_path)),
)
# IvfRq via new API
table.create_index("vector", config=IvfRq(distance_type="l2"))
@@ -936,51 +917,21 @@ def test_remote_create_index_new_api(tmp_path):
custom_stop_words=FtsStopWordsTable(table, "text"),
),
)
with pytest.raises(
ValueError,
match="remote table sources cannot guarantee a complete snapshot",
):
lancedb.tokenize(
"the cloud",
custom_stop_words=FtsStopWordsTable(table, "text"),
)
assert len(received_requests) == 9
assert len(received_requests) == 7
assert [req["column"] for req in received_requests] == [
"vector",
"category",
"text",
"text",
"text",
"text",
"text",
"vector",
"vector",
]
assert received_requests[2]["block_size"] == 256
assert received_requests[2].get("custom_stop_words") is None
assert received_requests[3]["custom_stop_words"] == []
assert received_requests[4]["custom_stop_words"] == ["cloud", "Keep Case"]
assert received_requests[5]["custom_stop_words"] == ["file-word"]
assert received_requests[6]["custom_stop_words"] == ["table-word"]
assert "path" not in json.dumps(received_requests[5])
assert "table" not in json.dumps(received_requests[5])
assert "table-word" in json.dumps(received_requests[6])
assert not {"source", "table", "path"} & received_requests[6].keys()
# A forked/unpickled RemoteTable may have no live handle. Stop-word
# validation must fail without trying to reopen through LOOP.run from
# the LOOP thread (which would deadlock).
table._table_handle = None
with pytest.raises(
ValueError,
match="remote table sources cannot guarantee a complete snapshot",
):
lancedb.tokenize(
"the cloud",
custom_stop_words=FtsStopWordsTable(table, "text"),
)
assert table._table_handle is None
assert received_requests[4]["custom_stop_words"] == ["file-word"]
assert str(stop_words_path) not in json.dumps(received_requests[4])
def test_table_wait_for_index_timeout():
@@ -1377,35 +1328,6 @@ def test_query_sync_fts():
)
def test_query_sync_fts_fuzziness_boundary():
query_requests = 0
def handler(body):
nonlocal query_requests
query_requests += 1
assert body["full_text_query"]["query"]["match"]["fuzziness"] == 0
return pa.table({"id": [1]})
with query_test_table(handler, server_version=Version("0.3.0")) as table:
table.search(
MatchQuery("puppy", "text", fuzziness=0),
query_type="fts",
).to_list()
assert query_requests == 1
with pytest.raises(
ValueError,
match="explicit fuzzy FTS queries are not supported.*fuzziness=0",
):
table.search(
MatchQuery("puppy", "text", fuzziness=1),
query_type="fts",
).to_list()
# Positive fuzziness fails before issuing another remote query.
assert query_requests == 1
def test_query_sync_hybrid():
def handler(body):
if "full_text_query" in body:
rust/lancedb/src/index/scalar.rs
+10
View File
@@ -358,6 +358,7 @@ mod tests {
use arrow_array::{
ArrayRef, Int32Array, LargeStringArray, RecordBatch, RecordBatchIterator, StringArray,
StringViewArray,
};
use arrow_schema::{DataType, Field, Schema};
use lance_namespace::LanceNamespace;
@@ -446,12 +447,14 @@ mod tests {
Arc::new(Schema::new(vec![
Field::new("word", DataType::Utf8, false),
Field::new("large_word", DataType::LargeUtf8, false),
Field::new("view_word", DataType::Utf8View, false),
Field::new("number", DataType::Int32, false),
Field::new("nullable_word", DataType::Utf8, true),
])),
vec![
Arc::new(StringArray::from(vec!["cat", "", "cat", " CAT "])) as ArrayRef,
Arc::new(LargeStringArray::from(vec!["large", "large", "", "other"])),
Arc::new(StringViewArray::from(vec!["view", "view", "", "other"])),
Arc::new(Int32Array::from(vec![1, 2, 3, 4])),
Arc::new(StringArray::from(vec![
Some("valid"),
@@ -482,6 +485,13 @@ mod tests {
.unwrap(),
vec!["large", "other"]
);
assert_eq!(
FtsStopWordsSource::table(table.clone(), "view_word")
.resolve()
.await
.unwrap(),
vec!["view", "other"]
);
let error = FtsStopWordsSource::table(table.clone(), "nullable_word")
.resolve()
rust/lancedb/src/remote/table.rs
+35 -3
View File
@@ -4726,13 +4726,18 @@ mod tests {
"/v1/table/my_table/create_index/" => {
let body = request.body().unwrap().as_bytes().unwrap();
let body: serde_json::Value = serde_json::from_slice(body).unwrap();
let request_number = create_requests_ref.fetch_add(1, Ordering::SeqCst);
let mut expected =
serde_json::to_value(InvertedIndexParams::default()).unwrap();
expected["column"] = "text".into();
expected[INDEX_TYPE_KEY] = "FTS".into();
expected["custom_stop_words"] = json!(["cat", " dog ", "CAT"]);
expected["custom_stop_words"] = match request_number {
0 | 1 => json!(["cat", " dog ", "CAT"]),
2 => json!([]),
3 => json!(["from-table"]),
_ => panic!("unexpected create-index request {request_number}"),
};
assert_eq!(body, expected);
create_requests_ref.fetch_add(1, Ordering::SeqCst);
http::Response::builder()
.status(200)
.body("{}".to_string())
@@ -4768,7 +4773,34 @@ mod tests {
.await
.unwrap();
assert_eq!(create_requests.load(Ordering::SeqCst), 2);
table
.create_index(&["text"], Index::FTS(Default::default()))
.custom_stop_words(FtsStopWordsSource::inline(Vec::new()))
.unwrap()
.execute()
.await
.unwrap();
let source_conn = crate::connect("memory://").execute().await.unwrap();
let source_batch = RecordBatch::try_new(
Arc::new(Schema::new(vec![Field::new("word", DataType::Utf8, false)])),
vec![Arc::new(StringArray::from(vec!["from-table"]))],
)
.unwrap();
let source_table = source_conn
.create_table("stop_words", source_batch)
.execute()
.await
.unwrap();
table
.create_index(&["text"], Index::FTS(Default::default()))
.custom_stop_words(FtsStopWordsSource::table(source_table, "word"))
.unwrap()
.execute()
.await
.unwrap();
assert_eq!(create_requests.load(Ordering::SeqCst), 4);
}
#[tokio::test]
rust/lancedb/src/table.rs
+15
View File
@@ -3613,6 +3613,7 @@ mod tests {
0,
3,
true,
true,
),
(
"empty_custom_stop_words",
@@ -3621,6 +3622,7 @@ mod tests {
2,
3,
true,
true,
),
(
"custom_stop_words",
@@ -3629,6 +3631,7 @@ mod tests {
2,
0,
true,
true,
),
(
"untrained_custom_stop_words",
@@ -3637,6 +3640,16 @@ mod tests {
2,
0,
false,
true,
),
(
"disabled_custom_stop_words",
Some(vec!["lance".to_string()]),
vec!["the".to_string(), "lance".to_string(), "data".to_string()],
2,
3,
true,
false,
),
];
@@ -3647,6 +3660,7 @@ mod tests {
expected_the_matches,
expected_lance_matches,
train,
remove_stop_words,
) in cases
{
let schema = Arc::new(Schema::new(vec![Field::new("text", DataType::Utf8, false)]));
@@ -3666,6 +3680,7 @@ mod tests {
.unwrap();
let params = crate::index::scalar::FtsIndexBuilder::default()
.stem(false)
.remove_stop_words(remove_stop_words)
.custom_stop_words(custom_stop_words.clone());
table
.create_index(&["text"], Index::FTS(params))