lancedb/docs/web/curation/inspect-datasets.mdx

---
title: "Inspect Datasets"
sidebarTitle: "Inspect Datasets"
description: "Query a table, a HuggingFace dataset, or a dataset in object storage the same way, with plain SELECT, WHERE, projection and LIMIT."
icon: database
keywords: ["select", "where", "filter", "projection", "limit", "huggingface", "storage integration", "sql", "inspect"]
availability:
  embedded: unavailable
  enterprise: available
---

The same `SELECT` statement inspects three kinds of source: a table
already in your database, a HuggingFace dataset, or a dataset in object
storage -- once a [storage integration](/management/storage-integration)
covers the latter two. Nothing is downloaded or imported until you choose
to import it.

## The basics: project, filter, limit {#basics}

A table name, a HuggingFace URI, and an object-storage URI all work the
same way after `FROM`:

<CodeGroup>
```sql SQL icon="database"
SELECT id, text
FROM documents
WHERE text IS NOT NULL
LIMIT 100
```
</CodeGroup>

- **Project** -- name the columns you need (`id, text`) instead of
  `SELECT *`, especially for a source with large or many columns.
- **Filter** -- `WHERE` narrows rows before they come back; combine
  predicates with `AND`/`OR` the same as any other table.
- **Limit** -- `LIMIT n` caps how many rows return. It bounds the result,
  not the read: a remote source is still fetched at file or row-group
  granularity, so a small `LIMIT` on a source with large files or row
  groups can still touch more data than the row count suggests.

The rest of this page swaps `documents` for a HuggingFace or
object-storage source -- the `SELECT`/`WHERE`/`LIMIT` syntax above doesn't
change.

## Inspect a HuggingFace dataset {#huggingface}

A single-quoted HuggingFace dataset URI works directly in a SQL `FROM`
clause, the same as a table name -- no separate download, no intermediate
file on disk:

<CodeGroup>
```sql SQL icon="database"
SELECT id, text
FROM 'hf://datasets/owner/repo/data/train-*.parquet'
WHERE text IS NOT NULL
LIMIT 100
```
</CodeGroup>

<CodeGroup>
```python Python icon="python"
reader = db.execute_query("""
    SELECT id, text
    FROM 'hf://datasets/owner/repo/data/train-*.parquet'
    WHERE text IS NOT NULL
    LIMIT 100
""")
sample = reader.read_all().to_pylist()
```
</CodeGroup>

The URI is `hf://datasets/<owner>/<repo>[@<revision>]/<path>`. `<path>` can
be one Parquet file, a directory, or a glob: `*` matches within one
directory, `**` matches across directories, and a repository root selects
every Parquet file beneath it. Select a dataset configuration or split by
including it in the path. The default revision is `main`; each statement
resolves the revision to a commit once, so every worker reads the same
snapshot. This source reads Parquet files -- it does not run a dataset's
own generation scripts, and it does not compute embeddings or build
indexes, so [chunk](/curation/chunking), embed, or index the result as a
separate step, the same as data from any other source.

<Note>
Files matched by one statement must have compatible schemas, and an
unmatched path is an error rather than a silent fallback to a different
split. Nested Parquet values come through as-is; a floating-point list is
not automatically converted into a fixed-size vector column.
</Note>

## Inspect a dataset in object storage {#object-storage}

An `s3://`, `gs://` or `az://` prefix covered by a storage integration
reads an existing **Lance** dataset at that prefix, the same way as a
managed table:

<CodeGroup>
```sql SQL icon="database"
SELECT id, text
FROM 's3://customer-data/raw/documents.lance'
WHERE text IS NOT NULL
LIMIT 100
```
</CodeGroup>

This reads a dataset another process already wrote in Lance format, not
an arbitrary file format -- there is no Parquet-style reader for cloud
storage the way there is for `hf://`. See
[Storage Integration](/sql/ddl/storage-integration#read-import) for every
provider's setup.

## Access a private repository or bucket {#access}

A private HuggingFace repository, one subject to HuggingFace's
authenticated rate limits, or any non-public bucket needs a
[storage integration](/management/storage-integration) covering it. For
HuggingFace that integration is bound to a token
[Secret](/management/secret); for a non-public S3 bucket it can instead be
an IAM role integration, which needs no Secret at all -- see
[AWS S3](/ingestion/define-storage-integration#aws).

<CodeGroup>
```python Python icon="python"
db.create_secret("hf_token", "<your-hugging-face-token>")

# create_storage_integration is not in a released SDK yet; use the SQL
# form below until it ships.
db.execute_query("""
    CREATE STORAGE INTEGRATION private_hub
        STORAGE_PROVIDER = 'HUGGINGFACE'
        STORAGE_PREFIX = 'hf://datasets/owner/private/data/'
        HUGGINGFACE_TOKEN_SECRET = hf_token
""").read_all()

reader = db.execute_query(
    "SELECT * FROM 'hf://datasets/owner/private/data/*.parquet' LIMIT 10"
)
```

```sql SQL icon="database"
CREATE SECRET hf_token TYPE = GENERIC_STRING SECRET_STRING = '<your-hugging-face-token>'

CREATE STORAGE INTEGRATION private_hub
    STORAGE_PROVIDER = 'HUGGINGFACE'
    STORAGE_PREFIX = 'hf://datasets/owner/private/data/'
    HUGGINGFACE_TOKEN_SECRET = hf_token

SELECT * FROM 'hf://datasets/owner/private/data/*.parquet' LIMIT 10
```
</CodeGroup>

{/*
Draft for the storage-integration SDK release (#4618, #4775). Keep this
section unpublished until create_storage_integration ships in a release; it
is not in v0.41.0.

<CodeGroup>
```python Python icon="python"
db.create_secret("hf_token", "<your-hugging-face-token>")

db.create_storage_integration(
    "private_hub", "hf://datasets/owner/private/data/",
    storage_provider="HUGGINGFACE",
    huggingface_token_secret="hf_token",
)

reader = db.execute_query(
    "SELECT * FROM 'hf://datasets/owner/private/data/*.parquet' LIMIT 10"
)
```
</CodeGroup>
*/}

A query never names the integration: the planner matches the most specific
`STORAGE_PREFIX` that covers the URI being read, across every integration
the caller has `USAGE` on. A HuggingFace integration always needs
`HUGGINGFACE_TOKEN_SECRET`; creating one without it is rejected. For public
access, register no integration for that prefix at all -- an `hf://` read
that matches no integration is anonymous, with no environment-variable
fallback. See
[Storage Integration](/management/storage-integration) for every provider
and the full credential model.

## Import into a table {#import}

Once you've inspected a source and it looks right, `CREATE TABLE ... AS
SELECT` imports it:

<CodeGroup>
```sql SQL icon="database"
CREATE TABLE imported AS
SELECT *
FROM 'hf://datasets/owner/repo/data/train-*.parquet'
```
</CodeGroup>

<CodeGroup>
```python Python icon="python"
db.execute_query("""
    CREATE TABLE imported AS
    SELECT * FROM 'hf://datasets/owner/repo/data/train-*.parquet'
""").read_all()
table = db.open_table("imported")
```
</CodeGroup>

This `CREATE TABLE AS SELECT` runs as a distributed import when the
deployment has an available plan-executor fleet: the planner assigns
disjoint file and row-group ranges to workers, each worker writes
uncommitted Lance files directly to the destination, and a coordinator
commits the completed table from their transaction metadata once every
worker finishes. A query with a global limit, aggregation or join keeps
its usual coordination; a small import can run as one local task without a
fleet. The table becomes queryable only after that final commit --
nothing partial is ever visible, and a failed or cancelled import can
resume the same durable job from its retained attempt rather than
restarting from nothing. To start over after a failed one-off import
instead, drop the partial table and rerun the statement.

<Note>
Importing requires a native database with `database_manifest_enabled`.
Combining a HuggingFace source with a WAL-backed destination table in one
CTAS is rejected, since the import cannot publish a base-only read that
omits the destination's unflushed data.
</Note>

## What to read next {#next}

<CardGroup cols={2}>
  <Card title="Storage Integration" icon="cloud" href="/management/storage-integration">
    The credential object behind private repository access.
  </Card>
  <Card title="Create a Function" icon="code" href="/curation/create-function">
    Add embeddings or derived columns to the imported data.
  </Card>
  <Card title="Chunk records into multiple rows" icon="diagram-subtask" href="/curation/chunking">
    Split long text or media fields after importing.
  </Card>
  <Card title="Table" icon="table" href="/sql/ddl/table#create">
    `CREATE TABLE AS SELECT` syntax and options in full.
  </Card>
</CardGroup>