feat: use oauth2 crate for OAuth with configurable client auth (#4181)

Stacked on #4173 (`jack/restore-oidc-flows`, base branch mirrored to
this repo so the diff shows only this change); context from review:
https://github.com/lancedb/lancedb/pull/4173#issuecomment-5674048100.
Rebase to `main` once #4173 and #4179 merge.

## What moved to the `oauth2` crate (5.0, no default features)

- Authorization URL generation and CSRF state (`authorize_url`,
`CsrfToken`)
- PKCE S256 challenge/verifier generation and code exchange
- Client-credentials, authorization-code, refresh-token, and device-code
grant request construction
- Device authorization request and the device token polling loop
(`authorization_pending`, `slow_down` +5s, expiry deadline, denial,
network backoff capped at 10s)
- Standard success/error response parsing (`RequestTokenError`)
- Token-endpoint client authentication and standards-compliant parameter
encoding (RFC 6749 2.3.1 Basic encoding)

LanceDB keeps ownership of OIDC discovery (compared `openidconnect`: no
measurable win for our 3-field metadata + strict validation, at real
dependency cost), HTTPS-or-loopback endpoint enforcement, the loopback
callback server, browser/stderr prompts, token caching and refresh
orchestration, and the dedicated hardened Azure IMDS source, which is
unchanged.

## Client authentication methods

New `ClientAuthMethod` enum (`none` | `client_secret_basic` |
`client_secret_post`), exposed in Rust, Python, and Node. Unset resolves
to `client_secret_basic` when a secret is present (RFC 6749 2.3.1
recommendation and the normal Okta confidential-app default, so a
default Okta app works without weakening its configuration) and to
`none` for public clients (PKCE/device). Explicit `none` with a secret,
or basic/post without one, is rejected. The method applies to client
credentials, code exchange, refresh, and device requests. Deliberate
behavior change: confidential clients previously always sent the secret
in the POST body; they now default to Basic (Keycloak accepts both).

No `audience`/`resource` parameters were added: the supported target is
an Okta custom authorization server with the API audience configured
server-side, so client-provided audience parameters are unnecessary;
`add_extra_param` support exists if a concrete provider contract ever
needs them.

## Device polling behavior changes (deliberate, tested)

- The first token poll now happens immediately rather than after one
interval (RFC 8628 allows both).
- Transient failures (HTTP 429, 5xx, `temporarily_unavailable`, network
errors) now retry with exponential backoff capped at 10s instead of
retrying at the fixed interval; polling never spins faster than once per
second even if a server reports a zero interval.

## Security and compatibility

- Issuer and discovered endpoints (and device verification URIs) still
require HTTPS except explicit loopback HTTP, enforced before any crate
URL type is built
- Token HTTP client keeps the hardened redirect policy that refuses
insecure redirect targets; regression test added
- Errors never embed raw response bodies (avoids leaking tokens through
parse failures); all credential types stay redacted in Debug
- Transient conditions (429/5xx/`temporarily_unavailable`) remain
retryable in device polling and hard errors elsewhere; refresh keeps
rotation and reauthentication semantics
- Existing public APIs stay source-compatible except the added
`OAuthConfig.client_auth_method` field

## Tests

Rust: client-auth methods across code
exchange/refresh/client-credentials/device (none/basic/post),
auth-method resolution and validation, transient device retries,
denial/expiry, redirect rejection, malformed-response leak check, PKCE
URL assertions, redaction. Python and Node: enum values, conversion,
unknown-method errors, config defaults.

Manual Okta validation recipe (no automated Okta credentials): create a
custom authorization server with an API audience, one confidential web
app (Basic) for authorization-code, one native app (PKCE, no secret),
one native device app; point `issuer_url` at the custom server, set
`client_auth_method` only for the POST-required case; verify token
acquisition, refresh after expiry, and `x-lancedb-credential-type: oidc`
against a LanceDB deployment. Never commit tenant URLs or secrets.

Co-authored-by: Xuanwo <github@xuanwo.io>

nodejs/__test__/remote.test.ts

@@ -5,9 +5,12 @@ import * as http from "http";
 import { RequestListener } from "http";
 import packageJson = require("../package.json");
 import {
+  ClientAuthMethod,
   ClientConfig,
   Connection,
   ConnectionOptions,
+  OAuthConfig,
+  OAuthFlowType,
   TlsConfig,
   connect,
 } from "../lancedb";
@@ -438,6 +441,40 @@ describe("remote connection", () => {
     ]);
   });
 
+  describe("OAuthConfig", () => {
+    it("should expose client auth method values", () => {
+      expect(ClientAuthMethod.None).toBe("none");
+      expect(ClientAuthMethod.ClientSecretBasic).toBe("client_secret_basic");
+      expect(ClientAuthMethod.ClientSecretPost).toBe("client_secret_post");
+    });
+
+    it("should accept a confidential client with basic auth", () => {
+      const config: OAuthConfig = {
+        issuerUrl: "https://issuer.example.com",
+        clientId: "client-id",
+        clientSecret: "secret",
+        scopes: ["openid"],
+        flow: OAuthFlowType.AuthorizationCode,
+        clientAuthMethod: ClientAuthMethod.ClientSecretBasic,
+      };
+
+      expect(config.clientAuthMethod).toBe(ClientAuthMethod.ClientSecretBasic);
+    });
+
+    it("should accept a public PKCE client without auth method or secret", () => {
+      const config: OAuthConfig = {
+        issuerUrl: "https://issuer.example.com",
+        clientId: "client-id",
+        scopes: ["openid"],
+        flow: OAuthFlowType.AuthorizationCode,
+        usePkce: true,
+      };
+
+      expect(config.clientSecret).toBeUndefined();
+      expect(config.clientAuthMethod).toBeUndefined();
+    });
+  });
+
   describe("TlsConfig", () => {
     it("should create TlsConfig with all fields", () => {
       const tlsConfig: TlsConfig = {

nodejs/lancedb/index.ts

@@ -172,6 +172,7 @@ export {
 } from "./header";
 
 export {
+  ClientAuthMethod,
   OAuthConfig,
   OAuthFlowType,
   OAuthSession,

nodejs/lancedb/oauth.ts

@@ -47,6 +47,33 @@ export interface TokenCacheOptions {
   lockTimeoutSecs?: number;
 }
 
+/**
+ * How the client authenticates to the OAuth token endpoint.
+ *
+ * The method applies to every OAuth request that carries client
+ * authentication: client-credentials, authorization-code exchange,
+ * refresh-token, and device-authorization requests. The Azure managed
+ * identity flow ignores this option.
+ */
+export enum ClientAuthMethod {
+  /**
+   * No client authentication, for public clients using PKCE or the device
+   * flow. Cannot be combined with `clientSecret`.
+   */
+  None = "none",
+  /**
+   * HTTP Basic authentication. This is the RFC 6749 recommended method and
+   * the normal default for confidential clients, including default Okta
+   * applications. Requires `clientSecret`.
+   */
+  ClientSecretBasic = "client_secret_basic",
+  /**
+   * Credentials in the request body, for providers configured to require it.
+   * Requires `clientSecret`.
+   */
+  ClientSecretPost = "client_secret_post",
+}
+
 /**
  * OAuth configuration for LanceDB authentication.
  *
@@ -140,6 +167,15 @@ export interface OAuthConfig {
   /** Client secret (required for ClientCredentials). */
   clientSecret?: string;
 
+  /**
+   * How the client authenticates to the token endpoint (default: auto).
+   * With a `clientSecret` the default is `ClientAuthMethod.ClientSecretBasic`,
+   * which matches the RFC 6749 recommendation and the default configuration
+   * of Okta confidential applications; without a secret the client is public
+   * and no client authentication is sent.
+   */
+  clientAuthMethod?: ClientAuthMethod;
+
   /** Loopback redirect URI for AuthorizationCode. */
   redirectUri?: string;
 

nodejs/src/remote.rs

@@ -197,6 +197,11 @@ pub struct OAuthConfig {
     pub flow: Option<String>,
     /// Client secret (required for client_credentials).
     pub client_secret: Option<String>,
+    /// How the client authenticates to the token endpoint: "none",
+    /// "client_secret_basic", or "client_secret_post". Defaults to
+    /// "client_secret_basic" when a client secret is set, and "none" for
+    /// public clients.
+    pub client_auth_method: Option<String>,
     /// Loopback redirect URI for authorization_code.
     pub redirect_uri: Option<String>,
     /// Port for the authorization_code loopback callback server.
@@ -227,6 +232,7 @@ impl std::fmt::Debug for OAuthConfig {
                 "client_secret",
                 &self.client_secret.as_deref().map(|_| "<redacted>"),
             )
+            .field("client_auth_method", &self.client_auth_method)
             .field("redirect_uri", &self.redirect_uri)
             .field("callback_port", &self.callback_port)
             .field("use_pkce", &self.use_pkce)
@@ -270,10 +276,27 @@ impl TryFrom<OAuthConfig> for lancedb::remote::oauth::OAuthConfig {
             }
         };
 
+        let client_auth_method = match config.client_auth_method.as_deref() {
+            Some("none") => Some(lancedb::remote::oauth::ClientAuthMethod::None),
+            Some("client_secret_basic") => {
+                Some(lancedb::remote::oauth::ClientAuthMethod::ClientSecretBasic)
+            }
+            Some("client_secret_post") => {
+                Some(lancedb::remote::oauth::ClientAuthMethod::ClientSecretPost)
+            }
+            None => None,
+            Some(other) => {
+                return Err(Error::InvalidInput {
+                    message: format!("Unknown OAuth client auth method: {other}"),
+                });
+            }
+        };
+
         Ok(Self {
             issuer_url: config.issuer_url,
             client_id: config.client_id,
             client_secret: config.client_secret,
+            client_auth_method,
             scopes: config.scopes,
             resource: config.resource,
             audience: config.audience,
@@ -427,6 +450,7 @@ mod tests {
             scopes: vec!["scope".to_string()],
             flow: Some("typo".to_string()),
             client_secret: None,
+            client_auth_method: None,
             redirect_uri: None,
             callback_port: None,
             use_pkce: None,
@@ -453,6 +477,7 @@ mod tests {
             scopes: vec!["scope".to_string()],
             flow: Some("client_credentials".to_string()),
             client_secret: Some("super-secret".to_string()),
+            client_auth_method: None,
             redirect_uri: None,
             callback_port: None,
             use_pkce: None,
@@ -476,6 +501,7 @@ mod tests {
             scopes: vec!["openid".to_string()],
             flow: Some("authorization_code".to_string()),
             client_secret: Some("secret".to_string()),
+            client_auth_method: None,
             redirect_uri: Some("http://127.0.0.1:9000/callback".to_string()),
             callback_port: Some(9000),
             use_pkce: Some(false),
@@ -508,6 +534,7 @@ mod tests {
             scopes: vec!["openid".to_string()],
             flow: Some("device_code".to_string()),
             client_secret: None,
+            client_auth_method: None,
             redirect_uri: None,
             callback_port: None,
             use_pkce: None,
@@ -524,4 +551,62 @@ mod tests {
             lancedb::remote::oauth::OAuthFlow::DeviceCode
         ));
     }
+
+    #[test]
+    fn test_client_auth_method_conversion() {
+        use lancedb::remote::oauth::ClientAuthMethod;
+
+        for (value, expected) in [
+            ("none", ClientAuthMethod::None),
+            ("client_secret_basic", ClientAuthMethod::ClientSecretBasic),
+            ("client_secret_post", ClientAuthMethod::ClientSecretPost),
+        ] {
+            let config = OAuthConfig {
+                issuer_url: "https://issuer.example.com".to_string(),
+                client_id: "client-id".to_string(),
+                scopes: vec!["openid".to_string()],
+                flow: Some("device_code".to_string()),
+                client_secret: None,
+                client_auth_method: Some(value.to_string()),
+                resource: None,
+                audience: None,
+                redirect_uri: None,
+                callback_port: None,
+                use_pkce: None,
+                managed_identity_client_id: None,
+                refresh_buffer_secs: None,
+                token_cache: None,
+            };
+
+            let converted = lancedb::remote::oauth::OAuthConfig::try_from(config).unwrap();
+            assert_eq!(converted.client_auth_method, Some(expected));
+        }
+    }
+
+    #[test]
+    fn test_unknown_client_auth_method_returns_invalid_input() {
+        let config = OAuthConfig {
+            issuer_url: "https://issuer.example.com".to_string(),
+            client_id: "client-id".to_string(),
+            scopes: vec!["openid".to_string()],
+            flow: Some("device_code".to_string()),
+            client_secret: None,
+            client_auth_method: Some("typo".to_string()),
+            resource: None,
+            audience: None,
+            redirect_uri: None,
+            callback_port: None,
+            use_pkce: None,
+            managed_identity_client_id: None,
+            refresh_buffer_secs: None,
+            token_cache: None,
+        };
+
+        let err = lancedb::remote::oauth::OAuthConfig::try_from(config).unwrap_err();
+        assert!(matches!(
+            err,
+            Error::InvalidInput { message }
+                if message == "Unknown OAuth client auth method: typo"
+        ));
+    }
 }