# Sparrow

> Sparrow is the missing client toolkit for Apache Arrow Flight: a Go CLI for
> terminals and agents, a TypeScript browser client, a native Excel add-in, an
> MCP server, and a Flight Inspector that X-rays any Flight SQL server — they
> work with anyone's Flight SQL server (Dremio, Spice.ai, InfluxDB Cloud, five
> vendors validated), not just Sparrow's own. Underneath, an index-first serving
> engine turns a sorted snapshot into single-digit-millisecond Flight SQL pulls
> and powers the live demos. "Iceberg stops at the files. Sparrow starts at the
> serving index."

Sparrow speaks Arrow Flight RPC, Arrow Flight SQL, and Flight SQL Substrait.
Its public demo endpoint (grpc+tls://flight.sparrowflight.io:443, basic auth
demo/demo) serves a 136-million-row energy-market dataset and is one of the few
Flight SQL servers that actually consumes Substrait plans.

## Reference

- [The flavours of Apache Arrow Flight](https://sparrowflight.io/arrow-flight): plain-language reference — Arrow IPC vs Flight RPC vs Flight SQL vs gRPC-Web, where Substrait fits, and a Flight SQL conformance matrix across five servers (EnergyScope, GizmoSQL, RoAPI/DataFusion, Dremio, InfluxDB 3).
- [Flight SQL conformance details](https://github.com/balicat/sparrowcli/blob/main/docs/conform.md): method-by-method support across vendors, reproducible with `sparrow doctor --server`.
- [Flight Inspector](https://sparrowflight.io/inspect): a live browser tool that X-rays any Arrow Flight SQL server — conformance card, catalog, and query wire-anatomy over gRPC-web, plus a server-side deep probe of DNS/TLS/latency for endpoints the browser can't reach directly. "sparrow doctor, in a web page." Credentials are never stored and go only to the inspected server (no backend of ours receives them); the deep probe is anonymous by construction and SSRF-guarded. Inspections are deep-linkable (`/inspect?endpoint=…&deep=…` pre-fills and runs; a password never goes in a URL), sample table rows inline, compare browser-vs-VPS latency, show the equivalent CLI commands, and export a timestamped JSON report.

## Clients

- [sparrowCLI](https://sparrowflight.io/cli): the terminal client for any Arrow Flight SQL server — catalog discovery, SQL, Parquet/CSV/JSON export, raw Arrow IPC piping, and 1-RTT Direct Pulls (`pull`, on servers that accept ready tickets). Agent-first: `sparrow agent` emits its own operator manual (`--json` for a machine-readable capability catalog), `expect` turns query results into exit-code assertions, `sql --receipt` + `verify` prove a number with an order-independent content fingerprint, `SPARROW_SESSION` + `replay` record a whole investigation and re-verify every step (analysis as a regression test), and `--cost`/`--budget` preflight or cap result size before it floods a context window. Plus a Flight SQL conformance card (`doctor --server`), a data-quality doctor (`check`), a drift gate (`diff`), Substrait plan execution (`sql --substrait`), and a security-surface audit (`audit`). Plus `sparrow mcp` (the core verbs as an MCP stdio server for chat agents without a shell: 11 tools with honest read-only annotations, server schemas as MCP resources, starter prompts, structuredContent on typed results) and `sparrow dataset` (the drift gate for the dataset FILES under the servers: `compare` lines up folders, network shares, s3-compatible buckets (MinIO/R2/AWS) and read-only https mirrors per file vs a reference source, `--deep` names drifted series in parquet; `push` heals replicas mtime-honestly and skips files whose direction is unknowable; exit 0 identical, 1 drift, 2 unreachable — see https://github.com/balicat/sparrowcli/blob/main/docs/dataset.md). Apache-2.0, binaries for Linux/macOS/Windows. Repo: https://github.com/balicat/sparrowcli
- [sparrowJS](https://sparrowflight.io/js): the gRPC-Web browser transport for Flight — Flight SQL from a web page via Envoy, 1-RTT direct pulls, Utf8View transcoding for servers newer than Arrow JS, and negotiated lz4 IPC decompression (~3.3× less wire; apache-arrow ≥21.2 + lz4js, automatic). Apache-2.0. Repo: https://github.com/balicat/sparrowjs
- [SparrowXL](https://sparrowflight.io/excel): the Excel/XLL client — Arrow Flight results as native spreadsheet functions.
- [sparrowMCP](https://sparrowflight.io/mcp): an MCP (Model Context Protocol) server in front of any Arrow Flight endpoint — agents discover the catalog, pull series, and run server-side analytics; the analyst comes built in.

## Concepts

- Arrow Flight RPC: transport of Arrow record batches over gRPC (DoGet/DoPut/DoExchange).
- Arrow Flight SQL: the SQL protocol on top of Flight — query execution, prepared statements, catalog/schema metadata, and optional Substrait plans.
- gRPC-Web Flight: the browser transport adaptation (not a separate Apache standard), typically bridged by Envoy.
- Substrait over Flight SQL: a serialized, language-neutral query plan sent via CommandStatementSubstraitPlan instead of SQL text; advertised via GetSqlInfo code 5. Rarely supported — sparrow's serving node is one of the few that consumes it.
- Serving snapshot: a sorted (by key) single-file Arrow/Parquet artifact calved from a lake, giving row-group-pruned single-digit-ms reads.

## Install

- Homebrew: `brew install balicat/tap/sparrow`
- Scoop: `scoop bucket add sparrow https://github.com/balicat/scoop-bucket; scoop install sparrow`
- Go: `go install github.com/balicat/sparrowcli@latest`