dedup

SQL dedup by normalized hash — prevents indexing the same SQL with different questions.

Without dedup, training the same SQL twice produces duplicate embeddings and noisy retrieval. indexed_sql_dedup tracks SQL hash → first question so callers can skip already-indexed SQLs.

Hash is computed over normalized SQL (lowercase, stripped, whitespace collapsed) so cosmetic variants hash the same.

Classes

DedupEntry — One row from indexed_sql_dedup. Frozen — read-only value object.
SqlDedupRepository — CRUD for the indexed_sql_dedup table in PostgreSQL.

Functions

normalize_sql — Normalize SQL: strip, remove trailing semicolons, lowercase, collapse whitespace.
sql_hash — SHA-256 (truncated to 16 hex chars) of the normalized SQL.

wittgenstein_pgvector_qsql.dedup.normalize_sql

normalize_sql(sql: str) → str

Normalize SQL: strip, remove trailing semicolons, lowercase, collapse whitespace.

wittgenstein_pgvector_qsql.dedup.sql_hash

sql_hash(sql: str) → str

SHA-256 (truncated to 16 hex chars) of the normalized SQL.

16 chars ≈ 2^64 combinations — collision probability is negligible for catalogs of thousands of SQLs.

wittgenstein_pgvector_qsql.dedup.DedupEntry

class DedupEntry()

One row from indexed_sql_dedup. Frozen — read-only value object.

wittgenstein_pgvector_qsql.dedup.SqlDedupRepository

class SqlDedupRepository(pg_dsn: str)

CRUD for the indexed_sql_dedup table in PostgreSQL.

Table DDL (created automatically by `ensure_table()`)

CREATE TABLE indexed_sql_dedup (
    sql_hash       TEXT PRIMARY KEY,
    sql_normalized TEXT NOT NULL,
    first_question TEXT NOT NULL,
    chart_config   JSONB,
    indexed_at     TIMESTAMP DEFAULT NOW()
);

Parameters

pg_dsn : str — psycopg2-compatible DSN, e.g. 'postgresql://user:pass@host:port/dbname'

Methods

ensure_table — Create the dedup table if it does not exist. Idempotent.
get_first_question — Return the first question that indexed this SQL, or None if unseen.
insert — Insert a new dedup entry. Caller must verify hash_value is unseen first.
list_all — Return all dedup entries ordered by recency (DESC).
find_chart_by_hash — Return the chart_config stored for a SQL hash, or None.

wittgenstein_pgvector_qsql.dedup.SqlDedupRepository.ensure_table

method SqlDedupRepository.ensure_table() → None

Create the dedup table if it does not exist. Idempotent.

wittgenstein_pgvector_qsql.dedup.SqlDedupRepository.get_first_question

method SqlDedupRepository.get_first_question(hash_value: str) → str | None

Return the first question that indexed this SQL, or None if unseen.

wittgenstein_pgvector_qsql.dedup.SqlDedupRepository.insert

method SqlDedupRepository.insert(hash_value: str, sql_normalized: str, first_question: str, chart_config: dict[str, Any] | None) → None

Insert a new dedup entry. Caller must verify hash_value is unseen first.

wittgenstein_pgvector_qsql.dedup.SqlDedupRepository.list_all

method SqlDedupRepository.list_all() → list[DedupEntry]

Return all dedup entries ordered by recency (DESC).

wittgenstein_pgvector_qsql.dedup.SqlDedupRepository.find_chart_by_hash

method SqlDedupRepository.find_chart_by_hash(hash_value: str) → dict[str, Any] | None

Return the chart_config stored for a SQL hash, or None.