Skip to content

catalog

module wittgenstein_catalog_builder.catalog

Generic catalog metadata inference from DataFrames.

CatalogBuilder wraps the inference logic (granularity, period, column stats) so any ETL pipeline can build a catalog.yaml without duplicating the heuristics. Consumer-specific wiring (report.json, metadata.yaml, YAML serialization) stays in the consuming ETL package.

Classes

  • CatalogBuilder — Infer catalog metadata from DataFrames and merge with existing YAML curations.

wittgenstein_catalog_builder.catalog.CatalogBuilder

class CatalogBuilder(extra_verified: set[str] | None = None)

Infer catalog metadata from DataFrames and merge with existing YAML curations.

Methods

  • infer_granularity — Detect temporal granularity from a date column.

  • get_period — Return (min, max) period strings from a date or year column.

  • get_distinct_values — Return up to max_values sorted distinct non-null string representations.

  • infer_confidence — Return 'verified' for known common columns, 'inferred' otherwise.

  • numeric_sample — Return n evenly-spaced values from sorted distinct uniques (deterministic).

  • date_strings — Return sorted distinct dates as YYYY-MM-DD strings.

  • build_column_meta — Build a ColumnMeta from a single DataFrame column.

  • build_table_entry — Build a complete TableMeta from a DataFrame.

  • build_from_strategy — Extract via strategy then infer — one call instead of loading the DataFrame yourself first (the Excel/CSV path this library originally assumed). **kwargs forwards to build_table_entry (description, column_descriptions, primary_key, ...).

  • load_existing — Load the tables section of an existing catalog.yaml.

  • merge_curations — Overlay manual curations from an existing catalog entry onto a fresh entry.

wittgenstein_catalog_builder.catalog.CatalogBuilder.infer_granularity

method CatalogBuilder.infer_granularity(df: pd.DataFrame, date_col: str = 'data_ref') → str

Detect temporal granularity from a date column.

Returns one of: diario / mensal / trimestral / anual / snapshot. Falls back to structural columns (trimestre, ano) when date_col is absent.

wittgenstein_catalog_builder.catalog.CatalogBuilder.get_period

method CatalogBuilder.get_period(df: pd.DataFrame, date_col: str = 'data_ref') → tuple[str | None, str | None]

Return (min, max) period strings from a date or year column.

Returns (None, None) when no temporal column is found.

wittgenstein_catalog_builder.catalog.CatalogBuilder.get_distinct_values

method CatalogBuilder.get_distinct_values(series: pd.Series, max_values: int = 30) → list[str]

Return up to max_values sorted distinct non-null string representations.

wittgenstein_catalog_builder.catalog.CatalogBuilder.infer_confidence

method CatalogBuilder.infer_confidence(col_name: str) → str

Return 'verified' for known common columns, 'inferred' otherwise.

wittgenstein_catalog_builder.catalog.CatalogBuilder.numeric_sample

method CatalogBuilder.numeric_sample(series: pd.Series, n: int = 5) → list

Return n evenly-spaced values from sorted distinct uniques (deterministic).

wittgenstein_catalog_builder.catalog.CatalogBuilder.date_strings

method CatalogBuilder.date_strings(series: pd.Series) → list[str]

Return sorted distinct dates as YYYY-MM-DD strings.

wittgenstein_catalog_builder.catalog.CatalogBuilder.build_column_meta

method CatalogBuilder.build_column_meta(col: str, series: pd.Series, description: str = '', confidence: str | None = None) → ColumnMeta

Build a ColumnMeta from a single DataFrame column.

wittgenstein_catalog_builder.catalog.CatalogBuilder.build_table_entry

method CatalogBuilder.build_table_entry(name: str, df: pd.DataFrame, description: str = '', column_descriptions: dict[str, str] | None = None, column_confidence: dict[str, str] | None = None, table_type: str = 'fact', primary_key: str | None = None, foreign_keys: list[dict] | None = None) → TableMeta

Build a complete TableMeta from a DataFrame.

Parameters

  • name : str — Table name (used for labelling only).

  • df : pd.DataFrame — Source DataFrame.

  • description : str — Human-readable table description.

  • column_descriptions : dict[str, str] | None — {col_name: description} override map.

  • column_confidence : dict[str, str] | None — {col_name: confidence} override map.

  • table_type : str — "fact" or "dimension".

  • primary_key : str | None — Primary key column name.

  • foreign_keys : list[dict] | None — List of dicts with keys column and references.

wittgenstein_catalog_builder.catalog.CatalogBuilder.build_from_strategy

method CatalogBuilder.build_from_strategy(strategy: SourceStrategy, name: str, **kwargs: Any) → TableMeta

Extract via strategy then infer — one call instead of loading the DataFrame yourself first (the Excel/CSV path this library originally assumed). **kwargs forwards to build_table_entry (description, column_descriptions, primary_key, ...).

wittgenstein_catalog_builder.catalog.CatalogBuilder.load_existing

method CatalogBuilder.load_existing(path: Path) → dict[str, dict]

Load the tables section of an existing catalog.yaml.

Returns an empty dict if the file is absent or unreadable.

wittgenstein_catalog_builder.catalog.CatalogBuilder.merge_curations

method CatalogBuilder.merge_curations(entry: TableMeta, existing: dict) → TableMeta

Overlay manual curations from an existing catalog entry onto a fresh entry.

Preserved from existing

  • Table description (when not blank)
  • Column description (when not blank)
  • Column confidence (when explicitly set)

Always recomputed from data

  • rows, granularity, period, type, is_categorical, unique_count, values, values_sample, range