Ir para o conteúdo

utils

module wittgenstein_catalog_builder.utils

Generic DataFrame cleaning and I/O helpers for ETL pipelines.

Functions

  • normalize_column_name — Convert a column name to snake_case without accents.

  • clean_numeric — Convert series to numeric, replacing '***', '-', 'X' with NaN.

  • clean_date — Normalize a date column to datetime, stripping the time component.

  • drop_null_columns — Drop columns that are 100% null or have no usable name.

  • trim_strings — Trim whitespace from all string values in object columns.

  • uppercase_text_columns — Uppercase all string values in object columns (canonical casing for chat ingest).

  • read_excel_sheet — Read an Excel sheet and apply basic cleaning (drop null columns, trim strings).

  • read_csv_clean — Read a CSV with utf-8-sig encoding and apply basic cleaning.

  • save_csv — Save a DataFrame as CSV and print a summary line.

  • rename_columns — Rename columns using a mapping dict (thin wrapper around DataFrame.rename).

  • normalize_all_columns — Apply snake_case normalization to all column names.

wittgenstein_catalog_builder.utils.normalize_column_name

normalize_column_name(name: str) → str

Convert a column name to snake_case without accents.

wittgenstein_catalog_builder.utils.clean_numeric

clean_numeric(series: pd.Series) → pd.Series

Convert series to numeric, replacing '***', '-', 'X' with NaN.

wittgenstein_catalog_builder.utils.clean_date

clean_date(series: pd.Series) → pd.Series

Normalize a date column to datetime, stripping the time component.

wittgenstein_catalog_builder.utils.drop_null_columns

drop_null_columns(df: pd.DataFrame) → pd.DataFrame

Drop columns that are 100% null or have no usable name.

wittgenstein_catalog_builder.utils.trim_strings

trim_strings(df: pd.DataFrame) → pd.DataFrame

Trim whitespace from all string values in object columns.

wittgenstein_catalog_builder.utils.uppercase_text_columns

uppercase_text_columns(df: pd.DataFrame) → pd.DataFrame

Uppercase all string values in object columns (canonical casing for chat ingest).

wittgenstein_catalog_builder.utils.read_excel_sheet

read_excel_sheet(path: Path, sheet_name: str, **kwargs) → pd.DataFrame

Read an Excel sheet and apply basic cleaning (drop null columns, trim strings).

wittgenstein_catalog_builder.utils.read_csv_clean

read_csv_clean(path: Path, **kwargs) → pd.DataFrame

Read a CSV with utf-8-sig encoding and apply basic cleaning.

wittgenstein_catalog_builder.utils.save_csv

save_csv(df: pd.DataFrame, output_dir: Path, table_name: str) → Path

Save a DataFrame as CSV and print a summary line.

wittgenstein_catalog_builder.utils.rename_columns

rename_columns(df: pd.DataFrame, mapping: dict) → pd.DataFrame

Rename columns using a mapping dict (thin wrapper around DataFrame.rename).

wittgenstein_catalog_builder.utils.normalize_all_columns

normalize_all_columns(df: pd.DataFrame) → pd.DataFrame

Apply snake_case normalization to all column names.