sav/spss converter

Summary

This adds a new TableConverter for SAV files (generated by SPSS).

Also:

• xfail for Python 3.13 in the pipeline.
• Documentation
• Linting, styling

Focus

Nothing special.

Test Environment

Run the unit tests.

Check List for the Author

Please, prepare your MR for a review. Be sure to write a summary and a focus and create gitlab comments for the reviewer. They should guide the reviewer through the changes, explain your changes and also point out open questions. For further good practices have a look at our review guidelines

• All automated tests pass
• Reference related issues caosdb/customers/interfach/management#6
• Up-to-date CHANGELOG.md (or not necessary)
• Up-to-date JSON schema (or not necessary)
• Appropriate user and developer documentation (or not necessary)
  • How do I use the software? Assume "stupid" users.
  • How do I develop or debug the software? Assume novice developers.
• Annotations in code (Gitlab comments)
  • Intent of new code
  • Problems with old code
  • Why this implementation?

Check List for the Reviewer

• I understand the intent of this MR
• All automated tests pass
• Up-to-date CHANGELOG.md (or not necessary)
• Appropriate user and developer documentation (or not necessary)
• The test environment setup works and the intended behavior is reproducible in the test environment
• In-code documentation and comments are up-to-date.
• Check: Are there specifications? Are they satisfied?

For further good practices have a look at our review guidelines.

src/caoscrawler/conv_impl/spss.py

@@ -21,6 +21,7 @@
 import argparse
 from collections import OrderedDict
 
+import numpy as np
 import pandas as pd
 import pyreadstat
 import yaml
@@ -53,15 +54,58 @@ class SPSSConverter(converters.TableConverter):
         # The default dtype backend "numpy_nullable" does not handle dates well.
         # Note that pandas.ArrowDtype is considered experimental (in Pandas 2.2).
         df = pd.io.spss.read_spss(element.path, dtype_backend="pyarrow")
+        dtypes = read_column_types(element.path)
 
         # if element.path.endswith(".sav"):
         #     sav_df, meta = pyreadstat.read_sav(element.path, metadataonly=True)
         # from IPython import embed
         # embed()
+        df.drop(range(15, len(df.index)), inplace=True)
+
+        # Fix datetime columns
+        for name, dtype in dtypes.items():
+            if dtype != "DATETIME":
+                continue
+            col = df.loc[:, name]
+            col.fillna(np.nan, inplace=True)
+            col.replace([np.nan], [None], inplace=True)
+
+        # from IPython import embed
+        # embed()
 
         return self._children_from_dataframe(df)
 
 
+def read_column_types(savfile: Optional[str] = None, meta: Optional = None) -> dict[str, str]:
+    """Read SAV file and return the column types.
+
+Optionally, take data from a previours reading.
+
+Parameters
+----------
+savfile : Optional[str]
+    The SAV file to read.
+
+meta : Optional
+    The meta data result from `pyreadstat.read_sav(...)`.
+
+Returns
+-------
+out : dict[str, str]
+    The column names and types.
+    """
+    if not meta:
+        _, meta = pyreadstat.read_sav(savfile, metadataonly=True)
+    elif savfile is not None:
+        raise ValueError("Only one of `savfile` and `meta` must be given.")
+    dtypes: dict[str, str] = {}
+    for name in meta.column_names:
+        datatype = ORIGINAL_TYPES.get(meta.original_variable_types[name],
+                                      READSTAT_TYPES[meta.readstat_variable_types[name]])
+        dtypes[name] = datatype
+    return dtypes
+
+
 def spss_to_yaml(savfile: str, yamlfile: str, cfood: Optional[str] = None) -> None:
     """Parse the *.sav and create basic datamodel in ``yamlfile``.
 
@@ -71,6 +115,7 @@ cfood: str
   If given, also create a cfood skeleton.
     """
     _, meta = pyreadstat.read_sav(savfile, metadataonly=True)
+    dtypes = read_column_types(meta=meta)
 
     cfood_str = """
 ---
@@ -147,13 +192,10 @@ directory: # corresponds to the directory given to the crawler
     properties = OrderedDict()
 
     for name in meta.column_names:
-        datatype = ORIGINAL_TYPES.get(meta.original_variable_types[name],
-                                      READSTAT_TYPES[meta.readstat_variable_types[name]])
         prop = {
-            "datatype": datatype,
+            "datatype": dtypes[name],
         }
         desc = meta.column_names_to_labels.get(name)
-
         if desc and desc != name:
             prop["description"] = desc
         # Handle categorial variables
@@ -246,7 +288,7 @@ DummyRT:
             # Which type?
             if propdef["datatype"] == "DOUBLE":
                 dtype = "FloatElement"
-            elif propdef["datatype"] == "TEXT":
+            elif propdef["datatype"] in ("TEXT", "DATETIME"):
                 dtype = None
             else:
                 reftype = propdef["datatype"]