Filling XLSX: Everything except multiple choice.

Summary

For caosdb/customers/dimr/management#219:

Filling XLSX templates from JSON works now.

Focus

• Correct handling of foreign keys
• Correct handling of other lists

Test Environment

Run the unit tests.

Check List for the Author

Please, prepare your MR for a review. Be sure to write a summary and a focus and create gitlab comments for the reviewer. They should guide the reviewer through the changes, explain your changes and also point out open questions. For further good practices have a look at our review guidelines

• All automated tests pass
• Reference related issues
• Up-to-date CHANGELOG.md (or not necessary)
• Up-to-date JSON schema (or not necessary)
• Appropriate user and developer documentation (or not necessary)
  • How do I use the software? Assume "stupid" users.
  • How do I develop or debug the software? Assume novice developers.
• Annotations in code (Gitlab comments)
  • Intent of new code
  • Problems with old code
  • Why this implementation?

Check List for the Reviewer

• I understand the intent of this MR
• All automated tests pass
• Up-to-date CHANGELOG.md (or not necessary)
• Appropriate user and developer documentation (or not necessary)
• The test environment setup works and the intended behavior is reproducible in the test environment
• In-code documentation and comments are up-to-date.
• Check: Are there specifications? Are they satisfied?

For further good practices have a look at our review guidelines.

src/caosadvancedtools/table_json_conversion/fill_xlsx.py

@@ -23,12 +23,14 @@
 from __future__ import annotations
 
 import json
+import pathlib
 from collections import OrderedDict
 from types import SimpleNamespace
-from typing import Any, Dict, List, Optional, Union, TextIO
+from typing import Any, Dict, List, Optional, TextIO, Union
 
 from jsonschema import FormatChecker, validate
-from openpyxl import load_workbook, Workbook
+from jsonschema.exceptions import ValidationError
+from openpyxl import Workbook, load_workbook
 from openpyxl.worksheet.worksheet import Worksheet
 
 from .table_generator import ColumnType, RowType
@@ -60,14 +62,6 @@ def _get_column_types(sheet: Worksheet) -> OrderedDict:
     return result
 
 
-def _get_deep_value(data: Dict[str, Any], path: List[str]):
-    """Return the value at ``path`` inside the dict ``data``.
-    """
-    if len(path) > 1:
-        return _get_deep_value(data[path[0]], path[1:])
-    return data[path[0]]
-
-
 def _get_foreign_key_columns(sheet: Worksheet) -> Dict[str, SimpleNamespace]:
     """Return the foreign keys of the worksheet.
 
@@ -190,10 +184,19 @@ class TemplateFiller:
             self._props[fullpath] = value
 
         def fill_from_data(self, data: Dict[str, Any]):
+            # TODO recursive for dicts and list?
             """Fill current level with all scalar elements of ``data``."""
             for name, value in data.items():
                 if not isinstance(value, (dict, list)):
                     self[name] = value
+                elif isinstance(value, dict):
+                    if not value or isinstance(list(value.items())[0], list):
+                        continue
+                    old_path = self._current_path
+                    new_path = self._current_path.copy() + [name]
+                    self._current_path = new_path
+                    self.fill_from_data(data=value)
+                    self._current_path = old_path
 
     def _create_index(self):
         """Create a sheet index for the workbook.
@@ -271,6 +274,9 @@ out: union[dict, None]
 
         insertables: Dict[str, Any] = {}
         for name, content in data.items():
+            # TODO is this the best way to do it????
+            if name == "file":
+                continue
             path = current_path + [name]
             next_context = context.next_level(name)
             # preprocessing
@@ -327,7 +333,11 @@ out: union[dict, None]
 
         # Insert foreign keys
         if insert_row is not None and sheet is not None and _is_exploded_sheet(sheet):
-            foreigns = _get_foreign_key_columns(sheet)
+            try:
+                foreigns = _get_foreign_key_columns(sheet)
+            except ValueError:
+                print(f"Sheet: {sheet}")
+                raise
             for index, path in ((f.index, f.path) for f in foreigns.values()):
                 value = context[path]
                 sheet.cell(row=insert_row+1, column=index+1, value=value)
@@ -360,10 +370,17 @@ validation_schema: dict, optional
     # Validation
     if validation_schema is not None:
         validation_schema = _read_or_dict(validation_schema)
-        validate(data, validation_schema, format_checker=FormatChecker())
+        try:
+            validate(data, validation_schema, format_checker=FormatChecker())
+        except ValidationError as ve:
+            print(ve.message)
+            raise RuntimeError("Validation failed")
 
     # Filling the data
     result_wb = load_workbook(template)
     template_filler = TemplateFiller(result_wb)
     template_filler.fill_data(data=data)
+
+    parentpath = pathlib.Path(result).parent
+    parentpath.mkdir(parents=True, exist_ok=True)
     result_wb.save(result)