Filling XLSX: Everything except multiple choice.

Summary

For caosdb/customers/dimr/management#219:

Filling XLSX templates from JSON works now.

Focus

• Correct handling of foreign keys
• Correct handling of other lists

Test Environment

Run the unit tests.

Check List for the Author

Please, prepare your MR for a review. Be sure to write a summary and a focus and create gitlab comments for the reviewer. They should guide the reviewer through the changes, explain your changes and also point out open questions. For further good practices have a look at our review guidelines

• All automated tests pass
• Reference related issues
• Up-to-date CHANGELOG.md (or not necessary)
• Up-to-date JSON schema (or not necessary)
• Appropriate user and developer documentation (or not necessary)
  • How do I use the software? Assume "stupid" users.
  • How do I develop or debug the software? Assume novice developers.
• Annotations in code (Gitlab comments)
  • Intent of new code
  • Problems with old code
  • Why this implementation?

Check List for the Reviewer

• I understand the intent of this MR
• All automated tests pass
• Up-to-date CHANGELOG.md (or not necessary)
• Appropriate user and developer documentation (or not necessary)
• The test environment setup works and the intended behavior is reproducible in the test environment
• In-code documentation and comments are up-to-date.
• Check: Are there specifications? Are they satisfied?

For further good practices have a look at our review guidelines.

src/caosadvancedtools/table_json_conversion/fill_xlsx.py

@@ -23,6 +23,7 @@
 from __future__ import annotations
 
 import json
+import pathlib
 from collections import OrderedDict
 from types import SimpleNamespace
 from typing import Any, Dict, List, Optional, Union, TextIO
@@ -60,14 +61,6 @@ def _get_column_types(sheet: Worksheet) -> OrderedDict:
     return result
 
 
-def _get_deep_value(data: Dict[str, Any], path: List[str]):
-    """Return the value at ``path`` inside the dict ``data``.
-    """
-    if len(path) > 1:
-        return _get_deep_value(data[path[0]], path[1:])
-    return data[path[0]]
-
-
 def _get_foreign_key_columns(sheet: Worksheet) -> Dict[str, SimpleNamespace]:
     """Return the foreign keys of the worksheet.
 
@@ -269,6 +262,8 @@ out: union[dict, None]
             context = TemplateFiller.Context()
         context.fill_from_data(data)
 
+        explode_later: List[dict] = []
+
         insertables: Dict[str, Any] = {}
         for name, content in data.items():
             path = current_path + [name]
@@ -282,16 +277,23 @@ out: union[dict, None]
                 if isinstance(content[0], dict):
                     # An array of objects: must go into exploded sheet
                     for entry in content:
-                        self._handle_data(data=entry, current_path=path, context=next_context)
+                        explode_later.append({"data": entry, "current_path": path,
+                                              "context": next_context})
                     continue
             elif isinstance(content, dict):
                 if not current_path:  # Special handling for top level
-                    self._handle_data(content, current_path=path, context=next_context)
+                    explode_later.append({"data": content, "current_path": path,
+                                          "context": next_context})
                     continue
                 insert = self._handle_data(content, current_path=path, context=next_context.copy(),
                                            only_collect_insertables=True)
                 assert isinstance(insert, dict)
                 assert not any(key in insertables for key in insert)
+
+                # print(insert)
+                # from IPython import embed
+                # embed()
+
                 insertables.update(insert)
                 continue
             else:  # scalars
@@ -308,8 +310,9 @@ out: union[dict, None]
             insertables[path_str] = value
         if only_collect_insertables:
             return insertables
-        if not current_path:
-            return None
+        # if not current_path:
+        #     print("returning early")
+        #     return None
 
         # actual data insertion
         insert_row = None
@@ -325,11 +328,21 @@ out: union[dict, None]
 
             sheet.cell(row=insert_row+1, column=col_index+1, value=value)
 
+        # continue in exploded sheets
+        for exploded_content in explode_later:
+            print(f"Exploding: {exploded_content}")
+            self._handle_data(**exploded_content)
+
         # Insert foreign keys
         if insert_row is not None and sheet is not None and _is_exploded_sheet(sheet):
             foreigns = _get_foreign_key_columns(sheet)
             for index, path in ((f.index, f.path) for f in foreigns.values()):
-                value = context[path]
+                try:
+                    value = context[path]
+                except KeyError as err:
+                    print(f"\nCurrent context: {context._props}")
+                    from IPython import embed
+                    embed()
                 sheet.cell(row=insert_row+1, column=index+1, value=value)
 
         return None
@@ -366,4 +379,7 @@ validation_schema: dict, optional
     result_wb = load_workbook(template)
     template_filler = TemplateFiller(result_wb)
     template_filler.fill_data(data=data)
+
+    parentpath = pathlib.Path(result).parent
+    parentpath.mkdir(parents=True, exist_ok=True)
     result_wb.save(result)