Filling XLSX: Everything except multiple choice.

Summary

For caosdb/customers/dimr/management#219:

Filling XLSX templates from JSON works now.

Focus

• Correct handling of foreign keys
• Correct handling of other lists

Test Environment

Run the unit tests.

Check List for the Author

Please, prepare your MR for a review. Be sure to write a summary and a focus and create gitlab comments for the reviewer. They should guide the reviewer through the changes, explain your changes and also point out open questions. For further good practices have a look at our review guidelines

• All automated tests pass
• Reference related issues
• Up-to-date CHANGELOG.md (or not necessary)
• Up-to-date JSON schema (or not necessary)
• Appropriate user and developer documentation (or not necessary)
  • How do I use the software? Assume "stupid" users.
  • How do I develop or debug the software? Assume novice developers.
• Annotations in code (Gitlab comments)
  • Intent of new code
  • Problems with old code
  • Why this implementation?

Check List for the Reviewer

• I understand the intent of this MR
• All automated tests pass
• Up-to-date CHANGELOG.md (or not necessary)
• Appropriate user and developer documentation (or not necessary)
• The test environment setup works and the intended behavior is reproducible in the test environment
• In-code documentation and comments are up-to-date.
• Check: Are there specifications? Are they satisfied?

For further good practices have a look at our review guidelines.

src/caosadvancedtools/table_json_conversion/fill_xlsx.py

@@ -262,6 +262,8 @@ out: union[dict, None]
             context = TemplateFiller.Context()
         context.fill_from_data(data)
 
+        explode_later: List[dict] = []
+
         insertables: Dict[str, Any] = {}
         for name, content in data.items():
             path = current_path + [name]
@@ -275,16 +277,23 @@ out: union[dict, None]
                 if isinstance(content[0], dict):
                     # An array of objects: must go into exploded sheet
                     for entry in content:
-                        self._handle_data(data=entry, current_path=path, context=next_context)
+                        explode_later.append({"data": entry, "current_path": path,
+                                              "context": next_context})
                     continue
             elif isinstance(content, dict):
                 if not current_path:  # Special handling for top level
-                    self._handle_data(content, current_path=path, context=next_context)
+                    explode_later.append({"data": content, "current_path": path,
+                                          "context": next_context})
                     continue
                 insert = self._handle_data(content, current_path=path, context=next_context.copy(),
                                            only_collect_insertables=True)
                 assert isinstance(insert, dict)
                 assert not any(key in insertables for key in insert)
+
+                # print(insert)
+                # from IPython import embed
+                # embed()
+
                 insertables.update(insert)
                 continue
             else:  # scalars
@@ -301,8 +310,9 @@ out: union[dict, None]
             insertables[path_str] = value
         if only_collect_insertables:
             return insertables
-        if not current_path:
-            return None
+        # if not current_path:
+        #     print("returning early")
+        #     return None
 
         # actual data insertion
         insert_row = None
@@ -318,11 +328,21 @@ out: union[dict, None]
 
             sheet.cell(row=insert_row+1, column=col_index+1, value=value)
 
+        # continue in exploded sheets
+        for exploded_content in explode_later:
+            print(f"Exploding: {exploded_content}")
+            self._handle_data(**exploded_content)
+
         # Insert foreign keys
         if insert_row is not None and sheet is not None and _is_exploded_sheet(sheet):
             foreigns = _get_foreign_key_columns(sheet)
             for index, path in ((f.index, f.path) for f in foreigns.values()):
-                value = context[path]
+                try:
+                    value = context[path]
+                except KeyError as err:
+                    print(f"\nCurrent context: {context._props}")
+                    from IPython import embed
+                    embed()
                 sheet.cell(row=insert_row+1, column=index+1, value=value)
 
         return None