ENH: add framework for converting json schema into table templates

Summary

Insert a meaningful description for this merge request here: What is the new/changed behavior? Which bug has been fixed? Are there related issues?

Focus

Point the reviewer to the core of the code change. Where should they start reading? What should they focus on (e.g. security, performance, maintainability, user-friendliness, compliance with the specs, finding more corner cases, concrete questions)?

Test Environment

How to set up a test environment for manual testing?

Check List for the Author

Please, prepare your MR for a review. Be sure to write a summary and a focus and create gitlab comments for the reviewer. They should guide the reviewer through the changes, explain your changes and also point out open questions. For further good practices have a look at our review guidelines

• All automated tests pass
• Reference related issues
• Up-to-date CHANGELOG.md (or not necessary)
• Up-to-date JSON schema (or not necessary)
• Appropriate user and developer documentation (or not necessary)
  • How do I use the software? Assume "stupid" users.
  • How do I develop or debug the software? Assume novice developers.
• Annotations in code (Gitlab comments)
  • Intent of new code
  • Problems with old code
  • Why this implementation?

Check List for the Reviewer

• I understand the intent of this MR
• All automated tests pass
• Up-to-date CHANGELOG.md (or not necessary)
• Appropriate user and developer documentation (or not necessary)
• The test environment setup works and the intended behavior is reproducible in the test environment
• In-code documentation and comments are up-to-date.
• Check: Are there specifications? Are they satisfied?

For further good practices have a look at our review guidelines.

src/caosadvancedtools/table_json_conversion/table_generator.py

@@ -103,7 +103,7 @@ class TableTemplateGenerator(ABC):
         if foreign_keys is None:
             foreign_keys = {}
         # here, we treat the top level
-        # sheets[sheetname][colname]= (COL_TYPE, [path])
+        # sheets[sheetname][colname]= (COL_TYPE, description, [path])
         sheets: dict[str, dict[str, tuple[str, list]]] = {}
         if "properties" not in schema:
             raise ValueError("Inappropriate JSON schema: The following part should contain "
@@ -139,24 +139,42 @@ class TableTemplateGenerator(ABC):
         return selected_keys
 
     def _treat_schema_element(self, schema: dict, sheets: dict = None, path: list = None,
-                              foreign_keys: dict = None, level_in_sheet_name: int = 1
-                              ) -> dict[str, tuple[str, list]]:
-        """ recursively transforms elements from the schema into column definitions """
+                              foreign_keys: dict = None, level_in_sheet_name: int = 1,
+                              array_paths: list = None
+                              ) -> dict[str, tuple[str, str, list]]:
+        """ recursively transforms elements from the schema into column definitions
+
+        sheets is modified in place.
+        Returns
+        -------
+
+            dict
+            describing the columns; see doc string of _generate_sheets_from_schema
+        """
         if not ("type" in schema or "enum" in schema or "oneOf" in schema or "anyOf" in schema):
             raise ValueError("Inappropriate JSON schema: The following part should contain the "
                              f"'type' key:\n{schema}\n")
 
+        if array_paths is None:
+            array_paths = [path]
+
         ctype = ColumnType.SCALAR
 
         # if it is an array, value defs are in 'items'
         if 'type' in schema and schema['type'] == 'array':
-            if 'type' in schema['items'] and schema['items']['type'] == 'object' and len(path) > 1:  # list of references; special treatment
+            if ('type' in schema['items'] and schema['items']['type'] == 'object'
+                    and len(path) > 1):  # list of references; special treatment
                 # we add a new sheet
-                sheets[path[-1]] = self._treat_schema_element(schema['items'], sheets, path,
-                                                              foreign_keys,
-                                                              len(path))
-                for c in self._get_foreign_keys(foreign_keys, path[:-1]):
-                    sheets[path[-1]].update({c: (ColumnType.FOREIGN, f"see sheet '{path[0]}'", path[:-1]+[c])})
+                sheetname = ".".join(path)
+                sheets[sheetname] = self._treat_schema_element(
+                    schema['items'], sheets, path, foreign_keys, len(path),
+                    array_paths=array_paths+[path])
+                print("Sel", path)
+                for p in array_paths:
+                    keys = self._get_foreign_keys(foreign_keys, p)
+                    for k in keys:
+                        print(k, p)
+                        sheets[sheetname].update({k: (ColumnType.FOREIGN, f"see sheet '{path[0]}'", p+[k])})
                 # columns are added to the new sheet, thus we do not return columns
                 return {}
             else:
@@ -175,7 +193,7 @@ class TableTemplateGenerator(ABC):
             for pname in schema["properties"].keys():
                 cols.update(self._treat_schema_element(
                     schema["properties"][pname], sheets, path+[pname], foreign_keys,
-                    level_in_sheet_name))
+                    level_in_sheet_name, array_paths=array_paths))
             return cols
         else:
             description = schema['description'] if 'description' in schema else None