ENH: JsonSchemaExporter can merge schemata and arrayize them

Summary

New utility functions for the Json schema exporter

• merging and arrays
• file properties

Focus

Functionality

Test Environment

Look at the unit tests:

pytest unittests/test_json_schema_exporter.py -k modification

Check List for the Author

Please, prepare your MR for a review. Be sure to write a summary and a focus and create gitlab comments for the reviewer. They should guide the reviewer through the changes, explain your changes and also point out open questions. For further good practices have a look at our review guidelines

• All automated tests pass
• Reference related issues
• Up-to-date CHANGELOG.md (or not necessary)
• Up-to-date JSON schema (or not necessary)
• Appropriate user and developer documentation (or not necessary)
  • How do I use the software? Assume "stupid" users.
  • How do I develop or debug the software? Assume novice developers.
• Annotations in code (Gitlab comments)
  • Intent of new code
  • Problems with old code
  • Why this implementation?

Check List for the Reviewer

• I understand the intent of this MR
• All automated tests pass
• Up-to-date CHANGELOG.md (or not necessary)
• Appropriate user and developer documentation (or not necessary)
• The test environment setup works and the intended behavior is reproducible in the test environment
• In-code documentation and comments are up-to-date.
• Check: Are there specifications? Are they satisfied?

For further good practices have a look at our review guidelines.

src/caosadvancedtools/models/data_model.py

@@ -29,8 +29,9 @@ from copy import deepcopy
 # remove this, when we drop support for old Python versions.
 from typing import List
 
-import caosdb as db
-from caosdb.apiutils import compare_entities, describe_diff
+import linkahead as db
+import linkahead.common.models as models
+from linkahead.apiutils import compare_entities, describe_diff
 
 
 CAOSDB_INTERNAL_PROPERTIES = [
@@ -263,28 +264,56 @@ class DataModel(dict):
 
         return list(all_ents.values())
 
-    def get_deep(self, name: str, visited: set = None):
+    def get_deep(self, name: str, visited_props: set = None, visited_parents: set = None):
         """Attempt to resolve references for the given ``name``.
 
-        This methods only uses data which is available in this datamodel, which acts kind of like a
-        cache pool.
+        The returned entity has all the properties it inherits from its ancestry and all properties
+        have the correct descriptions and datatypes.  This methods only uses data which is available
+        in this DataModel, which acts kind of like a cache pool.
+
+        Note that this may change this data model (subsequent "get" like calls may also return
+        deeper content.)
 
-        Note that this may change this data model (subsequent "get" like calls may also return deep
-        content.)
         """
         entity = self.get(name)
         if not entity:
             return entity
-        if not visited:
-            visited = set()
+        if not visited_props:
+            visited_props = set()
+        if not visited_parents:
+            visited_parents = set()
+
+        importances = {
+            models.OBLIGATORY: 0,
+            models.RECOMMENDED: 1,
+            models.SUGGESTED: 2,
+        }
+
+        for parent in list(entity.get_parents()):  # Make a change-resistant list copy.
+            if parent.name in visited_parents:
+                continue
+            visited_parents.add(parent.name)
+            parent_importance = importances.get(parent._flags.get("inheritance"), 999)
+            if parent.name in self:
+                deep_parent = self.get_deep(parent.name,  # visited_props=visited_props,
+                                            visited_parents=visited_parents
+                                            )
+
+                for prop in deep_parent.properties:
+                    importance = importances[deep_parent.get_importance(prop.name)]
+                    if (importance <= parent_importance
+                            and prop.name not in [prop.name for prop in entity.properties]):
+                        entity.add_property(prop)
+            else:
+                print(f"Referenced parent \"{parent.name}\" not found in data model.")
 
-        # new_props = []
         for prop in list(entity.get_properties()):  # Make a change-resistant list copy.
-            if prop.name in visited:
+            if prop.name in visited_props:
                 continue
-            visited.add(prop.name)
+            visited_props.add(prop.name)
             if prop.name in self:
-                deep_prop = self.get_deep(prop.name, visited=visited)
+                deep_prop = self.get_deep(prop.name, visited_props=visited_props,
+                                          visited_parents=visited_parents)
                 linked_prop = entity.get_property(prop)
                 if not linked_prop.datatype:
                     if deep_prop.role == "Property":
@@ -295,4 +324,5 @@ class DataModel(dict):
                     linked_prop.description = deep_prop.description
             else:
                 print(f"Referenced property \"{prop.name}\" not found in data model.")
+
         return entity