Extend json-schema model parser

Summary

For https://gitlab.indiscale.com/caosdb/customers/3dmmto/management/-/issues/9.

This implements

• pattern properties
• type null properties
• json schema references
• and json schemas without top-level recordtype specification

for the json schema datamodel parser.

Focus

Mainly additions to the JsonSchemaParser class. We also defined a treatment for type: array properties that lack a strict items specification (i.e., could be lists of elements with mixed types, so a manual specification has to be provided -- or they are ignored altogether).

Test Environment

New unit tests. You can also test it with the chemotion json schema in nextcloud/Customers/3dmmto/example_files/testdaten_chemotion/schema.json.

Check List for the Author

Please, prepare your MR for a review. Be sure to write a summary and a focus and create gitlab comments for the reviewer. They should guide the reviewer through the changes, explain your changes and also point out open questions. For further good practices have a look at our review guidelines

• All automated tests pass
• Reference related issues
• Up-to-date CHANGELOG.md (or not necessary)
• Up-to-date JSON schema (or not necessary)
• Appropriate user and developer documentation (or not necessary)
  • How do I use the software? Assume "stupid" users.
  • How do I develop or debug the software? Assume novice developers.
• Annotations in code (Gitlab comments)
  • Intent of new code
  • Problems with old code
  • Why this implementation?

Check List for the Reviewer

• I understand the intent of this MR
• All automated tests pass
• Up-to-date CHANGELOG.md (or not necessary)
• Appropriate user and developer documentation (or not necessary)
• The test environment setup works and the intended behavior is reproducible in the test environment
• In-code documentation and comments are up-to-date.
• Check: Are there specifications? Are they satisfied?

For further good practices have a look at our review guidelines.

src/caosadvancedtools/models/parser.py

@@ -694,10 +694,19 @@ class JsonSchemaParser(Parser):
                 # Check if this is a valid Json Schema
                 name = self._stringify(elt["title"], context=elt)
                 self._treat_element(elt, name)
-            elif "properties" in elt:
-                for key, prop in elt["properties"].items():
-                    name = self._get_name_from_property(key, prop)
-                    self._treat_element(prop, name)
+            elif "properties" in elt or "patternProperties":
+                # No top-level type but there are entities
+                if "properties" in elt:
+                    for key, prop in elt["properties"].items():
+                        name = self._get_name_from_property(key, prop)
+                        self._treat_element(prop, name)
+                if "patternProperties" in elt:
+                    # See also treatment in ``_treat_record_type``. Since here,
+                    # there is no top-level RT we use the prefix `__Pattern`,
+                    # i.e., the resulting Record Types will be called
+                    # `__PatternElement`.
+                    self._treat_pattern_properties(
+                        elt["patternProperties"], name_prefix="__Pattern")
             else:
                 # Neither RecordType itself, nor further properties in schema,
                 # so nothing to do here. Maybe add something in the future.
@@ -803,6 +812,17 @@ class JsonSchemaParser(Parser):
                     rt.add_property(prop_ent, importance=importance,
                                     datatype=db.LIST(prop_ent))
 
+        if "patternProperties" in elt:
+
+            pattern_property_rts = self_treat_pattern_properties(
+                elt["patternProperties"], name_prefix=name)
+            for ppr in pattern_property_rts:
+                # add reference to pattern property type. These can never be
+                # obligatory since pattern properties cannot be required in the
+                # original schema (since their actual names are not known a
+                # priori).
+                rt.add_property(ppr)
+
         if "description" in elt:
             rt.description = elt["description"]
         return rt
@@ -847,6 +867,21 @@ class JsonSchemaParser(Parser):
                 self.model[ref_rt.name] = ref_rt
                 return db.Property(name=name, datatype=db.LIST(ref_rt)), False
 
+    def _get_pattern_prop(self):
+        if "__pattern_property_pattern_property" in self.model:
+            return self.model["__pattern_property_pattern_property"]
+        pp = db.Property(name="__matched_pattern", datatype=db.TEXT)
+        self.model["__pattern_property_pattern_property"] = pp
+        return pp
+
+    def _treat_pattern_properties(self, pattern_elements, name_prefix=""):
+        """Special Treatment for pattern properties: A RecordType is created for each pattern property."""
+        num_patterns = len(pattern_elements)
+        pattern_prop = self._get_pattern_prop()
+        for ii, (key, element) in enumerate(pattern_elements.items()):
+            name_suffix = f"_{ii+1}" if num_patterns > 1 else ""
+            name = name_prefix + "Element" + name_suffix
+
 
 if __name__ == "__main__":
     parser = argparse.ArgumentParser(description=__doc__,