F schema export references

Summary

For https://gitlab.indiscale.com/caosdb/customers/dimr/management/-/issues/38 Extends !80 (merged) by treatment of references and list of references.

(List of) References are translated int (arrays of) "oneOf" properties: They include a subschema for the creation of new Records of the referenced types, and an enum with the existing Records. A special case are reference properties with the generic db.REFERENCE datatype. Since here, we can't create a new Record (since we don't know a type), only the enum with all Records and Files is used.

Focus

RecordType treatment has been refactored s.th. it can be used recursively to create subschemas. Apart from this and the generation of the enum values, the existing property treatment was already enough.

Test Environment

Unit and integration tests should be sufficient but feel free to test with your favorite RT from a live system, too.

Check List for the Author

Please, prepare your MR for a review. Be sure to write a summary and a focus and create gitlab comments for the reviewer. They should guide the reviewer through the changes, explain your changes and also point out open questions. For further good practices have a look at our review guidelines

• All automated tests pass
• Reference related issues
• Up-to-date CHANGELOG.md (or not necessary)
• Up-to-date JSON schema (or not necessary)
• Appropriate user and developer documentation (or not necessary)
  • How do I use the software? Assume "stupid" users.
  • How do I develop or debug the software? Assume novice developers.
• Annotations in code (Gitlab comments)
  • Intent of new code
  • Problems with old code
  • Why this implementation?

Check List for the Reviewer

• I understand the intent of this MR
• All automated tests pass
• Up-to-date CHANGELOG.md (or not necessary)
• Appropriate user and developer documentation (or not necessary)
• The test environment setup works and the intended behavior is reproducible in the test environment
• In-code documentation and comments are up-to-date.
• Check: Are there specifications? Are they satisfied?

For further good practices have a look at our review guidelines.

src/caosadvancedtools/json_schema_exporter.py

@@ -33,14 +33,22 @@ def _make_required_list(rt: db.RecordType):
             if rt.get_importance(prop.name) == db.OBLIGATORY]
 
 
-def _make_prop_from_prop(prop: db.Property, additional_options_for_text_props: Optional[dict],
-                         units_in_description: bool):
+def _make_segment_from_prop(prop: db.Property, additional_properties: bool,
+                            name_and_description_in_properties: bool,
+                            additional_options_for_text_props: Optional[dict],
+                            units_in_description: bool):
     """Return the JSON Schema segment for the given property
 
     Parameters
     ----------
     prop : db.Property
         the property to be transformed
+    additional_properties : bool, optional
+        Whether additional properties will be admitted in the resulting
+        schema. Optional, default is True.
+    name_and_description_in_properties : bool, optional
+        Whether to include name and description in the `properties` section of
+        the schema to be exported. Optional, default is False.
     additional_options_for_text_props : Optional[dict]
         dict of dicts that may contain the keys 'pattern' and 'format' to
         further define the rules for the JSON Schema segment
@@ -50,11 +58,6 @@ def _make_prop_from_prop(prop: db.Property, additional_options_for_text_props: O
         instead.
     """
 
-    if prop.is_reference():
-        raise NotImplementedError(
-            "Reference properties are not supported in this version of the json schema exporter."
-        )
-
     if not additional_options_for_text_props:
         additional_options_for_text_props = {}
     if prop.datatype == db.TEXT or prop.datatype == db.DATETIME:
@@ -96,8 +99,32 @@ def _make_prop_from_prop(prop: db.Property, additional_options_for_text_props: O
         json_prop["type"] = "array"
         list_element_prop = db.Property(
             name=prop.name, datatype=get_list_datatype(prop.datatype, strict=True))
-        json_prop["items"] = _make_prop_from_prop(
-            list_element_prop, additional_options_for_text_props, units_in_description)
+        json_prop["items"] = _make_segment_from_prop(
+            list_element_prop, additional_properties,
+            name_and_description_in_properties, additional_options_for_text_props,
+            units_in_description
+        )
+    elif prop.is_reference():
+        if prop.datatype == db.REFERENCE:
+            # No Record creation since no RT is specified and we don't know what
+            # schema to use, so only enum of all Records and all Files.
+            values = _retrieve_enum_values("RECORD") + _retrieve_enum_values("FILE")
+            json_prop["enum"] = values
+        elif prop.datatype == db.FILE:
+            # TODO: different issue
+            raise NotImplementedError("Files have not been implemented yet.")
+        else:
+            values = _retrieve_enum_values(f"RECORD '{prop.datatype}'")
+            rt = db.execute_query(f"FIND RECORDTYPE WITH name='{prop.datatype}'", unique=True)
+            subschema = _make_segment_from_recordtype(rt, additional_properties,
+                                                      name_and_description_in_properties,
+                                                      additional_options_for_text_props,
+                                                      units_in_description)
+            json_prop["oneOf"] = [
+                {"enum": values},
+                subschema
+            ]
+
     else:
         raise ValueError(
             f"Unknown or no property datatype. Property {prop.name} with type {prop.datatype}")
@@ -135,6 +162,54 @@ def _make_text_property(description="", text_format=None, text_pattern=None):
     return prop
 
 
+def _retrieve_enum_values(role: str):
+
+    possible_values = db.execute_query(f"SELECT name, id FROM {role}")
+
+    vals = []
+    for val in possible_values:
+        if val.name:
+            vals.append(f"{val.id}, {val.name}")
+        else:
+            vals.append(f"{val.id}")
+
+    return vals
+
+
+def _make_segment_from_recordtype(rt: db.RecordType, additional_properties: bool = True,
+                                  name_and_description_in_properties: bool = False,
+                                  additional_options_for_text_props: Optional[dict] = None,
+                                  units_in_description: bool = True):
+    """Return a Json schema segment for the given RecordType.
+    """
+    schema = {
+        "type": "object"
+    }
+
+    schema["required"] = _make_required_list(rt)
+    schema["additionalProperties"] = additional_properties
+
+    props = {}
+    if name_and_description_in_properties:
+        props["name"] = _make_text_property("The name of the Record to be created")
+        props["description"] = _make_text_property("The description of the Record to be created")
+
+    for prop in rt.properties:
+        if prop.name in props:
+            # Multi property
+            raise NotImplementedError(
+                "Creating a schema for multi-properties is not specified. "
+                f"Property {prop.name} occurs more than once."
+            )
+        props[prop.name] = _make_segment_from_prop(
+            prop, additional_properties, name_and_description_in_properties,
+            additional_options_for_text_props, units_in_description)
+
+    schema["properties"] = props
+
+    return schema
+
+
 def recordtype_to_json_schema(rt: db.RecordType, additional_properties: bool = True,
                               name_and_description_in_properties: bool = False,
                               additional_options_for_text_props: Optional[dict] = None,
@@ -171,32 +246,14 @@ def recordtype_to_json_schema(rt: db.RecordType, additional_properties: bool = T
     if additional_options_for_text_props is None:
         additional_options_for_text_props = {}
 
-    schema = {
-        "$schema": "https://json-schema.org/draft/2019-09/schema",
-        "type": "object"
-    }
+    schema = _make_segment_from_recordtype(rt, additional_properties,
+                                           name_and_description_in_properties,
+                                           additional_options_for_text_props,
+                                           units_in_description)
+    schema["$schema"] = "https://json-schema.org/draft/2019-09/schema"
     if rt.name:
         schema["title"] = rt.name
     if rt.description:
         schema["description"] = rt.description
 
-    schema["required"] = _make_required_list(rt)
-    schema["additionalProperties"] = additional_properties
-
-    props = {}
-    if name_and_description_in_properties:
-        props["name"] = _make_text_property("The name of the Record to be created")
-        props["description"] = _make_text_property("The description of the Record to be created")
-
-    for prop in rt.properties:
-        if prop.name in props:
-            # Multi property
-            raise NotImplementedError(
-                "Creating a schema for multi-properties is not specified. "
-                f"Property {prop.name} occurs more than once."
-            )
-        props[prop.name] = _make_prop_from_prop(
-            prop, additional_options_for_text_props, units_in_description)
-
-    schema["properties"] = props
     return schema