F dict heuristic

Summary

Part of https://gitlab.indiscale.com/caosdb/customers/3dmmto/management/-/issues/30. Introduce a converter that automatically transforms dictionary entries to property values without having to specify them explicitly. It also also creates referenced Records from object-typed dictionary values. Properties can be blacklisted.

The new converter extends the existing DictElementConverter, so we can still use the existing treatment via records and subtree entries in the converter definition. This allows for the special treatment of blacklisted dictionary keys, but it is not limited to them. We do not change the original create_children method, so all dictionary entries are also available in the subtree if needed elsewhere.

Focus

Mainly the new converter and its documentation.

Test Environment

make doc and check the new documentation. Is it understandable? Is it useful?

For actual testing the new unit test should be sufficient. Can you think of corner cases that they do not cover yet?

Check List for the Author

Please, prepare your MR for a review. Be sure to write a summary and a focus and create gitlab comments for the reviewer. They should guide the reviewer through the changes, explain your changes and also point out open questions. For further good practices have a look at our review guidelines

• All automated tests pass
• Reference related issues
• Up-to-date CHANGELOG.md (or not necessary)
• Up-to-date JSON schema (or not necessary)
• Appropriate user and developer documentation (or not necessary)
  • How do I use the software? Assume "stupid" users.
  • How do I develop or debug the software? Assume novice developers.
• Annotations in code (Gitlab comments)
  • Intent of new code
  • Problems with old code
  • Why this implementation?

Check List for the Reviewer

• I understand the intent of this MR
• All automated tests pass
• Up-to-date CHANGELOG.md (or not necessary)
• Appropriate user and developer documentation (or not necessary)
• The test environment setup works and the intended behavior is reproducible in the test environment
• In-code documentation and comments are up-to-date.
• Check: Are there specifications? Are they satisfied?

For further good practices have a look at our review guidelines.

src/caoscrawler/converters.py

@@ -811,10 +811,159 @@ class DictElementConverter(Converter):
         return match_name_and_value(self.definition, element.name, element.value)
 
 
+class PropertiesFromDictConverter(DictElementConverter):
+    """Extend the :py:class:`DictElementConverter` by a heuristic to set
+    property values from the dictionary keys.
+
+    """
+
+    def _validate_definition(self):
+
+        if "record_from_dict" not in self.definition or self.definition["record_from_dict"] is None:
+
+            raise ValueError(
+                "You need to specify the (root) record, the properties of "
+                f"which will be set from the dict in converter {self.name}."
+            )
+
+        if not "variable_name" in self.definition["record_from_dict"] or not self.definition["record_from_dict"]["variable_name"]:
+
+            raise ValueError(
+                f"The root record in converter {self.name} needs to have a "
+                "`variable_name` by which it is accessed in the subtree."
+            )
+
+    def __init__(self, definition: dict, name: str, converter_registry: dict,
+                 referenced_record_callback: Optional[callable] = None):
+
+        super().__init__(definition, name, converter_registry)
+        self._validate_definition()
+        self.referenced_record_callback = referenced_record_callback
+
+    def _recursively_create_records(self, subdict: dict, root_record: db.Record,
+                                    root_rec_name: str,
+                                    values: GeneralStore, records: RecordStore,
+                                    referenced_record_callback: callable,
+                                    keys_modified: list = []
+                                    ):
+        """Create a record form the given `subdict` and recursively create referenced records."""
+
+        blacklisted_keys = self.definition["record_from_dict"][
+            "properties_blacklist"] if "properties_blacklist" in self.definition["record_from_dict"] else []
+        special_references = self.definition["record_from_dict"]["references"] if "references" in self.definition["record_from_dict"] else [
+        ]
+
+        for key, value in subdict.items():
+
+            if key in blacklisted_keys:
+                # We ignore this in the automated property generation
+                continue
+            if isinstance(value, list):
+                if not any([isinstance(val, dict) for val in value]):
+                    # no dict in list, i.e., no references, so this is simple
+                    root_record.add_property(name=key, value=value)
+                else:
+                    if not all([isinstance(val, dict) for val in value]):
+                        # if this is not an error (most probably it is), this
+                        # needs to be handled manually for now.
+                        raise ValueError(
+                            f"{key} in {subdict} contains a mixed list of references and scalars.")
+                    ref_recs = []
+                    for ii, ref_dict in enumerate(value):
+                        ref_rec = db.Record()
+                        ref_var_name = f"{root_rec_name}.{key}.{ii}"
+                        if key in special_references:
+                            for par in special_references[key]["parents"]:
+                                ref_rec.add_parent(par)
+                        else:
+                            ref_rec.add_parent(key)
+                        records[ref_var_name] = ref_rec
+                        values[ref_var_name] = ref_rec
+                        keys_modified, ref_rec = self._recursively_create_records(
+                            subdict=ref_dict,
+                            root_record=ref_rec,
+                            root_rec_name=ref_var_name,
+                            values=values,
+                            records=records,
+                            referenced_record_callback=referenced_record_callback,
+                            keys_modified=keys_modified,
+                        )
+                        ref_recs.append(ref_rec)
+                    root_record.add_property(name=key, value=ref_recs)
+
+            elif isinstance(value, dict):
+                ref_rec = db.Record()
+                ref_var_name = f"{root_rec_name}.{key}"
+                if key in special_references:
+                    for par in special_references[key]["parents"]:
+                        ref_rec.add_parent(par)
+                else:
+                    ref_rec.add_parent(key)
+                records[ref_var_name] = ref_rec
+                values[ref_var_name] = ref_rec
+                keys_modified, ref_rec = self._recursively_create_records(
+                    subdict=value,
+                    root_record=ref_rec,
+                    root_rec_name=ref_var_name,
+                    values=values,
+                    records=records,
+                    referenced_record_callback=referenced_record_callback,
+                    keys_modified=keys_modified
+                )
+                root_record.add_property(key, ref_rec)
+            else:
+                if key.lower() in SPECIAL_PROPERTIES:
+                    setattr(root_record, key.lower(), value)
+                else:
+                    root_record.add_property(name=key, value=value)
+            keys_modified.append((root_rec_name, key))
+
+        if referenced_record_callback:
+            root_record = referenced_record_callback(root_record)
+
+        return keys_modified, root_record
+
+    def create_records(self, values: GeneralStore, records: RecordStore,
+                       element: StructureElement):
+
+        keys_modified = []
+
+        rfd = self.definition["record_from_dict"]
+        if rfd["variable_name"] not in records:
+            rec = db.Record()
+            if "name" in rfd:
+                rec.name = rfd["name"]
+            if "parents" in rfd:
+                for par in rfd["parents"]:
+                    rec.add_parent(par)
+            else:
+                rec.add_parent(rfd["variable_name"])
+            records[rfd["variable_name"]] = rec
+            values[rfd["variable_name"]] = rec
+
+        else:
+            rec = records[rfd["variable_name"]]
+
+        keys_modified, rec = self._recursively_create_records(
+            subdict=element.value,
+            root_record=rec,
+            root_rec_name=rfd["variable_name"],
+            values=values,
+            records=records,
+            referenced_record_callback=self.referenced_record_callback,
+            keys_modified=keys_modified,
+        )
+
+        keys_modified.extend(super().create_records(
+            values=values, records=records, element=element))
+
+        return keys_modified
+
+
 class DictConverter(DictElementConverter):
     def __init__(self, *args, **kwargs):
         warnings.warn(DeprecationWarning(
-            "This class is depricated. Please use DictConverter."))
+            "This class is depricated. Please use DictElementConverter."))
         super().__init__(*args, **kwargs)