F integration test baltasar

Summary

See #11. An integration test for a complex use case of the crawler. It was used during a live presentation of caosdb and the new crawler. The presentation covered a use case for inserting simulations with cardiac models by Baltasar Rüchardt.

Focus

Have a look at the assertions in the test file, please. Is there anything missing?

Test Environment

• A: Start an empty docker container and run pytest use_case_simple_presentation.py. This will interactively register the currently running docker container with this test.
• B: Use a previously registered docker container and run pytest use_case_simple_presentation.py.

Check List for the Author

Please, prepare your MR for a review. Be sure to write a summary and a focus and create gitlab comments for the reviewer. They should guide the reviewer through the changes, explain your changes and also point out open questions. For further good practices have a look at our review guidelines

• All automated tests pass
• Reference related issues
• Up-to-date CHANGELOG.md (or not necessary)
• Annotations in code (Gitlab comments)
  • Intent of new code
  • Problems with old code
  • Why this implementation?

Check List for the Reviewer

• I understand the intent of this MR
• All automated tests pass
• Up-to-date CHANGELOG.md (or not necessary)
• The test environment setup works and the intended behavior is reproducible in the test environment
• In-code documentation and comments are up-to-date.
• Check: Are there specifications? Are they satisfied?

For further good practices have a look at our review guidelines.

src/newcrawler/converters.py

@@ -39,6 +39,8 @@ from typing import Optional, Union
 from abc import abstractmethod
 import yaml_header_tools
 
+from caosdb.high_level_api import (CaosDBPythonEntity,
+                                   create_entity_container)
 import yaml
 
 # These are special properties which are (currently) treated differently
@@ -329,10 +331,61 @@ class SimpleFileConverter(Converter):
             return None
         return m.groupdict()
 
+class YamlFileCaosDBRecord(Converter):
+    """
+    Load a file using pylib high level API and convert the contained
+    record into caosdb records.
+    """
+
+    def typecheck(self, element: StructureElement):
+        return isinstance(element, File)
+
+    def create_children(self, generalStore: GeneralStore,
+                        element: StructureElement):
+        return list()
+
+    def match(self, element: StructureElement):
+        if not isinstance(element, File):
+            raise RuntimeError("Element must be a file.")
+        m = re.match(self.definition["match"], element.name)
+        if m is None:
+            return None
+        return m.groupdict()
+
+    def create_records(self, values: GeneralStore,
+                       records: RecordStore,
+                       element: StructureElement):
+        if not isinstance(element, File):
+            raise RuntimeError("A yaml file is needed to create children.")
+        
+        keys_modified = []
+
+        with open(element.path, "r") as f:
+            entries = yaml.safe_load(f)
+
+        entity = CaosDBPythonEntity.deserialize(entries)
+        entities = create_entity_container(entity)
+
+        for n, ent in enumerate(entities):
+            name = ent.name
+            if name is None:
+                name = "YamlRecord_{}".format(n + 1)
+            records[name] = ent
+            values[name] = ent
+
+            for propname in ent.properties:
+                keys_modified.append((name, propname.name))
+
+        # Process the records section of the yaml definition:
+        keys_modified.extend(
+            super().create_records(values, records, element))
+        
+        return keys_modified
+    
 
 class MarkdownFileConverter(Converter):
     def __init__(self, definition: dict, name: str,
-                 converter_registry: dict):
+                          converter_registry: dict):
         """
         Initialize a new directory converter.
         """