F auto converter and transformer registry

Summary

Part of https://gitlab.indiscale.com/caosdb/customers/horizon-batcat/horizon-batcat-meta/-/issues/150. Makes the behavior of scanner.scan_structure_elements consistent to scanner.scan_directory by auto-generating converter and and transformer registry from the crawler definition.

Focus

Just the new optional behavior. Since it's still ok to give both registries explicitly to reproduce the former behavior, we shouldn't lose any backwards compatibility.

Test Environment

Unit test and manually run the elabFTW crawler from the elabFTW showcase.

Check List for the Author

Please, prepare your MR for a review. Be sure to write a summary and a focus and create gitlab comments for the reviewer. They should guide the reviewer through the changes, explain your changes and also point out open questions. For further good practices have a look at our review guidelines

• All automated tests pass
• Reference related issues
• Up-to-date CHANGELOG.md (or not necessary)
• Up-to-date JSON schema (or not necessary)
• Appropriate user and developer documentation (or not necessary)
  • How do I use the software? Assume "stupid" users.
  • How do I develop or debug the software? Assume novice developers.
• Annotations in code (Gitlab comments)
  • Intent of new code
  • Problems with old code
  • Why this implementation?

Check List for the Reviewer

• I understand the intent of this MR
• All automated tests pass
• Up-to-date CHANGELOG.md (or not necessary)
• Appropriate user and developer documentation (or not necessary)
• The test environment setup works and the intended behavior is reproducible in the test environment
• In-code documentation and comments are up-to-date.
• Check: Are there specifications? Are they satisfied?

For further good practices have a look at our review guidelines.

src/caoscrawler/scanner.py

@@ -486,7 +486,7 @@ def scan_directory(dirname: Union[str, list[str]], crawler_definition_path: str,
 
 def scan_structure_elements(items: Union[list[StructureElement], StructureElement],
                             crawler_definition: dict,
-                            converter_registry: dict,
+                            converter_registry: Optional[dict] = None,
                             restricted_path: Optional[list[str]] = None,
                             debug_tree: Optional[DebugTree] = None,
                             registered_transformer_functions: Optional[dict] = None) -> (
@@ -508,6 +508,15 @@ def scan_structure_elements(items: Union[list[StructureElement], StructureElemen
          Traverse the data tree only along the given path. When the end of the
          given path is reached, traverse the full tree as normal. See docstring
          of 'scanner' for more details.
+    converter_registry: dict, optional
+         Optional dictionary containing the converter definitions
+         needed for the crawler definition. If none is given, it will
+         be generated from the `crawler_definition`. Default is None.
+    registered_transformer_functions: dict, optional
+         Optional dictionary containing the transformer function
+         definitions needed for the crawler definition. If none is
+         given, it will be generated from the
+         `crawler_definition`. Default is None.
 
     Returns
     -------
@@ -519,6 +528,10 @@ def scan_structure_elements(items: Union[list[StructureElement], StructureElemen
     if not isinstance(items, list):
         items = [items]
 
+    if converter_registry is None:
+        converter_registry = create_converter_registry(crawler_definition)
+    if registered_transformer_functions is None:
+        registered_transformer_functions = create_transformer_registry(crawler_definition)
     # TODO: needs to be covered somewhere else
     # self.run_id = uuid.uuid1()
     converters = initialize_converters(crawler_definition, converter_registry)

[Florian Spreckelsen]

This is the actual change. If none is given, create from the crawler definition which should contain the relevant information anyway.