FEAT(scanner): Auto-generate converter_registry and registered_transformer_functions

CHANGELOG.md

@@ -11,11 +11,17 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ### Changed ###
 
+- `scanner.scan_structure_elements` now auto-generates the
+  `converter_registry` and the `registered_transformer_functions` from
+  the `crawler_definition` if none are given. Therefore, the
+  `converter_registry` argument is now optional.
+
 ### Deprecated ###
 
 ### Removed ###
 
 ### Fixed ###
+
 - A RecordType with multiple Parents no longer causes an error during
   collection of identifiables
 

src/caoscrawler/scanner.py

@@ -486,7 +486,7 @@ def scan_directory(dirname: Union[str, list[str]], crawler_definition_path: str,
 
 def scan_structure_elements(items: Union[list[StructureElement], StructureElement],
                             crawler_definition: dict,
-                            converter_registry: dict,
+                            converter_registry: Optional[dict] = None,
                             restricted_path: Optional[list[str]] = None,
                             debug_tree: Optional[DebugTree] = None,
                             registered_transformer_functions: Optional[dict] = None) -> (
@@ -508,6 +508,15 @@ def scan_structure_elements(items: Union[list[StructureElement], StructureElemen
          Traverse the data tree only along the given path. When the end of the
          given path is reached, traverse the full tree as normal. See docstring
          of 'scanner' for more details.
+    converter_registry: dict, optional
+         Optional dictionary containing the converter definitions
+         needed for the crawler definition. If none is given, it will
+         be generated from the `crawler_definition`. Default is None.
+    registered_transformer_functions: dict, optional
+         Optional dictionary containing the transformer function
+         definitions needed for the crawler definition. If none is
+         given, it will be generated from the
+         `crawler_definition`. Default is None.
 
     Returns
     -------
@@ -519,6 +528,10 @@ def scan_structure_elements(items: Union[list[StructureElement], StructureElemen
     if not isinstance(items, list):
         items = [items]
 
+    if converter_registry is None:
+        converter_registry = create_converter_registry(crawler_definition)
+    if registered_transformer_functions is None:
+        registered_transformer_functions = create_transformer_registry(crawler_definition)
     # TODO: needs to be covered somewhere else
     # self.run_id = uuid.uuid1()
     converters = initialize_converters(crawler_definition, converter_registry)