Create a new scanner module and move functions from crawl module there

Summary

Refactoring of the crawler and scanner:

• I basically split the crawler into two parts:
  • Scanner module that walks through the file system and other structure elements and collects all the information into record types.
  • The crawler module (crawl.py) does everything else which needs a CaosDB interaction.

The main steps in refactoring were:

• Extraction of the functions related only to the scanning process.
• Slight renaming and consistency checks.
• Adapting all the tests to the new structure.

Left TODO:

• Fixing the integration tests

Focus

The best procedure for the review probably is to go through the individual commits. I tried to keep them as fine-grained as possible. Fixing the tests actually was very repetitive, because the API was changed slightly.

There is one thing that might be solved not completely ideal:

• Setting of the runid and the crawled_directory attributes. These are member variables that were previously set by functions now contained in the independent scanner module. This might need some cross-checking with @henrik who probably introduced these variables for logging.

Test Environment

Unittests

I did not run the integration tests manually, but this is probably done by the pipeline.

Check List for the Author

Please, prepare your MR for a review. Be sure to write a summary and a focus and create gitlab comments for the reviewer. They should guide the reviewer through the changes, explain your changes and also point out open questions. For further good practices have a look at our review guidelines

• All automated tests pass
• Reference related issues
• Up-to-date CHANGELOG.md (or not necessary)
• Up-to-date JSON schema (or not necessary)
• Appropriate user and developer documentation (or not necessary)
  • How do I use the software? Assume "stupid" users.
  • How do I develop or debug the software? Assume novice developers.
• Annotations in code (Gitlab comments)
  • Intent of new code
  • Problems with old code
  • Why this implementation?

Check List for the Reviewer

• I understand the intent of this MR
• All automated tests pass
• Up-to-date CHANGELOG.md (or not necessary)
• Appropriate user and developer documentation (or not necessary)
• The test environment setup works and the intended behavior is reproducible in the test environment
• In-code documentation and comments are up-to-date.
• Check: Are there specifications? Are they satisfied?

For further good practices have a look at our review guidelines.

src/caoscrawler/scanner.py

@@ -224,19 +224,21 @@ def initialize_converters(crawler_definition: dict, converter_registry: dict):
 # Main scanner function:
 # --------------------------------------------------------------------------------
 
-def _crawl(self,
-           items: list[StructureElement],
-           local_converters: list[Converter],
-           generalStore: GeneralStore,
-           recordStore: RecordStore,
-           structure_elements_path: list[str],
-           converters_path: list[str],
-           restricted_path: Optional[list[str]] = None):
+def scanner(self,
+            items: list[StructureElement],
+            converters: list[Converter],
+            generalStore: GeneralStore,
+            recordStore: RecordStore,
+            structure_elements_path: list[str],
+            converters_path: list[str],
+            restricted_path: Optional[list[str]] = None):
     """
     Crawl a list of StructureElements and apply any matching converters.
 
+    Formerly known as "_crawl".
+
     items: structure_elements (e.g. files and folders on one level on the hierarchy)
-    local_converters: locally defined converters for
+    converters: locally defined converters for
                         treating structure elements. A locally defined converter could be
                         one that is only valid for a specific subtree of the originally
                         cralwed StructureElement structure.
@@ -257,7 +259,7 @@ def _crawl(self,
         restricted_path = None
 
     for element in items:
-        for converter in local_converters:
+        for converter in converters:
 
             # type is something like "matches files", replace isinstance with "type_matches"
             # match function tests regexp for example
@@ -302,7 +304,7 @@ def _crawl(self,
                             structure_elements_path + [element.get_name()],
                             converters_path + [converter.name])
 
-                self._crawl(children, converter.converters,
+                self.scanner(children, converter.converters,
                             generalStore_copy, recordStore_copy,
                             structure_elements_path + [element.get_name()],
                             converters_path + [converter.name],
@@ -411,9 +413,9 @@ def scan_structure_elements(items: Union[list[StructureElement], StructureElemen
     self.run_id = uuid.uuid1()
     converters = initialize_converters(crawler_definition, converter_registry)
 
-    return _crawl(
+    return scanner(
         items=items,
-        local_converters=converters,
+        converters=converters,
         generalStore=self.generalStore,
         recordStore=self.recordStore,
         structure_elements_path=[],