Create a new scanner module and move functions from crawl module there

Summary

Refactoring of the crawler and scanner:

• I basically split the crawler into two parts:
  • Scanner module that walks through the file system and other structure elements and collects all the information into record types.
  • The crawler module (crawl.py) does everything else which needs a CaosDB interaction.

The main steps in refactoring were:

• Extraction of the functions related only to the scanning process.
• Slight renaming and consistency checks.
• Adapting all the tests to the new structure.

Left TODO:

• Fixing the integration tests

Focus

The best procedure for the review probably is to go through the individual commits. I tried to keep them as fine-grained as possible. Fixing the tests actually was very repetitive, because the API was changed slightly.

There is one thing that might be solved not completely ideal:

• Setting of the runid and the crawled_directory attributes. These are member variables that were previously set by functions now contained in the independent scanner module. This might need some cross-checking with @henrik who probably introduced these variables for logging.

Test Environment

Unittests

I did not run the integration tests manually, but this is probably done by the pipeline.

Check List for the Author

Please, prepare your MR for a review. Be sure to write a summary and a focus and create gitlab comments for the reviewer. They should guide the reviewer through the changes, explain your changes and also point out open questions. For further good practices have a look at our review guidelines

• All automated tests pass
• Reference related issues
• Up-to-date CHANGELOG.md (or not necessary)
• Up-to-date JSON schema (or not necessary)
• Appropriate user and developer documentation (or not necessary)
  • How do I use the software? Assume "stupid" users.
  • How do I develop or debug the software? Assume novice developers.
• Annotations in code (Gitlab comments)
  • Intent of new code
  • Problems with old code
  • Why this implementation?

Check List for the Reviewer

• I understand the intent of this MR
• All automated tests pass
• Up-to-date CHANGELOG.md (or not necessary)
• Appropriate user and developer documentation (or not necessary)
• The test environment setup works and the intended behavior is reproducible in the test environment
• In-code documentation and comments are up-to-date.
• Check: Are there specifications? Are they satisfied?

For further good practices have a look at our review guidelines.

src/caoscrawler/scanner.py

@@ -164,6 +164,8 @@ def create_converter_registry(definition: dict):
     - module is the name of the module to be imported which must be installed
     - class is the converter class to load and associate with this converter entry
 
+    Formerly known as "load_converters".
+
     all other info for the converter needs to be included in the converter plugin
     directory:
     schema.yml file
@@ -337,10 +339,12 @@ def _crawl(self,
 # --------------------------------------------------------------------------------
 
 
-def crawl_directory(dirname: str, crawler_definition_path: str,
+def scan_directory(dirname: str, crawler_definition_path: str,
                     restricted_path: Optional[list[str]] = None):
     """ Crawl a single directory.
 
+    Formerly known as "crawl_directory".
+
     Convenience function that starts the crawler (calls start_crawling)
     with a single directory as the StructureElement.
 
@@ -351,7 +355,7 @@ def crawl_directory(dirname: str, crawler_definition_path: str,
 
     crawler_definition = load_definition(crawler_definition_path)
     # Load and register converter packages:
-    converter_registry = load_converters(crawler_definition)
+    converter_registry = create_converter_registry(crawler_definition)
 
     if not dirname:
         raise ValueError(
@@ -366,21 +370,22 @@ def crawl_directory(dirname: str, crawler_definition_path: str,
             # dirname had a trailing '/'
             dir_structure_name = os.path.basename(dirname[:-1])
 
-    start_crawling(Directory(dir_structure_name,
-                                  dirname),
-                        crawler_definition,
-                        converter_registry,
-                        restricted_path=restricted_path
-                        )
+    return scan_structure_elements(Directory(dir_structure_name,
+                                             dirname),
+                                   crawler_definition,
+                                   converter_registry,
+                                   restricted_path=restricted_path)
     
 
-def start_crawling(items: Union[list[StructureElement], StructureElement],
-                   crawler_definition: dict,
-                   converter_registry: dict,
-                   restricted_path: Optional[list[str]] = None):
+def scan_structure_elements(items: Union[list[StructureElement], StructureElement],
+                            crawler_definition: dict,
+                            converter_registry: dict,
+                            restricted_path: Optional[list[str]] = None):
     """
     Start point of the crawler recursion.
 
+    Formerly known as "start_crawling".
+
     Parameters
     ----------
     items: list
@@ -400,27 +405,17 @@ def start_crawling(items: Union[list[StructureElement], StructureElement],
     """
 
     # This function builds the tree of converters out of the crawler definition.
-
-    if self.generalStore is None:
-        raise RuntimeError("Should not happen.")
-
     if not isinstance(items, list):
         items = [items]
 
     self.run_id = uuid.uuid1()
-    local_converters = Crawler.initialize_converters(crawler_definition, converter_registry)
+    converters = initialize_converters(crawler_definition, converter_registry)
 
-    # This recursive crawling procedure generates the update list:
-    self.crawled_data: list[db.Record] = []
-    self._crawl(
+    return _crawl(
         items=items,
-        local_converters=local_converters,
+        local_converters=converters,
         generalStore=self.generalStore,
         recordStore=self.recordStore,
         structure_elements_path=[],
         converters_path=[],
         restricted_path=restricted_path)
-    if self.debug:
-        self.debug_converters = local_converters
-
-    return self.crawled_data