MAINT: allow to start the crawler with generic structure elements

Summary

Early tests were always done with dictionaries. However, it should be possible to simply start the crawler using any other StructureElement as root. This required a small change in crawl.py were the first directory was previously treated specially.

Focus

Unfortunately, this change needs some changes to cfood.yml(root directory needs to be added) files and to the unittests (One more match exists.)

You might want to look at the changes using: git diff dev -w

Test Environment

How to set up a test environment for manual testing?

Check List for the Author

Please, prepare your MR for a review. Be sure to write a summary and a focus and create gitlab comments for the reviewer. They should guide the reviewer through the changes, explain your changes and also point out open questions. For further good practices have a look at our review guidelines

• All automated tests pass
• Reference related issues
• Up-to-date CHANGELOG.md (or not necessary)
• Annotations in code (Gitlab comments)
  • Intent of new code
  • Problems with old code
  • Why this implementation?

Check List for the Reviewer

• I understand the intent of this MR
• All automated tests pass
• Up-to-date CHANGELOG.md (or not necessary)
• The test environment setup works and the intended behavior is reproducible in the test environment
• In-code documentation and comments are up-to-date.
• Check: Are there specifications? Are they satisfied?

For further good practices have a look at our review guidelines.

src/newcrawler/crawl.py

@@ -286,13 +286,13 @@ class Crawler(object):
 
         return local_converters
 
-    def start_crawling(self, item: StructureElement,
+    def start_crawling(self, items,
                        crawler_definition: dict,
                        converter_registry: dict):
         """
         Start point of the crawler recursion.
 
-        item: A structure element that is used for generating the initial items for the crawler.
+        items: A list of structure elements that is used for generating the initial items for the crawler.
               This could e.g. be a Directory.
         crawler_definition: A dictionary representing the crawler definition, possibly from a yaml
               file.
@@ -305,11 +305,14 @@ class Crawler(object):
         if self.generalStore is None:
             raise RuntimeError("Should not happen.")
 
+        if not isinstance(items, list):
+            items = [items]
+
         local_converters = Crawler.create_local_converters(crawler_definition,
                                                            converter_registry)
         # This recursive crawling procedure generates the update list:
         self.updateList: list[db.Record] = []
-        self._crawl([item],
+        self._crawl(items,
                     self.global_converters, local_converters, self.generalStore, self.recordStore,
                     [], [])