Draft: New debug tree

Summary

This MR implements a completely overworked and heavily improved debug tree that can be used to:

• trace errors in the crawler code
• find the causes of bugs and unexpected behavior in cfood files
• write unit tests for the crawler

The functionality is implemented in the scanner module which receives an additional argument for the new_debug_tree. This argument is used recursively to generate a tree structure of roughly the following form:

- SE1:
    matching_converters:
      - C1:
          - SE1a
          - SE1b
          - SE1c
          - ...
- SE2
- SE3
- ...

So every level of structure elements (SEx) includes all information about a specific structure element that was scanned. The attribute matching_converters includes all information about matching converters (which recurses in its subtree).

There is a function in the scanner module now that allows turning this information into an HTML-file with collapsible tree elements (using CSS).

Here is an example of how the output looks like:

This MR furthermore implements a small change that allows to deactivate specific levels in the CFood using the "deactivate" key.

Test Environment

This should be manually tested by generating the scanner output files using the function save_debug_tree. Furthermore, there are some unit tests.

Check List for the Author

Please, prepare your MR for a review. Be sure to write a summary and a focus and create gitlab comments for the reviewer. They should guide the reviewer through the changes, explain your changes and also point out open questions. For further good practices have a look at our review guidelines

• All automated tests pass
• Reference related issues
• Up-to-date CHANGELOG.md (or not necessary)
• Up-to-date JSON schema (or not necessary)
• Appropriate user and developer documentation (or not necessary)
  • How do I use the software? Assume "stupid" users.
  • How do I develop or debug the software? Assume novice developers.
• Annotations in code (Gitlab comments)
  • Intent of new code
  • Problems with old code
  • Why this implementation?

Check List for the Reviewer

• I understand the intent of this MR
• All automated tests pass
• Up-to-date CHANGELOG.md (or not necessary)
• Appropriate user and developer documentation (or not necessary)
• The test environment setup works and the intended behavior is reproducible in the test environment
• In-code documentation and comments are up-to-date.
• Check: Are there specifications? Are they satisfied?

For further good practices have a look at our review guidelines.

src/caoscrawler/converters/converters.py

@@ -438,6 +438,15 @@ class Converter(object, metaclass=ABCMeta):
 
         self.setup()
 
+    def get_dict(self):
+        """
+        Return a dictionary containing all the attributes from this structure element.
+        This is used for the (new) debug tree.
+        """
+        return {"name": self.name,
+                "type": str(type(self))[8:-2],
+                "definition": self.definition}
+
     def setup(self):
         """
         Analogous to `cleanup`. Can be used to set up variables that are permanently