Allow to set breakpoints in crawler definitions

Summary

#187

This idea is very simple:

As soon as a Converter matches, activate the pydb if "breakpoint: true" is part of its definition.

This allows to interactively step through the scanning process at the point where a scanner matches.

Test Environment

Use a cfood of your choice, add "breakpoint: true" somewhere and run the scanner/crawler.

Validate with "breakpoint: false" that it is not run in this case.

Check List for the Author

Please, prepare your MR for a review. Be sure to write a summary and a focus and create gitlab comments for the reviewer. They should guide the reviewer through the changes, explain your changes and also point out open questions. For further good practices have a look at our review guidelines

• All automated tests pass
• Reference related issues
• Up-to-date CHANGELOG.md (or not necessary)
• Up-to-date JSON schema (or not necessary)
• Appropriate user and developer documentation (or not necessary)
  • How do I use the software? Assume "stupid" users.
  • How do I develop or debug the software? Assume novice developers.
• Annotations in code (Gitlab comments)
  • Intent of new code
  • Problems with old code
  • Why this implementation?

Check List for the Reviewer

• I understand the intent of this MR
• All automated tests pass
• Up-to-date CHANGELOG.md (or not necessary)
• Appropriate user and developer documentation (or not necessary)
• The test environment setup works and the intended behavior is reproducible in the test environment
• In-code documentation and comments are up-to-date.
• Check: Are there specifications? Are they satisfied?

For further good practices have a look at our review guidelines.

src/caoscrawler/scanner.py

@@ -348,6 +348,12 @@ def scanner(items: list[StructureElement],
             if (converter.typecheck(element) and (
                     restricted_path is None or element.name == restricted_path[0])
                     and converter.match(element) is not None):
+
+                # Set a breakpoint that opens pydb if "breakpoint" is defined for this converter
+                # and set to a true value:
+                if ("breakpoint" in converter.definition and
+                        bool(converter.definition["breakpoint"])):
+                    breakpoint()
                 path_found = True
                 general_store_copy = general_store.create_scoped_copy()
                 record_store_copy = record_store.create_scoped_copy()