Skip to content
Snippets Groups Projects

F auto converter and transformer registry

Merged F auto converter and transformer registry
f-auto-converter-and-transformer-registry into dev
2 unresolved threads
Merged Florian Spreckelsen requested to merge f-auto-converter-and-transformer-registry into dev
2 unresolved threads

Summary

Part of https://gitlab.indiscale.com/caosdb/customers/horizon-batcat/horizon-batcat-meta/-/issues/150. Makes the behavior of scanner.scan_structure_elements consistent to scanner.scan_directory by auto-generating converter and and transformer registry from the crawler definition.

Focus

Just the new optional behavior. Since it's still ok to give both registries explicitly to reproduce the former behavior, we shouldn't lose any backwards compatibility.

Test Environment

Unit test and manually run the elabFTW crawler from the elabFTW showcase.

Check List for the Author

Please, prepare your MR for a review. Be sure to write a summary and a focus and create gitlab comments for the reviewer. They should guide the reviewer through the changes, explain your changes and also point out open questions. For further good practices have a look at our review guidelines

  • All automated tests pass
  • Reference related issues
  • Up-to-date CHANGELOG.md (or not necessary)
  • Up-to-date JSON schema (or not necessary)
  • Appropriate user and developer documentation (or not necessary)
    • How do I use the software? Assume "stupid" users.
    • How do I develop or debug the software? Assume novice developers.
  • Annotations in code (Gitlab comments)
    • Intent of new code
    • Problems with old code
    • Why this implementation?

Check List for the Reviewer

  • I understand the intent of this MR
  • All automated tests pass
  • Up-to-date CHANGELOG.md (or not necessary)
  • Appropriate user and developer documentation (or not necessary)
  • The test environment setup works and the intended behavior is reproducible in the test environment
  • In-code documentation and comments are up-to-date.
  • Check: Are there specifications? Are they satisfied?

For further good practices have a look at our review guidelines.

Edited by Florian Spreckelsen

Merge request reports

Pipeline #62216 passed

Stage: info
Stage: setup
Stage: cert
Stage: style
Stage: test

Pipeline passed for e9e6bb45 on f-auto-converter-and-transformer-registry

Merged by Henrik tom WördenHenrik tom Wörden 1 month ago (Mar 25, 2025 9:39am UTC)

Loading

Pipeline #62217 passed

Stage: info
Stage: setup
Stage: cert
Stage: style
Stage: test

Pipeline passed for ac2bdf32 on dev

Activity

Filter activity
  • Approvals
  • Assignees & reviewers
  • Comments (from bots)
  • Comments (from users)
  • Commits & branches
  • Edits
  • Labels
  • Lock status
  • Mentions
  • Merge request status
  • Tracking
src/caoscrawler/scanner.py
519 528 if not isinstance(items, list):
520 529 items = [items]
521 530
531 if converter_registry is None:
532 converter_registry = create_converter_registry(crawler_definition)
533 if registered_transformer_functions is None:
534 registered_transformer_functions = create_transformer_registry(crawler_definition)
  • Florian Spreckelsen
    Florian Spreckelsen @florian started a thread on the diff
    • unittests/test_scanner.py
    61 61 converter_registry)
    62 62 assert len(recs) == 4
    63 63
    64 # Converter registry is generated automatically from the crawler
    65 # definition in scan_structure_elements.
    66 recs_without_converter_registry = scan_structure_elements(DictElement(name="", value=data),
    67 crawler_definition)
    68 assert len(recs) == len(recs_without_converter_registry)
  • Florian Spreckelsen requested review from @henrik

    requested review from @henrik

  • Florian Spreckelsen marked the checklist item All automated tests pass as completed

    marked the checklist item All automated tests pass as completed

  • Henrik tom Wörden mentioned in commit ac2bdf32

    mentioned in commit ac2bdf32

  • Henrik tom Wörden merged

    merged

    • Please register or sign in to reply