ENH: apply paging such that no unwanted jobs are being triggered

Summary

Performance was bad because a paged retrieve would also load and execute jobs for entities which are not on the requested page.

Focus

Two things happened:

1. There was a trivial but wrong implementation of Entity#skipJob(). Jobs would never be skipped, even if an entity had entity status IGNORE. This was a one-liner.
2. Because the paging was being applied after the query (and other jobs) some (and possibly A LOT) entity jobs would be added to the schedule which are not necessary because the entities are not the requested page. This is the rest of the changes.

Test Environment

Unit test should be sufficient.

Check List for the Author

Please, prepare your MR for a review. Be sure to write a summary and a focus and create gitlab comments for the reviewer. They should guide the reviewer through the changes, explain your changes and also point out open questions. For further good practices have a look at our review guidelines

• All automated tests pass
• Reference related issues
• Up-to-date CHANGELOG.md (or not necessary)
• Annotations in code (Gitlab comments)
  • Intent of new code
  • Problems with old code
  • Why this implementation?

Check List for the Reviewer

• I understand the intent of this MR
• All automated tests pass
• Up-to-date CHANGELOG.md (or not necessary)
• The test environment setup works and the intended behavior is reproducible in the test environment
• In-code documentation and comments are up-to-date.
• Check: Are there specifications? Are they satisfied?

For further good practices have a look at our review guidelines.

assigned to @henrik

added 1 commit

• abbe3cbc - FIX: query with no index

Compare with previous version

marked this merge request as draft

added 43 commits

• abbe3cbc...05379e64 - 42 commits from branch dev
• 65169ad0 - Merge branch 'dev' into f-enh-paging

Compare with previous version

added 39 commits

• 65169ad0...35a40981 - 38 commits from branch dev
• a09288a7 - Merge branch 'dev' into f-enh-paging

Compare with previous version

added 1 commit

• 8d288196 - WIP: enhanced paging

Compare with previous version

marked this merge request as ready

changed title from draft: ENH: apply paging earlier to ENH: apply paging such that no unwanted jobs are being triggered

changed the description

requested review from @henrik

assigned to @timm and unassigned @henrik

marked the checklist item All automated tests pass as completed

marked the checklist item Reference related issues as completed

added 1 commit

• 316db498 - DOC: update CHANGELOG

Compare with previous version

marked the checklist item Up-to-date CHANGELOG.md (or not necessary) as completed

marked the checklist item Annotations in code (Gitlab comments) as completed

added 1 commit

• a88a62cb - DOC: update CHANGELOG

Compare with previous version

resolved all threads

I am ok with this MR if the paging variables are name startIndex and stopIndex instead of index an length.

added 1 commit

• 7028a06b - MAINT: rename paging variables to startIndex, endIndex

Compare with previous version

resolved all threads

enabled an automatic merge when the pipeline for 7028a06b succeeds

mentioned in commit 31243e69

merged