Skip to content
Snippets Groups Projects

F fix crawler overwrite

Merged Alexander Schlemmer requested to merge f-fix-crawler-overwrite into dev
All threads resolved!

Summary

Fix for: https://gitlab.com/caosdb/caosdb-crawler/-/issues/23 and #47 (closed)

Test Environment

There is an integration test provided. For manual testing, you need pylib in the most recent dev branch, otherwise merge_entities is missing the force option.

Check List for the Author

Please, prepare your MR for a review. Be sure to write a summary and a focus and create gitlab comments for the reviewer. They should guide the reviewer through the changes, explain your changes and also point out open questions. For further good practices have a look at our review guidelines

  • All automated tests pass
  • Reference related issues
  • Up-to-date CHANGELOG.md (or not necessary)
  • Annotations in code (Gitlab comments)
    • Intent of new code
    • Problems with old code
    • Why this implementation?

Check List for the Reviewer

  • I understand the intent of this MR
  • All automated tests pass
  • Up-to-date CHANGELOG.md (or not necessary)
  • The test environment setup works and the intended behavior is reproducible in the test environment
  • In-code documentation and comments are up-to-date.
  • Check: Are there specifications? Are they satisfied?

For further good practices have a look at our review guidelines.

Edited by Henrik tom Wörden

Merge request reports

Pipeline #29962 passed

Pipeline passed for f109a398 on f-fix-crawler-overwrite

Merged by Henrik tom WördenHenrik tom Wörden 2 years ago (Nov 3, 2022 1:15pm UTC)

Loading

Pipeline #30076 passed

Pipeline passed for dec8858e on dev

Activity

Filter activity
  • Approvals
  • Assignees & reviewers
  • Comments (from bots)
  • Comments (from users)
  • Commits & branches
  • Edits
  • Labels
  • Lock status
  • Mentions
  • Merge request status
  • Tracking
  • Alexander Schlemmer changed the description

    changed the description

  • Problem, merge request causes merge problems in synchronize of the form:

    E                           Entity a (559, None) has a Property 'project' with value=558
    E                           Entity b (559, None) has a Property 'project' with value=<Record id="558">

    This could probably be resolved with a proper preparation of the objects to be merged in crawl.py:748:

    # side effect
                            record.id = identified_record.id
                            # Copy over checksum and size too if it is a file
                            if isinstance(record, db.File):
                                record._size = identified_record._size
                                record._checksum = identified_record._checksum
    
                            # TODO: Replace values that are Records which have an ID just with the ID
    
                            merge_entities(record, identified_record)
                            to_be_updated.append(record)
                            self.add_to_remote_existing_cache(record)
                            del flat[i]
                        resolved_references = True
  • Alexander Schlemmer marked this merge request as draft

    marked this merge request as draft

  • Florian Spreckelsen removed review request for @florian

    removed review request for @florian

  • Florian Spreckelsen assigned to @florian and unassigned @salexan

    assigned to @florian and unassigned @salexan

  • added 14 commits

    Compare with previous version

  • Florian Spreckelsen changed the description

    changed the description

  • added 1 commit

    • 53ce6836 - FIX: Use force merge with a deepcopy

    Compare with previous version

  • Florian Spreckelsen resolved all threads

    resolved all threads

  • Florian Spreckelsen marked the checklist item All automated tests pass as incomplete

    marked the checklist item All automated tests pass as incomplete

  • Florian Spreckelsen marked the checklist item Up-to-date CHANGELOG.md (or not necessary) as incomplete

    marked the checklist item Up-to-date CHANGELOG.md (or not necessary) as incomplete

  • Florian Spreckelsen marked the checklist item Annotations in code (Gitlab comments) as incomplete

    marked the checklist item Annotations in code (Gitlab comments) as incomplete

  • added 1 commit

    • 996672f9 - FIX: Keep existing names and description when updating properties

    Compare with previous version

  • Florian Spreckelsen marked the checklist item All automated tests pass as completed

    marked the checklist item All automated tests pass as completed

  • added 1 commit

    Compare with previous version

  • Florian Spreckelsen marked the checklist item Up-to-date CHANGELOG.md (or not necessary) as completed

    marked the checklist item Up-to-date CHANGELOG.md (or not necessary) as completed

  • Florian Spreckelsen marked this merge request as ready

    marked this merge request as ready

  • Florian Spreckelsen marked the checklist item Annotations in code (Gitlab comments) as completed

    marked the checklist item Annotations in code (Gitlab comments) as completed

  • added 1 commit

    • 4e368b01 - DEP: Increase minimum pylib version

    Compare with previous version

  • Florian Spreckelsen
  • Florian Spreckelsen requested review from @henrik

    requested review from @henrik

  • Henrik tom Wörden marked the checklist item I understand the intent of this MR as completed

    marked the checklist item I understand the intent of this MR as completed

  • Henrik tom Wörden marked the checklist item All automated tests pass as completed

    marked the checklist item All automated tests pass as completed

  • Henrik tom Wörden marked the checklist item Up-to-date CHANGELOG.md (or not necessary) as completed

    marked the checklist item Up-to-date CHANGELOG.md (or not necessary) as completed

  • Henrik tom Wörden marked the checklist item The test environment setup works and the intended behavior is reproducible in the test as completed

    marked the checklist item The test environment setup works and the intended behavior is reproducible in the test as completed

  • Henrik tom Wörden marked the checklist item In-code documentation and comments are up-to-date. as completed

    marked the checklist item In-code documentation and comments are up-to-date. as completed

  • Henrik tom Wörden marked the checklist item Check: Are there specifications? Are they satisfied? as completed

    marked the checklist item Check: Are there specifications? Are they satisfied? as completed

  • added 1 commit

    • 3aa7a907 - MAINT: Refactor fix into separate function

    Compare with previous version

  • Florian Spreckelsen
  • Florian Spreckelsen
  • Florian Spreckelsen
  • Henrik tom Wörden resolved all threads

    resolved all threads

  • added 1 commit

    Compare with previous version

  • mentioned in commit dec8858e

  • Please register or sign in to reply
    Loading