F fix crawler overwrite

Summary

Fix for: https://gitlab.com/caosdb/caosdb-crawler/-/issues/23 and #47 (closed)

Test Environment

There is an integration test provided. For manual testing, you need pylib in the most recent dev branch, otherwise merge_entities is missing the force option.

Check List for the Author

Please, prepare your MR for a review. Be sure to write a summary and a focus and create gitlab comments for the reviewer. They should guide the reviewer through the changes, explain your changes and also point out open questions. For further good practices have a look at our review guidelines

• All automated tests pass
• Reference related issues
• Up-to-date CHANGELOG.md (or not necessary)
• Annotations in code (Gitlab comments)
  • Intent of new code
  • Problems with old code
  • Why this implementation?

Check List for the Reviewer

• I understand the intent of this MR
• All automated tests pass
• Up-to-date CHANGELOG.md (or not necessary)
• The test environment setup works and the intended behavior is reproducible in the test environment
• In-code documentation and comments are up-to-date.
• Check: Are there specifications? Are they satisfied?

For further good practices have a look at our review guidelines.

assigned to @salexan

added 2 commits

• 0ca226cc - FIX: Crawler overwrites and deletes existing data
• 1838fee2 - TST: removed xfail from test for issue 23

Compare with previous version

marked this merge request as ready

added 1 commit

• 086b9593 - DOC: udpated changelog

Compare with previous version

requested review from @florian

marked the checklist item All automated tests pass as completed

marked the checklist item Reference related issues as completed

marked the checklist item Up-to-date CHANGELOG.md (or not necessary) as completed

marked the checklist item Annotations in code (Gitlab comments) as completed

changed the description

Problem, merge request causes merge problems in synchronize of the form:

E                           Entity a (559, None) has a Property 'project' with value=558
E                           Entity b (559, None) has a Property 'project' with value=<Record id="558">

This could probably be resolved with a proper preparation of the objects to be merged in crawl.py:748:

# side effect
                        record.id = identified_record.id
                        # Copy over checksum and size too if it is a file
                        if isinstance(record, db.File):
                            record._size = identified_record._size
                            record._checksum = identified_record._checksum

                        # TODO: Replace values that are Records which have an ID just with the ID

                        merge_entities(record, identified_record)
                        to_be_updated.append(record)
                        self.add_to_remote_existing_cache(record)
                        del flat[i]
                    resolved_references = True

marked this merge request as draft

removed review request for @florian

assigned to @florian and unassigned @salexan

added 14 commits

• 086b9593...8b1b03ae - 11 commits from branch dev
• 87697740 - Merge branch 'dev' into f-fix-crawler-overwrite
• f1411c30 - DOC: Update changelog
• 951c6e85 - TST: Remove test of merge_entities

Compare with previous version

changed the description

added 1 commit

• 53ce6836 - FIX: Use force merge with a deepcopy

Compare with previous version

resolved all threads