New f fix merge

Summary

Continuation of !120 (merged) that was weirdly merged, later reverted. this MR is still lacking documentation and explanation, esp. regarding the pending list. See #108 (closed).

After a closer look, this was actually the second half of the fix for #113 (closed). We postpone the merging of entities in case of unresolved reference properties.

Focus

The problem that is being solved is best described in the new integration test originally from !141 (merged). It describes the case of three records, an event, a basis, and a campaign. Both, event and campaign reference the basis, and event references campaign. Campaign and basis already exist on the server.

This is solved by postponing the merge, and by the new merge_id_with_resolved_entity argument of merge_entities.

Test Environment

New integration test should be sufficient.

Check List for the Author

Please, prepare your MR for a review. Be sure to write a summary and a focus and create gitlab comments for the reviewer. They should guide the reviewer through the changes, explain your changes and also point out open questions. For further good practices have a look at our review guidelines

• All automated tests pass
• Reference related issues
• Up-to-date CHANGELOG.md (or not necessary)
• Up-to-date JSON schema (or not necessary)
• Appropriate user and developer documentation (or not necessary)
  • How do I use the software? Assume "stupid" users.
  • How do I develop or debug the software? Assume novice developers.
• Annotations in code (Gitlab comments)
  • Intent of new code
  • Problems with old code
  • Why this implementation?

Check List for the Reviewer

• I understand the intent of this MR
• All automated tests pass
• Up-to-date CHANGELOG.md (or not necessary)
• Appropriate user and developer documentation (or not necessary)
• The test environment setup works and the intended behavior is reproducible in the test environment
• In-code documentation and comments are up-to-date.
• Check: Are there specifications? Are they satisfied?

For further good practices have a look at our review guidelines.

src/caoscrawler/crawl.py

@@ -170,6 +170,48 @@ def _resolve_datatype(prop: db.Property, remote_entity: db.Entity):
     return prop
 
 
+def _treat_merge_error_of(newrecord, record):
+    """
+    The parameters are two entities that cannot be merged with the merge_entities function.
+
+    # This function checks for two obvious cases where no merge will ever be possible:
+    # 1. Two Entities with differing IDs
+    # 2. Two non-Entity values which differ
+
+    It creates a more informative logger message and raises an Exception in those cases.
+    """
+    for this_p in newrecord.properties:
+        that_p = record.get_property(this_p.name)
+        if (isinstance(this_p.value, db.Entity)
+                and isinstance(that_p.value, db.Entity)):
+            if this_p.value.id is not None and that_p.value.id is not None:
+                if this_p.value.id != that_p.value.id:
+                    logger.error("The Crawler is trying to merge two entities "
+                                 "because they should be the same object (same"
+                                 " identifiables), but they reference "
+                                 "different Entities with the same Property."
+                                 f"Problematic Property: {this_p.name}\n"
+                                 f"Referenced Entities: {this_p.value.id} and "
+                                 f"{that_p.value.id}\n"
+                                 f"{record}\n{newrecord}")
+                    raise RuntimeError("Cannot merge Entities")
+        elif (not isinstance(this_p.value, db.Entity)
+              and not isinstance(that_p.value, db.Entity)):
+            if ((this_p.value != that_p.value)
+                # TODO can we also compare lists?
+                and not isinstance(this_p.value, list)
+                    and not isinstance(that_p.value, list)):
+                logger.error("The Crawler is trying to merge two entities "
+                             "because they should be the same object (same"
+                             " identifiables), but they have  "
+                             "different values for the same Property."
+                             f"Problematic Property: {this_p.name}\n"
+                             f"Values: {this_p.value} and "
+                             f"{that_p.value}\n"
+                             f"{record}\n{newrecord}")
+                raise RuntimeError("Cannot merge Entities")
+
+
 class SecurityMode(Enum):
     RETRIEVE = 0
     INSERT = 1
@@ -537,48 +579,6 @@ class Crawler(object):
 
         return references
 
-    @staticmethod
-    def _treat_merge_error_of(newrecord, record):
-        """
-        The parameters are two entities that cannot be merged with the merge_entities function.
-
-        # This function checks for two obvious cases where no merge will ever be possible:
-        # 1. Two Entities with differing IDs
-        # 2. Two non-Entity values which differ
-
-        It creates a more informative logger message and raises an Exception in those cases.
-        """
-        for this_p in newrecord.properties:
-            that_p = record.get_property(this_p.name)
-            if (isinstance(this_p.value, db.Entity)
-                    and isinstance(that_p.value, db.Entity)):
-                if this_p.value.id is not None and that_p.value.id is not None:
-                    if this_p.value.id != that_p.value.id:
-                        logger.error("The Crawler is trying to merge two entities "
-                                     "because they should be the same object (same"
-                                     " identifiables), but they reference "
-                                     "different Entities with the same Property."
-                                     f"Problematic Property: {this_p.name}\n"
-                                     f"Referenced Entities: {this_p.value.id} and "
-                                     f"{that_p.value.id}\n"
-                                     f"{record}\n{newrecord}")
-                        raise RuntimeError("Cannot merge Entities")
-            elif (not isinstance(this_p.value, db.Entity)
-                    and not isinstance(that_p.value, db.Entity)):
-                if ((this_p.value != that_p.value)
-                        # TODO can we also compare lists?
-                        and not isinstance(this_p.value, list)
-                        and not isinstance(that_p.value, list)):
-                    logger.error("The Crawler is trying to merge two entities "
-                                 "because they should be the same object (same"
-                                 " identifiables), but they have  "
-                                 "different values for the same Property."
-                                 f"Problematic Property: {this_p.name}\n"
-                                 f"Values: {this_p.value} and "
-                                 f"{that_p.value}\n"
-                                 f"{record}\n{newrecord}")
-                    raise RuntimeError("Cannot merge Entities")
-
     def split_into_inserts_and_updates(self, ent_list: list[db.Entity]):
         to_be_inserted: list[db.Entity] = []
         to_be_updated: list[db.Entity] = []
@@ -703,7 +703,7 @@ class Crawler(object):
                 record,
                 referencing_entities=referencing_entities)
             newrecord = self.get_from_any_cache(identifiable)
-            merge_entities(newrecord, record)
+            merge_entities(newrecord, record, merge_id_with_resolved_entity=True)
         if len(flat) > 0:
             circle = self.detect_circular_dependency(flat)
             if circle is None: