Treat dependencies among identifiables

Summary

Allow a CFood to have identifiables that depend on each other. E.g. an identifiable Project and an Experiment that references it.

Focus

Identifiables are now treated one after the other when checking whether they need to be inserted.

Test Environment

Automated tests should suffice. Note that tests where changed slightly such that they fail without this fix: https://gitlab.indiscale.com/caosdb/src/caosdb-advanced-user-tools/-/jobs/37342 Since identifiables are not treated correctly the experiment is inserted a second time which leads to the violation of the unique query.

Check List for the Author

Please, prepare your MR for a review. Be sure to write a summary and a focus and create gitlab comments for the reviewer. They should guide the reviewer through the changes, explain your changes and also point out open questions. For further good practices have a look at our review guidelines

• All automated tests pass
• Reference related Issues
• Up-to-date CHANGELOG.md
• Annotations in code (Gitlab comments)
  • Intent of new code
  • Problems with old code
  • Why this implementation?

Check List for the Reviewer

• I understand the intent of this MR
• All automated tests pass
• Up-to-date CHANGELOG.md
• The test environment setup works and the intended behavior is reproducible in the test environment
• In-code documentation and comments are up-to-date.
• Check: Are there spezifications? Are they satisfied?
• remove the demonstration branch f-make-deps-fail

For further good practices have a look at our review guidelines.

src/caosadvancedtools/crawler.py

@@ -576,44 +576,51 @@ carefully and if the changes are ok, click on the following link:
         # looking for matching entities in CaosDB when there is no valid id
         # i.e. there was none set from a cache
 
+        existing = []
+        inserted = []
+
         for ent in identifiables:
             if ent.id is None or ent.id < 0:
                 logger.debug("Looking for: {}".format(
                     ent.id if ent.id is not None else ent.name))
-                existing = Crawler.find_existing(ent)
+                found = Crawler.find_existing(ent)
 
-                if existing is not None:
-                    ent.id = existing.id
+                if found is not None:
+                    ent.id = found.id
             else:
                 logger.debug("Id is known of: {}".format(ent))
 
-        # insert missing, i.e. those which are not valid
-        missing_identifiables = db.Container()
-        missing_identifiables.extend([ent for ent in identifiables
-                                      if ent.id is None or ent.id < 0])
-        # TODO the following should not be necessary. Fix it
-
-        for ent in missing_identifiables:
-            ent.id = None
+            # insert missing, i.e. those which are not valid
+            if ent.id is None or ent.id < 0:
+                missing = ent
+                ent.id = None
+            else:
+                missing = None
+                existing.append(ent)
 
-        if len(missing_identifiables) > 0:
-            info = "Going to insert the following entities:\n"
+            if missing:
+                try:
+                    guard.safe_insert(missing, unique=False,
+                                      flags={"force-missing-obligatory": "ignore"})
+                    inserted.append(ent)
+                except Exception as e:
+                    DataModelProblems.evaluate_exception(e)
+        if len(existing) > 0:
+            info = "Identified the following existing entities:\n"
 
-            for ent in missing_identifiables:
+            for ent in existing:
                 info += str(ent)+"\n"
             logger.debug(info)
+        else:
+            logger.debug("Did not identify any existing entities")
+        if len(inserted) > 0:
+            info = "Inserted the following entities:\n"
 
-        if len(missing_identifiables) == 0:
-            logger.debug("No new entities to be inserted.")
+            for ent in inserted:
+                info += str(ent)+"\n"
+            logger.debug(info)
         else:
-            try:
-                logger.info(
-                    "Inserting {} Records...".format(
-                        len(missing_identifiables)))
-                guard.safe_insert(missing_identifiables, unique=False,
-                                  flags={"force-missing-obligatory": "ignore"})
-            except Exception as e:
-                DataModelProblems.evaluate_exception(e)
+            logger.debug("Did not insert any new entities")
 
         logger.debug("Retrieving entities from CaosDB...")
         identifiables.retrieve(unique=True, raise_exception_on_error=False)