fix importance bug

Summary

Fix for #146 (closed)

The place where the configuration is being set moved from the back-end (where it has been since the Middle Bronze Age) to the server (conf/core/rules.csv).

Also the Job implementing the check had some serious gaps (even not-too-cornerish cases) and TODOs which were closed/implemented. Effectively, this MR is a major refactoring of the CheckParOblPropPresent job.

Focus

• The RuleLoader and all related transaction classes have been removed. Instead the JobConfig class loads the rules for jobs from a csv file which contains the columns that have been stored to the "rules" table in the back-end previously.

• The Job class was extended by several convenient method I stole from the f-filesystem branch (most prominently resolve and isSubType.

• The Mode enum has been renamed to JobFailureSeverity because that is actually a much more speaking name. Also the values (remainders of the Early Bronze Age) have been changed to more speaking ones (and never-used values have been removed).

• On the go, I also used the new resolve function in some other places and also removed some occurrences where Jobs started other jobs explicitly. I noticed that this was not necessary and it only was there because the CheckParentValid and CheckPropertyValid jobs have been moved to the PRE_CHECK stage in earlier commits, but the then unnecessary explicit calls have never been removed - they did no harm, but they are not necessary.

Test Environment

Tests in caosdb-pyinttest!9 (merged)

Check List for the Author

Please, prepare your MR for a review. Be sure to write a summary and a focus and create gitlab comments for the reviewer. They should guide the reviewer through the changes, explain your changes and also point out open questions. For further good practices have a look at our review guidelines

• All automated tests pass
• Reference related Issues
• Up-to-date CHANGELOG.md
• Annotations in code (Gitlab comments)
  • Intent of new code
  • Problems with old code
  • Why this implementation?

Check List for the Reviewer

• I understand the intent of this MR
• All automated tests pass
• Up-to-date CHANGELOG.md
• The test environment setup works and the intended behavior is reproducible in the test environment
• In-code documentation and comments are up-to-date.
• Check: Are there spezifications? Are they satisfied?

For further good practices have a look at our review guidelines.

conf/core/jobs.csv

 #
 # This file is a part of the CaosDB Project.
 #
-# Copyright (C) 2018 Research Group Biomedical Physics,
-# Max-Planck-Institute for Dynamics and Self-Organization Göttingen
 # Copyright (C) 2021 Timm Fitsche <t.fitschen@indiscale.com>
 # Copyright (C) 2021 IndiScale GmbH <info@indiscale.com>
 #
@@ -21,92 +19,92 @@
 #
 
 # Five columns:
-# DOMAIN_ID, ENTITY_ID, TRANSACTION, JOB, MODE
+# DOMAIN_ID, ENTITY_ID, TRANSACTION, JOB, JOB_FAILURE_SEVERITY
 
 # general rules
-0,0,INSERT,CheckPropValid,MUST
-0,0,INSERT,CheckParValid,MUST
-0,0,INSERT,CheckParOblPropPresent,MUST
-0,0,INSERT,CheckValueParsable,MUST
-0,0,UPDATE,CheckPropValid,MUST
-0,0,UPDATE,CheckParValid,MUST
-0,0,UPDATE,CheckParOblPropPresent,MUST
-0,0,UPDATE,CheckValueParsable,MUST
-0,0,DELETE,CheckReferenceDependencyExistent,MUST
-0,0,DELETE,CheckChildDependencyExistent,MUST
+0,0,INSERT,CheckPropValid,ERROR
+0,0,INSERT,CheckParValid,ERROR
+0,0,INSERT,CheckParOblPropPresent,ERROR
+0,0,INSERT,CheckValueParsable,ERROR
+0,0,UPDATE,CheckPropValid,ERROR
+0,0,UPDATE,CheckParValid,ERROR
+0,0,UPDATE,CheckParOblPropPresent,ERROR
+0,0,UPDATE,CheckValueParsable,ERROR
+0,0,DELETE,CheckReferenceDependencyExistent,ERROR
+0,0,DELETE,CheckChildDependencyExistent,ERROR
 
 
 # role specific rules
 ## recordtype rules
-0,1,INSERT,CheckNamePresent,MUST
-0,1,INSERT,CheckPropPresent,SHOULD
-0,1,INSERT,SetImpToRecByDefault,MUST
+0,1,INSERT,CheckNamePresent,ERROR
+0,1,INSERT,CheckPropPresent,WARN
+0,1,INSERT,SetImpToRecByDefault,ERROR
 
-0,1,UPDATE,CheckNamePresent,MUST
-0,1,UPDATE,CheckPropPresent,SHOULD
-0,1,UPDATE,SetImpToRecByDefault,MUST
+0,1,UPDATE,CheckNamePresent,ERROR
+0,1,UPDATE,CheckPropPresent,WARN
+0,1,UPDATE,SetImpToRecByDefault,ERROR
 
 ## record rules
-0,2,INSERT,CheckNamePresent,SHOULD
-0,2,INSERT,CheckPropPresent,SHOULD
-0,2,INSERT,CheckParPresent,MUST
-0,2,INSERT,SetImpToFix,MUST
+0,2,INSERT,CheckNamePresent,WARN
+0,2,INSERT,CheckPropPresent,WARN
+0,2,INSERT,CheckParPresent,ERROR
+0,2,INSERT,SetImpToFix,ERROR
 
-0,2,UPDATE,CheckNamePresent,SHOULD
-0,2,UPDATE,CheckPropPresent,SHOULD
-0,2,UPDATE,CheckParPresent,MUST
-0,2,UPDATE,SetImpToFix,MUST
+0,2,UPDATE,CheckNamePresent,WARN
+0,2,UPDATE,CheckPropPresent,WARN
+0,2,UPDATE,CheckParPresent,ERROR
+0,2,UPDATE,SetImpToFix,ERROR
 
 ## file rules
-0,3,INSERT,CheckNamePresent,SHOULD
-0,3,INSERT,MatchFileProp,MUST
-0,3,INSERT,CheckTargetPathValid,MUST
-0,3,INSERT,SetImpToFix,MUST
+0,3,INSERT,CheckNamePresent,WARN
+0,3,INSERT,MatchFileProp,ERROR
+0,3,INSERT,CheckTargetPathValid,ERROR
+0,3,INSERT,SetImpToFix,ERROR
 
-0,3,UPDATE,CheckNamePresent,SHOULD
-0,3,UPDATE,MatchFileProp,MUST
-0,3,UPDATE,CheckTargetPathValid,MUST
-0,3,UPDATE,SetImpToFix,MUST
+0,3,UPDATE,CheckNamePresent,WARN
+0,3,UPDATE,MatchFileProp,ERROR
+0,3,UPDATE,CheckTargetPathValid,ERROR
+0,3,UPDATE,SetImpToFix,ERROR
 
 ## property rules
-0,4,INSERT,CheckDatatypePresent,MUST
-0,4,UPDATE,CheckDatatypePresent,MUST
-0,4,INSERT,CheckNamePresent,MUST
-0,4,UPDATE,CheckNamePresent,MUST
-0,4,INSERT,SetImpToFix,MUST
-0,4,UPDATE,SetImpToFix,MUST
+0,4,INSERT,CheckDatatypePresent,ERROR
+0,4,UPDATE,CheckDatatypePresent,ERROR
+0,4,INSERT,CheckNamePresent,ERROR
+0,4,UPDATE,CheckNamePresent,ERROR
+0,4,INSERT,SetImpToFix,ERROR
+0,4,UPDATE,SetImpToFix,ERROR
 
 ## query template rules
-0,8,UPDATE,CheckQueryTemplate,MUST
-0,8,INSERT,CheckQueryTemplate,MUST
+0,8,UPDATE,CheckQueryTemplate,ERROR
+0,8,INSERT,CheckQueryTemplate,ERROR
 
 # data type specific rules
 ## reference rules
-0,11,INSERT,CheckRefidPresent,SHOULD
-0,11,INSERT,CheckRefidValid,MUST
-0,11,INSERT,CheckRefidIsaParRefid,SHOULD
+0,11,INSERT,CheckRefidPresent,WARN
+0,11,INSERT,CheckRefidValid,ERROR
+0,11,INSERT,CheckRefidIsaParRefid,WARN
 
-0,11,UPDATE,CheckRefidPresent,SHOULD
-0,11,UPDATE,CheckRefidValid,MUST
-0,11,UPDATE,CheckRefidIsaParRefid,SHOULD
+0,11,UPDATE,CheckRefidPresent,WARN
+0,11,UPDATE,CheckRefidValid,ERROR
+0,11,UPDATE,CheckRefidIsaParRefid,WARN
 
 ## integer rules
-0,12,INSERT,CheckUnitPresent,SHOULD
-0,12,INSERT,ParseUnit,SHOULD
+0,12,INSERT,CheckUnitPresent,WARN
+0,12,INSERT,ParseUnit,WARN
 
-0,12,UPDATE,CheckUnitPresent,SHOULD
-0,12,UPDATE,ParseUnit,SHOULD
+0,12,UPDATE,CheckUnitPresent,WARN
+0,12,UPDATE,ParseUnit,WARN
 
 ## double rules
-0,13,INSERT,CheckUnitPresent,SHOULD
-0,13,INSERT,ParseUnit,SHOULD
+0,13,INSERT,CheckUnitPresent,WARN
+0,13,INSERT,ParseUnit,WARN
 
-0,13,UPDATE,CheckUnitPresent,SHOULD
-0,13,UPDATE,ParseUnit,SHOULD
+0,13,UPDATE,CheckUnitPresent,WARN
+0,13,UPDATE,ParseUnit,WARN
 
 ## filereference rules
-0,17,INSERT,CheckRefidValid,MUST
-0,17,INSERT,CheckRefidIsaParRefid,MUST
+0,17,INSERT,CheckRefidValid,ERROR
+0,17,INSERT,CheckRefidIsaParRefid,ERROR
 
-0,17,UPDATE,CheckRefidValid,MUST
-0,17,UPDATE,CheckRefidIsaParRefid,MUST
+0,17,UPDATE,CheckRefidValid,ERROR
+0,17,UPDATE,CheckRefidIsaParRefid,ERROR