ENH: setup logging and reporting for serverside execution

Summary

• crawler_run_model.yml contains a model for Records that contain informaiton on crawler jobs
• the new logging module sets up logging for serverside jobs
• crawl.py was refactored and now use the above additions

Focus

Point the reviewer to the core of the code change. Where should they start reading? What should they focus on (e.g. security, performance, maintainability, user-friendliness, compliance with the specs, finding more corner cases, concrete questions)?

Test Environment

I suggest local-testing setup of dpz.

• set pylib to package: "git+https://gitlab.indiscale.com/caosdb/src/caosdb-pylib.git@f-validate-caoscrawler"
• set caoscrawler to package: "git+https://gitlab.indiscale.com/caosdb/src/caosdb-crawler.git@f-email-notify"
• add the data model crawler_run_model.yml
• call python runcrawler.py
• check the email folder in docker /tmp/mail/somemail
• check the CrawlerRun Record in LinkAhead

Check List for the Author

Please, prepare your MR for a review. Be sure to write a summary and a focus and create gitlab comments for the reviewer. They should guide the reviewer through the changes, explain your changes and also point out open questions. For further good practices have a look at our review guidelines

• All automated tests pass
• Reference related issues
• Up-to-date CHANGELOG.md (or not necessary)
• Up-to-date JSON schema (or not necessary)
• Appropriate user and developer documentation (or not necessary)
  • How do I use the software? Assume "stupid" users.
  • How do I develop or debug the software? Assume novice developers.
• Annotations in code (Gitlab comments)
  • Intent of new code
  • Problems with old code
  • Why this implementation?

Check List for the Reviewer

• I understand the intent of this MR
• All automated tests pass
• Up-to-date CHANGELOG.md (or not necessary)
• Appropriate user and developer documentation (or not necessary)
• The test environment setup works and the intended behavior is reproducible in the test environment
• In-code documentation and comments are up-to-date.
• Check: Are there specifications? Are they satisfied?

For further good practices have a look at our review guidelines.

src/caoscrawler/config.py

+# This file is a part of the CaosDB Project.
+#
+# Copyright (C) 2023 Indiscale GmbH <info@indiscale.com>
+# Copyright (C) 2023 Henrik tom Wörden <h.tomwoerden@indiscale.com>
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Affero General Public License as
+# published by the Free Software Foundation, either version 3 of the
+# License, or (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU Affero General Public License for more details.
+#
+# You should have received a copy of the GNU Affero General Public License
+# along with this program. If not, see <https://www.gnu.org/licenses/>.
+#
+
+import caosdb as db
+
+DEFAULTS = {
+    "send_crawler_notifications": False,
+    "create_crawler_status_records": False,
+    "public_host_url": "/",
+}
+
+
+def get_config_setting(setting):
+    caosdb_config = db.configuration.get_config()
+    if "caoscrawler" in caosdb_config and setting in caosdb_config["caoscrawler"]:
+        return caosdb_config["caoscrawler"][setting]
+    else:
+        return DEFAULTS[setting]