F awi sams

Summary

For #1 (closed). Removes the dependencies on the specifics of the GEOMAR data model and makes things more configurable.

Focus

A lot of functionality is moved into small Python modules in scripting/bin/sample_helpers. They should define a reasonable default behavior for the datamodel in models_and_helper_scripts/sample-management-datamodel.yml and the AWI SAMS datamodel, and should be overwritten by files of the same name in a server profile's custom directory for customization.

Test Environment

So far, only manual testing.

Do round trips for:

• Container registration, export, update
• Sample registration, export, update

Check List for the Author

Please, prepare your MR for a review. Be sure to write a summary and a focus and create gitlab comments for the reviewer. They should guide the reviewer through the changes, explain your changes and also point out open questions. For further good practices have a look at our review guidelines

• All automated tests pass
• Reference related issues
• Up-to-date CHANGELOG.md (or not necessary)
• Up-to-date JSON schema (or not necessary)
• Appropriate user and developer documentation (or not necessary)
  • How do I use the software? Assume "stupid" users.
  • How do I develop or debug the software? Assume novice developers.
• Annotations in code (Gitlab comments)
  • Intent of new code
  • Problems with old code
  • Why this implementation?

Check List for the Reviewer

• I understand the intent of this MR
• All automated tests pass
• Up-to-date CHANGELOG.md (or not necessary)
• Appropriate user and developer documentation (or not necessary)
• The test environment setup works and the intended behavior is reproducible in the test environment
• In-code documentation and comments are up-to-date.
• Check: Are there specifications? Are they satisfied?

For further good practices have a look at our review guidelines.

sample-management-custom/caosdb-server/scripting/bin/bis_utils.py

@@ -32,84 +32,12 @@ from caosadvancedtools.serverside.helper import send_mail
 from caoscrawler.config import get_config_setting
 from linkahead import get_entity_by_name
 
-SPECIAL_TREATMENT_SAMPLE = [
-    "BIS ID",
-    "Collection",
-    "Date collected start",
-    "Date collected stop",
-    "Date sampled start",
-    "Date sampled stop",
-    "Gear configuration",
-    "Gear",
-    "Hol",
-    "Latitude start",
-    "Latitude stop",
-    "Longitude start",
-    "Longitude stop",
-    "Main User",
-    "Nagoya case number",
-    "PDFReport",
-    "PI",
-    "Parent BIS ID",
-    "Person",
-    "Sampling Person",
-    "Sampling depth start",
-    "Sampling depth stop",
-    "Sphere",
-    "Station ID",
-    "Station number",
-    "Storage Container Label",
-    "Storage ID",
-    "Subevent",
-    "Time collected start",
-    "Time collected stop",
-    "Time sampled start",
-    "Time sampled stop",
-    "Timezone",
-    "Water depth start",
-    "Water depth stop",
-]
-
-IGNORED_COLUMN_NAMES_SAMPLE = [
-    "BIS URL",
-    "Date",
-    "IGSN URL",
-    "IGSN",  # TODO This will be relevant for external IGSNs in the future.
-    "Parent Sample",
-    "Sampling depth",
-    "Storage chain",
-    "Water depth",
-]
-
-
-# This can be used for hard-coded descriptions
-COLUMN_DESCRIPTIONS = {
-    "BIS ID": "An ID generated by the BIS (either integer or URL to this entity). Do not change this column!",
-    "BIS label": "The label automatically assigned by the BIS. Do not change this column.",
-    "Collection": "Collection(s) to which this container or samples belongs (separated by ';')",
-    "Container Contents": "Notes concerning the contents of this container",
-    "Container size": "The size of the container (e.g 81 spaces)",
-    "Container type": "The type of container. Do not change this column.",
-    "Custom label": "Any additional custom label you put on the container",
-    "Date collected start": "The start date in format YYYY-MM-DD",
-    "Date collected stop": "The end date in format YYYY-MM-DD",
-    "Gear configuration": "How gear is configured, meshsize of the net, filter pore size, what kind of trawl",
-    "Latitude start": "Latitude where sampling started; in decimal degrees,use \".\" as decimal sign, use - for S",
-    "Latitude stop": "Latitude where sampling ended; in decimal degrees,use \".\" as decimal sign, use - for S",
-    "Longitude start": "Longitude where sampling started; in decimal degrees,use \".\" as decimal sign, use - for W",
-    "Longitude stop": "Longitude where sampling ended; in decimal degrees,use \".\" as decimal sign, use - for W",
-    "Notes / remarks": "Field for notes",
-    "PDFReport": "A PDF containing additional information on the container or sampe contents",
-    "PI": "The PI (by abbreviation) of the container or sample",
-    "Parent container": "The BIS ID (integer or URL) or name or BIS label of the container where this container is stored.",
-    "Sampling depth start": "The depth in meters where the sampling started as a positive value, not bottom depth",
-    "Sampling depth stop": "The depth in meters where the sampling ended as a positive value,not bottom depth",
-    "Storage ID": "BIS ID of the Container that holds this sample",
-    "Subevent": "Subevent given by D-Ship in Stationlog. Expl. AL123_14-4",
-    "Timezone": "Timezone: Either UTC, CET, .... or in the form +hh[:mm], -hh:[mm].",
-    "Water depth start": "The bottom depth in meters where the sampling started as a positive value",
-    "Water depth stop": "The bottom depth in meters where the sampling ended as a positive value",
-}
+from sample_helpers.sample_upload_column_definitions import (
+    IGNORED_COLUMN_NAMES_SAMPLE, SPECIAL_TREATMENT_SAMPLE)
+from sample_helpers.utils import CONSTANTS, return_value_if_not_none
+
+
+COLUMN_DESCRIPTIONS = CONSTANTS["csv_column_descriptions"]
 
 
 def get_do_not_insert_type_names(override_names: list[str] = []):
@@ -203,22 +131,6 @@ def replace_entity_urls_by_ids(data: pd.DataFrame, eid_columns: list[str] = ["BI
     return data
 
 
-def return_value_if_not_none(val):
-    """Workaround for somewhat weird pandas behavior. Return value if actual
-    value, otherwise, return None.
-
-    """
-    if isinstance(val, list):
-        if len(val) == 0:
-            return None
-        if len(val) == 1:
-            if pd.isnull(val[0]) or val[0] is None or f"{val[0]}".lower() == "nan" or f"{val[0]}" == "":
-                return None
-    elif pd.isnull(val) or val is None or f"{val}".lower() == "nan" or f"{val}" == "":
-        return None
-    return val
-
-
 def whitespace_cleanup_in_df(df: pd.DataFrame):
     """Strip all leading and trailing whitespaces from all str values in df."""