CHANGELOG.md

@@ -5,10 +5,12 @@ All notable changes to this project will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
-## [Unreleased] ##
+## [Unreleased]
 
 ### Added ###
 
+- New function in apiutils that copies an Entity.
+
 ### Changed ###
 
 ### Deprecated ###
@@ -17,8 +19,48 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ### Fixed ###
 
+* [#75](https://gitlab.indiscale.com/caosdb/src/caosdb-pylib/-/issues/75), [#103](https://gitlab.indiscale.com/caosdb/src/caosdb-pylib/-/issues/103) Fixed JSON schema to allow more sections, and correct requirements for
+  password method.
+
 ### Security ###
 
+### Documentation ###
+
+
+## [0.7.2] - 2022-03-25 ##
+(Timm Fitschen)
+
+### Added ###
+
+### Changed ###
+
+### Deprecated ###
+
+* In module `caosdb.apiutils`:
+  * `CaosDBPythonEntity` class
+  * `convert_to_entity` function
+  * `convert_to_python_object` function
+
+### Removed ###
+
+### Fixed ###
+
+* [caosdb-pylib#106](https://gitlab.indiscale.com/caosdb/src/caosdb-pylib/-/issues/106)
+  Parsing Error in class caosdb.common.models.ACL. This may lead to the
+  unintentional revocation of permissions for some users or roles during
+  updates. However, no additional permissions are being granted.
+
+### Security ###
+
+### Documentation ###
+
+## [0.7.1] - 2022-03-11 ##
+(Daniel Hornung)
+
+### Documentation ###
+
+- `timeout` option in example pycaosdb.ini
+
 ## [0.7.0] - 2022-01-21 ##
 
 ### Added ###

DEPENDENCIES.md

-* caosdb-server == 0.3
-* Python >= 3.5
+* caosdb-server >= 0.7.2
+* Python >= 3.6
 * pip >= 20.0.2
 
-
-Any other dependencies are being installed via pip
+Any other dependencies are defined in the setup.py and are being installed via pip

README_SETUP.md

@@ -82,60 +82,8 @@ pip3 install --user .[jsonschema]
 
 ## Configuration ##
 
-The  configuration is done using `ini` configuration files.
-PyCaosDB tries to read from the inifile specified in the environment variable `PYCAOSDBINI` or
-alternatively in `~/.pycaosdb.ini` upon import.  After that, the ini file `pycaosdb.ini` in the
-current working directory will be read additionally, if it exists.
-
-Here, we will look at the most common configuration options. For a full and 
-comprehensive description please check out 
-[pycaosdb.ini file](https://gitlab.com/caosdb/caosdb-pylib/-/blob/main/examples/pycaosdb.ini) 
-You can download this file and use it as a starting point.
-
-
-Typically, you need to change at least the `url` and `username` fields as required. 
-(Ask your CaosDB administrator or IT crowd if
-you do not know what to put there, but for the demo instances https://demo.indiscale.com, `username=admin`
-and `password=caosdb` should work).
-
-### Authentication ###
-
-The default configuration (that your are asked for your password when ever a connection is created
-can be changed by setting `password_method`:
-
-* with `password_method=input` password (and possibly user) will be queried on demand (**default**)
-* use the password manager [pass](https://www.passwordstore.org) by using `pass` as value, see also the [ArchWiki
-  entry](https://wiki.archlinux.org/index.php/Pass#Basic_usage). This also requires `password_identifier` which refers to the identifier within pass
-  for the desired password.
-* install the python package [keyring](https://pypi.org/project/keyring), to use the system keyring/wallet (macOS, GNOME, KDE,
-  Windows). The password will be queried on first usage.
-* with `password_method=plain` (**strongly discouraged**)
-
-The following illustrates the recommended options:
-
-```ini
-[Connection]
-# using "pass" password manager
-#password_method=pass
-#password_identifier=...
-
-# using the system keyring/wallet (macOS, GNOME, KDE, Windows)
-#password_method=keyring
-```
-
-### SSL Certificate ###
-In some cases (especially if you are testing CaosDB) you might need to supply 
-an SSL certificate to allow SSL encryption.
-
-```ini
-[Connection]
-cacert=/path/to/caosdb.ca.pem
-```
-
-### Further Settings ###
-As mentioned above, a complete list of options can be found in the 
-[pycaosdb.ini file](https://gitlab.com/caosdb/caosdb-pylib/-/blob/main/examples/pycaosdb.ini) in 
-the examples folder of the source code.
+The configuration is done using `ini` configuration files.  The content of these configuration files
+is described in detail in the [configuration section of the documentation](https://docs.indiscale.com/caosdb-pylib/configuration.html).
 
 ## Try it out ##
 
@@ -155,7 +103,10 @@ like this, check out the "Authentication" section in the [configuration document
 Now would be a good time to continue with the [tutorials](tutorials/index).
 
 ## Run Unit Tests
-tox
+
+- Run all tests: `tox` or `make unittest`
+- Run a specific test file: e.g. `tox -- unittests/test_schema.py`
+- Run a specific test function: e.g. `tox -- unittests/test_schema.py::test_config_files`
 
 ## Documentation ##
 

RELEASE_GUIDELINES.md

@@ -36,8 +36,10 @@ guidelines of the CaosDB Project
 9. Publish the release by executing `./release.sh` with uploads the caosdb
    module to the Python Package Index [pypi.org](https://pypi.org).
 
-10. Merge the main branch back into the dev branch.
+10. Create a gitlab release on gitlab.indiscale.com and gitlab.com
 
-11. After the merge of main to dev, start a new development version by
+11. Merge the main branch back into the dev branch.
+
+12. After the merge of main to dev, start a new development version by
     setting `ISRELEASED` to `False` and by increasing at least the `MICRO`
     version in [setup.py](./setup.py) and preparing CHANGELOG.md.

examples/pycaosdb.ini

@@ -67,3 +67,6 @@
 
 # This option is used internally and for testing. Do not override.
 # implementation=_DefaultCaosDBServerConnection
+
+# The timeout for requests to the server.
+# timeout=1000

setup.py

@@ -47,9 +47,13 @@ from setuptools import find_packages, setup
 
 ISRELEASED = False
 MAJOR = 0
-MINOR = 7
-MICRO = 1
-PRE = ""  # e.g. rc0, alpha.1, 0.beta-23
+MINOR = 8
+MICRO = 0
+# Do not tag as pre-release until this commit
+# https://github.com/pypa/packaging/pull/515
+# has made it into a release. Probably we should wait for pypa/packaging>=21.4
+# https://github.com/pypa/packaging/releases
+PRE = "" # "dev"  # e.g. rc0, alpha.1, 0.beta-23
 
 if PRE:
     VERSION = "{}.{}.{}-{}".format(MAJOR, MINOR, MICRO, PRE)
@@ -166,12 +170,13 @@ def setup_package():
         packages=find_packages('src'),
         python_requires='>=3.6',
         package_dir={'': 'src'},
-        install_requires=['lxml>=3.6.4',
-                          'PyYaml>=3.12', 'future', 'PySocks>=1.6.7'],
+        install_requires=['lxml>=4.6.3',
+                          'PyYAML>=6.0', 'future', 'PySocks>=1.6.7'],
         extras_require={'keyring': ['keyring>=13.0.0'],
-                        'jsonschema': ['jsonschema==4.0.1']},
+                        'jsonschema': ['jsonschema>=4.4.0']},
         setup_requires=["pytest-runner>=2.0,<3dev"],
-        tests_require=["pytest", "pytest-cov", "coverage>=4.4.2", "jsonschema==4.0.1"],
+        tests_require=["pytest", "pytest-cov", "coverage>=4.4.2",
+                       "jsonschema>=4.4.0"],
         package_data={
             'caosdb': ['cert/indiscale.ca.crt', 'schema-pycaosdb-ini.yml'],
         },

src/caosdb/apiutils.py

@@ -37,7 +37,9 @@ from caosdb.common.datatype import (BOOLEAN, DATETIME, DOUBLE, FILE, INTEGER,
                                     REFERENCE, TEXT, is_reference)
 from caosdb.common.models import (Container, Entity, File, Property, Query,
                                   Record, RecordType, execute_query,
-                                  get_config)
+                                  get_config, SPECIAL_ATTRIBUTES)
+
+import logging
 
 
 def new_record(record_type, name=None, description=None,
@@ -141,6 +143,9 @@ class CaosDBPythonEntity(object):
     _last_id = 0
 
     def __init__(self):
+        warnings.warn("The CaosDBPythonEntity class is deprecated, replacements will be provided by"
+                      " the high_level_api module.",
+                      DeprecationWarning, stacklevel=2)
         # Save a copy of the dry state
         # of this object in order to be
         # able to detect conflicts.
@@ -381,10 +386,7 @@ def _single_convert_to_python_object(robj, entity):
     return robj
 
 
-def _single_convert_to_entity(entity, robj, recursive_depth, **kwargs):
-    """
-    recursive_depth: disabled if 0
-    """
+def _single_convert_to_entity(entity, robj, **kwargs):
     if robj._id is not None:
         entity.id = robj._id
 
@@ -410,16 +412,16 @@ def _single_convert_to_entity(entity, robj, recursive_depth, **kwargs):
             else:
                 entity.add_parent(id=parent)
 
-    def add_property(entity, prop, name, _recursive=False, datatype=None):
+    def add_property(entity, prop, name, recursive=False, datatype=None):
         if datatype is None:
-            raise RuntimeError("Datatype must not be None.")
+            raise ArgumentError("datatype must not be None")
 
         if isinstance(prop, CaosDBPythonEntity):
             entity.add_property(name=name, value=str(
                 prop._id), datatype=datatype)
 
-            if _recursive and not prop.do_not_expand:
-                return convert_to_entity(prop, recursive=_recursive)
+            if recursive and not prop.do_not_expand:
+                return convert_to_entity(prop, recursive=recursive)
             else:
                 return []
         else:
@@ -429,11 +431,6 @@ def _single_convert_to_entity(entity, robj, recursive_depth, **kwargs):
 
             return []
 
-    if recursive_depth == 0:
-        recursive = False
-    else:
-        recursive = True
-
     for prop in robj._properties:
         value = robj.__getattribute__(prop)
 
@@ -447,7 +444,7 @@ def _single_convert_to_entity(entity, robj, recursive_depth, **kwargs):
 
                         if recursive and not v.do_not_expand:
                             children.append(convert_to_entity(
-                                v, recursive=recursive_depth-1))
+                                v, recursive=recursive))
                     else:
                         if isinstance(v, float) or isinstance(v, int):
                             lst.append(str(v))
@@ -477,6 +474,9 @@ def _single_convert_to_entity(entity, robj, recursive_depth, **kwargs):
 
 
 def convert_to_entity(python_object, **kwargs):
+    warnings.warn("The convert_to_entity function is deprecated, replacement will be provided by "
+                  "the high_level_api module.", DeprecationWarning, stacklevel=2)
+
     if isinstance(python_object, Container):
         # Create a list of objects:
 
@@ -498,6 +498,8 @@ def convert_to_entity(python_object, **kwargs):
 def convert_to_python_object(entity):
     """"""
 
+    warnings.warn("The convert_to_python_object function is deprecated, replacement will be "
+                  "provided by the high_level_api module.", DeprecationWarning, stacklevel=2)
     if isinstance(entity, Container):
         # Create a list of objects:
 
@@ -565,10 +567,6 @@ def getCommitIn(folder):
         return t.readline().strip()
 
 
-COMPARED = ["name", "role", "datatype", "description", "importance",
-            "id", "path", "checksum", "size"]
-
-
 def compare_entities(old_entity: Entity, new_entity: Entity):
     """
     Compare two entites.
@@ -592,7 +590,7 @@ def compare_entities(old_entity: Entity, new_entity: Entity):
     if old_entity is new_entity:
         return (olddiff, newdiff)
 
-    for attr in COMPARED:
+    for attr in SPECIAL_ATTRIBUTES:
         try:
             oldattr = old_entity.__getattribute__(attr)
             old_entity_attr_exists = True
@@ -681,10 +679,77 @@ def compare_entities(old_entity: Entity, new_entity: Entity):
     return (olddiff, newdiff)
 
 
+def merge_entities(entity_a: Entity, entity_b: Entity):
+    """
+    Merge entity_b into entity_a such that they have the same parents and properties.
+
+    datatype, unit, value, name and description will only be changed in entity_a if they
+    are None for entity_a and set for entity_b. If there is a corresponding value
+    for entity_a different from None a RuntimeError will be raised informing of an
+    unresolvable merge conflict.
+
+    The merge operation is done in place.
+
+    Returns entity_a.
+
+    WARNING: This function is currently experimental and insufficiently tested. Use with care.
+    """
+
+    logging.warning(
+        "This function is currently experimental and insufficiently tested. Use with care.")
+
+    # Compare both entities:
+    diff_r1, diff_r2 = compare_entities(entity_a, entity_b)
+
+    # Go through the comparison and try to apply changes to entity_a:
+    for key in diff_r2["parents"]:
+        entity_a.add_parent(entity_b.get_parent(key))
+
+    for key in diff_r2["properties"]:
+        if key in diff_r1["properties"]:
+            if ("importance" in diff_r1["properties"][key] and
+                    "importance" in diff_r2["properties"][key]):
+                if (diff_r1["properties"][key]["importance"] !=
+                        diff_r2["properties"][key]["importance"]):
+                    raise NotImplementedError()
+            elif ("importance" in diff_r1["properties"][key] or
+                  "importance" in diff_r2["properties"][key]):
+                raise NotImplementedError()
+
+            for attribute in ("datatype", "unit", "value"):
+                if diff_r1["properties"][key][attribute] is None:
+                    setattr(entity_a.get_property(key), attribute,
+                            diff_r2["properties"][key][attribute])
+                else:
+                    raise RuntimeError("Merge conflict.")
+        else:
+            # TODO: This is a temporary FIX for
+            #       https://gitlab.indiscale.com/caosdb/src/caosdb-pylib/-/issues/105
+            entity_a.add_property(id=entity_b.get_property(key).id,
+                                  name=entity_b.get_property(key).name,
+                                  datatype=entity_b.get_property(key).datatype,
+                                  value=entity_b.get_property(key).value,
+                                  unit=entity_b.get_property(key).unit,
+                                  importance=entity_b.get_importance(key))
+            # entity_a.add_property(
+            #     entity_b.get_property(key),
+            #     importance=entity_b.get_importance(key))
+
+    for special_attribute in ("name", "description"):
+        sa_a = getattr(entity_a, special_attribute)
+        sa_b = getattr(entity_b, special_attribute)
+        if sa_a != sa_b:
+            if sa_a is None:
+                setattr(entity_a, special_attribute, sa_b)
+            else:
+                raise RuntimeError("Merge conflict.")
+    return entity_a
+
+
 def describe_diff(olddiff, newdiff, name=None, as_update=True):
     description = ""
 
-    for attr in list(set(list(olddiff.keys())+list(newdiff.keys()))):
+    for attr in list(set(list(olddiff.keys()) + list(newdiff.keys()))):
         if attr == "parents" or attr == "properties":
             continue
         description += "{} differs:\n".format(attr)

src/caosdb/common/models.py

@@ -5,9 +5,9 @@
 #
 # Copyright (C) 2018 Research Group Biomedical Physics,
 # Max-Planck-Institute for Dynamics and Self-Organization Göttingen
-# Copyright (C) 2020 Indiscale GmbH <info@indiscale.com>
+# Copyright (C) 2020-2022 Indiscale GmbH <info@indiscale.com>
 # Copyright (C) 2020 Florian Spreckelsen <f.spreckelsen@indiscale.com>
-# Copyright (C) 2020 Timm Fitschen <t.fitschen@indiscale.com>
+# Copyright (C) 2020-2022 Timm Fitschen <t.fitschen@indiscale.com>
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License as
@@ -25,7 +25,14 @@
 # ** end header
 #
 
-"""missing docstring."""
+"""
+Collection of the central classes of the CaosDB client, namely the Entity class
+and all of its subclasses and the Container class which is used to carry out
+transactions.
+
+All additional classes are either important for the entities or the
+transactions.
+"""
 from __future__ import print_function, unicode_literals
 
 import re
@@ -72,6 +79,10 @@ ALL = "ALL"
 NONE = "NONE"
 
 
+SPECIAL_ATTRIBUTES = ["name", "role", "datatype", "description",
+                      "id", "path", "checksum", "size"]
+
+
 class Entity(object):
 
     """Entity is a generic CaosDB object.
@@ -114,6 +125,48 @@ class Entity(object):
         self.id = id
         self.state = None
 
+
+    def copy(self):
+        """
+        Return a copy of entity.
+
+        If deep == True return a deep copy, recursively copying all sub entities.
+
+        Standard properties are copied using add_property.
+        Special attributes, as defined by the global variable SPECIAL_ATTRIBUTES and additionaly
+        the "value" are copied using setattr.
+        """
+        if self.role == "File":
+            new = File()
+        elif self.role == "Property":
+            new = Property()
+        elif self.role == "RecordType":
+            new = RecordType()
+        elif self.role == "Record":
+            new = Record()
+        elif self.role == "Entity":
+            new = Entity()
+        else:
+            raise RuntimeError("Unkonwn role.")
+
+        # Copy special attributes:
+        # TODO: this might rise an exception when copying
+        #       special file attributes like checksum and size.
+        for attribute in SPECIAL_ATTRIBUTES + ["value"]:
+            val = getattr(self, attribute)
+            if val is not None:
+                setattr(new, attribute, val)
+
+        # Copy parents:
+        for p in self.parents:
+            new.add_parent(p)
+
+        # Copy properties:
+        for p in self.properties:
+            new.add_property(p, importance=self.get_importance(p))
+
+        return new
+
     @property
     def version(self):
         if self._version is not None or self._wrapped_entity is None:
@@ -269,14 +322,74 @@ class Entity(object):
         self.__pickup = new_pickup
 
     def grant(self, realm=None, username=None, role=None,
-              permission=None, priority=False):
+              permission=None, priority=False, revoke_denial=True):
+        """Grant a permission to a user or role for this entity.
+
+        You must specify either only the username and the realm, or only the
+        role.
+
+        By default a previously existing denial rule would be revoked, because
+        otherwise this grant wouldn't have any effect. However, for keeping
+        contradicting rules pass revoke_denial=False.
+
+        Parameters
+        ----------
+        permission: str
+            The permission to be granted.
+        username : str, optional
+            The username. Exactly one is required, either the `username` or the
+            `role`.
+        realm: str, optional
+            The user's realm. Required when username is not None.
+        role: str, optional
+            The role (as in Role-Based Access Control). Exactly one is
+            required, either the `username` or the `role`.
+        priority: bool, default False
+            Whether this permission is granted with priority over non-priority
+            rules.
+        revoke_denial: bool, default True
+            Whether a contradicting denial (with same priority flag) in this
+            ACL will be revoked.
+        """
+        # @review Florian Spreckelsen 2022-03-17
         self.acl.grant(realm=realm, username=username, role=role,
-                       permission=permission, priority=priority)
+                       permission=permission, priority=priority,
+                       revoke_denial=revoke_denial)
 
     def deny(self, realm=None, username=None, role=None,
-             permission=None, priority=False):
+             permission=None, priority=False, revoke_grant=True):
+        """Deny a permission to a user or role for this entity.
+
+        You must specify either only the username and the realm, or only the
+        role.
+
+        By default a previously existing grant rule would be revoked, because
+        otherwise this denial would override the grant rules anyways. However,
+        for keeping contradicting rules pass revoke_grant=False.
+
+        Parameters
+        ----------
+        permission: str
+            The permission to be denied.
+        username : str, optional
+            The username. Exactly one is required, either the `username` or the
+            `role`.
+        realm: str, optional
+            The user's realm. Required when username is not None.
+        role: str, optional
+            The role (as in Role-Based Access Control). Exactly one is
+            required, either the `username` or the `role`.
+        priority: bool, default False
+            Whether this permission is denied with priority over non-priority
+            rules.
+        revoke_grant: bool, default True
+            Whether a contradicting grant (with same priority flag) in this
+            ACL will be revoked.
+        """
+        # @review Florian Spreckelsen 2022-03-17
         self.acl.deny(realm=realm, username=username, role=role,
-                      permission=permission, priority=priority)
+                      permission=permission, priority=priority,
+                      revoke_grant=revoke_grant)
 
     def revoke_denial(self, realm=None, username=None,
                       role=None, permission=None, priority=False):
@@ -3636,13 +3749,15 @@ class ACI():
         self.permission = permission
 
     def __hash__(self):
-        return hash(str(self.realm) + ":" + str(self.username) +
-                    ":" + str(self.role) + ":" + str(self.permission))
+        return hash(self.__repr__())
 
     def __eq__(self, other):
         return isinstance(other, ACI) and (self.role is None and self.username == other.username and self.realm ==
                                            other.realm) or self.role == other.role and self.permission == other.permission
 
+    def __repr__(self):
+        return str(self.realm) + ":" + str(self.username) + ":" + str(self.role) + ":" + str(self.permission)
+
     def add_to_element(self, e):
         if self.role is not None:
             e.set("role", self.role)
@@ -3667,10 +3782,35 @@ class ACL():
             self.clear()
 
     def parse_xml(self, xml):
+        """Clear this ACL and parse the xml.
+
+        Iterate over the rules in the xml and add each rule to this ACL.
+
+        Contradicting rules will both be kept.
+
+        Parameters
+        ----------
+        xml : lxml.etree.Element
+            The xml element containing the ACL rules, i.e. <Grant> and <Deny>
+            rules.
+        """
         self.clear()
         self._parse_xml(xml)
 
     def _parse_xml(self, xml):
+        """Parse the xml.
+
+        Iterate over the rules in the xml and add each rule to this ACL.
+
+        Contradicting rules will both be kept.
+
+        Parameters
+        ----------
+        xml : lxml.etree.Element
+            The xml element containing the ACL rules, i.e. <Grant> and <Deny>
+            rules.
+        """
+        # @review Florian Spreckelsen 2022-03-17
         for e in xml:
             role = e.get("role")
             username = e.get("username")
@@ -3683,10 +3823,12 @@ class ACL():
 
                     if e.tag == "Grant":
                         self.grant(username=username, realm=realm, role=role,
-                                   permission=permission, priority=priority)
+                                   permission=permission, priority=priority,
+                                   revoke_denial=False)
                     elif e.tag == "Deny":
                         self.deny(username=username, realm=realm, role=role,
-                                  permission=permission, priority=priority)
+                                  permission=permission, priority=priority,
+                                  revoke_grant=False)
 
     def combine(self, other):
         """ Combine and return new instance."""
@@ -3764,12 +3906,42 @@ class ACL():
         if item in self._denials:
             self._denials.remove(item)
 
-    def grant(self, username=None, realm=None, role=None,
-              permission=None, priority=False):
+    def grant(self, permission, username=None, realm=None, role=None,
+              priority=False, revoke_denial=True):
+        """Grant a permission to a user or role.
+
+        You must specify either only the username and the realm, or only the
+        role.
+
+        By default a previously existing denial rule would be revoked, because
+        otherwise this grant wouldn't have any effect. However, for keeping
+        contradicting rules pass revoke_denial=False.
+
+        Parameters
+        ----------
+        permission: str
+            The permission to be granted.
+        username : str, optional
+            The username. Exactly one is required, either the `username` or the
+            `role`.
+        realm: str, optional
+            The user's realm. Required when username is not None.
+        role: str, optional
+            The role (as in Role-Based Access Control). Exactly one is
+            required, either the `username` or the `role`.
+        priority: bool, default False
+            Whether this permission is granted with priority over non-priority
+            rules.
+        revoke_denial: bool, default True
+            Whether a contradicting denial (with same priority flag) in this
+            ACL will be revoked.
+        """
+        # @review Florian Spreckelsen 2022-03-17
         priority = self._get_boolean_priority(priority)
         item = ACI(role=role, username=username,
                    realm=realm, permission=permission)
-        self._remove_item(item, priority)
+        if revoke_denial:
+            self._remove_item(item, priority)
 
         if priority is True:
             self._priority_grants.add(item)
@@ -3777,11 +3949,41 @@ class ACL():
             self._grants.add(item)
 
     def deny(self, username=None, realm=None, role=None,
-             permission=None, priority=False):
+             permission=None, priority=False, revoke_grant=True):
+        """Deny a permission to a user or role for this entity.
+
+        You must specify either only the username and the realm, or only the
+        role.
+
+        By default a previously existing grant rule would be revoked, because
+        otherwise this denial would override the grant rules anyways. However,
+        for keeping contradicting rules pass revoke_grant=False.
+
+        Parameters
+        ----------
+        permission: str
+            The permission to be denied.
+        username : str, optional
+            The username. Exactly one is required, either the `username` or the
+            `role`.
+        realm: str, optional
+            The user's realm. Required when username is not None.
+        role: str, optional
+            The role (as in Role-Based Access Control). Exactly one is
+            required, either the `username` or the `role`.
+        priority: bool, default False
+            Whether this permission is denied with priority over non-priority
+            rules.
+        revoke_grant: bool, default True
+            Whether a contradicting grant (with same priority flag) in this
+            ACL will be revoked.
+        """
+        # @review Florian Spreckelsen 2022-03-17
         priority = self._get_boolean_priority(priority)
         item = ACI(role=role, username=username,
                    realm=realm, permission=permission)
-        self._remove_item(item, priority)
+        if revoke_grant:
+            self._remove_item(item, priority)
 
         if priority is True:
             self._priority_denials.add(item)

src/caosdb/configuration.py

@@ -84,23 +84,28 @@ def config_to_yaml(config):
 
 
 def validate_yaml_schema(valobj):
-    # TODO: Re-enable warning once the schema has been extended to also cover
-    # SSS pycaosdb.inis and integration tests.
     if optional_jsonschema_validate:
         with open(os.path.join(os.path.dirname(__file__), "schema-pycaosdb-ini.yml")) as f:
             schema = yaml.load(f, Loader=yaml.SafeLoader)
         optional_jsonschema_validate(instance=valobj, schema=schema["schema-pycaosdb-ini"])
-    # else:
-    #     warnings.warn("""
-    #                     Warning: The validation could not be performed because `jsonschema` is not installed.
-    #                 """)
+    else:
+        warnings.warn("""
+            Warning: The validation could not be performed because `jsonschema` is not installed.
+        """)
 
 
 def _read_config_files():
-    """Function to read config files from different paths. Checks for path in $PYCAOSDBINI or home directory (.pycaosdb.ini) and in the current working directory (pycaosdb.ini).
+    """Function to read config files from different paths.
+
+    Checks for path either in ``$PYCAOSDBINI`` or home directory (``.pycaosdb.ini``), and
+    additionally in the current working directory (``pycaosdb.ini``).
+
+    Returns
+    -------
+
+    ini files: list
+      The successfully parsed ini-files. Order: env_var or home directory, cwd. Used for testing the function.
 
-    Returns:
-        [list]: list with successfully parsed ini-files. Order: env_var or home directory, cwd. Used for testing the function.
     """
     return_var = []
     if "PYCAOSDBINI" in environ:

src/caosdb/schema-pycaosdb-ini.yml

@@ -65,26 +65,39 @@ schema-pycaosdb-ini:
             properties:
               password_method:
                 const: input
+            required: [password_method]
           then:
             required: [url]
         - if:
             properties:
               password_method:
                 const: plain
+            required: [password_method]
           then:
             required: [url, username, password]
         - if:
             properties:
               password_method:
                 const: pass
+            required: [password_method]
           then:
             required: [url, username, password_identifier]
         - if:
             properties:
               password_method:
                 const: keyring
+            required: [password_method]
           then:
             required: [url, username]
     IntegrationTests:
       description: "Used by the integration test suite from the caosdb-pyinttest repo."
       additionalProperties: true
+    Misc:
+      description: "Some additional configuration settings."
+      additionalProperties: true
+    advancedtools:
+      description: "Configuration settings for the caosadvancedtools."
+      additionalProperties: true
+    sss_helper:
+      description: "Configuration settings for server-side scripting."
+      additionalProperties: true

src/doc/conf.py

@@ -25,14 +25,14 @@ import sphinx_rtd_theme  # noqa: E402
 # -- Project information -----------------------------------------------------
 
 project = 'pycaosdb'
-copyright = '2020, IndiScale GmbH'
+copyright = '2022, IndiScale GmbH'
 author = 'Daniel Hornung'
 
 # The short X.Y version
-version = '0.5.2'
+version = '0.8.0'
 # The full version, including alpha/beta/rc tags
 # release = '0.5.2-rc2'
-release = '0.5.2'
+release = '0.8.0-dev'
 
 
 # -- General configuration ---------------------------------------------------

src/doc/configuration.md

@@ -4,6 +4,15 @@ PyCaosDB tries to read from the inifile specified in the environment variable `P
 alternatively in `~/.pycaosdb.ini` upon import.  After that, the ini file `pycaosdb.ini` in the
 current working directory will be read additionally, if it exists.
 
+Here, we will look at the most common configuration options. For a full and comprehensive
+description please check out the [example pycaosdb.ini file](https://gitlab.com/caosdb/caosdb-pylib/-/blob/main/examples/pycaosdb.ini).  You can download this file and use
+it as a starting point.
+
+
+Typically, you need to change at least the `url` and `username` fields as required.  (Ask your
+CaosDB administrator or IT crowd if you do not know what to put there, but for the demo instance at
+https://demo.indiscale.com, `username=admin` and `password=caosdb` should work).
+
 ## Authentication ##
 
 The default configuration (that your are asked for your password when ever a connection is created
@@ -17,6 +26,8 @@ can be changed by setting `password_method`:
   Windows). The password will be queried on first usage.
 * with `password_method=plain` (**strongly discouraged**)
 
+The following illustrates the recommended options:
+
 ```ini
 [Connection]
 username=YOUR_USERNAME
@@ -35,7 +46,10 @@ username=YOUR_USERNAME
 
 ## SSL Certificate ##
 
-You can set the pass to the ssl certificate to be used:
+In some cases (especially if you are testing CaosDB) you might need to supply an SSL certificate to
+allow SSL encryption.
+
+The `cacert` option sets the path to the ssl certificate for the connection:
 
 ```ini
 [Connection]
@@ -49,6 +63,8 @@ with CaosDB which makes the experience much less verbose. Set it to 1 or 2 in ca
 debugging (which I hope will not be necessary for this tutorial) or if you want to learn more about
 the internals of the protocol. 
 
+`timeout` sets the timeout for requests to the server.
+
 A complete list of options can be found in the 
 [pycaosdb.ini file](https://gitlab.com/caosdb/caosdb-pylib/-/blob/main/examples/pycaosdb.ini) in 
 the examples folder of the source code.

tox.ini

@@ -9,3 +9,6 @@ deps = .
     pytest-cov
     jsonschema==4.0.1
 commands=py.test --cov=caosdb -vv {posargs}
+
+[flake8]
+max-line-length=100

unittests/test_acl.py

+# -*- encoding: utf-8 -*-
+#
+# This file is a part of the CaosDB Project.
+#
+# Copyright (C) 2022 Indiscale GmbH <info@indiscale.com>
+# Copyright (C) 2022 Timm Fitschen <f.fitschen@indiscale.com>
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Affero General Public License as
+# published by the Free Software Foundation, either version 3 of the
+# License, or (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU Affero General Public License for more details.
+#
+# You should have received a copy of the GNU Affero General Public License
+# along with this program. If not, see <https://www.gnu.org/licenses/>.
+#
+import caosdb as db
+from lxml import etree
+
+
+def test_parse_xml():
+    # @review Florian Spreckelsen 2022-03-17
+    xml_str = """
+        <EntityACL>
+          <Grant priority="False" role="role1">
+            <Permission name="RETRIEVE:ENTITY"/>
+          </Grant>
+          <Deny priority="False" role="role1">
+            <Permission name="RETRIEVE:ENTITY"/>
+          </Deny>
+          <Grant priority="True" role="role1">
+            <Permission name="RETRIEVE:ENTITY"/>
+          </Grant>
+          <Deny priority="True" role="role1">
+            <Permission name="RETRIEVE:ENTITY"/>
+          </Deny>
+        </EntityACL>"""
+    xml = etree.fromstring(xml_str)
+    left_acl = db.ACL(xml)
+
+    right_acl = db.ACL()
+    right_acl.grant(role="role1", permission="RETRIEVE:ENTITY",
+                    revoke_denial=False)
+    right_acl.deny(role="role1", permission="RETRIEVE:ENTITY",
+                   revoke_grant=False)
+    right_acl.grant(role="role1", permission="RETRIEVE:ENTITY",
+                    priority=True, revoke_denial=False)
+    right_acl.deny(role="role1", permission="RETRIEVE:ENTITY",
+                   priority=True, revoke_grant=False)
+
+    assert left_acl == right_acl

unittests/test_apiutils.py

@@ -31,10 +31,14 @@ import tempfile
 import caosdb as db
 import caosdb.apiutils
 from caosdb.apiutils import (apply_to_ids, compare_entities, create_id_query,
-                             resolve_reference)
+                             resolve_reference, merge_entities)
+
+from caosdb.common.models import SPECIAL_ATTRIBUTES
 
 from .test_property import testrecord
 
+import pytest
+
 
 def test_convert_object():
     r2 = db.apiutils.convert_to_python_object(testrecord)
@@ -230,3 +234,92 @@ def test_compare_special_properties():
             assert diff_r2[key] == 2
         assert len(diff_r1["properties"]) == 0
         assert len(diff_r2["properties"]) == 0
+
+
+def test_copy_entities():
+    r = db.Record(name="A")
+    r.add_parent(name="B")
+    r.add_property(name="C", value=4, importance="OBLIGATORY")
+    r.add_property(name="D", value=[3, 4, 7], importance="OBLIGATORY")
+    r.description = "A fancy test record"
+
+    c = r.copy()
+
+    assert c is not r
+    assert c.name == "A"
+    assert c.role == r.role
+    assert c.parents[0].name == "B"
+    # parent and property objects are not shared among copy and original:
+    assert c.parents[0] is not r.parents[0]
+
+    for i in [0, 1]:
+        assert c.properties[i] is not r.properties[i]
+        for special in SPECIAL_ATTRIBUTES:
+            assert getattr(c.properties[i], special) == getattr(r.properties[i], special)
+        assert c.get_importance(c.properties[i]) == r.get_importance(r.properties[i])
+
+
+def test_merge_entities():
+    r = db.Record(name="A")
+    r.add_parent(name="B")
+    r.add_property(name="C", value=4, importance="OBLIGATORY")
+    r.add_property(name="D", value=[3, 4, 7], importance="OBLIGATORY")
+    r.description = "A fancy test record"
+
+    r2 = db.Record()
+    r2.add_property(name="F", value="text")
+    merge_entities(r2, r)
+    assert r2.get_parents()[0].name == "B"
+    assert r2.get_property("C").name == "C"
+    assert r2.get_property("C").value == 4
+    assert r2.get_property("D").name == "D"
+    assert r2.get_property("D").value == [3, 4, 7]
+
+    assert r2.get_property("F").name == "F"
+    assert r2.get_property("F").value == "text"
+
+
+def test_merge_bug_109():
+    rt = db.RecordType(name="TestBug")
+    p = db.Property(name="test_bug_property", datatype=db.LIST(db.INTEGER))
+
+    r_b = db.Record(name="TestRecord")
+    r_b.add_parent(rt)
+    r_b.add_property(p, value=[18, 19])
+
+    r_a = db.Record(name="TestRecord")
+    r_a.add_parent(rt)
+
+    merge_entities(r_a, r_b)
+
+    assert r_b.get_property("test_bug_property").value == [18, 19]
+    assert r_a.get_property("test_bug_property").value == [18, 19]
+
+    assert "<Value>18</Value>\n    <Value>19</Value>" in str(r_b)
+    assert "<Value>18</Value>\n    <Value>19</Value>\n    <Value>18</Value>\n    <Value>19</Value>" not in str(r_b)
+
+    assert "<Value>18</Value>\n    <Value>19</Value>" in str(r_a)
+    assert "<Value>18</Value>\n    <Value>19</Value>\n    <Value>18</Value>\n    <Value>19</Value>" not in str(r_a)
+
+
+@pytest.mark.xfail
+def test_bug_109():
+    rt = db.RecordType(name="TestBug")
+    p = db.Property(name="test_bug_property", datatype=db.LIST(db.INTEGER))
+
+    r_b = db.Record(name="TestRecord")
+    r_b.add_parent(rt)
+    r_b.add_property(p, value=[18, 19])
+
+    r_a = db.Record(name="TestRecord")
+    r_a.add_parent(rt)
+    r_a.add_property(r_b.get_property("test_bug_property"))
+
+    assert r_b.get_property("test_bug_property").value == [18, 19]
+    assert r_a.get_property("test_bug_property").value == [18, 19]
+
+    assert "<Value>18</Value>\n    <Value>19</Value>" in str(r_b)
+    assert "<Value>18</Value>\n    <Value>19</Value>\n    <Value>18</Value>\n    <Value>19</Value>" not in str(r_b)
+
+    assert "<Value>18</Value>\n    <Value>19</Value>" in str(r_a)
+    assert "<Value>18</Value>\n    <Value>19</Value>\n    <Value>18</Value>\n    <Value>19</Value>" not in str(r_a)

unittests/test_configs/pycaosdb-IntegrationTests.ini

+# -*- mode:conf; -*-
+## This sections needs to exist in addition to the usual section
+[IntegrationTests]
+# test_server_side_scripting.bin_dir.local=/path/to/scripting/bin
+test_server_side_scripting.bin_dir.local=/home/myself/test/caosdb-server/scripting/bin
+# test_server_side_scripting.bin_dir.server=/opt/caosdb/git/caosdb-server/scripting/bin
+
+# # location of the files from the pyinttest perspective
+# test_files.test_insert_files_in_dir.local=/extroot/test_insert_files_in_dir/
+test_files.test_insert_files_in_dir.local=/home/myself/test/debug_advanced/paths/extroot/test_insert_files_in_dir
+# # location of the files from the caosdb_servers perspective
+test_files.test_insert_files_in_dir.server=/opt/caosdb/mnt/extroot/test_insert_files_in_dir/
+
+########## Files ##################
+## Used by tests of file handling. Specify the path to an existing
+## directory in which file tests are performed, once as seen by the
+## host and once as seen by the server.
+
+# location of the files from the pyinttest (i.e. host) perspective
+#test_files.test_insert_files_in_dir.local=/extroot/test_insert_files_in_dir/
+
+# location of the files from the caosdb server's perspective
+#test_files.test_insert_files_in_dir.server=/opt/caosdb/mnt/extroot/test_insert_files_in_dir/
+
+# # location of the one-time tokens from the pyinttest's perspective
+# test_authentication.admin_token_crud = /authtoken/admin_token_crud.txt
+# test_authentication.admin_token_expired = /authtoken/admin_token_expired.txt
+# test_authentication.admin_token_3_attempts = /authtoken/admin_token_3_attempts.txt
+
+
+## Insert your usual settings here
+[Connection]
+url=https://localhost:10443/
+username=admin
+password_method=plain
+password=caosdb
+

unittests/test_configs/pycaosdb-real-world-1.ini

+[Connection]
+url = https://localhost:10443
+cacert = /opt/caosdb/cert/caosdb.cert.pem
+debug = 0
+timeout = 5000
+
+[Misc]
+sendmail = /usr/local/bin/sendmail_to_file
+entity_loan.curator_mail_from=crawler-test@example.com
+entity_loan.curator_mail_to=crawler-test@example.com
+
+[sss_helper]
+external_uri = https://caosdb.example.com:443
+
+[advancedtools]
+crawler.from_mail=admin@example.com
+crawler.to_mail=admin@example.com

unittests/test_configs/pycaosdb-real-world-2.ini

+[Connection]
+url = https://samplemanager.example.com:443
+cacert = /opt/caosdb/cert/caosdb.cert.pem
+debug = 0
+timeout = 5000
+[Misc]
+sendmail = /usr/local/bin/sendmail_to_file
+entity_loan.curator_mail_from=crawler-test@example.com
+entity_loan.curator_mail_to=crawler-test@example.com
+[sss_helper]
+external_uri = https://localhost:10443
+[advancedtools]
+crawler.from_mail=crawler-test@example.com
+crawler.to_mail=crawler-test@example.com           
+

unittests/test_configs/pycaosdb-server-side-scripting.ini

+; this is the pycaosdb.ini for the server-side-scripting home.
+[Connection]
+url = https://caosdb-server:10443
+cacert = /opt/caosdb/cert/caosdb.cert.pem
+debug = 0
+timeout = 5000
+
+[Misc]
+sendmail = /usr/local/bin/sendmail_to_file