diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
index 9f8c131968c050fb18001b5dc7c5468d0ed26dae..72b4381aebcaf5edd16c10c479df6ca908128ec6 100644
--- a/.gitlab-ci.yml
+++ b/.gitlab-ci.yml
@@ -47,7 +47,7 @@ pylint:
tags: [ docker ]
stage: linting
script:
- - pylint3 --unsafe-load-any-extension=y -d all -e E,F src/caosdb/common
+ - pylint --unsafe-load-any-extension=y -d all -e E,F src/caosdb/common
allow_failure: true
diff --git a/.gitlab/merge_request_templates/Default.md b/.gitlab/merge_request_templates/Default.md
index 77a95da1cc40c815e4952a1283d345af56e80461..e43435cb31f415fc4f9c8447983b411627612ad7 100644
--- a/.gitlab/merge_request_templates/Default.md
+++ b/.gitlab/merge_request_templates/Default.md
@@ -25,6 +25,7 @@ guidelines](https://gitlab.com/caosdb/caosdb/-/blob/dev/REVIEW_GUIDELINES.md)
- [ ] All automated tests pass
- [ ] Reference related Issues
- [ ] Up-to-date CHANGELOG.md
+- [ ] Add type hints in created/changed code
- [ ] Annotations in code (Gitlab comments)
- Intent of new code
- Problems with old code
diff --git a/CHANGELOG.md b/CHANGELOG.md
index b93e013a03f5c716bad9f9337188f8715ac3c9c2..e109ba5a37dcccabcaf2b341212def848e676a99 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -9,20 +9,42 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
### Added ###
+- It is possible now to supply a password for caosdb_admin on the command line and also activate the user directly using "-c".
* Added examples for complex data models to documentation
* extended apiutils with `resolve_reference(Property)`
* is_reference function for Properties
### Changed ###
+* Retrievals of entities where the class does not match the entity role raise
+ a ValueError now. See
+ [here](https://gitlab.indiscale.com/caosdb/src/caosdb-pylib/-/issues/66) for more
+ information. Updating a role is now being done by setting the `Entity.role`
+ to the new role (as string).
+* Entity.add_property and Entity.add_parent do not accept `**kwargs`-style
+ keywords anymore. Formerly known keywords have been refactored into named
+ parameters.
+* [#35](https://gitlab.com/caosdb/caosdb-pylib/-/issues/35) Loggers now use the name of
+ the unit where they are called instead of a static name
+
### Deprecated ###
-* `id_query(ids)` in apiutils
+* `id_query(ids)` in apiutils (to be removed with >=0.5.4)
+* The whole yamlapi with the following functions (to be removed with >=0.5.4):
+ * `append_sublist`
+ * `kv_to_xml`
+ * `dict_to_xml`
+ * `yaml_to_xml`
+ * `process`
+ * `yaml_file_to_xml`
+ * `insert_yaml_file`
### Removed ###
### Fixed ###
+* #60 Unintuitive behavior of `Entity.role` after a `Entity(id).retrieve()`
+ Originally the role was `None`. The correct role is present now.
* #53 Documentation of inheritance
* #38 Dependencies in chunk-deletion of containers
@@ -37,6 +59,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
* `etag` property for the `caosdb.Query` class. The etag allows to debug the
caching and to decide whether the server has changed between queries.
* function `_read_config_files` to read `pycaosdb.ini` files from different paths.
+* function `retrieve_substructure` that recursively adds connected entities.
### Changed ###
diff --git a/README_SETUP.md b/README_SETUP.md
index 9da548395073643c16539cef180c4d6412dd8d46..e58f934ceba176e4b5ba42239565f8e3bd48171a 100644
--- a/README_SETUP.md
+++ b/README_SETUP.md
@@ -11,6 +11,10 @@ typically be installed automatically):
- `PyYaml`
- `PySocks`
+Optional packages:
+- `keyring`
+- `jsonschema`
+
### How to install ###
#### Linux ####
@@ -30,12 +34,14 @@ packages you will ever need out of the box. If you prefer, you may also install
After installation, open an Anaconda prompt from the Windows menu and continue in the [Generic
installation](#generic-installation) section.
-#### iOS ####
+#### MacOS ####
-If there is no Python 3 installed yet, there are two main ways to obtain it: Either get the binary
-package from [python.org](https://www.python.org/downloads/) or, for advanced users, install via [Homebrew](https://brew.sh/). After installation
-from python.org, it is recommended to also update the TLS certificates for Python (this requires
-administrator rights for your user):
+If there is no Python 3 installed yet, there are two main ways to
+obtain it: Either get the binary package from
+[python.org](https://www.python.org/downloads/) or, for advanced
+users, install via [Homebrew](https://brew.sh/). After installation
+from python.org, it is recommended to also update the TLS certificates
+for Python (this requires administrator rights for your user):
```sh
# Replace this with your Python version number:
@@ -45,7 +51,8 @@ cd /Applications/Python\ 3.9/
sudo ./Install\ Certificates.command
```
-After these steps, you may continue with the [Generic installation](#generic-installation).
+After these steps, you may continue with the [Generic
+installation](#generic-installation).
#### Generic installation ####
@@ -66,6 +73,13 @@ cd caosdb-pylib
pip3 install --user .
```
+For installation of optional packages, install with an additional option, e.g. for
+validating with the caosdb json schema:
+
+```sh
+pip3 install --user .[jsonschema]
+```
+
## Configuration ##
The configuration is done using `ini` configuration files.
diff --git a/setup.py b/setup.py
index e1d39458ea8d1b0b17ea12a82ebd7133b27b045a..d0e12f1a88f43883ede5e53c90512968ca310e93 100755
--- a/setup.py
+++ b/setup.py
@@ -159,11 +159,12 @@ def setup_package():
package_dir={'': 'src'},
install_requires=['lxml>=3.6.4',
'PyYaml>=3.12', 'future', 'PySocks>=1.6.7'],
- extras_require={'keyring': ['keyring>=13.0.0']},
+ extras_require={'keyring': ['keyring>=13.0.0'],
+ 'jsonschema': ['jsonschema==4.0.1']},
setup_requires=["pytest-runner>=2.0,<3dev"],
- tests_require=["pytest", "pytest-cov", "coverage>=4.4.2"],
+ tests_require=["pytest", "pytest-cov", "coverage>=4.4.2", "jsonschema==4.0.1"],
package_data={
- 'caosdb': ['cert/indiscale.ca.crt'],
+ 'caosdb': ['cert/indiscale.ca.crt', 'schema-pycaosdb-ini.yml'],
},
scripts=["src/caosdb/utils/caosdb_admin.py"]
)
diff --git a/src/caosdb/apiutils.py b/src/caosdb/apiutils.py
index 9192289a0b6518a2d6ec3abcdeca9d1d2c063d6d..b3d5df698baceacf671c5171430dfa48b3c881b5 100644
--- a/src/caosdb/apiutils.py
+++ b/src/caosdb/apiutils.py
@@ -517,9 +517,9 @@ def getOriginUrlIn(folder):
with open(t.name, "r") as t:
urlString = "Fetch URL:"
- for l in t.readlines():
- if urlString in l:
- return l[l.find(urlString) + len(urlString):].strip()
+ for line in t.readlines():
+ if urlString in line:
+ return line[line.find(urlString) + len(urlString):].strip()
return None
diff --git a/src/caosdb/common/administration.py b/src/caosdb/common/administration.py
index e7ba94182d7a4d8b60c6400cd1d804f62f7bf03c..dff461e7fb0ed5270119907bd4ad859503b3ce21 100644
--- a/src/caosdb/common/administration.py
+++ b/src/caosdb/common/administration.py
@@ -26,16 +26,12 @@
"""missing docstring."""
-from lxml import etree
-
from caosdb.common.utils import xml2str
from caosdb.connection.connection import get_connection
-from caosdb.exceptions import (HTTPClientError,
- HTTPForbiddenError,
- HTTPResourceNotFoundError,
- EntityDoesNotExistError,
- ServerConfigurationException,
- )
+from caosdb.exceptions import (EntityDoesNotExistError, HTTPClientError,
+ HTTPForbiddenError, HTTPResourceNotFoundError,
+ ServerConfigurationException)
+from lxml import etree
def set_server_property(key, value):
@@ -156,13 +152,22 @@ def _update_user(name, realm=None, password=None, status=None,
return con.put_form_data(entity_uri_segment="User/" + (realm + "/" + name if realm is not None else name), params=params, **kwargs).read()
except HTTPResourceNotFoundError as e:
e.msg = "User does not exist."
- raise
+ raise e
except HTTPForbiddenError as e:
e.msg = "You are not permitted to update this user."
- raise
+ raise e
except HTTPClientError as e:
if e.status == 409:
e.msg = "Entity does not exist."
+
+ if e.status == 422:
+ e.msg = """Maybe the password does not match the required standard?
+ The current requirements are:
+ - at least 8 characters
+ - at least 1 number
+ - at least 1 lower case character
+ - at least 1 upper case character
+ - at least 1 special character"""
raise
@@ -191,7 +196,13 @@ def _insert_user(name, password=None, status=None, email=None, entity=None, **kw
e.msg = "User name is already in use."
if e.status == 422:
- e.msg = "Maybe the password does not match the required standard?"
+ e.msg = """Maybe the password does not match the required standard?
+ The current requirements are:
+ - at least 8 characters
+ - at least 1 number
+ - at least 1 lower case character
+ - at least 1 upper case character
+ - at least 1 special character"""
raise e
diff --git a/src/caosdb/common/datatype.py b/src/caosdb/common/datatype.py
index 5434f5b6556a13f65754eacc66cb32231366e5b3..03ff6d023ab0d3005c37d56c65353c1a1072518e 100644
--- a/src/caosdb/common/datatype.py
+++ b/src/caosdb/common/datatype.py
@@ -62,10 +62,21 @@ def is_list_datatype(datatype):
def is_reference(datatype):
- """ returns whether the value is a reference
+ """Returns whether the value is a reference
FILE and REFERENCE properties are examples, but also datatypes that are
- RecordTypes
+ RecordTypes.
+
+ Parameters
+ ----------
+ datatype : str
+ The datatype to check.
+
+ Returns
+ -------
+ bool
+ True if the datatype is a not base datatype or a list of a base datatype.
+ Otherwise False is returned.
"""
if datatype is None:
@@ -80,6 +91,41 @@ def is_reference(datatype):
return True
+def get_referenced_recordtype(datatype):
+ """Return the record type of the referenced datatype.
+
+ Raises
+ ------
+ ValueError
+ In cases where datatype is not a reference, the list does not have
+ a referenced record type or the datatype is a FILE.
+
+ Parameters
+ ----------
+ datatype : str
+ The datatype to check.
+
+ Returns
+ -------
+ str
+ String containing the name of the referenced datatype.
+ """
+
+ if not is_reference(datatype):
+ raise ValueError("datatype must be a reference")
+
+ if is_list_datatype(datatype):
+ datatype = get_list_datatype(datatype)
+ if datatype is None:
+ raise ValueError("list does not have a list datatype")
+
+ if datatype == FILE:
+ raise ValueError(
+ "FILE references are not considered references with a record type")
+
+ return datatype
+
+
def get_id_of_datatype(datatype):
""" returns the id of a Record Type
diff --git a/src/caosdb/common/models.py b/src/caosdb/common/models.py
index fb16918241e30c57c8439ad316c89f441808bef6..15c67fc96d099e3e4d5a7e06364330decab9a25f 100644
--- a/src/caosdb/common/models.py
+++ b/src/caosdb/common/models.py
@@ -134,7 +134,10 @@ class Entity(object):
@role.setter
def role(self, role):
- self.__role = role
+ if role is not None and role.lower() == "entity":
+ self.__role = None
+ else:
+ self.__role = role
@property
def size(self):
@@ -202,6 +205,10 @@ class Entity(object):
def description(self, new_description):
self.__description = new_description
+ @property
+ def checksum(self):
+ return self._checksum
+
@property
def unit(self):
if self.__unit is not None or self._wrapped_entity is None:
@@ -345,65 +352,86 @@ class Entity(object):
def add_property(self, property: Union[int, str, Entity, None] = None,
value: Union[str, int, Entity, None] = None,
- **kwargs) -> Entity: # @ReservedAssignment
+ id: Optional[int] = None, name: Optional[str] = None,
+ description: Optional[str] = None, datatype: Optional[str] = None,
+ unit: Optional[str] = None, importance: Optional[str] = None,
+ inheritance: Optional[str] = None) -> Entity: # @ReservedAssignment
"""Add a property to this entity.
The first parameter is meant to identify the property entity. So the method expects an instance of
Entity, an integer or a string here. The second parameter is the value of the new property. Any
- other named parameter may be passed by means of the **kwargs. Accepted keywords are:
+ other named parameter may be passed by means of the keywwords. Accepted keywords are:
id, name, description, importance, inheritance, datatype, and unit. Any other keyword will be
ignored right now. But that may change in the future.
Parameters
----------
- property : int, str, Entity, optional
- An identifying parameter (name, id or abstract property), by default None
- value : str, int, Entity, optional
+ property : Union[int, str, Entity, None], optional
+ An identifying parameter, by default None
+ value : Union[str, int, Entity, None], optional
The value of the new property, by default None
+ id : Optional[int], optional
+ Id of the property, by default None
+ name : Optional[str], optional
+ Name of the property, by default None
+ description : Optional[str], optional
+ Description of the property, by default None
+ datatype : Optional[str], optional
+ Datatype of the property, by default None
+ unit : Optional[str], optional
+ Unit of the property, by default None
+ importance : Optional[str], optional
+ Importance of the property, by default None
+ inheritance : Optional[str], optional
+ Inheritance of the property, by default None
Returns
-------
Entity
The entity passed
+
Raises
------
+ ValueError
+ Unusual Property role
UserWarning
- If the first parameter is None then kwargs['id'] or kwargs['name'] must be defined and not be None.
- Otherwise a UserWarning is raised.
+ If the first parameter is None then id or name must be defined and not be None.
UserWarning
- If the first parameter is an integer then it is interpreted as the id and kwargs['id'] must be
- undefined or None. Otherwise a UserWarning is raised.
+ If the first parameter is an integer then it is interpreted as the id and id must be
+ undefined or None.
UserWarning
If the first parameter is not None and neither an instance of Entity nor an integer it is
- interpreted as the name and kwargs['name'] must be undefined or None. Otherwise a UserWarning is
- raised.
+ interpreted as the name and name must be undefined or None.
"""
- copy_kwargs = kwargs.copy()
- name = (kwargs['name'] if 'name' in kwargs else None)
- pid = (kwargs['id'] if 'id' in kwargs else None)
+
+ pid = id
abstract_property = None
if isinstance(property, Entity):
+ if property.role is not None and property.role.lower() in ["record", "file"]:
+ raise ValueError("The property parameter is a {0}. This "
+ "is very unusual and probably not what you "
+ "want. Otherwise, construct a property from "
+ "a {0} using the Property class and add "
+ "that to this entity.".format(property.role))
abstract_property = property
elif isinstance(property, int):
if pid is not None:
raise UserWarning("The first parameter was an integer which would normally be interpreted as the id of the property which is to be added. But you have also specified a parameter 'id' in the method call. This is ambiguous and cannot be processed.")
pid = property
- copy_kwargs['id'] = pid
+ id = pid
elif property is not None:
if name is not None:
raise UserWarning("The first parameter was neither an instance of Entity nor an integer. Therefore the string representation of your first parameter would normally be interpreted name of the property which is to be added. But you have also specified a parameter 'name' in the method call. This is ambiguous and cannot be processed.")
name = str(property)
- copy_kwargs['name'] = name
if property is None and name is None and pid is None:
raise UserWarning(
"This method expects you to pass at least an entity, a name or an id.")
- del copy_kwargs['importance']
- del copy_kwargs['inheritance']
- new_property = Property(**copy_kwargs)
+ new_property = Property(name=name, id=id, description=description, datatype=datatype,
+ value=value, unit=unit)
if abstract_property is not None:
new_property._wrap(property)
@@ -416,9 +444,7 @@ class Entity(object):
new_property.value = value
self.properties.append(
- property=new_property, importance=(
- kwargs['importance'] if 'importance' in kwargs else None), inheritance=(
- kwargs['inheritance'] if 'inheritance' in kwargs else None))
+ property=new_property, importance=importance, inheritance=inheritance)
return self
@@ -455,35 +481,38 @@ class Entity(object):
return self
- def add_parent(self, parent: Union[Entity, int, str, None] = None,
- **kwargs) -> Entity: # @ReservedAssignment
+ def add_parent(self, parent: Union[Entity, int, str, None] = None,
+ id: Optional[int] = None, name: Optional[str] = None,
+ inheritance: Optional[str] = None) -> Entity: # @ReservedAssignment
"""Add a parent to this entity.
Parameters
----------
- parent :Entity, int, str, optional
+ parent : Union[Entity, int, str, None], optional
The parent entity, either specified by the Entity object
itself, or its id or its name, by default None
- **kwargs : dict, optional
- Additional keyword arguments for specifying the parent by
- name or id, and for specifying the mode of inheritance.
- id : int
- Integer id of the parent entity. Ignored if `parent`
- is not None.
- name : str
- Name of the parent entity. Ignored if `parent is not
- none`.
- inheritance : str
- One of ``obligatory``, ``recommended``, ``suggested``, or ``fix``. Specifies the
- minimum importance which parent properties need to have to be inherited by this
- entity. If no `inheritance` is given, no properties will be inherited by the child.
- This parameter is case-insensitive.
-
- Note that the behaviour is currently not yet specified when assigning parents to
- Records, it only works for inheritance of RecordTypes (and Properties).
-
- For more information, it is recommended to look into the
- :ref:`data insertion tutorial<tutorial-inheritance-properties>`.
+ id : Optional[int], optional
+ Integer id of the parent entity. Ignored if `parent`
+ is not None, by default None
+ name : Optional[str], optional
+ Name of the parent entity. Ignored if `parent is not
+ none`, by default None
+ inheritance : Optional[str], optional
+ One of ``obligatory``, ``recommended``, ``suggested``, or ``fix``. Specifies the
+ minimum importance which parent properties need to have to be inherited by this
+ entity. If no `inheritance` is given, no properties will be inherited by the child.
+ This parameter is case-insensitive.
+
+ Note that the behaviour is currently not yet specified when assigning parents to
+ Records, it only works for inheritance of RecordTypes (and Properties).
+
+ For more information, it is recommended to look into the
+ :ref:`data insertion tutorial<tutorial-inheritance-properties>`, by default None
+
+ Returns
+ -------
+ Entity
+ The entity passed
Returns
-------
@@ -496,8 +525,8 @@ class Entity(object):
If neither a `parent` parameter, nor the `id`, nor `name`
parameter is passed to this method.
"""
- name = (kwargs['name'] if 'name' in kwargs else None)
- pid = (kwargs['id'] if 'id' in kwargs else None)
+
+ pid = id
parent_entity = None
if isinstance(parent, Entity):
@@ -511,9 +540,6 @@ class Entity(object):
raise UserWarning(
"This method expects you to pass at least an entity, a name or an id.")
- inheritance = (kwargs['inheritance']
- if 'inheritance' in kwargs else None)
-
addp = Parent(id=pid, name=name, inheritance=inheritance)
if parent_entity is not None:
@@ -682,6 +708,7 @@ class Entity(object):
ValueError
Wrong input type.
"""
+
# entity given
pattern_name = getattr(pattern, "name", None)
pattern_id = getattr(pattern, "id", None)
@@ -730,6 +757,7 @@ class Entity(object):
See also get_property_values()
"""
+
SPECIAL_SELECTORS = ["unit", "value", "description", "id", "name"]
if not isinstance(selector, (tuple, list)):
@@ -936,8 +964,9 @@ class Entity(object):
Raise an error if `xml` is not a lxml.etree.Element
"""
if xml is None:
- xml = etree.Element("Entity")
- #assert isinstance(xml, etree._Element)
+ # use role as xml tag name, fall-back to "Entity"
+ elem_tag = "Entity" if self.role is None else self.role
+ xml = etree.Element(elem_tag)
if not isinstance(xml, etree._Element):
raise TypeError("xml has to be a lxml.etree.Element")
@@ -1070,6 +1099,8 @@ class Entity(object):
TypeError
If a Child was neither a Property, nor a Parent, nor a Message.
"""
+ if isinstance(entity, Entity):
+ entity.role = elem.tag
entity._cuid = elem.get("cuid")
entity.id = elem.get("id") # @ReservedAssignment
entity.name = elem.get("name")
@@ -1450,6 +1481,7 @@ class QueryTemplate():
description: Optional[str] = None) -> None: # @ReservedAssignment
self.id = (int(id) if id is not None else None)
+ self.role = "QueryTemplate"
self.name = name
self.description = description
self.query = query
@@ -1766,32 +1798,33 @@ class Property(Entity):
def add_property(self, property: Union[int, str, Entity, None] = None,
value: Union[str, int, Entity, None] = None,
- **kwargs) -> Entity: # @ReservedAssignment
- copy_kwargs = kwargs.copy()
+ id=None, name=None, description=None, datatype=None,
+ unit=None, importance=FIX, inheritance=FIX) -> Entity: # @ReservedAssignment
- if 'importance' not in copy_kwargs:
- # set default importance
- copy_kwargs['importance'] = FIX
-
- if 'inheritance' not in copy_kwargs:
- # set default importance
- copy_kwargs['inheritance'] = FIX
+ return super().add_property(
+ property=property, id=id, name=name, description=description, datatype=datatype,
+ value=value, unit=unit, importance=importance, inheritance=inheritance)
- return super(Property, self).add_property(
- property=property, value=value, **copy_kwargs)
- def add_parent(self, parent: Union[Entity, str, int, None] = None, **kwargs) -> Entity:
+ def add_parent(self, parent: Union[Entity, str, int, None] = None, id=None, name=None, inheritance=FIX) -> Entity:
"""Add a parent Entity to this Property.
Parameters
----------
- parent : Entity, str, int, optional
- The parent entity, by default None
- **kwargs : dict, optional
- Additional keyword arguments specifying the parent Entity
- by id or name, and specifying the inheritance level. See
- :py:meth:`Entity.add_parent` for more information. Note
- that by default, `inheritance` is set to ``fix``.
+ parent : Entity or int or str or None
+ The parent entity, either specified by the Entity object
+ itself, or its id or its name. Default is None.
+ id : int
+ Integer id of the parent entity. Ignored if `parent`
+ is not None.
+ name : str
+ Name of the parent entity. Ignored if `parent is not
+ none`.
+ inheritance : str, default: FIX
+ One of ``obligatory``, ``recommended``, ``suggested``, or ``fix``. Specifies the
+ minimum importance which parent properties need to have to be inherited by this
+ entity. If no `inheritance` is given, no properties will be inherited by the child.
+ This parameter is case-insensitive.
Returns
-------
@@ -1801,13 +1834,8 @@ class Property(Entity):
--------
Entity.add_parent
"""
- copy_kwargs = kwargs.copy()
-
- if 'inheritance' not in copy_kwargs:
- # set default importance
- copy_kwargs['inheritance'] = FIX
- return super(Property, self).add_parent(parent=parent, **copy_kwargs)
+ return super(Property, self).add_parent(parent=parent, id=id, name=name, inheritance=inheritance)
def __init__(self, name=None, id=None, description=None, datatype=None,
value=None, unit=None):
@@ -1845,6 +1873,11 @@ class Property(Entity):
if server_retrieval:
tmp_prop = deepcopy(self)
+ """
+ remove role to avoid unnessecary ValueError while
+ retrieving the Entity.
+ """
+ tmp_prop.role = None
tmp_prop.retrieve()
return tmp_prop.is_reference()
@@ -1900,21 +1933,14 @@ class RecordType(Entity):
"""This class represents CaosDB's RecordType entities."""
- def add_property(self, property=None, value=None, **kwargs): # @ReservedAssignment
- copy_kwargs = kwargs.copy()
+ def add_property(self, property=None, value=None, id=None, name=None, description=None, datatype=None,
+ unit=None, importance=RECOMMENDED, inheritance=FIX): # @ReservedAssignment
- if 'importance' not in copy_kwargs:
- # set default importance
- copy_kwargs['importance'] = RECOMMENDED
-
- if 'inheritance' not in copy_kwargs:
- # set default importance
- copy_kwargs['inheritance'] = FIX
-
- return super().add_property(property=property, value=value,
- **copy_kwargs)
+ return super().add_property(
+ property=property, id=id, name=name, description=description, datatype=datatype,
+ value=value, unit=unit, importance=importance, inheritance=inheritance)
- def add_parent(self, parent=None, **kwargs):
+ def add_parent(self, parent=None, id=None, name=None, inheritance=OBLIGATORY):
"""Add a parent to this RecordType
Parameters
@@ -1922,24 +1948,30 @@ class RecordType(Entity):
parent : Entity or int or str or None, optional
The parent entity, either specified by the Entity object
itself, or its id or its name. Default is None.
- **kwargs : dict, optional
- Additional keyword arguments specifying the parent Entity by id or
- name, and specifying the inheritance level. See
- :py:meth:`Entity.add_parent` for more information. Note
- that by default, `inheritance` is set to ``obligatory``.
+ Parameters
+ ----------
+ parent : Entity or int or str or None
+ The parent entity, either specified by the Entity object
+ itself, or its id or its name. Default is None.
+ id : int
+ Integer id of the parent entity. Ignored if `parent`
+ is not None.
+ name : str
+ Name of the parent entity. Ignored if `parent is not
+ none`.
+ inheritance : str, default OBLIGATORY
+ One of ``obligatory``, ``recommended``, ``suggested``, or ``fix``. Specifies the
+ minimum importance which parent properties need to have to be inherited by this
+ entity. If no `inheritance` is given, no properties will be inherited by the child.
+ This parameter is case-insensitive.
See Also
--------
Entity.add_parent
"""
- copy_kwargs = kwargs.copy()
-
- if 'inheritance' not in copy_kwargs:
- # set default importance
- copy_kwargs['inheritance'] = OBLIGATORY
- return super().add_parent(parent=parent, **copy_kwargs)
+ return super().add_parent(parent=parent, id=id, name=name, inheritance=inheritance)
def __init__(self, name=None, id=None, description=None, datatype=None): # @ReservedAssignment
Entity.__init__(self, name=name, id=id, description=description,
@@ -1956,19 +1988,12 @@ class Record(Entity):
"""This class represents CaosDB's Record entities."""
- def add_property(self, property=None, value=None, **kwargs): # @ReservedAssignment
- copy_kwargs = kwargs.copy()
-
- if 'importance' not in copy_kwargs:
- # set default importance
- copy_kwargs['importance'] = FIX
-
- if 'inheritance' not in copy_kwargs:
- # set default importance
- copy_kwargs['inheritance'] = FIX
+ def add_property(self, property=None, value=None, id=None, name=None, description=None, datatype=None,
+ unit=None, importance=FIX, inheritance=FIX): # @ReservedAssignment
return super().add_property(
- property=property, value=value, **copy_kwargs)
+ property=property, id=id, name=name, description=description, datatype=datatype,
+ value=value, unit=unit, importance=importance, inheritance=inheritance)
def __init__(self, name=None, id=None, description=None): # @ReservedAssignment
Entity.__init__(self, name=name, id=id, description=description,
@@ -2120,19 +2145,12 @@ class File(Record):
return checksum.hexdigest()
- def add_property(self, property=None, value=None, **kwargs): # @ReservedAssignment
- copy_kwargs = kwargs.copy()
-
- if 'importance' not in copy_kwargs:
- # set default importance
- copy_kwargs['importance'] = FIX
-
- if 'inheritance' not in copy_kwargs:
- # set default importance
- copy_kwargs['inheritance'] = FIX
+ def add_property(self, property=None, id=None, name=None, description=None, datatype=None,
+ value=None, unit=None, importance=FIX, inheritance=FIX): # @ReservedAssignment
return super().add_property(
- property=property, value=value, **copy_kwargs)
+ property=property, id=id, name=name, description=description, datatype=datatype,
+ value=value, unit=unit, importance=importance, inheritance=inheritance)
class _Properties(list):
@@ -2636,15 +2654,45 @@ class _Messages(dict):
def _basic_sync(e_local, e_remote):
+ '''Copy all state from a one entity to another.
+
+ This method is used to syncronize an entity with a remote (i.e. a newly
+ retrieved) one.
+
+ Any entity state of the local one will be overriden.
+
+ Parameters
+ ----------
+ e_local : Entity
+ Destination of the copy.
+ e_local : Entity
+ Source of the copy.
+
+
+ Returns
+ -------
+ e_local : Entity
+ The syncronized entity.
+ '''
if e_local is None or e_remote is None:
return None
+ if e_local.role is None:
+ e_local.role = e_remote.role
+ elif e_remote.role is not None and not e_local.role.lower() == e_remote.role.lower():
+ raise ValueError("The resulting entity had a different role ({0}) "
+ "than the local one ({1}). This probably means, that "
+ "the entity was intialized with a wrong class "
+ "by this client or it has changed in the past and "
+ "this client did't know about it yet.".format(
+ e_remote.role, e_local.role))
+
e_local.id = e_remote.id
e_local.name = e_remote.name
e_local.description = e_remote.description
e_local.path = e_remote.path
e_local._checksum = e_remote._checksum
e_local._size = e_remote._size
- e_local.datatype = e_remote.datatype # @ReservedAssignment
+ e_local.datatype = e_remote.datatype
e_local.unit = e_remote.unit
e_local.value = e_remote.value
e_local.properties = e_remote.properties
@@ -3250,7 +3298,9 @@ class Container(list):
if is_reference(references.datatype):
# add only if it is a reference, not a property
- if isinstance(references.value, int):
+ if references.value is None:
+ continue
+ elif isinstance(references.value, int):
is_being_referenced.add(references.value)
elif is_list_datatype(references.datatype):
for list_item in references.value:
@@ -4538,7 +4588,7 @@ def _evaluate_and_add_error(parent_error, ent):
Parameters:
-----------
- parent_error : TrancactionError
+ parent_error : TransactionError
Parent error to which the new exception will be attached. This
exception will be a direct child.
ent : Entity
diff --git a/src/caosdb/configuration.py b/src/caosdb/configuration.py
index 6e8a9c6ff2083b0c30324722003fb3c08a592191..51e3749aaca3045afec9334ef987a174d5d19f26 100644
--- a/src/caosdb/configuration.py
+++ b/src/caosdb/configuration.py
@@ -21,6 +21,16 @@
#
# ** end header
#
+
+import os
+import yaml
+import warnings
+try:
+ optional_jsonschema_validate = None
+ from jsonschema import validate as optional_jsonschema_validate
+except ImportError:
+ pass
+
try:
# python2
from ConfigParser import ConfigParser
@@ -47,13 +57,45 @@ def configure(inifile):
_pycaosdbconf = None
if _pycaosdbconf is None:
_reset_config()
- return _pycaosdbconf.read(inifile)
+ read_config = _pycaosdbconf.read(inifile)
+ validate_yaml_schema(config_to_yaml(_pycaosdbconf))
+ return read_config
def get_config():
+ global _pycaosdbconf
return _pycaosdbconf
+def config_to_yaml(config):
+ valobj = {}
+ for s in config.sections():
+ valobj[s] = {}
+ for key, value in config[s].items():
+ # TODO: Can the type be inferred from the config object?
+ if key in ["timeout", "debug"]:
+ valobj[s][key] = int(value)
+ elif key in ["ssl_insecure"]:
+ valobj[s][key] = bool(value)
+ else:
+ valobj[s][key] = value
+
+ return valobj
+
+
+def validate_yaml_schema(valobj):
+ # TODO: Re-enable warning once the schema has been extended to also cover
+ # SSS pycaosdb.inis and integration tests.
+ if optional_jsonschema_validate:
+ with open(os.path.join(os.path.dirname(__file__), "schema-pycaosdb-ini.yml")) as f:
+ schema = yaml.load(f, Loader=yaml.SafeLoader)
+ optional_jsonschema_validate(instance=valobj, schema=schema["schema-pycaosdb-ini"])
+ # else:
+ # warnings.warn("""
+ # Warning: The validation could not be performed because `jsonschema` is not installed.
+ # """)
+
+
def _read_config_files():
"""Function to read config files from different paths. Checks for path in $PYCAOSDBINI or home directory (.pycaosdb.ini) and in the current working directory (pycaosdb.ini).
diff --git a/src/caosdb/connection/authentication/interface.py b/src/caosdb/connection/authentication/interface.py
index a364aeb564ee929d995b2f8098bd21e30e9733ab..f2cc5001cf8fa0f6d61ec65346f6a200ba0dfcd8 100644
--- a/src/caosdb/connection/authentication/interface.py
+++ b/src/caosdb/connection/authentication/interface.py
@@ -36,7 +36,7 @@ from caosdb.exceptions import LoginFailedError
# meta class compatible with Python 2 *and* 3:
ABC = ABCMeta('ABC', (object, ), {'__slots__': ()})
-_LOGGER = logging.getLogger("authentication")
+_LOGGER = logging.getLogger(__name__)
class AbstractAuthenticator(ABC):
diff --git a/src/caosdb/connection/connection.py b/src/caosdb/connection/connection.py
index 9e273a56778737033fda9f342f967f56946b501b..6c3946ee639872b0edb0b5b3c30a808cf8c028d4 100644
--- a/src/caosdb/connection/connection.py
+++ b/src/caosdb/connection/connection.py
@@ -60,7 +60,7 @@ except ImportError:
# pylint: disable=missing-docstring
-_LOGGER = logging.getLogger("connection")
+_LOGGER = logging.getLogger(__name__)
class _WrappedHTTPResponse(CaosDBHTTPResponse):
diff --git a/src/caosdb/schema-pycaosdb-ini.yml b/src/caosdb/schema-pycaosdb-ini.yml
new file mode 100644
index 0000000000000000000000000000000000000000..fd49c2ffa63e7bc334f8177ff79575f46718f38c
--- /dev/null
+++ b/src/caosdb/schema-pycaosdb-ini.yml
@@ -0,0 +1,87 @@
+schema-pycaosdb-ini:
+ type: object
+ additionalProperties: false
+ properties:
+ Container:
+ additionalProperties: false
+ properties:
+ debug:
+ default: 0
+ type: integer
+ enum: [0, 1, 2]
+ Connection:
+ description: Settings for the connection to the CaosDB server
+ additionalProperties: false
+ properties:
+ url:
+ description: URL of the CaosDB server
+ type: string
+ pattern: https://[-a-zA-Z0-9\.]+(:[0-9]+)?(/)?
+ examples: [https://demo.indiscale.com/, https://localhost:10443/]
+ username:
+ type: string
+ description: User name used for authentication with the server
+ examples: [admin]
+ password_method:
+ description: The password input method defines how the password is supplied that is used for authentication with the server.
+ type: string
+ default: input
+ enum: [input, plain, pass, keyring]
+ password_identifier:
+ type: string
+ password:
+ type: string
+ examples: [caosdb]
+ auth_token:
+ type: string
+ description: Using an authentication token to connect with the server. This setting is not recommended for users.
+ cacert:
+ type: string
+ description: If the server's SSL certificate cannot be validated by your installed certificates (default or installed by your admins), you may also need to supply the matching key (pem file)
+ examples: [/path/to/caosdb.ca.pem]
+ ssl_insecure:
+ description: If this option is set, the SSL certificate of the server will not be validated. This has the potential of a man-in-the-middle attack. Use with care!
+ type: boolean
+ default: false
+ ssl_version:
+ description: You may define the ssl version to be used. It has to be the name of the corresponding attribute in the Python ssl module.
+ examples: [PROTOCOL_TLS]
+ debug:
+ default: 0
+ type: integer
+ enum: [0, 1, 2]
+ description: The debug key allows control the verbosity. Set it to 1 or 2 in case you want to see debugging output or if you want to learn more about the internals of the protocol. 0 disables debugging output.
+ socket_proxy:
+ examples: [localhost:12345]
+ type: string
+ description: You can define a socket proxy to be used. This is for the case that the server sits behind a firewall which is being tunnelled with a socket proxy (SOCKS4 or SOCKS5) (e.g. via ssh's -D option or a dedicated proxy server).
+ implementation:
+ description: This option is used internally and for testing. Do not override.
+ examples: [_DefaultCaosDBServerConnection]
+ timeout:
+ type: integer
+ allOf:
+ - if:
+ properties:
+ password_method:
+ const: input
+ then:
+ required: [url]
+ - if:
+ properties:
+ password_method:
+ const: plain
+ then:
+ required: [url, username, password]
+ - if:
+ properties:
+ password_method:
+ const: pass
+ then:
+ required: [url, username, password_identifier]
+ - if:
+ properties:
+ password_method:
+ const: keyring
+ then:
+ required: [url, username]
diff --git a/src/caosdb/utils/caosdb_admin.py b/src/caosdb/utils/caosdb_admin.py
index 392d8bea2ce3d9a868c32854800ca6cb78f021ba..9fb94f57683036f5432a40198cc4ae98893665fb 100755
--- a/src/caosdb/utils/caosdb_admin.py
+++ b/src/caosdb/utils/caosdb_admin.py
@@ -116,28 +116,43 @@ def _promt_for_pw():
def do_create_user(args):
- password = None
+ password = args.user_password
if args.ask_password is True:
password = _promt_for_pw()
try:
admin._insert_user(name=args.user_name,
email=args.user_email, password=password)
+
+ if args.activate_user:
+ do_activate_user(args)
except HTTPClientError as e:
print(e.msg)
def do_activate_user(args):
- admin._update_user(name=args.user_name, status="ACTIVE")
+ try:
+ admin._update_user(name=args.user_name, status="ACTIVE")
+ except HTTPClientError as e:
+ print(e.msg)
def do_deactivate_user(args):
- admin._update_user(name=args.user_name, status="INACTIVE")
+ try:
+ admin._update_user(name=args.user_name, status="INACTIVE")
+ except HTTPClientError as e:
+ print(e.msg)
def do_set_user_password(args):
- password = _promt_for_pw()
- admin._update_user(name=args.user_name, password=password)
+ if args.user_password is None:
+ password = _promt_for_pw()
+ else:
+ password = args.user_password
+ try:
+ admin._update_user(name=args.user_name, password=password)
+ except HTTPClientError as e:
+ print(e.msg)
def do_add_user_roles(args):
@@ -301,10 +316,21 @@ USAGE
# users (CRUD)
subparser = subparsers.add_parser(
"create_user",
- help="Create a new user in caosdb's internal user database.")
+ help="Create a new user in caosdb's internal user database. You need "
+ " to activate the user before use.")
subparser.set_defaults(call=do_create_user)
- subparser.add_argument("-a", "--ask-password",
- help="Prompt for a password.", action="store_true")
+ mg = subparser.add_mutually_exclusive_group()
+ mg.add_argument("-a", "--ask-password",
+ help="Prompt for a password.", action="store_true")
+ mg.add_argument(
+ "--password",
+ dest="user_password",
+ default=None,
+ help="Alternative way to provide the new user's password. Please "
+ "consider to use the more secure, interactive way (-a option).")
+ subparser.add_argument("-c", "--activate-user",
+ help="Activate the user after creation.",
+ action="store_true")
subparser.add_argument(
metavar='USERNAME',
dest="user_name",
@@ -331,12 +357,20 @@ USAGE
subparser = subparsers.add_parser(
"set_user_password",
- help="Set a new password for a user. The password is not to be given on the command line for security reasons. You will be prompted for the password.")
+ help="Set a new password for a user. "
+ "By default, you will be prompted for the password.")
subparser.set_defaults(call=do_set_user_password)
subparser.add_argument(
metavar='USERNAME',
dest="user_name",
help="The name of the user who's password is to be set.")
+ subparser.add_argument(
+ metavar='PASSWORD',
+ nargs="?",
+ dest="user_password",
+ default=None,
+ help="Alternative way to provide the user's new password. "
+ "The more secure (and default way) is to provide it interactively.")
subparser = subparsers.add_parser(
"set_user_entity",
diff --git a/src/caosdb/utils/plantuml.py b/src/caosdb/utils/plantuml.py
index 75b96ee3aa28ba916adc69e418de398abfe23356..be34b2604f3682bb71b48bbd73e00fe854b3af51 100644
--- a/src/caosdb/utils/plantuml.py
+++ b/src/caosdb/utils/plantuml.py
@@ -36,11 +36,24 @@ plantuml FILENAME.pu -> FILENAME.png
import os
import caosdb as db
+from caosdb.common.datatype import is_reference, get_referenced_recordtype
REFERENCE = "REFERENCE"
def get_description(description_str):
+ """Extract and format a description string from a record type or property.
+
+ Parameters
+ ----------
+ description_str : str
+ The description string that is going to be formatted.
+
+ Returns
+ -------
+ str
+ The reformatted description ending in a line break.
+ """
words = description_str.split()
lines = []
lines.append("")
@@ -211,15 +224,76 @@ package \"The property P references an instance of D\" <<Rectangle>> {
return result
+def retrieve_substructure(start_record_types, depth, result_id_set=None, result_container=None, cleanup=True):
+ """Recursively retrieves CaosDB record types and properties, starting
+ from given initial types up to a specific depth.
+
+ Parameters
+ ----------
+ start_record_types : Iterable[db.Entity]
+ Iterable with the entities to be displayed. Starting from these
+ entities more entities will be retrieved.
+ depth : int
+ The maximum depth up to which to retriev sub entities.
+ result_id_set : set[int]
+ Used by recursion. Filled with already visited ids.
+ result_container : db.Container
+ Used by recursion. Filled with already visited entities.
+ cleanup : bool
+ Used by recursion. If True return the resulting result_container.
+ Don't return anything otherwise.
+
+ Returns
+ -------
+ db.Container
+ A container containing all the retrieved entites or None if cleanup is False.
+ """
+ # Initialize the id set and result container for level zero recursion depth:
+ if result_id_set is None:
+ result_id_set = set()
+ if result_container is None:
+ result_container = db.Container()
+
+ for entity in start_record_types:
+ entity.retrieve()
+ if entity.id not in result_id_set:
+ result_container.append(entity)
+ result_id_set.add(entity.id)
+ for prop in entity.properties:
+ if is_reference(prop.datatype) and prop.datatype != db.FILE and depth > 0:
+ rt = db.RecordType(name=get_referenced_recordtype(prop.datatype)).retrieve()
+ retrieve_substructure([rt], depth-1, result_id_set, result_container, False)
+
+ if prop.id not in result_id_set:
+ result_container.append(prop)
+ result_id_set.add(prop.id)
+
+ for parent in entity.parents:
+ rt = db.RecordType(id=parent.id).retrieve()
+ if parent.id not in result_id_set:
+ result_container.append(rt)
+ result_id_set.add(parent.id)
+ if depth > 0:
+ retrieve_substructure([rt], depth-1, result_id_set, result_container, False)
+
+ if cleanup:
+ return result_container
+ return None
+
+
def to_graphics(recordtypes, filename):
- """ calls recordtypes_to_plantuml_string(), saves result to file and
+ """Calls recordtypes_to_plantuml_string(), saves result to file and
creates an svg image
- plantuml needs to be installed
- @params:
- recordtypes: itrable with the record types to be displayed
- filname: filename of the image (e.g. data_structure; data_structure.pu and
- data_structure.svg will be created.
+ plantuml needs to be installed.
+
+ Parameters
+ ----------
+ recordtypes : Iterable[db.Entity]
+ Iterable with the entities to be displayed.
+ filename : str
+ filename of the image without the extension(e.g. data_structure;
+ data_structure.pu and data_structure.svg will be created.)
"""
pu = recordtypes_to_plantuml_string(recordtypes)
diff --git a/src/caosdb/yamlapi.py b/src/caosdb/yamlapi.py
index 9a69a5276804727084af65c6568b22833e8be596..69928af1568bf2150288844d74d606e39d598d0b 100644
--- a/src/caosdb/yamlapi.py
+++ b/src/caosdb/yamlapi.py
@@ -22,7 +22,7 @@
# ** end header
#
-"""YAML interface for the database (caosdb)"""
+"""!!! Deprecated !!! YAML interface for the database (caosdb)"""
import yaml
from lxml import etree
@@ -31,9 +31,14 @@ import re
import caosdb
import caosdb.common.utils as utils
from caosdb.connection.connection import get_connection
+import warnings
def append_sublist(v, newel, def_entity_type):
+ warnings.warn("""
+ This function is deprecated and will be removed with the next release.
+ Please use caosdb-advanced-user-tools/models/data_model.py for a
+ similar functionality.""", DeprecationWarning)
if v is None:
return
for i in v:
@@ -46,6 +51,10 @@ def append_sublist(v, newel, def_entity_type):
def kv_to_xml(k, v):
+ warnings.warn("""
+ This function is deprecated and will be removed with the next release.
+ Please use caosdb-advanced-user-tools/models/data_model.py for a
+ similar functionality.""", DeprecationWarning)
newel = Element(k)
# code.interact(local=locals())
if isinstance(v, list): # Top level loop
@@ -69,10 +78,18 @@ def dict_to_xml(d):
d: The dictionary (possibly loaded from yaml)
to convert to caosdb-xml.
"""
+ warnings.warn("""
+ This function is deprecated and will be removed with the next release.
+ Please use caosdb-advanced-user-tools/models/data_model.py for a
+ similar functionality.""", DeprecationWarning)
return kv_to_xml("Entities", d)
def yaml_to_xml(yamlstr):
+ warnings.warn("""
+ This function is deprecated and will be removed with the next release.
+ Please use caosdb-advanced-user-tools/models/data_model.py for a
+ similar functionality.""", DeprecationWarning)
"""Load a yaml document from yamlstr and converts it to XML.
Parameters
@@ -81,11 +98,15 @@ def yaml_to_xml(yamlstr):
The string to load the yaml document from.
"""
- return dict_to_xml(yaml.load(yamlstr))
+ return dict_to_xml(yaml.safe_load(yamlstr))
def process(text):
"""Do some replacements on the original file to obtain valid yaml."""
+ warnings.warn("""
+ This function is deprecated and will be removed with the next release.
+ Please use caosdb-advanced-user-tools/models/data_model.py for a
+ similar functionality.""", DeprecationWarning)
processed = re.sub(
"^(\\s*)-\\s*\\{?(.*)\\}?\\s*$",
"\\1- {\\2}",
@@ -98,6 +119,10 @@ def process(text):
def yaml_file_to_xml(yamlfilename):
+ warnings.warn("""
+ This function is deprecated and will be removed with the next release.
+ Please use caosdb-advanced-user-tools/models/data_model.py for a
+ similar functionality.""", DeprecationWarning)
with open(yamlfilename, "r") as f:
return yaml_to_xml(process(f.read()))
@@ -108,6 +133,10 @@ def insert_yaml_file(yamlfilename, simulate=False):
Set 'simulate' to True if you don't actually want to insert the xml,
but only receive what would be sent.
"""
+ warnings.warn("""
+ This function is deprecated and will be removed with the next release.
+ Please use caosdb-advanced-user-tools/models/data_model.py for a
+ similar functionality.""", DeprecationWarning)
con = get_connection()
prs = etree.XMLParser(remove_blank_text=True)
sent_xml = etree.tostring(
diff --git a/src/doc/tutorials/Data-Insertion.rst b/src/doc/tutorials/Data-Insertion.rst
index 22fb9461d6916003b2dad496ff3487df335c8dcc..84223db3615688a1fb900eefa2b86115b266b307 100644
--- a/src/doc/tutorials/Data-Insertion.rst
+++ b/src/doc/tutorials/Data-Insertion.rst
@@ -17,8 +17,10 @@ Property called “a” of datatype double. This is very easy in pylib:
a = db.Property(name="a", datatype=db.DOUBLE)
-There are a few basic datatypes: db.INTEGER, db.TEXT. See `data
-type <Specification/Datatype>`__ for a full list.
+There are a few basic datatypes like db.INTEGER, db.DOUBLE, or db.TEXT. See the
+`data types
+<https://docs.indiscale.com/caosdb-server/specification/Datatype.html>`_ for a
+full list.
We can create our own small data model for e.g. a simulation by adding
two more Properties and a RecordType:
@@ -49,11 +51,11 @@ resolution, but we'll omit this for the sake of brevity for now.
.. code:: python
rt = db.RecordType(name="2D_BarkleySimulation",
- description="Spatially extended Barkley simulation")
+ description="Spatially extended Barkley simulation")
# inherit all properties from the BarkleySimulation RecordType
rt.add_parent(name="BarkleySimulation", inheritance="all")
rt.insert()
-
+
print(rt.get_property(name="epsilon").importance) ### rt has a "epsilon" property with the same importance as "BarkleySimulation"
The parameter ``inheritance=(obligatory|recommended|fix|all|none)`` of
@@ -135,15 +137,37 @@ Useful is also, that you can insert directly tabular data.
.. code:: python
- from caosadvancedtools.table_converter import from_tsv
-
- recs = from_tsv("test.csv", "Experiment")
- print(recs)
- recs.insert()
+ from caosadvancedtools.table_converter import from_tsv
+
+ recs = from_tsv("test.csv", "Experiment")
+ print(recs)
+ recs.insert()
With this example file
`test.csv <uploads/4f2c8756a26a3984c0af09d206d583e5/test.csv>`__.
+List Properties
+---------------
+
+As you may already know, properties can also have list values instead of scalar
+values. They can be accessed, set, and updated as you would expect from any
+list-valued attribute in Python, as the following example illustrates.
+
+.. code:: python
+
+ import caosdb as db
+ db.Property(name="TestList", datatype=db.LIST(db.DOUBLE)).insert()
+ db.RecordType(name="TestType").add_property(name="TestList").insert()
+ db.Record(name="TestRec").add_parent("TestType").add_property(
+ name="TestList", value=[1,2,3]).insert()
+ retrieved = db.Record(name="TestRec").retrieve()
+ retrieved.get_property("TestList").value += [4,5]
+ retrieved.update()
+
+ # Check update
+ retrieved = db.Record(name="TestRec").retrieve()
+ print(retrieved.get_property("TestList").value)
+
File Update
-----------
diff --git a/tox.ini b/tox.ini
index 94c2dc8affb280d3e7f6cff4536636432c9f7749..22c89f765c612ff78572ee2cab20dfab2e740e84 100644
--- a/tox.ini
+++ b/tox.ini
@@ -7,4 +7,5 @@ deps = .
nose
pytest
pytest-cov
+ jsonschema==4.0.1
commands=py.test --cov=caosdb -vv {posargs}
diff --git a/unittests/broken_configs/pycaosdb1.ini b/unittests/broken_configs/pycaosdb1.ini
new file mode 100644
index 0000000000000000000000000000000000000000..71180286881399c35e251bba89a54a345cd948ac
--- /dev/null
+++ b/unittests/broken_configs/pycaosdb1.ini
@@ -0,0 +1,5 @@
+[Connection]
+cacert=/very/long/path/to/self/signed/pem/file/caosdb.ca.pem
+url=https://hostname:8833/playground
+username=username
+password_method=pass
diff --git a/unittests/broken_configs/pycaosdb2.ini b/unittests/broken_configs/pycaosdb2.ini
new file mode 100644
index 0000000000000000000000000000000000000000..6bdf58ea812b83a622e647b2a18b01bb1a1e3099
--- /dev/null
+++ b/unittests/broken_configs/pycaosdb2.ini
@@ -0,0 +1,9 @@
+[Connection]
+url=https://0.0.0.0/
+username=username
+password_identifier=SECTION/SUBSECTION/identifier
+password_method=pass
+cacert=/etc/ssl/cert.pem
+ssl_insecure=true
+timeout=10000
+debug=9
diff --git a/unittests/broken_configs/pycaosdb3.ini b/unittests/broken_configs/pycaosdb3.ini
new file mode 100644
index 0000000000000000000000000000000000000000..62d1fed9497c0c258c13aa2ed8fecb23a3006849
--- /dev/null
+++ b/unittests/broken_configs/pycaosdb3.ini
@@ -0,0 +1,12 @@
+[connection]
+ssl_insecure=true
+url=https://localhost:10443/
+password=caosdb
+username=admin
+password_method=plain
+
+timeout=10000
+debug=0
+
+[Container]
+debug=0
diff --git a/unittests/broken_configs/pycaosdb4.ini b/unittests/broken_configs/pycaosdb4.ini
new file mode 100644
index 0000000000000000000000000000000000000000..e96604ac453ff2be4e2b419aa9ccbbf3598fa231
--- /dev/null
+++ b/unittests/broken_configs/pycaosdb4.ini
@@ -0,0 +1,10 @@
+[Connection]
+ssl_insecure=true
+url=https://localhost:10443/
+password=caosdb
+username=admin
+password_method=plain
+
+timeout=10000
+debug=0
+key=bla
\ No newline at end of file
diff --git a/unittests/test_add_property.py b/unittests/test_add_property.py
index 5bae6c219732f0170f5c351eae58148c9d3d065a..0d3183b4c0ca5517ecea68d0e49bbf335bb2a13e 100644
--- a/unittests/test_add_property.py
+++ b/unittests/test_add_property.py
@@ -264,3 +264,33 @@ def test_add_list_of_entitities():
for e in rec.get_property("listOfEntities").value:
assert i == e.id
i += 1
+
+
+def test_add_property_with_wrong_role():
+ entity = db.Entity()
+
+ r = db.Record()
+ rt = db.RecordType()
+ p = db.Property()
+ f = db.File()
+ e = db.Entity()
+
+ entity.add_property(rt)
+ entity.add_property(p)
+ entity.add_property(e)
+
+ with raises(ValueError) as cm:
+ entity.add_property(r)
+ assert cm.value.args[0] == ("The property parameter is a Record. This is "
+ "very unusual and probably not what you want. "
+ "Otherwise, construct a property from a "
+ "Record using the Property class and add that "
+ "to this entity.")
+
+ with raises(ValueError) as cm:
+ entity.add_property(f)
+ assert cm.value.args[0] == ("The property parameter is a File. This is "
+ "very unusual and probably not what you want. "
+ "Otherwise, construct a property from a File "
+ "using the Property class and add that to "
+ "this entity.")
diff --git a/unittests/test_authentication_plain.py b/unittests/test_authentication_plain.py
index 10cbc418df8bd81c81568a4df0cf1e8a4ac498f8..146b59889c71c86ea77fb4ae962118cdda1afb06 100644
--- a/unittests/test_authentication_plain.py
+++ b/unittests/test_authentication_plain.py
@@ -65,4 +65,6 @@ def test_subclass_configure():
def test_plain_has_logger():
p = PlainTextCredentialsProvider()
assert hasattr(p, "logger")
- assert p.logger.name == "authentication"
+ assert "authentication" in p.logger.name
+ assert "connection" in p.logger.name
+ assert "caosdb" in p.logger.name
diff --git a/unittests/test_configs/pycaosdb-indiscale-demo.ini b/unittests/test_configs/pycaosdb-indiscale-demo.ini
new file mode 100644
index 0000000000000000000000000000000000000000..9010343467f78c7c9c3e25ea3a57520deac18c8e
--- /dev/null
+++ b/unittests/test_configs/pycaosdb-indiscale-demo.ini
@@ -0,0 +1,12 @@
+[Connection]
+url=https://demo.indiscale.com/
+username=admin
+password=caosdb
+password_method=plain
+cacert=/etc/ssl/cert.pem
+
+timeout=10000
+debug=0
+
+[Container]
+debug=0
\ No newline at end of file
diff --git a/unittests/test_configs/pycaosdb1.ini b/unittests/test_configs/pycaosdb1.ini
new file mode 100644
index 0000000000000000000000000000000000000000..dcfa7c21fac735d81ab92b33f0abd31df25fc1ad
--- /dev/null
+++ b/unittests/test_configs/pycaosdb1.ini
@@ -0,0 +1,6 @@
+[Connection]
+cacert=/very/long/path/to/self/signed/pem/file/caosdb.ca.pem
+url=https://hostname:8833/playground
+password_identifier=SECTION/caosdb
+username=username
+password_method=pass
diff --git a/unittests/test_configs/pycaosdb2.ini b/unittests/test_configs/pycaosdb2.ini
new file mode 100644
index 0000000000000000000000000000000000000000..e5493bde3be0f84f38e427edb4d42fba9c75482d
--- /dev/null
+++ b/unittests/test_configs/pycaosdb2.ini
@@ -0,0 +1,9 @@
+[Connection]
+url=https://0.0.0.0/
+username=username
+password_identifier=SECTION/SUBSECTION/identifier
+password_method=pass
+cacert=/etc/ssl/cert.pem
+ssl_insecure=true
+timeout=10000
+debug=0
diff --git a/unittests/test_configs/pycaosdb3.ini b/unittests/test_configs/pycaosdb3.ini
new file mode 100644
index 0000000000000000000000000000000000000000..6c4934039c99855566f38c69f7511d774f81efbd
--- /dev/null
+++ b/unittests/test_configs/pycaosdb3.ini
@@ -0,0 +1,12 @@
+[Connection]
+ssl_insecure=true
+url=https://localhost:10443/
+password=caosdb
+username=admin
+password_method=plain
+
+timeout=10000
+debug=0
+
+[Container]
+debug=0
diff --git a/unittests/test_entity.py b/unittests/test_entity.py
index e98dfbef5b6b5e5f691e8aecc2fa7d4a86991452..1e88702ac016d7dcfdf00919dd0f93b5d3345e00 100644
--- a/unittests/test_entity.py
+++ b/unittests/test_entity.py
@@ -24,6 +24,7 @@
"""Tests for the Entity class."""
# pylint: disable=missing-docstring
import unittest
+from lxml import etree
from caosdb import (INTEGER, Entity, Property, Record, RecordType,
configure_connection)
@@ -42,17 +43,54 @@ class TestEntity(unittest.TestCase):
def test_instance_variables(self):
entity = Entity()
self.assertTrue(hasattr(entity, "role"))
+ self.assertIsNone(entity.role)
self.assertTrue(hasattr(entity, "id"))
self.assertTrue(hasattr(entity, "name"))
self.assertTrue(hasattr(entity, "description"))
self.assertTrue(hasattr(entity, "parents"))
self.assertTrue(hasattr(entity, "properties"))
- def test_role(self):
+ def test_entity_role_1(self):
entity = Entity(role="TestRole")
self.assertEqual(entity.role, "TestRole")
entity.role = "TestRole2"
self.assertEqual(entity.role, "TestRole2")
- def test_instanciation(self):
+ def test_entity_role_2(self):
+ entity = Entity()
+
+ self.assertIsNone(entity.role)
+ self.assertEqual(entity.to_xml().tag, "Entity")
+
+ entity.role = "Record"
+ self.assertEqual(entity.role, "Record")
+ self.assertEqual(entity.to_xml().tag, "Record")
+
+ def test_recordtype_role(self):
+ entity = RecordType()
+
+ self.assertEqual(entity.role, "RecordType")
+ self.assertEqual(entity.to_xml().tag, "RecordType")
+
+ def test_property_role(self):
+ entity = Property()
+
+ self.assertEqual(entity.role, "Property")
+ self.assertEqual(entity.to_xml().tag, "Property")
+
+ def test_instantiation(self):
self.assertRaises(Exception, Entity())
+
+ def test_parse_role(self):
+ """During parsing, the role of an entity is set explicitely. All other
+ classes use the class name as a "natural" value for the role property.
+ """
+ parser = etree.XMLParser(remove_comments=True)
+ entity = Entity._from_xml(Entity(),
+ etree.parse("unittests/test_record.xml",
+ parser).getroot())
+
+ self.assertEqual(entity.role, "Record")
+ # test whether the __role property of this object has explicitely been
+ # set.
+ self.assertEqual(getattr(entity, "_Entity__role"), "Record")
diff --git a/unittests/test_schema.py b/unittests/test_schema.py
new file mode 100644
index 0000000000000000000000000000000000000000..387518c9a30aac45facc24d77ea8c8532a4a8b16
--- /dev/null
+++ b/unittests/test_schema.py
@@ -0,0 +1,26 @@
+#!/bin/python
+# Test configuration schema
+# A. Schlemmer, 01/2021
+
+from jsonschema.exceptions import ValidationError
+from pytest import raises
+from glob import glob
+import os
+from caosdb.configuration import config_to_yaml, validate_yaml_schema
+from configparser import ConfigParser
+
+
+def test_config_files():
+ for fn in glob(os.path.join(os.path.dirname(__file__), "test_configs", "*.ini")):
+ c = ConfigParser()
+ c.read(fn)
+ validate_yaml_schema(config_to_yaml(c))
+
+
+def test_broken_config_files():
+ for fn in glob(os.path.join(os.path.dirname(__file__), "broken_configs", "*.ini")):
+ print(fn)
+ with raises(ValidationError):
+ c = ConfigParser()
+ c.read(fn)
+ validate_yaml_schema(config_to_yaml(c))
diff --git a/unittests/test_yamlapi.py b/unittests/test_yamlapi.py
new file mode 100644
index 0000000000000000000000000000000000000000..4af8a53b1172be0ddac71444724dc4b69119d998
--- /dev/null
+++ b/unittests/test_yamlapi.py
@@ -0,0 +1,88 @@
+# -*- coding: utf-8 -*-
+#
+# ** header v3.0
+# This file is a part of the CaosDB Project.
+#
+# Copyright (C) 2021 IndiScale GmbH <info@indiscale.com>
+# Copyright (C) 2021 Alexander Kreft <akreft@trineo.org>
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Affero General Public License as
+# published by the Free Software Foundation, either version 3 of the
+# License, or (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU Affero General Public License for more details.
+#
+# You should have received a copy of the GNU Affero General Public License
+# along with this program. If not, see <https://www.gnu.org/licenses/>.
+#
+# ** end header
+#
+
+import os
+import warnings
+import tempfile
+from caosdb.yamlapi import (append_sublist, kv_to_xml,
+ dict_to_xml, yaml_to_xml,
+ process, yaml_file_to_xml)
+
+with warnings.catch_warnings(record=True) as w:
+ # Cause all warnings to always be triggered.
+ warnings.simplefilter("always")
+
+ append_sublist(None, None, None)
+
+ assert issubclass(w[-1].category, DeprecationWarning)
+ assert "This function is deprecated" in str(w[-1].message)
+
+with warnings.catch_warnings(record=True) as w:
+ # Cause all warnings to always be triggered.
+ warnings.simplefilter("always")
+
+ kv_to_xml("None", "None")
+ assert len(w) == 1
+ assert issubclass(w[-1].category, DeprecationWarning)
+ assert "This function is deprecated" in str(w[-1].message)
+
+with warnings.catch_warnings(record=True) as w:
+ # Cause all warnings to always be triggered.
+ warnings.simplefilter("always")
+
+ dict_to_xml(None)
+
+ assert issubclass(w[-1].category, DeprecationWarning)
+ assert "This function is deprecated" in str(w[-1].message)
+
+with warnings.catch_warnings(record=True) as w:
+ # Cause all warnings to always be triggered.
+ warnings.simplefilter("always")
+
+ yaml_to_xml("None")
+
+ assert issubclass(w[-1].category, DeprecationWarning)
+ assert "This function is deprecated" in str(w[-1].message)
+
+with warnings.catch_warnings(record=True) as w:
+ # Cause all warnings to always be triggered.
+ warnings.simplefilter("always")
+
+ process("None")
+
+ assert issubclass(w[-1].category, DeprecationWarning)
+ assert "This function is deprecated" in str(w[-1].message)
+
+with warnings.catch_warnings(record=True) as w:
+ # Cause all warnings to always be triggered.
+ warnings.simplefilter("always")
+
+ with tempfile.TemporaryDirectory() as tmpdirname:
+ tmpfile = os.path.join(tmpdirname, 'yamlfile')
+ with open(tmpfile, 'w') as tf:
+ tf.write("")
+ yaml_file_to_xml(tmpfile)
+
+ assert issubclass(w[-1].category, DeprecationWarning)
+ assert "This function is deprecated" in str(w[-1].message)