diff --git a/CHANGELOG.md b/CHANGELOG.md
index 503808ef36c690cd4e0ec3a31e7c2f4e21fbee97..1bf1187134b589f6f115900a7c5ac952cac6265b 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -5,7 +5,7 @@ All notable changes to this project will be documented in this file.
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
-## [Unreleased] ##
+## [Unreleased]
### Added ###
@@ -24,14 +24,43 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
### Fixed ###
+* [#75](https://gitlab.indiscale.com/caosdb/src/caosdb-pylib/-/issues/75), [#103](https://gitlab.indiscale.com/caosdb/src/caosdb-pylib/-/issues/103) Fixed JSON schema to allow more sections, and correct requirements for
+ password method.
+
+### Security ###
+
+### Documentation ###
+
+
+## [0.7.2] - 2022-03-25 ##
+(Timm Fitschen)
+
+### Added ###
+
+### Changed ###
+
+### Deprecated ###
+
+* In module `caosdb.apiutils`:
+ * `CaosDBPythonEntity` class
+ * `convert_to_entity` function
+ * `convert_to_python_object` function
+
+### Removed ###
+
+### Fixed ###
+
* [caosdb-pylib#106](https://gitlab.indiscale.com/caosdb/src/caosdb-pylib/-/issues/106)
- Parsing Error in class caosdb.common.models.ACL
+ Parsing Error in class caosdb.common.models.ACL. This may lead to the
+ unintentional revocation of permissions for some users or roles during
+ updates. However, no additional permissions are being granted.
### Security ###
### Documentation ###
## [0.7.1] - 2022-03-11 ##
+(Daniel Hornung)
### Documentation ###
diff --git a/DEPENDENCIES.md b/DEPENDENCIES.md
index 04e783a7fa31b1d5c3a600a1009c8f040db1620d..d6f91cb9e15982bd1b7b0366f94bd9b37286f85a 100644
--- a/DEPENDENCIES.md
+++ b/DEPENDENCIES.md
@@ -1,6 +1,5 @@
-* caosdb-server == 0.3
-* Python >= 3.5
+* caosdb-server >= 0.7.2
+* Python >= 3.6
* pip >= 20.0.2
-
-Any other dependencies are being installed via pip
+Any other dependencies are defined in the setup.py and are being installed via pip
diff --git a/README_SETUP.md b/README_SETUP.md
index e58f934ceba176e4b5ba42239565f8e3bd48171a..374ea306100aff4cb4f6cbb37f4e9f1eee6f7abc 100644
--- a/README_SETUP.md
+++ b/README_SETUP.md
@@ -82,60 +82,8 @@ pip3 install --user .[jsonschema]
## Configuration ##
-The configuration is done using `ini` configuration files.
-PyCaosDB tries to read from the inifile specified in the environment variable `PYCAOSDBINI` or
-alternatively in `~/.pycaosdb.ini` upon import. After that, the ini file `pycaosdb.ini` in the
-current working directory will be read additionally, if it exists.
-
-Here, we will look at the most common configuration options. For a full and
-comprehensive description please check out
-[pycaosdb.ini file](https://gitlab.com/caosdb/caosdb-pylib/-/blob/main/examples/pycaosdb.ini)
-You can download this file and use it as a starting point.
-
-
-Typically, you need to change at least the `url` and `username` fields as required.
-(Ask your CaosDB administrator or IT crowd if
-you do not know what to put there, but for the demo instances https://demo.indiscale.com, `username=admin`
-and `password=caosdb` should work).
-
-### Authentication ###
-
-The default configuration (that your are asked for your password when ever a connection is created
-can be changed by setting `password_method`:
-
-* with `password_method=input` password (and possibly user) will be queried on demand (**default**)
-* use the password manager [pass](https://www.passwordstore.org) by using `pass` as value, see also the [ArchWiki
- entry](https://wiki.archlinux.org/index.php/Pass#Basic_usage). This also requires `password_identifier` which refers to the identifier within pass
- for the desired password.
-* install the python package [keyring](https://pypi.org/project/keyring), to use the system keyring/wallet (macOS, GNOME, KDE,
- Windows). The password will be queried on first usage.
-* with `password_method=plain` (**strongly discouraged**)
-
-The following illustrates the recommended options:
-
-```ini
-[Connection]
-# using "pass" password manager
-#password_method=pass
-#password_identifier=...
-
-# using the system keyring/wallet (macOS, GNOME, KDE, Windows)
-#password_method=keyring
-```
-
-### SSL Certificate ###
-In some cases (especially if you are testing CaosDB) you might need to supply
-an SSL certificate to allow SSL encryption.
-
-```ini
-[Connection]
-cacert=/path/to/caosdb.ca.pem
-```
-
-### Further Settings ###
-As mentioned above, a complete list of options can be found in the
-[pycaosdb.ini file](https://gitlab.com/caosdb/caosdb-pylib/-/blob/main/examples/pycaosdb.ini) in
-the examples folder of the source code.
+The configuration is done using `ini` configuration files. The content of these configuration files
+is described in detail in the [configuration section of the documentation](https://docs.indiscale.com/caosdb-pylib/configuration.html).
## Try it out ##
diff --git a/RELEASE_GUIDELINES.md b/RELEASE_GUIDELINES.md
index e71234b8e2bc95f954ffbebdc26acf6edd8e0b2d..b4e38d643756798f0ba8b07d6eceec529cbb3054 100644
--- a/RELEASE_GUIDELINES.md
+++ b/RELEASE_GUIDELINES.md
@@ -36,8 +36,10 @@ guidelines of the CaosDB Project
9. Publish the release by executing `./release.sh` with uploads the caosdb
module to the Python Package Index [pypi.org](https://pypi.org).
-10. Merge the main branch back into the dev branch.
+10. Create a gitlab release on gitlab.indiscale.com and gitlab.com
-11. After the merge of main to dev, start a new development version by
+11. Merge the main branch back into the dev branch.
+
+12. After the merge of main to dev, start a new development version by
setting `ISRELEASED` to `False` and by increasing at least the `MICRO`
version in [setup.py](./setup.py) and preparing CHANGELOG.md.
diff --git a/setup.py b/setup.py
index cdb10b6b41b5c466187a4bc63a77f41bd04ec454..def734ac1b5b4a648df58a6904d51835229dbb86 100755
--- a/setup.py
+++ b/setup.py
@@ -47,9 +47,13 @@ from setuptools import find_packages, setup
ISRELEASED = False
MAJOR = 0
-MINOR = 7
-MICRO = 2
-PRE = "dev" # e.g. rc0, alpha.1, 0.beta-23
+MINOR = 8
+MICRO = 0
+# Do not tag as pre-release until this commit
+# https://github.com/pypa/packaging/pull/515
+# has made it into a release. Probably we should wait for pypa/packaging>=21.4
+# https://github.com/pypa/packaging/releases
+PRE = "" # "dev" # e.g. rc0, alpha.1, 0.beta-23
if PRE:
VERSION = "{}.{}.{}-{}".format(MAJOR, MINOR, MICRO, PRE)
@@ -166,12 +170,13 @@ def setup_package():
packages=find_packages('src'),
python_requires='>=3.6',
package_dir={'': 'src'},
- install_requires=['lxml>=3.6.4',
- 'PyYaml>=3.12', 'future', 'PySocks>=1.6.7'],
+ install_requires=['lxml>=4.6.3',
+ 'PyYAML>=6.0', 'future', 'PySocks>=1.6.7'],
extras_require={'keyring': ['keyring>=13.0.0'],
- 'jsonschema': ['jsonschema==4.0.1']},
+ 'jsonschema': ['jsonschema>=4.4.0']},
setup_requires=["pytest-runner>=2.0,<3dev"],
- tests_require=["pytest", "pytest-cov", "coverage>=4.4.2", "jsonschema==4.0.1"],
+ tests_require=["pytest", "pytest-cov", "coverage>=4.4.2",
+ "jsonschema>=4.4.0"],
package_data={
'caosdb': ['cert/indiscale.ca.crt', 'schema-pycaosdb-ini.yml'],
},
diff --git a/src/caosdb/configuration.py b/src/caosdb/configuration.py
index 51e3749aaca3045afec9334ef987a174d5d19f26..75827df0d00d6c82251c2c04fa47413ac2801928 100644
--- a/src/caosdb/configuration.py
+++ b/src/caosdb/configuration.py
@@ -84,23 +84,28 @@ def config_to_yaml(config):
def validate_yaml_schema(valobj):
- # TODO: Re-enable warning once the schema has been extended to also cover
- # SSS pycaosdb.inis and integration tests.
if optional_jsonschema_validate:
with open(os.path.join(os.path.dirname(__file__), "schema-pycaosdb-ini.yml")) as f:
schema = yaml.load(f, Loader=yaml.SafeLoader)
optional_jsonschema_validate(instance=valobj, schema=schema["schema-pycaosdb-ini"])
- # else:
- # warnings.warn("""
- # Warning: The validation could not be performed because `jsonschema` is not installed.
- # """)
+ else:
+ warnings.warn("""
+ Warning: The validation could not be performed because `jsonschema` is not installed.
+ """)
def _read_config_files():
- """Function to read config files from different paths. Checks for path in $PYCAOSDBINI or home directory (.pycaosdb.ini) and in the current working directory (pycaosdb.ini).
+ """Function to read config files from different paths.
+
+ Checks for path either in ``$PYCAOSDBINI`` or home directory (``.pycaosdb.ini``), and
+ additionally in the current working directory (``pycaosdb.ini``).
+
+ Returns
+ -------
+
+ ini files: list
+ The successfully parsed ini-files. Order: env_var or home directory, cwd. Used for testing the function.
- Returns:
- [list]: list with successfully parsed ini-files. Order: env_var or home directory, cwd. Used for testing the function.
"""
return_var = []
if "PYCAOSDBINI" in environ:
diff --git a/src/caosdb/schema-pycaosdb-ini.yml b/src/caosdb/schema-pycaosdb-ini.yml
index bfe8fe7c63679507bba795bb45d7afa2b097f07b..5dabdd89795e19a757209e03cc843776be705777 100644
--- a/src/caosdb/schema-pycaosdb-ini.yml
+++ b/src/caosdb/schema-pycaosdb-ini.yml
@@ -65,26 +65,39 @@ schema-pycaosdb-ini:
properties:
password_method:
const: input
+ required: [password_method]
then:
required: [url]
- if:
properties:
password_method:
const: plain
+ required: [password_method]
then:
required: [url, username, password]
- if:
properties:
password_method:
const: pass
+ required: [password_method]
then:
required: [url, username, password_identifier]
- if:
properties:
password_method:
const: keyring
+ required: [password_method]
then:
required: [url, username]
IntegrationTests:
description: "Used by the integration test suite from the caosdb-pyinttest repo."
additionalProperties: true
+ Misc:
+ description: "Some additional configuration settings."
+ additionalProperties: true
+ advancedtools:
+ description: "Configuration settings for the caosadvancedtools."
+ additionalProperties: true
+ sss_helper:
+ description: "Configuration settings for server-side scripting."
+ additionalProperties: true
diff --git a/src/doc/conf.py b/src/doc/conf.py
index b4dfcc9925eb8100b957b6c8c2c06c855e8d3ff0..ce1cfd261cb8b8d5aac8022d969c765b1c45fae3 100644
--- a/src/doc/conf.py
+++ b/src/doc/conf.py
@@ -29,10 +29,10 @@ copyright = '2022, IndiScale GmbH'
author = 'Daniel Hornung'
# The short X.Y version
-version = '0.7'
+version = '0.8.0'
# The full version, including alpha/beta/rc tags
# release = '0.5.2-rc2'
-release = '0.7.2-dev'
+release = '0.8.0-dev'
# -- General configuration ---------------------------------------------------
diff --git a/src/doc/configuration.md b/src/doc/configuration.md
index 802da4e91818ba65bd0184a9a5ac49f5c2ba02d2..02cbbd7b13d916a676ad26c277e370ae76bf3725 100644
--- a/src/doc/configuration.md
+++ b/src/doc/configuration.md
@@ -4,6 +4,15 @@ PyCaosDB tries to read from the inifile specified in the environment variable `P
alternatively in `~/.pycaosdb.ini` upon import. After that, the ini file `pycaosdb.ini` in the
current working directory will be read additionally, if it exists.
+Here, we will look at the most common configuration options. For a full and comprehensive
+description please check out the [example pycaosdb.ini file](https://gitlab.com/caosdb/caosdb-pylib/-/blob/main/examples/pycaosdb.ini). You can download this file and use
+it as a starting point.
+
+
+Typically, you need to change at least the `url` and `username` fields as required. (Ask your
+CaosDB administrator or IT crowd if you do not know what to put there, but for the demo instance at
+https://demo.indiscale.com, `username=admin` and `password=caosdb` should work).
+
## Authentication ##
The default configuration (that your are asked for your password when ever a connection is created
@@ -17,6 +26,8 @@ can be changed by setting `password_method`:
Windows). The password will be queried on first usage.
* with `password_method=plain` (**strongly discouraged**)
+The following illustrates the recommended options:
+
```ini
[Connection]
username=YOUR_USERNAME
@@ -35,7 +46,10 @@ username=YOUR_USERNAME
## SSL Certificate ##
-You can set the pass to the ssl certificate to be used:
+In some cases (especially if you are testing CaosDB) you might need to supply an SSL certificate to
+allow SSL encryption.
+
+The `cacert` option sets the path to the ssl certificate for the connection:
```ini
[Connection]
diff --git a/unittests/test_configs/pycaosdb-IntegrationTests.ini b/unittests/test_configs/pycaosdb-IntegrationTests.ini
new file mode 100644
index 0000000000000000000000000000000000000000..cb9871708f7f23c489de0cbc8f4fbda15dfa6ad0
--- /dev/null
+++ b/unittests/test_configs/pycaosdb-IntegrationTests.ini
@@ -0,0 +1,37 @@
+# -*- mode:conf; -*-
+## This sections needs to exist in addition to the usual section
+[IntegrationTests]
+# test_server_side_scripting.bin_dir.local=/path/to/scripting/bin
+test_server_side_scripting.bin_dir.local=/home/myself/test/caosdb-server/scripting/bin
+# test_server_side_scripting.bin_dir.server=/opt/caosdb/git/caosdb-server/scripting/bin
+
+# # location of the files from the pyinttest perspective
+# test_files.test_insert_files_in_dir.local=/extroot/test_insert_files_in_dir/
+test_files.test_insert_files_in_dir.local=/home/myself/test/debug_advanced/paths/extroot/test_insert_files_in_dir
+# # location of the files from the caosdb_servers perspective
+test_files.test_insert_files_in_dir.server=/opt/caosdb/mnt/extroot/test_insert_files_in_dir/
+
+########## Files ##################
+## Used by tests of file handling. Specify the path to an existing
+## directory in which file tests are performed, once as seen by the
+## host and once as seen by the server.
+
+# location of the files from the pyinttest (i.e. host) perspective
+#test_files.test_insert_files_in_dir.local=/extroot/test_insert_files_in_dir/
+
+# location of the files from the caosdb server's perspective
+#test_files.test_insert_files_in_dir.server=/opt/caosdb/mnt/extroot/test_insert_files_in_dir/
+
+# # location of the one-time tokens from the pyinttest's perspective
+# test_authentication.admin_token_crud = /authtoken/admin_token_crud.txt
+# test_authentication.admin_token_expired = /authtoken/admin_token_expired.txt
+# test_authentication.admin_token_3_attempts = /authtoken/admin_token_3_attempts.txt
+
+
+## Insert your usual settings here
+[Connection]
+url=https://localhost:10443/
+username=admin
+password_method=plain
+password=caosdb
+
diff --git a/unittests/test_configs/pycaosdb-empty.ini b/unittests/test_configs/pycaosdb-empty.ini
new file mode 100644
index 0000000000000000000000000000000000000000..e69de29bb2d1d6434b8b29ae775ad8c2e48c5391
diff --git a/unittests/test_configs/pycaosdb-real-world-1.ini b/unittests/test_configs/pycaosdb-real-world-1.ini
new file mode 100644
index 0000000000000000000000000000000000000000..e524f1d3465c61d89ae4a4dda54536a722f99837
--- /dev/null
+++ b/unittests/test_configs/pycaosdb-real-world-1.ini
@@ -0,0 +1,17 @@
+[Connection]
+url = https://localhost:10443
+cacert = /opt/caosdb/cert/caosdb.cert.pem
+debug = 0
+timeout = 5000
+
+[Misc]
+sendmail = /usr/local/bin/sendmail_to_file
+entity_loan.curator_mail_from=crawler-test@example.com
+entity_loan.curator_mail_to=crawler-test@example.com
+
+[sss_helper]
+external_uri = https://caosdb.example.com:443
+
+[advancedtools]
+crawler.from_mail=admin@example.com
+crawler.to_mail=admin@example.com
diff --git a/unittests/test_configs/pycaosdb-real-world-2.ini b/unittests/test_configs/pycaosdb-real-world-2.ini
new file mode 100644
index 0000000000000000000000000000000000000000..5ebd115a4a4de189d22180130acca2a4b78b6daf
--- /dev/null
+++ b/unittests/test_configs/pycaosdb-real-world-2.ini
@@ -0,0 +1,15 @@
+[Connection]
+url = https://samplemanager.example.com:443
+cacert = /opt/caosdb/cert/caosdb.cert.pem
+debug = 0
+timeout = 5000
+[Misc]
+sendmail = /usr/local/bin/sendmail_to_file
+entity_loan.curator_mail_from=crawler-test@example.com
+entity_loan.curator_mail_to=crawler-test@example.com
+[sss_helper]
+external_uri = https://localhost:10443
+[advancedtools]
+crawler.from_mail=crawler-test@example.com
+crawler.to_mail=crawler-test@example.com
+
diff --git a/unittests/test_configs/pycaosdb-server-side-scripting.ini b/unittests/test_configs/pycaosdb-server-side-scripting.ini
new file mode 100644
index 0000000000000000000000000000000000000000..de2867f8dc66b3e81f10f35e40c36f9cb8591604
--- /dev/null
+++ b/unittests/test_configs/pycaosdb-server-side-scripting.ini
@@ -0,0 +1,9 @@
+; this is the pycaosdb.ini for the server-side-scripting home.
+[Connection]
+url = https://caosdb-server:10443
+cacert = /opt/caosdb/cert/caosdb.cert.pem
+debug = 0
+timeout = 5000
+
+[Misc]
+sendmail = /usr/local/bin/sendmail_to_file
diff --git a/unittests/test_configs/pycaosdb4.ini b/unittests/test_configs/pycaosdb4.ini
new file mode 100644
index 0000000000000000000000000000000000000000..ddbc7ca6f969e55ea6131d96f091177a13687ece
--- /dev/null
+++ b/unittests/test_configs/pycaosdb4.ini
@@ -0,0 +1,4 @@
+[Connection]
+url=https://localhost:10443/
+username=admin
+password_method=input
diff --git a/unittests/test_configs/pycaosdb5.ini b/unittests/test_configs/pycaosdb5.ini
new file mode 100644
index 0000000000000000000000000000000000000000..3f365efdd92641a39b742e22f825033a69e12dc5
--- /dev/null
+++ b/unittests/test_configs/pycaosdb5.ini
@@ -0,0 +1,4 @@
+[Connection]
+url=https://localhost:10443/
+username=admin
+# No password method: should be "input" by default
diff --git a/unittests/test_schema.py b/unittests/test_schema.py
index 1552179a3e43dacb3ecca705466bb7ff84d330cf..fc3f63a4cbaeadcac3c1cb9be2d861a0688fe4b0 100644
--- a/unittests/test_schema.py
+++ b/unittests/test_schema.py
@@ -2,7 +2,9 @@
#
# This file is a part of the CaosDB Project.
#
+# Copyright (C) 2022 Indiscale GmbH <info@indiscale.com>
# Copyright (C) 2021 Alexander Schlemmer
+# Copyright (C) 2022 Daniel Hornung <d.hornung@indiscale.com>
#
# This program is free software: you can redistribute it and/or modify
# it under the terms of the GNU Affero General Public License as
@@ -31,15 +33,18 @@ from configparser import ConfigParser
def test_config_files():
for fn in glob(os.path.join(os.path.dirname(__file__), "test_configs", "*.ini")):
+ print(f"Testing {fn}.")
c = ConfigParser()
c.read(fn)
+ print(config_to_yaml(c))
validate_yaml_schema(config_to_yaml(c))
def test_broken_config_files():
for fn in glob(os.path.join(os.path.dirname(__file__), "broken_configs", "*.ini")):
- print(fn)
+ print(f"Testing {fn}.")
with raises(ValidationError):
c = ConfigParser()
c.read(fn)
+ print(config_to_yaml(c))
validate_yaml_schema(config_to_yaml(c))