CHANGELOG.md

@@ -5,7 +5,7 @@ All notable changes to this project will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
-## [Unreleased] ##
+## [Unreleased]
 
 ### Added ###
 
@@ -17,14 +17,43 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ### Fixed ###
 
+* [#75](https://gitlab.indiscale.com/caosdb/src/caosdb-pylib/-/issues/75), [#103](https://gitlab.indiscale.com/caosdb/src/caosdb-pylib/-/issues/103) Fixed JSON schema to allow more sections, and correct requirements for
+  password method.
+
+### Security ###
+
+### Documentation ###
+
+
+## [0.7.2] - 2022-03-25 ##
+(Timm Fitschen)
+
+### Added ###
+
+### Changed ###
+
+### Deprecated ###
+
+* In module `caosdb.apiutils`:
+  * `CaosDBPythonEntity` class
+  * `convert_to_entity` function
+  * `convert_to_python_object` function
+
+### Removed ###
+
+### Fixed ###
+
 * [caosdb-pylib#106](https://gitlab.indiscale.com/caosdb/src/caosdb-pylib/-/issues/106)
-  Parsing Error in class caosdb.common.models.ACL
+  Parsing Error in class caosdb.common.models.ACL. This may lead to the
+  unintentional revocation of permissions for some users or roles during
+  updates. However, no additional permissions are being granted.
 
 ### Security ###
 
 ### Documentation ###
 
 ## [0.7.1] - 2022-03-11 ##
+(Daniel Hornung)
 
 ### Documentation ###
 

DEPENDENCIES.md

-* caosdb-server == 0.3
-* Python >= 3.5
+* caosdb-server >= 0.7.2
+* Python >= 3.6
 * pip >= 20.0.2
 
-
-Any other dependencies are being installed via pip
+Any other dependencies are defined in the setup.py and are being installed via pip

README_SETUP.md

@@ -82,60 +82,8 @@ pip3 install --user .[jsonschema]
 
 ## Configuration ##
 
-The  configuration is done using `ini` configuration files.
-PyCaosDB tries to read from the inifile specified in the environment variable `PYCAOSDBINI` or
-alternatively in `~/.pycaosdb.ini` upon import.  After that, the ini file `pycaosdb.ini` in the
-current working directory will be read additionally, if it exists.
-
-Here, we will look at the most common configuration options. For a full and 
-comprehensive description please check out 
-[pycaosdb.ini file](https://gitlab.com/caosdb/caosdb-pylib/-/blob/main/examples/pycaosdb.ini) 
-You can download this file and use it as a starting point.
-
-
-Typically, you need to change at least the `url` and `username` fields as required. 
-(Ask your CaosDB administrator or IT crowd if
-you do not know what to put there, but for the demo instances https://demo.indiscale.com, `username=admin`
-and `password=caosdb` should work).
-
-### Authentication ###
-
-The default configuration (that your are asked for your password when ever a connection is created
-can be changed by setting `password_method`:
-
-* with `password_method=input` password (and possibly user) will be queried on demand (**default**)
-* use the password manager [pass](https://www.passwordstore.org) by using `pass` as value, see also the [ArchWiki
-  entry](https://wiki.archlinux.org/index.php/Pass#Basic_usage). This also requires `password_identifier` which refers to the identifier within pass
-  for the desired password.
-* install the python package [keyring](https://pypi.org/project/keyring), to use the system keyring/wallet (macOS, GNOME, KDE,
-  Windows). The password will be queried on first usage.
-* with `password_method=plain` (**strongly discouraged**)
-
-The following illustrates the recommended options:
-
-```ini
-[Connection]
-# using "pass" password manager
-#password_method=pass
-#password_identifier=...
-
-# using the system keyring/wallet (macOS, GNOME, KDE, Windows)
-#password_method=keyring
-```
-
-### SSL Certificate ###
-In some cases (especially if you are testing CaosDB) you might need to supply 
-an SSL certificate to allow SSL encryption.
-
-```ini
-[Connection]
-cacert=/path/to/caosdb.ca.pem
-```
-
-### Further Settings ###
-As mentioned above, a complete list of options can be found in the 
-[pycaosdb.ini file](https://gitlab.com/caosdb/caosdb-pylib/-/blob/main/examples/pycaosdb.ini) in 
-the examples folder of the source code.
+The configuration is done using `ini` configuration files.  The content of these configuration files
+is described in detail in the [configuration section of the documentation](https://docs.indiscale.com/caosdb-pylib/configuration.html).
 
 ## Try it out ##
 

RELEASE_GUIDELINES.md

@@ -36,8 +36,10 @@ guidelines of the CaosDB Project
 9. Publish the release by executing `./release.sh` with uploads the caosdb
    module to the Python Package Index [pypi.org](https://pypi.org).
 
-10. Merge the main branch back into the dev branch.
+10. Create a gitlab release on gitlab.indiscale.com and gitlab.com
 
-11. After the merge of main to dev, start a new development version by
+11. Merge the main branch back into the dev branch.
+
+12. After the merge of main to dev, start a new development version by
     setting `ISRELEASED` to `False` and by increasing at least the `MICRO`
     version in [setup.py](./setup.py) and preparing CHANGELOG.md.

setup.py

@@ -47,9 +47,13 @@ from setuptools import find_packages, setup
 
 ISRELEASED = False
 MAJOR = 0
-MINOR = 7
-MICRO = 2
-PRE = "dev"  # e.g. rc0, alpha.1, 0.beta-23
+MINOR = 8
+MICRO = 0
+# Do not tag as pre-release until this commit
+# https://github.com/pypa/packaging/pull/515
+# has made it into a release. Probably we should wait for pypa/packaging>=21.4
+# https://github.com/pypa/packaging/releases
+PRE = "" # "dev"  # e.g. rc0, alpha.1, 0.beta-23
 
 if PRE:
     VERSION = "{}.{}.{}-{}".format(MAJOR, MINOR, MICRO, PRE)
@@ -166,12 +170,13 @@ def setup_package():
         packages=find_packages('src'),
         python_requires='>=3.6',
         package_dir={'': 'src'},
-        install_requires=['lxml>=3.6.4',
-                          'PyYaml>=3.12', 'future', 'PySocks>=1.6.7'],
+        install_requires=['lxml>=4.6.3',
+                          'PyYAML>=6.0', 'future', 'PySocks>=1.6.7'],
         extras_require={'keyring': ['keyring>=13.0.0'],
-                        'jsonschema': ['jsonschema==4.0.1']},
+                        'jsonschema': ['jsonschema>=4.4.0']},
         setup_requires=["pytest-runner>=2.0,<3dev"],
-        tests_require=["pytest", "pytest-cov", "coverage>=4.4.2", "jsonschema==4.0.1"],
+        tests_require=["pytest", "pytest-cov", "coverage>=4.4.2",
+                       "jsonschema>=4.4.0"],
         package_data={
             'caosdb': ['cert/indiscale.ca.crt', 'schema-pycaosdb-ini.yml'],
         },

src/caosdb/configuration.py

@@ -84,23 +84,28 @@ def config_to_yaml(config):
 
 
 def validate_yaml_schema(valobj):
-    # TODO: Re-enable warning once the schema has been extended to also cover
-    # SSS pycaosdb.inis and integration tests.
     if optional_jsonschema_validate:
         with open(os.path.join(os.path.dirname(__file__), "schema-pycaosdb-ini.yml")) as f:
             schema = yaml.load(f, Loader=yaml.SafeLoader)
         optional_jsonschema_validate(instance=valobj, schema=schema["schema-pycaosdb-ini"])
-    # else:
-    #     warnings.warn("""
-    #                     Warning: The validation could not be performed because `jsonschema` is not installed.
-    #                 """)
+    else:
+        warnings.warn("""
+            Warning: The validation could not be performed because `jsonschema` is not installed.
+        """)
 
 
 def _read_config_files():
-    """Function to read config files from different paths. Checks for path in $PYCAOSDBINI or home directory (.pycaosdb.ini) and in the current working directory (pycaosdb.ini).
+    """Function to read config files from different paths.
+
+    Checks for path either in ``$PYCAOSDBINI`` or home directory (``.pycaosdb.ini``), and
+    additionally in the current working directory (``pycaosdb.ini``).
+
+    Returns
+    -------
+
+    ini files: list
+      The successfully parsed ini-files. Order: env_var or home directory, cwd. Used for testing the function.
 
-    Returns:
-        [list]: list with successfully parsed ini-files. Order: env_var or home directory, cwd. Used for testing the function.
     """
     return_var = []
     if "PYCAOSDBINI" in environ:

src/caosdb/schema-pycaosdb-ini.yml

@@ -65,26 +65,39 @@ schema-pycaosdb-ini:
             properties:
               password_method:
                 const: input
+            required: [password_method]
           then:
             required: [url]
         - if:
             properties:
               password_method:
                 const: plain
+            required: [password_method]
           then:
             required: [url, username, password]
         - if:
             properties:
               password_method:
                 const: pass
+            required: [password_method]
           then:
             required: [url, username, password_identifier]
         - if:
             properties:
               password_method:
                 const: keyring
+            required: [password_method]
           then:
             required: [url, username]
     IntegrationTests:
       description: "Used by the integration test suite from the caosdb-pyinttest repo."
       additionalProperties: true
+    Misc:
+      description: "Some additional configuration settings."
+      additionalProperties: true
+    advancedtools:
+      description: "Configuration settings for the caosadvancedtools."
+      additionalProperties: true
+    sss_helper:
+      description: "Configuration settings for server-side scripting."
+      additionalProperties: true

src/doc/conf.py

@@ -29,10 +29,10 @@ copyright = '2022, IndiScale GmbH'
 author = 'Daniel Hornung'
 
 # The short X.Y version
-version = '0.7'
+version = '0.8.0'
 # The full version, including alpha/beta/rc tags
 # release = '0.5.2-rc2'
-release = '0.7.2-dev'
+release = '0.8.0-dev'
 
 
 # -- General configuration ---------------------------------------------------

src/doc/configuration.md

@@ -4,6 +4,15 @@ PyCaosDB tries to read from the inifile specified in the environment variable `P
 alternatively in `~/.pycaosdb.ini` upon import.  After that, the ini file `pycaosdb.ini` in the
 current working directory will be read additionally, if it exists.
 
+Here, we will look at the most common configuration options. For a full and comprehensive
+description please check out the [example pycaosdb.ini file](https://gitlab.com/caosdb/caosdb-pylib/-/blob/main/examples/pycaosdb.ini).  You can download this file and use
+it as a starting point.
+
+
+Typically, you need to change at least the `url` and `username` fields as required.  (Ask your
+CaosDB administrator or IT crowd if you do not know what to put there, but for the demo instance at
+https://demo.indiscale.com, `username=admin` and `password=caosdb` should work).
+
 ## Authentication ##
 
 The default configuration (that your are asked for your password when ever a connection is created
@@ -17,6 +26,8 @@ can be changed by setting `password_method`:
   Windows). The password will be queried on first usage.
 * with `password_method=plain` (**strongly discouraged**)
 
+The following illustrates the recommended options:
+
 ```ini
 [Connection]
 username=YOUR_USERNAME
@@ -35,7 +46,10 @@ username=YOUR_USERNAME
 
 ## SSL Certificate ##
 
-You can set the pass to the ssl certificate to be used:
+In some cases (especially if you are testing CaosDB) you might need to supply an SSL certificate to
+allow SSL encryption.
+
+The `cacert` option sets the path to the ssl certificate for the connection:
 
 ```ini
 [Connection]

unittests/test_configs/pycaosdb-IntegrationTests.ini

+# -*- mode:conf; -*-
+## This sections needs to exist in addition to the usual section
+[IntegrationTests]
+# test_server_side_scripting.bin_dir.local=/path/to/scripting/bin
+test_server_side_scripting.bin_dir.local=/home/myself/test/caosdb-server/scripting/bin
+# test_server_side_scripting.bin_dir.server=/opt/caosdb/git/caosdb-server/scripting/bin
+
+# # location of the files from the pyinttest perspective
+# test_files.test_insert_files_in_dir.local=/extroot/test_insert_files_in_dir/
+test_files.test_insert_files_in_dir.local=/home/myself/test/debug_advanced/paths/extroot/test_insert_files_in_dir
+# # location of the files from the caosdb_servers perspective
+test_files.test_insert_files_in_dir.server=/opt/caosdb/mnt/extroot/test_insert_files_in_dir/
+
+########## Files ##################
+## Used by tests of file handling. Specify the path to an existing
+## directory in which file tests are performed, once as seen by the
+## host and once as seen by the server.
+
+# location of the files from the pyinttest (i.e. host) perspective
+#test_files.test_insert_files_in_dir.local=/extroot/test_insert_files_in_dir/
+
+# location of the files from the caosdb server's perspective
+#test_files.test_insert_files_in_dir.server=/opt/caosdb/mnt/extroot/test_insert_files_in_dir/
+
+# # location of the one-time tokens from the pyinttest's perspective
+# test_authentication.admin_token_crud = /authtoken/admin_token_crud.txt
+# test_authentication.admin_token_expired = /authtoken/admin_token_expired.txt
+# test_authentication.admin_token_3_attempts = /authtoken/admin_token_3_attempts.txt
+
+
+## Insert your usual settings here
+[Connection]
+url=https://localhost:10443/
+username=admin
+password_method=plain
+password=caosdb
+

unittests/test_configs/pycaosdb-real-world-1.ini

+[Connection]
+url = https://localhost:10443
+cacert = /opt/caosdb/cert/caosdb.cert.pem
+debug = 0
+timeout = 5000
+
+[Misc]
+sendmail = /usr/local/bin/sendmail_to_file
+entity_loan.curator_mail_from=crawler-test@example.com
+entity_loan.curator_mail_to=crawler-test@example.com
+
+[sss_helper]
+external_uri = https://caosdb.example.com:443
+
+[advancedtools]
+crawler.from_mail=admin@example.com
+crawler.to_mail=admin@example.com

unittests/test_configs/pycaosdb-real-world-2.ini

+[Connection]
+url = https://samplemanager.example.com:443
+cacert = /opt/caosdb/cert/caosdb.cert.pem
+debug = 0
+timeout = 5000
+[Misc]
+sendmail = /usr/local/bin/sendmail_to_file
+entity_loan.curator_mail_from=crawler-test@example.com
+entity_loan.curator_mail_to=crawler-test@example.com
+[sss_helper]
+external_uri = https://localhost:10443
+[advancedtools]
+crawler.from_mail=crawler-test@example.com
+crawler.to_mail=crawler-test@example.com           
+

unittests/test_configs/pycaosdb-server-side-scripting.ini

+; this is the pycaosdb.ini for the server-side-scripting home.
+[Connection]
+url = https://caosdb-server:10443
+cacert = /opt/caosdb/cert/caosdb.cert.pem
+debug = 0
+timeout = 5000
+
+[Misc]
+sendmail = /usr/local/bin/sendmail_to_file

unittests/test_configs/pycaosdb4.ini

+[Connection]
+url=https://localhost:10443/
+username=admin
+password_method=input

unittests/test_configs/pycaosdb5.ini

+[Connection]
+url=https://localhost:10443/
+username=admin
+# No password method: should be "input" by default

unittests/test_schema.py

@@ -2,7 +2,9 @@
 #
 # This file is a part of the CaosDB Project.
 #
+# Copyright (C) 2022 Indiscale GmbH <info@indiscale.com>
 # Copyright (C) 2021 Alexander Schlemmer
+# Copyright (C) 2022 Daniel Hornung <d.hornung@indiscale.com>
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License as
@@ -31,15 +33,18 @@ from configparser import ConfigParser
 
 def test_config_files():
     for fn in glob(os.path.join(os.path.dirname(__file__), "test_configs", "*.ini")):
+        print(f"Testing {fn}.")
         c = ConfigParser()
         c.read(fn)
+        print(config_to_yaml(c))
         validate_yaml_schema(config_to_yaml(c))
 
 
 def test_broken_config_files():
     for fn in glob(os.path.join(os.path.dirname(__file__), "broken_configs", "*.ini")):
-        print(fn)
+        print(f"Testing {fn}.")
         with raises(ValidationError):
             c = ConfigParser()
             c.read(fn)
+            print(config_to_yaml(c))
             validate_yaml_schema(config_to_yaml(c))