Merge branch 'dev' into f-grpc-main

CHANGELOG.md

@@ -10,6 +10,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 ### Added
 
 * An openAPI specification of the XML api
+* New server configuration option `SERVER_BIND_ADDRESS`, which is the address to listen to. See [server.conf](conf/core/server.conf).
 
 ### Changed
 
@@ -19,12 +20,15 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ### Fixed
 
+* #183 No reasonable error when using bad datetime format.
+  (https://gitlab.indiscale.com/caosdb/src/caosdb-server/-/issues/183)
 * #127 "nan" as value (list item) in properties with data type "LIST<DOUBLE>"
   return with "Cannot parse value to double" error.
 * #170 Updating an abstract list-type property with a default value fails with
   "unkown error".
 * #145 Documentation of importances and inheritance
 * Missing sources of the easy-unit dependency.
+* #178 Formatting of tables in documentation
 
 ### Security
 

README_SETUP.md

 # Getting Started with the CaosDB Server
-Here, you find information on requirements, the installation, configuration and more.
+
+Here, you find information on requirements, the installation, configuration and
+more.
 
 ## Requirements
 
@@ -9,6 +11,7 @@ Here, you find information on requirements, the installation, configuration and
 * caosdb-mysqlbackend=3.0
 
 ### Third-party Software
+
 * `>=Java 8`
 * `>=Apache Maven 3.0.4`
 * `>=Python 3.4`
@@ -24,6 +27,7 @@ Here, you find information on requirements, the installation, configuration and
 - `easy-units` >= 0.0.1    https://gitlab.com/timm.fitschen/easy-units
 
 #### Install the requirements on Debian
+
 On Debian, the required packages can be installed with:
 
     apt-get install git make mariadb-server maven openjdk-11-jdk-headless \
@@ -80,7 +84,8 @@ server:
      Replace `localhost` by your host name, if you want.
    - `keytool -importkeystore -srckeystore caosdb.jks -destkeystore caosdb.p12 -deststoretype PKCS12 -srcalias selfsigned`
    - Export the public part only: `openssl pkcs12 -in caosdb.p12 -nokeys -out cert.pem`.
-     The resulting `cert.pem` can safely be given to users to allow ssl verification.
+     The resulting `cert.pem` can safely be given to users to allow ssl
+     verification.
    - You can check the content of the certificate with `openssl x509 -in cert.pem -text`
 
    Alternatively, you can create a keystore from certificate files that you already have:
@@ -88,13 +93,16 @@ server:
    - `keytool -importkeystore -srckeystore all-certs.pkcs12 -srcstoretype PKCS12  -deststoretype pkcs12 -destkeystore caosdb.jks`
 3. Install/configure the MySQL back-end: see the `README_SETUP.md` of the
    `caosdb-mysqlbackend` repository
-4. Create an authtoken config (e.g. copy `conf/core/authtoken.example.yaml` to `conf/ext/authtoken.yml` and change it)
+4. Create an authtoken config (e.g. copy `conf/core/authtoken.example.yaml` to
+   `conf/ext/authtoken.yml` and change it)
 5. Copy `conf/core/server.conf` to `conf/ext/server.conf` and change it
    appropriately:
     * Setup for MySQL back-end:
       specify the fields `MYSQL_USER_NAME`, `MYSQL_USER_PASSWORD`,
       `MYSQL_DATABASE_NAME`, and `MYSQL_HOST`.
-    * Choose the ports under which CaosDB will be accessible.
+    * Choose the host and ports under which CaosDB will be accessible. The host
+      is the IP address the server should listen on. Blank (the default) means
+      all interfaces, and `127.0.0.1` means only localhost.
     * Setup the SSL certificate: Assuming that there is an appropriate `Java Key
       Store` file (see above), change the fields `CERTIFICATES_KEY_PASSWORD`,
       `CERTIFICATES_KEY_STORE_PATH`, and `CERTIFICATES_KEY_STORE_PASSWORD`.
@@ -150,6 +158,9 @@ type `https://localhost:10443` in your Browser, assuming you used 10443 as port.
 Note, that you will get a security warning if you are using a self-signed
 certificate.
 
+You can use `make run-single` to directly start the server without rebuilding
+sources (if you called `make run` previously).
+
 ## Run Unit Tests
 
 `$ make test`
@@ -213,3 +224,60 @@ cd javasphinx
 git checkout 659209069603a
 pip3 install .
 ```
+
+## Installation under special conditions
+
+### Installation without or with restricted internet connection on the host system
+
+It is of course possible to install CaosDB on servers which do not have an
+internet connection or only have limited access to the internet.
+However, maven, which is used to build the CaosDB server, typically accesses
+the internet very often, and some reconfiguration might be necessary to prevent
+timeout issues and similar headache.
+
+A recommendation to proceed is:
+- Clone the repositories (caosdb-server, caosdb-mysqlbackend and subrepository
+  caosdb-webui) on a second machine that has internet connection.
+- Run `make install` and `make run` in the caosdb-server repository. This will
+  download all required dependencies and setup all maven-related artifacts in
+  a folder called `.m2` typically located in the home directory.
+- In addition to copying over the repositories for the server, mysqlbackend and
+  webui to the target host machine, also copy the complete `.m2` folder to the
+  **home directory of the user that is running the CaosDB server**.
+- On the host machine, open the file `pom.xml` in the server repository.
+- Look for the section `<repositories>`. There should be a `<respository>` with
+  `<id>local-maven-repo</id>`. It should be moved on top of the list of
+  repositories, so that the xml block appears directly after `<repositories>`
+  and before the repository maven-central.
+
+## Troubleshooting / FAQ
+
+### I set up and run the server, but I get an error that XSLT parsing failed in the WebUI
+
+You probably forgot to `make` the webui.
+
+Make sure that you:
+- Installed the WebUI according to the section [Web UI]
+- Run `make install` in the webui sub folder
+
+### How can I run the CaosDB server using ports 443 and 80?
+
+On standard linux setups, ports 443 and 80 cannot be opened by non-root processes.
+
+There are multiple solutions for this problem:
+- Use iptables to redirect a low port to a high port.
+- Use `CAP_NET_BIND_SERVICE`.
+- Use authbind.
+
+Here are some resources to read about these options:
+- [Question on Stackoverflow](https://stackoverflow.com/questions/413807/is-there-a-way-for-non-root-processes-to-bind-to-privileged-ports-on-linux)
+- [Questoin on Superuser](https://superuser.com/questions/710253/allow-non-root-process-to-bind-to-port-80-and-443)
+- [iptables tutorial](https://www.frozentux.net/iptables-tutorial/iptables-tutorial.html#REDIRECTTARGET)
+
+Using authbind is a simple solution that is briefly described here:
+- Install the package authbind
+- Create (empty) files `/etc/authbind/byport/80` (and `.../443`), e.g. using `touch`
+- Grant execution permissions for the user who runs the server to the new files.
+- Run the CaosDB server using authbind with the `--deep` option:
+  `authbind --deep make run`. The `--deep` option is necessary because the
+  server starts a subprocess which actually opens the ports.

conf/core/server.conf

@@ -77,6 +77,10 @@ MYSQL_SCHEMA_VERSION=v5.0
 # The context root is a prefix which allows running multiple instances of CaosDB using the same
 # hostname and port. Must start with "/".
 CONTEXT_ROOT=
+# Server bind/host address, which is the address to listen to. Set to blank, or
+# 0.0.0.0 in IPv4, to listen to all. Set to 127.0.0.1 to make it available to
+# localhost only.
+SERVER_BIND_ADDRESS=
 # HTTPS port of this server instance.
 SERVER_PORT_HTTPS=443
 # HTTP port of this server instance.

src/doc/specification/Authentication.md

-# Authentication
- Some features of CaosDB are available to registered users only. Making any changes to the data stock via HTTP requires authentication by `username` _plus_ `password`. They are to be send as a HTTP header, while the password is to be hashed by the sha512 algorithm:
-
-| `username:` | `$username` | 
-|-------------|-------------|-
-| `password:` | `$SHA512ed_password` |
-
-
-## Sessions
-
-### Login
-
-#### Request Challenge
-
- * `GET http://host:port/mpidsserver/login?username=$username`
- * `GET http://host:port/mpidsserver/login` with `username` header
-
-*no password required to be sent over http*
-
-The request returns an AuthToken with a login challenge as a cookie. The AuthToken is a dictionary of the following form:
-
-
-        {scope=$scope;
-        mode=LOGIN;
-        offerer=$offerer;
-        auth=$auth
-        expires=$expires;
-        date=$date;
-        hash=$hash;
-        session=$session;
-        }
-
- $scope:: A uri pattern string. Example: ` {**/*} `
- $mode:: `ONETIME`, `SESSION`, or `LOGIN`
- $offerer:: A valid username
- $auth:: A valid username
- $expires:: A `YYYY-MM-DD HH:mm:ss[.nnnn]` date string
- $date:: A `YYYY-MM-DD HH:mm:ss[.nnnn]` date string
- $hash:: A string
- $session:: A string
-
-The challenge is solved by concatenating the `$hash` string and the user's `$password` string and calculating the sha512 hash of both. Pseudo code:
-
-
-        $solution = sha512($hash + sha512($password))
-
-#### Send Solution
-
-The old $hash string in the cookie has to be replaces by $solution and the cookie is to be send with the next request:
-
-`PUT http://host:port/mpidsserver/login`
-
-The server will return the user's entity in the HTTP body, e.g.
-
-
-        <Response ...>
-          <User name="$username" ...>
-            ...
-          </User>
-        </Response>
-
-and a new AuthToken with `$mode=SESSION` and a new expiration date and so on. This AuthToken cookie is to be send with every request.
-
-### Logout
-
-Send 
-
-`PUT http://host:port/mpidsserver/logout`
-
-with a valid AuthToken cookie. No new AuthToken will be returned and no AuthToken with that `$session` will be accepted anymore.
-
-### Commandline solution with `curl` ##
-
-To use curl for talking with the server, first save your password into a variable:
-`PW=$(cat)`
-
-The create a cookie in `cookie.txt` like this (note that this makes your password visible for a short time to everyone on your system:
-`curl -X POST -c cookie.txt -D head.txt -H "Content-Type: application/x-www-form-urlencoded" -d username=<USERNAME> -d password="$PW" --insecure "https://<SERVER>/login`
-
-To use the cookie, pass it on with later requests:
-`curl -X GET -b cookie.txt --insecure "https://<SERVER>/Entity/12345"`
-

src/doc/specification/Authentication.rst

+Authentication
+==============
+
+Some features of CaosDB are available to registered users only. Making any
+changes to the data stock via HTTP requires authentication by ``username`` **plus**
+``password``. They are to be send as a HTTP header, while the password is to be
+hashed by the sha512 algorithm:
+
+============= ======================
+username:     password:
+============= ======================
+``$username`` ``$SHA512ed_password``
+============= ======================
+
+Sessions
+--------
+
+Login
+^^^^^
+
+Request Challenge
+^^^^^^^^^^^^^^^^^
+
+* ``GET http://host:port/mpidsserver/login?username=$username``
+* ``GET http://host:port/mpidsserver/login`` with ``username`` header
+
+**No password is required to be sent over http.**
+
+The request returns an AuthToken with a login challenge as a cookie.
+The AuthToken is a dictionary of the following form:
+
+.. code-block::
+
+   {scope=$scope;
+    mode=LOGIN;
+    offerer=$offerer;
+    auth=$auth
+    expires=$expires;
+    date=$date;
+    hash=$hash;
+    session=$session;
+   }
+
+where
+
+* ``$scope`` :: A uri pattern string. Example: ``{ **/* }``
+* ``$mode`` :: ``ONETIME``, ``SESSION``, or ``LOGIN``
+* ``$offerer`` :: A valid username
+* ``$auth`` :: A valid username
+* ``$expires`` :: A ``YYYY-MM-DD HH:mm:ss[.nnnn]`` date string
+* ``$date`` :: A ``YYYY-MM-DD HH:mm:ss[.nnnn]`` date string
+* ``$hash`` :: A string
+* ``$session`` :: A string
+
+The challenge is solved by concatenating the ``$hash`` string and
+the user's ``$password`` string and calculating the sha512 hash of both.
+Pseudo code:
+
+.. code-block::
+
+   $solution = sha512($hash + sha512($password))
+
+Send Solution
+^^^^^^^^^^^^^
+
+The old ``$hash`` string in the cookie has to be replaces by ``$solution`` and
+ the cookie is to be send with the next request:
+
+``PUT http://host:port/mpidsserver/login``
+
+The server will return the user's entity in the HTTP body, e.g.
+
+.. code-block::
+
+   <Response ...>
+     <User name="$username" ...>
+      ...
+     </User>
+   </Response>
+
+and a new AuthToken with ``$mode=SESSION`` and a new expiration date and so
+on. This AuthToken cookie is to be send with every request.
+
+Logout
+^^^^^^
+
+Send
+
+``PUT http://host:port/mpidsserver/logout``
+
+with a valid AuthToken cookie. No new AuthToken will be returned and no
+AuthToken with that ``$session`` will be accepted anymore.
+
+Commandline solution with ``curl``
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+To use curl for talking with the server, first save your password into a
+variable: ``PW=$(cat)``
+
+The create a cookie in ``cookie.txt`` like this (note that this makes your
+password visible for a short time to everyone on your system:
+
+.. code-block:: sh
+
+   curl -X POST -c cookie.txt -D head.txt -H "Content-Type: application/x-www-form-urlencoded" -d username=<USERNAME> -d password="$PW" --insecure "https://<SERVER>/login
+
+To use the cookie, pass it on with later requests:
+
+.. code-block:: sh
+
+   curl -X GET -b cookie.txt --insecure "https://<SERVER>/Entity/12345"

src/main/java/org/caosdb/server/CaosDBServer.java

@@ -333,6 +333,13 @@ public class CaosDBServer extends Application {
   }
 
   private static void initWebServer() throws Exception {
+    /* For the host, the property can't be used directly since blank should mean
+    all interfaces, not localhost; which means replacing a blank value with
+    null. */
+    final String server_bind_address_property =
+        getServerProperty(ServerProperties.KEY_SERVER_BIND_ADDRESS);
+    final String server_bind_address =
+        server_bind_address_property.length() == 0 ? null : server_bind_address_property;
     final int port_https =
         Integer.parseInt(getServerProperty(ServerProperties.KEY_SERVER_PORT_HTTPS));
     final int port_http =
@@ -350,10 +357,15 @@ public class CaosDBServer extends Application {
         Integer.parseInt(getServerProperty(ServerProperties.KEY_MAX_CONNECTIONS));
 
     if (NO_TLS) {
-      runHTTPServer(port_http, initialConnections, maxTotalConnections);
+      runHTTPServer(server_bind_address, port_http, initialConnections, maxTotalConnections);
     } else {
       runHTTPSServer(
-          port_https, port_http, port_redirect_https, initialConnections, maxTotalConnections);
+          server_bind_address,
+          port_https,
+          port_http,
+          port_redirect_https,
+          initialConnections,
+          maxTotalConnections);
     }
     GRPCServer.startServer();
   }
@@ -372,7 +384,10 @@ public class CaosDBServer extends Application {
    * @throws Exception
    */
   private static void runHTTPServer(
-      final int port_http, final int initialConnections, final int maxTotalConnections)
+      final String server_bind_address,
+      final int port_http,
+      final int initialConnections,
+      final int maxTotalConnections)
       throws Exception {
     Engine.getInstance()
         .getRegisteredServers()
@@ -385,7 +400,7 @@ public class CaosDBServer extends Application {
         new Server(
             (Context) null,
             Arrays.asList(Protocol.HTTP),
-            null,
+            server_bind_address,
             port_http,
             (Restlet) null,
             "org.restlet.ext.jetty.HttpServerHelper");
@@ -422,6 +437,7 @@ public class CaosDBServer extends Application {
    * connections on `port_http` and redirect any http connections to `port_redirect_https`.
    *
    * @author Timm Fitschen
+   * @param server_bind_address IP address to listen on (null means all interfaces).
    * @param port_https Listen on this port for https connections.
    * @param port_http Listen on this port for http connections and send http-to-https redirect with
    *     different port.
@@ -429,6 +445,7 @@ public class CaosDBServer extends Application {
    * @throws Exception if problems occur starting up this server.
    */
   private static void runHTTPSServer(
+      final String server_bind_address,
       final int port_https,
       final int port_http,
       final int port_redirect_https,
@@ -445,7 +462,7 @@ public class CaosDBServer extends Application {
         new Server(
             (Context) null,
             Arrays.asList(Protocol.HTTPS),
-            null,
+            server_bind_address,
             port_https,
             (Restlet) null,
             "org.caosdb.server.CaosDBServerConnectorHelper");
@@ -456,7 +473,7 @@ public class CaosDBServer extends Application {
       logger.info("Redirecting to " + port_redirect_https);
       component
           .getServers()
-          .add(Protocol.HTTP, port_http)
+          .add(Protocol.HTTP, server_bind_address, port_http)
           .setNext(new HttpToHttpsRedirector(port_redirect_https));
     }
 

src/main/java/org/caosdb/server/ServerProperties.java

@@ -60,6 +60,7 @@ public class ServerProperties extends Properties {
   public static final String KEY_CONTEXT_ROOT = "CONTEXT_ROOT";
   public static final String KEY_POLICY_COMPONENT = "POLICY_COMPONENT";
 
+  public static final String KEY_SERVER_BIND_ADDRESS = "SERVER_BIND_ADDRESS";
   public static final String KEY_SERVER_PORT_HTTPS = "SERVER_PORT_HTTPS";
   public static final String KEY_SERVER_PORT_HTTP = "SERVER_PORT_HTTP";
   public static final String KEY_REDIRECT_HTTP_TO_HTTPS_PORT = "REDIRECT_HTTP_TO_HTTPS_PORT";

src/main/java/org/caosdb/server/datatype/DateTimeDatatype.java

@@ -37,12 +37,19 @@ public class DateTimeDatatype extends AbstractDatatype {
     try {
       if (value instanceof UTCDateTime || value instanceof Date) {
         return (DateTimeInterface) value;
-      } else if (value instanceof GenericValue) {
-        return DateTimeFactory2.valueOf(((GenericValue) value).toDatabaseString());
+      }
+
+      DateTimeInterface result;
+      if (value instanceof GenericValue) {
+        result = DateTimeFactory2.valueOf(((GenericValue) value).toDatabaseString());
       } else {
-        return DateTimeFactory2.valueOf(value.toString());
+        result = DateTimeFactory2.valueOf(value.toString());
       }
-    } catch (final IllegalArgumentException e) {
+
+      // Test if this is storable
+      result.toDatabaseString();
+      return result;
+    } catch (final UnsupportedOperationException | IllegalArgumentException e) {
       throw ServerMessages.CANNOT_PARSE_DATETIME_VALUE;
     }
   }