diff --git a/src/doc/administration/server_side_scripting.rst b/src/doc/administration/server_side_scripting.rst new file mode 100644 index 0000000000000000000000000000000000000000..f85b8a709b294db01fec3bfc892caf4d1d065bf6 --- /dev/null +++ b/src/doc/administration/server_side_scripting.rst @@ -0,0 +1,83 @@ +Server-Side Scripting +===================== + +Introduction +------------ + +Small computation task, like some visualization, might be easily implemented in Python or some other language, but cumbersome to integrate into the server. Furthermore, the CaosDB server should stay a general tool without burden from specific projects. Also, one might want to trigger some standardized processing task from the web interface for convenience. For these situations the "server side scripting" is intended. + +Concepts +------------ + +The basic idea is that a script or program (script in the following) can be called to run on the server (or elsewhere in future) to do some calculations. This triggering of the script is done over the API so it can be done with any client. Input arguments can be passed to the script and the STDOUT and STDERR are returned. + +Each script is executed in a temporary home directory, which is automatically clean up. However, scripts can store files in the "$SHARED" folder and for example provide users a link that allows them to download files. + +Write and Install a Script +-------------------------- + +A server-side script must accept at least the ``--auth-token=AUTH_TOKEN`` option. All other command-line parameters which are passed to the script are not specified by the API and maybe defined by the script itself. + +So a minimal bash script would be + +.. code-block:: sh + #!/bin/bash + + echo Hello, World! + +thereby just ignoring the ``--auth-token`` option. + +The script has to be executable and must be placed somewhere in one of the directory trees which are configured by the server config :doc:`SERVER_SIDE_SCRIPTING_BIN_DIRS <configuration>`. + +Users will need the ``SCRIPTING:EXECUTE:path:to:the:script`` permission. Here the path to the script is of course relativet to the ``SERVER_SIDE_SCRIPTING_BIN_DIRS`` where it is located. + +For more information see the :doc:`specification of the API <../specs/Server-side-scripting>` + +Invocation +------------ + +Server side scripts are triggered by sending a POST to the `/scripting` interface (see [API scripting](path??). There are the following arguments that can be provided: + +- `call`: the name of the script to be called +- `-pN`: positional arguments (e.g. `-p0`, `-p1` etc.) +- `-ONAME`: named arguments (e.g. `-Otest`, `-Onumber` etc.) + +The arguments will be passed to the script. + +An invocation via a button in javascript could look like: + +.. code-block:: javascript + var _make_sss_button = function (entity) { + const script = "script.py"; + + const scripting_form = $(` + <form class="btn-group-xs ${_css_class_export_button}" + method="POST" + action="/scripting"> + <input type="hidden" name="call" value="${script}"/> + <input name="-p0" value=""/> + <button type="submit" class="btn btn-link">Start script</button> + </form>`); + return scripting_form[0]; + } + +For more information see the :doc:`specification of the API <../specs/Server-side-scripting>` + +Environment +------------ + +The script is called with several special environment variables to accommodate +for its special location. + +`HOME` +^^^^^^^^^^^^ +To be able to run with reduced privileges, the script has its `HOME` environment +variable set to a special directory with write access. This directory will be +deleted after the script has terminated. Its content is freshly copied for each +script invocation from a skeleton directory, located in the server directory, in +`scripting/home/`. By default, this directory contains the following: + +- `readme.md` :: A small text file describing the purpose of the directory. + +Users of CaosDB are invited to populate the directory with whatever their +scripts need. diff --git a/src/doc/specs/Server-side-scripting.md b/src/doc/specs/Server-side-scripting.md new file mode 100644 index 0000000000000000000000000000000000000000..49f56af66a6b65855e36de12144955b2925a4f51 --- /dev/null +++ b/src/doc/specs/Server-side-scripting.md @@ -0,0 +1,187 @@ +# Server-Side Scripting API (v0.1) + +The CaosDB Server can execute scripts (bash/python/perl) and compiled executables. The scripts can be invoked by a remote-procedure-call (RPC) protocol. Both, the requirements for the scripts and the RPC are described in this document. + +## Configuration of the Server + +* The CaosDB Server has two relevant properties: + 1. `SERVER_SIDE_SCRIPTING_BIN_DIR` is the directory where executable scripts are to be placed. The server will not execute scripts which are out side of this directory. This directory must be readable and executable for the server. But it should not be readable or executable for anyone else. + 2. `SERVER_SIDE_SCRIPTING_WORKING_DIR` is the directory under which the server creates temporary working directories. The server needs writing, reading and executing permissions. The temporary working directories are deleted after the scripts have finished and the server has collected the results of the scripts. + +## Installing a Server-Side Script (SSS) + +* Put your script into the `SERVER_SIDE_SCRIPTING_BIN_DIR` or in any subdirectory and make it executable. Executable files below this directory are called `SSS`. +* All other files in that directory MUST be ignored by the server, i.e. the server will never call them directly. + However, they MAY contain additional data, different implementations, libraries etc. +* A symlink pointing to an executable MUST be treated as SSS, too. + +### Example SERVER\_SIDE\_SCRIPTING\_BIN\_DIR + +``` +/ ++- script1.py (EXECUTABLE) ++- script2.sh (EXECUTABLE) ++- subdir1/ + +-script3.pl (EXECUTABLE) ++- subdir2/ + +- data_for_script2 + +- another_script.py (EXECUTABLE) ++- script4 -> ./subdir2/another_script.py (SIMLINK to EXECUTABLE) ++- script5 -> /usr/local/bin/external (SIMLINK to EXECUTABLE) +``` + +The files `scripts1.py`, `scripts2.sh`, `scripts3.pl` and `another_script.py` are SSS. +Also the `script4` can be called which would result in calling `another_script.py` +The script5 points to an executable which is not stored in the `SSD_DIR`. + +## Calling a Server-Side Script Via Remote Procedure Call + +* Users can invoke scripts via HTTP under the uri `https://$HOST:$HTTPS_PORT/$CONTEXT_ROOT/scripting`. The server accepts POST requests with content types `application/x-www-form-urlencoded` and `multipart/form-data`. +* There are 6 types of form fields which are processed by the server: + 1. A single parameter form field with name="call" (the path to script, relative to the `SERVER_SIDE_SCRIPTING_BIN_DIR`). If there are more than one fields with that name, the behavior is not defined. + 2. Zero or more parameter form fields with a unique name which starts with the string `-O` (command line options). + 3. Zero or more parameter form fields with a unique name which starts with the string `-p` (positional command line arguments). + 4. Zero or more file form fields which have a unique field name and a unique file name (upload files). If the field names or file names are not unique, the behavior is not defined. + 5. A parameter form field with name="timeout" (the request timeout) + 6. A parameter form field with name="auth-token" and a valid CaosDB AuthToken string or "generate" as value. + +## How Does the Server Call the Script? +* The server executes the script in a temporary working directory, called *PWD* (which will be deleted afterwards). +* The form parameter `call`, the options, the arguments and file fields construct the string which is executed in a shell. The value of *call* begins the command line. +* For any option paramter with the name `-Ooption_name` and a value `option_value` a resulting `--option_name=option_value` is appended to the commandLine in no particular order. +* The values of the positional arguments are appended sorted alphabetically by their field name. E.g. a parameter `-p0` with value `v0` and a parameter `-p1` with value `v1` result in appending `v0 v1` to the command line. +* All files will be loaded into the directory `$PWD/.upload_files` with their file name (i.e. the form field property). If the file names contain slashes '/' they will build sub-directories in the `./upload_files`. +* If a file form field has a field name which begins with either `-p` or `-O` the file name with prefix `.upload_files/` is passed as value of an option or as a positional argument to the script. Thus it is possible to distinguish several uploaded files from one another. +* If there is a "auth-token" field present, another command line options `--auth-token=...` is appended. The value is either the string which was submitted with the POST request, or, if the value was "generate", a refreshed, valid AuthToken which authenticates as the user of the request (why? see below). + +### Example HTML Form + +```html +<form action="/scripting" method="post" enctype="multipart/form-data"> +<input type="hidden" name="call" value="my/script.py"/> +<input type="file" name="-Oconfig-file"/> +<input type="file" name="-p1"/> +<input type="text" name="-p0" value="analyze"/> +<input type="text" name="user"/> +<input type="text" name="-Oalgorithm" value="fast"/> +<input type="submit" value="Submit"> +</form> +``` + +where the user uploads `my.conf` as `-Oconfig-file` and `my.input.tsv` as `-p1`, would result in this command line: + +``` +$SERVER_SIDE_SCRIPTING_BIN_DIR/my/script.py --config-file=.upload_files/my.conf --algorithm=fast analyze .upload_files/my.input.tsv +``` +## CaosDB Server Response + +The CaosDB Server responds with an xml document. The root element is the usual `/Response`. If no errors occurred (which would be represented with `/Response/Error` elements) the result of the script execution is represented as a `/Response/script/` element. + +* It has a `code` attribute which contains the exit code value of the execution. +* It has `stdout` and `stderr` children which contain the dump of the stdout and stderr file of the execution environment. +* It has a `call` child which contains the command line which was executed (but without a possible `--auth-token` option and with a relative path to the executable). + +### Example XML Response + +```xml +<Response> +<script code="0"> +<call>my/script.py --config-file=.upload_files/my.conf --algorithm=fast analyze .upload_files/my.input.tsv</call> +<stdout>Result: 0.5</stdout> +<stderr>Warning: 8 Lines did not contain enough columns</stderr> +</Response> +``` + +## CaosDB Clients and Authentication Token + +A special use case for server side scripting is the automated execution of CaosDB clients. These clients need to connect to the CaosDB Server and thus need a way to authenticate themselves. +For this special case the server can pass an Authentication Token which can be used by the script to authenticate itself. If the invocation request send a particulare AuthToken in the form, this AuthToken will be passed to the script with the `--auth-token` option. Otherwise, if the `auth-token` field has "generate" as value, a fresh AuthToken is generated which belongs to the user who requested the script execution. +Thus the script is executed (and connects back to the server) as the user who called the script in the first place. + +A CaosDB client might use the python client library to connect to the server with that AuthToken. + +### Example Script + +```python +#!/usr/bin/env python3 +# -*- coding: utf-8 -*- +# +# ** header v3.0 +# This file is a part of the CaosDB Project. +# +# Copyright (C) 2018 Research Group Biomedical Physics, +# Max-Planck-Institute for Dynamics and Self-Organization Göttingen +# +# This program is free software: you can redistribute it and/or modify +# it under the terms of the GNU Affero General Public License as +# published by the Free Software Foundation, either version 3 of the +# License, or (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU Affero General Public License for more details. +# +# You should have received a copy of the GNU Affero General Public License +# along with this program. If not, see <https://www.gnu.org/licenses/>. +# +# ** end header +# +"""server_side_script.py. + +An example which implements a minimal server-side script. + +1) This script expects to find a *.txt file in the .upload_files dir which is +printed to stdout. + +2) It executes a "Count stars" query and prints the result to stdout. + +3) It will return with code 0 if everything is ok, or with any code that is +specified with the commandline option --exit +""" + +import sys +from os import listdir +from caosdb import configure_connection, execute_query + + +# parse --auth-token option and configure connection +CODE = 0 +QUERY = "COUNT stars" +for arg in sys.argv: + if arg.startswith("--auth-token="): + auth_token = arg[13:] + configure_connection(auth_token=auth_token) + if arg.startswith("--exit="): + CODE = int(arg[7:]) + if arg.startswith("--query="): + QUERY = arg[8:] + + +############################################################ +# 1 # find and print *.txt file ############################ +############################################################ + +try: + for fname in listdir(".upload_files"): + if fname.endswith(".txt"): + with open(".upload_files/{}".format(fname)) as f: + print(f.read()) +except FileNotFoundError: + pass + + +############################################################ +# 2 # query "COUNT stars" ################################## +############################################################ + +RESULT = execute_query(QUERY) +print(RESULT) + +############################################################ +# 3 ######################################################## +############################################################ + +sys.exit(CODE) +``` +