Skip to content
Snippets Groups Projects
Verified Commit 647dc4b9 authored by Timm Fitschen's avatar Timm Fitschen
Browse files

Merge branch 'f-filesystem-cleanup' into f-filesystem-core

parents 83bb9d88 a6253517
No related branches found
No related tags found
1 merge request!75Draft: ENH: file system: core
Pipeline #35636 failed
Showing with 377 additions and 125 deletions
...@@ -39,6 +39,10 @@ variables: ...@@ -39,6 +39,10 @@ variables:
CPPINT: "" CPPINT: ""
MYSQLBACKEND: "" MYSQLBACKEND: ""
workflow:
rules:
- if: $CI_PIPELINE_SOURCE != "merge_request_event" && $CI_COMMIT_REF_NAME != $CI_COMMIT_TAG
image: $CI_REGISTRY_IMAGE image: $CI_REGISTRY_IMAGE
stages: stages:
- info - info
...@@ -70,8 +74,11 @@ build-testenv: ...@@ -70,8 +74,11 @@ build-testenv:
image: docker:20.10 image: docker:20.10
stage: setup stage: setup
timeout: 3h timeout: 3h
only: rules:
- schedules - if: $CI_PIPELINE_SOURCE == "schedule"
- if: $CI_PIPELINE_SOURCE != "schedule"
changes:
- src/test/docker/Dockerfile
script: script:
- cd src/test/docker - cd src/test/docker
- docker login -u gitlab-ci-token -p $CI_JOB_TOKEN $CI_REGISTRY - docker login -u gitlab-ci-token -p $CI_JOB_TOKEN $CI_REGISTRY
...@@ -136,9 +143,8 @@ trigger_inttest: ...@@ -136,9 +143,8 @@ trigger_inttest:
pages_prepare: &pages_prepare pages_prepare: &pages_prepare
tags: [ cached-dind ] tags: [ cached-dind ]
stage: deploy stage: deploy
only: rules:
refs: - if: $CI_COMMIT_REF_NAME =~ /^release-.*$/i
- /^release-.*$/i
script: script:
- echo "Deploying..." - echo "Deploying..."
- make doc - make doc
...@@ -148,6 +154,5 @@ pages_prepare: &pages_prepare ...@@ -148,6 +154,5 @@ pages_prepare: &pages_prepare
- public - public
pages: pages:
<<: *pages_prepare <<: *pages_prepare
only: rules:
refs: - if: $CI_COMMIT_REF_NAME == 'main'
- main
## Summary
*Please give a short summary of what the issue is.*
## Expected Behavior
*What did you expect how the software should behave?*
## Actual Behavior
*What did the software actually do?*
## Steps to Reproduce the Problem
*Please describe, step by step, how others can reproduce the problem. Please try these steps for yourself on a clean system.*
1.
2.
3.
## Specifications
- Version: *Which version of this software?*
- Platform: *Which operating system, which other relevant software versions?*
## Possible fixes
*Do you have ideas how the issue can be resolved?*
# Summary
*Insert a meaningful description for this merge request here: What is the new/changed behavior?
Which bug has been fixed? Are there related issues?*
# Focus
*Point the reviewer to the core of the code change. Where should they start reading? What should
they focus on (e.g. security, performance, maintainability, user-friendliness, compliance with the
specs, finding more corner cases, concrete questions)?*
# Test Environment
*How to set up a test environment for manual testing?*
# Check List for the Author
Please, prepare your MR for a review. Be sure to write a summary and a focus and create gitlab
comments for the reviewer. They should guide the reviewer through the changes, explain your changes
and also point out open questions. For further good practices have a look at [our review
guidelines](https://gitlab.com/caosdb/caosdb/-/blob/dev/REVIEW_GUIDELINES.md)
- [ ] All automated tests pass
- [ ] Reference related issues
- [ ] Up-to-date CHANGELOG.md (or not necessary)
- [ ] Up-to-date JSON schema (or not necessary)
- [ ] Appropriate user and developer documentation (or not necessary)
- How do I use the software? Assume "stupid" users.
- How do I develop or debug the software? Assume novice developers.
- [ ] Annotations in code (Gitlab comments)
- Intent of new code
- Problems with old code
- Why this implementation?
# Check List for the Reviewer
- [ ] I understand the intent of this MR
- [ ] All automated tests pass
- [ ] Up-to-date CHANGELOG.md (or not necessary)
- [ ] Appropriate user and developer documentation (or not necessary)
- [ ] The test environment setup works and the intended behavior is reproducible in the test
environment
- [ ] In-code documentation and comments are up-to-date.
- [ ] Check: Are there specifications? Are they satisfied?
For further good practices have a look at [our review guidelines](https://gitlab.com/caosdb/caosdb/-/blob/dev/REVIEW_GUIDELINES.md).
/assign me
/target_branch dev
...@@ -5,23 +5,59 @@ All notable changes to this project will be documented in this file. ...@@ -5,23 +5,59 @@ All notable changes to this project will be documented in this file.
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
## [Unreleased] ## [Unreleased] ##
### Added ### Added ###
* SELECT query support for the GRPC API ### Changed ###
### Changed * The default behavior of the query `FIND SomeName [...]` (as well as COUNT and SELECT) is being
made configurable and changes:
* `FIND SomeName` will be interpreted as `FIND <FIND_QUERY_DEFAULT_ROLE>
SomeName` from now on where `FIND_QUERY_DEFAULT_ROLE` is a newly introduced
server property.
* The new `FIND_QUERY_DEFAULT_ROLE` server property defaults to `RECORD`
which is why the behavior of the server api has a **breaking change**.
* The semantics of `FIND *` are affected as well. `FIND *` is equivalent to
`FIND <FIND_QUERY_DEFAULT_ROLE>`.
* Of course, administrators can choose to retain the old behavior by setting
`FIND_QUERY_DEFAULT_ROLE=ENTITY`.
* CQL now treats `WITH` and `WITH A` equivalently. Issue: [#192](https://gitlab.com/caosdb/caosdb-server/-/issues/192)
* The InsertFilesInDir FlagJob now creates File entities without a name. The previous behavior
caused severe performance problems for very large numbers of files. Issue: [#197](https://gitlab.com/caosdb/caosdb-server/-/issues/197)
### Deprecated ### Deprecated ###
### Removed ### Removed ###
### Fixed ### Fixed ###
### Security * Denying a role permission has no effect
[#196](https://gitlab.com/caosdb/caosdb-server/-/issues/196). See security
notes below.
* Missing RecordType leads to unexpected server error
[#166](https://gitlab.com/caosdb/caosdb-server/-/issues/166)
### Documentation ### Security ###
* Fixed [#196](https://gitlab.com/caosdb/caosdb-server/-/issues/196). This was
an error in the authorization procedure which allowed unprivileged users
execute insert, update or delete transactions on entities. However, the
unprivileged users would also need the correct entity permissions to do that.
Without backup, this means possible data loss. Also there was the possibility
to spam the database by creating unwanted entities.
### Documentation ###
- Nested queries.
- Global entity permissions.
## [0.9.0] - 2023-01-19
### Added
* SELECT query support for the GRPC API
## [0.8.1] - 2022-11-07 ## [0.8.1] - 2022-11-07
(Timm Fitschen) (Timm Fitschen)
......
cff-version: 1.2.0
message: "If you use this software, please cite it as below."
authors:
- family-names: Fitschen
given-names: Timm
orcid: https://orcid.org/0000-0002-4022-432X
- family-names: Schlemmer
given-names: Alexander
orcid: https://orcid.org/0000-0003-4124-9649
- family-names: Hornung
given-names: Daniel
orcid: https://orcid.org/0000-0002-7846-6375
- family-names: tom Wörden
given-names: Henrik
orcid: https://orcid.org/0000-0002-5549-578X
- family-names: Spreckelsen
given-names: Florian
orcid: https://orcid.org/0000-0002-6856-2910
- family-names: Parlitz
given-names: Ulrich
orcid: https://orcid.org/0000-0003-3058-1435
- family-names: Luther
given-names: Stefan
orcid: https://orcid.org/0000-0001-7214-8125
title: "CaosDB - Server"
version: 0.8.1
doi: 10.3390/data4020083
date-released: 2022-11-07
...@@ -2,11 +2,12 @@ ...@@ -2,11 +2,12 @@
## For Building and Running the Server ## For Building and Running the Server
* `>=caosdb-proto 0.2.0` * `>=caosdb-proto 0.3.0`
* `>=caosdb-mysqlbackend 5.0.0` * `>=caosdb-mysqlbackend 5.0.0`
* `>=Java 11` * `>=Java 11`
* `>=Apache Maven 3.6.0` * `>=Apache Maven 3.6.0`
* `>=Make 4.2` * `>=Make 4.2`
* `>=gcc 8` (if PAM authentication is required)
* `libpam` (if PAM authentication is required) * `libpam` (if PAM authentication is required)
* More dependencies are being pulled and installed automatically by Maven. See the complete list of dependencies in the [pom.xml](pom.xml) * More dependencies are being pulled and installed automatically by Maven. See the complete list of dependencies in the [pom.xml](pom.xml)
......
# Features # Features
* The CaosDB Server implements a CaosDB GRPC API Endpoint (v0.2.0) * The CaosDB Server implements a CaosDB GRPC API Endpoint (v0.3.0)
Authentication is supported via a Basic scheme, using the well-known Authentication is supported via a Basic scheme, using the well-known
"authentication" header. "authentication" header.
Notable limitations of the current implementation of the API: Notable limitations of the current implementation of the API:
......
...@@ -15,12 +15,18 @@ See [DEPENDENCIES.md](DEPENDENCIES.md). ...@@ -15,12 +15,18 @@ See [DEPENDENCIES.md](DEPENDENCIES.md).
On Debian, the required packages can be installed with: On Debian, the required packages can be installed with:
apt-get install git make mariadb-server maven openjdk-11-jdk-headless \ apt-get install make mariadb-server maven openjdk-11-jdk-headless \
python3-pip screen libpam0g-dev unzip python3-pip libpam0g-dev unzip
Note that installing MariaDB will uninstall existing MySQL packages and vice Note that installing MariaDB will uninstall existing MySQL packages and vice
versa. versa.
#### Install the requirements on Fedora
On Fedora, the required packages can be installed with:
sudo dnf install make pam-devel mariadb-server mariadb python3 java-17-openjdk-headless unzip gcc
### System ### System
* `>=Linux 4.0.0`, `x86_64`, e.g. Ubuntu 18.04 * `>=Linux 4.0.0`, `x86_64`, e.g. Ubuntu 18.04
...@@ -160,20 +166,47 @@ sources (if you called `make run` previously). ...@@ -160,20 +166,47 @@ sources (if you called `make run` previously).
## Setup Eclipse ## Setup Eclipse
1. Open Eclipse (recommended version: Oxygen.1a Release (4.7.1a)) 1. Open Eclipse (tested with 2022-R12)
2. `File > New > Java Project`: Choose a project name and specify the location 2. File > Import > Maven > Existing Maven Projects: Specify location.
of this repo. The JRE and Project layout should be configured automatically. 3. You will most likely encounter "Plugin execution not covered by lifecycle
Now, the project should initially have two source-folders: `./src/main/java` configuration: ..." errors. Adapt the file
and `./src/test/java`. After a build, another one, `<eclipse-workspace>/.metadata/.plugins/org.eclipse.m2e.core/lifecycle-mapping-metadata.xml`.
`./target/generated-sources/antlr4` should be generated. If there are more
than these three source-folders, reconfigure the projects source folders Example:
appropriately with `Project > Properties > Java Build Path > Source`.
3. In the `Package Explorer` view, right-click on the project and `Configure > ```xml
Convert to Maven Project`. <?xml version="1.0" encoding="UTF-8"?>
4. In the `Package Explorer` view, right-click on the project and `Maven > <lifecycleMappingMetadata>
Update Project`. <pluginExecutions>
5. Usually a build of the project is started automatically. Otherwise `Project > <pluginExecution>
Build Project`. <pluginExecutionFilter>
<groupId>com.coveo</groupId>
<artifactId>fmt-maven-plugin</artifactId>
<versionRange>2.5.1</versionRange>
<goals>
<goal>format</goal>
</goals>
</pluginExecutionFilter>
<action>
<ignore />
</action>
</pluginExecution>
<pluginExecution>
<pluginExecutionFilter>
<groupId>org.codehaus.mojo</groupId>
<artifactId>buildnumber-maven-plugin</artifactId>
<versionRange>1.4</versionRange>
<goals>
<goal>create-metadata</goal>
</goals>
</pluginExecutionFilter>
<action>
<ignore />
</action>
</pluginExecution>
</pluginExecutions>
</lifecycleMappingMetadata>
```
Done! Done!
...@@ -211,7 +244,7 @@ Stand-alone documentation is built using Sphinx: `make doc` ...@@ -211,7 +244,7 @@ Stand-alone documentation is built using Sphinx: `make doc`
(`l_` not found): (`l_` not found):
```sh ```sh
git clone git@github.com:simgrid/javasphinx.git git clone https://github.com/simgrid/javasphinx.git
cd javasphinx cd javasphinx
git checkout 659209069603a git checkout 659209069603a
pip3 install . pip3 install .
......
...@@ -22,6 +22,7 @@ guidelines of the CaosDB Project ...@@ -22,6 +22,7 @@ guidelines of the CaosDB Project
* [pom.xml](./pom.xml) (probably this means to remove the `-SNAPSHOT`) * [pom.xml](./pom.xml) (probably this means to remove the `-SNAPSHOT`)
* `src/doc/conf.py` * `src/doc/conf.py`
* `CHANGELOG.md` * `CHANGELOG.md`
* `CITATION.cff` (update version and date)
5. Merge the release branch into the main branch. 5. Merge the release branch into the main branch.
......
caosdb-proto @ c6405e53
Subproject commit bbab921f9aff5ac1945b299973e2b2acac75128a Subproject commit c6405e538c179d2a8af952f85d9e9dc51fbadb92
caosdb-webui @ 421b2dce
Subproject commit bc199886b78a5dd39eb3bccd3a1451c8ab1e7552 Subproject commit 421b2dce5199a5a2c96bcc638543c1fd51d48870
...@@ -64,7 +64,7 @@ MYSQL_DATABASE_NAME=caosdb ...@@ -64,7 +64,7 @@ MYSQL_DATABASE_NAME=caosdb
# User name for connecting to mysql # User name for connecting to mysql
MYSQL_USER_NAME=caosdb MYSQL_USER_NAME=caosdb
# Password for the user # Password for the user
MYSQL_USER_PASSWORD=caosdb MYSQL_USER_PASSWORD=random1234
# Schema of mysql procedures and tables which is required by this CaosDB instance # Schema of mysql procedures and tables which is required by this CaosDB instance
MYSQL_SCHEMA_VERSION=v6.0-SNAPSHOT MYSQL_SCHEMA_VERSION=v6.0-SNAPSHOT
...@@ -204,6 +204,15 @@ USER_NAME_INVALID_MESSAGE=User names must have a length from 1 to 32 characters. ...@@ -204,6 +204,15 @@ USER_NAME_INVALID_MESSAGE=User names must have a length from 1 to 32 characters.
PASSWORD_VALID_REGEX=^((?=.*\\d)(?=.*[a-z])(?=.*[A-Z])(?=.*[\\p{Punct}]).{8,128})$ PASSWORD_VALID_REGEX=^((?=.*\\d)(?=.*[a-z])(?=.*[A-Z])(?=.*[\\p{Punct}]).{8,128})$
PASSWORD_INVALID_MESSAGE=Passwords must have a length from 8 to 128 characters. THe must contain at least one [A-Z], one [a-z], one [0-9] and a special character e.g. [!§$%&/()=?.;,:#+*+~]. PASSWORD_INVALID_MESSAGE=Passwords must have a length from 8 to 128 characters. THe must contain at least one [A-Z], one [a-z], one [0-9] and a special character e.g. [!§$%&/()=?.;,:#+*+~].
# --------------------------------------------------
# Query Settings
# --------------------------------------------------
# FIND blablabla is short for FIND $FIND_QUERY_DEFAULT_ROLE blablabla. Should be
# either ENTITY or RECORD. RECORDTYPE, FILE, and PROPERTY will work as well but
# are rather unexpected for human users.
FIND_QUERY_DEFAULT_ROLE=RECORD
# -------------------------------------------------- # --------------------------------------------------
# Extensions # Extensions
# -------------------------------------------------- # --------------------------------------------------
......
...@@ -219,6 +219,12 @@ The following query returns entities which have a _pname1_ property with any val ...@@ -219,6 +219,12 @@ The following query returns entities which have a _pname1_ property with any val
`FIND ename WITH pname1` `FIND ename WITH pname1`
`FIND ename WITH A pname1`
`FIND ename WITH A PROPERTY pname1`
`FIND ename WITH PROPERTY pname1`
`FIND ename . pname1` `FIND ename . pname1`
`FIND ename.pname1` `FIND ename.pname1`
...@@ -338,7 +344,7 @@ Any result set can be filtered by logically combining POV filters or back refere ...@@ -338,7 +344,7 @@ Any result set can be filtered by logically combining POV filters or back refere
* NOT:: The logical negation. Equivalent expressions: `NOT, DOESN'T HAVE A PROPERTY, DOES NOT HAVE A PROPERTY, DOESN'T HAVE A, DOES NOT HAVE A, DOES NOT, DOESN'T, IS NOT, ISN'T, !` * NOT:: The logical negation. Equivalent expressions: `NOT, DOESN'T HAVE A PROPERTY, DOES NOT HAVE A PROPERTY, DOESN'T HAVE A, DOES NOT HAVE A, DOES NOT, DOESN'T, IS NOT, ISN'T, !`
* OR:: The logical _or_. Equivalent expressions: `OR, |` * OR:: The logical _or_. Equivalent expressions: `OR, |`
* RECORD,RECORDTYPE,FILE,PROPERTY:: Role expression for restricting the result set to a specific role. * RECORD,RECORDTYPE,FILE,PROPERTY:: Role expression for restricting the result set to a specific role.
* WHICH:: The marker for the beginning of the filters. Equivalent expressions: `WHICH, WHICH HAS A, WHICH HAS A PROPERTY, WHERE, WITH, .` * WHICH:: The marker for the beginning of the filters. Equivalent expressions: `WHICH, WHICH HAS A, WHICH HAS A PROPERTY, WHERE, WITH (A), .`
* REFERENCE:: This one is tricky: `REFERENCE TO` expresses a the state of _having_ a reference property. `REFERENCED BY` expresses the state of _being_ referenced by another entity. * REFERENCE:: This one is tricky: `REFERENCE TO` expresses a the state of _having_ a reference property. `REFERENCED BY` expresses the state of _being_ referenced by another entity.
* COUNT:: `COUNT` works like `FIND` but doesn't return the entities. * COUNT:: `COUNT` works like `FIND` but doesn't return the entities.
......
...@@ -25,7 +25,7 @@ ...@@ -25,7 +25,7 @@
<modelVersion>4.0.0</modelVersion> <modelVersion>4.0.0</modelVersion>
<groupId>org.caosdb</groupId> <groupId>org.caosdb</groupId>
<artifactId>caosdb-server</artifactId> <artifactId>caosdb-server</artifactId>
<version>0.9.0-SNAPSHOT</version> <version>0.9.1-SNAPSHOT</version>
<packaging>jar</packaging> <packaging>jar</packaging>
<name>CaosDB Server</name> <name>CaosDB Server</name>
<scm> <scm>
......
# CaosDB Query Language # CaosDB Query Language Examples
**WIP This is going to be the specification. CQL tutorials are in the webui**
## Example queries See syntax specification in [CaosDB Query Language Syntax](query-syntax).
### Simple FIND Query ## Simple FIND Query
The following query will return any entity which has the name _ename_ and all its children.
`FIND ename`
The following queries are equivalent and will return any entity which has the name _ename_ and all its children, but only if they are genuin records. Of course, the returned set of entities (henceforth referred to as _resultset_) can also be restricted to recordtypes, properties and files. The following query will return any record which has the name _somename_ and all
record children of any entity with that name.
`FIND RECORD ename` `FIND somename`
`FIND RECORDS ename` On server in the default configuration, the following queries are equivalent to this one.
Wildcards use `*` for any characters or none at all. Wildcards for single characters (like the '_' wildcard from mysql) are not implemented yet. `FIND RECORD somename`
`FIND RECORD en*` returns any entity which has a name beginning with _en_. `FIND RECORDS somename`
Regular expressions must be surrounded by _<<_ and '>>': Of course, the returned set of entities (henceforth referred to as _resultset_) can also be
restricted to RecordTypes (`FIND RECORDTYPE ...`), Properties (`FIND PROPERTY ...`) and Files (`FIND
FILE ...`).
`FIND RECORD <<e[aemn]{2,5}>>` You can include all entities (Records, RecordTypes, Properties, ...) into the results by using the
`ENTITY` keyword:
`FIND RECORD <<[cC]am_[0-9]*>>` `FIND ENTITY somename`
*TODO* (Timm): Wildcards use `*` for any characters or none at all. Wildcards for single characters (like the `_` wildcard from mysql) are not implemented yet.
Describe escape sequences like `\\ `, `\*`, `\<<` and `\>>`.
Currently, wildcards and regular expressions are only available for the _simple-find-part_ of the query, i. e. no wildcards/regexps for filters. `FIND en*` returns any record which has a name beginning with _en_.
### Simple COUNT Query Regular expressions must be surrounded by _<<_ and _>>_:
This query counts entities which have certain properties. `FIND <<e[aemn]{2,5}>>`
`COUNT ename` `FIND <<[cC]amera_[0-9]*>>`
will return the number of entities which have the name _ename_ and all their children.
The syntax of the COUNT queries is equivalent to the FIND queries in any respect (this also applies to wildcards and regular expressions) but one: The prefix is to be `COUNT` instead of `FIND`. *TODO*:
Describe escape sequences like `\\\\ `, `\*`, `\<<` and `\>>`.
Unlike the FIND queries, the COUNT queries do not return any entities. The result of the query is the number of entities which _would be_ returned if the query was a FIND query. Currently, wildcards and regular expressions are only available for the _simple-find-part_ of the
query, i.e. not for property-operator-value filters (see below).
## Simple COUNT Query
COUNT queries count entities which have certain properties.
`COUNT ... rname ...`
will return the number of records which have the name _rname_ and all record
children of any entity with that name.
The syntax of the COUNT queries is equivalent to the FIND queries in any
respect (this also applies to wildcards and regular expressions) but one: The
prefix is to be `COUNT` instead of `FIND`.
Unlike the FIND queries, the COUNT queries do not return any entities. The result of the query is
the number of entities which _would be_ returned if the query was a FIND query.
## Filters ## Filters
### POV - Property-Operator-Value ### POV - Property-Operator-Value
The following queries are equivalent and will restrict the result set to entities which have a property named _pname1_ that has a value _val1_. The following queries are equivalent and will restrict the result set to records which have a property named _pname1_ that has a value _val1_.
`FIND ename.pname1=val1` `FIND ename.pname1=val1`
...@@ -53,31 +70,33 @@ The following queries are equivalent and will restrict the result set to entitie ...@@ -53,31 +70,33 @@ The following queries are equivalent and will restrict the result set to entitie
`FIND ename WHICH HAS A pname1=val1` `FIND ename WHICH HAS A pname1=val1`
Again, the resultset can be restricted to records: Again, the resultset can be restricted to any other entity role as well:
`FIND RECORD ename WHICH HAS A pname1=val1` `FIND RECORDTYPE ename WHICH HAS A pname1=val1`
_currently known operators:_ `=, !=, <=, <, >=, >` (and cf. next paragraphes!) _currently known operators:_ `=, !=, <=, <, >=, >` (and cf. next paragraphes!)
#### Special Operator: LIKE #### Special Operator: LIKE
The _LIKE_ can be used with wildcards. The `*` is a wildcard for any (possibly empty) sequence of characters. Examples: The _LIKE_ can be used with wildcards. The `*` is a wildcard for any (possibly empty) sequence of characters.
Examples:
`FIND RECORD ename WHICH HAS A pname1 LIKE va*` `FIND ename WHICH HAS A pname1 LIKE va*`
`FIND RECORD ename WHICH HAS A pname1 LIKE va*1` `FIND ename WHICH HAS A pname1 LIKE va*1`
`FIND RECORD ename WHICH HAS A pname1 LIKE *al1` `FIND ename WHICH HAS A pname1 LIKE *al1`
_Note:_ The _LIKE_ operator is will only produce expectable results with text properties. _Note:_ The _LIKE_ operator will only produce expectable results with text properties.
#### Special Case: References #### Special Case: References
In general a reference can be addressed just like a POV filter. So In general a reference can be addressed just like a POV filter. So
`FIND ename1.pname1=ename2` `FIND ename1 WITH pname1=ename2`
will also return any entity named _ename1_ which references the entity with name or id _ename2_ via a reference property named _pname1_. However, it will also return any entity with a text property of that name with the string value _ename2_. In order to restrict the result set to reference properties one may make use of special reference operators: will also return any record named _ename1_ which references the entity with name or id _ename2_ via a reference property named _pname1_. However, it will also return any entity with a text property of that name with the string value _ename2_. In order to restrict the result set to reference properties one may make use of special reference operators:
_reference operators:_ `->, REFERENCES, REFERENCE TO` _reference operators:_ `->, REFERENCES, REFERENCE TO`
...@@ -88,7 +107,7 @@ The query looks like this: ...@@ -88,7 +107,7 @@ The query looks like this:
`FIND ename1 WHICH HAS A pname1->ename2` `FIND ename1 WHICH HAS A pname1->ename2`
#### Time Special Case: DateTime #### Special Case: DateTime
_DateTime operators:_ `=, !=, <, >, IN, NOT IN` _DateTime operators:_ `=, !=, <, >, IN, NOT IN`
...@@ -116,15 +135,15 @@ Examples: ...@@ -116,15 +135,15 @@ Examples:
* `2015-04-03T00:00:00.0!=2015-04-03T00:00:00.0` is false. * `2015-04-03T00:00:00.0!=2015-04-03T00:00:00.0` is false.
* `2015-04-03T00:00:00!=2015-04-03T00:00:00` is false. * `2015-04-03T00:00:00!=2015-04-03T00:00:00` is false.
* `2015-04!=2015-05` is true. * `2015-04!=2015-05` is true.
* `2015-04!=2015-04` is false. * `2015-04!=2015-04` is false
##### `d1>d2`: Transitive, non-symmetric relation. ##### `d1>d2`: Transitive, non-symmetric relation.
Semantics depend on the flavors of d1 and d2. If both are... Semantics depend on the flavors of d1 and d2. If both are...
###### [UTCDateTime](Datatype#datetime) ###### [UTCDateTime](specification/Datatype.html#datetime)
* ''True'' iff the time of d1 is after the the time of d2 according to [UTC](https://en.wikipedia.org/wiki/Coordinated_Universal_Time). * ''True'' iff the time of d1 is after the the time of d2 according to [UTC](https://en.wikipedia.org/wiki/Coordinated_Universal_Time).
* ''False'' otherwise. * ''False'' otherwise.
###### [SemiCompleteDateTime](Datatype#datetime) ###### [SemiCompleteDateTime](specification/Datatype.html#datetime)
* ''True'' iff `d1.ILB>d2.EUB` is true or `d1.ILB=d2.EUB` is true. * ''True'' iff `d1.ILB>d2.EUB` is true or `d1.ILB=d2.EUB` is true.
* ''False'' iff `d1.EUB<d2.ILB}} is true or {{{d1.EUB=d2.ILB` is true. * ''False'' iff `d1.EUB<d2.ILB}} is true or {{{d1.EUB=d2.ILB` is true.
* ''Undefined'' otherwise. * ''Undefined'' otherwise.
...@@ -142,11 +161,11 @@ Examples: ...@@ -142,11 +161,11 @@ Examples:
##### `d1<d2`: Transitive, non-symmetric relation. ##### `d1<d2`: Transitive, non-symmetric relation.
Semantics depend on the flavors of d1 and d2. If both are... Semantics depend on the flavors of d1 and d2. If both are...
###### [UTCDateTime](Datatype#datetime) ###### [UTCDateTime](specification/Datatype.html#datetime)
* ''True'' iff the time of d1 is before the the time of d2 according to [UTC](https://en.wikipedia.org/wiki/Coordinated_Universal_Time) * ''True'' iff the time of d1 is before the the time of d2 according to [UTC](https://en.wikipedia.org/wiki/Coordinated_Universal_Time)
* ''False'' otherwise. * ''False'' otherwise.
###### [SemiCompleteDateTime](Datatype#datetime) ###### [SemiCompleteDateTime](specification/Datatype.html#datetime)
* ''True'' iff `d1.EUB<d2.ILB` is true or `d1.EUB=d2.ILB` is true. * ''True'' iff `d1.EUB<d2.ILB` is true or `d1.EUB=d2.ILB` is true.
* ''False'' iff `d1.ILB>d2.EUB}} is true or {{{d1.ILB=d2.EUB` is true. * ''False'' iff `d1.ILB>d2.EUB}} is true or {{{d1.ILB=d2.EUB` is true.
* ''Undefined'' otherwise. * ''Undefined'' otherwise.
...@@ -164,7 +183,7 @@ Examples: ...@@ -164,7 +183,7 @@ Examples:
##### `d1 IN d2`: Transitive, non-symmetric relation. ##### `d1 IN d2`: Transitive, non-symmetric relation.
Semantics depend on the flavors of d1 and d2. If both are... Semantics depend on the flavors of d1 and d2. If both are...
###### [SemiCompleteDateTime](Datatype#datetime) ###### [SemiCompleteDateTime](specification/Datatype.html#datetime)
* ''True'' iff (`d1.ILB>d2.ILB` is true or `d1.ILB=d2.ILB` is true) and (`d1.EUB<d2.EUB` is true or `d1.EUB=d2.EUB` is true). * ''True'' iff (`d1.ILB>d2.ILB` is true or `d1.ILB=d2.ILB` is true) and (`d1.EUB<d2.EUB` is true or `d1.EUB=d2.EUB` is true).
* ''False'' otherwise. * ''False'' otherwise.
...@@ -175,9 +194,9 @@ Examples: ...@@ -175,9 +194,9 @@ Examples:
* `2015 IN 2015-01-01` is false. * `2015 IN 2015-01-01` is false.
* `2015-01-01 IN 2015-01-01T20:15:30` is false. * `2015-01-01 IN 2015-01-01T20:15:30` is false.
##### `d1 NOT IN d2`: Non-symmetric relation. ##### `d1 NOT IN d2`: Transitive, non-symmetric relation.
Semantics depend on the flavors of d1 and d2. If both are... Semantics depend on the flavors of d1 and d2. If both are...
###### [SemiCompleteDateTime](Datatype#datetime) ###### [SemiCompleteDateTime](specification/Datatype.html#datetime)
* ''True'' iff `d1.ILB IN d2.ILB` is false. * ''True'' iff `d1.ILB IN d2.ILB` is false.
* ''False'' otherwise. * ''False'' otherwise.
...@@ -193,7 +212,7 @@ These semantics follow a three-valued logic with ''true'', ''false'' and ''undef ...@@ -193,7 +212,7 @@ These semantics follow a three-valued logic with ''true'', ''false'' and ''undef
#### Omitting the Property or the Value #### Omitting the Property or the Value
One doesn't have to specify the property or the value at all. The following query filters the result set for entities which have any property with a value greater than _val1_. One doesn't have to specify the property or the value at all. The following query filters the result set for records which have any property with a value greater than _val1_.
`FIND ename WHICH HAS A PROPERTY > val1` `FIND ename WHICH HAS A PROPERTY > val1`
...@@ -213,7 +232,7 @@ And for references... ...@@ -213,7 +232,7 @@ And for references...
`FIND ename1.->ename2` `FIND ename1.->ename2`
The following query returns entities which have a _pname1_ property with any value. The following query returns records which have a _pname1_ property with any value.
`FIND ename WHICH HAS A PROPERTY pname1` `FIND ename WHICH HAS A PROPERTY pname1`
...@@ -227,22 +246,17 @@ The following query returns entities which have a _pname1_ property with any val ...@@ -227,22 +246,17 @@ The following query returns entities which have a _pname1_ property with any val
### TransactionFilter ### TransactionFilter
`FIND ename WHICH (sugar|negated_sugar)? (NOT)? (CREATED|INSERTED|UPDATED) (by_clause time_clause?| time_clause by_clause?)`
*Definition* *Definition*
sugar:: `HAS BEEN` | `HAVE BEEN` | `HAD BEEN` | `WAS` | `IS` | sugar:: `HAS BEEN` | `HAVE BEEN` | `HAD BEEN` | `WAS` | `IS`
negated_sugar:: `HAS NOT BEEN` | `HASN'T BEEN` | `WAS NOT` | `WASN'T` | `IS NOT` | `ISN'T` | `HAVN'T BEEN` | negated_sugar:: `HAS NOT BEEN` | `HASN'T BEEN` | `WAS NOT` | `WASN'T` | `IS NOT` | `ISN'T` | `HAVN'T BEEN` | `HAVE NOT BEEN` | `HADN'T BEEN` | `HAD NOT BEEN`
`HAVE NOT BEEN` | `HADN'T BEEN` | `HAD NOT BEEN`
by_clause:: `BY (ME | username | SOMEONE ELSE (BUT ME)? | SOMEONE ELSE BUT username)` by_clause:: `BY (ME | username | SOMEONE ELSE (BUT ME)? | SOMEONE ELSE BUT username)`
datetime:: A datetime string of the form `YYYY[-MM[-DD(T| )[hh[:mm[:ss[.nnn][(+|-)zzzz]]]]]]` or `TODAY`.
date:: A date string of the form `YYYY-MM-DD` time_clause:: `[AT|ON|IN|BEFORE|AFTER|UNTIL|SINCE] (datetime) `
datetime:: A datetime string of the form `YYYY-MM-DD hh:mm:ss`
time_clause:: `ON ($date|$datetime) ` Here is plenty of room for more syntactic sugar, e.g. a `TODAY` keyword, and more funcionality, e.g. ranges.
`FIND ename WHICH ($sugar|$negated_sugar)? (NOT)? (CREATED|INSERTED|UPDATED|DELETED) (by_clause time_clause?| time_clause by_clause?)`
*Examples* *Examples*
...@@ -252,12 +266,13 @@ The following query returns entities which have a _pname1_ property with any val ...@@ -252,12 +266,13 @@ The following query returns entities which have a _pname1_ property with any val
`FIND ename WHICH HAS BEEN CREATED BY erwin ON 2014-12-24` `FIND ename WHICH HAS BEEN CREATED BY erwin ON 2014-12-24`
`FIND ename WHICH HAS BEEN CREATED BY SOMEONE ELSE BUT erwin ON 2014-12-24` `FIND ename WHICH HAS BEEN UPDATED BY SOMEONE ELSE BUT erwin ON 2014-12-24`
`FIND ename WHICH HAS BEEN CREATED BY erwin` `FIND ename WHICH HAS BEEN INSERTED BY erwin`
`FIND ename . CREATED BY erwin ON ` `FIND ename WHICH HAS BEEN INSERTED SINCE 2021-04`
Note that `SINCE` and `UNTIL` are inclusive, while `BEFORE` and `AFTER` are not.
### File Location ### File Location
...@@ -297,7 +312,7 @@ Find any file in a directory which begins with `2016-02`: ...@@ -297,7 +312,7 @@ Find any file in a directory which begins with `2016-02`:
### Back References ### Back References
The back reference filters for entities that are referenced by another entity. The following query returns entities of the type _ename1_ which are referenced by _ename2_ entities via the reference property _pname1_. The back reference filters for entities that are referenced by another entity. The following query returns records of the type _ename1_ which are referenced by _ename2_ entities via the reference property _pname1_.
* `FIND ename1 WHICH IS REFERENCED BY ename2 AS A pname1` * `FIND ename1 WHICH IS REFERENCED BY ename2 AS A pname1`
* `FIND ename1 WITH @ ename2 / pname1` * `FIND ename1 WITH @ ename2 / pname1`
...@@ -310,6 +325,13 @@ One may omit the property specification: ...@@ -310,6 +325,13 @@ One may omit the property specification:
* `FIND ename1 WITH @ ename2` * `FIND ename1 WITH @ ename2`
* `FIND ename1 . @ ename2` * `FIND ename1 . @ ename2`
### Nested queries, or filtering by sub-properties ###
Nested queries can easily be searched by simply concatenating `WHICH` or `WITH` expressions:
* `FIND ename WHICH HAS A pname WHICH HAS A subpname=val`
* For example: `FIND AN experiment WHICH HAS A camera WHICH HAS A 'serial number'= 1234567890`
### Combining Filters with Propositional Logic ### Combining Filters with Propositional Logic
Any result set can be filtered by logically combining POV filters or back reference filters: Any result set can be filtered by logically combining POV filters or back reference filters:
...@@ -356,7 +378,7 @@ Any result set can be filtered by logically combining POV filters or back refere ...@@ -356,7 +378,7 @@ Any result set can be filtered by logically combining POV filters or back refere
In contrast to `FIND` queries, which always return the complete entity, there are `SELECT` queries which only return the entity with only those properties which are specified in the query. The syntax is very similar to `FIND` queries - just replace the `FIND` by `SELECT <comma separated list of selectors> FROM`: In contrast to `FIND` queries, which always return the complete entity, there are `SELECT` queries which only return the entity with only those properties which are specified in the query. The syntax is very similar to `FIND` queries - just replace the `FIND` by `SELECT <comma separated list of selectors> FROM`:
`SELECT p1, p2, p3 FROM Record ename` `SELECT p1, p2, p3 FROM ename`
However, the `SELECT` query can also return properties of referenced entities and thereby are a means of joining entities together and return a custom view or projection: However, the `SELECT` query can also return properties of referenced entities and thereby are a means of joining entities together and return a custom view or projection:
...@@ -391,14 +413,22 @@ Since Caosdb 0.2 entities are optionally version controlled. The query language ...@@ -391,14 +413,22 @@ Since Caosdb 0.2 entities are optionally version controlled. The query language
* The `ANY VERSION OF` modifier currently the only expression for taking the versioning into account when using the query language. * The `ANY VERSION OF` modifier currently the only expression for taking the versioning into account when using the query language.
* Subproperties are not supported yet, e.g. `FIND ANY VERSION OF ENTITY WHICH IS REFERENCED BY ename WITH ...`. This applies to all cases where you specify properties of *referenced* entities or *referencing* entities. * Subproperties are not supported yet, e.g. `FIND ANY VERSION OF ENTITY WHICH IS REFERENCED BY ename WITH ...`. This applies to all cases where you specify properties of *referenced* entities or *referencing* entities.
### Future ## Configuration
* Add `(LATEST|LAST|OLDEST|NEWEST|FIRST) VERSION OF` modifiers. In CaosDB Server implementations before version 0.9.0, the `FIND ename` query
* Add `(ANY|LATEST|LAST|OLDEST|NEWEST|FIRST) VERSION (BEFORE|AFTER) (<timestamp>|<transaction id>|<entity@version>) OF` modifier. would return any entity with that name and all children, regardless of the
* Add support for subproperties, e.g. `FIND ANY VERSION OF ENTITY WHICH IS REFERENCED BY ename WITH ...`. entity's role. Basically, `FIND ename` *was* equivalent to `FIND ENTITY ename`.
Since 0.9.0 the default behavior has changed and now `FIND ename` is equivalent
to `FIND RECORD ename`. This default is, however, configurable via the
`FIND_QUERY_DEFAULT_ROLE` server property. See [Server Configuration](./administration/configuration.rst).
## Future ## Future
* *Sub Queries* (or *Sub Properties*): `FIND ename WHICH HAS A pname WHICH HAS A subpname=val`. This is like: `FIND AN experiment WHICH HAS A camera WHICH HAS A 'serial number'= 1234567890` * Additional versioning queries:
* Add `(LATEST|LAST|OLDEST|NEWEST|FIRST) VERSION OF` modifiers.
* Add `(ANY|LATEST|LAST|OLDEST|NEWEST|FIRST) VERSION (BEFORE|AFTER) (<timestamp>|<transaction
id>|<entity@version>) OF` modifier.
* Add support for subproperties, e.g. `FIND ANY VERSION OF ENTITY WHICH IS REFERENCED BY ename
WITH ...`.
* Find deleted entities: `FIND ename WHICH WAS DELETED (BY ME | ON 2014-12-24)`
* *More Logic*, especially `ANY`, `ALL`, `NONE`, and `SUCH THAT` key words (and equivalents) for logical quantisation: `FIND ename1 SUCH THAT ALL ename2 WHICH HAVE A REFERENCE TO ename1 HAVE A pname=val`. This is like `FIND experiment SUCH THAT ALL person WHICH ARE REFERENCED BY THIS experiment AS conductor HAVE AN 'academic title'=professor.` * *More Logic*, especially `ANY`, `ALL`, `NONE`, and `SUCH THAT` key words (and equivalents) for logical quantisation: `FIND ename1 SUCH THAT ALL ename2 WHICH HAVE A REFERENCE TO ename1 HAVE A pname=val`. This is like `FIND experiment SUCH THAT ALL person WHICH ARE REFERENCED BY THIS experiment AS conductor HAVE AN 'academic title'=professor.`
...@@ -17,6 +17,7 @@ ...@@ -17,6 +17,7 @@
# sys.path.insert(0, os.path.abspath('../caosdb')) # sys.path.insert(0, os.path.abspath('../caosdb'))
import sphinx_rtd_theme import sphinx_rtd_theme
from os.path import dirname, abspath
# -- Project information ----------------------------------------------------- # -- Project information -----------------------------------------------------
...@@ -25,9 +26,9 @@ copyright = '2022, IndiScale GmbH' ...@@ -25,9 +26,9 @@ copyright = '2022, IndiScale GmbH'
author = 'Daniel Hornung, Timm Fitschen' author = 'Daniel Hornung, Timm Fitschen'
# The short X.Y version # The short X.Y version
version = '0.9.0' version = '0.9.1'
# The full version, including alpha/beta/rc tags # The full version, including alpha/beta/rc tags
release = '0.9.0-SNAPSHOT' release = '0.9.1-SNAPSHOT'
# -- General configuration --------------------------------------------------- # -- General configuration ---------------------------------------------------
...@@ -48,6 +49,7 @@ extensions = [ ...@@ -48,6 +49,7 @@ extensions = [
"sphinx.ext.autosectionlabel", # Allow reference sections using its title "sphinx.ext.autosectionlabel", # Allow reference sections using its title
"sphinx_rtd_theme", "sphinx_rtd_theme",
"sphinxcontrib.plantuml", # PlantUML diagrams "sphinxcontrib.plantuml", # PlantUML diagrams
"sphinx_a4doc", # antl4
] ]
# Add any paths that contain templates here, relative to this directory. # Add any paths that contain templates here, relative to this directory.
...@@ -223,3 +225,7 @@ autodoc_default_options = { ...@@ -223,3 +225,7 @@ autodoc_default_options = {
# -- Options for autosectionlabel -------------------------------------------- # -- Options for autosectionlabel --------------------------------------------
autosectionlabel_prefix_document = True autosectionlabel_prefix_document = True
# -- Options for sphinx_a4doc ------------------------------------------------
a4_base_path = abspath(dirname(__file__) + '/../main/java/org/caosdb/server/query')
...@@ -18,7 +18,7 @@ Welcome to caosdb-server's documentation! ...@@ -18,7 +18,7 @@ Welcome to caosdb-server's documentation!
Changelog <CHANGELOG> Changelog <CHANGELOG>
specification/index.rst specification/index.rst
Glossary Glossary
API documentation<_apidoc/packages> Server Internals<_apidoc/packages>
Welcome to the CaosDB, the flexible semantic data management toolkit! Welcome to the CaosDB, the flexible semantic data management toolkit!
......
...@@ -124,8 +124,8 @@ How to set permissions ...@@ -124,8 +124,8 @@ How to set permissions
---------------------- ----------------------
There are multiple ways to set role and entity permissions. The most There are multiple ways to set role and entity permissions. The most
common and best tested way currently is to set global default entity permissions common and best tested way currently is to set global default *entity* permissions
in the ``global_entity_permissions.xml`` config file, and role-based role in the ``global_entity_permissions.xml`` config file, and role-based *role*
permissions with the ``caosdb_admin.py`` `utility script permissions with the ``caosdb_admin.py`` `utility script
<https://gitlab.com/caosdb/caosdb-pylib/-/blob/main/src/caosdb/utils/caosdb_admin.py>`__ <https://gitlab.com/caosdb/caosdb-pylib/-/blob/main/src/caosdb/utils/caosdb_admin.py>`__
of CaosDB's Python library which is also used to `manage users and of CaosDB's Python library which is also used to `manage users and
...@@ -138,8 +138,8 @@ find a more detailed description of the possible ways of setting permissions. ...@@ -138,8 +138,8 @@ find a more detailed description of the possible ways of setting permissions.
you can set the default permissions that every entity on the server has. The you can set the default permissions that every entity on the server has. The
global default permissions can **only** be set in this file; all other ways global default permissions can **only** be set in this file; all other ways
below can only change the permissions of individual entities. Note that you below can only change the permissions of individual entities. Note that you
can add more rules but you can never remove rules set in the can add more rules in the ``global_entity_permissions.xml``, but you can not remove rules by
``global_entity_permissions.xml``. Thus, it might not be possible to overrule writing to this file. Thus, it might not be possible to overrule
permissions defined here (see :ref:`Permission permissions defined here (see :ref:`Permission
calculation<Calculation>`). Note also that, as the name suggests, only calculation<Calculation>`). Note also that, as the name suggests, only
:ref:`entity permissions<entity-permissions>` can be set this way. The :ref:`entity permissions<entity-permissions>` can be set this way. The
......
CaosDB Query Language Syntax
============================
This is the documentation of the CaosDB Query Language Syntax. The
authoritative specification of the syntax is the ANTLR4 grammar you can find in
the source code:
`CQLParser.g4 <https://gitlab.com/caosdb/caosdb-server/-/blob/main/src/main/java/org/caosdb/server/query/CQLParser.g4>`__
and
`CQLLexer.g4 <https://https://gitlab.com/caosdb/caosdb-server/-/blob/main/src/main/java/org/caosdb/server/query/CQLLexer.g4>`__
See examples in :doc:`Query Language<CaosDB-Query-Language>`.
.. a4:autogrammar:: CQLParser.g4
.. a4:autogrammar:: CQLLexer.g4
...@@ -10,7 +10,7 @@ users may have the same role, and there may be roles without any users. ...@@ -10,7 +10,7 @@ users may have the same role, and there may be roles without any users.
The user and their roles are always returned by the server in answers to requests The user and their roles are always returned by the server in answers to requests
and can thus be interpreted and used by clients. The most important use though and can thus be interpreted and used by clients. The most important use though
is [permission](permissions.rst) checking in the server: Access and is [permission](permissions) checking in the server: Access and
modification of modification of
entities can be controlled via roles, so that users of a given role are allowed entities can be controlled via roles, so that users of a given role are allowed
or denied certain actions. Incidentally, the permission to edit the permissions or denied certain actions. Incidentally, the permission to edit the permissions
...@@ -32,4 +32,4 @@ There are some special roles, which are automatically assigned to users: ...@@ -32,4 +32,4 @@ There are some special roles, which are automatically assigned to users:
Except for the `anonymous` role, these special roles are not returned by the Except for the `anonymous` role, these special roles are not returned by the
server, but can nevertheless be used to define server, but can nevertheless be used to define
[permissions](permissions.rst). [permissions](permissions.html#permissions).
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Please register or to comment