Skip to content
Snippets Groups Projects
Verified Commit da2fba2a authored by Timm Fitschen's avatar Timm Fitschen
Browse files

Merge branch 'f-filesystem-core' into f-filesystem-directory

parents 605faa81 647dc4b9
No related branches found
No related tags found
1 merge request!74Draft: ENH: file system: directory
Pipeline #35637 failed
Stage: info
Stage: setup
Stage: test
Stage: deploy
Hide whitespace changes
Inline Side-by-side
Showing
with 377 additions and 125 deletions
.gitlab-ci.yml
+ 13
− 8
View file @ da2fba2a
......@@ -39,6 +39,10 @@ variables:
CPPINT: ""
MYSQLBACKEND: ""
workflow:
rules:
- if: $CI_PIPELINE_SOURCE != "merge_request_event" && $CI_COMMIT_REF_NAME != $CI_COMMIT_TAG
image: $CI_REGISTRY_IMAGE
stages:
- info
......@@ -70,8 +74,11 @@ build-testenv:
image: docker:20.10
stage: setup
timeout: 3h
only:
- schedules
rules:
- if: $CI_PIPELINE_SOURCE == "schedule"
- if: $CI_PIPELINE_SOURCE != "schedule"
changes:
- src/test/docker/Dockerfile
script:
- cd src/test/docker
- docker login -u gitlab-ci-token -p $CI_JOB_TOKEN $CI_REGISTRY
......@@ -136,9 +143,8 @@ trigger_inttest:
pages_prepare: &pages_prepare
tags: [ cached-dind ]
stage: deploy
only:
refs:
- /^release-.*$/i
rules:
- if: $CI_COMMIT_REF_NAME =~ /^release-.*$/i
script:
- echo "Deploying..."
- make doc
......@@ -148,6 +154,5 @@ pages_prepare: &pages_prepare
- public
pages:
<<: *pages_prepare
only:
refs:
- main
rules:
- if: $CI_COMMIT_REF_NAME == 'main'
.gitlab/issue_templates/Default.md 0 → 100644
+ 28
− 0
View file @ da2fba2a
## Summary
*Please give a short summary of what the issue is.*
## Expected Behavior
*What did you expect how the software should behave?*
## Actual Behavior
*What did the software actually do?*
## Steps to Reproduce the Problem
*Please describe, step by step, how others can reproduce the problem. Please try these steps for yourself on a clean system.*
1.
2.
3.
## Specifications
- Version: *Which version of this software?*
- Platform: *Which operating system, which other relevant software versions?*
## Possible fixes
*Do you have ideas how the issue can be resolved?*
.gitlab/merge_request_templates/Default.md 0 → 100644
+ 54
− 0
View file @ da2fba2a
# Summary
*Insert a meaningful description for this merge request here: What is the new/changed behavior?
Which bug has been fixed? Are there related issues?*
# Focus
*Point the reviewer to the core of the code change. Where should they start reading? What should
they focus on (e.g. security, performance, maintainability, user-friendliness, compliance with the
specs, finding more corner cases, concrete questions)?*
# Test Environment
*How to set up a test environment for manual testing?*
# Check List for the Author
Please, prepare your MR for a review. Be sure to write a summary and a focus and create gitlab
comments for the reviewer. They should guide the reviewer through the changes, explain your changes
and also point out open questions. For further good practices have a look at [our review
guidelines](https://gitlab.com/caosdb/caosdb/-/blob/dev/REVIEW_GUIDELINES.md)
- [ ] All automated tests pass
- [ ] Reference related issues
- [ ] Up-to-date CHANGELOG.md (or not necessary)
- [ ] Up-to-date JSON schema (or not necessary)
- [ ] Appropriate user and developer documentation (or not necessary)
- How do I use the software? Assume "stupid" users.
- How do I develop or debug the software? Assume novice developers.
- [ ] Annotations in code (Gitlab comments)
- Intent of new code
- Problems with old code
- Why this implementation?
# Check List for the Reviewer
- [ ] I understand the intent of this MR
- [ ] All automated tests pass
- [ ] Up-to-date CHANGELOG.md (or not necessary)
- [ ] Appropriate user and developer documentation (or not necessary)
- [ ] The test environment setup works and the intended behavior is reproducible in the test
environment
- [ ] In-code documentation and comments are up-to-date.
- [ ] Check: Are there specifications? Are they satisfied?
For further good practices have a look at [our review guidelines](https://gitlab.com/caosdb/caosdb/-/blob/dev/REVIEW_GUIDELINES.md).
/assign me
/target_branch dev
CHANGELOG.md
+ 45
− 9
View file @ da2fba2a
......@@ -5,23 +5,59 @@ All notable changes to this project will be documented in this file.
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
## [Unreleased]
## [Unreleased] ##
### Added
### Added ###
* SELECT query support for the GRPC API
### Changed ###
### Changed
* The default behavior of the query `FIND SomeName [...]` (as well as COUNT and SELECT) is being
made configurable and changes:
* `FIND SomeName` will be interpreted as `FIND <FIND_QUERY_DEFAULT_ROLE>
SomeName` from now on where `FIND_QUERY_DEFAULT_ROLE` is a newly introduced
server property.
* The new `FIND_QUERY_DEFAULT_ROLE` server property defaults to `RECORD`
which is why the behavior of the server api has a **breaking change**.
* The semantics of `FIND *` are affected as well. `FIND *` is equivalent to
`FIND <FIND_QUERY_DEFAULT_ROLE>`.
* Of course, administrators can choose to retain the old behavior by setting
`FIND_QUERY_DEFAULT_ROLE=ENTITY`.
* CQL now treats `WITH` and `WITH A` equivalently. Issue: [#192](https://gitlab.com/caosdb/caosdb-server/-/issues/192)
* The InsertFilesInDir FlagJob now creates File entities without a name. The previous behavior
caused severe performance problems for very large numbers of files. Issue: [#197](https://gitlab.com/caosdb/caosdb-server/-/issues/197)
### Deprecated
### Deprecated ###
### Removed
### Removed ###
### Fixed
### Fixed ###
### Security
* Denying a role permission has no effect
[#196](https://gitlab.com/caosdb/caosdb-server/-/issues/196). See security
notes below.
* Missing RecordType leads to unexpected server error
[#166](https://gitlab.com/caosdb/caosdb-server/-/issues/166)
### Documentation
### Security ###
* Fixed [#196](https://gitlab.com/caosdb/caosdb-server/-/issues/196). This was
an error in the authorization procedure which allowed unprivileged users
execute insert, update or delete transactions on entities. However, the
unprivileged users would also need the correct entity permissions to do that.
Without backup, this means possible data loss. Also there was the possibility
to spam the database by creating unwanted entities.
### Documentation ###
- Nested queries.
- Global entity permissions.
## [0.9.0] - 2023-01-19
### Added
* SELECT query support for the GRPC API
## [0.8.1] - 2022-11-07
(Timm Fitschen)
......
CITATION.cff 0 → 100644
+ 28
− 0
View file @ da2fba2a
cff-version: 1.2.0
message: "If you use this software, please cite it as below."
authors:
- family-names: Fitschen
given-names: Timm
orcid: https://orcid.org/0000-0002-4022-432X
- family-names: Schlemmer
given-names: Alexander
orcid: https://orcid.org/0000-0003-4124-9649
- family-names: Hornung
given-names: Daniel
orcid: https://orcid.org/0000-0002-7846-6375
- family-names: tom Wörden
given-names: Henrik
orcid: https://orcid.org/0000-0002-5549-578X
- family-names: Spreckelsen
given-names: Florian
orcid: https://orcid.org/0000-0002-6856-2910
- family-names: Parlitz
given-names: Ulrich
orcid: https://orcid.org/0000-0003-3058-1435
- family-names: Luther
given-names: Stefan
orcid: https://orcid.org/0000-0001-7214-8125
title: "CaosDB - Server"
version: 0.8.1
doi: 10.3390/data4020083
date-released: 2022-11-07
DEPENDENCIES.md
+ 2
− 1
View file @ da2fba2a
......@@ -2,11 +2,12 @@
## For Building and Running the Server
* `>=caosdb-proto 0.2.0`
* `>=caosdb-proto 0.3.0`
* `>=caosdb-mysqlbackend 5.0.0`
* `>=Java 11`
* `>=Apache Maven 3.6.0`
* `>=Make 4.2`
* `>=gcc 8` (if PAM authentication is required)
* `libpam` (if PAM authentication is required)
* More dependencies are being pulled and installed automatically by Maven. See the complete list of dependencies in the [pom.xml](pom.xml)
......
FEATURES.md
+ 1
− 1
View file @ da2fba2a
# Features
* The CaosDB Server implements a CaosDB GRPC API Endpoint (v0.2.0)
* The CaosDB Server implements a CaosDB GRPC API Endpoint (v0.3.0)
Authentication is supported via a Basic scheme, using the well-known
"authentication" header.
Notable limitations of the current implementation of the API:
......
README_SETUP.md
+ 50
− 17
View file @ da2fba2a
......@@ -15,12 +15,18 @@ See [DEPENDENCIES.md](DEPENDENCIES.md).
On Debian, the required packages can be installed with:
apt-get install git make mariadb-server maven openjdk-11-jdk-headless \
python3-pip screen libpam0g-dev unzip
apt-get install make mariadb-server maven openjdk-11-jdk-headless \
python3-pip libpam0g-dev unzip
Note that installing MariaDB will uninstall existing MySQL packages and vice
versa.
#### Install the requirements on Fedora
On Fedora, the required packages can be installed with:
sudo dnf install make pam-devel mariadb-server mariadb python3 java-17-openjdk-headless unzip gcc
### System
* `>=Linux 4.0.0`, `x86_64`, e.g. Ubuntu 18.04
......@@ -160,20 +166,47 @@ sources (if you called `make run` previously).
## Setup Eclipse
1. Open Eclipse (recommended version: Oxygen.1a Release (4.7.1a))
2. `File > New > Java Project`: Choose a project name and specify the location
of this repo. The JRE and Project layout should be configured automatically.
Now, the project should initially have two source-folders: `./src/main/java`
and `./src/test/java`. After a build, another one,
`./target/generated-sources/antlr4` should be generated. If there are more
than these three source-folders, reconfigure the projects source folders
appropriately with `Project > Properties > Java Build Path > Source`.
3. In the `Package Explorer` view, right-click on the project and `Configure >
Convert to Maven Project`.
4. In the `Package Explorer` view, right-click on the project and `Maven >
Update Project`.
5. Usually a build of the project is started automatically. Otherwise `Project >
Build Project`.
1. Open Eclipse (tested with 2022-R12)
2. File > Import > Maven > Existing Maven Projects: Specify location.
3. You will most likely encounter "Plugin execution not covered by lifecycle
configuration: ..." errors. Adapt the file
`<eclipse-workspace>/.metadata/.plugins/org.eclipse.m2e.core/lifecycle-mapping-metadata.xml`.
Example:
```xml
<?xml version="1.0" encoding="UTF-8"?>
<lifecycleMappingMetadata>
<pluginExecutions>
<pluginExecution>
<pluginExecutionFilter>
<groupId>com.coveo</groupId>
<artifactId>fmt-maven-plugin</artifactId>
<versionRange>2.5.1</versionRange>
<goals>
<goal>format</goal>
</goals>
</pluginExecutionFilter>
<action>
<ignore />
</action>
</pluginExecution>
<pluginExecution>
<pluginExecutionFilter>
<groupId>org.codehaus.mojo</groupId>
<artifactId>buildnumber-maven-plugin</artifactId>
<versionRange>1.4</versionRange>
<goals>
<goal>create-metadata</goal>
</goals>
</pluginExecutionFilter>
<action>
<ignore />
</action>
</pluginExecution>
</pluginExecutions>
</lifecycleMappingMetadata>
```
Done!
......@@ -211,7 +244,7 @@ Stand-alone documentation is built using Sphinx: `make doc`
(`l_` not found):
```sh
git clone git@github.com:simgrid/javasphinx.git
git clone https://github.com/simgrid/javasphinx.git
cd javasphinx
git checkout 659209069603a
pip3 install .
......
RELEASE_GUIDELINES.md
+ 1
− 0
View file @ da2fba2a
......@@ -22,6 +22,7 @@ guidelines of the CaosDB Project
* [pom.xml](./pom.xml) (probably this means to remove the `-SNAPSHOT`)
* `src/doc/conf.py`
* `CHANGELOG.md`
* `CITATION.cff` (update version and date)
5. Merge the release branch into the main branch.
......
caosdb-proto @ c6405e53
Subproject commit bbab921f9aff5ac1945b299973e2b2acac75128a
Subproject commit c6405e538c179d2a8af952f85d9e9dc51fbadb92
caosdb-webui @ 421b2dce
Subproject commit bc199886b78a5dd39eb3bccd3a1451c8ab1e7552
Subproject commit 421b2dce5199a5a2c96bcc638543c1fd51d48870
conf/core/server.conf
+ 11
− 2
View file @ da2fba2a
......@@ -64,7 +64,7 @@ MYSQL_DATABASE_NAME=caosdb
# User name for connecting to mysql
MYSQL_USER_NAME=caosdb
# Password for the user
MYSQL_USER_PASSWORD=caosdb
MYSQL_USER_PASSWORD=random1234
# Schema of mysql procedures and tables which is required by this CaosDB instance
MYSQL_SCHEMA_VERSION=v6.0-SNAPSHOT
......@@ -204,6 +204,15 @@ USER_NAME_INVALID_MESSAGE=User names must have a length from 1 to 32 characters.
PASSWORD_VALID_REGEX=^((?=.*\\d)(?=.*[a-z])(?=.*[A-Z])(?=.*[\\p{Punct}]).{8,128})$
PASSWORD_INVALID_MESSAGE=Passwords must have a length from 8 to 128 characters. THe must contain at least one [A-Z], one [a-z], one [0-9] and a special character e.g. [!§$%&/()=?.;,:#+*+~].
# --------------------------------------------------
# Query Settings
# --------------------------------------------------
# FIND blablabla is short for FIND $FIND_QUERY_DEFAULT_ROLE blablabla. Should be
# either ENTITY or RECORD. RECORDTYPE, FILE, and PROPERTY will work as well but
# are rather unexpected for human users.
FIND_QUERY_DEFAULT_ROLE=RECORD
# --------------------------------------------------
# Extensions
# --------------------------------------------------
......@@ -213,4 +222,4 @@ ENTITY_VERSIONING_ENABLED=true
# Enabling the state machine extension
# EXT_STATE_ENTITY=ENABLE
# EXT_STATE_ENTITY=ENABLE
\ No newline at end of file
doc/Query.md
+ 7
− 1
View file @ da2fba2a
......@@ -219,6 +219,12 @@ The following query returns entities which have a _pname1_ property with any val
`FIND ename WITH pname1`
`FIND ename WITH A pname1`
`FIND ename WITH A PROPERTY pname1`
`FIND ename WITH PROPERTY pname1`
`FIND ename . pname1`
`FIND ename.pname1`
......@@ -338,7 +344,7 @@ Any result set can be filtered by logically combining POV filters or back refere
* NOT:: The logical negation. Equivalent expressions: `NOT, DOESN'T HAVE A PROPERTY, DOES NOT HAVE A PROPERTY, DOESN'T HAVE A, DOES NOT HAVE A, DOES NOT, DOESN'T, IS NOT, ISN'T, !`
* OR:: The logical _or_. Equivalent expressions: `OR, |`
* RECORD,RECORDTYPE,FILE,PROPERTY:: Role expression for restricting the result set to a specific role.
* WHICH:: The marker for the beginning of the filters. Equivalent expressions: `WHICH, WHICH HAS A, WHICH HAS A PROPERTY, WHERE, WITH, .`
* WHICH:: The marker for the beginning of the filters. Equivalent expressions: `WHICH, WHICH HAS A, WHICH HAS A PROPERTY, WHERE, WITH (A), .`
* REFERENCE:: This one is tricky: `REFERENCE TO` expresses a the state of _having_ a reference property. `REFERENCED BY` expresses the state of _being_ referenced by another entity.
* COUNT:: `COUNT` works like `FIND` but doesn't return the entities.
......
pom.xml
+ 1
− 1
View file @ da2fba2a
......@@ -25,7 +25,7 @@
<modelVersion>4.0.0</modelVersion>
<groupId>org.caosdb</groupId>
<artifactId>caosdb-server</artifactId>
<version>0.9.0-SNAPSHOT</version>
<version>0.9.1-SNAPSHOT</version>
<packaging>jar</packaging>
<name>CaosDB Server</name>
<scm>
......
src/doc/CaosDB-Query-Language.md
+ 104
− 74
View file @ da2fba2a
# CaosDB Query Language
**WIP This is going to be the specification. CQL tutorials are in the webui**
# CaosDB Query Language Examples
## Example queries
See syntax specification in [CaosDB Query Language Syntax](query-syntax).
### Simple FIND Query
The following query will return any entity which has the name _ename_ and all its children.
`FIND ename`
## Simple FIND Query
The following queries are equivalent and will return any entity which has the name _ename_ and all its children, but only if they are genuin records. Of course, the returned set of entities (henceforth referred to as _resultset_) can also be restricted to recordtypes, properties and files.
The following query will return any record which has the name _somename_ and all
record children of any entity with that name.
`FIND RECORD ename`
`FIND somename`
`FIND RECORDS ename`
On server in the default configuration, the following queries are equivalent to this one.
Wildcards use `*` for any characters or none at all. Wildcards for single characters (like the '_' wildcard from mysql) are not implemented yet.
`FIND RECORD somename`
`FIND RECORD en*` returns any entity which has a name beginning with _en_.
`FIND RECORDS somename`
Regular expressions must be surrounded by _<<_ and '>>':
Of course, the returned set of entities (henceforth referred to as _resultset_) can also be
restricted to RecordTypes (`FIND RECORDTYPE ...`), Properties (`FIND PROPERTY ...`) and Files (`FIND
FILE ...`).
`FIND RECORD <<e[aemn]{2,5}>>`
You can include all entities (Records, RecordTypes, Properties, ...) into the results by using the
`ENTITY` keyword:
`FIND RECORD <<[cC]am_[0-9]*>>`
`FIND ENTITY somename`
*TODO* (Timm):
Describe escape sequences like `\\ `, `\*`, `\<<` and `\>>`.
Wildcards use `*` for any characters or none at all. Wildcards for single characters (like the `_` wildcard from mysql) are not implemented yet.
Currently, wildcards and regular expressions are only available for the _simple-find-part_ of the query, i. e. no wildcards/regexps for filters.
`FIND en*` returns any record which has a name beginning with _en_.
### Simple COUNT Query
Regular expressions must be surrounded by _<<_ and _>>_:
This query counts entities which have certain properties.
`FIND <<e[aemn]{2,5}>>`
`COUNT ename`
will return the number of entities which have the name _ename_ and all their children.
`FIND <<[cC]amera_[0-9]*>>`
The syntax of the COUNT queries is equivalent to the FIND queries in any respect (this also applies to wildcards and regular expressions) but one: The prefix is to be `COUNT` instead of `FIND`.
*TODO*:
Describe escape sequences like `\\\\ `, `\*`, `\<<` and `\>>`.
Unlike the FIND queries, the COUNT queries do not return any entities. The result of the query is the number of entities which _would be_ returned if the query was a FIND query.
Currently, wildcards and regular expressions are only available for the _simple-find-part_ of the
query, i.e. not for property-operator-value filters (see below).
## Simple COUNT Query
COUNT queries count entities which have certain properties.
`COUNT ... rname ...`
will return the number of records which have the name _rname_ and all record
children of any entity with that name.
The syntax of the COUNT queries is equivalent to the FIND queries in any
respect (this also applies to wildcards and regular expressions) but one: The
prefix is to be `COUNT` instead of `FIND`.
Unlike the FIND queries, the COUNT queries do not return any entities. The result of the query is
the number of entities which _would be_ returned if the query was a FIND query.
## Filters
### POV - Property-Operator-Value
The following queries are equivalent and will restrict the result set to entities which have a property named _pname1_ that has a value _val1_.
The following queries are equivalent and will restrict the result set to records which have a property named _pname1_ that has a value _val1_.
`FIND ename.pname1=val1`
......@@ -53,31 +70,33 @@ The following queries are equivalent and will restrict the result set to entitie
`FIND ename WHICH HAS A pname1=val1`
Again, the resultset can be restricted to records:
Again, the resultset can be restricted to any other entity role as well:
`FIND RECORD ename WHICH HAS A pname1=val1`
`FIND RECORDTYPE ename WHICH HAS A pname1=val1`
_currently known operators:_ `=, !=, <=, <, >=, >` (and cf. next paragraphes!)
#### Special Operator: LIKE
The _LIKE_ can be used with wildcards. The `*` is a wildcard for any (possibly empty) sequence of characters. Examples:
The _LIKE_ can be used with wildcards. The `*` is a wildcard for any (possibly empty) sequence of characters.
Examples:
`FIND RECORD ename WHICH HAS A pname1 LIKE va*`
`FIND ename WHICH HAS A pname1 LIKE va*`
`FIND RECORD ename WHICH HAS A pname1 LIKE va*1`
`FIND ename WHICH HAS A pname1 LIKE va*1`
`FIND RECORD ename WHICH HAS A pname1 LIKE *al1`
`FIND ename WHICH HAS A pname1 LIKE *al1`
_Note:_ The _LIKE_ operator is will only produce expectable results with text properties.
_Note:_ The _LIKE_ operator will only produce expectable results with text properties.
#### Special Case: References
In general a reference can be addressed just like a POV filter. So
In general a reference can be addressed just like a POV filter. So
`FIND ename1.pname1=ename2`
`FIND ename1 WITH pname1=ename2`
will also return any entity named _ename1_ which references the entity with name or id _ename2_ via a reference property named _pname1_. However, it will also return any entity with a text property of that name with the string value _ename2_. In order to restrict the result set to reference properties one may make use of special reference operators:
will also return any record named _ename1_ which references the entity with name or id _ename2_ via a reference property named _pname1_. However, it will also return any entity with a text property of that name with the string value _ename2_. In order to restrict the result set to reference properties one may make use of special reference operators:
_reference operators:_ `->, REFERENCES, REFERENCE TO`
......@@ -88,12 +107,12 @@ The query looks like this:
`FIND ename1 WHICH HAS A pname1->ename2`
#### Time Special Case: DateTime
#### Special Case: DateTime
_DateTime operators:_ `=, !=, <, >, IN, NOT IN`
##### `d1=d2`: Equivalence relation.
* ''True'' iff d1 and d2 are equal in every respect (same DateTime flavor, same fields are defined/undefined and all defined fields are equal respectively).
##### `d1=d2`: Equivalence relation.
* ''True'' iff d1 and d2 are equal in every respect (same DateTime flavor, same fields are defined/undefined and all defined fields are equal respectively).
* ''False'' iff they have the same DateTime flavor but have different fields defined or fields with differing values.
* ''Undefined'' otherwise.
......@@ -103,11 +122,11 @@ Examples:
* `2015-04-03T00:00:00.0=2015-04-03T00:00:00.0` is true.
* `2015-04-03T00:00:00=2015-04-03T00:00:00` is true.
* `2015-04=2015-05` is false.
* `2015-04=2015-04` is true.
* `2015-04=2015-04` is true.
##### `d1!=d2`: Intransitive, symmetric relation.
* ''True'' iff `d1=d2` is false.
* ''False'' iff `d1=d2` is true.
##### `d1!=d2`: Intransitive, symmetric relation.
* ''True'' iff `d1=d2` is false.
* ''False'' iff `d1=d2` is true.
* ''Undefined'' otherwise.
Examples:
......@@ -116,15 +135,15 @@ Examples:
* `2015-04-03T00:00:00.0!=2015-04-03T00:00:00.0` is false.
* `2015-04-03T00:00:00!=2015-04-03T00:00:00` is false.
* `2015-04!=2015-05` is true.
* `2015-04!=2015-04` is false.
* `2015-04!=2015-04` is false
##### `d1>d2`: Transitive, non-symmetric relation.
Semantics depend on the flavors of d1 and d2. If both are...
###### [UTCDateTime](Datatype#datetime)
###### [UTCDateTime](specification/Datatype.html#datetime)
* ''True'' iff the time of d1 is after the the time of d2 according to [UTC](https://en.wikipedia.org/wiki/Coordinated_Universal_Time).
* ''False'' otherwise.
###### [SemiCompleteDateTime](Datatype#datetime)
###### [SemiCompleteDateTime](specification/Datatype.html#datetime)
* ''True'' iff `d1.ILB>d2.EUB` is true or `d1.ILB=d2.EUB` is true.
* ''False'' iff `d1.EUB<d2.ILB}} is true or {{{d1.EUB=d2.ILB` is true.
* ''Undefined'' otherwise.
......@@ -142,11 +161,11 @@ Examples:
##### `d1<d2`: Transitive, non-symmetric relation.
Semantics depend on the flavors of d1 and d2. If both are...
###### [UTCDateTime](Datatype#datetime)
###### [UTCDateTime](specification/Datatype.html#datetime)
* ''True'' iff the time of d1 is before the the time of d2 according to [UTC](https://en.wikipedia.org/wiki/Coordinated_Universal_Time)
* ''False'' otherwise.
###### [SemiCompleteDateTime](Datatype#datetime)
###### [SemiCompleteDateTime](specification/Datatype.html#datetime)
* ''True'' iff `d1.EUB<d2.ILB` is true or `d1.EUB=d2.ILB` is true.
* ''False'' iff `d1.ILB>d2.EUB}} is true or {{{d1.ILB=d2.EUB` is true.
* ''Undefined'' otherwise.
......@@ -164,7 +183,7 @@ Examples:
##### `d1 IN d2`: Transitive, non-symmetric relation.
Semantics depend on the flavors of d1 and d2. If both are...
###### [SemiCompleteDateTime](Datatype#datetime)
###### [SemiCompleteDateTime](specification/Datatype.html#datetime)
* ''True'' iff (`d1.ILB>d2.ILB` is true or `d1.ILB=d2.ILB` is true) and (`d1.EUB<d2.EUB` is true or `d1.EUB=d2.EUB` is true).
* ''False'' otherwise.
......@@ -175,9 +194,9 @@ Examples:
* `2015 IN 2015-01-01` is false.
* `2015-01-01 IN 2015-01-01T20:15:30` is false.
##### `d1 NOT IN d2`: Non-symmetric relation.
##### `d1 NOT IN d2`: Transitive, non-symmetric relation.
Semantics depend on the flavors of d1 and d2. If both are...
###### [SemiCompleteDateTime](Datatype#datetime)
###### [SemiCompleteDateTime](specification/Datatype.html#datetime)
* ''True'' iff `d1.ILB IN d2.ILB` is false.
* ''False'' otherwise.
......@@ -193,7 +212,7 @@ These semantics follow a three-valued logic with ''true'', ''false'' and ''undef
#### Omitting the Property or the Value
One doesn't have to specify the property or the value at all. The following query filters the result set for entities which have any property with a value greater than _val1_.
One doesn't have to specify the property or the value at all. The following query filters the result set for records which have any property with a value greater than _val1_.
`FIND ename WHICH HAS A PROPERTY > val1`
......@@ -213,7 +232,7 @@ And for references...
`FIND ename1.->ename2`
The following query returns entities which have a _pname1_ property with any value.
The following query returns records which have a _pname1_ property with any value.
`FIND ename WHICH HAS A PROPERTY pname1`
......@@ -227,22 +246,17 @@ The following query returns entities which have a _pname1_ property with any val
### TransactionFilter
`FIND ename WHICH (sugar|negated_sugar)? (NOT)? (CREATED|INSERTED|UPDATED) (by_clause time_clause?| time_clause by_clause?)`
*Definition*
sugar:: `HAS BEEN` | `HAVE BEEN` | `HAD BEEN` | `WAS` | `IS` |
sugar:: `HAS BEEN` | `HAVE BEEN` | `HAD BEEN` | `WAS` | `IS`
negated_sugar:: `HAS NOT BEEN` | `HASN'T BEEN` | `WAS NOT` | `WASN'T` | `IS NOT` | `ISN'T` | `HAVN'T BEEN` |
`HAVE NOT BEEN` | `HADN'T BEEN` | `HAD NOT BEEN`
negated_sugar:: `HAS NOT BEEN` | `HASN'T BEEN` | `WAS NOT` | `WASN'T` | `IS NOT` | `ISN'T` | `HAVN'T BEEN` | `HAVE NOT BEEN` | `HADN'T BEEN` | `HAD NOT BEEN`
by_clause:: `BY (ME | username | SOMEONE ELSE (BUT ME)? | SOMEONE ELSE BUT username)`
date:: A date string of the form `YYYY-MM-DD`
datetime:: A datetime string of the form `YYYY-MM-DD hh:mm:ss`
time_clause:: `ON ($date|$datetime) ` Here is plenty of room for more syntactic sugar, e.g. a `TODAY` keyword, and more funcionality, e.g. ranges.
`FIND ename WHICH ($sugar|$negated_sugar)? (NOT)? (CREATED|INSERTED|UPDATED|DELETED) (by_clause time_clause?| time_clause by_clause?)`
datetime:: A datetime string of the form `YYYY[-MM[-DD(T| )[hh[:mm[:ss[.nnn][(+|-)zzzz]]]]]]` or `TODAY`.
time_clause:: `[AT|ON|IN|BEFORE|AFTER|UNTIL|SINCE] (datetime) `
*Examples*
......@@ -252,12 +266,13 @@ The following query returns entities which have a _pname1_ property with any val
`FIND ename WHICH HAS BEEN CREATED BY erwin ON 2014-12-24`
`FIND ename WHICH HAS BEEN CREATED BY SOMEONE ELSE BUT erwin ON 2014-12-24`
`FIND ename WHICH HAS BEEN UPDATED BY SOMEONE ELSE BUT erwin ON 2014-12-24`
`FIND ename WHICH HAS BEEN CREATED BY erwin`
`FIND ename WHICH HAS BEEN INSERTED BY erwin`
`FIND ename . CREATED BY erwin ON `
`FIND ename WHICH HAS BEEN INSERTED SINCE 2021-04`
Note that `SINCE` and `UNTIL` are inclusive, while `BEFORE` and `AFTER` are not.
### File Location
......@@ -297,7 +312,7 @@ Find any file in a directory which begins with `2016-02`:
### Back References
The back reference filters for entities that are referenced by another entity. The following query returns entities of the type _ename1_ which are referenced by _ename2_ entities via the reference property _pname1_.
The back reference filters for entities that are referenced by another entity. The following query returns records of the type _ename1_ which are referenced by _ename2_ entities via the reference property _pname1_.
* `FIND ename1 WHICH IS REFERENCED BY ename2 AS A pname1`
* `FIND ename1 WITH @ ename2 / pname1`
......@@ -310,6 +325,13 @@ One may omit the property specification:
* `FIND ename1 WITH @ ename2`
* `FIND ename1 . @ ename2`
### Nested queries, or filtering by sub-properties ###
Nested queries can easily be searched by simply concatenating `WHICH` or `WITH` expressions:
* `FIND ename WHICH HAS A pname WHICH HAS A subpname=val`
* For example: `FIND AN experiment WHICH HAS A camera WHICH HAS A 'serial number'= 1234567890`
### Combining Filters with Propositional Logic
Any result set can be filtered by logically combining POV filters or back reference filters:
......@@ -356,7 +378,7 @@ Any result set can be filtered by logically combining POV filters or back refere
In contrast to `FIND` queries, which always return the complete entity, there are `SELECT` queries which only return the entity with only those properties which are specified in the query. The syntax is very similar to `FIND` queries - just replace the `FIND` by `SELECT <comma separated list of selectors> FROM`:
`SELECT p1, p2, p3 FROM Record ename`
`SELECT p1, p2, p3 FROM ename`
However, the `SELECT` query can also return properties of referenced entities and thereby are a means of joining entities together and return a custom view or projection:
......@@ -391,14 +413,22 @@ Since Caosdb 0.2 entities are optionally version controlled. The query language
* The `ANY VERSION OF` modifier currently the only expression for taking the versioning into account when using the query language.
* Subproperties are not supported yet, e.g. `FIND ANY VERSION OF ENTITY WHICH IS REFERENCED BY ename WITH ...`. This applies to all cases where you specify properties of *referenced* entities or *referencing* entities.
### Future
## Configuration
* Add `(LATEST|LAST|OLDEST|NEWEST|FIRST) VERSION OF` modifiers.
* Add `(ANY|LATEST|LAST|OLDEST|NEWEST|FIRST) VERSION (BEFORE|AFTER) (<timestamp>|<transaction id>|<entity@version>) OF` modifier.
* Add support for subproperties, e.g. `FIND ANY VERSION OF ENTITY WHICH IS REFERENCED BY ename WITH ...`.
In CaosDB Server implementations before version 0.9.0, the `FIND ename` query
would return any entity with that name and all children, regardless of the
entity's role. Basically, `FIND ename` *was* equivalent to `FIND ENTITY ename`.
Since 0.9.0 the default behavior has changed and now `FIND ename` is equivalent
to `FIND RECORD ename`. This default is, however, configurable via the
`FIND_QUERY_DEFAULT_ROLE` server property. See [Server Configuration](./administration/configuration.rst).
## Future
* *Sub Queries* (or *Sub Properties*): `FIND ename WHICH HAS A pname WHICH HAS A subpname=val`. This is like: `FIND AN experiment WHICH HAS A camera WHICH HAS A 'serial number'= 1234567890`
* *More Logic*, especially `ANY`, `ALL`, `NONE`, and `SUCH THAT` key words (and equivalents) for logical quantisation: `FIND ename1 SUCH THAT ALL ename2 WHICH HAVE A REFERENCE TO ename1 HAVE A pname=val`. This is like `FIND experiment SUCH THAT ALL person WHICH ARE REFERENCED BY THIS experiment AS conductor HAVE AN 'academic title'=professor.`
* Additional versioning queries:
* Add `(LATEST|LAST|OLDEST|NEWEST|FIRST) VERSION OF` modifiers.
* Add `(ANY|LATEST|LAST|OLDEST|NEWEST|FIRST) VERSION (BEFORE|AFTER) (<timestamp>|<transaction
id>|<entity@version>) OF` modifier.
* Add support for subproperties, e.g. `FIND ANY VERSION OF ENTITY WHICH IS REFERENCED BY ename
WITH ...`.
* Find deleted entities: `FIND ename WHICH WAS DELETED (BY ME | ON 2014-12-24)`
* *More Logic*, especially `ANY`, `ALL`, `NONE`, and `SUCH THAT` key words (and equivalents) for logical quantisation: `FIND ename1 SUCH THAT ALL ename2 WHICH HAVE A REFERENCE TO ename1 HAVE A pname=val`. This is like `FIND experiment SUCH THAT ALL person WHICH ARE REFERENCED BY THIS experiment AS conductor HAVE AN 'academic title'=professor.`
src/doc/conf.py
+ 8
− 2
View file @ da2fba2a
......@@ -17,6 +17,7 @@
# sys.path.insert(0, os.path.abspath('../caosdb'))
import sphinx_rtd_theme
from os.path import dirname, abspath
# -- Project information -----------------------------------------------------
......@@ -25,9 +26,9 @@ copyright = '2022, IndiScale GmbH'
author = 'Daniel Hornung, Timm Fitschen'
# The short X.Y version
version = '0.9.0'
version = '0.9.1'
# The full version, including alpha/beta/rc tags
release = '0.9.0-SNAPSHOT'
release = '0.9.1-SNAPSHOT'
# -- General configuration ---------------------------------------------------
......@@ -48,6 +49,7 @@ extensions = [
"sphinx.ext.autosectionlabel", # Allow reference sections using its title
"sphinx_rtd_theme",
"sphinxcontrib.plantuml", # PlantUML diagrams
"sphinx_a4doc", # antl4
]
# Add any paths that contain templates here, relative to this directory.
......@@ -223,3 +225,7 @@ autodoc_default_options = {
# -- Options for autosectionlabel --------------------------------------------
autosectionlabel_prefix_document = True
# -- Options for sphinx_a4doc ------------------------------------------------
a4_base_path = abspath(dirname(__file__) + '/../main/java/org/caosdb/server/query')
src/doc/index.rst
+ 1
− 1
View file @ da2fba2a
......@@ -18,7 +18,7 @@ Welcome to caosdb-server's documentation!
Changelog <CHANGELOG>
specification/index.rst
Glossary
API documentation<_apidoc/packages>
Server Internals<_apidoc/packages>
Welcome to the CaosDB, the flexible semantic data management toolkit!
......
src/doc/permissions.rst
+ 4
− 4
View file @ da2fba2a
......@@ -124,8 +124,8 @@ How to set permissions
----------------------
There are multiple ways to set role and entity permissions. The most
common and best tested way currently is to set global default entity permissions
in the ``global_entity_permissions.xml`` config file, and role-based role
common and best tested way currently is to set global default *entity* permissions
in the ``global_entity_permissions.xml`` config file, and role-based *role*
permissions with the ``caosdb_admin.py`` `utility script
<https://gitlab.com/caosdb/caosdb-pylib/-/blob/main/src/caosdb/utils/caosdb_admin.py>`__
of CaosDB's Python library which is also used to `manage users and
......@@ -138,8 +138,8 @@ find a more detailed description of the possible ways of setting permissions.
you can set the default permissions that every entity on the server has. The
global default permissions can **only** be set in this file; all other ways
below can only change the permissions of individual entities. Note that you
can add more rules but you can never remove rules set in the
``global_entity_permissions.xml``. Thus, it might not be possible to overrule
can add more rules in the ``global_entity_permissions.xml``, but you can not remove rules by
writing to this file. Thus, it might not be possible to overrule
permissions defined here (see :ref:`Permission
calculation<Calculation>`). Note also that, as the name suggests, only
:ref:`entity permissions<entity-permissions>` can be set this way. The
......
src/doc/query-syntax.rst 0 → 100644
+ 15
− 0
View file @ da2fba2a
CaosDB Query Language Syntax
============================
This is the documentation of the CaosDB Query Language Syntax. The
authoritative specification of the syntax is the ANTLR4 grammar you can find in
the source code:
`CQLParser.g4 <https://gitlab.com/caosdb/caosdb-server/-/blob/main/src/main/java/org/caosdb/server/query/CQLParser.g4>`__
and
`CQLLexer.g4 <https://https://gitlab.com/caosdb/caosdb-server/-/blob/main/src/main/java/org/caosdb/server/query/CQLLexer.g4>`__
See examples in :doc:`Query Language<CaosDB-Query-Language>`.
.. a4:autogrammar:: CQLParser.g4
.. a4:autogrammar:: CQLLexer.g4
src/doc/roles.md
+ 2
− 2
View file @ da2fba2a
......@@ -10,7 +10,7 @@ users may have the same role, and there may be roles without any users.
The user and their roles are always returned by the server in answers to requests
and can thus be interpreted and used by clients. The most important use though
is [permission](permissions.rst) checking in the server: Access and
is [permission](permissions) checking in the server: Access and
modification of
entities can be controlled via roles, so that users of a given role are allowed
or denied certain actions. Incidentally, the permission to edit the permissions
......@@ -32,4 +32,4 @@ There are some special roles, which are automatically assigned to users:
Except for the `anonymous` role, these special roles are not returned by the
server, but can nevertheless be used to define
[permissions](permissions.rst).
[permissions](permissions.html#permissions).
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Please register or to comment