Skip to content
Snippets Groups Projects

Compare revisions

Changes are shown as if the source revision was being merged into the target revision. Learn more about comparing revisions.

Source

Select target project
No results found

Target

Select target project
  • caosdb/src/caosdb-pylib
1 result
Show changes
Commits on Source (70)
Showing with 279 additions and 62 deletions
......@@ -29,9 +29,9 @@ variables:
image: $CI_REGISTRY_IMAGE
stages:
- setup
- code_style
- linting
- setup
- test
- deploy
......@@ -53,6 +53,14 @@ pylint:
- make lint
allow_failure: true
mypy:
tags: [ docker ]
stage: linting
script:
- pip install mypy types-PyYAML types-jsonschema types-requests types-setuptools types-lxml types-python-dateutil
- make mypy
allow_failure: true
# run unit tests
unittest_py3.7:
tags: [ docker ]
......@@ -62,7 +70,7 @@ unittest_py3.7:
script: &python_test_script
# Python docker has problems with tox and pip so use plain pytest here
- touch ~/.pylinkahead.ini
- pip install nose pytest pytest-cov python-dateutil jsonschema>=4.4.0
- pip install pynose pytest pytest-cov jsonschema>=4.4.0 setuptools
- pip install .
- python -m pytest unittests
......@@ -100,6 +108,22 @@ unittest_py3.11:
image: python:3.11
script: *python_test_script
unittest_py3.12:
tags: [ docker ]
stage: test
needs: [ ]
image: python:3.12
script: *python_test_script
unittest_py3.13:
allow_failure: true
tags: [ docker ]
stage: test
needs: [ ]
image: python:3.13-rc
script: *python_test_script
# Trigger building of server image and integration tests
trigger_build:
stage: deploy
......@@ -126,6 +150,7 @@ build-testenv:
stage: setup
only:
- schedules
- web
script:
- cd unittests/docker
- docker login -u gitlab-ci-token -p $CI_JOB_TOKEN $CI_REGISTRY
......@@ -148,7 +173,7 @@ pages_prepare: &pages_prepare
refs:
- /^release-.*$/i
script:
- echo "Deploying"
- echo "Deploying documentation"
- make doc
- cp -r build/doc/html public
artifacts:
......
......@@ -5,14 +5,11 @@ All notable changes to this project will be documented in this file.
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
## [Unreleased] - Date
## [Unreleased] ##
### Added ###
* `apiutils.merge_entities` now has a `merge_id_with_resolved_entity` keyword
which allows to identify property values with each other in case that one is
an id and the other is an Entity with this id. Default is ``False``, so no
change to the default behavior.
* Support for Python 3.12
### Changed ###
......@@ -22,10 +19,30 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
### Fixed ###
* [#104](https://gitlab.com/linkahead/linkahead-pylib/-/issues/104) Selecting
parts of a `Container` with a `slice` used to return a `list` object instead
of a `Container`, removing all useful methods of the `Container` class. This
has been fixed and using a `slice` such as `[:2]` now returns a new
`Container`.
### Security ###
### Documentation ###
## [0.14.0] - 2024-02-20
### Added ###
* `utils.merge_entities` now has a `merge_id_with_resolved_entity` keyword
which allows to identify property values with each other in case that one is
an id and the other is an Entity with this id. Default is ``False``, so no
change to the default behavior.
* `apiutils.escape_quoted_text` for escaping text in queries.
### Changed ###
* `cached_query()` now also caches uniqueness related exceptions.
## [0.13.2] - 2023-12-15
### Fixed ###
......
......@@ -20,6 +20,6 @@ authors:
given-names: Stefan
orcid: https://orcid.org/0000-0001-7214-8125
title: CaosDB - Pylib
version: 0.13.2
version: 0.14.0
doi: 10.3390/data4020083
date-released: 2023-10-11
date-released: 2024-02-20
......@@ -32,7 +32,7 @@ doc:
install:
@echo "Not implemented yet, use pip for installation."
check: style lint
check: style lint mypy
.PHONY: check
style:
......@@ -43,6 +43,10 @@ lint:
pylint --unsafe-load-any-extension=y -d all -e E,F src/linkahead/common
.PHONY: lint
mypy:
mypy src/linkahead
.PHONY: mypy
unittest:
tox -r
.PHONY: unittest
......@@ -48,8 +48,8 @@ from setuptools import find_packages, setup
ISRELEASED = False
MAJOR = 0
MINOR = 13
MICRO = 3
MINOR = 14
MICRO = 1
# Do not tag as pre-release until this commit
# https://github.com/pypa/packaging/pull/515
# has made it into a release. Probably we should wait for pypa/packaging>=21.4
......@@ -183,6 +183,7 @@ def setup_package():
package_dir={'': 'src'},
install_requires=['lxml>=4.6.3',
"requests[socks]>=2.26",
"setuptools",
"python-dateutil>=2.8.2",
'PyYAML>=5.4.1',
'future',
......
from linkahead.utils.escape import *
from warnings import warn
warn(("CaosDB was renamed to LinkAhead. Please import this library as `import linkahead.utils.escape`. Using the"
" old name, starting with caosdb, is deprecated."), DeprecationWarning)
......@@ -29,10 +29,10 @@ copyright = '2023, IndiScale GmbH'
author = 'Daniel Hornung'
# The short X.Y version
version = '0.13.3'
version = '0.14.1'
# The full version, including alpha/beta/rc tags
# release = '0.5.2-rc2'
release = '0.13.3-dev'
release = '0.14.1-dev'
# -- General configuration ---------------------------------------------------
......
......@@ -15,6 +15,7 @@ Welcome to PyLinkAhead's documentation!
High Level API <high_level_api>
Code gallery <gallery/index>
API documentation <_apidoc/linkahead>
Related Projects <related_projects/index>
Back to Overview <https://docs.indiscale.com/>
......
Related Projects
++++++++++++++++
.. toctree::
:maxdepth: 2
:caption: Contents:
:hidden:
.. container:: projects
For in-depth documentation for users, administrators and developers, you may want to visit the subproject-specific documentation pages for:
:`Server <https://docs.indiscale.com/caosdb-server>`_: The Java part of the LinkAhead server.
:`MySQL backend <https://docs.indiscale.com/caosdb-mysqlbackend>`_: The MySQL/MariaDB components of the LinkAhead server.
:`WebUI <https://docs.indiscale.com/caosdb-webui>`_: The default web frontend for the LinkAhead server.
:`Advanced user tools <https://docs.indiscale.com/caosdb-advanced-user-tools>`_: The advanced Python tools for LinkAhead.
:`LinkAhead Crawler <https://docs.indiscale.com/caosdb-crawler/>`_: The crawler is the main tool for automatic data integration in LinkAhead.
:`LinkAhead <https://docs.indiscale.com/caosdb-deploy>`_: Your all inclusive LinkAhead software package.
:`Back to Overview <https://docs.indiscale.com/>`_: LinkAhead Documentation.
......@@ -215,6 +215,10 @@ def compare_entities(old_entity: Entity, new_entity: Entity, compare_referenced_
if old_entity is new_entity:
return (olddiff, newdiff)
if type(old_entity) is not type(new_entity):
raise ValueError(
"Comparison of different Entity types is not supported.")
for attr in SPECIAL_ATTRIBUTES:
try:
oldattr = old_entity.__getattribute__(attr)
......@@ -501,11 +505,11 @@ def describe_diff(olddiff, newdiff, name=None, as_update=True):
if len(olddiff["parents"]) > 0:
description += ("Parents that are only in the old version:\n"
+ ", ".join(olddiff["parents"]))
+ ", ".join(olddiff["parents"]) + "\n")
if len(newdiff["parents"]) > 0:
description += ("Parents that are only in the new version:\n"
+ ", ".join(olddiff["parents"]))
+ ", ".join(olddiff["parents"]) + "\n")
for prop in list(set(list(olddiff["properties"].keys())
+ list(newdiff["properties"].keys()))):
......
......@@ -36,6 +36,7 @@ from enum import Enum
from functools import lru_cache
from typing import Union
from .exceptions import EmptyUniqueQueryError, QueryNotUniqueError
from .utils import get_entity
from .common.models import execute_query, Entity, Container
......@@ -80,16 +81,22 @@ If a query phrase is given, the result must be unique. If this is not what you
if count != 1:
raise ValueError("You must supply exactly one argument.")
result = (None, )
if eid is not None:
return _cached_access(AccessType.EID, eid, unique=True)
result = _cached_access(AccessType.EID, eid, unique=True)
if name is not None:
return _cached_access(AccessType.NAME, name, unique=True)
result = _cached_access(AccessType.NAME, name, unique=True)
if path is not None:
return _cached_access(AccessType.PATH, path, unique=True)
result = _cached_access(AccessType.PATH, path, unique=True)
if query is not None:
return _cached_access(AccessType.QUERY, query, unique=True)
result = _cached_access(AccessType.QUERY, query, unique=True)
raise ValueError("Not all arguments may be None.")
if result != (None, ):
if isinstance(result, (QueryNotUniqueError, EmptyUniqueQueryError)):
raise result
return result
raise RuntimeError("This line should never be reached.")
def cached_query(query_string) -> Container:
......@@ -98,7 +105,10 @@ def cached_query(query_string) -> Container:
All additional arguments are at their default values.
"""
return _cached_access(AccessType.QUERY, query_string, unique=False)
result = _cached_access(AccessType.QUERY, query_string, unique=False)
if isinstance(result, (QueryNotUniqueError, EmptyUniqueQueryError)):
raise result
return result
@lru_cache(maxsize=DEFAULT_SIZE)
......@@ -111,14 +121,17 @@ def _cached_access(kind: AccessType, value: Union[str, int], unique=True):
if value in _DUMMY_CACHE:
return _DUMMY_CACHE[value]
if kind == AccessType.QUERY:
return execute_query(value, unique=unique)
if kind == AccessType.NAME:
return get_entity.get_entity_by_name(value)
if kind == AccessType.EID:
return get_entity.get_entity_by_id(value)
if kind == AccessType.PATH:
return get_entity.get_entity_by_path(value)
try:
if kind == AccessType.QUERY:
return execute_query(value, unique=unique)
if kind == AccessType.NAME:
return get_entity.get_entity_by_name(value)
if kind == AccessType.EID:
return get_entity.get_entity_by_id(value)
if kind == AccessType.PATH:
return get_entity.get_entity_by_path(value)
except (QueryNotUniqueError, EmptyUniqueQueryError) as exc:
return exc
raise ValueError(f"Unknown AccessType: {kind}")
......
......@@ -4,9 +4,10 @@
#
# Copyright (C) 2018 Research Group Biomedical Physics,
# Max-Planck-Institute for Dynamics and Self-Organization Göttingen
# Copyright (C) 2020-2023 Indiscale GmbH <info@indiscale.com>
# Copyright (C) 2020-2024 IndiScale GmbH <info@indiscale.com>
# Copyright (C) 2020-2023 Florian Spreckelsen <f.spreckelsen@indiscale.com>
# Copyright (C) 2020-2022 Timm Fitschen <t.fitschen@indiscale.com>
# Copyright (C) 2024 Joscha Schmiedt <joscha@schmiedt.dev>
#
# This program is free software: you can redistribute it and/or modify
# it under the terms of the GNU Affero General Public License as
......@@ -63,8 +64,7 @@ from ..exceptions import (AmbiguousEntityError, AuthorizationError,
UniqueNamesError, UnqualifiedParentsError,
UnqualifiedPropertiesError)
from .datatype import (BOOLEAN, DATETIME, DOUBLE, INTEGER, TEXT,
get_list_datatype,
is_list_datatype, is_reference)
get_list_datatype, is_list_datatype, is_reference)
from .state import State
from .timezone import TimeZone
from .utils import uuid, xml2str
......@@ -82,7 +82,7 @@ NONE = "NONE"
SPECIAL_ATTRIBUTES = ["name", "role", "datatype", "description",
"id", "path", "checksum", "size"]
"id", "path", "checksum", "size", "value"]
class Entity:
......@@ -156,7 +156,7 @@ class Entity:
# Copy special attributes:
# TODO: this might rise an exception when copying
# special file attributes like checksum and size.
for attribute in SPECIAL_ATTRIBUTES + ["value"]:
for attribute in SPECIAL_ATTRIBUTES:
val = getattr(self, attribute)
if val is not None:
setattr(new, attribute, val)
......@@ -1537,7 +1537,12 @@ def _parse_value(datatype, value):
return float(value)
if datatype == INTEGER:
return int(str(value))
if isinstance(value, int):
return value
elif isinstance(value, float) and value.is_integer():
return int(value)
else:
return int(str(value))
if datatype == BOOLEAN:
if str(value).lower() == "true":
......@@ -3058,6 +3063,14 @@ class Container(list):
def __repr__(self):
return xml2str(self.to_xml())
def __getitem__(self, key):
self_as_list_slice = super().__getitem__(key)
if isinstance(self_as_list_slice, list):
# Construct new Container from list slice
return Container().extend(self_as_list_slice)
else:
return self_as_list_slice
@staticmethod
def from_xml(xml_str):
"""Creates a Container from the given xml string.
......@@ -4571,8 +4584,9 @@ def execute_query(q, unique=False, raise_exception_on_error=True, cache=True,
Whether an exception should be raised when there are errors in the
resulting entities. Defaults to True.
cache : bool
Whether to use the query server-side cache (equivalent to adding a
"cache" flag). Defaults to True.
Whether to use the server's query cache (equivalent to adding a
"cache" flag) to the Query object. Defaults to True. Not to be
confused with the ``cached`` module.
flags : dict of str
Flags to be added to the request.
page_length : int
......@@ -4620,6 +4634,8 @@ class Info():
def __init__(self):
self.messages = Messages()
self.user_info = None
self.time_zone = None
self.sync()
def sync(self):
......
# -*- coding: utf-8 -*-
#
# This file is a part of the LinkAhead Project.
#
# Copyright (C) 2024 Henrik tom Wörden <h.tomwoerden@indiscale.com>
# Copyright (C) 2024 IndiScale GmbH <info@indiscale.com>
#
# This program is free software: you can redistribute it and/or modify
# it under the terms of the GNU Affero General Public License as
# published by the Free Software Foundation, either version 3 of the
# License, or (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU Affero General Public License for more details.
#
# You should have received a copy of the GNU Affero General Public License
# along with this program. If not, see <https://www.gnu.org/licenses/>.
#
import warnings
def escape_squoted_text(text: str) -> str:
r"""Return an escaped version of the argument.
The characters ``\``, ``*`` and ``'`` need to be escaped if used in single quoted
expressions in the query language.
This function returns the given string where the characters ``\``, ``'`` and ``*`` are
escaped by a ``\`` (backslash character).
Parameters
----------
text : str
The text to be escaped.
Returns
-------
out : str
The escaped text.
"""
return text.replace("\\", r"\\").replace("'", r"\'").replace("*", r"\*")
def escape_dquoted_text(text: str) -> str:
r"""Return an escaped version of the argument.
The characters ``\``, ``*`` and ``"`` need to be escaped if used in double quoted
expressions in the query language.
This function returns the given string where the characters ``\``, ``"`` and ``*`` are
escaped by a ``\`` (backslash character).
Parameters
----------
text : str
The text to be escaped.
Returns
-------
out : str
The escaped text.
"""
return text.replace("\\", r"\\").replace('"', r"\"").replace("*", r"\*")
def escape_quoted_text(text: str) -> str:
"""
Please use escape_squoted_text or escape_dquoted_text instead of this function.
"""
warnings.warn("Please use escape_squoted_text or escape_dquoted_text", DeprecationWarning)
return escape_squoted_text(text)
......@@ -22,7 +22,9 @@
"""Convenience functions to retrieve a specific entity."""
from typing import Union
from ..common.models import execute_query, Entity
from ..common.models import Entity, execute_query
from .escape import escape_squoted_text
def get_entity_by_name(name: str) -> Entity:
......@@ -30,6 +32,7 @@ def get_entity_by_name(name: str) -> Entity:
Submits the query "FIND ENTITY WITH name='{name}'".
"""
name = escape_squoted_text(name)
return execute_query(f"FIND ENTITY WITH name='{name}'", unique=True)
......
[tox]
envlist=py37, py38, py39, py310, py311
envlist=py37, py38, py39, py310, py311, py312, py313
skip_missing_interpreters = true
[testenv]
deps = .
nose
pynose
pytest
pytest-cov
mypy
jsonschema>=4.4.0
commands=py.test --cov=linkahead -vv {posargs}
......@@ -17,3 +18,4 @@ max-line-length=100
testpaths = unittests
xfail_strict = True
addopts = -x -vv --cov=linkahead
pythonpath = src
......@@ -12,4 +12,4 @@ ARG COMMIT="dev"
# TODO Rename to linkahead
RUN git clone -b dev https://gitlab.indiscale.com/caosdb/src/caosdb-pylib.git linkahead-pylib && \
cd linkahead-pylib && git checkout $COMMIT && pip3 install .
RUN pip3 install recommonmark sphinx-rtd-theme
RUN pip3 install recommonmark sphinx-rtd-theme mypy
......@@ -26,13 +26,12 @@
# A. Schlemmer, 02/2018
import pytest
import linkahead as db
import linkahead.apiutils
from linkahead.apiutils import (apply_to_ids, compare_entities, create_id_query,
empty_diff, EntityMergeConflictError,
resolve_reference, merge_entities)
import pytest
from linkahead.apiutils import (EntityMergeConflictError, apply_to_ids,
compare_entities, create_id_query, empty_diff,
merge_entities, resolve_reference)
from linkahead.common.models import SPECIAL_ATTRIBUTES
......@@ -104,6 +103,8 @@ def test_compare_entities():
r1.add_parent("lopp")
r1.add_property("test", value=2)
r2.add_property("test", value=2)
r1.add_property("testi", importance=linkahead.SUGGESTED, value=2)
r2.add_property("testi", importance=linkahead.RECOMMENDED, value=2)
r1.add_property("tests", value=3)
r2.add_property("tests", value=45)
r1.add_property("tester", value=3)
......@@ -115,8 +116,8 @@ def test_compare_entities():
assert len(diff_r1["parents"]) == 1
assert len(diff_r2["parents"]) == 0
assert len(diff_r1["properties"]) == 3
assert len(diff_r2["properties"]) == 3
assert len(diff_r1["properties"]) == 4
assert len(diff_r2["properties"]) == 4
assert "test" not in diff_r1["properties"]
assert "test" not in diff_r2["properties"]
......@@ -124,6 +125,9 @@ def test_compare_entities():
assert "tests" in diff_r1["properties"]
assert "tests" in diff_r2["properties"]
assert "testi" in diff_r1["properties"]
assert "testi" in diff_r2["properties"]
assert "tester" in diff_r1["properties"]
assert "tester" in diff_r2["properties"]
......@@ -212,7 +216,6 @@ def test_compare_special_properties():
assert len(diff_r2["properties"]) == 0
@pytest.mark.xfail
def test_compare_properties():
p1 = db.Property()
p2 = db.Property()
......@@ -223,21 +226,12 @@ def test_compare_properties():
assert len(diff_r1["properties"]) == 0
assert len(diff_r2["properties"]) == 0
p1.importance = "SUGGESTED"
diff_r1, diff_r2 = compare_entities(p1, p2)
assert len(diff_r1["parents"]) == 0
assert len(diff_r2["parents"]) == 0
assert len(diff_r1["properties"]) == 0
assert len(diff_r2["properties"]) == 0
assert "importance" in diff_r1
assert diff_r1["importance"] == "SUGGESTED"
# TODO: I'm not sure why it is not like this:
# assert diff_r2["importance"] is None
# ... but:
assert "importance" not in diff_r2
p2.importance = "SUGGESTED"
p1.value = 42
p2.value = 4
......
......@@ -5,7 +5,8 @@
# This file is a part of the LinkAhead Project.
#
# Copyright (C) 2020 Timm Fitschen <t.fitschen@indiscale.com>
# Copyright (C) 2020 IndiScale GmbH <info@indiscale.com>
# Copyright (C) 2024 Joscha Schmiedt <joscha@schmiedt.dev>
# Copyright (C) 2020-2024 IndiScale GmbH <info@indiscale.com>
#
# This program is free software: you can redistribute it and/or modify
# it under the terms of the GNU Affero General Public License as
......@@ -178,3 +179,23 @@ def test_container_deletion_with_references():
assert len(deps14a) == 1 and deps14a.pop() == -1
assert len(deps14b) == 1 and deps14b.pop() == -1
assert len(deps15) == 1 and deps15.pop() == -1
def test_container_slicing():
cont = db.Container()
cont.extend([db.Record(name=f"TestRec{ii+1}") for ii in range(5)])
assert isinstance(cont, db.common.models.Container)
container_slice = cont[:2]
assert isinstance(container_slice, db.common.models.Container), \
f"Container slice should be Container, was {type(container_slice)}"
for element in container_slice:
assert isinstance(element, db.Record), \
f"element in slice was not Record, but {type(element)}"
assert len(container_slice) == 2
assert cont[-1].name == "TestRec5"
with pytest.raises(TypeError):
cont["stringkey"]
with pytest.raises(TypeError):
cont[[0, 2, 3]]
......@@ -23,8 +23,10 @@
#
"""Tests for linkahead.common.utils."""
from __future__ import unicode_literals
from lxml.etree import Element
from linkahead.common.utils import xml2str
from linkahead.utils.escape import (escape_dquoted_text, escape_squoted_text)
from lxml.etree import Element
def test_xml2str():
......@@ -32,3 +34,12 @@ def test_xml2str():
element = Element(name)
serialized = xml2str(element)
assert serialized == "<Björn/>\n"
def test_escape_quoted_text():
assert escape_squoted_text("bla") == "bla"
assert escape_squoted_text(r"bl\a") == r"bl\\a"
assert escape_squoted_text("bl*a") == r"bl\*a"
assert escape_squoted_text(r"bl*ab\\lab\*labla") == r"bl\*ab\\\\lab\\\*labla"
assert escape_squoted_text("bl'a") == r"bl\'a"
assert escape_dquoted_text('bl"a') == r'bl\"a'