Skip to content
Snippets Groups Projects

Compare revisions

Changes are shown as if the source revision was being merged into the target revision. Learn more about comparing revisions.

Source

Select target project
No results found

Target

Select target project
  • caosdb/src/caosdb-pylib
1 result
Show changes
Commits on Source (92)
Showing with 188 additions and 864 deletions
......@@ -13,6 +13,7 @@ __pycache__/
dist/
build/
src/caosdb/version.py
src/linkahead/version.py
# documentation
_apidoc
......
#
# This file is a part of the CaosDB Project.
# This file is a part of the LinkAhead Project.
#
# Copyright (C) 2018 Research Group Biomedical Physics,
# Max-Planck-Institute for Dynamics and Self-Organization Göttingen
......@@ -44,7 +44,7 @@ code_style:
- make style
allow_failure: true
# pylint tests for pycaosdb
# pylint tests for pylinkahead
pylint:
tags: [ docker ]
stage: linting
......@@ -61,7 +61,7 @@ unittest_py3.7:
image: python:3.7
script: &python_test_script
# Python docker has problems with tox and pip so use plain pytest here
- touch ~/.pycaosdb.ini
- touch ~/.pylinkahead.ini
- pip install nose pytest pytest-cov python-dateutil jsonschema>=4.4.0
- pip install .
- python -m pytest unittests
......@@ -82,7 +82,7 @@ unittest_py3.9:
script:
# verify that this actually is Python 3.9
- python3 -c "import sys; assert sys.version.startswith('3.9')"
- touch ~/.pycaosdb.ini
- touch ~/.pylinkahead.ini
- make unittest
......
......@@ -28,6 +28,7 @@ guidelines](https://gitlab.com/caosdb/caosdb/-/blob/dev/REVIEW_GUIDELINES.md)
- [ ] Up-to-date CHANGELOG.md (or not necessary)
- [ ] Up-to-date JSON schema (or not necessary)
- [ ] Appropriate user and developer documentation (or not necessary)
- Update / write published documentation (`make doc`).
- How do I use the software? Assume "stupid" users.
- How do I develop or debug the software? Assume novice developers.
- [ ] Annotations in code (Gitlab comments)
......@@ -41,7 +42,8 @@ guidelines](https://gitlab.com/caosdb/caosdb/-/blob/dev/REVIEW_GUIDELINES.md)
- [ ] I understand the intent of this MR
- [ ] All automated tests pass
- [ ] Up-to-date CHANGELOG.md (or not necessary)
- [ ] Appropriate user and developer documentation (or not necessary)
- [ ] Appropriate user and developer documentation (or not necessary), also in published
documentation.
- [ ] The test environment setup works and the intended behavior is reproducible in the test
environment
- [ ] In-code documentation and comments are up-to-date.
......
......@@ -5,6 +5,57 @@ All notable changes to this project will be documented in this file.
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
## [Unreleased] - Date
### Added ###
### Changed ###
### Deprecated ###
### Removed ###
### Fixed ###
### Security ###
### Documentation ###
## [0.13.1] - 2023-10-11 ##
### Fixed ###
* no Error when no configuration file is used
[Issue](https://gitlab.com/linkahead/linkahead-pylib/-/issues/107)
## [0.13.0] - 2023-10-10 ##
### Added ###
* New `page_length` parameter for `caosdb.execute_query` and
`caosdb.Query.execute`. See docstrings for more details.
* `Entity.remove_value_from_property` function that removes a given value from a
property and optionally removes the property if it is empty afterwards.
### Changed ###
* `_Messages` is now `Messages` and inherits from list instead of dict
* `Message.__init__` signature changed and `type` defaults to "Info" now.
* `Message.__eq__` changed. Equality is equality of `type`, `code`, and
`description` now.
* Rename from CaosDB to LinkAhead. For proper migration, follow the instructions
in `migration_to_linkahead.md` and check the documentation at [docs.indiscale.com](https://docs.indiscale.com/caosdb-pylib/README_SETUP.html#migration).
### Deprecated ###
* The API of Messages has been simplified and some ways to interact with
messages have been deprecated. Warnings are raised correspondingly.
* `Message.get_code`. Use the `code` property instead.
### Fixed ###
- Detection for cyclic references when converting entites using the high level API.
## [0.12.0] - 2023-06-02 ##
### Added ###
......@@ -31,7 +82,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
## [0.11.2] - 2023-03-14 ##
### Fixed ###
- root logger is no longer used to create warnings. Fixes undesired output in
- root logger is no longer used to create warnings. Fixes undesired output in
stderr
## [0.11.1] - 2023-03-07 ##
......
......@@ -20,6 +20,6 @@ authors:
given-names: Stefan
orcid: https://orcid.org/0000-0001-7214-8125
title: CaosDB - Pylib
version: 0.12.0
version: 0.13.1
doi: 10.3390/data4020083
date-released: 2023-06-02
\ No newline at end of file
date-released: 2023-10-11
# ** header v3.0
# This file is a part of the CaosDB Project.
# This file is a part of the LinkAhead Project.
#
# Copyright (C) 2020 IndiScale GmbH <info@indiscale.com>
# Copyright (C) 2020 Daniel Hornung <d.hornung@indiscale.com>
......@@ -40,7 +40,7 @@ style:
.PHONY: style
lint:
pylint --unsafe-load-any-extension=y -d all -e E,F src/caosdb/common
pylint --unsafe-load-any-extension=y -d all -e E,F src/linkahead/common
.PHONY: lint
unittest:
......
......@@ -41,7 +41,7 @@ the preferred way is also a merge request as describe above (the documentation r
However, you can also create an issue for it.
* You can also contact us at **info (AT) caosdb.org** and join the
CaosDB community on
[#caosdb:matrix.org](https://matrix.to/#/!unwwlTfOznjEnMMXxf:matrix.org).
[#linkahead:matrix.org](https://matrix.to/#/!unwwlTfOznjEnMMXxf:matrix.org).
## License
......
......@@ -128,3 +128,6 @@ Build documentation in `build/` with `make doc`.
### Troubleshooting ###
If the client is to be executed directly from the `/src` folder, an initial `.\setup.py install --user` must be called.
## Migration ##
TODO
# Release Guidelines for the CaosDB Python Client Library
This document specifies release guidelines in addition to the general release
guidelines of the CaosDB Project
guidelines of the LinkAhead Project
([RELEASE_GUIDELINES.md](https://gitlab.com/caosdb/caosdb/blob/dev/RELEASE_GUIDELINES.md))
## General Prerequisites
......
#!/usr/bin/env python3
# encoding: utf-8
#
# ** header v3.0
# This file is a part of the CaosDB Project.
#
# Copyright (C) 2021 Indiscale GmbH <info@indiscale.com>
# Copyright (C) 2021 Henrik tom Wörden <h.tomwoerden@indiscale.com>
#
# This program is free software: you can redistribute it and/or modify
# it under the terms of the GNU Affero General Public License as
# published by the Free Software Foundation, either version 3 of the
# License, or (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU Affero General Public License for more details.
#
# You should have received a copy of the GNU Affero General Public License
# along with this program. If not, see <https://www.gnu.org/licenses/>.
#
# ** end header
#
import os
base_root = "src/linkahead/"
initcontent = """
from {module} import *
from warnings import warn
warn(("CaosDB was renamed to LinkAhead. Please import this library as `import {module}`. Using the"
" old name, starting with caosdb, is deprecated."), DeprecationWarning)
"""
for root, dirs, files in os.walk(base_root, topdown=False):
if root.endswith("__pycache__"):
continue
cdir = os.path.join("src/caosdb", root[len(base_root):])
os.makedirs(cdir, exist_ok=True)
for fi in files:
if not fi.endswith(".py"):
continue
path = os.path.join(cdir, fi)
with open(path, 'w') as cur:
if fi == "__init__.py":
cur.write(initcontent.format(module=".".join(
os.path.join(root, fi[:-3]).split('/')[1:-1])))
else:
cur.write(initcontent.format(module=".".join(
os.path.join(root, fi[:-3]).split('/')[1:])))
#!/usr/bin/env python3
"""A small example to get started with caosdb-pylib.
Make sure that a `pycaosdb.ini` is readable at one of the expected locations.
Make sure that a `pylinkahead.ini` is readable at one of the expected locations.
"""
import random
......
# To be found be the caosdb package, the INI file must be located either in
# - $CWD/pycaosdb.ini
# - $HOME/.pycaosdb.ini
# - $CWD/pylinkahead.ini
# - $HOME/.pylinkahead.ini
# - the location given in the env variable PYCAOSDBINI
[Connection]
......
......@@ -2,7 +2,7 @@
# -*- coding: utf-8 -*-
#
# ** header v3.0
# This file is a part of the CaosDB Project.
# This file is a part of the LinkAhead Project.
#
# Copyright (C) 2018 Research Group Biomedical Physics,
# Max-Planck-Institute for Dynamics and Self-Organization Göttingen
......
#!/usr/bin/env python3
# ** header v3.0
# This file is a part of the CaosDB Project.
# This file is a part of the LinkAhead Project.
#
# Copyright (c) 2019 IndiScale GmbH
# Copyright (c) 2019 Daniel Hornung <d.hornung@indiscale.com>
......@@ -25,7 +25,7 @@
As a result, only a specific user or group may access it.
This script assumes that the user specified in the pycaosdb.ini
This script assumes that the user specified in the pylinkahead.ini
configuration can create new entities.
"""
......
git merge linkahead-rename-step-1
# resolve potential conflicts and commit
rm -rf src/linkahead
git mv src/caosdb/ src/linkahead
rm -rf src/caosdb
python3 create_slim_linkahead_wrapper.py
git add src
git ci -m "MAINT: rename caosdb to linkahead (module)"
git merge linkahead-rename-step-2
# resolve potential conflicts and commit
git merge dev
#!/bin/bash
rm -rf dist/ build/ .eggs/
python setup.py sdist bdist_wheel
python -m twine upload -s dist/*
export PKGNAME=caosdb
python setup.py sdist bdist_wheel
python -m twine upload dist/*
......@@ -2,8 +2,9 @@
# -*- encoding: utf-8 -*-
#
#
"""caosdb"""
"""linkahead"""
import os
import re
import subprocess
import sys
......@@ -45,15 +46,15 @@ from setuptools import find_packages, setup
# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
########################################################################
ISRELEASED = True
ISRELEASED = False
MAJOR = 0
MINOR = 12
MICRO = 0
MINOR = 13
MICRO = 2
# Do not tag as pre-release until this commit
# https://github.com/pypa/packaging/pull/515
# has made it into a release. Probably we should wait for pypa/packaging>=21.4
# https://github.com/pypa/packaging/releases
PRE = "" # "dev" # e.g. rc0, alpha.1, 0.beta-23
PRE = "" # "dev" # e.g. rc0, alpha.1, 0.beta-23
if PRE:
VERSION = "{}.{}.{}-{}".format(MAJOR, MINOR, MICRO, PRE)
......@@ -91,25 +92,26 @@ def git_version():
def get_version_info():
# Adding the git rev number needs to be done inside write_version_py(),
# otherwise the import of caosdb.version messes up the build under
# otherwise the import of linkahead.version messes up the build under
# Python 3.
FULLVERSION = VERSION
# Magic which is only really needed in the pipelines. Therefore: a lot of dark pipeline magic.
GIT_REVISION = "Unknown"
if os.path.exists('.git'):
GIT_REVISION = git_version()
elif os.path.exists('caosdb_pylib_commit'):
with open('caosdb_pylib_commit', 'r') as f:
elif os.path.exists('linkahead_pylib_commit'):
with open('linkahead_pylib_commit', 'r') as f:
GIT_REVISION = f.read().strip()
elif os.path.exists('src/caosdb/version.py'):
elif os.path.exists('src/linkahead/version.py'):
# must be a source distribution, use existing version file
try:
from caosdb.version import git_revision as GIT_REVISION
except ImportError:
raise ImportError("Unable to import git_revision. Try removing "
"src/caosdb/version.py and the build directory "
"before building.")
else:
GIT_REVISION = "Unknown"
with open('src/linkahead/version.py') as fi:
rev_pattern = re.compile(r"^git_revision = '(?P<rev>.*)'$")
for line in fi.readlines():
match = rev_pattern.match(line)
if match is not None:
GIT_REVISION = match.group('rev')
break
if not ISRELEASED:
FULLVERSION += '.dev0+' + GIT_REVISION[:7]
......@@ -117,9 +119,9 @@ def get_version_info():
return FULLVERSION, GIT_REVISION
def write_version_py(filename='src/caosdb/version.py'):
def write_version_py(filename='src/linkahead/version.py'):
cnt = """
# THIS FILE IS GENERATED FROM caosdb SETUP.PY
# THIS FILE IS GENERATED FROM linkahead SETUP.PY
#
short_version = '%(version)s'
version = '%(version)s'
......@@ -153,15 +155,21 @@ def setup_package():
# Rewrite the version file everytime
write_version_py()
if 'PKGNAME' in os.environ and os.environ['PKGNAME'] == 'caosdb':
pname = 'caosdb'
pdesc = 'Deprecated! Please install linkahead.'
else:
pname = 'linkahead'
pdesc = 'Python Interface for LinkAhead'
metadata = dict(
name='caosdb',
name=pname,
version=get_version_info()[0],
description='Python Interface for CaosDB',
description=pdesc,
long_description=long_description,
long_description_content_type="text/markdown",
author='Timm Fitschen',
author_email='t.fitschen@indiscale.com',
url='https://www.caosdb.org',
url='https://www.linkahead.org',
license="AGPLv3+",
classifiers=[
"Programming Language :: Python :: 3",
......@@ -178,16 +186,19 @@ def setup_package():
"python-dateutil>=2.8.2",
'PyYAML>=5.4.1',
'future',
],
],
extras_require={'keyring': ['keyring>=13.0.0'],
'jsonschema': ['jsonschema>=4.4.0']},
setup_requires=["pytest-runner>=2.0,<3dev"],
tests_require=["pytest", "pytest-cov", "coverage>=4.4.2",
"jsonschema>=4.4.0"],
package_data={
'caosdb': ['cert/indiscale.ca.crt', 'schema-pycaosdb-ini.yml'],
'linkahead': ['cert/indiscale.ca.crt', 'schema-pycaosdb-ini.yml'],
},
scripts=["src/caosdb/utils/caosdb_admin.py"]
scripts=[
"src/linkahead/utils/caosdb_admin.py",
"src/linkahead/utils/linkahead_admin.py"
]
)
try:
setup(**metadata)
......
# -*- coding: utf-8 -*-
#
# ** header v3.0
# This file is a part of the CaosDB Project.
#
# Copyright (C) 2018 Research Group Biomedical Physics,
# Max-Planck-Institute for Dynamics and Self-Organization Göttingen
#
# This program is free software: you can redistribute it and/or modify
# it under the terms of the GNU Affero General Public License as
# published by the Free Software Foundation, either version 3 of the
# License, or (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU Affero General Public License for more details.
#
# You should have received a copy of the GNU Affero General Public License
# along with this program. If not, see <https://www.gnu.org/licenses/>.
#
# ** end header
#
"""CaosDB Python bindings.
from linkahead import *
from warnings import warn
Tries to read from the inifile specified in the environment variable `PYCAOSDBINI` or alternatively
in `~/.pycaosdb.ini` upon import. After that, the ini file `pycaosdb.ini` in the current working
directory will be read additionally, if it exists.
"""
from os import environ, getcwd
# Import of the connection function (which is used to connect to the DB):
from os.path import expanduser, join
# Import of convenience methods:
import caosdb.apiutils
from caosdb.common import administration
from caosdb.common.datatype import (BOOLEAN, DATETIME, DOUBLE, FILE, INTEGER,
LIST, REFERENCE, TEXT)
from caosdb.common.state import State, Transition
# Import of the basic API classes:
from caosdb.common.models import (ACL, ALL, FIX, NONE, OBLIGATORY, RECOMMENDED,
SUGGESTED, Container, DropOffBox, Entity,
File, Info, Message, Permissions, Property,
Query, QueryTemplate, Record, RecordType,
delete, execute_query, get_global_acl,
get_known_permissions, raise_errors)
from caosdb.utils.get_entity import get_entity_by_name, get_entity_by_path, get_entity_by_id
from caosdb.configuration import _read_config_files, configure, get_config
from caosdb.connection.connection import configure_connection, get_connection
from caosdb.exceptions import *
try:
from caosdb.version import version as __version__
except ModuleNotFoundError:
version = "uninstalled"
__version__ = version
_read_config_files()
warn(("CaosDB was renamed to LinkAhead. Please import this library as `import linkahead`. Using the"
" old name, starting with caosdb, is deprecated."), DeprecationWarning)
This diff is collapsed.
# -*- coding: utf-8 -*-
#
# This file is a part of the CaosDB Project.
#
# Copyright (C) 2023 IndiScale GmbH <info@indiscale.com>
# Copyright (C) 2023 Henrik tom Wörden <h.tomwoerden@indiscale.com>
# Copyright (C) 2023 Daniel Hornung <d.hornung@indiscale.com>
#
# This program is free software: you can redistribute it and/or modify
# it under the terms of the GNU Affero General Public License as
# published by the Free Software Foundation, either version 3 of the
# License, or (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU Affero General Public License for more details.
#
# You should have received a copy of the GNU Affero General Public License
# along with this program. If not, see <https://www.gnu.org/licenses/>.
#
"""
This module provides some cached versions of functions that retrieve Entities from a remote server.
"""
from linkahead.cached import *
from warnings import warn
from enum import Enum
from functools import lru_cache
from typing import Union
from .utils import get_entity
from .common.models import execute_query, Entity, Container
# roughly 1GB for typical entity sizes
DEFAULT_SIZE = 33333
# This dict cache is solely for filling the real cache manually (e.g. to reuse older query results)
_DUMMY_CACHE = {}
class AccessType(Enum):
"""Different access types for cached queries. Needed for filling the cache manually with
:func:`cache_fill` .
"""
QUERY = 1
PATH = 2
EID = 3
NAME = 4
def cached_get_entity_by(eid: Union[str, int] = None, name: str = None, path: str = None, query:
str = None) -> Entity:
"""Return a single entity that is identified uniquely by one argument.
You must supply exactly one argument.
If a query phrase is given, the result must be unique. If this is not what you need, use
:func:`cached_query` instead.
"""
count = 0
if eid is not None:
count += 1
if name is not None:
count += 1
if path is not None:
count += 1
if query is not None:
count += 1
if count != 1:
raise ValueError("You must supply exactly one argument.")
if eid is not None:
return _cached_access(AccessType.EID, eid, unique=True)
if name is not None:
return _cached_access(AccessType.NAME, name, unique=True)
if path is not None:
return _cached_access(AccessType.PATH, path, unique=True)
if query is not None:
return _cached_access(AccessType.QUERY, query, unique=True)
raise ValueError("Not all arguments may be None.")
def cached_query(query_string) -> Container:
"""A cached version of :func:`caosdb.execute_query<caosdb.common.models.execute_query>`.
All additional arguments are at their default values.
"""
return _cached_access(AccessType.QUERY, query_string, unique=False)
@lru_cache(maxsize=DEFAULT_SIZE)
def _cached_access(kind: AccessType, value: Union[str, int], unique=True):
# This is the function that is actually cached.
# Due to the arguments, the cache has kind of separate sections for cached_query and
# cached_get_entity_by with the different AccessTypes. However, there is only one cache size.
# The dummy dict cache is only for filling the cache manually, it is deleted afterwards.
if value in _DUMMY_CACHE:
return _DUMMY_CACHE[value]
if kind == AccessType.QUERY:
return execute_query(value, unique=unique)
if kind == AccessType.NAME:
return get_entity.get_entity_by_name(value)
if kind == AccessType.EID:
return get_entity.get_entity_by_id(value)
if kind == AccessType.PATH:
return get_entity.get_entity_by_path(value)
raise ValueError(f"Unknown AccessType: {kind}")
def cache_clear() -> None:
"""Empty the cache that is used by `cached_query` and `cached_get_entity_by`."""
_cached_access.cache_clear()
def cache_info():
"""Return info about the cache that is used by `cached_query` and `cached_get_entity_by`.
Returns
-------
out: named tuple
See the standard library :func:`functools.lru_cache` for details."""
return _cached_access.cache_info()
def cache_initialize(maxsize=DEFAULT_SIZE) -> None:
"""Create a new cache with the given size for `cached_query` and `cached_get_entity_by`.
This implies a call of :func:`cache_clear`, the old cache is emptied.
"""
cache_clear()
global _cached_access
_cached_access = lru_cache(maxsize=maxsize)(_cached_access.__wrapped__)
def cache_fill(items: dict, kind: AccessType = AccessType.EID, unique: bool = True) -> None:
"""Add entries to the cache manually.
This allows to fill the cache without actually submitting queries. Note that this does not
overwrite existing entries with the same keys.
Parameters
----------
items: dict
A dictionary with the entries to go into the cache. The keys must be compatible with the
AccessType given in ``kind``
kind: AccessType, optional
The AccessType, for example ID, name, path or query.
unique: bool, optional
If True, fills the cache for :func:`cached_get_entity_by`, presumably with
:class:`caosdb.Entity<caosdb.common.models.Entity>` objects. If False, the cache should be filled
with :class:`caosdb.Container<caosdb.common.models.Container>` objects, for use with
:func:`cached_query`.
"""
# 1. add the given items to the corresponding dummy dict cache
_DUMMY_CACHE.update(items)
# 2. call the cache function with each key (this only results in a dict look up)
for key in items.keys():
_cached_access(kind, key, unique=unique)
# 3. empty the dummy dict cache again
_DUMMY_CACHE.clear()
warn(("CaosDB was renamed to LinkAhead. Please import this library as `import linkahead.cached`. Using the"
" old name, starting with caosdb, is deprecated."), DeprecationWarning)