Skip to content
Snippets Groups Projects
Commit 7fa244c8 authored by Henrik tom Wörden's avatar Henrik tom Wörden
Browse files

Merge branch 'dev' into f-reuse-yaml-doc

parents 0908c2dc bd19f0ee
Branches
Tags
2 merge requests!107Release v0.11.0,!57FIX: if multiple updates for one entity exist, the retrieve would result in an...
Pipeline #51025 passed
Showing
with 360 additions and 64 deletions
[html]
show_contexts = True
...@@ -3,6 +3,7 @@ ...@@ -3,6 +3,7 @@
src/caosadvancedtools/version.py src/caosadvancedtools/version.py
# compiled python and dist stuff # compiled python and dist stuff
.venv
*.egg *.egg
.eggs .eggs
*.egg-info/ *.egg-info/
...@@ -17,3 +18,5 @@ build/ ...@@ -17,3 +18,5 @@ build/
# documentation # documentation
_apidoc _apidoc
/dist/
*~
...@@ -123,15 +123,58 @@ linting: ...@@ -123,15 +123,58 @@ linting:
- make lint - make lint
allow_failure: true allow_failure: true
unittest: unittest_py39:
tags: [docker] tags: [docker]
stage: unittest stage: unittest
image: $CI_REGISTRY_IMAGE image: $CI_REGISTRY_IMAGE
needs: [build-testenv] needs: [build-testenv]
script: script:
- python3 -c "import caosdb; print('CaosDB Version:', caosdb.__version__)" # First verify that system Python actually is 3.9
- python3 -c "import sys; assert sys.version.startswith('3.9')"
- python3 -c "import linkahead; print('LinkAhead Version:', linkahead.__version__)"
- tox - tox
unittest_py38:
tags: [docker]
stage: unittest
image: python:3.8
script: &python_test_script
- pip install pynose pandas pytest pytest-cov gitignore-parser openpyxl>=3.0.7 xlrd==1.2 h5py
- pip install git+https://gitlab.indiscale.com/caosdb/src/caosdb-pylib.git@dev
- pip install .
- pytest --cov=caosadvancedtools unittests
unittest_py310:
tags: [docker]
stage: unittest
image: python:3.10
script: *python_test_script
unittest_py311:
tags: [docker]
stage: unittest
image: python:3.11
script: *python_test_script
unittest_py312:
tags: [docker]
stage: unittest
image: python:3.12
script: *python_test_script
unittest_py313:
tags: [docker]
stage: unittest
image: python:3.13-rc
script:
# TODO: Replace by '*python_test_script' as soon as 3.13 has been officially released.
- apt update && apt install -y cargo || true
- pip install meson[ninja] meson-python || true
- pip install pynose pandas pytest pytest-cov gitignore-parser openpyxl>=3.0.7 xlrd==1.2 h5py || true
- pip install git+https://gitlab.indiscale.com/caosdb/src/caosdb-pylib.git@dev || true
- pip install . || true
- pytest --cov=caosadvancedtools unittests || true
# Build the sphinx documentation and make it ready for deployment by Gitlab Pages # Build the sphinx documentation and make it ready for deployment by Gitlab Pages
# Special job for serving a static website. See https://docs.gitlab.com/ee/ci/yaml/README.html#pages # Special job for serving a static website. See https://docs.gitlab.com/ee/ci/yaml/README.html#pages
pages_prepare: &pages_prepare pages_prepare: &pages_prepare
......
## Summary
*Please give a short summary of what the issue is.*
## Expected Behavior
*What did you expect how the software should behave?*
## Actual Behavior
*What did the software actually do?*
## Steps to Reproduce the Problem
*Please describe, step by step, how others can reproduce the problem. Please try these steps for yourself on a clean system.*
1.
2.
3.
## Specifications
- Version: *Which version of this software?*
- Platform: *Which operating system, which other relevant software versions?*
## Possible fixes
*Do you have ideas how the issue can be resolved?*
# Summary
*Insert a meaningful description for this merge request here: What is the new/changed behavior?
Which bug has been fixed? Are there related issues?*
# Focus
*Point the reviewer to the core of the code change. Where should they start reading? What should
they focus on (e.g. security, performance, maintainability, user-friendliness, compliance with the
specs, finding more corner cases, concrete questions)?*
# Test Environment
*How to set up a test environment for manual testing?*
# Check List for the Author
Please, prepare your MR for a review. Be sure to write a summary and a focus and create gitlab
comments for the reviewer. They should guide the reviewer through the changes, explain your changes
and also point out open questions. For further good practices have a look at [our review
guidelines](https://gitlab.com/caosdb/caosdb/-/blob/dev/REVIEW_GUIDELINES.md)
- [ ] All automated tests pass
- [ ] Reference related issues
- [ ] Up-to-date CHANGELOG.md (or not necessary)
- [ ] Up-to-date JSON schema (or not necessary)
- [ ] Appropriate user and developer documentation (or not necessary)
- Update / write published documentation (`make doc`).
- How do I use the software? Assume "stupid" users.
- How do I develop or debug the software? Assume novice developers.
- [ ] Annotations in code (Gitlab comments)
- Intent of new code
- Problems with old code
- Why this implementation?
# Check List for the Reviewer
- [ ] I understand the intent of this MR
- [ ] All automated tests pass
- [ ] Up-to-date CHANGELOG.md (or not necessary)
- [ ] Appropriate user and developer documentation (or not necessary), also in published
documentation.
- [ ] The test environment setup works and the intended behavior is reproducible in the test
environment
- [ ] In-code documentation and comments are up-to-date.
- [ ] Check: Are there specifications? Are they satisfied?
For further good practices have a look at [our review guidelines](https://gitlab.com/caosdb/caosdb/-/blob/dev/REVIEW_GUIDELINES.md).
/assign me
/target_branch dev
...@@ -8,12 +8,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ...@@ -8,12 +8,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
### Added ### ### Added ###
- Unauthorized inserts can now be cached. Note that the Crawler cannot postpone - XLSX handling: conversion from XLSX to Json
inserts but the Cache has the functionality now.
- caosdbignore; You can add one or more `.caosdbignore` files to the directory
structure that you want to make available in CaosDB and the run loadFiles.
The syntax is that of `.gitignore` files. For more information see `loadFiles`
section of the Crawler in the documentation.
### Changed ### ### Changed ###
...@@ -25,6 +20,111 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ...@@ -25,6 +20,111 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
### Security ### ### Security ###
### Documentation ###
## [0.10.0] - 2024-04-24 ##
### Added ###
- XLSX handling: template generator and conversion from Json to XLSX
- Json schema exporter:
- has new parameter `use_rt_pool`
- propagates more properties in the `make_array` function
- Support for Python 3.12 and experimental support for 3.13
### Changed ###
* `table_converter.to_table` now returns an empty DataFrame instead of raising a
ValueError when called with an empty container.
### Removed ###
* The deprecated `parent` keyword from the YAML datamodel specification. Use
`inherit_from_{obligatory|recommended|suggested}` instead.
* Support for Python 3.7
### Fixed ###
- Json schema exporter handles reference properties better.
- [#59](https://gitlab.com/linkahead/linkahead-advanced-user-tools/-/issues/59)
`to_table` failed on lists as values.
### Documentation ###
* Added documentation for json-schema datamodel export
## [0.9.0] - 2023-11-27 ##
### Added ###
* Added support for passing callables as `find_func` to the `BaseTableExporter`.
* Added member `BaseTableExporter.all_keys`
* Parsing from YAML now allows to give an existing model to which the YAML data model shall be
added.
* The `json_schema_exporter` module which introduces tools to create a json
schema from a RecordType, e.g., for the usage in web forms.
* `DataModel.get_deep(name: str)` method which uses the DataModel as a kind of cache pool.
### Changed ###
* A bit better error handling in the yaml model parser.
* `TableImporter.check_datatypes` allows numeric values in string columns if
`strict=False` (default).
### Fixed ###
* `TableImporter.check_missing` in case of array-valued fields in table
* YAML model parser has better description handling.
### Documentation ###
* Test coverage reports are now generated in `.tox/cov_html/` by tox.
## [0.8.0] - 2023-05-30 ##
(Florian Spreckelsen)
### Added ###
- TableImporter now accepts a `existing_columns` argument which demands that certain columns exist
- The `JsonSchemaParser` class supports `patternProperties`
- The `JsonSchemaParser` calss supports json-schema references (`$ref`)
### Changed ###
- The converters and datatype arguments of TableImporter now may have keys for nonexisting columns
- The `JsonSchemaParser` class does not require the top-level entry of a json
schema definition to specify a RecordType.
### Fixed ###
- refactored to work with the new default key word in FIND queries: RECORD
## [0.7.0] - 2023-03-09 ##
(Florian Spreckelsen)
### Added ###
- `create_entity_link` function to create html links to entities; useful for
logging
## [0.6.1] - 2023-01-20##
### Added ###
* Re-introduced support for Python 3.7
## [0.6.0] - 2022-10-11 ##
(Florian Spreckelsen)
### Added ###
- Unauthorized inserts can now be cached. Note that the Crawler cannot postpone
inserts but the Cache has the functionality now.
- caosdbignore; You can add one or more `.caosdbignore` files to the directory
structure that you want to make available in CaosDB and the run loadFiles.
The syntax is that of `.gitignore` files. For more information see `loadFiles`
section of the Crawler in the documentation.
## [0.5.0] - 2022-09-05 ## ## [0.5.0] - 2022-09-05 ##
(Florian Spreckelsen) (Florian Spreckelsen)
......
cff-version: 1.2.0
message: "If you use this software, please cite it as below."
authors:
- family-names: Fitschen
given-names: Timm
orcid: https://orcid.org/0000-0002-4022-432X
- family-names: Schlemmer
given-names: Alexander
orcid: https://orcid.org/0000-0003-4124-9649
- family-names: Hornung
given-names: Daniel
orcid: https://orcid.org/0000-0002-7846-6375
- family-names: tom Wörden
given-names: Henrik
orcid: https://orcid.org/0000-0002-5549-578X
- family-names: Parlitz
given-names: Ulrich
orcid: https://orcid.org/0000-0003-3058-1435
- family-names: Luther
given-names: Stefan
orcid: https://orcid.org/0000-0001-7214-8125
title: CaosDB - Advanced User Tools
version: 0.10.0
doi: 10.3390/data4020083
date-released: 2024-04-24
\ No newline at end of file
...@@ -36,10 +36,10 @@ unittest: ...@@ -36,10 +36,10 @@ unittest:
pytest-3 unittests pytest-3 unittests
style: style:
pycodestyle --count src unittests --exclude=swagger_client pycodestyle --count --exclude=swagger_client src unittests
autopep8 -ar --diff --exit-code --exclude swagger_client . autopep8 -ar --diff --exit-code --exclude swagger_client src unittests
.PHONY: style .PHONY: style
lint: lint:
pylint --unsafe-load-any-extension=y -d all -e E,F --ignore=swagger_client src/caosadvancedtools pylint --unsafe-load-any-extension=y --fail-under=9.72 -d R,C --ignore=swagger_client src/caosadvancedtools
.PHONY: lint .PHONY: lint
...@@ -4,6 +4,7 @@ ...@@ -4,6 +4,7 @@
This is the **CaosDB Advanced User Tools** repository and a part of the This is the **CaosDB Advanced User Tools** repository and a part of the
CaosDB project. CaosDB project.
This project contains tools that are beyond the typical use of This project contains tools that are beyond the typical use of
the CaosDB python client. Especially, this includes the crawler which will the CaosDB python client. Especially, this includes the crawler which will
typically be used by a data curator. typically be used by a data curator.
...@@ -43,7 +44,7 @@ Conduct](https://gitlab.com/caosdb/caosdb/-/blob/dev/CODE_OF_CONDUCT.md). ...@@ -43,7 +44,7 @@ Conduct](https://gitlab.com/caosdb/caosdb/-/blob/dev/CODE_OF_CONDUCT.md).
[documentation](https://docs.indiscale.com/caosdb-advanced-user-tools/), the [documentation](https://docs.indiscale.com/caosdb-advanced-user-tools/), the
preferred way is also a merge request as describe above (the documentation preferred way is also a merge request as describe above (the documentation
resides in `src/doc`). However, you can also create an issue for it. resides in `src/doc`). However, you can also create an issue for it.
- You can also contact us at **info (AT) caosdb.de** and join the CaosDB - You can also contact us at **info (AT) caosdb.org** and join the CaosDB
community on community on
[#caosdb:matrix.org](https://matrix.to/#/!unwwlTfOznjEnMMXxf:matrix.org). [#caosdb:matrix.org](https://matrix.to/#/!unwwlTfOznjEnMMXxf:matrix.org).
...@@ -51,7 +52,7 @@ Conduct](https://gitlab.com/caosdb/caosdb/-/blob/dev/CODE_OF_CONDUCT.md). ...@@ -51,7 +52,7 @@ Conduct](https://gitlab.com/caosdb/caosdb/-/blob/dev/CODE_OF_CONDUCT.md).
* Copyright (C) 2018 Research Group Biomedical Physics, Max Planck Institute * Copyright (C) 2018 Research Group Biomedical Physics, Max Planck Institute
for Dynamics and Self-Organization Göttingen. for Dynamics and Self-Organization Göttingen.
* Copyright (C) 2020-2021 Indiscale GmbH <info@indiscale.com> * Copyright (C) 2020-2023 Indiscale GmbH <info@indiscale.com>
All files in this repository are licensed under a [GNU Affero General Public All files in this repository are licensed under a [GNU Affero General Public
License](LICENCE.md) (version 3 or later). License](LICENCE.md) (version 3 or later).
...@@ -11,7 +11,7 @@ git clone 'https://gitlab.com/caosdb/caosdb-advanced-user-tools' ...@@ -11,7 +11,7 @@ git clone 'https://gitlab.com/caosdb/caosdb-advanced-user-tools'
Dependencies will be installed automatically if you use the below described Dependencies will be installed automatically if you use the below described
procedure. procedure.
- `caosdb>=0.6.0` - `caosdb>=0.6.0`
- `openpyxl>=3.0.0` - `openpyxl>=3.0.7`
- `xlrd>=1.2.0` - `xlrd>=1.2.0`
- `pandas>=1.2.0` - `pandas>=1.2.0`
- `numpy>=1.17.3` - `numpy>=1.17.3`
...@@ -32,7 +32,10 @@ Optional h5-crawler: ...@@ -32,7 +32,10 @@ Optional h5-crawler:
- `pip install .[h5-crawler] --user` - `pip install .[h5-crawler] --user`
## Run Unit Tests ## Run Unit Tests
`tox`
- All tests: `tox`
- One specific test with tox: `tox -- unittests/test_myusecase.py -k expression`
- Or even using only pytest: `pytest unittests/test_myusecase.py -k expression`
## Run Integration Tests Locally ## Run Integration Tests Locally
...@@ -52,6 +55,8 @@ Optional h5-crawler: ...@@ -52,6 +55,8 @@ Optional h5-crawler:
`make style` `make style`
## Documentation # ## Documentation #
We use sphinx to create the documentation. Docstrings in the code should comply
with the Googly style (see link below).
Build documentation in `build/` with `make doc`. Build documentation in `build/` with `make doc`.
...@@ -59,4 +64,11 @@ Build documentation in `build/` with `make doc`. ...@@ -59,4 +64,11 @@ Build documentation in `build/` with `make doc`.
- `sphinx` - `sphinx`
- `sphinx-autoapi` - `sphinx-autoapi`
- `sphinx-rtd-theme`
- `recommonmark >= 0.6.0` - `recommonmark >= 0.6.0`
### How to contribute ###
- [Google Style Python Docstrings](https://sphinxcontrib-napoleon.readthedocs.io/en/latest/example_google.html)
- [Google Style Python Docstrings 2nd reference](https://github.com/google/styleguide/blob/gh-pages/pyguide.md#38-comments-and-docstrings)
- [References to other documentation](https://www.sphinx-doc.org/en/master/usage/extensions/intersphinx.html#role-external)
...@@ -24,6 +24,7 @@ guidelines of the CaosDB Project ...@@ -24,6 +24,7 @@ guidelines of the CaosDB Project
- `version` variables in `src/doc/conf.py` - `version` variables in `src/doc/conf.py`
- Version on [setup.py](./setup.py): Check the `MAJOR`, `MINOR`, `MICRO`, `PRE` variables and set - Version on [setup.py](./setup.py): Check the `MAJOR`, `MINOR`, `MICRO`, `PRE` variables and set
`ISRELEASED` to `True`. Use the possibility to issue pre-release versions for testing. `ISRELEASED` to `True`. Use the possibility to issue pre-release versions for testing.
- `CITATION.cff` (update version and date)
5. Merge the release branch into the main branch. 5. Merge the release branch into the main branch.
......
...@@ -16,7 +16,7 @@ fi ...@@ -16,7 +16,7 @@ fi
OUT=/tmp/crawler.output OUT=/tmp/crawler.output
ls ls
cat pycaosdb.ini cat pycaosdb.ini
python3 -c "import caosdb; print('CaosDB Version:', caosdb.__version__)" python3 -c "import linkahead; print('LinkAhead Version:', linkahead.__version__)"
rm -rf /tmp/caosdb_identifiable_cache.db rm -rf /tmp/caosdb_identifiable_cache.db
set -e set -e
echo "Clearing database" echo "Clearing database"
...@@ -43,21 +43,23 @@ mv DataAnalysis/2010_TestProject/2019-02-03_something/README.xlsx \ ...@@ -43,21 +43,23 @@ mv DataAnalysis/2010_TestProject/2019-02-03_something/README.xlsx \
DataAnalysis/2010_TestProject/2019-02-03_something/README.xlsx_back DataAnalysis/2010_TestProject/2019-02-03_something/README.xlsx_back
cd .. cd ..
echo "run crawler" echo "run crawler"
./crawl.py / | tee $OUT ./crawl.py / | tee "$OUT"
# rename the moved file # rename the moved file
mv extroot/DataAnalysis/2010_TestProject/2019-02-03_something/README.xlsx_back \ mv extroot/DataAnalysis/2010_TestProject/2019-02-03_something/README.xlsx_back \
extroot/DataAnalysis/2010_TestProject/2019-02-03_something/README.xlsx extroot/DataAnalysis/2010_TestProject/2019-02-03_something/README.xlsx
# check whether there was something UNAUTHORIZED # check whether there was something UNAUTHORIZED
grep "There where unauthorized changes" $OUT grep "There where unauthorized changes" "$OUT"
# get the id of the run which is the last field of the output string # get the id of the run which is the last field of the output string
RUN_ID=$(grep "run id:" $OUT | awk '{ print $NF }') RUN_ID=$(grep "run id:" "$OUT" | awk '{ print $NF }')
echo $RUN_ID echo $RUN_ID
echo "run crawler again" echo "run crawler again"
echo "./crawl.py -a $RUN_ID /" echo "./crawl.py -a $RUN_ID /"
./crawl.py -a $RUN_ID / | tee $OUT ./crawl.py -a $RUN_ID / | tee "$OUT"
set +e set +e
if grep "There where unauthorized changes" $OUT if grep "There where unauthorized changes" "$OUT"
then then
echo "There still were unauthorized changes, which should not have happend!"
echo "Test FAILED"
exit 1 exit 1
fi fi
set -e set -e
...@@ -94,5 +96,8 @@ python3 -m pytest test_json_schema_datamodel_parser.py ...@@ -94,5 +96,8 @@ python3 -m pytest test_json_schema_datamodel_parser.py
echo "Testing yaml datamodel parser" echo "Testing yaml datamodel parser"
python3 -m pytest test_yaml_parser.py python3 -m pytest test_yaml_parser.py
echo "Testing json-schema exporter"
python3 -m pytest test_json_schema_exporter.py
# Obsolete due to teardown in the above test. # Obsolete due to teardown in the above test.
# echo "/n/n/n YOU NEED TO RESTART THE SERVER TO REDO TESTS!!!" # echo "/n/n/n YOU NEED TO RESTART THE SERVER TO REDO TESTS!!!"
...@@ -33,7 +33,7 @@ from caosadvancedtools.guard import (global_guard, RETRIEVE, UPDATE) ...@@ -33,7 +33,7 @@ from caosadvancedtools.guard import (global_guard, RETRIEVE, UPDATE)
def setup(): def setup():
"""Delete all test entities.""" """Delete all test entities."""
db.execute_query("FIND Test*").delete(raise_exception_on_error=False) db.execute_query("FIND ENTITY Test*").delete(raise_exception_on_error=False)
def setup_module(): def setup_module():
...@@ -105,13 +105,13 @@ def test_add_to_empty_list(): ...@@ -105,13 +105,13 @@ def test_add_to_empty_list():
db.Record(name="TestReferencingRecord").add_parent( db.Record(name="TestReferencingRecord").add_parent(
referencing_rt).add_property(list_prop, value=[]).insert() referencing_rt).add_property(list_prop, value=[]).insert()
referenced_rec = db.execute_query("FIND TestReferencedRecord", unique=True) referenced_rec = db.execute_query("FIND ENTITY TestReferencedRecord", unique=True)
referencing_rec = db.execute_query( referencing_rec = db.execute_query(
"FIND TestReferencingRecord", unique=True) "FIND ENTITY TestReferencingRecord", unique=True)
assure_object_is_in_list(referenced_rec, referencing_rec, list_prop.name) assure_object_is_in_list(referenced_rec, referencing_rec, list_prop.name)
referencing_rec = db.execute_query( referencing_rec = db.execute_query(
"FIND TestReferencingRecord", unique=True) "FIND ENTITY TestReferencingRecord", unique=True)
assert referencing_rec.get_property(list_prop.name).value == [ assert referencing_rec.get_property(list_prop.name).value == [
referenced_rec.id] referenced_rec.id]
...@@ -81,7 +81,7 @@ def insert_entities(): ...@@ -81,7 +81,7 @@ def insert_entities():
def setup_module(): def setup_module():
"""Clear all test entities""" """Clear all test entities"""
try: try:
db.execute_query("FIND Test*").delete() db.execute_query("FIND ENTITY Test*").delete()
except BaseException: except BaseException:
pass pass
......
...@@ -33,7 +33,7 @@ from caosadvancedtools.cache import UpdateCache ...@@ -33,7 +33,7 @@ from caosadvancedtools.cache import UpdateCache
class CacheTest(unittest.TestCase): class CacheTest(unittest.TestCase):
def empty_db(self): def empty_db(self):
try: try:
db.execute_query("FIND Test*").delete() db.execute_query("FIND ENTITY Test*").delete()
except Exception: except Exception:
pass pass
...@@ -63,6 +63,12 @@ class CacheTest(unittest.TestCase): ...@@ -63,6 +63,12 @@ class CacheTest(unittest.TestCase):
update = UpdateCache(db_file=self.cache) update = UpdateCache(db_file=self.cache)
run_id = "a" run_id = "a"
print(db.execute_query("FIND Record TestRecord", unique=True))
print(db.execute_query("FIND entity with id="+str(rec.id), unique=True))
try:
print(db.execute_query("FIND Record "+str(rec.id), unique=True))
except BaseException:
print("Query does not work as expected")
update.insert(cont, run_id) update.insert(cont, run_id)
assert len(update.get_updates(run_id)) == 1 assert len(update.get_updates(run_id)) == 1
......
...@@ -74,7 +74,7 @@ def test_crawler_with_data_model_problems(): ...@@ -74,7 +74,7 @@ def test_crawler_with_data_model_problems():
deleted_entities = {"Experiment", "Poster", "results"} deleted_entities = {"Experiment", "Poster", "results"}
for ent in deleted_entities: for ent in deleted_entities:
db.execute_query("FIND "+ent).delete() db.execute_query("FIND ENTITY "+ent).delete()
# Do the crawling # Do the crawling
def access(x): return "extroot" + x def access(x): return "extroot" + x
......
...@@ -40,7 +40,7 @@ def setup_module(): ...@@ -40,7 +40,7 @@ def setup_module():
"""Clear all test entities. Allow insertions.""" """Clear all test entities. Allow insertions."""
guard.set_level(INSERT) guard.set_level(INSERT)
try: try:
db.execute_query("FIND Test*").delete() db.execute_query("FIND ENTITY Test*").delete()
except Exception: except Exception:
pass pass
......
...@@ -30,7 +30,7 @@ from caosdb.apiutils import retrieve_entity_with_id ...@@ -30,7 +30,7 @@ from caosdb.apiutils import retrieve_entity_with_id
def get_entity_with_id(eid): def get_entity_with_id(eid):
return db.execute_query("FIND "+str(eid), unique=True) return db.execute_query("FIND ENTITY "+str(eid), unique=True)
class LoadFilesTest(unittest.TestCase): class LoadFilesTest(unittest.TestCase):
...@@ -49,7 +49,7 @@ class CrawlerTest(unittest.TestCase): ...@@ -49,7 +49,7 @@ class CrawlerTest(unittest.TestCase):
# # dummy for dependency test experiment # # # # dummy for dependency test experiment # #
######################## ########################
exp = db.execute_query( exp = db.execute_query(
"FIND Experiment with date=2019-02-04 and identifier=empty_identifier", "FIND ENTITY Experiment with date=2019-02-04 and identifier=empty_identifier",
unique=True) unique=True)
######################## ########################
...@@ -59,7 +59,7 @@ class CrawlerTest(unittest.TestCase): ...@@ -59,7 +59,7 @@ class CrawlerTest(unittest.TestCase):
# vanishing of the property # vanishing of the property
# thus an x is used here. Needs to be fixed. # thus an x is used here. Needs to be fixed.
exp = db.execute_query( exp = db.execute_query(
"FIND Experiment with date=2019-02-03 and identifier=empty_identifier", "FIND ENTITY Experiment with date=2019-02-03 and identifier=empty_identifier",
unique=True) unique=True)
# There should be a Project with name TestProject which is referenced # There should be a Project with name TestProject which is referenced
...@@ -99,7 +99,7 @@ class CrawlerTest(unittest.TestCase): ...@@ -99,7 +99,7 @@ class CrawlerTest(unittest.TestCase):
# # second experiment # # # # second experiment # #
######################### #########################
exp = db.execute_query( exp = db.execute_query(
"FIND Experiment with date=2019-02-03 and identifier='something'", "FIND ENTITY Experiment with date=2019-02-03 and identifier='something'",
unique=True) unique=True)
# Should be the same project # Should be the same project
...@@ -120,7 +120,7 @@ class CrawlerTest(unittest.TestCase): ...@@ -120,7 +120,7 @@ class CrawlerTest(unittest.TestCase):
# # first analysis # # # # first analysis # #
###################### ######################
ana = db.execute_query( ana = db.execute_query(
"FIND Analysis with date=2019-02-03 and identifier='empty_identifier'", "FIND ENTITY Analysis with date=2019-02-03 and identifier='empty_identifier'",
unique=True) unique=True)
# There should be a Project with name TestProject which is referenced # There should be a Project with name TestProject which is referenced
...@@ -164,7 +164,7 @@ class CrawlerTest(unittest.TestCase): ...@@ -164,7 +164,7 @@ class CrawlerTest(unittest.TestCase):
# # second analysis # # # # second analysis # #
####################### #######################
ana = db.execute_query( ana = db.execute_query(
"FIND Analysis with date=2019-02-03 and identifier='something'", "FIND ENTITY Analysis with date=2019-02-03 and identifier='something'",
unique=True) unique=True)
# Should be the same project # Should be the same project
...@@ -197,7 +197,7 @@ class CrawlerTest(unittest.TestCase): ...@@ -197,7 +197,7 @@ class CrawlerTest(unittest.TestCase):
# # first simulation # # # # first simulation # #
###################### ######################
sim = db.execute_query( sim = db.execute_query(
"FIND Simulation with date=2019-02-03 and identifier='empty_identifier'", "FIND ENTITY Simulation with date=2019-02-03 and identifier='empty_identifier'",
unique=True) unique=True)
# There should be a Project with name TestProject which is referenced # There should be a Project with name TestProject which is referenced
...@@ -228,7 +228,7 @@ class CrawlerTest(unittest.TestCase): ...@@ -228,7 +228,7 @@ class CrawlerTest(unittest.TestCase):
# # second simulation # # # # second simulation # #
######################### #########################
sim = db.execute_query( sim = db.execute_query(
"FIND Simulation with date=2019-02-03 and identifier='something'", "FIND ENTITY Simulation with date=2019-02-03 and identifier='something'",
unique=True) unique=True)
sources = [get_entity_with_id(el) for el in sources = [get_entity_with_id(el) for el in
...@@ -273,7 +273,7 @@ class CrawlerTest(unittest.TestCase): ...@@ -273,7 +273,7 @@ class CrawlerTest(unittest.TestCase):
######################### #########################
# # first publication # # # # first publication # #
######################### #########################
pub = db.execute_query("FIND *really_cool_finding", unique=True) pub = db.execute_query("FIND ENTITY *really_cool_finding", unique=True)
# There should be a file as result attached with path poster.pdf # There should be a file as result attached with path poster.pdf
datfile_id = pub.get_property("results").value[0] datfile_id = pub.get_property("results").value[0]
...@@ -291,7 +291,7 @@ class CrawlerTest(unittest.TestCase): ...@@ -291,7 +291,7 @@ class CrawlerTest(unittest.TestCase):
########################## ##########################
# # second publication # # # # second publication # #
########################## ##########################
pub = db.execute_query("FIND *paper_on_exciting_stuff ", unique=True) pub = db.execute_query("FIND ENTITY *paper_on_exciting_stuff ", unique=True)
# Test type # Test type
self.assertEqual(pub.parents[0].name, "Thesis") self.assertEqual(pub.parents[0].name, "Thesis")
...@@ -311,10 +311,10 @@ class CrawlerTest(unittest.TestCase): ...@@ -311,10 +311,10 @@ class CrawlerTest(unittest.TestCase):
# # first software version # # # # first software version # #
############################## ##############################
ana = db.execute_query( ana = db.execute_query(
"FIND Software with version='V1.0-rc1'", unique=True) "FIND ENTITY Software with version='V1.0-rc1'", unique=True)
sw = db.execute_query( sw = db.execute_query(
"FIND Software with name='2010_TestSoftware'", unique=True) "FIND ENTITY Software with name='2010_TestSoftware'", unique=True)
assert sw.get_property("alias").value == "TestSoftware" assert sw.get_property("alias").value == "TestSoftware"
# The software record should inherit from the correct software # The software record should inherit from the correct software
...@@ -360,10 +360,10 @@ class CrawlerTest(unittest.TestCase): ...@@ -360,10 +360,10 @@ class CrawlerTest(unittest.TestCase):
# # second software version # # # # second software version # #
####################### #######################
ana = db.execute_query( ana = db.execute_query(
"FIND Software with version='v0.1'", unique=True) "FIND ENTITY Software with version='v0.1'", unique=True)
sw = db.execute_query( sw = db.execute_query(
"FIND Software with name='2010_TestSoftware'", unique=True) "FIND ENTITY Software with name='2010_TestSoftware'", unique=True)
# The software record should inherit from the correct software # The software record should inherit from the correct software
assert sw.id == ana.get_parents()[0].id assert sw.id == ana.get_parents()[0].id
...@@ -393,11 +393,11 @@ class CrawlerTest(unittest.TestCase): ...@@ -393,11 +393,11 @@ class CrawlerTest(unittest.TestCase):
# # third software version # # # # third software version # #
####################### #######################
ana = db.execute_query( ana = db.execute_query(
"FIND Software with date='2020-02-04' and not version", "FIND ENTITY Software with date='2020-02-04' and not version",
unique=True) unique=True)
sw = db.execute_query( sw = db.execute_query(
"FIND Software with name='2020NewProject0X'", unique=True) "FIND ENTITY Software with name='2020NewProject0X'", unique=True)
# The software record should inherit from the correct software # The software record should inherit from the correct software
assert sw.id == ana.get_parents()[0].id assert sw.id == ana.get_parents()[0].id
...@@ -438,11 +438,11 @@ class CrawlerTest(unittest.TestCase): ...@@ -438,11 +438,11 @@ class CrawlerTest(unittest.TestCase):
# # fourth software version # # # # fourth software version # #
####################### #######################
ana = db.execute_query( ana = db.execute_query(
"FIND Software with date='2020-02-03' and not version", "FIND ENTITY Software with date='2020-02-03' and not version",
unique=True) unique=True)
sw = db.execute_query( sw = db.execute_query(
"FIND Software with name='2020NewProject0X'", unique=True) "FIND ENTITY Software with name='2020NewProject0X'", unique=True)
assert sw.get_property("alias").value == "NewProject0X" assert sw.get_property("alias").value == "NewProject0X"
# The software record should inherit from the correct software # The software record should inherit from the correct software
...@@ -479,10 +479,10 @@ class CrawlerTest(unittest.TestCase): ...@@ -479,10 +479,10 @@ class CrawlerTest(unittest.TestCase):
# # fifth software version # # # # fifth software version # #
############################## ##############################
ana = db.execute_query( ana = db.execute_query(
"FIND Software with version='second'", unique=True) "FIND ENTITY Software with version='second'", unique=True)
sw = db.execute_query( sw = db.execute_query(
"FIND Software with name='2020NewProject0X'", unique=True) "FIND ENTITY Software with name='2020NewProject0X'", unique=True)
assert sw.get_property("alias").value == "NewProject0X" assert sw.get_property("alias").value == "NewProject0X"
# The software record should inherit from the correct software # The software record should inherit from the correct software
......
import unittest import unittest
import pytest
import caosdb as db import caosdb as db
from caosadvancedtools.models.data_model import DataModel from caosadvancedtools.models.data_model import DataModel
...@@ -55,9 +56,22 @@ class DataModelTest(unittest.TestCase): ...@@ -55,9 +56,22 @@ class DataModelTest(unittest.TestCase):
assert len(exist) == 1 assert len(exist) == 1
assert exist[0].name == "TestRecord" assert exist[0].name == "TestRecord"
def test_large_data_model(self):
# create RT and one property
dm = DataModel()
long = "Long" * 50
first_RT = db.RecordType(name=f"TestRecord_first")
for index in range(20):
this_RT = db.RecordType(name=f"TestRecord_{long}_{index:02d}")
first_RT.add_property(this_RT)
dm.append(this_RT)
dm.append(first_RT)
dm.sync_data_model(noquestion=True) # Insert
dm.sync_data_model(noquestion=True) # Check again
def tearDown(self): def tearDown(self):
try: try:
tests = db.execute_query("FIND test*") tests = db.execute_query("FIND ENTITY test*")
tests.delete() tests.delete()
except Exception: except Exception:
pass pass
...@@ -39,7 +39,7 @@ def setup_module(): ...@@ -39,7 +39,7 @@ def setup_module():
"""Clear problem sets and delete possible test entities""" """Clear problem sets and delete possible test entities"""
DataModelProblems.missing.clear() DataModelProblems.missing.clear()
try: try:
db.execute_query("FIND Test*").delete() db.execute_query("FIND Entity Test*").delete()
except Exception as delete_exc: except Exception as delete_exc:
print(delete_exc) print(delete_exc)
......
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Please register or to comment