caosdb-advanced-user-tools issueshttps://gitlab.indiscale.com/caosdb/src/caosdb-advanced-user-tools/-/issues2023-09-11T11:19:56Zhttps://gitlab.indiscale.com/caosdb/src/caosdb-advanced-user-tools/-/issues/113Yaml model: Missing documentation for `role: Record`2023-09-11T11:19:56ZDaniel HornungYaml model: Missing documentation for `role: Record`## Summary
The user documentation should have a chapter for creating Records via the Yaml model parser.
The documentation should contain:
- A description how the `role: Record` and `value: foo` work.
- A note that a large `extern:` se...## Summary
The user documentation should have a chapter for creating Records via the Yaml model parser.
The documentation should contain:
- A description how the `role: Record` and `value: foo` work.
- A note that a large `extern:` section may be required.
- A tutorial-like example with
- A YAML data model
- In a different file, the YAML Records data
- A short script for inserting the data.Alexander SchlemmerAlexander Schlemmerhttps://gitlab.indiscale.com/caosdb/src/caosdb-advanced-user-tools/-/issues/112yaml parser: remove short-cut values2024-03-13T20:27:16ZDaniel Hornungyaml parser: remove short-cut values## Summary
Short cuts for value assignment should be removed. Details are given in #109
## Expected Behavior
Short cut usage should raise an error.
## Actual Behavior
Short cuts are interpreted as value.
## Steps to Reproduce the ...## Summary
Short cuts for value assignment should be removed. Details are given in #109
## Expected Behavior
Short cut usage should raise an error.
## Actual Behavior
Short cuts are interpreted as value.
## Steps to Reproduce the Problem
```yaml
A:
description: some RecordType
B:
obligatory_properties:
A: LIST<A>
```
## Specifications
- Version: lates dev
- Platform: Linux
## Possible fixes
- Remove the short cuts.
- Reactive existing unit tests for this behavior.Daniel HornungDaniel Hornunghttps://gitlab.indiscale.com/caosdb/src/caosdb-advanced-user-tools/-/issues/111hidden dependency in utility function (for tests?)2023-08-29T09:31:25ZAlexander Schlemmerhidden dependency in utility function (for tests?)I am a bit confused by the following function defined in `loadFiles.py`:
```python
def compile_file_list(caosdbignore, localpath):
"""creates a list of files that contain all files under localpath except
those excluded by caosdb...I am a bit confused by the following function defined in `loadFiles.py`:
```python
def compile_file_list(caosdbignore, localpath):
"""creates a list of files that contain all files under localpath except
those excluded by caosdbignore
"""
from gitignore_parser import parse_gitignore
matches = parse_gitignore(caosdbignore)
# (...)
```
It imports (upon calling) from an optional dependency `gitignore_parser`, although the function is not contained in a test file.
Am I missing anything or should rather either:
- the function be contained in a test file or test module only?
- the gitignore_parser added as a real dependency and the import be located on top of the file?Daniel HornungDaniel Hornunghttps://gitlab.indiscale.com/caosdb/src/caosdb-advanced-user-tools/-/issues/107Where should the crawler use versioned references?2023-06-30T07:55:52ZHenrik tom WördenWhere should the crawler use versioned references?Florian SpreckelsenFlorian Spreckelsenhttps://gitlab.indiscale.com/caosdb/src/caosdb-advanced-user-tools/-/issues/103Use library function has_parent2023-02-22T08:37:17ZHenrik tom WördenUse library function has_parent## Summary
The following commit introduced the library function `has_parent` in a unit test. However this caused a failure.
0e6af72ece9591c95ed11d4f88800dd79e619e3c
Investigate how it is possible, that a parent with value `None` inste...## Summary
The following commit introduced the library function `has_parent` in a unit test. However this caused a failure.
0e6af72ece9591c95ed11d4f88800dd79e619e3c
Investigate how it is possible, that a parent with value `None` instead of db.Entity can exist and fix it.https://gitlab.indiscale.com/caosdb/src/caosdb-advanced-user-tools/-/issues/102loadpath ignores files starting with a "."?2023-02-20T20:20:01ZHenrik tom Wördenloadpath ignores files starting with a "."?## Summary
I have the impression, that files which have a file name that start with a "." are being ignored. Is that documented??? TODO finish this issue.
## Expected Behavior
*What did you expect how the software should behave?*
## ...## Summary
I have the impression, that files which have a file name that start with a "." are being ignored. Is that documented??? TODO finish this issue.
## Expected Behavior
*What did you expect how the software should behave?*
## Actual Behavior
*What did the software actually do?*
## Steps to Reproduce the Problem
*Please describe, step by step, how others can reproduce the problem. Please try these steps for yourself on a clean system.*
1.
2.
3.
## Specifications
- Version: *Which version of this software?*
- Platform: *Which operating system, which other relevant software versions?*
## Possible fixes
*Do you have ideas how the issue can be resolved?*https://gitlab.indiscale.com/caosdb/src/caosdb-advanced-user-tools/-/issues/90Deletion of data sources and the CaosDB Crawler2022-05-18T09:15:49ZHenrik tom WördenDeletion of data sources and the CaosDB CrawlerThe CaosDB Crawler was designed to be able to scan some (File-)Structure and insert Records in CaosDB as needed. The design, that it reacts to what it finds (e.g. there is a `README.md` in a folder -> create a DataSet Record with appropr...The CaosDB Crawler was designed to be able to scan some (File-)Structure and insert Records in CaosDB as needed. The design, that it reacts to what it finds (e.g. there is a `README.md` in a folder -> create a DataSet Record with appropriate Properties) imply an important caveat: It does not react to what it does not find. I.e. it does not react to removed stuff.
Example: A folder with a `README.md` file is translated into a data set. If that folder is removed, it is not found by the crawler. The data set remains in the CaosDB server.
Users might not expect this behavior since they expect some synchronization. If an addition is treated automatically, why wouldn't a deletion???
This also impacts moving or renaming files.
In many cases a reverse check would not even be possible since Records can stem from multiple sources. I.e. you cannot simply remove data sets that would not be created by a crawler run because a user could have created them manually or there could be another crawler etc.
One possibility would be to create some user assistance based on caching: Track, what files (or structure elements in general) were used by a particular crawler to create Records. If such a file is removed, it might be at least indicated to the user. This could be expanded with advice, what to do or even the option to automatically fix it (adjust Records after renaming of a file or deletion of Records due to file removal).Henrik tom WördenHenrik tom Wördenhttps://gitlab.indiscale.com/caosdb/src/caosdb-advanced-user-tools/-/issues/89It's not possible in the yaml parser to define a property, use it and assign ...2023-12-13T13:59:17ZAlexander SchlemmerIt's not possible in the yaml parser to define a property, use it and assign a value for the property of the recordThe following is currently not possible:
```yaml
prop:
datatype: INTEGER
rec:
obligatory_properties:
prop:
value: 2
```The following is currently not possible:
```yaml
prop:
datatype: INTEGER
rec:
obligatory_properties:
prop:
value: 2
```Florian SpreckelsenFlorian Spreckelsenhttps://gitlab.indiscale.com/caosdb/src/caosdb-advanced-user-tools/-/issues/84Add unittests for datamodel schema and warn in case of validation error2022-06-13T07:00:20ZFlorian SpreckelsenAdd unittests for datamodel schema and warn in case of validation error- [ ] A warning is raised instead of an error in case of a validation error ("... will be necessary in the future ...")
- [ ] Collect datamodel.ymls from customer repos and add them to validation unit tests
- [ ] Also test faulty datamod...- [ ] A warning is raised instead of an error in case of a validation error ("... will be necessary in the future ...")
- [ ] Collect datamodel.ymls from customer repos and add them to validation unit tests
- [ ] Also test faulty datamodel.ymls whether a validation warning is raisedFlorian SpreckelsenFlorian Spreckelsenhttps://gitlab.indiscale.com/caosdb/src/caosdb-advanced-user-tools/-/issues/83`to_table` fails on sub-properties2022-04-19T10:00:44ZDaniel Hornung`to_table` fails on sub-properties## Summary
The SELECT query allows to return sub-properties, for example `SELECT MusicalInstrument.price FROM MusicalAnalysis`. The resulting object however is not converted properly to a DataFrame by the `to_table` function.
## Expec...## Summary
The SELECT query allows to return sub-properties, for example `SELECT MusicalInstrument.price FROM MusicalAnalysis`. The resulting object however is not converted properly to a DataFrame by the `to_table` function.
## Expected Behavior
The columns of the DataFrame should correspond to the property name chain, for example `MusicalInstrument.Manufacturer.Location`.
## Actual Behavior
Currently, `to_table` just creates a column with the first level, like `MusicalInstrument`. The content is the Entity's XML string representation.
## Steps to Reproduce the Problem
Using the demo server:
```python
data = db.execute_query("SELECT quality_factor, MusicalInstrument.price, MusicalInstrument.Manufacturer FROM RECORD MusicalAnalysis" )
table = to_table(data)
print(table)
```
## Specifications
- Version: Current dev branch of pycaosdb and caosadvancedtools.
- Platform: Linux
## Possible fixes
The `to_table` could walk through all properties of the entities in the returned object. Possibly the XML parser needs to be improved for that.Daniel HornungDaniel Hornunghttps://gitlab.indiscale.com/caosdb/src/caosdb-advanced-user-tools/-/issues/82Should global configuration file be used in unittests2022-04-04T13:32:50ZHenrik tom WördenShould global configuration file be used in unittests## Summary
A setting in the global `.ini` file (e.g. `debug=1`) changes the behavior of the unittests. Is this desired? If not, how should this be changed?## Summary
A setting in the global `.ini` file (e.g. `debug=1`) changes the behavior of the unittests. Is this desired? If not, how should this be changed?https://gitlab.indiscale.com/caosdb/src/caosdb-advanced-user-tools/-/issues/80Clean up scifolder linting2023-06-30T12:51:48ZDaniel HornungClean up scifolder linting## Summary
Currently the scifolder code only passes the linter with a massive amount of `disable` directives. Many of these probably mean that it is very easy to cause exceptions by proper usage of the code.
## Expected Behavior
`pyl...## Summary
Currently the scifolder code only passes the linter with a massive amount of `disable` directives. Many of these probably mean that it is very easy to cause exceptions by proper usage of the code.
## Expected Behavior
`pylint` should just accept the code.
## Actual Behavior
```console
src/caosadvancedtools/scifolder$ grep -rn "pylint: disable"
simulation_cfood.py:91: if SOURCES.key in self.header: # pylint: disable=unsupported-membership-test
simulation_cfood.py:96: globs=get_glob(self.header[SOURCES.key]), # pylint: disable=unsubscriptable-object
simulation_cfood.py:102: if REVISIONOF.key in self.header: # pylint: disable=unsupported-membership-test
simulation_cfood.py:105: recordtypes=[dm.Software], # pylint: disable=no-member
simulation_cfood.py:107: globs=get_glob(self.header[dm.revisionOf]), # pylint: disable=unsubscriptable-object
utils.py:115: return list(df.ID) # pylint: disable=no-member
withreadme.py:124: if self.crawled_path.lower().endswith(".md"): # pylint: disable=no-member
withreadme.py:126: fileguide.access(self.crawled_path)) # pylint: disable=no-member
withreadme.py:127: elif self.crawled_path.lower().endswith(".xlsx"): # pylint: disable=no-member
withreadme.py:129: fileguide.access(self.crawled_path)) # pylint: disable=no-member
withreadme.py:148: globs, prefix=os.path.dirname(self.crawled_path)) # pylint: disable=no-member
withreadme.py:163: self.attached_filenames.extend(flat_list) # pylint: disable=no-member
withreadme.py:188: for field in self.win_paths: # pylint: disable=no-member
withreadme.py:248: to_be_updated=self.to_be_updated, # pylint: disable=no-member
withreadme.py:258: if field.key not in self.header: # pylint: disable=no-member
withreadme.py:262: for item in self.header[field.key]: # pylint: disable=no-member
withreadme.py:267: os.path.dirname(self.crawled_path), # pylint: disable=no-member
```
## Steps to Reproduce the Problem
Run `pylint` on the code before the `disable` directives were added. Or:
```python
from caosadvancedtools.scifolder.simulation_cfood import SimulationCFood
scf = SimulationCFood(".")
scf.update_identifiables()
# raises AttributeError: 'SimulationCFood' object has no attribute 'simulation'
```
## Specifications
- Version: Latest dev branch commit
- Platform: Debian/Linux
## Possible fixes
- Maybe add more values to the `__init__()` methods?Alexander SchlemmerAlexander Schlemmerhttps://gitlab.indiscale.com/caosdb/src/caosdb-advanced-user-tools/-/issues/77Yaml data model sync confuses "new" and "old" in interactive output2022-03-16T11:06:14ZAlexander SchlemmerYaml data model sync confuses "new" and "old" in interactive outputWhen displaying changes, the yaml data model syncer refers to the new entities with the word "old" and vice versa...
That should be changed.When displaying changes, the yaml data model syncer refers to the new entities with the word "old" and vice versa...
That should be changed.Alexander SchlemmerAlexander Schlemmerhttps://gitlab.indiscale.com/caosdb/src/caosdb-advanced-user-tools/-/issues/76Units are not updated by `DataModel.sync_data_model`2022-03-16T11:03:18ZFlorian SpreckelsenUnits are not updated by `DataModel.sync_data_model`## Summary
When re-defining a property in, e.g., a `datamodel.yml` with a different unit than the already existing property, the unit is not updated.
## Expected Behavior
The unit should be updated.
## Actual Behavior
`sync_data_mod...## Summary
When re-defining a property in, e.g., a `datamodel.yml` with a different unit than the already existing property, the unit is not updated.
## Expected Behavior
The unit should be updated.
## Actual Behavior
`sync_data_model` doesn't see any necessary changes.
## Steps to Reproduce the Problem
Consider
```python
import caosdb as db
from caosadvancedtools.models.parser import parse_model_from_string
db.Property(name="TestProp", datatype="INTEGER", unit="m").insert()
model = """
TestProp:
datatype: INTEGER
unit: cm
"""
entities = parse_model_from_string(model)
entities.sync_data_model()
```
Results in
```
No new entities.
Inspecting changes that will be made...
No differences found. No update
```
## Specifications
- Version: `dev`Florian SpreckelsenFlorian Spreckelsenhttps://gitlab.indiscale.com/caosdb/src/caosdb-advanced-user-tools/-/issues/75Yaml-model: Implement `valued_properties` keyword2022-03-14T16:51:21ZFlorian SpreckelsenYaml-model: Implement `valued_properties` keywordIn addition to the existing `obligatory_properties`, there should be a `valued_properties` keyword, especially for `role: Record`. Contents of this keyword could look like
```yaml
TestRecord:
role: Record
valued_properties:
a: ...In addition to the existing `obligatory_properties`, there should be a `valued_properties` keyword, especially for `role: Record`. Contents of this keyword could look like
```yaml
TestRecord:
role: Record
valued_properties:
a: 1
b:
value: 2
```Alexander SchlemmerFlorian SpreckelsenAlexander Schlemmerhttps://gitlab.indiscale.com/caosdb/src/caosdb-advanced-user-tools/-/issues/74Yaml model: Implement `role: File`2022-03-14T12:52:49ZFlorian SpreckelsenYaml model: Implement `role: File`This also needs a mandatory `path` keyword.This also needs a mandatory `path` keyword.Alexander SchlemmerFlorian SpreckelsenAlexander Schlemmerhttps://gitlab.indiscale.com/caosdb/src/caosdb-advanced-user-tools/-/issues/73Replace `latest` image tag in pipeline2022-05-17T07:42:32ZFlorian SpreckelsenReplace `latest` image tag in pipelineSince all (?) images are tagged `latest`, images may be mixed up when running pipelines for different f-branches at the same time or in short succession of each other.
- [ ] The pipeline tags `testenv` images with the branch name (or an...Since all (?) images are tagged `latest`, images may be mixed up when running pipelines for different f-branches at the same time or in short succession of each other.
- [ ] The pipeline tags `testenv` images with the branch name (or any other suitable unique tag)Florian SpreckelsenFlorian Spreckelsenhttps://gitlab.indiscale.com/caosdb/src/caosdb-advanced-user-tools/-/issues/71Setting values for previously declared properties in yml data model currently...2022-03-05T13:03:24ZAlexander SchlemmerSetting values for previously declared properties in yml data model currently failsE.g.:
```yaml
parameter:
datatype: INTEGER
Simulation:
role: Record
obligatory_properties:
parameter: 26
```E.g.:
```yaml
parameter:
datatype: INTEGER
Simulation:
role: Record
obligatory_properties:
parameter: 26
```Alexander SchlemmerAlexander Schlemmerhttps://gitlab.indiscale.com/caosdb/src/caosdb-advanced-user-tools/-/issues/70Trigger advancedtools pipeline when pyinttests have succeeded2024-03-18T10:22:55ZFlorian SpreckelsenTrigger advancedtools pipeline when pyinttests have succeededIn order not to overlook bugs, the caosadvancedtools pipeline should be triggered automatically, too.
- [x] advancedtools pipeline is triggered automatically
- [ ] tested also with f-branch logic (i.e., f-something branch in other libra...In order not to overlook bugs, the caosadvancedtools pipeline should be triggered automatically, too.
- [x] advancedtools pipeline is triggered automatically
- [ ] tested also with f-branch logic (i.e., f-something branch in other libraries should trigger f-something in advancedtools if present)Florian SpreckelsenFlorian Spreckelsenhttps://gitlab.indiscale.com/caosdb/src/caosdb-advanced-user-tools/-/issues/63Sync in data_model show misleading warnings2022-01-05T13:30:16ZAlexander SchlemmerSync in data_model show misleading warningsCurrently, running the `model.sync_data_model(noquestion=True)` function from data_model twice on the same yml file (assuming an empty database leads to e.g. the following output:
```
## Difference between the old and the new version of...Currently, running the `model.sync_data_model(noquestion=True)` function from data_model twice on the same yml file (assuming an empty database leads to e.g. the following output:
```
## Difference between the old and the new version of Simulation
property responsible differs:
old version: {'datatype': None}
new version: {'datatype': 'LIST<Person>'}
property date differs:
old version: {'datatype': None}
new version: {'datatype': 'DATETIME'}
property identifier differs:
old version: {'datatype': None}
new version: {'datatype': 'TEXT'}
```
So, there is a difference reported, that cannot exist. Probably due to information that was not downloaded by the client before the comparison.
You can use the following yaml to reproduce:
```yaml
Simulation:
obligatory_properties:
date:
datatype: DATETIME
description: 'date of the simulation'
identifier:
datatype: TEXT
description: 'identifier of the simulation'
responsible:
datatype: LIST<Person>
```
Probably the xml of the retrieved properties of the record types do not contain the datatype attributes and need to be retrieved explicitly before the comparison.Florian SpreckelsenFlorian Spreckelsen