Newer
Older
# Changelog
All notable changes to this project will be documented in this file.
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
- XLSX handling: template generator and conversion from Json to XLSX
- Json schema exporter:
- has new parameter `use_rt_pool`
- propagates more properties in the `make_array` function
- Support for Python 3.12 and experimental support for 3.13
* `table_converter.to_table` now returns an empty DataFrame instead of raising a
ValueError when called with an empty container.
* The deprecated `parent` keyword from the YAML datamodel specification. Use
`inherit_from_{obligatory|recommended|suggested}` instead.
- Json schema exporter handles reference properties better.
- [#59](https://gitlab.com/linkahead/linkahead-advanced-user-tools/-/issues/59)
`to_table` failed on lists as values.
* Added documentation for json-schema datamodel export
* Added support for passing callables as `find_func` to the `BaseTableExporter`.
* Added member `BaseTableExporter.all_keys`
* Parsing from YAML now allows to give an existing model to which the YAML data model shall be
added.
* The `json_schema_exporter` module which introduces tools to create a json
schema from a RecordType, e.g., for the usage in web forms.
* `DataModel.get_deep(name: str)` method which uses the DataModel as a kind of cache pool.
* A bit better error handling in the yaml model parser.
* `TableImporter.check_datatypes` allows numeric values in string columns if
`strict=False` (default).
* `TableImporter.check_missing` in case of array-valued fields in table
* Test coverage reports are now generated in `.tox/cov_html/` by tox.
## [0.8.0] - 2023-05-30 ##
(Florian Spreckelsen)
Henrik tom Wörden
committed
- TableImporter now accepts a `existing_columns` argument which demands that certain columns exist
- The `JsonSchemaParser` class supports `patternProperties`
- The `JsonSchemaParser` calss supports json-schema references (`$ref`)
Henrik tom Wörden
committed
- The converters and datatype arguments of TableImporter now may have keys for nonexisting columns
- The `JsonSchemaParser` class does not require the top-level entry of a json
schema definition to specify a RecordType.
- refactored to work with the new default key word in FIND queries: RECORD
## [0.7.0] - 2023-03-09 ##
(Florian Spreckelsen)
- `create_entity_link` function to create html links to entities; useful for
logging
## [0.6.0] - 2022-10-11 ##
(Florian Spreckelsen)
- Unauthorized inserts can now be cached. Note that the Crawler cannot postpone
inserts but the Cache has the functionality now.
- caosdbignore; You can add one or more `.caosdbignore` files to the directory
structure that you want to make available in CaosDB and the run loadFiles.
The syntax is that of `.gitignore` files. For more information see `loadFiles`
section of the Crawler in the documentation.
## [0.5.0] - 2022-09-05 ##
(Florian Spreckelsen)
- You can now use `python -m caosadvancedtools.models.parser model_file` to
- [#36](https://gitlab.com/caosdb/caosdb-advanced-user-tools/-/issues/36)
`parent` keyword in yaml datamodel definition (replaced by
`inherit_from_{obligatory|recommended|suggested}` keywords).
## [0.4.1] - 2022-05-03 ##
(Henrik tom Wörden)
### Changed ###
- `JsonSchemaParser` now identifies `name` properties in the schema with the
CaosDB name property.
### Fixed ###
- [#40](https://gitlab.com/caosdb/caosdb-advanced-user-tools/-/issues/40)
`assure_object_is_in_list` now handles adding objects to an initially empty list correctly.
## [0.4.0] - 2022-04-05 ##
### Added ###
- `generic_analysis.py` allows to easily call scripts to perform analyses in
- **EXPERIMENTAL:** Models parser can import from Json Schema files now:
`models.parser.parse_model_from_json_schema(...)`. See the documentation of
`models.parser.JsonSchemaParser` for the limitations of the current
implementation.
- New keyword "role" in yaml data model that allows creation of Records and Files.
- It is now possible to set values of properties and default values of properties
directly in the yaml model.
- `TableConverter` now converts int to float and vice versa to match the desired dtype.
### Deprecated ###
### Removed ###
### Fixed ###
- CaosDB internal properties `name`, `unit` and `description` can now be used via the `extern`
keyword in YAML datamodel specifications.
- `check_reference_field` function to check whether entities with provided ids
exits (for example when importing data from a table)
- added the `datatypes` argument to `TableImporter` for columns that do not
## [0.3.0] - 2021-11-02 ##
### Added ###
- Error handling for missing files when using the crawler
- `send_mail` function in `caosadvancedtools.serverside.helper` module
- New class to collect possible problems with the data model
- Function to get a file path to a shared resource directory
- Function to setup logging appropriate for server side scripts with webui
- New class for collecting information for exporting tables, e.g., to
metadata repositories
- new name parsing
- new test for software folder structure
- new assure_name_is function
- two utility functions when working with files: NameCollector and
- Automated documentation builds: `make doc`
- Introduce a cfood that can create a Record structure based on the contents of a hdf5 file
- table importer implementations for csv and tsv
- string-in-list check for table imports
- AbtractCFood has new property, `update_flags`.
- identifiables of single CFoods are now treated one after the other. This
allows them to have dependencies among each other if they are ordered
- `caosadvancedtools.serverside.helper.init_data_model` also checks the role
- The `caosadvancedtools.table_importer.date_converter` now actually returns
`datetime.date` instance. A new
`caosadvancedtools.table_importer.datetime_converter` replaces the old
`date_converter` and returns a `datetime.datetime` instance.
- The suppression module is now a logging filter.
- The WebUIHandler is now a python logging formatter.
- instead of `get_entity`, type-specific functions are used in
`cfood.py` when the type of the entity in question is known.
- Logger is used instead of `print` for errors in `crawler.py`.
- complies with new exception handling, i.e., TransactionErros with
children being raised in all cases of EntityErrors (see
[#32](https://gitlab.com/caosdb/caosdb-pylib/-/issues/32) in
caosdb-pylib)
- `caosadvancedtools.cfood.assure_object_is_in_list` conducts in-place
updates if no `to_be_updated` object is supplied.
- An exception in collect_information does no longer lead to a break down.
- Removed dependency on discontiued xlrd version
- Fixed an issue where `caosadvancedtools.cache.UpdateCache` would
cause an `sqlite3.IntegrityError` if more than one change was cached
for the same entity.
- #40 Insertion of identifiables with missing obligatory properties
- Before, a Property with the datatype "LIST(TEXT)" would lead to the creation
- #52 `XLSimporter.read_xls` throwed a wrong error when reading from a file with a wrong ending.
Now, a `DataInconsistencyError` is raised instead of a ValueError.
- List properties are no longer updated unnecessarily by the crawler.