Newer
Older
# Changelog
All notable changes to this project will be documented in this file.
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
## [Unreleased] ##
### Added ###
* New setup extras `test` and `doc` which install the dependencies for testing and documentation.
- Using the official name "LinkAhead" wherever possible without large effort. This includes the
following exposed names / features:
- `models.data_model.LINKAHEAD_INTERNAL_PROPERTIES`
- `export_related.export` exports to `linkahead_data.xml` now.
### Deprecated ###
### Removed ###
- Bloxberg code snippets. These were just a proof of concept, untested and never used in production.
### Fixed ###
### Security ###
### Documentation ###
* Added documentation of `caosadvancedtools.loadFiles` module.
Henrik tom Wörden
committed
- Possibility to configure custom CSS to be used in static HTML sites created
by the crawler
- The static HTML pages that are generated by the crawler now require the
webinterface to be built with `JS_DIST_BUNDLE=TRUE`, i.e. all js source files
bundled into one `webcaosdb.dist.js`. This is the default and highly
recommended for production systems in any case but it will break the display
of these static files in certain corner cases where you may want to use
`JS_DIST_BUNDLE=FALSE` for development or debugging reasons.
Henrik tom Wörden
committed
- Rendering of entities in static HTML sites created by the crawler.
- [#63](https://gitlab.com/linkahead/linkahead-advanced-user-tools/-/issues/63)
The `TableImporter` now gives a detailed output in what row/column
which TypeError occurred.
- `linkahead-loadfiles` now also is a standalone script in the installation
- The `TableImporter` and its subclasses now change all integer datatypes to the
nullable `pandas.Int64Datatype` so that integer columns with empty fields can be
treated properly. In case you don't want the datatypes to be changed
automatically, initialize the `TableImporter` with
`convert_int_to_nullable_int=False`.
- [#62](https://gitlab.com/linkahead/linkahead-advanced-user-tools/-/issues/62)
The `TableImporter` now handles empty fields in integer columns by supporting
the corresponding [nullable integer
types](https://pandas.pydata.org/docs/user_guide/integer_na.html) in Pandas.
- Rudimentary documentation for `table_importer` module
- XLSX handling: template generator and conversion from Json to XLSX
- Json schema exporter:
- has new parameter `use_rt_pool`
- propagates more properties in the `make_array` function
- Support for Python 3.12 and experimental support for 3.13
* `table_converter.to_table` now returns an empty DataFrame instead of raising a
ValueError when called with an empty container.
* The deprecated `parent` keyword from the YAML datamodel specification. Use
`inherit_from_{obligatory|recommended|suggested}` instead.
- Json schema exporter handles reference properties better.
- [#59](https://gitlab.com/linkahead/linkahead-advanced-user-tools/-/issues/59)
`to_table` failed on lists as values.
* Added documentation for json-schema datamodel export
* Added support for passing callables as `find_func` to the `BaseTableExporter`.
* Added member `BaseTableExporter.all_keys`
* Parsing from YAML now allows to give an existing model to which the YAML data model shall be
added.
* The `json_schema_exporter` module which introduces tools to create a json
schema from a RecordType, e.g., for the usage in web forms.
* `DataModel.get_deep(name: str)` method which uses the DataModel as a kind of cache pool.
* A bit better error handling in the yaml model parser.
* `TableImporter.check_datatypes` allows numeric values in string columns if
`strict=False` (default).
* `TableImporter.check_missing` in case of array-valued fields in table
* YAML model parser shows "LinkAhead" and "the yaml file" in its comparison display
instead of "old" and "new".
* Test coverage reports are now generated in `.tox/cov_html/` by tox.
## [0.8.0] - 2023-05-30 ##
(Florian Spreckelsen)
Henrik tom Wörden
committed
- TableImporter now accepts a `existing_columns` argument which demands that certain columns exist
- The `JsonSchemaParser` class supports `patternProperties`
- The `JsonSchemaParser` calss supports json-schema references (`$ref`)
Henrik tom Wörden
committed
- The converters and datatype arguments of TableImporter now may have keys for nonexisting columns
- The `JsonSchemaParser` class does not require the top-level entry of a json
schema definition to specify a RecordType.
- refactored to work with the new default key word in FIND queries: RECORD
## [0.7.0] - 2023-03-09 ##
(Florian Spreckelsen)
- `create_entity_link` function to create html links to entities; useful for
logging
## [0.6.0] - 2022-10-11 ##
(Florian Spreckelsen)
- Unauthorized inserts can now be cached. Note that the Crawler cannot postpone
inserts but the Cache has the functionality now.
- caosdbignore; You can add one or more `.caosdbignore` files to the directory
structure that you want to make available in CaosDB and the run loadFiles.
The syntax is that of `.gitignore` files. For more information see `loadFiles`
section of the Crawler in the documentation.
## [0.5.0] - 2022-09-05 ##
(Florian Spreckelsen)
- You can now use `python -m caosadvancedtools.models.parser model_file` to
- [#36](https://gitlab.com/caosdb/caosdb-advanced-user-tools/-/issues/36)
`parent` keyword in yaml datamodel definition (replaced by
`inherit_from_{obligatory|recommended|suggested}` keywords).
## [0.4.1] - 2022-05-03 ##
(Henrik tom Wörden)
### Changed ###
- `JsonSchemaParser` now identifies `name` properties in the schema with the
CaosDB name property.
### Fixed ###
- [#40](https://gitlab.com/caosdb/caosdb-advanced-user-tools/-/issues/40)
`assure_object_is_in_list` now handles adding objects to an initially empty list correctly.
## [0.4.0] - 2022-04-05 ##
### Added ###
- `generic_analysis.py` allows to easily call scripts to perform analyses in
- **EXPERIMENTAL:** Models parser can import from Json Schema files now:
`models.parser.parse_model_from_json_schema(...)`. See the documentation of
`models.parser.JsonSchemaParser` for the limitations of the current
implementation.
- New keyword "role" in yaml data model that allows creation of Records and Files.
- It is now possible to set values of properties and default values of properties
directly in the yaml model.
- `TableConverter` now converts int to float and vice versa to match the desired dtype.
### Deprecated ###
### Removed ###
### Fixed ###
- CaosDB internal properties `name`, `unit` and `description` can now be used via the `extern`
keyword in YAML datamodel specifications.
- `check_reference_field` function to check whether entities with provided ids
exits (for example when importing data from a table)
- added the `datatypes` argument to `TableImporter` for columns that do not
## [0.3.0] - 2021-11-02 ##
### Added ###
- Error handling for missing files when using the crawler
- `send_mail` function in `caosadvancedtools.serverside.helper` module
- New class to collect possible problems with the data model
- Function to get a file path to a shared resource directory
- Function to setup logging appropriate for server side scripts with webui
- New class for collecting information for exporting tables, e.g., to
metadata repositories
- new name parsing
- new test for software folder structure
- new assure_name_is function
- two utility functions when working with files: NameCollector and
- Automated documentation builds: `make doc`
- Introduce a cfood that can create a Record structure based on the contents of a hdf5 file
- table importer implementations for csv and tsv
- string-in-list check for table imports
- AbtractCFood has new property, `update_flags`.
- identifiables of single CFoods are now treated one after the other. This
allows them to have dependencies among each other if they are ordered
- `caosadvancedtools.serverside.helper.init_data_model` also checks the role
- The `caosadvancedtools.table_importer.date_converter` now actually returns
`datetime.date` instance. A new
`caosadvancedtools.table_importer.datetime_converter` replaces the old
`date_converter` and returns a `datetime.datetime` instance.
- The suppression module is now a logging filter.
- The WebUIHandler is now a python logging formatter.
- instead of `get_entity`, type-specific functions are used in
`cfood.py` when the type of the entity in question is known.
- Logger is used instead of `print` for errors in `crawler.py`.
- complies with new exception handling, i.e., TransactionErros with
children being raised in all cases of EntityErrors (see
[#32](https://gitlab.com/caosdb/caosdb-pylib/-/issues/32) in
caosdb-pylib)
- `caosadvancedtools.cfood.assure_object_is_in_list` conducts in-place
updates if no `to_be_updated` object is supplied.
- An exception in collect_information does no longer lead to a break down.
- Removed dependency on discontiued xlrd version
- Fixed an issue where `caosadvancedtools.cache.UpdateCache` would
cause an `sqlite3.IntegrityError` if more than one change was cached
for the same entity.
- #40 Insertion of identifiables with missing obligatory properties
- Before, a Property with the datatype "LIST(TEXT)" would lead to the creation
- #52 `XLSimporter.read_xls` throwed a wrong error when reading from a file with a wrong ending.
Now, a `DataInconsistencyError` is raised instead of a ValueError.
- List properties are no longer updated unnecessarily by the crawler.