# Changelog
All notable changes to this project will be documented in this file.

The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).

## [0.10.0] - 2024-04-24 ##

### Added ###

- XLSX handling: template generator and conversion from Json to XLSX
- Json schema exporter:
  - has new parameter `use_rt_pool`
  - propagates more properties in the `make_array` function
- Support for Python 3.12 and experimental support for 3.13

### Changed ###

* `table_converter.to_table` now returns an empty DataFrame instead of raising a
  ValueError when called with an empty container.

### Removed ###

* The deprecated `parent` keyword from the YAML datamodel specification. Use
  `inherit_from_{obligatory|recommended|suggested}` instead.
* Support for Python 3.7

### Fixed ###

- Json schema exporter handles reference properties better.
- [#59](https://gitlab.com/linkahead/linkahead-advanced-user-tools/-/issues/59)
  `to_table` failed on lists as values.

### Documentation ###

* Added documentation for json-schema datamodel export

## [0.9.0] - 2023-11-27 ##

### Added ###

* Added support for passing callables as `find_func` to the `BaseTableExporter`.
* Added member `BaseTableExporter.all_keys`
* Parsing from YAML now allows to give an existing model to which the YAML data model shall be
  added.
* The `json_schema_exporter` module which introduces tools to create a json
  schema from a RecordType, e.g., for the usage in web forms.
* `DataModel.get_deep(name: str)` method which uses the DataModel as a kind of cache pool.

### Changed ###

* A bit better error handling in the yaml model parser.
* `TableImporter.check_datatypes` allows numeric values in string columns if
  `strict=False` (default).

### Fixed ###

* `TableImporter.check_missing` in case of array-valued fields in table
* YAML model parser has better description handling.

### Documentation ###

* Test coverage reports are now generated in `.tox/cov_html/` by tox.

## [0.8.0] - 2023-05-30 ##
(Florian Spreckelsen)

### Added ###

- TableImporter now accepts a `existing_columns` argument which demands that certain columns exist
- The `JsonSchemaParser` class supports `patternProperties`
- The `JsonSchemaParser` calss supports json-schema references (`$ref`)

### Changed ###

- The converters and datatype arguments of TableImporter now may have keys for nonexisting columns
- The `JsonSchemaParser` class does not require the top-level entry of a json
  schema definition to specify a RecordType.

### Fixed ###

- refactored to work with the new default key word in FIND queries: RECORD

## [0.7.0] - 2023-03-09 ##
(Florian Spreckelsen)

### Added ###

- `create_entity_link` function to create html links to entities; useful for
  logging

## [0.6.1] - 2023-01-20##

### Added ###

* Re-introduced support for Python 3.7

## [0.6.0] - 2022-10-11 ##
(Florian Spreckelsen)

### Added ###

- Unauthorized inserts can now be cached. Note that the Crawler cannot postpone
  inserts but the Cache has the functionality now.
- caosdbignore; You can add one or more `.caosdbignore` files to the directory
  structure that you want to make available in CaosDB and the run loadFiles.
  The syntax is that of `.gitignore` files. For more information see `loadFiles`
  section of the Crawler in the documentation.

## [0.5.0] - 2022-09-05 ##
(Florian Spreckelsen)

### Added ###

- You can now use `python -m caosadvancedtools.models.parser model_file` to
  parse and potentially synchronize data models.

### Deprecated ###

- [#36](https://gitlab.com/caosdb/caosdb-advanced-user-tools/-/issues/36)
  `parent` keyword in yaml datamodel definition (replaced by
  `inherit_from_{obligatory|recommended|suggested}` keywords).

## [0.4.1] - 2022-05-03 ##
(Henrik tom Wörden)

### Changed ###

- `JsonSchemaParser` now identifies `name` properties in the schema with the
  CaosDB name property.

### Fixed ###

- [#40](https://gitlab.com/caosdb/caosdb-advanced-user-tools/-/issues/40)
  `assure_object_is_in_list` now handles adding objects to an initially empty list correctly.

## [0.4.0] - 2022-04-05  ##

### Added ###

- CFood that creates a Record for each line in a csv file
- `generic_analysis.py` allows to easily call scripts to perform analyses in
  server side scripting [EXPERIMENTAL]
- **EXPERIMENTAL:** Models parser can import from Json Schema files now:
  `models.parser.parse_model_from_json_schema(...)`. See the documentation of
  `models.parser.JsonSchemaParser` for the limitations of the current
  implementation.
- New keyword "role" in yaml data model that allows creation of Records and Files.
- It is now possible to set values of properties and default values of properties
  directly in the yaml model.

### Changed ###

- `TableConverter` now converts int to float and vice versa to match the desired dtype.

### Deprecated ###

### Removed ###

### Fixed ###

- CaosDB internal properties `name`, `unit` and `description` can now be used via the `extern`
  keyword in YAML datamodel specifications.

### Security ###

## [0.3.1] - 2021-12-06  ##

### Added ###
- `check_reference_field` function to check whether entities with provided ids
  exits (for example when importing data from a table)
- added the `datatypes` argument to `TableImporter` for columns that do not
  need a special conversion function

## [0.3.0] - 2021-11-02 ##

### Added ###

- Error handling for missing files when using the crawler
- included the scifolder module
- included the caosmodels module
- `send_mail` function in `caosadvancedtools.serverside.helper` module
- New class to collect possible problems with the data model
- New class for checking and importing tables
- Function to get a file path to a shared resource directory
- Function to setup logging appropriate for server side scripts with webui
  output
- New class for collecting information for exporting tables, e.g., to
  metadata repositories
- new name parsing
- new test for software folder structure
- new assure_name_is function
- two utility functions when working with files: NameCollector and
  get_file_via_download
- Automated documentation builds: `make doc`
- Crawler documentation
- Proof-of-concept integration with Bloxberg.
- Introduce a cfood that can create a Record structure based on the contents of a hdf5 file
  h5py is now an optional dependency
- table importer implementations for csv and tsv
- string-in-list check for table imports
- AbtractCFood has new property, `update_flags`.

### Changed ###

- identifiables of single CFoods are now treated one after the other. This
  allows them to have dependencies among each other if they are ordered
  correctly
- identifiables must have at least one property or a name
- `caosadvancedtools.serverside.helper.init_data_model` also checks the role
  and data type of entities.
- The `caosadvancedtools.table_importer.date_converter` now actually returns
  `datetime.date` instance. A new
  `caosadvancedtools.table_importer.datetime_converter` replaces the old
  `date_converter` and returns a `datetime.datetime` instance.
- The suppression module is now a logging filter.
- The WebUIHandler is now a python logging formatter.
- instead of `get_entity`, type-specific functions are used in
  `cfood.py` when the type of the entity in question is known.
- Logger is used instead of `print` for errors in `crawler.py`.
- complies with new exception handling, i.e., TransactionErros with
  children being raised in all cases of EntityErrors (see
  [#32](https://gitlab.com/caosdb/caosdb-pylib/-/issues/32) in
  caosdb-pylib)
- `caosadvancedtools.cfood.assure_object_is_in_list` conducts in-place
  updates if no `to_be_updated` object is supplied.

### Fixed ###
- An exception in collect_information does no longer lead to a break down.
- Removed dependency on discontiued xlrd version
- Fixed an issue where `caosadvancedtools.cache.UpdateCache` would
  cause an `sqlite3.IntegrityError` if more than one change was cached
  for the same entity.
- #40 Insertion of identifiables with missing obligatory properties
- Before, a Property with the datatype "LIST(TEXT)" would lead to the creation
  of a RecordType. This is fixed now.
- #52 `XLSimporter.read_xls` throwed a wrong error when reading from a file with a wrong ending.
  Now, a `DataInconsistencyError` is raised instead of a ValueError.
- List properties are no longer updated unnecessarily by the crawler.

## [0.2.0] - 2020-05-28 ##

### Added
- everything