# Changelog
All notable changes to this project will be documented in this file.

The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).

## [Unreleased] ##

### Added ###

* Official support for Python 3.13
* New setup extras `test` and `doc` which install the dependencies for testing and documentation.

### Changed ###

- Using the official name "LinkAhead" wherever possible without large effort. This includes the
  following exposed names / features:
  - `models.data_model.LINKAHEAD_INTERNAL_PROPERTIES`
  - `export_related.export` exports to `linkahead_data.xml` now.

### Deprecated ###

### Removed ###

- Bloxberg code snippets. These were just a proof of concept, untested and never used in production.

### Fixed ###

### Security ###

### Documentation ###

* Added documentation of `caosadvancedtools.loadFiles` module.

## [0.12.0] - 2024-07-31 ##

### Added ###

- Possibility to configure custom CSS to be used in static HTML sites created
  by the crawler

### Changed ###

- The static HTML pages that are generated by the crawler now require the
  webinterface to be built with `JS_DIST_BUNDLE=TRUE`, i.e. all js source files
  bundled into one `webcaosdb.dist.js`. This is the default and highly
  recommended for production systems in any case but it will break the display
  of these static files in certain corner cases where you may want to use
  `JS_DIST_BUNDLE=FALSE` for development or debugging reasons.

### Fixed ###

- Rendering of entities in static HTML sites created by the crawler.
- [#63](https://gitlab.com/linkahead/linkahead-advanced-user-tools/-/issues/63)
  The `TableImporter` now gives a detailed output in what row/column
  which TypeError occurred.

## [0.11.0] - 2024-07-09 ##

### Added ###

- XLSX handling: conversion from XLSX to Json
- `linkahead-loadfiles` now also is a standalone script in the installation

### Changed ###

- The `TableImporter` and its subclasses now change all integer datatypes to the
  nullable `pandas.Int64Datatype` so that integer columns with empty fields can be
  treated properly. In case you don't want the datatypes to be changed
  automatically, initialize the `TableImporter` with
  `convert_int_to_nullable_int=False`.

### Fixed ###

- Blacklisted buggy openpyxl version
- [#62](https://gitlab.com/linkahead/linkahead-advanced-user-tools/-/issues/62)
  The `TableImporter` now handles empty fields in integer columns by supporting
  the corresponding [nullable integer
  types](https://pandas.pydata.org/docs/user_guide/integer_na.html) in Pandas.

### Documentation ###

- loadFiles has better `-h` documentation now
- Rudimentary documentation for `table_importer` module

## [0.10.0] - 2024-04-24 ##

### Added ###

- XLSX handling: template generator and conversion from Json to XLSX
- Json schema exporter:
  - has new parameter `use_rt_pool`
  - propagates more properties in the `make_array` function
- Support for Python 3.12 and experimental support for 3.13

### Changed ###

* `table_converter.to_table` now returns an empty DataFrame instead of raising a
  ValueError when called with an empty container.

### Removed ###

* The deprecated `parent` keyword from the YAML datamodel specification. Use
  `inherit_from_{obligatory|recommended|suggested}` instead.
* Support for Python 3.7

### Fixed ###

- Json schema exporter handles reference properties better.
- [#59](https://gitlab.com/linkahead/linkahead-advanced-user-tools/-/issues/59)
  `to_table` failed on lists as values.

### Documentation ###

* Added documentation for json-schema datamodel export

## [0.9.0] - 2023-11-27 ##

### Added ###

* Added support for passing callables as `find_func` to the `BaseTableExporter`.
* Added member `BaseTableExporter.all_keys`
* Parsing from YAML now allows to give an existing model to which the YAML data model shall be
  added.
* The `json_schema_exporter` module which introduces tools to create a json
  schema from a RecordType, e.g., for the usage in web forms.
* `DataModel.get_deep(name: str)` method which uses the DataModel as a kind of cache pool.

### Changed ###

* A bit better error handling in the yaml model parser.
* `TableImporter.check_datatypes` allows numeric values in string columns if
  `strict=False` (default).

### Fixed ###

* `TableImporter.check_missing` in case of array-valued fields in table
* YAML model parser has better description handling.
* YAML model parser shows "LinkAhead" and "the yaml file" in its comparison display
  instead of "old" and "new".

### Documentation ###

* Test coverage reports are now generated in `.tox/cov_html/` by tox.

## [0.8.0] - 2023-05-30 ##
(Florian Spreckelsen)

### Added ###

- TableImporter now accepts a `existing_columns` argument which demands that certain columns exist
- The `JsonSchemaParser` class supports `patternProperties`
- The `JsonSchemaParser` calss supports json-schema references (`$ref`)

### Changed ###

- The converters and datatype arguments of TableImporter now may have keys for nonexisting columns
- The `JsonSchemaParser` class does not require the top-level entry of a json
  schema definition to specify a RecordType.

### Fixed ###

- refactored to work with the new default key word in FIND queries: RECORD

## [0.7.0] - 2023-03-09 ##
(Florian Spreckelsen)

### Added ###

- `create_entity_link` function to create html links to entities; useful for
  logging

## [0.6.1] - 2023-01-20##

### Added ###

* Re-introduced support for Python 3.7

## [0.6.0] - 2022-10-11 ##
(Florian Spreckelsen)

### Added ###

- Unauthorized inserts can now be cached. Note that the Crawler cannot postpone
  inserts but the Cache has the functionality now.
- caosdbignore; You can add one or more `.caosdbignore` files to the directory
  structure that you want to make available in CaosDB and the run loadFiles.
  The syntax is that of `.gitignore` files. For more information see `loadFiles`
  section of the Crawler in the documentation.

## [0.5.0] - 2022-09-05 ##
(Florian Spreckelsen)

### Added ###

- You can now use `python -m caosadvancedtools.models.parser model_file` to
  parse and potentially synchronize data models.

### Deprecated ###

- [#36](https://gitlab.com/caosdb/caosdb-advanced-user-tools/-/issues/36)
  `parent` keyword in yaml datamodel definition (replaced by
  `inherit_from_{obligatory|recommended|suggested}` keywords).

## [0.4.1] - 2022-05-03 ##
(Henrik tom Wörden)

### Changed ###

- `JsonSchemaParser` now identifies `name` properties in the schema with the
  CaosDB name property.

### Fixed ###

- [#40](https://gitlab.com/caosdb/caosdb-advanced-user-tools/-/issues/40)
  `assure_object_is_in_list` now handles adding objects to an initially empty list correctly.

## [0.4.0] - 2022-04-05  ##

### Added ###

- CFood that creates a Record for each line in a csv file
- `generic_analysis.py` allows to easily call scripts to perform analyses in
  server side scripting [EXPERIMENTAL]
- **EXPERIMENTAL:** Models parser can import from Json Schema files now:
  `models.parser.parse_model_from_json_schema(...)`. See the documentation of
  `models.parser.JsonSchemaParser` for the limitations of the current
  implementation.
- New keyword "role" in yaml data model that allows creation of Records and Files.
- It is now possible to set values of properties and default values of properties
  directly in the yaml model.

### Changed ###

- `TableConverter` now converts int to float and vice versa to match the desired dtype.

### Deprecated ###

### Removed ###

### Fixed ###

- CaosDB internal properties `name`, `unit` and `description` can now be used via the `extern`
  keyword in YAML datamodel specifications.

### Security ###

## [0.3.1] - 2021-12-06  ##

### Added ###
- `check_reference_field` function to check whether entities with provided ids
  exits (for example when importing data from a table)
- added the `datatypes` argument to `TableImporter` for columns that do not
  need a special conversion function

## [0.3.0] - 2021-11-02 ##

### Added ###

- Error handling for missing files when using the crawler
- included the scifolder module
- included the caosmodels module
- `send_mail` function in `caosadvancedtools.serverside.helper` module
- New class to collect possible problems with the data model
- New class for checking and importing tables
- Function to get a file path to a shared resource directory
- Function to setup logging appropriate for server side scripts with webui
  output
- New class for collecting information for exporting tables, e.g., to
  metadata repositories
- new name parsing
- new test for software folder structure
- new assure_name_is function
- two utility functions when working with files: NameCollector and
  get_file_via_download
- Automated documentation builds: `make doc`
- Crawler documentation
- Proof-of-concept integration with Bloxberg.
- Introduce a cfood that can create a Record structure based on the contents of a hdf5 file
  h5py is now an optional dependency
- table importer implementations for csv and tsv
- string-in-list check for table imports
- AbtractCFood has new property, `update_flags`.

### Changed ###

- identifiables of single CFoods are now treated one after the other. This
  allows them to have dependencies among each other if they are ordered
  correctly
- identifiables must have at least one property or a name
- `caosadvancedtools.serverside.helper.init_data_model` also checks the role
  and data type of entities.
- The `caosadvancedtools.table_importer.date_converter` now actually returns
  `datetime.date` instance. A new
  `caosadvancedtools.table_importer.datetime_converter` replaces the old
  `date_converter` and returns a `datetime.datetime` instance.
- The suppression module is now a logging filter.
- The WebUIHandler is now a python logging formatter.
- instead of `get_entity`, type-specific functions are used in
  `cfood.py` when the type of the entity in question is known.
- Logger is used instead of `print` for errors in `crawler.py`.
- complies with new exception handling, i.e., TransactionErros with
  children being raised in all cases of EntityErrors (see
  [#32](https://gitlab.com/caosdb/caosdb-pylib/-/issues/32) in
  caosdb-pylib)
- `caosadvancedtools.cfood.assure_object_is_in_list` conducts in-place
  updates if no `to_be_updated` object is supplied.

### Fixed ###
- An exception in collect_information does no longer lead to a break down.
- Removed dependency on discontiued xlrd version
- Fixed an issue where `caosadvancedtools.cache.UpdateCache` would
  cause an `sqlite3.IntegrityError` if more than one change was cached
  for the same entity.
- #40 Insertion of identifiables with missing obligatory properties
- Before, a Property with the datatype "LIST(TEXT)" would lead to the creation
  of a RecordType. This is fixed now.
- #52 `XLSimporter.read_xls` throwed a wrong error when reading from a file with a wrong ending.
  Now, a `DataInconsistencyError` is raised instead of a ValueError.
- List properties are no longer updated unnecessarily by the crawler.

## [0.2.0] - 2020-05-28 ##

### Added
- everything