CHANGELOG.md

Changelog

All notable changes to this project will be documented in this file.

The format is based on Keep a Changelog, and this project adheres to Semantic Versioning.

[Unreleased]

Added

• Official support for Python 3.13

Changed

• Using the official name "LinkAhead" wherever possible without large effort. This includes the following exposed names / features:
  • models.data_model.LINKAHEAD_INTERNAL_PROPERTIES
  • export_related.export exports to linkahead_data.xml now.
• Renamed (and added) installation "extra" options:
  • h5 instead of h5-crawler
  • doc, test and all are new, they install the dependencies for testing, documentation and everything.

Deprecated

Removed

• Bloxberg code snippets. These were just a proof of concept, untested and never used in production.

Fixed

Security

Documentation

• Added documentation of caosadvancedtools.loadFiles module.

[0.12.0] - 2024-07-31

Added

• Possibility to configure custom CSS to be used in static HTML sites created by the crawler

Changed

• The static HTML pages that are generated by the crawler now require the webinterface to be built with JS_DIST_BUNDLE=TRUE, i.e. all js source files bundled into one webcaosdb.dist.js. This is the default and highly recommended for production systems in any case but it will break the display of these static files in certain corner cases where you may want to use JS_DIST_BUNDLE=FALSE for development or debugging reasons.

Fixed

• Rendering of entities in static HTML sites created by the crawler.
• #63 The TableImporter now gives a detailed output in what row/column which TypeError occurred.

[0.11.0] - 2024-07-09

Added

• XLSX handling: conversion from XLSX to Json
• linkahead-loadfiles now also is a standalone script in the installation

Changed

• The TableImporter and its subclasses now change all integer datatypes to the nullable pandas.Int64Datatype so that integer columns with empty fields can be treated properly. In case you don't want the datatypes to be changed automatically, initialize the TableImporter with convert_int_to_nullable_int=False.

Fixed

• Blacklisted buggy openpyxl version
• #62 The TableImporter now handles empty fields in integer columns by supporting the corresponding nullable integer types in Pandas.

Documentation

• loadFiles has better -h documentation now
• Rudimentary documentation for table_importer module

[0.10.0] - 2024-04-24

Added

• XLSX handling: template generator and conversion from Json to XLSX
• Json schema exporter:
  • has new parameter use_rt_pool
  • propagates more properties in the make_array function
• Support for Python 3.12 and experimental support for 3.13

Changed

• table_converter.to_table now returns an empty DataFrame instead of raising a ValueError when called with an empty container.

Removed

• The deprecated parent keyword from the YAML datamodel specification. Use inherit_from_{obligatory|recommended|suggested} instead.
• Support for Python 3.7

Fixed

• Json schema exporter handles reference properties better.
• #59 to_table failed on lists as values.

Documentation

• Added documentation for json-schema datamodel export

[0.9.0] - 2023-11-27

Added

• Added support for passing callables as find_func to the BaseTableExporter.
• Added member BaseTableExporter.all_keys
• Parsing from YAML now allows to give an existing model to which the YAML data model shall be added.
• The json_schema_exporter module which introduces tools to create a json schema from a RecordType, e.g., for the usage in web forms.
• DataModel.get_deep(name: str) method which uses the DataModel as a kind of cache pool.

Changed

• A bit better error handling in the yaml model parser.
• TableImporter.check_datatypes allows numeric values in string columns if strict=False (default).

Fixed

• TableImporter.check_missing in case of array-valued fields in table
• YAML model parser has better description handling.
• YAML model parser shows "LinkAhead" and "the yaml file" in its comparison display instead of "old" and "new".

Documentation

• Test coverage reports are now generated in .tox/cov_html/ by tox.

[0.8.0] - 2023-05-30

(Florian Spreckelsen)

Added

• TableImporter now accepts a existing_columns argument which demands that certain columns exist
• The JsonSchemaParser class supports patternProperties
• The JsonSchemaParser calss supports json-schema references ($ref)

Changed

• The converters and datatype arguments of TableImporter now may have keys for nonexisting columns
• The JsonSchemaParser class does not require the top-level entry of a json schema definition to specify a RecordType.

Fixed

• refactored to work with the new default key word in FIND queries: RECORD

[0.7.0] - 2023-03-09

(Florian Spreckelsen)

Added

• create_entity_link function to create html links to entities; useful for logging

[0.6.1] - 2023-01-20##

Added

• Re-introduced support for Python 3.7

[0.6.0] - 2022-10-11

(Florian Spreckelsen)

Added

• Unauthorized inserts can now be cached. Note that the Crawler cannot postpone inserts but the Cache has the functionality now.
• caosdbignore; You can add one or more .caosdbignore files to the directory structure that you want to make available in CaosDB and the run loadFiles. The syntax is that of .gitignore files. For more information see loadFiles section of the Crawler in the documentation.

[0.5.0] - 2022-09-05

(Florian Spreckelsen)

Added

• You can now use python -m caosadvancedtools.models.parser model_file to parse and potentially synchronize data models.

Deprecated

• #36 parent keyword in yaml datamodel definition (replaced by inherit_from_{obligatory|recommended|suggested} keywords).

[0.4.1] - 2022-05-03

(Henrik tom Wörden)

Changed

• JsonSchemaParser now identifies name properties in the schema with the CaosDB name property.

Fixed

• #40 assure_object_is_in_list now handles adding objects to an initially empty list correctly.

[0.4.0] - 2022-04-05

Added

• CFood that creates a Record for each line in a csv file
• generic_analysis.py allows to easily call scripts to perform analyses in server side scripting [EXPERIMENTAL]
• EXPERIMENTAL: Models parser can import from Json Schema files now: models.parser.parse_model_from_json_schema(...). See the documentation of models.parser.JsonSchemaParser for the limitations of the current implementation.
• New keyword "role" in yaml data model that allows creation of Records and Files.
• It is now possible to set values of properties and default values of properties directly in the yaml model.

Changed

• TableConverter now converts int to float and vice versa to match the desired dtype.

Deprecated

Removed

Fixed

• CaosDB internal properties name, unit and description can now be used via the extern keyword in YAML datamodel specifications.

Security

[0.3.1] - 2021-12-06

Added

• check_reference_field function to check whether entities with provided ids exits (for example when importing data from a table)
• added the datatypes argument to TableImporter for columns that do not need a special conversion function

[0.3.0] - 2021-11-02

Added

• Error handling for missing files when using the crawler
• included the scifolder module
• included the caosmodels module
• send_mail function in caosadvancedtools.serverside.helper module
• New class to collect possible problems with the data model
• New class for checking and importing tables
• Function to get a file path to a shared resource directory
• Function to setup logging appropriate for server side scripts with webui output
• New class for collecting information for exporting tables, e.g., to metadata repositories
• new name parsing
• new test for software folder structure
• new assure_name_is function
• two utility functions when working with files: NameCollector and get_file_via_download
• Automated documentation builds: make doc
• Crawler documentation
• Proof-of-concept integration with Bloxberg.
• Introduce a cfood that can create a Record structure based on the contents of a hdf5 file h5py is now an optional dependency
• table importer implementations for csv and tsv
• string-in-list check for table imports
• AbtractCFood has new property, update_flags.

Changed

• identifiables of single CFoods are now treated one after the other. This allows them to have dependencies among each other if they are ordered correctly
• identifiables must have at least one property or a name
• caosadvancedtools.serverside.helper.init_data_model also checks the role and data type of entities.
• The caosadvancedtools.table_importer.date_converter now actually returns datetime.date instance. A new caosadvancedtools.table_importer.datetime_converter replaces the old date_converter and returns a datetime.datetime instance.
• The suppression module is now a logging filter.
• The WebUIHandler is now a python logging formatter.
• instead of get_entity, type-specific functions are used in cfood.py when the type of the entity in question is known.
• Logger is used instead of print for errors in crawler.py.
• complies with new exception handling, i.e., TransactionErros with children being raised in all cases of EntityErrors (see #32 in caosdb-pylib)
• caosadvancedtools.cfood.assure_object_is_in_list conducts in-place updates if no to_be_updated object is supplied.

Fixed

• An exception in collect_information does no longer lead to a break down.
• Removed dependency on discontiued xlrd version
• Fixed an issue where caosadvancedtools.cache.UpdateCache would cause an sqlite3.IntegrityError if more than one change was cached for the same entity.
• #40 (closed) Insertion of identifiables with missing obligatory properties
• Before, a Property with the datatype "LIST(TEXT)" would lead to the creation of a RecordType. This is fixed now.
• #52 (closed) XLSimporter.read_xls throwed a wrong error when reading from a file with a wrong ending. Now, a DataInconsistencyError is raised instead of a ValueError.
• List properties are no longer updated unnecessarily by the crawler.

[0.2.0] - 2020-05-28

Added

• everything