From 6fe2ac1cea9fd7552b9ed757163567d8ed647505 Mon Sep 17 00:00:00 2001 From: Florian Spreckelsen <f.spreckelsen@indiscale.com> Date: Thu, 24 Oct 2024 11:15:27 +0200 Subject: [PATCH 01/18] REL: Begin next release cycle --- CHANGELOG.md | 70 +++++++++++++++++++++++++++++++--------------------- doc/conf.py | 4 +-- 2 files changed, 44 insertions(+), 30 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 9b8478f..b100138 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -5,6 +5,20 @@ All notable changes to this project will be documented in this file. The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). +## [Unreleased] + +### Added + +### Changed + +### Deprecated + +### Removed + +### Fixed + +### Security + ## [8.0.0] - 2024-10-24 ## ### Added ### @@ -170,29 +184,29 @@ This is a major update. Switching from integer ids for internal and external use * `get_version_history` returns two additional columns, `child_username` and `child_realm`. * Added a `versioned` flag to the following procedures: - * `applyBackReference` - * `applyIDFilter` - * `applyPOV` - * `applyRefPOV` - * `makeStmt` - * `calcComplementUnion` - * `calcDifference` - * `calcIntersection` - * `finishSubProperty` - * `getChildren` - * `initEmptyTargetSet` - * `initDisjunctionFilter` - * `initEntity` - * `initQuery` - * `createTmpTable` + * `applyBackReference` + * `applyIDFilter` + * `applyPOV` + * `applyRefPOV` + * `makeStmt` + * `calcComplementUnion` + * `calcDifference` + * `calcIntersection` + * `finishSubProperty` + * `getChildren` + * `initEmptyTargetSet` + * `initDisjunctionFilter` + * `initEntity` + * `initQuery` + * `createTmpTable` * Added a `direct` column to `archive_isa` table ### Removed ### * unused procedures: - * `initNegationFilter` - * `initConjunctionFilter` - * `finishNegationFilter` + * `initNegationFilter` + * `initConjunctionFilter` + * `finishNegationFilter` ### Fixed ### @@ -229,20 +243,20 @@ This is a major update. Switching from integer ids for internal and external use with a lot of additions to the API. E.g. * New `entity_version`. * All `*_data` tables have a new twin, the `archive_*_data` table, where all - old versions of entities are stored. The `*_data` tables only contain the - data of the latest version. + old versions of entities are stored. The `*_data` tables only contain the + data of the latest version. * Additional `archive_isa` for the history entities' parents. * Additional `_iversion` column for the `reference_data` table for storing - references to particular versions of an entity. + references to particular versions of an entity. * New `setFileProperties` and `retrieveQueryTemplateDef` procedures which reduce server code and let the - backend decide which tables to use. Also, this is necessary for the - versioning, because these procedures behave differently depending on the - ENTITY_VERSIONING feature being enabled or disabled. + backend decide which tables to use. Also, this is necessary for the + versioning, because these procedures behave differently depending on the + ENTITY_VERSIONING feature being enabled or disabled. * Several functions and procedures for the interaction with the - `entity_version` table and the `transactions` table. E.g. - `insert_single_child_version`, `delete_all_entity_versions`, - `get_iversion`, `get_primary_parent_version`, `get_version_timestamp`, - `get_head_version`, `get_head_relative`, `get_version_history`. + `entity_version` table and the `transactions` table. E.g. + `insert_single_child_version`, `delete_all_entity_versions`, + `get_iversion`, `get_primary_parent_version`, `get_version_timestamp`, + `get_head_version`, `get_head_relative`, `get_version_history`. The versions are tracked internally by the `_iversion` field which is an integer and which should not be used outside of the backend. * New makefile targets for testing with MariaDB instance from Docker image: call diff --git a/doc/conf.py b/doc/conf.py index d7968a2..b679a2b 100644 --- a/doc/conf.py +++ b/doc/conf.py @@ -26,9 +26,9 @@ copyright = '2021 - 2024, IndiScale GmbH' author = 'Daniel Hornung' # The short X.Y version -version = '8.0.0' +version = '8.0.1' # The full version, including alpha/beta/rc tags -release = '8.0.0' +release = '8.0.1-SNAPSHOT' # -- General configuration --------------------------------------------------- -- GitLab From b8f6d501f7363764b2a6778b7804a16ba1804273 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Henrik=20tom=20W=C3=B6rden?= <h.tomwoerden@indiscale.com> Date: Fri, 22 Dec 2023 22:10:47 +0100 Subject: [PATCH 02/18] FIX: typo --- README_SETUP.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README_SETUP.md b/README_SETUP.md index 4e599ac..94a2e9d 100644 --- a/README_SETUP.md +++ b/README_SETUP.md @@ -21,7 +21,7 @@ ## Setup the SQL database for CaosDB -* Run `make install`. If a there is a database with the name you have choosen +* Run `make install`. If a there is a database with the name you have chosen during the configuration, you need to reconfigure or delete the database first. * *Required database privileges:* -- GitLab From d23ffacb912fdf247dc79df92f79dae42f929479 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Henrik=20tom=20W=C3=B6rden?= <h.tomwoerden@indiscale.com> Date: Tue, 5 Nov 2024 10:43:45 +0100 Subject: [PATCH 03/18] DOC: add usage hint for dump fix --- dump_updates/README.md | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/dump_updates/README.md b/dump_updates/README.md index 704ea80..92ce4ba 100644 --- a/dump_updates/README.md +++ b/dump_updates/README.md @@ -4,6 +4,11 @@ This directory contains scripts to update database dumps to newer versions of the MariaDB server. +## Usage ## + +`2024-10-02.dump_fix_mariadb_10_6.sh < yourdump.sql > yourdump.fixed.sql` + + ## Background ## In some cases, restoring the database content from existing SQL dumps may not be possible in a -- GitLab From ba3d8e95e5807a002a9ce52cf85fda734a4dfab2 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Henrik=20tom=20W=C3=B6rden?= <h.tomwoerden@indiscale.com> Date: Tue, 5 Nov 2024 10:56:56 +0100 Subject: [PATCH 04/18] Revert "DOC: add usage hint for dump fix" This reverts commit d23ffacb912fdf247dc79df92f79dae42f929479. --- dump_updates/README.md | 5 ----- 1 file changed, 5 deletions(-) diff --git a/dump_updates/README.md b/dump_updates/README.md index 92ce4ba..704ea80 100644 --- a/dump_updates/README.md +++ b/dump_updates/README.md @@ -4,11 +4,6 @@ This directory contains scripts to update database dumps to newer versions of the MariaDB server. -## Usage ## - -`2024-10-02.dump_fix_mariadb_10_6.sh < yourdump.sql > yourdump.fixed.sql` - - ## Background ## In some cases, restoring the database content from existing SQL dumps may not be possible in a -- GitLab From 4a34ffb997aac5d54cea9aba7ef8dc1ad34288fb Mon Sep 17 00:00:00 2001 From: Daniel <d.hornung@indiscale.com> Date: Fri, 8 Nov 2024 13:55:55 +0100 Subject: [PATCH 05/18] TEST: Applying dump upgrade script twice fails. --- dump_updates/test/test_all.py | 15 +++++++++++++++ 1 file changed, 15 insertions(+) diff --git a/dump_updates/test/test_all.py b/dump_updates/test/test_all.py index 8e97d2a..acdeeec 100644 --- a/dump_updates/test/test_all.py +++ b/dump_updates/test/test_all.py @@ -19,6 +19,11 @@ # along with this program. If not, see <https://www.gnu.org/licenses/>. """Testing the dump update scripts + +As a general rule, dump updaters should be idempotent, so tests should run the update again after +the first checks are successlful. + +TODO: Reduce the boilerplate for the tests. """ import filecmp @@ -68,3 +73,13 @@ def test_2024_10_02(tmpdir): check=True ) assert filecmp.cmp(output.name, expectedfile), "Output does not match expected output." + with (NamedTemporaryFile(dir=tmpdir, suffix=".sql", delete=True) as output2, + open(output.name, mode="rb") as infile_stream + ): + run([script_fullname], + stdin=infile_stream, + stdout=output2, + check=True + ) + assert filecmp.cmp(output2.name, expectedfile), ( + "Run 2: Output does not match expected output.") -- GitLab From f3d56b51db5d0196dbdf86b9042e515816708fd6 Mon Sep 17 00:00:00 2001 From: Daniel <d.hornung@indiscale.com> Date: Fri, 8 Nov 2024 13:57:13 +0100 Subject: [PATCH 06/18] Fix: Applying dump upgrade script twice no longer fails. --- CHANGELOG.md | 2 ++ dump_updates/2024-10-02.dump_fix_mariadb_10_6.sh | 4 ++-- 2 files changed, 4 insertions(+), 2 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index b100138..5ea239a 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -17,6 +17,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### Fixed +- [SQL dump migration can now be applied twice](https://gitlab.indiscale.com/caosdb/src/caosdb-mysqlbackend/-/issues/60): Fixed an error in the regex. + ### Security ## [8.0.0] - 2024-10-24 ## diff --git a/dump_updates/2024-10-02.dump_fix_mariadb_10_6.sh b/dump_updates/2024-10-02.dump_fix_mariadb_10_6.sh index 4450b83..8b9e484 100755 --- a/dump_updates/2024-10-02.dump_fix_mariadb_10_6.sh +++ b/dump_updates/2024-10-02.dump_fix_mariadb_10_6.sh @@ -35,10 +35,10 @@ set -euo pipefail IFS=$'\n\t' script=' -s/Offset INT UNSIGNED) RETURNS varbinary(255)/HeadOffset INT UNSIGNED) RETURNS varbinary(255)/ +/^[[:blank:]]+Offset INT UNSIGNED/s/Offset INT UNSIGNED\) RETURNS varbinary\(255\)/HeadOffset INT UNSIGNED\) RETURNS varbinary(255)/ s/LIMIT 1 OFFSET Offset/LIMIT 1 OFFSET HeadOffset/ ' -sed -e "$script" +sed -E -e "$script" unset script -- GitLab From 4ea8cd918eef39c9bb4513953ba873e1f4551eb9 Mon Sep 17 00:00:00 2001 From: Daniel <d.hornung@indiscale.com> Date: Thu, 30 Jan 2025 14:01:33 +0100 Subject: [PATCH 07/18] MAINT: Renamed function "check_version", added documentation. --- utils/patch_header.sh | 65 ++++++++++++++++++++++++++++++++++--------- 1 file changed, 52 insertions(+), 13 deletions(-) diff --git a/utils/patch_header.sh b/utils/patch_header.sh index 33fbff8..c7d25ae 100644 --- a/utils/patch_header.sh +++ b/utils/patch_header.sh @@ -1,12 +1,10 @@ -# -# ** header v3.0 -# This file is a part of the CaosDB Project. +# This file is a part of the LinkAhead Project. # # Copyright (C) 2018 Research Group Biomedical Physics, # Max-Planck-Institute for Dynamics and Self-Organization Göttingen -# Copyright (C) 2019, 2020 Daniel Hornung <d.hornung@indiscale.com> +# Copyright (C) 2019, 2020, 2025 Daniel Hornung <d.hornung@indiscale.com> # Copyright (C) 2020 Henrik tom Wörden <h.tomwoerden@indiscale.com> -# Copyright (C) 2020 IndiScale <info@indiscale.com> +# Copyright (C) 2020, 2025 IndiScale <info@indiscale.com> # # This program is free software: you can redistribute it and/or modify # it under the terms of the GNU Affero General Public License as @@ -21,12 +19,20 @@ # You should have received a copy of the GNU Affero General Public License # along with this program. If not, see <https://www.gnu.org/licenses/>. # -# ** end header -# ############################################################################### # header for patch scripts # ############################################################################### +# +# This file defines the following functions (see below for more detailed descriptions): +# +# - check_version_or_exit: Check if the stored version is as expected. +# - update_version: Set the stored version. +# - dump_table: Store table content into a file. +# - redo_table: Apply the table content stored by dump_table. +# +# This file also sets the `errexit` option (`set -e`). +# set -e @@ -63,7 +69,7 @@ configuration make file. Notable examples are: - DATABASE_USER_PW EOF - ) + ) function _print_help() { echo -e "$USAGE" @@ -115,21 +121,51 @@ done if [ -n "$PATCH" ]; then echo -ne "applying patch $PATCH to $DATABASE_NAME ... " fi -# @param $1: db version string, e.g. v2.0.0 -# @return: 0 on success, 1 on failure -function check_version { + +# usage: check_version_or_exit version +# +# Check if the version string is as expected. Otherwise, `exit 0`. +# +# Arguments +# --------- +# +# version: str +# The expected database version string, e.g. 'v2.0.0' +# +# Returns +# ------- +# +# 0 on success +function check_version_or_exit { local version=$($MYSQL_CMD $(get_db_args) -B -e "Select CaosDBVersion();") - if [[ "$(echo $version | sed 's/^CaosDBVersion()\s//')" = "$1" ]]; then + local version="$(echo $version | sed 's/^CaosDBVersion()\s//')" + if [[ "$version" = "$1" ]]; then return 0 fi uptodate } -# @param $1: new version string +# Deprecated name, call "check_version_or_exit" directly instead. +function check_version { + check_version_or_exit "$1" +} + +# usage: update_version version +# +# Set the version string. +# +# Arguments +# --------- +# +# version: str +# The new database version string, e.g. 'v2.0.0' function update_version { mysql_execute "DROP FUNCTION IF EXISTS CaosDBVersion; CREATE FUNCTION CaosDBVersion() RETURNS VARCHAR(255) DETERMINISTIC RETURN '$1';" } +# usage: dump_table table +# +# Dump the table given in the argument into file "caosdb.<table>.<OLD_VERSION>.dump.sql" function dump_table { if [[ -z $MYSQLDUMP_CMD ]]; then echo "Cannot find mysqldump program!" >&2 @@ -139,6 +175,9 @@ function dump_table { > ${DATABASE_NAME}.${1}.${OLD_VERSION}.dump.sql } +# usage: redo_table table +# +# Apply the dump from `dump_table` to the database. Takes the same argument. function redo_table { $MYSQL_CMD $(get_db_args) < ${DATABASE_NAME}.${1}.${OLD_VERSION}.dump.sql } -- GitLab From 167be92ecbce8773c2a55d20379e875787056915 Mon Sep 17 00:00:00 2001 From: Daniel <d.hornung@indiscale.com> Date: Thu, 30 Jan 2025 15:09:51 +0100 Subject: [PATCH 08/18] WIP: Making database ready for MariaDB 11.2. --- CHANGELOG.md | 4 ++++ README.md | 4 ++-- README_SETUP.md | 23 ++++++++-------------- RELEASE_GUIDELINES.md | 2 +- utils/sanity_check.sql | 44 ++++++++++++++++++++++++++++++++++++++---- 5 files changed, 55 insertions(+), 22 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 5ea239a..b93f14a 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -11,10 +11,14 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### Changed +- New version requirement: MariaDB 11.2 or later. + ### Deprecated ### Removed +- MySQL support. See the deprecation in the 7.0.0 release. + ### Fixed - [SQL dump migration can now be applied twice](https://gitlab.indiscale.com/caosdb/src/caosdb-mysqlbackend/-/issues/60): Fixed an error in the regex. diff --git a/README.md b/README.md index 7094464..6b3d3a7 100644 --- a/README.md +++ b/README.md @@ -3,7 +3,7 @@ ## Welcome -This is the **LinkAhead MySQL Backend** repository and a part of the +This is the **LinkAhead MariaDB Backend** repository and a part of the LinkAhead project. ## Setup @@ -43,7 +43,7 @@ Conduct](https://gitlab.com/linkahead/linkahead/-/blob/main/CODE_OF_CONDUCT.md). * Copyright (C) 2018 Research Group Biomedical Physics, Max Planck Institute for Dynamics and Self-Organization Göttingen. -* Copyright (C) 2020-2021 Indiscale GmbH <info@indiscale.com> +* Copyright (C) 2020-2025 Indiscale GmbH <info@indiscale.com> All files in this repository are licensed under a [GNU Affero General Public License](LICENCE.md) (version 3 or later). diff --git a/README_SETUP.md b/README_SETUP.md index 94a2e9d..a92db35 100644 --- a/README_SETUP.md +++ b/README_SETUP.md @@ -1,8 +1,8 @@ -# Setup of the CaosDB SQL back end +# Setup of the LinkAhead MariaDB back end ## Dependencies -* `MariaDB Client 10.1` or later, `MySQL Client >=5.5, <=5.7.36`. +* `MariaDB Client 11.2` or later. * make ## Create the configuration @@ -11,7 +11,7 @@ parameter that you want to change, add a corresponding line in your `.config` file. You probably want to change the passwords. As the passwords are stored unencrypted in the `.config` file, make sure nobody else can read it. - * If there is no `mysql-config-editor` (`MySQL 5.5`. and `MariaDB`) then the + * If there is no `mysql-config-editor` program, then the `MYSQL_USER_PASSWORD` must be provided, that is the password of the `MYSQL_USER`. * If you are using MariaDB and the `root` database user uses pam @@ -102,18 +102,6 @@ caosdb`. ### Troubleshooting -#### MySQL has failing tests - -Our test suite is developed with MariaDB. That is why some tests, those which -check constraints based on auto-generated names, fail because MySQL generates -other names. - -Also some test fail with the message "Expected: NULL / Was: NULL" which is -obviously what was expected. - -Please look carefully through the tests. If something more serious than that -comes up, please report a bug. - #### Failure to restore a database dump created with an older MariaDB version #### Have a look into the `dump_updates/README.md`. In cases of version @@ -121,6 +109,11 @@ incompatibilities, the necessary steps to migrate the dump are probably described there and scripts for the migrations are provided in the same directory. +#### MySQL has failing tests + +*Note: Since we switched from MySQL to MariaDB, the two DBMS projects have diverged quite far. As a +result, we no longer support MySQL. + ## Documentation # Build documentation in `build/` with `make doc`. diff --git a/RELEASE_GUIDELINES.md b/RELEASE_GUIDELINES.md index c1d7335..72f4534 100644 --- a/RELEASE_GUIDELINES.md +++ b/RELEASE_GUIDELINES.md @@ -1,4 +1,4 @@ -# Release Guidelines for the CaosDB MySQL Backend +# Release Guidelines for the LinkAhead MariaDB Backend This document specifies release guidelines in addition to the general release guidelines of the CaosDB Project diff --git a/utils/sanity_check.sql b/utils/sanity_check.sql index c29e7c8..5e3d580 100644 --- a/utils/sanity_check.sql +++ b/utils/sanity_check.sql @@ -55,9 +55,29 @@ sanityCheckBody: BEGIN ("user_roles") ; -SELECT COUNT(WRONG) INTO @count_wrong FROM ( SELECT l.table_name AS MATCHED, r.table_name AS WRONG FROM expected_tables AS l RIGHT OUTER JOIN information_schema.tables AS r ON (r.table_name COLLATE utf8_unicode_ci = l.table_name) WHERE r.table_schema = database()) AS temp WHERE temp.MATCHED IS NULL; +SELECT COUNT(WRONG) INTO @count_wrong FROM ( + SELECT l.table_name AS MATCHED, + r.table_name AS WRONG + FROM + expected_tables AS l + RIGHT OUTER JOIN + information_schema.tables AS r + ON (r.table_name COLLATE utf8_unicode_ci = l.table_name) + WHERE r.table_schema = database() AND r.table_type != 'TEMPORARY' + ) + AS temp WHERE temp.MATCHED IS NULL; -SELECT COUNT(MISSING) INTO @count_missing FROM ( SELECT l.table_name AS MISSING, r.table_name AS MATCHED FROM expected_tables AS l LEFT OUTER JOIN information_schema.tables AS r ON (r.table_name COLLATE utf8_unicode_ci = l.table_name) WHERE r.table_schema = database() OR r.table_schema IS NULL) AS temp WHERE temp.MATCHED IS NULL; +SELECT COUNT(MISSING) INTO @count_missing FROM ( + SELECT l.table_name AS MISSING, + r.table_name AS MATCHED + FROM + expected_tables AS l + LEFT OUTER JOIN + information_schema.tables AS r + ON (r.table_name COLLATE utf8_unicode_ci = l.table_name) + WHERE r.table_schema = database() OR r.table_schema IS NULL + ) + AS temp WHERE temp.MATCHED IS NULL; IF @count_missing = 0 AND @count_wrong = 0 THEN LEAVE sanityCheckBody; @@ -66,12 +86,28 @@ END IF; SELECT "--------------"; SELECT @count_missing AS "Number of missing tables"; -SELECT MISSING AS "Missing tables" FROM ( SELECT l.table_name AS MISSING, r.table_name AS MATCHED FROM expected_tables AS l LEFT OUTER JOIN information_schema.tables AS r ON (r.table_name COLLATE utf8_unicode_ci = l.table_name) WHERE r.table_schema = database() OR r.table_schema IS NULL) AS temp WHERE temp.MATCHED IS NULL; +SELECT MISSING AS "Missing tables" FROM ( + SELECT l.table_name AS MISSING, r.table_name AS MATCHED FROM + expected_tables AS l + LEFT OUTER JOIN + information_schema.tables AS r + ON (r.table_name COLLATE utf8_unicode_ci = l.table_name) + WHERE r.table_schema = database() OR r.table_schema IS NULL + ) + AS temp WHERE temp.MATCHED IS NULL; SELECT "--------------"; SELECT @count_wrong AS "Number of tables which should not exist"; -SELECT WRONG AS "Tables which should not exist" FROM ( SELECT l.table_name AS MATCHED, r.table_name AS WRONG FROM expected_tables AS l RIGHT OUTER JOIN information_schema.tables AS r ON (r.table_name COLLATE utf8_unicode_ci = l.table_name) WHERE r.table_schema = database()) AS temp WHERE temp.MATCHED IS NULL; +SELECT WRONG AS "Tables which should not exist" FROM ( + SELECT expec.table_name AS MATCHED, info.table_name AS WRONG FROM + expected_tables AS expec + RIGHT OUTER JOIN + information_schema.tables AS info + ON (info.table_name COLLATE utf8_unicode_ci = expec.table_name) + WHERE info.table_schema = database() and info.table_type != 'TEMPORARY' + ) + AS temp WHERE temp.MATCHED IS NULL; SELECT "--------------"; SELECT "ERROR" from sanity_check_failed; -- GitLab From 43456eec73a87b751a39103999ea81205786ae29 Mon Sep 17 00:00:00 2001 From: Daniel <d.hornung@indiscale.com> Date: Thu, 30 Jan 2025 16:30:54 +0100 Subject: [PATCH 09/18] FIX: Set character set and collation so accents are not ignored. --- CHANGELOG.md | 2 ++ README_SETUP.md | 2 +- RELEASE_GUIDELINES.md | 2 +- patches/patch20250130-8.0.1/patch.sh | 47 ++++++++++++++++++++++++++++ utils/helpers.sh | 2 +- 5 files changed, 52 insertions(+), 3 deletions(-) create mode 100755 patches/patch20250130-8.0.1/patch.sh diff --git a/CHANGELOG.md b/CHANGELOG.md index b93f14a..366a97c 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -22,6 +22,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### Fixed - [SQL dump migration can now be applied twice](https://gitlab.indiscale.com/caosdb/src/caosdb-mysqlbackend/-/issues/60): Fixed an error in the regex. +- [linahead-server#264 Queries ignore accents and umlauts](https://gitlab.com/linkahead/linkahead-server/-/issues/264): Changed to more appropriate + collation and character sets. ### Security diff --git a/README_SETUP.md b/README_SETUP.md index a92db35..ec01218 100644 --- a/README_SETUP.md +++ b/README_SETUP.md @@ -1,4 +1,4 @@ -# Setup of the LinkAhead MariaDB back end +# Setup of the LinkAhead MariaDB backend ## Dependencies diff --git a/RELEASE_GUIDELINES.md b/RELEASE_GUIDELINES.md index 72f4534..41c4713 100644 --- a/RELEASE_GUIDELINES.md +++ b/RELEASE_GUIDELINES.md @@ -1,4 +1,4 @@ -# Release Guidelines for the LinkAhead MariaDB Backend +# Release Guidelines for the LinkAhead MariaDB backend This document specifies release guidelines in addition to the general release guidelines of the CaosDB Project diff --git a/patches/patch20250130-8.0.1/patch.sh b/patches/patch20250130-8.0.1/patch.sh new file mode 100755 index 0000000..d274e72 --- /dev/null +++ b/patches/patch20250130-8.0.1/patch.sh @@ -0,0 +1,47 @@ +#!/bin/bash +# +# This file is a part of the Linkahead Project. +# +# Copyright (C) 2025 IndiScale GmbH <info@indiscale.com> +# Copyright (C) 2025 Daniel Hornung <d.hornung@indiscale.com> +# +# This program is free software: you can redistribute it and/or modify +# it under the terms of the GNU Affero General Public License as +# published by the Free Software Foundation, either version 3 of the +# License, or (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU Affero General Public License for more details. +# +# You should have received a copy of the GNU Affero General Public License +# along with this program. If not, see <https://www.gnu.org/licenses/>. +# + +# Update mysql schema to version v8.1.0 +# +# Set the collation so that accents on characters become relevant. + +NEW_VERSION="v8.0.0" +OLD_VERSION="v8.0.1" + +if [ -z "$UTILSPATH" ]; then + UTILSPATH="../utils" +fi + +. $UTILSPATH/patch_header.sh $* + +check_version $OLD_VERSION + +# Update charsets and collations. + +# Get all tables, drop first line, take first column +tables=$(mysql_execute "SHOW TABLE status where Collation='utf8mb3_unicode_ci';" -B | tail +2 | awk '{print $1}') +for table in tables; do + mysql_execute "ALTER TABLE $table CONVERT TO CHARACTER SET utf8mb4 COLLATE uca1400_as_ci;" +done + +update_version $NEW_VERSION + +success diff --git a/utils/helpers.sh b/utils/helpers.sh index f335dfa..ff4b189 100644 --- a/utils/helpers.sh +++ b/utils/helpers.sh @@ -34,7 +34,7 @@ function mysql_execute { #TODO is it meaningful that here always the database user is used??? set -e - $MYSQL_CMD $(get_db_args) -e "$1" + $MYSQL_CMD $(get_db_args) -e "$@" ret=${PIPESTATUS[0]} if [ "$ret" -ne 0 ]; then failure "MYSQL ERROR" -- GitLab From cd2bc1675538c7d8727710d9105daca4de859034 Mon Sep 17 00:00:00 2001 From: Daniel <d.hornung@indiscale.com> Date: Mon, 3 Feb 2025 10:30:54 +0100 Subject: [PATCH 10/18] FIX: versions in patch --- CHANGELOG.md | 2 +- patches/patch20250130-8.0.1/patch.sh | 31 ++++++++++++++++++++++++++-- 2 files changed, 30 insertions(+), 3 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 366a97c..32b3348 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -22,7 +22,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### Fixed - [SQL dump migration can now be applied twice](https://gitlab.indiscale.com/caosdb/src/caosdb-mysqlbackend/-/issues/60): Fixed an error in the regex. -- [linahead-server#264 Queries ignore accents and umlauts](https://gitlab.com/linkahead/linkahead-server/-/issues/264): Changed to more appropriate +- [linkahead-server#264 Queries ignore accents and umlauts](https://gitlab.com/linkahead/linkahead-server/-/issues/264): Changed to more appropriate collation and character sets. ### Security diff --git a/patches/patch20250130-8.0.1/patch.sh b/patches/patch20250130-8.0.1/patch.sh index d274e72..ccaaee8 100755 --- a/patches/patch20250130-8.0.1/patch.sh +++ b/patches/patch20250130-8.0.1/patch.sh @@ -23,8 +23,8 @@ # # Set the collation so that accents on characters become relevant. -NEW_VERSION="v8.0.0" -OLD_VERSION="v8.0.1" +OLD_VERSION="v8.0.0" +NEW_VERSION="v8.0.1" if [ -z "$UTILSPATH" ]; then UTILSPATH="../utils" @@ -41,6 +41,33 @@ tables=$(mysql_execute "SHOW TABLE status where Collation='utf8mb3_unicode_ci';" for table in tables; do mysql_execute "ALTER TABLE $table CONVERT TO CHARACTER SET utf8mb4 COLLATE uca1400_as_ci;" done +# mysql_execute "UPDATE data_type SET datatype=17 WHERE datatype=3" + + +# cd git/caosdb-mysqlbackend/ +# UTILSPATH=utils +# . utils/patch_header.sh + + +# show table status where Collation='utf8mb3_unicode_ci'; + +# name_data +# text_data + +# show table status like 'name_data'; +# select * from name_data where value like 'Yield%'; +# update name_data SET value='Yield-stréss' where entity_id=354; +# update name_data SET value='Yield-strés😋s' where entity_id=354; +# select * from name_data where value='Yield-stress'; + +# # Unicode 14.0.0 +# ALTER TABLE name_data CONVERT TO CHARACTER SET utf8mb4 COLLATE uca1400_as_ci; + +# FULL_COLLATION_NAME: utf8mb4_uca1400_as_ci; +# COLLATION_NAME: uca1400_as_ci; + +# mariadb -h sqldb -u caosdb -prandom1234 --default-character-set=utf8mb4 caosdb + update_version $NEW_VERSION -- GitLab From f7a81cf493024a58c0b20fe56fc3261df1350d09 Mon Sep 17 00:00:00 2001 From: Daniel <d.hornung@indiscale.com> Date: Mon, 3 Feb 2025 11:30:31 +0100 Subject: [PATCH 11/18] FIX: patch shell script --- patches/patch20250130-8.0.1/patch.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/patches/patch20250130-8.0.1/patch.sh b/patches/patch20250130-8.0.1/patch.sh index ccaaee8..bad63f1 100755 --- a/patches/patch20250130-8.0.1/patch.sh +++ b/patches/patch20250130-8.0.1/patch.sh @@ -38,7 +38,7 @@ check_version $OLD_VERSION # Get all tables, drop first line, take first column tables=$(mysql_execute "SHOW TABLE status where Collation='utf8mb3_unicode_ci';" -B | tail +2 | awk '{print $1}') -for table in tables; do +for table in $tables; do mysql_execute "ALTER TABLE $table CONVERT TO CHARACTER SET utf8mb4 COLLATE uca1400_as_ci;" done # mysql_execute "UPDATE data_type SET datatype=17 WHERE datatype=3" -- GitLab From ed7a4977a2fe1e2d7c34a1a24874d0c2283871e1 Mon Sep 17 00:00:00 2001 From: Daniel <d.hornung@indiscale.com> Date: Mon, 3 Feb 2025 11:37:55 +0100 Subject: [PATCH 12/18] WIP: Patch --- patches/patch20250130-8.0.1/patch.sh | 11 +++++++++++ 1 file changed, 11 insertions(+) diff --git a/patches/patch20250130-8.0.1/patch.sh b/patches/patch20250130-8.0.1/patch.sh index bad63f1..863d112 100755 --- a/patches/patch20250130-8.0.1/patch.sh +++ b/patches/patch20250130-8.0.1/patch.sh @@ -36,11 +36,22 @@ check_version $OLD_VERSION # Update charsets and collations. +# Remove some constraints first. +mysql_execute 'ALTER TABLE user_info DROP FOREIGN KEY `subjects_ibfk_2`;' + # Get all tables, drop first line, take first column tables=$(mysql_execute "SHOW TABLE status where Collation='utf8mb3_unicode_ci';" -B | tail +2 | awk '{print $1}') for table in $tables; do mysql_execute "ALTER TABLE $table CONVERT TO CHARACTER SET utf8mb4 COLLATE uca1400_as_ci;" done + +# Add constraints again. +mysql_execute 'ALTER TABLE user_info MODIFY COLUMN `entity` VARCHAR(255) COLLATE utf8mb4_bin DEFAULT NULL;' +mysql_execute 'ALTER TABLE user_info ADD CONSTRAINT `subjects_ibfk_2` FOREIGN KEY (`entity`) REFERENCES `entity_ids` (`id`);' + + + + # mysql_execute "UPDATE data_type SET datatype=17 WHERE datatype=3" -- GitLab From eb5e16959b1a4972e223ef7f26c7cf4acab213c4 Mon Sep 17 00:00:00 2001 From: Daniel <d.hornung@indiscale.com> Date: Tue, 4 Feb 2025 17:35:11 +0100 Subject: [PATCH 13/18] WIP: Patch --- .gitlab-ci.yml | 36 +++--- Makefile | 2 +- README_SETUP.md | 14 +++ patches/patch20170316-2.0.27/patch.sh | 7 +- patches/patch20250130-8.0.1/patch.sh | 24 +++- tests/test_autotap.sql | 156 +++++++++++++------------- 6 files changed, 136 insertions(+), 103 deletions(-) diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml index 310aad6..561f671 100644 --- a/.gitlab-ci.yml +++ b/.gitlab-ci.yml @@ -70,24 +70,24 @@ test-dump-update: script: - make test-dump-update -# Run the unit tests with MySQL 5 -unittests-mysql-5: - tags: [ docker ] - stage: test - # Should not stop the pipeline from continuing. - allow_failure: true - services: - - name: mysql:5.7.36 - command: ["--default-authentication-plugin=mysql_native_password"] - - script: - # remove some lines from autotap because the checks of column default - # values don't work with mysql-5 - - sed -i "/col_default_is.*NULL/d" tests/test_autotap.sql - - sed -i "/col_default_is.*INACTIVE/d" tests/test_autotap.sql - - sed -i "/col_default_is.*SHA/d" tests/test_autotap.sql - - sed -i "s/utf8mb3/utf8/" tests/test_autotap.sql - - make pipeline-test SQL_HOST=mysql +# # Run the unit tests with MySQL 5 +# unittests-mysql-5: +# tags: [ docker ] +# stage: test +# # Should not stop the pipeline from continuing. +# allow_failure: true +# services: +# - name: mysql:5.7.36 +# command: ["--default-authentication-plugin=mysql_native_password"] +# +# script: +# # remove some lines from autotap because the checks of column default +# # values don't work with mysql-5 +# - sed -i "/col_default_is.*NULL/d" tests/test_autotap.sql +# - sed -i "/col_default_is.*INACTIVE/d" tests/test_autotap.sql +# - sed -i "/col_default_is.*SHA/d" tests/test_autotap.sql +# - sed -i "s/utf8mb3/utf8/" tests/test_autotap.sql +# - make pipeline-test SQL_HOST=mysql ######## Deploy ######## diff --git a/Makefile b/Makefile index 0204e01..4bfdc03 100644 --- a/Makefile +++ b/Makefile @@ -64,7 +64,7 @@ test-dump-update: test-docker: @docker kill caosdb-mysqlserver-test || true @docker container rm -v caosdb-mysqlserver-test || true - @docker run --name caosdb-mysqlserver-test -p "3306:3306" \ + @docker run --name caosdb-mysqlserver-test -p "3307:3306" \ -e MYSQL_ROOT_PASSWORD="pass-for-test" -d mariadb:10.11 @sleep 10 MAINPATH=$(realpath tests/docker_env) utils/make_db test --fresh diff --git a/README_SETUP.md b/README_SETUP.md index ec01218..8034022 100644 --- a/README_SETUP.md +++ b/README_SETUP.md @@ -99,6 +99,20 @@ After the first start, you need to install the database: `make install`. Then, you can connect to it with `mariadb --protocol=TCP -u caosdb -prandom1234 caosdb`. +### Developing tests ### + +To use the mytap framework: + +```console +$ cd libs; unzip mytap-1.0.zip; cd mytap-1.0 +# Use the necessary mariadb options for connecting to the server in the following commands +$ mariadb ... < mytap.sql # Set up the "tap" database and functions. +$ mariadb ... < scripts/autotap.sql # Insert the "autotap" function. +# Create autotap file (for comparison with current state) +$ mariadb ... --raw -B --skip-column-names -e "call tap.autotap('_caosdb_schema_unit_tests')" > test_autotap.current.sql +# Run the autotap tests (may fail unfortunately with current MariaDB): +$ mariadb ... < test_autotap.current.sql +``` ### Troubleshooting diff --git a/patches/patch20170316-2.0.27/patch.sh b/patches/patch20170316-2.0.27/patch.sh index ecb892e..7a2d0a6 100755 --- a/patches/patch20170316-2.0.27/patch.sh +++ b/patches/patch20170316-2.0.27/patch.sh @@ -35,7 +35,12 @@ fi check_version $OLD_VERSION -mysql_execute 'DELETE FROM `entities` WHERE id=6; ALTER TABLE `entities` MODIFY COLUMN `role` ENUM("RECORDTYPE","RECORD","FILE","DOMAIN","PROPERTY","DATATYPE","ROLE","QUERYTEMPLATE") NOT NULL; INSERT INTO entities (id, name, description, role, acl) VALUES (8,"QUERYTEMPLATE","The QueryTemplate role.","ROLE",0); UPDATE `entities` SET role="ROLE" WHERE id<100 and name=role; CREATE TABLE `query_template_def` (id INT UNSIGNED PRIMARY KEY, definition MEDIUMTEXT NOT NULL, CONSTRAINT `query_template_def_ibfk_1` FOREIGN KEY (`id`) REFERENCES `entities` (`id`));' +mysql_execute ' +DELETE FROM `entities` WHERE id=6; +ALTER TABLE `entities` MODIFY COLUMN `role` ENUM("RECORDTYPE","RECORD","FILE","DOMAIN","PROPERTY","DATATYPE","ROLE","QUERYTEMPLATE") NOT NULL; +INSERT INTO entities (id, name, description, role, acl) VALUES (8,"QUERYTEMPLATE","The QueryTemplate role.","ROLE",0); +UPDATE `entities` SET role="ROLE" WHERE id<100 and name=role; +CREATE TABLE `query_template_def` (id INT UNSIGNED PRIMARY KEY, definition MEDIUMTEXT NOT NULL, CONSTRAINT `query_template_def_ibfk_1` FOREIGN KEY (`id`) REFERENCES `entities` (`id`));' update_version $NEW_VERSION diff --git a/patches/patch20250130-8.0.1/patch.sh b/patches/patch20250130-8.0.1/patch.sh index 863d112..aca1f7a 100755 --- a/patches/patch20250130-8.0.1/patch.sh +++ b/patches/patch20250130-8.0.1/patch.sh @@ -36,6 +36,17 @@ check_version $OLD_VERSION # Update charsets and collations. +# Change database default. +mysql_execute "ALTER DATABASE \`$DATABASE_NAME\` COLLATE = 'uca1400_as_ci';" + +# Some columns should be changed manually to prevent text length changes. +mysql_execute "ALTER TABLE desc_overrides MODIFY description TEXT CHARACTER SET utf8mb4;" +mysql_execute "ALTER TABLE entities MODIFY description TEXT CHARACTER SET utf8mb4;" +mysql_execute "ALTER TABLE permissions MODIFY permissions MEDIUMTEXT CHARACTER SET utf8mb4 NOT NULL;" +mysql_execute "ALTER TABLE query_template_def MODIFY definition MEDIUMTEXT CHARACTER SET utf8mb4 NOT NULL;" +mysql_execute "ALTER TABLE roles MODIFY description MEDIUMTEXT CHARACTER SET utf8mb4;" +mysql_execute "ALTER TABLE text_data MODIFY value TEXT CHARACTER SET utf8mb4 NOT NULL;" + # Remove some constraints first. mysql_execute 'ALTER TABLE user_info DROP FOREIGN KEY `subjects_ibfk_2`;' @@ -49,11 +60,18 @@ done mysql_execute 'ALTER TABLE user_info MODIFY COLUMN `entity` VARCHAR(255) COLLATE utf8mb4_bin DEFAULT NULL;' mysql_execute 'ALTER TABLE user_info ADD CONSTRAINT `subjects_ibfk_2` FOREIGN KEY (`entity`) REFERENCES `entity_ids` (`id`);' +# Set custom collations for columns +mysql_execute "ALTER TABLE transaction_log MODIFY entity_id VARCHAR(255) CHARACTER SET utf8mb4 COLLATE utf8mb4_bin NOT NULL;" + +### Finished ### + +update_version $NEW_VERSION +success -# mysql_execute "UPDATE data_type SET datatype=17 WHERE datatype=3" +##### Useful commands ##### # cd git/caosdb-mysqlbackend/ # UTILSPATH=utils @@ -79,7 +97,3 @@ mysql_execute 'ALTER TABLE user_info ADD CONSTRAINT `subjects_ibfk_2` FOREIGN KE # mariadb -h sqldb -u caosdb -prandom1234 --default-character-set=utf8mb4 caosdb - -update_version $NEW_VERSION - -success diff --git a/tests/test_autotap.sql b/tests/test_autotap.sql index 168758c..be7c63f 100644 --- a/tests/test_autotap.sql +++ b/tests/test_autotap.sql @@ -38,7 +38,7 @@ SELECT tap.has_schema('_caosdb_schema_unit_tests',''); -- *************************************************************** SELECT tap.has_table('_caosdb_schema_unit_tests','archive_isa',''); -SELECT tap.table_collation_is('_caosdb_schema_unit_tests','archive_isa','utf8mb3_unicode_ci',''); +-- SELECT tap.table_collation_is('_caosdb_schema_unit_tests','archive_isa','utf8mb4_uca1400_as_ci',''); SELECT tap.table_engine_is('_caosdb_schema_unit_tests','archive_isa','InnoDB',''); -- COLUMNS @@ -116,7 +116,7 @@ SELECT tap.fk_on_update('_caosdb_schema_unit_tests','archive_isa','archive_isa_i -- *************************************************************** SELECT tap.has_table('_caosdb_schema_unit_tests','collection_type',''); -SELECT tap.table_collation_is('_caosdb_schema_unit_tests','collection_type','utf8mb3_unicode_ci',''); +-- SELECT tap.table_collation_is('_caosdb_schema_unit_tests','collection_type','utf8mb4_uca1400_as_ci',''); SELECT tap.table_engine_is('_caosdb_schema_unit_tests','collection_type','InnoDB',''); -- COLUMNS @@ -155,8 +155,8 @@ SELECT tap.has_column('_caosdb_schema_unit_tests','collection_type','collection' SELECT tap.col_column_type_is('_caosdb_schema_unit_tests','collection_type','collection','varchar(255)',''); SELECT tap.col_extra_is('_caosdb_schema_unit_tests','collection_type','collection','',''); SELECT tap.col_default_is('_caosdb_schema_unit_tests','collection_type','collection',NULL,''); -SELECT tap.col_charset_is('_caosdb_schema_unit_tests','collection_type','collection','utf8mb3',''); -SELECT tap.col_collation_is('_caosdb_schema_unit_tests','collection_type','collection','utf8mb3_unicode_ci',''); +SELECT tap.col_charset_is('_caosdb_schema_unit_tests','collection_type','collection','utf8mb4',''); +SELECT tap.col_collation_is('_caosdb_schema_unit_tests','collection_type','collection','utf8mb4_uca1400_as_ci',''); -- INDEXES SELECT tap.indexes_are('_caosdb_schema_unit_tests','collection_type','`domain_id`,`entity_id`,`property_id`',''); @@ -208,7 +208,7 @@ SELECT tap.fk_on_update('_caosdb_schema_unit_tests','collection_type','collectio -- *************************************************************** SELECT tap.has_table('_caosdb_schema_unit_tests','data_type',''); -SELECT tap.table_collation_is('_caosdb_schema_unit_tests','data_type','utf8mb3_unicode_ci',''); +-- SELECT tap.table_collation_is('_caosdb_schema_unit_tests','data_type','utf8mb4_uca1400_as_ci',''); SELECT tap.table_engine_is('_caosdb_schema_unit_tests','data_type','InnoDB',''); -- COLUMNS @@ -300,7 +300,7 @@ SELECT tap.constraint_type_is('_caosdb_schema_unit_tests','data_type','datatype_ -- *************************************************************** SELECT tap.has_table('_caosdb_schema_unit_tests','date_data',''); -SELECT tap.table_collation_is('_caosdb_schema_unit_tests','date_data','utf8mb3_unicode_ci',''); +-- SELECT tap.table_collation_is('_caosdb_schema_unit_tests','date_data','utf8mb4_uca1400_as_ci',''); SELECT tap.table_engine_is('_caosdb_schema_unit_tests','date_data','InnoDB',''); -- COLUMNS @@ -348,8 +348,8 @@ SELECT tap.has_column('_caosdb_schema_unit_tests','date_data','status',''); SELECT tap.col_column_type_is('_caosdb_schema_unit_tests','date_data','status','enum(\'OBLIGATORY\',\'RECOMMENDED\',\'SUGGESTED\',\'FIX\')',''); SELECT tap.col_extra_is('_caosdb_schema_unit_tests','date_data','status','',''); SELECT tap.col_default_is('_caosdb_schema_unit_tests','date_data','status','NULL',''); -SELECT tap.col_charset_is('_caosdb_schema_unit_tests','date_data','status','utf8mb3',''); -SELECT tap.col_collation_is('_caosdb_schema_unit_tests','date_data','status','utf8mb3_unicode_ci',''); +SELECT tap.col_charset_is('_caosdb_schema_unit_tests','date_data','status','utf8mb4',''); +SELECT tap.col_collation_is('_caosdb_schema_unit_tests','date_data','status','utf8mb4_uca1400_as_ci',''); -- COLUMN date_data.pidx @@ -398,7 +398,7 @@ SELECT tap.fk_on_update('_caosdb_schema_unit_tests','date_data','date_ov_forkey_ -- *************************************************************** SELECT tap.has_table('_caosdb_schema_unit_tests','datetime_data',''); -SELECT tap.table_collation_is('_caosdb_schema_unit_tests','datetime_data','utf8mb3_unicode_ci',''); +-- SELECT tap.table_collation_is('_caosdb_schema_unit_tests','datetime_data','utf8mb4_uca1400_as_ci',''); SELECT tap.table_engine_is('_caosdb_schema_unit_tests','datetime_data','InnoDB',''); -- COLUMNS @@ -437,8 +437,8 @@ SELECT tap.has_column('_caosdb_schema_unit_tests','datetime_data','status',''); SELECT tap.col_column_type_is('_caosdb_schema_unit_tests','datetime_data','status','enum(\'OBLIGATORY\',\'RECOMMENDED\',\'SUGGESTED\',\'FIX\',\'REPLACEMENT\')',''); SELECT tap.col_extra_is('_caosdb_schema_unit_tests','datetime_data','status','',''); SELECT tap.col_default_is('_caosdb_schema_unit_tests','datetime_data','status',NULL,''); -SELECT tap.col_charset_is('_caosdb_schema_unit_tests','datetime_data','status','utf8mb3',''); -SELECT tap.col_collation_is('_caosdb_schema_unit_tests','datetime_data','status','utf8mb3_unicode_ci',''); +SELECT tap.col_charset_is('_caosdb_schema_unit_tests','datetime_data','status','utf8mb4',''); +SELECT tap.col_collation_is('_caosdb_schema_unit_tests','datetime_data','status','utf8mb4_uca1400_as_ci',''); -- COLUMN datetime_data.pidx @@ -505,7 +505,7 @@ SELECT tap.fk_on_update('_caosdb_schema_unit_tests','datetime_data','dat_propert -- *************************************************************** SELECT tap.has_table('_caosdb_schema_unit_tests','desc_overrides',''); -SELECT tap.table_collation_is('_caosdb_schema_unit_tests','desc_overrides','utf8mb3_unicode_ci',''); +-- SELECT tap.table_collation_is('_caosdb_schema_unit_tests','desc_overrides','utf8mb4_uca1400_as_ci',''); SELECT tap.table_engine_is('_caosdb_schema_unit_tests','desc_overrides','InnoDB',''); -- COLUMNS @@ -544,8 +544,8 @@ SELECT tap.has_column('_caosdb_schema_unit_tests','desc_overrides','description' SELECT tap.col_column_type_is('_caosdb_schema_unit_tests','desc_overrides','description','text',''); SELECT tap.col_extra_is('_caosdb_schema_unit_tests','desc_overrides','description','',''); SELECT tap.col_default_is('_caosdb_schema_unit_tests','desc_overrides','description','NULL',''); -SELECT tap.col_charset_is('_caosdb_schema_unit_tests','desc_overrides','description','utf8mb3',''); -SELECT tap.col_collation_is('_caosdb_schema_unit_tests','desc_overrides','description','utf8mb3_unicode_ci',''); +SELECT tap.col_charset_is('_caosdb_schema_unit_tests','desc_overrides','description','utf8mb4',''); +SELECT tap.col_collation_is('_caosdb_schema_unit_tests','desc_overrides','description','utf8mb4_uca1400_as_ci',''); -- INDEXES SELECT tap.indexes_are('_caosdb_schema_unit_tests','desc_overrides','`desc_ov_dom_ent_idx`',''); @@ -590,7 +590,7 @@ SELECT tap.constraint_type_is('_caosdb_schema_unit_tests','desc_overrides','desc -- *************************************************************** SELECT tap.has_table('_caosdb_schema_unit_tests','double_data',''); -SELECT tap.table_collation_is('_caosdb_schema_unit_tests','double_data','utf8mb3_unicode_ci',''); +-- SELECT tap.table_collation_is('_caosdb_schema_unit_tests','double_data','utf8mb4_uca1400_as_ci',''); SELECT tap.table_engine_is('_caosdb_schema_unit_tests','double_data','InnoDB',''); -- COLUMNS @@ -638,8 +638,8 @@ SELECT tap.has_column('_caosdb_schema_unit_tests','double_data','status',''); SELECT tap.col_column_type_is('_caosdb_schema_unit_tests','double_data','status','enum(\'OBLIGATORY\',\'RECOMMENDED\',\'SUGGESTED\',\'FIX\',\'REPLACEMENT\')',''); SELECT tap.col_extra_is('_caosdb_schema_unit_tests','double_data','status','',''); SELECT tap.col_default_is('_caosdb_schema_unit_tests','double_data','status',NULL,''); -SELECT tap.col_charset_is('_caosdb_schema_unit_tests','double_data','status','utf8mb3',''); -SELECT tap.col_collation_is('_caosdb_schema_unit_tests','double_data','status','utf8mb3_unicode_ci',''); +SELECT tap.col_charset_is('_caosdb_schema_unit_tests','double_data','status','utf8mb4',''); +SELECT tap.col_collation_is('_caosdb_schema_unit_tests','double_data','status','utf8mb4_uca1400_as_ci',''); -- COLUMN double_data.pidx @@ -697,7 +697,7 @@ SELECT tap.fk_on_update('_caosdb_schema_unit_tests','double_data','dou_property_ -- *************************************************************** SELECT tap.has_table('_caosdb_schema_unit_tests','entities',''); -SELECT tap.table_collation_is('_caosdb_schema_unit_tests','entities','utf8mb3_unicode_ci',''); +-- SELECT tap.table_collation_is('_caosdb_schema_unit_tests','entities','utf8mb4_uca1400_as_ci',''); SELECT tap.table_engine_is('_caosdb_schema_unit_tests','entities','InnoDB',''); -- COLUMNS @@ -718,8 +718,8 @@ SELECT tap.has_column('_caosdb_schema_unit_tests','entities','description',''); SELECT tap.col_column_type_is('_caosdb_schema_unit_tests','entities','description','text',''); SELECT tap.col_extra_is('_caosdb_schema_unit_tests','entities','description','',''); SELECT tap.col_default_is('_caosdb_schema_unit_tests','entities','description','NULL',''); -SELECT tap.col_charset_is('_caosdb_schema_unit_tests','entities','description','utf8mb3',''); -SELECT tap.col_collation_is('_caosdb_schema_unit_tests','entities','description','utf8mb3_unicode_ci',''); +SELECT tap.col_charset_is('_caosdb_schema_unit_tests','entities','description','utf8mb4',''); +SELECT tap.col_collation_is('_caosdb_schema_unit_tests','entities','description','utf8mb4_uca1400_as_ci',''); -- COLUMN entities.role @@ -727,8 +727,8 @@ SELECT tap.has_column('_caosdb_schema_unit_tests','entities','role',''); SELECT tap.col_column_type_is('_caosdb_schema_unit_tests','entities','role','enum(\'RECORDTYPE\',\'RECORD\',\'FILE\',\'_REPLACEMENT\',\'PROPERTY\',\'DATATYPE\',\'ROLE\',\'QUERYTEMPLATE\')',''); SELECT tap.col_extra_is('_caosdb_schema_unit_tests','entities','role','',''); SELECT tap.col_default_is('_caosdb_schema_unit_tests','entities','role',NULL,''); -SELECT tap.col_charset_is('_caosdb_schema_unit_tests','entities','role','utf8mb3',''); -SELECT tap.col_collation_is('_caosdb_schema_unit_tests','entities','role','utf8mb3_unicode_ci',''); +SELECT tap.col_charset_is('_caosdb_schema_unit_tests','entities','role','utf8mb4',''); +SELECT tap.col_collation_is('_caosdb_schema_unit_tests','entities','role','utf8mb4_uca1400_as_ci',''); -- COLUMN entities.acl @@ -760,7 +760,7 @@ SELECT tap.col_is_pk('_caosdb_schema_unit_tests','entities','`id`',''); -- *************************************************************** SELECT tap.has_table('_caosdb_schema_unit_tests','entity_acl',''); -SELECT tap.table_collation_is('_caosdb_schema_unit_tests','entity_acl','utf8mb3_unicode_ci',''); +-- SELECT tap.table_collation_is('_caosdb_schema_unit_tests','entity_acl','utf8mb4_uca1400_as_ci',''); SELECT tap.table_engine_is('_caosdb_schema_unit_tests','entity_acl','InnoDB',''); -- COLUMNS @@ -807,7 +807,7 @@ SELECT tap.col_is_pk('_caosdb_schema_unit_tests','entity_acl','`id`',''); -- *************************************************************** SELECT tap.has_table('_caosdb_schema_unit_tests','enum_data',''); -SELECT tap.table_collation_is('_caosdb_schema_unit_tests','enum_data','utf8mb3_unicode_ci',''); +-- SELECT tap.table_collation_is('_caosdb_schema_unit_tests','enum_data','utf8mb4_uca1400_as_ci',''); SELECT tap.table_engine_is('_caosdb_schema_unit_tests','enum_data','InnoDB',''); -- COLUMNS @@ -855,8 +855,8 @@ SELECT tap.has_column('_caosdb_schema_unit_tests','enum_data','status',''); SELECT tap.col_column_type_is('_caosdb_schema_unit_tests','enum_data','status','enum(\'OBLIGATORY\',\'RECOMMENDED\',\'SUGGESTED\',\'FIX\')',''); SELECT tap.col_extra_is('_caosdb_schema_unit_tests','enum_data','status','',''); SELECT tap.col_default_is('_caosdb_schema_unit_tests','enum_data','status','NULL',''); -SELECT tap.col_charset_is('_caosdb_schema_unit_tests','enum_data','status','utf8mb3',''); -SELECT tap.col_collation_is('_caosdb_schema_unit_tests','enum_data','status','utf8mb3_unicode_ci',''); +SELECT tap.col_charset_is('_caosdb_schema_unit_tests','enum_data','status','utf8mb4',''); +SELECT tap.col_collation_is('_caosdb_schema_unit_tests','enum_data','status','utf8mb4_uca1400_as_ci',''); -- COLUMN enum_data.pidx @@ -905,7 +905,7 @@ SELECT tap.fk_on_update('_caosdb_schema_unit_tests','enum_data','enum_ov_forkey_ -- *************************************************************** SELECT tap.has_table('_caosdb_schema_unit_tests','files',''); -SELECT tap.table_collation_is('_caosdb_schema_unit_tests','files','utf8mb3_unicode_ci',''); +-- SELECT tap.table_collation_is('_caosdb_schema_unit_tests','files','utf8mb4_uca1400_as_ci',''); SELECT tap.table_engine_is('_caosdb_schema_unit_tests','files','InnoDB',''); -- COLUMNS @@ -926,8 +926,8 @@ SELECT tap.has_column('_caosdb_schema_unit_tests','files','path',''); SELECT tap.col_column_type_is('_caosdb_schema_unit_tests','files','path','varchar(255)',''); SELECT tap.col_extra_is('_caosdb_schema_unit_tests','files','path','',''); SELECT tap.col_default_is('_caosdb_schema_unit_tests','files','path',NULL,''); -SELECT tap.col_charset_is('_caosdb_schema_unit_tests','files','path','utf8mb3',''); -SELECT tap.col_collation_is('_caosdb_schema_unit_tests','files','path','utf8mb3_unicode_ci',''); +SELECT tap.col_charset_is('_caosdb_schema_unit_tests','files','path','utf8mb4',''); +SELECT tap.col_collation_is('_caosdb_schema_unit_tests','files','path','utf8mb4_uca1400_as_ci',''); -- COLUMN files.size @@ -977,7 +977,7 @@ SELECT tap.col_is_pk('_caosdb_schema_unit_tests','files','`file_id`',''); -- *************************************************************** SELECT tap.has_table('_caosdb_schema_unit_tests','integer_data',''); -SELECT tap.table_collation_is('_caosdb_schema_unit_tests','integer_data','utf8mb3_unicode_ci',''); +-- SELECT tap.table_collation_is('_caosdb_schema_unit_tests','integer_data','utf8mb4_uca1400_as_ci',''); SELECT tap.table_engine_is('_caosdb_schema_unit_tests','integer_data','InnoDB',''); -- COLUMNS @@ -1025,8 +1025,8 @@ SELECT tap.has_column('_caosdb_schema_unit_tests','integer_data','status',''); SELECT tap.col_column_type_is('_caosdb_schema_unit_tests','integer_data','status','enum(\'OBLIGATORY\',\'RECOMMENDED\',\'SUGGESTED\',\'FIX\',\'REPLACEMENT\')',''); SELECT tap.col_extra_is('_caosdb_schema_unit_tests','integer_data','status','',''); SELECT tap.col_default_is('_caosdb_schema_unit_tests','integer_data','status',NULL,''); -SELECT tap.col_charset_is('_caosdb_schema_unit_tests','integer_data','status','utf8mb3',''); -SELECT tap.col_collation_is('_caosdb_schema_unit_tests','integer_data','status','utf8mb3_unicode_ci',''); +SELECT tap.col_charset_is('_caosdb_schema_unit_tests','integer_data','status','utf8mb4',''); +SELECT tap.col_collation_is('_caosdb_schema_unit_tests','integer_data','status','utf8mb4_uca1400_as_ci',''); -- COLUMN integer_data.pidx @@ -1084,7 +1084,7 @@ SELECT tap.fk_on_update('_caosdb_schema_unit_tests','integer_data','int_property -- *************************************************************** SELECT tap.has_table('_caosdb_schema_unit_tests','isa_cache',''); -SELECT tap.table_collation_is('_caosdb_schema_unit_tests','isa_cache','utf8mb3_unicode_ci',''); +-- SELECT tap.table_collation_is('_caosdb_schema_unit_tests','isa_cache','utf8mb4_uca1400_as_ci',''); SELECT tap.table_engine_is('_caosdb_schema_unit_tests','isa_cache','InnoDB',''); -- COLUMNS @@ -1114,8 +1114,8 @@ SELECT tap.has_column('_caosdb_schema_unit_tests','isa_cache','rpath',''); SELECT tap.col_column_type_is('_caosdb_schema_unit_tests','isa_cache','rpath','varchar(255)',''); SELECT tap.col_extra_is('_caosdb_schema_unit_tests','isa_cache','rpath','',''); SELECT tap.col_default_is('_caosdb_schema_unit_tests','isa_cache','rpath',NULL,''); -SELECT tap.col_charset_is('_caosdb_schema_unit_tests','isa_cache','rpath','utf8mb3',''); -SELECT tap.col_collation_is('_caosdb_schema_unit_tests','isa_cache','rpath','utf8mb3_unicode_ci',''); +SELECT tap.col_charset_is('_caosdb_schema_unit_tests','isa_cache','rpath','utf8mb4',''); +SELECT tap.col_collation_is('_caosdb_schema_unit_tests','isa_cache','rpath','utf8mb4_uca1400_as_ci',''); -- CONSTRAINTS SELECT tap.constraints_are('_caosdb_schema_unit_tests','isa_cache','`PRIMARY`,`isa_cache_child_entity`,`isa_cache_parent_entity`',''); @@ -1145,7 +1145,7 @@ SELECT tap.col_is_pk('_caosdb_schema_unit_tests','isa_cache','`child`,`parent`,` -- *************************************************************** SELECT tap.has_table('_caosdb_schema_unit_tests','name_data',''); -SELECT tap.table_collation_is('_caosdb_schema_unit_tests','name_data','utf8mb3_unicode_ci',''); +-- SELECT tap.table_collation_is('_caosdb_schema_unit_tests','name_data','utf8mb4_uca1400_as_ci',''); SELECT tap.table_engine_is('_caosdb_schema_unit_tests','name_data','InnoDB',''); -- COLUMNS @@ -1184,8 +1184,8 @@ SELECT tap.has_column('_caosdb_schema_unit_tests','name_data','value',''); SELECT tap.col_column_type_is('_caosdb_schema_unit_tests','name_data','value','varchar(255)',''); SELECT tap.col_extra_is('_caosdb_schema_unit_tests','name_data','value','',''); SELECT tap.col_default_is('_caosdb_schema_unit_tests','name_data','value',NULL,''); -SELECT tap.col_charset_is('_caosdb_schema_unit_tests','name_data','value','utf8mb3',''); -SELECT tap.col_collation_is('_caosdb_schema_unit_tests','name_data','value','utf8mb3_unicode_ci',''); +SELECT tap.col_charset_is('_caosdb_schema_unit_tests','name_data','value','utf8mb4',''); +SELECT tap.col_collation_is('_caosdb_schema_unit_tests','name_data','value','utf8mb4_uca1400_as_ci',''); -- COLUMN name_data.status @@ -1193,8 +1193,8 @@ SELECT tap.has_column('_caosdb_schema_unit_tests','name_data','status',''); SELECT tap.col_column_type_is('_caosdb_schema_unit_tests','name_data','status','enum(\'OBLIGATORY\',\'RECOMMENDED\',\'SUGGESTED\',\'FIX\',\'REPLACEMENT\')',''); SELECT tap.col_extra_is('_caosdb_schema_unit_tests','name_data','status','',''); SELECT tap.col_default_is('_caosdb_schema_unit_tests','name_data','status',NULL,''); -SELECT tap.col_charset_is('_caosdb_schema_unit_tests','name_data','status','utf8mb3',''); -SELECT tap.col_collation_is('_caosdb_schema_unit_tests','name_data','status','utf8mb3_unicode_ci',''); +SELECT tap.col_charset_is('_caosdb_schema_unit_tests','name_data','status','utf8mb4',''); +SELECT tap.col_collation_is('_caosdb_schema_unit_tests','name_data','status','utf8mb4_uca1400_as_ci',''); -- COLUMN name_data.pidx @@ -1266,7 +1266,7 @@ SELECT tap.fk_on_update('_caosdb_schema_unit_tests','name_data','name_data_prope -- *************************************************************** SELECT tap.has_table('_caosdb_schema_unit_tests','name_overrides',''); -SELECT tap.table_collation_is('_caosdb_schema_unit_tests','name_overrides','utf8mb3_unicode_ci',''); +-- SELECT tap.table_collation_is('_caosdb_schema_unit_tests','name_overrides','utf8mb4_uca1400_as_ci',''); SELECT tap.table_engine_is('_caosdb_schema_unit_tests','name_overrides','InnoDB',''); -- COLUMNS @@ -1305,8 +1305,8 @@ SELECT tap.has_column('_caosdb_schema_unit_tests','name_overrides','name',''); SELECT tap.col_column_type_is('_caosdb_schema_unit_tests','name_overrides','name','varchar(255)',''); SELECT tap.col_extra_is('_caosdb_schema_unit_tests','name_overrides','name','',''); SELECT tap.col_default_is('_caosdb_schema_unit_tests','name_overrides','name','NULL',''); -SELECT tap.col_charset_is('_caosdb_schema_unit_tests','name_overrides','name','utf8mb3',''); -SELECT tap.col_collation_is('_caosdb_schema_unit_tests','name_overrides','name','utf8mb3_unicode_ci',''); +SELECT tap.col_charset_is('_caosdb_schema_unit_tests','name_overrides','name','utf8mb4',''); +SELECT tap.col_collation_is('_caosdb_schema_unit_tests','name_overrides','name','utf8mb4_uca1400_as_ci',''); -- INDEXES SELECT tap.indexes_are('_caosdb_schema_unit_tests','name_overrides','`name_ov_dom_ent_idx`',''); @@ -1351,7 +1351,7 @@ SELECT tap.constraint_type_is('_caosdb_schema_unit_tests','name_overrides','name -- *************************************************************** SELECT tap.has_table('_caosdb_schema_unit_tests','null_data',''); -SELECT tap.table_collation_is('_caosdb_schema_unit_tests','null_data','utf8mb3_unicode_ci',''); +-- SELECT tap.table_collation_is('_caosdb_schema_unit_tests','null_data','utf8mb4_uca1400_as_ci',''); SELECT tap.table_engine_is('_caosdb_schema_unit_tests','null_data','InnoDB',''); -- COLUMNS @@ -1390,8 +1390,8 @@ SELECT tap.has_column('_caosdb_schema_unit_tests','null_data','status',''); SELECT tap.col_column_type_is('_caosdb_schema_unit_tests','null_data','status','enum(\'OBLIGATORY\',\'RECOMMENDED\',\'SUGGESTED\',\'FIX\')',''); SELECT tap.col_extra_is('_caosdb_schema_unit_tests','null_data','status','',''); SELECT tap.col_default_is('_caosdb_schema_unit_tests','null_data','status','NULL',''); -SELECT tap.col_charset_is('_caosdb_schema_unit_tests','null_data','status','utf8mb3',''); -SELECT tap.col_collation_is('_caosdb_schema_unit_tests','null_data','status','utf8mb3_unicode_ci',''); +SELECT tap.col_charset_is('_caosdb_schema_unit_tests','null_data','status','utf8mb4',''); +SELECT tap.col_collation_is('_caosdb_schema_unit_tests','null_data','status','utf8mb4_uca1400_as_ci',''); -- COLUMN null_data.pidx @@ -1440,7 +1440,7 @@ SELECT tap.fk_on_update('_caosdb_schema_unit_tests','null_data','null_forkey_pro -- *************************************************************** SELECT tap.has_table('_caosdb_schema_unit_tests','passwd',''); -SELECT tap.table_collation_is('_caosdb_schema_unit_tests','passwd','utf8mb3_unicode_ci',''); +-- SELECT tap.table_collation_is('_caosdb_schema_unit_tests','passwd','utf8mb4_uca1400_as_ci',''); SELECT tap.table_engine_is('_caosdb_schema_unit_tests','passwd','InnoDB',''); -- COLUMNS @@ -1470,8 +1470,8 @@ SELECT tap.has_column('_caosdb_schema_unit_tests','passwd','alg',''); SELECT tap.col_column_type_is('_caosdb_schema_unit_tests','passwd','alg','varchar(255)',''); SELECT tap.col_extra_is('_caosdb_schema_unit_tests','passwd','alg','',''); SELECT tap.col_default_is('_caosdb_schema_unit_tests','passwd','alg','\'SHA-512\'',''); -SELECT tap.col_charset_is('_caosdb_schema_unit_tests','passwd','alg','utf8mb3',''); -SELECT tap.col_collation_is('_caosdb_schema_unit_tests','passwd','alg','utf8mb3_unicode_ci',''); +SELECT tap.col_charset_is('_caosdb_schema_unit_tests','passwd','alg','utf8mb4',''); +SELECT tap.col_collation_is('_caosdb_schema_unit_tests','passwd','alg','utf8mb4_uca1400_as_ci',''); -- COLUMN passwd.it @@ -1505,7 +1505,7 @@ SELECT tap.col_is_pk('_caosdb_schema_unit_tests','passwd','`principal`',''); -- *************************************************************** SELECT tap.has_table('_caosdb_schema_unit_tests','permissions',''); -SELECT tap.table_collation_is('_caosdb_schema_unit_tests','permissions','utf8mb3_unicode_ci',''); +-- SELECT tap.table_collation_is('_caosdb_schema_unit_tests','permissions','utf8mb4_uca1400_as_ci',''); SELECT tap.table_engine_is('_caosdb_schema_unit_tests','permissions','InnoDB',''); -- COLUMNS @@ -1526,8 +1526,8 @@ SELECT tap.has_column('_caosdb_schema_unit_tests','permissions','permissions','' SELECT tap.col_column_type_is('_caosdb_schema_unit_tests','permissions','permissions','mediumtext',''); SELECT tap.col_extra_is('_caosdb_schema_unit_tests','permissions','permissions','',''); SELECT tap.col_default_is('_caosdb_schema_unit_tests','permissions','permissions',NULL,''); -SELECT tap.col_charset_is('_caosdb_schema_unit_tests','permissions','permissions','utf8mb3',''); -SELECT tap.col_collation_is('_caosdb_schema_unit_tests','permissions','permissions','utf8mb3_unicode_ci',''); +SELECT tap.col_charset_is('_caosdb_schema_unit_tests','permissions','permissions','utf8mb4',''); +SELECT tap.col_collation_is('_caosdb_schema_unit_tests','permissions','permissions','utf8mb4_uca1400_as_ci',''); -- CONSTRAINTS SELECT tap.constraints_are('_caosdb_schema_unit_tests','permissions','`PRIMARY`,`perm_name_roles`',''); @@ -1550,7 +1550,7 @@ SELECT tap.col_is_pk('_caosdb_schema_unit_tests','permissions','`role`',''); -- *************************************************************** SELECT tap.has_table('_caosdb_schema_unit_tests','query_template_def',''); -SELECT tap.table_collation_is('_caosdb_schema_unit_tests','query_template_def','utf8mb3_unicode_ci',''); +-- SELECT tap.table_collation_is('_caosdb_schema_unit_tests','query_template_def','utf8mb4_uca1400_as_ci',''); SELECT tap.table_engine_is('_caosdb_schema_unit_tests','query_template_def','InnoDB',''); -- COLUMNS @@ -1571,8 +1571,8 @@ SELECT tap.has_column('_caosdb_schema_unit_tests','query_template_def','definiti SELECT tap.col_column_type_is('_caosdb_schema_unit_tests','query_template_def','definition','mediumtext',''); SELECT tap.col_extra_is('_caosdb_schema_unit_tests','query_template_def','definition','',''); SELECT tap.col_default_is('_caosdb_schema_unit_tests','query_template_def','definition',NULL,''); -SELECT tap.col_charset_is('_caosdb_schema_unit_tests','query_template_def','definition','utf8mb3',''); -SELECT tap.col_collation_is('_caosdb_schema_unit_tests','query_template_def','definition','utf8mb3_unicode_ci',''); +SELECT tap.col_charset_is('_caosdb_schema_unit_tests','query_template_def','definition','utf8mb4',''); +SELECT tap.col_collation_is('_caosdb_schema_unit_tests','query_template_def','definition','utf8mb4_uca1400_as_ci',''); -- CONSTRAINTS SELECT tap.constraints_are('_caosdb_schema_unit_tests','query_template_def','`PRIMARY`,`query_template_def_ibfk_1`',''); @@ -1595,7 +1595,7 @@ SELECT tap.fk_on_update('_caosdb_schema_unit_tests','query_template_def','query_ -- *************************************************************** SELECT tap.has_table('_caosdb_schema_unit_tests','reference_data',''); -SELECT tap.table_collation_is('_caosdb_schema_unit_tests','reference_data','utf8mb3_unicode_ci',''); +-- SELECT tap.table_collation_is('_caosdb_schema_unit_tests','reference_data','utf8mb4_uca1400_as_ci',''); SELECT tap.table_engine_is('_caosdb_schema_unit_tests','reference_data','InnoDB',''); -- COLUMNS @@ -1643,8 +1643,8 @@ SELECT tap.has_column('_caosdb_schema_unit_tests','reference_data','status',''); SELECT tap.col_column_type_is('_caosdb_schema_unit_tests','reference_data','status','enum(\'OBLIGATORY\',\'RECOMMENDED\',\'SUGGESTED\',\'FIX\',\'REPLACEMENT\')',''); SELECT tap.col_extra_is('_caosdb_schema_unit_tests','reference_data','status','',''); SELECT tap.col_default_is('_caosdb_schema_unit_tests','reference_data','status',NULL,''); -SELECT tap.col_charset_is('_caosdb_schema_unit_tests','reference_data','status','utf8mb3',''); -SELECT tap.col_collation_is('_caosdb_schema_unit_tests','reference_data','status','utf8mb3_unicode_ci',''); +SELECT tap.col_charset_is('_caosdb_schema_unit_tests','reference_data','status','utf8mb4',''); +SELECT tap.col_collation_is('_caosdb_schema_unit_tests','reference_data','status','utf8mb4_uca1400_as_ci',''); -- COLUMN reference_data.pidx @@ -1722,7 +1722,7 @@ SELECT tap.fk_on_update('_caosdb_schema_unit_tests','reference_data','reference_ -- *************************************************************** SELECT tap.has_table('_caosdb_schema_unit_tests','roles',''); -SELECT tap.table_collation_is('_caosdb_schema_unit_tests','roles','utf8mb3_unicode_ci',''); +-- SELECT tap.table_collation_is('_caosdb_schema_unit_tests','roles','utf8mb4_uca1400_as_ci',''); SELECT tap.table_engine_is('_caosdb_schema_unit_tests','roles','InnoDB',''); -- COLUMNS @@ -1743,8 +1743,8 @@ SELECT tap.has_column('_caosdb_schema_unit_tests','roles','description',''); SELECT tap.col_column_type_is('_caosdb_schema_unit_tests','roles','description','mediumtext',''); SELECT tap.col_extra_is('_caosdb_schema_unit_tests','roles','description','',''); SELECT tap.col_default_is('_caosdb_schema_unit_tests','roles','description','NULL',''); -SELECT tap.col_charset_is('_caosdb_schema_unit_tests','roles','description','utf8mb3',''); -SELECT tap.col_collation_is('_caosdb_schema_unit_tests','roles','description','utf8mb3_unicode_ci',''); +SELECT tap.col_charset_is('_caosdb_schema_unit_tests','roles','description','utf8mb4',''); +SELECT tap.col_collation_is('_caosdb_schema_unit_tests','roles','description','utf8mb4_uca1400_as_ci',''); -- CONSTRAINTS SELECT tap.constraints_are('_caosdb_schema_unit_tests','roles','`PRIMARY`',''); @@ -1760,7 +1760,7 @@ SELECT tap.col_is_pk('_caosdb_schema_unit_tests','roles','`name`',''); -- *************************************************************** SELECT tap.has_table('_caosdb_schema_unit_tests','stats',''); -SELECT tap.table_collation_is('_caosdb_schema_unit_tests','stats','utf8mb3_unicode_ci',''); +-- SELECT tap.table_collation_is('_caosdb_schema_unit_tests','stats','utf8mb4_uca1400_as_ci',''); SELECT tap.table_engine_is('_caosdb_schema_unit_tests','stats','InnoDB',''); -- COLUMNS @@ -1772,8 +1772,8 @@ SELECT tap.has_column('_caosdb_schema_unit_tests','stats','name',''); SELECT tap.col_column_type_is('_caosdb_schema_unit_tests','stats','name','varchar(255)',''); SELECT tap.col_extra_is('_caosdb_schema_unit_tests','stats','name','',''); SELECT tap.col_default_is('_caosdb_schema_unit_tests','stats','name',NULL,''); -SELECT tap.col_charset_is('_caosdb_schema_unit_tests','stats','name','utf8mb3',''); -SELECT tap.col_collation_is('_caosdb_schema_unit_tests','stats','name','utf8mb3_unicode_ci',''); +SELECT tap.col_charset_is('_caosdb_schema_unit_tests','stats','name','utf8mb4',''); +SELECT tap.col_collation_is('_caosdb_schema_unit_tests','stats','name','utf8mb4_uca1400_as_ci',''); -- COLUMN stats.value @@ -1798,7 +1798,7 @@ SELECT tap.col_is_pk('_caosdb_schema_unit_tests','stats','`name`',''); -- *************************************************************** SELECT tap.has_table('_caosdb_schema_unit_tests','text_data',''); -SELECT tap.table_collation_is('_caosdb_schema_unit_tests','text_data','utf8mb3_unicode_ci',''); +-- SELECT tap.table_collation_is('_caosdb_schema_unit_tests','text_data','utf8mb4_uca1400_as_ci',''); SELECT tap.table_engine_is('_caosdb_schema_unit_tests','text_data','InnoDB',''); -- COLUMNS @@ -1837,8 +1837,8 @@ SELECT tap.has_column('_caosdb_schema_unit_tests','text_data','value',''); SELECT tap.col_column_type_is('_caosdb_schema_unit_tests','text_data','value','text',''); SELECT tap.col_extra_is('_caosdb_schema_unit_tests','text_data','value','',''); SELECT tap.col_default_is('_caosdb_schema_unit_tests','text_data','value',NULL,''); -SELECT tap.col_charset_is('_caosdb_schema_unit_tests','text_data','value','utf8mb3',''); -SELECT tap.col_collation_is('_caosdb_schema_unit_tests','text_data','value','utf8mb3_unicode_ci',''); +SELECT tap.col_charset_is('_caosdb_schema_unit_tests','text_data','value','utf8mb4',''); +SELECT tap.col_collation_is('_caosdb_schema_unit_tests','text_data','value','utf8mb4_uca1400_as_ci',''); -- COLUMN text_data.status @@ -1846,8 +1846,8 @@ SELECT tap.has_column('_caosdb_schema_unit_tests','text_data','status',''); SELECT tap.col_column_type_is('_caosdb_schema_unit_tests','text_data','status','enum(\'OBLIGATORY\',\'RECOMMENDED\',\'SUGGESTED\',\'FIX\',\'REPLACEMENT\')',''); SELECT tap.col_extra_is('_caosdb_schema_unit_tests','text_data','status','',''); SELECT tap.col_default_is('_caosdb_schema_unit_tests','text_data','status',NULL,''); -SELECT tap.col_charset_is('_caosdb_schema_unit_tests','text_data','status','utf8mb3',''); -SELECT tap.col_collation_is('_caosdb_schema_unit_tests','text_data','status','utf8mb3_unicode_ci',''); +SELECT tap.col_charset_is('_caosdb_schema_unit_tests','text_data','status','utf8mb4',''); +SELECT tap.col_collation_is('_caosdb_schema_unit_tests','text_data','status','utf8mb4_uca1400_as_ci',''); -- COLUMN text_data.pidx @@ -1896,7 +1896,7 @@ SELECT tap.fk_on_update('_caosdb_schema_unit_tests','text_data','str_property_id -- *************************************************************** SELECT tap.has_table('_caosdb_schema_unit_tests','transaction_log',''); -SELECT tap.table_collation_is('_caosdb_schema_unit_tests','transaction_log','utf8mb3_unicode_ci',''); +-- SELECT tap.table_collation_is('_caosdb_schema_unit_tests','transaction_log','utf8mb4_uca1400_as_ci',''); SELECT tap.table_engine_is('_caosdb_schema_unit_tests','transaction_log','InnoDB',''); -- COLUMNS @@ -1908,8 +1908,8 @@ SELECT tap.has_column('_caosdb_schema_unit_tests','transaction_log','transaction SELECT tap.col_column_type_is('_caosdb_schema_unit_tests','transaction_log','transaction','varchar(255)',''); SELECT tap.col_extra_is('_caosdb_schema_unit_tests','transaction_log','transaction','',''); SELECT tap.col_default_is('_caosdb_schema_unit_tests','transaction_log','transaction',NULL,''); -SELECT tap.col_charset_is('_caosdb_schema_unit_tests','transaction_log','transaction','utf8mb3',''); -SELECT tap.col_collation_is('_caosdb_schema_unit_tests','transaction_log','transaction','utf8mb3_unicode_ci',''); +SELECT tap.col_charset_is('_caosdb_schema_unit_tests','transaction_log','transaction','utf8mb4',''); +SELECT tap.col_collation_is('_caosdb_schema_unit_tests','transaction_log','transaction','utf8mb4_uca1400_as_ci',''); -- COLUMN transaction_log.entity_id @@ -1970,7 +1970,7 @@ SELECT tap.is_indexed('_caosdb_schema_unit_tests','transaction_log','`entity_id` -- *************************************************************** SELECT tap.has_table('_caosdb_schema_unit_tests','units_lin_con',''); -SELECT tap.table_collation_is('_caosdb_schema_unit_tests','units_lin_con','utf8mb3_unicode_ci',''); +-- SELECT tap.table_collation_is('_caosdb_schema_unit_tests','units_lin_con','utf8mb4_uca1400_as_ci',''); SELECT tap.table_engine_is('_caosdb_schema_unit_tests','units_lin_con','InnoDB',''); -- COLUMNS @@ -2044,7 +2044,7 @@ SELECT tap.col_is_pk('_caosdb_schema_unit_tests','units_lin_con','`signature_fro -- *************************************************************** SELECT tap.has_table('_caosdb_schema_unit_tests','user_info',''); -SELECT tap.table_collation_is('_caosdb_schema_unit_tests','user_info','utf8mb3_unicode_ci',''); +-- SELECT tap.table_collation_is('_caosdb_schema_unit_tests','user_info','utf8mb4_uca1400_as_ci',''); SELECT tap.table_engine_is('_caosdb_schema_unit_tests','user_info','InnoDB',''); -- COLUMNS @@ -2083,8 +2083,8 @@ SELECT tap.has_column('_caosdb_schema_unit_tests','user_info','status',''); SELECT tap.col_column_type_is('_caosdb_schema_unit_tests','user_info','status','enum(\'ACTIVE\',\'INACTIVE\')',''); SELECT tap.col_extra_is('_caosdb_schema_unit_tests','user_info','status','',''); SELECT tap.col_default_is('_caosdb_schema_unit_tests','user_info','status','\'INACTIVE\'',''); -SELECT tap.col_charset_is('_caosdb_schema_unit_tests','user_info','status','utf8mb3',''); -SELECT tap.col_collation_is('_caosdb_schema_unit_tests','user_info','status','utf8mb3_unicode_ci',''); +SELECT tap.col_charset_is('_caosdb_schema_unit_tests','user_info','status','utf8mb4',''); +SELECT tap.col_collation_is('_caosdb_schema_unit_tests','user_info','status','utf8mb4_uca1400_as_ci',''); -- COLUMN user_info.entity @@ -2125,7 +2125,7 @@ SELECT tap.fk_on_update('_caosdb_schema_unit_tests','user_info','subjects_ibfk_2 -- *************************************************************** SELECT tap.has_table('_caosdb_schema_unit_tests','user_roles',''); -SELECT tap.table_collation_is('_caosdb_schema_unit_tests','user_roles','utf8mb3_unicode_ci',''); +-- SELECT tap.table_collation_is('_caosdb_schema_unit_tests','user_roles','utf8mb4_uca1400_as_ci',''); SELECT tap.table_engine_is('_caosdb_schema_unit_tests','user_roles','InnoDB',''); -- COLUMNS @@ -2179,7 +2179,7 @@ SELECT tap.fk_on_update('_caosdb_schema_unit_tests','user_roles','user_roles_ibf -- *************************************************************** SELECT tap.has_table('_caosdb_schema_unit_tests','entity_version',''); -SELECT tap.table_collation_is('_caosdb_schema_unit_tests','entity_version','utf8mb3_unicode_ci',''); +-- SELECT tap.table_collation_is('_caosdb_schema_unit_tests','entity_version','utf8mb4_uca1400_as_ci',''); SELECT tap.table_engine_is('_caosdb_schema_unit_tests','entity_version','InnoDB',''); -- COLUMNS -- GitLab From ffed88c04643b187150a9bf76bc885f83994ac6d Mon Sep 17 00:00:00 2001 From: Daniel <d.hornung@indiscale.com> Date: Tue, 4 Feb 2025 17:37:37 +0100 Subject: [PATCH 14/18] WIP: Patch --- .docker/docker-compose.yml | 2 +- Makefile | 12 ++++++------ doc/Maintenance.rst | 7 ++----- 3 files changed, 9 insertions(+), 12 deletions(-) diff --git a/.docker/docker-compose.yml b/.docker/docker-compose.yml index 1a63292..8c6555e 100644 --- a/.docker/docker-compose.yml +++ b/.docker/docker-compose.yml @@ -1,7 +1,7 @@ version: '3.6' services: sqldb: - image: mariadb:10.5 + image: mariadb:11.4 volumes: - type: volume source: "caosdb-sqldata" diff --git a/Makefile b/Makefile index 4bfdc03..9bd45f7 100644 --- a/Makefile +++ b/Makefile @@ -62,10 +62,10 @@ test-dump-update: # Run tests with a database which is started in a MariaDB Docker container .PHONY: test-docker test-docker: - @docker kill caosdb-mysqlserver-test || true - @docker container rm -v caosdb-mysqlserver-test || true - @docker run --name caosdb-mysqlserver-test -p "3307:3306" \ - -e MYSQL_ROOT_PASSWORD="pass-for-test" -d mariadb:10.11 + @docker kill linkahead-mysqlserver-test || true + @docker container rm -v linkahead-mysqlserver-test || true + @docker run --name linkahead-mysqlserver-test -p "3306:3306" \ + -e MYSQL_ROOT_PASSWORD="pass-for-test" -d mariadb:11.4 @sleep 10 MAINPATH=$(realpath tests/docker_env) utils/make_db test --fresh make test-docker-stop @@ -73,8 +73,8 @@ test-docker: # if automatic stopping failed .PHONY: test-docker-stop test-docker-stop: - docker kill caosdb-mysqlserver-test - docker container rm -v caosdb-mysqlserver-test + docker kill linkahead-mysqlserver-test + docker container rm -v linkahead-mysqlserver-test # Compile the standalone documentation .PHONY: doc diff --git a/doc/Maintenance.rst b/doc/Maintenance.rst index 39b2ad3..b8bedee 100644 --- a/doc/Maintenance.rst +++ b/doc/Maintenance.rst @@ -1,22 +1,19 @@ - Maintenance =========== - Creating a Backup ----------------- - You can use the Python script ``utils/backup.py`` to create a backup (SQL dump) of the SQL-Backend:: ./utils/backup.py -d folder/where/the/backup/is/created -You can do this while CaosDB is online. +You can do this while LinkAhead is online. Restoring a Backup ------------------ -CaosDB should be offline for restoring. +LinkAhead should be offline for restoring. You can use the Bash script ``utils/make_db`` to restore a backup (SQL dump) of the SQL-Backend:: -- GitLab From b3638eb88ff718eb185d93057304014a49ae7c23 Mon Sep 17 00:00:00 2001 From: Daniel <d.hornung@indiscale.com> Date: Wed, 5 Feb 2025 13:15:30 +0100 Subject: [PATCH 15/18] DOC: Explain why tests are commented out. --- tests/test_autotap.sql | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/tests/test_autotap.sql b/tests/test_autotap.sql index be7c63f..6d62a0c 100644 --- a/tests/test_autotap.sql +++ b/tests/test_autotap.sql @@ -38,6 +38,10 @@ SELECT tap.has_schema('_caosdb_schema_unit_tests',''); -- *************************************************************** SELECT tap.has_table('_caosdb_schema_unit_tests','archive_isa',''); +-- ******* +-- Collation testing of tables fails for uca1400 collations, see upstream issue: +-- https://github.com/hepabolu/mytap/issues/58 +-- ******* -- SELECT tap.table_collation_is('_caosdb_schema_unit_tests','archive_isa','utf8mb4_uca1400_as_ci',''); SELECT tap.table_engine_is('_caosdb_schema_unit_tests','archive_isa','InnoDB',''); -- GitLab From c0ec024527918714a85209306bff8e4fa7333e35 Mon Sep 17 00:00:00 2001 From: Florian Spreckelsen <f.spreckelsen@indiscale.com> Date: Fri, 7 Feb 2025 10:02:16 +0100 Subject: [PATCH 16/18] MAINT: Use correct patch version --- CHANGELOG.md | 14 +++++++++----- .../patch.sh | 2 +- 2 files changed, 10 insertions(+), 6 deletions(-) rename patches/{patch20250130-8.0.1 => patch20250130-8.1.0}/patch.sh (99%) diff --git a/CHANGELOG.md b/CHANGELOG.md index 32b3348..e102be5 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -9,10 +9,10 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### Added -### Changed - - New version requirement: MariaDB 11.2 or later. +### Changed + ### Deprecated ### Removed @@ -21,9 +21,13 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### Fixed -- [SQL dump migration can now be applied twice](https://gitlab.indiscale.com/caosdb/src/caosdb-mysqlbackend/-/issues/60): Fixed an error in the regex. -- [linkahead-server#264 Queries ignore accents and umlauts](https://gitlab.com/linkahead/linkahead-server/-/issues/264): Changed to more appropriate - collation and character sets. +- [SQL dump migration can now be applied + twice](https://gitlab.indiscale.com/caosdb/src/caosdb-mysqlbackend/-/issues/60): + Fixed an error in the regex. +- [linkahead-server#264](https://gitlab.com/linkahead/linkahead-server/-/issues/264) + Queries ignore accents and umlauts: Changed to more appropriate + collation and character sets. This requires MariaDB 11.2 or later, + see above. ### Security diff --git a/patches/patch20250130-8.0.1/patch.sh b/patches/patch20250130-8.1.0/patch.sh similarity index 99% rename from patches/patch20250130-8.0.1/patch.sh rename to patches/patch20250130-8.1.0/patch.sh index aca1f7a..5a09215 100755 --- a/patches/patch20250130-8.0.1/patch.sh +++ b/patches/patch20250130-8.1.0/patch.sh @@ -24,7 +24,7 @@ # Set the collation so that accents on characters become relevant. OLD_VERSION="v8.0.0" -NEW_VERSION="v8.0.1" +NEW_VERSION="v8.1.0" if [ -z "$UTILSPATH" ]; then UTILSPATH="../utils" -- GitLab From bcb35195aea75cf371390a05b50324dc905e53f8 Mon Sep 17 00:00:00 2001 From: Florian Spreckelsen <f.spreckelsen@indiscale.com> Date: Fri, 27 Jun 2025 15:58:52 +0200 Subject: [PATCH 17/18] BUILD: Extend .gitignore --- .gitignore | 2 ++ 1 file changed, 2 insertions(+) diff --git a/.gitignore b/.gitignore index 6f952db..4ae81f3 100644 --- a/.gitignore +++ b/.gitignore @@ -15,3 +15,5 @@ _doxygen/ _generated/ /build/ *~ + +__pycache__ \ No newline at end of file -- GitLab From f585772d01a6267d5041c1b9089a66a74a3b9204 Mon Sep 17 00:00:00 2001 From: Florian Spreckelsen <f.spreckelsen@indiscale.com> Date: Fri, 27 Jun 2025 16:05:15 +0200 Subject: [PATCH 18/18] BUILD: Bump versions for release 8.1.0 --- CHANGELOG.md | 8 +------- CITATION.cff | 4 ++-- doc/conf.py | 4 ++-- 3 files changed, 5 insertions(+), 11 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index e102be5..cb4d9fb 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -5,16 +5,12 @@ All notable changes to this project will be documented in this file. The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). -## [Unreleased] +## [8.1.0] - 2025-06-27 ### Added - New version requirement: MariaDB 11.2 or later. -### Changed - -### Deprecated - ### Removed - MySQL support. See the deprecation in the 7.0.0 release. @@ -29,8 +25,6 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 collation and character sets. This requires MariaDB 11.2 or later, see above. -### Security - ## [8.0.0] - 2024-10-24 ## ### Added ### diff --git a/CITATION.cff b/CITATION.cff index 3d997f9..49e4228 100644 --- a/CITATION.cff +++ b/CITATION.cff @@ -23,6 +23,6 @@ authors: given-names: Stefan orcid: https://orcid.org/0000-0001-7214-8125 title: "LinkAhead - MariaDB Backend" -version: 8.0.0 +version: 8.1.0 doi: 10.3390/data4020083 -date-released: 2024-10-10 +date-released: 2025-06-27 diff --git a/doc/conf.py b/doc/conf.py index b679a2b..8b94ae3 100644 --- a/doc/conf.py +++ b/doc/conf.py @@ -26,9 +26,9 @@ copyright = '2021 - 2024, IndiScale GmbH' author = 'Daniel Hornung' # The short X.Y version -version = '8.0.1' +version = '8.1.0' # The full version, including alpha/beta/rc tags -release = '8.0.1-SNAPSHOT' +release = '8.1.0' # -- General configuration --------------------------------------------------- -- GitLab