Skip to content
Snippets Groups Projects

Enable validation in table_json_conversion.convert.to_dict for use in XLSX-converter

Enable validation in table_json_conversion.convert.to_dict for use in XLSX-converter
f-bug-138-validate-table_json_conversion into dev
1 unresolved thread
Merged I. Nüske requested to merge f-bug-138-validate-table_json_conversion into dev
1 unresolved thread

Summary

Reenables failing validation test. Changes some test data and adds preprocessing to the jsonschema validation.
Addresses #138 (closed) / https://gitlab.indiscale.com/caosdb/customers/dimr/management/-/issues/268

Focus

  • Are the data_schemas an acceptable solution? Were the original schemas automatically generated? Does this necessitate any changes to other code?
  • Allowing nullable properties can also be done in a json schema. The current solution using filtering is more lenient, but in my opinion it might be desirable to at some point switch to using oneof(null, validType) everywhere applicable, to adhere more closely to the standard.
  • I did not find a way to validate the dates properly. A custom TypeChecker does not fit this use case, as we could accept datetimes as string everywhere, but not only when the format contains 'date'. Using a custom FormatChecker also does not work, as the datetime in type:string format:datetime fields already fails on the typecheck. This left filtering as the most convenient solution.

Test Environment

Manual Testing.

Check List for the Authoe

  • All automated tests pass
  • Reference related issues
  • Up-to-date CHANGELOG.md or not necessary
  • Up-to-date JSON schema or not necessary
  • Appropriate user and developer documentation (or not necessary)
  • Annotations in code (Gitlab comments)

Check List for the Reviewer

  • I understand the intent of this MR
  • All automated tests pass
  • Up-to-date CHANGELOG.md (or not necessary)
  • Appropriate user and developer documentation (or not necessary)
  • The test environment setup works and the intended behavior is reproducible in the test environment
  • In-code documentation and comments are up-to-date.
  • Check: Are there specifications? Are they satisfied?
Edited by Daniel Hornung

Merge request reports

Loading
Loading

Activity

Filter activity
  • Approvals
  • Assignees & reviewers
  • Comments (from bots)
  • Comments (from users)
  • Commits & branches
  • Edits
  • Labels
  • Lock status
  • Mentions
  • Merge request status
  • Tracking
  • I. Nüske
  • I. Nüske marked the checklist item Annotations in code (Gitlab comments) as completed

    marked the checklist item Annotations in code (Gitlab comments) as completed

  • Daniel Hornung marked the checklist item I understand the intent of this MR as completed

    marked the checklist item I understand the intent of this MR as completed

  • Daniel Hornung resolved all threads

    resolved all threads

  • Daniel Hornung added 2 commits

    added 2 commits

    • 67c3f944 - MAINT: Using array_schema_from_model_schema instead of hard coding.
    • 815d30d1 - DOC: Small typo.

    Compare with previous version

  • Daniel Hornung
    Daniel Hornung @daniel started a thread on the diff
    • unittests/table_json_conversion/test_read_xlsx.py
    53 53 json: dict
    54 54 The result of the conversion.
    55 55 """
    56 # FIXME Set default "validate" back to True, after implementation of
    57 # https://gitlab.indiscale.com/caosdb/src/caosdb-advanced-user-tools/-/issues/138
    58 result = convert.to_dict(xlsx=xlsx_file, schema=schema_file, validate=validate)
    56 with open(schema_file, encoding="utf8", mode="r") as sch_f:
    57 model_schema = json.load(sch_f)
    58 data_schema = xlsx_utils.array_schema_from_model_schema(model_schema)
  • Daniel Hornung added 6 commits

    added 6 commits

    Compare with previous version

  • Daniel Hornung marked the checklist item Up-to-date CHANGELOG.md (or not necessary) as completed

    marked the checklist item Up-to-date CHANGELOG.md (or not necessary) as completed

  • Daniel Hornung marked the checklist item Appropriate user and developer documentation (or not necessary) as completed

    marked the checklist item Appropriate user and developer documentation (or not necessary) as completed

  • Daniel Hornung marked the checklist item The test environment setup works and the intended behavior is reproducible in the test as completed

    marked the checklist item The test environment setup works and the intended behavior is reproducible in the test as completed

  • Daniel Hornung marked the checklist item In-code documentation and comments are up-to-date. as completed

    marked the checklist item In-code documentation and comments are up-to-date. as completed

  • Daniel Hornung marked the checklist item Check: Are there specifications? Are they satisfied? as completed

    marked the checklist item Check: Are there specifications? Are they satisfied? as completed

  • Daniel Hornung marked the checklist item All automated tests pass as completed

    marked the checklist item All automated tests pass as completed

  • I. Nüske mentioned in commit 0be67e04

    mentioned in commit 0be67e04

  • I. Nüske merged

    merged

  • I. Nüske mentioned in issue #138 (closed)

    mentioned in issue #138 (closed)

    • Please register or sign in to reply