ENH: Implement queries and entity retrieval

Summary

See https://gitlab.indiscale.com/caosdb/customers/lfpb/management/-/issues/386.

Focus

Point the reviewer to the core of the code change. Where should they start
reading? What should they focus on (e.g. security, performance,
maintainability, user-friendliness, compliance with the specs, finding more
corner cases, concrete questions)?

Test Environment

How to set up a test environment for manual testing?

Check List for the Author

Please, prepare your MR for a review. Be sure to write a summary and a focus and create gitlab comments for the reviewer. They should guide the reviewer through the changes, explain your changes and also point out open questions. For further good practices have a look at our review guidelines

• All automated tests pass
• Reference related Issues
• Up-to-date CHANGELOG.md
• Annotations in code (Gitlab comments)
  • Intent of new code
  • Problems with old code
  • Why this implementation?

Check List for the Reviewer

• I understand the intent of this MR
• All automated tests pass
• Up-to-date CHANGELOG.md
• The test environment setup works and the intended behavior is reproducible in the test environment
• In-code documentation and comments are up-to-date.
• Check: Are there spezifications? Are they satisfied?

For further good practices have a look at our review guidelines.

mentioned in merge request caosdb-juliainttest!3 (merged)

marked the checklist item Reference related Issues as completed

added 8 commits

• d7537e93 - MAINT: Add caosdb_client.json for tests
• c01a8864 - DOC: Add hint to config file
• 1cf223b6 - DRAFT: Move Transaction module to separate file
• 82d75b25 - DRAFT: Begin tests for transactions
• 3daa6ee5 - ENH: Implement create_transaction
• e40a8c77 - MAINT: Add caosdb config file
• 381fff84 - ENH: Implement retrievals and update docstrings
• a1ce9bdd - STY: whitespaces

Compare with previous version

Unfortunately, Documenter.jl doesn't support the Private argument in @index so we have to join everything into one index. I still like having one now that we have so many elements in our API documentation, though.

This is kind of neat; Julia automatically converts anything that looks like a Vector{String} to a C-array of char pointers.

changed this line in version 4 of the diff

added 1 commit

• 4773e5ac - DRAFT: Pipeline

Compare with previous version

added 6 commits

• 45dabd84 - DRAFT: Begin implementation of entities
• 5ef97a84 - ENH: Add Exception type for client exceptions
• 475a378f - FIX: Correct arguments for retrieve_by_ids
• 2b379aff - DRAFT: Begin implementation of entities
• f5bd5529 - ENH: Add test for entities
• 0be053a7 - DRAFT: Add a lot of setters, getters, and helper functions

Compare with previous version

added 2 commits

• 379a133b - ENH: Add convenience wrappers for Integer indices
• 20939d95 - ENH: Add size checking to getters

Compare with previous version

added 1 commit

• b3e4ab79 - MAINT: Move remaining modules to separate files

Compare with previous version

added 1 commit

• bdcaceb1 - ENH: Finish retrieve and query transactions

Compare with previous version

added 1 commit

• 1e651982 - DOC: Add hint on COUNT queries

Compare with previous version

Nothing changed, just moved to separate file.

Moved the remaining modules to separate files since CaosDB.jl became quite crowded.

The _deletable flag isn't actually needed on the Julia side, but we still have it in order to prevent any strange memory problems if the structs in Julia didn't exactly match the C structs.

Set it to false in the constructor to reflect the default value in the C structs.

Mostly just moved ...

This one is new. It's meant for CaosDB specific errors on the Julia side.

Get the correct error code for errors in the higher-order client.

Nothing new, just moved.

This is never created on its own, only as a part of a transaction. That's why we don't add a destructor (i.e., finalizer) here (there aren't even create_result_set and delete_result_set functions in ccaosdb).

Overload the creator s.th. users only have to type create_connection() and get one for the default connection which is probably what the users would want in most cases. They can also call this with the string name of the connection they want to use, or with the connection object itself if they already have created one.

This one is a little tricky on the C side. It is in principle owned by the connection object, but we release it so that it may continue existing after the connection goes out of scope. Afterwards we have to care for the deletion ourselves, hence managed_by_julia = true.

Again overloaded s.th. retrieval can be done either with a single id or a list of ids.

Julia syntax for "ids is a Vector (i.e., list) of things that look like strings."

Apparently, even though Julia identifies Cstring with Ptr{Cchar}, i.e., char *, we need the explicit Ptr{Ptr{Cchar}} (or, equivalently, Ptr{Ptr{UInt32}}) for char **.

Once it's implemented in the C++ client, we'll want to have execute_asynchrously here, too.

The only way to actually create a result set right now. Since it's owned by the transaction, we don't have to care for it's destruction explicitly.

This allows the users to omit the ResultSet alltogether in almost all cases which I expect to become the norm. get_results(transaction) returns the resulting entities directly without caring about a ResultSet.

Convenience functions further simplifying queries and retrievals: The users don't have to care about connections (though they may still specify them by name or connection object if needed), transactions, adding subrequests, and getting the results after execution; they simply type execute_query() or retrieve() and are returned the entities directly.

Overloaded again to allow to specify the connection by name or by connection Object.

Even more overloaded since it can be called with a single id (in which case, a single entity is returned) or with a list of ids, and the connection can be specified as above.

Nothing new, just moved.

This test will and should fail should we ever decide to change this code.

We're lacking the debugging means to create dummy connections, transactions, and result sets for unit testing, so everything that's related to the actual execution of the transactions and their results is part of the integration tests for now.

Test some setters and getters, also the appending and removal of parents and properties. Since the wrapped methods are already tested extensively in (c)caosdb, we're only testing some to verify the Julia code.

This one is quite long but also very repetitive (similar to the entity functions in ccaosdb). Even more so, since they are overloaded at many points to increase convenience (allow accessing members with Integers instead of Cints, getting all parents, properties, ... instead of getting them one-by-one, etc.).

It's kind of ugly but for users to be able to type

using Caosdb
entity = create_entity()

Instead of having to use CaosDB.Entity.create_entity() everywhere, this has to be exported by the Entity submodule first and then again by the CaosDB module itself.

Note that this is the Julia way of extending the global namespace with the exported members of used modules in contrast to imported modules (import CaosDB instead of using CaosDB). In the latter case, nothing is forwarded to the global namespace even when exported.

Even more convenience functions that set the return the entity with the correct role.

Have to make the distinction between real entities with role = "Property" and pure property objects.

Really stressing it in the docstring but I'm pretty sure that this will cause some confusion.

getters and setters have been overloaded when possible.

Yet another way of providing a char pointer to a ccall. If the string is written into by the ccall, this is the way to initialize it. The actual value (undef) and the actual length (256 was the one all the examples used, so I stuck with it) don't really matter, since memory and the new values are managed by the called c function anyway.

changed this line in version 20 of the diff

There may be pathological cases in which Julia's garbage collector kills out after deferencing it by pointer(out) but before it is converted into a string and returned. That's why we we prevent this with the @preserve macro.

changed this line in version 20 of the diff

We're interested in the actual value, not the reference to it, so it is deferenced by [].

Overloaded to allow get_error(entity, 1) instead of get_error(entity, Cint(1)).

I suppose Julia users would want to have the indices starting at 1.

The only case in which an explicit CaosDB exception is thrown since this is new to the C(++)-based clients in contrast to the Python clients. You don't create an entity, set its id, and retrieve it, but you just execute a retrieval transaction.

Allow to add a list of parents, too.

Re-introduce the usual convenience methods that Extern C is lacking.

added 1 commit

• 174f9d96 - DRAFT: Pipeline

Compare with previous version