# Crawler 2.0
The current CaosDB crawler has several limitations. The concept of 
identifiables is for example not able to incorporate conditions like 
referencing entities (only entities that are being referenced; other direction).
Another aspect is that crawler setup should be more easy. This should probably 
result in less code (since custom and possibly untested code is error prone).
Optimally, setup/configuration can be done using a visual tool or is (in part) automated. 

One approach  to these goals would be to:
1. generalize some aspects of the crawler (e.g. the identifiable)
2. use a more configuration based approach that requires as little programming
   as possible
   
The datastructures that we encountered in the past were inherently hierarchical:
- folder sturctures
- standardized containers, like HDF5 files
- ASCII "container" formats, like JSON files

The Crawler 2.0 should be able treat an arbitrary hierarchical structures and
convert them to interconnected Records that are consistent with a predefined
semantic data model.

The configuration must define:
- How the structure is created
  Example: Does the content of a file need to be considered and added to the tree?
- How  the structure and its contained data is mapped to the semantic data model:
  Example The Record "Experiment" will store the data from the folder name and the
  email address from a JSON file as CaosDB properties.


## Structure Mapping
In the following, it is described how the above can be done on an abstract level.

The hierarchical structure is assumed to be constituted of a tree of
StructureElements. The tree is created on the fly by so-called Converters which
are defined the configuration. The tree of StructureElements is a model
of the existing data.
Example: A tree of Python file objects (StructureElements) could represent a file tree
         that exists on some file server.

Converters treat StructureElements and thereby create the StructureElements that
are the children of the treated StructureElement.
Example: A StructureElement represents a folder and a Converter defines that for each file in the folder 
         another StructureElement is created.
Converters therefore create the above named tree. The definition of a Converter also contains what
Converters shall be used to treat the generated child-StructureElements. The definition is therefore a tree itself.

> Alex: The previous paragraph is difficult to understand. The reference "above named" is a little unclear.

> Side discussion
> Question: Should there be global Converters
> that are always checked when treating a StructureElement? Should Converters be
> associated with generated child-StructureElements? Currently, all children are
> created and checked against all Converters. It could be that one would like to
> check file-StructureElements against one set of Converters and
> directory-StructureElements against another)
>
> Alex' opinion: I would rather go for a macro/variable/template-based solution, so that the employment of a globally predefined
>                converter is explicitely mentioned instead of "silently and automatically" applied.

Each StructureElement in the tree has a set of data values, i.e a dictionary 
of key-value pairs.
Some of those values may be set due to the kind of StructureElement. For example,
a file could always have the file name as such a key value pair: 'filename': <sth>.
Converters may define additional functions that create further values. For
example, a regular expression could be used to get a date from a file name.

## Identifiables
The concept of an identifiable should be broadend to how an entity can be 
identified. Suggestion: Definition through a unique query
Example: "FIND RECORD Fish WITH FishNumber=A AND WHICH IS REFERENCED BY B"
Note that the second part can not be specified as condition with the old 
identifiable concept.
The query must return 1 or 0 entities. If no entity is returned the respective 
object may be created and if one is returned it is the one we were looking for.
If more than one is returned, then there is a mistake in the definition or in 
the data set. It is the responsibility of the designer of the Query for the identifiable
to make sure, that it returns either zero or one Entity.

## Entity Construction

In the simplest case an entity is constructed at a given node from its key-
value pairs. However, the data for a given entity might be distributed over different levels of
the tree.

Two different approaches are possible:
1. During the construction of an entity at a given node also key-value pairs
   from other nodes are used. For example, key-value pairs from parent nodes might
   be made accessible. Or key-value pairs might be accessed by providing the path
   to them in the tree. 
2. Information is added to an entity at other nodes. The simplest case uses the
   identifiable definition to add information. I.e. it is checked whether the 
   respective entity does already exist in the server, if not it is inserted and
   then the information is added. 
Additionally, it could be made possible to add information to entities that are
constructed in other nodes without the use of the identifiable. For example, 
it could be allowed to add information to entities that were created at parent 
nodes.

> Alex: I haven't really understood the variant at 2..
 
## Value computation
It is quite straight forward how to set a Property of a Record with a value 
that is contained in the hierarchical structure. However, the example with the
regular expression illustrates that the desired value might not be present.
For example, the desired value might be `firstname+" "+lastname`. Since the 
computation might not be trivial, it is likely that writing code for these 
computations might be necessary. Still, these would be tiny parts that probably
can easily be unit tested. There is also no immediated security risk since the 
configuration plus code replace the old scripts (i.e. only code). One could 
define small functions that are vigorously unit tested and the function names 
are used in the configuration.