Skip to content
Snippets Groups Projects
Commit 65008741 authored by Henrik tom Wörden's avatar Henrik tom Wörden
Browse files

DOC: include workflow documentation

parent b523af38
No related branches found
No related tags found
2 merge requests!222Release 0.12.0,!220F doc workflow
......@@ -9,6 +9,7 @@ CaosDB-Crawler Documentation
Getting started<getting_started/index>
Tutorials<tutorials/index>
Workflow<workflow>
Concepts<concepts>
Converters<converters/index>
CFoods (Crawler Definitions)<cfood>
......
Crawler Workflow
================
The CaosDB crawler aims to provide a very flexible framework for synchronizing
The LinkAhead crawler aims to provide a very flexible framework for synchronizing
data on file systems (or potentially other sources of information) with a
running CaosDB instance. The workflow that is used in the scientific environment
running LinkAhead instance. The workflow that is used in the scientific environment
should be choosen according to the users needs. It is also possible to combine multiple workflow or use them in parallel.
In this document we will describe several workflows for crawler operation.
......@@ -14,13 +14,13 @@ Local Crawler Operation
A very simple setup that can also reliably used for testing (e.g. in local
docker containers) sets up the crawler on a local computer. The files that
are being crawled need to be visible to both, the local computer and the
machine, running the CaosDB.
machine, running the LinkAhead.
Prerequisites
+++++++++++++
- Make sure that CaosDB is running, that your computer has a network connection to CaosDB and
that your pycaosdb.ini is pointing to the correct instance of CaosDB. Please refer to the
- Make sure that LinkAhead is running, that your computer has a network connection to LinkAhead and
that your pycaosdb.ini is pointing to the correct instance of LinkAhead. Please refer to the
pylib manual for questions related to the configuration in pycaosdb.ini
(https://docs.indiscale.com/caosdb-pylib/README_SETUP.html).
- Make sure that caosdb-crawler and caosdb-advanced-user-tools are installed (e.g. using pip).
......@@ -46,15 +46,15 @@ The generic syntax is:
python3 -m caosadvancedtools.loadFiles -p <prefix-in-caosdb-file-system> <path-to-crawled-folder>
Important: The <path-to-crawled-folder> is the location of the files **as seen by CaosDB**, e.g. for a CaosDB instance running in a docker container (e.g. see: https://gitlab.com/caosdb/caosdb-docker) the command line could look like:
Important: The <path-to-crawled-folder> is the location of the files **as seen by LinkAhead**, e.g. for a LinkAhead instance running in a docker container (e.g. see: https://gitlab.com/caosdb/caosdb-docker) the command line could look like:
python3 -m caosadvancedtools.loadFiles -p / /opt/caosdb/mnt/extroot/ExperimentalData
This command line would load the folder "ExperimentalData" contained in the extroot folder within the docker container to the CaosDB-prefix "/" which is the root prefix.
This command line would load the folder "ExperimentalData" contained in the extroot folder within the docker container to the LinkAhead-prefix "/" which is the root prefix.
Running the crawler
)))))))))))))))))))
The following command line assumes that the extroot folder visible in the CaosDB docker container is located in "../extroot":
The following command line assumes that the extroot folder visible in the LinkAhead docker container is located in "../extroot":
caosdb-crawler -i identifiables.yml --prefix /extroot --debug --provenance=provenance.yml -s update cfood.yml ../extroot/ExperimentalData/
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment