Skip to content
GitLab
Explore
Sign in
Register
Primary navigation
Search or go to…
Project
CaosDB Crawler
Manage
Activity
Members
Labels
Plan
Issues
Issue boards
Milestones
Iterations
Wiki
Code
Merge requests
Repository
Branches
Commits
Tags
Repository graph
Compare revisions
Snippets
Locked files
Build
Pipelines
Jobs
Pipeline schedules
Artifacts
Deploy
Releases
Package registry
Container registry
Model registry
Operate
Environments
Terraform modules
Monitor
Incidents
Analyze
Value stream analytics
Contributor analytics
CI/CD analytics
Repository analytics
Code review analytics
Issue analytics
Model experiments
Help
Help
Support
GitLab documentation
Compare GitLab plans
Community forum
Contribute to GitLab
Provide feedback
Keyboard shortcuts
?
Snippets
Groups
Projects
Show more breadcrumbs
caosdb
Software
CaosDB Crawler
Commits
3d49c985
Commit
3d49c985
authored
3 years ago
by
Alexander Schlemmer
Browse files
Options
Downloads
Patches
Plain Diff
ENH: added some functions from the old crawler framework
parent
f4b751e2
No related branches found
No related tags found
1 merge request
!53
Release 0.1
Changes
2
Hide whitespace changes
Inline
Side-by-side
Showing
2 changed files
src/newcrawler/identifiable_adapters.py
+35
-0
35 additions, 0 deletions
src/newcrawler/identifiable_adapters.py
src/newcrawler/utils.py
+24
-0
24 additions, 0 deletions
src/newcrawler/utils.py
with
59 additions
and
0 deletions
src/newcrawler/identifiable_adapters.py
+
35
−
0
View file @
3d49c985
...
...
@@ -25,6 +25,7 @@
import
caosdb
as
db
from
abc
import
abstractmethod
from
utils
import
get_value
class
IdentifiableAdapter
(
object
):
"""
...
...
@@ -43,6 +44,40 @@ class IdentifiableAdapter(object):
from the database.
"""
@staticmethod
def
create_query_for_identifiable
(
ident
:
db
.
Record
):
"""
This function is taken from the old crawler:
caosdb-advanced-user-tools/src/caosadvancedtools/crawler.py
uses the properties of ident to create a query that can determine
whether the required record already exists.
"""
# TODO multiple parents are ignored! Sufficient?
if
len
(
ident
.
get_parents
())
==
0
:
raise
ValueError
(
"
The identifiable must have at least one parent.
"
)
query_string
=
"
FIND Record
"
+
ident
.
get_parents
()[
0
].
name
query_string
+=
"
WITH
"
if
ident
.
name
is
None
and
len
(
ident
.
get_properties
())
==
0
:
raise
ValueError
(
"
The identifiable must have features to identify it.
"
)
if
ident
.
name
is
not
None
:
query_string
+=
"
name=
'
{}
'
AND
"
.
format
(
ident
.
name
)
for
p
in
ident
.
get_properties
():
if
p
.
datatype
is
not
None
and
p
.
datatype
.
startswith
(
"
LIST<
"
):
for
v
in
p
.
value
:
query_string
+=
(
"
references
"
+
str
(
v
.
id
if
isinstance
(
v
,
db
.
Entity
)
else
v
)
+
"
AND
"
)
else
:
query_string
+=
(
"'"
+
p
.
name
+
"'
=
'"
+
str
(
get_value
(
p
))
+
"'
AND
"
)
# remove the last AND
return
query_string
[:
-
4
]
@abstractmethod
def
get_registered_identifiable
(
self
,
record
:
db
.
Record
):
"""
...
...
This diff is collapsed.
Click to expand it.
src/newcrawler/utils.py
+
24
−
0
View file @
3d49c985
...
...
@@ -24,6 +24,7 @@
#
import
caosdb
as
db
from
datetime
import
datetime
# Some utility functions, e.g. for extending pylib.
...
...
@@ -38,3 +39,26 @@ def has_parent(entity: db.Entity, name: str):
if
parent
.
name
==
name
:
return
True
return
False
def
get_value
(
prop
):
"""
Returns the value of a Property
This function is taken from the old crawler:
caosdb-advanced-user-tools/src/caosadvancedtools/crawler.py
Parameters
----------
prop : The property of which the value shall be returned.
Returns
-------
out : The value of the property; if the value is an entity, its ID.
"""
if
isinstance
(
prop
.
value
,
db
.
Entity
):
return
prop
.
value
.
id
elif
isinstance
(
prop
.
value
,
datetime
):
return
prop
.
value
.
isoformat
()
else
:
return
prop
.
value
This diff is collapsed.
Click to expand it.
Preview
0%
Loading
Try again
or
attach a new file
.
Cancel
You are about to add
0
people
to the discussion. Proceed with caution.
Finish editing this message first!
Save comment
Cancel
Please
register
or
sign in
to comment