Skip to content
GitLab
Explore
Sign in
Register
Primary navigation
Search or go to…
Project
CaosDB Crawler
Manage
Activity
Members
Labels
Plan
Issues
Issue boards
Milestones
Iterations
Wiki
Code
Merge requests
Repository
Branches
Commits
Tags
Repository graph
Compare revisions
Snippets
Locked files
Build
Pipelines
Jobs
Pipeline schedules
Artifacts
Deploy
Releases
Package registry
Container registry
Model registry
Operate
Environments
Terraform modules
Monitor
Incidents
Analyze
Value stream analytics
Contributor analytics
CI/CD analytics
Repository analytics
Code review analytics
Issue analytics
Model experiments
Help
Help
Support
GitLab documentation
Compare GitLab plans
GitLab community forum
Contribute to GitLab
Provide feedback
Keyboard shortcuts
?
Snippets
Groups
Projects
Show more breadcrumbs
caosdb
Software
CaosDB Crawler
Commits
5cd4075c
Commit
5cd4075c
authored
Nov 18, 2021
by
Henrik tom Wörden
Browse files
Options
Downloads
Patches
Plain Diff
MAINT: move code parts
parent
e9f1a33f
No related branches found
No related tags found
1 merge request
!53
Release 0.1
Changes
2
Show whitespace changes
Inline
Side-by-side
Showing
2 changed files
src/newcrawler/crawl.py
+24
-28
24 additions, 28 deletions
src/newcrawler/crawl.py
tests/test_functions.py
+1
-2
1 addition, 2 deletions
tests/test_functions.py
with
25 additions
and
30 deletions
src/newcrawler/crawl.py
+
24
−
28
View file @
5cd4075c
...
...
@@ -101,6 +101,9 @@ class Store(object):
self
.
storage
=
dict
()
def
__getitem__
(
self
,
key
):
if
key
not
in
self
.
storage
:
self
.
storage
[
key
]
=
db
.
Record
()
return
self
.
storage
[
key
]
def
update
(
self
,
other
):
...
...
@@ -256,11 +259,30 @@ class Converter(object):
element
:
StructureElement
):
pass
@abstractmethod
def
create_records
(
self
,
values
:
GeneralStore
,
records
:
RecordStore
,
element
:
StructureElement
):
pass
if
"
records
"
not
in
self
.
definition
:
return
for
name
,
record
in
self
.
definition
[
"
records
"
].
items
():
c_record
=
records
.
storage
[
name
]
if
"
parents
"
in
record
:
for
parent
in
record
[
"
parents
"
]:
c_record
.
add_parent
(
name
=
parent
)
else
:
c_record
.
add_parent
(
name
=
name
)
for
key
,
value
in
record
.
items
():
if
key
==
"
parents
"
:
continue
propvalue
=
value
for
varname
,
varvalue
in
values
.
storage
.
items
():
propvalue
=
propvalue
.
replace
(
"
$
"
+
varname
,
varvalue
)
c_record
.
add_property
(
name
=
key
,
value
=
propvalue
)
@abstractmethod
def
typecheck
(
self
,
element
:
StructureElement
):
...
...
@@ -295,32 +317,6 @@ class DirectoryConverter(Converter):
return
self
.
create_children_from_directory
(
element
)
def
create_records
(
self
,
values
:
GeneralStore
,
records
:
RecordStore
,
element
:
StructureElement
):
if
"
records
"
not
in
self
.
definition
:
return
for
name
,
record
in
self
.
definition
[
"
records
"
].
items
():
if
name
not
in
records
.
storage
:
c_record
=
db
.
Record
()
records
.
storage
[
name
]
=
c_record
c_record
=
records
.
storage
[
name
]
if
"
parents
"
in
record
:
for
parent
in
record
[
"
parents
"
]:
c_record
.
add_parent
(
name
=
parent
)
else
:
c_record
.
add_parent
(
name
=
name
)
for
key
,
value
in
record
.
items
():
if
key
==
"
parents
"
:
continue
propvalue
=
value
for
varname
,
varvalue
in
values
.
storage
.
items
():
propvalue
=
propvalue
.
replace
(
"
$
"
+
varname
,
varvalue
)
c_record
.
add_property
(
name
=
key
,
value
=
propvalue
)
def
typecheck
(
self
,
element
:
StructureElement
):
return
isinstance
(
element
,
Directory
)
...
...
This diff is collapsed.
Click to expand it.
tests/test_functions.py
+
1
−
2
View file @
5cd4075c
...
...
@@ -2,5 +2,4 @@
# Tests for main functions of crawler
# A. Schlemmer, 07/2021
from
newcrawler
import
match_complete
#from newcrawler import match_complete
This diff is collapsed.
Click to expand it.
Preview
0%
Loading
Try again
or
attach a new file
.
Cancel
You are about to add
0
people
to the discussion. Proceed with caution.
Finish editing this message first!
Save comment
Cancel
Please
register
or
sign in
to comment