Skip to content
GitLab
Explore
Sign in
Register
Primary navigation
Search or go to…
Project
C
caosdb-advanced-user-tools
Manage
Activity
Members
Labels
Plan
Issues
Issue boards
Milestones
Iterations
Wiki
Code
Merge requests
Repository
Branches
Commits
Tags
Repository graph
Compare revisions
Snippets
Locked files
Build
Pipelines
Jobs
Pipeline schedules
Artifacts
Deploy
Releases
Package registry
Container registry
Model registry
Operate
Environments
Terraform modules
Monitor
Incidents
Analyze
Value stream analytics
Contributor analytics
CI/CD analytics
Repository analytics
Code review analytics
Issue analytics
Model experiments
Help
Help
Support
GitLab documentation
Compare GitLab plans
Community forum
Contribute to GitLab
Provide feedback
Keyboard shortcuts
?
Snippets
Groups
Projects
Show more breadcrumbs
caosdb
Software
caosdb-advanced-user-tools
Commits
eaa1e276
Commit
eaa1e276
authored
2 years ago
by
Florian Spreckelsen
Browse files
Options
Downloads
Patches
Plain Diff
STY: docstrings and changelog
parent
a61a52ab
No related branches found
No related tags found
2 merge requests
!59
FIX: if multiple updates for one entity exist, the retrieve would result in an...
,
!55
ENH: include caosdbignore
Changes
2
Hide whitespace changes
Inline
Side-by-side
Showing
2 changed files
CHANGELOG.md
+4
-4
4 additions, 4 deletions
CHANGELOG.md
src/caosadvancedtools/loadFiles.py
+20
-11
20 additions, 11 deletions
src/caosadvancedtools/loadFiles.py
with
24 additions
and
15 deletions
CHANGELOG.md
+
4
−
4
View file @
eaa1e276
...
...
@@ -7,13 +7,13 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
## [Unreleased] ##
### Added ###
-
caosdbignore; You can add one or more
`.caosdbignore`
files to the directory
structure that you want to make available in CaosDB and the run loadFiles.
The syntax is that of
`.gitignore`
files. For more information see
`loadFiles`
section of the Crawler in the documentation.
-
Unauthorized inserts can now be cached. Note that the Crawler cannot postpone
inserts but the Cache has the functionality now.
-
caosdbignore; You can add one or more
`.caosdbignore`
files to the directory
structure that you want to make available in CaosDB and the run loadFiles.
The syntax is that of
`.gitignore`
files. For more information see
`loadFiles`
section of the Crawler in the documentation.
### Changed ###
...
...
This diff is collapsed.
Click to expand it.
src/caosadvancedtools/loadFiles.py
+
20
−
11
View file @
eaa1e276
...
...
@@ -51,10 +51,13 @@ def convert_size(size):
def
combine_ignore_files
(
caosdbignore
,
localignore
,
dirname
=
None
):
"""
appends the contents of localignore to caosdbignore
and saves the result and returns the name
"""
appends the contents of localignore to caosdbignore and saves the result
and returns the name
"""
tmp
=
NamedTemporaryFile
(
delete
=
False
,
mode
=
"
w
"
,
dir
=
dirname
,
prefix
=
"
.caosdbignore
"
)
tmp
=
NamedTemporaryFile
(
delete
=
False
,
mode
=
"
w
"
,
dir
=
dirname
,
prefix
=
"
.caosdbignore
"
)
with
open
(
caosdbignore
,
"
r
"
)
as
base
:
tmp
.
write
(
base
.
read
())
with
open
(
localignore
,
"
r
"
)
as
local
:
...
...
@@ -64,9 +67,11 @@ def combine_ignore_files(caosdbignore, localignore, dirname=None):
def
compile_file_list
(
caosdbignore
,
localpath
):
"""
creates a list of files
that contain all files under localpath except those excluded by caosdbignore
"""
creates a list of files that contain all files under localpath except
those excluded by caosdbignore
"""
from
gitignore_parser
import
parse_gitignore
matches
=
parse_gitignore
(
caosdbignore
)
current_ignore
=
caosdbignore
...
...
@@ -89,7 +94,8 @@ def compile_file_list(caosdbignore, localpath):
# create a new local ignore file
if
"
.caosdbignore
"
in
files
:
current_ignore
=
combine_ignore_files
(
current_ignore
,
os
.
path
.
join
(
root
,
"
.caosdbignore
"
),
os
.
path
.
join
(
root
,
"
.caosdbignore
"
),
# due to the logic of gitignore_parser the file
# has to be written to this folder
dirname
=
root
)
...
...
@@ -105,9 +111,10 @@ def compile_file_list(caosdbignore, localpath):
def
create_re_for_file_list
(
files
,
localroot
,
remoteroot
):
"""
creates a regular expression
that matches file paths contained in the files argument and all parent directories.
The prefix localroot is replaced by the prefix remoteroot.
"""
creates a regular expression that matches file paths contained in the
files argument and all parent directories. The prefix localroot is replaced
by the prefix remoteroot.
"""
regexp
=
""
for
fi
in
files
:
...
...
@@ -232,13 +239,15 @@ exclude is given preference over include.
args
=
parser
.
parse_args
()
if
args
.
caosdbignore
and
(
args
.
exclude
or
args
.
include
):
raise
ValueError
(
"
Do not use a caosdbignore file and in- or exclude simultaneously!
"
)
raise
ValueError
(
"
Do not use a caosdbignore file and in- or exclude simultaneously!
"
)
if
args
.
caosdbignore
and
not
args
.
localpath
:
raise
ValueError
(
"
To use caosdbignore you must supply a local path!
"
)
if
args
.
localpath
and
(
args
.
exclude
or
args
.
include
):
raise
ValueError
(
"
Do not use a localpath and in- or exclude simultaneously!
"
)
raise
ValueError
(
"
Do not use a localpath and in- or exclude simultaneously!
"
)
con
=
db
.
get_connection
()
con
.
timeout
=
float
(
args
.
timeout
)
...
...
This diff is collapsed.
Click to expand it.
Preview
0%
Loading
Try again
or
attach a new file
.
Cancel
You are about to add
0
people
to the discussion. Proceed with caution.
Finish editing this message first!
Save comment
Cancel
Please
register
or
sign in
to comment