XML Converter

Summary

Base functionality for matching and converting XML trees is implemented.

See: #145 (closed)

This should be reviewed and merged relatively soon. Follow-up issues will be created for the following tasks:

• user and API documentation (#159)
• design and implement more unit tests (#160)
• integration tests (#161)
• issue for creating attribute node converter and completing the text node converter (#158 (closed))

Focus

Have a look at #145 (closed) and check whether the requirements are met.

Test Environment

There are unit tests.

Check List for the Author

Please, prepare your MR for a review. Be sure to write a summary and a focus and create gitlab comments for the reviewer. They should guide the reviewer through the changes, explain your changes and also point out open questions. For further good practices have a look at our review guidelines

• All automated tests pass
• Reference related issues
• Up-to-date CHANGELOG.md (or not necessary)
• Up-to-date JSON schema (or not necessary)
• Appropriate user and developer documentation (or not necessary)
  • How do I use the software? Assume "stupid" users.
  • How do I develop or debug the software? Assume novice developers.
• Annotations in code (Gitlab comments)
  • Intent of new code
  • Problems with old code
  • Why this implementation?

Check List for the Reviewer

• I understand the intent of this MR
• All automated tests pass
• Up-to-date CHANGELOG.md (or not necessary)
• Appropriate user and developer documentation (or not necessary)
• The test environment setup works and the intended behavior is reproducible in the test environment
• In-code documentation and comments are up-to-date.
• Check: Are there specifications? Are they satisfied?

For further good practices have a look at our review guidelines.

src/caoscrawler/xml_converter.py

@@ -39,7 +39,7 @@ from .stores import GeneralStore, RecordStore
 from .structure_elements import (BooleanElement, DictElement, Directory, File,
                                  FloatElement, IntegerElement, JSONFile,
                                  ListElement, NoneElement, StructureElement,
-                                 TextElement, XMLTagElement, XMLTextNode)
+                                 TextElement, XMLTagElement, XMLTextNode, XMLAttributeNode)
 from .utils import has_parent
 
 import lxml.etree
@@ -93,6 +93,13 @@ class XMLTagConverter(Converter):
         - When attribs_as_children is set to true, attribute nodes will be generated from the attributes
           of the matched tags.
 
+        Namespaces
+        ----------
+        The default is to take the namespace map from the current node and use it in xpath queries.
+        Because default namespaces cannot be handled by xpath, it is possible to remap the default namespace
+        using the key "default_namespace".
+        The key "nsmap" can be used to define additional nsmap entries.
+
         """
         if not isinstance(element, XMLTagElement):
             raise TypeError("Element must be an instance of XMLTagElement.")
@@ -106,6 +113,11 @@ class XMLTagConverter(Converter):
             nsmap[default_namespace] = nsmap[None]
             del nsmap[None]
 
+        # Set additional nsmap entries from the converter definition:
+        if "nsmap" in self.definition:
+            for key, value in self.definition["nsmap"].items():
+                nsmap[key] = value
+
         xpath = self.definition.get("xpath", "child::*")
         children = element.tag.xpath(xpath, namespaces=nsmap)
         el_lst = []
@@ -113,7 +125,13 @@ class XMLTagConverter(Converter):
             if isinstance(el, str):
                 raise RuntimeError("Only standard xml nodes are supported as results of xpath queries.")
             elif isinstance(el, lxml.etree._Element):
-                el_lst.append(XMLTagElement(el))
+                if self.definition.get("tags_as_children", True):
+                    el_lst.append(XMLTagElement(el))
+                if self.definition.get("attribs_as_children", False):
+                    for attrib in el.attrib:
+                        el_lst.append(XMLAttributeNode(el, attrib))
+                if self.definition.get("text_as_children", False):
+                    el_lst.append(XMLTextNode(el))
             else:
                 raise RuntimeError("Unsupported child type.")
         return el_lst