XML Converter

Summary

Base functionality for matching and converting XML trees is implemented.

See: #145 (closed)

This should be reviewed and merged relatively soon. Follow-up issues will be created for the following tasks:

• user and API documentation (#159)
• design and implement more unit tests (#160)
• integration tests (#161)
• issue for creating attribute node converter and completing the text node converter (#158 (closed))

Focus

Have a look at #145 (closed) and check whether the requirements are met.

Test Environment

There are unit tests.

Check List for the Author

Please, prepare your MR for a review. Be sure to write a summary and a focus and create gitlab comments for the reviewer. They should guide the reviewer through the changes, explain your changes and also point out open questions. For further good practices have a look at our review guidelines

• All automated tests pass
• Reference related issues
• Up-to-date CHANGELOG.md (or not necessary)
• Up-to-date JSON schema (or not necessary)
• Appropriate user and developer documentation (or not necessary)
  • How do I use the software? Assume "stupid" users.
  • How do I develop or debug the software? Assume novice developers.
• Annotations in code (Gitlab comments)
  • Intent of new code
  • Problems with old code
  • Why this implementation?

Check List for the Reviewer

• I understand the intent of this MR
• All automated tests pass
• Up-to-date CHANGELOG.md (or not necessary)
• Appropriate user and developer documentation (or not necessary)
• The test environment setup works and the intended behavior is reproducible in the test environment
• In-code documentation and comments are up-to-date.
• Check: Are there specifications? Are they satisfied?

For further good practices have a look at our review guidelines.

unittests/test_xml_converter.py

@@ -54,7 +54,7 @@ from caoscrawler.structure_elements import (BooleanElement, DictElement,
                                             TextElement, XMLTagElement)
 from caoscrawler.xml_converter import XMLTagConverter
 
-import lxml
+from lxml.etree import fromstring
 
 UNITTESTDIR = Path(__file__).parent
 
@@ -91,7 +91,7 @@ match_tag: a
 match_attrib:  # default is the empty dictionary
     "(?P<ref>(href|url))": "test(?P<number>[0-9])"  # either the "href" or the "url" attribute must be set
     alt: (.+)  # this attribute must be present and contain at least one character
-match_text: .*  # allow any text, also empty (this is the default)
+match_text: \\s*(?P<node_text>.+)\\s*
 
 # _*_ marks the default:
 attribs_as_children: true  # true / _false_
@@ -106,6 +106,12 @@ subtree:
         type: TextElement
         match_name: alt
         match_value: ^(?P<text>.*)$
+
+    img:
+        type: XMLTag
+        match_name: img
+        match_attrib:
+            src: test2
 """)
     return xml_cfood
 
@@ -127,13 +133,72 @@ def test_simple_xml(basic_xmltag_converter):
     </a>
     """
 
-    tag = XMLTagElement("tag", lxml.etree.fromstring(xml_text))
+    tag = XMLTagElement("tag", fromstring(xml_text))
     m = basic_xmltag_converter.match(tag)
 
     assert m is not None
     assert m["ref"] == "href"
     assert m["number"] == "1"
+    assert m["node_text"] == "test "
 
 
 def test_not_matching(basic_xmltag_converter):
-    pass
+    m = basic_xmltag_converter.match(XMLTagElement("tag", fromstring("""
+        <a href="test1">
+        test <img src="test2"/>
+        </a>
+        """)))
+
+    assert m is None  # alt-attribute was missing
+
+    m = basic_xmltag_converter.match(XMLTagElement("tag", fromstring("""
+        <a href="test" alt="no link">
+        test <img src="test2"/>
+        </a>
+        """)))
+
+    assert m is None  # href attribute did not match
+
+    m = basic_xmltag_converter.match(XMLTagElement("tag", fromstring("""
+        <a href="test1" url="http" alt="no link">
+        test <img src="test2"/>
+        </a>
+        """)))
+
+    assert m is None  # href and url must not be present simultaneously
+
+    m = basic_xmltag_converter.match(XMLTagElement("tag", fromstring("""
+        <a href="test1" alt="no link"><img src="test2"/></a>
+        """)))
+
+    assert m is None  # text node is empty
+
+    m = basic_xmltag_converter.match(XMLTagElement("tag", fromstring("""
+        <a href="test1" alt="no link"/>
+        """)))
+
+    assert m is None  # text node is empty
+
+    # TODO: adapt converter -> empty (==None) text node is equivalent to empty string text node
+    # TODO: adapt tests
+    # TODO: how to match "  ajskdlfjaldsf ajsdklfjadkl " without the whitespaces in regexp correctly?
+
+
+
+def test_nested_simple_xml(basic_xmltag_converter):
+    """
+    Test for xml conversion including children.
+    """
+    xml_text = """
+    <a href="test1" alt="no link">
+    test <img src="test2"/>
+    </a>
+    """
+
+    tag = XMLTagElement("tag", fromstring(xml_text))
+    m = basic_xmltag_converter.match(tag)
+    assert m is not None
+
+    general_store = GeneralStore()
+
+    children = basic_xmltag_converter.create_children(general_store, tag)