MAINT: enhance output when table cannot be parsed

Summary

Previously, when there was a problem with a table that shall be parsed by the crawler (e.g. wrong datatype, bad format of a date and so on) there was simply the statement: 'Cannot parse FILENAME'

Now the message that is contained in the ValueError is actually printed (not only in debugging logs).

Also, the check_datatypes method was missing an appropriate call to logging.warning.

Test Environment

Load a table using the webinterface where the table causes a ValueError in one of the converter functions.

Check List for the Author

Please, prepare your MR for a review. Be sure to write a summary and a focus and create gitlab comments for the reviewer. They should guide the reviewer through the changes, explain your changes and also point out open questions. For further good practices have a look at our review guidelines

• All automated tests pass
• Reference related Issues
• Up-to-date CHANGELOG.md
• Annotations in code (Gitlab comments)
  • Intent of new code
  • Problems with old code
  • Why this implementation?

Check List for the Reviewer

• I understand the intent of this MR
• All automated tests pass
• Up-to-date CHANGELOG.md
• The test environment setup works and the intended behavior is reproducible in the test environment
• In-code documentation and comments are up-to-date.
• Check: Are there spezifications? Are they satisfied?

For further good practices have a look at our review guidelines.

src/caosadvancedtools/table_importer.py

@@ -51,7 +51,7 @@ def assure_name_format(name):
     name = str(name)
 
     if len(name.split(",")) != 2:
-        raise ValueError("Name field should be 'LastName, FirstName'."
+        raise ValueError("The field value should be 'LastName, FirstName'. "
                          "The supplied value was '{}'.".format(name))
 
     return name
@@ -303,14 +303,20 @@ class TableImporter(object):
         """
 
         for key, datatype in self.datatypes.items():
-            for idx, val in df.loc[pd.notnull(df.loc[:, key]), key].iteritems():
+            for idx, val in df.loc[
+                    pd.notnull(df.loc[:, key]), key].iteritems():
+
                 if not isinstance(val, datatype):
-                    raise DataInconsistencyError(
-                        "In row no. {rn} and column {c} of file '{fi}' the "
+                    msg = (
+                        "In row no. {rn} and column '{c}' of file '{fi}' the "
                         "datatype was {was} but it should be "
                         "{expected}".format(rn=idx, c=key, fi=filename,
-                                            was=type(val), expected=datatype)
+                                            was=str(type(val)).strip("<>"),
+                                            expected=str(datatype).strip("<>"))
                     )
+                    logger.warning(msg, extra={'identifier': filename,
+                                               'category': "inconsistency"})
+                    raise DataInconsistencyError(msg)
 
     def check_missing(self, df, filename=None):
         """
@@ -394,7 +400,7 @@ class XLSImporter(TableImporter):
             df = xls_file.parse(converters=self.converters, **kwargs)
         except Exception as e:
             logger.warning(
-                "Cannot parse {}.".format(filename),
+                "Cannot parse {}.\n{}".format(filename, e),
                 extra={'identifier': str(filename),
                        'category': "inconsistency"})
             raise DataInconsistencyError(*e.args)
@@ -411,7 +417,7 @@ class CSVImporter(TableImporter):
                              **kwargs)
         except ValueError as ve:
             logger.warning(
-                "Cannot parse {}.".format(filename),
+                "Cannot parse {}.\n{}".format(filename, ve),
                 extra={'identifier': str(filename),
                        'category': "inconsistency"})
             raise DataInconsistencyError(*ve.args)
@@ -428,7 +434,7 @@ class TSVImporter(TableImporter):
                              **kwargs)
         except ValueError as ve:
             logger.warning(
-                "Cannot parse {}.".format(filename),
+                "Cannot parse {}.\n{}".format(filename, ve),
                 extra={'identifier': str(filename),
                        'category': "inconsistency"})
             raise DataInconsistencyError(*ve.args)