Pylib bug: Container with files and non-ascii characters

changed iteration to Sprint Nov 11, 2024 - Nov 24, 2024

added Statusdoing label and removed Statustodo label

Further reduced code snippet for reproducing the bug:

file1 = db.File(file="/dev/null", path="/1Ä Ö Ü ÖR ¢")
file1.insert()

Cursory testing suggests that any insertion of a file with at least five non-ASCII characters anywhere (names, parents, properties, path) will trigger this.
The characters do not have to be part of the file entity, this code:

file1 = db.File(file="/dev/null", path="/1")
rec1 = db.Record(name="Ä Ö Ü ÖR ¢")
db.Container().extend([file1, rec1]).insert()

does also trigger the error.
The file does seem to be necessary, as are the five non-ASCII characters, four and less non-ASCII characters works.

changed the description

I did not find a way to reproduce the error over the webinterface or with a record, which makes it seem likely that the bug is caused somewhere in the encoding of a multipart request in the pylib.

However, using the two calls db.File(file="/dev/null", path="/1Ä Ö Ü ÖR ¢").insert() and db.File(file="/dev/null", path="/1Ä Ö Ü OR ¢").insert() whose only difference is O vs. Ö in the path, the requests seem equivalent down to urllib3/connection.py, which then calls the python standard lib http, and I think any bug there would have been noticed at some point before now. A quick search found no corresponding results, so either I missed something (which is very possible, not an expert on http requests) or the problem might not be in the request sent by the pylib after all.

In my opinion the approach most likely to yield results would be to check whether the requests are equivalent when they reach the server and then go from there, but this is currently outside the allotted timeframe.

added Statustodo label and removed Statusdoing label

added (C label

set weight to 4

added Statusdoing label and removed Statustodo label

One string representation of the chunks generated with the first call above in urllib3 is:

ChunksAndContentLength(chunks=(b'username=admin&password=caosdb',), content_length=30):
b'username=admin&password=caosdb'

ChunksAndContentLength(chunks=<generator object body_to_chunks.<locals>.chunk_readable at 0x7fef4d915380>, content_length=None):
b'--0b6e013f3f864b22a79ab723573bd321\r\nContent-Disposition: form-data; name="FileRepresentation"\r\nContent-Type: text/plain; charset=utf-8\r\n\r\n<Insert>\n  <File id="-1" cuid="-1--0d5619ac-63d4-417c-a15f-e32edf27f6f2" path="/1\xc3\x84 \xc3\x96 \xc3\x9c \xc3\x96R \xc2\xa2" checksum="cf83e1357eefb8bdf1542850d66d8007d620e4050b5715dc83f4a921d36ce9ce47d0d13c5d85f2b0ff8318d2877eec2f63b931bd47417a81a538327af927da3e" upload="0x4d47d61ec9cea472"/>\n</Insert>\n\r\n--0b6e013f3f864b22a79ab723573bd321\r\nContent-Disposition: form-data; name="0x36e29629b4ac346f"; filename="0x4d47d61ec9cea472"\r\nContent-Type: text/plain; charset=utf-8\r\n\r\n\r\n--0b6e013f3f864b22a79ab723573bd321--\r\n'

with the only differences between both calls being different hex codes (naturally) and different paths: path="/1\xc3\x84 \xc3\x96 \xc3\x9c \xc3\x96R \xc2\xa2" vs path="/1\xc3\x84 \xc3\x96 \xc3\x9c OR \xc2\xa2", as would be expected.

However, while one of them successfully terminates when called from the pythonlib and the other causes an error, both are successfully uploaded when calling the curl command
curl -b cookie.txt -F "FileRepresentation=<file.xml" -F "testfile.bla=@testfile.bla" "<SERVER>/Entity" --insecure
with file.xml containing either

<Insert>
  <File id="-1" cuid="-1--d81ed08c-06cb-4246-82e7-4c98e3732310" path="/1\xc3\x84 \xc3\x96 \xc3\x9c \xc3\x9c \xc2\xa2" upload="testfile.bla"/>
</Insert>

or

<Insert>
  <File path="/1&#196; &#214; &#220; &#214;R &#162;" upload="testfile.bla"/>
</Insert>

which are the closest xml versions to the xml from the chunk and to_xml() respectively that do not trigger any other errors. Additionally, while there are other server errors that potentially may have masked this one, any xml I tested that lead to a successful upload via curl will still upload successfully when adding any number of non-ASCII chars.

This makes it rather likely to be a problem with the pylib calls again. My next approach would be to check whether there is any reasonably quick way of intercepting the final calls from python and curl themselves, probably via proxy, and comparing them. However, as this will take up more time than is currently available, further investigation of the bug has been deferred.