[Date Prev][Date Next][Thread Prev][Thread Next][Author Index][Date Index][Thread Index]
Re: Non-ASCII characters in Green
Hi Aaron. I've not looked at it specifically but wondered how it ought
to be supported. I believe your problem is that while you are fixing
the Python code to be utf-8, the server portion written in C isn't. Now
having said that, you -ought- to at least get it to store and return an
arbitrary byte sequence as the server doesn't trim high order bits. I
presume that utf-8 strings don't have an embedded zero byte that would
mess up C code.
I'd look at the portion of the Python code that transmits the (utf-8)
string over the TCP socket and insure that that translation is occurring
correctly. I'm ignorant of what happens when a Unicode string is passed
to a socket write call. You mention you changed String_write() but did
you change String_read() to examine the returned string and treat it as
Unicode as appropriate?
Re how to support it in the bigger scheme, the original Xanadoers
believed that it ought to be transparent to the backend and to indicate
whether the byte sequence in a particular document is 8 or 16 bits or
encoded in some manner, a link would be added by the front-end
indicating that. Of course all front-ends must then query for and
respect that link, but no such standardization has yet been done.
I wonder whether 16-bit chars ought to be done with a different resource
type (1 = bytes, 2 = links, 3 = words) so that it isn't even possible to
address the bytes out-of-phase as you could using a link-type. I
wouldn't use a different resource type for each encoding though, just
for each physical chunk size.
Aaron Bingham wrote:
Has anyone looked at supporting this? I looked at it briefly
yesterday and came across a couple of problems in Pyxi, but after
fixing these I was still getting incorrect data from the backend, so
there must be a deeper issue here.
In order to get Pyxi to handle non-ASCII chars, I had to change
x88.XuConn.write() to pass both regular and Unicode strings to
String_write(). I also had to change the default encoding to utf-8.
After doing this I was able to insert some German text, but when I
reloaded it, regular ASCII characters were substituted. For example,
a 223 LATIN SMALL LETTER SHARP S became 67 LATIN CAPITAL LETTER C.