Python Unicode to String conversion

[thijs.braem]

Hi everyone,

I'm having quite some troubles trying to convert Unicode to String
(for use in psycopg, which apparently doesn't know how to cope with
unicode strings).

The error I keep having is something like this:
ERREUR: Séquence d'octets invalide pour le codage «UTF8» : 0xe02063

(sorry, locale is french, it means "byte sequence invalid for encoding
<<utf8>>", the value is probably an e with one of the french accents)

I've found lots of stuff about this googling the error, but I don't
seem to be able to find a "works always"-function just to convert a
unicode variable back to string...

If someone could find me a solution, that'd really be a lifesaver.
I've been losing hours and hours over this one :s

thijs

 

[Larry Bates]

Hi everyone,

I'm having quite some troubles trying to convert Unicode to String
(for use in psycopg, which apparently doesn't know how to cope with
unicode strings).

The error I keep having is something like this:
ERREUR: Séquence d'octets invalide pour le codage «UTF8» : 0xe02063

(sorry, locale is french, it means "byte sequence invalid for encoding
<<utf8>>", the value is probably an e with one of the french accents)

I've found lots of stuff about this googling the error, but I don't
seem to be able to find a "works always"-function just to convert a
unicode variable back to string...

If someone could find me a solution, that'd really be a lifesaver.
I've been losing hours and hours over this one :s

thijs

Question is, what should it be converted to when there is no corresponding
ascii character to map the offending character to? Do you want to throw it
away? Your expectation "works always" is very ill-defined. Write a small
function to process the strings yourself and set up a substitution dictionary to
do the conversion yourself.

-Larry

 

[Chris Mellon]

Hi everyone,

I'm having quite some troubles trying to convert Unicode to String
(for use in psycopg, which apparently doesn't know how to cope with
unicode strings).

The error I keep having is something like this:
ERREUR: Séquence d'octets invalide pour le codage «UTF8» : 0xe02063

(sorry, locale is french, it means "byte sequence invalid for encoding
<<utf8>>", the value is probably an e with one of the french accents)

I've found lots of stuff about this googling the error, but I don't
seem to be able to find a "works always"-function just to convert a
unicode variable back to string...

encode().

You didn't post the code that was failing, I can encode that value
into UTF-8 (and unless I'm very much mistaken, you should be able to
encode any unicode string to UTF-8).

 

[John Machin]

Hi everyone,

I'm having quite some troubles trying to convert Unicode to String
(for use in psycopg, which apparently doesn't know how to cope with
unicode strings).

The error I keep having is something like this:
ERREUR: Séquence d'octets invalide pour le codage «UTF8» : 0xe02063

(sorry, locale is french, it means "byte sequence invalid for encoding
<<utf8>>",

I'm a pig-ignorant Anglo; it's news to me that Python error messages
varied by locale; I thought they always came out in ASCII as G(od|
uido) intended Does that message emanate from Python or psycopg?
In either case, it is saying that it is expecting a UTF8-encoded
string, but the string given to it is not a valid UTF8-encoded string.

the value is probably an e with one of the french accents)

PROBABLY?? (1) Please try to understand that computers are quite
deterministic. (2) If you want help, stop guessing and use something
like
print repr(the_value)
and tell us what it *actually* is. Also show us the *relevant* parts
of your code, so that we can see how your are trying to convert your
data, and how you are trying to pass it to psycopg. Also show us the
full traceback that you get.

I've found lots of stuff about this googling the error, but I don't
seem to be able to find a "works always"-function just to convert a
unicode variable back to string...

If someone could find me a solution, that'd really be a lifesaver.
I've been losing hours and hours over this one :s

1. Find out what your input data actually is (e.g. unicode)
2. Find out what form psycopg requires (e.g. utf8-encoded str).
3. unicode to utf8 is quite simple:
u'S\xe9quence'
# round trip works as expected

Here is what ASCII-Python says about malformed UTF8:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "C:\python25\lib\encodings\utf_8.py", line 16, in decode
return codecs.utf_8_decode(input, errors, True)
UnicodeDecodeError: 'utf8' codec can't decode bytes in position 0-2:
invalid data

Cheers,
John

 

[John Machin]

encode().

You didn't post the code that was failing, I can encode that value
into UTF-8

What is "that value"?
(1) unichr(0xe02063)? You must have a wide unicode build of Python ...
(2) u"\xe0\x20\x63"? Of course you can encode it; so what?

(and unless I'm very much mistaken, you should be able to
encode any unicode string to UTF-8).

That is true, by definition. However you are barking this truism up
the wrong tree. The unknown complainant's whinge is that it is
expecting a sequence of octets (an 8-bit string) that is valid UTF8,
but the actuality is something else. It is *NOT* trying to say that a
unicode input can't be converted to UTF8.

 

[Carsten Haese]

Hi everyone,

I'm having quite some troubles trying to convert Unicode to String
(for use in psycopg, which apparently doesn't know how to cope with
unicode strings).

The error I keep having is something like this:
ERREUR: Séquence d'octets invalide pour le codage «UTF8» : 0xe02063

I'm guessing that you are passing a latin-1 encoded string and pretend
(or psycopg assumes) incorrectly that it's UTF-8 encoded. In latin-1
encoding, 0xe0 is a small letter a with a grave accent, 0x20 is a space,
and 0x63 is a small letter c. While this is a perfectly valid latin-1
encoded character string, it doesn't represent a valid UTF-8 encoded
character string.

It seems that you need to pass a UTF-8 encoded string to the database.
To give you specific advice on how to do that, we'd have to see your
code. For now, I'll give you the generic advice of taking a look at
http://www.amk.ca/python/howto/unicode .

HTH,

 

[Lawrence D'Oliveiro]

In message <[email protected]>,

The error I keep having is something like this:
ERREUR: Séquence d'octets invalide pour le codage «UTF8» : 0xe02063

It would be useful to see some actual code snippet, traceback listing etc.

 

[iapain]

First make sure your DB encoding is UTF-8 not the latin1

The error I keep having is something like this:
ERREUR: Séquence d'octets invalide pour le codage «UTF8» : 0xe02063

then try this:

def smart_str(s, encoding='utf-8', errors='strict'):
"""
Returns a bytestring version of 's', encoded as specified in
'encoding'.
"""
if not isinstance(s, basestring):
try:
return str(s)
except UnicodeEncodeError:
return unicode(s).encode(encoding, errors)
elif isinstance(s, unicode):
return s.encode(encoding, errors)
elif s and encoding != 'utf-8':
return s.decode('utf-8', errors).encode(encoding, errors)
else:
return s

 

[thijs.braem]

Sorry for answering so late. Thanks a million! This code snippet
helped me solve the problem.

I think I will be using SQLAlchemy for these sorts of things from now
on though, it seems to be taking care of these things itself, on top
of being one hell of a handy ORM of course

thijs

 

[Richard Levasseur]

It took me days to figure out what was going on when dealing with
unicode, ascii, latin1, utf8, decodeerrors, etc, so I'm just chiming
in to echo something similar iapain's comments:

When dealing with unicode, i've run into situations where I have
multiple encodings in the same string, usually latin1 and utf8
(latin1 != ascii, and latin1 != utf8, and they don't play nice
together). So, for future readers, if you have problems dealing with
unicode encode and decode, try using a mix of latin1 and utf8
encodings to figure out whats going on, and what characters are
fubar'ing the process.

 

[Gabriel Genellina]

En Mon, 17 Sep 2007 01:33:14 -0300, Richard Levasseur

When dealing with unicode, i've run into situations where I have
multiple encodings in the same string, usually latin1 and utf8
(latin1 != ascii, and latin1 != utf8, and they don't play nice
together). So, for future readers, if you have problems dealing with
unicode encode and decode, try using a mix of latin1 and utf8
encodings to figure out whats going on, and what characters are
fubar'ing the process.

Life is easier if you follow these guidelines:
- work internally always in Unicode (not strings)
- All input data (read from files, coming from an Internet connection,
typed by user...) should be decoded from byte strings into unicode as
early as possible. (You should know which encoding your data comes in, in
each case)
- All output data (written to files, printing to screen, etc) is encoded
from unicode into byte strings as late as possible.

This way, unless your input data is garbage, you never could mix strings
from different encodings.
For further information, read the Unicode Howto
<http://www.amk.ca/python/howto/unicode> and this excerpt form the "Python
Cookbook", by Alex Martelli
<http://www.onlamp.com/pub/a/python/excerpt/pythonckbk_chap1/index.html>