[UnicodeEncodeError] Don't know what else to try

[Gilles Ganault]

Hello

Data that I download from the web seems to be using different code
pages at times, and Python doesn't like this.

Google returned a way to handle this, but I'm still getting an error:
========
print output.decode('utf-8')
File "C:\Python25\lib\encodings\utf_8.py", line 16, in decode
return codecs.utf_8_decode(input, errors, True)
UnicodeEncodeError: 'ascii' codec can't encode character u'\xe2' in
position 47:
ordinal not in range(128)
========

Here's the code:
========
try:
output = "Different: (%s) %s : %s # %s" %
(id[2],id[3],id[0],id[1])
print output.decode('utf-8')

except UnicodeDecodeError:

try:
output = "Different: (%s) %s : %s # %s" %
(id[2],id[3],id[0],id[1])
print output.decode('iso8859-15')

except UnicodeDecodeError:
output = "Different: (%s) %s : %s # %s" %
(id[2],id[3],id[0],id[1])
print output.decode('cp1252')
========

Am I doing it wrong? Is there something else I should try?

Thank you.

 

[Martin v. Löwis]

print output.decode('utf-8')

File "C:\Python25\lib\encodings\utf_8.py", line 16, in decode
return codecs.utf_8_decode(input, errors, True)
UnicodeEncodeError: 'ascii' codec can't encode character u'\xe2' in
position 47:
ordinal not in range(128)

Notice that it complains about the 'ascii' codec, when you were using
the utf-8 codec. Also, it complains about *en*coding, when you try
decoding.
For this, there could be two possible causes:

1. output is already a Unicode object - decoding it is not a meaningful
operation. So Python first *en*codes it with ascii, then would decode
it with UTF-8. The first step fails.
2. decoding from utf-8 works fine. It then tries to print output, which
requires an encoding. By default, it encodes as ascii, which fails.

========

Here's the code:
========
try:
output = "Different: (%s) %s : %s # %s" %
(id[2],id[3],id[0],id[1])

Add
print type(output)
here. If it says "unicode", reconsider the next line

print output.decode('utf-8')

Regards,
Martin

 

[Gilles Ganault]

Add
print type(output)
here. If it says "unicode", reconsider the next line

In case the string fetched from a web page turns out not to be Unicode
and Python isn't happy, what is the right way to handle this, know
what codepage is being used?

Thank you.

 

[Marc 'BlackJack' Rintsch]

In case the string fetched from a web page turns out not to be Unicode
and Python isn't happy, what is the right way to handle this, know what
codepage is being used?

How do you fetch the data? If you simply download it with `urllib` or
`urllib` you never get `unicode` but ordinary `str`\s. The you have to
figure out the encoding by looking at the headers from the server and/or
looking at the fetched data if it contains hints.

And when ``print``\ing you should explicitly *encode* the data again
because sooner or later you will come across a `stdout` where Python
can't determine what the process at the other end expects, for example if
output is redirected to a file.

Ciao,
Marc 'BlackJack' Rintsch

 

[Martin v. Löwis]

In case the string fetched from a web page turns out not to be Unicode
and Python isn't happy, what is the right way to handle this, know
what codepage is being used?

Can you first please report what happened when you add the print statement?

Thanks,
Martin

 

[Gilles Ganault]

Can you first please report what happened when you add the print statement?

Thanks guys, I found how to handle this:

===========
for id in rows:
#Says Unicode, but it's actually not
#print type(id[1])
#<type 'unicode'>

try:
print id[1];
except UnicodeEncodeError:
print "Not unicode"
try:
print id[1].encode('iso8859-15')
print "iso"
except UnicodeEncodeError:
print id[1].encode('cp1252')
print "Windows"
===========

Thank you.

 

[Marc 'BlackJack' Rintsch]

Can you first please report what happened when you add the print
statement?

Thanks guys, I found how to handle this:

===========
for id in rows:
#Says Unicode, but it's actually not
#print type(id[1])
#<type 'unicode'>

If it says `unicode` *it is* `unicode`.

try:
print id[1];
except UnicodeEncodeError:
print "Not unicode"

But it *is* `unicode` if `type()` says so!

Your code still fails when ``id[1]`` can't be encoded in `sys.encoding`,
'iso8859-15', or 'cp1252'. Even worse: The output may be even encoded in
different encodings this way. That's garbage you can't decode properly
with one encoding anymore.

A clean solution would be just one ``print`` with a call of `encode()`
and an explicit encoding. I'd use 'utf-8' as default but give the user
of the program a possibility to make another choice.

Ciao,
Marc 'BlackJack' Rintsch

 

[John Machin]

Thanks guys, I found how to handle this:

No you didn't.

===========
for id in rows:
        #Says Unicode, but it's actually not

If it's not unicode, what is it? What is your basis for that assertion
(which implies there is a bug in the version of Python that you are
using)? The probability that type() ever become so buggy that it
misreports whether a value is unicode or not, is extremely small.
Further the probability that you or I would be the first to notice the
problem is vanishingly small.

        #print type(id[1])
        #<type 'unicode'>

You didn't reply to Martin's question, instead you are tilting at a
windmill whose probability of existence lies between epsilon and zero.
When you ask for help, you should act on reasonable requests from your
helpers. Here's a reasonable request; insert some more verbose
debugging code:
print 'id[1] is', type(id[1]), repr(id[1])
AND tell us what you see, *before* you try to "fix" it.

For the future, please remember these:
(1) When you are worried about exactly what data some name refers to,
do
print 'name is', type(name), repr(name)
(2) Examine the more plausible sources of error first ... here's a
partial ordering: Ganault, ..., Gates, ..., Guido, God

HTH,
John