Encodign issue in Python 3.3.1 (once again)

[Michael Torrie]

I have checked the database through phpMyAdmin and it is indeed UTF-8.

I have no idea why python 3.3.1 chooses to work with latin-iso only....

It's not python that is doing this here...

If you look at the source code to pymysql, I'm sure you will identify
the problem. In fact I'll even walk you through things.

The traceback you posted tells you what's going on and why, with
superfluous details removed for clarity:

File "/opt/python3/lib/python3.3/site-packages/pymysql/cursors.py", line
108, in execute:
query = query.encode(charset)
UnicodeEncodeError: 'latin-1' codec can't encode characters in position
46-52: ordinal not in range(256)

So there we have it. pymysql is actually explicitly calling .encode()
on a string and is using whatever character set is specified by the
local variable charset. If you look in cursors.py at that line, and
then a few lines above, you will find that charset is assigned to
conn.charset. This means that the charset is actually defined in the
connection object. So go look at connections.py. Sure enough, it shows
that charset is defined by default as "latin-1." That's no good for
you. So take a look at the __init__ method in connections.py. In there
you should find the necessary keyword argument you need to use when
creating the mysql connection to make sure the charset is utf-8.

 

[Íßêïò Ãêñ33ê]

I do not know here to find connections.py Michael.

But i do not understand since iam suing the following 2 statements, why a unicode error remains.

#needed line, script does *not* work without it
sys.stdout = os.fdopen(1, 'w', encoding='utf-8')

# connect to database
con = pymysql.connect( db = 'pelatologio', host = 'localhost', user = 'myself', passwd = 'mypass', init_command='SET NAMES UTF8' )
cur = con.cursor()

Shall i chnage the connector form 'pymysql' => 'MySQLdb' ?

 

[MRAB]

I do not know here to find connections.py Michael.

But i do not understand since iam suing the following 2 statements, why a unicode error remains.

#needed line, script does *not* work without it
sys.stdout = os.fdopen(1, 'w', encoding='utf-8')

# connect to database
con = pymysql.connect( db = 'pelatologio', host = 'localhost', user = 'myself', passwd = 'mypass', init_command='SET NAMES UTF8' )
cur = con.cursor()

Shall i chnage the connector form 'pymysql' => 'MySQLdb' ?

A quick look at the documentation tells me that the charset can be
specified in the 'connect' call, something like this:

con = pymysql.connect( db = 'pelatologio', host = 'localhost', user =
'myself', passwd = 'mypass', init_command='SET NAMES UTF8', charset =
'utf-8' )

 

[Michael Torrie]

I do not know here to find connections.py Michael.

It's part of the pymysql package that you installed. Look in there (the
traceback even shows you where the file is). But you actually don't
even need to look at it on your server. You can see the source code for
it here:

https://github.com/petehunt/PyMySQL/tree/master/pymysql

But i do not understand since iam suing the following 2 statements,
why a unicode error remains.

The traceback told you why the error is happening. inside the pymysql
library, there is a call to .encode(charset) on your query string. And
the default charset is latin-1. You've got to work out a way to change
that and I've already told you exactly how to find out what you need to
do.

#needed line, script does *not* work without it sys.stdout =
os.fdopen(1, 'w', encoding='utf-8')

# connect to database con = pymysql.connect( db = 'pelatologio', host
= 'localhost', user = 'myself', passwd = 'mypass', init_command='SET
NAMES UTF8' ) cur = con.cursor()

HINT: this line ^^^^^^ is what you have to modify by passing in a
particular keyword argument that sets the character set (no it's not the
init_command one). You will have to look at connections.py's __init__
declaration and you should see what you need to pass in.

Yes I could have saved myself a lot of time and typing by giving you the
answer, but that isn't going to help you. On the other hand, if you
want the easy way out, paypal some money to me and I'll just give you
the answer instead of trying to give you the tools you need to debug
this issue. The answers are all in the source code to pymysql, which is
open source so you can always read (either on your own server file
system or on the git repository).

Shall i chnage the connector form 'pymysql' => 'MySQLdb' ?

No.

 

[Íßêïò Ãêñ33ê]

con = pymysql.connect( db = 'pelatologio', host = 'localhost', user = 'blabla', passwd = 'blabla', init_command='SET NAMES UTF8', charset= 'utf-8' )

produces this "God knows what" error traceback....

/home/nikos/public_html/cgi-bin/metrites.py in ()
217 template = htmldata + counter
218 elif page.endswith('.py'):
=> 219 htmldata = subprocess.check_output( '/home/nikos/public_html/cgi-bin/' + page )
220 template = htmldata.decode('utf-8').replace( 'Content-type: text/html; charset=utf-8', '' ) + counter
221
htmldata undefined, subprocess = <module 'subprocess' from '/opt/python3/lib/python3.3/subprocess.py'>, subprocess.check_output = <function check_output>, page = 'pelatologio.py'
/opt/python3/lib/python3.3/subprocess.py in check_output(timeout=None, *popenargs=('/home/nikos/public_html/cgi-bin/pelatologio.py',), **kwargs={})
584 retcode = process.poll()
585 if retcode:
=> 586 raise CalledProcessError(retcode, process.args, output=output)
587 return output
588
global CalledProcessError = <class 'subprocess.CalledProcessError'>, retcode = 1, process = <subprocess.Popen object>, process.args = '/home/nikos/public_html/cgi-bin/pelatologio.py', output = b'<!--: spam\nContent-Type: text/html\n\n<body b...Type\' object has no attribute \'id\'\n\n-->\n\n'
CalledProcessError: Command '/home/nikos/public_html/cgi-bin/pelatologio.py' returned non-zero exit status 1
args = (1, '/home/nikos/public_html/cgi-bin/pelatologio.py')
cmd = '/home/nikos/public_html/cgi-bin/pelatologio.py'
output = b'<!--: spam\nContent-Type: text/html\n\n<body b...Type\' object has no attribute \'id\'\n\n-->\n\n'
returncode = 1
with_traceback = <built-in method with_traceback of CalledProcessError object>

 

[Michael Torrie]

con = pymysql.connect( db = 'pelatologio', host = 'localhost', user = 'blabla', passwd = 'blabla', init_command='SET NAMES UTF8', charset = 'utf-8' )

produces this "God knows what" error traceback....

Hey at least your database code is now working.

/home/nikos/public_html/cgi-bin/metrites.py in ()
217 template = htmldata + counter
218 elif page.endswith('.py'):
=> 219 htmldata = subprocess.check_output( '/home/nikos/public_html/cgi-bin/' + page )
220 template = htmldata.decode('utf-8').replace( 'Content-type: text/html; charset=utf-8', '' ) + counter
221
htmldata undefined, subprocess = <module 'subprocess' from '/opt/python3/lib/python3.3/subprocess.py'>, subprocess.check_output = <function check_output>, page = 'pelatologio.py'
/opt/python3/lib/python3.3/subprocess.py in check_output(timeout=None, *popenargs=('/home/nikos/public_html/cgi-bin/pelatologio.py',), **kwargs={})
584 retcode = process.poll()
585 if retcode:
=> 586 raise CalledProcessError(retcode, process.args, output=output)
587 return output
588
global CalledProcessError = <class 'subprocess.CalledProcessError'>, retcode = 1, process = <subprocess.Popen object>, process.args = '/home/nikos/public_html/cgi-bin/pelatologio.py', output = b'<!--: spam\nContent-Type: text/html\n\n<body b...Type\' object has no attribute \'id\'\n\n-->\n\n'
CalledProcessError: Command '/home/nikos/public_html/cgi-bin/pelatologio.py' returned non-zero exit status 1
args = (1, '/home/nikos/public_html/cgi-bin/pelatologio.py')
cmd = '/home/nikos/public_html/cgi-bin/pelatologio.py'
output = b'<!--: spam\nContent-Type: text/html\n\n<body b...Type\' object has no attribute \'id\'\n\n-->\n\n'
returncode = 1
with_traceback = <built-in method with_traceback of CalledProcessError object>

Again the traceback is telling you where to look for the problem. And
the problem is in your own script, pelatologio.py, though the line
number that's causing the problem is obscured.

Basically you want pelatologio.py to run and then you process the output
through a template, correct?

Well the traceback is telling you that pelatologio.py is erroring out.
In fact, subprocess is telling you that this process (your script) quit
with an error. Though the line number in pelatologio.py is unknown (the
traceback is obscured by your html processing code in your template
processor), the error message itself is not. Inside pelatologio.py
there is some object that you are trying to reference the "id" attribute
on it does not contain id.

So the problem is in pelatologio.py. Double check your code there. Try
to make a standalone test file you can run locally.

 

[nagia.retsina]

Τη ΤÏίτη, 28 ΜαÎου 2013 8:17:05 μ.μ. UTC+3, ο χÏήστης Michael Torrie έγÏαψε:

Basically you want pelatologio.py to run and then you process the output
through a template, correct?

That is correct.

Inside pelatologio.py
there is some object that you are trying to reference the "id" attribute
on it does not contain id.
So the problem is in pelatologio.py. Double check your code there. Try
to make a standalone test file you can run locally.

I have, i see no error, the error appears only when i append the charset directive into the connector, if i remove it the error dissapears.

 

[Joel Goldstick]

Τη ΤÏίτη, 28 ΜαÎου 2013 8:17:05 μ.μ. UTC+3, ο χÏήστης Michael Torrie
έγÏαψε:


That is correct.


I have, i see no error, the error appears only when i append the charset
directive into the connector, if i remove it the error dissapears.

http://en.wikipedia.org/wiki/Dunning–Kruger_effect

Could this be the problem?

 

[Michael Torrie]

Τη ΤÏίτη, 28 ΜαÎου 2013 8:17:05 μ.μ. UTC+3, ο χÏήστης Michael Torrie
έγÏαψε:


That is correct.


I have, i see no error, the error appears only when i append the
charset directive into the connector, if i remove it the error
dissapears.

No, when you remove the charset directive your database call fails and
thus things fail even earlier.

 

[nagia.retsina]

Τη ΤÏίτη, 28 ΜαÎου 2013 9:19:21 μ.μ. UTC+3, ο χÏήστης Michael Torrie έγÏαψε:

No, when you remove the charset directive your database call fails and

thus things fail even earlier.

Yes, it does create issues.
i just tried to add it to metrites.py for testing and it breaks too.

https://github.com/petehunt/PyMySQL/issues/38

Switching back to MySQLdb.

 

[nagia.retsina]

But i think iam gonna fight it some more because mysqldb in python 3 has issues too

 

[nagia.retsina]

What makes us o sure it is a pymysql issue and not python's encoding issue?

 

[nagia.retsina]

Τη ΤετάÏτη, 29 ΜαÎου 2013 1:30:15 μ.μ. UTC+3, ο χÏήστης (e-mail address removed) έγÏαψε:

What makes us o sure it is a pymysql issue and not python's encoding issue?

Someone help please.

 

[nagia.retsina]

Can ypou tell me how to install MySQLdb in python 3 using pip?

pip install MySQLdb doesnt find the module.

 

[Michael Torrie]

What makes us o sure it is a pymysql issue and not python's encoding
issue?

The original traceback, which showed that the encoding error was
happening in
"/opt/python3/lib/python3.3/site-packages/pymysql/cursors.py", line 108.
As was said, you solve that by passing a charset="utf-8" to the
connection string.

So doing that solved the encoding problem (a query is now being
successfully built and sent to mysql) and went on to expose another
problem (bug) in your code, but I cannot tell what that is, since the
error happened in a subprocess and the traceback got sent to /dev/null.
I suspect is has something to do with how the query results are being
returned, or it could have something to do with the query itself.
Python DB API does not specify exactly which style of prepared
statements should be used by a given third-party module. So differences
in syntax between how pymysql and MysqlDB define the variables could be
the problem.

In any case your course is clear. Run pelatologio.py outside of your
templating system and see what the traceback says exactly now that the
charset issue is fixed.

 

[nagia.retsina]

Τη Πέμπτη, 30 ΜαÎου 2013 12:29:56 μ.μ. UTC+3, ο χÏήστης Michael Torrie έγÏαψε:

The original traceback, which showed that the encoding error was

happening in

"/opt/python3/lib/python3.3/site-packages/pymysql/cursors.py", line 108.

As was said, you solve that by passing a charset="utf-8" to the

connection string.



So doing that solved the encoding problem (a query is now being

successfully built and sent to mysql) and went on to expose another

problem (bug) in your code, but I cannot tell what that is, since the

error happened in a subprocess and the traceback got sent to /dev/null.

I suspect is has something to do with how the query results are being

returned, or it could have something to do with the query itself.

Python DB API does not specify exactly which style of prepared

statements should be used by a given third-party module. So differences

in syntax between how pymysql and MysqlDB define the variables could be

the problem.



In any case your course is clear. Run pelatologio.py outside of your

templating system and see what the traceback says exactly now that the

charset issue is fixed.

Good morning Michael,

I'am afraid as much as you dont want to admin it that the moment i append the charset directive into the connections tring i receive a huge error which it can be displayed in:

http://superhost.gr/cgi-bin/pelatologio.py

This is run directly isolated form the templating system.

 

[nagia.retsina]

Τη Πέμπτη, 30 ΜαÎου 2013 1:53:33 μ.μ. UTC+3, ο χÏήστης (e-mail address removed) έγÏαψε:

Τη Πέμπτη, 30 ΜαÎου 2013 12:29:56 μ.μ. UTC+3, ο χÏήστης Michael Torrie έγÏαψε:




Good morning Michael,



I'am afraid as much as you dont want to admin it that the moment i appendthe charset directive into the connections tring i receive a huge error which it can be displayed in:



http://superhost.gr/cgi-bin/pelatologio.py



This is run directly isolated form the templating system.

AttributeError: 'NoneType' object has no attribute 'id'
args = ("'NoneType' object has no attribute 'id'",)
with_traceback = <built-in method with_traceback of AttributeError object>

is the error while i have no id varibale just an 'ID' inside:

cur.execute('''SELECT ID FROM clients WHERE name = %s''', (name,) )

 

[Chris Angelico]

Good morning Michael,

I'am afraid as much as you dont want to admin it that the moment i append the charset directive into the connections tring i receive a huge error which it can be displayed in:

http://superhost.gr/cgi-bin/pelatologio.py

This is run directly isolated form the templating system.

Your valid character set names can be found here:

https://github.com/petehunt/PyMySQL/blob/master/pymysql/charset.py

I found this simply by googling 'pymysql' and working through the call tree.

Find what you want. Use it.

You can probably get this information from the docs, too. Read them.

ChrisA

 

[Íßêïò Ãêñ33ê]

Ôç ÐÝìðôç, 30 ÌáÀïõ 2013 2:33:56 ì.ì. UTC+3, ï ÷ñÞóôçò Chris Angelico Ýãñáøå:

Your valid character set names can be found here:



https://github.com/petehunt/PyMySQL/blob/master/pymysql/charset.py



I found this simply by googling 'pymysql' and working through the call tree.



Find what you want. Use it.



You can probably get this information from the docs, too. Read them.



ChrisA

I cant "fucking believe it".

The moen i switched "charset = 'utf-8'" => "charset = 'utf8'" all started to work properly!

Thank you very much Chris, i cnat belive i was lookign 3 days for that error and it was a matter of a dash removal.....My God!

 

[Michael Torrie]

The moen i switched "charset = 'utf-8'" => "charset = 'utf8'" all
started to work properly!

Glad you have it working.

Perhaps this should be a lesson to you, Nick. Chris was able to spot
your problem by READING THE DOCUMENTATION, which he probably found
through some quick searching on Google. These are things you can and
should do too. As well as looking at the source code of the modules you
may have downloaded and are using, and understanding how the code works
(or fits together) and how to debug. You seem to have a tendency to try
to attack your problems with rather blunt instruments, and then come to
the list where one of the list members invariable does the footwork for you.