Unicode issue with Python v3.3

[nagia.retsina]

Ôç ÓÜââáôï, 13 Áðñéëßïõ 2013 4:41:57 ð.ì. UTC+3, ï ÷ñÞóôçò Cameron Simpson Ýãñáøå:

| Ôç ÐÝìðôç, 11 Áðñéëßïõ 2013 1:45:22 ì..ì. UTC+3, ï ÷ñÞóôçò Cameron Simpson Ýãñáøå:


| > | the doctype is coming form the attempt of script metrites.py to open and read the 'index.html' file.

| > | But i don't know how to try to open it as a byte file instead of antetxt file.



Lele Gaifax showed one way:



from codecs import open

with open('index.html', encoding='utf-8') as f:

content = f.read()



But a plain open() should also do:



with open('index.html') as f:

content = f.read()



if you're not taking tight control of the file encoding.



The point here is to get _text_ (i.e. str) data from the file, not bytes.



If the text turns out to be incorrectly decoded (i.e. incorrectly

reading the file bytes and assembling them into text strings) because

the default encoding is wrong, then you may need to read for Lele's

more verbose open() example to select the correct encoding.



But first ignore that and get text (str) instead of bytes.

If you're already getting text from the file, something later is

making bytes and handing it to print().



Another approach to try is to use

sys.stdout.write()

instead of

print()



The print() function will take _anything_ and write text of some form.

The write() function will throw an exception if it gets the wrong type ofdata.



If sys.stdout is opened in binary mode then write() will require

bytes as data; strings will need to be explicitly turned into bytes

via .encode() in order to not raise an exception.



If sys.stdout is open in text mode, write() will require str data.

The sys.stdout file itself will transcribe to bytes for you.



If you take that route, at least you will not have confusion about

str versus bytes.



For an HTML output page I would advocate arranging that sys.stdout

is in text mode; that way you can do the natural thing and .write()

str data and lovely UTF-8 bytes will come out the other end.



If the above test (using .write() instead of print()) shows it to

be in binary mode we can fix that. But you need to find out.



You will want access to the error messages from the CGI environment;

do you have access to the web servers error_log? You can tail that

in a terminal while you reload the page to see what's going on.



| This works in the shell, but doesn't work on my website:

|

| $ cat utf8.txt

| õëéêü!Ðñüêåéôáé ã



Ok, so your terminal is using UTF-8 as its output coding. (And so

is your mail posting program, since we see it unmangled on my screen

here.)



| $ python3

| Python 3.2.3 (default, Oct 19 2012, 20:10:41)

| [GCC 4.6.3] on linux2

| Type "help", "copyright", "credits" or "license" for more information.

| >>> data = open('utf8.txt').read()

| >>> print(data)

| õëéêü!Ðñüêåéôáé ã



Likewise.



However, in an exciting twist, I seem to recall that Python invoked

interactively with aterminal as output will have the default terminal

encoding in place on sys.stdout. Producing what you expect. _However_,

python invoked in a batch environment where stdout is not a terminal

(such as in the CGI environment producing your web page), that is

_not_ necessarily the case.



| >>> print(data.encode('utf-8'))

| b'\xcf\x85\xce\xbb\xce\xb9\xce\xba\xcf\x8c!\xce\xa0\xcf\x81\xcf\x8c\xce\xba\xce\xb5\xce\xb9\xcf\x84\xce\xb1\xce\xb9 \xce\xb3\n'

|

| See, the last line is what i'am getting on my website.



The above line takes your Unicode text in "data" and transcribed

it to bytes using UTF-8 as the encoding. And print() is then receiving

that bytes object and printing its str() representation as "b'....'".

That str is itself unicode, and when print passes it to sys.stdout,

_that_ transcribed the unicode "b'...'" string as bytes to your

terminal. Using UTF-8 based on the previous examples above, but

since all those characters are in the bottom 127 code range the

byte sequence will be the same if it uses ASCII or ISO8859-1 or

almost anything else



As you can see, there's a lot of encoding/decoding going on behind

the scenes even in this superficially simple example.



| If i remove

| the encode('utf-8') part in metrites.py, the webpage will not show

| anything at all...



Ah, but data will be being output. The print() function _will_ be

writing "data" out in some form. I suggest you remove the .encode()

and then examine the _source_ text of the web page, not its visible

form.



So: remove .encode(), reload the web page, "view page source"

(depends on your browser, it is ctrl-U in Firefox ((Cmd-U in firefox

on a Mac))).



I think a lot of the issue you have in this thread is that your

page is too complex. Make another page to do the same thing, and

start with nothing. Add stuff to it a single item at a time until

the page behaves incorrectly. Then you will know the exact item of

code that introduced the issue. And then that single item can be

examined in detail for the decode/encode issues.



The other issue in the thread is that people losing patience get

snarky. Respond only to the technical content. If a message is only

snarky, _ignore_ it. People like the last word; let them have it

and you won't get sidetracked into arguments.



Cheers,

--

Cameron Simpson <[email protected]>



PCs are like a submarine, it will work fine till you open Windows. - zollie101

First of all thank you very much Cameron for your detailed help and effort to write to me:

It seems another issue had happened without my knowledge, i was uploading stuff at /root/public_html/cgi-bin instead of /home/nikos/public_html/cgi-bin.

I realized that when i deliberately made error to metrites.py scropt and i got still the same page.

Ookey after that is corrected, i then tried the plain solution and i got this response back form the shell:

Traceback (most recent call last):
File "metrites.py", line 213, in &lt;module&gt;
htmldata = f.read()
File "/root/.local/lib/python2.7/lib/python3.3/encodings/iso8859_7.py", line 23, in decode
return codecs.charmap_decode(input,self.errors,decoding_table)[0]
UnicodeDecodeError: 'charmap' codec can't decode byte 0xae in position 47: character maps to &lt;undefined&gt;

then i switched to:

with open('/home/nikos/www/' + page, encoding='utf-8') as f:
htmldata = f.read()

and i got no error at all, just pure run *from the shell*!
But i get internal server error when i try to run the webpage from the browser(Chrome).

So, can you tell me please where can i find the apache error log so to display here please?

Apcher error_log is always better than running 'python3 metrites.py' because even if the python script has no error apache will also display more web related things?

Thank you Cameron.

 

[nagia.retsina]

Ôç ÓÜââáôï, 13 Áðñéëßïõ 2013 4:41:57 ð.ì. UTC+3, ï ÷ñÞóôçò Cameron Simpson Ýãñáøå:

| Ôç ÐÝìðôç, 11 Áðñéëßïõ 2013 1:45:22 ì..ì. UTC+3, ï ÷ñÞóôçò Cameron Simpson Ýãñáøå:


| > | the doctype is coming form the attempt of script metrites.py to open and read the 'index.html' file.

| > | But i don't know how to try to open it as a byte file instead of antetxt file.



Lele Gaifax showed one way:



from codecs import open

with open('index.html', encoding='utf-8') as f:

content = f.read()



But a plain open() should also do:



with open('index.html') as f:

content = f.read()



if you're not taking tight control of the file encoding.



The point here is to get _text_ (i.e. str) data from the file, not bytes.



If the text turns out to be incorrectly decoded (i.e. incorrectly

reading the file bytes and assembling them into text strings) because

the default encoding is wrong, then you may need to read for Lele's

more verbose open() example to select the correct encoding.



But first ignore that and get text (str) instead of bytes.

If you're already getting text from the file, something later is

making bytes and handing it to print().



Another approach to try is to use

sys.stdout.write()

instead of

print()



The print() function will take _anything_ and write text of some form.

The write() function will throw an exception if it gets the wrong type ofdata.



If sys.stdout is opened in binary mode then write() will require

bytes as data; strings will need to be explicitly turned into bytes

via .encode() in order to not raise an exception.



If sys.stdout is open in text mode, write() will require str data.

The sys.stdout file itself will transcribe to bytes for you.



If you take that route, at least you will not have confusion about

str versus bytes.



For an HTML output page I would advocate arranging that sys.stdout

is in text mode; that way you can do the natural thing and .write()

str data and lovely UTF-8 bytes will come out the other end.



If the above test (using .write() instead of print()) shows it to

be in binary mode we can fix that. But you need to find out.



You will want access to the error messages from the CGI environment;

do you have access to the web servers error_log? You can tail that

in a terminal while you reload the page to see what's going on.



| This works in the shell, but doesn't work on my website:

|

| $ cat utf8.txt

| õëéêü!Ðñüêåéôáé ã



Ok, so your terminal is using UTF-8 as its output coding. (And so

is your mail posting program, since we see it unmangled on my screen

here.)



| $ python3

| Python 3.2.3 (default, Oct 19 2012, 20:10:41)

| [GCC 4.6.3] on linux2

| Type "help", "copyright", "credits" or "license" for more information.

| >>> data = open('utf8.txt').read()

| >>> print(data)

| õëéêü!Ðñüêåéôáé ã



Likewise.



However, in an exciting twist, I seem to recall that Python invoked

interactively with aterminal as output will have the default terminal

encoding in place on sys.stdout. Producing what you expect. _However_,

python invoked in a batch environment where stdout is not a terminal

(such as in the CGI environment producing your web page), that is

_not_ necessarily the case.



| >>> print(data.encode('utf-8'))

| b'\xcf\x85\xce\xbb\xce\xb9\xce\xba\xcf\x8c!\xce\xa0\xcf\x81\xcf\x8c\xce\xba\xce\xb5\xce\xb9\xcf\x84\xce\xb1\xce\xb9 \xce\xb3\n'

|

| See, the last line is what i'am getting on my website.



The above line takes your Unicode text in "data" and transcribed

it to bytes using UTF-8 as the encoding. And print() is then receiving

that bytes object and printing its str() representation as "b'....'".

That str is itself unicode, and when print passes it to sys.stdout,

_that_ transcribed the unicode "b'...'" string as bytes to your

terminal. Using UTF-8 based on the previous examples above, but

since all those characters are in the bottom 127 code range the

byte sequence will be the same if it uses ASCII or ISO8859-1 or

almost anything else



As you can see, there's a lot of encoding/decoding going on behind

the scenes even in this superficially simple example.



| If i remove

| the encode('utf-8') part in metrites.py, the webpage will not show

| anything at all...



Ah, but data will be being output. The print() function _will_ be

writing "data" out in some form. I suggest you remove the .encode()

and then examine the _source_ text of the web page, not its visible

form.



So: remove .encode(), reload the web page, "view page source"

(depends on your browser, it is ctrl-U in Firefox ((Cmd-U in firefox

on a Mac))).



I think a lot of the issue you have in this thread is that your

page is too complex. Make another page to do the same thing, and

start with nothing. Add stuff to it a single item at a time until

the page behaves incorrectly. Then you will know the exact item of

code that introduced the issue. And then that single item can be

examined in detail for the decode/encode issues.



The other issue in the thread is that people losing patience get

snarky. Respond only to the technical content. If a message is only

snarky, _ignore_ it. People like the last word; let them have it

and you won't get sidetracked into arguments.



Cheers,

--

Cameron Simpson <[email protected]>



PCs are like a submarine, it will work fine till you open Windows. - zollie101

First of all thank you very much Cameron for your detailed help and effort to write to me:

It seems another issue had happened without my knowledge, i was uploading stuff at /root/public_html/cgi-bin instead of /home/nikos/public_html/cgi-bin.

I realized that when i deliberately made error to metrites.py scropt and i got still the same page.

Ookey after that is corrected, i then tried the plain solution and i got this response back form the shell:

Traceback (most recent call last):
File "metrites.py", line 213, in &lt;module&gt;
htmldata = f.read()
File "/root/.local/lib/python2.7/lib/python3.3/encodings/iso8859_7.py", line 23, in decode
return codecs.charmap_decode(input,self.errors,decoding_table)[0]
UnicodeDecodeError: 'charmap' codec can't decode byte 0xae in position 47: character maps to &lt;undefined&gt;

then i switched to:

with open('/home/nikos/www/' + page, encoding='utf-8') as f:
htmldata = f.read()

and i got no error at all, just pure run *from the shell*!
But i get internal server error when i try to run the webpage from the browser(Chrome).

So, can you tell me please where can i find the apache error log so to display here please?

Apcher error_log is always better than running 'python3 metrites.py' because even if the python script has no error apache will also display more web related things?

Thank you Cameron.

 

[Cameron Simpson]

| Ookey after that is corrected, i then tried the plain solution and i got this response back form the shell:
|
| Traceback (most recent call last):
| File "metrites.py", line 213, in &lt;module&gt;
| htmldata = f.read()
| File "/root/.local/lib/python2.7/lib/python3.3/encodings/iso8859_7.py", line 23, in decode
| return codecs.charmap_decode(input,self.errors,decoding_table)[0]
| UnicodeDecodeError: 'charmap' codec can't decode byte 0xae in position 47: character maps to &lt;undefined&gt;
|
| then i switched to:
|
| with open('/home/nikos/www/' + page, encoding='utf-8') as f:
| htmldata = f.read()
|
| and i got no error at all, just pure run *from the shell*!

Ok, so you need to specify utf-8 to decode the file. Good.

| But i get internal server error when i try to run the webpage from the browser(Chrome).

That is standard for a CGI script that does not complete successfully.

| So, can you tell me please where can i find the apache error log so to display here please?

That depends on the install. Have a look in /var/log/apache or similar.
Otherwise you need to find the httpd.conf for the apache and look
for its log coniguration settings.

| Apcher error_log is always better than running 'python3 metrites.py' because even if the python script has no error apache will also display more web related things?

The error log is where error messages from CGI scripts go. And other error messages.
It is very useful when testing CGI scripts.

Of course, it's best to work out as much as possible from the command
line first; you have much more direct control and access to errors
there. That only gets you so far though; the environment the CGI
script runs in is not the same as your command line, and some
different behaviour can come from this.

BTW, are you sure python3 is running your CGI script?
Also, the CGI script may not be running as you, but as the apache user.
In that case, it may fail if it does not has permission to access various
files owned by you.

Anyway, you need to see the error messages to work this out.

If you can't find the error log you can divert stderr at the
start of your python program:

sys.stderr = open('/home/nikos/cgi.err.out', 'a')

and watch that in a shell:

tail -f cgi.err.out

Cheers,

 

[nagia.retsina]

Τη Σάββατο, 13 ΑπÏιλίου 2013 1:28:07 μ.μ. UTC+3, οχÏήστης Cameron Simpson έγÏαψε:

| Ookey after that is corrected, i then tried the plain solution and i got this response back form the shell:

|

| Traceback (most recent call last):

| File "metrites.py", line 213, in &lt;module&gt;

| htmldata = f.read()

| File "/root/.local/lib/python2.7/lib/python3.3/encodings/iso8859_7.py", line 23, in decode

| return codecs.charmap_decode(input,self.errors,decoding_table)[0]

| UnicodeDecodeError: 'charmap' codec can't decode byte 0xae in position 47: character maps to &lt;undefined&gt;

|

| then i switched to:

|

| with open('/home/nikos/www/' + page, encoding='utf-8') as f:

| htmldata = f.read()

|

| and i got no error at all, just pure run *from the shell*!



Ok, so you need to specify utf-8 to decode the file. Good.



| But i get internal server error when i try to run the webpage from the browser(Chrome).



That is standard for a CGI script that does not complete successfully.



| So, can you tell me please where can i find the apache error log so to display here please?



That depends on the install. Have a look in /var/log/apache or similar.

Otherwise you need to find the httpd.conf for the apache and look

for its log coniguration settings.



| Apcher error_log is always better than running 'python3 metrites.py' because even if the python script has no error apache will also display more web related things?



The error log is where error messages from CGI scripts go. And other error messages.

It is very useful when testing CGI scripts.



Of course, it's best to work out as much as possible from the command

line first; you have much more direct control and access to errors

there. That only gets you so far though; the environment the CGI

script runs in is not the same as your command line, and some

different behaviour can come from this.



BTW, are you sure python3 is running your CGI script?

Also, the CGI script may not be running as you, but as the apache user.

In that case, it may fail if it does not has permission to access various

files owned by you.



Anyway, you need to see the error messages to work this out.



If you can't find the error log you can divert stderr at the

start of your python program:



sys.stderr = open('/home/nikos/cgi.err.out', 'a')



and watch that in a shell:



tail -f cgi.err.out



Cheers,

--

Cameron Simpson <[email protected]>



If you 'aint falling off, you ar'nt going hard enough. - Fred Gassit

root@macgyver [/home/nikos/public_html/cgi-bin]# ls ../cgi.err.out
.../cgi.err.out
root@macgyver [/home/nikos/public_html/cgi-bin]# cat ../cgi.err.out
root@macgyver [/home/nikos/public_html/cgi-bin]#

Also i have foudn the error log and i tried to view it but it was empty andthen i removed it and then run the script both from shell and broswer but it didnt reappeared.

root@macgyver [/home/nikos/public_html/cgi-bin]# cat /var/log/httpd/error_log
cat: /var/log/httpd/error_log: No such file or directory
root@macgyver [/home/nikos/public_html/cgi-bin]#

Maybe its somehtign wron with my enviroment?
Should we check the Apache and CGI enviroment somehow and also make sure asyou say that *I* run the CGI scripts and not user 'Apache' ?

Tell me what commands i should issues please and i will display the output to you.

Thank you Cameron, for helpimg me. Somehow the script doesnt seem to be theissue in my VPS.

 

[nagia.retsina]

Τη Σάββατο, 13 ΑπÏιλίου 2013 1:28:07 μ.μ. UTC+3, οχÏήστης Cameron Simpson έγÏαψε:

| Ookey after that is corrected, i then tried the plain solution and i got this response back form the shell:

|

| Traceback (most recent call last):

| File "metrites.py", line 213, in &lt;module&gt;

| htmldata = f.read()

| File "/root/.local/lib/python2.7/lib/python3.3/encodings/iso8859_7.py", line 23, in decode

| return codecs.charmap_decode(input,self.errors,decoding_table)[0]

| UnicodeDecodeError: 'charmap' codec can't decode byte 0xae in position 47: character maps to &lt;undefined&gt;

|

| then i switched to:

|

| with open('/home/nikos/www/' + page, encoding='utf-8') as f:

| htmldata = f.read()

|

| and i got no error at all, just pure run *from the shell*!



Ok, so you need to specify utf-8 to decode the file. Good.



| But i get internal server error when i try to run the webpage from the browser(Chrome).



That is standard for a CGI script that does not complete successfully.



| So, can you tell me please where can i find the apache error log so to display here please?



That depends on the install. Have a look in /var/log/apache or similar.

Otherwise you need to find the httpd.conf for the apache and look

for its log coniguration settings.



| Apcher error_log is always better than running 'python3 metrites.py' because even if the python script has no error apache will also display more web related things?



The error log is where error messages from CGI scripts go. And other error messages.

It is very useful when testing CGI scripts.



Of course, it's best to work out as much as possible from the command

line first; you have much more direct control and access to errors

there. That only gets you so far though; the environment the CGI

script runs in is not the same as your command line, and some

different behaviour can come from this.



BTW, are you sure python3 is running your CGI script?

Also, the CGI script may not be running as you, but as the apache user.

In that case, it may fail if it does not has permission to access various

files owned by you.



Anyway, you need to see the error messages to work this out.



If you can't find the error log you can divert stderr at the

start of your python program:



sys.stderr = open('/home/nikos/cgi.err.out', 'a')



and watch that in a shell:



tail -f cgi.err.out



Cheers,

--

Cameron Simpson <[email protected]>



If you 'aint falling off, you ar'nt going hard enough. - Fred Gassit

root@macgyver [/home/nikos/public_html/cgi-bin]# ls ../cgi.err.out
.../cgi.err.out
root@macgyver [/home/nikos/public_html/cgi-bin]# cat ../cgi.err.out
root@macgyver [/home/nikos/public_html/cgi-bin]#

Also i have foudn the error log and i tried to view it but it was empty andthen i removed it and then run the script both from shell and broswer but it didnt reappeared.

root@macgyver [/home/nikos/public_html/cgi-bin]# cat /var/log/httpd/error_log
cat: /var/log/httpd/error_log: No such file or directory
root@macgyver [/home/nikos/public_html/cgi-bin]#

Maybe its somehtign wron with my enviroment?
Should we check the Apache and CGI enviroment somehow and also make sure asyou say that *I* run the CGI scripts and not user 'Apache' ?

Tell me what commands i should issues please and i will display the output to you.

Thank you Cameron, for helpimg me. Somehow the script doesnt seem to be theissue in my VPS.

 

[Chris Angelico]

Also i have foudn the error log and i tried to view it but it was empty and then i removed it and then run the script both from shell and broswer but it didnt reappeared.

root@macgyver [/home/nikos/public_html/cgi-bin]# cat /var/log/httpd/error_log
cat: /var/log/httpd/error_log: No such file or directory
root@macgyver [/home/nikos/public_html/cgi-bin]#

https://www.google.com/search?q=log+file+rotation

ChrisA

 

[Cameron Simpson]

| root@macgyver [/home/nikos/public_html/cgi-bin]# ls ../cgi.err.out
| ../cgi.err.out

I prefer "ls -ld" myself.

| root@macgyver [/home/nikos/public_html/cgi-bin]# cat ../cgi.err.out
|
| Also i have foudn the error log and i tried to view it but it was
| empty and then i removed it and then run the script both from shell
| and broswer but it didnt reappeared.

Never remove it. It is only created by the web server at startup or log rotation time. So now you need to restart the apache to get it back.

Just open a spare terminal and run:

tail -f /var/log/httpd/error_log

| Should we check the Apache and CGI enviroment somehow and also
| make sure as you say that *I* run the CGI scripts and not user
| 'Apache' ?

Well, it is helpful to know. if the CGI script tries to write any data to files,
if it runs as a different user it will need different permissions on the files.

| Tell me what commands i should issues please and i will display the output to you.

I would be tempter to wrap the CGI script in a shell script.
Suppose your script is named foo.py.

You can move the script to foo-py and make a shell script called "foo.py" looking like this:

#!/bin/sh
exec 2>>/home/nikos/cgi.err.out
echo "$0 $*" >&2
id >&2
env | sort >&2
set -x
exec /full/path/to/foo-py ${1+"$@"}

and make sure it, like the original, is readable and executable:

chmod a+rx foo.py foo-py

Make sure cgi.err.out is publicly writable (in case the apache is
not running the CGIs are you):

chmod a+w cgi.err.out

Then:

tail -f cgi.err.out

in a spare window.

Then try the script.

It should transcribe information about the script's user and
environment and also catch errors.

This should help in debugging.

Cheers,

 

[nagia.retsina]

Τη ΤετάÏτη, 10 ΑπÏιλίου 2013 12:10:13 Ï€.μ. UTC+3, ο χÏήστης Îίκος ΓκÏ33κ έγÏαψε:

Hello, iam still trying to alter the code form python 2.6 => 3.3



Everyrging its setup except that unicode error that you can see if you goto http://superhost.gr



Can anyone help with this?

I even tried to change print() with sys.stdout.buffer() but still i get the same unicode issue.



I don't know what to try anymore.

root@nikos [/home/nikos/public_html/foo-py]# pwd
/home/nikos/public_html/foo-py
root@nikos [/home/nikos/public_html/foo-py]# cat foo.py
#!/bin/sh
exec 2>>/home/nikos/cgi.err.out
echo "$0 $*" >&2
id >&2
env | sort >&2
set -x
exec /full/path/to/foo-py ${1+"$@"}

root@nikos [/home/nikos/public_html/foo-py]# python3 foo.py
File "foo.py", line 2
exec 2>>/home/nikos/cgi.err.out
^
SyntaxError: invalid syntax
root@nikos [/home/nikos/public_html/foo-py]#

As far as thr tail -f of the error_log:

root@nikos [/home/nikos/public_html]# touch /var/log/httpd/error_log
root@nikos [/home/nikos/public_html]# tail -f /var/log/httpd/error_log

and its empty even when at the exact same time i run 'python3 metrites.py' from another interactive prompt when it supposed to give live feed of the error messages.

Cameron would it be too much to ask to provide you with root access to my VPS server so you can have a look there too?

i can pay you if you like if you wait a few days to gather some money.

 

[Cameron Simpson]

| root@nikos [/home/nikos/public_html/foo-py]# pwd
| /home/nikos/public_html/foo-py
| root@nikos [/home/nikos/public_html/foo-py]# cat foo.py
| #!/bin/sh
| exec 2>>/home/nikos/cgi.err.out
| echo "$0 $*" >&2
| id >&2
| env | sort >&2
| set -x
| exec /full/path/to/foo-py ${1+"$@"}
|
| root@nikos [/home/nikos/public_html/foo-py]# python3 foo.py
| File "foo.py", line 2
| exec 2>>/home/nikos/cgi.err.out
| ^
| SyntaxError: invalid syntax

That is because foo.py isn't a python script anymore, it is a shell script.
Its purpose is to divert stderr to a file and to recite various
things about the environment to that file in addition to any error
messages.

Just run it directly:

./foo.py

The #! line should cause it to be run by the shell.

I also recommend you try to do all this as your normal user account.
Root is for administration, such as stopping/starting apache and
so on. Not test running scripts from the command line; consider:
if the script has bugs, as root it can do an awful lot of damage.

| root@nikos [/home/nikos/public_html/foo-py]#
| As far as thr tail -f of the error_log:
| root@nikos [/home/nikos/public_html]# touch /var/log/httpd/error_log

That won't do you much good; apache has not opened it, and so it
will not be writing to it. It was writing to a file of that name,
but you removed that file. Apache probably still has its hooks in the old
file (which now has no name).

Restarting apache should open (or create if missing) this file for you.

| root@nikos [/home/nikos/public_html]# tail -f /var/log/httpd/error_log
| and its empty even when at the exact same time i run 'python3
| metrites.py' from another interactive prompt when it supposed to
| give live feed of the error messages.

No, _apache_ writes to that file. So only when you visit the web
page will stuff appear there.

If you just run things from the command line, error messages will appear on your terminal. Or, after this line of the wrapper script:

exec 2>>/home/nikos/cgi.err.out

the error messages will appear in cgi.err.out.

| Cameron would it be too much to ask to provide you with root
| access to my VPS server so you can have a look there too?
| i can pay you if you like if you wait a few days to gather some money.

I really do not recommend that:

- it is nuts to blithely allow a stranger root access to your system
- you won't learn anything about CGI scripts

What you need for further debugging of your python issues is access
to the error messages from the CGI script. That is the purpose of
the wrapper script.

Get the wrapper running on the command line and then test it via the browser.

Cheers,
--
Cameron Simpson <[email protected]>

Lord grant me the serenity to accept the things I can not change,
the courage to change the things that I can,
and the wisdom to hide the bodies of those people I had to kill
because they pissed me off.
- Jeffrey Papen <[email protected]>

 

[nagia.retsina]

Τη ΚυÏιακή, 14 ΑπÏιλίου 2013 12:28:32 μ.μ. UTC+3, ο χÏήστης Cameron Simpson έγÏαψε:

| root@nikos [/home/nikos/public_html/foo-py]# pwd

| /home/nikos/public_html/foo-py

| root@nikos [/home/nikos/public_html/foo-py]# cat foo.py

| #!/bin/sh

| exec 2>>/home/nikos/cgi.err.out

| echo "$0 $*" >&2

| id >&2

| env | sort >&2

| set -x

| exec /full/path/to/foo-py ${1+"$@"}

|

| root@nikos [/home/nikos/public_html/foo-py]# python3 foo.py

| File "foo.py", line 2

| exec 2>>/home/nikos/cgi.err.out

| ^

| SyntaxError: invalid syntax



That is because foo.py isn't a python script anymore, it is a shell script.

Its purpose is to divert stderr to a file and to recite various

things about the environment to that file in addition to any error

messages.



Just run it directly:



./foo.py



The #! line should cause it to be run by the shell.



I also recommend you try to do all this as your normal user account.

Root is for administration, such as stopping/starting apache and

so on. Not test running scripts from the command line; consider:

if the script has bugs, as root it can do an awful lot of damage.



| root@nikos [/home/nikos/public_html/foo-py]#

| As far as thr tail -f of the error_log:

| root@nikos [/home/nikos/public_html]# touch /var/log/httpd/error_log



That won't do you much good; apache has not opened it, and so it

will not be writing to it. It was writing to a file of that name,

but you removed that file. Apache probably still has its hooks in the old

file (which now has no name).



Restarting apache should open (or create if missing) this file for you.



| root@nikos [/home/nikos/public_html]# tail -f /var/log/httpd/error_log

| and its empty even when at the exact same time i run 'python3

| metrites.py' from another interactive prompt when it supposed to

| give live feed of the error messages.



No, _apache_ writes to that file. So only when you visit the web

page will stuff appear there.



If you just run things from the command line, error messages will appear on your terminal. Or, after this line of the wrapper script:



exec 2>>/home/nikos/cgi.err.out



the error messages will appear in cgi.err.out.



| Cameron would it be too much to ask to provide you with root

| access to my VPS server so you can have a look there too?

| i can pay you if you like if you wait a few days to gather some money.



I really do not recommend that:



- it is nuts to blithely allow a stranger root access to your system

- you won't learn anything about CGI scripts



What you need for further debugging of your python issues is access

to the error messages from the CGI script. That is the purpose of

the wrapper script.



Get the wrapper running on the command line and then test it via the browser.



Cheers,

--

Cameron Simpson <[email protected]>



Lord grant me the serenity to accept the things I can not change,

the courage to change the things that I can,

and the wisdom to hide the bodies of those people I had to kill

because they pissed me off.

- Jeffrey Papen <[email protected]>

Well i trust you because you are the only one along with Lele that are helpimg me here:

i tried what you said:

root@nikos [/home/nikos/public_html/cgi-bin]# service httpd restart
root@nikos [/home/nikos/public_html/cgi-bin]# python3 metrites.py
root@nikos [/home/nikos/public_html]# cd foo-py/
root@nikos [/home/nikos/public_html/foo-py]# ls
../ ../ foo.py*
root@nikos [/home/nikos/public_html/foo-py]# ./foo.py
root@nikos [/home/nikos/public_html/foo-py]# cd ..
root@nikos [/home/nikos/public_html]# cat cgi.err.out
root@nikos [/home/nikos/public_html/cgi-bin]# cat /var/log/httpd/error_log
root@nikos [/home/nikos/public_html/cgi-bin]#

and i have run the script form browser but i still see nothing.

I insist that you will make the most of this if you access the VPS yourself..
it runs CentOS 6.4

Please accept, i trust you.

 

[nagia.retsina]

Τη ΚυÏιακή, 14 ΑπÏιλίου 2013 12:28:32 μ.μ. UTC+3, ο χÏήστης Cameron Simpson έγÏαψε:

| root@nikos [/home/nikos/public_html/foo-py]# pwd

| /home/nikos/public_html/foo-py

| root@nikos [/home/nikos/public_html/foo-py]# cat foo.py

| #!/bin/sh

| exec 2>>/home/nikos/cgi.err.out

| echo "$0 $*" >&2

| id >&2

| env | sort >&2

| set -x

| exec /full/path/to/foo-py ${1+"$@"}

|

| root@nikos [/home/nikos/public_html/foo-py]# python3 foo.py

| File "foo.py", line 2

| exec 2>>/home/nikos/cgi.err.out

| ^

| SyntaxError: invalid syntax



That is because foo.py isn't a python script anymore, it is a shell script.

Its purpose is to divert stderr to a file and to recite various

things about the environment to that file in addition to any error

messages.



Just run it directly:



./foo.py



The #! line should cause it to be run by the shell.



I also recommend you try to do all this as your normal user account.

Root is for administration, such as stopping/starting apache and

so on. Not test running scripts from the command line; consider:

if the script has bugs, as root it can do an awful lot of damage.



| root@nikos [/home/nikos/public_html/foo-py]#

| As far as thr tail -f of the error_log:

| root@nikos [/home/nikos/public_html]# touch /var/log/httpd/error_log



That won't do you much good; apache has not opened it, and so it

will not be writing to it. It was writing to a file of that name,

but you removed that file. Apache probably still has its hooks in the old

file (which now has no name).



Restarting apache should open (or create if missing) this file for you.



| root@nikos [/home/nikos/public_html]# tail -f /var/log/httpd/error_log

| and its empty even when at the exact same time i run 'python3

| metrites.py' from another interactive prompt when it supposed to

| give live feed of the error messages.



No, _apache_ writes to that file. So only when you visit the web

page will stuff appear there.



If you just run things from the command line, error messages will appear on your terminal. Or, after this line of the wrapper script:



exec 2>>/home/nikos/cgi.err.out



the error messages will appear in cgi.err.out.



| Cameron would it be too much to ask to provide you with root

| access to my VPS server so you can have a look there too?

| i can pay you if you like if you wait a few days to gather some money.



I really do not recommend that:



- it is nuts to blithely allow a stranger root access to your system

- you won't learn anything about CGI scripts



What you need for further debugging of your python issues is access

to the error messages from the CGI script. That is the purpose of

the wrapper script.



Get the wrapper running on the command line and then test it via the browser.



Cheers,

--

Cameron Simpson <[email protected]>



Lord grant me the serenity to accept the things I can not change,

the courage to change the things that I can,

and the wisdom to hide the bodies of those people I had to kill

because they pissed me off.

- Jeffrey Papen <[email protected]>

Well i trust you because you are the only one along with Lele that are helpimg me here:

i tried what you said:

root@nikos [/home/nikos/public_html/cgi-bin]# service httpd restart
root@nikos [/home/nikos/public_html/cgi-bin]# python3 metrites.py
root@nikos [/home/nikos/public_html]# cd foo-py/
root@nikos [/home/nikos/public_html/foo-py]# ls
../ ../ foo.py*
root@nikos [/home/nikos/public_html/foo-py]# ./foo.py
root@nikos [/home/nikos/public_html/foo-py]# cd ..
root@nikos [/home/nikos/public_html]# cat cgi.err.out
root@nikos [/home/nikos/public_html/cgi-bin]# cat /var/log/httpd/error_log
root@nikos [/home/nikos/public_html/cgi-bin]#

and i have run the script form browser but i still see nothing.

I insist that you will make the most of this if you access the VPS yourself..
it runs CentOS 6.4

Please accept, i trust you.

 

[Íßêïò Ãêñ33ê]

Hello, can you still help me please?

 

[Íßêïò Ãêñ33ê]

Hello, can you still help me please?

 

[nagia.retsina]

Τη ΚυÏιακή, 14 ΑπÏιλίου 2013 12:28:32 μ.μ. UTC+3, ο χÏήστης Cameron Simpson έγÏαψε:

| root@nikos [/home/nikos/public_html/foo-py]# pwd

| /home/nikos/public_html/foo-py

| root@nikos [/home/nikos/public_html/foo-py]# cat foo.py

| #!/bin/sh

| exec 2>>/home/nikos/cgi.err.out

| echo "$0 $*" >&2

| id >&2

| env | sort >&2

| set -x

| exec /full/path/to/foo-py ${1+"$@"}

|

| root@nikos [/home/nikos/public_html/foo-py]# python3 foo.py

| File "foo.py", line 2

| exec 2>>/home/nikos/cgi.err.out

| ^

| SyntaxError: invalid syntax



That is because foo.py isn't a python script anymore, it is a shell script.

Its purpose is to divert stderr to a file and to recite various

things about the environment to that file in addition to any error

messages.



Just run it directly:



./foo.py



The #! line should cause it to be run by the shell.



I also recommend you try to do all this as your normal user account.

Root is for administration, such as stopping/starting apache and

so on. Not test running scripts from the command line; consider:

if the script has bugs, as root it can do an awful lot of damage.



| root@nikos [/home/nikos/public_html/foo-py]#

| As far as thr tail -f of the error_log:

| root@nikos [/home/nikos/public_html]# touch /var/log/httpd/error_log



That won't do you much good; apache has not opened it, and so it

will not be writing to it. It was writing to a file of that name,

but you removed that file. Apache probably still has its hooks in the old

file (which now has no name).



Restarting apache should open (or create if missing) this file for you.



| root@nikos [/home/nikos/public_html]# tail -f /var/log/httpd/error_log

| and its empty even when at the exact same time i run 'python3

| metrites.py' from another interactive prompt when it supposed to

| give live feed of the error messages.



No, _apache_ writes to that file. So only when you visit the web

page will stuff appear there.



If you just run things from the command line, error messages will appear on your terminal. Or, after this line of the wrapper script:



exec 2>>/home/nikos/cgi.err.out



the error messages will appear in cgi.err.out.



| Cameron would it be too much to ask to provide you with root

| access to my VPS server so you can have a look there too?

| i can pay you if you like if you wait a few days to gather some money.



I really do not recommend that:



- it is nuts to blithely allow a stranger root access to your system

- you won't learn anything about CGI scripts



What you need for further debugging of your python issues is access

to the error messages from the CGI script. That is the purpose of

the wrapper script.



Get the wrapper running on the command line and then test it via the browser.



Cheers,

--

Cameron Simpson <[email protected]>



Lord grant me the serenity to accept the things I can not change,

the courage to change the things that I can,

and the wisdom to hide the bodies of those people I had to kill

because they pissed me off.

- Jeffrey Papen <[email protected]>

cameron,

can you help please or tell me what else i need to try?
Hello

 

[nagia.retsina]

Τη ΚυÏιακή, 14 ΑπÏιλίου 2013 12:28:32 μ.μ. UTC+3, ο χÏήστης Cameron Simpson έγÏαψε:

| root@nikos [/home/nikos/public_html/foo-py]# pwd

| /home/nikos/public_html/foo-py

| root@nikos [/home/nikos/public_html/foo-py]# cat foo.py

| #!/bin/sh

| exec 2>>/home/nikos/cgi.err.out

| echo "$0 $*" >&2

| id >&2

| env | sort >&2

| set -x

| exec /full/path/to/foo-py ${1+"$@"}

|

| root@nikos [/home/nikos/public_html/foo-py]# python3 foo.py

| File "foo.py", line 2

| exec 2>>/home/nikos/cgi.err.out

| ^

| SyntaxError: invalid syntax



That is because foo.py isn't a python script anymore, it is a shell script.

Its purpose is to divert stderr to a file and to recite various

things about the environment to that file in addition to any error

messages.



Just run it directly:



./foo.py



The #! line should cause it to be run by the shell.



I also recommend you try to do all this as your normal user account.

Root is for administration, such as stopping/starting apache and

so on. Not test running scripts from the command line; consider:

if the script has bugs, as root it can do an awful lot of damage.



| root@nikos [/home/nikos/public_html/foo-py]#

| As far as thr tail -f of the error_log:

| root@nikos [/home/nikos/public_html]# touch /var/log/httpd/error_log



That won't do you much good; apache has not opened it, and so it

will not be writing to it. It was writing to a file of that name,

but you removed that file. Apache probably still has its hooks in the old

file (which now has no name).



Restarting apache should open (or create if missing) this file for you.



| root@nikos [/home/nikos/public_html]# tail -f /var/log/httpd/error_log

| and its empty even when at the exact same time i run 'python3

| metrites.py' from another interactive prompt when it supposed to

| give live feed of the error messages.



No, _apache_ writes to that file. So only when you visit the web

page will stuff appear there.



If you just run things from the command line, error messages will appear on your terminal. Or, after this line of the wrapper script:



exec 2>>/home/nikos/cgi.err.out



the error messages will appear in cgi.err.out.



| Cameron would it be too much to ask to provide you with root

| access to my VPS server so you can have a look there too?

| i can pay you if you like if you wait a few days to gather some money.



I really do not recommend that:



- it is nuts to blithely allow a stranger root access to your system

- you won't learn anything about CGI scripts



What you need for further debugging of your python issues is access

to the error messages from the CGI script. That is the purpose of

the wrapper script.



Get the wrapper running on the command line and then test it via the browser.



Cheers,

--

Cameron Simpson <[email protected]>



Lord grant me the serenity to accept the things I can not change,

the courage to change the things that I can,

and the wisdom to hide the bodies of those people I had to kill

because they pissed me off.

- Jeffrey Papen <[email protected]>

cameron,

can you help please or tell me what else i need to try?
Hello

 

[Chris Angelico]

can you help please or tell me what else i need to try?

You need to try trimming quoted text in replies, not double-spacing,
and paying for help.

ChrisA

 

[Chris Angelico]

can you help please or tell me what else i need to try?

You need to try trimming quoted text in replies, not double-spacing,
and paying for help.

ChrisA

 

[Cameron Simpson]

| > | Cameron would it be too much to ask to provide you with root
| > | access to my VPS server so you can have a look there too?
| > | i can pay you if you like if you wait a few days to gather some money.
| >
| > I really do not recommend that:
| > - it is nuts to blithely allow a stranger root access to your system
| > - you won't learn anything about CGI scripts
[...]
| I insist that you will make the most of this if you access the VPS yourself.
| it runs CentOS 6.4
| Please accept, i trust you.

Very well.

Let's take this off list to personal email (note that the reply-to
on this message is just myself, not the list/group).

We can return here after sorting CGI issues, should there be any further python
specific issues.

Reply to this message. I will email you my ssh public key. Please make me an
_ordinary_ user account called "cameron" and send me the ssh details of your VPS.

 

[Íßêïò Ãêñ33ê]

Ôç ÐÝìðôç, 18 Áðñéëßïõ 2013 2:00:48 ð.ì.. UTC+3, ï ÷ñÞóôçò Cameron Simpson Ýãñáøå:

Reply to this message. I will email you my ssh public key. Please make mean
_ordinary_ user account called "cameron" and send me the ssh details of your
VPS.

Thank you very much Cameron, i appreciate all your help and i'am willing toopen you a free lifetime premium account at my webhosting as a token of appreciation.

I have just mail you the login credentials.

 

[Íßêïò Ãêñ33ê]

Ôç ÐÝìðôç, 18 Áðñéëßïõ 2013 2:00:48 ð.ì.. UTC+3, ï ÷ñÞóôçò Cameron Simpson Ýãñáøå:

Reply to this message. I will email you my ssh public key. Please make mean
_ordinary_ user account called "cameron" and send me the ssh details of your
VPS.

Thank you very much Cameron, i appreciate all your help and i'am willing toopen you a free lifetime premium account at my webhosting as a token of appreciation.

I have just mail you the login credentials.