data: protocol

[Robin Becker]

I have an outstanding request for ReportLab to allow images to be opened using
the data: scheme. That used to be supported in python 2.7 using urllib, but in
python 3.3 urllib2 --> urllib and at least the default urlopener doesn't support
data:

Is there a way to use the residual legacy of the old urllib code that's now in
urllib.URLopener to open unusual schemes? I know it can be used directly eg

urllib.request.URLopener().open('data:.........')

but that seems to leave the splitting & testing logic up to me when it logically
belongs in some central place ie urllib.request.urlopen.

 

[Steven D'Aprano]

I have an outstanding request for ReportLab to allow images to be opened
using the data: scheme. That used to be supported in python 2.7 using
urllib, but in python 3.3 urllib2 --> urllib and at least the default
urlopener doesn't support data:

It looks like you intended to show an example, but left it out.

Is there a way to use the residual legacy of the old urllib code that's
now in urllib.URLopener to open unusual schemes? I know it can be used
directly eg

urllib.request.URLopener().open('data:.........')

but that seems to leave the splitting & testing logic up to me when it
logically belongs in some central place ie urllib.request.urlopen.

You may need to explain in a little more detail. When you say "splitting
and testing", what are you splitting and testing? It may also help if you
show some Python 2.7 code that works, and what happens in 3.3.

 

[Robin Becker]

It looks like you intended to show an example, but left it out.


You may need to explain in a little more detail. When you say "splitting
and testing", what are you splitting and testing? It may also help if you
show some Python 2.7 code that works, and what happens in 3.3.

OK not sure about 3.4, but in 3.3 the urllib module cannot open a request like this

C:\code-trunk\hg-repos\reportlab\tests>\python33\python.exe
Python 3.3.3 (v3.3.3:c3896275c0f6, Nov 18 2013, 21:18:40) [MSC v.1600 32 bit
(Intel)] on win32
Type "help", "copyright", "credits" or "license" for more information.urllib.request.urlopen('data:image/gif;base64,R0lGODdhAQABAIAAAP///////ywAAAAAAQABAAACAkQBADs=').read()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "C:\python33\lib\urllib\request.py", line 156, in urlopen
return opener.open(url, data, timeout)
File "C:\python33\lib\urllib\request.py", line 469, in open
response = self._open(req, data)
File "C:\python33\lib\urllib\request.py", line 492, in _open
'unknown_open', req)
File "C:\python33\lib\urllib\request.py", line 447, in _call_chain
result = func(*args)
File "C:\python33\lib\urllib\request.py", line 1310, in unknown_open
raise URLError('unknown url type: %s' % type)

in python27 one can do

C:\tmp>python
Python 2.7.6 (default, Nov 10 2013, 19:24:18) [MSC v.1500 32 bit (Intel)] on win32
Type "help", "copyright", "credits" or "license" for more information.
and as indicated by Ian Kelly in 3.4
C:\tmp>\python34\python.exe
Python 3.4.0 (v3.4.0:04f714765c13, Mar 16 2014, 19:24:06) [MSC v.1600 32 bit
(Intel)] on win32
Type "help", "copyright", "credits" or "license" for more information.35



in 3.3 we have the old code URLopener class. However, when I use that I see this

C:\code-trunk\hg-repos\reportlab\tests>\python33\python.exe
Python 3.3.3 (v3.3.3:c3896275c0f6, Nov 18 2013, 21:18:40) [MSC v.1600 32 bit
(Intel)] on win32
Type "help", "copyright", "credits" or "license" for more information.'Date: Thu, 08 May 2014 10:21:45 GMT\nContent-type: image/gif\nContent-Length:
35\n\nGIF87a\x01\x00\x01\x00\x80\x00\x00├
â”├â”├â”├â”├â”├â”,\x00\x00\x00\x00\x01\x00\x01\x00\x00\x02\x02D\x01\x00;'
so I seem to be getting the real data and some headers now. I think this is
different from what is expected, but that code is labelled as old/deprecated and
possibly going away.

Since urllib doesn't always work as expected in 3.3 I've had to write a small
stub for the special data: case. Doing all the splitting off of the headers
seems harder than just doing the special case.

However, there are a lot of these 'schemes' so should I be doing this sort of
thing? Apparently it's taken 4 versions of python to get urllib in 3.4 to do
this so it's not clear to me whether all schemes are supposed to hang off
urllib.request.urlopen or if instead of special casing the 3.3 data: I should
have special cased a handler for it and injected that into my opener (or
possibly the default opener). Doing the handler means I do have to handle the
headers stuff whereas my stub is just returning the data bits.

 

[Ian Kelly]

Since urllib doesn't always work as expected in 3.3 I've had to write a
small stub for the special data: case. Doing all the splitting off of the
headers seems harder than just doing the special case.

However, there are a lot of these 'schemes' so should I be doing this sort
of thing? Apparently it's taken 4 versions of python to get urllib in 3.4 to
do this so it's not clear to me whether all schemes are supposed to hang off
urllib.request.urlopen or if instead of special casing the 3.3 data: I
should have special cased a handler for it and injected that into my opener
(or possibly the default opener). Doing the handler means I do have to
handle the headers stuff whereas my stub is just returning the data bits.

The DataHandler class in 3.4 is all of 14 lines of code. My first
instinct would be to backport that and add it to some OpenerDirector
instance.

 

[Robin Becker]

On 08/05/2014 16:07, Ian Kelly wrote:
.........

The DataHandler class in 3.4 is all of 14 lines of code. My first

which makes me wonder why the old URLopener is still there and why all this
wasn't done before 3.0 appeared.

instinct would be to backport that and add it to some OpenerDirector
instance.

Using a director would be possible, but it's not much cleaner than special
casing the function and I don't have to worry about which opener to use; I use
the one provided by the user