Oh look, another language (ceylon)

[Dave Angel]

- 15 bits for a length.

15 bits give you a maximum length of 32767. There are ways around that.
E.g. a length of 0 through 32766 means exactly what it says; a length of
32767 means that the next two bytes are part of the length too, giving
you a maximum of 4294967295 characters per string. That's an 8GB string.
Surely big enough for anyone

If you use nearly all of the possible 2 byte values then adding 2
more bytes won't give you anywhere near 4 bI'll ion characters.
You're perhaps thinking of bringing in four more bytes when the
length exceeds 32k.

 

[Chris Angelico]

I suppose that's not terrible, except for the O(n) string operations
which is just dumb. Yes, it's better than buggy, broken strings. But
still dumb, because those aren't the only choices. For example, for the
sake of an extra two bytes at the start of each string, they could store
a flag and a length:

True, but I suspect that _any_ variance from JS strings would have
significant impact on the performance of everything that crosses the
boundary. If anything, I'd be looking at a permanent 32-bit shim on
the string (rather than the 16-or-32-bit that you describe, or the
16-or-48-bit that Dave clarifies your theory as needing); that would
allow strings up to 2GB (31 bits of pure binary length), and exceeding
that could just raise a RuntimeError. Then, passing any string to a JS
method would simply mean trimming off the first two code units.

But the problem is also with strings coming back from JS. Every time
you get something crossing from JS to Ceylon, you have to walk it,
count up its length, and see if it has any surrogates (and somehow
deal with mismatched surrogates). Every string, even if all you're
going to do is give it straight back to JS in the next line of code.
Potentially quite expensive, and surprisingly so - as opposed to
simply saying "string indexing can be slow on large strings", which
puts the cost against a visible line of code.

ChrisA

 

[Tim Daneliuk]

can I safely assume that we'll be seeing a PEP fairly shortly?

For Immediate Press Release:


We at TundraWare are now entering our 10th year of debate in the YAPDL
design as to what ought to be a statement and what ought to be a function.
The Statementists are currently winning 3 bouts to 2 over the
Functionists but there is much more gnashing of teeth and wringing of
hands to come. We remain true to the original vision of the language as
an unwanted appendage to Python which will promote fractionalisation and
thus improve opportunity for future billings.

We are also contemplating an offshoot language that melds the best of Java
into YAPDL. Known as JAPDL ("Jah.piddle") it is targeted particularly
to Rastafri programmers worldwide. The primary contribution of JAPDL
to the language arts is the replacement of the GIL (Global Interpreter Lock)
with the much simpler, DR (Dread Lock).

 

[Steven D'Aprano]

But the problem is also with strings coming back from JS.

Just because you call it a "string" in Ceylon, doesn't mean you have to
use the native Javascript string type unchanged.

Since the Ceylon compiler controls what Javascript operations get called
(the user never writes any Javascript directly), the compiler can tell
which operations potentially add surrogates. Since strings are immutable
in Ceylon, a slice of a BMP-only string is also BMP-only; concatenating
two BMP-only strings gives a BMP-only string. I expect that uppercasing
or lowercasing such strings will also keep the same invariant, but if
not, well, you already have to walk the string to convert it, walking it
again should be no more expensive.

The point is not that my off-the-top-of-my-head pseudo-implementation was
optimal in all details, but that *text strings* should be decent data
structures with smarts, not dumb arrays of variable-width characters. If
that means avoiding dumb-array-of-char naive implementations, and writing
your own, that's part of the compiler writers job.

Python strings can include null bytes, unlike C, even when built on top
of C. They know their length, unlike C, even when built on top of C. Just
because the native Java and Javascript string types doesn't do these
things, doesn't mean that they can't be done in Javascript.

- as opposed to simply saying "string
indexing can be slow on large strings", which puts the cost against a
visible line of code.

For all we know, Ceylon already does something like this, but merely
doesn't advertise the fact that while it *can* be slow, it can *also* be
fast. It's an implementation detail, perhaps, much like string
concatenation in Python officially requires building a new string, but in
CPython sometimes it can append to the original string.


Still, given that Pike and Python have already solved this problem, and
have O(1) string indexing operations and length for any Unicode string,
SMP and BMP, it is a major disappointment that Ceylon doesn't.

 

[Steven D'Aprano]

If you use nearly all of the possible 2 byte values then adding 2 more
bytes won't give you anywhere near 4 bI'll ion characters. You're
perhaps thinking of bringing in four more bytes when the length exceeds
32k.

Yep, I screwed up. Thanks for the correction.

 

[Chris Angelico]

Just because you call it a "string" in Ceylon, doesn't mean you have to
use the native Javascript string type unchanged.

Indeed not, but there are going to be many MANY cases where a JS
string has to become a Ceylon string and vice versa - a lot more often
than CPython drops to C. For instance, suppose you run your Ceylon
code inside a web browser. Pick up pretty much any piece of JavaScript
code from any web page - how much string manipulation does it do, and
how much does it call on various DOM methods? In CPython, only a small
number of Python functions will end up dropping to C APIs to do their
work (and most of those will have to do some manipulation along the
way somewhere - eg chances are print()/sys.stdout.write() will
eventually have to encode its output to 8-bit before passing it to
some byte-oriented underlying stream, so the actual representation of
a Python string doesn't matter); in browser-based work, that is
inverted.

However, Ceylon can actually be implemented on multiple backends (Java
and JavaScript listed). It's fully possible that an
"application-oriented" backend might use Pike-strings internally,
while a "browser-oriented" backend could still use the underlying
string representation. The questions are entirely of performance,
since it's been guaranteed already to have the same semantics.

I would really like to see JavaScript replaced in web browsers, since
the ECMAScript folks have stated explicitly (in response to a question
from me) that UTF-16 representation *must* stay, for backward compat.
JS is a reasonable language - it's not terrible - but it has a number
of glaring flaws. Ceylon could potentially be implemented in browsers,
using Pike-strings internally, and then someone could write a
JavaScript engine that compiles to Ceylon (complete with
bug-compatibility stupid-code that encodes all strings UTF-16 before
indexing into them). It would be an overall improvement, methinks.

ChrisA

 

[Chris Angelico]

Still, given that Pike and Python have already solved this problem, and
have O(1) string indexing operations and length for any Unicode string,
SMP and BMP, it is a major disappointment that Ceylon doesn't.

And of course, the part that's "solved" here is not the O(1) string
indexing, but the fact that UTF-32 semantics with less memory usage
than UTF-16. It's easy to get perfect indexing semantics if you don't
mind how much space your strings take up.

ChrisA

 

[Rick Johnson]

I've never *really* been crazy about the plus operator
concatenating strings anyhow, however, the semantics of "+"
seem to navigate the "perilous waters of intuition" far
better than "*".

Addition of numeric types is well defined in maths:
Take N inputs values and *reduce* them into a single
value that represents the mathematical summation of
all inputs.

HOWEVER,

Addition of strings (concatenation) requires
interpreting the statement as a more simplistic
"joining" process of : take N inputs and join them
together in a *linear fashion* until they become a
single value.

As you might already know the latter is a far less elegant
process, although the transformation remains "sane". Even
though in the first case: with "numeric addition", the
individual inputs are *sacrificed* to the god of maths; and
in the second case: for "string addition", the individual
inputs are *retained*; BOTH implementations of the plus
operator expose a CONSISTENT interface -- and like any good
interface the dirty details are hidden from the caller!

INTERFACES ARE THE KEY TO CODE INTEGRITY and LONGEVITY!

HOWEVER, HOWEVER. O

There is an inconsistency when applying the "*" operator
between numerics and strings. In the case of numerics the
rules are widely understood and quite logical, HOWEVER, in
the case of "string products", not only are rules missing,
any attempt to create a rule is illogical, AND, we've broken
the consistency of the "*" interface!

py> "a" * "4"
'aaaa'

Okay, that makes sense, but what about:

py> "a" * "aaaa"

That will haunt your nightmares!

But even the previous example, whilst quite logical, is
violating the "contract of transformations" and can ONLY
result in subtle bugs, language designer woes, and endless
threads on Pyhon-ideas that result in no elegant solution.

THERE EXISTS NO PATH TO ELEGANCE VIA GLUTTONY

Operator overloading must be restricted. Same goes for
syntactic sugars. You can only do SO much with a sugar
before it mutates into a salt.

TOO MUCH OF A GOOD THING... well, ya know!

 

[Steven D'Aprano]

I've never *really* been crazy about the plus operator concatenating
strings anyhow, however, the semantics of "+" seem to navigate the
"perilous waters of intuition" far better than "*".

Addition of numeric types is well defined in maths: Take N inputs
values and *reduce* them into a single value that represents the
mathematical summation of all inputs.

Which sum would that be?

Addition of vectors, matrices, quaternions, tensors, something else?

Do you perhaps mean the Whitney Sum?
http://mathworld.wolfram.com/WhitneySum.html

Ah, no, you're talking about addition of Real numbered values, where
nothing can *possibly* go wrong:

py> 0.1 + 0.1 + 0.1 == 0.3
False

Hmmm. Oh well, at least we know that adding 1 to a number is guaranteed
to make it bigger:

py> 1e16 + 1 > 1e16
False

Surely though, the order you do the addition doesn't matter:

py> 1.5 + (1.3 + 1.9) == (1.5 + 1.3) + 1.9
False


Dammit maths, why do you hate us so???


So, explain to me again, what is the *precise* connection between the
mathematical definition of addition, as we learn about in school, and
what computers do?

HOWEVER,

Addition of strings (concatenation) requires interpreting the
statement as a more simplistic "joining" process of : take N inputs
and join them together in a *linear fashion* until they become a
single value.

Ah, you mean like addition in base-1, otherwise known as the unary number
system, also known as a tally.

So if you want to add (decimal) 3 and 5 using base-1, we would write:

||| + |||||

and concatenating the tallies together gives:

||||||||


which if I'm not mistaken makes 8 in decimal.

There is an inconsistency when applying the "*" operator between
numerics and strings. In the case of numerics the rules are widely
understood and quite logical, HOWEVER, in the case of "string products",
not only are rules missing, any attempt to create a rule is illogical,
AND, we've broken the consistency of the "*" interface!

A foolish consistency is the hobgoblin of little minds.

Just because you can't define a sensible meaning for str * str doesn't
mean you can't define a sensible meaning for str * int.

py> "a" * "4"
'aaaa'

Okay, that makes sense, but what about:

py> "a" * "aaaa"

That will haunt your nightmares!

You're easily terrified if you have nightmares about that. I can't
imagine what you would do if faced with the M-combinator applied to
itself.

But even the previous example, whilst quite logical, is violating the
"contract of transformations"

What contract of transformations?

 

[Chris Angelico]

You're easily terrified if you have nightmares about that. I can't
imagine what you would do if faced with the M-combinator applied to
itself.

Not to mention that he has to construct his own nightmares. This not
being PHP, it's unlikely to work quite the way he thinks it does:
Traceback (most recent call last):
File "<pyshell#51>", line 1, in <module>
"a" * "4"
TypeError: can't multiply sequence by non-int of type 'str'

Unless he has some strange Python interpreter that coalesces
integer-like strings to integers, of course, in which case I
completely understand why he's having nightmares.

ChrisA

 

[wxjmfauth]

Le lundi 18 novembre 2013 14:31:33 UTC+1, Steven D'Aprano a écrit :

... choose one of the three bad choices: ...



* choose UTF-16 or UTF-8, and have O(n) primitive string operations (like

Haskell and, apparently, Ceylon);



* or UTF-16 without support for the supplementary planes (which makes it

virtually UCS-2), like Javascript;



* choose UTF-32, and use two or four times as much memory as needed.

Nothing can beat the coding schemes endorsed by Unicode.

They are all working on the smallest possible entity
level (Unicode Transformation *Units*) with a unique
set of these entities.

To not forget. A set of characters is an artificial
construction and by nature it can not follow the
logic of a more "natural" set, eg. integers.

jmf

 

[Gregory Ewing]

Which sum would that be?

Addition of vectors, matrices, quaternions, tensors, something else?

Considering vectors, multiplying a vector by a scalar
can be thought of as putting n copies of the vector
together nose-to-tail.

That's not very much different from putting n copies
of a string one after another.

 

[Bob Martin]

Le lundi 18 novembre 2013 14:31:33 UTC+1, Steven D'Aprano a =E9crit=A0:

Nothing can beat the coding schemes endorsed by Unicode.

They are all working on the smallest possible entity
level (Unicode Transformation *Units*) with a unique
set of these entities.

To not forget.

Is that an egg-corn?