Dealing with marketing types...

[Stephen Kellett]

Sounds like someone has read about AJAX and decided that is what is
next. They probably put 2 and 2 together and came up with 5 thinking the
J stands for Java rather than Javascript and that your sever end must be
Java. Well if they really want performance play the C++ (or assembler!)
trump card and watch them squirm

I think you best approach is some serious education of upper management
on the benefits and pitfalls of each language, and of switching
languages. Also point out the huge costs:

o Total write-off of all development costs of V1.0.

o Total write off of all intellectual property assets of V1.0 (well if
you are building V2.0 on something else, you've put V1.0 in the bin with
zero re-use)

o Total slap in the face and moral-crusher to the development team and
support staff for V1.0. You will most likely see an exodus of talented
staff after the change, if it happens.

o Effectively starting from ground-zero, making the cost for
implementing V2.0 the entire development cost, rather than the
incremental cost for the jump to V2.0 from V1.0 using the existing
language.

The costs in human, timescale and financial terms for what these people
are proposing are huge. This company may not survive the change. If they
change you may want to consider the "abandon ship" approach and find a
more reliable place to devote you skills to.

Finally, read "The Peter Principle" and realise there are people like
these with their sights set on getting to the top of the greasy pole
without any consideration for the damage they cause to others. You need
to identify such people and steer clear of them (they generally do not
infect all companies).

All the best.

Stephen

 

[Terry Reedy]

http://naeblis.cx/rtomayko/2005/05/28/ibm-poop-heads

which is probably what you meant.

Thanks for digging this up. It solidified my understanding of why LAMP.

TJR

 

[Paul Rubin]

Thanks for digging this up. It solidified my understanding of why LAMP.

That article makes a lot of bogus claims and is full of hype. LAMP is
a nice way to throw a small site together without much fuss, sort of
like fancy xerox machines are a nice way to print a small-run
publication without much fuss. If you want to do something big, you
still need an actual printing press. Hint: Google.com is not a LAMP
site despite that article's insinuation. Yes, Google uses Python for
lots of internal projects and maybe administrative pages. The search
engine is not a Python app.

 

[Philippe C. Martin]

This is the never ending story of the cyclic (I'm being redundant) life
cycle of many companies: R&D driven versus Marketing driver.

My belief is that none work as the trades do not attempt to reach the same
goal:
1) R&D should not try to define products
2) Marketing should not try to impose the tools/means necessary to implement
the products they have defined.

I know that does not help

 

[Drazen Gemic]

The sequence goes like this:
1) When there is little or no money to be made, you start out with an
implied status as a partner. This means you work long + extra hours for
little pay on the promise that you will be rewarded when/if success comes.

Well, customers are NOT partners. You need a contract when you deal with
customers. It is not a team work, it is selling and buying. And you are
selling. This is a third year that I am working as a freelance, and,
during unofficial conversations with customers representatives,
I have heard many phrases like:
"I am your ally", "covering your back", "we are the team", and I never fell
for any of them. I agreed in general, never been impolite, but went on my
way. And I never forgot, neither let customers forget, that I am selling, and they are
buying, and we are not the team.

DG

 

[Andrew Dalke]

That article makes a lot of bogus claims and is full of hype. LAMP is
a nice way to throw a small site together without much fuss, sort of
like fancy xerox machines are a nice way to print a small-run
publication without much fuss. If you want to do something big, you
still need an actual printing press.

In the comments the author does say he's trying to be provocative.

My question to you is - what is "something big"? I've not been
on any project for which "LAMP" can't be used, and nor do I
expect to be. After all, there's only about 100,000 people in
the world who might possibly interested using my software. (Well,
the software I get paid to do; not, say, the couple of patches I've
sent in to Python).

I had one client consider moving from Python/CGI/flat files to
Java/WebLogic/Oracle. The old code took nearly 10 seconds to
display a page (!). They were convinced that they had gone past
the point where Python/CGI was useful, and they needed to use a
more scalable enterprise solution. The conviction meant they
didn't profile the system. With about a day of work I got the
performance down to under a second by removing some needless imports,
delaying others until they were needed, making sure all the
..pyc files existed, etc.

I could have gotten more performance switching to a persistent
Python web server and using a database instead of a bunch of
flat files in a directory, but that wasn't worth the time.

Andrew
(e-mail address removed)

 

[Paul Rubin]

My question to you is - what is "something big"? I've not been
on any project for which "LAMP" can't be used, and nor do I
expect to be. After all, there's only about 100,000 people in
the world who might possibly interested using my software. (Well,
the software I get paid to do; not, say, the couple of patches I've
sent in to Python).

If you're running a web site with 100k users (about 1/3 of the size of
Slashdot) that begins to be the range where I'd say LAMP starts
running out of gas. Yes, Slashdot is a LAMP site, but it's split
across a rack full of servers and is spending kilobucks a month on
colo space and hosting fees. Other similarly sized sites face similar
expenses. It seems to me that by using implementation methods that
map more directly onto the hardware, a site with Slashdot's traffic
levels could run on a single modest PC (maybe a laptop). I believe
LiveJournal (which has something more like a million users) uses
methods like that, as does ezboard. There was a thread about it here
a year or so ago.

As a simple example, that article's advice of putting all fine grained
session state into the database (so that every single browser hit sets
off SQL queries) is crazy. One site I worked on got a huge speedup by
simply storing the most frequently used stuff from the user session in
a browser cookie. That required zero extra work to handle multiple
servers (whichever server got the query, got the cookie) and it saved
a ton of SQL traffic.

As for "big", hmm, I'd say as production web sites go, 100k users is
medium sized, Slashdot is "largish", Ebay is "big", Google is huge.

 

[Andrew Dalke]

Paul Rubin replied to me:

If you're running a web site with 100k users (about 1/3 of the size of
Slashdot) that begins to be the range where I'd say LAMP starts
running out of gas.

Let me elaborate a bit. That claim of 100K from me is the
entire population of people who would use bioinformatics or
chemical informatics. It's the extreme upper bound of the
capacity I ever expect. It's much more likely I'll only
need to handle a few thousand users.

I believe
LiveJournal (which has something more like a million users) uses
methods like that, as does ezboard. There was a thread about it here
a year or so ago.

I know little about it, though I read at
http://goathack.livejournal.org/docs.html
] LiveJournal source is lots of Perl mixed up with lots of MySQL

I found more details at
http://jeremy.zawodny.com/blog/archives/001866.html

It's a bunch of things - Perl, C, MySQL-InnoDB, MyISAM, Akamai,
memcached. The linked slides say "lots of MySQL usage."
60 servers.

I don't see that example as validating your statement that
LAMP doesn't scale for mega-numbers of hits any better than
whatever you might call "printing press" systems.

As a simple example, that article's advice of putting all fine grained
session state into the database (so that every single browser hit sets
off SQL queries) is crazy.

To be fair, it does say "database plus cache" though the author
suggests the place for the cache is at the HTTP level and not
at the DB level. I would have considered something like memcached
perhaps backed by an asychronous write to a db if you want the
user state saved even after the cache is cleared/reset.

How permanent though does the history need to be? Your
approach wipes history when the user clears the cookie and it
might not be obvious that doing so should clear the history.

In any case, the implementation cost for this is likely
higher than what you did. I mention it to suggest an
alternative.

As for "big", hmm, I'd say as production web sites go, 100k users is
medium sized, Slashdot is "largish", Ebay is "big", Google is huge.

I'ld say that few sites have >100k users, much less
daily users with personalized information. As a totally made-up
number, only few dozens of sites (maybe a couple hundred?) would
need to worry about those issues.

If that's indeed the case then I'll also argue that each of
them is going to have app-specific choke points which are best
hand-optimized and not framework optimized. Is there enough
real-world experience to design a EnterpriseWeb-o-Rama (your
"printing press") which can handle those examples you gave
any better than starting off with a LAMP system and hand-caching
the parts that need it?

Andrew
(e-mail address removed)

 

[Paul Rubin]

I know little about it, though I read at
http://goathack.livejournal.org/docs.html
] LiveJournal source is lots of Perl mixed up with lots of MySQL

I found more details at
http://jeremy.zawodny.com/blog/archives/001866.html

It's a bunch of things - Perl, C, MySQL-InnoDB, MyISAM, Akamai,
memcached. The linked slides say "lots of MySQL usage." 60 servers.

LM uses MySQL extensively but what I don't know is whether it serves
up individual pages by the obvious bunch of queries like a smaller BBS
might. I have the impression that it's more carefully tuned than that.

I don't see that example as validating your statement that
LAMP doesn't scale for mega-numbers of hits any better than
whatever you might call "printing press" systems.

What example? Slashdot? It uses way more hardware than it needs to,
at least ten servers and I think a lot more. If LJ is using 6x as
many servers and taking 20x (?) as much traffic as Slashdot, then LJ
is doing something more efficiently than Slashdot.

How permanent though does the history need to be? Your
approach wipes history when the user clears the cookie and it
might not be obvious that doing so should clear the history.

The cookie is set at user login and it only has to persist through the
login session. It's not as if the info only exists in the cookie and
nowhere else.

I'ld say that few sites have >100k users, much less
daily users with personalized information. As a totally made-up
number, only few dozens of sites (maybe a couple hundred?) would
need to worry about those issues.

Yes, but for those of us interested in how big sites are put together,
those are the types of sites we have to think about ;-). I'd say
there's more than a few hundred of them, but it's not like there's
millions. And some of them really can't afford to waste so much
hardware--look at the constant Wikipedia fundraising pitches for more
server iron because the Wikimedia software (PHP/MySQL, natch) can't
handle the load.

If that's indeed the case then I'll also argue that each of
them is going to have app-specific choke points which are best
hand-optimized and not framework optimized. Is there enough
real-world experience to design a EnterpriseWeb-o-Rama (your
"printing press") which can handle those examples you gave
any better than starting off with a LAMP system and hand-caching
the parts that need it?

Yes, of course there is. Look at the mainframe transaction systems of
the 60's-70's-80's, for example. Look at Google. Then there's the
tons of experience we all have with LAMP systems. By putting some
effort into seeing where the resources in those things go, I believe
we can do a much better job. In particular, those sites like Slashdot
are really not update intensive in the normal database sense. They
can be handled almost entirely with some serial log files plus some
ram caching. At that point almost all the SQL overhead and a lot of
the context switching can go away.

 

[Kay Schluehr]

With Java I depend very little on customers IT staff, sysadmins, etc. If
I need additional functionality in form library, extension, whatever, all
I need is to drop another JAR in, and that does it.

Maybe this is for you?

http://peak.telecommunity.com/DevCenter/PythonEggs

Kay

 

[Mike Meyer]

Paul Rubin replied to me:
I'ld say that few sites have >100k users, much less
daily users with personalized information. As a totally made-up
number, only few dozens of sites (maybe a couple hundred?) would
need to worry about those issues.

I'd say quite a *lot* of sites have >100k users. A small client of
mine was a (now defunct .com) that was focused on "community
building". They had a user base of a couple of million people, and
you've probably never heard of The Park. They ran six servers,
thousands of simultaneous users, and it was all built on LAMP.

If you go looking for sites that offer the same kinds of things they
did - free web hosting, free web-based email, web-based chat,
calendering services, etc., you'll find a lot of such sites, and they
all probably have more than 100K users.

Of course, when you're talking about millions of web sites, a "few
sites" could be a a fairly large number of them.

An article I read recently made the point that I think you're trying
to make. The author argued that for most sites, scalability just
wasn't that big an issue. Web sites are cheap enough that they are
affordable to relatively small communities, and in many cases a
service that would bomb if they tried to go global with it would be a
big success in a small community. As such, he expects the web to be
dominated by sites that are really only of interest to a small
community. For those sites, LAMP will work just fine.

<mike

 

[Steve Jorgensen]

....

Let me add an Item #3 -
If you have some entrepeneurial savvy and can keep your emotions out of
it tou can simply tell them you have decided strike out on your own and
tell them that you will be available. They will be happy to hear you are
leaving and happier still to hear you can be available for backup.
Their goals and fears are addressed at the same time. AND there is a very
high possibility that they will *need* you at a later date for which you
can charge them dearly.

That last item #3 has actually worked for me with (2) prior employers.
I did have to eat my indignation and keep it friendly but it did pay off
in the end.
Thomas Bartkus

I have to say that, although I have yet to write a line of Python code for
money, I've found that this concept basically works. When you realize that
your employer is cutting you out to save the cost of paying you, funny enough,
they'll be willing to -really- pay you as a consultant later when they get
stuck, and one or more paying customers are impacted. They also win't mind
figuring out how to have you come in after hours so it won't conflict with
your new day job if you have one. In my case, the work was in VB/VBA, but the
same principle should apply to any technology.

Do make sure that your contract with any new employer allows you to do at
least small amounts of moonlighting - they probably won't object. They will
insist that any moonlighting shall not compete with their line of business,
and you should agree to that stipulation.

 

[Aahz]

What example? Slashdot? It uses way more hardware than it needs to,
at least ten servers and I think a lot more. If LJ is using 6x as
many servers and taking 20x (?) as much traffic as Slashdot, then LJ
is doing something more efficiently than Slashdot.

So what? I think you're missing the real point of the article: using
LAMP scales *DOWN* in a way that enterprise systems don't. Getting your
first prototype up and running is far more important than sheer
scalability, and LAMP does have many mechanisms to obtain scalability
when it's needed.

 

[Terry Reedy]

Yes, of course there is. Look at the mainframe transaction systems of
the 60's-70's-80's, for example. Look at Google.

Based on what I've read, if we could look at Google, we would see 150,000
to 200,000 servers (about half bought with IPO money). We would see a
highly customized dynamic cluster computing infrastructure that can be
utilized with high-level (Python-like) commands. The need to throw
hundreds of machines at each web request strikes me as rather specialized,
though definitely not limited to search. So while not LAMP, I don't see it
as generic EWeboRama either.

Terry J. Reedy

 

[Paul Rubin]

So what? I think you're missing the real point of the article: using
LAMP scales *DOWN* in a way that enterprise systems don't. Getting your
first prototype up and running is far more important than sheer
scalability,

There comes a day when your first prototype can no longer handle the
traffic. The question then is how do you deal with that.

and LAMP does have many mechanisms to obtain scalability when it's
needed.

LAMP seems to have no solution other than scaling (i.e. blowing more
money on hardware and colo space). One really gets the impression
that a more thoughtful design could handle the traffic without needing
to scale the hardware.

 

[Andrew Dalke]

LM uses MySQL extensively but what I don't know is whether it serves
up individual pages by the obvious bunch of queries like a smaller BBS
might. I have the impression that it's more carefully tuned than that.

The linked page links to a PDF describing the architecture.
The careful tuning comes in part from a high-performance caching
system - memcached.

What example? Slashdot?

Livejournal. You gave it as a counter example to the LAMP
architecture used by /.

] It seems to me that by using implementation methods that
] map more directly onto the hardware, a site with Slashdot's
] traffic levels could run on a single modest PC (maybe a laptop).
] I believe LiveJournal (which has something more like a million
] users) uses methods like that, as does ezboard.

Since LJ uses a (highly hand-tuned) LAMP architecture, it isn't
an effective counterexample.

It uses way more hardware than it needs to,
at least ten servers and I think a lot more. If LJ is using 6x as
many servers and taking 20x (?) as much traffic as Slashdot, then LJ
is doing something more efficiently than Slashdot.

I don't know where the 20x comes from. Registered users? I
read /. but haven't logged into it in 5+ years. I know I
hit a lot /. more often than I do LJ (there's only one diary
I follow there). The use is different as well; all people
hit one story / comments page, and the comments are ranked
based on reader-defined evaluations. LJ has no one journal
that gets anywhere as many hits and there is no ranking scheme.

Yes, but for those of us interested in how big sites are put together,
those are the types of sites we have to think about ;-).

My apologies since I know this sounds snide, but then why didn't
you (re)read the LJ architecture overview I linked to above?
That sounds like something you would have been interested in
reading and would have directly provided information that
counters what you said in your followup.

The "ibm-poop-heads" article by Ryan Tomayko gives pointers to
several other large-scale LAMP-based web sites. You didn't
like the Google one. I checked a couple of the others:

IMDB -
http://www.findarticles.com/p/articles/mi_zdpcm/is_200408/ai_ziff130634
As you might expect, the site is now co-located with other Amazon.com
sites, served up from machines running Linux and Apache, but ironically,
most of the IMDb does not use a traditional database back end. Its
message boards are built on PostgreSQL, and certain parts of IMDb
Pro-including its advanced search-use MySQL, but most of the site is
built with good old Perl script.

del.icio.us
Took some digging but I found
http://lists.del.icio.us/pipermail/discuss/2004-November/001421.html
"The database gets corrupted because the machine gets power-cycled,
not through any fault of MySQL's."

The point is that LAMP systems do scale, both down and up. It's
a polemic against "architecture astronauts" who believe the only
way to handle large sites (and /., LJ, IMDB, and del.icio.us are
larger than all but a few sites) is with some spiffy "enterprise"
architecture framework.

I'd say
there's more than a few hundred of them, but it's not like there's
millions. And some of them really can't afford to waste so much
hardware--look at the constant Wikipedia fundraising pitches for more
server iron because the Wikimedia software (PHP/MySQL, natch) can't
handle the load.

Could that have, for example, bought EnterpriseWeb-O-Rama and done
any better/cheaper? Could they have even started the project
had they gone that route?

Yes, of course there is [exprience in large-scale web apps].
Look at the mainframe transaction systems of the 60's-70's-80's, for
example. Look at Google.

For the mainframe apps you'll have to toss anything processed
in batch mode, like payrolls. What had the customization level
and scale comparable to 100K+ sites of today? ATMs? Stock trading?

Google is a one-off system. At present there's no other system
I know of - especially one with that many users - where a single
user request can trigger searches from hundreds of machines.
That's all custom software. Or should most servers implement
what is in essence a new distributed operating system just to
run a web site?

Then there's the tons of experience we all have with LAMP systems. By
putting some effort into seeing where the resources in those things go,
I believe we can do a much better job. In particular, those sites like
Slashdot are really not update intensive in the normal database sense.
They can be handled almost entirely with some serial log files plus some
ram caching. At that point almost all the SQL overhead and a lot of the
context switching can go away.

Is /. an appropriate comparable? I get the idea that it hasn't
changed much in the last, say, 5 years and the user base hasn't
grown much either. What you propose requires programming effort.
If the system doesn't need work, if money in > money out (even with
expensive hardware), and if the extra work doesn't get much benefit,
then is it worthwhile to them to rearchitect the system?

Perhaps in a couple years it'll run on two machines (one as the
backup), with no change to the code, and simply because the
hardware is good enough and cheap enough.

Andrew
(e-mail address removed)

 

[Aahz]

There comes a day when your first prototype can no longer handle the
traffic. The question then is how do you deal with that.

Then you scale with hardware or do profiling and rewrite chunks,
whichever costs less FOR THE BUSINESS.

LAMP seems to have no solution other than scaling (i.e. blowing more
money on hardware and colo space). One really gets the impression
that a more thoughtful design could handle the traffic without needing
to scale the hardware.

Is there some reason you keep repeating yourself without actually paying
attention to other people?

 

[Thomas Bartkus]

...

I have to say that, although I have yet to write a line of Python code for
money, I've found that this concept basically works. When you realize that
your employer is cutting you out to save the cost of paying you, funny enough,
they'll be willing to -really- pay you as a consultant later when they get
stuck, and one or more paying customers are impacted.

Yup! It's theold stupid + greedy double whammy that means they end up paying
more.
Your not feeling sorry for them, are you?

They also win't mind
figuring out how to have you come in after hours so it won't conflict with
your new day job if you have one. In my case, the work was in VB/VBA, but the
same principle should apply to any technology.

Do make sure that your contract with any new employer allows you to do at
least small amounts of moonlighting - they probably won't object. They will
insist that any moonlighting shall not compete with their line of business,
and you should agree to that stipulation.

How much of *my* time are they buying with a salary? 40Hrs a week? 24/7 ?
You want to see that your contract as an employee doesn't somehow forbid you
from earning extra on your own time. Unless, of course, they are paying
enough to make you happy to sell them *all* your time. Sometimes you are
hired mainly to keep your skills unavailable to their competitors. Thats ok
as long as they pay you enough to keep you happy with this. Unless they are
paying for it, your own free time is none of their business.

Thomas Bartkus

 

[fuzzylollipop]

I agree about the stiffing the guys that brought you to the party, that
was 100% the DotCom plan, offer crap salary and tonnes of "options"
then fire/replace all the people that worked for _FREE_ practically
when the next round of funding comes in, rinse, repeat.

Either way . . . I think the guy needs to move on. I know I would.

 

[fuzzylollipop]

man this is the worst advice I have ever heard, you can't "walk away
with code" someone else paid you to write. Regardless of what your
perceived slight is.

NEVER take code you were paid to write unless you have it in writing
that you can, you will lose that one everytime.