OT: why do web BBS's and blogs get so slow?

[Paul Rubin]

As of right now, says that > 217,000 users have updated their
livejournal in the last 24 hours. That is 217,000 database writes in
the last 24 hours. Nothing to shake a stick at.

Well, that depends on what the hardware is. Does anyone know?

However, most every post has the option of people commenting on
it. Those stats don't count comments, which all require a DB write.

Have you looked at the code and confirmed that each comment post
requires a DB write in the usual sense? Part of this discussion has been
about how to avoid DB writes for such common operations.

Or really, from my above numbers, > 200,000 visitors/day.

That really is pretty impressive, I had no idea LJ was that big. I
wonder how much bandwidth they're using, what their expenses are, how
many staff they have, how many paying users, etc.

Yeah. The trick is that most anything that is really intellectually
stimulating, is difficult to do. Solving the dynamic-page-generation
problem can be intellectually stimulating, but it is not easy. Adding
into it the page formatting with templates problem, database load,
bandwidth problem, etc. It starts getting unweildy very quickly.

I don't see any of this stuff as being terribly difficult, it's just
work, and I'm not currently involved in operating such a site, so it's
not something that I have a burning desire to work on for free. There
are times when I've been interested in running such a site, but the
headaches involved in that would probably be even worse than writing
the code.

LJ doesn't really have the right format for a discussion board, but
maybe it could be adapted.

 

[Leif K-Brooks]

Or even faster if "Hi <username>" was two gifs that could be
loaded.

Please, don't lose accessibility for some petty speed gain.

 

[Max M]

Please, don't lose accessibility for some petty speed gain.

I was talking about how to optimize for speed, and how designs can be
changed for the purpose of speed.

When you do that, somethings got to give.

Sometimes you don't have the technological possibility to do it right.
Then a "hack" like that can be in order.


regards Max M

 

[Aahz]

Well, that depends on what the hardware is. Does anyone know?

Go look at their site! ;-)

Have you looked at the code and confirmed that each comment post
requires a DB write in the usual sense? Part of this discussion has
been about how to avoid DB writes for such common operations.

They require writes at *some* point, because they're persistent. What
kind of system they use (if any) to batch things up for efficiency, I've
no idea.

That really is pretty impressive, I had no idea LJ was that big. I
wonder how much bandwidth they're using, what their expenses are, how
many staff they have, how many paying users, etc.

LJ is huge. I'd guess probably >50% of their active userbase is paying.

LJ doesn't really have the right format for a discussion board, [...]

Which is precisely why I said that I was loathe to mention it. ;-)
(I'm well-known in my social circle for my antipathy to LJ.)

Nevertheless, LJ is an excellent example of a high-performance web
application where the software is available as Open Source. You can
probably learn a lot if you dig into it.
--
Aahz ([email protected]) <*> http://www.pythoncraft.com/

"The joy of coding Python should be in seeing short, concise, readable
classes that express a lot of action in a small amount of clear code --
not in reams of trivial code that bores the reader to death." --GvR

 

[Paul Rubin]

I was talking about how to optimize for speed, and how designs can be
changed for the purpose of speed.

You could also do it with client side javascript in a static page:

<script>
document.write("hi " + get_username(document.cookie))
</script>

if the username is stored in a browser cookie. More generality you
can simulate a client side include:

<script src=username.js></script>

where the server responds to requests for username.js with

document.write("hi [username]")

after determining the username from the sent cookie. The server to
use cache control headers very aggressively to stop the browser from
showing the wrong name if someone logs off and then logs back in as
another user. I remember cackling somewhat madly when I came up with
this, because it was after having spent about a whole day coding up
generating a messy page dynamically and realizing I could do the above
instead (more complicated situation of course).

 

[Paul Rubin]

They require writes at *some* point, because they're persistent. What
kind of system they use (if any) to batch things up for efficiency, I've
no idea.

I think of "DB write" as something different than just appending
something to a file, i.e. a DB write means SQL parsing, transactional

LJ is huge. I'd guess probably >50% of their active userbase is paying.

That sounds that they can deal with scaling issues by just throwing
more hardware at the problem. Especially since I think their
high-traffic pages (like the home page) are static. The dynamic pages
are user journals which I think don't interact with each other that
much and which (unlike Slashdot's comment threads) don't individually
get very heavy traffic. They could basically have completely separate
servers for usernames beginning with "a", "b", "c", etc. and have a
simple load balancer redirecting browser requests.

Nevertheless, LJ is an excellent example of a high-performance web
application where the software is available as Open Source. You can
probably learn a lot if you dig into it.

Yeah, but I don't know if I could stomach looking at that much Perl code.

 

[Paul Rubin]

Go look at their site! ;-)

I took a quick look and didn't see anything specific about the hardware.
Got a url?

 

[Leif K-Brooks]

I was talking about how to optimize for speed, and how designs can be
changed for the purpose of speed.

When you do that, somethings got to give.

Sometimes you don't have the technological possibility to do it right.
Then a "hack" like that can be in order.

I would take a slightly slower site over one that I can't use any day.
How on earth do you expect blind people to use your site? What about
people with text browsers or images disabled?

 

[Josiah Carlson]

If you check their stats page (that I linked):
http://livejournal.com/stats

They have ~80,000 paid members. Even when buying in a yearly package,
that's around $2/month, which puts them at around $160k/month in
revenue, not including those that pay the additional ~$1/month for extra
livejournal icons.

That sounds that they can deal with scaling issues by just throwing
more hardware at the problem. Especially since I think their
high-traffic pages (like the home page) are static. The dynamic pages
are user journals which I think don't interact with each other that
much and which (unlike Slashdot's comment threads) don't individually
get very heavy traffic. They could basically have completely separate
servers for usernames beginning with "a", "b", "c", etc. and have a
simple load balancer redirecting browser requests.

This was discussed some months ago on the news journal:
http://livejournal.com/users/news

If you are interested, read up.

Yeah, but I don't know if I could stomach looking at that much Perl code.

I agree.

- Josiah

 

[Aahz]

I took a quick look and didn't see anything specific about the hardware.
Got a url?

Nope. I know as little about LJ as I can get away with. But the bits
I've seen indicate that they do a pretty good job of documentation; you
might have to poke around, but I'm sure it's there. Or just get an LJ
account and ask.
--
Aahz ([email protected]) <*> http://www.pythoncraft.com/

"The joy of coding Python should be in seeing short, concise, readable
classes that express a lot of action in a small amount of clear code --
not in reams of trivial code that bores the reader to death." --GvR

 

[Aahz]

I think of "DB write" as something different than just appending
something to a file, i.e. a DB write means SQL parsing, transactional
Dunno.


That sounds that they can deal with scaling issues by just throwing
more hardware at the problem. Especially since I think their
high-traffic pages (like the home page) are static. The dynamic pages
are user journals which I think don't interact with each other that
much and which (unlike Slashdot's comment threads) don't individually
get very heavy traffic. They could basically have completely separate
servers for usernames beginning with "a", "b", "c", etc. and have a
simple load balancer redirecting browser requests.

That's certainly possible. I believe that some community journals get
heavy traffic, though.

Yeah, but I don't know if I could stomach looking at that much Perl code.

Repeat my earlier comment with extra emphasis. ;-) ("I'm loathe...")
--
Aahz ([email protected]) <*> http://www.pythoncraft.com/

"The joy of coding Python should be in seeing short, concise, readable
classes that express a lot of action in a small amount of clear code --
not in reams of trivial code that bores the reader to death." --GvR