Really slow, painfullu slow asp.net web app

[michael]

Hi,

I am trying to write an ASP.NET web app, in Visual Basic.NET, using
Visual Studio.NET 2004, .NET framework 1.1.4322 SP1
Running the project/app on localhost while in dev/write/debug stage

When I say "trying", I do have it written, and it works ... sort of,
for some cases.
The problems/issues?

1. it runs very slowly
2. it seems to either hang up/lock up aspnet_wp.exe
3. or timesouts way to soon
4. if/when it hangs/locks aspnet_wp.exe, "End Process" in Task Manager
is only way to get aspnet_wp.exe to stop (and it restarts) thus
bringing VS.NET back into "write mode" from debug mode (closing web
page or hitting stop debugging in VS.NET does not actually detach/stop
aspnet_wp.exe)

In psuedo code form my app takes several images (ideally)
for each image
go through each pixel and gets its color
if the color for the pixel is already on a dataset.table then
increment the frequency count for that color (ARGB)
if not on the table then add that color (ARGB) to the
dataset.table
next pixel
when done with that image then store the values/rows (colors &
frequencies) in the SQL Server 2000 database

next image

FIRST - let me stress it is not pulling th data from SQL Server - it is
finding/crunching the data in the image and then when done processing
the image/pixels, then at that point it rights the results to database
SECOND - it is not the writing to the db that is slow - have, in
attempt to pin issues down commented out actual writing to db

If I run my app on an image of size 100x100 pixels it runs pretty fast
Can even do a batch with 3 or 4 smaller images

But if I run it on an image of say 600 x 600 pixels ... it just bogs
down.

I can understand a larger image would have more colors, and as it loops
through the pixels, it would take longer to sort the table for that
color (to see if it needs to add or increment)

But it just comes to a grinding halt.
As it never actually finishes a larger image don't lnow exactly how
many colors it is finding, and therefore how many rows the tabel has,
but near as I can tell, on one image there is about 70,000 colors,
meaning the table would have about 70,000 lines.

Is that too big? As in too many lines for ASP.NET to handle?

And if I either:
1. just check aspnet_wp.exe while running, or
2. close the web page, or
3. stop debugging in VS.net

Mem usage for aspnet_wp.exe is psikingat like 98% of memory.
Stopping debugging does not stop aspnet_wp.exe from spiking ... even if
I give the computer/VS>NET an hour or more to actually stop debugging.

I have to go into Task Manager and end process on aspnet_wp.exe then
vs.net actually stops debugging.

I did write sort of a prototype of this as a windows app and it goes
very fast, even with very large files & a large number of files (and my
code for this protoype is way less efficient)

I would expect some slowness moving to a web app, but not for it to
just simply hang (and if I let it run long enough ... it just times out
.... with aspnet_wp.exe spiking for literally days ... e.g. I let the
app run over the weekend, came back today ... and timeout, no data and
aspnet_wp.exe still consuming 98% of mem.)

Any help appreciated
Mike

 

[Kevin Spencer]

Let me guess: You're using the Bitmap.GetPixel method, right? If you want to
process an image pixel by pixel, you want to use unsafe code and pointer
arithmetic, which you won't be able to do using VB.Net. The GetPixel method
is safe, and slow. Also, depending upon how you're using your Dataset, you
may be running into some performance issues there as well. Regardless of
whether you're writing to the database all at once, you may be running into
some type-checking issues, again, depending upon how you're using the
DataSet. But I'd be willing to guess that 90% of your problem is with your
image processing.

--
HTH,

Kevin Spencer
Microsoft MVP
..Net Developer
We got a sick zebra a hat,
you ultimate tuna.

 

[Mike]

Hi Kevin,

Re:
"Let me guess: You're using the Bitmap.GetPixel method, right?"

Yes, using bitmap.getpixel.

"But I'd be willing to guess that 90% of your problem is with your
image processing. "

Ordinarily, a good safe bet. ;-)
But not in this case.

I noticed, even in winapp version that bitmap.getpixel is slow, so I
inserted a function that reads the bitmap color info into an array and
then work with the array.
Reading the bitmpa into the array is lightening fast, and then doing
the crunching on the pixel data in the array is not where I seem to be
taking the hit.

"depending upon how you're using the DataSet."

I am pretty sure this is where the issue lies ... but can't seem to pin
it down as to what about how I am using the ds is doing it.
Right now, for the part causing the slow down/hang up it is a typed
dataset.
I actually instatiate 2 instances of the table:
1. holds the intermediate, per pixel data, a temptable if you will
2. then I check to see if that intermediate color/info is already on
the complete/permanent table, if that color is there increment
frequency, if not there then add it.

How?
I have tried every way I can think of, using select on table,
dataviews, datarowviews, defaultview, filtering, looping through, etc.
etc. etc. nothing seems to make any difference (which leads me back to
thinking there is either a bug in aspnet_wp.exe or I have a bad install
of it ... given it hangs/gets stuck).

Here is the code, as I have it now (this is about the 30th version) of
the section/procedure that seems to be causing the traffic jam:

Private Shared Sub AddPixelColorDataToTable(ByRef DtlRowView As
System.Data.DataRowView _
, ByRef RawDV As DataView _
, ByRef dsMCT As WhatIsBlue.MaterialColorTemp _
, ByRef arrayMatColorTempRows() As
WhatIsBlue.MaterialColorTemp.MaterialColorTempRow _
, ByRef DtlRowCount As Integer _
, ByRef MCTDV As System.Data.DataView)

Dim StartLoop As Integer = 1
If (DtlRowCount = 1) Then
StartLoop = 0
End If

Dim foundrow As Integer = -1
Dim useMCTIndex As Int64

For i As Integer = StartLoop To UBound(arrayMatColorTempRows)
If (i = 0) Then
useMCTIndex = RawDV(0)("MCTIndex")
Else
useMCTIndex = DtlRowView("MCTIndex")
End If

MCTDV.RowFilter = "MCTIndex = '" & useMCTIndex.ToString &
"'"


foundrow = MCTDV.Find(New Object() {useMCTIndex _
, arrayMatColorTempRows(i).Alpha,
arrayMatColorTempRows(i).Red _
, arrayMatColorTempRows(i).Green,
arrayMatColorTempRows(i).Blue})

'CurrRow = dsMCT.MaterialColorTemp.Rows.Find(New Object()
{useMCTIndex _
' , arrayMatColorTempRows(i).Alpha,
arrayMatColorTempRows(i).Red _
' , arrayMatColorTempRows(i).Green,
arrayMatColorTempRows(i).Blue})

'If (CurrRow Is Nothing) Then
If (foundrow = -1) Then
Dim CurrRow As
WhatIsBlue.MaterialColorTemp.MaterialColorTempRow

CurrRow = dsMCT.MaterialColorTemp.NewRow

CurrRow.MCTIndex = useMCTIndex
CurrRow.Alpha = arrayMatColorTempRows(i).Alpha
CurrRow.Red = arrayMatColorTempRows(i).Red
CurrRow.Green = arrayMatColorTempRows(i).Green
CurrRow.Blue = arrayMatColorTempRows(i).Blue
CurrRow.WeightedPct =
arrayMatColorTempRows(i).WeightedPct
CurrRow.RawPct = arrayMatColorTempRows(i).RawPct
CurrRow.WeightedFrequency =
arrayMatColorTempRows(i).WeightedFrequency
CurrRow.RawFrequency =
arrayMatColorTempRows(i).RawFrequency
CurrRow.Accent = arrayMatColorTempRows(i).Accent
CurrRow.Accept = arrayMatColorTempRows(i).Accept

dsMCT.MaterialColorTemp.Rows.Add(CurrRow)

Else
MCTDV(foundrow)("RawFrequency") =
MCTDV(foundrow)("RawFrequency") + 1
MCTDV(foundrow)("WeightedFrequency") =
MCTDV(foundrow)("WeightedFrequency") +
arrayMatColorTempRows(i).WeightedFrequency

'CurrRow("RawFrequency") = CurrRow("RawFrequency") + 1
'CurrRow("WeightedFrequency") =
CurrRow("WeightedFrequency") +
arrayMatColorTempRows(i).WeightedFrequency
End If
Next

End Sub

Yes a bit odd ... as I said, sort of trying this & that.
BTW: it actually gets, and here tries to store 2 values for each pixel,
one is the raw/as is color data for the pixel (from the array) and the
other is the calculated color (for our needs, sort of have to do an
averaging of color values in the area around a pixel)

Yes, eliminating the raw data does indeedy speed things up, a bit, but
on the larger images, it still comes to a grinding halt, hangs, need to
"end process" on aspnet_wp.exe

Really, puzzled on this ... just comes to a grinding halt ... I'd even
be happy if it took a couple of hours to do an image ... at least it
does the image (even if slow) but it just comes to a grinding halt ...
left it run over night a couple of times now, on 1 image and each
morning get the generic "The page cannot be displayed" message in web
browser

When I say generic page ... this is after it has been processing for a
while and the page suggest things like hitting refresh, trying again
later, check my internet settings, yada yada.

It just simply comes to a grinding halt ... slow would be better than a
grinding halt.

On the smaller images z(100x100) it does them in about 2 minutes ...
larger ... comes to a halt.

Thanks for the help.

Mike

 

[Mike]

Hi,

Fixed it.

As one might see from above code, I was using DataView's &
DataRowView's to filter the tables (which gets fairly large as it
processes an image).

Don't use DataView's or DRV's (even though documentation stresses them
heavily).
Not saying, never use them, but if you are adding data, large amounts
of it, to tables via/thru DVs, it will bog down.

I moved code to a windows app, just to see if it was ASP.NET related
issue.
But windows app, alos slowed way down (oh, btw, in prototype windows I
was not using tables but rather storing data in arrays, hashtables,
collections, etc. which is why prototype was fast).

End up, in trying one thing after another, creating several (variable
number depending upon how an image is to be processed) seperate tables,
then using table.select methods to find/sort/add and it zips right
along.
Then moved it back to a web app.

With DV method ran it twice overnight ... always bombed out, yet
aspnet_wp.exe still churning away in the background ... 16 hours or so
.... nver finishing, web page ceasing to display, aspnet_wp.exe churning
away ....

By working directly with tables (an array of them) it can process the
image 3 times, essentially, in 3.75 minutes, with no lock up, boom
done.

HTH someone.

Mike