Saving web.config caused W2K8 SP1 Machine Restart

[Howard Hoffman]

We have an ASP.NET application that makes heavy use of the
System.Threading.ThreadPool for background work. We use the Global.asax
Application_End end to set a ManualResetEvent that tells the threads to
cease work. However, if they are very busy it can take as long as 10
minutes for all of them to end.

During a period where the threads were indeed very busy, an engineer:

1) Made a change to web.config for the application. Saved the file.
2) Immediately surfed to one of the application's web pages.

There was a delay of about 10 seconds in bringing the page up on step
2...and the machine restarted itself.

Stunned, we tried it again an hour later and got the same result.

There was no dump file, and nothing in the Windows Event Log other than the
'starting Windows' parade of events. We also noted that our application log
file had an entry for Application_End, and appeared to end normally (except
the background threads never reported back that they finished ...).

What are the expected restart semantics here? I know that saving
web.config triggers an application-end, which really equates to the
AppDomain shutting down. I'd thought, however, that any in-flight HTTP
requests or background threads would continue to work in the to-be-ended
AppDomain, and that once all that work was finished the AppDomain would
indeed go away. At the same time a new AppDomain is created to be used for
new HTTP requests and anything else that is done in the application
following the Application_Start event.

Has this changed for W2K8, or am is my understanding of application-restart
semantics incorrect.

This machine does have a post W2K8 SP1 / ASP.NET 2.0/3.5 SP1 hotfix :
967535.

Any ideas or clarifications greatly appreciated.

Howard Hoffman

 

[Guest]

We have an ASP.NET application that makes heavy use of the
System.Threading.ThreadPool for background work.  We use the Global.asax
Application_End end to set a ManualResetEvent that tells the threads to
cease work.  However, if they are very busy it can take as long as 10
minutes for all of them to end.

During a period where the threads were indeed very busy, an engineer:

1) Made a change to web.config for the application.  Saved the file.
2) Immediately surfed to one of the application's web pages.

There was a delay of about 10 seconds in bringing the page up on step
2...and the machine restarted itself.

Stunned, we tried it again an hour later and got the same result.

There was no dump file, and nothing in the Windows Event Log other than the
'starting Windows' parade of events.  We also noted that our application log
file had an entry for Application_End, and appeared to end normally (except
the background threads never reported back that they finished ...).

What are the expected restart semantics here?   I know that saving
web.config triggers an application-end, which really equates to the
AppDomain shutting down.  I'd thought, however, that any in-flight HTTP
requests or background threads would continue to work in the to-be-ended
AppDomain, and that once all that work was finished the AppDomain would
indeed go away.  At the same time a new AppDomain is created to be used for
new HTTP requests and anything else that is done in the application
following the Application_Start event.

Has this changed for W2K8, or am is my understanding of application-restart
semantics incorrect.

This machine does have a post W2K8 SP1 / ASP.NET 2.0/3.5 SP1 hotfix :
967535.

Any ideas or clarifications greatly appreciated.

Howard Hoffman

Hi Howard,

did you check how Rapid Fail Protection is configired on IIS? Maybe
you use /rebootonerror option to reboot the whole server as it is
shown on the following tutorial?

http://blogs.msdn.com/vijaysk/archi...-when-rapid-fail-protection-is-triggered.aspx

 

[Howard Hoffman]

Alexey -

Thanks for the tip and the link.

No, we do not have Shutdown Exectuable set on our App Pool. Just the
default value of nothing for Exectuable and also nothing for Exectuable
Parameters.

ApplicationHost.config for the app-pool also has not <failure> element, just
a <processModel> (we have a custom account as the App Pool identity) and a
<recycling> element.

Howard

We have an ASP.NET application that makes heavy use of the
System.Threading.ThreadPool for background work. We use the Global.asax
Application_End end to set a ManualResetEvent that tells the threads to
cease work. However, if they are very busy it can take as long as 10
minutes for all of them to end.

During a period where the threads were indeed very busy, an engineer:

1) Made a change to web.config for the application. Saved the file.
2) Immediately surfed to one of the application's web pages.

There was a delay of about 10 seconds in bringing the page up on step
2...and the machine restarted itself.

Stunned, we tried it again an hour later and got the same result.

There was no dump file, and nothing in the Windows Event Log other than
the
'starting Windows' parade of events. We also noted that our application
log
file had an entry for Application_End, and appeared to end normally
(except
the background threads never reported back that they finished ...).

What are the expected restart semantics here? I know that saving
web.config triggers an application-end, which really equates to the
AppDomain shutting down. I'd thought, however, that any in-flight HTTP
requests or background threads would continue to work in the to-be-ended
AppDomain, and that once all that work was finished the AppDomain would
indeed go away. At the same time a new AppDomain is created to be used for
new HTTP requests and anything else that is done in the application
following the Application_Start event.

Has this changed for W2K8, or am is my understanding of
application-restart
semantics incorrect.

This machine does have a post W2K8 SP1 / ASP.NET 2.0/3.5 SP1 hotfix :
967535.

Any ideas or clarifications greatly appreciated.

Howard Hoffman

Hi Howard,

did you check how Rapid Fail Protection is configired on IIS? Maybe
you use /rebootonerror option to reboot the whole server as it is
shown on the following tutorial?

http://blogs.msdn.com/vijaysk/archi...-when-rapid-fail-protection-is-triggered.aspx

 

[Howard Hoffman]

It now appears this a hardware problem.

Since my original post, the restarts have spread to random times-of-the-day
when there is no activity on the machine, or something simple such as a user
using Windows Explorer to look at file properties.

Power supply is the current guess.

Thanks all for your help.

Howard Hoffman

 

[Guest]

It now appears this a hardware problem.

Since my original post, the restarts have spread to random times-of-the-day
when there is no activity on the machine, or something simple such as a user
using Windows Explorer to look at file properties.

Power supply is the current guess.

Thanks all for your help.

Hopefully it helps