Every time we upload something to the primary node and then enable the application provisioning, we get following errors on one of the nodes. It was working fine few weeks ago. It happened earlier and for some reason, rebooting the master node resolved the issue but this time rebooting didn't do anything.
Any idea?
2012-08-15T13:27:54.3679353-04:00 Error secure-XXXX 10.2.8.120 Failed to run operation 'ProvisionApplications'. Failed to run method 'Microsoft.Web.Farm.SyncApplicationsRemoteMethod' on server '10.2.8.120'. Exception in response stream. Invalid index. (Exception from HRESULT: 0x80070585). Exception in response stream. Invalid index. (Exception from HRESULT: 0x80070585). Microsoft.Web.Farm.WebFarmException: Failed to run method 'Microsoft.Web.Farm.SyncApplicationsRemoteMethod' on server '10.2.8.120'. Exception in response stream. Invalid index. (Exception from HRESULT: 0x80070585). ---> Microsoft.Web.Farm.WebFarmException: Exception in response stream ---> System.Runtime.InteropServices.COMException (0x80070585): Invalid index. (Exception from HRESULT: 0x80070585)
at Microsoft.Web.Deployment.AxNative.AppHostAddRawCollectionElement(IAppHostElementCollection collection, IAppHostElement element, Int32 index)
at Microsoft.Web.Deployment.AppHostElementProvider.AddChild(DeploymentObject source, Int32 position, Boolean whatIf)
at Microsoft.Web.Deployment.DeploymentObject.AddChild(DeploymentObject source, Int32 position, DeploymentSyncContext syncContext)
at Microsoft.Web.Deployment.DeploymentSyncContext.HandleAddChild(DeploymentObject destParent, DeploymentObject sourceObject, Int32 position)
at Microsoft.Web.Deployment.DeploymentSyncContext.SyncChildrenOrder(DeploymentObject dest, DeploymentObject source)
at Microsoft.Web.Deployment.DeploymentSyncContext.HandleAddChild(DeploymentObject destParent, DeploymentObject sourceObject, Int32 position)
at Microsoft.Web.Deployment.DeploymentSyncContext.SyncChildrenOrder(DeploymentObject dest, DeploymentObject source)
at Microsoft.Web.Deployment.DeploymentSyncContext.SyncChildrenNoOrder(DeploymentObject dest, DeploymentObject source)
at Microsoft.Web.Deployment.DeploymentSyncContext.SyncChildrenNoOrder(DeploymentObject dest, DeploymentObject source)
at Microsoft.Web.Deployment.DeploymentSyncContext.SyncChildrenNoOrder(DeploymentObject dest, DeploymentObject source)
at Microsoft.Web.Deployment.DeploymentSyncContext.SyncChildrenOrder(DeploymentObject dest, DeploymentObject source)
at Microsoft.Web.Deployment.DeploymentSyncContext.SyncChildrenOrder(DeploymentObject dest, DeploymentObject source)
at Microsoft.Web.Deployment.DeploymentSyncContext.ProcessSync(DeploymentObject destinationObject, DeploymentObject sourceObject)
at Microsoft.Web.Deployment.DeploymentObject.SyncToInternal(DeploymentObject destObject, DeploymentSyncOptions syncOptions, PayloadTable payloadTable, ContentRootTable contentRootTable)
at Microsoft.Web.Deployment.DeploymentObject.SyncTo(DeploymentProviderOptions providerOptions, DeploymentBaseOptions baseOptions, DeploymentSyncOptions syncOptions)
at Microsoft.Web.Deployment.DeploymentObject.SyncTo(DeploymentBaseOptions baseOptions, DeploymentSyncOptions syncOptions)
at Microsoft.Web.Farm.SyncApplicationsRemoteMethod.RunRemote(ServerRemoteMethodContext methodContext, Object[] parameters)
at Microsoft.Web.Farm.WebFarmAgentService.HandleRunMethod(HttpListenerContext context)
--- End of inner exception stack trace ---
at Microsoft.Web.Farm.RunRemoteAsyncResult.ResponseCompleteCallback(IAsyncResult result)
--- End of inner exception stack trace ---
at Microsoft.Web.Farm.ProvisionApplicationsOperationProvider.RunOperationAsyncResult.EndProvision()
at Microsoft.Web.Farm.ProvisionApplicationsOperationProvider.EndRunOperation(IAsyncResult result)
at Microsoft.Web.Farm.RunOperationContext.FinalOperationCompleteCallback(IAsyncResult result)
2012-08-15T13:27:54.3679353-04:00 Error secure-XXXX 10.2.8.120 Failed to run operation 'HandleConfigurationChange'. Failed to run method 'Microsoft.Web.Farm.SyncApplicationsRemoteMethod' on server '10.2.8.120'. Exception in response stream. Invalid index. (Exception from HRESULT: 0x80070585). Exception in response stream. Invalid index. (Exception from HRESULT: 0x80070585). Microsoft.Web.Farm.WebFarmException: Failed to run method 'Microsoft.Web.Farm.SyncApplicationsRemoteMethod' on server '10.2.8.120'. Exception in response stream. Invalid index. (Exception from HRESULT: 0x80070585). ---> Microsoft.Web.Farm.WebFarmException: Exception in response stream ---> System.Runtime.InteropServices.COMException (0x80070585): Invalid index. (Exception from HRESULT: 0x80070585)
at Microsoft.Web.Deployment.AxNative.AppHostAddRawCollectionElement(IAppHostElementCollection collection, IAppHostElement element, Int32 index)
at Microsoft.Web.Deployment.AppHostElementProvider.AddChild(DeploymentObject source, Int32 position, Boolean whatIf)
at Microsoft.Web.Deployment.DeploymentObject.AddChild(DeploymentObject source, Int32 position, DeploymentSyncContext syncContext)
at Microsoft.Web.Deployment.DeploymentSyncContext.HandleAddChild(DeploymentObject destParent, DeploymentObject sourceObject, Int32 position)
at Microsoft.Web.Deployment.DeploymentSyncContext.SyncChildrenOrder(DeploymentObject dest, DeploymentObject source)
at Microsoft.Web.Deployment.DeploymentSyncContext.HandleAddChild(DeploymentObject destParent, DeploymentObject sourceObject, Int32 position)
at Microsoft.Web.Deployment.DeploymentSyncContext.SyncChildrenOrder(DeploymentObject dest, DeploymentObject source)
at Microsoft.Web.Deployment.DeploymentSyncContext.SyncChildrenNoOrder(DeploymentObject dest, DeploymentObject source)
at Microsoft.Web.Deployment.DeploymentSyncContext.SyncChildrenNoOrder(DeploymentObject dest, DeploymentObject source)
at Microsoft.Web.Deployment.DeploymentSyncContext.SyncChildrenNoOrder(DeploymentObject dest, DeploymentObject source)
at Microsoft.Web.Deployment.DeploymentSyncContext.SyncChildrenOrder(DeploymentObject dest, DeploymentObject source)
at Microsoft.Web.Deployment.DeploymentSyncContext.SyncChildrenOrder(DeploymentObject dest, DeploymentObject source)
at Microsoft.Web.Deployment.DeploymentSyncContext.ProcessSync(DeploymentObject destinationObject, DeploymentObject sourceObject)
at Microsoft.Web.Deployment.DeploymentObject.SyncToInternal(DeploymentObject destObject, DeploymentSyncOptions syncOptions, PayloadTable payloadTable, ContentRootTable contentRootTable)
at Microsoft.Web.Deployment.DeploymentObject.SyncTo(DeploymentProviderOptions providerOptions, DeploymentBaseOptions baseOptions, DeploymentSyncOptions syncOptions)
at Microsoft.Web.Deployment.DeploymentObject.SyncTo(DeploymentBaseOptions baseOptions, DeploymentSyncOptions syncOptions)
at Microsoft.Web.Farm.SyncApplicationsRemoteMethod.RunRemote(ServerRemoteMethodContext methodContext, Object[] parameters)
at Microsoft.Web.Farm.WebFarmAgentService.HandleRunMethod(HttpListenerContext context)
--- End of inner exception stack trace ---
at Microsoft.Web.Farm.RunRemoteAsyncResult.ResponseCompleteCallback(IAsyncResult result)
--- End of inner exception stack trace ---
at Microsoft.Web.Farm.ServerOperationProvider.RunOperationAsyncResult.EndRunOperation()
at Microsoft.Web.Farm.ServerOperationProvider.EndRunOperation(IAsyncResult result)
at Microsoft.Web.Farm.RunOperationContext.FinalOperationCompleteCallback(IAsyncResult result)
2012-08-15T13:27:54.3679353-04:00 Error secure-XXXX 10.2.8.120 Server marked 'Unhealthy' due to failure to run operation 'HandleConfigurationChange' Microsoft.Web.Farm.WebFarmException: Failed to run method 'Microsoft.Web.Farm.SyncApplicationsRemoteMethod' on server '10.2.8.120'. Exception in response stream. Invalid index. (Exception from HRESULT: 0x80070585). ---> Microsoft.Web.Farm.WebFarmException: Exception in response stream ---> System.Runtime.InteropServices.COMException (0x80070585): Invalid index. (Exception from HRESULT: 0x80070585)
at Microsoft.Web.Deployment.AxNative.AppHostAddRawCollectionElement(IAppHostElementCollection collection, IAppHostElement element, Int32 index)
at Microsoft.Web.Deployment.AppHostElementProvider.AddChild(DeploymentObject source, Int32 position, Boolean whatIf)
at Microsoft.Web.Deployment.DeploymentObject.AddChild(DeploymentObject source, Int32 position, DeploymentSyncContext syncContext)
at Microsoft.Web.Deployment.DeploymentSyncContext.HandleAddChild(DeploymentObject destParent, DeploymentObject sourceObject, Int32 position)
at Microsoft.Web.Deployment.DeploymentSyncContext.SyncChildrenOrder(DeploymentObject dest, DeploymentObject source)
at Microsoft.Web.Deployment.DeploymentSyncContext.HandleAddChild(DeploymentObject destParent, DeploymentObject sourceObject, Int32 position)
at Microsoft.Web.Deployment.DeploymentSyncContext.SyncChildrenOrder(DeploymentObject dest, DeploymentObject source)
at Microsoft.Web.Deployment.DeploymentSyncContext.SyncChildrenNoOrder(DeploymentObject dest, DeploymentObject source)
at Microsoft.Web.Deployment.DeploymentSyncContext.SyncChildrenNoOrder(DeploymentObject dest, DeploymentObject source)
at Microsoft.Web.Deployment.DeploymentSyncContext.SyncChildrenNoOrder(DeploymentObject dest, DeploymentObject source)
at Microsoft.Web.Deployment.DeploymentSyncContext.SyncChildrenOrder(DeploymentObject dest, DeploymentObject source)
at Microsoft.Web.Deployment.DeploymentSyncContext.SyncChildrenOrder(DeploymentObject dest, DeploymentObject source)
at Microsoft.Web.Deployment.DeploymentSyncContext.ProcessSync(DeploymentObject destinationObject, DeploymentObject sourceObject)
at Microsoft.Web.Deployment.DeploymentObject.SyncToInternal(DeploymentObject destObject, DeploymentSyncOptions syncOptions, PayloadTable payloadTable, ContentRootTable contentRootTable)
at Microsoft.Web.Deployment.DeploymentObject.SyncTo(DeploymentProviderOptions providerOptions, DeploymentBaseOptions baseOptions, DeploymentSyncOptions syncOptions)
at Microsoft.Web.Deployment.DeploymentObject.SyncTo(DeploymentBaseOptions baseOptions, DeploymentSyncOptions syncOptions)
at Microsoft.Web.Farm.SyncApplicationsRemoteMethod.RunRemote(ServerRemoteMethodContext methodContext, Object[] parameters)
at Microsoft.Web.Farm.WebFarmAgentService.HandleRunMethod(HttpListenerContext context)
--- End of inner exception stack trace ---
at Microsoft.Web.Farm.RunRemoteAsyncResult.ResponseCompleteCallback(IAsyncResult result)
--- End of inner exception stack trace ---
at Microsoft.Web.Farm.ServerOperationProvider.RunOperationAsyncResult.EndRunOperation()
at Microsoft.Web.Farm.ServerOperationProvider.EndRunOperation(IAsyncResult result)
at Microsoft.Web.Farm.RunOperationContext.FinalOperationCompleteCallback(IAsyncResult result)