This project is read-only.

Configuring MSE in a Distributive Architecture

Topics: Usage Scenarios, Technical Questions
Jun 11, 2008 at 5:58 PM
I have MSE configured as a messenger on a web server (let's call this server1) and MSE as a broker and messenger on another server (let's call this server 2). When invoking services using the server2, I have no problems getting a response back. When invoking these same services via server1, I get the following error when looking at the service logs:

System.ServiceModel.EndpointNotFoundException, System.ServiceModel, Version=3.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089
There was no endpoint listening at http://server2/ProcessMessage that could accept the message. This is often caused by an incorrect address or SOAP action. See InnerException, if present, for more details.

Stack trace:
System.ServiceModel.Channels.HttpChannelUtilities.ProcessGetResponseWebException(WebException webException, HttpWebRequest request, HttpAbortReason abortReason)
System.ServiceModel.Channels.HttpChannelFactory.HttpRequestChannel.HttpChannelAsyncRequest.CompleteGetResponse(IAsyncResult result)
System.ServiceModel.Channels.HttpChannelFactory.HttpRequestChannel.HttpChannelAsyncRequest.OnGetResponse(IAsyncResult result)
System.ServiceModel.Diagnostics.Utility.AsyncThunk.UnhandledExceptionFrame(IAsyncResult result)
System.Net.LazyAsyncResult.Complete(IntPtr userToken)
System.Net.ContextAwareResult.CompleteCallback(Object state)
System.Threading.ExecutionContext.runTryCode(Object userData)
System.Runtime.CompilerServices.RuntimeHelpers.ExecuteCodeWithGuaranteedCleanup(TryCode code, CleanupCode backoutCode, Object userData)
System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
System.Net.ContextAwareResult.Complete(IntPtr userToken)
System.Net.LazyAsyncResult.ProtectedInvokeCallback(Object result, IntPtr userToken)
System.Net.HttpWebRequest.ProcessResponse()
System.Net.HttpWebRequest.SetResponse(CoreResponseData coreResponseData)
System.Net.HttpWebRequest.SetAndOrProcessResponse(Object responseOrException)
System.Net.ConnectionReturnResult.SetResponses(ConnectionReturnResult returnResult)
System.Net.Connection.ReadComplete(Int32 bytesRead, WebExceptionStatus errorStatus)
System.Net.Connection.ReadCallback(IAsyncResult asyncResult)
System.Net.Connection.ReadCallbackWrapper(IAsyncResult asyncResult)
System.Net.LazyAsyncResult.Complete(IntPtr userToken)
System.Net.ContextAwareResult.Complete(IntPtr userToken)
System.Net.LazyAsyncResult.ProtectedInvokeCallback(Object result, IntPtr userToken)
System.Net.Sockets.BaseOverlappedAsyncResult.CompletionPortCallback(UInt32 errorCode, UInt32 numBytes, NativeOverlapped* nativeOverlapped)
System.Threading._IOCompletionCallback.PerformIOCompletionCallback(UInt32 errorCode, UInt32 numBytes, NativeOverlapped* pOVERLAP)

I can't figure out what endpoint ProcessMessage is in order to get this work, has someone been able to configure a distributed architecture or ran into this issue before? Appreciate the help.
Jun 11, 2008 at 7:04 PM
When configuring a runtime as a messenger only, the BrokerURI must point to a runtime that is configured as a Broker only.  When a runtime is configured as a messenger+broker, the broker is available only to the local messenger (we do not construct an endpoint for the broker to listen at).

If I understand your situation correctly, you have server1 pointing to server2 in which server2 is a messenger+broker.

If you want to keep server2 configured the way it is, you'll need to introduce a server3 which operates as a broker only.  Make sure server3's ListenerURI and server1's BrokerURI are the same. The ListenerURI tells the broker what endpoint to construct to receiving messages.  Also, you will need to ensure the BrokerBinding on for both is either wsHttp(Soap12) or NetTcp.




ramav wrote:
I have MSE configured as a messenger on a web server (let's call this server1) and MSE as a broker and messenger on another server (let's call this server 2). When invoking services using the server2, I have no problems getting a response back. When invoking these same services via server1, I get the following error when looking at the service logs:

System.ServiceModel.EndpointNotFoundException, System.ServiceModel, Version=3.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089
There was no endpoint listening at http://server2/ProcessMessage that could accept the message. This is often caused by an incorrect address or SOAP action. See InnerException, if present, for more details.

Stack trace:
System.ServiceModel.Channels.HttpChannelUtilities.ProcessGetResponseWebException(WebException webException, HttpWebRequest request, HttpAbortReason abortReason)
System.ServiceModel.Channels.HttpChannelFactory.HttpRequestChannel.HttpChannelAsyncRequest.CompleteGetResponse(IAsyncResult result)
System.ServiceModel.Channels.HttpChannelFactory.HttpRequestChannel.HttpChannelAsyncRequest.OnGetResponse(IAsyncResult result)
System.ServiceModel.Diagnostics.Utility.AsyncThunk.UnhandledExceptionFrame(IAsyncResult result)
System.Net.LazyAsyncResult.Complete(IntPtr userToken)
System.Net.ContextAwareResult.CompleteCallback(Object state)
System.Threading.ExecutionContext.runTryCode(Object userData)
System.Runtime.CompilerServices.RuntimeHelpers.ExecuteCodeWithGuaranteedCleanup(TryCode code, CleanupCode backoutCode, Object userData)
System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
System.Net.ContextAwareResult.Complete(IntPtr userToken)
System.Net.LazyAsyncResult.ProtectedInvokeCallback(Object result, IntPtr userToken)
System.Net.HttpWebRequest.ProcessResponse()
System.Net.HttpWebRequest.SetResponse(CoreResponseData coreResponseData)
System.Net.HttpWebRequest.SetAndOrProcessResponse(Object responseOrException)
System.Net.ConnectionReturnResult.SetResponses(ConnectionReturnResult returnResult)
System.Net.Connection.ReadComplete(Int32 bytesRead, WebExceptionStatus errorStatus)
System.Net.Connection.ReadCallback(IAsyncResult asyncResult)
System.Net.Connection.ReadCallbackWrapper(IAsyncResult asyncResult)
System.Net.LazyAsyncResult.Complete(IntPtr userToken)
System.Net.ContextAwareResult.Complete(IntPtr userToken)
System.Net.LazyAsyncResult.ProtectedInvokeCallback(Object result, IntPtr userToken)
System.Net.Sockets.BaseOverlappedAsyncResult.CompletionPortCallback(UInt32 errorCode, UInt32 numBytes, NativeOverlapped* nativeOverlapped)
System.Threading._IOCompletionCallback.PerformIOCompletionCallback(UInt32 errorCode, UInt32 numBytes, NativeOverlapped* pOVERLAP)

I can't figure out what endpoint ProcessMessage is in order to get this work, has someone been able to configure a distributed architecture or ran into this issue before? Appreciate the help.



Jun 11, 2008 at 11:13 PM
When you say that a third server would have to be introduced, do you mean another physical machine is necessary, or just an additional instance of the runtime server on one of the existing machines?
Jun 11, 2008 at 11:55 PM

The installer doesn't support adding multiple Runtime Server services on the same machine (i.e. each registered as a windows service), so you'd need a different physical machine.  When you setup a runtime on a server to act as a messenger only, the implication is that you will also have a second physical machine with a runtime server acting as a broker.

So if you don't have a server3 to install the MSE on, you could reconfigure the runtime on server2 to be a broker only.

 


kdstock wrote:
When you say that a third server would have to be introduced, do you mean another physical machine is necessary, or just an additional instance of the runtime server on one of the existing machines?



Jun 12, 2008 at 3:17 PM
Following your advice, and not having a spare server I have done the following. I re-purposed server2 to be just a broker by unclicking messenger on the runtime through the admin console. In addition I modified the listener URI on server2 and the broker uri for server1 to match. Having made those changes, we are able to see our request transactions process; however we send a response back when receiving a request and this is failing in a distributed environment.

This is what we are seeing in the service logs on the runtime server that is just the messenger:

System.ServiceModel.ProtocolException, System.ServiceModel, Version=3.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089
The header 'Security' from the namespace 'http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-wssecurity-secext-1.0.xsd' was not understood by the recipient of this message, causing the message to not be processed.  This error typically indicates that the sender of this message has enabled a communication protocol that the receiver cannot process.  Please ensure that the configuration of the client's binding is consistent with the service's binding.

Stack trace:
System.ServiceModel.Dispatcher.ProxyOperationRuntime.AfterReply(ProxyRpc& rpc)
System.ServiceModel.Channels.ServiceChannel.HandleReply(ProxyOperationRuntime operation, ProxyRpc& rpc)
System.ServiceModel.Channels.ServiceChannel.EndCall(String action, Object[] outs, IAsyncResult result)
System.ServiceModel.Channels.ServiceChannelProxy.InvokeEndService(IMethodCallMessage methodCall, ProxyOperationRuntime operation)
System.ServiceModel.Channels.ServiceChannelProxy.Invoke(IMessage message)
System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type)
Microsoft.MSE.Runtime.Contracts.Broker.IBroker.EndRequest(IAsyncResult ar)
Microsoft.MSE.Runtime.Services.Broker.BrokerServiceProxy.EndRequest(IAsyncResult ar)
Microsoft.MSE.Runtime.Services.Messenger.VirtualOperationInvoker.InvokeEnd(Object instance, Object[]& outputs, IAsyncResult result)
System.ServiceModel.Dispatcher.DispatchOperationRuntime.InvokeEnd(MessageRpc& rpc)
System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.ProcessMessage7(MessageRpc& rpc)
System.ServiceModel.Dispatcher.MessageRpc.Process(Boolean isOperationContextSet)
System.ServiceModel.Dispatcher.MessageRpc.Wrapper.Resume(Boolean& alreadyResumedNoLock)
System.ServiceModel.Dispatcher.ThreadBehavior.CleanThreadCallback(Object state)
System.ServiceModel.Channels.IOThreadScheduler.CriticalHelper.WorkItem.Invoke2()
System.ServiceModel.Channels.IOThreadScheduler.CriticalHelper.WorkItem.OnSecurityContextCallback(Object o)
System.Security.SecurityContext.Run(SecurityContext securityContext, ContextCallback callback, Object state)
System.ServiceModel.Channels.IOThreadScheduler.CriticalHelper.WorkItem.Invoke()
System.ServiceModel.Channels.IOThreadScheduler.CriticalHelper.ProcessCallbacks()
System.ServiceModel.Channels.IOThreadScheduler.CriticalHelper.CompletionCallback(Object state)
System.ServiceModel.Channels.IOThreadScheduler.CriticalHelper.ScheduledOverlapped.IOCallback(UInt32 errorCode, UInt32 numBytes, NativeOverlapped* nativeOverlapped)
System.ServiceModel.Diagnostics.Utility.IOCompletionThunk.UnhandledExceptionFrame(UInt32 error, UInt32 bytesRead, NativeOverlapped* nativeOverlapped)
System.Threading._IOCompletionCallback.PerformIOCompletionCallback(UInt32 errorCode, UInt32 numBytes, NativeOverlapped* pOVERLAP)


We believe the error is somehow connected to how the messenger and broker are interacting as we have successfully processed requests and sent responses when the services are routed through MSE on a configuration of a messenger and broker (i.e. a non-distributed architecture and also tried it on both servers).

Can someone help us understand where the http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-wssecurity-secext-1.0.xsd is being referenced, or if I need to add this schema to MSE? Is there any other configuration that we need to do make the distributive model work as well, do we need to populate our header with any additional info? We appreciate your help on this.



Jun 13, 2008 at 8:17 PM
A couple questions:

What appears in the logs for the broker?  Can you post the logs for this?  
What is the binding your service uses?
What is the binding selected for the Broker?
Are you able to verify that your service is in fact called, and something happens with the response?  Or does the request never make it to your service?

Thanks.

ramav wrote:
Following your advice, and not having a spare server I have done the following. I re-purposed server2 to be just a broker by unclicking messenger on the runtime through the admin console. In addition I modified the listener URI on server2 and the broker uri for server1 to match. Having made those changes, we are able to see our request transactions process; however we send a response back when receiving a request and this is failing in a distributed environment.

This is what we are seeing in the service logs on the runtime server that is just the messenger:

System.ServiceModel.ProtocolException, System.ServiceModel, Version=3.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089
The header 'Security' from the namespace 'http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-wssecurity-secext-1.0.xsd' was not understood by the recipient of this message, causing the message to not be processed.  This error typically indicates that the sender of this message has enabled a communication protocol that the receiver cannot process.  Please ensure that the configuration of the client's binding is consistent with the service's binding.

Stack trace:
System.ServiceModel.Dispatcher.ProxyOperationRuntime.AfterReply(ProxyRpc& rpc)
System.ServiceModel.Channels.ServiceChannel.HandleReply(ProxyOperationRuntime operation, ProxyRpc& rpc)
System.ServiceModel.Channels.ServiceChannel.EndCall(String action, Object[] outs, IAsyncResult result)
System.ServiceModel.Channels.ServiceChannelProxy.InvokeEndService(IMethodCallMessage methodCall, ProxyOperationRuntime operation)
System.ServiceModel.Channels.ServiceChannelProxy.Invoke(IMessage message)
System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type)
Microsoft.MSE.Runtime.Contracts.Broker.IBroker.EndRequest(IAsyncResult ar)
Microsoft.MSE.Runtime.Services.Broker.BrokerServiceProxy.EndRequest(IAsyncResult ar)
Microsoft.MSE.Runtime.Services.Messenger.VirtualOperationInvoker.InvokeEnd(Object instance, Object[]& outputs, IAsyncResult result)
System.ServiceModel.Dispatcher.DispatchOperationRuntime.InvokeEnd(MessageRpc& rpc)
System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.ProcessMessage7(MessageRpc& rpc)
System.ServiceModel.Dispatcher.MessageRpc.Process(Boolean isOperationContextSet)
System.ServiceModel.Dispatcher.MessageRpc.Wrapper.Resume(Boolean& alreadyResumedNoLock)
System.ServiceModel.Dispatcher.ThreadBehavior.CleanThreadCallback(Object state)
System.ServiceModel.Channels.IOThreadScheduler.CriticalHelper.WorkItem.Invoke2()
System.ServiceModel.Channels.IOThreadScheduler.CriticalHelper.WorkItem.OnSecurityContextCallback(Object o)
System.Security.SecurityContext.Run(SecurityContext securityContext, ContextCallback callback, Object state)
System.ServiceModel.Channels.IOThreadScheduler.CriticalHelper.WorkItem.Invoke()
System.ServiceModel.Channels.IOThreadScheduler.CriticalHelper.ProcessCallbacks()
System.ServiceModel.Channels.IOThreadScheduler.CriticalHelper.CompletionCallback(Object state)
System.ServiceModel.Channels.IOThreadScheduler.CriticalHelper.ScheduledOverlapped.IOCallback(UInt32 errorCode, UInt32 numBytes, NativeOverlapped* nativeOverlapped)
System.ServiceModel.Diagnostics.Utility.IOCompletionThunk.UnhandledExceptionFrame(UInt32 error, UInt32 bytesRead, NativeOverlapped* nativeOverlapped)
System.Threading._IOCompletionCallback.PerformIOCompletionCallback(UInt32 errorCode, UInt32 numBytes, NativeOverlapped* pOVERLAP)


We believe the error is somehow connected to how the messenger and broker are interacting as we have successfully processed requests and sent responses when the services are routed through MSE on a configuration of a messenger and broker (i.e. a non-distributed architecture and also tried it on both servers).

Can someone help us understand where the http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-wssecurity-secext-1.0.xsd is being referenced, or if I need to add this schema to MSE? Is there any other configuration that we need to do make the distributive model work as well, do we need to populate our header with any additional info? We appreciate your help on this.






Jun 16, 2008 at 3:48 PM
Our service takes a request that comes in and writes it out to a database, so we are able to see the data be written out to our database; however we need to send a response back to the calling application to confirm that we processed the request and this is where we are failing with the error posted in my previous post.

Re: Your question on the logs: I gave a small sample in my previous post of what we see in the logs, and I will have to see how I can send the logs to you since it's quite large.
Binding for Broker is: WsHttp (Soap12)
Binding for our service is a custom binding listed below: 

bindings xmlns="">

  <wsHttpBinding>

    <binding name="Ndex_WSHttpBinding" bypassProxyOnLocal="false"

transactionFlow="false" hostNameComparisonMode="StrongWildcard"

maxBufferPoolSize="524288" maxReceivedMessageSize="999999999"

messageEncoding="Text" textEncoding="utf-8" useDefaultWebProxy="true"

receiveTimeout="00:15:00">

      <readerQuotas maxDepth="2147483647"

maxStringContentLength="2147483647" maxArrayLength="2147483647"

maxBytesPerRead="2147483647" maxNameTableCharCount="2147483647">

      </readerQuotas>

      <reliableSession ordered="true" inactivityTimeout="00:15:00"

enabled="false">

      </reliableSession>

      <security mode="Message">

        <transport clientCredentialType="Windows"

proxyCredentialType="None" realm="">

        </transport>

        <message clientCredentialType="UserName">

        </message>

      </security>

    </binding>

  </wsHttpBinding>

</bindings>