I imported a large schema, that I will attach to this issue. After importing the endpoints wouldn't register. I kept getting this message when I restared the MSE runtime service:
MSE Runtime is unable to connect to the catalog.
Communication Exception raised in GetRuntimeServerFromCatalog : The maximum message size quota for incoming messages (65536) has been exceeded. To increase the quota, use the MaxReceivedMessageSize property on the appropriate binding element.
I've been working with WCF long enough to be very familiar with this error. However, I couldn't figure out which binding was causing the issue. I assumed it was a binding in the Runtime Servers setting of the MMC snap-in, but that proved to be elusive. I finally
figured out that I needed to increase the MaxReceivedMessageSize in the Microsoft.MSE.Repository.Service.exe.config file on the netNamedPipeBinding.
I would suggest that you increase this by default to a very high number. Additionally, I would add that it's highly confusing that I have to configure the catalog settings in 2 places for the runtime.