Written by Irmak Tevfik on 09 - Nov - 2014

Why and where to use Circuit Breaker Pattern on Microsoft Azure?

It is almost certain that every single person reading this article feels annoyed to wait for an elevator which is held and blocked by a person. In this case elevator might be a third party web service and person blocking it might be the developer maintaining it. OK, there might be a problem on the elevator, or the guy blocking it might have a reasonable explanation, but at some point you need to take the stairs. Same rule applies to the projects running on Microsoft Azure.

There needs to be a threshold and a time-out for an operation to complete. So lets say, if you are working with an external web service, you have a slight chance that the service might be down. 
So, you need to carry on your transaction without tolerating this to hold your process. For this kind of situation, we have a pattern called Circuit Breaker Pattern.

Simply, pattern acts as proxy on processes that has a chance to fail. We implement the pattern as a state machine containing three states as Open, Half-open and Closed.
Idea is to monitor the failures (log as necessary) and give decision. Lets take a look at the states:

Open: States that the circuit is in open state running the transaction. Gets set on start of the trip.
Half-open: States that circuit time-out elapsed and running until the threshold reached. Vital to have this state as process might be delayed for a reason and we should not flood it with requests.
Closed: Initial state or state set after trip resets.

Which can be seen on the diagram below.


If we take a look at to the code I wrote, we might have a better perspective. Note that, constructor takes default time-out of 3 seconds.
public class CircuitBreaker
{
    private event EventHandler CircuitStateChanged;
    private Timer timer;
    private uint treshold;
    private uint timeout;
    private CircuitState circuitState;
    private uint failureCount;
    private object returnObject;
 
    #region Ctors
    public CircuitBreaker()
        : this(3000, 5)
    {
 
    }
 
    public CircuitBreaker(uint timeout, uint treshold)
    {
        this.timeout = timeout;
        this.treshold = treshold;
        this.circuitState = CircuitState.Closed;
        this.failureCount = 0;
 
        this.timer = new Timer(timeout);
        timer.Elapsed += new System.Timers.ElapsedEventHandler(timer_Elapsed);
    }
    #endregion
 
    /// <summary>
    /// Entry point. method will take the delegate with parameters to invoke.
    /// </summary>
    /// <param name="delOperation"></param>
    /// <param name="parameters"></param>
    /// <returns></returns>
    public object Run(Delegate delOperation, object[] parameters)
    {
        if (circuitState == CircuitState.Open) throw new OpenCircuitException("Process is currently open", null);
        try
        {
            Trip();
            returnObject = delOperation.DynamicInvoke(parameters);
        }
        catch (Exception ex)
        {
            ResultHandle();               
        }
 
        if (this.circuitState == CircuitState.HalfOpen)
        {
            // is in a half-open state, then reset
            Reset();
        }
 
        return returnObject;
    }
 
    private void ResultHandle()
    {
        if (this.circuitState == CircuitState.HalfOpen)
        {
            Trip();
        }
        else if (failureCount < treshold)
        {
            failureCount++;
        }
        else if (failureCount >= treshold)
        {
            Trip();
        }
 
        throw new OperationFailedException("Operation failed", null);
    }
 
    private void timer_Elapsed(object sender, ElapsedEventArgs e)
    {
        ChangeState(CircuitState.HalfOpen);
        this.timer.Stop();
        if (returnObject == null)
        {
            failureCount++;
            if (failureCount > treshold)
            {
                ChangeState(CircuitState.Closed);
                throw new OperationFailedException("Operation failed", null);
            }
            Trip();
        }
    }
 
    private void ChangeState(CircuitState newState)
    {
        this.circuitState = newState;
    }
 
    private void Trip()
    {
        ChangeState(CircuitState.Open);
        this.timer.Start();
    }
 
    private void Reset()
    {
        ChangeState(CircuitState.Closed);
        this.timer.Stop();
    }
}

If you like, you can download the code from github link below. To test, I have published a simple service to Azure running on http://servicetests.azurewebsites.net/api/test/GetDelayedTest and simply run the following method as:
private object TestRunMethod(string address)
{
        httpClient = new HttpClient();
        //httpClient = new HttpClient(handle);
        httpClient.BaseAddress = new Uri("http://servicetests.azurewebsites.net/api/");
        httpClient.DefaultRequestHeaders.TryAddWithoutValidation("User-Agent",
"Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/33.0.1750.146 Safari/537.36");
        var res = this.httpClient.GetAsync(address).Result;
        return res;
}
 
.... execute the code as 
new CircuitBreaker.CircuitBreaker().Run(new Func<string, object>(TestRunMethod), new object[] { "http://servicetests.azurewebsites.net/api/test/GetDelayedTest" });

Download from:
https://github.com/irmaktevfik/CircuitBreakerPattern

Enjoy Coding:)
comments powered by Disqus