Improving Asynchronous Tests for Silverlight

Christopher Bennage — Mon, 17 Jan 2011 16:29:00 GMT

First a confession, and I know I’ll invoke shame for this one: I have done Very Little Testing with Silverlight. I won’t bore you with reasons or excuses, but I wasn’t up on the state of unit testing in Silverlight until very recently. (In fact, I might still be missing some chunks.)

The Problem

Last week, I started writing some tests for Raven DB. I won’t call them ‘unit’ tests because they were really ‘integration’ tests. Specifically, I wanted to test the budding Silverlight client for Raven. This mean that my tests had to make calls to the Raven server, wait for the result, then assert something.

Large portions of the Silverlight client are cross-compiled and I wanted to do the same with the relevant tests. On the .NET side, a demonstrative test looks like this:

[Fact]
public void Can_insert_async_and_get_sync()
{
    using (var server = GetNewServer(port, path))
    {
        var documentStore = new DocumentStore { Url = "http://localhost:" + port };
        documentStore.Initialize();

        var entity = new Company { Name = "Async Company" };
        using (var session = documentStore.OpenAsyncSession())
        {
            session.Store(entity);
            session.SaveChangesAsync().Wait();
        }

        using (var session = documentStore.OpenSession())
        {
            var company = session.Load<Company>(entity.Id); // the SL client won’t have sync methods!

            Assert.Equal("Async Company", company.Name);
        }
    }
}

The client is making use of Task from System.Threading.Tasks, and in case you didn’t know, this namespace has been ported to Silverlight as part of the CTP for the new C# 5 async syntax.The idea here is to make the async code read synchronously.

Using XunitLight, I was able to cross compile a similar test and run it with the Silverlight Unit Test Framework. Only, it didn’t work.

It doesn’t work because the test is running on the UI thread, and the call to Wait() blocks the execution on the current thread while the Task returned from SaveChangesAsync() executes. This task is making use of WebRequest and deep in the bowels of Silverlight some calls are being marshaled back onto the UI thread. Only that thread is being blocked by Wait(), so nothing happens.

The Standard Solution

The Silverlight Unit Test Framework has some infrastructure for handling this sort of problem. Jeff Wilcox, who built the framework, talks about asynchronous test support in this post. Using the standard approach, I ended up with this test:

[Asynchronous]
[TestMethod]
public void Can_insert_async_and_load_async()
{
    var documentStore = new DocumentStore { Url = "http://localhost:" + port };
    documentStore.Initialize();

    var entity = new Company { Name = "Async Company #1" };
    using (var session_for_storing = documentStore.OpenAsyncSession())
    {
        session_for_storing.Store(entity);
        var result = session_for_storing.SaveChangesAsync();
        EnqueueConditional(() => result.IsCompleted || result.IsFaulted);
    }

    EnqueueCallback(() =>
    {
        using (var session_for_loading = documentStore.OpenAsyncSession())
        {
            var task = session_for_loading.LoadAsync<Company>(entity.Id);
            EnqueueConditional(() => task.IsCompleted || task.IsFaulted);
            EnqueueCallback(() => 
            {
                Assert.Equal(entity.Name, task.Result.Name);
                EnqueueTestComplete();
            });
        }
    });
}

The Asynchronous attribute tells the framework to use a queuing mechanism to execute the test. Then in your test, you must carefully arrange the code with various enqueue methods. This was actually one of the simpler tests, a couple of them had even more nesting of EnqueueCallback. It was functional, but my heart hurt.

I should also note that the code you see here is a mix of several things. From the Silverlight Unit Test Framework itself we have [Asynchronous] and the enqueue methods. (These methods are provided on a base class SilverlightTest). In addition, [TestMethod] is part of the Visual Studio Team Test (VSTT). In my .NET example above, I was using [Fact] from XUnit. The Silverlight Unit Test Framework includes a default provider for VSTT. Finally, the Assert.Equal call is from XUnit (well, from XunitLight).

The Alternative Solution

Writing the tests like this was killing me. So I lifted the syntax used in Caliburn for coroutines and I was able to convert the test to this:

[Asynchronous]
[TestMethod]
public IEnumerable<Task> Can_insert_async_and_load_async()
{
    var documentStore = new DocumentStore { Url = "http://localhost:" + port };
    documentStore.Initialize();

    var entity = new Company {Name = "Async Company #1"};
    using (var session_for_storing = documentStore.OpenAsyncSession(dbname))
    {
        session_for_storing.Store(entity);
        yield return session_for_storing.SaveChangesAsync();
    }

    using (var session_for_loading = documentStore.OpenAsyncSession(dbname))
    {
        var task = session_for_loading.LoadAsync<Company>(entity.Id);
        yield return task;

        Assert.Equal(entity.Name, task.Result.Name);
    }
}

How This Works

I’m leveraging the fact that all of the asynchronous work is performed with Task. In Caliburn, we use IResult for the same purpose. You’ll note that my test returns an IEnumerable of Task. I took the default implementation of the provider for VSTT, and I found the bit where the test was actually invoked and I changed it to this (note that methodInfo contains the test method):

public virtual void Invoke(object instance)
{
    var type = typeof (IEnumerable<>).MakeGenericType(new[] {typeof(Task)});
    if(type.IsAssignableFrom(methodInfo.ReturnType))
    {
        var executor = instance.GetType().GetMethod("ExecuteTest");
        executor.Invoke(instance, new[] {methodInfo});
    } else
    {
        methodInfo.Invoke(instance, None); // this is the original implementation
    }
}

I check to see if my test method returns an IEnumerable<T> and if it does, I then look for a method named ExecuteTest on my test class and I invoke that passing in my actual test method as a parameter.

public void ExecuteTest(MethodInfo test)
{
    var tasks = (IEnumerable<Task>)test.Invoke(this, new object[] { });
    IEnumerator<Task> enumerator = tasks.GetEnumerator();
    ExecuteTestStep(enumerator);
}

Yes, this code is far from bullet proof. It should check to make sure our test class contains an ExecuteTest method (among other things).

Finally, ExecuteTestStep takes each of the Task instances that are yielded and calls the various enqueue methods from SilverlightTest.

private void ExecuteTestStep(IEnumerator<Task> enumerator)
{
    bool moveNextSucceeded = false;
    try
    {
        moveNextSucceeded = enumerator.MoveNext();
    }
    catch (Exception ex)
    {
        EnqueueTestComplete();
        return;
    }

    if (moveNextSucceeded)
    {
        try
        {
            Task next = enumerator.Current;
            EnqueueConditional(() => next.IsCompleted || next.IsFaulted);
            EnqueueCallback(() => ExecuteTestStep(enumerator));
        }
        catch (Exception ex)
        {
            EnqueueTestComplete();
            return;
        }
    }
    else EnqueueTestComplete();
}

If you are familiar with the source from Caliburn Micro, you’ll notice that this is very similar to the logic for executing IResult instances.

The complete source, along with a few examples tests, is available on github.

I’ll likely improve it during the next couple of weeks. For example, there’s no real needs for the attributes when the method returns IEnumerable<Task>.

Feedback is welcome.

Christopher Bennage : Caliburn, Reflection

Improving Asynchronous Tests for Silverlight

The Problem

The Standard Solution

The Alternative Solution

How This Works