Sunday, May 8, 2011

Introduction to IL Rewriting with Cecil, Part 1–Rewriting FizzBuzz and the Art of Redirecting Method Calls

lesson1

The simplest possible code example that anyone can learn from.

Introduction

When I first started learning IL rewriting and Cecil about several years ago, one of the difficulties that I struggled with was the fact that there were very few practical examples on how to take an existing assembly and modify it at runtime. In many ways, I was stranded in heavily undocumented territory, and needless to say, this lack of documentation made it very difficult to learn how to do anything useful with Cecil.

Meanwhile, in the Year 2011…

It’s now 2011, and I think it’s safe to say that for many people, IL rewriting (much less Cecil) is still a “big mystery wrapped in an enigma containing frustration”. Indeed, Cecil is an incredible library that can let you do some incredible things, but at the same time, it can be very frustrating since the learning curve is still steep and there are still no practical guides for using it. As a user, there must be some sample code out there that shows how to do the most basic tasks with Cecil, right?

Establishing the Feedback Loop

In order to learn any skill (such as IL rewriting), we need to establish a simple feedback loop that allows users to easily experiment with the tools they are given so that they know:

  • What went wrong if it doesn’t work
  • Where to fix it if it breaks
  • How to see the results of their experiments without getting mired in the implementation details of the tests themselves

In this case, we’ll need to set up a basic environment that will let users experiment and learn how to modify assemblies at runtime with Cecil. We will need:

  1. A test fixture that loads a sample assembly and gives users the chance to modify it before reloading the modified assembly into memory (An NUnit base fixture)
  2. A way to display/diagnose any invalid assembly errors that occur due to making changes to the original assembly (PEVerify)
  3. To make it easy to change so that we can experiment with different approaches to modifying IL, thus “closing” the feedback loop

Lost in Bytecode

Given these requirements, where would we even begin? It’s not every day that one decides to randomly parse .NET assemblies and learn how to change the underlying bytecode that ultimately defines their behavior. This can seem like a daunting task for even the most intelligent of budding interlopers, but fortunately for my readers, most of the work has already been done for you in these examples. All you need to do is sit back and scroll down the page, as I proceed to tell you the “ins” and “outs” about Cecil, and the practical lessons learned from rewriting IL. With that in mind, let’s get started!

A WriteLine for Another WriteLine

One of the simplest things that you can possibly do with Cecil is to swap a single static method call for another static method call with the same parameters and the same return type. (It’s a fairly simple operation since both methods have the same signature, and you don’t need to add any additional instructions to make it happen). In this case, I opted to swap all calls to Console.WriteLine() with calls to FakeConsole.WriteLine():

As you can see from the example above, I used a simple LINQ query to identify all the call instructions that needed to be modified. More experienced Cecil users will probably notice that I decided to rewrite all method calls to point to FakeConsole.WriteLine() instead of individually checking to make sure that the method call I was replacing was indeed Console.WriteLine(). Indeed, that was an intentional move, given that the FizzBuzz.Print() method doesn’t make any other external calls to any other methods besides Console.WriteLine().

Assuming that I somehow created an instruction that caused an invalid modified assembly, however, how would I be able to know what went wrong, much less know how to fix it?

PEVerify, how do I love and hate thee…

As it turns out, there is a tool called PEVerify.exe that can tell you whether or not the assemblies that you modify with Cecil are valid or invalid. For example, if I were to remove all the IL instructions out of the FizzBuzz.Print() method, PEVerify would give me the following error message:

peverify1

(Believe me, it’s much prettier when it’s zoomed out)

PEVerify will examine any given assembly and be able to tell you whether or not the compiler (or in this case, you, the human compiler) made any mistakes in creating the assembly. It can be a very useful tool, and that’s why I modified the sample test fixture to run PEVerify right after the user modifies the sample assembly. If you don’t already have PEVerify installed, make sure you download it and configure the Lesson1 app.config file to point to where PEVerify is installed:


Once PEVerify has been configured as part of the tests, the rest is up to your imagination.

Exploring Method Replacement and Beyond

Now that the basic IL rewriting setup has been laid out for you, the onus is on you to explore the possibilities with Cecil and IL rewriting, even if it means that you have to start with some small, basic steps. In the next installment in this series, I’ll show you how to use PEVerify and Cecil to keep the stack balanced so that you can do things like swap static method calls for instance method calls, and even do things like install runtime hooks so you can change your code as your application is running. Stay tuned!

Wednesday, May 4, 2011

Dynamically Intercepting Thrown Exceptions with LinFu.AOP 2.0

exception

A screenshot of LinFu dynamically catching a thrown exception.

On Error, Resume Interception

Another useful thing that LinFu.AOP allows you to do is to intercept (and rethrow) exceptions within your applications at runtime. LinFu makes it so easy, in fact, that all you have to do is add the following lines to your CSProj file:


Exceptionally Simple

To use LinFu.AOP's dynamic exception handling capabilities, all you need to do is make the following call to handle all exceptions being thrown in your application:


Try/Catch Me, If You Can

The call to ExceptionHandlerRegistry.SetHandler tells LinFu to hook the SampleExceptionHandler into your application so that all exceptions that will be thrown will automatically be handled by the given exception handler. Under normal circumstances (where interception is disabled), the call to account.Deposit() will cause the app to crash, but as this example shows, LinFu.AOP was able to intercept the thrown exception before it could crash the rest of the app.

What makes this even more interesting, however, is the IExceptionHandlerInfo instance that describes the context from which the exception was thrown:


More Information Than You Can Throw An Exception At


The IExceptionHandlerInfo interface has enough information to describe the method that caused the exception, as well as having properties such as ShouldSkipRethrow that allow you to decide whether or not LinFu should just swallow the exception and keep running the program as if an exception was never thrown. The ReturnValue property, in turn, allows you to alter the return value of a given method in case you want to resume the method and provide an alternate return value as if no exceptions were ever thrown.

As you can see, LinFu.AOP makes it really easy to transparently handle exceptions in your applications, and if this post saves at least a few developers a few headaches from having to manually diagnose their applications, then I'd consider it to be a gratifying success.

Enjoy!

EDIT: You can get the code examples for LinFu.AOP's dynamic exception handling here.

Thursday, April 28, 2011

Intercepting Console.WriteLine and Other Third-Party Method Calls with LinFu.AOP 2.0

ConsoleInterceptor

Worth a thousand words

In case you are wondering, yes, that is a screenshot of LinFu.AOP intercepting calls to Console.WriteLine() at runtime.
One of the more useful things that LinFu.AOP can do is intercept calls to third-party assemblies that aren’t necessarily under your control. In fact, LinFu makes it so easy that all you have to do to make the interception happen is add the reference to LinFu like the following lines to your CSProj file just like I did with my SampleLibrary.csproj file:

<PropertyGroup>
<PostWeaveTaskLocation>$(MSBuildProjectDirectory)\$(OutputPath)\..\..\..\lib\LinFu.Core.dll</PostWeaveTaskLocation>
</PropertyGroup>
<UsingTask TaskName="PostWeaveTask" AssemblyFile="$(PostWeaveTaskLocation)" />
<Target Name="AfterBuild">
<PostWeaveTask TargetFile="$(MSBuildProjectDirectory)\$(OutputPath)$(MSBuildProjectName).dll" InterceptAllMethodCalls="true" />
</Target>

‘Automagically’ Delicious

Once you reload and rebuild the solution, LinFu.AOP will automatically modify your code after the build runs so that you can intercept it at runtime. LinFu does this by adding hooks to your code so you can change it as the program is running. In this case, I casted the modified BankAccount class to an IModifiableType instance so that I could add my custom ConsoleInterceptor instance:



// Create the BankAccount class just like normal...
var account = new BankAccount(100);
// Notice how LinFu.AOP automatically implements IModifiableType so you can intercept/replace method calls at runtime
var modifiableType = account as IModifiableType;
if (modifiableType != null)
modifiableType.MethodCallReplacementProvider = new WriteLineMethodReplacementProvider();

account.Deposit(100);
The WriteLineMethodReplacementProvider class, in turn, determines the method calls that should be intercepted at runtime:

public class WriteLineMethodReplacementProvider : IMethodReplacementProvider
{
public bool CanReplace(object host, IInvocationInfo info)
{
var declaringType = info.TargetMethod.DeclaringType;
if (declaringType != typeof(System.Console))
return false;

// We're only interested in replacing Console.WriteLine()
var targetMethod = info.TargetMethod;
return targetMethod.Name == "WriteLine";
}

public IInterceptor GetMethodReplacement(object host, IInvocationInfo info)
{
return new ConsoleInterceptor();
}
}

Choosing which methods to intercept

As you can see from the example above, this class ensures that only calls to Console.WriteLine() are ever intercepted. The ConsoleInterceptor itself is responsible for replacing and intercepting the Console.WriteLine() method itself:

public class ConsoleInterceptor : IInterceptor
{
public object Intercept(IInvocationInfo info)
{
var targetType = info.TargetMethod.DeclaringType;
var target = info.Target;
var targetMethod = info.TargetMethod;
var arguments = info.Arguments;

Console.WriteLine("Intercepted method named '{0}'", targetMethod.Name);

// Call the original WriteLine method
targetMethod.Invoke(null, arguments);

// Console.WriteLine doesn't have a return value so it's OK to return null)
return null;
}
}
The most interesting part about the code example is how LinFu.AOP adds the method call hooks without touching a single line of the source code. All of the IL rewriting is done behind the scenes so you won’t have to worry about the gory details of using an AOP framework in your legacy code. The beauty of this approach is that it allows you to intercept any method call, even if that method call is a part of the .NET base class libraries.
You can find the LinFu.AOP.Examples library here at Github.
NOTE: Please intercept BCL method calls responsibly.

Wednesday, April 27, 2011

Beyond Duck Typing with LinFu.DynamicObject: Creating Types that can Change at Runtime

A Post-Easter Egg

One of the hidden features that LinFu.DynamicObject has is the ability to dynamically add properties and methods to itself using a shared type definition at runtime. In other words, you can have two or more LinFu.DynamicObject instances share the same DynamicType, and any changes you make to that type will be propagated to all LinFu.DynamicObject instances that share that same type:
using LinFu.Reflection.Extensions;
using NUnit.Framework;

namespace LinFu.Reflection.Tests
{
[TestFixture]
public class DynamicTypeTests
{
[Test]
public void ShouldBeAbleToShareTheSameDynamicType()
{
var typeSpec = new TypeSpec() { Name = "Person" };

// Add an age property 
typeSpec.AddProperty("Age", typeof(int));

// Attach the DynamicType named 'Person' to a bunch of dynamic objects
var personType = new DynamicType(typeSpec);
var first = new DynamicObject();
var second = new DynamicObject();

first += personType;
second += personType;

// Use both objects as persons
IPerson firstPerson = first.CreateDuck<IPerson>();
IPerson secondPerson = second.CreateDuck<IPerson>();

firstPerson.Age = 18;
secondPerson.Age = 21;

Assert.AreEqual(18, firstPerson.Age);
Assert.AreEqual(21, secondPerson.Age);

// Change the type so that it supports the INameable interface
typeSpec.AddProperty("Name", typeof(string));
INameable firstNameable = first.CreateDuck<INameable>();
INameable secondNameable = second.CreateDuck<INameable>();

firstNameable.Name = "Foo";
secondNameable.Name = "Bar";

Assert.AreEqual("Foo", firstNameable.Name);
Assert.AreEqual("Bar", secondNameable.Name);
}
}
}

Evolving Ducks

Most of the code above is self-explanatory, and the most interesting part about this code is the fact that it has two DynamicObject instances that share the same DynamicType instance. Once the Age property was added to the DynamicType definition, both first and second DynamicObjects automatically ‘inherited’ the additional Age property that was added to the DynamicType at runtime. Another interesting piece of code was the duck typing call to the IPerson interface, which wasn’t possible until after the Age property was added:

// Use both objects as persons
IPerson firstPerson = first.CreateDuck();
IPerson secondPerson = second.CreateDuck();

firstPerson.Age = 18;
secondPerson.Age = 21;

Assert.AreEqual(18, firstPerson.Age);
Assert.AreEqual(21, secondPerson.Age);


As you can see from the example above, LinFu.DynamicObject is smart enough to change its definition every time the attached DynamicType definition changes, and that’s why it was also able to duck type itself to the INameable interface:

// Change the type so that it supports the INameable interface // Change the type so that it supports the INameable interface
            typeSpec.AddProperty("Name", typeof(string));
            INameable firstNameable = first.CreateDuck<INameable>();
            INameable secondNameable = second.CreateDuck<INameable>();

            firstNameable.Name = "Foo";
            secondNameable.Name = "Bar";

            Assert.AreEqual("Foo", firstNameable.Name);
            Assert.AreEqual("Bar", secondNameable.Name);
Pretty straightforward, isn't it? (LinFu.DynamicObject has had this feature for well over 4 years, but I never got around to publishing it until now). Enjoy!

Tuesday, April 26, 2011

Duck Typing with LinFu & C# 4.0’s Dynamic Keyword

The Lame Duck

When C# 4.0 came out with the dynamic keyword, I was pretty excited over the prospect of having Ruby-like features finally being baked in to the C# language itself, but as one of my twitter followers and friends pointed out in one of their posts from a few years ago, C# 4.0 still lacks duck typing support, and the .NET BCL doesn’t seem to have anything that could do something similar to the following code:

public interface ICanAdd
{
int Add(int a, int b);
}
public class SomethingThatAdds
{
private ICanAdd _adder;
public SomethingThatAdds(ICanAdd adder)
{
_adder = adder;
}
public int FirstNumber { get; set; }
public int SecondNumber { get; set; }
public int AddNumbers()
{
return _adder.Add(FirstNumber, SecondNumber);
}
}

There has to be some way to construct an object at runtime and map it to an ICanAdd interface, but the problem is that the current .NET Base Class Libraries don’t seem to have a solution for this problem. As Dave Tchepak pointed out in his post, the following dynamic code will fail miserably at runtime:

public class Dynamic : DynamicObject
{
Dictionary<String, object> members = new Dictionary<string, object>();
public override bool TrySetMember(SetMemberBinder binder, object value)
{
members[binder.Name] = value;
return true;
}
public override bool TryGetMember(GetMemberBinder binder, out object result)
{
return members.TryGetValue(binder.Name, out result);
}
}
// ..and the test would look like:
[Test]
public void CannotUseDynamicAdderForAnythingUseful()
{
dynamic adder = new Dynamic();
adder.Add = new Func<int, int, int>((first, second) => first + second);
var somethingThatCanAdd = new SomethingThatAdds(adder); /* Fails here at runtime */
somethingThatCanAdd.FirstNumber = 10;
somethingThatCanAdd.SecondNumber = 20;
Assert.That(somethingThatCanAdd.AddNumbers(), Is.EqualTo(30));
}



The problem is that the runtime isn’t smart enough to figure out that there has to be a duck-typing cast to the ICanAdd interface in order to use the SomethingThatCanAdd class, and that’s where LinFu’s DynamicObject comes in handy.


If it walks and quacks like a duck, then it’s all good


LinFu.DynamicObject is flexible enough that it can let you build object instances at runtime and then ‘strongly’ duck type those object instances to any interface that matches the intended duck type. In this case, we need to find a way to build up something that can map to an ICanAdd interface instance so that it can be used by the SomethingThatAdds class:

[Test]
public void CanCreateADynamicAdder()
{
var adder = new DynamicObject();
CustomDelegate addBody = delegate(object[] args)
{
int a = (int)args[0];
int b = (int)args[1];
return a + b;
};
// Map LinFu's DynamicObject to an ICanAdd interface
var linfuDynamicObject = new DynamicObject(new object());
var returnType = typeof(int);
var parameterTypes = new Type[] { typeof(int), typeof(int) };
linfuDynamicObject.AddMethod("Add", addBody, returnType, parameterTypes);
// If it looks like a duck...
Assert.IsTrue(linfuDynamicObject.LooksLike<ICanAdd>());
// ...then it must be a duck, right?
var somethingThatCanAdd = new SomethingThatAdds(adder.CreateDuck<ICanAdd>());
somethingThatCanAdd.FirstNumber = 10;
somethingThatCanAdd.SecondNumber = 20;
Assert.AreEqual(somethingThatCanAdd.AddNumbers(), 30);
}
 The call to the DynamicObject.CreateDuck() method does all the heavy lifting for you so you don’t have to worry about the details of how to make the object behave like a duck. It just works, and that’s the power that LinFu offers.
(EDIT: You can grab the source code and examples for LinFu.DynamicObject here at Github)
 
Technorati Tags: ,,

Sunday, April 24, 2011

A CQRS and an AOP expert walk into an Oredev bar…

5166054200_220bec62b0_b

A mind-bending experience

Back in November 2010 of last year, I remember that a stranger walked up to me at the Green Lion Inn Bar after the Oredev conference and told me that my notion of aspects, and of object-oriented design was actually a limited version of some of the message passing capabilities of some old language named Smalltalk.

He asked me to imagine for a moment that everything in OOD is a message. In original OO terminology, he said, there were methods, and messages. Whenever you call a method (in today’s terminology), what you are implicitly doing is sending a message to an object telling it to execute the code associated with that message (the method).

For me, the concept of methods executing as a result of implicitly sent messages seemed fairly straightforward, but the discussion took an interesting turn.

He said, “Imagine that these messages can be sent from anywhere. They can be running within the same process, running on your machine, or they can even be coming from a machine that is located halfway across the world.”

The concept of Remote Procedure Calls wasn’t anything new—but the next thing he said piqued my interest.

“If every method is merely a message to be sent, then any method call can be remoted [if you rewrite it with IL].”

In effect, what Greg was proposing was to create a messaging system that was so granular that one could transparently take any method call on any given machine and have it execute on another machine or process without modifying the original source code. In theory, one could use such a system to split a single process so that it runs parallel on multiple machines, all without modifying the original source code.

In other words, he was proposing automatic parallelization of any application using AOP, and IL rewriting.

Needless to say, my jaw hit the floor, and I was left speechless.

NDC Talk Preview: “Ten Simple Rules for Rewriting IL on the Common Language Runtime”

Introduction

One of the things that frustrated me when I first started learning how to write (and rewrite) IL back in the early 2000’s was the fact that there were very few practical examples on how to do anything useful with IL. In fact, the only reliable learning tools that were available around at the time was PEVerify.exe, and IL disassembler. In the six to seven years since I first started writing/rewriting IL, I often asked myself what it would take to write a useful IL tutorial that would be easy enough for someone with a minimal C# background to learn, and yet be informative enough that it would be useful to someone who has written an AOP framework of their own.

Fast forward to 2011

frustrated

Over the years, I have learned some really hard lessons about IL, and now that I’ll be speaking at NDC 2011, it will be the first time that I will get to share what I have learned with the .NET community. Make no mistake—IL writing can be a “mystery wrapped in frustration containing an enigma”, but the good news is that I have been through many of those pitfalls, and now I can help other developers understand what’s going on when those errors occur, and I can even show you how to fix those errors.

Ten Lessons Learned

In fact, there are ten basic principles that you can follow that will make it easier for you to understand and work with IL writing/rewriting. These principles are:

  1. Rule #1: If you don’t understand IL, then learn from the C# compiler
  2. Rule #2: Always keep the stack in a balanced state; One push = one pop
  3. Rule #3: Don’t forget to explicitly box/unbox your types
  4. Rule #4: PEVerify.exe can be your best friend and worst enemy at the same time
  5. Rule #5: Use Console.WriteLine() to triangulate errors in your IL at runtime; Use it well, and use it often
  6. Rule #6: GOTO statements in IL are a Good Thing™
  7. Rule #7: Emit as few IL instructions as possible
  8. Rule #8: Almost anything can be modified, including sealed classes and final methods
  9. Rule #9: You can replace any given method call as long as you replace it with a method that leaves the stack in the same state
  10. Rule #10: IL can be learned like any other programming language; practice makes perfect.

Many of these lessons learned are fairly straightforward, while some principles will require a bit more explanation during the talk. No matter what your level of experience in IL might be, I guarantee that this is one informative session that you won’t want to miss. Stay tuned, and I’ll see you all at NDC 2011!

Saturday, April 23, 2011

I’m joining Readify in Sydney, Australia!

readify

Way more than the average job

After a year of working for an investment bank in Hong Kong, I realized that I needed a job more meaningful than just churning out number-crunching apps for people that I would never meet or rarely ever see. Opportunity, as the saying goes, rarely ever knocks twice at a person’s door, and when Readify said that they would let me work for them after I turned them down last year to go to Hong Kong, I knew that this opportunity would never come again, and I decided to take it. It’s not every day that one gets a chance to work with some of the best developers in the .NET community, and I am humbled by the fact that I will get the chance to work with them.

Hong Kong, in Retrospect

In hindsight, living in Hong Kong has been a very comfortable experience. The city is relatively small, it has an excellent public transportation system, and you can practically get anything you want within a few keystrokes on your keyboard, a call from your mobile phone, or just by going to Mongkok to shop for the latest gadgets. For me, the problem wasn’t living in the city per se—it was the idea of having to indefinitely work on the same business apps within the same business domain for years upon years on end. Working in investment banking IT has certainly been a lucrative venture for me, but I wanted a role that would let me go out and help other clients and developers be more effective at what they do. At the same time, I wanted to work within the confines of a relatively smaller and close-knit company that gave their people more control over how they work rather than having them go through so many processes because of legal restrictions or mandated institutional regulations.

To Sydney, and Beyond

3563650121_b659b9de96

Shortly after my upcoming talk at NDC 2011, I’ll be packing all my things and I’ll be moving to Sydney, Australia to work with the folks at Readify. There’s no telling what the future holds for me, but if this job opportunity is any indication, then the future looks bright indeed.

Friday, April 22, 2011

Speaking about “Ten Simple Rules for Rewriting IL on the Common Language Runtime” at NDC 2011

 

Red-Pill-vs.-Blue-Pill

Red or Blue

Have you ever wondered what happens under the hood of an AOP framework? Have you ever tried writing IL, only to get hit by an unintelligible wall of PEVerify.exe errors?

Learning how to (re)write IL can be hard, but in this session, I will show you some techniques for rewriting almost any .NET assembly with the least amount of effort and the most minimum amount of frustration possible. The art of rewriting IL will give you the ability to change programs as if you (the developer) are the compiler itself. With IL rewriting, you can add, remove, replace, unseal, and redirect method calls from any type. You can change almost any method you can possibly imagine, including methods in third party libraries that don’t have any source code publicly available. The possibilities are endless, and there’s a whole other hidden world that goes on when you compile your code into IL using your favorite .NET language, and this talk will show you just how different things are once take the red pill.

Join me at NDC 2011 in Oslo at 11:40AM-12:40PM on Track 5, and I’ll show you just how deep that rabbit hole really goes.

Ratings by outbrain