Showing posts with label protobuf. Show all posts
Showing posts with label protobuf. Show all posts

Wednesday, 23 May 2012

WCF Bindings

Carrying on from an excellent MSDN blog post by Rick Rainey comparing the performance of various WCF bindings here is an example including NetNamePipes and ProtoBuf-Net:

Test results

Original blog post:  http://blogs.msdn.com/b/rickrain/archive/2012/05/20/which-wcf-binding-is-best.aspx

protobuf-net

Is available via Nuget or its Google project hosting: http://code.google.com/p/protobuf-net/

AddressAccessDeniedException

If you come across the following exception:

HTTP could not register URL http://+:9001/. Your process does not have access rights to this namespace (see http://go.microsoft.com/fwlink/?LinkId=70353 for details).

Either run Visual Studio using “Run as Administrator” or visit the following link for a more complete resolution:

http://blogs.msdn.com/b/amitlale/archive/2007/01/29/addressaccessdeniedexception-cause-and-solution.aspx

Source

using System;
using System.Collections.Generic;
using System.Linq;
using System.Runtime.Serialization;
using System.Text;
using System.ServiceModel;
using System.ServiceModel.Channels;
using System.Diagnostics;
using ProtoBuf;
using ProtoBuf.ServiceModel;

namespace BindingTest
{
[ServiceContract]
public interface ITestService
{
[OperationContract]
void Execute(TestData data);
}

[DataContract]
[ProtoContract]
public class TestData
{
[DataMember]
[ProtoMember(1)]
public List<string> Data { get; set; }
}

public class TestService : ITestService
{
public void Execute(TestData data)
{
// empty service implementation
}
}

class Program
{
static void Main(string[] args)
{
var bindingElements = new BindingElement[] {
new BinaryMessageEncodingBindingElement(),
new ReliableSessionBindingElement(true),
new HttpTransportBindingElement()
};

var bindings = new Binding[]
{
new WSHttpBinding() { MaxReceivedMessageSize = Int32.MaxValue },
new NetTcpBinding
{
Name = "NetTcpBinding (SecurityMode.Message)",
Security = { Mode = SecurityMode.Message },
MaxReceivedMessageSize = Int32.MaxValue
},
new CustomBinding(bindingElements)
{
Name = "CustomBinding (HTTP/Binary/ReliableSession(ordered)/NoSecurity"
},
new NetTcpBinding(),
new NetNamedPipeBinding(NetNamedPipeSecurityMode.Transport),
new NetNamedPipeBinding(NetNamedPipeSecurityMode.None)
};

var payloadSize = 15000;
var iterations = 5;

OutputTitle(payloadSize, iterations);

// Compare all the bindings in the bindings array.
CompareBindings(bindings, payloadSize, iterations, false);

// Compare all the bindings using protobuf-net
CompareBindings(bindings, payloadSize, iterations, true);

OutputNote();
}

public static void OutputTitle(int payloadSize, int iterations)
{
Console.WriteLine("Test parameters: {0} byte payload, {1} iterations",
payloadSize.ToString("N0"),
iterations);
Console.WriteLine();
}

private static void OutputNote()
{
Console.WriteLine();
Console.WriteLine("NOTE: NetNamedPipe is only used for same machine inter process communication");
Console.WriteLine();
}

public static string GenerateRandomStringData(int stringSize)
{
var r = new Random();
var sb = new StringBuilder(stringSize);
while (sb.Length < stringSize)
sb.Append((char)(r.Next(0, 256)));
return sb.ToString();
}

public static TestData GenerateTestData(int payloadSize)
{
const int itemDataSize = 4096;

var data = new TestData {Data = new List<string>()};

int numDataItems = payloadSize / itemDataSize;

for (int item = 0; item < numDataItems; item++)
data.Data.Add(GenerateRandomStringData(itemDataSize));

data.Data.Add(GenerateRandomStringData(payloadSize - (numDataItems * itemDataSize)));

return data;
}

public static void CompareBindings(Binding[] bindings, int payloadSize, int iterations, bool useProtoBuf)
{
Console.WriteLine("Using {0} serialisation", useProtoBuf ? "protobuf" : "normal");
var data = GenerateTestData(payloadSize);

foreach (var b in bindings)
{
try
{
// Run the test on this binding.
Stopwatch result = TestBinding(b, data, iterations, useProtoBuf);

// Output the results.
Console.Write(b.Name);
//Console.WriteLine(string.Format("Payload Size: {0}, Iterations: {1}", payloadSize, iterations));
//Console.WriteLine(" Binding Elements:");
//foreach (BindingElement be in b.CreateBindingElements())
// Console.WriteLine(" {0}", be.GetType());
Console.WriteLine(", time: {0} ms", result.ElapsedMilliseconds);
}
catch (Exception e)
{
Console.WriteLine("EXCEPTION: {0}", e.Message);
}
}
Console.WriteLine();
}

public static Stopwatch TestBinding(Binding binding, TestData data, int iterations, bool useProtoBuf)
{
var address = GetAddress(binding);

// Start the host using the binding provided
var host = new ServiceHost(typeof(TestService), new Uri(address));
var endpoint = host.AddServiceEndpoint(typeof(ITestService), binding, "");

if (useProtoBuf)
endpoint.Behaviors.Add(new ProtoEndpointBehavior());

host.Open();

var sw = new Stopwatch();
sw.Restart();

// Create a client proxy using the binding provided
var cf = new ChannelFactory<ITestService>(binding, address);
ITestService proxy = cf.CreateChannel();

// Call the service
for (int count = 0; count < iterations; count++)
proxy.Execute(data);

// Close the client proxy and host.
cf.Close();
sw.Stop();
host.Close();

return sw;
}

private static string GetAddress(Binding binding)
{
var machine = "localhost";
var port = (binding.Scheme == "net.pipe")
? ""
: ":9001";

return string.Concat(binding.Scheme, "://", machine, port, "/", Guid.NewGuid().ToString());
}
}
}

Download:  http://stevenhollidge.com/blog-source-code/BindingTestSolution.zip

Friday, 13 April 2012

ServiceStack: REST with ProtoBuf

Service Stack

Service Stack makes services easy! 

You can download their excellent software and examples from the website:  http://www.servicestack.net/

This blog post combines three of their examples into one project: 

  • Web Service
  • REST Services
  • ProtoBuf plugin

Web Services in 5 easy Steps

1. Create an empty ASP.NET Web Application

new-web

2. Add ServiceStack via Nuget

servicestack-via-nuget

3. Create your DTO, Response and Service

// DTO
public class Hello
{
public string Name { get; set; }
}

// Response
public class HelloResponse
{
public string Result { get; set; }
}

//Service
public class HelloService : IService<Hello>
{
public object Execute(Hello request)
{
return new HelloResponse { Result = "Hello, " + request.Name };
}
}

4. Add your Global.asax


Notice typeof(AppHost).Assembly on line 11.  This allows us to pass in multiple assemblies that contain our services.  In this case we’re just working within the one assembly.

using System;
using System.Web;
using Funq;
using ServiceStack.Demo.WebService;
using ServiceStack.WebHost.Endpoints;

namespace ServiceStack.Demo
{
public class AppHost : AppHostBase
{
public AppHost() : base("ServiceStack makes services easy!", typeof(AppHost).Assembly) { }

public override void Configure(Container container)
{
Routes
.Add<Hello>("/hello")
.Add<Hello>("/hello/{Name}");
}
}

public class Global : HttpApplication
{
protected void Application_Start(object sender, EventArgs e)
{
new AppHost().Init();
}
}
}

5. Update your Web.Config


Notice line 6: <location path=”servicestack”>  This creates a virtual directory that will contain your services, in this example we’ve called it servicestack but it could be called anything we like.

<?xml version="1.0"?>
<configuration>
<system.web>
<compilation debug="true" targetFramework="4.0" />
</system.web>
<location path="servicestack">
<system.web>
<httpHandlers>
<add path="*" type="ServiceStack.WebHost.Endpoints.ServiceStackHttpHandlerFactory, ServiceStack" verb="*"/>
</httpHandlers>
</system.web>

<!-- Required for IIS 7.0 -->
<system.webServer>
<modules runAllManagedModulesForAllRequests="true"/>
<validation validateIntegratedModeConfiguration="false" />
<handlers>
<add path="*" name="ServiceStack.Factory" type="ServiceStack.WebHost.Endpoints.ServiceStackHttpHandlerFactory, ServiceStack" verb="*" preCondition="integratedMode" resourceType="Unspecified" allowPathInfo="true" />
</handlers>
</system.webServer>
</location>

</configuration>


That’s it, ready to run!


image


Don’t panic, remember we set a virtual directory to hold all our services (see step 5)?

All we need to do is navigate to the servicestack folder.

Out of the box you get a nice metadata landing page, which lists all your hosted services:


image


If we run the default web service URL we see a nice HTML rendered presentation of our data.


image


Or we can append the format of our choice to the URL:


image


For Google Chrome to be able to render JSON I use the following plugin:


https://chrome.google.com/webstore/detail/chklaanhfefbnpoihckbnefhakgolnmc 


image


?format=JSV would return us the following data:


image


JSV is a compact version of JSON, which by default Google Chrome cannot display in the browser.


NOTE TO CLIENTS: When using other clients to your services you can set the HTTP Accept header on the request to determine which response format you’ll receive.


image


I’m using RestClient-Tool to test my services: http://code.google.com/a/eclipselabs.org/p/restclient-tool/


Rest Services


Two simples steps:


1. Add your Services

using System;
using System.Collections.Generic;
using System.ComponentModel;
using System.Net;
using ServiceStack.Common.Web;
using ServiceStack.ServiceInterface;
using ServiceStack.Text;

namespace ServiceStack.Demo.Rest
{
[Description("GET or DELETE a single movie by Id. Use POST to create a new Movie and PUT to update it")]
public class Movie
{
public Movie()
{
this.Genres = new List<string>();
}

public int Id { get; set; }
public string Title { get; set; }
public decimal Rating { get; set; }
public string Director { get; set; }
public DateTime ReleaseDate { get; set; }
public List<string> Genres { get; set; }
}

public class MovieResponse
{
public Movie Movie { get; set; }
}

public class MovieService : RestServiceBase<Movie>
{
/// GET /movies/{Id}
public override object OnGet(Movie movie)
{
// normally you would return a movie from the db
return new MovieResponse { Movie = movie };
}

/// POST /movies
///
/// returns HTTP Response =>
/// 201 Created
/// Location: http://localhost/ServiceStack.MovieRest/movies/{newMovieId}
///
/// {newMovie DTO in [xml|json|jsv|etc]}
public override object OnPost(Movie movie)
{
// insert a movie into your database... returns the new Id = 999
var newId = 999;
var newMovie = new MovieResponse { Movie = new Movie() { Id = newId } };

return new HttpResult(newMovie)
{
StatusCode = HttpStatusCode.Created,
Headers = {{ HttpHeaders.Location, this.RequestContext.AbsoluteUri.WithTrailingSlash() + newId }}
};
}

/// PUT /movies/{id}
public override object OnPut(Movie movie)
{
// save to db
return null;
}

/// DELETE /movies/{Id}
public override object OnDelete(Movie request)
{
// delete from db
return null;
}
}

[Description("Find movies by genre, or all movies if no genre is provided")]
public class Movies
{
public string Genre { get; set; }
}

public class MoviesResponse
{
public List<Movie> Movies { get; set; }
}

public class MoviesService : RestServiceBase<Movies>
{
/// GET /movies
/// GET /movies/genres/{Genre}
public override object OnGet(Movies request)
{
return new MoviesResponse { Movies = new List<Movie>() { new Movie() { Id=10 }} };
}
}
}

2. Add your routes to the Global.asax

public override void Configure(Container container)
{
Routes
.Add<Hello>("/hello")
.Add<Hello>("/hello/{Name}");

Routes
.Add<Movie>("/movies", "POST,PUT,DELETE")
.Add<Movie>("/movies/{Id}")
.Add<Movies>("/movies")
.Add<Movies>("/movies/genres/{Genre}");
}

Run the app!


Our services have been detected.


image


image


Adding ProtoBuf for Raw Performance


1. Add the plugin via Nuget


protobuf


Which adds WebActivator framework which runs code before the start of application in the ASP.NET pipeline.


ProtoBuf_AppStart is added and will run to handle the configuration.


2. Add Attributes to your DTOs


ProtoBuf requires attributes to explain the de/serialization order for the properties.

using System;
using System.Collections.Generic;
using System.ComponentModel;
using System.Net;
using System.Runtime.Serialization;
using ServiceStack.Common.Web;
using ServiceStack.ServiceInterface;
using ServiceStack.Text;

namespace ServiceStack.Demo.Rest
{
[DataContract]
[Description("GET or DELETE a single movie by Id. Use POST to create a new Movie and PUT to update it")]
public class Movie
{
public Movie()
{
this.Genres = new List<string>();
}

[DataMember(Order = 1)] public int Id { get; set; }
[DataMember(Order = 2)] public string Title { get; set; }
[DataMember(Order = 3)] public decimal Rating { get; set; }
[DataMember(Order = 4)] public string Director { get; set; }
[DataMember(Order = 5)] public DateTime ReleaseDate { get; set; }
[DataMember(Order = 6)] public List<string> Genres { get; set; }
}

[DataContract]
public class MovieResponse
{
[DataMember(Order = 0)]
public Movie Movie { get; set; }
}

public class MovieService : RestServiceBase<Movie>
{
/// GET /movies/{Id}
public override object OnGet(Movie movie)
{
// normally you would return a movie from the db
return new MovieResponse { Movie = movie };
}

/// POST /movies
///
/// returns HTTP Response =>
/// 201 Created
/// Location: http://localhost/ServiceStack.MovieRest/movies/{newMovieId}
///
/// {newMovie DTO in [xml|json|jsv|etc]}
public override object OnPost(Movie movie)
{
// insert a movie into your database... returns the new Id = 999
var newId = 999;
var newMovie = new MovieResponse { Movie = new Movie() { Id = newId } };

return new HttpResult(newMovie)
{
StatusCode = HttpStatusCode.Created,
Headers = {{ HttpHeaders.Location, this.RequestContext.AbsoluteUri.WithTrailingSlash() + newId }}
};
}

/// PUT /movies/{id}
public override object OnPut(Movie movie)
{
// save to db
return null;
}

/// DELETE /movies/{Id}
public override object OnDelete(Movie request)
{
// delete from db
return null;
}
}

[DataContract]
[Description("Find movies by genre, or all movies if no genre is provided")]
public class Movies
{
[DataMember(Order = 1)]
public string Genre { get; set; }
}

[DataContract]
public class MoviesResponse
{
[DataMember(Order=1)]
public List<Movie> Movies { get; set; }
}


public class MoviesService : RestServiceBase<Movies>
{
/// GET /movies
/// GET /movies/genres/{Genre}
public override object OnGet(Movies request)
{
return new MoviesResponse { Movies = new List<Movie>() { new Movie() { Id=10 }} };
}
}
}

Ready to run!


Notice how the X-PROTOBUF column has been added.


image


image


image


image


Framework tests for ServiceStack,Plugins.ProtoBuf can be found here:



Blog Source

http://stevenhollidge.com/blog-source-code/ServiceStack.Demo.zip

Wednesday, 6 July 2011

Comparing ProtoBuf-net Serialisation

ProtoBuf-net is a C# engine for Protocol Buffers, a binary serialisation format created by Google.

The project is written and maintained by Marc Gravell who currently works for Stack Overflow. 

The source for this project can be found here: http://code.google.com/p/protobuf-net/

probuf-net_logo

I’ll be comparing the size of the output along with the speed for serialisation and deserialisation speed for the following methods:

  • Xml Serializer with binary formatter using the [Serializable] attribute
  • Data Contract Xml Serializer using the [DataContract] & [DataMember] attributes
  • Data Contract Json Serializer again using the [DataContract] & [DataMember]
  • ProtoBuf-net v2 BETA (but used by StackOverflow) using the reflection and [ProtoContract] & [ProtoMember] attributes
  • ProtoBuf-net v2 BETA (but used by StackOverflow) using a compiled model with no attributes on my data classes
  • ProtoSharp ALPHA (another C# Protocol Buffers serialisation engine by Torbjorn Gyllebring)

For my tests I’ll be using the following model:

model

Here’s the object model view when populated with data:

Data

Simple to use

Here is an example of how to create a Protobuf-net compiled model as opposed to attributes on the class and fields:

Write and Read from Memory Stream

ScreenShot077

Shared Object References

If you have multiple pointers to the same object you can utilise the graph support added in v2.

This example was taken from a Stack Overflow question answered by Marc Gravell:

http://stackoverflow.com/questions/6063729/does-protocol-buffers-support-serialization-of-object-graphs-with-shared-referenc/6063837#6063837

Conclusion

With Protobuf-net serialisation and deserialisation times are reduced so you should get a quick win out of the box.

If the largest bottleneck in your solution is the transfer speed of data between processes this engine can greatly benefit your project as the data footprint is dramatically reduced.

In order to accurately calculate if your solution would benefit from the use of ProtoBuf-net, you’ll need to set up scenarios that mimic your real world requirements as closely as possible.

Downloads

You can download the source code here:

http://stevenhollidge.com/blog-source-code/Protobuf-net-Tests.zip