Wednesday, August 31, 2005

Solidsoft are recruiting in the USA! 

I hadn't realised quite how long it's been since I've blogged, oops! Anyway, just to give you an indication as to how busy I am, Solidsoft have now become incorporated in the USA with an office in Chicago and we are currently looking at hiring top BizTalk Architects (initially, anyway!).

The advert has gone up on Monster.com. Here is the specific link to the details:

Click this link to go to job advert details

Monday, December 13, 2004

BizTalk and SOA 

So, it's been a while since I've blogged. Life has been hectic and I'm currently spreading my time between two projects so blog time has been in short supply, even though my blog-to-do list is growing!

Anyway, so what is this post all about? It's about where BizTalk fits architecturally within the new "hot" world of Service Oriented Architectures.

BizTalk Server, as we know, is a middleware product that enables integration between LOB applications and systems in an EAI scenario and trading partners in a B2B scenario. As part of this broad area of responsibility BizTalk is able to execute business processes that implement complex integration or workflow scenarios, transform data from one format into another, route messages based on message metadata or actual content, perform security duties such as encryption and signing, capture tracking information for administrators or business analysts, communicate with a variety of communication protocols and mechanisms and so on and so on and so on. In short, the platform of BizTalk Server offers a complete business process and integration suite, hence the title of Microsoft's new division that is in charge of all of this stuff.

Knowing what we know of the features of the BizTalk Server platform how does that fit into an SOA world, one in which client applications "e;patch"e; together software services that provide specific business functions and offer a cohesive whole? Ultimately, why would you want to put an integration and business process engine into the mix when effectively the client is just calling a web service or two in order to offer specific functionality to a user?

Well, my belief is that BizTalk does offer value in an SOA when it is operating within 3 distinct patterns: Service Broker, Service Aggregator and Integration Enabler.

Service Broker

A service broker sits between the client and the web service and provides brokering services on behalf of the web service to the client. By putting a broker in between, it can act as a secure boundary that allows you to provide controlled access to internal web services. The broker might be the only supported route for consumers of an organisation's web services (see figure 1).

Service Broker
Figure 1: Service Broker

In addition to brokering based on security boundaries, the broker might act as a routing engine capable of determining the correct web service based on either message metadata content (such as HTTP or SOAP headers) or message content itself, in a content based routing scenario (see figure 2).

Service Broker using CBR
Figure 2: Service Broker using CBR

The service broker, depending on caller requirements, may operate in a request-response or one-way scenario.

If the broker operates in a request-response scenario, it is likely that it will provide some form of sync-on-async operation, as in the case of BizTalk Server 2004, in order to be able to provide a synchronous response to the client.

If the broker operates in a one-way scenario, no response is required and BizTalk Server is able to process the request in an asynchronous manner.

It may well be that a response can be asynchronously delivered, in which case the client issues a one-way request to BizTalk Server that brokers the call to the appropriate web service asynchronously (to the client), while further on in the process after a response has been received from the brokered web service, BizTalk Server asynchronously delivers a notification to the client containing the response.

Either way, BizTalk Server is used as middleware to broker calls between a client and its target web services. Introducing BizTalk Server as a broker will have performance implications due to an extra hop between two endpoints, however, the benefits of imposing a routing engine and security boundary offer an attraction to naturally security sensitive organisations that would like to expose business functionality as web services.

Service Aggregator

The service aggregator I tend to term a "super service" - a service that offers an aggregated business function view of discrete separate business functions.

In this scenario, BizTalk Server is used to consume multiple web services and aggregate their results behind the facade of a web service offering a single, probably coarse grained, business function (see figure 3). The aggregator, in effect, offers the ability to create composite applications using business process rules that tie multiple finer grained web services together.

Service Aggregator
Figure 3: Service Aggregator

Creating an aggregator, or super service, in BizTalk Server would involve building an Orchestration (the composite application logic) that enable the results from multiple web service calls to be aggregated together to provide a single response.

Indeed, BizTalk Server, in this scenario, operates as both a broker and an aggregator in one - brokering all calls to internal web services but also aggregating responses.

This pattern, I expect, will be the most oft used pattern with regards to BizTalk Server and an SOA. With the power of the Orchestration Engine and Business Rules, as well as the ability to inspect, manipulate and transform messages, BizTalk Server offers a great deal as a service aggregator within a SOA.

Integration Enabler

The previous two patterns are probably the most obvious with respect to a SOA; however, there is quite often the need to integrate with LOB applications behind the web service facade in an organisation. Just because BizTalk Server may sit between the client and the web service offering brokering and aggregation services, doesn't mean to say that its role as an application integration tool disappears - far from it (see figure 4).

Integration Enabler
Figure 4: Integration Enabler

An organisation may offer a web service that integrates both ERP and warehousing systems to external trading partners. While it is probably quite easy (depending on the interfaces into each application) to write code to achieve this integration, we get back into the mindset that we have been trying to get away from. Writing point-to-point interfaces between applications is bad! It has been proven time and again that P2P interfaces are short term and never stack up, both in terms of costs and support, in the long term.

Therefore, integrating between a web service's private implementation and LOB applications should still be achieved using an integration tool such as BizTalk Server, its bread and butter purpose.


So, in conclusion, I think that BizTalk Server 2004 fits quite appropriately into these 3 patterns of operation within an SOA. BizTalk Server itself is trumpeted as a friend to SOA with whitepapers and patterns and practices on the subject. On the whole, it isn't just marketing fluff and Microsoft is right. However, there are caveats - the most important caveat being performance and scalability. It is possible, it's just that there are pitfalls that you need to be aware of when designing architectures around BizTalk at the heart of an SOA. That, my friend, is an article for another day...

Reserved Keywords and Distinguished Fields 

Just a quick issue I discovered the other day with one of my schemas. I added an element called timeout with some child elements to the schema (days, hours, minutes, seconds), promoted the child elements as distinguished fields and then tried to use the schema in my orchestration.

Every time I tried to use the promoted property days using myMessage.Blah.timeout.days, I kept getting a squiggly red line under timeout and days with the error: unexpected keyword: timeout, cannot find symbol ''

The issue is because the timeout name is an XLANG/s reserved keyword. Therefore, if you want to promote a field as a distinguished field to be used in an orchestration, then make sure that no elements exist that are named the same as XLANG/s reserved keywords in the path up to the root element.

When oh when will Microsoft junk XLANG/s and just use C#???

Monday, November 15, 2004

BTS Guys Understand SOA and Contract First 

Aaron Skonnard has put up a good post about why BTS developers understand SOA and contract first. This is a hot topic at Solidsoft Towers and James and Duncan certainly have championed this at work for some time.

I was talking with James the other day about the ASHX HTTP handler that bypasses all of the unnecessary XML->Object and Object->XML deserialisation and serialisation that goes on. We agreed that as a lightweight 'web service' for BTS running at the core of a SOA architecture it is essential as it reduces the evil of latency that BTS can introduce (due to its primary task of ensuring messages are durable which means trips to SQL Server that could be avoided in a request-response synchronous scenario).

Aaron Skonnard's post can be found at http://pluralsight.com/blogs/aaron/archive/2004/11/11/3440.aspx.

I've been quite busy recently and haven't been blogging as much as I'd like. I'm due to rectify this with some posts on BizTalk Server 2004 and SOA soon, certainly in the performance area as it is possible to do SOA with BizTalk with reasonable latency results provided you know what you are doing.

Tuesday, October 05, 2004

The Bloggers Guide to BizTalk 

There is a tome out called The Bloggers Guide to BizTalk put together by Alan Smith that amalgamates a lot of the BizTalk blogging content into a very handy guide in CHM format. Incredible effort and highly useful! All I can say is WOW!

Alan's blog post can be found here and the guide itself is on GotDotNet here.


Monday, October 04, 2004

DateTime Serialization Part 2 

A colleague has just suggested that it may be because the BizTalk NT Service account may not have a profile, therefore no timezone information. So, setting a promoted property in Orchestration with the value DateTime.Now would potentially produce a time based on GMT with no timezone or daylight savings offsets in effect.

A mightily sound theory, but I haven't been able to try it yet.

Will post here when I've had a chance to do it and prove the theory once and for all...

Duncan's blog post on the subject can be found here

Wednesday, September 29, 2004

DateTime Serialization in BizTalk - Possible Bug? 

On a recent project, I came across an interesting issue with assigning the current date and time to a promoted property in a Message Assignment shape.

The problem highlighted itself in our Audit database where we were auditing process state changes. The submission date of each audited record appeared to be 1 hour ahead than the current local time (based on GMT with Daylight Savings Time in effect - meaning we are currently +1 to UTC).

Our scenario involves writing an audit record from the web site that is initiating the process, then each subsequent audit is performed at various steps within a BizTalk orchestration. Each audit record contains some meta data extracted from the envelope, plus the XML fragment of the header within the envelope and optionally the body from within the envelope.

The first audit record is written by the web site. It uses a class created by xsd.exe which is based on the envelope schema deployed in BizTalk (not a BizTalk Envelope schema though). The code populates the class, then uses the XmlSerialization class to serialise the object into XML.

Each subsequent audit within the BizTalk orchestration effectively creates a new message (based on the envelope schema), updates the messageId, processStatus and submissionDate promoted properties within a Message Assignment shape, and then calls the Audit Web Service passing the message (as a string - long story, but that's the way it currently is) as an argument.

The first audit record has the correct time in the Oracle database, but every subsequent audit record has a time that is 1 hour ahead!

In the web site I'm using the expression DateTime.Now to populate the current date/time in the submissionDate property of my object. When serialised by .NET using XmlSerialization, it converts this date into a Sortable Date Time pattern (identified as 's' in .NET) as defined by ISO 8601. In addition, the DateTime.Now value is in local time, and it takes this into account when building the ISO 8601 date string. For example:


From this date value you can clearly see that the date/time is in UTC format with a +01:00 for the adjustment for daylight savings time. So, this is a correct date/time for British Summertime.

When I set the submissionDate promoted property in BizTalk using the expression DateTime.Now, it should output the same or similar value, taking into account that DateTime.Now returns a local time, and that the date string should encode the daylight savings offset. However, it doesn't...

The following is a date string output from BizTalk. Bearing in mind that the message is serialised into XML and then passed to the Audit Web Service. This is what is received by the web service:


The date/time string format is based on the Universal Sortable Date Time pattern (identified by 'u' in .NET) and is the correct local time but does not have any daylight savings adjustment associated with it! This means that when the Audit Web Service converts the string into a DateTime object using either System.Convert.ToString() or DateTime.Parse(), it takes the daylight savings adjustment into account and adds 1 hour!. It didn't work with DateTime.ParseExact( dateTimeString, "u", System.Globalization.DateTimeInfo.InvariantInfo ) either.

So, in order to fix this, we had to use the DateTime.UtcNow in the orchestration whenever we set the submission date promoted property so that the DateTime.Parse method could take into account current regional settings when converting the string to a DateTime.

Saturday, September 18, 2004

Cool XSD Validation Function for Orchestration 

As I promised in an earlier post, here's a cool XSD validation function that can be called from within orchestration. Just add to a library - I tend to put it into a BizTalk Utilities library full of useful little functions that complement orchestration (most of which should be provided by the orchestration engine, but that's a rant for another day!).

public static void ValidateDocument( XmlDocument businessDocument, string schemaStrongName )
  // Constants
  const int PART_CLASS_NAME = 0;

  // Parse schema strong name
  string[] assemblyNameParts = schemaStrongName.Split( new char[] { ',' }, PARTS_IN_SCHEMA_STRONG_NAME );
  string className = assemblyNameParts[PART_CLASS_NAME].Trim();
  string fullyQualifiedAssemblyName = assemblyNameParts[PART_QUALIFIED_ASSEMBLY_NAME].Trim();

  // Load assembly
  Assembly schemaAssembly = Assembly.Load( fullyQualifiedAssemblyName );

  // Create instance of the BTS schema in order to get to the actual schemas
  Type schemaType = schemaAssembly.GetType( className );
  Microsoft.XLANGs.BaseTypes.SchemaBase btsSchemaCollection = ( Microsoft.XLANGs.BaseTypes.SchemaBase )Activator.CreateInstance( schemaType );

  // Set up XML validating reader and validate document
  XmlParserContext parserContext = new XmlParserContext( null, null, "", XmlSpace.None );
  XmlValidatingReader reader = new XmlValidatingReader( businessDocument.OuterXml, XmlNodeType.Document, parserContext );
  reader.ValidationType = ValidationType.Schema;
  reader.Schemas.Add( btsSchemaCollection.SchemaCollection );
  while( reader.Read() ) {}

In order to call in orchestration, just create an expression shape and use the following:

ValidateDocument( myBtsMessage, myBtsMessage( BTS.SchemaStrongName ) );

Validating using Business Rules 

I tend to get about at work (not in the, ahem, naughty sense!) and while in Norwich a client wanted to validate their document to ensure that the end users had filled in all of the correct details. Simple standard stuff. However, when I explained that validating the document using a pipeline would suspend the message at the first error, they didn't exactly jump with joy!

They wanted to validate the document, but provide all of the errors in one go to the client if the document failed validation.

In order to do this, we proposed following the pattern used by the Swift Accelerator for BizTalk. The accelerator uses a disassembler that implements validation using business rules (run from the disassembler itself) and stores all errors in a separate document. It does not fail at the first error, but simply tries all of its rules and outputs all errors associated with the document in one go by returning the original document plus a document containing the errors.

This allows all sorts of interesting scenarios to occur, such as implementing an orchestration that handles errors allowing resubmission, notification to interested parties, etc. Quite powerful!

In the case of my client, we suggested running the business rules from within orchestration, but the concept remains the same.

Filter Subscriptions on Orchestrations 

I've been working for a large defence client in Portsmouth in the UK and recently come across another limitation in the filter expressions that can be applied to an activatable receive shape of an orchestration.

First some background though. When you drag an operation from a port to an activatable receive, you are effectively creating a subscription for that orchestration in the message box. When you deploy the orchestration, bind it and then enlist, the subscription will be created for that orchestration based on the message type that was defined on the operation that was bound to the receive shape.

This is important: the subscription is based on the message type of the operation defined on the port type. If you look at the subscription viewer and find your orchestration subscription (look for ServiceType of XLANG and the name will be prefixed with Activate), then you'll notice that the subscription has been automatically created with a MessageType filter.

Therein lies the problem. You see, because the subscription has been based on the message type being received, you can't set a filter in the receive shape that points to, say an envelope's promoted properties. This in my opinion is a serious flaw. If I have an envelope that contains meta data about my message, then I want to subscribe to properties on the envelope, but actually receive the message contained in the body of the envelope.

If I want to do this, then I have to define the envelope schema as a normal schema, that is, not a BizTalk Envelope schema. I then ensure that the body element is extensible, in other words, appending an <Any> element with ProcessContents set to Skip and the Namespace to ##any.

Once this has been done, receive the envelope in the orchestration (defined on the receive shape and port type) and then extract, plus optionally validate, the body inside orchestration.

This has an advantage:

The body document can be validated at a 'business' level - effectively allowing any errors from the validation to be handled within the orchestration, rather than being automatically suspended. I'll post some neat code that achieves this in a later post.

However, and this brings me all the way back to the point of my post, if I try and modularise my envelope schema by separating my header schema and then importing it into the main envelope schema, I have a major problem.

The envelope schema exists in one namespace and the header schema in another. The header schema has been imported into the envelope schema. In so far as XSD is concerned, I have a valid schema. In so far as BizTalk is concerned, it can see the header schema and can thus validate the contents (as opposed to using the Any element trick).

However, if I go to set a filter expression on the receive shape, then all of the promoted properties that are defined in the header schema do not display. Only promoted properties of the envelope schema will be displayed!

I've been bitten by filter subscriptions with BizTalk Envelopes and now with imported schemas. I so wish this will be sorted out as it limits the flexibility of the schema design that could be possible, but due to restrictions imposed by BizTalk, isn't.

I would be interested to hear some other view points on this.

BizTalk Hotfix for Business Rules 

It's been a while since my last update, so I'll start off gently with news about a hotfix for business rules if you have installed .NET Framework 1.1 SP1.

Click here for more information.

About the Author

You may be wondering who I am (or may not!), but I've been in the industry 16+ years working on a variety of systems from IBM mainframes as a CICS systems programmer, to developing on Unix and Windows based systems. At the moment, I'm currently working for a Microsoft Gold Partner in the UK called Solidsoft who specialise in systems integration using BizTalk Server. My position is generally dictated by what I'm doing, but normally as a solutions architect/consultant helping clients with their integration projects involving, yes you guessed it: BizTalk Server.

Disclaimer: all the views expressed here are my own and do not necessarily represent the views of Solidsoft Ltd or indeed any other company.