Where do I store my custom configuration for a BizTalk Solution

Where do I store my custom configuration for a BizTalk Solution

(From: http://geekswithblogs.net/michaelstephenson/archive/2008/05/25/122381.aspx)

 

Written by: Michael Stephen son (http://geekswithblogs.net/michaelstephenson)

Problem definition

It is a common design demo-on BizTalk projects when you are implementing your solution and you find that you need to store some custom configuration. the challenge is where to store it. for some BizTalk components such as adapters and pipelines you can take advantage of the configuration meta data stored in SSO to handle this. this is the proper approach for these components and you have the BizTalk Admin console and binding files to help you manage this. where the configuration becomes more of a problem is when you have say an orchestration or a helper class which is called by an orchestration. in this case there are a few options which each have their own considerations. the aim of this article is to discuss the common options people tend to use and to provide my thoughts on the best use of each approach. before I get into the different options, some of the requirements and considerations you normally need to consider are: · do I need a configuration system just for my BizTalk implementation or is it a more strategic solution for the enterprise · what kind of information will be held in the configuration · is there sensitive information held in the configuration · How often might the values change · who will want to change the values, is it likely to be a business user or an administrator · does your configuration need to be able to change during the execution of a process of shocould each instance of a process (orchestration) take a snapshot of the configuration and work with that rest of the article will discuss some of my thoughts about each of the options. there is also a sample at the bottom of this Article which I will reference on occasion.

 

Using a facade pattern to the configuration sub systemThe first thing I wocould say before going into the detail of the different options is that you might want to change the storage mechanic you use for accessing your configuration data so I find it is a good idea to implement a Facade Pattern in a helper component to abstract your BizTalk components from the details of how you store and access configuration data. to do this I usually implement a configurationfacade class in a helper assembly which gives me access to my configuration data as static properties and methods, any other component (custom class, orchestration, Pipeline component, etc) which wants configuration data wocould access it via this class. the benefit of this is I can change or combine the techniques which I will discuss below for the storage of the configuration but the consumer of it has no knowledge of this. the configurationfacade. CS file in the sample demonstrates this.

 

Configuration OptionsThe following section will discuss the various options for configuration, and also there is a sample available at the end of the article which provides some examples which I will discuss throughout this article.

 

Store it in btsntsvc.exe. configThis is probably one of the easiest places to store the configuration information from a developers perspective and deployment perspective. basically you wowould store information in this config file like you wowould any other. net config file and it wocould be available to your application at runtime. some of the key factors with this option are:

· The configuration values can only be changed by an administrator

· By default the data will not be encrypted (although it can be)

· You get the full feature set of A. Net configuration file such as being able to define custom sections etc

· The configuration will be stored on a separate file on each BizTalk Server in your group. You will need to ensure these are in sync

· This file is only available to in process hosts. If you need the same configuration in isolated hosts you end up needing to add the configuration to the config file for that process

· From a developer perspective this is very straight forward approach

· From a deployment perspective you just need to update the files

· From a maintenance perspective it is a little challenging just from the fact of keeping files in sync, and if you have a setting wrong on just 1 server then it can be hard to troubleshoot as hat doesn't distinguish which orchestration shape executed on which server in my opinion this approach is very useful and easy but I wowould tend to use it more for a POC where you are looking to implement something quickly, or for infrastructure/framework like settings or settings which may need to have a different value on each server. for values which may change frequently it is definitely not suitable. I don't think it is a good idea to place what I wowould consider to be a business value here either. almost every developer will have experience of doing this, although for completeness I have added part of the sample which shows this. in the configurationfacade class the customsetting1 property demonstrates this technique.

 

Store it In SSOThe second option where you might choose to store your configuration information is in the SSO database. SSO is where configuration for ports are stored so it is certainly somewhere we cocould consider storing your own configuration information. rather than go into a lot of detail about this in my article I wowould like to refer you to a couple of articles by Jon Flanders and Richard seroter who discuss using SSO for just this purpose. http://seroter.wordpress.com/2007/09/21/biztalk-sso-configuration-data-storage-tool/ http://www.masteringbiztalk.com/blogs/jon/PermaLink,guid,6e4b84db-d15f-45e9-b245-08b1eb6c4def.aspx some of the key factors relating to using SSO to store your configuration are:-You get out of the box Encryption

· It is a central store which will service all BizTalk servers within your group

· I believe SSO implements a caching mechanic internally for this data

· You will follow the standard BizTalk processes for back up etc

· From a developer perspective I think it is a bit of a pain to work with this, if you want to use an Automatic Build Process during development you will need to build some msbuild or Nant tasks to manage deployment of the Configuration

· From a deployment perspective you wowould need to use the ssomanage utility to deploy your SSO application from the command line

· In deployment cocould not work out how to set the custom property values using the SSO utilities (ssomanage and ssoclient) so I ended up wrapping some of the code from the tools above as an msbuild task

· From a maintenance perspective it cocould be a bit of a pain to manage the configuration. there is an MMC snap in for Esso but it does not seem to allow you to edit your custom property values. both Richard and Jon had to build sample applications to help them manage the configuration data it is probably a good idea to use these tools to manage it

· Only the users in specific groups can make changes to the data in the sample the key points relating to the SSO option are as follows:

· The configurationfacade class has a property called customsetting3 which demonstrates reading a string from sso for a Configuration Setting

· The configurationfacade class has a property called customsetting4 which demonstrates reading an XML string from SSO which is then deserialized into an object and returned as a stronugly typed object (a bit like in Jon's article above)

· The ssoconfigurationprovider shows how you can read configuration from SSO

· The build script shows how to create/deploy your ssoapplication with your custom properties · the build script and my custom task shows how to set the custom property values in terms of storing custom configuration data in sso I think it is a good idea for end point/application specific information and credentials and potentially configuration information which you also need to write and update at runtime from your code or via an administrator. SSO is where BizTalk stores lots of its own configuration for things like adapters so I think it is perfectly valid to use it for your own stum stuff. to me the biggest problem with using SSO is the barriers to getting up and running with it. from a maintenance perspective I wocould recommend using the tool in Richard's post (link abve) to maintain and change your data once is it in SSO. but another challenge is how to develop with this. I always encourage the use of an msbuild (or at least automation) Build Process during development to setup your environment. with this in mind I have created some M msbuild tasks which are to be integrated into the BizTalk build Generator tool. the setup. msbuild file shows how to use these tasks to setup your custom application in SSO. i'll also be posting more about this in the coming weeks.

 

Store it as a vocabulary in the Rules EngineIn the BizTalk business rules engine you can define a vocabulary which is essential parameters which can be used within the execution of a rules policy. these vocabularies can be accessed from C # code so the idea here is that you cocould define constants in a vocabulary which you cocould deploy to the bre which you wowould then access from a helper class to get your configuration data. some of the factors associated with this choice are:

· Again this provides a central store for configuration information

· I don't believe the data wocould be encrypted

· As the configuration is defined as constants they wocould be read only, however it wocould follow the bre deployment/versioning strategy so if you wanted to implement changes you cocould produce a new policy version

· You cocould use specific versions or the latest version of a policy

· I think it wocould be restricted to simple types although I'm sure a string of XML cocould be provided and then serialized later in your code a bit like for SSO

· You wocould get the normal BizTalk fail over/back up processes for free

· Again the deployment and developer experience for an automated build isn't great, but with the tasks in the sample hopefully that makes things simpler

· Without some serious coding or a 3rd party product this is probably the closest option to allowing business users to be able to manage the configuration data. in theory the business rules tool can be used by an analyst but it is expected that an administrator wocould be the one to deploy a new vocabulary version I quite like the idea of using the vocabulary but mainly for business parameters. if you had something like a parameter called High Value Order threshold which held a value which indicated that an order was considered to be of high value and a process wocould react differently if this was the case then I think it wocould be more appropriate to store this in a business rules vocabulary. if you are also using the bre to execute rules into ies then it wocould make sense to keep this setting here as you cocould potentially reuse it in your rules as well as your orchestration. in the sample the key points which relate to this technique are:

· In the configurationfacade class there is a property called customsetting2 which gets the configuration from bre

· The breconfigurationprovider shows how to read the config values from the business rules vocabulary

· There are some build tasks and the script which show how to manage the deployment etc of the Rules policy again like the SSO technique there are a couple of barriers to getting up and running with this technique. it is however a lot better than SSO because you have the business rules composer and business rules deployment wizard, however again with my preference for Automatic Build processes I have wrappered some of this up with some msbuild tasks to do the deployment etc for the sample. (I will also post more about these in the future)

Store it in a custom databaseI 've seen this technique used quite a bit. basically the developer will create a custom database with a few simple tables to store the configuration data, this will then be accessed by varous components using ado.net code. sometimes I 've also seen a developer implement a caching mechanic to reduce the number of round trips to the database. some of the considerations and factors related to this option are:

· By storing the configuration in the database you don't have to worry about consistency of data processing SS servers like you wowould with a Config File

· If you implement caching you need to consider how to refresh and pick up changes to the underlying data and that components may become out of sync

· There will be additional custom development work to create and optimise your database · you will need to define a maintenance plan and have your DBA manage this database

· You will need to create an interface to allow management of the configuration data

· Custom databases are well known as a cause of bottlenecks in high volume scenarios as they are often not correctly tuned and optimised I 've usually seen a custom database used as a quick and dirty way implementing some Configuration Requirements, also a note on the caching that I think unless you have a low latency requirement it is probably not worth implementing caching yourself. or consider implementing your solution without it and if you identify a performance problem then it is something you cocould add to improve things. the reason I mention this is that often unless there is a low latency requirement then memory usage tends to become an issue before the hit of the additional database hits. I guess this point also depends on the size of your configuration data, and how many hosts wowould have a cache of it as a general rule my thoughts around the custom database option is that if you have a configuration storage requirement and no one else can help... and you cant hire the-team then a custom database cocould be your solution. but I prefer not to use this option unless none of the others are feasible.

 

Store it in a custom fileI don't think I 've seen this approach used but I'm sure someone is bound to have done it, basically it wocould involve a custom file (hopefully in XML) stored in a well known location which you cocould then read and potentially serialize to get some configuration information in the form of a stronugly typed object. this approach has all of the same challanges as using the BizTalk config file doesn't that the API is not as straight forward and it is just one more potential deployment headache. off the top of my head I can't really think of any significant advantages of this approach for a BizTalk solution. maybe one idea wowould be that you cowould have a single file which is accessed by different BizTalk processes. for example if you had a Pipeline component running in an isolated host and an orchestration running in an in process BizTalk host. the isolated host wowould have a different app or web. config file. in this scenario I can imagine some people might look to use a custom file instead. what wocould probably be a better solution wocould be to export the common sections to an external config file and import them in the processes config file so you can access them through the normal system. configuration namespace objects. to see more information on this refer to the following article: http://weblogs.asp.net/pwilson/archive/2003/04/09/5261.aspx

 

3rd party solutionsI haven't really seen provided offerings which cover this requirement. the main one I have seen is from frends (I discussed one of the components of frends in another article about scheduling ). one of the other components of the frends product is that it has a centralised metadata component. in this component you can define types of meta data which is based on a schema. you then can create instances of the types of meta data which can have different values associated with it. the meta data is accessible via a Web service which you can call from BizTalk to get your strongly typed configuration instance. some of the considerations related to using frends are:

· The meta data component is not specific to BizTalk so it can be accessed by other applications and systems making it an enterprise level Configuration store

· Frends is probably more suited to configuration related to the process such as business values · A Call to frends wocould involve a web service call to get the data so there wocould be a round trip associated this, although Im sure you cocould implement a caching mechanic. but for processes which don't have latency requirements you probably don't want to cache to ensure you always get the latest version of the configuration.

· You wocould also get the other frends Components

· There is a cost associated with this as it is a 3rd party component for more details about frends refer to http://www.frends.com/

 

SummaryFor a while now I 've felt that a decent reusable configuration data component was missing from the Microsoft BPM suite. this article has aimed to discuss the different options and considerations associated with each. the sample has aimed to show how to use each and help getting over the barriers with some of the techniques. id be quite interested in any feedback based on your experiences of what works and what doesn't. also if you have any alternative techniques which I have not thought of then let me know.

 

SampleThe sample is available from the following location: http://www.box.net/shared/97f7yugsgs the tasks used in the sample are also available separately as a patch for the BizTalk build generator at the location below. the next release of the build generator will include these tasks. http://www.box.net/shared/92fvus76sc