Rules for Framework Development

Something i came across while i was developing a framework for one of my clients.

Source : Brian Button's Blog

Brian’s Handy Dandy Rules for Framework Development

  1. Clients come before frameworks
  2. Ease of use trumps ease of implementation
  3. Quality, Quality, Quality
  4. Be an enabler
  5. It’s a people problem

1. Clients come before frameworks

This was easily the most commonly heard bit of advice my poll revealed, and it matches up very nicely with my own experiences. The best frameworks are not created, they are mined out of existing code. The worst frameworks seem to have sprung from the twisted imagination of a rogue architect somewhere…

2. Ease of use trumps ease of implementation

As the author of a framework, you need to keep your eye on the ball of making others’ jobs easier. If you ever have to make a tradeoff between changing something to make it easier to implement and changing something to make it easier for your framework clients to use, always choose the latter. Write good documentation, ship your unit tests, and test out your APIs through writing lots of in-house client code. Lots of eyes on the API helps as well

3. Quality, Quality, Quality

Use Test Driven Development on your framework code, and enable framework users to use it on theirs. This means to avoid things like sealed and overzealous use of CAS. ‘Nuff said.

4. Be an enabler

You can’t read people’s minds, so don’t presume to know what their key uses of your framework will be. Allow people to embrace and extend your framework through extension points. Document these extension points, give lots of examples, and create automation to help users with these tasks.

5. It’s a people problem

Building a framework is decidedly not a difficult technical problem. We’ve built these things before, we’ll build them again. It is primarily an issue of balancing people and their conflicting interests. Good architects know this and are adept at balancing these concerns.

Why I’d like to visit Malaysia more often!

Finally I managed to make it for the IASA Executive meeting at Genting, Malaysia, and boy was it awesome…


Apart from the wonderful weather and excellent hospitality of the members of IASA Malaysia and Microsoft Malaysia, it was a pleasure knowing so many wonderful people who foster community spirit and give a damn about it…


I met many council members of IASA Malaysia who are really passionate about community spirit. It goes to show that the very word ‘community’ is much more than running a forum or conducting monthly meetings or administering a user group. Much much more…


It’s the community Spirit. A passion. A truly intense desire to help and create awareness which is not driven by monitory needs and does not provide any monitory benefits. What could drive these people in this age to spend so much time and effort to do something that does not provide visible, immediate, tangible benefits? Why would people who are working in full time jobs, leave their busy, busy day jobs and bother about community? Why?


Because they see beyond the tangible benefits, they see far beyond their immediate and work needs; they see IT as a community which solves problems and creates awareness. These people are bounded by this very vision, a brotherhood so to speak, where everyone has one goal, one vision, and where everyone is driven by the spirit of community.


It was my honor to meet people like Daniel Tan (Executive Director for PATIMAS COMPUTERS), Aaron Tan Dani (Microsoft Architect Evangelist), Yap Ke Lip (Director for Nuzo Systems), Teh Tiong Keen (Solution Architect for ING Insurance), Chin Koon Siang (CEO for Startfos Consulting), Jimmy Yeoh (CEO for MAAGNET Systems), Lau Bee Eng (Consultant for Microsoft Services, Malaysia), Liang Yong Soon (Technical Director for Persys Technologies), Mike (Senior Manager for iTrain), Ho Chin Kuan (Lecturer for Multimedia University, Malaysia), Vico Vicknesh (Microsoft BCC Program Manager) and many others who left their busy hectic job schedules and made it all the way to Genting.


Microsoft DPE Lead in Malaysia – Tyson Dowd also joined us at the end of the day and was very supportive of the community efforts. A big thanks to Tyson and Microsoft Malaysia to even take care of my accommodation and making this community event a success with all the support provided over the past months.


I must admit, I have never seen such warm welcome and hospitality that I’ve received during the past 2 days in Malaysia.


Thanks a lot Tyson and Aaron for putting so much effort into this community and believing in community spirit. Also, thank you so much for believing in the MVP program and taking such wonderful care of the needs of MVP’s. So could I come over as a Malaysia MVP? 😉
Also thanks for assuring that any of my business needs in Malaysia will be met by Microsoft Malaysia. You guys rock!
Looking forward to your invite again! 🙂


Thanks TK for taking care of logistics.


Thanks Mike for the awesome time after the Executive Meeting in Genting.


Thanks Vico for dropping me off at the right place.


And thanks to everyone who supports this sprit of community and devote their time and effort for it.


Special Thanks to Genting Highlands for sponsoring this Meeting and nurturing the spirit of community.


Cheers to this very spirit!

Does SOA require ESB?

More of my notes on Udi Dahan‘s podcast on – Does SOA requires ESB?


 


Does SOA require ESB?


 


A single system that will be divided by autonomous services.


 


Autonomous services as opposed to simple services, that a lot of the industry is talking about, add another layer of richness into what goes into a service and what a service’s behavior is. If we were to take a concrete example of what the difference between a simple service and an autonomous services is, an example of simple service would be something like address validation service – which is a passive kind of service that while provides a much needed functionality doesn’t contain any data of its own and doesn’t really do any work on its own, until someone asks it a question. Whereas an autonomous service example would be a marketing service or sales service or a business level service, that even if no one asks it to do anything, has its own business, has its own data, does its own kind of thing when it needs to do it.


We can see the different kind of behavior and the different kind of interaction we’d have with each one of those services.


While with a simple service, passive service if you will, the kind of interaction we have is pretty much low level request and response – gimme something and the service gives it back.


With a business level services, an autonomous service, the kinds of interactions change quite a bit, since an autonomous service has to behave a certain way, has its own data, that has to version and manage, wont just let anybody come in and mingle its own data. The kinds of requests its exposes to the outside world, its message schema would be quite a bit different, obviously richer in the number of message types it would be willing to accept. So we are talking about two totally different scales of service. So in a context of distributed system, when we go about dividing that system into services, I would suggest going the autonomous services route, and not focusing so much on system level of low level passive services.


So once we go to that level of primary building block, for building a distributed system and we recognize that a unit of distribution is an autonomous service. That is not to say that a single autonomous service has to run in a single autonomous server. A single autonomous service can run on many servers, it could run on an entire datacenter, it could run on multiple datacenters. Rather the way that we wanna look at it is saying – things that we are gonna distribute are going to be autonomous services.


The opposite of that is that if it is not an autonomous service, we will not be distributing it in the same way that we distribute autonomous services. This is very important to prevent us from going down to the distributed objects path of creating tiny little distributed objects that chat amongst themselves over the network as if everything was done in memory.


Now, what exactly does an autonomous allow you to do with it. The message schema than an autonomous service exposes is exactly that, the message schema belongs to the service. You will not be sending these messages to another autonomous service, there are not general purpose messages that happen to be floating around in the air. A message is defined by the autonomous service that exposes it, that same autonomous services has an endpoint or many endpoints where it is willing to accept those messages, other autonomous services have their own messages.


So you wouldn’t be having the same message going to a number of autonomous services. So going back to the distributing thing, when you as a client or another service need to communicate with other autonomous services, the very act of saying – I’m going to send this message, you already know where it has to be sent to. So it becomes an implementation detail, where that autonomous service’s endpoints are is not an application level concern for you. You just wanna say, send this message, where it is supposed to sent to is defined in the implementation detail of your higher level bus type layer. That bus would know that this kind of message needs to be sent to that autonomous service, and that autonomous services has these endpoints and it will send it to that endpoint.


So we’ve simplified the kinds of communications we have, everything is  a message, you can’t do a RPC with an autonomous service. Simply because the autonomous service just won’t let you. Autonomous service will not be holding resources open and threads just waiting around for you to do a whole bunch of tidy conversations with it. You need to bundle up everything you need to say in a message, you send it to the service and if it feels like it, it will send you a message back at some point in the future – but don’t hold your breath.


So getting back to issue of the bus, what is a bus and what do we need the bus to do?


First, when we are talking about the API which we want the bus to expose, we’ve already identified one really simple method in there, which is send the message. This method takes the message as its parameter. The application cant say, where the message should be sent to, it doesn’t know what the destination is, doesn’t know what the endpoint is.


If someone did know that when they were writing code, it would change when the system got deployed. So there is no way in the world we can have our application level code dealing with destination of messages. So a bus, when we talk to it, send a message, and the bus will just handle where the message has to be sent to. So that’s the first thing. The other side of it is how we receive messages, or rather the way the application hooks into the bus when my autonomous service via this bus receives a message, what happes? Well if we just look at it at a purely logical level, a message is handled by a message handler. A message handler is a logical construct that embodies the service layer code of what to do with a  message when it arrives.


So what we would like to do on the autonomous service side when it comes to receiving messages is to register a number of message handlers with the bus that in essence, we are saying to the bus – when this message arrives, dispatch it to this message handler, when a different kind of message arrives, dispatch it to a different kind of message handler. And that’s about it. We don’t have to use receive semantics or synchronous receive semantics, because the flow themselves are asynchronous. The fact is that we are bundling up a message, sending it off, that’s it! You don’t have to do all weird multithreading things within your message handlers that can deplete your thread pool and give you terrible performance and heavy loads and cause your service to crash. It really is very simple.


When you autonomous service hooks into the bus, it registers a bunch  of message handlers and that’s about it. Once a bus receives a message, it just passes it along to a message handler.


There is one more special case that a bus should support and that is client side code. Because once the autonomous services use a bus to communicate with them, then end user application need to talk to the autonomous service using that same bus protocol.


Now request response is something that needs to be done when it comes to client side code. Client side code wants to say – Add this customer and it really needs to get a response back sometime. And it needs to be able to correlate the response message to the original request message. This is something else that the bus should do.


So we have identified two and half major areas that a bus is responsible for.


1.      Sending a message


2.      What to do when a message arrives – dispatch it to the message handler


3.      How to deal with very basic request response semantics for client side code


Anything beyond that, we are already getting to workflow and workflow and persistent workflow and all those interesting stuff that happens when you go to route messages is totally out of the scope of the basic bus functions.


Now what is that a bus has to give us?


1.      Send message when a message arrives


2.      Dispatch it to the message handler


Now if we take a look at MSMQ or rendezvous or JMS, etc, it’s really rich, has a lot of power and you don’t really need it in the application level.


Back to the example of Add customer.


The client code as a result of user clicking OK on a forum would request the bus to send an ‘Add Customer’ message. And that ‘Add Customer’ message would be routed by the bus to the ‘Customer’ autonomous service, or maybe our ‘Sales’ service or maybe our ‘CRM’ service. That ‘CRM’ service would have bus take the message once it arrives and pass it to our ‘Add Customer’ message handler. That ‘Add Customer’ message handler would in some way or another talk to the database and save the customer details in the database, and would now like to send the data of the customer back to the client. Obviously the most important thing is getting that customer id back to the client.


So there are really two thing that we need to send back


Data – The customer id


That we were successful


This can be done is a number of ways, in this concrete example we’ll use one that I found very successful:


It is to send that same ‘Add Customer’ message back with an error code saying success. Your message was successful and this is the exact message you sent me and it was successfully handled. That is one message that the service would send back to the client. The client will resolve this and register success to the user and close the forum.


Now we still haven’t handled how the data gets back from the ‘Add Customer’ service to the client.


The nice thing about using a bus and this entire asynchronous messaging style of communication is that you can have more than one response messages, which is really great. So on the service side we got our ‘Add Customer’ message handler and after sending the above mentioned message, it sends back another message which is the ‘Customer Details Updated’ message, where it contains all the data about the customer.


The nice thing about this kind of message is that the customer service can decide that not only it wants to send the message back to the same client which sent the request, but it also wants to publish it, to all the clients that are interested in this message. So we get much richer semantics that are available once we go the asynchronous messaging route. We get a lot more capabilities of deciding what we want to say to who, when and have that all embodied in messaging.


So you can’t really do SOA or you can’t built loosely coupled autonomous services without having a bus between them.

Does SOA mean the end of OO?

Just to share some notes I took from Udi Dahan‘s excellent podcast – Does SOA mean the end of OO?


 


Why did we need SOA in the first place? And what are we using OO for?


Both of them are attacking the problem of software development – OO from bottom up and SOA from top down.


 


Let’s start looking at OO design of a given application.


 


OO was started in the single process era – a single process in a single application on a single thread on a single machine.


 


If you are building a single app, SOA probably is not going to solve any problems that OO does not cover.


 


Presently, in a typical application, we have numerous processes on numerous machines on numerous datacenters.


 


25 years ago, when the idea of distributed objects came, it didn’t do wonders. OO failed to bring the advantages it gave to single process, single applications to distributed applications. So we lurched from OO.


 


SOA came in this scenario advocating services. The interactions between objects are different from that of services. The interactions between objects are often chatty, in process, we except each call to occur blindingly fast. Whereas, when we are calling inter process or inter machine, we have to acknowledge the fact that there is network latency and that we don’t necessarily know what the distance is between those two computers that are talking. So we can’t take  the same object oriented semantics and make services behave the same way, or we will get the same problem that we had with distributed objects.


 


So that’s really when SOA came in and said that what you have are services. Service is something that is quite a bit larger than an object. Services will probably contain numerous objects within itself, managing its own state, whether in memory or in a database. Being hosted separately and being run on a separate machine.


 


Facts that SOA brings to the table is that distributing is a primary concern and it totally affects the way services interact together.


 


Services interact differently than objects, services will be built probably out of objects in a OO language.


 


So does SOA mean the end of OO? Absolutely not!


 


It solves a different problem, not only different problem, it solves a problem that was previously solved very poorly by OO. It came to fill a void. And that is really the coexistence between services and objects. The fact that SOA is the future doesn’t mean that anything in the past is no good anymore.


 

International Association of Software Architects – Singapore Chapter

We’re in the midst of establishing the Singapore chapter of IASA, and tomorrow is the key members gathering to discuss some of the crucial issus like the constitution. 


The Singapore IASA chapter is located here (Please excuse us, for the site is not fully functional yet).


For those who do not know about IASA, its an international community of Software Architects with many chapters all around the world; you can read more about the Association here.


We’ve managed to secure some of the most prominent members of Singapore IT community to join us in establishing this chapter.


The Singapore chapter will be one of the first exclusive association for software Architects in Singapore.


With all this excitement in the air, I just want to thank all you guys who are joining us as council members for all your support and enthusiasm. Especial thanks to Aaron Tan Dani who is the APAC Chairman of IASA.


Cheers!

The BMW Analogy : Why must there be a UAT?

An interesting conversation sparked up today when I was together with my users assisting them for UAT.


UAT. User Acceptance Test. It’s a norm that is inserted in all software development lifecycles. It’s a part of software development. To us in Info Tech, UAT is something we don’t really give a second thought to. We learnt it in our schools, we’ve seen our seniors prepare for it when we started our careers, we’ve heard our colleagues talk about it. There is nothing special about it – to us it has become a process that must be followed.


Ever stopped to ask why? Why should there be a UAT?


Well, this is the very question, these interesting users asked about.


Why must there be a UAT?


It’s the first time I’ve heard that question!


Now the users who have been conducting UAT for my projects – be it for any company, are generally IT educated. UAT is a norm that they have been following in their companies. It’s a part of software development in their companies. UAT is something we dont really give a second thought to. They’ve learnt it in their schools, they’ve seen their seniors prepare for it when they started their careers, they’ve heard their colleagues talk about it. There is nothing special about it – to them it has been a process that must be followed. The company says so… (See the trend?)


But this user was asking us today – Why must there be a UAT?


And the interesting analogy (considering that the company I am working for is one of the best solution developer companies in the world) that he gave was this:


User : If I buy a customized car from BMW, does the salesperson tell me “Thanks for the tons of money you paid for this car. Now drive this car around, and if the steering wheel comes off while you are driving, do come back to us and we will fix it for ya – for free!”
Me : Errrmm…. (Thinking : Is this a rhetorical question? Or do you want me to answer it?)


Hey, hey, hey, hold your horses… I know being in IT, you can rebut the BMW analogy with a million bullets, so can I…


But I took a step back and thought from the BMW point of view, can we make solutions that do not require UATs? Solutions that users can buy just like they buy products?


Just a thought…