Hosted MongoDB

In my last NoSQL talk, the topic of hosting MongoDB came up.

There’s lots of data-dependant applications that are hosted on third-party systems.  Many of these systems offer data hosting as well, like SQL Server or MySQL.  But, MongoDB is a newcomer to the database arena and hosting services haven’t embraced MongoDB as much as they have with SQL Server or MySQL.

I’ve been looking into various options for using MongoDB when you’re not hosting on your own hardware; fortunately, there’s a few companies that are filling the gap here.  I’ve been taking the point of view that I’ve already got a system hosted and that host doesn’t offer MongoDB services so I’ve been looking at companies that just provide MongoDB hosting.  There’s several, with varying options and varying prices.  There’s many that offer free hosting up to a certain amount of data, and almost all of them offer dedicated processing power.  Here’s what I’ve found so far:

Name URL Free option Dedicated option
MongoHQ https://mongohq.com/ Yes Yes
MongoLab https://mongolab.com/ Yes Yes
MongoMachine https://www.mongomachine.com/ No Yes

 

Some web hosting services are starting to offer MongoDB as an option, like ServerBeach, but, moving your data access to a different provider is much easier than moving your entire application to a different provider so I haven’t delved into these hosting companies much.

Robustness with RabbitMQ in .NET

Recently I’ve been doing a bit of work with queues, in part with RabbitMQ.

Unfortunately, (or fortunately, depending on your point of view) network connections where I am have had a tendency to be unreliable.  That’s a story for the pub; but, needless to say we needed our clients of RabbitMQ to be robust in light of disconnections.  Fortunately, RabbitMQ has ways of dealing with this.

A bit about queues

Queues are a two way communication mechanism.  You can enqueue messages and dequeue messages.  In a distributed system you often have a local queue that you asynchronously deal with when you enqueue and dequeue messages.  MSMQ works in this way.  RabbitMQ can be used in this way too.  Eventually, though, a connection to a remote computer needs to be made for the message to be delivered or received from the queue.  RabbitMQ can support asynchronous communications; which means failure can occur outside of when you give your message to the local queue (or “client”, in RabbitMQ parlance) or receive a message from the local queue.  RabbitMQ doesn’t have the luxury of a the OS having a built-in queue client like MSMQ to deal with connection issues out-of-process—the RabbitMQ client is in-process to your application, so it can be a bit tricky dealing with disconnections.

RabbitMQ supports many “patterns” of messaging like pub/sub, push/pull, etc.  I’ve been working in a more pub/sub model; so I’m focusing on publication and subscription aspects of RabbitMQ

Robustness can be a complex beast; and I’m not going to discuss many of the aspects certain systems need to deal with.  I’m going to take the stance that the queue and it’s client API deals with correct delivery, receipt, acknowledgement, and resending of unacknowledged messages.

With that said, I’ve alluded to two points of failure.  During the enqueue or during the dequeue.  This is where I’ll detail how we can make our system to be more robust with regard to RabbitMQ and disconnections.  With a subscription model in RabbitMQ, the local queue client basically proactively gathers messages for you and places them in a local cache (itself a queue) and you “enumerate” through the messages in the queue.  One way of doing this is with the Subscrption.Next() method.  One overload of Next simply blocks and returns the next message in the queue.  In my circumstance we simply couldn’t block the thread indeterminately like that, so we used a different overload Subscription.Next(int timeout, out BasicDeliveryEventArgs result).  This is effectively the same as the first Next method but will timeout if there are no messages in the queue.  The calling code is then free to do anything else it needs to do, like abort and/or try again.  During the call to Next the connection to the queue server could have been severed.  If it has been severed one of two things will happen. 

In the case of Next(int, BasicDeliveryEventArgs), the BasicDeliveryEventArgs instance may be null but Next will return true.  (i.e. “succeeded” but there’s no message).  This may happen because the connection was lost prior to the call to Next.  This may also happen if the connection is severed while data is being streamed from the server—resulting internally with an end of stream exception.

Another possibility is OperationInterruptedException will be thrown from Subscription.Next.  I’m not sure why things like end of stream exceptions aren’t translated into an OperationInterruptedException instead of returning true and setting BaiscDeliveryEventArgs to null.

In either case, this tells you that the connection is no longer good.  With RabbitMQ you have to try to recreate your connection and try your dequeue again.

Of course, you’re not sure what the problem may be and thus  don’t know when the server may be available again; so, to simply immediately try to reconnect and retry will most likely mean several failures until it succeeds.  e.g. if the server was rebooted due to an update; we might be able to try thousands of times before we’re successful.  To avoid these needless retries, adding a delay in there limits the number of times we retry and gives other threads a chance at CPU time.  I’ve found that when we don’t delay at all, it’s more common that we never seem to be able to reconnect.

Following is some code that deals with disconnections in a robust way.

 

while (!abort)
{


	if (subscription == null)
	{
		try
		{
			subscription = CreateSubscription(connectionFactory, "mainqueue", out connection, out model);
		}
		catch (BrokerUnreachableException)
		{
			Thread.Sleep(1000);
			continue;
		}
	}
	try
	{
		BasicDeliverEventArgs basicDeliveryEventArgs;
		if (subscription.Next(500, out basicDeliveryEventArgs))
		{
			if (basicDeliveryEventArgs == null)
			{
				throw new OperationInterruptedException(
					new ShutdownEventArgs(ShutdownInitiator.Application, 0, "null BasicDeliveryEventArgs"));
			}
			// TODO: something with basicDeliveryEventArgs.Body
			subscription.Ack(basicDeliveryEventArgs);
		}
	}
	catch (OperationInterruptedException)
	{
		// don't bother with connection, it will throw IOException due to disconnection
		using (model) using (subscription)
		{
			subscription = null;
			model = null;
			connection = null;
		}
		Thread.Sleep(1000);
	}


}



 



It, of course, doesn’t go into detail about how to create a subscription instance—I’ve left that up to you to create a subscription how your application needs it, and the disposal of the model and the connection isn’t detailed as your application will implement this code in different ways influencing when and how these two instances need to be disposed.

Certification Caveats

There’s lots of organizations and company’s that offer "certification"–either on their own products/processes or someone else’s.  They’ll all try to tell you the virtues of their certification.  But, despite their claims, certification isn’t a panacea.  Let’s look at some caveats of certification.

The biggest area I’ve seen is what "certification" means.  "Certified" as what, or able to do what?  Really, certification only certifies that the person can and has taken an exam.  Let’s look at software development for a moment.  I’m never going to hire someone whose sole purpose will be to take that exam.  There’s the possibility that they used their knowledge and experience to pass the exam; but the exam itself cannot gauge that.  There’s a certain amount of faith that needs to be put into that certification.

This leads to the next biggest problem of "certifications": the people who either memorized the answers or only studied exam questions.  These people, at worst, posses only enough knowledge on the technology under certification to pass the test.  It’s highly unlikely that anyone with just that knowledge is going to be any use to me.  All joking aside, I have seen people pass certification exams doing only this.  I’ve worked with "certified" peers, "engineers", "PhD’s", etc. that were effectively dysfunctional and ended up being more a burden than a benefit to the team and the project.

This raises the question, what should I be expecting from certifications? Certification alone tells you nothing.  In conjunction with experience and empirical evidence that you yourself have verified about a candidate, certifications can be information that can tell you something more about a candidate; but, rarely should be a mandatory requirement for hire.

Okay, so if certifications are not a panacea, then how do i make them useful?  Well, for one, make them a "nice to have".  At face value they alone can’t tell you a candidate meets your criteria.  This assumes, of course, you have criteria and it’s more than just "passed certification exam".  So, to get value from whether or not someone is certified, you need to know what this candidate will be doing for you and how to judge her work.  If you cannot answer either of those two questions, whether or not someone is certified does you no good.  Okay, so you do know what you need this candidate to do and come up with some valid business-driven criteria to judge their work once hired…  Is there anything about the certification that would help you verify the candidates work once hired?  e.g. I need to hire someone to write a Windows Phone 7 application.  There isn’t a huge pool of resources with loads of experience developing for WP7.  Certification in Windows Phone 7 and experience developing Silverlight or WPF applications would give me the qualitative criteria to make a judgement call over another candidate.

There are circumstances where the certification itself can be the only criteria.  I would argue that it’s fairly rare; but, when new technologies come out and there are simply no one with any quantifiable experience with a technology, certifications can be useful.  It’s rare in the software industry you want someone simply for their knowledge in one particular technology.  Most of the time you’re looking more for someone who has other more general attributes like problem solving, ability to learn, works well under pressure, team player, bleah, bleah, bleah.  My point is, that if they posses none of those other qualities, whether they passed a certification exam is meaningless to you–the candidate is simply unacceptable before even getting to the fact they’re certified.  I would also argue that you need to perform some sort of due diligence and either learn about the technology you hope to hire someone for or get cheat sheet—expecting a certification to be a replacement for that is only going to cause you disappointment.

Criteria for Success

I was having a  conversation with a client recently and the topic of “11th hour” came up.  He seemed to think it wasn’t possible to deliver a software project without some sort of “11th hour” panic.  I disagreed

I disagree because I’ve done it on small and large projects.  Don’t get me wrong; it’s very easy to lose sight of things and easily get a project into a state where “11th hour” panic is inevitable.  But, it’s not something that can’t be avoided.

One of the problems with software projects, it seems, (and with other projects, I suppose) is losing sight of things and not having a “destination”.  This is what I refer to by “criteria for success”.

Criteria for success is basically the criteria by which what is expected to be delivered will be evaluated.

This criteria no only gives you a focused destination, it also provides a focus for the team.  Without some sort of criteria it’s hard to gauge whether the team is side-tracked on something that doesn’t add value to the project.  Criteria for success not only gives you a destination to strive for; but also means your efforts can be focused on that destination and be able to get to that destination quicker.  This means the project can be completed quicker and progress can be judged more accurately.

Without defining criteria for success, software projects that do succeed succeed by accident.  I don’t know about the rest of the industry; but I’m not really comfortable with completing my work “by accident”.

Defining “Done”

Defining “Done” is a very similar.  Defining done is another popular mantra; but, I think “criteria for success” is more clear in its goal in that it’s almost explicit that acceptance criteria is what we’re looking for.  “Done” can be a little wishy-washy and open for interpretation.

Defining Criteria

I don’t think it’s enough to complain that people have a problem defining success or “done”; so, I think it’s also important to help people get their own criteria for success.  i.e. people have a problem coming up with criteria for success because no one has defined criteria for success for defining criteria for success—so to speak.

There’s various tools to help with this sort of thing; the most obvious is Agile.  Defining user stories and getting acceptance from stakeholders on these stories is an excellent first step.  Many people simply view user stories as a check-list of items.  This can be extremely useful; but explicitly defining criteria (acceptance criteria is the more general term) isn’t an explicit task for most people.

Often we can’t expect out stakeholders to know how to define the acceptance criteria.  “Just do it” or “make it work” are often the definition of done—which doesn’t leave us much in the way of unambiguous.

Agile kind of get’s away from “requirements” because requirements are often associated with waterfall methodologies and people expect that “requirements” mean a full set of requirements for the project before the project commences and doesn’t change for the life of the project.  This really isn’t what requirements are; but, we’ll continue using terms like “criteria for success” and “defining ‘done’” to avoid these associations.  But, theories and practices for effectively defining requirements can give us some valuable tools for defining good criteria.  Karl Wiegers has a couple of excellent books that has some great criteria that we can reuse for our criteria:

Correct

Is the criteria correct (easy to say)?  Is how we’re judging the deliverable right?  If we’re judging x against y, is y what will actually be used as the criteria?  If it’s really z, we can’t deliver something that satisfies y on purpose.

Feasible

Is the criteria even attainable?  If we simply can’t reach that criteria, we’re going to fail…  This can be a bit tricky because we don’t always know whether what we’re actually working is actually possible.  With Agile, we like to separate what is known to be possible from what is not known to be possible with “spikes”.  We purposely research things we’re not sure are possible (or not entirely sure how long things may take) into spikes so we can gather the information we need to define our criteria for success correctly.  This mitigates the risk and gives is the ability to separate and time-box these aspects of the project because if we’re not really sure if it’s feasible or how long it’s going to take we need to prioritize it differently and explicitly.

Necessary

This is kind of obvious; but without this explicit criteria our criteria may lack focus and our work not be prioritized correctly and we end up working on tasks that are not important (i.e. not necessary).  It’s critical that the criteria actually define necessary qualities.

Prioritized

I’ve already alluded to this; but prioritized criteria is essential for getting a project done properly.  “Prioritized” is a bit ambiguous; so, it’s important to define what we mean by prioritization.  The most essential part is that criteria is prioritized in relation to other criteria.  There’s a tendency for everything to be prioritized as “important” and this has the opposite effect because it also means everything is defined as “unimportant” and the order that tasks get performed is open to interpretation (ease of implementation, interest in the task, coolness of the task, etc. none of which are stakeholder criteria).  If it’s not important enough to prioritize a task correctly, it’s not important enough to actually perform the task.

Unambiguous

It may seem obvious; be we need to make sure our criteria is not ambiguous.  It’s easy for criteria to be open for interpretation and end up being ambiguous.  See the next point…

Verifiable

The criteria that Weigers’ details is that the criteria needs to be verifiable.  It’s not enough to say “done” or “works”, we need to be able to unambiguously and empirically verify our work against specific criteria.  Criteria like “works” just doesn’t cut it. This is stakeholder-driven.  Often we’ll need to help our stakeholders think through what they’re ultimately going to use for verification—we want them to think about this before we begin the work, not after (which is very common).