Infosys | Microsoft

The biggest and most vexing problem with information today is that it is available everywhere. It is ubiquitous. Information overload is no longer a futuristic term but we are experiencing it now. Think back to an experience where you were hunting for information on a particular topic. Think of the places you ran a search tool on, folders you clicked on, documents you browsed through before you finally were able to create a neat single document / email / text file etc that captured all relevant required information from different sources. And we are not even talking of information that is, say embedded inside a line of business application, a video file or archived somewhere and so on!

Thus, there is a pressing need to:
 - Integrate information from a vast variety of sources, and
 - Filter that integrated information for a given context.

So, what's the best way to go about doing this? At the very outset we must, of course, understand the possible sources of information in an enterprise. That is, what are the various places you would find relevant business information inside a enterprise? To keep it simple and easy to understand, we can divide the information store into two big buckets:
 - Structured Information Stores
 - Unstructured Information Stores

The next immediate question could be: What is the basis for such a segregation? The answer is the way information is stored. When we think of structured information stores, it typically means that the information is stored in 'well known' way. It is classified and tagged in that store so that it is easy to manage and retrieve. The information in a structured store always has some explicit or implicit metadata associated with it. Thus, stores like well known databases such as SQL Server, Oracle, Sybase; ERP systems like SAP, CRM systems, document management systems like Documentum etc are all structures information stores. In contrast, unstructured information stores do not exhibit such a robust and disciplined way of information storage. They are mostly 'anything goes' kind of stores. Various file formats are just stored as dumped files on a storage media in an unstructured information store. We all have some kind of 'miscellaneous' folder full of information debris, which apart from the slightly insulting moniker is still a rich source of information to help get work done. Another key point to keep in mind is that we are just not looking at the enterprise servers (i.e. the Intranet) for this information. Relevant business data can be found in other places such as user's desktops or laptops, mobile devices and of course the Internet.

Another important reason for doing the segregation above is that there are fundamentally different ways of extracting relevant information from structured and unstructured stores. We need to understand how to gather information from both of these sources, integrate the information in a common binding business context and then present that integrated contextual information to the user. Structured Information sources are places from where information is easier to retrieve since they provide well known ways of extracting information from their stores. These could be API / SDK's, Web Services or even descriptive logs. For unstructured information stores, crawling and indexing them through a search engine is an optimal way of retrieving relevant data. Another option could be using the properties of the file system where unstructured information is stored. In both of these cases, one of the key challenges is taking care of security permissions of the requesting user. Since the end user of the information here is viewing structured data in an entirely different application, her permissions need to be matched with the permission of the actual store before returning back contextual information. Another key challenge is performance. Since a lot of information stores are going to be queried, performance issues have to be addressed upfront.

The business benefit of contextual integrated information is illustrated very easily. Just substitute information for people in this example in the previous blog and you will see it. You may have seen the pattern now with both these blogs, that is how for effective collaboration we require people connections and information in the right context. There is of course one more key part to this and that is Communication mechanisms, which will be covered in a later post.