Introduction:

In this post, we learn about building a basic search engine or document retrieval system using Vector space model. This use case is widely used in information retrieval systems. Given a set of documents and search term(s)/query we need to retrieve relevant documents that are similar to the search query. 

Problem statement:

The problem statement explained above is represented as in below image. …

Continue

 

This post is the third part of the multi-part series on how to build a search engine –

• How to build a search engine – Part 1: Installing the tools and getting the technology stack ready
• How to build a search engine – Part 2: …

Continue

This post is the second part of the multi-part series on how to build a search engine –

• How to build a search engine - Part 1: Installing the tools and getting the technology stack ready
• How to build a search engine - Part 2: Configuring elasticsearch
• How to build a search engine - Part 3: …

Continue

"Information is the oil of the 21st century, and analytics is the combustion engine" Peter Sondergaard, SVP, Gartner Research

In analytics, we retrieve information from various data sources; it can be structured or unstructured. The biggest challenge here is to retrieve information from unstructured data mainly texts. Here machine learning comes into the picture to overcome this challenge. Different algorithms have been designed in different platforms…

Continue

I am pleased to announce my new book on social media analytics. The book offers concepts, tools, tutorials, and cases studies to understand and analyze the seven layers of social media data, including text, actions, networks, apps, hyperlinks, search engine, and location…

Continue

As a coincidence, I noticed from Google Insights for Search that there was some sort of correlation between web search interests in "American Idol" (singing competition on US television) and "iPhone" (Apple smartphone).  Looking at the time series graph, it appears that American Idol searches are seasonal from January to May each year, which correspond to the months which this television series is broadcast.  As soon as American Idol interest drops in June, searches for iPhone spike in the…

Continue

This is an excerpt from my blogpost Working With Large Data Sets...

For the past 18 months I’ve moved from working on the SMTP proxy to working on our other systems, all of which make use of the data we collect from each connection. It’s a fair amount of data and it can be up to 2Kb in size for each connection. Our servers receive approximately 1000 of these pieces of data per second, which is fairly sustained due to our global…

Continue

I have submitted a post on my blog to describe the basics of monitoring brand using discourse analysis. See http://discourseweb.wordpress.com/2009/08/28/monitoring-brand/