Are you interested in IBM Watson Explorer vs Elasticsearch? Recently, I had to do some comparisons between IBM Watson Explorer and Elasticsearch for a project. I spent some time going through the features of both platforms and found some interesting features and comparisons. For this comparison, I’m only looking at IBM Watson Explorer Foundational Components and the core Elasticsearch component.
What is Watson Explorer?
IBM Watson Explorer (WEX) is a full-fledged search application that allows users to search across multiple data sources with ease. WEX ships with several different data connectors to allow easy ingestion of data into the search engine. It features search capabilities that you would expect from an enterprise search engine and includes two different fully built user interfaces: the classic WEX Foundational Components legacy search interface and IBM Watson Explorer Application Builder (AppBuilder). The AppBuilder interface allows for rapid application development and an entity model that can build relationships between your data. It also allows for easy to create widgets on the search and entity pages. IBM Watson Explorer is an on-premise application.
What is Elasticsearch?
Elasticsearch is an opensource search and data analysis engine. It’s primarily developed by the Elastic company. It has the standard search engine features you would expect and is great at indexing and querying text but it’s also designed for data analytics and numerical data. Elasticsearch allows you to query your data, to summarise it, calculated averages, and more. I said that this post would focus on the Elasticsearch component itself but I still need to mention that it is the central piece of an ecosystem of products called the Elastic Stack. The other tools help you visualize data (Kibana), ingest(Logstash, Beats), and manage data that is indexed in Elasticsearch. Elasticsearch, like Apache SOLR, is built upon the open source Apache Lucene search engine library.
Watson Explorer VS Elasticsearch
In general, I find that WEX and Elasticsearch are pretty comparable in the indexing and retrieval of search results. Both seem to support common search functionality like advanced query syntax, faceting, etc. Elasticsearch seems to be the leader if you’re looking for data analytics but I’m more concerned about an enterprise search scenario.
I wouldn’t think of Elasticsearch as a fully flushed-out search application. It’s really only the index and a JSON search API that you could build your application on top of. WEX is more of a full-featured application that is made for rapid application development of search apps. WEX also features a SOAP and REST API (XML) that can be leveraged by other applications.
I find that WEX is much easier to ingest data into as it comes with a connector framework that supports everything from websites, to file systems, to SharePoint, to database systems, etc. It appears that you would need to build out these connectors for Elasticsearch and push documents into the index.
Data conversion is another area I explored. While WEX uses older XML/XSLT data converters, they work very well to modify documents prior to ingestion. The idea of virtual documents is also nice where you can ingest documents at separate times but tie them together using a vse-key so that the data appears as one single WEX document. Elasticsearch doesn’t include a conversion framework out-of-box. However, there are some libraries such as Apache Tika to help with document metadata and things like that. Still, it appears that data conversion needs to be done in a separate program with Elasticsearch.
As far as user interfaces go, Elasticsearch doesn’t seem to have an officially supported enterprise search interface that is available as part of the open-source package. There seem to be a few GitHub projects trying to tackle this though. The Elastic company does have a beta Workplace Search that seems to be targetting the enterprise search space but is limited in connectors and conversions still.
The AppBuilder UI is really a strong point of IBM Watson Explorer. It allows you to build search applications extremely fast. It has the ability to build customized data widgets in Ruby, HTML, CSS, and JS. It leverages that entity model that I talked about before to draw relationship between your data.
Watson Explorer also features a great admin user interface for developing and managing search collections and features.
What about the cost? This is something that is hard to determine. On one side the open-source Elasticsearch component is free while Watson Explorer is probably hundreds of thousands of dollars for a license. I don’t know how much WEX Foundational costs to be honest. Though WEX has a higher upfront cost it can make up for some of that in less development time since it is fully featured. I think that Elasticsearch projects would have to include more development time built in just due to the lack of features. So there is a bit of a trade-off in costs.
One place where Elasticsearch probably wins is the ability to find developers that are familiar with the tool. Being an open-source tool, Elasticsearch has an advantage that just about anyone can learn the application. I’d also venture to guess that Elasticsearch has a much wider adoption than Watson Explorer so you won’t have as difficult of a time finding developers for Elasticsearch. You also have the option to go to Elastic.co to get help from their expert services team. For WEX you need to go directly to IBM or find a good Watson Explorer consultant.
Conclusion
I have found that WEX is the best choice if you are looking for a fully-featured enterprise search engine with data connectors, conversion, and a friendly UI all built into the solution. You can get WEX up and running very quickly and minimize development time and effort. You can do proof-of-concept type work in no time to flush out your ideas. However, since WEX is a full application it is less flexible with customization than Elasticsearch.
If you’re looking to really build out a custom application around a search library, then Elasticsearch might be the right solution for you. Since Elasticsearch isn’t as full-featured as WEX it gives you more flexibility in what you want your application to be. Another benefit of Elasticsearch is that many cloud providers such as AWS are offering cloud-based search services based on Elasticsearch.
Note: It also seems clear to me that the Elastic company is much more interested in the data analysis functionality of the engine versus the enterprise search space. I’m basing this on their marketing but they do have some small interest in enterprise search as evident by the Workplace Search.
My Bias: Full Disclosure
I have been working as an IBM Watson Explorer Consultant for the better part of my career. That being said, I may have some bias towards the WEX tool simply because I know its feature set better than I know Elasticsearch. I gladly welcome you to tell me how wrong I am, hopefully in a nice way. I’d love to learn that there are features I simply missed with Elasticsearch such as a good connector/conversion framework.
