Copy and Modified Documents with a Watson Explorer Converter

A common task when crawling and indexing a document in Watson Explorer Engine (WEX) is making changes to a document during the conversion process. The most common occurrence is needing to copy all the contents in the application-vxml document while making some changes to one or a few of those contents. To do this, there is a recursive copy template that can be used. I’ll show you how to apply it.

First, I’m going to use the out-of-box “example-metadata” collection. Navigate to that collection and click the test-it button.

wex collection screenshot test it

After clicking test-it you will see a listing of documents. Click on the test-it button for the “blowout” record.

watson explorer test-it results

On the resulting page, scroll down and look at the conversion trace. There is a converter called “Create Metadata from Content”. This is the converter that ships with WEX to convert the HTML files into v:xml documents. Each of the links on the left side represent input and output of that conversion step. We want to click on the output of this converter to see what the document looks like.

watson explorer conversion trace

You will see the output of your current V:XML document. Note that I have a Google Chrome plugin that is converting my XML output for display.

watson explorer converter output

For the sake of this exercise, let’s change the title field to contain the actual title and the author. Like this: Blowout – Lucy Spring. To do this we go back to the previous page and click “add new converter” further down the page.
watson explorer add converter

We want a custom converter

watson explorer add custom converter

Now you will see the configuration screen for a custom converter
wex_converter_08You want to set both the type-in and type-out to application/vxml-unnormalized as we want to apply this template to application/vxml-unnormalized and we will provide application/vxml-unnormalized as output. I use “unnormalized” because I want the normal WEX normalization functions to still apply after this transformation. Also give your converter a name.

wex custom converter configuration

The next section is the conditional setting. This is where you can determine the matches that will cause the converter to apply. In this case we want to match all so I just add a wildcard (*).

wex converter conditional settings

You can skip the advanced section and focus on the Action section. First, the needs to be set to XSL since we’re applying an XSL template to an XML document.

watson explorer custom converter action

Now we’ll use a standard template that allows you to copy nodes with special processing.

The template above will only copy the document if you run it this way. We want to modify this to merge our title and author by matching on the title content and copying some things.

As you can see I’ve added comments in the code above. The important thing to note is that I want to modify the title content so I match it and the mode is always copy due to the way this template works. Then I just copy the attributes, and concat the two values I wanted.

Save this converter and click test-it again at the top of the Watson Explorer page. You will now see your new converter in the conversion trace.

wex custom converter conversion trace

Now if we check the input and output we’ll see the difference.

The before:

wex before converter

Now the title after:

wex converter after

Now if you crawl this collection your titles will include the author name in the search results.
watson explorer search results

 

 

The Cognitive Call Center on IBM Watson

One of the major use cases I see for Watson Explorer (WEX) is in call centers. No matter the specific industry the major goal for call centers are decreasing call time and increasing customer satisfaction rates. The way to do this is to get the correct information in front of the Customer Service Representative (CSR) as fast as possible. This is an excellent use case for Watson Explorer Application Builder (WEX AppBuilder). I’m going to keep this post mostly high level and not get too deep into the technical aspects of such solution.

watson explorer application builder screenshot
A simple example of a Watson Explorer Application Builder display.

WEX AppBuilder works together with Watson Explorer Engine (WEX Engine) to present the user with an 360 degree view of the information they need. If you think of Cognitive Computing as a pyramid, then WEX Foundational is the base layer of that pyramid. Watson Explorer Engine can crawl, convert, and index data into high speed positional indexes. Once the data is indexed in the WEX Engine, you can leverage other applications and APIs, both IBM and external, such as WEX AppBuilder or the IBM Watson Developer Cloud services.

For a call center scenario you would first index your data sources in Watson Explorer Engine. WEX Engine connects to all types of data sources, but if a connector doesn’t exist for your data source there are even push APIs available. The WEX Engine allows some industry leading search features and can even be used as a standalone enterprise search application.

Once the data is indexed, then you can build a pretty compelling user interface very fast in WEX AppBuilder. The WEX AppBuilder product allows you to quickly connect to engine data sources and build pages and widgets to display the data. There are several out of box widgets but it also allows for custom widgets using Ruby, JavaScript, HTML, and CSS.  I don’t want to give the idea that WEX AppBuilder is simply a display framework though. The product allows you to define entities and associations from your data. Those associations can be used to bring together related data into that unified 360 degree view. AppBuilder also has the concept of endpoints that allow you to connect to APIs in real time to bring in additional data, and allow other system to connect to AppBuilder to retrieve data.

Say your call center is caller centric. Note: That seems obvious but is not always the case. A call may have many products. Your CSR rep gets the critical information from the call, performs a search, and lands on a page that tells you all about the caller. The system will tell you all of the products they use, and don’t use. It may tell you if they are past due. It could also tell you if they had support tickets open recently. Most call center reps will need to navigation through several different systems to get this data currently. You can see how gathering all of this information into a single view can be beneficial rather quickly.

watson personality insights
An example IBM Watson Personality profile

Getting the call center onto a 360 view is the first step. The real power comes from having that data in the IBM Watson Explorer platform. You can then start taking the system in a cognitive direction. Instead simply displaying that data to the rep, what if the system was able to predict what the call was about? Using WEX Content Analytics and the Watson Developer Cloud you can start to put together those predictions. We can analyze data points such as mailers that were sent out, past due bills, reasons for calls, etc. You could analyze the call logs and help tickets to get a general sentiment of the client. Would you like to be able to predict if a client is about to change providers? Why wait for them to tell you about it. You could even use the Watson Personality Insights to build a profile of the client. This can give ideas of what types of communication turn them off.

When your CSRs are armed with this type of information it can really change the interaction. Imagine how you would feel if you called in and the CSR already knew about your issue. What if you got notified of the issue before you even thought to call about it? The system can even evolve more. You could use something like the Watson Conversation Service to handle some of these types of questions now before the client even has to speak to a person. This frees up your CSRs to handle higher value calls. When people do call the IVR system can pass the information to Watson Explorer and have the display populated for the CSR.

As you can see in this high level view there is a lot of value to be gained from starting with the Watson Explorer platform. Getting your call center on the Watson Explorer platform is the first step in this cognitive journey.

If you’d like to learn more feel free to comment below or contact me.

IBM Watson Explorer

A few people have contacted me about what I do at my day job as a Watson Explorer Consultant. Since this is my personal site I don’t usually focus on what I do at my 9-5. I’m going to write a few posts to explain what exactly it’s like working for the IBM Watson Group and what applications I work with.

Disclaimer: This article contains my own thoughts and opinions and in no way represents IBM or any IBM products. This post is not sponsored nor affiliated with IBM in anyway. Please see the official IBM Watson Explorer website for information directly from IBM.

What is IBM Watson Explorer?

Watson Explorer is a data discovery tool. It allows you to explore vast amounts of enterprise data. The tool allows you to consume and index data from various data sources. Out of the box Watson Explorer ships with many popular connectors for enterprise data systems. Using it’s own proprietary indexing technology, Watson Explorer can leverage natural language processing to deliver relevant query results to end users. The product can also utilize Query Routing to route queries to websites and return the results within it’s own interface. This data can be integrated into a single, 360 degree view, application on the front-end.

IBM Watson Explorer ships with several different modules:

  • WEX Foundational Components
    • IBM Watson Explorer Engine
    • IBM Watson Explorer Results Module
    • IBM Watson Explorer Application Builder
  • WEX Analytical Components
    • IBM Watson Explorer Content Analytics Admin Console
    • IBM Watson Explorer Content Analytics Miner
    • IBM Watson Explorer Content Analytics Search
    • IBM Watson Explorer Content Analytics Studio

IBM Watson Explorer Engine

The Watson Explorer Engine component is the key backend component of the of the foundational components. The foundational components come from IBM’s acquisition of a startup called Vivisimo based out of Pittsburgh, PA. Engine basically acts as an enterprise search engine that can be leveraged to crawl and indexed large amounts of data both structured and unstructured. The documents are stored as XML documents, similar to Apache SOLR. During the crawling process XSLT can be utilized to modify the data of the document before storing it to the index. Engine can be configured to be distributed among many servers to meet big data needs and scale quickly. The web based admin interface allows IT users a simple way to manage this powerful application. For enterprise search applications engine comes with it’s own search interface. To leverage 360 degree views engine must be combined with IBM Watson Explorer Application Builder.

Watson Explorer Engine Admin Interface
The Watson Explorer Engine admin interface

IBM Watson Explorer Results Module

The results module component allows non-technical business users to manage feature of the search results within Watson Explorer. Users can use the spotlight manager to configure spotlights that will show a boosted content above standard search results based on specific keywords. You can also use Results Module’s terminology manager to easily manage spelling suggesting, synonyms, and related terms.

IBM Watson Explorer Results Module
A screenshots of the IBM Watson Explorer Results Module interface

 

IBM Watson Explorer Application Builder

The Application Builder module is used to build 360 views of enterprise data. This applications connects to Watson Explorer Engine and displays the indexed data to the end user. One of the primary benefits of Application Builder is that you can leverage the entity model. By creating an entity for your data you can then define relationships which allows a developer to easily combine and display related data to the end user. Users can choose to follow specific entities that they are interested. Application Builder will then provide them with the most relevant information based on what the user follows.

The technology behind Application Builder is Ruby on Rails. Specifically App Builder uses jRuby which means that the application runs inside a JVM. So for IT purposed it can be considered as another Java application.

App Builder ships with several out of the box widgets that can be configured to display various types of data. Most deployments of Application Builder use many custom widgets. Custom widgets allow a developer to utilize Ruby (ERB), HTML, CSS and JavaScript to deliver a custom experience. This tool is very powerful and is currently being used by many enterprise customers to get the entire 360 view of their data so they can make educated business decisions.

IBM Watson Explorer Application Builder
An example of a “360 View” using IBM Watson Explorer Application Builder.

 IBM Watson Content Analytics

The content analytics module is a separate piece of software from the foundational components. IBM Watson Content Analytics allows you to collect and analyze different types of content. It stores this content in it’s own indices which are currently separate from Watson Explorer Engine. It can consume both structured and unstructured data from documents, email, databases, websites, and other enterprise repositories.  You can then perform text analytics across the data that is indexed by Watson Content Analytics.

What is a Watson Explorer Consultant

I’m a Watson Explorer Consultant. That means I work directly with customers to conceptualize and deliver Watson Explorer Solutions. My primary focus is on the Watson Explorer Engine foundational components. I use the engine, app builder, and results module components to deliver solutions to data problems at companies big and small. I’m currently one of the experts on my team for IBM Watson Explorer Application Builder. I’m able to utilize my past web development experience to deliver some highly customized solutions to customer data problems.

Our team is based out of Pittsburgh, PA but we are also distributed across the world. I currently work from my home in Ohio full time. I spend a portion of my time traveling to client sites to consult with them directly and deliver solutions in person. It takes a special kind of person to be able to handle problems with both technology and humans. If you’d like to reach out to me please use my contact form.