Regular Expression Converter for Watson Explorer Engine

Sometimes it’s useful to extract data from a Watson Explorer content node using regular expressions. In this post, I’ll show you how to extract data using a regular expression and create a new content node for that specific data.

To start off we will use the default example-metadata collection. We will attempt to extract any 3 digit number from the snippet content to make the regex easy. You can do much more advanced regular expressions if necessary.

First go to the example-metadata collection and click “test-it”

Then click on “Test-it” next to the first result:

Now scroll down and look at the output of the ” Create Metadata from Content” converter:

In the output, you will see the snippet content has the number 500 in it.


We will make a converter that will extract any 3 digit number into a new content. First, add a new converter:

Select the Regex entity extraction converter and click Add.

In the converter configuration, in the list of entities node names enter “my-regex-node” and the target node of “snippet”. Then click OK.

Now on the sidebar of WEX click the + next to XML.

enter the following names:

Now update the xml node to include your regular expression like below. Note that my regex is “[0-9]{3}” to match 3 digits. Save the node.

Return to the collection and do a test-it, as we did above, down to that same first result. If you look at the converter trace you will see the regex converter is running.

Click on the 910 output to see your new content node:

Now you can use the new “regex-rule” content in your search application.


John Ward is an IBM Watson Explorer Consultant with several years of experience deploying and customizing Watson Explorer solutions.

Using crawl-url nodes to enqueue data along with a url in Watson Explorer

viv:crawl-enqueue-url is documented as having 1 argument – the url to enqueue.
However, it has an optional second form with two arguments which allows much more flexibility. If the first argument is set to the url and the second is a variable containing a crawl-url, that also works. (note the use of entities to create the content tags)
For example:
When www.someurl.com is crawled, the content ‘inside’ will be added to it (after running through the normalization converter, probably), and from the converter, the attributes ‘url’ (automatically added) and ‘another-attribute’ (manually added) will be available to you via viv:current-node()/@attribute-name. If you want to add multiple contents in your crawl-data node, you’ll need to give them a root node – ‘document’ will work:
The prototype of the two-argument form actually looks something like: viv:crawl-enqueue-url(object, node). If the first argument evaluates to true, it is used as the url attribute on the node. That’s why the url is automatically added, above. If it evaluates to false, the node is used as-is. Thus, an equivalent form of the above is:


John Ward is an IBM Watson Explorer Consultant with several years of experience deploying and customizing Watson Explorer solutions.

Watson Explorer XSL Tips and Tricks

Using the Chico Application to Test XSLT

Here’s an example use of Chico! I was testing something to do with evaluating XSLT and wanted to run it quickly:
Get to chico by going to your Velocity’s ‘velocity’ script and adding v.app=chico. Enter AXL in the box on the left, then submit it to see the processed results on the right.

Match   within strings

Use to match non-breaking spaces within strings in XSL:


For loop with XSL

You can emulate a for loop in XSL by creating a string of some length and tokenizing it:

If you intend on using this “for-loop” in a converter and you are taking data from a web page then you must do two things to make it work.

  • Save the entire page in a xsl variable outside the loop. Once in the loop you will lose access to the webpage.
  • If you need to iterate in your xpaths then save position() in a variable. If you try to call it directly in your xpaths then they will not work correctly.


Get a random number within a specific range


Copy a nodeset with special processing
  • Use the following XSL to copy a nodeset verbatim. Enter any special processing templates in between the comments as indicated


Boost parser

  • Based on the above, this is a generic recipe for a parser you can add to a source where you want to boost its results. You’ll probably want to edit the values in uppercase. Note that we’re throwing away binning information and boost-onlying the results.
PROTIP:If you are adding this boost parser to a source which accesses a Velocity Search Engine collection, you **must** set the parser type toxsland **not**html-xslor you may spend hours debugging your parser. You’ll know if you made this mistake when the content nodes are empty.


Parse XML that uses an XML namespace

Define a new prefix with the xmlns attribute in each xsl:template and then prepend your new prefix to each XPath (in this case, I’ve set the namespace to the ‘a’ prefix).


Parse an XML file from the command line
  • Use transform (from your checkout/vivisimo/util directory):

You can set up an alias in your .bash_profile if you don’t want to type the whole path every time.

Or you can cd to your installation directory and run from there, then the software will find your vivisimo.conf.
  • Stub XSL for a starting point:

Using xsl:key

For nodesets that are frequently accessed, creating a hash lookup with xsl:key() can significantly improve performance (in my experience, accessing a value using a key is about 0.07ms or about as fast as accessing a variable value). Here’s a simple example of creating a key and using it on the /*/settings node in the display:
For a given input XML of:

You can create a key that compares against [email protected] of eachsettingnode:


Then you can access the value of that setting node with a call tokey(). The first parameter is the name of the key and the second parameter is the value to compare against [email protected]


Using a nodeset/result-tree as the scope and context[edit]

Normally in XSL when you execute a command, the global context for xpaths is the incoming XML and the scope is the current node.
The easiest way to change the scope is to use<apply-templates />; the new scope is whatever node that the template matches on. However, the context still remains the incoming XML.
To use a completely different context (and naturally the scope), you can use xsl:for-each with a constructed nodeset.
Example:

actually returns


This can be used with document()and AXL variables (which don’t have any context and will generate an error if you use a key() in them).

Note that running it in a exsl:node-set() context (or for that matter any context that you don’t really know about (i.e., anything other than / and document()) can generate weird results:

actually returns


Grouping – The Muenchian Method[edit]

Suppose you have the following xml:

To list each city grouped by state, use the following xsl:

  • The first line produces a key called location-key for each location using the value of state.
  • The outer for-each loop scans through all location nodes whose state node is the first in the group of locations with that state value. In other words, it loops one time for each state.
  • The inner for-each loop scans through all location nodes whose state node has the value of the current state node. In other words, it loops once for each city in the current state.
The above XSL will produce something like this:

The method described above is especially great for grouping large amounts of data as it is orders of magnitude faster than manually scanning through every node and comparing it with siblings.

One place that I found a particularly crazy use for the Muenchian Method was in a converter. I wanted to output the sum of numeric values in the 2nd column of an html table for groups of rows. A group of rows in the table was defined by all rows having the same values in their 6th, 3rd and 1st columns. In that case, I wanted to make a key for each row in the table using a value of the concatenation of the 6th, 3rd, and 1st columns in the row.

Next, I used a for-each loop to get one row from each group.


Inside that loop, I found the sum of all the numeric values in column 2 of each row in the current group.


If your head isn’t hurting yet, a more detailed explanation of the Muenchian Method is available here


How to Enqueue Javascript Links

This is a question that comes up a lot from customers; how do I enqueue links on web pages that use Javascript?
It’s actually a pretty simple process, but the way Javascript links are parsed by the browser won’t work when using a converter. Below is a sample XHTML file that has a few Javascript links in it that we’d like to convert and enqueue.

Below is the stylesheet we would use in the converter to extract the links for this HTML file.


This is a simple example, but it can be used as a good starting point for customers.


Directory, basename, and extension extraction

Sometimes it’s quite useful to extract the directory paths, the basename, and the extension from a given path. These three templates will do that.

 

Observe thatget-basenameandget-extare basically the same thing, the exception being the difference in the separator. These two templates could be combined in a singleafter-lasttemplate which takes as params the string and the separator.
Note that the behavior when an extension is not present is not defined. A better recursion termination condition would fix that.

Empty content remover

Database seeds sometimes generate empty content elements. This is bad form for the finished product. This customer XSL template will remove empty contents, i.e. contents without child nodes.

Parse HTML in XML

Sometimes people put HTML into RSS feeds and the like, and it needs to be parsed. This is not a straightforward task, but this code should help you out. This is taken from the parser for a custom RSS feed parser Colin Dean wrote for NIH. See VO #1024for a little discussion and a full source parser example.
You may be inclined to follow some online tutorials about using two passes to parse: one outputs the HTML with output escaping disabled, and the second pass actually performs the intended parsing. One may think to use Velocity’s secondary parser in a source to handle this. This is incorrect! It can be done in one pass with some magical viv XSL extensions.


Uglify Text for Content Name Attribute

First normalizes (trims) whitespace. Converts spaces to dashes. Lowercases, and strips non-alpha-numeric (and dash) characters.

John Ward is an IBM Watson Explorer Consultant with several years of experience deploying and customizing Watson Explorer solutions.

Copy and Modified Documents with a Watson Explorer Converter

A common task when crawling and indexing a document in Watson Explorer Engine (WEX) is making changes to a document during the conversion process. The most common occurrence is needing to copy all the contents in the application-vxml document while making some changes to one or a few of those contents. To do this, there is a recursive copy template that can be used. I’ll show you how to apply it.

First, I’m going to use the out-of-box “example-metadata” collection. Navigate to that collection and click the test-it button.

wex collection screenshot test it

After clicking test-it you will see a listing of documents. Click on the test-it button for the “blowout” record.

watson explorer test-it results

On the resulting page, scroll down and look at the conversion trace. There is a converter called “Create Metadata from Content”. This is the converter that ships with WEX to convert the HTML files into v:xml documents. Each of the links on the left side represent input and output of that conversion step. We want to click on the output of this converter to see what the document looks like.

watson explorer conversion trace

You will see the output of your current V:XML document. Note that I have a Google Chrome plugin that is converting my XML output for display.

watson explorer converter output

For the sake of this exercise, let’s change the title field to contain the actual title and the author. Like this: Blowout – Lucy Spring. To do this we go back to the previous page and click “add new converter” further down the page.
watson explorer add converter

We want a custom converter

watson explorer add custom converter

Now you will see the configuration screen for a custom converter
wex_converter_08You want to set both the type-in and type-out to application/vxml-unnormalized as we want to apply this template to application/vxml-unnormalized and we will provide application/vxml-unnormalized as output. I use “unnormalized” because I want the normal WEX normalization functions to still apply after this transformation. Also give your converter a name.

wex custom converter configuration

The next section is the conditional setting. This is where you can determine the matches that will cause the converter to apply. In this case we want to match all so I just add a wildcard (*).

wex converter conditional settings

You can skip the advanced section and focus on the Action section. First, the needs to be set to XSL since we’re applying an XSL template to an XML document.

watson explorer custom converter action

Now we’ll use a standard template that allows you to copy nodes with special processing.

The template above will only copy the document if you run it this way. We want to modify this to merge our title and author by matching on the title content and copying some things.

As you can see I’ve added comments in the code above. The important thing to note is that I want to modify the title content so I match it and the mode is always copy due to the way this template works. Then I just copy the attributes, and concat the two values I wanted.

Save this converter and click test-it again at the top of the Watson Explorer page. You will now see your new converter in the conversion trace.

wex custom converter conversion trace

Now if we check the input and output we’ll see the difference.

The before:

wex before converter

Now the title after:

wex converter after

Now if you crawl this collection your titles will include the author name in the search results.
watson explorer search results


John Ward is an IBM Watson Explorer Consultant with several years of experience deploying and customizing Watson Explorer solutions.

 

The Cognitive Call Center on IBM Watson

One of the major use cases I see for Watson Explorer (WEX) is in call centers. No matter the specific industry the major goal for call centers are decreasing call time and increasing customer satisfaction rates. The way to do this is to get the correct information in front of the Customer Service Representative (CSR) as fast as possible. This is an excellent use case for Watson Explorer Application Builder (WEX AppBuilder). I’m going to keep this post mostly high level and not get too deep into the technical aspects of such solution.

watson explorer application builder screenshot
A simple example of a Watson Explorer Application Builder display.

WEX AppBuilder works together with Watson Explorer Engine (WEX Engine) to present the user with an 360 degree view of the information they need. If you think of Cognitive Computing as a pyramid, then WEX Foundational is the base layer of that pyramid. Watson Explorer Engine can crawl, convert, and index data into high speed positional indexes. Once the data is indexed in the WEX Engine, you can leverage other applications and APIs, both IBM and external, such as WEX AppBuilder or the IBM Watson Developer Cloud services.

For a call center scenario you would first index your data sources in Watson Explorer Engine. WEX Engine connects to all types of data sources, but if a connector doesn’t exist for your data source there are even push APIs available. The WEX Engine allows some industry leading search features and can even be used as a standalone enterprise search application.

Once the data is indexed, then you can build a pretty compelling user interface very fast in WEX AppBuilder. The WEX AppBuilder product allows you to quickly connect to engine data sources and build pages and widgets to display the data. There are several out of box widgets but it also allows for custom widgets using Ruby, JavaScript, HTML, and CSS.  I don’t want to give the idea that WEX AppBuilder is simply a display framework though. The product allows you to define entities and associations from your data. Those associations can be used to bring together related data into that unified 360 degree view. AppBuilder also has the concept of endpoints that allow you to connect to APIs in real time to bring in additional data, and allow other system to connect to AppBuilder to retrieve data.

Say your call center is caller centric. Note: That seems obvious but is not always the case. A call may have many products. Your CSR rep gets the critical information from the call, performs a search, and lands on a page that tells you all about the caller. The system will tell you all of the products they use, and don’t use. It may tell you if they are past due. It could also tell you if they had support tickets open recently. Most call center reps will need to navigation through several different systems to get this data currently. You can see how gathering all of this information into a single view can be beneficial rather quickly.

watson personality insights
An example IBM Watson Personality profile

Getting the call center onto a 360 view is the first step. The real power comes from having that data in the IBM Watson Explorer platform. You can then start taking the system in a cognitive direction. Instead simply displaying that data to the rep, what if the system was able to predict what the call was about? Using WEX Content Analytics and the Watson Developer Cloud you can start to put together those predictions. We can analyze data points such as mailers that were sent out, past due bills, reasons for calls, etc. You could analyze the call logs and help tickets to get a general sentiment of the client. Would you like to be able to predict if a client is about to change providers? Why wait for them to tell you about it. You could even use the Watson Personality Insights to build a profile of the client. This can give ideas of what types of communication turn them off.

When your CSRs are armed with this type of information it can really change the interaction. Imagine how you would feel if you called in and the CSR already knew about your issue. What if you got notified of the issue before you even thought to call about it? The system can even evolve more. You could use something like the Watson Conversation Service to handle some of these types of questions now before the client even has to speak to a person. This frees up your CSRs to handle higher value calls. When people do call the IVR system can pass the information to Watson Explorer and have the display populated for the CSR.

As you can see in this high-level view there is a lot of value to be gained from starting with the Watson Explorer platform. Getting your call center on the Watson Explorer platform is the first step in this cognitive journey.

If you’d like to learn more feel free to comment below or contact me.


John Ward is an IBM Watson Explorer Consultant with several years of experience deploying and customizing Watson Explorer solutions.

Building a Business Around a 3rd Party API

Today I read an interesting post on HackerNews about a price comparison site, PriceZombie, being kicked out of the Amazon Associates program. The Amazon Associates program is Amazon’s affiliate network. Amazon pays a commision for any sales you bring in through affiliate links. Since I have a pretty extensive background with affiliate marketing I have a few thoughts on the dilemma PriceZombie is facing.

After being previously told we were in 100% compliance with the rules, our Amazon affiliate account was closed a few months ago. Amazon claimed we were violating their rules against showing product and price information that was more than 24 hours old. Obviously, this is something ALL price history trackers do, not just PriceZombie.

It seems obvious that PriceZombie was in contact with a representative of Amazon and that they thought they were complying with the terms of their affiliate agreement. However, the account was then banned for “violating their rules against showing product and price information that was more than 24 hours old.”

My first thought, not unlike the HackerNews commenters, was that PriceZombie violated the terms of their agreement and they need to suck it up. However, I’m an Amazon Associate and I don’t remember anything about not using historical pricing data in the agreement. I took at look at the newest affiliate agreement and couldn’t find any mention.

PriceZombie uses the Amazon Advertising API so I took a look at that agreement. No mentions of not being allowed to use historical pricing data. In fact this agreement seems to suggest you are allowed to show historical prices as long as you display a date with that price:

(o) You will include a date/time stamp adjacent to your display of pricing or availability information on your application if you obtain Product Advertising Content from a Data Feed, or if you call the Product Advertising API or refresh the Product Advertising Content displayed on your application less frequently than hourly. However, during the same day on which you requested and refreshed the pricing and availability information displayed on your application, you may omit the date portion of the stamp

I did find a section stating that you cannot store images and that you should update the content every 24 hours. This probably covers the PriceZombie case but it’s not very explicit.

(n) You will not store or cache Product Advertising Content consisting of an image, but you may store a link to Product Advertising Content consisting of an image for up to 24 hours. You may store other Product Advertising Content that does not consist of images for caching purposes for up to 24 hours, but if you do so you must immediately thereafter refresh and re-display the Product Advertising Content by making a call to the Product Advertising API or retrieving a new Data Feed and refreshing the Product Advertising Content on your application immediately thereafter.

I will assume PriceZombie was kicked out for a technicality in the previous rule. If so then why are so many other Amazon price tracking sites still operational? Amazon was making money from those affiliate sales. Why would they want to stop that revenue? Were the margins too thin on the PriceZombie conversions? Is Amazon trying to cover up the fact that it’s been raising prices? None of this really matters.

The Real Lesson Today

People are caught up in the debate about the possible violation of Amazon Terms of Service, but they’re missing the real point here. If you are planning to build a business around any 3rd party API you are exposing yourself to the greatest risk I can think of. Your business can die in a blink of an eye and it’s completely beyond your control. Someone else controls your revenue and can turn it off with the flip of a switch.

It doesn’t matter if PriceZombie violated the TOS. Amazon has no obligation to allow anyone access to it’s API or affiliate program. They don’t even need a reason to deny you access. They can allow Joe’s price tracker to keep running even though they killed PriceZombie. There is no implied equality when it comes to business agreements. They could terminate your agreement at anytime and for any or no reason. Kind of like my employer can terminate my employment without reason.

The real thing we learned today is that you can’t build the core of your business around a 3rd party API. Situations like this happen all the time and people still put their bread and butter in somebody else’s basket. PriceZombie could have taken some steps to ensure they would survive if that Amazon money stopped rolling in but they didn’t. Now it’s too late.

I’m even guilty of this. I’ve built sites in the past with Amazon Associates as the sole monetization but I’ve never seen them as a long term business and I always knew the risk of termination was there. I’ve changed my ways of thinking in the past few years. One of the major changes is the realization that if you are relying on 3rd party monetization then you have hobby, not a business.

Real businesses have something to sell.

I’m not suggesting anyone do this but there is a part of me that would find it extremely hilarious if PriceZombie would 301 redirect all of their pages to the same product on Jet.com. I really want them to update all the links on the price tracker to Jet links but that would be a legal problem using Amazon’s API content improperly. So 301 redirect would probably be safer. Obviously I’m not a lawyer.  

Tracking Ad Blockers Using Google Analytics

If you’ve read any of my older posts you probably know that I’ve been involved in internet advertising, in some form, for a long time now. Lately I’ve been seeing more and more posts about how ad blocking is growing at a tremendous rate. I still have some sites that make money from display advertising so I was curious just how many of my users are actually using ad blocking tools. I decided to see if there was a way I could track the users who do use tools like AdBlock vs the users who do not.

First, I came across a script called Block Ad Block. The intention of the script appears to be to detect ad blockers and allow the site owner to act on that knowledge. As a site owner I could detect that a user is using an ad blocker and decide not to display the content to that user. I could also replace the ad block with something like a donations link, an image that tells the user they are a leech, etc. This is all done client side via JavaScript so ultimately the user could still display the content if they disabled JS. I don’t have an interest in punishing people but I saw a way I could use this script to capture the data I wanted.

I noticed that the script allows custom code to be executed when adblock is detected or not detected. Then I remembered that Google Analytics allows you to set custom variables. Combining these two functions we should be able to track which users are using ad blockers in Google Analytics.

First you want to download blockadblock.js. Rename it to something else, I called mine GA.js and include it in your site’s header.

Then create the following Javascript file and include it in the header.

This script include google analytics so you don’t want to include it anywhere else. Also there is a section where you can trigger your own JavaScript to run if adblock is detected. For example you could beg for donations if the user is using adblock.

After this runs for awhile it will start collection custom variable data. You can check the data in the Google Analytics dashboard by going to Your Site -> Audience -> Custom -> Custom Variables. You will see one that says “adblock”. If you click the link it will then take you to a report that shows if a user has adblock enabled or disabled. Yes means adblock was detected:

Google Analytics Detect Ad Blockers

When I look at my data I generally see that somewhere around 20% of my users are using ad blockers. Seeing as the script is written in client side JavaScript and so is Google Analytics, you won’t be able to capture any data if the user has JavaScript disabled. That still seems to be even a more extreme minority than the number of AdBlock users. If I get bored or there is enough interest I may turn this into a WordPress plugin.

 

The Best WordPress Email Plugin

Thinking about adding a WordPress email plugin to your site? One of the most power marketing tools that anyone publishing online can take advantage of is email marketing. According to the Direct Marketing Association email advertising has an average return on investment of %4,300 in the United States. Email is consistently the best performing media in conversion rates. McKinsey & Co. stated that email marketing usually out performs social media with 3 times the conversions.

If you’re not collecting email address, you should be.

We know that email is the best way to connect to our audiences but with all of the plugins out there for WordPress how can you tell which ones are worth it? I’m going to make it easy for you. Just use Bloom by Elegant Themes.

What makes Bloom the best WordPress email list plugin that I have used so far?

I just relaunched my blog a few weeks ago and one of my posts was starting to go viral but I had a big problem. I had no way to collect email addresses. I was in a hurry to add something as my traffic was building. I probably looked at hundreds of plugins, some paid and some free, before I remembered I had a subscription to Elegant Themes. That’s when I found Bloom and fell in love.

If you aren’t familiar with Elegant Themes they’ve been making premium WordPress themes and plugins for years. Bloom is a newer creation and it’s simply beautiful. I haven’t found another plugin that offers the same ease of configuration with the thousands of pre-built opt-in forms that Bloom offers.

What makes Bloom different from your standard opt-in plugin?

bloom wordpress email plugin providers
Bloom for WordPress integrates with these popular providers

First of all Bloom has easy integration for 12 of the most popular email providers including:

  • AWeber
  • MailChimp
  • Campaign Monitor
  • Constant Contact
  • Mad Mimi
  • InfusionSoft
  • iContact
  • GetResponse
  • mail poet
  • feed blitz
  • send in blue
  • OntraPort

I personally use MailChimp and the integration couldn’t have been easier. I just supplied my API-Key and had simple drop downs to select which list I wanted to use.

Bloom has hundreds of pre-made templates to use…

bloom opt in form designs for wordpress

…and they look amazing. You can use these awesome templates with a click of the mouse. You can even customize them using the built in customizer or custom CSS code. I haven’t seen another plugin offer this level of customization while also looking so elegant. You can customize colors, borders, text, buttons, fields and almost anything you can think of.

Even more customization out of the box

bloom wordpress triggersBloom gives you tons of options for how and when your email opt-in should appear. You can have those cool automated pop-in type boxes, a static sidebar form, a bottom of the post opt-in like I have, or even things like post-sales opt-ins. You can control when to trigger things like the pop-in too. So you can wait for a certain amount of time or when the users scrolls, or after the user comments, or even when the user becomes inactive.

You can tell Bloom when and where to show or not show your opt-in form. If you want to exclude categories that’s a simple mouse click. It can add the form to all of your posts by default or you can choose to manual add the forms to posts. You can even get down to the post/page level when determining if you want the opt-in form to show on specific pages.

The control panel

bloom wordpress control panel

The control panel is where bloom really shines. You can get quick stats about the number of subscribers, new subscribers, etc and they are also presented in a beautiful format. You can see all of your various forms and edit them.

You can also run split test very easily. Want to test which color works better? Bloom does that. Want to test a different copy? Bloom does that. The split testing tool is really cool and makes optimizing your opt-in sequence so much easier.

Bloom is responsive and retina ready. 

bloom wordpress responsive

No matter what device your users are on your opt-in forms will look beautiful.

Get the Bloom and access to over 90 other WordPress themes and plugins

That’s right. When you purchase bloom you also get access to more than 90 other premium WordPress themes and plugins from Elegant Themes. I’ve used these themes for years and they are well developed, designed, and easy to use. You won’t be disappointed with a membership to Elegant Themes.

Click here to join Elegant Themes today and get access to the best premium WordPress themes and plugins on the web.

I’ve been a lifetime member for over 8 years and I could not be happier. 282,273 customers can’t be wrong.

[Affiliate Disclaimer]

Are Products the Road to Prosperity?

A few weeks ago a wrote a somewhat popular post about my past experiences trying to make money online. I went through the ups and downs of working as an affiliate promoting other people’s products. At the conclusion of the post I said that I wanted to focus on building some type of product. So why do I think products are the way to the lifestyle I want to build?

For a long time my mindset was just to grind it out and make as much extra income online as I could. This led me to affiliate marketing, blogging, and even some low quality “thin” affiliate sites that most of you would label “spam”. I spent so much time promoting other people’s products that I never really sat back and took a look at the big picture. An advertiser I was working with would pull their products from the affiliate network or I would get removed from the campaign for various reasons (sometimes warranted). I spent my own time and energy selling products I had no ownership in. The people who produced the products were really getting all the value. For a small one time fee I would send them a new customer. In most cases they could continue to sell to the same people I brought in through my advertising tactics. I, on the other hand, had to constantly be looking for new traffic, new products, and new ways to connect with people who will open their wallets. I had no path to a sustainable business model.

On top of that my day job is in the professional services realm. I started working as a consultant for a start up that eventually as acquired by IBM. Now I’m part of the IBM Watson team working as an IBM Watson Solutions Consultant. I’ve learned a lot of things over the past five years of services work. There is one business lesson that seems to stick out to me: services don’t scale. This is especially evident in a young startup that is just trying to make ends meet. You constantly battle the problem of having too much work and not enough people or too many people and not enough work. That’s because they way you scale services is to add people. Now there are ways to “productize” services but in the end it’s still usually tied to an hourly rate. Meaning to make more money you either have to hire more people, raise rates, or find a way to bill more time. Now there are a lot of people who have built successful service businesses but it’s clear to me that is not the route I want to go.

During the acquisition process I also learned that sometimes equity doesn’t pay off for everyone even when there is a successful exit. It paid off for some of the first employees. I was a later hire and it didn’t really work out all that well considering I’d taking significantly less salary than I probably should have been making. My final payout from the acquisition ended up not even being enough to make up for one year of the lower salary I took for three years.

My Mindset Changed

enlightenment

In the winter of 2012 I signed up for the 30×500 product development class that’s put on by Amy Hoy and Alex Hillman. You know how you often hear those stories about some startup getting bought by Facebook for bazillions of dollars when they have no revenue? What you don’t hear are all the many, many, failures that didn’t win the startup lottery. I technically did win the startup lottery and I was left with basically nothing except a new corporate job title and my trusty 2009 Macbook Pro that I still get to use today.

Why am I talking about startups? Well, because 30×500 is basically the polar opposite of the startup lottery world. It’s a class focused on bootstrapping. Startups are usually started with some type of an idea. Then proof of concept/prototype/beta is built to test to see if people actually want the product or not. Startups are basically designed to either fail quickly or have hockey stick like growth (many times that’s not revenue growth either). How is 30×500 different? The class teaches you to do research to find what people actually want and then create a product to fill that need. It’s much easier to sell something to people who already want to buy what you are selling than it is to try to force a product to “fit a market”.

The best lessons I got out of the class were probably focused around money. Everything I’ve learned about money most of my life was tied to time. From working for $5.35 an hour in high school. To earning a salary which means your money is tied to 40 hours of your time each week. Then you end up working another 10-20 hours per week for basically free because that’s how most salary positions work in my experience. My parents, friends, relatives always talked in hourly wages. Amy’s class and probably Tim Ferriss’ 4 Hour Work Week taught me that the best way to free yourself is to separate your time and money.

All Hail Products

The best way to separate your time from money is by producing a product. Now what exactly is a product? It could really be anything. I think of products as anything that you can bundle into some type of packages physical or digital that you can sell to multiple people. Ebooks, training courses, software as a service (SaaS) are all ideas of digital products. For all of these things you basically make a large initial time commitment and then smaller ones to support the sales and the product itself.

Now if you notice above I’m not talking about “monetization”. I’m talking about sales. Monetization is what you do when you have a bunch of people who don’t want to pay you for anything so you try to extract what little you can from them, typically in the form of advertising. This really only works well for huge sites like Facebook for example. Even sites as huge as twitter are struggling with the “monetization” path. I’ve gone through trying to monetize traffic in the past and it feels like the wrong approach if you want more predictable success and to limit obvious paths to failure.

The biggest upside to products is the ability to scale. This is especially true of digital products. While scaling physical products might take more work to pump out more products it takes almost nothing to sell another copy of an ebook. How much of my time would it take me to build 1,000 custom websites for clients? Ewww I’m not even sure I want to think about dealing with that many picky clients. If I had to put a time on that I would say 10 per year might be a good estimate (for me) which would give us a total 10 years. 10 years. 10 years of hearing clients argue why that particular shade of blue is not right. How long would it take to sell a WordPress plugin to 1,000 people? I’ve seen people do that in hours. Even if I couldn’t do it in hours I could continue to scale the sales of a product like that while pursuing other things because once the product is created my time is mostly freed from it aside from support and sales.

Yeah but I Can’t Actually Make a Product

There are probably a few of you who got this far but are thinking that you can’t make a product and market a product. You might even think you need a huge following to sell something. After all these bloggers and authors were all famous way before they sold products, right? Wrong!

  • 37signals, the creators of popular project management software basecamp, started off as an agency making web applications for clients. Their first foray into products was a actually an ebook about e-commerce search engines. Basecamp came out of a need they had internally and eventually they turned it into a very successful product. They’ve since branched out with more products.
  • Amy Hoy and Thomas Fuchs started out doing consulting work. They started doing some training classes for corporations and eventually directly to the end user. Their first product was a book about JS called Javascript Performance Rocks. At the time JS was becoming more popular and everyone was making terrible slow web apps. The book help solved that pain. Then they developed Freckle time tracking software for freelancers. Then eventually Amy took what she learned about creating products and packaged that knowledge as a product called 30×500. Amy even had a failure in there when she tried to reinvent email but kept on moving once she cut her losses.
  • Patrick McKenzie created software for teachers to use to generate bingo cards. Yes, he literally built a successful business selling to probably one of the cheapest markets I can think of. Teachers in the US are severely underpaid in most cases and they often spend their own money buying supplies and things like this. Yet they are willing to shell out money for this software. He’s solving some kind of pain.
  • Brennan Dunn ran a successful consulting business. His big entry into the product space was a book called Double Your Freelancing Rate. He no longer runs a consultancy but he does have a project management SaaS application called Planscope and is building more products including training classes.
  • Nathan Barry basically went from being an unknown web developer to supporting his family off a few ebooks.

I could go on and on. There are thousands of people in markets I’ve never even heard about making a living off of products. These stories might not be the “20 year old that started the next Facebook and became a multimillionaire” kind of story. The truth is I’m never going to create the next instagram, but these seemingly ordinary people are making a few hundred thousand dollars from small product businesses. Mostly because they just started blogging with solutions to problems people in their space have. They built an audience by helping people. Eventually those people were also interested in buying their product. The best part of selling a product is that there is a good chance those same people are going to be interested in buying your next product (provided it targets a similar interest). All of the people I listed above have an email list that they have built over the years. I bet each one of them could send out an email announcing a new product and makes thousands of dollars in pre-sales right this instant.

Conclusion

You might be asking yourself: If this guy is so sure products are the bee’s knees then where are his products? Well there is one thing no class can do and that is make a person take action. You will find that most people won’t take action. You could give them an exact blueprint to print money and they won’t do it.

One of the first things 30×500 will teach you is to sell to people like yourself. You already know that market. Well, I have a hard time with introspection so I have been stuck trying to define exactly what it is that I am. I’m not really a web developer. I’m not really a big data engineer. I’m not really a consultant. Maybe I suffer from imposter syndrome? Maybe I’m just realistic in my skill level? I’m probably just more of a generalist but what ever the problem is I’m going to make it my goal to work towards solving it.

There is a lifestyle I want to work towards that I feel like only having ownership in my own business and products can provide. I used to think it required millions of dollars. I don’t think that now. I just need enough to support my family, cover my current expenses, and then a little bit to have fun with. I’ve already worked out a remote work relationship with my employer, but I want to be able to travel with my wife while still making a living. I want to be able to decide one morning that I’m going to move to somewhere sunny (so long as they have internet) and actually be able to do it without permission from a boss. I’ve seen posts from Amy about being too sick to work for weeks at a time but still making an income from her product business. If you had a job and were sick for that long there a chance you would be fired or at least forced to go on Family Medical Leave and lose your income. If you were a consultant and couldn’t consult you would lose your income. With a product business the business can still chug along while you are out dealing with whatever life throws your way. That’s the kind of lifestyle I’m going to work towards building starting now.

 

Setup A Life You Don’t Need to Escape From

I waste too much time online, especially on social networks like Facebook. I saw a quote the other day from Seth Godin:

Instead of wondering when your next vacation is, maybe you should set up a life you don’t need to escape from. -Seth Godin

This really hits home with me. I often find myself running away from life. I don’t really have a clear path. Now if I end up on my Facebook profile I see this:

facebook screenshot

I decided that since I waste so much time on Facebook at least I can give my future self a reminder of the goal I should be working towards. Will it help? I don’t know. I’m sure at some point in the future I’ll forget about it and maybe it will give me an extra boost that day.

Anyway, here’s the template if anyone is interested in it.

facebook profile header template