Applying AI & NLP: Wit.ai, Watson, Lex

"It is our choices, Harry, that show what we truly are, far more than our abilities."

-- Albus Dumbledore

It's no secret that I am a big fan of the Harry Potter books and movies. Not only are they great entertainment, but there are valuable lessons to be learned from the adventures of "the boy who lived." From loyalty to bravery, Harry and his friends express qualities worthy of emulation.

I led off with that quote from Professor Dumbledore because it's something I truly believe. I have known many people who were blessed with talents I fervently envied. Some were great musicians. Some could do things on the playing field that humbled me to the core. However, too many squandered those gifts through bad, and ultimately, self-destructive choices. Unfortunately, having the ability to be great doesn't always make you great.

Of course, not all choices are bad. This is true in both our personal and professional lives. Every day we are met with challenges that require us to choose between a number of competing options. Whether it's "what do I have for breakfast," or "what technology should I deploy," our days are a nearly endless series of choices.

Choices

For the past several months, I have been immersed in the studies of artificial intelligence (AI) and natural language processing (NLP). Not only are they two extremely interesting technologies, but the inroads they've made to our day-to-day lives is breathtaking. Not a day goes by where I don't talk to my phone or ask Alexa a question.

AI and NLP are also used in less obvious ways. For example, I recently did a Web chat with AT&T. A year or so ago, I would have been certain that I was conversing with a human being. Today, I am not so sure. Thinking back on my questions and AT&T's answers, there was nothing in our conversation that couldn't have been said by a machine. The same is probably true for my recent chats with Comcast and CenturyLink. Where once businesses hired people to sit at desks to handle customer care requests, they now are turning to machines to lower expenses while still keeping their customers happy.

As part of my endeavor to learn the fundamentals of AI and NLP, I wrote software to exercise three of the leading commercial platforms -- Wit.ai, IBM Watson, and Amazon Lex. While I will not pretend that I was able to explore every function that these platforms provide, I did get a pretty good feeling as to how they were similar and how they differed.

So, without any further delay, here are some broad-brush impressions of these platforms. I will stick to text for now and save audio for a subsequent article.

[Note: The rest of this article uses terminology used extensively by AI and NLP. If you are not familiar with these terms, please read Enhancing the Customer Experience with Natural Language Processing.]

Wit.ai

This was the first platform I worked with and one of the easiest to get up and running. It provides an intuitive user interface to build bots, so you simply train your bot with sample nuances that get parsed into intents and entities. Bot developers can define their own intents and entities, or use those that have already been built into Wit.ai.

Pros:

• Lots of built-in intents and entities
• Easy to use bot creation tool
• Extensive language support (including Zulu and Mongolian)
• An open source model for sharing apps
• Multiple ways to parse utterances (trait, free-text, keywords, free-text and keywords)
• An inbox tool to classify utterances from a running bot
• Free (There is no charge to use Wit.ai.)

Cons:

• No support for bot dialogs -- responses are built and maintained outside of Wit.ai
• It's unclear how Wit.ai scales. Its FAQ asks you to contact them if you plan to "hit" their API "quite heavily."
• How much R&D will Wit.ai receive as a free service?

IBM Watson Conversations

Watson is clearly a heavy hitter in this space, and IBM appears to be fully committed to its success. Like with Wit.ai, it's an easy process to create a Watson Conversation. The Web interface is straightforward, yet powerful. It comes with an impressive number of pre-built intents that can be applied to any workspace (Watson's term for bot). It also has a very extensive dialog tool to create multiple possible responses for every intent.

Pros:

• Easy to use
• Bot dialog responses
• IBM's strong commitment to this technology
• A highly accurate utterance parsing engine
• Lots of built-in entities
• A very extensive API with an on-line test tool (Swagger)
• Watson Conversations are not free, but IBM provides a very generous development tier. I have yet to pay for any of my work.

Cons:

• Compared to Wit.ai, IBM is limited in its language support. Bots can speak English, German, Spanish, French, Italian, Japanese, and Arabic, but where is the support for Somali and Afrikaner?

Amazon Lex

Amazon Lex is my most recent research project, and it may become my favorite. It does everything that Wit.ai and Watson do, but it also supports the concept of a bot state machine. This allows the controlling application to take different actions depending on the current state of a conversation. Additionally, Lex allows developers to create slots for their applications. A slot can be thought of as a variable that can be used when creating utterances. For example, "I would like to make an appointment for a {AppointmentType}" can be used to train the bot.

Pros:

• Lex's Lambda interface allows developers to tie business logic to a user's utterances
• A powerful state machine
• Slots can be used to classify utterances and drive the bot's state machine
• Lex comes with sample bots that can be used as the basis of a new bot. (For example, the dental ScheduleAppointment sample bot can be easily modified to support any appointment type)
• Bots can be versioned
• Lex isn't free, but Amazon offers a generous development tier. Like Watson, I have done all my exploration without having to shell out a single nickel.

Cons:

• The process to create a bot is not nearly as easy as it is for Wit.ai and Watson
• Once deployed, Lex bots appear to require time before they are truly active. On several occasions I have had to wait minutes before my changes kicked in.
• The authentication methodology is quite complicated. Of course, that can be seen as a pro. The more difficult authentication is, the greater the security.
• I could only find support for English.

Mischief Managed

I will be the first to admit that I have only scratched the surface when it comes to understanding and using Wit.ai, Watson, and Lex. While I am happy with the progress I've made, there are subtle nuances of each one that will only become clear as I write more bots. That said, I am comfortable in saying that I could use any of the three to build text bots that could mimic the actions of a human being when it comes to easy to moderately involved conversations.

I fully understand that the AI and NLP space isn't limited to these three platforms. I have already begun working with Google Cloud Natural Language and there are some smaller niche players on my radar. Who knows what goodies I might find? Thank goodness for choice!

Related content:

• Enhancing Customer Experience with Natural Language Processing
• Enhancing Customer Experience with NLP -- Part 2
• Putting Watson to Work

Follow Andrew Prokop on Twitter and LinkedIn!
@ajprokop
Andrew Prokop on LinkedIn