Tuesday, October 17, 2017

Databricks: Scratching the surface of artificial intelligence

A “dark art” perhaps, machine learning and deep learning are emerging as game changers in a world where more than 2.5 Exabytes of data are created per day. João Marques Lima talks to Databricks’ executives on the role of AI and barriers to wide adoption.

The use of data today is becoming more common, yet, businesses are still learning the ropes of addressing their – in some cases huge – data lakes.

Adding to that, is the penetration of artificial intelligence (AI) capabilities into the mix of tools used in the data analytics process.

Machine learning, deep dreaming and deep learning are just some of the most outstanding AI today. However, there is still a long way to go until these make it into the wider enterprise layer.

Ali Ghodsi, co-founder and CEO of Databricks

So, says Ali Ghodsi, co-founder and CEO of Databricks, an open source data software startup. “Humanity in general, when it comes to machine learning or deep learning, we are just scratching the surface,” he said.

“In the last three to four years, machine learning just popped up. Essentially people realised they know what to do with all those data sets in the cloud and all the data that they have.

“When big data is added to cloud computing, we can do machine learning on it and do predictions and amazing things can happen.”

Following the explosion of machine learning, Ghodsi highlighted another technological explosion that has just happened in the last two years and a half: deep learning.

“That particular mode of machine learning is much better than any other modes, it beats humans at many things.

“There are a lot of innovations that are happening all the time addressing the question of how do we simplify them and how do we make them accessible to mere mortals. By mere mortals I mean people that have computer science degrees, who have been doing this, intelligent mere mortals. How can we enable them to actually use this stuff?

“There is a lot that is going to happen until this technology becomes acceptable and it will become acceptable one day, but we have far to go. Right now, it is like a dark art, it is very magical and not everyone can do it.”

Despite maybe being some sort of “dark art”, the executive points that innovation in this space is accelerating, and humans might surprise and accept it faster than anyone thinks, “possibly in the next five years”.

“The culture is the hard part. People will overcome this very slowly unfortunately, that is a generational thing. There is a generation of people that wants to do things in a certain way and you cannot really tell them to do it in a different way.”


Teaching relevance

Databricks CTO Matei Zaharia

While the cultural change will take its time, a key component of preparing and future-proofing AI technologies, and any other technology by that matter, is education.

With Databricks’ routes dating back from when most of its founders were students at the University of California, we asked CTO Matei Zaharia about what he thinks of the current state of computing education in the light of the explosion in interest for AI tools.

He said: “First, the Curriculum in schools needs to be kept up to date. Often people create a course but then keep the same course for ten years and in a space like this, and in general in computing, the technology changes pretty fast.

“This results in people learning stuff that was relevant ten years ago but then there was no distributive systems, no cloud.”

On the industry side, Zaharia highlighted the existence of several online courses that people are using to learn more about computing and the new wave of IT.

“We actually ran some courses on Spark that got a lot of views and companies like Microsoft and IBM are doing the same. I think that is something that is helping.”


This article originally appeared in the Data Economy magazine. To read more on data centres, cloud and data, visit here