AI and big data go perfectly together -- sometimes
The combination of big data and AI tools enables new forms of analytics and automation, but use of the technologies in enterprise applications is still evolving.
In January, my colleagues at Dun & Bradstreet issued the results of a recent survey, which found that 40% of polled organizations are adding more jobs as a result of deploying AI. This finding appears to counter fears that AI adoption will reduce the availability of human jobs, with only eight of the 100 survey respondents saying that their organizations are cutting jobs due to AI.
The Dun & Bradstreet team polled attendees at the AI World Conference & Expo in Boston last December to glean these findings, which raise a larger question as to how companies are adapting to emerging technologies, such as AI and big data -- especially as we're in the midst of an unprecedented era of digital disruption that is only increasing in intensity.
Companies are using new technology to disrupt in new ways. And, as leaders in organizations face the realities of digital disruption, the cost of adopting even a fast-follower strategy on AI is simply untenable. The rapid advancement of technology and issues such as AI's impact on the future of work -- how jobs will change and the increasing impermanence of knowledgeable workers in the enterprise -- are forcing our hands.
The cost of doing nothing on AI is not nothing
The Dun & Bradstreet survey found that AI is largely being used for analytics, automation and data management. New capabilities are being enabled that make otherwise unapproachable domains far more accessible. For example, college professors can now use a host of tools to detect cheating, once a largely manual and experience-based process. In HR departments, technologies have emerged that can screen resumes, predict the future success of applicants and perform many other tasks that were once thought to be largely intractable for all but their most basic elements.
It isn't simply the new AI capabilities that make these applications more feasible; it's also the reimagining of operational tasks to take advantage of the available data and open up new ways of thinking. At the same time, changing privacy laws and increasingly clever malefactors who use advanced technologies in alarming new ways are forcing a greater portion of leadership mind share to be occupied with issues related to data security and governance.
As complex and potentially overwhelming as today's environment is, responding to this confluence of disruption will never be any easier. With a second generation of digital natives -- those born with technologies like the internet already in their life -- now emerging, we would do well to step back and examine how we're using the ever-growing abundance of AI and big data in businesses.
According to Dun & Bradstreet's survey, AI is currently in use to some degree at a majority of organizations. That finding is consistent with other industry studies, which have noted the transition from awareness and early-stage adoption of AI technology to full implementation and the of added business value from its use.
The reality is that many AI applications, especially those that require rich corpora of stable data from which to draw conclusions, have been stymied by the complexities of data discovery and curation. However, as big data technology has evolved to enable organizations to keep and manage increasingly large volumes of data, new applications that take advantage of things like IoT and mobile networks are starting to produce promising results. Some examples include facial recognition in law enforcement, smart city technology and autonomous devices, such as self-driving cars and drones.
Who's doing what with enterprise AI?
Surveys of AI practitioners have generally cited three groups: those with active, well-honed AI applications already deployed; those who have active projects underway but are still looking for the right balance of innovation and ROI; and those who are still exploring the technology or have yet to make a serious commitment to AI in the enterprise.
There's significant debate over the relative magnitude of these three groups. In Dun & Bradstreet's survey, which was conducted at an AI-focused event, nearly half of the respondents -- 44% -- said their businesses were in the process of deploying the technology, while 20% had fully deployed it in their organizations and 23% were implementations.
Businesses looking to AI to solve complex problems are sometimes left feeling a bit perplexed and less than satisfied with the outcome, suggesting that there is an explainability problem. If AI methods aren't well understood, it's difficult for humans to accept results that seem counterintuitive. This was evident in the Dun & Bradstreet survey results, with 46% of the respondents saying that understanding how AI arrives at its conclusions is an issue in their organizations. Only one-third stated that they fully understand how their AI systems come to conclusions.
Some of the other reasons for dissatisfaction with AI outcomes stem from basic problem formulation. For example, supervised AI methods trained by humans run the risk of making decisions based on potentially misleading reinforcement of existing knowledge, especially when the right steps aren't taken in advance to address bias -- in the data, the algorithms themselves or the interpretation of the results they produce.
Problem formulation relies on the ability of data scientists to ensure that the right methods and data are used and the right questions are asked to support the derived conclusions. The risk of incomplete problem formulation underscores the need to have explainable AI and more conversations about diversity of thought and methodology so that the technology can be more valuable to the enterprise.
The right mix of AI and big data
It's equally important to carefully consider the data used by AI. The lack of the right data was one of the greatest hurdles to further implementing AI in the organizations polled by Dun & Bradstreet -- both it and a lack of internal expertise were cited by 28% of the survey respondents.
As data continues to be produced and stored in exponentially increasing quantities, we will begin to see AI systems adapt and improve.
Though we as practitioners may have a reasonable handle on data volume, the velocity of change in big data environments remains a significant issue for some AI applications. Streaming data is a great example of a phenomenon often overlooked with the infamous data sample that fails to encompass random and assignable causes of variance, which may hold the key to the next big advancement -- or the next epic failure.
Data veracity is another issue that is increasingly relevant, especially to classification methodologies and other unsupervised AI methods. Data is the foundation upon which any technology -- especially AI -- must be built. A faulty data foundation -- for example, using data that contains bias or has been improperly manipulated -- often results in a faulty technology approach yielding faulty insights. This fallacy of construction often goes unnoticed and can be reinforced in negative ways by pressure to go to market sooner or to respond quickly to emergent business conditions.
AI's evolution crucial to its business value
But, as data continues to be produced and stored in exponentially increasing quantities, we will begin to see AI systems adapt and improve. This evolution is inherent to the business value of AI. Just as technology has the capability to self-diagnose to some extent, we will begin to see complex systems that can learn not only from human agents, but also from experience -- good examples include adversarial AI and ensemble methodologies.
In addition, the AI and data science practitioners of the next generation of digital natives will be far more clinical in making observations of systems so complex that they're not understandable in their entirety. These future data scientists will establish differential diagnoses much like today's medical practitioners do to distinguish between diseases with similar symptoms.
The combination of AI and big data will continue to evolve, and organizations are assured to continue to increase their experimentation with and deployment of the technology. There's no guarantee that this evolution will trend in a positive direction, however. In fact, some great modern minds have foretold quite the opposite.
The new science of digital disruption is entwined with the evolution of business and AI. What seems certain is that the pace and rate of that evolution will continue to increase. AI and big data are, in fact, not always perfect together today. What will ultimately make the difference in this domain is the increasing maturity of our analytical methods and our thinking.
