In the last two years we’ve witnessed a pivotal moment in adoption of AI in real live applications; from chat bots that employ natural language classifiers and can drive a meaningful interaction with humans to advanced applications capable of using combined algorithms to solve more complex problems.

In practice, testing these AI applications becomes a real challenge, specifically because unlike traditional coding, the testing activity should not focus on covering the code-base but rather on the evaluation of the AI performance in relation to the training data. This fundamental change in the testing approach raises some interesting questions, like how to have real world data to test the AI against, how to handle regressions, how to automate.

Another fundamental change is that with AI, specifically with unsupervised learning models, testing would change the system, as it would act as training data. In our talk we’ll try to formulate some of the challenges that testing teams will face when interacting with AI applications in real life and propose some possible approaches that would work in this context.