Study reveals new methods to improve ‘plug-and-play’ ML components
Dr Alex Cummaudo of the Applied Artificial Intelligence Institute (A²I²) aims to improve the quality of software that interacts with pre-trained machine learning (ML) models offered by cloud vendors. Over the past two years his research has developed an innovative framework that can be adopted to make usage of such AI components more reliable.
Multiple leading cloud vendors offer pre-trained ML models to software developers via their web service APIs. Common examples of this technology include object detection in images or speech-to-text services. These ‘intelligent’ web services, often marketed as ‘plug-and-play’ AI, aim to reduce technical knowledge required to infuse AI into applications, thereby reducing time-to-market.
A deeper look into how the services work suggests that the technology isn’t as simple as it looks. Dr Cummaudo’s research has highlighted that there are undocumented risks in how these services behave over time, with many developers facing complicated issues they cannot easily resolve.
“Our study has shown that AI-empowered cloud services aren’t as robust as people think. What is needed is more clear communication from the cloud vendors that empowers software engineers to build robust software by discussing potential risks,” Dr Cummaudo said.
“Inconsistencies can result in reliability issues, reducing software quality, and the lack of proper understanding by developers can lead to lost productivity in software teams.”
To overcome these issues, a facade architecture was developed and tested over a five-month study against a prominent computer vision service. Designed to assist developers in handling service behaviour, the approach identified 331 cases of evolution that would have otherwise ‘leaked’ into client systems and affected end-users. Dr Cummaudo is confident that this research will be used to better inform developers on such technology risks.
“This research aims to better inform industry professionals on how they can use intelligent web services to maintain overall system reliability in their development of complex applications.”
The findings from the study have been published in a series of top-tier software engineering journals and conferences. For more information, read Dr Cummaudo’s blog post and the peer-reviewed articles below.
- Beware the Evolving ‘Intelligent’ Web Service! An Integration Architecture Tactic to Guard AI-first Components, presented at the 2020 European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE’20)
- Interpreting Cloud Computer Vision Pain-Points: a Mining Study of Stack Overflow, presented at the 2020 International Conference on Software Engineering (ICSE’20)
- Losing Confidence in Quality: Unspoken Evolution of Computer Vision Services, presented at the 2019 International Conference on Software Maintenance and Evolution (ICSME’19)
- Requirements of API Documentation: A Case Study into Computer Vision Services, published in the IEEE Transactions on Software Engineering (TSE)