We are glad to announce that DeepTest 2020 will take place online on the Zoom platform on Wednesday, July 1, 2020. There will be two thematic sessions from 6:45 AM to 10 AM UTC Time, and from 2:45 PM to 6 PM UTC Time, with discussions focusing on topics covered in the accepted papers, along with panels with influential experts, and two exciting keynotes.
Workshop Theme
Software development is going through a paradigm shift, where decision making is increasingly shifting from hand-coded program logic to Deep Learning (DL). Popular applications in Speech Processing, Image Recognition, Robotics, Go game, use Deep Learning as their core components. DL is also widely adopted in safety-critical systems like autonomous cars, medical diagnosis, and aircraft collision avoidance systems.
Thus, it has become crucial to rigorously test DL applications with realistic corner cases to ensure high reliability.
Due to its effectiveness to learn from historical data, DL is also being applied for devising novel program analysis and software testing techniques related to malware detection, fuzz testing, bug-finding, and type-checking.
Therefore, testing DL-based applications and DL-based testing techniques are emerging and important Software Engineering topics.
DeepTest is a workshop targeting research at the intersection of testing and deep learning. This interdisciplinary workshop will explore issues related to:
Deep learning applied to software testing.
Testing applied to Deep Learning applications.
The workshop will consist of invited talks, presentations based on abstract submissions, and a panel discussion, where all participants are invited to share their insights and ideas.
The aim of the workshop is to bring together an international group of researchers with both Machine Learning, Testing, and Verification backgrounds to come discuss their research, share datasets, and generally help the field build momentum.
Registration 📅
Registration for DeepTest 2020 will open on 10 June 2020 on the main ICSE 2020 web site.
Guidelines for Presenters and Audience 📋
We will use Zoom as a main platform. Guidelines on the organization of the virtual workshop are summarized
here. Given the virtual nature of the workshop, the document also contains a few tips for presenters and attendees. Feel free to download and use the official DeepTest 2020 Zoom's virtual background here.
Keynotes ✨
Prof. Shin Yoo, KAIST - Korea Advanced Institute of Science & Technology.
Searching for Cost Effective Test Inputs for DNN Testing Abstract:
DNNs are a good example of the "non-testable" programs, a concept proposed by Weyuker in the 80s. Due to the challenges in generating test oracles for them, many existing testing techniques for DNNs are based on the concept of metamorphic testing. An implication of this is that, unlike traditional programs, we can neither randomly sample nor search for inputs with specific properties, instead bound by pre-determined perturbations. This talk will look at some of the recent first steps in search based test data generation, which allows us to navigate the input space guided by Surprise Adequacy (SA). We will also look at how can be used to mitigate the problem of manual labelling cost.
Prof. Matt Fredrikson, School of Computer Science Carnegie Mellon University (CMU).
How to find ML bugs that expose training data and bias outcomes Abstract:
Systems and services that essentially rely on data to provide functionality are widespread, as a growing number of high-profile success stories drives their adoption into new domains. Increasingly, the technology underpinning this trend is deep learning, which has enabled new applications that had previously eluded traditional software development methods. However, this development has also been met with concerns around the privacy of individuals' data, and the potential for these systems to discriminate in unintended and harmful ways. In this talk, we will see how privacy and fairness in ML applications concerns can be related through the lens of protected information use, and show that tools developed to help characterize ML models' use of such information can uncover new types of "bugs" that expose private training data and lead to unwarranted discrimination. Finally, we will discuss promising techniques that address these issues through novel data representations and model post-processing, leading towards ML applications that solve important problems without jeopardizing the privacy and fairness concerns of their users.
Accepted talks 💻
Evaluating Surprise Adequacy for Question Answering by Seah Kim and Shin Yoo.
OffSide: Learning to Identify Mistakes in Boundary Conditions by Jón Briem, Jordi Smit, Hendrig Sellik, Pavel Rapoport, Georgios Gousios and Maurício Aniche.
Deep Learning for Software Defect Prediction: A Survey by Safa Omri and Carsten Sinz.
Does Neuron Coverage Matter for Deep Reinforcement Learning? A Preliminary Study by Miller Trujillo, Mario Linares-Vásquez, Camilo Escobar-Velásquez, Ivana Dusparic and Nicolás Cardozo.
DeepFuzzSL: Generating Models with Deep Learning to Find Bugs in the Simulink Compiler by Sohil Shrestha, Shafiul Azam Chowdhury and Christoph Csallner.
Manifold-based Test Generation for Image Classifiers by Taejoon Byun and Sanjai Rayadurgam.
Click the Zoom Link, or use the Meeting ID: 669 619 7635 and the password deeptest20. Feel free to download and use the official DeepTest 2020 Zoom's virtual background here.
06:45-07:00 Opening and Welcome from the Chairs
07:00-08:00 Keynote 1: Searching for Cost Effective Test Inputs for DNN Testing by Shin Yoo
08:00-08:20 Evaluating Surprise Adequacy for Question Answering.
Seah Kim and Shin Yoo
08:20-08:40 Deep Learning for Software Defect Prediction: A Survey
Safa Omri and Carsten Sinz
08:40-09:00 OffSide: Learning to Identify Mistakes in Boundary Conditions
Jón Briem, Jordi Smit, Hendrig Sellik, Pavel Rapoport, Georgios Gousios and Maurício Aniche
Click the Zoom Link, or use the Meeting ID: 669 619 7635 and the password deeptest20. Feel free to download and use the official DeepTest 2020 Zoom's virtual background here.
14:45-15:00 Welcome from the Chairs
15:00-16:00 Keynote 2: How to find ML bugs that expose training data and bias outcomes by Matt Fredrikson
16:00-16:20 Manifold-based Test Generation for Image Classifiers
Taejoon Byun and Sanjai Rayadurgam
16:20-16:40 DeepFuzzSL: Generating Models with Deep Learning to Find Bugs in the Simulink Compiler
Sohil Shrestha, Shafiul Azam Chowdhury and Christoph Csallner
16:40-17:00 Does Neuron Coverage Matter for Deep Reinforcement Learning? A Preliminary Study
Miller Trujillo, Mario Linares-Vásquez, Camilo Escobar-Velásquez, Ivana Dusparic and Nicolás Cardozo
We accept one type of submission:
presentation abstracts up to 6-page papers describing work related to the workshop topics. Relevant topics include, but are not limited to:
Benchmarks for evaluating DL approaches
Techniques to aid interpretable DL techniques
Techniques to improve designing reliable models
DL-aided software development approaches
DL for fault prediction, localization and repair
Metamorphic testing as software quality assurance
Security in DL systems
Fault Localization and anomaly detection
Use of DL for analyzing natural language-like artifacts such as code, or user reviews
DL techniques to support automated software testing
Formal verification of DL-based systems
DL to aid program comprehension, program transformation, and program generation
Safety and security of DL based systems
New approaches to uncertainty measures and estimation
Both preliminary work and new insights in previous work are encouraged.
Note: Although the main focus is on deep learning we encourage submissions that are related more broadly to machine learning and testing and the relationship between the two.
Submissions should be in ACM Master two-column format (https://www.acm.org/publications/proceedings-template).
Presentation abstracts will not be included in the ACM Digital Library, but will be included in an informal pre-proceedings on the website.
We very much welcome presentation abstracts about work already published elsewhere, or giving an overview of an existing system, and the format is designed not to preclude future publication.
If you have any questions, or wonder whether your submission is in scope, please do not hesitate to contact the organizers.
Organizers
Andrea Stocco, Università della Svizzera italiana (USI), Switzerland
Gunel Jahangirova, Università della Svizzera italiana (USI), Switzerland
Vincenzo Riccio, Università della Svizzera italiana (USI), Switzerland
