BlackBox toolkit: intelligent assistance to UI design

Sermuga Pandian, Vinoth Pandian; Jarke, Matthias (Thesis advisor); Schröder, Ulrich J. (Thesis advisor)

Aachen : RWTH Aachen University (2022)
Dissertation / PhD Thesis

Dissertation, RWTH Aachen University, 2022

Abstract

User Interface (UI) design is an iterative process where designers iterate over multiple prototyping fidelities to finalise an aesthetic and usable interface. Prior research on adding Artificial Intelligence (AI) to the UI design process focused on automating the process while sacrificing the autonomy of designers. In this dissertation, we conduct systematic research using a human-centred approach to provide AI assistance to UI designers before, during, and after the traditional LoFi prototyping process. As a result, this research aims to provide coherent AI assistance throughout the repetitive and arduous LoFi prototyping task without sacrificing the autonomy of UI designers. In doing so, we contribute the BlackBox Toolkit. This toolkit assists designers by creating four large-scale, diverse, open-access benchmark datasets and three AI tools that assist UI designers throughout the LoFi prototyping process.Blackbox toolkit contributes the following datasets: UISketch dataset, ~18k UI element sketches; Syn & SynZ datasets, ~300k synthetic LoFi sketches; LoFi Sketch dataset, ~4.5k real-life LoFi sketches and Wired dataset, ~2.7k semantically annotated UI screenshots. Each of these datasets targets one of the two types of LoFi prototypes: LoFi sketches and LoFi wireframes. The datasets ensure ample diversity of designers and developers by data collection from a wide range of countries, input media, and prior experience.Moving on to the AI tools, Akin is a UI wireframe generator that uses a modified conditional SAGAN model and assists UI designers before LoFi prototyping by generating multiple UI wireframes for a given UI design pattern. Evaluation results show that the quality of Akin-generated UI wireframes was adequate. Further, the user evaluation shows that UI/UX designers considered Akin-generated wireframes as good as designer-made wireframes. Further, designers identified Akin-generated wireframes as designer-made 50% of the time.RUITE is a UI wireframe refiner that uses a Transformer-Encoder model and assists UI designers during LoFi prototyping by aligning and grouping UI elements in a given UI wireframe. On almost all evaluation metrics, it provides satisfactory results. The qualitative feedback indicates that designers prefer UI wireframe refinement using RUITE and expressed interest in using it.MetaMorph is a UI element detector that uses the RetinaNet object detection model and assists UI designers after LoFi prototyping by detecting the constituent UI elements of LoFi sketches and their location and dimension. It enables the transformation of LoFi sketches to higher-fidelities. Upon evaluation with hand-drawn LoFi sketches, MetaMorph provides 47.8% mAP. Further, the qualitative feedback shows that MetaMorph reduced designers’ effort in transforming LoFi prototype to higher fidelities by providing them with a head-start. In summary, from our qualitative feedback, the UI designers perceive utilising AI for UI design as an exciting and practical approach and expressed their eagerness to adopt such tools. Moreover, the user satisfaction studies conducted using After Scenario Questionnaires show an above-average designer satisfaction level upon using all three AI assistance tools. This research aims to understand the impact of AI tools in UI designer workflow and assess their satisfaction upon using these AI tools. Further, it sets a baseline for future research on UI wireframe generation, refinement and transformation.