A Usability Study of Interpretable Aesthetic Evaluation as Decision Support in Full-Site AI Website Generators for Novice Users

Generative artificial intelligence systems increasingly enable novice users to create complex websites with minimal technical expertise; however, these users often struggle to evaluate the visual quality of AI-generated outputs and make confident refinement decisions. This study investigates whether an aesthetics-guided interaction design, integrating interpretable computational aesthetic evaluation into AI-powered website generation workflows can support users’ evaluative judgment, reduce cognitive effort, and improve usability and user experience. We conducted a within-subjects usability study (N = 30) comparing a widely used commercial full-site generator (Wix ADI) with an aesthetics-guided system that provides multi-dimensional aesthetic evaluation (e.g., color, layout, typography, and details) along with actionable improvement suggestions. Results showed that the guided system significantly improved perceived usability (p <.01, d = 0.36), AI literacy (p <.01, d = 0.31), AI self-efficacy (p <.01, d = 0.19), and perceived design quality (p <.01, d = 0.51). It also significantly reduced task completion time across all tasks (p <.05), indicating improved operational efficiency, while task success rates remained comparable. Eye-tracking results provided preliminary evidence consistent with a modest reduction in cognitive effort, although such findings should be interpreted cautiously due to potential confounding factors such as luminance variation. Qualitative findings suggest a shift in users’ questions from high-level conceptual uncertainty toward more operational and action-oriented concerns. Overall, the findings suggest that user-facing evaluative feedback may help novices externalize quality criteria and make more informed and confident design decisions.