A Qualitative Assessment of Using ChatGPT as Large Language Model for Scientific Workflow Development

Abstract

Bac kgr ound: Scientific w orkflow systems ar e incr easingl y popular for expr essing and e xecuting comple x data anal ysis pipelines ov er large datasets, as they offer r e pr oducibility, de penda bility, and scala bility of anal yses by automatic parallelization on large compute clusters. How ever, implementing w orkflows is difficult due to the inv olv ement of many black-box tools and the deep infrastructure stack necessary for their execution. Simultaneously, user-supporting tools are r are , and the number of available examples is much lower than in classical programming languages. Results: To address these c hallenges, w e investigate the efficiency of large language models (LLMs), specifically ChatGPT, to support users when dealing with scientific workflows. We performed 3 user studies in 2 scientific domains to evaluate ChatGPT for comprehending, adapting, and extending workflows. Our results indicate that LLMs efficiently interpret workflows but achieve lower performance for exchanging components or purposeful workflow extensions. We c har acterize their limitations in these challenging scenarios and suggest future resear c h directions. Conclusions: Our results show a high accuracy for comprehending and explaining scientific workflows while achieving a reduced performance for modifying and extending workflow descriptions. These findings clearly illustrate the need for further resear c h in this area. Ke yw ords: lar g e languag e models, scientific workflows, user support, ChatGPT Key points: r We explore large language models (LLMs) to support users who de v elop scientific w orkflo ws. r We are the first to conduct user studies involving domain experts. r We conduct 3 studies to assess LLMs in scientific workflow compr ehension, ada ptation, and extension. r Our results indicate that LLMs efficientl y inter pr et workflows. r Our results show room for improvement regarding component adaptation and w orkflo w extension.