Developing the questionnaire of self-efficacy and needs in using large-language model-based AI services
Journal
Current Psychology
Journal Volume
44
Journal Issue
9
Start Page
8158-8176
ISSN
1046-1310
1936-4733
Date Issued
2025-01-04
Author(s)
Ju, Yu-Jeng
Wang, Yi-Ching
Liu, Cheng-Heng
Liu, Jen-Hsuan
Abstract
The rising prevalence of Large-Language Model-based AI Services (LLMAIs) underscores the importance of understanding users’ self-efficacy and specific needs in engaging with these technologies. Such insights are pivotal for evaluating LLMAI instructional effectiveness and guiding future developments of these technologies. However, current instruments fail to adequately capture these aspects. The study aimed to develop and validate the Questionnaire of Self-efficacy and Needs in using core features of LLMAIs (Q-SNELL). The Q-SNELL was developed through an extensive review of LLMAI core features. Its validation involved content and face validity assessments with a diverse group of experts and common users. Convergent validity, known-groups validity, and test-retest reliability were further evaluated through first and second round online surveys completed by users of LLMAIs. The Q-SNELL comprises three parts: the first focuses on self-efficacy and needs related to the eight core features of LLMAIs; the second includes two items identifying the most frequently used features; and the third assesses general self-efficacy. Content validity showed high agreement (76–100%), while face validity was also strong (80–100%), except for the ease of responding to the scale design (59% and 68% agreement). Validation with 398 participants showed mixed outcomes for convergent validity, strong support for known-groups validity, and moderate to high test-retest reliability (ICC = 0.47 to 0.74). The Q-SNELL is a pioneering instrument measuring users’ self-efficacy and needs regarding LLMAIs’ 8 core features and general usage. Its flexible structure allows for selective application of specific items, tailored to the needs of researchers.
Subjects
Artificial intelligence
ChatGPT
Large language model
Psychometric properties
Questionnaire
Self-efficacy
Publisher
Springer Science and Business Media LLC
Type
journal article