This dissertation reports development and validation studies of a Web-based
standardized test of Japanese as a foreign language (JFL), designed to measure learners’
off-line grammatical and pragmatic knowledge in multiple-choice format. Targeting
Japanese majors in the U.S. universities and colleges, the test is designed to explore
possible relationships between JFL learners’ grammatical knowledge and domains of
pragmatic knowledge. Pragmatic knowledge is defined as knowledge of “[linguistic]
forms necessary for productively conveying speech intentions, and assigning pragmatic
meaning to interlocutor utterance” (Röver 2005), and is operationalized as the ability to
associate linguistic features in specific contexts with appropriate speech acts, routines,
and speech styles. Grammatical knowledge includes well-formedness of isolated
sentences at the morpho-syntactic level, independent of contextual information. Research
questions address reliability of the test, the relationship between JFL learners’
grammatical and pragmatic knowledge (Bardovi-Harlig and Dörnyei 1998), as well as the
relationship between the overall scores of this multiple-choice test and the ACTFL
Proficiency Guidelines Levels for speaking.The data from 110 participants of JFL learners in four different universities
revealed that the test was overall reliable. The total scores of the test positively correlated with length of study/exposure, levels of instruction, and proficiency ratings based on the
ACTFL Guidelines. As for the relationship among grammatical and domains of
pragmatic knowledge, the study found complex interrelationships: correlation analyses
showed that the four subsections (Grammar, Routines, Speech Acts, and Speech Styles)
are significantly correlated overall, but the strengths of correlation varied depending on
the levels of instruction and proficiency levels. The Routines section was somewhat
separate and was not highly correlated with other sections at upper levels. The data also
suggest that, as far as JFL is concerned, the knowledge of speech styles is in closer
relationship with the knowledge of grammar and speech acts, compared with studies on
learners of English as a Second/Foreign Language. Exploratory factor analysis, however,
found unexpected patterns of factor loadings, not matching the internal structure of the
test, suggesting that the hypothesized components of language ability may not be
empirically distinguishable. Construct validity of the test is defensible only for the overall
score being a valid indication of communicative language ability in general. The extent
to which each section scores are valid indications of corresponding hypothesized separate
components of the language ability needs to be further investigated in future studies.
Pedagogical implications for testing knowledge of natives’ prototypical communication
patterns, limitations of the study, and future development of the test are discussed. |