Abstract
Large-scale natural language annotation projects have been costly, time consuming but indispensable tasks for computational linguistic research.\r Recently, however, with the advancement of web-based crowdsourcing services, a lot of attempts in computational linguistics research ahs shown that crowdsourcing platform enables rapid and affordable annotation of natural language.\r \r However, crowdsourced annotation schemes should be carefully designed for non-expert crowd annotators, decomposing and simplifying the complexities of laboratory-level annotation tasks.\r In this research, we explore the Pairwise Similarity Judgment (PSJ) framework as a decompositional toolkit for linguistic annotation tasks. We try to validate a non-expert annotation scheme inspired by this framework with an age-old complicated problem in computational linguistics, Semantic Role Labeling (SRL).\r \r We implement a PSJ annotation scheme for a SRL task, and then see how non-expert crowd annotators perform on the task using Amazon Mechanical Turk (AMT) platform. \r In our results, we found the PSJ framework to be useful to design a crowdsourced annotation task for even complex problems.