LINGUIST List 33.1796
Thu May 19 2022
FYI: Call for Participation in the NLP Community Metasurvey
Editor for this issue: Everett Green <everettlinguistlist.org>
Date: 19-May-2022
From: Julian Michael <nlp-metasurvey-admin
nyu.edu>
Subject: Call for Participation in the NLP Community Metasurvey
E-mail this message to a friend What do NLPers think about controversial NLP issues? And perhaps more interestingly: What do NLPers think other NLPers think about controversial NLP issues?
TL;DR: We invite you to take a (~20 minute) survey that not only gauges the beliefs of the NLP community on actively debated ideas and positions, but also asks: How does the NLP community view itself?
For the first 1,000 respondents, we will donate $10 to a non-profit that you can choose at the end of the survey.
Take the survey »
https://nyu.qualtrics.com/jfe/form/SV_5iECGAxs8SnufSm Recent rapid changes in the NLP landscape have raised many questions. For example:
* Are we devoting too many resources to scaling up?
* Do language models understand language? Will they ever?
* Is the traditional paradigm of model benchmarking still tenable?
* What kinds of predictive models are ethical for researchers to build and release?
* Will most influential advances come from industry or academic labs over the next 10 years?
On these and similar questions, it’s often unclear what counts as general consensus, what is still actively debated, or what is widely considered overturned. We believe it would benefit the field to have a better understanding of the current landscape of community opinions and beliefs.
To this end, we urge researchers in computational linguistics and natural language processing to take the NLP Community Metasurvey. This survey covers more than 30 potentially controversial positions, asking not only whether you agree, but what percentage of the community you think agrees with the position, thus exposing the difference between "what people think" and "what people think people think." This idea was inspired by the PhilPapers Surveys (
http://philpapers.org/surveys).
The results and analysis will be descriptive, not prescriptive, as such issues cannot be resolved by majority vote. By necessity, we are covering a subjective and biased subset of questions and reducing many complex issues into simplified scales, but we hope that the outcome can serve as a reference point for community discussion and for future surveys. This is not the final word in any debate, but we hope it will spark new discussions as an initial study of the range of positions people hold and ways in which the community may mis-model itself.
The results will be written up in a report and visualizations of the data will be made accessible online (suitably aggregated to prevent deanonymization) at
https://nlpsurvey.net.
We hope to cover as much of our large and growing community as possible, so please participate! We also encourage you to share it in other NLP/CL spaces like email lists, school or lab Slack groups, and Twitter.
Thanks for taking part in something we hope will help us all understand our community better!
—The NLP Community Metasurvey Team
More information at
https://nlpsurvey.net Linguistic Field(s): Computational Linguistics
Page Updated: 19-May-2022