Following up on Peter's point 
> 
> So the P&P parser that Sproat and Lappin envision would accomplish 
> much more than comparable statistical parsers, which makes the 
> proposed accuracy metric a poor yardstick for comparison 
> 
 
In addition to capturing the distinction between learnable and unlearnable 
languages, P&P has as an important goal capturing the distinction between 
well-formed (grammatical) and ill-formed (ungrammatical) sentences 
within a language. As I understand it, the challenge demands only correct 
parsing of grammatical sentences, not correct rejection of ungrammatical 
ones. This represents another case where the P&P system, by virtue of the 
goals of the theory, is being subjected to greater demands than the 
statistical parsers. 
 
Comp Ling isn't my field either, but I gather it is a desideratum for at least 
some statistical parsers that they be robust in the face of noisy input, 
certainly during training but perhaps also during parsing, if they are to 
avoid being completely thrown off by the occasional typo or unfamiliar 
word. So it strikes me as an interesting empirical question whether such 
robustness, if indeed the best statistical parsers have it, hinders them from 
being able to detect ungrammaticality in general. Of course humans too 
can "cope with" ill-formedness of various kinds (as Sproat and Lappin 
note), but they mostly know when they are having to do so, i.e., ill- 
formedness is still detected. 
 
So, I would like to suggest a revised version of the challenge that 
incorporates a second corpus consisting of ungrammatical sentences that 
are to be identified as such. (Earlier P&P parsers such as Fong's were 
designed to do this, but it's not obvious that this ability will easily scale up 
with broader coverage, so I don't think this is a sucker's bet.) Furthermore, 
since the computationalists got to choose the corpus of good sentences, it 
would seem only fair that the theoreticians get to choose the corpus of bad 
sentences :-) 
 
P.S. The statistical parsers will still be getting off easy, in my view, because 
the unfamiliar sentences they *are* supposed to parse as well-formed are 
drawn from the same sample as the training set. The set of novel sentences 
humans [and P&P parsers, we hope] parse as grammatical arguably 
includes sentence types that do not occur in the language learner's input. 
 
-- 
 
Carson T. Schutze Department of Linguistics, UCLA 
Web: http://www.linguistics.ucla.edu/people/cschutze

In response to Richard Sproat and Shalom Lappin's challenge (16.1156), 
Peter Hallman (16.1251) draws a contrast between the Principles and 
Parameters (P&P) approach and statistical approaches to parsing. 
 
 A statistical parser can, within physical limitations, recognize 
 and learn any statistically significant pattern, not merely those 
 patterns that occur in human languages.... The P&P framework 
 seeks to answer the question 
 
 (Q) What is a possible human language (type)? 
 
 The P&P parser that Sproat and Lappin envision would answer this 
 question; comparable statistical parsers do not. 
 
He suspects that it would be "unrealistic" for a P&P parser to reach accuracy 
comparable to current statistical parsers in three years, for two reasons. 
First, as the paragraph above concludes, a P&P parser would accomplish 
more than current statistical parsers. Second, current P&P theory may not 
be "ready to form the basis of a trainable parser". 
 
I am more optimistic for P&P. To me, these same two reasons indicate 
Sproat and Lappin's challenge to be realistic rather than unrealistic. 
 
First, a statistical parser is only hindered when it recognizes patterns that 
do not occur in human languages. The larger the space of hypotheses to 
explore, the less effective machine learning can be. Conversely, many 
advances in statistical parsing (going back as far as probabilistic regular 
and context-free grammars) are made precisely by better 
delineating "those patterns that occur in human languages", such as locality 
and hierarchy. In other words, a statistical parser embodies an 
(approximate) answer to the question Q, just as a P&P parser or theory 
does. A better answer should give rise to a better parser. 
 
Second, the attention that the P&P approach pays to language acquisition 
corresponds directly to payoffs in parsing performance. For example, a 
parser whose design addresses the poverty of the stimulus should require 
less training data, less supervision, or both. Such a parser would be able to 
learn from the Penn Treebank better, take advantage of vast amounts of 
unlabeled corpora, or both. 
 
In sum, a parser that better "connect[s] typological universals to the 
mechanism of language learning" will fare better in accuracy, all other 
things being equal. That one linguistic theory may be more "ready" than 
another for implementation reflects on not just the focus of different 
communities (as Martha McGinnis points out, 16.1251), but also the 
theories themselves. Trying to answer the question Q is no excuse for poor 
parsing. All other things being equal, poor (or unknown) parsing 
performance indicates failure at (resp. disinterest in) answering Q.

The challenge suggested by Richard Sproat is in my opinion a most 
important research idea, vital to the further development and expansion of 
P&P, although I share some of the worries expressed by previous writers 
here. 
 
I would like to add that the positive approach to this challenge should 
be “how can P&P be made to work”, and not “let’s see how P&P fails to meet 
its claims”. 
 
There is growing skepticism in psycholinguistic circles that P&P, though 
accepted, does not deliver: It provides no practical gain in answering the 
question how language is acquired (satisfying explanatory adequacy). 
 
Moreover, the MP is considered to be too complicated, only accessible and 
understood by a small isolated group of people, therefore of no practical 
use, although it makes claims about explanatory adequacy. Quantum 
physics is also extremely complex and difficult to understand, yet nobody 
has claimed that it is of no practical use or isolated from the real world. 
 
Generative circles have already identified the growing disparity between 
P&P and psycholinguistic research. Currently, a broad research program 
headed by Janet Dean Fodor et al. (CUNY) is carried out to satisfy 
explanatory adequacy – to meet Sproat’s challenge. 
 
I hope that one of the researchers will post a message here, or that Richard 
Sproat will post their messages on the matter, should he get any. 
 
Cheers, 
-Oren.