LINGUIST List 5.492

Thu 28 Apr 1994

Qs: Ristad review, Idiom dictionary, Linguistic knowledge

Editor for this issue: <>

Directory

1. , Q: Reviews of Ristad's 'Language Complexity Game'
2. , Queries: What's the latest on ...
3. David W. Talmage, Looking for idiom dictionary
4. , Query: Boundary between Language and Other Faculties

I am writing one, and I was wondering if there have been
others, or if anybody else is writing one.

(1) Ape sign language?
(2) The claim of a gene for grammatical endings?
(3) Alingual adults (farm workers, I think) who can very
 well learn sign language?
(4) Manual babbling?

I am wondering if anybody has been following up on these
and would be willing to send me the latest references?

[I'm posting this for a colleague who is not on the Linguist list.
Please reply to me at dwtalideas.com. I'll forward any replies
to him.

Summary: Ideas, Inc. wants lists of English idioms to use in an information
retrieval benchmark.
]

Text Retrieval Processing

 Please indicate whether you know of any on-going work in the
linguistics or text retrieval fields pertaining to the analysis of text to
identify and/or count idiomatic phrases, any public domain idiom
dictionaries, or any potential sources of information in this area.

 We are participating in the development and implementation of a Boolean
benchmark test for full-text document retrieval products. The model for
the Boolean benchmark exercise will be Samuel DeFazio's Full-Text Document
Retrieval (FTR) Benchmark, hereafter referred to as the FTR Benchmark. The
DeFazio FTR benchmark depends on the identification and statistical
analysis of tokens within the body of textual data to be accessed under the
FTR benchmark test, including counting of each and every token appearing
 within the body of text. Subsequent analysis throws out the 50 most
frequently occurring tokens (on the theory that these are articles such as
"a," "and," or "the," and as such are not candidate tokens for search) and
categorizes each remaining token as high use, medium use, or low use,
depending on the token's count, or number of occurrences within the body of
text. During execution of the FTR benchmark, query search expressions are
generated from the list of tokens by category, randomly forming Boolean
expressions by combining tokens and Boolean operators (AND, AND NOT, OR,
etc.).

 Some modern, state-of-the-art text retrieval products perform idiomatic
processing based on dictionaries of commonly used idioms, or phrases, and
as such, are able to process queries for idioms within the text database.
We are therefore exploring the feasibility of including idiomatic phrases
in the FTR benchmark test by incorporating an idiom dictionary and idiom
identification capability in the token/phrase analysis part of the FTR
benchmark test.

 Therefore we request that you please identify any on-going work in the
linguistics or text retrieval field of which you are aware, pertaining to
the analysis of text to identify and/or count idioms, any public domain
idiom dictionaries, or any potential sources of information in this area.

 Thanks in advance for your assistance and cooperation.

[1] Full-Text Document Retrieval Benchmark, Version 1.1, November, 1992,
Samuel DeFazio, Sequent Computer Systems, Inc.


 ==============================================================================
David W. Talmage (uunet!aquin!luvthang!talmage)
"Once more. This is deixis. This is your brain on deixis. Any questions?"

Suppose we have the following discourse:

John's brothers are Harry, Mike, and Phil. John's brothers
both want to be pilots.

As compared to

John's brothers are Harry, Mike, and Phil. John's brothers,
except for Mike, both want to be pilots

Does anybody know of any discussion of where the boundary
between linguistic knowledge and other kinds of reasoning
abilities comes in in deciding the the first is "deviant"
and the second not? Or if not, would anybody care to
contribute any ideas on this?