LINGUIST List 23.1026: Qs: Looking for a Web Crawler for Corpus Analysis

LINGUIST List 23.1026
Wed Feb 29 2012 Qs: Looking for a Web Crawler for Corpus Analysis Editor for this issue: Zac Smith <zaclinguistlist.org>
We'd like to remind readers that the responses to queries are usually best posted to the individual asking the question. That individual is then strongly encouraged to post a summary to the list. This policy was instituted to help control the huge volume of mail on LINGUIST; so we would appreciate your cooperating with it whenever it seems appropriate. In addition to posting a summary, we'd like to remind people that it is usually a good idea to personally thank those individuals who have taken the trouble to respond to the query. To post to LINGUIST, use our convenient web form at http://linguistlist.org/LL/posttolinguist.cfm.
Date: 23-Feb-2012 From: Ana Popescu <discusurihotmail.com> Subject: Looking for a Web Crawler for Corpus Analysis E-mail this message to a friend Dear All, I would like to know if there is a web crawler that could download websites in text format - I have a list of aprox. 100 links from which I would like to collect the text (not the pdfs) and then take the .txt files and run them in a concording programme (I already have Wordsmith and another one). I want to be able to get word lists, but also to be able to find the file where a particular word originates - this is why I need the text files. I am interested in a crawler suitable for OS/Windows. Also, I want the crawler to be able to download the sites recursively, if asked to do so. I have found different free software of this kind on the Internet but they don't do everything I need. Thanks. Linguistic Field(s): Computational Linguistics This Year the LINGUIST List hopes to raise $67,000. This money will go to help keep the List running by supporting all of our Student Editors for the coming year. See below for donation instructions, and don't forget to check out Fund Drive 2012 site! http://linguistlist.org/fund-drive/2012/ There are many ways to donate to LINGUIST! You can donate right now using our secure credit card form at https://linguistlist.org/donation/donate/donate1.cfm Alternatively you can also pledge right now and pay later. To do so, go to: https://linguistlist.org/donation/pledge/pledge1.cfm For all information on donating and pledging, including information on how to donate by check, money order, or wire transfer, please visit: http://linguistlist.org/donation/ The LINGUIST List is under the umbrella of Eastern Michigan University and as such can receive donations through the EMU Foundation, which is a registered 501(c) Non Profit organization. Our Federal Tax number is 38-6005986. These donations can be offset against your federal and sometimes your state tax return (U.S. tax payers only). For more information visit the IRS Web-Site, or contact your financial advisor. Many companies also offer a gift matching program, such that they will match any gift you make to a non-profit organization. Normally this entails your contacting your human resources department and sending us a form that the EMU Foundation fills in and returns to your employer. This is generally a simple administrative procedure that doubles the value of your gift to LINGUIST, without costing you an extra penny. Please take a moment to check if your company operates such a program. Thank you very much for your support of LINGUIST! Read more issues\|LINGUIST home page\|Top of issue

Page Updated: 29-Feb-2012

Supported in part by the National Science Foundation

About LINGUIST | Contact Us

While the LINGUIST List makes every effort to ensure the linguistic relevance of sites listed on its pages, it cannot vouch for their contents.