skip to content
Web corpus construction Preview this item
ClosePreview this item
Checking...

Web corpus construction

Author: Roland Schäfer; Felix Bildhauer; Morgan & Claypool Publishers.
Publisher: [San Rafael, Calif.] : Morgan & Claypool Publishers, ©2013.
Series: Synthesis lectures on human language technologies, # 22.
Edition/Format:   Print book : EnglishView all editions and formats
Database:WorldCat
Summary:
The World Wide Web constitutes the largest existing source of texts written in a great variety of languages. A feasible and sound way of exploiting this data for linguistic research is to compile a static corpus for a given language. There are several advantages of this approach: (i) Working with such corpora obviates the problems encountered when using Internet search engines in quantitative linguistic research  Read more...
Rating:

(not yet rated) 0 with reviews - Be the first.

Subjects
More like this

 

Find a copy in the library

&AllPage.SpinnerRetrieving; Finding libraries that hold this item...

Details

Document Type: Book
All Authors / Contributors: Roland Schäfer; Felix Bildhauer; Morgan & Claypool Publishers.
ISBN: 1608459837 9781608459834
OCLC Number: 847941536
Notes: "This volume is a printed version of a work that appears in the Synthesis digital library of engineering and computer science"--Page 4 of cover.
Description: xv, 129 pages : illustrations ; 24 cm.
Contents: 1. Web corpora --
2. Data collection --
2.1 Introduction --
2.2 The structure of the web --
2.2.1 General properties --
2.2.2 Accessibility and stability of web pages --
2.2.3 What's in a (national) top level domain? --
2.2.4 Problematic segments of the web --
2.3 Crawling basics --
2.3.1 Introduction --
2.3.2 Corpus construction from search engine results --
2.3.3 Crawlers and crawler performance --
2.3.4 Configuration details and politeness --
2.3.5 Seed URL generation --
2.4 More on crawling strategies --
2.4.1 Introduction --
2.4.2 Biases and the pagerank --
2.4.3 Focused crawling --
3. Post-processing --
3.1 Introduction --
3.2 Basic cleanups --
3.2.1 HTML stripping --
3.2.2 Character references and entities --
3.2.3 Character sets and conversion --
3.2.4 Further normalization --
3.3 Boilerplate removal --
3.3.1 Introduction to boilerplate --
3.3.2 Feature extraction --
3.3.3 Choice of the machine learning method --
3.4 Language identification --
3.5 Duplicate detection --
3.5.1 Types of duplication --
3.5.2 Perfect duplicates and hashing --
3.5.3 Near duplicates, Jaccard coefficients, and shingling --
4. Linguistic processing --
4.1 Introduction --
4.2 Basics of tokenization, part-of-speech tagging, and lemmatization --
4.2.1 Tokenization --
4.2.2 Part-of-speech tagging --
4.2.3 Lemmatization --
4.3 Linguistic post-processing of noisy data --
4.3.1 Introduction --
4.3.2 Treatment of noisy data --
4.4 Tokenizing web texts --
4.4.1 Example: missing whitespace --
4.4.2 Example: emoticons --
4.5 POS tagging and lemmatization of web texts --
4.5.1 Tracing back errors in POS tagging --
4.6 Orthographic normalization --
4.7 Software for linguistic post-processing --
5. Corpus evaluation and comparison --
5.1 Introduction --
5.2 Rough quality check --
5.2.1 Word and sentence lengths --
5.2.2 Duplication --
5.3 Measuring corpus similarity --
5.3.1 Inspecting frequency lists --
5.3.2 Hypothesis testing with --
5.3.3 Hypothesis testing with Spearman's rank correlation --
5.3.4 Using test statistics without hypothesis testing --
5.4 Comparing keywords --
5.4.1 Keyword extraction with x2 --
5.4.2 Keyword extraction using the ratio of relative frequencies --
5.4.3 Variants and refinements --
5.5 Extrinsic evaluation --
5.6 Corpus composition --
5.6.1 Estimating corpus composition --
5.6.2 Measuring corpus composition --
5.6.3 Interpreting corpus composition --
5.7 Summary.
Series Title: Synthesis lectures on human language technologies, # 22.
Other Titles: Synthesis digital library of engineering and computer science.
Responsibility: Roland Schäfer and Felix Bildhauer.

Abstract:

The World Wide Web constitutes the largest existing source of texts written in a great variety of languages. A feasible and sound way of exploiting this data for linguistic research is to compile a static corpus for a given language. There are several advantages of this approach: (i) Working with such corpora obviates the problems encountered when using Internet search engines in quantitative linguistic research (such as non-transparent ranking algorithms). (ii) Creating a corpus from web data is virtually free. (iii) The size of corpora compiled from the WWW may exceed by several orders of magnitudes the size of language resources offered elsewhere. (iv) The data is locally available to the user, and it can be linguistically post-processed and queried with the tools preferred by her/him. This book addresses the main practical tasks in the creation of web corpora up to giga-token size. Among these tasks are the sampling process (i. e., web crawling) and the usual cleanups including boilerplate removal and removal of duplicated content. Linguistic processing and problems with linguistic processing coming from the different kinds of noise in web corpora are also covered. Finally, the authors show how web corpora can be evaluated and compared to other corpora (such as traditionally compiled corpora).

Reviews

User-contributed reviews
Retrieving GoodReads reviews...
Retrieving DOGObooks reviews...

Tags

Be the first.
Confirm this request

You may have already requested this item. Please select Ok if you would like to proceed with this request anyway.

Linked Data


Primary Entity

<http://www.worldcat.org/oclc/847941536> # Web corpus construction
    a schema:CreativeWork, schema:Book ;
   library:oclcnum "847941536" ;
   library:placeOfPublication <http://experiment.worldcat.org/entity/work/data/1371127490#Place/san_rafael_calif> ; # San Rafael, Calif.
   library:placeOfPublication <http://id.loc.gov/vocabulary/countries/cau> ;
   rdfs:seeAlso <http://experiment.worldcat.org/entity/work/data/1371127490#CreativeWork/synthesis_digital_library_of_engineering_and_computer_science> ; # Synthesis digital library of engineering and computer science.
   schema:about <http://experiment.worldcat.org/entity/work/data/1371127490#Topic/sprache> ; # Sprache
   schema:about <http://id.loc.gov/authorities/subjects/sh85077224> ; # Computational linguistics
   schema:about <http://experiment.worldcat.org/entity/work/data/1371127490#Topic/internet> ; # Internet
   schema:about <http://experiment.worldcat.org/entity/work/data/1371127490#Topic/korpus> ; # Korpus
   schema:about <http://id.worldcat.org/fast/1173234> ; # Web search engines
   schema:about <http://experiment.worldcat.org/entity/work/data/1371127490#Topic/corpora_linguistics_data_processing> ; # Corpora (Linguistics)--Data processing
   schema:about <http://id.worldcat.org/fast/871998> ; # Computational linguistics
   schema:about <http://experiment.worldcat.org/entity/work/data/1371127490#Topic/computerlinguistik> ; # Computerlinguistik
   schema:about <http://id.loc.gov/authorities/subjects/sh97007463> ; # Web search engines
   schema:bookFormat bgn:PrintBook ;
   schema:contributor <http://viaf.org/viaf/131713828> ; # Morgan & Claypool Publishers.
   schema:contributor <http://experiment.worldcat.org/entity/work/data/1371127490#Person/bildhauer_felix> ; # Felix Bildhauer
   schema:copyrightYear "2013" ;
   schema:creator <http://experiment.worldcat.org/entity/work/data/1371127490#Person/schafer_roland_1974> ; # Roland Schäfer
   schema:datePublished "2013" ;
   schema:description "1. Web corpora -- 2. Data collection -- 2.1 Introduction -- 2.2 The structure of the web -- 2.2.1 General properties -- 2.2.2 Accessibility and stability of web pages -- 2.2.3 What's in a (national) top level domain? -- 2.2.4 Problematic segments of the web -- 2.3 Crawling basics -- 2.3.1 Introduction -- 2.3.2 Corpus construction from search engine results -- 2.3.3 Crawlers and crawler performance -- 2.3.4 Configuration details and politeness -- 2.3.5 Seed URL generation -- 2.4 More on crawling strategies -- 2.4.1 Introduction -- 2.4.2 Biases and the pagerank -- 2.4.3 Focused crawling -- 3. Post-processing -- 3.1 Introduction -- 3.2 Basic cleanups -- 3.2.1 HTML stripping -- 3.2.2 Character references and entities -- 3.2.3 Character sets and conversion -- 3.2.4 Further normalization -- 3.3 Boilerplate removal -- 3.3.1 Introduction to boilerplate -- 3.3.2 Feature extraction -- 3.3.3 Choice of the machine learning method -- 3.4 Language identification -- 3.5 Duplicate detection -- 3.5.1 Types of duplication -- 3.5.2 Perfect duplicates and hashing -- 3.5.3 Near duplicates, Jaccard coefficients, and shingling -- 4. Linguistic processing -- 4.1 Introduction -- 4.2 Basics of tokenization, part-of-speech tagging, and lemmatization -- 4.2.1 Tokenization -- 4.2.2 Part-of-speech tagging -- 4.2.3 Lemmatization -- 4.3 Linguistic post-processing of noisy data -- 4.3.1 Introduction -- 4.3.2 Treatment of noisy data -- 4.4 Tokenizing web texts -- 4.4.1 Example: missing whitespace -- 4.4.2 Example: emoticons -- 4.5 POS tagging and lemmatization of web texts -- 4.5.1 Tracing back errors in POS tagging -- 4.6 Orthographic normalization -- 4.7 Software for linguistic post-processing -- 5. Corpus evaluation and comparison -- 5.1 Introduction -- 5.2 Rough quality check -- 5.2.1 Word and sentence lengths -- 5.2.2 Duplication -- 5.3 Measuring corpus similarity -- 5.3.1 Inspecting frequency lists -- 5.3.2 Hypothesis testing with -- 5.3.3 Hypothesis testing with Spearman's rank correlation -- 5.3.4 Using test statistics without hypothesis testing -- 5.4 Comparing keywords -- 5.4.1 Keyword extraction with x2 -- 5.4.2 Keyword extraction using the ratio of relative frequencies -- 5.4.3 Variants and refinements -- 5.5 Extrinsic evaluation -- 5.6 Corpus composition -- 5.6.1 Estimating corpus composition -- 5.6.2 Measuring corpus composition -- 5.6.3 Interpreting corpus composition -- 5.7 Summary."@en ;
   schema:description "The World Wide Web constitutes the largest existing source of texts written in a great variety of languages. A feasible and sound way of exploiting this data for linguistic research is to compile a static corpus for a given language. There are several advantages of this approach: (i) Working with such corpora obviates the problems encountered when using Internet search engines in quantitative linguistic research (such as non-transparent ranking algorithms). (ii) Creating a corpus from web data is virtually free. (iii) The size of corpora compiled from the WWW may exceed by several orders of magnitudes the size of language resources offered elsewhere. (iv) The data is locally available to the user, and it can be linguistically post-processed and queried with the tools preferred by her/him. This book addresses the main practical tasks in the creation of web corpora up to giga-token size. Among these tasks are the sampling process (i. e., web crawling) and the usual cleanups including boilerplate removal and removal of duplicated content. Linguistic processing and problems with linguistic processing coming from the different kinds of noise in web corpora are also covered. Finally, the authors show how web corpora can be evaluated and compared to other corpora (such as traditionally compiled corpora)."@en ;
   schema:exampleOfWork <http://worldcat.org/entity/work/id/1371127490> ;
   schema:inLanguage "en" ;
   schema:isPartOf <http://experiment.worldcat.org/entity/work/data/1371127490#Series/synthesis_lectures_on_human_language_technologies> ; # Synthesis lectures on human language technologies ;
   schema:isPartOf <http://worldcat.org/issn/1947-4059> ; # Synthesis lectures on human language technologies,
   schema:name "Web corpus construction"@en ;
   schema:productID "847941536" ;
   schema:publication <http://www.worldcat.org/title/-/oclc/847941536#PublicationEvent/san_rafael_calif_morgan_&_claypool_publishers_2013> ;
   schema:publisher <http://experiment.worldcat.org/entity/work/data/1371127490#Agent/morgan_&_claypool_publishers> ; # Morgan & Claypool Publishers
   schema:workExample <http://worldcat.org/isbn/9781608459834> ;
   wdrs:describedby <http://www.worldcat.org/title/-/oclc/847941536> ;
    .


Related Entities

<http://experiment.worldcat.org/entity/work/data/1371127490#Agent/morgan_&_claypool_publishers> # Morgan & Claypool Publishers
    a bgn:Agent ;
   schema:name "Morgan & Claypool Publishers" ;
    .

<http://experiment.worldcat.org/entity/work/data/1371127490#CreativeWork/synthesis_digital_library_of_engineering_and_computer_science> # Synthesis digital library of engineering and computer science.
    a schema:CreativeWork ;
   schema:name "Synthesis digital library of engineering and computer science." ;
    .

<http://experiment.worldcat.org/entity/work/data/1371127490#Person/bildhauer_felix> # Felix Bildhauer
    a schema:Person ;
   schema:familyName "Bildhauer" ;
   schema:givenName "Felix" ;
   schema:name "Felix Bildhauer" ;
    .

<http://experiment.worldcat.org/entity/work/data/1371127490#Person/schafer_roland_1974> # Roland Schäfer
    a schema:Person ;
   schema:birthDate "1974" ;
   schema:familyName "Schäfer" ;
   schema:givenName "Roland" ;
   schema:name "Roland Schäfer" ;
    .

<http://experiment.worldcat.org/entity/work/data/1371127490#Place/san_rafael_calif> # San Rafael, Calif.
    a schema:Place ;
   schema:name "San Rafael, Calif." ;
    .

<http://experiment.worldcat.org/entity/work/data/1371127490#Series/synthesis_lectures_on_human_language_technologies> # Synthesis lectures on human language technologies ;
    a bgn:PublicationSeries ;
   schema:hasPart <http://www.worldcat.org/oclc/847941536> ; # Web corpus construction
   schema:name "Synthesis lectures on human language technologies ;" ;
    .

<http://id.loc.gov/authorities/subjects/sh85077224> # Computational linguistics
    a schema:Intangible ;
   schema:name "Computational linguistics"@en ;
    .

<http://id.loc.gov/authorities/subjects/sh97007463> # Web search engines
    a schema:Intangible ;
   schema:name "Web search engines"@en ;
    .

<http://id.worldcat.org/fast/1173234> # Web search engines
    a schema:Intangible ;
   schema:name "Web search engines"@en ;
    .

<http://id.worldcat.org/fast/871998> # Computational linguistics
    a schema:Intangible ;
   schema:name "Computational linguistics"@en ;
    .

<http://viaf.org/viaf/131713828> # Morgan & Claypool Publishers.
    a schema:Organization ;
   schema:name "Morgan & Claypool Publishers." ;
    .

<http://worldcat.org/isbn/9781608459834>
    a schema:ProductModel ;
   schema:isbn "1608459837" ;
   schema:isbn "9781608459834" ;
    .

<http://worldcat.org/issn/1947-4059> # Synthesis lectures on human language technologies,
    a bgn:PublicationSeries ;
   schema:hasPart <http://www.worldcat.org/oclc/847941536> ; # Web corpus construction
   schema:issn "1947-4059" ;
   schema:name "Synthesis lectures on human language technologies," ;
    .


Content-negotiable representations

Close Window

Please sign in to WorldCat 

Don't have an account? You can easily create a free account.