skip to content
Differential effect of correct name translation on human and automated judgments of translation acceptability : a pilot study Preview this item
ClosePreview this item
Checking...

Differential effect of correct name translation on human and automated judgments of translation acceptability : a pilot study

Author: Michelle Vanni; James Walrath; U.S. Army Research Laboratory.
Publisher: Adelphi, MD : Army Research Laboratory, [2008]
Edition/Format:   eBook : Document : EnglishView all editions and formats
Summary:
This study proffers two important findings: (1) automated machine translation (MT) evaluation is insensitive to the cognitive gravitas of proper names, contributing to its weak modeling of human judgments of higher quality MT output, and (2) there is a "new" methodology that produces superior measurement of translation acceptability. Twenty Arabic sentences, each with average name density of 3.7 names in 22 words,  Read more...
Rating:

(not yet rated) 0 with reviews - Be the first.

Subjects
More like this

 

Find a copy online

Find a copy in the library

&AllPage.SpinnerRetrieving; Finding libraries that hold this item...

Details

Additional Physical Format: Print version:
Vanni, Michelle.
Differential effect of correct name translation on human and automated judgments of translation acceptability
iv, 22 p.
(OCoLC)472225135
Material Type: Document, Internet resource
Document Type: Internet Resource, Computer File
All Authors / Contributors: Michelle Vanni; James Walrath; U.S. Army Research Laboratory.
OCLC Number: 318693074
Notes: Title from PDF title screen (viewed on Dec. 2, 2009).
"September 2008."
Technical rept.
Description: 1 online resource (iv, 22 pages)
Responsibility: Michelle Vanni and James Walrath.

Abstract:

This study proffers two important findings: (1) automated machine translation (MT) evaluation is insensitive to the cognitive gravitas of proper names, contributing to its weak modeling of human judgments of higher quality MT output, and (2) there is a "new" methodology that produces superior measurement of translation acceptability. Twenty Arabic sentences, each with average name density of 3.7 names in 22 words, were translated into English with a research-grade MT system, to produce a 20-output-sentence Control Stimulus Set. Manual correction of 25% of the name translations resulted in an Enhanced Stimulus Set. A Magnitude Estimation (ME) methodology task had each of two teams of five subjects judge Control and Enhanced Sets against human reference translations. As is customary in ME studies, subjects made direct numerical estimations of the magnitude of the stimuli, in this case the degree to which sentences in the Sets conveyed the meaning in the reference sentences. Average estimates for Control and Enhanced Sets were 4.57 and 6.16, respectively, a 34.8% difference. Automated evaluation with the Metric for Evaluation of Translation with Explicit word ORdering (METEOR) produced scores of .446 and .546, a 22% difference. ME detected a differential effect, a finding which suggests that weighting proper name rendering in automated evaluation systems may improve correlations with human judgments on higher quality output.

Reviews

User-contributed reviews
Retrieving GoodReads reviews...
Retrieving DOGObooks reviews...

Tags

Be the first.
Confirm this request

You may have already requested this item. Please select Ok if you would like to proceed with this request anyway.

Linked Data


Primary Entity

<http://www.worldcat.org/oclc/318693074> # Differential effect of correct name translation on human and automated judgments of translation acceptability : a pilot study
    a schema:MediaObject, schema:CreativeWork, schema:Book ;
   library:oclcnum "318693074" ;
   library:placeOfPublication <http://experiment.worldcat.org/entity/work/data/504945386#Place/adelphi_md> ; # Adelphi, MD
   library:placeOfPublication <http://id.loc.gov/vocabulary/countries/mdu> ;
   schema:about <http://experiment.worldcat.org/entity/work/data/504945386#Thing/automated_evaluation> ; # AUTOMATED EVALUATION
   schema:about <http://id.loc.gov/authorities/subjects/sh85006311> ; # Arabic language--Translating
   schema:about <http://experiment.worldcat.org/entity/work/data/504945386#Topic/mt_evaluation> ; # MT evaluation
   schema:about <http://experiment.worldcat.org/entity/work/data/504945386#Topic/linguistics> ; # Linguistics
   schema:about <http://experiment.worldcat.org/entity/work/data/504945386#Topic/stimuli> ; # Stimuli
   schema:about <http://experiment.worldcat.org/entity/work/data/504945386#Thing/name_translation> ; # NAME TRANSLATION
   schema:about <http://experiment.worldcat.org/entity/work/data/504945386#Topic/arabic_language> ; # Arabic language
   schema:about <http://experiment.worldcat.org/entity/work/data/504945386#Topic/machine_translation> ; # Machine translation
   schema:about <http://experiment.worldcat.org/entity/work/data/504945386#Topic/humans> ; # Humans
   schema:about <http://id.worldcat.org/fast/812447> ; # Arabic language--Translating
   schema:about <http://experiment.worldcat.org/entity/work/data/504945386#Topic/judgement_psychology> ; # Judgement(psychology)
   schema:about <http://experiment.worldcat.org/entity/work/data/504945386#Topic/comprehension> ; # Comprehension
   schema:about <http://experiment.worldcat.org/entity/work/data/504945386#Thing/machine_translation_evaluation> ; # MACHINE TRANSLATION EVALUATION
   schema:about <http://experiment.worldcat.org/entity/work/data/504945386#Topic/foreign_languages> ; # Foreign languages
   schema:about <http://experiment.worldcat.org/entity/work/data/504945386#Thing/proper_names> ; # PROPER NAMES
   schema:about <http://experiment.worldcat.org/entity/work/data/504945386#Thing/human_understanding> ; # HUMAN UNDERSTANDING
   schema:about <http://experiment.worldcat.org/entity/work/data/504945386#Topic/metrics> ; # Metrics
   schema:about <http://experiment.worldcat.org/entity/work/data/504945386#Topic/name_translation> ; # Name translation
   schema:about <http://experiment.worldcat.org/entity/work/data/504945386#Topic/estimates> ; # Estimates
   schema:about <http://experiment.worldcat.org/entity/work/data/504945386#Thing/magnitude_estimation> ; # MAGNITUDE ESTIMATION
   schema:about <http://experiment.worldcat.org/entity/work/data/504945386#Topic/magnitude_estimation> ; # Magnitude estimation
   schema:about <http://id.worldcat.org/fast/1154816> ; # Translating machines
   schema:bookFormat schema:EBook ;
   schema:contributor <http://experiment.worldcat.org/entity/work/data/504945386#Person/walrath_james> ; # James Walrath
   schema:contributor <http://viaf.org/viaf/149860432> ; # U.S. Army Research Laboratory.
   schema:creator <http://experiment.worldcat.org/entity/work/data/504945386#Person/vanni_michelle> ; # Michelle Vanni
   schema:datePublished "2008" ;
   schema:description "This study proffers two important findings: (1) automated machine translation (MT) evaluation is insensitive to the cognitive gravitas of proper names, contributing to its weak modeling of human judgments of higher quality MT output, and (2) there is a "new" methodology that produces superior measurement of translation acceptability. Twenty Arabic sentences, each with average name density of 3.7 names in 22 words, were translated into English with a research-grade MT system, to produce a 20-output-sentence Control Stimulus Set. Manual correction of 25% of the name translations resulted in an Enhanced Stimulus Set. A Magnitude Estimation (ME) methodology task had each of two teams of five subjects judge Control and Enhanced Sets against human reference translations. As is customary in ME studies, subjects made direct numerical estimations of the magnitude of the stimuli, in this case the degree to which sentences in the Sets conveyed the meaning in the reference sentences. Average estimates for Control and Enhanced Sets were 4.57 and 6.16, respectively, a 34.8% difference. Automated evaluation with the Metric for Evaluation of Translation with Explicit word ORdering (METEOR) produced scores of .446 and .546, a 22% difference. ME detected a differential effect, a finding which suggests that weighting proper name rendering in automated evaluation systems may improve correlations with human judgments on higher quality output."@en ;
   schema:exampleOfWork <http://worldcat.org/entity/work/id/504945386> ;
   schema:inLanguage "en" ;
   schema:isSimilarTo <http://www.worldcat.org/oclc/472225135> ;
   schema:name "Differential effect of correct name translation on human and automated judgments of translation acceptability : a pilot study"@en ;
   schema:productID "318693074" ;
   schema:publication <http://www.worldcat.org/title/-/oclc/318693074#PublicationEvent/adelphi_md_army_research_laboratory_2008> ;
   schema:publisher <http://experiment.worldcat.org/entity/work/data/504945386#Agent/army_research_laboratory> ; # Army Research Laboratory
   schema:url <http://handle.dtic.mil/100.2/ADA488000> ;
   schema:url <http://purl.access.gpo.gov/GPO/LPS117123> ;
   wdrs:describedby <http://www.worldcat.org/title/-/oclc/318693074> ;
    .


Related Entities

<http://experiment.worldcat.org/entity/work/data/504945386#Agent/army_research_laboratory> # Army Research Laboratory
    a bgn:Agent ;
   schema:name "Army Research Laboratory" ;
    .

<http://experiment.worldcat.org/entity/work/data/504945386#Person/vanni_michelle> # Michelle Vanni
    a schema:Person ;
   schema:familyName "Vanni" ;
   schema:givenName "Michelle" ;
   schema:name "Michelle Vanni" ;
    .

<http://experiment.worldcat.org/entity/work/data/504945386#Person/walrath_james> # James Walrath
    a schema:Person ;
   schema:familyName "Walrath" ;
   schema:givenName "James" ;
   schema:name "James Walrath" ;
    .

<http://experiment.worldcat.org/entity/work/data/504945386#Thing/automated_evaluation> # AUTOMATED EVALUATION
    a schema:Thing ;
   schema:name "AUTOMATED EVALUATION" ;
    .

<http://experiment.worldcat.org/entity/work/data/504945386#Thing/human_understanding> # HUMAN UNDERSTANDING
    a schema:Thing ;
   schema:name "HUMAN UNDERSTANDING" ;
    .

<http://experiment.worldcat.org/entity/work/data/504945386#Thing/machine_translation_evaluation> # MACHINE TRANSLATION EVALUATION
    a schema:Thing ;
   schema:name "MACHINE TRANSLATION EVALUATION" ;
    .

<http://experiment.worldcat.org/entity/work/data/504945386#Thing/magnitude_estimation> # MAGNITUDE ESTIMATION
    a schema:Thing ;
   schema:name "MAGNITUDE ESTIMATION" ;
    .

<http://experiment.worldcat.org/entity/work/data/504945386#Thing/name_translation> # NAME TRANSLATION
    a schema:Thing ;
   schema:name "NAME TRANSLATION" ;
    .

<http://experiment.worldcat.org/entity/work/data/504945386#Topic/judgement_psychology> # Judgement(psychology)
    a schema:Intangible ;
   schema:name "Judgement(psychology)"@en ;
    .

<http://experiment.worldcat.org/entity/work/data/504945386#Topic/machine_translation> # Machine translation
    a schema:Intangible ;
   schema:name "Machine translation"@en ;
    .

<http://experiment.worldcat.org/entity/work/data/504945386#Topic/magnitude_estimation> # Magnitude estimation
    a schema:Intangible ;
   schema:name "Magnitude estimation"@en ;
    .

<http://id.loc.gov/authorities/subjects/sh85006311> # Arabic language--Translating
    a schema:Intangible ;
   schema:name "Arabic language--Translating"@en ;
    .

<http://id.worldcat.org/fast/1154816> # Translating machines
    a schema:Intangible ;
   schema:name "Translating machines"@en ;
    .

<http://id.worldcat.org/fast/812447> # Arabic language--Translating
    a schema:Intangible ;
   schema:name "Arabic language--Translating"@en ;
    .

<http://viaf.org/viaf/149860432> # U.S. Army Research Laboratory.
    a schema:Organization ;
   schema:name "U.S. Army Research Laboratory." ;
    .

<http://www.worldcat.org/oclc/472225135>
    a schema:CreativeWork ;
   rdfs:label "Differential effect of correct name translation on human and automated judgments of translation acceptability" ;
   schema:description "Print version:" ;
   schema:isSimilarTo <http://www.worldcat.org/oclc/318693074> ; # Differential effect of correct name translation on human and automated judgments of translation acceptability : a pilot study
    .

<http://www.worldcat.org/title/-/oclc/318693074>
    a genont:InformationResource, genont:ContentTypeGenericResource ;
   schema:about <http://www.worldcat.org/oclc/318693074> ; # Differential effect of correct name translation on human and automated judgments of translation acceptability : a pilot study
   schema:dateModified "2017-12-25" ;
   void:inDataset <http://purl.oclc.org/dataset/WorldCat> ;
    .


Content-negotiable representations

Close Window

Please sign in to WorldCat 

Don't have an account? You can easily create a free account.