Webmaster's Note: The whole dataset is available Here. Please download the dataset instead of crawling the website.
Basic Info:
id: J93-2004
title: Building A Large Annotated Corpus Of English: The Penn Treebank
venue: CL
year: 1993
pdf: link
title: Building A Large Annotated Corpus Of English: The Penn Treebank
venue: CL
year: 1993
pdf: link
Abstract
Dow Jones Newswire stories Dept. of Agriculture bulletins Library of America texts MUC-3 messages IBM Manual sentences WBUR radio transcripts ATIS sentences Brown Corpus, retagged 231,404 231,404 3,065,776 1,061,166 78,555 78,555 105,652 105,652 111,828 111,828 89,121 89,121 11,589 11,589 19,832 19,832 1,172,041 1,172,041 Total: 4,885,798 2,881,188 Some comments on the materials included: • Department of Energy abstracts are scientific abstracts from a variety of disciplines. • All of the skeletally parsed Dow Jones Newswire materials are also available as digitally recorded read speech as part of the DARPA WSJ-CSR1 corpus, available through the Linguistic Data Consortium. • The Department of Agriculture materials include short bulletins on such topics as when to plant various flowers and how to can various vegetables and fruits. • The Library of America texts are 5,000-10,000 word passages, mainly book chapters, from a variety of American authors including Mark Twain, Henry Adams, Willa Cather, Herman Melville, W. E. B. Dubois, and Ralph Waldo Emerson. • The MUC-3 texts are all news stories from the Federal News Service about terrorist activities in South America. Some of these texts are translations of Spanish news stories or transcripts of radio broadcasts. They are taken from training materials for the Third Message Understanding Conference. • The Brown Corpus materials were completely retagged by the Penn Treebank project starting from the untagged version of the Brown Corpus (Francis 1964). • The IBM sentences are taken from IBM computer manuals; they are chosen to contain a vocabulary of 3,000 words, and are limited in length. • The ATIS sentences are transcribed versions of spontaneous sentences collected as training materials for the DARPA Air Travel Information System project. The entire corpus has been tagged for POS information, at an estimated error rate 327 Computational Linguistics Volume 19, Number 2 of approximately 3%. The POS-tagged version of the Library of America texts and the Department of Agriculture bulletins have been corrected twice (each by a different annotator), "and the corrected files were then carefully adjudicated; we estimate the error rate of the adjudicated version at well under 1%. Using a version of PARTS retrained on the entire preliminary corpus and adjudicating between the output of the retrained version and the preliminary version of the corpus, we plan to reduce the error rate of the final version of the corpus to approximately 1%. All the skeletally parsed materials have been corrected once, except for the Brown materials, which have been quickly proofread an additional time for gross parsing errors. 5.2 Future Directions A large number of research efforts, both at the University of Pennsylvania and else- where, have relied on the output of the Penn Treebank Project to date. A few examples already in print: a number of projects investigating stochastic parsing have used either the POS-tagged materials (Magerman and Marcus 1990; Brill et al. 1990; Brill 1991) or the skeletally parsed corpus (Weischedel et al. 1991; Pereira and Schabes 1992). The POS-tagged corpus has also been used to train a number of different POS taggers in- cluding Meteer, Schwartz, and Weischedel (1991), and the skeletally parsed corpus has been used in connection with the development of new methods to exploit intonational cues in disambiguating the parsing of spoken sentences (Veilleux and Ostendorf 1992). The Penn Treebank has been used to bootstrap the development of lexicons for particu- lar applications (Robert Ingria, personal communication) and is being used as a source of examples for linguistic theory and psychological modelling (e.g. Niv 1991). To aid in the search for specific examples of grammatical phenomena using the Treebank, Richard Pito has developed tgrep, a tool for very fast context-free pattern matching against the skeletally parsed corpus, which is available through the Linguistic Data Consortium. While the Treebank is being widely used, the annotation scheme employed has a variety of limitations. Many otherwise clear argument/adjunct relations in the corpus are not indicated because of the current Treebank's essentially context-free represen- tation. For example, there is at present no satisfactory representation for sentences in which complement noun phrases or clauses occur after a sentential level adverb. Either the adverb is trapped within the VP, so that the complement can occur within the VP where it belongs, or else the adverb is attached to the S, closing off the VP and forcing the complement to attach to the S. This "trapping" problem serves as a limitation for groups that currently use Treebank material semiautomatically to derive lexicons for particular applications. For most of these problems, however, solutions are possible on the basis of mechanisms already used by the Treebank Project. For example, the pseudo-attachment notation can be extended to indicate a variety of crossing depen- dencies. We have recently begun to use this mechanism to represent various kinds of dislocations, and the Treebank annotators themselves have developed a detailed proposal to extend pseudo-attachment to a wide range of similar phenomena. A variety of inconsistencies in the annotation scheme used within the Treebank have also become apparent with time. The annotation schemes for some syntactic categories should be unified to allow a consistent approach to determining predicate- argument structure. To take a very simple example, sentential adverbs attach under VP when they occur between auxiliaries and predicative ADJPs, but attach under S when they occur between auxiliaries and VPs. These structures need to be regularized. As the current Treebank has been exploited by a variety of users, a significant number have expressed a need for forms of annotation richer than provided by the project's first phase. Some users would like a less skeletal form of annotation of surface 328 Mitchell P. Marcus et al. Building a Large Annotated Corpus of English grammatical structure, expanding the essentially context-free analysis of the current Penn Treebank to indicate a wide variety of noncontiguous structures and dependen- cies. A wide range of Treebank users now strongly desire a level of annotation that makes explicit some form of predicate-argument structure. The desired level of rep- resentation would make explicit the logical subject and logical object of the verb, and would indicate, at least in clear cases, which subconstituents serve as arguments of the underlying predicates and which serve as modifiers. During the next phase of the Treebank project, we expect to provide both a richer analysis of the existing corpus and a parallel corpus of predicate-argument structures. This will be done by first enriching the annotation of the current corpus, and then automatically extracting predicate-argument structure, at the level of distinguishing logical subjects and objects, and distinguishing arguments from adjuncts for clear cases. Enrichment will be achieved by automatically transforming the current Penn Treebank into a level of structure close to the intended target, and then completing the conversion by hand. Acknowledgments The work reported here was partially supported by DARPA grant No. N0014-85-K0018, by DARPA and AFOSR jointly under grant No. AFOSR-90-0066 and by ARO grant No. DAAL 03-89-C0031 PRI. Seed money was provided by the General Electric Corporation under grant No. J01746000. We gratefully acknowledge this support. We would also like to acknowledge the contribution of the annotators who have worked on the Penn Treebank Project: Florence Dong, Leslie Dossey, Mark Ferguson, Lisa Frank, Elizabeth Hamilton, Alissa Hinckley, Chris Hudson, Karen Katz, Grace Kim, Robert MacIntyre, Mark Parisi, Britta Schasberger, Victoria Tredinnick and Matt Waters; in addition, Rob Foye, David Magerman, Richard Pito and Steven Shapiro deserve our special thanks for their administrative and programming support. We are grateful to AT&T Bell Labs for permission to use Kenneth Church's PARTS part-of-speech labeler and Donald Hindle's Fidditch parser. Finally, we would like to thank Sue Marcus for sharing with us her statistical expertise and providing the analysis of the time data of the experiment reported in Section 3. The design of that experiment is due to the first two authors; they alone are responsible for its shortcomings. References Brill, Eric (1991). "Discovering the lexical features of a language." In Proceedings, 29th Annual Meeting of the Association for Computational Linguistics. Berkeley CA. Brill, Eric; Magerman, David; Marcus, Mitchell P.; and Santorini, Beatrice (1990). "Deducing linguistic structure from the statistics of large corpora." In Proceedings, DARPA Speech and Natural Language Workshop. June 1990, 275-282. Church, Kenneth W. (1980). Memory limitations in natural language processing. Master's dissertation, Massachusetts Institute of Technology, Cambridge MA. Church, Kenneth W. (1988). "A stochastic parts program and noun phrase parser for unrestricted text." In Proceedings, Second Conference on Applied Natural Language Processing. 136--143. Francis, W. Nelson (1964). "A standard sample of present-day English for use with digital computers." Report to the U.S Office of Education on Cooperative Research Project No. E-007. Brown University, Providence RI. Francis, W. Nelson, and Ku~era, Henry (1982). Frequency Analysis of English Usage: Lexicon and Grammar. Houghton Mifflin. Garside, Roger; Leech, Geoffrey; and Sampson, Geoffrey (1987). The Computational Analysis of English: A Corpus-Based Approach. Longman. Hindle, Donald (1983). "User manual for Fidditch." Technical memorandum 7590-142, Naval Research Laboratory. Hindle, Donald (1989). "Acquiring disambiguation rules from text." In Proceedings, 27th Annual Meeting of the Association for Computational Linguistics. Lewis, Bil; LaLiberte, Dan; and the GNU Manual Group (1990). The GNU Emacs Lisp reference manual. Free Software Foundation, Cambridge, MA. Magerman, David, and Marcus, Mitchell P. (1990). "Parsing a natural language using 329 Computational Linguistics Volume 19, Number 2 mutual information statistics." In Proceedings of AAAI-90. Meteer, Marie; Schwartz, Richard; and Weischedel, Ralph (1991). "Studies in part of speech labelling." In Proceedings, Fourth DARPA Speech and Natural Language Workshop. February 1991. Niv, Michael (1991). "Syntactic disambiguation." In The Penn Review of Linguistics, 14, 120-126. Pereira, Fernando, and Schabes, Yves (1992). "Inside-outside reestimation from partially bracketed corpora." In Proceedings, 30th Annual Meeting of the Association for Computational Linguistics. Santorini, Beatrice (1990). "Part-of-speech tagging guidelines for the Penn Treebank Project." Technical report MS-CIS-90-47, Department of Computer and Information Science, University of Pennsylvania. Santorini, Beatrice, and Marcinkiewicz, Mary Ann (1991). "Bracketing guidelines for the Penn Treebank Project." Unpublished manuscript, Department of Computer and Information Science, University of Pennsylvania. Veilleux, N. M., and Ostendorf, Mari (1992). "Probabilistic parse scoring based on prosodic features." In Proceedings, Fifth DARPA Speech and Natural Language Workshop. February 1992. Weischedel, Ralph; Ayuso, Damaris; Bobrow, R.; Boisen, Sean; Ingria, Robert; and Palmucci, Jeff (1991). "Partial parsing: a report of work in progress." In Proceedings, Fourth DARPA Speech and Natural Language Workshop. February 1991. 330
| Stat | Rank | Value |
|---|---|---|
| Incoming Citations | 1(1) | 593(590) |
| Outgoing Citations | 1854(1773) | 8(7) |
| PageRank | 7 | 6212 |
| PageRank per Year | 5 | 414.13 |
| By Title |
|---|
| By Abstract |
|---|
| By Full Text |
|---|
| By Co-citation |
|---|
| Citing sentences |
|---|
| W05-0404 1 82:165 In our framework, we employ a simple HMM-based tagger, where the most probable tag sequence, a29a30, given the words, a31, is output (Weischedel et al. , 1993): a29 a30 a20a22a32a34a33a36a35a38a37a39a32a41a40 a42 a43a45a44 a30a47a46 a31a49a48a17a20a22a32a34a33a50a35a38a37a39a32a41a40 a42 a43a45a44 a31 a46a30 a48 a43a51a44 a30 a48 Since we do not have enough data which is manually tagged with part-of-speech tags for our applications, we used Penn Treebank (Marcus et al. , 1994) as our training set. |
| P06-1063 2 8:174 Large treebanks are available for major languages, however these are often based on a speci c text type or genre, e.g. nancial newspaper text (the Penn-II Treebank (Marcus et al. , 1993)). |
| W04-1602 3 13:163 (Marcus, et al. , 1993), (Marcus, et al. , 1994) In addition to the usual issues involved with the complex annotation of data, we have come to terms with a number of issues that are specific to a highly inflected language with a rich history of traditional grammar. |
| C96-1003 4 136:199 to estimale a model (clustering words), and measured the I(L distancd ~ between 'l'he K\], distance (relative Clt|,l:Opy), which is widely used in information theory and sta, tist, ics, is a, nleasur,2 of 'dista, n<:c' l>~\[,wcen two distributions 5.2 Experiment 2: Qualitative Evaluation We extracted roughly 180,000 case fl:anles from the bracketed WSJ (Wall Street Journal) corpus of the Penn Tree Bank (Marcus et al. , 1993) as co-occurrence data. |
| C96-1003 5 7:199 have been proposed (Hindle, 1990; Brown et al. , 1992; Pereira et al. , 1993; Tokunaga et al. , 1995). |
| C96-1003 6 169:199 In particular, we used this method with WordNet (Miller et al. , 1993) and using the same training data. |
| W01-0908 7 51:179 4.1 Data We used Penn-Treebank (Marcus et al. , 1993) data, presented in Table 1. |
| W99-0621 8 140:205 These data sets were based on the Wall Street Journal corpus in the Penn Treebank (Marcus et al. , 1993). |
| L08-1018 9 19:77 264-285. |
| L08-1018 10 13:77 http://duc.nist.gov |
| L08-1018 11 2:77 Statistics in linguistics, Oxford.: Basil Blackwell. |
| L08-1018 12 74:77 In acknowledgment of this fact, a series of conferences like Text Retrieval Conferences (TREC) (Voorhees and Harman, 1999), Message Understanding Conferences (MUC) (Chinchor et al, 1993), TIPSTER SUMMAC Text Summarization Evaluation (Mani et al, 1998), Document Understanding Conference (DUC) (DUC, 2004), and Text Summar |
| W04-2003 13 29:197 Although grammatical function and empty nodes annotation expressing long-distance dependencies are provided in Treebanks such as the Penn Treebank (Marcus et al. , 1993), most statistical Treebank trained parsers fully or largely ignore them 1, which entails two problems: first, the training cannot profit from valuable annotation data. |
| W06-1601 14 132:193 5 Datasets and Evaluation We train our models with verb instances extracted from three parsed corpora: (1) the Wall Street Journal section of the Penn Treebank (PTB), which was parsed by human annotators (Marcus et al. , 1993), (2) the Brown Laboratory for Linguistic Information Processing corpus of Wall Street Journal text (BLLIP), which was parsed automatically by the Charniak parser (Charniak, 2000), and (3) the Gigaword corpus of raw newswire text (GW), which we parsed ourselves with the Stanford parser. |
| P03-1013 15 8:266 However, most of the existing models have been developed for English and trained on the Penn Treebank (Marcus et al. , 1993), which raises the question whether these models generalize to other languages, and to annotation schemes that differ from the Penn Treebank markup. |
| P03-1013 16 43:266 The annotation scheme (Skut et al. , 1997) is modeled to a certain extent on that of the Penn Treebank (Marcus et al. , 1993), with crucial differences. |
| W03-1903 17 8:157 Part-of-Speech (POS) annotation for example can be seen as the task of choosing the appropriate tag for a word from an ontology of word categories (compare for example the Penn Treebank POS tagset as described in (Marcus et al. , 1993)). |
| W03-1903 18 14:157 Ontologies are formal specifications of a conceptualization (Gruber, 1993) so that it seems straightforward to formalize annotation schemes as ontologies and make use of semantic annotation tools such as OntoMat (Handschuh et al. , 2001) for the purpose of linguistic annotation. |
| W07-1001 19 106:186 The CDR (Morris, 1993) is assigned with access to clinical and cognitive test information, independent of performance on the battery of neuropsychological tests used for this research study, and has been shown to have high expert inter-annotator reliability (Morris et al. , 1997). |
| W07-1001 20 13:186 Narrative retellings provide a natural, conversational speech sample that can be analyzed for many of the characteristics of speech and language that have been shown to discriminate between healthy and impaired subjects, including syntactic complexity (Kemper et al. , 1993; Lyons et al. , 1994) and mean pause duration (Singh et al. , 2001). |
| P08-1109 21 102:173 5 Experiments For all experiments, we trained and tested on the Penn treebank (PTB) (Marcus et al., 1993). |
| N06-2033 22 5:93 Much of this work has been fueled by the availability of large corpora annotated with syntactic structures, especially the Penn Treebank (Marcus et al. , 1993). |
| C04-1197 23 181:216 The training set is extracted from TreeBank (Marcus et al. , 1993) section 1518, the development set, used in tuning parameters of the system, from section 20, and the test set from section 21. |
| P06-2009 24 183:229 4 Experiments and Results We use the standard corpus for this task, the Penn Treebank (Marcus et al. , 1993). |
| P06-2009 25 80:229 Policy #Shift #Left #Right Start over 156545 26351 27918 Stay 117819 26351 27918 Step back 43374 26351 27918 Table 1: The number of actions required to build all the trees for the sentences in section 23 of Penn Treebank (Marcus et al. , 1993) as a function of the focus point placement policy. |
| W99-0104 26 196:264 We use the finite-state parses of FaSTU$ (Appelt et al. , 1993) for recognizing these entities, but the method extends to any basic phrasal parser 4. |
| W99-0104 27 52:264 The first one makes use of the advances in the parsing technology or on the availability of large parsed corpora (e.g. Trcebank (Marcus et al.1993)) to produce algorithms inspired by Hobbs' baseline method (Hobbs, 1978). |
| W99-0104 28 251:264 This knowledge is represented in axiomatic form, using the notation proposed in (Hobbs et al. , 1993) and previously implemented in TACITUS. |
| W99-0104 29 48:264 Such methods were presented in (Hoblm et al. , 1993) and ~flensky, 1978). |
| D07-1126 30 116:136 The pchemtb-closed shared task (Marcus et al. , 1993; Johansson and Nugues, 2007; Kulick et al. , 2004) is used to illustrate our models. |
| D07-1126 31 82:136 3 Experimental Results and Discussion We test our parsing models on the CONLL-2007 (Hajic et al. , 2004; Aduriz et al. , 2003; Mart et al. , 2007; Chen et al. , 2003; Bohmova et al. , 2003; Marcus et al. , 1993; Johansson and Nugues, 2007; Prokopidis et al. , 2005; Csendes et al. , 2005; Montemagni et al. , 2003; Oflazer et al. , 2003) data set on various languages including Arabic, Basque, Catalan, Chinese, English, Italian, Hungarian, and Turkish. |
| W05-1510 32 23:201 We evaluated the generator on the Penn Treebank (Marcus et al. , 1993), which is highly reliable corpus consisting of real-world texts. |
| W98-1121 33 46:230 Dialogs Speakers Turns Words Fragments Distinct Words Distinct Words/POS Singleton Words Singleton Words/POS Intonational Phrases Speech Repairs 98 34 6163 58298 756 859 1101 252 350 10947 2396 Table 1: Size of the Trains Corpus 2.1 POS Annotations Our POS tagset is based on the Penn Treebank tagset (Marcus et al. , 1993), but modified to include tags for discourse markers and end-of-turns, and to provide richer syntactic information (Heeman, 1997). |
| W98-1121 34 80:230 (Charniak et al. , 1993)) simplify these probability distributions, as given in Equations 9 and 10. |
| D08-1093 35 11:194 The other recipe that is currently used on a large scale is to measure the performance of a parser on existing treebanks, such as WSJ (Marcus et al., 1993), and assume that the accuracy measure will carry over to the domains of interest. |
| C02-1138 36 74:196 One kind is the Penn Treebank (Marcus et al. , 1993). |
| W95-0104 37 56:297 The performance figures given below are based on training each method on the 1-million-word Brown corpus \[Ku~:era and Francis, 1967\] and testing it on a 3/4-million-word corpus of Wall Street Journal text \[Marcus et al. , 1993\]. |
| W99-0701 38 57:94 Experiments We have conducted a series of lexical acquisition experiments with the above algorithm on largescale English corpora, e.g., the Brown corpus \[Francis and Kucera 1982\] and the PTB WSJ corpus \[Marcus et al. 1993\]. |
| J94-4005 39 32:230 Lexical collocation functions, especially those determined statistically, have recently attracted considerable attention in computational linguistics (Calzolari and Bindi 1990; Church and Hanks 1990; Sekine et al. 1992; Hindle and Rooth 1993) mainly, though not exclusively, for use in disambiguation. |
| J94-4005 40 42:230 More specifically, the work on optimizing preference factors and semantic collocations was done as part of a project on spoken language translation in which the CLE was used for analysis and generation of both English and Swedish (AgnSs et al. 1993). |
| I05-6007 41 8:83 Since the texts in the RST Treebank are taken from the syntactically annotated Penn Treebank (Marcus et al. , 1993), it is natural to ask what the relation is between the discourse structures in the RST Treebank and the syntactic structures of the Penn Treebank. |
| W99-0606 42 240:246 Ralph Weischedel et al. 1993. |
| W99-0606 43 84:246 3 Tagging 3.1 Corpus To facilitate comparison with previous results, we used the UPenn Treebank corpus (Marcus et al. , 1993). |
| C00-1034 44 73:229 2'\]'he WSJ corpus (Marcus et al. , 1993). |
| C00-1034 45 7:229 (levelopment of cor1)ora with morl)ho-synta(:ti(: and syntacti(: mmotation (Marcus et al. , 1993), (Sampson, 1995). |
| N06-1054 46 102:249 We use data from the CoNLL-2004 shared taskthe PropBank (Palmer et al. , 2005) annotations of the Penn Treebank (Marcus et al. , 1993), with sections 1518 as the training set and section 20 as the development set. |
| C96-2114 47 119:184 (Marcus et al. 1993, 316). |
| C96-2114 48 32:184 The tagger used is thus one that does not need tagged and disambiguated material to be trained on, namely the XPOST originally constructed at Xerox Parc (Cutting et al. 1992, Cutting and Pedersen 1993). |
| P08-1039 49 8:216 This is because their training data, the Penn Treebank (Marcus et al., 1993), does not fully annotate NP structure. |
| W07-2052 50 44:76 We parsed the TimeEval data using MSTParser v0.2 (McDonald and Pereira, 2006), which is trained with all Penn Treebank (Marcus et al. , 1993) without dependency label. |
| P04-1052 51 183:285 4 Evaluation As our algorithm works in open domains, we were able to perform a corpus-based evaluation using the Penn WSJ Treebank (Marcus et al. , 1993). |
| P04-1052 52 42:285 Also, attribute classi cation is a hard problem and there is no existing classi cation scheme that can be used for open domains like newswire; for example, WordNet (Miller et al. , 1993) organises adjectives as concepts that are related by the non-hierarchical relations of synonymy and antonymy (unlike nouns that are related through hierarchical links such as hyponymy, hypernymy and metonymy). |
| P98-1083 53 181:292 The main reason behind this lies in the difference between the two corpora used: Penn Treebank (Marcus et al. , 1993) and EDR corpus (EDR, 1995). |
| P98-1083 54 182:292 Penn Treebank(Marcus et al. , 1993) was also used to induce part-of-speech (POS) taggers because the corpus contains very precise and detailed POS markers as well as bracket, annotations. |
| D07-1100 55 12:206 We participated in the multilingual track of the CoNLL 2007 shared task (Nivre et al. , 2007), and evaluated the system on data sets of 10 languages (Hajic et al. , 2004; Aduriz et al. , 2003; Mart et al. , 2007; Chen et al. , 2003; Bohmova et al. , 2003; Marcus et al. , 1993; Johansson and Nugues, 2007; Prokopidis et al. , 2005; Csendes et al. , 2005; Montemagni et al. , 2003; Oflazer et al. , 2003). |
| W97-0105 56 11:267 In all other respects, our work departs from previous research on broad--coverage 16 I t I I I I I i ! I i I I I I I I I I I I I i I 1, I. I I I I i I 1 I I I I probabilistic parsing, which either attempts to learn to predict gr~rarn~tical structure of test data directly from a training treebank (Brill, 1993; Collins, 1996; Eisner, 1996; Jelinek et al. , 1994; Magerman, 1995; S~kine and Orishman, 1995; Sharman et al. , 1990), or employs a grammar and sometimes a dictionary to capture linguistic expertise directly (Black et al. , 1993a; GrinBerg et al. , 1995; Schabes; 1992), but arguably at a less detailed and informative level than in the research reported here. |
| W97-0105 57 188:267 Table 3 shows the differences between the treebank~ utilized in (Jelinek et al. , 1994) on the one hand, and in the work reported here, on the other, is Table 4 shows relevant lSFigures for Average Sentence Length ('l~raLuing Corpus) and Training Set Size, for the IBM ManuaLs Corpus, are approximate, and cz~e fzom (Black et aL, 1993a). |
| W97-0105 58 187:267 Clearly the present research task is quite considerably harder than the parsing and tagging tasks undertaken in (Jelinek et al. , 1994; Magerman, 1995; Black et al. , 1993b), which would seem to be the closest work to ours, and any comparison between this work and ours must be approached with extreme caution. |
| W97-0105 59 34:267 For example, the feature 1 On the ATR English Grammar, see below; for a detailed description of a precursor to the Gr-r~raar, see (Black et al. , 1993a). |
| W97-0105 60 106:267 Slrs Parse Base (Black et al. , 1993a) is 1.76. |
| W97-0105 61 26:267 By labelling Treeb~n~ nodes with Gr~ramar rule names, and not with phrasal and clausal n~raes, as in other (non-gr~rarnar-based) treebanks' (Eyes and Leech, 1993; Garside and McEnery, 1993; Marcus et al. , 1993), we gain access to all information provided by the Grammar regarding each ~reebank node. |
| W06-1652 62 101:189 The OP data consists of 2,452 documents from the Penn Treebank (Marcus et al. , 1993). |
| W97-1005 63 31:223 Both data were extracted from the Penn Treebank Wall Street Journal (WSJ) Corpus (Marcus et al. , 1993). |
| W97-1005 64 46:223 Statistical and information theoretic approaches (Hindle and Rooth, 1993), (Ratnaparkhi et al. , 1994),(Collins and Brooks, 1995), (Franz, 1996) Using lexical collocations to determine PPA with statistical techniques was first proposed by (Hindle and Rooth, 1993). |
| W06-1615 65 20:260 There are many choices for modeling co-occurrence data (Brown et al. , 1992; Pereira et al. , 1993; Blei et al. , 2003). |
| W06-1615 66 132:260 5 Data Sets and Supervised Tagger 5.1 Source Domain: WSJ We used sections 02-21 of the Penn Treebank (Marcus et al. , 1993) for training. |
| E99-1031 67 54:89 Our experiments created translation modules for two evaluation corpora: written news stories from the Penn Treebank corpus (Marcus et al. , 1993) and spoken task-oriented dialogues from the TRAINS93 corpus (Heeman and Allen, 1995). |
| D07-1003 68 56:243 Similarly, Murdock and Croft (2005) adopted a simple translation model from IBM model 1 (Brown et al. , 1990; Brown et al. , 1993) and applied it to QA. |
| D07-1003 69 156:243 This sort of problem can be solved in principle by conditional variants of the Expectation-Maximization algorithm (Baum et al. , 1970; Dempster et al. , 1977; Meng and Rubin, 1993; Jebara and Pentland, 1999). |
| D07-1003 70 97:243 The tree is produced by a state-of-the-art dependency parser (McDonald et al. , 2005) trained on the Wall Street Journal Penn Treebank (Marcus et al. , 1993). |
| C02-2024 71 66:153 The data set consisting of 249,994 TFSs was generated by parsing the Figure 3: The size of Dpi; for the size of the data set 800 bracketed sentences in the Wall Street Journal corpus (the first 800 sentences in Wall Street Journal 00) in the Penn Treebank (Marcus et al. , 1993) with the XHPSG grammar (Tateisi et al. , 1998). |
| W06-3604 72 37:152 As the third test set we selected all tokens of the Brown corpus part of the Penn Treebank (Marcus et al. , 1993), a selected portion of the original one-million word Brown corpus (Kucera and Francis, 1967), a collection of samples of American English in many different genres, from sources printed in 1961; we refer to this test set as BROWN. |
| X98-1014 73 37:315 Training Data Our source for syntactically annotated training data was the Penn Treebank (Marcus et al. , 1993). |
| W04-2708 74 10:184 Since Czech is a language with relatively high degree of word-order freedom, and its sentences contain certain syntactic phenomena, such as discontinuous constituents (non-projective constructions), which cannot be straightforwardly handled using the annotation scheme of Penn Treebank (Marcus et al. , 1993; Linguistic Data Consortium, 1999), based on phrase-structure trees, we decided to adopt for the PCEDT the dependency-based annotation scheme of the Prague Dependency Treebank PDT (Linguistic Data Consortium, 2001). |
| A00-1025 75 125:233 The approach is able to achieve 94% precision and recall for base NPs derived from the Penn Treebank Wall Street Journal (Marcus et al. , 1993). |
| W07-1530 76 85:119 It has been difficult to identify all and only those cases where a token functions as a discourse connective, and in many cases, the syntactic analysis in the Penn TreeBank (Marcus et al. , 1993) provides no help. |
| W07-1530 77 74:119 6 Penn Discourse Treebank (Bonnie Webber, Edinburgh) The Penn Discourse TreeBank (Miltsakaki et al. , 2004; Prasad et al. , 2004; Webber, 2005) annotates discourse relations over the Wall Street Journal corpus (Marcus et al. , 1993), in terms of discourse connectives and their arguments. |
| D07-1124 78 76:98 5 Results and Discussion The system with online learning and Nivres parsing algorithm was trained on the data released by CoNLL Shared Task Organizers for all the ten languages (Hajic et al. , 2004; Aduriz et al. , 2003; Mart et al. , 2007; Chen et al. , 2003; Bohmova et al. , 2003; Marcus et al. , 1993; Johansson and Nugues, 2007; Prokopidis et al. , 2005; Csendes et al. , 2005; Montemagni et al. , 2003; Oflazer et al. , 2003). |
| W02-1507 79 35:133 For instance (Chiang, 2000), (Xia, 2001) (Chen, 2001) all automatically acquire large TAGs for English from the Penn Treebank (Marcus et al. , 1993). |
| W04-0508 80 118:182 Although LDD annotation is actually provided in Treebanks such as the Penn Treebank (Marcus et al. , 1993) over which they are typically trained, most probabilistic parsers largely or fully ignore this information. |
| W99-0704 81 78:165 The WSJNPVP set consists of part-of speech tagged Wall Street Journal material (Marcus, Santorini & Marcinkiewicz, 1993), supplemented with syntactic tags indicating noun phrase and verb phrase boundaries (Daelemans et al, 1999iii). |
| P03-1055 82 27:180 We used the Wall Street Journal (WSJ) part of the Penn Treebank (Marcus et al. , 1993), where extraction is represented by co-indexing an empty terminal element (henceforth EE) to its antecedent. |
| P98-2234 83 128:278 6 Experiments 6.1 Data preparation Our experiments were conducted with data made available through the Penn Treebank annotation effort (Marcus et al. , 1993). |
| P98-2234 84 35:278 By core phrases, we mean the kind of nonrecursive simplifications of the NP and VP that in the literature go by names such as noun/verb groups (Appelt et al. , 1993) or chunks, and base NPs (Ramshaw and Marcus, 1995). |
| W07-2211 85 70:238 Examples of this work include a system by Liu et al (1990), and experiments by Hindle and Rooth (1993), and Resnik and Hearst (1993).2 These efforts had mixed success, suggesting that while multi-level preference scores are problematic, integrating some corpus data does not solve the problems. |
| P01-1010 86 34:160 While this technique has been successfully applied to parsing the ATIS portion in the Penn Treebank (Marcus et al. 1993), it is extremely time consuming. |
| P04-1082 87 1:164 Using linguistic principles to recover empty categories Richard CAMPBELL Microsoft Research One Microsoft Way Redmond, WA 98052 USA richcamp@microsoft.com Abstract This paper describes an algorithm for detecting empty nodes in the Penn Treebank (Marcus et al. , 1993), finding their antecedents, and assigning them function tags, without access to lexical information such as valency. |
| P04-1082 88 7:164 In the Penn Treebank (Marcus et al. , 1993), null elements, or empty categories, are used to indicate non-local dependencies, discontinuous constituents, and certain missing elements. |
| W05-0305 89 7:201 1 Introduction The overall goal of the Penn Discourse Treebank (PDTB) is to annotate the million word WSJ corpus in the Penn TreeBank (Marcus et al. , 1993) with a layer of discourse annotations. |
| W05-1509 90 10:202 State-of-the-art statistical parsers trained on the Penn Treebank (PTB) (Marcus et al. , 1993) proS a8a8 a8a8a8 a72a72 a72a72a72 NP-SBJ a16a16a16 a80a80a80the authority VP a16a16a16 a16a16a16a16 a0 a0a0 a64 a64a64 a80a80a80 a80a80a80a80 VBD dropped PP-TMP a8a8 a72a72IN at NP NN midnight NP-TMP NNP Tuesday PP-DIR a8a8 a72a72TO to NP QP a16a16a16 a80a80a80$ 2.80 trillion Figure 1: A sample syntactic structure with function labels. |
| A00-2020 91 20:157 We evaluate this method over the part of speech tagged portion of the Penn Treebank corpus (Marcus et al. , 1993). |
| P97-1024 92 109:230 3.5 Adding Context to the Model Next, we added of a stochastic POS tagger (Charniak et al. , 1993) to provide a model of context. |
| P97-1024 93 91:230 3.3 Accuracy Results (Weischedel et al. , 1993) describe a model for unknown words that uses four features, but treats the features ms independent. |
| W99-0204 94 28:218 jp/et I/nl/GDA/t agset, html 2http://www.uic.edu:80/orgs/tei/ 25 ing insights from EAGLES s, Penn TreeBank \[Marcus et al. , 1993\], and so forth. |
| P05-3018 95 13:103 a time-consuming process (Litman and Pan, 2002; Marcus et al. , 1993; Xia et al. , 2000; Wiebe, 2002). |
| W06-1608 96 21:168 The parser is trained on dependencies extracted from the English Penn Treebank version 3.0 (Marcus et al. , 1993) by using the head-percolation rules of (Yamada and Matsumoto, 2003). |
| W06-1666 97 65:278 5 Experimental Evaluation To perform empirical evaluations of the proposed methods, we considered the task of parsing the Penn Treebank Wall Street Journal corpus (Marcus et al. , 1993). |
| P01-1044 98 9:205 We used treebank grammars induced directly from the local trees of the entire WSJ section of the Penn Treebank (Marcus et al. , 1993) (release 3). |
| W04-0214 99 15:155 Since parsing is just an initial stage of natural language understanding, the project was focused not just on obtaining syntactic trees alone (as is done in many other parsed corpora, for example, Penn TreeBank (Marcus et al. , 1993) or Tiger (Brants and Plaehn, 2000)). |
| P98-2201 100 96:161 21418 examples of structures of the kind 'VB N1 PREP N2' were extracted from the Penn-TreeBank Wall Street Journal (Marcus et al. 1993). |
| W03-1002 101 35:186 POS tagging and phrase chunking in English were done using the trained systems provided with the fnTBL Toolkit (Ngai and Florian, 2001); both were trained from the annotated Penn Treebank corpus (Marcus et al. , 1993). |
| W03-1002 102 15:186 2 Prior Work Statistical machine translation, as pioneered by IBM (e.g. Brown et al. , 1993), is grounded in the noisy channel model. |
| N06-1021 103 36:189 For English, we used the Penn Treebank version 3.0 (Marcus et al. , 1993) and extracted dependency relations by applying the head-finding rules of (Yamada and Matsumoto, 2003). |
| W04-1903 104 90:97 Annotated reference corpora, such as the Brown Corpus (Kucera, Francis, 1967), the Penn Treebank (Marcus et al. , 1993), and the BNC (Leech et al. , 2001.), have helped both the development of English computational linguistics tools and English corpus linguistics. |
| W05-1619 105 37:135 Since text planners cannot generate either the requisite syntactic variation or quantity of text, [Langkilde-Geary, 2002] developed an evaluation strategy for HALOGEN employing a substitute: sentence parses from the Penn TreeBank [Marcus et al. , 1993], a corpus that includes texts from newspapers such as the Wall Street Journal, and which have been hand-annotated for syntax by linguists. |
| W05-1619 106 11:135 For instance, the HALOGEN statistical realizer [LangkildeGeary, 2002] underwent the most comprehensive evaluation of any surface realizer, which was conducted by measuring sentences extracted from the Penn TreeBank [Marcus et al. , 1993], converting them into its input formalism, and then producing output strings. |
| P06-2028 107 19:163 Typically, the local context around the 215 word to be sense-tagged is used to disambiguate the sense (Yarowsky, 1993), and it is common for linguistic resources such as WordNet (Li et al. , 1995; Mihalcea and Moldovan, 1998; Ramakrishnan and Prithviraj, 2004), or bilingual data (Li and Li, 2002) to be employed as well as more longrange context. |
| P99-1009 108 21:119 1 To train their system, R&M used a 200k-word chunk of the Penn Treebank Parsed Wall Street Journal (Marcus et al. , 1993) tagged using a transformation-based tagger (Brill, 1995) and extracted base noun phrases from its parses by selecting noun phrases that contained no nested noun phrases and further processing the data with some heuristics (like treating the possessive marker as the first word of a new base noun phrase) to flatten the recursive structure of the parse. |
| W03-1712 109 9:152 For example, the Penn Treebank (Marcus et al. , 1993) was annotated with skeletal syntactic structure, and many syntactic parsers were evaluated and compared on the corpus. |
| W03-1712 110 14:152 Although few corpora annotated with semantic knowledge are available now, there are some valuable lexical databases describing the lexical semantics in dictionary form, for example English WordNet (Miller et al. , 1993) and Chinese HowNet (Dong and Dong, 2001). |
| D07-1096 111 116:410 918 English For English we used the Wall Street Journal section of the Penn Treebank (Marcus et al. , 1993). |
| D07-1096 112 135:410 3.2 Domain Adaptation Track As mentioned previously, the source data is drawn from a corpus of news, specifically the Wall Street Journal section of the Penn Treebank (Marcus et al. , 1993). |
| W06-1638 113 154:255 We used sections 220 of the Penn Treebank 2 Wall Street Journal corpus (Marcus et al. , 1993) for training, section 22 as development set and section 23 for testing. |
| W06-1638 114 149:255 321 Jensen-Shannon divergence is defined as D(q,r) = 12 parenleftbigg D parenleftbigg q|| q +r2 parenrightbigg +D parenleftbigg r|| q +r2 parenrightbiggparenrightbigg These experiments are a kind of poor mans version of the deterministic annealing clustering algorithm (Pereira et al. , 1993; Rose, 1998), which gradually increases the number of clusters during the clustering process. |
| D07-1121 115 19:193 (1993), Johansson and Nugues (2007), Prokopidis et al. |
| W95-0105 116 28:297 (1993) found that direct annotation takes twice as long as automatic tagging plus correction, for partof-speech annotation); and the output quality reflects the difficulty of the task (inter-annotator disagreement is on the order of 10%, as contrasted with the approximately 3% error rate reported for part-of-speech annotation by Marcus et al.). |
| W95-0105 117 8:297 (Bensch and Savitch, 1992; Brill, 1991; Brown et al. , 1992; Grefenstette, 1994; McKcown and Hatzivassiloglou, 1993; Pereira et al. , 1993; Schtltze, 1993)). |
| W95-0105 118 55:297 The traditional method of evaluating similarity in a semantic network by measuring the path length between two nodes (Lee et al. , 1993; Rada et al. , 1989) also captures this, albeit indirectly, when the semantic network is just an IS-A hierarchy: if the minimal path of IS-A links between two nodes is long, that means it is necessary to go high in the taxonomy, to more abstract concepts, in order to find their least upper bound. |
| P03-2006 119 22:162 The corpus was automatically derived from the Penn Treebank II corpus (Marcus et al. , 1993), by means of the script chunklink.pl (Buchholz, 2002) that we modified to fit our purposes. |
| P03-2006 120 105:162 We performed experiments with two statistical classifiers: the decision tree induction system C4.5 (Quinlan, 1993) and the Tilburg Memory-Based Learner (TiMBL) (Daelemans et al. , 2002). |
| C96-1041 121 12:246 '\['here are three main approaches in tagging problem: rule-based approach (Klein and Simmons 1%3; Brodda 1982; Paulussen and Martin 1992; Brill et al. 1990), statistical approach (Church :1988; Merialdo 1994; Foster 1991; Weischedel et al. 1993; Kupiec 1992) and connectionist approach (Benello et al. 1989; Nakanmra et al. 1989). |
| P05-2004 122 91:237 4.2 Data The data comes from the CoNLL 2000 shared task (Sang and Buchholz, 2000), which consists of sentences from the Penn Treebank Wall Street Journal corpus (Marcus et al. , 1993). |
| D08-1008 123 17:220 Most systems for automatic role-semantic analysis have used constituent syntax as in the Penn Treebank (Marcus et al., 1993), although there has also been much research on the use of shallow syntax (Carreras and Mrquez, 2004) in SRL. |
| A00-2007 124 25:180 The main data set consist of four sections (15-18) of the Wall Street Journal (WSJ) part of the Penn Treebank (Marcus et al. , 1993) as training material and one section (20) as test material 1. |
| J99-2004 125 62:445 2.2 Statistical Parsers Pioneered by the IBM natural language group (Fujisaki et al. 1989) and later pursued by, for example, Schabes, Roth, and Osborne (1993), Jelinek et al. |
| J99-2004 126 90:445 Supertags Part-of-speech disambiguation techniques (POS taggers) (Church 1988; Weischedel et al. 1993; Brill 1993) are often used prior to parsing to eliminate (or substantially reduce) the part-of-speech ambiguity. |
| P08-1042 127 33:217 Treebank (Marcus et al., 1993), six of which are errors. |
| D07-1078 128 6:178 1 Introduction Syntax-based translation models (Eisner, 2003; Galley et al. , 2006; Marcu et al. , 2006) are usually built directly from Penn Treebank (PTB) (Marcus et al. , 1993) style parse trees by composing treebank grammar rules. |
| J06-1005 129 42:684 5 The SemCor collection (Miller et al., 1993) is a subset of the Brown Corpus and consists of 352 news articles distributed into three sets in which the nouns, verbs, adverbs, and adjectives have been manually tagged with their corresponding WordNet senses and part-of-speech tags using Brills tagger (1995). |
| W00-0721 130 78:97 The data sets used are the standard data sets for this problem (Ramshaw and Maxcus, 1995; Argamon et al. , 1999; Mufioz et al. , 1999; Tjong Kim Sang and Veenstra, 1999) taken from the Wall Street Journal corpus in the Penn Treebank (Marcus et al. , 1993). |
| P99-1016 131 12:143 Some of the data comes from the parsed files 2-21 of the Wall Street Journal Penn Treebank corpus (Marcus et al. , 1993), and additional parsed text was obtained by parsing the 1987 Wall Street Journal text using the parser described in Charniak et al. |
| W99-0622 132 15:280 As an example, consider the fiat NP structures that are in the Penn Treebank (Marcus et al. , 1993). |
| W02-1039 133 43:151 When an S alignment exists, there will always also exist a P alignment such that P a65 S. The English sentences were parsed using a state-of-the-art statistical parser (Charniak, 2000) trained on the University of Pennsylvania Treebank (Marcus et al. , 1993). |
| W02-1039 134 6:151 The first work in SMT, done at IBM (Brown et al. , 1993), developed a noisy-channel model, factoring the translation process into two portions: the translation model and the language model. |
| D08-1056 135 88:319 These categories were automatically generated using the labeled parses in Penn Treebank (Marcus et al., 1993) and the labeled semantic roles of PropBank (Kingsbury et al., 2002). |
| C98-2196 136 96:162 21418 examples of structures of the kind 'VB N1 PR,EP N2' were extracted from the Penn-TreeBank Wall Street Journal (Marcus et al. 1993). |
| C94-2149 137 104:212 Of the 1600 IBM sentences that have been parsed (those available from the Penn Treebank \[Marcus et al. , 19931), only 67 overlapped with the IBM-manual treebank that was bracketed by University of Lancaster. |
| C94-2149 138 184:212 \[Marcus et al. , 1993\] Marcus, M. , Santorini, B. , and Malvinkiewicz, M.A. |
| P05-1073 139 21:185 In the February 2004 version of the PropBank corpus, annotations are done on top of the Penn TreeBank II parse trees (Marcus et al. , 1993). |
| C02-1100 140 42:177 2 Background Default unification has been investigated by many researchers (Bouma, 1990; Russell et al. , 1991; Copestake, 1993; Carpenter, 1993; Lascarides and Copestake, 1999) in the context of developing lexical semantics. |
| C02-1100 141 72:177 As other researchers pursued efficient default unification (Bouma, 1990; Russell et al. , 1991; Copestake, 1993), we also propose another definition of default unification, which we call lenient default unification. |
| C02-1100 142 143:177 Without removing them, extracted rules cannot be triggered until when completely the same strings appear in a text.4 6 Performance Evaluation We measured the performance of our robust parsing algorithm by measuring coverage and degree of overgeneration for the Wall Street Journal in the Penn Treebank (Marcus et al. , 1993). |
| J00-4003 143 666:766 Only recently have robust knowledge-based methods for some of these tasks begun to appear, and their performance is still not very good, as seen above in our discussion of using WordNet as a semantic network; 33 as for checking the plausibility of a hypothesis on the basis of causal knowledge about the world, we now have a much better theoretical grasp of how such inferences could be made (see, for example, Hobbs et al. \[1993\] and Lascarides and Asher \[1993\]), but we are still quite a long way from a general inference engine. |
| W99-0301 144 172:269 html\] provided by Lynette Hirschman; syntactic structures in the style of the Penn TreeBank (Marcus et al. , 1993) provided by Ann Taylor; and an alternative annotation for the F0 aspects of prosody, known as Tilt (Taylor, 1998) and provided by its inventor, Paul Taylor. |
| P05-1040 145 13:165 (2001) compare taggers trained and tested on the Wall Street Journal (WSJ, Marcus et al. , 1993) and the Lancaster-Oslo-Bergen (LOB, Johansson, 1986) corpora and find that the results for the WSJ perform significantly worse. |
| P05-1040 146 16:165 Given the estimated 3% error rate of the WSJ tagging (Marcus et al. , 1993), they argue that the difference in performance is not sufficient to establish which of the two taggers is actually better. |
| P05-1040 147 22:165 On the other hand, structural annotation such as that used in syntactic treebanks (e.g. , Marcus et al. , 1993) assigns a syntactic category to a contiguous sequence of corpus positions. |
| D07-1125 148 21:184 Building heavily on the ideas of History-based parsing (Black et al. , 1993; Nivre, 2006), training the parser means essentially running the parsing algorithms in a learning mode on the data in order to gather training instances for the memory-based learner. |
| W00-1301 149 94:127 The training and test set were derived by finding all instances of the confusable words in the Brown Corpus, using the Penn Treebank parts of speech and tokenization (Marcus, Santorini et al. 1993), and then dividing this set into 80% for training and 20% for testing. |
| P99-1010 150 25:182 The study is conducted on both a simple Air Travel Information System (ATIS) corpus (Hemphill et al. , 1990) and the more complex Wall Street Journal (WSJ) corpus (Marcus et al. , 1993). |
| J97-3003 151 22:319 Such methods can achieve better performance, reaching tagging accuracy of up to 85% on unknown words for English (Brill 1994; Weischedel et al. 1993). |
| J97-3003 152 84:319 It has been noticed (as in \[Weischedel et al. , 1993\], for example) that capitalized and hyphenated words have a different distribution from other words. |
| W06-3122 153 41:91 We retrained the parser on lowercased Penn Treebank II (Marcus et al. , 1993), to match the lowercased output of the MT decoder. |
| W03-0806 154 78:162 Machine learning methods should be interchangeable: Transformation-based learning (TBL) (Brill, 1993) and Memory-based learning (MBL) (Daelemans et al. , 2002) have been applied to many different problems, so a single interchangeable component should be used to represent each method. |
| W03-0806 155 17:162 For example, 10 million words of the American National Corpus (Ide et al. , 2002) will have manually corrected POS tags, a tenfold increase over the Penn Treebank (Marcus et al. , 1993), currently used for training POS taggers. |
| P98-2140 156 114:149 The simplest "period-space-capital_letter" approach works well for simple texts but is rather unreliable for texts with many proper names and abbreviations at the end of sentence as, for instance, the Wall Street Journal (WSJ) corpus ( (Marcus et al. , 1993) ). |
| D08-1105 157 40:155 Building on the annotations from the Wall Street Journal (WSJ) portion of the Penn Treebank (Marcus et al., 1993), the project added several new layers of semantic annotations, such as coreference information, word senses, etc. In its first release (LDC2007T21) through the Linguistic Data Consortium (LDC), the project manually sense-tagged more than 40,000 examples belonging to hundreds of noun and verb types with an ITA of 90%, based on a coarse-grained sense inventory, where each word has an average of only 3.2 senses. |
| P05-2010 158 46:117 4 Analysis of Experimental Data Most of the existing research in computational linguistics that uses human annotators is within the framework of classification, where an annotator decides, for every test item, on an appropriate tag out of the pre-specified set of tags (Poesio and Vieira, 1998; Webber and Byron, 2004; Hearst, 1997; Marcus et al. , 1993). |
| W03-1009 159 126:175 4.1 Experimental Setup We use the whole Penn Treebank corpus (Marcus et al. , 1993) as our data set. |
| I05-3005 160 160:316 (Ng and Low 2004, Toutanova et al, 2003, Brants 2000, Ratnaparkhi 1996, Samuelsson 1993). |
| W97-0308 161 30:128 We have processed the Susanne corpus (Sampson, 1995) and Penn treebank (Marcus et al, 1993) to provide tables of word and subtree alignments. |
| C00-1044 162 93:188 302 (Marcus et al. , 1993) was nlanually annotated with subjeciivity chlssifications. |
| W05-0402 163 73:232 The list is obtained by first extracting the phrases with -TMP function tags from the PennTree bank, and taking the words in these phrases (Marcus et al. , 1993). |
| P00-1061 164 13:164 In all of the cited approaches, the Penn Wall Street Journal Treebank (Marcus et al. , 1993) is used, the availability of whichobviates the standard eort required for treebank traininghandannotating large corpora of specic domains of specic languages with specic parse types. |
| C04-1140 165 9:196 This is most prominently evidenced by the PENN TREEBANK (Marcus et al. , 1993). |
| C04-1140 166 39:196 The Brill tagger comes with an English default version also trained on general-purpose language corpora like the PENN TREEBANK (Marcus et al. , 1993). |
| W04-0302 167 119:145 The elementary trees were extracted from the parse trees in sections 02-21 of the Wall Street Journal in Penn Treebank (Marcus et al. , 1993), which is transformed by using parent-child annotation and left factoring (Roark and Johnson, 1999). |
| W08-2122 168 6:117 In this vein, the CoNLL 2008 shared task sets the challenge of learning jointly both syntactic dependencies (extracted from the Penn Treebank (Marcus et al., 1993) ) and semantic dependencies (extracted both from PropBank (Palmer et al., 2005) c2008. |
| P07-1080 169 125:171 We used the Penn Treebank WSJ corpus (Marcus et al. , 1993) to perform the empirical evaluation of the considered approaches. |
| E95-1022 170 30:171 157 ena or the linguist's abstraction capabilities (e.g. knowledge about what is relevant in the context), they tend to reach a 95-97% accuracy in the analysis of several languages, in particular English (Marshall 1983; Black et aL 1992; Church 1988; Cutting et al. 1992; de Marcken 1990; DeRose 1988; Hindle 1989; Merialdo 1994; Weischedel et al. 1993; Brill 1992; Samuelsson 1994; Eineborg and Gamb~ick 1994, etc.). |
| E95-1022 171 82:171 Note in passing that the ratio 1.04-1.08/99.7% compares very favourably with other systems; c.f. 3.0/99.3% by POST (Weischedel et al. 1993) and 1.04/97.6% or 1.09/98.6% by de Marcken (1990). |
| C98-1034 172 8:225 The bracketed portions of Figure 1, for example, show the base NPs in one sentence from the Penn Treebank Wall Street Journal (WSJ) corpus (Marcus et al., 1993). |
| N06-2026 173 32:89 PropBank encodes propositional information by adding a layer of argument structure annotation to the syntactic structures of the Penn Treebank (Marcus et al. , 1993). |
| W06-2902 174 7:192 This research has focused mostly on the development of statistical parsers trained on large annotated corpora, in particular the Penn Treebank WSJ corpus (Marcus et al. , 1993). |
| W06-2902 175 36:192 We use the Penn Treebank Wall Street Journal corpus as the large corpus and individual sections of the Brown corpus as the target corpora (Marcus et al. , 1993). |
| W00-1320 176 73:288 3.2 Probability structure of the original model We use p to denote the unlexicalized nonterminal corresponding to P, and similarly for li, ri and h. We now present the top-level generation probabilities, along with examples from 4The inclusion of the word feature in the BBN model was due to the work described in (Weischedel et al. , 1993), where word features helped reduce part of speech ambiguity for unknown words. |
| W00-1320 177 23:288 2.2 Motivation from previous work 2.2.1 Parsing In recent years, the success of statistical parsing techniques can be attributed to several factors, such as the increasing size of computing machinery to accommodate larger models, the availability of resources such as the Penn Treebank (Marcus et al. , 1993) and the success of machine learning techniques for lowerlevel NLP problems, such as part-of-speech tagging (Church, 1988; Brill, 1995), and PPattachment (Brill and Resnik, 1994; Collins and Brooks, 1995). |
| W07-2217 178 170:259 5 Parsing experiments 5.1 Data and setup We used the standard partitions of the Wall Street Journal Penn Treebank (Marcus et al. , 1993); i.e., sections 2-21 for training, section 22 for development and section 23 for evaluation. |
| E95-1015 179 179:230 4.1 The test environment For our experiments, we used a manually corrected version of the Air Travel Information System (ATIS) spoken language corpus (Hemphill et al. , 1990) annotated in the Pennsylvania Treebank (Marcus et al. , 1993). |
| P06-1123 180 108:221 3.3 Methods We parsed the English side of each bilingual bitext and both sides of each English/English bitext using an off-the-shelf syntactic parser (Bikel, 2004), which was trained on sections 02-21 of the Penn English Treebank (Marcus et al. , 1993). |
| P07-1035 181 126:171 For both experiments, we used dependency trees extracted from the Penn Treebank (Marcus et al. , 1993) using the head rules and dependency extractor from Yamada and Matsumoto (2003). |
| C08-1050 182 37:214 Statistical dependency parsers of English must therefore rely on dependency structures automatically converted from a constituent corpus such as the Penn Treebank (Marcus et al., 1993). |
| C08-1050 183 16:214 By habit, most systems for automatic role-semantic analysis have used Pennstyle constituents (Marcus et al., 1993) produced by Collins (1997) or Charniaks (2000) parsers. |
| W01-0706 184 61:129 Parsers Precision(a4 ) Recall(a4 ) a5a7a6 (a4 ) a8KM00 a9 93.45 93.51 93.48 a8Hal00 a9 93.13 93.51 93.32 a8CSCL a9 * 93.41 92.64 93.02 a8TKS00 a9 94.04 91.00 92.50 a8ZST00 a9 91.99 92.25 92.12 a8Dej00 a9 91.87 91.31 92.09 a8Koe00 a9 92.08 91.86 91.97 a8Osb00 a9 91.65 92.23 91.94 a8VB00 a9 91.05 92.03 91.54 a8PMP00 a9 90.63 89.65 90.14 a8Joh00 a9 86.24 88.25 87.23 a8VD00 a9 88.82 82.91 85.76 Baseline 72.58 82.14 77.07 2.2 Data Training was done on the Penn Treebank (Marcus et al. , 1993) Wall Street Journal data, sections 02-21. |
| W01-0706 185 11:129 First, it has been noted that in many natural language applications it is sufficient to use shallow parsing information; information such as noun phrases (NPs) and other syntactic sequences have been found useful in many large-scale language processing applications including information extraction and text summarization (Grishman, 1995; Appelt et al. , 1993). |
| W97-0121 186 256:275 We collected training samples from the Brown Corpus distributed with the Penn Treebank (Marcus et al.1993 ). |
| D07-1058 187 158:308 For a second set of parsing experiments, we used the WSJ portion of the Penn Tree Bank (Marcus et al. , 1993) and Helmut Schmids enrichment program tmod (Schmid, 2006). |
| W02-0817 188 22:149 One of the first large scale hand tagging efforts is reported in (Miller et al. , 1993), where a subset of the Brown corpus was tagged with WordNet July 2002, pp. |
| W02-0817 189 55:149 3.1 Data The starting corpus we use is formed by a mix of three different sources of data, namely the Penn Treebank corpus (Marcus et al. , 1993), the Los Angeles Times collection, as provided during TREC conferences1, and Open Mind Common Sense2, a collection of about 400,000 commonsense assertions in English as contributed by volunteers over the Web. |
| P06-2089 190 95:169 4 Experiments We evaluated our classifier-based best-first parser on the Wall Street Journal corpus of the Penn Treebank (Marcus et al. , 1993) using the standard split: sections 2-21 were used for training, section 22 was used for development and tuning of parameters and features, and section 23 was used for testing. |
| P06-2089 191 12:169 However, evaluations on the widely used WSJ corpus of the Penn Treebank (Marcus et al. , 1993) show that the accuracy of these parsers still lags behind the state-of-theart. |
| W04-2412 192 58:292 3 Data The data consists of six sections of the Wall Street Journal part of the Penn Treebank (Marcus et al. , 1993), and follows the setting of past editions of the CoNLL shared task: training set (sections 15-18), development set (section 20) and test set (section 21). |
| W97-1502 193 37:125 Many mainstream systems and formalisms would satisfy these criteria, including ones such as the University of Pennsylvania Treebank (Marcus et al, 1993) which are purely syntactic (though of course, only syntactic properties could then be extracted). |
| W99-0706 194 211:255 Our training and test corpora, for instance, are lessthan-gargantuan compared to such collections as the Penn Treebank \[Marcus et al. , 1993\]. |
| W99-0706 195 6:255 Many systems (e.g. , the KERNEL system \[Palmer et al. , 1993\]) use these relationships as an intermediate, form when determining the semantics of syntactically parsed text. |
| W99-0706 196 202:255 The figures given above were the original (1998) results for the system in \[Argamon et al. , 1998\], which came from training and testing on data derived from the Penn Treebank corpus \[Marcus et al. , 1993\] in which the added null elements (like null subjects) were left in. |
| P07-1079 197 80:260 4 Experiments We evaluate the accuracy of HPSG parsing with dependencyconstraintsontheHPSGTreebank(Miyao et al. , 2003), which is extracted from the Wall Street Journal portion of the Penn Treebank (Marcus et al. , 1993)1. |
| P07-1079 198 54:260 We created a dependency training corpus based on the Penn Treebank (Marcus et al. , 1993), or more specifically on the HPSG Treebank generated from the Penn Treebank (see section 2.2). |
| C04-1040 199 68:230 Dependency Analyzer PP-Attachment Resolver Root-Node Finder Base NP Chunker (POS Tagger) = SVM, = Preference Learning Figure 2: Module layers in the system That is, we use Penn Treebanks Wall Street Journal data (Marcus et al. , 1993). |
| P07-1026 200 147:194 The data consists of sections of the Wall Street Journal part of the Penn TreeBank (Marcus et al. , 1993), with information on predicate-argument structures extracted from the PropBank corpus (Palmer et al. , 2005). |
| C98-2179 201 25:155 To clarify the nature of the differences between various corpora and to find the causes of these differences, we analyzed 1122 psychological sentence production data (Connine et al. 1984), written discourse (Brown and WSJ ltom Penn Treebank Marcus et al. 1993), and conversational data (Switchboard Godfrey et al. 1992). |
| C98-2179 202 11:155 1 Introduction The probabilistic relation between verbs and their arguments plays an important role in modern statistical parsers and supertaggers (Charniak 1995, Collins 1996/1997, Joshi and Srinivas 1994, Kim, Srinivas, and Trueswell 1997, Stolcke et al. 1997), and in psychological theories of language processing (Clifton et al. 1984, Ferreira & McClure 1997, Garnsey et al. 1997, Jur'afsky 1996, MacDonald 1994, Mitchell & Holmes 1985, Tanenhaus et al. 1990, Trueswell et al. 1993). |
| C98-2179 203 20:155 This can be done automatically with unparsed corpora (Briscoe and Carroll 1997, Manning 1993, Ushioda et al. 1993), from parsed corpora such as Marcus et al.'s (1993) Treebank (Merlo 1994, Franlis 1994) or manually as was done for COMLEX (Macleod and Grishman 1994). |
| P03-1069 204 99:237 It achieves 90.1% average precision/recall for sentences with maximum length 40 and 89.5% for sentences with maximum length 100 when trained and tested on the standard sections of the Wall Street Journal Treebank (Marcus et al. , 1993). |
| N07-1049 205 112:180 Occasionally, in 59 sentences out of 2416 on section 23 of the Wall Street Journal Penn Treebank (Marcus et al. , 1993), the shift-reduce parser fails to attach a node to a head, producing a disconnected graph. |
| W06-3327 206 18:42 We measured the accuracy of the POS tagger trained in three settings: Original: The tagger is trained with the union of Wall Street Journal (WSJ) section of Penn Treebank (Marcus et al 1993), GENIA, and Penn BioIE. |
| P06-2004 207 31:217 The labeled corpus is the Penn Wall Street Journal treebank (Marcus et al. , 1993). |
| P06-2004 208 7:217 1 Introduction The best performing systems for many tasks in natural language processing are based on supervised training on annotated corpora such as the Penn Treebank (Marcus et al. , 1993) and the prepositional phrase data set first described in (Ratnaparkhi et al. , 1994). |
| P06-2004 209 195:217 With the exception of (Hindle and Rooth, 1993), most unsupervised work on PP attachment is based on superficial analysis of the unlabeled corpus without the use of partial parsing (Volk, 2001; Calvo et al. , 2005). |
| W00-1208 210 137:137 By comparing derivation trees for parallel sentences in two languages, instances of structural divergences (Dorr, 1993; Dorr, 1994; Palmer et al. , 1998) can be automatically detected. |
| W00-1208 211 22:137 2.2 Three Treebanks The Treebanks that we used in this paper are the English Penn Treebank II (Marcus et al. , 1993), the Chinese Penn Treebank (Xia et al. , 2000b), and the Korean Penn Treebank (Chung-hye Han, 2000). |
| W97-0201 212 48:114 3 The Effect of Training Corpus Size A number of past research work on WSD, such as (Leacock et al. , 1993; Bruce and Wiebe, 1994; Mooney, 1996), were tested on a small number of words like "line" and "interest". |
| W06-2110 213 83:170 4 Data Collection We evaluated out method by running RASP over Brown Corpus and Wall Street Journal, as contained in the Penn Treebank (Marcus et al. , 1993). |
| I08-2099 214 10:150 Some notable efforts in this direction for other languages have been the Penn Tree Bank (Marcus et al., 1993) for English and the Prague Dependency Bank (Hajicova, 1998) for Czech. |
| I08-2099 215 62:150 Other works based on this scheme like (Bharati et al., 1993; Bharati et al., 2002; Pedersen et al., 2004) have shown promising results. |
| D08-1091 216 152:223 Set Test Set ENGLISH-WSJ Sections Section 22 Section 23 (Marcus et al., 1993) 2-21 ENGLISH-BROWN see 10% of 10% of the (Francis et al. 2002) ENGLISH-WSJ the data6 the data6 FRENCH7 Sentences Sentences Sentences (Abeille et al., 2000) 1-18,609 18,610-19,609 19,609-20,610 GERMAN Sentences Sentences Sentences (Skut et al., 1997) 1-18,602 18,603-19,602 19,603-20,602 Table 1: Corpora and standard experimental setups. |
| W03-1707 217 12:178 The creation of the Penn English Treebank (Marcus et al. , 1993), a syntactically interpreted corpus, played a crucial role in the advances in natural language parsing technology (Collins, 1997; Collins, 2000; Charniak, 2000) for English. |
| W04-0305 218 135:189 6 The Experiments To investigate the e ects of lookahead on our family of deterministic parsers, we ran empirical experiments on the standard the Penn Treebank (Marcus et al. , 1993) datasets. |
| W06-0305 219 17:177 2 The Penn Discourse TreeBank (PDTB) The PDTB contains annotations of discourse relations and their arguments on the Wall Street Journal corpus (Marcus et al. , 1993). |
| N03-1033 220 12:202 Secondly, while all taggers use lexical information, and, indeed, it is well-known that lexical probabilities are much more revealing than tag sequence probabilities (Charniak et al. , 1993), most taggers make quite limited use of lexical probabilities (compared with, for example, the bilexical probabilities commonly used in current statistical parsers). |
| C98-2246 221 13:84 Charniak (Charniak et al., 1993) gives a thorough explanation of the equations for an IIMM model, and Kupiec (Kupiec, 1992) describes an ltMM tagging system in detail. |
| C98-2246 222 8:84 An example set of tags can be found in the Penn Treebank project (Marcus et al., 1993). |
| C98-2246 223 17:84 Weischedel's group (Weischedel et al., 1993) examines unknown words in the context of part-of-speech tagging. |
| W00-0709 224 74:132 4 Experiments The experiments described here were conducted using the Wall Street Journal Penn Treebank corpus (Marcus et al. , 1993). |
| W96-0203 225 137:240 The class based disambiguation operator is the Mutual Conditioned Plausibility (MCPI) (Basili et al. ,1993a). |
| W96-0203 226 108:240 Clusters are created by means of distributional techniques in (Ratnaparkhi et al, 1994), while in (Resnik and Hearst, 1993) low level synonim sets in WordNet are used. |
| W96-0203 227 45:240 This incremental process can be iterated to the point that the system 1 It is not just a matter of time, but also of required linguistic skills (see for example (Marcus et al, 1993)). |
| W96-0203 228 48:240 These later inductive phases may rely on some level of a priori knowledge, like for example the naive case relations used in the ARIOSTO_LEX system (Basili et al, 1993c, 1996). |
| W96-0203 229 62:240 To simplify, the plausibility of a detected esl is roughly inversely proportional to the number of mutually excluding syntactic structures in the text segment that generated the esl (see (Basili et al, 1993a) for details). |
| W96-0203 230 26:240 In general the training set is the parsed Wall Street Journal (Marcus et al, 1993), with few exceptions, and the size of the training samples is around 10-20,000 test cases. |
| W96-0203 231 123:240 This method is described hereafter, while the subsequent steps, that use deeper (rulebased) levels of knowledge, are implemented into the ARIOSTO_LEX lexical learning system, described in (Basili et al. , 1993b, 1933c and 1996). |
| W06-0609 232 20:157 (Marcus, et al. 1993; Santorini 1990) The syntactic annotation task consists of marking constituent boundaries, inserting empty categories (traces of movement, PRO, pro), showing the relationships between constituents (argument/adjunct structures), and specifying a particular subset of adverbial roles. |
| W98-0717 233 59:203 (1994) from the Penn Treebank (Marcus et al. , 1993) WSJ corpus. |
| W99-0611 234 12:220 (1998) present a probabilistic model for pronoun resolution trained on a small subset of the Penn Treebank Wall Street Journal corpus (Marcus et al. , 1993). |
| I05-2041 235 15:107 The first approaches are used for Penn Treebank (Marcus et al. , 1993) and the KAIST language resource (Lee et al. , 1997; Choi, 2001). |
| I05-2041 236 5:107 This kind of corpus has served as an extremely valuable resource for computational linguistics applications such as machine translation and question answering (Lee et al. , 1997; Choi, 2001), and has also proved useful in theoretical linguistics research (Marcus et al. , 1993). |
| I05-2041 237 26:107 However, most parsers still tend to show low performance on the long sentences (Li et al. , 1990; Doi et al. , 1993; Kim et al. , 2000). |
| D07-1018 238 219:223 For example, given that each semantic class exhibits a particular syntactic behaviour, information on the semantic class should improve POStagging for adjective-noun and adjective-participle ambiguities, probably the most difficult distinctions both for humans and computers (Marcus et al. , 1993; Brants, 2000). |
| W07-1602 239 84:164 For this reason, each preposition and verb was assigned a weight based on the proportion of occurrences of that word in the Penn Treebank (Marcus et al. , 1993) which are labelled with a spatial meaning. |
| C08-1025 240 12:206 For instance, about 38% of verbs in the training sections of the Penn Treebank (PTB) (Marcus et al., 1993) occur only once the lexical properties of these verbs (such as their most common subcategorization frames ) cannot be represented accurately in a model trained exclusively on the Penn Treebank. |
| W01-0904 241 56:181 3.1 The Corpus The systems are applied to examples from the Penn Treebank (Marcus et al. , 1993; Marcus et al. , 1994; Bies et al. , 1994) a corpus of over 4.5 million words of American English annotated with both part-of-speech and syntactic tree information. |
| W01-0904 242 92:181 Firstly, there is also H(RB) A(ADVP) declined H(VBD) H(VP) the dollar A(DT) H(NN) C(NP-SBJ) H(VP) H(S) Figure 2: A tree with constituents marked the top-down method, which is a version of the algorithm described by Hockenmaier et al (Hockenmaier et al. , 2000), but used for translating into simple (AB) CG rather than the Steedmans Combinatory Categorial Grammar (CCG) (Steedman, 1993). |
| W01-0904 243 62:181 However, as Categorial Grammar formalisms do not usually change the lexical entries of words to deal with movement, but use further rules (Wood, 1993; Steedman, 1993; Hockenmaier et al. , 2000), the lexicons learned here will be valid over corpora with movement. |
| W01-0904 244 10:181 For example, the Penn Treebank (Marcus et al. , 1993; Marcus et al. , 1994; Bies et al. , 1994) provides a large corpus of syntactically annotated examples mostly from the Wall Street Journal. |
| W01-0904 245 45:181 Hockenmaier et al (Hockenmaier et al. , 2000), although to some extent following the approach of Xia (Xia, 1999) where LTAGs are extracted, have pursued an alternative by extracting Combinatory Categorial Grammar (CCG) (Steedman, 1993; Wood, 1993) lexicons from the Penn Treebank. |
| N06-1031 246 16:157 The yield of this tree gives the target translation: the gunman was killed by police . The Penn English Treebank (PTB) (Marcus et al. , 1993) is our source of syntactic information, largely due to the availability of reliable parsers. |
| A00-1031 247 20:207 As two examples, (Rabiner, 1989) and (Charniak et al. , 1993) give good overviews of the techniques and equations used for Markov models and part-ofspeech tagging, but they are not very explicit in the details that are needed for their application. |
| A00-1031 248 11:207 Recent comparisons of approaches that can be trained on corpora (van Halteren et al. , 1998; Volk and Schneider, 1998) have shown that in most cases statistical aproaches (Cutting et al. , 1992; Schmid, 1995; Ratnaparkhi, 1996) yield better results than finite-state, rule-based, or memory-based taggers (Brill, 1993; Daelemans et al. , 1996). |
| A00-1031 249 23:207 Additionally, we present results of the tagger on the NEGRA corpus (Brants et al. , 1999) and the Penn Treebank (Marcus et al. , 1993). |
| A00-1031 250 164:207 The annotation consists of four parts: 1) a context-free structure augmented with traces to mark movement and discontinuous constituents, 2) phrasal categories that are annotated as node labels, 3) a small set of grammatical functions that are annotated as extensions to the node labels, and 4) part-of-speech tags (Marcus et al. , 1993). |
| W05-0307 251 18:435 A third of this is syntactically parsed as part of the Penn Treebank (Marcus et al. , 1993) and has dialog act annotation (Shriberg et al. , 1998). |
| W07-2048 252 21:92 We trained the parser on the Penn Treebank (Marcus et al. , 1993). |
| C00-1011 253 18:160 We compared this nonprobabilistic DOP model against tile probabilistic DOP model (which estimales the most probable parse for each sentence) on three different domains: tbe Penn ATIS treebank (Marcus et al. 1993), the Dutch OVIS treebank (Bonnema el al. 1997) and tile Penn Wall Street Journal (WSJ) treebank (Marcus el al. 1993). |
| C00-1011 254 71:160 While this technique has been sttccessfully applied to parsing lhe ATIS portion in the Penn Treebank (Marcus et al. 1993), it is extremely time consuming. |
| C00-1011 255 94:160 Experimental Comparison 4.1 Experiments on the ATIS corpus For our first comparison, we used I0 splits from the Penn ATIS corpus (Marcus et al. 1993) into training sets of 675 sentences and test sets of 75 sentences. |
| W05-0310 256 125:157 6 Discussion Lack of interannotator agreement presents a significant problem in annotation efforts (see, e.g., Marcus et al. 1993). |
| W05-0310 257 20:157 Post-editing of automatic annotation has been pursued in various projects (e.g. , Brants 2000, and Marcus et al. 1993). |
| W05-0310 258 21:157 The latter group did an experiment early on in which they found that manual tagging took about twice as long as correcting [automated tagging], with about twice the interannotator disagreement rate and an error rate that was about 50% higher (Marcus et al. 1993). |
| N03-3006 259 19:165 The parser has been trained, developed and tested on a large collection of syntactically analyzed sentences, the Penn Treebank (Marcus et al. , 1993). |
| H05-1083 260 105:218 As for parser, we train three off-shelf maximum-entropy parsers (Ratnaparkhi, 1999) using the Arabic, Chinese and English Penn treebank (Maamouri and Bies, 2004; Xia et al. , 2000; Marcus et al. , 1993). |
| H05-1083 261 19:218 This is possible because of the availability of statistical parsers, which can be trained on human-annotated treebanks (Marcus et al. , 1993; Xia et al. , 2000; Maamouri and Bies, 2004) for multiple languages; (2) The binding theory is used as a guideline and syntactic structures are encoded as features in a maximum entropy coreference system; (3) The syntactic features are evaluated on three languages: Arabic, Chinese and English (one goal is to see if features motivated by the English language can help coreference resolution in other languages). |
| H94-1034 262 43:235 In a test set of 756 utterances containing 26 repairs (Dowding et al. , 1993), they obtained a detection recall rate of 42% and a precision of 84.6%; for correction, they obtained a recall rate of 30% and a precision rate of 62%. |
| H94-1034 263 78:235 Good partof-speech results can be obtained using only the preceding category (Weischedel et al. , 1993), which is what we will be using. |
| W00-1201 264 7:100 The success of statistical methods in particular has been quite evident in the area of syntactic parsing, most recently with the outstanding results of (Charniak, 2000) and (Colhns, 2000) on the now-standard English test set of the Penn Treebank (Marcus et al. , 1993). |
| E99-1050 265 23:58 The tags sets we shall examine are the set used in the Penn Tree Bank (PTB) (Marcus et al. , 1993) and the C5 tag-set used by the CLAWS part-of-speech tagger (Garside, 1996). |
| N06-2019 266 41:84 For our out-of-domain training condition, the parser was trained on sections 2-21 of the Wall Street Journal (WSJ) corpus (Marcus et al. , 1993). |
| W98-1119 267 9:134 This program differs from earlier work in its almost complete lack of hand-crafting, relying instead on a very small corpus of Penn Wall Street Journal Tree-bank text (Marcus et al. , 1993) that has been marked with co-reference information. |
| D07-1102 268 14:95 This task evaluated parsing performance on 10 languages: Arabic, Basque, Catalan, Chinese, Czech, English, Greek, Hungarian, Italian, and Turkish using data originating from a wide variety of dependency treebanks, and transformations of constituency-based treebanks (Hajic et al. , 2004; Aduriz et al. , 2003; Mart et al. , 2007; Chen et al. , 2003; Bohmova et al. , 2003; Marcus et al. , 1993; Johansson and Nugues, 2007; Prokopidis et al. , 2005; Csendes et al. , 2005; Montemagni et al. , 2003; Oflazer et al. , 2003). |
| W95-0112 269 256:266 General purpose text annotations, such as part-of-speech tags and noun-phrase bracketing, are costly to obtain but have wide applicability and have been used successfully to develop statistical NLP systems (e.g. , \[Church, 1989; Weischedel et al. , 1993\]). |
| W95-0112 270 12:266 Furthermore, training corpora for information extraction are typically annotated with domain-specific tags, in contrast to general-purpose annotations such as part-of-speech tags or noun-phrase bracketing (e.g. , the Brown Corpus \[Francis and Kucera, 1982\] and the Penn Treebank \[Marcus et al. , 1993\]). |
| P06-1043 271 73:213 3.2 Wall Street Journal Our out-of-domain data is the Wall Street Journal (WSJ) portion of the Penn Treebank (Marcus et al. , 1993) which consists of about 40,000 sentences (one million words) annotated with syntactic information. |
| P06-1043 272 14:213 But the lack of corpora has led to a situation where much of the current work on parsing is performed on a single domain using training data from that domain the Wall Street Journal (WSJ) section of the Penn Treebank (Marcus et al. , 1993). |
| P08-1067 273 123:181 5 Experiments We compare the performance of our forest reranker against n-best reranking on the Penn English Treebank (Marcus et al., 1993). |
| P08-2026 274 20:102 One possible use for this technique is for parser adaptation initially training the parser on one type of data for which hand-labeled trees are available (e.g., Wall Street Journal (M. Marcus et al., 1993)) and then self-training on a second type of data in order to adapt the parser to the second domain. |
| C02-1126 275 72:139 For Penn Treebank II style annotation (Marcus et al. , 1993), in which a nonterminal symbol is a category together with zero or more functional tags, we adopt the following scheme: the atomic pattern a matches any label with category a or functional tag a; moreover, we define Boolean operators^,_, and:. |
| D07-1031 276 28:275 2 Evaluation All of the experiments described below have the same basic structure: an estimator is used to infer a bitag HMM from the unsupervised training corpus (the words of Penn Treebank (PTB) Wall Street Journal corpus (Marcus et al. , 1993)), and then the resulting model is used to label each word of that corpus with one of the HMMs hidden states. |
| E95-1029 277 17:143 A more optimistic view can be found in (Leech and Eyes 1993, p. 39; Marcus et al. 1993, p. 328); they argue that a near-100% interjudge agreement is possible, provided the part-of-speech annotation is done carefully by experts. |
| E95-1029 278 137:143 Our results agree, at least at the level of morphology, with (Leech and Eyes 1993; Marcus et al. 1993). |
| W97-0202 279 8:153 1 Introduction This paper reports on our experience hand tagging the senses of 25 of the most frequent verbs in 12,925 sentences of the Wall Street Journal Treebank corpus (Marcus et al. 1993). |
| W97-0202 280 3:153 edu Abstract This paper reports on our experience hand tagging the senses of 25 of the most frequent verbs in 12,925 sentences of the Wall Street Journal Treebank corpus (Marcus et al. 1993). |
| P06-2010 281 123:184 The data consist of sections of the Wall Street Journal (WSJ) part of the Penn TreeBank (Marcus et al. , 1993), with information on predicate-argument structures extracted from the PropBank corpus (Palmer et al. , 2005). |
| I05-4002 282 8:131 For many languages, large-scale syntactically annotated corpora have been built (e.g. the Penn Treebank (Marcus et al. , 1993)), and many parsing algorithms using CFGs have been proposed. |
| P05-1025 283 48:182 Unlabeled dependencies can be readily obtained by processing constituent trees, such as those in the Penn Treebank (Marcus et al. , 1993), with a set of rules to determine the lexical heads of constituents. |
| P94-1034 284 48:165 Several frameworks for finding translation equivalents or translation units in machine translation, such as \[Chang and Su 1993, Isabelle et al.1993\] and other example-based MT approaches, might be used to select the preferred mapping. |
| P94-1034 285 10:165 One major resource for corpus-based research is the treebanks available in many research organizations \[Marcus et al.1993\], which carry skeletal syntactic structures or 'brackets' that have been manually verified. |
| E06-1034 286 29:197 For example, in the WSJ corpus, part of the Penn Treebank 3 release (Marcus et al. , 1993), the string in (1) is a variation 12-gram since off is a variation nucleus that in one corpus occurrence is tagged as a preposition (IN), while in another it is tagged as a particle (RP). |
| W00-0726 287 15:150 We have chosen to work with a corpus with parse information, the Wall Street Journal WSJ part of the Penn Treebank II corpus (Marcus et al. , 1993), and to extract chunk information from the parse trees in this corpus. |
| W04-1501 288 21:124 Also in the Penn Treebank ((Marcus et al. , 1993), (Marcus et al. , 1994)) a limited set of relations is placed over the constituencybased annotation in order to make explicit the (morpho-syntactic or semantic) roles that the constituents play. |
| P08-1098 289 122:197 The WSJ corpus is based on the WSJ part of the PENN TREEBANK (Marcus et al., 1993); we used the first 10,000 sentences of section 2-21 as the pool set, and section 00 as evaluation set (1,921 sentences). |
| P08-1098 290 30:197 In the general language UPenn annotation efforts for the WSJ sections of the Penn Treebank (Marcus et al., 1993), sentences are annotated with POS tags, parse trees, as well as discourse annotation from the Penn Discourse Treebank (Miltsakaki et al., 2008), while verbs and verb arguments are annotated with Propbank rolesets (Palmer et al., 2005). |
| W03-1006 291 7:259 The PropBank superimposes an annotation of semantic predicate-argument structures on top of the Penn Treebank (PTB) (Marcus et al. , 1993; Marcus et al. , 1994). |
| W97-0209 292 9:195 The approach combines statistical and knowledge-based methods, but unlike many recent corpus-based approaches to sense disambiguation (arowsky, 1993; Bruce and Wiebe, 1994; Miller et al. , 1994), it takes as its starting point the assumption that senseannotated training text is not available. |
| W97-0209 293 55:195 Test and training materials were derived from the Brown corpus of American English, all of which has been parsed and manually verified by the Penn T~eebank project (Marcus et al. , 1993) and parts of which have been manually sense-tagged by the WordNet group (Miller et al. , 1993). |
| W97-0209 294 11:195 In marked contrast to annotated training material for partof-speech tagging, (a) there is no coarse-level set of sense distinctions widely agreed upon (whereas part-of-speech tag sets tend to differ in the details); (b) sense annotation has a comparatively high error rate (Miller, personal communication, reports an upper bound for human annotators of around 90% for ambiguous cases, using a non-blind evaluation method that may make even this estimate overly optimistic); and (c) no fully automatic method provides high enough quality output to support the "annotate automatically, correct manually" methodology used to provide high volume annotation by data providers like the Penn Treebank project (Marcus et al. , 1993). |
| C96-2125 295 7:134 Recently, we can see an important development in natural language processing and computational linguistics towards the use of empirical learning methods (for instance, (Charniak, 1993; Marcus et al. , 1993; Wermter, 11995; Jones, 1995; Werml;er et al. , 1996)). |
| A00-2005 296 35:181 2.3 Experiment The training set for these experiments was sections 01-21 of the Penn Treebank (Marcus et al. , 1993). |
| P98-2184 297 27:157 1984), written discourse (Brown and WSJ from Penn Treebank Marcus et al. 1993), and conversational data (Switchboard Godfrey et al. 1992). |
| P98-2184 298 21:157 This can be done automatically with unparsed corpora (Briscoe and Carroll 1997, Manning 1993, Ushioda et al. 1993), from parsed corpora such as Marcus et al.'s (1993) Treebank (Merlo 1994, Framis 1994) or manually as was done for COMLEX (Macleod and Grishman 1994). |
| P98-2184 299 11:157 1 Introduction The probabilistic relation between verbs and their arguments plays an important role in modern statistical parsers and supertaggers (Charniak 1995, Collins 1996/1997, Joshi and Srinivas 1994, Kim, Srinivas, and Trueswell 1997, Stolcke et al. 1997), and in psychological theories of language processing (Clifton et al. 1984, Ferfeira & McClure 1997, Gamsey et al. 1997, Jurafsky 1996, MacDonald 1994, Mitchell & Holmes 1985, Tanenhaus et al. 1990, Trueswell et al. 1993). |
| P06-1060 300 40:177 There are cases, though, where the labels consist of several related, but not entirely correlated, properties; examples include mention detectionthe task we are interested in, syntactic parsing with functional tag assignment (besides identifying the syntactic parse, also label the constituent nodes with their functional category, as defined in the Penn Treebank (Marcus et al. , 1993)), and, to a lesser extent, part-of-speech tagging in highly inflected languages.4 The particular type of mention detection that we are examining in this paper follows the ACE general definition: each mention in the text (a reference to a real-world entity) is assigned three types of information:5 An entity type, describing the type of the entity it points to (e.g. person, location, organization, etc) An entity subtype, further detailing the type (e.g. organizations can be commercial, governmental and non-profit, while locations can be a nation, population center, or an international region) A mention type, specifying the way the entity is realized a mention can be named (e.g. John Smith), nominal (e.g. professor), or pronominal (e.g. she). |
| W96-0213 301 102:123 ~ gtPdl= |&.allm~WI.Lqlf IDW,t~lIO, r I~"1~~ ~ II, Mlmulm, IP, il~,,lllb, l~ ~ I I I I I I I I I 0 200 400 600 800 1000 1200 1400 1600 1800 Article# 2000 Figure 1: Distribution of Tags for the word "about" vs. Article# Training Size(wrds)I Test571190 Size(wrds) I Baseline44478 97.04% Specialized 197.13% Table 10: Performance of Baseline ~ Specialized Model When Tested on Consistent Subset of Development Set 139 POS Tag 35 30 25 2O 15 10 5 0 1 I o. Oho m I I I B ~ m M I I I 2 3 4 Annotator Figure 2: Distribution of Tags for the word "about" vs. Annotator (Weischedel et al. , 1993) provide the results from a battery of "tri-tag" Markov Model experiments, in which the probability P(W,T) of observing a word sequence W = {wl,w2,,wn} together with a tag sequence T = {tl,t2,,tn} is given by: P(TIW)P(W) p(tl)p(t21tl) H P(tilti-lti-2) p(wilti i=3 Furthermore, p(wilti) for unknown words is computed by the following heuristic, which uses a set of 35 pre-determined endings: p(wilti) p(unknownwordlti ) x p(capitalfeature\[ti) x p(endings, hypenationlti ) This approximation works as well as the MaxEnt model, giving 85% unknown word accuracy(Weischedel et al. , 1993) on the Wall St. Journal, but cannot be generalized to handle more diverse information sources. |
| W96-0213 302 12:123 Previous uses of this model include language modeling(Lau et al. , 1993), machine translation(Berger et al. , 1996), prepositional phrase attachment(Ratnaparkhi et al. , 1994), and word morphology(Della Pietra et al. , 1995). |
| W96-0213 303 92:123 Comparison With Previous Work Most of the recent corpus-based POS taggers in the literature are either statistically based, and use Markov Model(Weischedel et al. , 1993, Merialdo, 1994) or Statistical Decision Tree(Jelinek et al. , 1994, Magerman, 1995)(SDT) techniques, or are primarily rule based, such as Drill's Transformation Based Learner(Drill, 1994)(TBL). |
| W96-0213 304 23:123 In practice, 7-/ is very large and the model's expectation Efj cannot be computed directly, so the following approximation(Lau et al. , 1993) is used: n E fj,~ E15(hi)p(tilhi)fj(hi,ti) i=1 where fi(hi) is the observed probability of the history hi in the training set. |
| W08-2101 305 33:190 2 The Data Our experiments on joint syntactic and semantic parsing use data that is produced automatically by merging the Penn Treebank (PTB) with PropBank (PRBK) (Marcus et al., 1993; Palmer et al., 2005), as shown in Figure 1. |
| P06-1023 306 6:187 In an evaluation on the PENN treebank (Marcus et al. , 1993), the parser outperformed other unlexicalized PCFG parsers in terms of labeled bracketing fscore. |
| P06-1023 307 8:187 1 Introduction Empty categories (also called null elements) are used in the annotation of the PENN treebank (Marcus et al. , 1993) in order to represent syntactic phenomena like constituent movement (e.g. whextraction), discontinuous constituents, and missing elements (PRO elements, empty complementizers and relative pronouns). |
| C08-1113 308 133:177 As mentioned in Section 2.2, there are words which have two or more candidate POS tags in the PTB corpus (Marcus et al., 1993). |
| W96-0112 309 17:254 1993; Chang et al. , 1992; Collins and Brooks, 1995; Fujisaki, 1989; Hindle and Rooth, 1991; Hindle and Rooth, 1993; Jelinek et al. , 1990; Magerman and Marcus, 1991; Magerman, 1995; Ratnaparkhi et al. , 1994; Resnik, 1993; Su and Chang, 1988). |
| W96-0112 310 197:254 We extracted 181,250 case frames from the WSJ (Wall Street Journal) bracketed corpus of the Penn Tree Bank (Marcus et al. , 1993). |
| W96-0112 311 25:254 For subproblem (a), we have devised a new method, based on LPR, which has some good properties not shared by the methods proposed so far (Alshawi and Carter, 1995; Chang et al. , 1992; Collins and Brooks, 1995; Hindle and Rooth, 1991; Ratnaparkhi et al. , 1994; Resnik, 1993). |
| H05-1070 312 9:265 Examples are the Penn Treebank (Marcus et al. , 1993) for American English annotated at the University of Pennsylvania, the French treebank (Abeille and Clement, 1999) developed in Paris, the TIGER Corpus (Brants et al. , 2002) for German annotated at the Universities of Saarbrcurrency1ucken and This research was funded by a German Science Foundation grant (DFG SFB441-6). |
| P99-1021 313 69:177 Both taggers used the Penn Treebank tagset and were trained on the Wall Street Journal corpus (Marcus et al. , 1993). |
| W06-0602 314 24:182 This corpus contains annotations of semantic PASs superimposed on the Penn Treebank (PTB) (Marcus et al. , 1993; Marcus et al. , 1994). |
| D07-1129 315 70:93 4 Experiments Our experiments were conducted on CoNLL-2007 shared task domain adaptation track (Nivre et al. , 2007) using treebanks (Marcus et al. , 1993; Johansson and Nugues, 2007; Kulick et al. , 2004). |
| P05-1036 316 38:201 The K&M model creates a packed parse forest of all possible compressions that are grammatical with respect to the Penn Treebank (Marcus et al. , 1993). |
| P98-2251 317 8:84 An example set of tags can be found in the Penn Treebank project (Marcus et al. , 1993). |
| P98-2251 318 13:84 Charniak (Charniak et al. , 1993) gives a thorough explanation of the equations for an HMM model, and Kupiec (Kupiec, 1992) describes an HMM tagging system in detail. |
| P98-2251 319 17:84 Weischedel's group (Weischedel et al. , 1993) examines unknown words in the context of part-of-speech tagging. |
| W06-2303 320 9:147 PropBank encodes propositional information by adding a layer of argument structure annotation to the syntactic structures of the Penn Treebank (Marcus et al. , 1993). |
| W03-0310 321 40:130 This cost can often be substantial, as with the Penn Treebank (Marcus et al. , 1993). |
| P06-2067 322 8:193 1 Introduction Robust statistical syntactic parsers, made possible by new statistical techniques (Collins, 1999; Charniak, 2000; Bikel, 2004) and by the availability of large, hand-annotated training corpora such as WSJ (Marcus et al. , 1993) and Switchboard (Godefrey et al. , 1992), have had a major impact on the field of natural language processing. |
| W02-1028 323 10:187 Even for relatively general texts, such as the Wall Street Journal (Marcus et al. , 1993) or terrorism articles (MUC4 Proceedings, 1992), Roark and Charniak (Roark and Charniak, 1998) reported that 3 of every 5 terms generated by their semantic lexicon learner were not present in WordNet. |
| W98-1115 324 82:145 4 The Experiment For our experiment, we used a tree-bank grammar induced from sections 2-21 of the Penn Wall Street Journal text (Marcus et al. , 1993), with section 22 reserved for testing. |
| D08-1071 325 145:260 4.1 Data Sets Our results are based on syntactic data drawn from the Penn Treebank (Marcus et al., 1993), specifically the portion used by CoNLL 2000 shared task (Tjong Kim Sang and Buchholz, 2000). |
| D08-1071 326 70:260 We use 3500 sentences from CoNLL (Tjong Kim Sang and De Meulder, 2003) as the NER data and section 20-23 of the WSJ (Marcus et al., 1993; Ramshaw and Marcus, 1995) as the POS/chunk data (8936 sentences). |
| C08-1038 327 9:151 1999), OpenCCG (White, 2004) and XLE (Crouch et al., 2007), or created semi-automatically (Belz, 2007), or fully automatically extracted from annotated corpora, like the HPSG (Nakanishi et al., 2005), LFG (Cahill and van Genabith, 2006; Hogan et al., 2007) and CCG (White et al., 2007) resources derived from the Penn-II Treebank (PTB) (Marcus et al., 1993). |
| C96-1020 328 3:125 Treebanks have been used within the field of natural language processing as a source of training data for statistical part og speech taggers (Black et al. , 1992; Brill, 1994; Merialdo, 1994; Weischedel et al. , 1993) and for statistical parsers (Black et al. , 1993; Brill, 1993; aelinek et al. , 1994; Magerman, 1995; Magerman and Marcus, 1991). |
| C96-1020 329 8:125 All of the features of the ATR/Lancaster Treebank that are described below represent a radical departure from extant large-scale (Eyes and Leech, 1993; Garside and McEnery, 1993; Marcus et al. , 1993) treebanks. |
| C00-2157 330 5:77 1 Introduction Syntactically annotated corpora like the Penn Treebank (Marcus et al. , 1993), the NeGra corpus (Skut et al. , 1998) or the statistically dismnbiguated parses in (Bell et al. , 1999) provide a wealth of intbrmation, which can only be exploited with an adequate query language. |
| C00-2157 331 13:77 (Carpenter, 1992), (Copestake, 1999), (DSrre and Dorna, 1993), (D6I're et al. , 1996), (Emele and Zajac, 1990), (H6ht~ld and Smolka, 1988)), and to pick those ingredients which are known to be con~i)utationally 'tractable' in some sense. |
| P04-1043 332 144:222 4.1 Corpora set-up The above kernels were experimented over two corpora: PropBank (www.cis.upenn.edu/ ace) along with Penn TreeBank5 2 (Marcus et al. , 1993) and FrameNet. |
| W04-2208 333 57:226 The definitions of part-of-speech (POS) categories and syntactic labels follow those of the Treebank I style (Marcus et al. , 1993). |
| W04-2208 334 11:226 On the other hand, high-quality treebanks such as the Penn Treebank (Marcus et al. , 1993) and the Kyoto University text corpus (Kurohashi and Nagao, 1997) have contributed to improving the accuracies of fundamental techniques for natural language processing such as morphological analysis and syntactic structure analysis. |
| C96-1038 335 90:256 4 Experiments The Penn Treebank (Marcus et al. , 1993) is used as the testing corpus. |
| C98-2177 336 30:151 To identify conjunctions, lists, and appositives, we first parsed the corpus, using an efficient statistical parser (Charniak et al., 1998), trained on the Penn Wall Street Journal Treebank (Marcus el; al., 1993). |
| D07-1097 337 8:119 1 Introduction In the multilingual track of the CoNLL 2007 shared task on dependency parsing, a single parser must be trained to handle data from ten different languages: Arabic (Hajic et al. , 2004), Basque (Aduriz et al. , 2003), Catalan, (Mart et al. , 2007), Chinese (Chen et al. , 2003), Czech (Bohmova et al. , 2003), English (Marcus et al. , 1993; Johansson and Nugues, 2007), Greek (Prokopidis et al. , 2005), Hungarian (Csendes et al. , 2005), Italian (Montemagni et al. , 2003), and Turkish (Oflazer et al. , 2003).1 Our contribution is a study in multilingual parser optimization using the freely available MaltParser system, which performs 1For more information about the task and the data sets, see Nivre et al. |
| N03-1031 338 6:194 1 Introduction Current state-of-the-art statistical parsers (Collins, 1999; Charniak, 2000) are trained on large annotated corpora such as the Penn Treebank (Marcus et al. , 1993). |
| P96-1043 339 11:256 A similar approach was taken in (Weischedel et al. , 1993) where an unknown word was guessed given the probabilities for an unknown word to be of a particular POS, its capitalisation feature and its ending. |
| P96-1043 340 168:256 These texts were not seen at the training phase which means that neither the 6Since Brill's tagger was trained on the Penn tag-set (Marcus et al. , 1993) we provided an additional mapping. |
| W05-1506 341 26:254 This paper, however, aims at the k-best tree algorithms whose packed representations are hypergraphs (Gallo et al. , 1993; Klein and Manning, 2001) (equivalently, and/or graphs or packed forests), which includes most parsers and parsing-based MT decoders. |
| W05-1506 342 198:254 For this experiment, we used sections 02 21 of the Penn Treebank (PTB) (Marcus et al. , 1993) as the training data and section 23 (2416 sentences) for evaluation, as is now standard. |
| W05-1506 343 61:254 3 Formulation Following Klein and Manning (2001), we use weighted directed hypergraphs (Gallo et al. , 1993) as an abstraction of the probabilistic parsing problem. |
| W05-1506 344 52:254 (1993) study the shortest hyperpath problem and Nielsen et al. |
| A00-2033 345 28:135 The PT grammar 2 was extracted from the Penn Treebank (Marcus et al. , 1993). |
| J99-4003 346 457:812 This source is very important for repairs that do not have initial retracing, and is the mainstay of the "parser-first" approach (e.g. , 550 Heeman and Allen Modeling Speakers' Utterances Dowding et al. 1993)--keep trying alternative corrections until one of them parses. |
| J99-4003 347 721:812 In a test set containing 26 repairs Dowding et al. 1993, they obtained a detection recall rate of 42% with a precision of 85%, and a correction recall rate of 31% with a precision of 62%. |
| J99-4003 348 227:812 These are the same distributions that are needed by previous POS-based language models (Equation 5) and POS taggers (Church 1988; Charniak et al. 1993). |
| P98-1034 349 8:227 The bracketed portions of Figure 1, for example, show the base NPs in one sentence from the Penn Treebank Wall Street Journal (WSJ) corpus (Marcus et al. , 1993). |
| P04-1013 350 126:185 6 The Experiments We used the Penn Treebank (Marcus et al. , 1993) to perform empirical experiments on the proposed parsing models. |
| N03-1014 351 121:204 6 The Experimental Results We used the Penn Treebank (Marcus et al. , 1993) to perform empirical experiments on this parsing model. |
| P06-2088 352 121:156 The experiment used all 578 sentences in the ATIS corpus with a parse tree, in the Penn Treebank (Marcus et al. 1993). |
| N06-2015 353 16:75 2 Treebanking The Penn Treebank (Marcus et al. , 1993) is annotated with information to make predicate-argument structure easy to decode, including function tags and markers of empty categories that represent displaced constituents. |
| P99-1054 354 86:168 The grammars were induced from sections 2-21 of the Penn Wall St. Journal Treebank (Marcus et al. , 1993), and tested on section 23. |
| D07-1111 355 17:130 In the multilingual parsing track, participants train dependency parsers using treebanks provided for ten languages: Arabic (Hajic et al. , 2004), Basque (Aduriz et al. 2003), Catalan (Mart et al. , 2007), Chinese (Chen et al. , 2003), Czech (Bhmova et al. , 2003), English (Marcus et al. , 1993; Johansson and Nugues, 2007), Greek (Prokopidis et al. , 2005), Hungarian (Czendes et al. , 2005), Italian (Montemagni et al. , 2003), and Turkish (Oflazer et al. , 2003). |
| D07-1111 356 18:130 In the domain adaptation track, participants were provided with English training data from the Wall Street Journal portion of the Penn Treebank (Marcus et al. , 1993) converted to dependencies (Johansson and Nugues, 2007) to train parsers to be evaluated on material in the biological (development set) and chemical (test set) domains (Kulick et al. , 2004), and optionally on text from the CHILDES database (MacWhinney, 2000; Brown, 1973). |
| W02-1509 357 5:133 With the availability of large natural language corpora annotated for syntactic structure, the treebanks, e.g., (Marcus et al. , 1993), automatic grammar extraction became possible (Chen and VijayShanker, 2000; Xia, 1999). |
| P08-1061 358 117:148 For experiment on English, we used the English Penn Treebank (PTB) (Marcus et al., 1993) and the constituency structures were converted to dependency trees using the same rules as (Yamada and Matsumoto, 2003). |
| W06-0611 359 44:127 Section 4 concludes the paper with a critical assessment of the proposed approach and a discussion of the prospects for application in the construction of corpora comparable in size and quality to existing treebanks (such as, for example, the Penn Treebank for English (Marcus et al. , 1993) or the TIGER Treebank for German (Brants et al. , 2002)). |
| W08-1008 360 6:159 Other languagesfor which this is the case include English (with the Penn treebank (Marcus et al., 1993), the Susanne Corpus (Sampson, 1993), and the British section of the ICE Corpus (Wallis and Nelson, 2006)) and Italian (with ISST (Montegmagni et al., 2000) and TUT (Bosco et al., 2000)). |
| I05-2019 361 13:145 eBonsai first performs syntactic analysis of a sentence using a parser based on GLR algorithm (MSLR parser) (Tanaka et al. , 1993), and provides candidates of its syntactic structure. |
| I05-2019 362 27:145 The MSLR parser (Tanaka et al. , 1993) performs syntactic analysis of the sentence. |
| I05-2019 363 7:145 Particularly, syntactically annotated corpora (treebanks), such as Penn Treebank (Marcus et al. , 1993), Negra Corpus (Skut et al. , 1997) and EDR Corpus (Jap, 1994), contribute to improve the performance of morpho-syntactic analysis systems. |
| W96-0111 364 50:275 To deal with this question, we use ATIS p-o-s trees as found in the Penn Treebank (Marcus et al. , 1993). |
| W96-0111 365 186:275 The latter approach has become increasingly popular (e.g. Schabes et al. , 1993; Weischedel et al. , 1993; Briscoe, 1994; Magerman, 1995; Collins, 1996). |
| W96-0111 366 13:275 In previous work, we tested the DOP method on a cleaned-up set of analyzed part-of-speech strings from the Penn Treebank (Marcus et al. , 1993), achieving excellent test results (Bod, 1993a, b). |
| D07-1112 367 10:131 We were given around 15K sentences of labeled text from the Wall Street Journal (WSJ) (Marcus et al. , 1993; Johansson and Nugues, 2007) as well as 200K unlabeled sentences. |
| D07-1112 368 42:131 The annotation guidelines for the Penn Treebank flattened noun phrases to simplify annotation (Marcus et al. , 1993), so there is no complex structure to NPs. |
| P08-1082 369 93:236 This probability is computed using IBMs Model 1 (Brown et al., 1993): P(Q|A) = productdisplay qQ P(q|A) (3) P(q|A) = (1)Pml(q|A)+Pml(q|C) (4) Pml(q|A) = summationdisplay aA (T(q|a)Pml(a|A)) (5) where the probability that the question term q is generated from answer A, P(q|A), is smoothed using the prior probability that the term q is generated from the entire collection of answers C, Pml(q|C). |
| P08-1082 370 136:236 The text was split at the sentence level, tokenized and PoS tagged, in the style of the Wall Street Journal Penn TreeBank (Marcus et al., 1993). |
| W04-0212 371 4:198 1 Introduction Large scale annotated corpora such as the Penn TreeBank (Marcus et al. , 1993) have played a central role in speech and natural language research. |
| W04-0212 372 97:198 However, developing the PDTB may help facilitate the production of more such corpora, through an initial pass of automatic annotation, followed by manual correction, much as was done in developing the PTB (Marcus et al. , 1993).) |
| C08-1012 373 48:171 2 Data Sets for the Experiments 2.1 Coordination Annotation in the PENN TREEBANK For our experiments, we used the WSJ part of the PENN TREEBANK (Marcus et al., 1993). |
| W99-0707 374 17:186 The approach is evaluated by cross-validation on the WSJ treebank corpus \[Marcus et al. , 1993\]. |
| J98-2001 375 18:640 In the past two or three years, this kind of verification has been attempted for other aspects of semantic interpretation: by Passonneau and Litman (1993) for segmentation and by Kowtko, Isard, and Doherty (1992) and Carletta et al. |
| W00-0735 376 3:60 While the tag features, containing WSJ paxt-ofspeech tags (Marcus et al. , 1993), have about 45 values, the word features have more than 10,000 values. |
| W00-0725 377 13:51 The experiments were performed using the Wall Street Journal (WSJ) corpus of the University of Pennsylvania (Marcus et al. , 1993) modified as described in (Charniak, 1996) and (Johnson, 1998). |
| P08-1117 378 78:233 4 Corpus Annotation For our corpus, we selected 1,000 sentences containing at least one comma from the Penn Treebank (Marcus et al., 1993) WSJ section 00, and manually annotated them with comma information3. |
| P08-1117 379 59:233 In (Bayraktar et al., 1998) the WSJ PennTreebank corpus (Marcus et al., 1993) is analyzed and a very detailed list of syntactic patterns that correspond to different roles of commas is created. |
| W05-0309 380 6:122 1 Introduction There is a pressing need for a consensus on a taskoriented level of semantic representation that can enable the development of powerful new semantic analyzers in the same way that the Penn Treebank (Marcus et al. , 1993) enabled the development of statistical syntactic parsers (Collins, 1999; Charniak, 2001). |
| N07-1058 381 89:201 The default training set of Penn Treebank (Marcus et al. 1993) was used for the parser because the domain and style of those texts actually matches fairly well with the domain and style of the texts on which a reading level predictor for second language learners might be used. |
| H05-1099 382 12:185 Li and Roth demonstrated that their shallow parser, trained to label shallow constituents along the lines of the well-known CoNLL2000 task (Sang and Buchholz, 2000), outperformed the Collins parser in correctly identifying these constituents in the Penn Wall Street Journal (WSJ) Treebank (Marcus et al. , 1993). |
| P94-1044 383 7:106 In addition, corpus-based stochastic modelling of lexical patterns (see Weischedel et al. , 1993) may provide information about word sense frequency of the kind advocated since (Ford et al. , 1982). |
| P06-2069 384 99:185 TB TBR JJ, JJR, JJS JJ RB,RBR,RBS RB CD, LS CD CC CC DT, WDT, PDT DT FW FW MD, VB, VBD, VBG, VBN, VBP, VBZ, VH, VHD, VHG, VHN, VHP, VHZ MD NN, NNS, NP, NPS NN PP, WP, PP$, WP$, EX, WRB PP IN, TO IN POS PO RP RP SYM SY UH UH VV, VVD, VVG, VVN, VVP, VVZ VB (Marcus et al. , 1993). |
| J01-4003 385 12:334 Next we use the conclusions from two psycholinguistic experiments on ranking the Cf-list, the salience of discourse entities in prepended phrases (Gordon, Grosz, and Gilliom 1993) and the ordering of possessor and possessed in complex NPs (Gordon et al. 1999), to try to improve the performance of LRC. |
| W07-1517 386 61:81 The Penn Treebank annotation (Marcus et al. , 1993) was chosen to be the first among equals: it is the starting point for the merger and data from other annotations are attached at tree nodes. |
| W00-1304 387 106:212 The corpus consists of sections 15-18 and section 20 of the Penn Treebank (Marcus et al. , 1993), and is pre-divided into a 8936-sentence (211727 tokens) training set and a 2012-sentence (47377 tokens) test set. |
| W97-0301 388 52:134 3 Probability Model This paper takes a "history-based" approach (Black et al. , 1993) where each tree-building procedure uses a probability model p(alb), derived from p(a, b), to weight any action a based on the available context, or history, b. First, we present a few simple categories of contextual predicates that capture any information in b that is useful for predicting a. Next, the predicates are used to extract a set of features from a corpus of manually parsed sentences. |
| D08-1070 389 69:189 The model was trained on sections 221 from the English Penn Treebank (Marcus et al., 1993). |
| W07-0738 390 73:270 Tag sets for English are derived from the Penn Treebank (Marcus et al. , 1993). |
| W08-2121 391 119:401 html 162 3.1.1 Penn Treebank 3 The Penn Treebank 3 corpus (Marcus et al., 1993) consists of hand-coded parses of the Wall Street Journal (test, development and training) and a small subset of the Brown corpus (W. N. Francis and H. Kucera, 1964) (test only). |
| W08-2121 392 150:401 3.2 Conversion to Dependencies 3.2.1 Syntactic Dependencies There exists no large-scale dependency treebank for English, and we thus had to construct a dependency-annotated corpus automatically from the Penn Treebank (Marcus et al., 1993). |
| N07-2045 393 50:82 We also test our language model using leave-one-out cross-validation on the Penn Treebank (Marcus et al. , 1993) (WSJ), giving us 86.74% accuracy (see Table 1). |
| N07-2045 394 46:82 2.1 Training the model As with (Minnen et al. , 2000), we train the language model on the Penn Treebank (Marcus et al. , 1993). |
| W00-1205 395 7:144 Introduction The Penn Treebank (Marcus et al. 1993) initiated a new paradigm in corpus-based research. |
| W06-1612 396 35:186 A third of the corpus is syntactically parsed as part of the Penn Treebank (Marcus et al. , 1993) 2This type corresponds to Princes (1981; 1992) inferrables. |
| W04-2703 397 6:239 The Penn TreeBank (PTB) is an example of such a resource with worldwide impact on natural language processing (Marcus et al. , 1993). |
| D07-1122 398 9:90 We took part the Multilingual Track of all ten languages provided by the CoNLL-2007 shared task organizers(Hajic et al. , 2004; Aduriz et al. , 2003; Mart et al. , 2007; Chen et al. , 2003; Bohmova et al. , 2003; Marcus et al. , 1993; Johansson and Nugues, 2007; Prokopidis et al. , 2005; Csendes et al. , 2005; Montemagni et al. , 2003; Oflazer et al. , 2003). |
| C96-2187 399 17:96 Successflfl examples of reuse of data resources include: the WordNet thesaurus (Miller el; al. , 1993); the Penn Tree Bank (Marcus et al. , 1993); the Longmans Dictionary of Contemporary English (Summers, 1995). |
| N01-1023 400 147:209 6 Experiment 6.1 Setup The experiments we report were done on the Penn Treebank WSJ Corpus (Marcus et al. , 1993). |
| N01-1023 401 6:209 They train from the Penn Treebank (Marcus et al. , 1993); a collection of 40,000 sentences that are labeled with corrected parse trees (approximately a million word tokens). |
| W07-2216 402 7:259 Figure 1 gives an example dependency graph for the sentence Mr. Tomash will remain as a director emeritus, whichhasbeenextractedfromthe Penn Treebank (Marcus et al. , 1993). |
| P06-3014 403 7:139 1 Introduction Robust statistical syntactic parsers, made possible by new statistical techniques (Collins, 1999; Charniak, 2000; Bikel, 2004) and by the availability of large, hand-annotated training corpora such as WSJ (Marcus et al. , 1993) and Switchboard (Godefrey et al. , 1992), have had a major impact on the field of natural language processing. |
| P07-1120 404 43:209 First, we trained a finitestate shallow parser on base phrases extracted from the Penn Wall St. Journal (WSJ) Treebank (Marcus et al. , 1993). |
| N06-1020 405 68:208 3.3 Corpora Our labeled data comes from the Penn Treebank (Marcus et al. , 1993) and consists of about 40,000 sentences from Wall Street Journal (WSJ) articles 153 annotated with syntactic information. |
| W00-0716 406 14:103 The syntactic and part-of-speech informations were obtained from the part of the corpus processed in the Penn Treebank project (Marcus et al. , 1993). |
| W05-1513 407 108:152 We trained and tested the parser on the Wall Street Journal corpus of the Penn Treebank (Marcus et al. , 1993) using the standard split: sections 2-21 were used for training, section 22 was used for development and tuning of parameters and features, and section 23 was used for testing. |
| P07-1062 408 31:192 The RST-DT consists of 385 documents from the Wall Street Journal, about 176,000 words, which overlaps with the Penn Wall St. Journal (WSJ) Treebank (Marcus et al. , 1993). |
| W95-0101 409 106:188 Unsupervised Learning: Results To test the effectiveness of the above unsupervised learning algorithm, we ran a number of experiments using two different corpora and part of speech tag sets: the Penn Treebank Wall Street Journal Corpus \[Marcus et al. , 1993\] and the original Brown Corpus \[Francis and Kucera, 1982\]. |
| W95-0101 410 80:188 Below is an example of the initial-state tagging of a sentence from the Penn Treebank \[Marcus et al. , 1993\], where an underscore is to be read as or. |
| W95-0101 411 27:188 Transformation-based error-driven learning has been applied to a number of natural language problems, including part of speech tagging, prepositional phrase attachment disambiguation, speech generation and syntactic parsing \[Brill, 1992; Brill, 1994; Ramshaw and Marcus, 1994; Roche and Schabes, 1995; Brill and Resnik, 1994; Huang et al. , 1994; Brill, 1993a; Brill, 1993b\]. |
| W95-0101 412 16:188 \[Francis and Kucera, 1982; Marcus et al. , 1993\]), training on a corpus of one type and then applying the tagger to a corpus of a different type usually results in a tagger with low accuracy \[Weischedel et al. , 1993\]. |
| W95-0101 413 6:188 Almost all of the work in the area of automatically trained taggers has explored Markov-model based part of speech tagging \[Jelinek, 1985; Church, 1988; Derose, 1988; DeMarcken, 1990; Cutting et al. , 1992; Kupiec, 1992; Charniak et al. , 1993; Weischedel et al. , 1993; Schutze and Singer, 1994; Lin et al. , 1994; Elworthy, 1994; Merialdo, 1995\]. |
| E06-1015 414 122:212 4.1 Experimental Set-up We used two different corpora: PropBank (www.cis.upenn.edu/ace) along with PennTree bank 2 (Marcus et al. , 1993) and FrameNet. |
| W02-2001 415 42:169 2.2 Corpus occurrence In order to get a feel for the relative frequency of VPCs in the corpus targeted for extraction, namely 0 5 10 15 20 25 30 35 40 0 10 20 30 40 50 60 70 VPC types (%) Corpus frequency Figure 1: Frequency distribution of VPCs in the WSJ Tagger correctextracted Prec Rec Ffl=1 Brill 135135 1.000 0.177 0.301 Penn 667800 0.834 0.565 0.673 Table 1: POS-based extraction results the WSJ section of the Penn Treebank, we took a random sample of 200 VPCs from the Alvey Natural Language Tools grammar (Grover et al. , 1993) and did a manual corpus search for each. |
| W02-2001 416 18:169 Any linguistic annotation required during the extraction process, therefore, is produced through automatic means, and it is only for reasons of accessibility and comparability with other research that we choose to work over the Wall Street Journal section of the Penn Treebank (Marcus et al. , 1993). |
| H05-1078 417 17:199 Statistical parsers trained on the Penn Treebank (PTB) (Marcus et al. , 1993) produce trees annotated with bare phrase structure labels (Collins, 1999; Charniak, 2000). |
| A97-1017 418 49:179 2.2.2 ENGLISH TRAINING DATA For training in the English experiments, we used WSJ (Marcus et al. , 1993). |
| P03-2036 419 44:82 (2003) from Sections 2-21 of the Wall Street Journal (WSJ) in the Penn Treebank (Marcus et al. , 1993) and its subsets.3 We then converted them into strongly equivalent HPSG-style grammars using the grammar conversion described in Section 2.1. |
| P03-2036 420 14:82 We performed a comparison between the existing CFG filtering techniques for LTAG (Poller and Becker, 1998) and HPSG (Torisawa et al. , 2000), using strongly equivalent grammars obtained by converting LTAGs extracted from the Penn Treebank (Marcus et al. , 1993) into HPSG-style. |
| C08-1094 421 55:177 Hence our classifier evaluation omits those two word positions, leading to n2 classifications for a string of length n. Table 1 shows statistics from sections 2-21 of the Penn WSJ Treebank (Marcus et al., 1993). |
| P02-1026 422 46:100 Penn Treebank corpus (Marcus et al. , 1993) sections 0-20 were used for training, sections 2124 for testing. |
| W04-2407 423 86:153 Thus, the Penn Treebank of American English (Marcus et al. , 1993) has been used to train and evaluate the best available parsers of unrestricted English text (Collins, 1999; Charniak, 2000). |
| W00-1427 424 86:182 2.5 Evaluation Minnen and Carroll (Under review) report an evaluation of the accuracy of the morphological generator with respect to the CELEX lexical database (version 2.5; Baayen et al. , 1993). |
| W00-1427 425 82:182 The analyser--and therefore the generator-includes exception lists derived from WordNet (version 1.5: Miller et al. , 1993). |
| W00-1427 426 85:182 corpus (Garside et al. , 1987), the Penn Treebank (Marcus et al. , 1993), the SUSANNE corpus (Sampson, 1995), the Spoken English Corpus (Taylor and Knowles, 1988), the Oxford Psycholinguistic Database (Quinlan, 1992), and the "Computer-Usable" version of the Oxford Advanced Learner's Dictionary of Current English (OALDCE; Mitton, 1.9.92). |
| C98-2135 427 114:149 The simplest "period-space-capital_letter" approach works well for simple texts but is rather unreliable for texts with many proper names and at)breviations at the end of sentence as, for instance, the Wall Street Journal (WSJ) corpus ( (Marcus et al., 1993) ). |
| W04-2002 428 14:60 A quick search in the Penn Treebank (Marcus et al. , 1993) shows that about 17% of all sentences contain parentheticals or other sentence fragments, interjections, or unbracketable constituents. |
| P02-1018 429 4:149 Evaluating the algorithm on the output of Charniaks parser (Charniak, 2000) and the Penn treebank (Marcus et al. , 1993) shows that the patternmatching algorithm does surprisingly well on the most frequently occuring types of empty nodes given its simplicity. |
| W05-0302 430 6:179 Introduction The creation of the Penn Treebank (Marcus et al, 1993) and the word sense-annotated SEMCOR (Fellbaum, 1997) have shown how even limited amounts of annotated data can result in major improvements in complex natural language understanding systems. |
| D07-1119 431 130:133 The following treebanks were used for training the parser: (Aduriz et al. , 2003; Bhmov et al. , 2003; Chen et al. , 2003; Haji et al. , 2004; Marcus et al. , 1993; Mart et al. , 2002; Montemagni et al. 2003; Oflazer et al. , 2003; Prokopidis et al. , 2005; Csendes et al. , 2005). |
| D07-1101 432 72:204 To train models, we used projectivized versions of the training dependency trees.2 1We are grateful to the providers of the treebanks that constituted the data for the shared task (Hajic et al. , 2004; Aduriz et al. , 2003; Mart et al. , 2007; Chen et al. , 2003; Bohmova et al. , 2003; Marcus et al. , 1993; Johansson and Nugues, 2007; Prokopidis et al. , 2005; Csendes et al. , 2005; Montemagni et al. , 2003; Oflazer et al. , 2003). |
| W99-0629 433 56:225 The data for all our experiments was extracted from the Penn Treebank II Wall Street Journal (WSJ) corpus (Marcus et al. , 1993). |
| W07-1502 434 16:184 While significant time savings have already been reported on the basis of automatic pre-tagging (e.g. , for POS and parse tree taggings in the Penn TreeBank (Marcus et al. , 1993), or named entity taggings for the Genia corpus (Ohta et al. , 2002)), this kind of pre-processing does not reduce the number of text tokens actually to be considered. |
| W07-1502 435 10:184 After the success in syntactic (Penn TreeBank (Marcus et al. , 1993)) and propositional encodings (Penn PropBank (Palmer et al. , 2005)), more sophisticated semantic data (such as temporal (Pustejovsky et al. , 2003) or opinion annotations (Wiebe et al. , 2005)) and discourse data (e.g. , for anaphora resolution (van Deemter and Kibble, 2000) and rhetorical parsing (Carlson et al. , 2003)) are being generated. |
| N07-1051 436 205:232 ENGLISH GERMAN CHINESE (Marcus et al. , 1993) (Skut et al. , 1997) (Xue et al. , 2002) TrainSet Section 2-21 Sentences 1-18,602 Articles 26-270 DevSet Section 22 18,603-19,602 Articles 1-25 TestSet Section 23 19,603-20,602 Articles 271-300 Table 3: Experimental setup. |
| W03-0402 437 19:239 In recent years, reranking techniques have been successfully applied to the so-called history-based models (Black et al. , 1993), especially to parsing (Collins, 2000; Collins and Duffy, 2002). |
| P05-1023 438 90:176 The most sophisticated of these techniques (such as Support Vector Machines) are unfortunately too computationally expensive to be used on large datasets like the Penn Treebank (Marcus et al. , 1993). |
| P05-1023 439 118:176 5 The Experimental Results We used the Penn Treebank WSJ corpus (Marcus et al. , 1993) to perform empirical experiments on the proposed parsing models. |
| D07-1082 440 124:195 6 Evaluation 6.1 Data The data used for our comparison experiments were developed as part of the OntoNotes project (Hovy et al. , 2006), which uses the WSJ part of the Penn Treebank (Marcus et al. , 1993). |
| W04-0707 441 12:160 2 Detecting Discourse-New Definite Descriptions 2.1 Vieira and Poesio Poesio and Vieira (1998) carried out corpus studies indicating that in corpora like the Wall Street Journal portion of the Penn Treebank (Marcus et al. , 1993), around 52% of DDs are discourse-new (Prince, 1992), and another 15% or so are bridging references, for a total of about 66-67% firstmention. |
| I05-3016 442 65:163 The implementation of the algorithm is one that has a core of code that can run on either the Penn Treebank (Marcus et al. , 1993) or on the Chinese Treebank. |
| P99-1079 443 46:104 3 Evaluation of Algorithms All four algorithms were run on a 3900 utterance subset of the Penn Treebank annotated corpus (Marcus et al. , 1993) provided by Charniak and Ge (1998). |
| P97-1062 444 161:179 with parse action sequences for 40,000 Wall Street Journal sentences derived from the Penn Treebank (Marcus et al. , 1993). |
| W01-1626 445 119:251 One judge annotated allarticles in four datasets of the Wall Street Journal Treebank corpus (Marcus et al. , 1993) (W9-4, W9-10, W9-22, and W933, each approximately 160K words) as well as thecorpusofWall Street Journal articles used in (Wiebe et al. , 1999) (called WSJ-SE below). |
| W01-1626 446 64:251 3 Previous Work on Subjectivity Tagging In previous work (Wiebe et al. , 1999;; Bruce and Wiebe, 1999), a corpus of sentences from the Wall Street Journal Treebank Corpus (Marcus et al. , 1993) was manually annotated with subjectivity classi cations bymultiplejudges. |
| W03-0902 447 19:190 Our work so far has focused on data in the Penn Treebank (Marcus et al. , 1993), particularly the Brown corpus and some examples from the Wall Street Journal corpus. |
| W98-0701 448 137:168 Because our algorithm does not consider the context given by the preceding sentences, we have conducted the following experiment to see to what extent the discourse context could improve the performance of the wordsense disambiguation: Using the semantic concordance files (Miller et al. , 1993), we have counted the occurrences of content words which previously appear in the same discourse file. |
| W98-0701 449 19:168 Both for the training and for the testing of our algorithm, we used the syntactically analysed sentences of the Brown Corpus (Marcus, 1993), which have been manually semantically tagged (Miller et al. , 1993) into semantic concordance files (SemCor). |
| H05-1066 450 208:223 Table 2 shows the results for English projective dependency trees extracted from the Penn Treebank (Marcus et al. , 1993) using the rules of Yamada and Matsumoto (2003). |
| H05-1066 451 14:223 In fact, the largest source of English dependency trees is automatically generated from the Penn Treebank (Marcus et al. , 1993) and is by convention exclusively projective. |
| D07-1099 452 62:90 4 Experiments We evaluated the ISBN parser on all the languages considered in the shared task (Hajic et al. , 2004; Aduriz et al. , 2003; Mart et al. , 2007; Chen et al. , 2003; Bohmova et al. , 2003; Marcus et al. , 1993; Johansson and Nugues, 2007; Prokopidis et al. , 2005; Csendes et al. , 2005; Montemagni et al. , 2003; Oflazer et al. , 2003). |
| W05-1002 453 112:191 PB, available at www.cis.upenn.edu/ace, is used along with the Penn TreeBank 2 (www.cis.upenn.edu /treebank) (Marcus et al. , 1993). |
| W01-0720 454 6:189 CLL has then been applied to a corpus of declarative sentences from the Penn Treebank (Marcus et al. , 1993; Marcus et al. , 1994) on which it has been shown to perform comparatively well with respect to much less psychologically plausible systems, which are significantly more supervised and are applied to somewhat simpler problems. |
| W01-0720 455 161:189 Here, we present experiments performed using two complex corpora, C1 and C2, extracted from the Penn Treebank (Marcus et al. , 1993; Marcus et al. , 1994). |
| W05-0106 456 49:84 A model was trained using Maximum Likelihood from the UPenn Treebank (Marcus et al. , 1993). |
| W01-0712 457 26:210 It consists of sections 15-18 of the Wall Street Journal part of the Penn Treebank II (Marcus et al. , 1993) as training data (211727 tokens) and section 20 as test data (47377 tokens). |
| W01-0712 458 109:210 We have used three different algorithms: the nearest neighbour algorithm IB1IG, which is part of the Timbl software package (Daelemans et al. , 1999), the decision tree learner IGTREE, also from Timbl, and C5.0, a commercial version of the decision tree learner C4.5 (Quinlan, 1993). |
| W07-1217 459 34:230 Empirical evaluation has been done with the ERG on a small set of texts from the Wall Street Journal Section 22 of the Penn Treebank (Marcus et al. , 1993). |
| P99-1051 460 201:212 For instance, the to-PP frame is poorly' represented in the syntactically annotated version of the Penn Treebank (Marcus et al. , 1993). |
| P02-1055 461 30:174 In one experiment, it has to be performed on the basis of the gold-standard, assumed-perfect POS taken directly from the training data, the Penn Treebank (Marcus et al. , 1993), so as to abstract from a particular POS tagger and to provide an upper bound. |
| P02-1055 462 48:174 Our chunks and functions are based on the annotations in the third release of the Penn Treebank (Marcus et al. , 1993). |
| P02-1055 463 126:174 task (Church, 1988; Brill, 1993; Ratnaparkhi, 1996; Daelemans et al. , 1996), and reported errors in the range of 26% are common. |
| J95-4004 464 144:404 Part-of-speech tagging is an active area of research; a great deal of work has been done in this area over the past few years (e.g. , Jelinek 1985; Church 1988; Derose 1988; Hindle 1989; DeMarcken 1990; Merialdo 1994; Brill 1992; Black et al. 1992; Cutting et al. 1992; Kupiec 1992; Charniak et al. 1993; Weischedel et al. 1993; Schutze and Singer 1994). |
| J95-4004 465 154:404 Almost all recent work in developing automatically trained part-of-speech taggers has been on further exploring Markovmodel based tagging (Jelinek 1985; Church 1988; Derose 1988; DeMarcken 1990; Merialdo 1994; Cutting et al. 1992; Kupiec 1992; Charniak et al. 1993; Weischedel et al. 1993; Schutze and Singer 1994). |
| J95-4004 466 11:404 Endemic structural ambiguity, which can lead to such difficulties as trying to cope with the many thousands of possible parses that a grammar can assign to a sentence, can be greatly reduced by adding empirically derived probabilities to grammar rules (Fujisaki et al. 1989; Sharman, Jelinek, and Mercer 1990; Black et al. 1993) and by computing statistical measures of lexical association (Hindle and Rooth 1993). |
| J95-4004 467 10:404 A number of part-of-speech taggers are readily available and widely used, all trained and retrainable on text corpora (Church 1988; Cutting et al. 1992; Brill 1992; Weischedel et al. 1993). |
| W08-1301 468 14:172 First, we noted how frequently WordNet (Fellbaum, 1998) gets used compared to other resources, such as FrameNet (Fillmore et al., 2003) or the Penn Treebank (Marcus et al., 1993). |
| W01-0702 469 15:165 The system is tested on base noun-phrase (NP) chunking using the Wall Street Journal corpus (Marcus et al. , 1993). |
| W07-2204 470 36:69 The sentences included in the gold standard were chosen at random from the BNC, subject to the condition that they contain a verb which does not occur in the training sections of the WSJ section of the PTB (Marcus et al. , 1993). |
| P06-2002 471 62:181 2.2 Generalization pseudocode In order to identify the portions in common between the patterns, and to generalize them, we apply the following pseudocode (Ruiz-Casado et al. , in press): 1All the PoS examples in this paper are done with Penn Treebank labels (Marcus et al. , 1993). |
| P99-1023 472 142:186 The tagger was tested on two corpora-the Brown corpus (from the Treebank II CDROM (Marcus et al. , 1993)) and the Wall Street Journal corpus (from the same source). |
| P99-1023 473 10:186 Much research has been done to improve tagging accuracy using several different models and methods, including: hidden Markov models (HMMs) (Kupiec, 1992), (Charniak et al. , 1993); rule-based systems (Brill, 1994), (Brill, 1995); memory-based systems (Daelemans et al. , 1996); maximum-entropy systems (Ratnaparkhi, 1996); path voting constraint systems (Tiir and Oflazer, 1998); linear separator systems (Roth and Zelenko, 1998); and majority voting systems (van Halteren et al. , 1998). |
| P99-1023 474 7:186 The Penn Treebank documentation (Marcus et al. , 1993) defines a commonly used set of tags. |
| P99-1023 475 48:186 Most work in the area of unknown words and tagging deals with predicting part-of-speech information based on word endings and affixation information, as shown by work in (Mikheev, 1996), (Mikheev, 1997), (Weischedel et al. , 1993), and (Thede, 1998). |
| P99-1023 476 160:186 The MBT (Daelemans et al. , 1996) 180 Tagger Type Standard Trigram (Weischedel et al. , 1993) MBT (Daelemans et al. , 1996) Rule-based (Brill, 1994) Maximum-Entropy (Ratnaparkhi, 1996) Full Second-Order HMM SNOW (Roth and Zelenko, 1998) Voting Constraints (Tiir and Oflazer, 1998) Full Second-Order HMM Known Unknown Overall Open/Closed Lexicon? |
| W00-1309 477 24:149 The data used for all our experiments is extracted from the PENN" WSJ Treebank (Marcus et al. 1993) by the program provided by Sabine Buchholz from Tilbug University. |
| W00-1306 478 24:185 This paper presents an empirical study measuring the effectiveness of our evaluation functions at selecting training sentences from the Wall Street Journal (WSJ) corpus (Marcuset al. , 1993) for inducing grammars. |
| W04-1114 479 59:271 The segmentation is based on the guidelines, given in the Chinese national standard GB13715, (Liu et al. 1993) and the POS tagging specification was developed according to the Grammatical Knowledge-base of contemporary Chinese. |
| W04-1114 480 261:271 Word association norms, mutual information, and lexicography, Computational Linguistics, 16(1): 22-29 Marcus, M. et al. 1993. |
| W04-1114 481 269:271 Collocation Dictionary of Modern Chinese Lexical Words, Business Publisher, China Yuan Liu, et al. 1993. |
| P05-1038 482 29:222 Compared to the Penn Treebank (PTB; Marcus et al. 1993), the POS tagset of the French Treebank is smaller (13 tags vs. 36 tags): all punctuation marks are represented as the single PONCT tag, there are no separate tags for modal verbs, whwords, and possessives. |
| P01-1003 483 99:155 4 Experimental Work A part of the Wall Street Journal (WSJ) which had been processed in the Penn Treebanck Project (Marcus et al. , 1993) was used in the experiments. |
| A00-2023 484 152:154 so they conform to the Penn Treebank corpus (Marcus et al. , 1993) annotation style, and then do experiments using models built with Treebank data. |
| I08-2096 485 5:170 Evaluations are typically carried out on newspaper texts, i.e. on section 23 of the Penn Treebank (PTB) (Marcus et al., 1993). |
| W05-0407 486 141:199 As referring dataset, we used the PropBank corpora available at www.cis.upenn.edu/ace, along with the Penn TreeBank 2 (www.cis.upenn.edu/treebank) (Marcus et al. , 1993). |
| W03-2102 487 77:211 3 Previous Work on Subjectivity Tagging In previous work (Wiebe et al., 1999), a corpus of sentences from the Wall Street Journal Treebank Corpus (Marcus et al., 1993) was manually anno- tated with subjectivity classifications by multiple judges. |
| P05-1012 488 133:209 3 Experiments We tested our methods experimentally on the English Penn Treebank (Marcus et al. , 1993) and on the Czech Prague Dependency Treebank (Hajic, 1998). |
| W05-1008 489 103:160 4.4 Corpora We ran the three syntactic preprocessors over a total of three corpora, of varying size: the Brown corpus (460K tokens) and Wall Street Journal corpus (1.2M tokens), both derived from the Penn Treebank (Marcus et al. , 1993), and the written component of the British National Corpus (98M tokens: Burnard (2000)). |
| W07-1505 490 156:212 Currently, the scheme supports PhraseChunks with subtypes such as NP, VP, PP, or ADJP (Marcus et al. , 1993). |
| W07-1505 491 22:212 The Dublin Core Metadata Initiative3 established a de facto standard for the Semantic Web.4 For (computational) linguistics proper, syntactic annotation schemes, such as the one from the Penn Treebank (Marcus et al. , 1993), or semantic annotations, such as the one underlying ACE (Doddington et al. , 2004), are increasingly being used in a quasi standard way. |
| W07-1505 492 123:212 With respect to already available POS tagsets, the scheme allows corresponding extensions of the supertype POSTag to, e.g., PennPOSTag (for the Penn Tag Set (Marcus et al. , 1993)) or GeniaPOSTag (for the GENIA Tag Set (Ohta et al. , 2002)). |
| C00-1041 493 69:130 5.1 The Prague Dependency Tree Bank (PDT in the sequel), which has been inspired by the build-up of the Penn Treebank (Marcus, Santorini & Marcinkiewicz 1993; Marcus, Kim, Marcinkiewicz et al. 1994), is aimed at a complex annotation of (a part of) the Czech National Corpus (CNC in the sequel), the creation of which is under progress at the Department of Czech National Corpus at the Faculty of Philosophy, Charles University (the corpus currently comprises about 100 million tokens of word forms). |
| W00-1307 494 119:249 3.5 The Experiments We have ran LexTract on the one-millionword English Penn Treebank (Marcus et al. , 1993) and got two Treebank grammars. |
| W04-2403 495 136:209 4 The Experiments For the experiments, we used PropBank (www.cis.upenn.edu/ace) along with PennTreeBank5 2 (www.cis.upenn.edu/treebank) (Marcus et al. , 1993). |
| W99-0623 496 12:158 These three parsers have given the best reported parsing results on the Penn Treebank Wall Street Journal corpus (Marcus et al. , 1993). |
| W05-1512 497 101:162 Data and Parameters To facilitate comparison with previous work, we trained our models on sections 2-21 of the WSJ section of the Penn tree-bank (Marcus et al. , 1993). |
| W02-1009 498 196:288 Each dataset consisted of a collection of flat rules such as Sput!NP put NP PP extracted from the Penn Treebank (Marcus et al. , 1993). |
| P99-1032 499 129:194 Two disjoint corpora are used in steps 2 and 5, both consisting of complete articles taken from the Wall Street Journal Treebank Corpus (Marcus et al. , 1993). |
| D07-1128 500 33:76 We use a hand-written competence grammar, combined with performance-driven disambiguation obtained from the Penn Treebank (Marcus et al. , 1993). |
| D07-1128 501 11:76 We have achieved average results in the CoNLL domain adaptation track open submission (Marcus et al. , 1993; Johansson and Nugues, 2007; Kulick et al. , 2004; MacWhinney, 2000; Brown, 1973). |
| W06-2112 502 91:234 3.1 Results for English We used sections 0 to 12 of the WSJ part of the Penn Treebank (Marcus et al. , 1993) with a total of 24,618 sentences for our experiments. |
| W06-2112 503 100:234 Neither (Hindle and Rooth, 1993) with 67% nor (Ratnaparkhi et al. , 1994) with 59% noun attachment were anywhere close to this figure. |
| W06-2112 504 39:234 But it makes obvious that (Ratnaparkhi et al. , 1994) were tackling a problem different from (Hindle and Rooth, 1993) given the fact that their baseline was at 59% guessing noun attachment (rather than 67% in the Hindle and Rooth experiments).3 Of course, the baseline is not a direct indicator of the difficulty of the disambiguation task. |
| M98-1009 505 26:276 Training Data Our source for syntactically annotated training data was the Penn Treebank (Marcus et al. , 1993). |
| N04-1016 506 149:228 The simplest model of compound noun disambiguation compares the frequencies of the two competing analyses and opts for the most frequent one (Pustejovsky et al. , Model Alta BNC Baseline 63.93 63.93 f (n1;n2) : f (n2;n3) 77.86 66.39 f (n1;n2) : f (n1;n3) 78.68# 65.57 f (n1;n2)= f (n1) : f (n2;n3)= f (n2) 68.85 65.57 f (n1;n2)= f (n2) : f (n2;n3)= f (n3) 70.49 63.11 f (n1;n2)= f (n2) : f (n1;n3)= f (n3) 80.32 66.39 f (n1;n2) : f (n2;n3) (NEAR) 68.03 63.11 f (n1;n2) : f (n1;n3) (NEAR) 71.31 67.21 f (n1;n2)= f (n1) : f (n2;n3)= f (n2) (NEAR) 61.47 62.29 f (n1;n2)= f (n2) : f (n2;n3)= f (n3) (NEAR) 65.57 57.37 f (n1;n2)= f (n2) : f (n1;n3)= f (n3) (NEAR) 75.40 68.03# Table 8: Performance of Altavista counts and BNC counts for compound bracketing (data from Lauer 1995) Model Accuracy Baseline 63.93 Best BNC 68.036 Lauer (1995): adjacency 68.90 Lauer (1995): dependency 77.50 Best Altavista 78.686 Lauer (1995): tuned 80.70 Upper bound 81.50 Table 9: Performance comparison with the literature for compound bracketing 1993). |
| N04-1016 507 114:228 Table 6 shows 3An exception is Golding (1995), who uses the entire Brown corpus for training (1M words) and 3/4 of the Wall Street Journal corpus (Marcus et al. , 1993) for testing. |
| N04-1016 508 144:228 6 Bracketing of Compound Nouns The first analysis task we consider is the syntactic disambiguation of compound nouns, which has received a fair amount of attention in the NLP literature (Pustejovsky et al. , 1993; Resnik, 1993; Lauer, 1995). |
| W02-1017 509 121:176 In one set of experiments, we generated lexicons for PEOPLE and ORGANIZATIONS using 2500 Wall Street Journal articles from the Penn Treebank (Marcus et al. , 1993). |
| P07-1071 510 35:200 The current version of the dataset gives semantic tags for the same sentencesas inthe PennTreebank (Marcuset al. , 1993), whichareexcerptsfromtheWallStreetJournal. |
| W01-1605 511 13:189 The resulting corpus contains 385 documents of American English selected from the Penn Treebank (Marcus et al. , 1993), annotated in the framework of Rhetorical Structure Theory. |
| W01-1605 512 23:189 Previous research has shown that RST trees can play a crucial role in building natural language generation systems (Hovy, 1993; Moore and Paris, 1993; Moore, 1995) and text summarization systems (Marcu, 2000); can be used to increase the naturalness of machine translation outputs (Marcu et al. 2000); and can be used to build essayscoring systems that provide students with discourse-based feedback (Burstein et al. , 2001). |
| N06-1040 513 26:188 Finally, Section 4 reports the results of parsing experiments using our exhaustive k-best CYK parser with the concise PCFGs induced from the Penn WSJ treebank (Marcus et al. , 1993). |
| W05-1511 514 21:178 Probabilistic models where probabilities are assigned to the CFG backbone of the unification-based grammar have been developed (Kasper et al. , 1996; Briscoe and Carroll, 1993; Kiefer et al. , 2002), and the most probable parse is found by PCFG parsing. |
| W05-1511 515 53:178 Most of them were developed for exhaustive parsing, i.e., producing all parse results that are given by the grammar (Matsumoto et al. , 1983; Maxwell and Kaplan, 1993; van Noord, 1997; Kiefer et al. , 1999; Malouf et al. , 2000; Torisawa et al. , 2000; Oepen et al. , 2002; Penn and Munteanu, 2003). |
| P04-1006 516 129:172 The first stage parser is a best-first PCFG parser trained on sections 2 through 22, and 24 of the Penn WSJ treebank (Marcus et al. , 1993). |
| D07-1023 517 191:264 We use as our English corpus the Wall Street Journal (WSJ) portion of the Penn Treebank (Marcus et al. , 1993). |
| P08-1068 518 19:218 We show that our semi-supervised approach yields improvements for fixed datasets by performing parsing experiments on the Penn Treebank (Marcus et al., 1993) and Prague Dependency Treebank (Hajic, 1998; Hajic et al., 2001) (see Sections 4.1 and 4.3). |
| P08-1068 519 93:218 The English experiments were performed on the Penn Treebank (Marcus et al., 1993), using a standard set of head-selection rules (Yamada and Matsumoto, 2003) to convert the phrase structure syntax of the Treebank to a dependency tree representation.6 We split the Treebank into a training set (Sections 221), a development set (Section 22), and several test sets (Sections 0,7 1, 23, and 24). |
| N03-1013 520 8:158 Resources specifying the relations among lexical items such as WordNet (Fellbaum, 1998) and HowNet (Dong, 2000) (among others) have inspired the work of many researchers in NLP (Carpuat et al. , 2002; Dorr et al. , 2000; Resnik, 1999; Hearst, 1998; Voorhees, 1993). |
| N03-1013 521 58:158 3 Building the CatVar The CatVar database was developed using a combination of resources and algorithms including the Lexical Conceptual Structure (LCS) Verb and Preposition Databases (Dorr, 2001), the Brown Corpus section of the Penn Treebank (Marcus et al. , 1993), an English morphological analysis lexicon developed for PC-Kimmo (Englex) (Antworth, 1990), NOMLEX (Macleod et al. , 1998), Longman Dictionary of Contemporary English 2For a deeper discussion and classification of Porter stemmers errors, see (Krovetz, 1993). |
| J95-2001 522 11:410 Rule-based taggers (Brill 1992; Elenius 1990; Jacobs and Zernik 1988; Karlsson 1990; Karlsson et al. 1991; Voutilainen, Heikkila, and Antitila 1992; Voutilainen and Tapanainen 1993) use POS-dependent constraints defined by experienced linguists. |
| J95-2001 523 16:410 (~) 1995 Association for Computational Linguistics Computational Linguistics Volume 21, Number 2 and Mancini 1991; Meteer, Schwartz, and Weischedel 1991; Merialdo 1991; Pelillo, Moro, and Refice 1992; Weischedel et al. 1993; Wothke et al. 1993). |
| J95-2001 524 38:410 When the training text is adequate to estimate the tagger parameters, more efficient stochastic taggers (Dermatas and Kokkinakis 1994; Maltese and Mancini 1991; Weischedel et al. 1993) and training methods can be implemented (Merialdo 1994). |
| J95-2001 525 22:410 Recently, several solutions to the problem of tagging unknown words have been presented (Charniak et al. 1993; Meteer, Schwartz, and Weischedel 1991). |
| J95-2001 526 14:410 Stochastic taggers use both contextual and morphological information, and the model parameters are usually defined or updated automatically from tagged texts (Cerf-Danon and E1-Beze 1991; Church 1988; Cutting et al. 1992; Dermatas and Kokkinakis 1988, 1990, 1993, 1994; Garside, Leech, and Sampson 1987; Kupiec 1992; Maltese * Department of Electrical Engineering, Wire Communications Laboratory (WCL), University of Patras, 265 00 Patras, Greece. |
| J95-2001 527 23:410 Hypotheses for unknown words, both stochastic (Dermatas and Kokkinakis 1993, 1994; Maltese and Mancini 1991; Weischedel et al. 1993), and connectionist (Eineborg and Gamback 1993; Elenius 1990) have been applied to unlimited vocabulary taggers. |
| C00-1009 528 98:138 The part of the 1Release 2 of this data set can be obtained t'rmn the Linguistic Data Consortium with Catalogue number LDC94T4B (http://www.ldc.upenn.edu/ldc/nofranm.html) 2There are 48 labels defined in (Marcus et al. , 1993), however, three of ttmm do not appear in the corpus. |
| W96-0208 529 48:170 Learning to Disambiguate Word Senses Several recent research projects have taken a corpus-based approach to lexical disambiguation (Brown, Della-Pietra, Della-Pietra, & Mercer, 1991; Gale, Church, & Yarowsky, 1992b; Leacock et al. , 1993b; Lehman, 1994). |
| W02-1031 530 60:215 We have observed in several experiments that the number of SuperARVs does not grow signi cantly as training set size increases; the moderate-sized Resource Management corpus (Price et al. , 1988) with 25,168 words produces 328 SuperARVs, compared to 538 SuperARVs for the 1 million word Wall Street Journal (WSJ) Penn Treebank set (Marcus et al. , 1993), and 791 for the 37 million word training set of the WSJ continuous speech recognition task. |
| C98-2229 531 126:208 6 Experiments 6.1 Data preparation Our experiments were conducted with data made available through the Penn Treebank annotation effort (Marcus et al., 1993). |
| C98-2229 532 36:208 By core phrases, we mean the kind of nonrecursive simplifications of the NP and VP that in the literature go by names such as noun/verb groups (Appelt et al., 1993) or chunks, and base NPs (Ibumshaw and Marcus, 1995). |
| W98-1114 533 19:161 Systems which are able to acquire a small number of verbal subcategorisation classes automatically from corpus text have been described by Brent (1991, 1993), and Ushioda et al. |
| W06-1205 534 10:189 1 Introduction A "pain in the neck" (Sag et al. , 2002) for NLP in languages of the Indo-Aryan family (e.g. Hindi-Urdu, Bangla and Kashmiri) is the fact that most verbs (nearly half of all instances in Hindi) occur as complex predicates multi-word complexes which function as a single verbal unit in terms of argument and event structure (Hook, 1993; Butt and Geuder, 2003; Raina and Mukerjee, 2005). |
| W06-1205 535 73:189 4.2 Word alignment We have used IBM models proposed by Brown (Brown et al. , 1993) for word aligning the parallel corpus. |
| C04-1004 536 96:125 Experimentation The corpus used in shallow parsing is extracted from the PENN TreeBank (Marcus et al. 1993) of 1 million words (25 sections) by a program provided by Sabine Buchholz from Tilburg University. |
| C04-1004 537 10:125 In recent years, HMMs have enjoyed great success in many tagging applications, most notably part-of-speech (POS) tagging (Church 1988; Weischedel et al 1993; Merialdo 1994) and named entity recognition (Bikel et al 1999; Zhou et al 2002). |
| P07-1031 538 84:226 RECALL F-SCORE Brackets 89.17 87.50 88.33 Dependencies 96.40 96.40 96.40 Brackets, revised 97.56 98.03 97.79 Dependencies, revised 99.27 99.27 99.27 Table 1: Agreement between annotators few weeks, and increased to about 1000 words per hour after gaining more experience (Marcus et al. , 1993). |
| P07-1031 539 8:226 1 Introduction The Penn Treebank (Marcus et al. , 1993) is perhaps the most in uential resource in Natural Language Processing (NLP). |
| P05-1039 540 18:187 It is available in several formats, and in this paper, we use the Penn Treebank (Marcus et al. , 1993) format of NEGRA. |
| W99-0502 541 4:9 It us widely acknowledged that word sense d~samblguatmn (WSD) us a central problem m natural language processing In order for computers to be able to understand and process natural language beyond simple keyword matching, the problem of d~samblguatmg word sense, or dlscermng the meamng of a word m context, must be effectively dealt with Advances in WSD v, ill have slgmficant Impact on apphcatlons hke information retrieval and machine translation For natural language subtasks hke part-of-speech tagging or s)ntactm parsing, there are relatlvely well defined and agreed-upon cnterm of what it means to have the "correct" part of speech or syntactic structure assigned to a word or sentence For instance, the Penn Treebank corpus (Marcus et al, 1993) pro~ide~,t large repo.~tory of texts annotated w~th partof-speech and s}ntactm structure mformatlon Tv.o independent human annotators can achieve a high rate of agreement on assigning part-of-speech tags to words m a g~ven sentence Unfortunately, th~s us not the case for word sense assignment F~rstly, it is rarely the case that any two dictionaries will have the same set of sense defimtmns for a g~ven word Different d~ctlonanes tend to carve up the "semantic space" m a different way, so to speak Secondly, the hst of senses for a word m a typical dmtmnar~ tend to be rather refined and comprehensive This is especmlly so for the commonly used words which have a large number of senses The sense dustmctmn between the different senses for a commonly used word m a d~ctmnary hke WoRDNET (Miller, 1990) tend to be rather fine Hence, two human annotators may genuinely dusagree m their sense assignment to a word m context The agreement rate between human annotators on word sense assignment us an Important concern for the evaluatmn of WSD algorithms One would prefer to define a dusamblguatlon task for which there us reasonably hlgh agreement between human annotators The agreement rate between human annotators will then form the upper ceiling against whmh to compare the performance of WSD algorithms For instance, the SENSEVAL exerclse has performed a detaded study to find out the raterannotator agreement among ~ts lexicographers taggrog the word senses (Kllgamff, 1998c, Kllgarnff, 1998a, Kflgarrlff, 1998b) 2 A Case Study In this-paper, we examine the ~ssue of raterannotator agreement by comparing the agreement rate of human annotators on a large sense-tagged corpus of more than 30,000 instances of the most frequently occurring nouns and verbs of Enghsh This corpus is the intersection of the WORDNET Semcor corpus (Miller et al, 1993) and the DSO corpus (Ng and Lee, 1996, Ng, 1997), which has been independently tagged wlth the refined senses of WORDNET by two separate groups of human annotators The Semcor corpus us a subset of the Brown corpus tagged with ~VoRDNET senses, and consists of more than 670,000 words from 352 text files Sense taggmg was done on the content words (nouns, ~erbs, adjectives and adverbs) m this subset The DSO corpus consists of sentences drawn from the Brown corpus and the Wall Street Journal For each word w from a hst of 191 frequently occurring words of Enghsh (121 nouns and 70 verbs), sentences containing w (m singular or plural form, and m its various reflectional verb form) are selected and each word occurrence w ~s tagged w~th a sense from WoRDNET There ~s a total of about 192,800 sentences in the DSO corpus m which one word occurrence has been sense-tagged m each sentence The intersection of the Semcor corpus and the DSO corpus thus consists of Brown corpus sentences m which a word occurrence w is sense-tagged m each sentence, where w Is one of.the 191 frequently oc-,currmg English nouns or verbs Since this common pomon has been sense-tagged by two independent groups of human annotators, ~t serves as our data set for investigating inter-annotator agreement in this paper 3 Sentence Matching To determine the extent of inter-annotator agreement, the first step ~s to match each sentence m Semcor to its corresponding counterpart In the DSO corpus This step ~s comphcated by the following factors 1 Although the intersected portion of both corpora came from Brown corpus, they adopted different tokemzatmn convention, and segmentartan into sentences differed sometimes 2 The latest versmn of Semcor makes use of the senses from WORDNET 1 6, whereas the senses used m the DSO corpus were from WoRDNET 15 1 To match the sentences, we first converted the senses m the DSO corpus to those of WORDNET 1 6 We ignored all sentences m the DSO corpus m which a word is tagged with sense 0 or -1 (A word is tagged with sense 0 or -1 ff none of the given senses m WoRDNFT applies ) 4, sentence from Semcor is considered to match one from the DSO corpus ff both sentences are exactl) ldent~cal or ff the~ differ only m the pre~ence or absence of the characters " (permd) or -' (hyphen) For each remaining Semcor sentence, taking into account word ordering, ff 75% or more of the words m the sentence match those in a DSO corpus sentence, then a potential match ~s recorded These i -kctua\[ly, the WORD~q'ET senses used m the DSO corpus were from a shght variant of the official WORDNE'I 1 5 release Th~s ssas brought to our attention after the pubhc release of the DSO corpus potential matches are then manually verffied to ensure that they are true matches and to ~eed out any false matches Using this method of matching, a total of 13,188 sentence-palrs contasnmg nouns and 17,127 sentence-pa~rs containing verbs are found to match from both corpora, ymldmg 30,315 sentences which form the intersected corpus used m our present study 4 The Kappa Statistic Suppose there are N sentences m our corpus where each sentence contains the word w Assume that w has M senses Let 4 be the number of sentences which are assigned identical sense b~ two human annotators Then a simple measure to quantify the agreement rate between two human annotators Is Pc, where Pc, = A/N The drawback of this simple measure is that it does not take into account chance agreement between two annotators The Kappa statistic a (Cohen, 1960) is a better measure of rater-annotator agreement which takes into account the effect of chance agreement It has been used recently w~thm computatmnal hngu~stlcs to measure raterannotator agreement (Bruce and Wmbe, 1998, Carletta, 1996, Veroms, 1998) Let Cj be the sum of the number of sentences which have been assigned sense 3 by annotator 1 and the number of sentences whmh have been assigned sense 3 by annotator 2 Then P~-P~ 1-P~ where M j=l and Pe measures the chance agreement between two annotators A Kappa ~alue of 0 indicates that the agreement is purely due to chance agreement, whereas a Kappa ~alue of 1 indicates perfect agreement A Kappa ~alue of 0 8 and above is considered as mdmatmg good agreement (Carletta, 1996) Table 1 summarizes the inter-annotator agreement on the mtersected corpus The first (becond) row denotes agreement on the nouns (xerbs), wh~le the lass row denotes agreement on all words combined The a~erage ~ reported m the table is a s~mpie average of the individual ~ value of each word The agreement rate on the 30,315 sentences as measured by P= is 57% This tallies with the figure reported ~n our earlier paper (Ng and Lee, 1996) where we performed a quick test on a subset of 5,317 sentences,n the intersection of both the Semcor corpus and the DSO corpus 10 \[\] mm m m m m m mm m m m m mm m m m Type Num of v, ords A N \[ P~ Avg Nouns 121 7,676 13,188 I 0 582 0 300 Verbs 70 9,520 17,127 I 0 555 0 347 All I 191 I 17,196 30,315 I 056T 0317 Table 1 Raw inter-annotator agreement 5 Algorithm Since the rater-annotator agreement on the intersected corpus is not high, we would like to find out how the agreement rate would be affected if different sense classes were in use In this section, we present a greedy search algorithm that can automatmalb derive coarser sense classes based on the sense tags assigned by two human annotators The resulting derived coarse sense classes achmve a higher agreement rate but we still maintain as many of the original sense classes as possible The algorithm is given m Figure 1 The algorithm operates on a set of sentences where each sentence contains an occurrence of the word w whmh has been sense-tagged by two human annotators At each Iteration of the algorithm, tt finds the pair of sense classes Ct and Cj such that merging these two sense classes results in the highest t~ value for the resulting merged group of sense classes It then proceeds to merge Cz and C~ Thin process Is repeated until the ~ value reaches a satisfactory value ~,~t,~, which we set as 0 8 Note that this algorithm is also applicable to deriving any coarser set of classes from a refined set for any NLP tasks in which prior human agreement rate may not be high enough Such NLP tasks could be discourse tagging, speech-act categorization, etc 6 Results For each word w from the list of 121 nouns and 70 verbs, ~e applied the greedy search algorithm to each set of sentences in the intersected corpus contaming w For a subset of 95 words (53 nouns and 42 verbs), the algorithm was able to derive a coarser set of 2 or more senses for each of these 95 words such that the resulting Kappa ~alue reaches 0 8 or higher For the other 96 words, m order for the Kappa value to reach 0 8 or higher, the algorithm collapses all senses of the ~ord to a single (trivial) class Table 2 and 3 summarizes the results for the set of 53 nouns and 42 ~erbs, respectively Table 2 md~cates that before the collapse of sense classes, these 53 nouns have an average of 7 6 senses per noun There is a total of 5,339 sentences in the intersected corpus containing these nouns, of which 3,387 sentences were assigned the same sense by the two groups of human annotators The average Kappa statistic (computed as a simple average of the Kappa statistic of ~he mdlwdual nouns) is 0 463 After the collapse of sense classes by the greedy search algorithm, the average number of senses per noun for these 53 nouns drops to 40 Howe~er, the number of sentences which have been asmgned the same coarse sense by the annotators increases to 5,033 That is, about 94 3% of the sentences have been assigned the same coarse sense, and that the average Kappa statistic has improved to 0 862, mgmfymg high rater-annotator agreement on the derived coarse senses Table3 gl~es the analogous figures for the 42 verbs, agmn mdmatmg that high agreement is achieved on the coarse sense classes den~ed for verbs 7 Discussion Our findings on rater-annotator agreement for word sense tagging indicate that for average language users, it is quite dl~cult to achieve high agreement when they are asked to assign refned sense tags (such as those found in WORDNET) given only the scanty definition entries m the WORDNET dlctionary and a few or no example sentences for the usage of each word sense Thin observation agrees wlth that obtmned m a recent study done by (Veroms, 1998), where the agreement on sense-tagging by naive users was also not hlgh Thus It appears that an average language user is able to process language wlthout needing to perform the task of dlsamblguatmg word sense to a very fine-grained resolutmn as formulated m a tradltlonal dmtlonary In contrast, expert lexicographers tagged the ~ ord sense in the sentences used m the SENSEVAL exerclse, where high rater-annotator agreement was reported There are also fuller dlctlonary entries m the HECTOR dlctlonary used and more e |
| W05-0620 542 73:334 3 Data The data consists of sections of the Wall Street Journal part of the Penn TreeBank (Marcus et al. , 1993), with information on predicate-argument structures extracted from the PropBank corpus (Palmer et al. , 2005). |
| W05-0620 543 64:334 2.2 Closed Challenge Setting The organization provided training, development and test sets derived from the standard sections of the Penn TreeBank (Marcus et al. , 1993) and PropBank (Palmer et al. , 2005) corpora. |
| P06-1064 544 5:185 1 Introduction A number of wide-coverage TAG, CCG, LFG and HPSG grammars (Xia, 1999; Chen et al. , 2005; Hockenmaier and Steedman, 2002a; ODonovan et al. , 2005; Miyao et al. , 2004) have been extracted from the Penn Treebank (Marcus et al. , 1993), and have enabled the creation of widecoverage parsers for English which recover local and non-local dependencies that approximate the underlying predicate-argument structure (Hockenmaier and Steedman, 2002b; Clark and Curran, 2004; Miyao and Tsujii, 2005; Shen and Joshi, 2005). |
| J98-2002 545 20:393 Some of these methods make use of prior knowledge in the form of an existing thesaurus (Resnik 1993a, 1993b; Framis 1994; Almuallim et al. 1994; Tanaka 1996; Utsuro and Matsumoto 1997), while others do not rely on any prior knowledge (Pereira, Tishby, and Lee 1993; Grishman and Sterling 1994; Tanaka 1994). |
| J98-2002 546 270:393 The second approach (Sekine et al. 1992; Chang, Luo, and Su 1992; Resnik 1993a; Grishman and Sterling 1994; Alshawi and Carter 1994) takes triples (verb, prep, noun2) and (nounl, prep, noun2), like those in Table 10, as training data for acquiring semantic knowledge and performs PP-attachment disambiguation on quadruples. |
| D07-1028 547 20:188 When tested on f-structures for all sentences from Section 23 of the Penn Wall Street Journal (WSJ) treebank (Mar267 cus et al. , 1993), the techniques described in this paper improve BLEU score from 66.52 to 68.82. |
| W02-1504 548 17:111 In this paper, we give an overview of NLPWin, a multi-application natural language analysis and generation system under development at Microsoft Research (Jensen et al. , 1993; Gamon et al. , 1997; Heidorn 2000), incorporating analysis systems for 7 languages (Chinese, English, French, German, Japanese, Korean and Spanish). |
| W02-1504 549 52:111 consistency among raters who may have different levels of fluency in the source language, raters are not shown the original French or Spanish sentence (for similar methodologies, see Ringger et al. , 2001; White et al. , 1993). |
| W02-1504 550 7:111 The most common answer is component testing, where the component is compared against a standard of goodness, usually the Penn Treebank for English (Marcus et al. , 1993), allowing a numerical score of precision and recall (e.g. Collins, 1997). |
| C08-1026 551 34:203 For example, in the WSJ corpus, part of the Penn Treebank 3 release (Marcus et al., 1993), the string in (1) is a variation 12-gram since off is a variation nucleus that is tagged preposition (IN) in one corpus occurrence and particle (RP) in another.1 Dickinson (2005) shows that examining those cases with identical local contextin this case, lookingat ward off aresultsinanestimated error detection precision of 92.5%. |
| W08-0614 552 12:43 The current release of PDTB2.0 contains the annotations of 1,808 Wall Street Journal articles (~1 million words) from the Penn TreeBank (Marcus et al. 1993) II distribution and a total of 40,600 discourse connective tokens (Prasad et al. 2008b). |
| W08-0614 553 10:43 1 Introduction Large scale annotated corpora, e.g., the Penn TreeBank (PTB) project (Marcus et al. 1993), have played an important role in text-mining. |
| D07-1131 554 13:154 In this year, CoNLL-2007 shared task (Nivre et al. , 2007) focuses on multilingual dependency parsing based on ten different languages (Hajic et al. , 2004; Aduriz et al. , 2003; Mart et al. , 2007; Chen et al. , 2003; Bhmova et al. , 2003; Marcus et al. , 1993; Johansson and Nugues, 2007; Prokopidis et al. , 2005; Czendes et al. , 2005; Montemagni et al. , 2003; Oflazer et al. , 2003) and domain adaptation for English (Marcus et al. , 1993; Johansson and Nugues, 2007; Kulick et al. , 2004; MacWhinney, 2000; Brown, 1973) without taking the languagespecific knowledge into consideration. |
| C04-1082 555 26:173 The tagger described in this paper is based on the standard Hidden Markov Model architecture (Charniak et al. , 1993; Brants, 2000). |
| C04-1082 556 94:173 3.1 Experiments The model described in section 2 has been tested on the Brown corpus (Francis and Kucera, 1982), tagged with the 45 tags of the Penn treebank tagset (Marcus et al. , 1993), which constitute the initial tagset T0. |
| P02-1034 557 120:185 The Penn Wall Street Journal treebank (Marcus et al. 1993) was used as training and test data. |
| C04-1010 558 10:293 To some extent, this can probably be explained by the strong tradition of constituent analysis in Anglo-American linguistics, but this trend has been reinforced by the fact that the major treebank of American English, the Penn Treebank (Marcus et al. , 1993), is annotated primarily with constituent analysis. |
| C04-1010 559 81:293 The learning algorithm used is the IB1 algorithm (Aha et al. , 1991) with k = 5, i.e. classification based on 5 nearest neighbors.4 Distances are measured using the modified value difference metric (MVDM) (Stanfill and Waltz, 1986; Cost and Salzberg, 1993) for instances with a frequency of at least 3 (and the simple overlap metric otherwise), and classification is based on distance weighted class voting with inverse distance weighting (Dudani, 1976). |
| P98-2182 560 31:152 To identify conjunctions, lists, and appositives, we first parsed the corpus, using an efficient statistical parser (Charniak et al. , 1998), trMned on the Penn Wall Street Journal Treebank (Marcus et al. , 1993). |
| C04-1055 561 58:119 The TRIPS structure generally has more levels of structure (roughly corresponding to levels in X-bar theory) than the Penn Treebank analyses (Marcus et al. , 1993), in particular for base noun phrases. |
| H05-1035 562 126:147 The difference in accuracy between a SVM model applied to RRR dataset (RRR-basic experiment) and the same experiment applied to TB2 dataset (TB2278 Description Accuracy Data Extra Supervision Always noun 55.0 RRR Most likely for each P 72.19 RRR Most likely for each P 72.30 TB2 Most likely for each P 81.73 FN Average human, headwords (Ratnaparkhi et al. , 1994) 88.2 RRR Average human, whole sentence (Ratnaparkhi et al. , 1994) 93.2 RRR Maximum Likelihood-based (Hindle and Rooth, 1993) 79.7 AP Maximum entropy, words (Ratnaparkhi et al. , 1994) 77.7 RRR Maximum entropy, words & classes (Ratnaparkhi et al. , 1994) 81.6 RRR Decision trees (Ratnaparkhi et al. , 1994) 77.7 RRR Transformation-Based Learning (Brill and Resnik, 1994) 81.8 WordNet Maximum-Likelihood based (Collins and Brooks, 1995) 84.5 RRR Maximum-Likelihood based (Collins and Brooks, 1995) 86.1 TB2 Decision trees & WSD (Stetina and Nagao, 1997) 88.1 RRR WordNet Memory-based Learning (Zavrel et al. , 1997) 84.4 RRR LexSpace Maximum entropy, unsupervised (Ratnaparkhi, 1998) 81.9 Maximum entropy, supervised (Ratnaparkhi, 1998) 83.7 RRR Neural Nets (Alegre et al. , 1999) 86.0 RRR WordNet Boosting (Abney et al. , 1999) 84.4 RRR Semi-probabilistic (Pantel and Lin, 2000) 84.31 RRR Maximum entropy, ensemble (McLauchlan, 2001) 85.5 RRR LSA SVM (Vanschoenwinkel and Manderick, 2003) 84.8 RRR Nearest-neighbor (Zhao and Lin, 2004) 86.5 RRR DWS FN dataset, w/o semantic features (FN-best-no-sem) 91.79 FN PR-WWW FN dataset, w/ semantic features (FN-best-sem) 92.85 FN PR-WWW TB2 dataset, best feature set (TB2-best) 93.62 TB2 PR-WWW Table 5: Accuracy of PP-attachment ambiguity resolution (our results in bold) basic experiment) is 2.9%. |
| H05-1035 563 16:147 But if one limits the information used for disambiguation of the PPattachment to include only the verb, the noun representing its object, the preposition and the main noun in the PP, the accuracy for human decision degrades from 93.2% to 88.2% (Ratnaparkhi et al. , 1994) on a dataset extracted from Penn Treebank (Marcus et 273 al. , 1993). |
| D08-1050 564 6:252 1 Introduction Most state-of-the-art wide-coverage parsers are based on the Penn Treebank (Marcus et al., 1993), making such parsers highly tuned to newspaper text. |
| A00-2030 565 96:164 Word features are introduced primarily to help with unknown words, as in (Weischedel et al. 1993). |
| A00-2030 566 36:164 We were already using a generative statistical model for part-of-speech tagging (Weischedel et al. 1993), and more recently, had begun using a generative statistical model for name finding (Bikel et al. 1997). |
| A00-2030 567 124:164 However, because these estimates are too sparse to be relied upon, we use interpolated estimates consisting of mixtures of successively lowerorder estimates (as in Placeway et al. 1993). |
| W06-1636 568 9:210 1 Introduction and Previous Research It is by now commonplace knowledge that accurate syntactic parsing is not possible given only a context-free grammar with standard Penn Treebank (Marcus et al. , 1993) labels (e.g. , S, NP, etc). |
| J01-2004 569 152:462 It has been shown repeatedly--e.g. , Briscoe and Carroll (1993), Charniak (1997), Collins (1997), Inui et al. |
| N03-1030 570 5:187 1 Introduction By exploiting information encoded in human-produced syntactic trees (Marcus et al. , 1993), research on probabilistic models of syntax has driven the performance of syntactic parsers to about 90% accuracy (Charniak, 2000; Collins, 2000). |
| W07-1524 571 67:167 Some of them are based upon syntactic structure, with PropBank (Kingsbury and Palmer, 2003) being one of the most relevant, building the annotation upon the syntactic representation of the TreeBank corpus (Marcus et al. , 1993). |
| W98-1126 572 159:229 The data consists of 2,544 main clauses from the Wall Street Journal Treebank corpus (Marcus et al. , 1993). |
| W00-0905 573 6:116 Introduction Verb subcategorizafion probabilities play an important role in both computational linguistic applications (e.g. Carroll, Minnen, and Briscoe 1998, Charniak 1997, Collins 1996/1997, Joshi and Srinivas 1994, Kim, Srinivas, and Tmeswell 1997, Stolcke et al. 1997) and psycholinguisfic models of language processing (e.g. Boland 1997, Clifton et al. 1984, Ferreira & McClure 1997, Fodor 1978, Garnsey et al. 1997, Jurafsky 1996, MacDonald 1994, Mitchell & Holmes 1985, Tanenhaus et al. 1990, Trueswell et al. 1993). |
| W00-0905 574 29:116 1 Data Data for 64 verbs (shown in Table 1) was collected from three corpora; The British National Corpus (BNC) (http'J/info.ox.ac.uk/bnc/index.html), the Penn Treehank parsed version of the Brown Corpus (Brown), and the Penn Treebank Wall Street Journal corpas (WSJ) (Marcus et al. 1993). |
| W03-0505 575 148:233 Table 3 compares precision, recall, and F scores for our system with CoNLL-2001 results training on sections 15-18 of the Penn Treebank and testing on section 21 (Marcus et al. , 1993). |
| C96-2185 576 46:105 4 Information Base 4.1 Text Corpus Text corpora are essential to statistical modeling, in developing formal theories of the grammars, investigating prosodic phenomena in speech, and evaluating or comparing the adequacy of parsing models (Marcus et al. , 1993). |
| W99-0628 577 54:183 A very impor232 Author Best Hindle and Rooth (1993) 80.0 % Resnik and Hearst (1993) 83.9 % WN Resnik and Hearst (1993) 75.0 % Ratnaparkhi et al. |
| W99-0628 578 89:183 In this data set the 4-tuples of the test and training sets were extracted from Penn Treebank Wall Street Journal \[Marcus et al. 1993\]. |
| W99-0628 579 22:183 Some works \[Woods et al, 1972\], \[Boguraev, 1979\], \[Marcus et al. 1993\] suggested several strategies that based their 231 decision-making on the relationships existing between predicates and argumentswhat \[Katz and Fodor, 1963\] called selectional restrictions. |
| P99-1018 580 109:177 4 The Corpus We used two corpora for our analysis: hospital discharge summaries from 1991 to 1997 from the Columbia-Presbyterian Medical Center, and the January 1996 part of the Wall Street Journal corpus from the Penn TreeBank \[Marcus et al. 1993\]. |
| P99-1018 581 172:177 In the future, we will experiment with semantic (rather than positional) clustering of premoditiers, using techniques such as those proposed in \[Hatzivassiloglou and McKeown 1993; Pereira et al. 1993\]. |
| D07-1127 582 78:167 3 Experiments and Results All experiments were conducted on the treebanks provided in the shared task (Hajic et al. , 2004; Aduriz et al. , 2003; Mart et al. , 2007; Chen et al. , 2003; Bhmov et al. , 2003; Marcus et al. , 1993; Johansson and Nugues, 2007; Prokopidis et al. , 2005; Csendes et al. , 2005; Montemagni et al. , 2003; Oflazer et al. , 2003). |
| A94-1009 583 42:191 Preparing tagged corpora either by hand is labour-intensive and potentially error-prone, and although a semi-automatic approach can be used (Marcus et al. , 1993), it is a good thing to reduce the human involvement as much as possible. |
| P06-1021 584 86:158 For instance, the Penn Treebank policy (Marcus et al. , 1993; Marcus et al. , 1994) is to annotate the lowest node that is unfinished with an -UNF tag as in Figure 4(a). |
| N06-2025 585 70:94 4.2 Experiments on SRL dataset We used two different corpora: PropBank (www.cis.upenn.edu/ace) along with Penn Treebank 2 (Marcus et al. , 1993) and FrameNet. |