Proceedings of the ACL 2007 Demo and Poster Sessions, pages 57–60, Prague, June 2007. c 2007 Association for Computational Linguistics Support Vector Machines for Query-focused Summarization trained and evaluated on Pyramid data Maria Fuentes TALP Research Center Universitat Polit`ecnica de Catalunya mfuentes@lsi.upc.edu Enrique Alfonseca Computer Science Departament Universidad Aut´onoma de Madrid Enrique.Alfonseca@gmail.com Horacio Rodr ´ ıguez TALP Research Center Universitat Polit`ecnica de Catalunya horacio@lsi.upc.edu Abstract This paper presents the use of Support Vector Machines (SVM) to detect rele- vant information to be included in a query- focused summary. Several SVMs are trained using information from pyramids of summary content units. Their per- formance is compared with the best per- forming systems in DUC-2005, using both ROUGE and autoPan, an automatic scor- ing method for pyramid evaluation. 1 Introduction Multi-Document Summarization (MDS) is the task of condensing the most relevant information from several documents in a single one. In terms of the DUC contests 1 , a query-focused summary has to provide a “brief, well-organized, fluent answer to a need for information”, described by a short query (two or three sentences). DUC participants have to synthesize 250-word sized summaries for fifty sets of 25-50 documents in answer to some queries. In previous DUC contests, from 2001 to 2004, the manual evaluation was based on a comparison with a single human-written model. Much information in the evaluated summaries (both human and auto- matic) was marked as “related to the topic, but not directly expressed in the model summary”. Ideally, this relevant information should be scored during the evaluation. The pyramid method (Nenkova and Pas- sonneau, 2004) addresses the problem by using mul- tiple human summaries to create a gold-standard, 1 http://www-nlpir.nist.gov/projects/duc/ and by exploiting the frequency of information in the human summaries in order to assign importance to different facts. However, the pyramid method re- quires to manually matching fragments of automatic summaries (peers) to the Semantic Content Units (SCUs) in the pyramids. AutoPan (Fuentes et al., 2005), a proposal to automate this matching process, and ROUGE are the evaluation metrics used. As proposed by Copeck and Szpakowicz (2005), the availability of human-annotated pyramids con- stitutes a gold-standard that can be exploited in or- der to train extraction models for the summary au- tomatic construction. This paper describes several models trained from the information in the DUC- 2006 manual pyramid annotations using Support Vector Machines (SVM). The evaluation, performed on the DUC-2005 data, has allowed us to discover the best configuration for training the SVMs. One of the first applications of supervised Ma- chine Learning techniques in summarization was in Single-Document Summarization (Ishikawa et al., 2002). Hirao et al. (2003) used a similar approach for MDS. Fisher and Roark (2006)’s MDS system is based on perceptrons trained on previous DUC data. 2 Approach Following the work of Hirao et al. (2003) and Kazawa et al. (2002), we propose to train SVMs for ranking the candidate sentences in order of rele- vance. To create the training corpus, we have used the DUC-2006 dataset, including topic descriptions, document clusters, peer and manual summaries, and pyramid evaluations as annotated during the DUC- 2006 manual evaluation. From all these data, a set 57 of relevant sentences is extracted in the following way: first, the sentences in the original documents are matched with the sentences in the summaries (Copeck and Szpakowicz, 2005). Next, all docu- ment sentences that matched a summary sentence containing at least one SCU are extracted. Note that the sentences from the original documents that are not extracted in this way could either be positive (i.e. contain relevant data) or negative (i.e. irrelevant for the summary), so they are not yet labeled. Finally, an SVM is trained, as follows, on the annotated data. Linguistic preprocessing The documents from each cluster are preprocessed using a pipe of general purpose processors performing tokenization, POS tagging, lemmatization, fine grained Named Enti- ties (NE)s Recognition and Classification, anaphora resolution, syntactic parsing, semantic labeling (us- ing WordNet synsets), discourse marker annotation, and semantic analysis. The same tools are used for the linguistic processing of the query. Using these data, a semantic representation of the sentence is produced, that we call environment. It is a semantic- network-like representation of the semantic units (nodes) and the semantic relations (edges) holding between them. This representation will be used to compute the (Fuentes et al., 2006) lexico-semantic measures between sentences. Collection of positive instances As indicated be- fore, every sentence from the original documents matching a summary sentence that contains at least one SCU is considered a positive example. We have used a set of features that can be classified into three groups: those extracted from the sentences, those that capture a similarity metric between the sentence and the topic description (query), and those that try to relate the cohesion between a sentence and all the other sentences in the same document or collection. The attributes collected from the sentences are: • The position of the sentence in its document. • The number of sentences in the document. • The number of sentences in the cluster. • Three binary attributes indicating whether the sentence contains positive, negative and neutral discourse markers, respectively. For instance, what’s more is positive, while for example and incidentally indicate lack of relevance. • Two binary attributes indicating whether the sentence contains right-directed discourse markers (that affect the relevance of fragment after the marker, e.g. first of all), or discourse markers affecting both sides, e.g. that’s why. • Several boolean features to mark whether the sentence starts with or contains a particular word or part-of-speech tag. • The total number of NEs included in the sen- tence, and the number of NEs of each kind. • SumBasic score (Nenkova and Vanderwende, 2005) is originally an iterative procedure that updates word probabilities as sentences are se- lected for the summary. In our case, word prob- abilities are estimated either using only the set of words in the current document, or using all the words in the cluster. The attributes that depend on the query are: • Word-stem overlapping with the query. • Three boolean features indicating whether the sentence contains a subject, object or indirect object dependency in common with the query. • Overlapping between the environment predi- cates in the sentence and those in the query. • Two similarity metrics calculated by expanding the query words using Google. • SumFocus score (Vanderwende et al., 2006). The cohesion-based attributes 2 are: • Word-stem overlapping between this sentence and the other sentences in the same document. • Word-stem overlapping between this sentence and the other sentences in the same cluster. • Synset overlapping between this sentence and the other sentences in the same document. • Synset overlapping with other sentences in the same collection. Model training In order to train a traditional SVM, both positive and negative examples are nec- essary. From the pyramid data we are able to iden- tify positive examples, but there is not enough ev- idence to classify the remaining sentences as posi- tive or negative. Although One-Class Support Vec- tor Machine (OSVM) (Manevitz and Yousef, 2001) can learn from just positive examples, according to Yu et al. (2002) they are prone to underfitting and overfitting when data is scant (which happens in 2 The mean, median, standard deviation and histogram of the overlapping distribution are calculated and included as features. 58 this case), and a simple iterative procedure called Mapping-Convergence (MC) algorithm can greatly outperform OSVM (see the pseudocode in Figure 1). Input: positive examples, P OS, unlabeled examples U Output: hypothesis at each iteration h ′ 1 , h ′ 2 , , h ′ k 1. Train h to identify “strong negatives” in U : N 1 := examples from U classified as negative by h P 1 := examples from U classified as positive by h 2. Set NEG := ∅ and i := 1 3. Loop until N i = ∅, 3.1. N EG := NEG ∪ N i 3.2. Train h ′ i from P OS and NEG 3.3. Classify P i by h ′ i : N i+1 = examples from P i classified as negative P i+1 = examples from P i classified as positive 5. Return {h ′ 1 , h ′ 2 , , h ′ k } Figure 1: Mapping-Convergence algorithm. The MC starts by identifying a small set of in- stances that are very dissimilar to the positive exam- ples, called strong negatives. Next, at each iteration, a new SVM h ′ i is trained using the original positive examples, and the negative examples found so far. The set of negative instances is then extended with the unlabeled instances classified as negative by h ′ i . The following settings have been tried: • The set of positive examples has been collected either by matching document sentences to peer summary sentences (Copeck and Szpakowicz, 2005) or by matching document sentences to manual summary sentences. • The initial set of strong negative examples for the MC algorithm has been either built auto- matically as described by Yu et al. (2002), or built by choosing manually, for each cluster, the two or three automatic summaries with lowest manual pyramid scores. • Several SVM kernel functions have been tried. For training, there were 6601 sentences from the original documents, out of which around 120 were negative examples and either around 100 or 500 pos- itive examples, depending on whether the document sentences had been matched to the manual or the peer summaries. The rest were initially unlabeled. Summary generation Given a query and a set of documents, the trained SVMs are used to rank sen- tences. The top ranked ones are checked to avoid re- dundancy using a percentage overlapping measure. 3 Evaluation Framework The SVMs, trained on DUC-2006 data, have been tested on the DUC-2005 corpus, using the 20 clus- ters manually evaluated with the pyramid method. The sentence features were computed as described before. Finally, the performance of each system has been evaluated automatically using two differ- ent measures: ROUGE and autoPan. ROUGE, the automatic procedure used in DUC, is based on n-gram co-occurrences. Both ROUGE-2 (henceforward R-2) and ROUGE-SU4 (R-SU4) has been used to rank automatic summaries. AutoPan is a procedure for automatically match- ing fragments of text summaries to SCUs in pyra- mids, in the following way: first, the text in the SCU label and all its contributors is stemmed and stop words are removed, obtaining a set of stem vectors for each SCU. The system summary text is also stemmed and freed from stop words. Next, a search for non-overlapping windows of text which can match SCUs is carried. Each match is scored taking into account the score of the SCU as well as the number of matching stems. The solution which globally maximizes the sum of scores of all matches is found using dynamic programming techniques. According to Fuentes et al. (2005), autoPan scores are highly correlated to the manual pyramid scores. Furthermore, autoPan also correlates well with man- ual responsiveness and both ROUGE metrics. 3 3.1 Results Positive Strong neg. R-2 R-SU4 autoPan peer pyramid scores 0.071 0.131 0.072 (Yu et al., 2002) 0.036 0.089 0.024 manual pyramid scores 0.025 0.075 0.024 (Yu et al., 2002) 0.018 0.063 0.009 Table 1: ROUGE and autoPan results using different SVMs. Table 1 shows the results obtained, from which some trends can be found: firstly, the SVMs trained using the set of positive examples obtained from peer summaries consistently outperform SVMs trained using the examples obtained from the man- ual summaries. This may be due to the fact that the 3 In DUC-2005 pyramids were created using 7 manual sum- maries, while in DUC-2006 only 4 were used. For that reason, better correlations are obtained in DUC-2005 data. 59 number of positive examples is much higher in the first case (on average 48,9 vs. 12,75 examples per cluster). Secondly, generating automatically a set with seed negative examples for the M-C algorithm, as indicated by Yu et al. (2002), usually performs worse than choosing the strong negative examples from the SCU annotation. This may be due to the fact that its quality is better, even though the amount of seed negative examples is one order of magnitude smaller in this case (11.9 examples in average). Fi- nally, the best results are obtained when using a RBF kernel, while previous summarization work (Hirao et al., 2003) uses polynomial kernels. The proposed system attains an autoPan value of 0.072, while the best DUC-2005 one (Daum´e III and Marcu, 2005) obtains an autoPan of 0.081. The dif- ference is not statistically significant. (Daum´e III and Marcu, 2005) system also scored highest in re- sponsiveness (manually evaluated at NIST). However, concerning ROUGE measures, the best participant (Ye et al., 2005) has an R-2 score of 0.078 (confidence interval [0.073–0.080]) and an R- SU4 score of 0.139 [0.135–0.142], when evaluated on the 20 clusters used here. The proposed sys- tem again is comparable to the best system in DUC- 2005 in terms of responsiveness, Daum´e III and Marcu (2005)’s R-2 score was 0.071 [0.067–0.074] and R-SU4 was 0.126 [0.123–0.129] and it is better than the DUC-2005 Fisher and Roark supervised ap- proach with an R-2 of 0.066 and an R-SU4 of 0.122. 4 Conclusions and future work The pyramid annotations are a valuable source of information for training automatically text sum- marization systems using Machine Learning tech- niques. We explore different possibilities for apply- ing them in training SVMsto rank sentences in order of relevance to the query. Structural, cohesion-based and query-dependent features are used for training. The experiments have provided some insights on which can be the best way to exploit the annota- tions. Obtaining the positive examples from the an- notations of the peer summaries is probably better because most of the peer systems are extract-based, while the manual ones are abstract-based. Also, us- ing a very small set of strong negative example seeds seems to perform better than choosing them auto- matically with Yu et al. (2002)’s procedure. In the future we plan to include features from ad- jacent sentences (Fisher and Roark, 2006) and use rouge scores to initially select negative examples. Acknowledgments Work partially funded by the CHIL project, IST-2004506969. References T. Copeck and S. Szpakowicz. 2005. Leveraging pyramids. In Proc. DUC-2005, Vancouver, Canada. Hal Daum´e III and Daniel Marcu. 2005. Bayesian summariza- tion at DUC and a suggestion for extrinsic evaluation. In Proc. DUC-2005, Vancouver, Canada. S. Fisher and B. Roark. 2006. Query-focused summarization by supervised sentence ranking and skewed word distribu- tions. In Proc. DUC-2006, New York, USA. M. Fuentes, E. Gonz`alez, D. Ferr´es, and H. Rodr´ıguez. 2005. QASUM-TALP at DUC 2005 automatically evaluated with the pyramid based metric autopan. In Proc. DUC-2005. M. Fuentes, H. Rodr´ıguez, J. Turmo, and D. Ferr´es. 2006. FEMsum at DUC 2006: Semantic-based approach integrated in a flexible eclectic multitask summarizer architecture. In Proc. DUC-2006, New York, USA. T. Hirao, J. Suzuki, H. Isozaki, and E. Maeda. 2003. Ntt’s multiple document summarization system for DUC2003. In Proc. DUC-2003. K. Ishikawa, S. Ando, S. Doi, and A. Okumura. 2002. Train- able automatic text summarization using segmentation of sentence. In Proc. 2002 NTCIR 3 TSC workshop. H. Kazawa, T. Hirao, and E. Maeda. 2002. Ranking SVM and its application tosentence selection. In Proc. 2002 Workshop on Information-Based Induction Science (IBIS-2002). L.M. Manevitz and M. Yousef. 2001. One-class SVM for docu- ment classification. Journal of Machine Learning Research. A. Nenkova and R. Passonneau. 2004. Evaluating content se- lection in summarization: The pyramid method. In Proc. HLT/NAACL 2004, Boston, USA. A. Nenkova and L. Vanderwende. 2005. The impact of frequency on summarization. Technical Report MSR-TR- 2005-101, Microsoft Research. L. Vanderwende, H. Suzuki, and C. Brockett. 2006. Mi- crosoft research at DUC 2006: Task-focused summarization with sentence simplification and lexical expansion. In Proc. DUC-2006, New York, USA. S. Ye, L. Qiu, and T.S. Chua. 2005. NUS at DUC 2005: Under- standing documents via concept links. In Proc. DUC-2005. H. Yu, J. Han, and K. C-C. Chang. 2002. PEBL: Positive example-based learning for web page classification using SVM. In Proc. ACM SIGKDD International Conference on Knowledge Discovery in Databases (KDD02), New York. 60 . Demo and Poster Sessions, pages 57–60, Prague, June 2007. c 2007 Association for Computational Linguistics Support Vector Machines for Query-focused Summarization. scor- ing method for pyramid evaluation. 1 Introduction Multi-Document Summarization (MDS) is the task of condensing the most relevant information from several