PERCEIVED PROMINENCE AND THE METRICAL-PROSODIC STRUCTURE OF DUTCH SENTENCES

PostScript version

RTF version

In the past ten to fifteen years different prosodic parsing models have been advanced for the purpose of text-to-speech (TTS) systems. But a close to natural speech sound realization of written texts continues to be an extremely difficult matter. With this paper, we do not intend to resolve the existing problems, but to provide new information on the subject which may lead to improvements of TTS systems.

In comparison with previously proposed prosodic parsing models for Dutch TTS systems (cf. Baart 1987, Dirksen & Quené 1993, and Quené & Kager 1993), we claim that our proposal is easier to implement, and at the same time more in accordance with the metrical variation that is observed in natural sentences. Instead of a set of metrical principles translating syntactic phrase structure rules, our model indirectly comprises syntactic relations by assigning different metrical values to verbs, nouns and modifiers. This metrical variation, in turn, in combination with a set of prosodic well-formedness constraints, gives rise to four degrees of prominence, each of which has to be translated in its proper acoustic values. The earlier-mentioned systems recognize only two degrees of prominence, no prominence, and accent prominence.

Since our corpus of analysis consists of newspaper sentences presented out of their context, the semantic-pragmatic distinction between given and new information is not included in our metrical parsing constraints.

It is generally assumed that prosodic parsing includes accentuation as well as phrasing. The perception task on the basis of which we define the metrical constraints, is restricted to information about accentuation (prominence), however. No constraints are thus formulated that refer to (different strengths of) boundaries.

The speech material is selected from the Dutch Polyphone corpus. This corpus consists of 12500 newspaper sentences. A total of 5000 speakers were asked to read five sentences, to be recorded over the telephone (for more details see Damhuis et al. 1994). From this corpus, we took a random set of 50 sentences for the purpose of a metrical-prosodic analysis. Although the grammatical structure of the sentences varies, they are all declarative.

The 50 newspaper sentences consist, on average, of 10.38 words per sentence, and the average number of syllables per sentence is 18.48, as shown in table 1. About half of the words are function words and the other half are content words. As expected, function words are perceived as being less prominent than content words.

Table 1: Number and means of words and syllables over 50 sentences and per sentence.

The 50 sentences are part of a much larger set of sentences selected by Streefkerk for the purpose of a study on prominence perception (Streefkerk 1997). A first perception experiment, executed by Streefkerk, involved 500 sentences spoken by 50 male and 50 female speakers. Ten listeners, all students from the Humanities Faculty at the University of Amsterdam, were asked to indicate which words were realized with emphasis. The 500 sentences were presented in 4 random order sessions, which differed per listener, to compensate for possible learning effects. The first two sessions contained 150, and the last two sessions contained 125 sentences. The perception experiment was performed on a UNIX workstation, and the results of each listener were automatically stored. While hearing the sentence through closed headphones, the listeners saw the sentence on a monitor. Under each word, on the monitor, a button was placed. The subjects had to click on the button when a given word was perceived as being spoken with emphasis. To test the consistency of the listeners, 50 sentences were presented twice to each listener. This set of 50 sentences is used for the metrical-prosodic analysis.

An example of the perception results for one sentence is given in table 2. The sentence De vliegtuigkaping werd tijdens de vlucht opgelost. ‘The airplane hijacking was solved during the flight’ was scored twice by the 10 listeners. For each word, the 20 judgements are added together, giving rise to a score between 0 (no mark) and 20 (all listeners marked this word twice as emphasized). We assume that the resulting scale of judgments is an indication of the involved degree of prominence: the higher the score the more prominent a given word is.

Table 2: Example of the results of the listening experiment. The table shows the cumulative judgments of the listeners and the resulting degrees of prominence.

It should be mentioned that the listeners differ quite remarkably with respect to the number of emphasized words they perceive in one and the same sentence. While some listeners assign a mean number of 1 prominence per sentence (see e.g. listener 7, table 3), others assign 4 prominences (see e.g. listener 9, table 3). It strikes, for instance, that only 4 times all listeners (20 marks) agree that a certain word is emphasized. These facts argue in favor of a relative, instead of an absolute, metrical representation. That is, a prosodic analysis which rigidly translates the syntactic surface structure into prosodic constituents, as proposed for instance by Nespor & Vogel (1986), leads to an abundance of prosodic heads and boundaries which have no acoustic and perceptional correspondents.

Table 3 also shows the existence of a learning effect. The listeners 4, 6, 8 and 9 mark substantially more words as prominent during the second parsing than during the first one. In Streefkerk & Pols (1998) it is shown that the set of marked words in the first parsing is mostly a subset of the marked words in the second parsing. Although the differences within and between listeners are rather strong, we still consider the cumulative judgements to be a useful alternative for prominence labeling of the speech material.

Table 3: Number of prominence judgments per listener after first and second parsing.

In table 4 we present the distribution of prominence marks according to a threefold distinction: (i) 0 marks, (ii) from 1 to 10 marks, and (iii) from 11 to 20 marks. The mean number of 0 marked words is 4.88 per sentence. This equals the amount of function words per sentence, as illustrated in table 1. The sum of marked words gives rise to a mean of 5.5 words per sentence. This equals the amount of content words per sentence (see table 1).

Table 4: Distribution of prominence marks between 0 and 20.

A final observation must be made here with regard to the listening experiment. Since the listeners were asked to indicate emphasized words, and not emphasized syllables, and they were not asked to indicate degrees of emphasis, the results do not give us information about (i) the location of the emphasis within the word, and (ii) the presence of weakly stressed words/syllables. With respect to (i), in general, the lexically stressed syllable of the word is also the syllable actually realized with prominence. Only in a very few cases, lexical stress shifts can be observed. With respect to (ii), unfortunately the prominence values of polysyllabic function words and of secondary stresses within relatively long content words are not perceptually tested. But in a similar experiment on a different set of sentences from the Polyphone corpus, the task included perceived prominence of words versus perceived prominence of syllables. The listeners indicated a mean number of 2.9 words, but a mean number of 5.1 syllables per sentence as being prominent (Streefkerk et al. 1997). This result seems to confirm that weakly stressed syllables are indeed perceived when the perception task is formulated differently.

Other perception tests in which degrees of prominence were asked for, also indicate that listeners are able to differentiate between unstressed, weakly stressed and strongly stressed syllables (see Helsloot 1993, 1995). In addition, the acoustic signals as well as careful listening of short trunks of the Dutch sentences reveal the presence of different stress degrees.

With the results of the listening experiment on the one hand, and rather elaborated theories of prosodic phonology on the other (cf. among others, Selkirk 1980, 1986), we hypothesized that a mapping of the two would be feasible. The six-level organization assumed by prosodic phonology, i.e. the syllable, the foot, the prosodic word, the phonological phrase, the intonation phrase and the phonological utterance, appeared to be a far too rich as well as a far too rigid system, however. Instead of six levels, only four levels could be distinguished. And the classical assignment of prosodic constituent structure to sentences based on morphological and syntactic information gave rise to absolutely determined heads and constituents which very often were not encountered in the perception results. A relational-based metrical grid representation, as initially proposed by Liberman & Prince (1977), extended by restrictions on the number of hierarchical levels, instead, allows for a more adequate analysis of the metrical structure of the material.

Since we are dealing with read sentences and not with spontaneous speech, almost all syllable inputs are properly realized in the output. Syllable deletion or syllable insertion, well-known phenomena in Dutch spontaneous speech (see Kuijpers & van Donselaar 1998), occur in just a few cases. In the 50 sentences, we found four instances of syllable deletion, and two instances of syllable insertion. The involved vowel is always a schwa:

Generally put, in a sequence of two or more unstressed syllables, the left-most schwa tends to be deleted, and in pre-boundary position, or in a sequence of adjacent stresses, a schwa is inserted if permitted by the segmental environment. In other words, rhythmic lapses and clashes are possibly resolved at the syllable level. Obviously, a TTS system must include these rhythmically-driven syllable deletions and insertions.

Apart from these insertions/deletions, input syllables are realized at the surface. This observation is metrically represented by a mark on the lowest metrical grid level:

(2) Syllable Constraint

All syllables receive a level-1 mark on the metrical grid.

Thus, the sentence in (3) is initially parsed into a sequence of level-1 marks:

Violations of the Sentence-Final Constraint , giving rise to an iambic pattern, do occur, however. Of the 50 metrically analyzed sentences, 7 have an iambic ending. Rhythmic alternation and domain maximality are the underlying reasons here. To start with the latter phenomenon, domain maximality, a sentence-final iamb is created when the preceding content word is ‘too far away’ from the final word. More precisely, the trochee must be realized within a maximum number of syllables. If the syllables intervening the head syllables of the final and pre-final content word exceed the number of four, the input pattern is left unchanged. Two examples are given below (head syllables are given in boldface, and the prominence marks refer to the words in italics):

Between the two head syllables vat and won in (28a) occur five syllables, and between the head syllables vol and tiek in (28b), six syllables. In Helsloot (1995), a chapter is dedicated to prosodic phrase maximality in Italian. Roughly said, in Italian the maximal phrase includes six to seven syllables, parsed into three feet. The final parts of the strings in (28) give rise to such maximal phrases: [dat ze had gewonnen] and [tegen de politiek]. In fact, in addition to the sentence-final verb gewonnen the complementizer dat is perceived by the listeners as having low prominence, and in (b), the disyllabic preposition tegen, although not perceived as bearing prominence, is expected to be realized by a low degree of prominence, on the basis of our Function Word Constraint II . The trochaic pattern required by the Sentence-Final Constraint cannot be realized because of maximality conditions on metrical phrasing. In the prosodic framework we are presenting here, phrases and boundaries are not explicit entities, however. The observed violation of the Sentence-Final Constraint can also be accounted for by a constraint on rhythmic alternation, which refers to alternating stress degrees instead of phrase size.

Prosodic well-formedness mainly involves rhythmic alternation: at the phrasal level stresses must be distributed in accordance to principles of recurrence and hierarchical ordering (cf. amongst others, Prince 1983, Hayes 1984, Halle & Vergnaud 1987, Helsloot 1995). Recurrence is generally assumed to be of a binary nature: a strong beat is followed by a weak beat. This alternation applies at hierarchically ordered levels of organization. In other words, binary alternation is observed from the smallest to the largest domains of prosodic organization. Metrical grids insightfully represent this organization. Well-formed rhythmic alternation is shown in (29a), ill-formed alternations in (29b) and (29c).

Obviously, the words of a language when grouped into phrases do not always give rise to well-formed rhythmically alternating patterns. The input may show up with a so-called stress clash (as illustrated in 29b). Since stress clashes are hard to produce physically, they generally do not occur in the actual realization of the phrase. A clash can be resolved in various ways. By means of (i) deletion of one of the stresses, (ii) movement of one of the stresses, or (iii) lengthening of the interval between the stresses. Whether one resolution or the other is selected depends on the linguistic context. In Dutch, all three resolutions are found. And it is mostly the stress on the right (i.e. the second stress) that will be modified. In case a clash exists between three stresses, things are becoming more complex.

In section 4.2, we referred to instances violating the Sentence-Final Constraint . Instead of a final trochee, an iamb is realized. The violation is due to stress-clash avoidance. Consider the metrical representation of the sentence in (30).

The Sentence-Final Constraint deletes the sentence-final level-3 mark. However, the perceived prominence pattern is 5 on gebrek and 11 on plichtsbesef. A clear iambic pattern. The representation in (30), in fact, gives rise to a clash with the level-4 mark of the preceding word, onthutsend.

In our corpus, about ten instances are found in which a sequence of three content words gives rise to a double stress clash. On the basis of the perceived prominences, we formulate a Clash Resolution Constraint which requires left-headedness at the highest level of organization.

On the basis of this constraint, the above sentence is represented as follows:

That is, the level-3 mark on gebrek will not be realized in the output.

The two relevant constraints are ranked with respect to one another: the Clash Resolution Constraint dominates the Sentence-Final Constraint , which explains the fact that the latter is violated.

Other examples from our corpus subject to the Clash Resolution Constraint , but not in sentence-final position, are given in (33).

The Modifier Constraint assigns a level-4 mark to the modifiers vol and groene in (33a), and aanzienlijk, minder and mensenschuw in (33b). Since these level-4 marks are not separated from each other by intervening level-3 and level-2 marks, they are subject to the Clash Resolution Constraint . This latter constraint outranks the Modifier Constraint , i.e., it is satisfied at the expense of the Modifier Constraint .

The rhythmic counterpart of the clash is the likewise ill-formed lapse, illustrated in (29c) above. The metrical input may give rise to a relatively long sequence of syllables that are only characterized by level-1 marks. Such a pattern gives rise to a monotonous realization. Generally, such patterns are avoided in natural speech. The Lapse Resolution Constraint accounts for this avoidance.

(34) Lapse Resolution Constraint :

In a sequence of more than three level-1 marks a level-2 mark is assigned to the central syllable with a full vowel.

In section 3.2, we mentioned that monosyllabic function words are sometimes perceived by the listeners as being realized with a low degree of prominence. The avoidance of a rhythmic lapse explains this prominence (the mark between square brackets is added):

As said before, listeners were asked to indicate which words in the sentences were realized with emphasis. Word-internal weakly stressed syllables are thus not separately marked. Another kind of perception test is needed in order to get judgements about the presence or absence of weakly stressed syllables. However, TTS models which do not translate weak (or level-2) stresses in acoustic terms, will produce monotonous realizations of such weakly stressed intervals. The Lapse Resolution Constraint prevents such realizations to occur.

The perception-based analysis and the metrical-prosodic analysis of the Dutch read-aloud sentences give rise to a metrical grid representation containing four distinct levels of prominence. The lowest level corresponds to the syllable level; the second level to weakly stressed syllables which either belong to a function word, to a non-head syllable of a long content word, or to the head syllable of a verb; the third level corresponds to the head syllable (lexically stressed syllable) of nouns; and the fourth level to the head syllable of adjectives and adverbs, and to negative particles and deictically used demonstrative pronouns. This representation is accounted for by a set of prosodic input constraints: the Syllable Constraint, the Function Word Constraints I and II, the Content Word Constraints I and II, the Compound Constraints, and the Modifier Constraints.

In addition, a number of sentence-level output constraints are formulated which account for the rhythmic well-formedness of the sentences: the Sentence-Initial Constraint, the Sentence-Final Constraint, the Clash Resolution Constraint, the Lapse Resolution Constraint.

The prosodic output constraints are often in conflict with the prosodic input constraints. The surface realizations indicate that the former are higher ranked in the constraint hierarchy than the latter. Output constraints themselves may also be in conflict with one another. For instance, the Sentence-Final Constraint can be violated by the higher ranked Clash Resolution Constraint.

The metrical grid representations resulting from the constraints match extremely well with the presence versus absence of prominences as perceived by the listeners: (i) of the 275 words perceived as prominent only eleven do not receive a proper metrical representation (see the prominence marks assigned to monosyllabic function words in table 5), (ii) and only one word with a level-3 mark (or higher) is not perceived as prominent at all. With respect to level-2, level-3 and level-4 marks on the one hand, and perception marks on the other, it is observed that the correspondences are relatively and locally manifested, but not absolutely. The fact that each sentence was read by just one speaker did not allow us to correct for speaker-dependent pronunciation. The next step is to verify the proposal on the basis of a different set of sentences from the same corpus of read-aloud sentences, as well as on the basis of a corpus which takes into a account the pronunciation of a larger group of speakers.

Baart, J. (1987). Focus, Syntax and Accent Placement . Dissertation, Leiden University.

Booij, G. (1995). The Phonology of Dutch . Oxford University Press.

Damhuis M., Boogaart T., in ‘t Veld C., Versteijlen M., Schelvis W., Bos L., Boves L. (1994). “Creation and analysis of the Dutch Polyphone corpus”, ICSLP 94 , Yokohama 1803 - 1803.

Dirksen, A. & H. Quené (1993). “Prosodic analysis: The next generation”, in V. J. van Heuven & L. C. W. Pols (eds.) Analysis and Synthesis of Speech , Mouton de Gruyter, Berlin-New York, 131-144.

Dirksen, A. & L. Menert (1997). Fluent Dutch Text-to-Speech , Version 1.0, Fluency Speech Technology, Utrecht.

Halle, M. & J-R. Vergnaud (1987). An essay on stress . Cambridge, Mass.: MIT Press.

Hayes, B. (1984). The Phonology of Rhythm in English. Linguistic Inquiry 15. 33-74.

Helsloot, C.J. (1995). Metrical Prosody. A Template-and-Constraint Approach to Phonological Phrasing in Italian . HIL Dissertation 16, HAG, Den Haag.

Kruyt, J. (1985). Accents from Speakers to Listeners . Dissertation , Leiden University.

Kuijpers, C., van Donselaar, M.(1998). “The Influence of Rhythmic Context on Schwa Epenthesis and Schwa Deletion in Dutch”, Language and Speech 41 ( 1), 87-108.

Lea, W. (1979). “Testing linguistic stress rules with listeners’ perception”, in J. Wolf & D.H. Klatt (eds.) Speech Communication Papers presented at the 97th meeting of the ASA , New York.

Liberman, M. & A. Prince (1977). On stress and linguistic rhythm. Linguistic Inquiry 8. 249-336.

Prince, A. (1983). Relating to the grid. Linguistic Inquiry 14. 19-100.

Quené, H. & R. Kager (1993). “Prosodic sentence analysis without exhaustive parsing”, in V. J. van Heuven & L. C. W. Pols (eds.) Analysis and Synthesis of Speech , Mouton de Gruyter, Berlin-New York, 115-130.

Selkirk, E. (1980). “Prosodic domains in phonology: Sanskrit revisited”, in M. Aronoff and M.-L. Kean (eds.), Juncture ( Studia linguistica et philologica 7 ). Saratoga, California: Anma Libri. 107-129.

Selkirk, E. (1986). “On Derived Domains in Sentence Phonology”, Phonology Yearbook 3. 371-405.

Streefkerk B. M. (1997). “Acoustical correlates of prominence: A design for research”, Proceedings of the Institute of Phonetic Sciences of the University of Amsterdam , 21 131-142.

Streefkerk, B. M., Pols, L. C. W. and Ten Bosch, L. F. M. (1997) ”Prominence in read aloud sentences, as marked by listeners and classified automatically”, Proceedings of the Institute of Phonetic Sciences of the University of Amsterdam , 21: 101-116.

Streefkerk, B. M. & L. Pols (1998). “Prominence in read aloud Dutch sentences as marked by naive listeners” Tagungsband KONVENS-98 , Frankfurt a.M., 201-205.

O’Shaughnessy, D. (1976). Modelling Fundamental Frequency and its Relationship to Syntax, Semantics and Phonetics . Dissertation MIT, Cambridge, Mass.

O’Shaughnessy, D & J. Allen (1983). “Linguistic modality effects on fundamental frequency in speech”, Journal of the Acoustic Society of America 74/4, p. 1155-1171.

Visch, E. (1989). A Metrical Theory of Rhythmic Stress Phenomena . Dordrecht: Foris.

[*] Studio Taalwetenschap Helsloot Verrips, Amsterdam.