Many of the approaches in the literature can be differentiated according to how MWE processing is timed with respect to underlying use cases. It also elucidates the interactions between MWE processing and two use cases: Parsing and machine translation. It offers a shared understanding of what is meant by “MWE processing,” distinguishing the subtasks of MWE discovery and identification. We propose a conceptual framework within which challenges and research contributions can be positioned. The emergence of solutions in the absence of guiding principles motivates this survey, whose aim is not only to provide a focused review of MWE processing, but also to clarify the nature of interactions between MWE processing and downstream applications. The issue of MWE handling is crucial for NLP applications, where it raises a number of challenges. The structure of linguistic processing that depends on the clear distinction between words and phrases has to be re-thought to accommodate MWEs.
The obligatory rewrite relation \(R(T)\) is also regular, thus we can build the corresponding transducer.Multiword expressions (MWEs) are a class of linguistic forms spanning conventional word boundaries that are both idiosyncratic and pervasive across different languages. We denote the empty string as \( \epsilon \) and \( \Sigma^* \) as the set of all words over an alphabet \( \Sigma \). For example, the machine in Figure 1 recognizes strings that start with an ‘a’ and are followed by an arbitrary number of ‘b’s. FSAs are used for recognizing patterns in text. The transition relation means that from a given state on a symbol from the alphabet we transition into a set of states (non-determinism). \( \Delta \subseteq Q \times \Sigma \to 2^Q \) is a transition relation.\( F \subseteq Q \) is a set of final (accepting) states.\( I \subseteq Q \) is a set of initial states.\( \Sigma \) is a finite set of symbols (alphabet).If there’s no transition on a given input, the machine terminates.įigure 1: Finite automaton for the expression ab*įormally we denote FSA as a 5-tuple \( A = \langle \Sigma, Q, I, F, \Delta \rangle \) where A state machine has no memory, that is, it does not keep track of the previous states it has been in. It’s always in one of its states and while it reads an input, it switches from state to state. Finite AutomataĪ finite-state automaton (FSA) is an abstract device that has states and transitions between those states. Let’s introduce some definitions before proceeding to the main part. We’ll see how they can be applied to implement powerful text rewrites that replace the matching parts of the input string in a single scan.
In this article, we’ll describe the process of text rewriting from a set-theoretic perspective and define the relevant formalisms. Those APIs, however, often go beyond the realm of regular languages, but rewriters based on regular expressions are still very powerful and can to solve most of the translation tasks that occur in practice. Because such tasks arise quite often in practice, many programming languages provide built-in text rewriting APIs. Applications of this process include text highlighting, annotation, lexical analysis, normalization, etc. Text rewriting is a programming task when parts of a given input string are replaced with other strings.
0 kommentar(er)
