Recognition of tokens Set Theory Recognition of tokens

Recognition of tokens & Set Theory

Recognition of tokens • The main task of lexical analysis is to read input characters in the code and produce tokens. • Lexical analyzer scans the entire source code of the program. It identifies each token one by one. Scanners are usually implemented to produce tokens only when requested by a parser. Here is how this works-

Recognition of tokens

Set Theory • A set is a collection of objects (called members or elements) that is regarded as being a single object. To indicate that an object x is a member of a set A one writes x ∊ A, while x ∉ A indicates that x is not a member of A. • Operations on sets • intersection

Intersection of Set • • The intersection operation is denoted by the symbol ∩. The set A ∩ B—read “A intersection B” or “the intersection of A and B”—is defined as the set composed of all elements that belong to both A and B. The symbol for denoting intersection of sets is ‘∩‘. For example: Let set A = {2, 3, 4, 5, 6} and set B = {3, 5, 7, 9} • • In this two sets, the elements 3 and 5 are common. The set containing these common elements i. e. , {3, 5} is the intersection of set A and B. 1. If A = {2, 4, 6, 8, 10} and B = {1, 3, 8, 4, 6}. Find intersection of two set A and B. Solution: • A ∩ B = {4, 6, 8} Therefore, 4, 6 and 8 are the common elements in both the sets. • •

Union of Set • Union of two given sets is the smallest set which contains all the elements of both the sets. • To find the union of two given sets A and B is a set which consists of all the elements of A and all the elements of B such that no element is repeated. • The symbol for denoting union of sets is ‘∪’. • For example; • Let set A = {2, 4, 5, 6} and set B = {4, 6, 7, 8} • Taking every element of both the sets A and B, without repeating any element, we get a new set = {2, 4, 5, 6, 7, 8}

Concatenation of set and string • The concatenation of two or more numbers is the number formed by concatenating their numerals. For example, the concatenation of 1, 234, and 5678 is 12345678. The value of the result depends on the numeric base, which is typically understood from context. • In formal language theory, a string is defined as a finite sequence of members of an underlying base set; this set is called the alphabet of a string or collection of strings. • The members of the set are called symbols, and are typically thought of as representing letters, characters, or digits.

NULL String or Empty String • Empty string also known as null string means a string with length 0 (zero). It is denoted by λ (lemda) symbol. Note: Empty Language is denoted by symbol Φ (Phi). • For example Φ={ }. • String terminology

String terminology • Substring - A sequence of consecutive symbols from a string. "beg" is a substring of "begin". • Prefix - A substring that comes at the beginning of a string. • Suffix - A substring that comes at the end of the string.