String Tokenization What is String Tokenization The String

String Tokenization Ø What is String Tokenization? Ø The String. Tokenizer class Ø Examples 1

What is String Tokenization? Ø So far we have been reading our input one value at a time. Ø Sometimes it is more natural to read a group of input at a time. Ø For example, when reading records of students from a text file, it is natural to read a whole record at a time. "995432 Al-Suhaim Adil 3. 5" Ø The read. Line() method of the Buffered Reader class can read a group of input as a single string object. Ø The problem is, how do we break this string object into individual words known as tokens? "995432" "Al-Suhaim Adil” "3. 5“ Ø This process is what String tokenization is about. 2

The String. Tokenizer Class Ø Ø The String. Tokenizer class, of the java. util package, is used to break a string object into individual tokens. It has the following constructors: Constructor function String. Tokenizer(String str) Creates a String. Tokenizer object that uses white space characters as delimiters. String. Tokenizer(String str, String delimiters) Creates a String. Tokenizer object that uses the characters in delimiters as separators. String. Tokenizer(String str, String delimiters, boolean return. Tokens) Creates a String. Tokenizer object that uses characters in delimiters as separators and treats separators as tokens. 3

String. Tokenizer Methods Ø The following are the main methods of the String. Tokenizer class: Method String next. Token() throws No. Such. Element. Exception function int Returns the count of tokens in this String. Tokenizer object that are not yet processed by next. Token() -- initially all. count. Tokens() boolean has. More. Tokens() Returns the next token as a string from this String. Tokenizer object. Throws an exception if there are no more tokens. Returns true if there are more tokens not yet processed by next. Token(). 4

How to apply the methods Ø To break a string into tokens, first, a String. Tokenizer object is created. String my. String = "I like Java very much"; String. Tokenizer tokenizer = new String. Tokenizer(my. String); Ø Then any of the following loops can be used to process the tokens: while(tokenizer. has. More. Tokens()){ String token = tokenizer. next. Token(); // process token } or int token. Count = tokenizer. count. Tokens(); for(int k = 1; k <= token. Count; k++){ String token = tokenizer. next. Token(); // process token } 5

Example 1 Ø The following program reads grades from the keyboard and finds the average. The grades are read in one line. import java. io. *; import java. util. String. Tokenizer; public class Tokenizer. Examplel{ public static void main(String[] args)throws IOException{ Buffered. Reader stdin = new Buffered. Reader(new Input. Stream. Reader(System. in)); System. out. print("Enter grades in one line: "); String input. Line = stdin. read. Line(); String. Tokenizer tokenizer = new String. Tokenizer(input. Line); int count = tokenizer. count. Tokens(); double sum = 0; while(tokenizer. has. More. Tokens()) sum += Double. parse. Double(tokenizer. next. Token()); System. out. println("n. The average = "+ sum / count); } } 6

Example 2 Ø This example shows how to use the second constructor of String. Tokenizer class. Ø It tokenizes the words in a string, such that the punctuation characters following the words are not appended to the resulting tokens. import java. util. String. Tokenizer; public class Tokenizer. Example 2{ public static void main(String[] args){ String input. Line = "Hi there, do you like Java? I do; very much. "; String. Tokenizer tokenizer = new String. Tokenizer (input. Line, ", . ? ; : ! trn"); while(tokenizer. has. More. Tokens()) System. out. println(tokenizer. next. Token()); } } Output: Hi there do you like Java I do very much 7

Example 3 Ø This example shows how to use third constructor of String. Tokenizer class. Ø It tokenizes an arithmetic expression based on the operators and returns both the operands and the operators as tokens. import java. util. String. Tokenizer; public class Tokenizer. Example 3{ public static void main(String[] args){ String input. Line = "(2+5)/(10 -1)"; String. Tokenizer tokenizer = new String. Tokenizer(input. Line, “+—*/()", true); while(tokenizer. has. More. Tokens()) System. out. println(tokenizer. next. Token()); } } Output: ( 2 + 5 ) / 10 1 ) 8