String and StringTokenizer

Question

What is difference between String and StringTokenizer?

Lakshmanan Sethu · Answer

@Vulpes yeah it is related to Java.

Munesh Sharma · Answer

Obviously StringTokenizer as it is suggested by its name tokenizes a String supplied to it as an argument to its constructor and the character based on which tokens of that string are to be made.The default tokenizing character is space  . You can think of the StringTokenizer as a specialized form of StreamTokenizer  where each token that is returned is a String . Depending upon what you want to do with the resultant tokens and/or how complex of a parse job you want to deal with lets you decide which to use. The StringTokenizer parses a string into a series of tokens. It is then your responsibility to figure out what each token returned is beyond the definition of a token separated by delimiters. Usually you just treat things as words when using StringTokenizer . On the other hand a StreamTokenizer allows you to ask is the next token a number word quoted string end-of-file end-of-line comment or whitespace. In this case the StreamTokenizer is smarter though can be considerably harder to use.

Vulpes · Answer

@Jerome Note that this question relates to Java not C# (and Lakshmanan has answered accordingly) because .NET doesn't have a StringTokenizer class.

Lakshmanan Sethu · Answer

Hi Please find answers below -> String.split() and Pattern.split() give you an easy syntax for doing the latter but that's essentially all that they do. If you want to parse the resulting strings or change the delimiter halfway through depending on a particular token they won't help you with that. -> StringTokenizer is even more restrictive than String.split() and also a bit fiddlier to use. It is essentially designed for pulling out tokens delimited by fixed substrings. Because of this restriction it's about twice as fast as String.split(). (See my comparison of String.split() and StringTokenizer.) It also predates the regular expressions API of which String.split() is a part. You'll note from my timings that String.split() can still tokenize thousands of strings in a few milliseconds on a typical machine. In addition it has the advantage over StringTokenizer that it gives you the output as a string array which is usually what you want. Using an Enumeration as provided by StringTokenizer is too syntactically fussy most of the time. From this point of view StringTokenizer is a bit of a waste of space nowadays and you may as well just use