Date: 2012jun20 Language: Java Keywords: token, tokenize Q. How can I convert a string into tokens? A. Java has these classes/methods builtin:

String.split() java.io.StreamTokenizer java.util.StringTokenizer java.util.Scanner

If you want something fancier look for a "lexer". Or look for a parser for the language you are dealing with. eg JSON, XML, RSS etc have parsers.