Date: 2012jun20 Language: Java Keywords: token, tokenize Q. Java: Convert a string into tokens A. Java has these classes/methods builtin:

String.split() java.io.StreamTokenizer java.util.StringTokenizer java.util.Scanner

If you want something fancier look for a "lexer". Or look for a parser for the language you are dealing with. eg JSON, XML, RSS etc have parsers.