Programming Tips - How can I convert a string into tokens?

Date: 2012jun20
Language: Java
Keywords: token, tokenize

Q.  How can I convert a string into tokens?

A.  Java has these classes/methods builtin:

String.split()
java.io.StreamTokenizer
java.util.StringTokenizer
java.util.Scanner

If you want something fancier look for a "lexer".

Or look for a parser for the language you are dealing with.
eg JSON, XML, RSS etc have parsers.