Copyright © 2008-2017, dave - Code samples on Dave's Brain is licensed under the Creative Commons Attribution 2.5 License. However other material, including English text has all rights reserved.
Date: 2012jun20 Language: Java Keywords: token, tokenize Q. How can I convert a string into tokens? A. Java has these classes/methods builtin:String.split() java.io.StreamTokenizer java.util.StringTokenizer java.util.ScannerIf you want something fancier look for a "lexer". Or look for a parser for the language you are dealing with. eg JSON, XML, RSS etc have parsers.
Add a commentSign in to add a comment