Tokenizer Split text into tokens according to a set of regular expressions you specify
Download

Tokenizer Ranking & Summary

Tokenizer Tags

develop learn RealBasic realbasic development realbasic example learn REALbasic

Tokenizer Description

Split text into tokens according to a set of regular expressions you specify The REALbasic Tokenizer class allows developers to easily split text into tokens according to a set of regular expressions you specify. Tokenizer can be used for a variety of purposes - for parsing data files, or even as part of the first stage for a programming language parser. Its functionality is similar to the LEXX library from the Unix world. The Tokenizer test application demonstrates using the Tokenizer class to parse a simple BASIC program into its constituent tokens, which are displayed in a list beneath the text entry field. The example program contains an intentional error - the word "nteger" should be "integer" - which serves to demonstrate how the Tokenizer handles unexpected tokens.The application is set to re-tokenize the text whenever it is changed, so try editing the program and observe how the token list is regenerated. Requirements: · REALbasic What's New in This Release: · Fixed issue with the use of \\ to match the \ character immediately before a ( or [ bracket.

Tokenizer Related Software