KinoSearch::Analysis::Tokenizer

KinoSearch::Analysis::Tokenizer is a customizable tokenizing.
Download

KinoSearch::Analysis::Tokenizer Ranking & Summary

Advertisement

  • Rating:
  • License:
  • Perl Artistic License
  • Price:
  • FREE
  • Publisher Name:
  • Marvin Humphrey
  • Publisher web site:
  • http://search.cpan.org/~creamyg/KinoSearch-0.162/lib/KinoSearch/Search/SearchServer.pm

KinoSearch::Analysis::Tokenizer Tags


KinoSearch::Analysis::Tokenizer Description

KinoSearch::Analysis::Tokenizer is a customizable tokenizing. KinoSearch::Analysis::Tokenizer is a customizable tokenizing.SYNOPSIS my $whitespace_tokenizer = KinoSearch::Analysis::Tokenizer->new( token_re => qr/S+/, ); # or... my $word_char_tokenizer = KinoSearch::Analysis::Tokenizer->new( token_re => qr/w+/, ); # or... my $apostrophising_tokenizer = KinoSearch::Analysis::Tokenizer->new; # then... once you have a tokenizer, put it into a PolyAnalyzer my $polyanalyzer = KinoSearch::Analysis::PolyAnalyzer->new( analyzers => , );Generically, "tokenizing" is a process of breaking up a string into an array of "tokens". # before: my $string = "three blind mice"; # after: @tokens = qw( three blind mice );KinoSearch::Analysis::Tokenizer decides where it should break up the text based on the value of token_re. # before: my $string = "Eats, Shoots and Leaves."; # tokenized by $whitespace_tokenizer @tokens = qw( Eats, Shoots and Leaves. ); # tokenized by $word_char_tokenizer @tokens = qw( Eats Shoots and Leaves );Requirements:· Perl Requirements: · Perl


KinoSearch::Analysis::Tokenizer Related Software

About SoftwareSea