Tokenizer (olingo-odata2-core 2.9.5-SP.5 API)

java.lang.Object
- org.apache.olingo.odata2.core.uri.expression.Tokenizer

```
public class Tokenizer
extends Object
```
Expression tokenizer

Constructor Summary

Constructors
Constructor Description

Tokenizer(String expression)

Method Summary

All Methods Instance Methods Concrete Methods
Modifier and Type	Method	Description
`Tokenizer`	`setFlagWhiteSpace(Boolean flagIncludeWhitespace)`	Inform the Tokenizer whether extra tokens for whitespace characters should be added to the token list or not.
`TokenList`	`tokenize()`	Tokenizes an expression as defined per OData specification

Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

- Constructor Detail
  - Tokenizer
```
public Tokenizer(String expression)
```
- Method Detail
  - setFlagWhiteSpace
```
public Tokenizer setFlagWhiteSpace(Boolean flagIncludeWhitespace)
```
    Inform the Tokenizer whether extra tokens for whitespace characters should be added to the token list or not.
    
    Parameters:
    
    flagIncludeWhitespace - True -> Whitespace token will be added to token list; False otherwise
    
    Returns:
    
    this
  - tokenize
```
public TokenList tokenize()
                   throws TokenizerException,
                          ExpressionParserException
```
    Tokenizes an expression as defined per OData specification
    
    Returns:
    
    Token list
    
    Throws:
    
    TokenizerException
    
    ExpressionParserException