The VocabularyCharacters class manages the data vocabulary of characters. More...

Inheritance diagram for MyCaffe.layers.gpt.VocabularyCharacter:

Public Member Functions
	VocabularyCharacter (Random random, bool bAddBos, bool bAddEos, bool bEnablePad)
	The constructor. More...

void	Add (char ch)
	Adds a new character to the vocabulary. More...

void	Add (string str)
	Add a string of characters to the vocabulary. More...

int	Build ()
	Builds the vocabulary from all characters added. More...

int	BuildFromString (string strData)
	Build the vocabulary from a string. More...

int[]	CreateTarget (int[] rgSrc)
	Create a target that is offset from the source by one and ends with a EOS. More...

List< int >	Tokenize (string str1, bool bMustExist=true)
	Tokenize a character into its corresponding index token. More...

int[]	Tokenize (string str, bool bAddBos, bool bAddEos)
	Tokenize a string of data. More...

string	Detokenize (int nIdxToken, bool bIgnoreBos, bool bIgnoreEos)
	Detokenize an index token into its corresponding character. More...

string	Detokenize (float[] rgf, bool bIgnoreBos, bool bIgnoreEos)
	Detokenize an array into a string. More...

Properties
int?	Count `[get]`
	Returns the size of the vocabulary. More...

char	BOS `[get]`
	Returns the special BOS character. More...

char	EOS `[get]`
	Returns the special EOS character. More...

Properties inherited from MyCaffe.layers.gpt.IVocabulary
int	Count `[get]`
	Returns the size of the vocabulary. More...

char	BOS `[get]`
	Returns the special BOS character. More...

char	EOS `[get]`
	Returns the special EOS character. More...

Detailed Description

The VocabularyCharacters class manages the data vocabulary of characters.

Definition at line 12 of file VocabularyCharacter.cs.

Constructor & Destructor Documentation

◆ VocabularyCharacter()

MyCaffe.layers.gpt.VocabularyCharacter.VocabularyCharacter	(	Random	random,
		bool	bAddBos,
		bool	bAddEos,
		bool	bEnablePad
	)

The constructor.

Parameters

random	Specifies the random number generator used.
bAddBos	Specifies to include the special BOS character in the vocabulary.
bAddEos	Specifies to include the special EOS character in the vocabulary.
bEnablePad	Specifies to enable the 0 based padding by adding the 0 pad key to the vocabulary.

Definition at line 28 of file VocabularyCharacter.cs.

Member Function Documentation

◆ Add() [1/2]

void MyCaffe.layers.gpt.VocabularyCharacter.Add ( char ch )

Adds a new character to the vocabulary.

Parameters

ch	Specifies the character

Definition at line 54 of file VocabularyCharacter.cs.

◆ Add() [2/2]

void MyCaffe.layers.gpt.VocabularyCharacter.Add ( string str )

Add a string of characters to the vocabulary.

Parameters

str	Specifies the string to add.

Implements MyCaffe.layers.gpt.IVocabulary.

Definition at line 64 of file VocabularyCharacter.cs.

◆ Build()

int MyCaffe.layers.gpt.VocabularyCharacter.Build ( )

Builds the vocabulary from all characters added.

Returns: The vocabulary size is returned.

Implements MyCaffe.layers.gpt.IVocabulary.

Definition at line 76 of file VocabularyCharacter.cs.

◆ BuildFromString()

int MyCaffe.layers.gpt.VocabularyCharacter.BuildFromString ( string strData )

Build the vocabulary from a string.

Parameters

strData Specifies the data to build the vocabulary from.

Returns: The vocabulary size is returned.

Implements MyCaffe.layers.gpt.IVocabulary.

Definition at line 100 of file VocabularyCharacter.cs.

◆ CreateTarget()

int[] MyCaffe.layers.gpt.VocabularyCharacter.CreateTarget ( int[] rgSrc )

Create a target that is offset from the source by one and ends with a EOS.

Parameters

rgSrc Specifies the source to create the target from.

Returns: The tokenized target is returned.

Implements MyCaffe.layers.gpt.IVocabulary.

Definition at line 131 of file VocabularyCharacter.cs.

◆ Detokenize() [1/2]

string MyCaffe.layers.gpt.VocabularyCharacter.Detokenize	(	float[]	rgf,
		bool	bIgnoreBos,
		bool	bIgnoreEos
	)

Detokenize an array into a string.

Parameters

rgf	Specifies the array of tokens to detokenize.
bIgnoreBos	Specifies to ignore the BOS token.
bIgnoreEos	Specifies to ignore the EOS token.

Returns: The detokenized string is returned.

Implements MyCaffe.layers.gpt.IVocabulary.

Definition at line 235 of file VocabularyCharacter.cs.

◆ Detokenize() [2/2]

string MyCaffe.layers.gpt.VocabularyCharacter.Detokenize	(	int	nIdxToken,
		bool	bIgnoreBos,
		bool	bIgnoreEos
	)

Detokenize an index token into its corresponding character.

Parameters

nIdxToken	Specifies the token to detokenize.
bIgnoreBos	Specifies to ignore the BOS token.
bIgnoreEos	Specifies to ignore the EOS token.

Returns: The detokenized string is returned (which may just be a character).

Implements MyCaffe.layers.gpt.IVocabulary.

Definition at line 199 of file VocabularyCharacter.cs.

◆ Tokenize() [1/2]

int[] MyCaffe.layers.gpt.VocabularyCharacter.Tokenize	(	string	str,
		bool	bAddBos,
		bool	bAddEos
	)

Tokenize a string of data.

Parameters

str	Specifies the string to tokenize.
bAddBos	Specifies to add the BOS at the start of the tokenized data.
bAddEos	Specifies to add the EOS to the end of the tokenized data.

Returns: The array of tokens is returned.

Implements MyCaffe.layers.gpt.IVocabulary.

Definition at line 174 of file VocabularyCharacter.cs.

◆ Tokenize() [2/2]

List< int > MyCaffe.layers.gpt.VocabularyCharacter.Tokenize	(	string	str1,
		bool	bMustExist = `true`
	)

Tokenize a character into its corresponding index token.

Parameters

str1	Specifies a single element (character or word) to tokenize.
bMustExist	Optionally, specifies to throw an error if the item is not in the vocabulary (default = true).

Returns: A list of tokens corresponding to the character is returned (typically just a single token).

Implements MyCaffe.layers.gpt.IVocabulary.

Definition at line 147 of file VocabularyCharacter.cs.

Property Documentation

◆ BOS

char MyCaffe.layers.gpt.VocabularyCharacter.BOS

get

Returns the special BOS character.

Definition at line 113 of file VocabularyCharacter.cs.

◆ Count

int? MyCaffe.layers.gpt.VocabularyCharacter.Count

get

Returns the size of the vocabulary.

Definition at line 45 of file VocabularyCharacter.cs.

◆ EOS

char MyCaffe.layers.gpt.VocabularyCharacter.EOS

get

Returns the special EOS character.

Definition at line 121 of file VocabularyCharacter.cs.

The documentation for this class was generated from the following file:

C:/Data/Data/SS_Projects/Intelligence/GitHub/MyCaffe/MyCaffe.layers.gpt/layers.gpt/VocabularyCharacter.cs

Public Member Functions

Properties

Detailed Description

Constructor & Destructor Documentation

◆ VocabularyCharacter()

Member Function Documentation

◆ Add() [1/2]

◆ Add() [2/2]

◆ Build()

◆ BuildFromString()

◆ CreateTarget()

◆ Detokenize() [1/2]

◆ Detokenize() [2/2]

◆ Tokenize() [1/2]

◆ Tokenize() [2/2]

Property Documentation

◆ BOS

◆ Count

◆ EOS