Stochastic Gradient Descent solver with momentum updates weights by a linear combination of the negative gradient and the previous weight update. More...

Inheritance diagram for MyCaffe.solvers.SGDSolver< T >:

Public Member Functions
	SGDSolver (CudaDnn< T > cuda, Log log, SolverParameter p, CancelEvent evtCancel, AutoResetEvent evtForceSnapshot, AutoResetEvent evtForceTest, IXDatabaseBase db, IXPersist< T > persist, int nSolverCount=1, int nSolverRank=0, Net< T > shareNet=null, onGetWorkspace getws=null, onSetWorkspace setws=null)
	The SGDSolver constructor. More...

void	PreSolve ()
	Runs the pre-solve which prepares the Solver to start Solving. More...

double	GetLearningRate (int nIterationOverride=-1)
	Return the current learning rate. More...

override double	ApplyUpdate (int nIterationOverride=-1)
	Compute the update values and apply them to the training Net. More...

virtual void	Normalize (int param_id)
	Normalize a learnable Blob of the training Net. More...

virtual void	Regularize (int param_id)
	Regularize a learnable Blob of the training net. More...

virtual void	ComputeUpdateValue (int param_id, double dfRate, int nIterationOverride=-1)
	Compute the SGD update value that will be applied to a learnable blobs in the training Net. More...

virtual void	ClipGradients ()
	Clip the gradients of all learnable blobs in the training Net. More...

Public Member Functions inherited from MyCaffe.solvers.Solver< T >
	Solver (CudaDnn< T > cuda, Log log, SolverParameter p, CancelEvent evtCancel, AutoResetEvent evtForceSnapshot, AutoResetEvent evtForceTest, IXDatabaseBase db, IXPersist< T > persist, int nSolverCount=1, int nSolverRank=0, Net< T > shareNet=null, onGetWorkspace getws=null, onSetWorkspace setws=null)
	The Solver constructor. More...

void	Dispose ()
	Discards the resources (GPU and Host) used by this Solver. More...

bool	ForceOnTrainingIterationEvent ()
	Force an OnTrainingIterationEvent to fire. More...

void	Init (SolverParameter p, Net< T > shareNet=null)
	Initializes the Solver. More...

void	Reset ()
	Reset the iterations of the net. More...

virtual void	Solve (int nIterationOverride=-1, byte[] rgWeights=null, byte[] rgState=null, TRAIN_STEP step=TRAIN_STEP.NONE)
	The main entry of the solver function. In default, iter will be zero. Pass in a non-zero iter number to resume training for a pre-trained net. More...

bool	Step (int nIters, TRAIN_STEP step=TRAIN_STEP.NONE, bool bZeroDiffs=true, bool bApplyUpdates=true, bool bDisableOutput=false, bool bDisableProgress=false, double? dfLossOverride=null, bool? bAllowSnapshot=null)
	Steps a set of iterations through a training cycle. More...

void	Restore (byte[] rgWeights, byte[] rgState, string strSkipBlobTypes=null)
	The restore method simply calls the RestoreSolverState method of the inherited class. More...

void	Snapshot (bool bForced, bool bScheduled, bool bUpdateDatabase=true)
	The snapshot function implements the basic snapshotting utility that stores the learned net. This method calls the SnapshotSolverState method of the inherited class. More...

SnapshotArgs	GetSnapshotArgs (byte[] rgState, byte[] rgWeights, double dfAccuracy, double dfError, int nIteration, SNAPSHOT_WEIGHT_UPDATE_METHOD wtUpdt)
	The GetSnapshotArgs method fills out a snapshot args structure. More...

double	TestAll (int nIterationOverride=-1)
	Run a TestAll by running all test Nets. More...

double	TestDetection (int nIterationOverride=-1, int nTestNetId=0)
	Run an SSD detection test on a given test Net by running it through its iterations. More...

double	TestClassification (int nIterationOverride=-1, int nTestNetId=0)
	Run a test on a given test Net by running it through its iterations. More...

void	UpdateSmoothedLoss (double dfLoss, int nStartIter, int nAverageLoss=0)
	Update the avaraged loss value. More...

Protected Member Functions
override void	dispose ()
	Releases all resources (GPU and Host) used by the Solver. More...

override void	RestoreSolverState (byte[] rgState)
	Restore the state of the Solver. More...

override byte[]	SnapshotSolverState ()
	Take a snapshot of the Solver state. More...

Protected Member Functions inherited from MyCaffe.solvers.Solver< T >
void	InitTrainNet (Net< T > shareNet=null)
	Initializes the Net used by the solver for training. More...

void	InitTestNets ()
	Initializes the Net used by the Solver for testing. More...

Protected Attributes
BlobCollection< T >	m_colHistory = new BlobCollection<T>()
	History maintains the historical momentum data. More...

BlobCollection< T >	m_colTemp = new BlobCollection<T>()
	Update maintains update related data and is not needed in snapshots. More...

Protected Attributes inherited from MyCaffe.solvers.Solver< T >
CudaDnn< T >	m_cuda
	Specifies the instance of CudaDnn used by the Solver that provides a connection to Cuda. More...

Log	m_log
	Specifies the Log for output. More...

SolverParameter	m_param
	Specifies the SolverParameter that defines how the Solver operates. More...

Net< T >	m_net
	Specifies the training Net. More...

List< Net< T > >	m_rgTestNets = new List<Net<T>>()
	Specifies the testing Nets. More...

int	m_nIter
	Specifies the current iteration. More...

int	m_nCurrentStep
	Specifies the current step. More...

List< double >	m_rgLosses = new List<double>()
	Specifies the Losses used to calculate the smoothed Loss. More...

double	m_dfSmoothedLoss = 0
	Specifies the smoothed loss protected for derived classes to use. More...

double?	m_dfIterAccuracy = null
	Specifies the iteration accuracy calculated when a blob exists with the name 'accuracy'. More...

int	m_nSolverCount = 1
	Specifies the Solver count in a multi-GPU training session. More...

int	m_nSolverRank = 0
	Specifies the Solver rank of this solver, where rank == 0 is the root Solver. More...

IXPersist< T >	m_persist
	Specifies the persistance object used to save weight and solver states. More...

double	m_dfLearningRateOverride = 0
	Optionally, specifies a learning rate override (default = 0, which ignores this setting). More...

Properties
BlobCollection< T >	history `[get]`
	Returns the history BlobCollection containing historical momentum data. More...

Properties inherited from MyCaffe.solvers.Solver< T >
double	LearningRateOverride `[getset]`
	Get/set the learning rate override. When 0, this setting is ignored. More...

int	TrainingTimeLimitInMinutes `[getset]`
	Get/set the training time limit in minutes. When set to 0, no time limit is imposed on training. More...

SNAPSHOT_WEIGHT_UPDATE_METHOD	SnapshotWeightUpdateMethod `[getset]`
	Get/set the snapshot weight update method. More...

IXDatabaseBase	Database `[get]`
	Returns the in-memory MyCaffeDatabase used. More...

bool	EnableTesting `[getset]`
	When enabled, the training cycle calls TestAll periodically based on the SolverParameter. Otherwise testing is not performed. More...

bool	EnableBlobDebugging `[getset]`
	When enabled, the OnTrainingIteration event is set extra debugging information describing the state of each Blob used by the Solver. More...

bool	EnableLayerDebugging `[getset]`
	Enable/disable layer debugging which causes each layer to check for NAN/INF on each forward/backward pass and throw an exception when found. More...

bool	EnableBreakOnFirstNaN `[getset]`
	When enabled (requires EnableBlobDebugging = true), the Solver immediately stop training upon detecting the first NaN withing the training Net. More...

bool	EnableDetailedNanDetection `[getset]`
	When enabled (requires EnableBlobDebugging = true), the detailed Nan (and Infinity) detection is perofmed on each blob when training Net. More...

bool	EnableSingleStep `[getset]`
	When enabled (requires EnableBlobDebugging = true), the Solver only runs one training cycle. More...

bool	WeightsUpdated `[getset]`
	Get/set when the weights have been updated. More...

object	Tag `[getset]`
	Returns a generic tag associated with the Solver. More...

Net< T >	TestingNet `[get]`
	Returns the testing Net used by the solver. More...

Net< T >	TrainingNet `[get]`
	Returns the training Net used by the solver. More...

CudaDnn< T >	Cuda `[get]`
	Returns the CudaDnn instance used by the Solver. More...

string	ActiveLabelCounts `[get]`
	Returns a string describing the labels detected in the training along with the % that each label has participated in the training. More...

string	LabelQueryHitPercents `[get]`
	Return the label query hit percentages for the active datasource. More...

string	LabelQueryEpochs `[get]`
	Return the label query epochs for the active datasource. More...

int	CurrentIteration `[get]`
	Returns the current training iteration. More...

int	MaximumIteration `[get]`
	Returns the maximum training iterations. More...

int	TrainingIterations `[get]`
	Returns the current training iterations remaining. More...

int?	TestingIterations `[get]`
	Returns the current testing iterations remaining. More...

int	TrainingIterationOverride `[getset]`
	Get/set the training iteration override. More...

int	TestingIterationOverride `[getset]`
	Get/set the testing iteration override. More...

AutoResetEvent	CompletedEvent `[get]`
	Returns an auto reset event that is set upon training completion. More...

CancelEvent	CancelEvent `[get]`
	Returns the cancel event which when set cancels the current operation run by the Solver. More...

double	smoothed_loss `[get]`
	Returns the smoothed loss. More...

SolverParameter	parameter `[get]`
	Returns the SolverParameter used. More...

Net< T >	net `[get]`
	Returns the main training Net. More...

List< Net< T > >	test_nets `[get]`
	Returns the testing Nets. More...

int	iter `[get]`
	Returns the current training iteration. More...

SolverParameter.SolverType	type `[get]`
	Returns the type of solver. More...

bool	forceSnapshot `[get]`
	Returns whether or not a snapshot has been forced. More...

bool	forceTest `[get]`
	Returns whether or not a test has been forced. More...

int	solver_count `[get]`
	Returns the solver count in a multi-GPU session. More...

int	solver_rank `[get]`
	Returns this Solver's rank in a multi-GPU session. More...

bool?	is_root_solver `[get]`
	Returns whether or not this is the root solver. More...

Additional Inherited Members
Static Public Member Functions inherited from MyCaffe.solvers.Solver< T >
static SGDSolver< T >	Create (CudaDnn< T > cuda, Log log, ProjectEx p, CancelEvent evtCancel, AutoResetEvent evtForceSnapshot, AutoResetEvent evtForceTest, IXDatabaseBase db, IXPersist< T > persist, int nSolverCount=1, int nSolverRank=0, Net< T > shareNet=null, onGetWorkspace getws=null, onSetWorkspace setws=null)
	Create a new Solver based on the project containing the SolverParameter. More...

static SGDSolver< T >	Create (CudaDnn< T > cuda, Log log, SolverParameter solverParam, CancelEvent evtCancel, AutoResetEvent evtForceSnapshot, AutoResetEvent evtForceTest, IXDatabaseBase db, IXPersist< T > persist, int nSolverCount=1, int nSolverRank=0, Net< T > shareNet=null, onGetWorkspace getws=null, onSetWorkspace setws=null)
	Create a new Solver based on the project containing the SolverParameter. More...

Events inherited from MyCaffe.solvers.Solver< T >
EventHandler	OnStart
	The OnStart event fires at the start of each training iteration. More...

EventHandler	OnAborted
	The OnAborted event fires after aborting a training cycle. More...

EventHandler< GradientsReadyArgs >	OnGradientsReady
	The OnGradientsReady event fires after the gradients of a Solver are ready for distribution to other Solvers in a multi-GPU training session. More...

EventHandler< SnapshotArgs >	OnSnapshot
	The OnSnapshot event fires when the Solver detects that a snapshot is needed. More...

EventHandler< TrainingIterationArgs< T > >	OnTrainingIteration
	The OnTrainingIteration event fires at the end of each training iteration. More...

EventHandler< TestingIterationArgs< T > >	OnTestingIteration
	The OnTestingIteration event fires at the end of each testing iteration. More...

EventHandler< TestResultArgs< T > >	OnTestResults
	When specified, the OnTestResults event fires after each single test run. The recipient is responsible for setting the accuracy. More...

EventHandler< TestArgs >	OnTest
	When specified, the OnTest event fires during a TestAll and overrides the call to Test. More...

EventHandler	OnTestStart
	The OnTestStart event fires at the start of each testing iteration. More...

EventHandler< CustomForwardBackArgs< T > >	OnCustomForwardBack
	The OnCustomForwardBack allows for overriding the forward/backward operations within the solver. More...

EventHandler< WorkspaceArgs >	OnGetWorkspace
	Specifies the OnGetWorkspace event that fires when the getWorkspace() function is called by a layer to get a shareable workspace to conserve GPU memory. More...

EventHandler< WorkspaceArgs >	OnSetWorkspace
	Specifies the OnSetWorkspace event that fires when the setWorkspace() function is called by a layer to get a shareable workspace to conserve GPU memory. More...

Detailed Description

Stochastic Gradient Descent solver with momentum updates weights by a linear combination of the negative gradient and the previous weight update.

See also: Stochastic Gradient Descent Wikipedia.

Template Parameters

T	Specifies the base type float or double. Using float is recommended to conserve GPU memory.

Definition at line 21 of file SGDSolver.cs.

Constructor & Destructor Documentation

◆ SGDSolver()

MyCaffe.solvers.SGDSolver< T >.SGDSolver	(	CudaDnn< T >	cuda,
		Log	log,
		SolverParameter	p,
		CancelEvent	evtCancel,
		AutoResetEvent	evtForceSnapshot,
		AutoResetEvent	evtForceTest,
		IXDatabaseBase	db,
		IXPersist< T >	persist,
		int	nSolverCount = `1`,
		int	nSolverRank = `0`,
		Net< T >	shareNet = `null`,
		onGetWorkspace	getws = `null`,
		onSetWorkspace	setws = `null`
	)

The SGDSolver constructor.

Parameters

cuda	Specifies the instance of CudaDnn to use.
log	Specifies the Log for output.
p	Specifies teh SolverParameter.
evtCancel	Specifies a CancelEvent used to cancel the current operation (e.g. training, testing) for which the Solver is performing.
evtForceSnapshot	Specifies an automatic reset event that causes the Solver to perform a Snapshot when set.
evtForceTest	Specifies an automatic reset event that causes teh Solver to run a testing cycle when set.
db	Specifies the in-memory MyCaffeDatabase.
persist	Specifies the peristence used for loading and saving weights.
nSolverCount	Specifies the number of Solvers participating in a multi-GPU session.
nSolverRank	Specifies the rank of this Solver in a multi-GPU session.
shareNet	Optionally, specifies the net to share when creating the training network (default = null, meaning no share net is used).
getws	Optionally, specifies the handler for getting the workspace.
setws	Optionally, specifies the handler for setting the workspace.

Definition at line 55 of file SGDSolver.cs.

Member Function Documentation

◆ ApplyUpdate()

override double MyCaffe.solvers.SGDSolver< T >.ApplyUpdate ( int nIterationOverride = -1 )

virtual

Compute the update values and apply them to the training Net.

Parameters

nIterationOverride Optionally, specifies an iteration override, or -1 which is ignored.

Returns: The learning rate used is returned.

Implements MyCaffe.solvers.Solver< T >.

Definition at line 192 of file SGDSolver.cs.

◆ ClipGradients()

virtual void MyCaffe.solvers.SGDSolver< T >.ClipGradients ( )

virtual

Clip the gradients of all learnable blobs in the training Net.

Definition at line 340 of file SGDSolver.cs.

◆ ComputeUpdateValue()

virtual void MyCaffe.solvers.SGDSolver< T >.ComputeUpdateValue	(	int	param_id,
		double	dfRate,
		int	nIterationOverride = `-1`
	)

virtual

Compute the SGD update value that will be applied to a learnable blobs in the training Net.

Parameters

param_id	Specifies the id of the Blob.
dfRate	Specifies the learning rate.
nIterationOverride	Optionally, specifies an iteration override, or -1 which is ignored.

Reimplemented in MyCaffe.solvers.AdaDeltaSolver< T >, MyCaffe.solvers.AdaGradSolver< T >, MyCaffe.solvers.AdamSolver< T >, MyCaffe.solvers.AdamWSolver< T >, MyCaffe.solvers.NesterovSolver< T >, and MyCaffe.solvers.RmsPropSolver< T >.

Definition at line 321 of file SGDSolver.cs.

◆ dispose()

override void MyCaffe.solvers.SGDSolver< T >.dispose ( )

protectedvirtual

Releases all resources (GPU and Host) used by the Solver.

Reimplemented from MyCaffe.solvers.Solver< T >.

Definition at line 64 of file SGDSolver.cs.

◆ GetLearningRate()

double MyCaffe.solvers.SGDSolver< T >.GetLearningRate ( int nIterationOverride = -1 )

Return the current learning rate.

The currently implemented learning rate policies are as follows:

fixed: always return .
step: return $base_lr * gamma ^ {floor{iter / step}}$
exp: return
inv: return $base_lr * {1 + gamma * iter} ^ {-power}$
multistep: similar to step but it allows non-uniform steps defined by stepvalue.
poly: the effective learning rate follows a polynomial decay, to be zero by the max_iter. return $base_lr * {1 - iter/max_iter} ^ {power}$
sigmoid: the effective learning rate follows a sigmoid decay. return $base_lr * {1/{1 + exp{-gamma * {iter - stepsize}}}}$

where base_lr, max_iter, gamma, step, stepvalue and power are defined int the solver protocol buffer, and iter is the current iteration.

Parameters

nIterationOverride Optionally, specifies an iteration override, or -1 which is ignored.

Returns: The learning rate value.

Definition at line 129 of file SGDSolver.cs.

◆ Normalize()

virtual void MyCaffe.solvers.SGDSolver< T >.Normalize ( int param_id )

virtual

Normalize a learnable Blob of the training Net.

Parameters

param_id Specifies the id of the Blob.

Definition at line 268 of file SGDSolver.cs.

◆ PreSolve()

void MyCaffe.solvers.SGDSolver< T >.PreSolve ( )

Runs the pre-solve which prepares the Solver to start Solving.

Definition at line 92 of file SGDSolver.cs.

◆ Regularize()

virtual void MyCaffe.solvers.SGDSolver< T >.Regularize ( int param_id )

virtual

Regularize a learnable Blob of the training net.

Parameters

param_id Specifies the id of the Blob.

Definition at line 287 of file SGDSolver.cs.

◆ RestoreSolverState()

override void MyCaffe.solvers.SGDSolver< T >.RestoreSolverState ( byte[] rgState )

protectedvirtual

Restore the state of the Solver.

Parameters

rgState Specifies the state of the Solver.

Implements MyCaffe.solvers.Solver< T >.

Definition at line 230 of file SGDSolver.cs.

◆ SnapshotSolverState()

override byte[] MyCaffe.solvers.SGDSolver< T >.SnapshotSolverState ( )

protectedvirtual

Take a snapshot of the Solver state.

Returns: The Solver state snapshot is returned.

Implements MyCaffe.solvers.Solver< T >.

Definition at line 250 of file SGDSolver.cs.

Member Data Documentation

◆ m_colHistory

BlobCollection<T> MyCaffe.solvers.SGDSolver< T >.m_colHistory = new BlobCollection<T>()

protected

History maintains the historical momentum data.

Definition at line 26 of file SGDSolver.cs.

◆ m_colTemp

BlobCollection<T> MyCaffe.solvers.SGDSolver< T >.m_colTemp = new BlobCollection<T>()

protected

Update maintains update related data and is not needed in snapshots.

Temp maintains other information that might be needed in computation of gradients/updates and is not needed in snapshots.

Definition at line 37 of file SGDSolver.cs.

Property Documentation

◆ history

BlobCollection<T> MyCaffe.solvers.SGDSolver< T >.history

get

Returns the history BlobCollection containing historical momentum data.

Definition at line 84 of file SGDSolver.cs.

The documentation for this class was generated from the following file:

C:/Data/Data/SS_Projects/Intelligence/GitHub/MyCaffe/MyCaffe/solvers/SGDSolver.cs

Public Member Functions

Protected Member Functions

Protected Attributes

Properties

Additional Inherited Members

Detailed Description

Constructor & Destructor Documentation

◆ SGDSolver()

Member Function Documentation

◆ ApplyUpdate()

◆ ClipGradients()

◆ ComputeUpdateValue()

◆ dispose()

◆ GetLearningRate()

◆ Normalize()

◆ PreSolve()

◆ Regularize()

◆ RestoreSolverState()

◆ SnapshotSolverState()

Member Data Documentation

◆ m_colHistory

◆ m_colTemp

Property Documentation

◆ history